When you build a data warehouse, the important question is how to ingest data from the source system to the data warehouse. If the table is small you can fully reload a table on a regular basis, however, if the table is large a common technique is to perform incremental table updates. This post demonstrates how you can enhance incremental pipeline performance when you ingest data into BigQuery.

Setting up a standard incremental data ingestion pipeline

We will use the below example to illustrate a common ingestion pipeline that incrementally updates a data warehouse table. Let’s say that you ingest data into BigQuery from a large and frequently updated table in the source system, and you have Staging and Reporting areas (datasets) in BigQuery.

The Reporting area in BigQuery stores the most recent, full data that has been ingested from the source system tables. Usually you create the base table as a full snapshot of the source system table. In our running example, we use BigQuery public data as the source system and create reporting.base_table as shown below. In our example each row is identified by a unique key which consists of two columns: block_hash and log_index.

  CREATE TABLE reporting.base_table  --156 GB processed
PARTITION BY TIMESTAMP_TRUNC(block_timestamp, DAY) AS
SELECT log_index, data, topics, block_timestamp, block_hash
FROM bigquery-public-data.crypto_ethereum.logs
WHERE block_timestamp BETWEEN TIMESTAMP '2020-01-01' AND TIMESTAMP '2020-11-30';

In data warehouses it is common to partition a large base table by a datetime column that has a business meaning. For example, it may be a transaction timestamp, or datetime when some business event happened, etc. The idea is that data analysts who use the data warehouse usually need to analyze only some range of dates and rarely need the full data. In our example, we partition the base table by block_timestamp which comes from the source system.

After ingesting the initial snapshot you need to capture changes that happen in the source system table and update the reporting base table accordingly. This is when the Staging area comes into the picture. The staging table will contain captured data changes that you will merge into the base table. Let’s say that in our source system on a regular basis we have a set of new rows and also some updated records. In our example we mock the staging data as follows: first, we create new data, than we mock the updated records:

  CREATE TABLE staging.load_delta AS --5 GB processed
SELECT log_index, data, topics, block_timestamp, block_hash
FROM bigquery-public-data.crypto_ethereum.logs
WHERE block_timestamp BETWEEN TIMESTAMP '2020-12-01' AND TIMESTAMP '2020-12-07';
 
INSERT INTO staging.load_delta --2 GB processed
SELECT log_index, CONCAT(data, RAND()), topics, block_timestamp, block_hash
FROM bigquery-public-data.crypto_ethereum.logs TABLESAMPLE SYSTEM (5 PERCENT)
WHERE block_timestamp BETWEEN TIMESTAMP '2020-10-01' AND TIMESTAMP '2020-11-30';

Next, the pipeline merges the staging data into the base table. It joins two tables by unique key and than updates the changed value or inserts a new row

  MERGE INTO reporting.base_table T --161 GB processed
USING staging.load_delta S
ON T.block_hash = S.block_hash
 AND T.log_index = S.log_index
WHEN MATCHED THEN UPDATE SET 
  T.data = S.data
WHEN NOT MATCHED THEN INSERT (log_index, data, topics, block_timestamp, block_hash)
VALUES (log_index, data, topics, block_timestamp, block_hash);

It is often the case that the staging table contains keys from various partitions but the number of those partitions are relatively small. It holds, for instance, because in the source system the recently added data may get changed due to some initial errors or ongoing processes but older records are rarely updated. However, when the above MERGE gets executed, BigQuery scans all partitions in the base table and processes 161 GB of data. You might add additional join condition on block_timestamp:

  MERGE INTO reporting.base_table T --161 GB processed
USING staging.load_delta S
ON T.block_hash = S.block_hash
 AND T.log_index = S.log_index
 AND T.block_timestamp = S.block_timestamp
WHEN MATCHED THEN UPDATE SET 
  T.data = S.data
WHEN NOT MATCHED THEN INSERT (log_index, data, topics, block_timestamp, block_hash)
VALUES (log_index, data, topics, block_timestamp, block_hash);

But BigQuery would still scan all partitions in the base table because condition T.block_timestamp = S.block_timestamp is a dynamic predicate and BigQuery doesn’t automatically push such predicates down from one table to another in MERGE.

Can you improve the MERGE efficiency by making it scan less data? The answer is Yes. 

As described in the MERGE documentation, pruning conditions may be located in a subquery filter, a merge_condition filter, or a search_condition filter. In this post we show how you can leverage the first two. The main idea is to turn a dynamic predicate into a static predicate.

Steps to enhance your ingestion pipeline

The initial step is to compute the range of partitions that will be updated during the MERGE and store it in a variable. As was mentioned above, in data ingestion pipelines, staging tables are usually small so the cost of the computation is relatively low.

  DECLARE src_range STRUCT<date_min TIMESTAMP, date_max TIMESTAMP> --115 MB processed
DEFAULT(SELECT STRUCT(
  MIN(block_timestamp) AS date_min,  
  MAX(block_timestamp) AS date_max) FROM staging.load_delta);

Based on your existing ETL/ELT pipeline, you can add the above code as-is to your pipeline or you can compute date_min, data_max as part of some already existing transformation step. Alternatively, date_min, data_max can be computed on the Source System side while capturing the next ingestion data batch.

After computing date_min, date_max you pass those values to the MERGE statement as static predicates. There are several ways to enhance the MERGE and prune partitions in the base table based on precomputed date_min, data_max. 

If your initial MERGE statement uses a subquery, you can incorporate a new filter into it:

  BEGIN 
DECLARE src_range STRUCT<date_min TIMESTAMP, date_max TIMESTAMP> --115 MB processed
DEFAULT(SELECT STRUCT(
  MIN(block_timestamp) AS date_min,  
  MAX(block_timestamp) AS date_max) FROM staging.load_delta);

MERGE INTO reporting.base_table T --41 GB processed
USING (
  SELECT *
  FROM staging.load_delta
  WHERE block_timestamp BETWEEN src_range.date_min AND src_range.date_max) S 
ON T.block_hash = S.block_hash
 AND T.log_index = S.log_index
 AND T.block_timestamp = S.block_timestamp
WHEN MATCHED THEN UPDATE SET 
  T.data = S.data
WHEN NOT MATCHED THEN INSERT (log_index, data, topics, block_timestamp, block_hash)
VALUES (log_index, data, topics, block_timestamp, block_hash);
END;

Note that you add the static filter to the staging table and keep T.block_timestamp = S.block_timestamp to convey to BigQuery that it can push that filter to the base table. This MERGE processes 41 GB of data in contrast to the initial 161 GB. You can see in the query plan that BigQuery pushes the partition filter from the staging table to the base table:

This type of optimization, when a pruning condition is pushed from a subquery to a large partitioned or clustered table, is not unique for MERGE. It also works for other types of queries. For instance:

  SELECT * -- 41 GB processed
FROM reporting.base_table T
INNER JOIN staging.load_delta S
ON T.block_hash = S.block_hash
 AND T.log_index = S.log_index
 AND T.block_timestamp = S.block_timestamp
WHERE S.block_timestamp BETWEEN TIMESTAMP '2020-10-05' AND TIMESTAMP '2020-12-07'

And you can check the query plan to verify that BigQuery pushed down the partition filter from one table to another.

Moreover, for SELECT statements, BigQuery can automatically infer a filter predicate on a join column and push it down from one table to another if your query meets the following criteria:

• The target table must be clustered or partitioned. 
• The result size of the other table, i.e. after applying all filters, must qualify for broadcast join. Namly, the result set must be relatively small, less than ~100MB.

In our running example, reporting.base_table is partitioned by block_timestamp. If you define a selective filter on staging.load_delta and join two tables, you can see an inferred filter on the join key pushed to the target table

  SELECT * 
FROM reporting.base_table T
INNER JOIN staging.load_delta S
ON T.block_timestamp = S.block_timestamp
WHERE S.block_hash = '0x0c1caa16b34d94843aabfebc0d5a961db358135988f7498a6fdc450ad55f0870'

There is no requirement to join tables by partitioning or clustering key to kick off this type of optimization. However, in this case the pruning effect on the target table would be less significant.

But let us get back to the pipeline optimizations. Another way to enhance MERGE is to modify the merge_condition filter by adding static predicate on the base table:

  BEGIN 
DECLARE src_range STRUCT<date_min TIMESTAMP, date_max TIMESTAMP> --115 MB processed
DEFAULT(SELECT STRUCT(
  MIN(block_timestamp) AS date_min,  
  MAX(block_timestamp) AS date_max) FROM staging.load_delta);

MERGE INTO reporting.base_table T --41 GB processed
USING staging.load_delta S
ON T.block_hash = S.block_hash
 AND T.log_index = S.log_index
 AND T.block_timestamp BETWEEN src_range.date_min AND src_range.date_max
WHEN MATCHED THEN UPDATE SET 
  T.data = S.data
WHEN NOT MATCHED THEN INSERT (log_index, data, topics, block_timestamp, block_hash)
VALUES (log_index, data, topics, block_timestamp, block_hash);
END;

To summarize, here are the steps that you can perform to enhance incremental ingestion pipelines in BigQuery. First you compute the range of updated partitions based on the small staging table. Next, you tweak the MERGE statement a bit to let BigQuery know to prune data in the base table.

All the enhanced MERGE statements scanned 41 GB of data, and setting up the src_range variable took 115 MB.  Compare it with the initial 161 GB scan. Moreover, given that computing src_range may be incorporated into some existing transformation in your ETL/ELT, it results in a good performance improvement which you can leverage in your pipelines. 

In this post we described how to enhance data ingestion pipelines by turning dynamic filter predicates into static predicates and letting BiQuery prune data for us. You can find more tips on BigQuery DML tuning here.

Special thanks to Daniel De Leo, who helped with examples and provided valuable feedback on this content.

With the rise of technologies like smartphones, retailers have felt the pressure to meet evolving consumer needs and expectations. Seven-Eleven Japan(“SEJ”) has long been on the forefront of this thanks to the way they develop and invest in IT. However, in recent years, Japan’s leading convenience store chain has struggled to maintain its complex legacy systems at the rate needed to keep up with today’s rapid digitization, spurred on by the increasing proliferation of smartphones and an IT vendor-dependent structure.  

Legacy systems limiting real-time responsiveness and innovation 

Since its early days, SEJ has been proactive in adopting information technology, mainly relying on technology solutions from Japan’s leading vendors. But as the systems have grown, key business issues have been resolved using a vendor-dependent structure rather than being driven by SEJ’s own needs.  

Datasets and business logic were combined and built into legacy environments, gradually leading to data silos. As a result, data was distributed across multiple systems, causing a variety of problems, including the inability to efficiently retrieve data when needed, delays in accessing data collected in individual stores, and difficulties taking measurements at the right time in business operations that require real-time responsiveness.

Connecting different systems also takes time and money, and the lead time for introducing new services—from planning to development and launch—has been longer than expected. 

To solve these problems, SEJ’s IT department built “Seven Central”—a new platform for practical data use launched in 2020 to support the company’s future IT strategies and digital transformation initiatives.

At its core, Seven Central’s ultimate purpose is to allow real-time data views. Versatile, real-time datasets—such as point-of-sale (POS) data from 7-Eleven stores—are consolidated into a centralized location in the cloud. They created a simple data mart that provides data via an API to enable them to respond more quickly to requests from individual departments. 

“In such uncertain times, it’s vital to use data to make quick decisions,” says Izuru Nishimura, Executive Officer and Head of ICT Department. “Each department across the entire company will be able to gain an immediate understanding of the situation based on the most up-to-date data and respond accordingly. This is why we built Seven Central.”

Google Cloud selected to help SEJ build and grow their data cloud

Today’s rapidly changing business environment has also highlighted the risk of IT support becoming a bottleneck. The long-term strategy is to gradually expand the datasets managed and collected in Seven Central according to business needs. 

In the first phase, SEJ collected POS data from all 21,000+ stores to enable real-time analysis. Moving forward, they would like to collect other relevant data—for example, unstructured data, such as images and videos, or master datasets that are currently stored externally. 

Google Cloud was already a top contender when SEJ started developing Seven Central in 2019. They compared various public cloud services besides Google Cloud, focusing on three main capabilities.

“We placed particular emphasis on service scalability to drive future digital transformation; security when handling data, which is the lifeline of our company; and finally, openness,” says Nishimura. He emphasizes that openness was perhaps the most important factor for choosing Google Cloud. Breaking away from the negative aspects of an entirely vendor-dependent system enabled them to build an agile development system with multiple vendors. 

Google Cloud technologies including BigQuery and API management platform, Apigee, play a vital role in Seven Central. BigQuery’s high-speed processing at petabyte scale and fully managed infrastructure helped keep costs low during development and verification.   

“Data is stored in a way that allows you to share it easily across organizations, which helps solve the issue of data silos from the perspective of scalability. I also like the fact there are some interesting features that could be used in the future—like BigQuery ML, which enables machine learning on BigQuery,” says Nishimura. 

Apigee allows SEJ to separate datasets and business logic, which is one of the key points of Seven Central. While the trend these days is to standardize interfaces using an API, the reality tends to involve many different APIs rather than the introduction of one unified API. With Apigee, SEJ provides a single unified API for all of its data cloud, and they can now understand what data is used thanks to Apigee’s API usage visualizations.  

“Right now, we collect data from all 21,000+ stores,” says Nishimura. “But in anticipation of a future expansion in business operations, we have designed a system that can scale up and run without issue, even if we were to have 30,000 stores, with 1,000 customers per store per day, purchasing five items per person.”

Real-time insights with BigQuery and Cloud Spanner

Google Cloud partner Cloud Ace came on board early in the planning phases. Based on their recommendations, SEJ decided to continue making full use of BigQuery to analyze data collected from all 21,000+ stores throughout Japan, while also using Cloud Spanner’s availability, near-unlimited scalability and transactional consistency to help achieve the real-time results needed for the project.

“Given that both the data and the regularity with which it is accessed are expected to steadily increase in the future, we chose Cloud Spanner as backend storage for data delivery via API. We consider it a good choice,” says Shota Kikuchi, General Manager, Consulting Department, Technology Division, Cloud Ace Co., Ltd.

Finally, they chose to use Google Cloud’s Stream Analytics Solutions messaging service for collecting POS data in real time, which can then be put to immediate use with Cloud Spanner and BigQuery. 

High-speed responses exceed targets and create new value 

Seven Central went live in September 2020 with surprising results. 

They initially set a target time of one hour from when a customer makes a purchase to the point when Seven Central can use that data. But when the final system was first tried—it took barely a minute. Moving forward they estimate that the latest inventory data from the service side will become available within a few minutes of being added to the system.  

“This is real innovation, and I must admit that I am quite surprised. As well as being able to solve existing issues, we also hope it will lead to new improvements and services that have been unimaginable up until now,” says Nishimura.

The team hopes to roll out the Seven Central platform in all companies affiliated with Seven & i Holdings—not just SEJ. They also plan to explore Google Cloud AI and machine learning technologies to take on challenges in new areas. For example, they are investigating the idea of clustering individual stores using BigQuery ML.

Seven Central has already attracted attention from many departments and received a lot of requests. Nishimura and his team say they hope to continue to grow Seven Central while still observing their fundamental principles—not including business logic, maintaining real-time results, and staying true to the uniqueness of SEJ.

Learn more about Google Cloud smart analytics solutions.

If you’re self-managing relational databases such as MySQL, PostgreSQL or SQL Server, you may be thinking about the pros and cons of cloud-based database services. Regardless of whether you’re running your databases on premises or in the cloud, self-managed databases can be inefficient and expensive, requiring significant effort around patching, hardware maintenance, backups, and tuning. Are managed database services a better option?

To answer this question, Google Cloud sponsored a business value white paper by IDC, based on the real-life experiences of eight Cloud SQL customers. Cloud SQL is an easy-to-use, fully-managed database service for running MySQL, PostgreSQL and SQL Server workloads. More than 90% of the top 100 Google Cloud customers use Cloud SQL.

The study found that migration to Cloud SQL unlocked significant efficiencies and cost reductions for these customers. Let’s take a look at the key benefits in this infographic.

A deeper dive into Cloud SQL benefits

To read the full IDC white paper, you can download it here: The Business Value of Cloud SQL: Google Cloud’s Relational Database Service for MySQL, PostgreSQL, and SQL Server, by Carl W. Olofson, Research Vice President, Data Management Software, IDC and Matthew Marden, Research Vice President, Business Value Strategy Practice, IDC.

Looking for commentary from IDC? Listen to the on-demand webinar, How Enterprises Have Achieved Greater Efficiency and Improved Business Performance using Google Cloud SQL, where Carl Olofson discusses the downsides of self-managed databases and the benefits of managed services like Cloud SQL, including the cost savings and improved business performance realized by the customers interviewed in the study.

Getting started

You can use our Database Migration Service for an easy, secure migration to Cloud SQL. Since Cloud SQL supports the same database versions, extensions and configuration flags as your existing MySQL, PostgreSQL and SQL Server instances, a simple lift-and-shift migration is usually all you need. So let Google Cloud take routine database administration tasks off your hands, and enjoy the scalability, reliability and openness that the cloud has to offer.

Start your journey with a Cloud SQL free trial.