When Blue Apron had issues running its data warehouse on another cloud provider, it built an analytics platform using Looker and Google BigQuery to enable faster business decisions about food inventory.

Blue Apron’s mission is to make incredible home cooking accessible to everyone. Launched in 2012, Blue Apron is reimagining the way that food is produced, distributed, and consumed.

Industries: Retail & Consumer Goods

Location: United States

Google Cloud Results

• Enables near real-time business decisions to better manage food inventory and delivery by reducing query times exponentially
• Helps improve customer service and optimize business processes with faster insights
• Reduces costs while reclaiming up to a week per month of engineering

The popularity of meal kit delivery services has surged in recent years as consumer attitudes toward home cooking and grocery shopping have shifted. As a pioneer in the category, Blue Apron helps its customers create incredible home cooking experiences by sending culinary-driven recipes with high-quality ingredients and step-by-step instructions straight to customers’ doors. Blue Apron also offers a monthly wine subscription service and a la carte culinary tools and products through its marketplace.

If that sounds simple, it isn’t. Ingredients for the meal kits must be sourced at the right time, quality, and price. Orders must be packed efficiently and in exactly the right proportions. Most importantly, meal kits must be delivered to the customer fresh and on time.

To meet these criteria and make data meaningful and intuitive to its managers, one of the tools Blue Apron relies on is Looker, an analytics platform that lets business users explore data and ask sophisticated questions using familiar terms.

A Google Cloud partner and winner of the 2016 Google Cloud Global Partner Award for Solution Innovation, Looker integrates its solution with Google Cloud Platform to help customers modernize their analytics.

Blue Apron previously used Looker with a single database instance hosted on another cloud provider. As data volumes grew and queries became more complex, it became difficult to scale. Blue Apron’s only options were choosing ever-larger server classes and increasing storage throughput by purchasing a higher number of provisioned IOPS. To improve speed, scalability, and cost efficiency, Blue Apron moved its data warehouse to Google BigQuery.

“The combination of Looker and Google BigQuery is powerful, allowing us to get data-hungry analysts essential information much faster,” says Sam Chase, Tech Lead, Data Operations at Blue Apron. “Because we choose to pay by the query, it’s also flexible and cost effective—plus storage is cheap, so we can just put data in and query what we need.”

The analytics platform of the future

When you’re making business decisions about a customer’s dinner, speed matters. Looker takes full advantage of the power of Google BigQuery, making it easy to build a data exploration platform. Blue Apron’s applications publish event data to Kafka—approximately 140 million events per day—and data is then streamed into Google BigQuery, which performs lightning-fast queries on both streamed and static data. Now, business users and analytics teams can make decisions based on near real-time information in Looker, instead of waiting until the next business day for results.

“After we moved to Google BigQuery, query time was reduced exponentially. It’s an astonishing difference, allowing us to run 300 queries per day,” says Sam.

Previously, Blue Apron spent up to a week out of every month optimizing its data warehouse to attempt to improve query performance. With Google BigQuery, all maintenance is handled by Google, reclaiming 25% of up to two engineers’ time. Even when multiple people are using Looker concurrently, query performance never degrades and storage never runs out.

“Because Google BigQuery is architected as a giant, shared cluster, growth is smooth,” says Lloyd Tabb, Founder and CTO of Looker. “Like a race car going from 0 to 120 mph, there are no shift points, just smooth acceleration. To us, it looks like the future.”

An empowering, integrated toolset

Looker takes advantage of aggressive caching and support for date-based table partitioning in Google BigQuery to increase performance, simplify the load process, and improve data manageability. By partitioning data by time, Blue Apron can also take advantage of better long-term storage pricing without sacrificing query performance. When using Google BigQuery with Looker, analysts can easily see how much data is going to be scanned before each query is run.

Blue Apron is also using Looker for Google BigQuery Data Transfer Service to provide actionable analytics for all of the company’s Google marketing data from Google AdWords and DoubleClick by Google in one place to understand campaign performance across channels, saving its data operations team months of work. Using Looker Blocks, marketers can quickly make sense of the data with reports and dashboards, and set alerts when campaign performance hits certain thresholds.

Looker Blocks for Google AdWords and DoubleClick by Google provide all the analysis you’d get straight from the Google console, plus additional value-add analysis that’s impossible to replicate without SQL. Complex metrics such as ROI on ad spend, flexible multi-touch attribution, and predictive lifetime value empower marketers with a better understanding of their customers and where to spend their next dollar.

In addition to these turnkey dashboards and pieces of analysis, marketers can customize views to meet their unique needs and workflows. These capabilities help the Blue Apron marketing team make decisions regarding the allocation of spend to maximize customer acquisition and retention.

“Everyone at Blue Apron is excited about using Google BigQuery with Looker,” says Sam. “Business users and marketers are more empowered to look for answers, instead of waiting for analytics teams. Because users know they can get results rapidly, our business processes are evolving and improving.”

For data cleansing and transformation, Blue Apron uses Google Cloud Dataproc to run fully managed Apache Spark clusters on Google Cloud Platform. It’s also leveraging Google BigQuery integration with Google Workspace to bring data into Google Sheets for further distribution and analysis.

“Transferring data between Google tools is fast because it all happens on the Google network,” says Sam. “We can pull data from Google BigQuery, run transformations with Spark, and then write it back to Google BigQuery. That’s very helpful in providing our business users and data analysts with the richest, most current data.”

A perfect match for better insights

As Blue Apron seeks to expand its reach and deepen its engagement with customers, it is making Google BigQuery and Looker available to more users, providing a high-quality interactive analytics experience. “Our ability to pull a lot of data in and compute fast results affects everyone in our company,” says Sam. “Using Google BigQuery and Looker to iterate quickly and build new models to make our operations more efficient will directly impact our customers.”

For Looker, Google BigQuery represents the next step in data warehouse evolution. “Google BigQuery is a perfect match for Looker, combining easy setup with near infinite scale-out and elasticity,” says Lloyd. “People can make smarter decisions faster that directly benefit their business and customers.”

May was a very busy month for data analytics product innovation. If you didn’t have the chance to attend our inaugural Data Cloud Summit, video replays of all our sessions are now available so feel free to watch them at your own pace. 

In this blog, I’d like to share some background behind the innovations we released in May, why we built them the way we did, and the type of value they can bring your company and your team.

But first, a huge thank you!

This week, we had the honor to announce that Google has been named a Leader in The Forrester Wave™: Streaming Analytics, Q2 2021 report. Forrester gave Dataflow a score of 5 out of 5 across 12 different criteria, stating: “Google Cloud Dataflow has strengths in data sequencing, advanced analytics, performance, and high-availability”. 

Google has more than a decade of experience in building real-time and internet-scale systems for its own needs, and we are excited to see that our ability to provide customers with a reliable, scalable, and performant platform is bearing fruit. 

This announcement comes on the back of the release of The Forrester Wave™: Cloud Data Warehouse, Q1 2021 report, which also named Google Cloud as a Leader.

We couldn’t be more excited about the recognition and appreciate all your feedback and trust in the work that we do to support your goal in accelerating data-powered innovation.

Innovation galore

Your feedback and your passion is the fuel that drives our ambition to deliver more and better services to you. That’s why, this year, we didn’t want to wait until Google Cloud Next to share some great products we have been working on. On May 26, our team announced a slew of new products, services and programs. Watch a quick summary below:

https://youtube.com/watch?v=DG1mOPMXJvw%3Fenablejsapi%3D1%26

Meeting you where you are

An important design principle behind all of our services is “meeting you where you are”. This means we aim to provide you with the tools and software you need to innovate on your own terms. Here are three new services that will help you do just that:

Datastream

Datastream, our new serverless change data capture (CDC) and replication service, allows your company to synchronize data across heterogeneous databases, storage systems, and applications reliably and with minimal latency to support real-time analytics, database replication, and event-driven architectures. Datastream delivers change streams from Oracle and MySQL databases into Google Cloud services such as BigQuery, Cloud SQL, Cloud Storage, and Cloud Spanner, saving time and resources while ensuring your data is accurate and up-to-date. 

• Under the hood, Datastream reads CDC events (inserts, updates, and deletes) from source databases, and writes those events with minimal latency to a data destination. It leverages the fact that each database source has its own CDC log—binlog for MySQL and LogMiner for Oracle—which it uses for its own internal replication and consistency purposes. 
• Datastream integrates with purpose-built and extensible Dataflow templates to pull the change streams written to Cloud Storage, and create up-to-date replicated tables in BigQuery for analytics. It also leverages Dataflow templates to replicate and synchronize databases into Cloud SQL or Cloud Spanner for database migrations and hybrid cloud configurations. 
• Datastream also powers a Google-native Oracle connector in Cloud Data Fusion’s new replication feature for easy ETL/ELT pipelining. By delivering change streams directly into Cloud Storage, customers can leverage Datastream to implement modern, event-driven architectures.

Looker and BigQuery Omni on Microsoft Azure

Research on multi cloud adoption is unequivocal — 92% of businesses in 2021 report having a multi cloud strategy. We want to continue supporting your choice by providing the flexibility you need to see your strategy through. 

• This past month, we introduced Looker, hosted on Microsoft Azure. For the first time, you can now choose Azure, Google Cloud, or AWS for your Looker instance. You can also self-host your Looker instance on-premises.
• We also introduced BigQuery Omni for Azure, which along with last year’s introduction of BigQuery Omni for AWS, will help you access and securely analyze data across Google Cloud, AWS, and Azure. 

The cost of moving data between cloud providers isn’t sustainable for many, and it’s still difficult to seamlessly work across clouds. BigQuery Omni represents a new way of analyzing data stored in multiple public clouds, which is made possible by BigQuery’s separation of compute and storage. By decoupling these two, BigQuery provides scalable storage that can reside in Google Cloud or other public clouds, and stateless resilient compute that executes standard SQL queries. 

• Unlike competitors, BigQuery Omni doesn’t require you to move or copy your data from one public cloud to another, where you might incur egress costs. You also benefit from the same BigQuery interface on Google Cloud, enabling you to query data stored in Google Cloud, AWS, and Azure without any cross-cloud movement or copies of data. 
• BigQuery Omni’s query engine runs the necessary compute on clusters in the same region where your data resides. For example, you can query Google Analytics 360 Ads data stored in Google Cloud and query logs data from your ecommerce platform and applications that are stored in AWS S3 and/or Microsoft Azure. 

Then, using Looker, you can build a dashboard that allows you to visualize your audience behavior and purchases alongside your advertising spend. 

Dataplex

We understand that most organizations still struggle to make high-quality data easily discoverable and accessible for analytics, across multiple silos, to a growing number of people and tools within their organization. 

They are often forced to make tradeoffs. For instance, moving and duplicating data across silos to enable diverse analytics use cases or leaving their data distributed but limiting the agility of decisions. 

• Dataplex provides an intelligent data fabric that enables you to centrally manage, monitor, and govern your data across data lakes, data warehouses, and data marts, while also ensuring data is securely accessible to a variety of analytics and data science tools. 
• One of the core tenets of Dataplex is letting you organize and manage your data in a way that makes sense for your business, without data movement or duplication. For that, we provide logical constructs like lakes, data zones, and assets. These constructs enable you to abstract away the underlying storage systems and become the foundation for setting policies around data access, security, lifecycle management, and so on. 
• For example, you can create a lake per department within your organization (e.g. Retail, Sales, Finance, etc.) and create data zones that map to data readiness and usage (e.g. landing, raw, curated_data_analytics, curated_data_science, etc.). 

Once you have your lakes and zones setup, you can attach data to these zones as assets. You can add data from different types of storage (e.g. GCS Bucket and BigQuery dataset) under the same zone. You can also attach data across multiple projects under the same zone. You can ingest data into your lakes and zones using the tools of your choice, including services such as Dataflow, Data Fusion, Dataproc, Pub/Sub, or choose from one of our partner products. Dataplex comes with built-in 1-click templates for common data management tasks. 
To find out more about Dataplex, head to cloud.google.com/dataplex or watch the video below:

https://youtube.com/watch?v=bbFeAt7cw1g%3Fenablejsapi%3D1%26

Helping you innovate everyday

Sharing data is hard. Traditional data sharing techniques use batch data pipelines that are expensive to run, create late arriving data, and can break with any changes to the source data. These techniques also create multiple copies of data, which brings unnecessary costs and can bypass data governance processes. They also fail to offer features for data monetization, such as managing subscriptions and entitlements. Altogether, these challenges mean that organizations are unable to realize the full potential of transforming their business with shared data.

Analytics Hub

To address these limitations, we are introducing Analytics Hub, a new fully managed service that helps organizations unlock the value of data sharing, leading to new insights and increased business value. 

This new service is built on the tremendous experience and feedback we have received over the years. For example, BigQuery has had cross-organizational, in-place data sharing capabilities since its inception in 2010—and the functionality is very popular. Over a 7-day period in April, we had over 3,000 different organizations sharing over 200 petabytes of data. These numbers don’t include data sharing between departments within the same organization.

Analytics Hub takes sharing to the next level, making it easy for you to publish, discover, and subscribe to valuable datasets that you can combine with your own data to derive unique insights. 

This includes: 

• Shared datasets: As a data publisher, you create shared datasets that contain the views of data that you want to deliver to your subscribers. Data subscribers can search through the datasets that are available across all exchanges for which they have access and subscribe to relevant datasets. In addition, the publisher can track subscribers, disable subscriptions, and see aggregated usage information for the shared data.
• Curated, self-service data exchanges: Exchanges are collections used to organize and secure shared datasets. By default, exchanges are completely private, but granular roles and permissions make it easy to deliver data to the right audience—whether internal or public. 

This is just the beginning for Analytics Hub. Please sign up for the preview, which is scheduled to be available in the third quarter of 2021.

Dataflow Prime

At Google Cloud, we have the great privilege of working with some of the most innovative organizations in the world. And this work provides us with a unique perspective into the future of big data processing. Dataflow Prime is a new platform based on a serverless, no-ops, and auto-tuning architecture that brings unparalleled resource utilization and radical operational simplicity to big data processing. This new service introduces a large number of exciting capabilities but I’d like to highlight three key aspects of the product:

• Vertical Autoscaling: Dataflow Prime dynamically adjusts the compute capacity allocated to each worker based on utilization, detecting when jobs are limited by worker resources and automatically adding more resources. Vertical Autoscaling works hand in hand with Horizontal Autoscaling to seamlessly scale workers to best fit the needs of the pipeline. As a result, it no longer takes hours or days to determine the perfect worker configuration to maximize utilization. 
• Right Fitting: Each stage of a pipeline typically has a different resource requirement than the others. Until now, either all workers in the pipeline would have had the higher memory and GPU, or none of them would. Pipelines either had to waste resources or suffer slower workloads. Right Fitting solves this problem by creating stage-specific pools of resources, optimized for each stage. 
• Smart Recommendations: Smart Recommendations automatically detects problems in your pipeline and shows potential fixes. For example, if your pipeline is running into permissions issues, a Smart Recommendation will detect which IAM permissions you need to enable to unblock your job. If you are using an inefficient coder in your job, Smart Recommendations will surface more performant coder implementations that can help you save on costs.

What’s next

We’re excited to hear your thoughts and feedback about all these exciting new services. I would also highly recommend that you connect with members of the community to learn more about their story and journey. A good example to start with is the Data To Value customer panel we produced at our inaugural Data Cloud Summit with the Chief Data Officers of Keybank and Rackspace. You can watch it for free below:

https://youtube.com/watch?v=ITI2Q3MkxuA%3Fenablejsapi%3D1%26

Hundreds of millions of residents of South East Asia have only limited access to banking and finance services. However, help is at hand. One of South East Asia’s largest fintech businesses, Ascend Money is using digital technologies to realize its mission of enabling as many people as possible to access innovative financial services and live better lives.

According to Ascend Money, 60 percent of the region’s 620 million residents do not have access to a full suite of financial services. Even a relatively small proportion of this market represents a big opportunity for the business.

Ascend Money’s offerings include the TrueMoney regional payment platform for underserved and digital consumers. The TrueMoney platform supports more than 40 million consumers across six countries: Cambodia, Indonesia, Myanmar, the Philippines, Thailand, and Vietnam.

“We found, based on value, ease of use, effectiveness, and the skills and adaptability of our own team members, BigQuery and other Google Cloud Platform services were the best fit for our business.”

– Abraham Jarrett, Head of Engineering, Ascend Money

TrueMoney also provides an e-wallet app that provides easy ways to top up mobile phones, undertake online shopping, and pay for products and services; a network of 65,000 branded shops; and international remittances, initially between Thailand and Myanmar. In addition, Ascend Money offers financial products to small to medium businesses.

Ascend Money is part of the South East Asian online business Ascend Group, which also operates e-commerce, e-procurement, data centers, cloud services, fulfilment, and digital marketing services.

Cost is a key criteria

Ascend Money initially started operations using physical infrastructure and on-premises workforce productivity applications. However, as the business grew its customer base and expanded into new markets, its technology leaders began to explore options to improve value for money; reduce the maintenance load on in-house team members; improve its data management and analysis; and collaborate more effectively.

“Operating our own on-premises infrastructure entails a higher cost of ownership and maintenance, as well as requiring us to scale up our hardware as needed,” says Abraham Jarrett, Head of Engineering, Ascend Money. “We conducted an evaluation and found that moving to the cloud would deliver a range of benefits.”

G Suite and Google Cloud Platform the best fit

The business evaluated solutions available in the market and opted to move to G Suite and Google Cloud Platform.

“Moving to Google Cloud Platform was an optimization exercise, with lowering our costs our primary goal, and achieving better performance an added benefit,” says Jarrett. Google Cloud Platform monitoring, diagnostic, and analytics tools have enabled Ascend Money to reduce its infrastructure spending, becoming more efficient and cost-effective in the process.

“Managing software is a big cost for us. Through G Suite, we are significantly reducing our software deployment, licensing, and repair costs.”

– Abraham Jarrett, Head of Engineering, Ascend Money

The Ascend Money team saw Google Kubernetes Engine as enabling a seamless way of transitioning from an on-premises data center to a cloud service. “We considered industry trends such as what people were adopting and who we could hire when making our decision,” says Jarrett. “Engineering teams are focusing on using Kubernetes open source container management at scale and as a way of doing business, which was attractive to us.

“Google Kubernetes Engine presented the easiest way to manage and orchestrate our containers, and drive us away from our existing infrastructure.”

BigQuery best for cost and ease of use

Ascend Money also reviewed data infrastructure solutions, with its business intelligence and data platform teams considering a range of options. The teams quickly ruled out an on-premises solution due to the capital investment required and opted for a cloud service. Following a rigorous evaluation of Google Cloud Platform against the services offered by another cloud provider, Ascend Money opted to run on Google Cloud.

“We found, based on value, ease of use, effectiveness, and the skills and adaptability of our own team members, BigQuery and other Google Cloud Platform services were the best fit for our business,” says Jarrett. The business is now running an architecture comprising a BigQuery analytics data warehouse; Cloud Storage to store raw and archive data; Cloud Dataflow to process stream and batch data, and Cloud Pub/Sub for event ingestion and delivery.

“We’re expanding our utilization of BigQuery and other services on a daily basis,” says Jarrett.

Ascend Money is also using a range of Google Cloud Platform services for the infrastructure outside its data platform, including Stackdriver logging and monitoring; Cloud KMS to manage cryptographic keys for its cloud services, Cloud Build to undertake continuous integration, delivery, and deployment; Cloud DNS to provide domain name system services; and Cloud Functions to enable its developers to run and scale code in the cloud.

With Google Cloud Platform, Ascend Money is now processing about 12GB of batch data per day and streaming data from about 200 data marts per day in Thailand alone.

Cost effective scalability and faster to market

Google Kubernetes Engine has enabled the business to scale and deliver to market faster. “We can build on the platform, take the application and container and deploy them to scale out,” says Jarrett. “The Google Kubernetes Engine orchestration mechanism that provides containerized, elastic scalability is extremely important in allowing us to manage costs and expend our effort efficiently. Through Kubernetes, we can write once and deploy everywhere.”

With Google Cloud Platform, the business has saved 3,000,000 THB (about US$90,000) in licensing costs and, through automation tools, reduced the time to complete infrastructure activities by 50 percent.

“We don’t have as many meetings since we deployed G Suite and we can work effectively in a distributed fashion.”

– Abraham Jarrett, Head of Engineering, Ascend Money

Ascend Money’s entire workforce is now using G Suite to collaborate and operate productively. The business is achieving a range of benefits including being able to get new team members up and running more quickly and reduced cost. “Managing software is a big cost for us,” says Jarrett. “Through G Suite, we are significantly reducing our software deployment, licensing, and repair costs.” Meanwhile, the shorter time to onboard team members to G Suite is paying off with improved productivity and an accelerated ability to collaborate.

Ascend Money team members primarily use Sheets, Slides, and Docs to create internal materials, including product requirement documents in Docs and workforce and project planning spreadsheets in Sheets.

The organization also uses Hangouts Meet to collaborate when working from different locations. “We don’t have as many meetings since we deployed G Suite and we can work effectively in a distributed fashion,” says Jarrett. “People can work from home and we have five regions plus Thailand where they can collaborate live on a document, such as a slide deck for board meetings or meetings with our chief executive officer, head of engineering, or other senior managers.”

20 hours per week saved

Jarrett describes the time savings of using web browser-based productivity software and G Suite as saving them an average of 20 hours per week. This has improved productivity and the quality of life for Ascend Money’s hard-working team, increasing overall staff satisfaction.

“The ability to work remotely is a big win for us,” Jarrett says. “Our team members can keep one computer at work and one at home and access the same information, they can work in a plane while it sits on the tarmac, or update a Google Doc or a Google Sheet in real time on their phone. They can work anytime, from any location, as long as they have an internet connection. They have less dead time, meaning more uptime.”

With Google Cloud Platform and G Suite, Ascend Money is ideally positioned to support growing demand for its fintech services in South East Asia, particularly among people with limited access to banking. “We have the agility and dynamism to support rising demand and look forward to continuing to work with Google to realize our business ambitions,” says Jarrett.