A Comprehensive Guide for Understanding and Optimizing Your Dataflow Costs
1178
Of your peers have already read this article.
3:30 Minutes
The most insightful time you'll spend today!
Dataflow is the industry-leading platform that provides unified batch and streaming data processing capabilities, and supports a wide variety of analytics and machine learning use cases. It’s a fully managed service that comes with flexible development options (from Flex Templates and Notebooks to Apache Beam SDKs for Java, Python and Go), and a rich set of built-in management tools. It seamlessly integrates with not just Google Cloud products like Pub/Sub, BigQuery, Vertex AI, GCS, Spanner, and BigTable, but also third-party services like Kafka and AWS S3, to best meet your analytics and machine learning needs.
Dataflow’s adoption has ramped up in the past couple of years, and numerous customers now rely on Dataflow for everything from designing small proof of concepts, to large-scale production deployments. As customers are always trying to optimize their spend and do more with less, we naturally get questions related to cost optimization for Dataflow.
In this post, we’ve put together a comprehensive list of actions you can take to help you understand and optimize your Dataflow costs and business outcomes, based on our real-world experiences and product knowledge. We’ll start by helping you understand your current and near-term costs. We’ll then share how you can continuously evaluate and optimize your Dataflow pipelines over time. Let’s dive in!
Understand your current and near-term costs
The first step in most cost optimization projects is to understand your current state. For Dataflow, this involves the ability to effectively:
- Understand Dataflow’s cost components
- Predict the cost of potential jobs
- Monitor the cost of submitted jobs
Understanding Dataflow’s cost components
Your Dataflow job will have direct and indirect costs. Direct costs reflect resources consumed by your job, while indirect costs are incurred by the broader analytics/machine learning solution that your Dataflow pipeline enables. Examples of indirect costs include usage of different APIs invoked from your pipeline, such as:
- BigQuery Storage Read and Write APIs
- Cloud Storage APIs
- BigQuery queries
- Pub/Sub subscriptions and publishing, and
- Network egress, if any
Direct and indirect costs influence the total cost of your Dataflow pipeline. Therefore, it’s important to develop an intuitive understanding of both components, and use that knowledge to adopt strategies and techniques that help you arrive at a truly optimized architecture and cost for your entire analytics or machine learning solution. For more information about Dataflow pricing, see the Dataflow pricing page.
Predict the cost of potential jobs
You can predict the cost of a potential Dataflow job by initially running the job on a small scale. Once the small job is successfully completed, you can use its results to extrapolate the resource consumption of your production pipeline. Plugging those estimates into the Google Cloud Pricing Calculator should give you the predicted cost of your production pipeline. For more details about predicting the cost of potential jobs, see this (old, but very applicable) blog post on predicting the cost of a Dataflow job.
Monitor the cost of submitted jobs
You can monitor the overall cost of your Dataflow jobs using a few simple tools that Dataflow provides out of the box. Also, as you optimize your existing pipelines, possibly using recommendations from this blog post, you can monitor the impact of your changes on performance, cost, or other aspects of your pipeline that you care about. Handy techniques for monitoring your Dataflow pipelines include:
- Use metrics within the Dataflow UI to monitor key aspects of your pipeline.
- For CPU intensive pipelines, profile the pipeline to gain insight into how CPU resources are being used throughout your pipeline.
- Experience real-time cost control by creating monitoring alerts on Dataflow job metrics which ship out of the box, and that can be good proxies for the cost of your job. These alerts send real-time notifications once the metrics associated with a running pipeline exceeds a predetermined threshold. We recommend that you only do this for your critical pipelines, so you can avoid notification fatigue.
- Enable Billing Export into BiqQuery, and perform deep, ad-hoc analyses of your Dataflow costs that help you understand not just the key drivers of your costs, but also how these drivers are trending over time.
- Create a labeling taxonomy, and add labels to your Dataflow jobs that help facilitate cost attribution during the ad-hoc analyses of your Dataflow cost data using BigQuery. Check out this blog post for some great examples of how to do this.
- Run your Dataflow jobs using a custom Service Account. While this is great from a security perspective, it also has the added benefit of enabling the easy identification of APIs used by your Dataflow job.
Optimize your Dataflow costs
Once you have a good understanding of your Dataflow costs, the next step is to explore opportunities for optimization. Topics to be considered during this phase of your cost optimization journey include:
- Your optimization goals
- Key factors driving the cost of your pipeline
- Considerations for developing optimized batch and streaming pipelines
Let’s look at each of these topics in detail.
Goals
The goal of a cost optimization effort may seem obvious: “reduce my pipeline’s cost.” However, your business may have other priorities that have to be carefully considered and balanced with the cost reduction goal. From our conversations with customers, we have found that most Dataflow cost optimization programs have two main goals:
1. Reduce the pipeline’s cost
2. Continue meeting the service level agreements (SLAs) required by the business
Cost factors
Opportunities for optimizing the cost of your Dataflow pipeline will be based on the key factors driving your costs. While most of these factors will be identified through the process of understanding your current and near term costs, we have identified a set of frequently recurring cost factors, which we have grouped into three buckets: Dataflow configuration, performance, and business requirements.
Dataflow configuration includes factors like:
- Should your pipeline be streaming or batch?
- Are you using Dataflow services like Streaming Engine, or FlexRS?
- Are you using a suitable machine type and disk size?
- Should your pipeline be using GPUs?
- Do you have the right number of initial and maximum workers?
Performance includes factors like:
- Are your SDK versions (Java, Python, Go) up to date?
- Are you using IO connectors efficiently? One of the strengths of Apache Beam is a large library of IO connectors to different storage and queueing systems. Apache Beam IO connectors are already optimized for maximum performance. However, there may be cases where there are trade-offs between cost and performance. For example, BigQueryIO supports several write methods, each of them with somewhat different performance and cost characteristics. For more details, see slides 19 – 22 of the Beam Summit session on cost optimization.
- Are you using efficient coders? Coders affect the size of the data that needs to be saved to disk or transferred to a dedicated shuffle service in the intermediate pipeline stages, and some coders are more efficient than others. Metrics like total shuffle data processed and total streaming data processed can help you identify opportunities for using more efficient coders. As a general rule, you should consider whether the data that appears in your pipeline contains redundant data that can be eliminated by both filtering out unused columns as early as possible, and using efficient coders such as AvroCoder or RowCoder. Also, remember that if stages of your pipeline are fused, the coders in the intermediate steps become irrelevant.
- Do you have sufficient parallelism in your pipeline? The execution details tab, and metrics like processing parallelism keys can help you determine whether your pipeline code is taking full advantage of Apache Beam’s ability to do massively parallel processing (for more details, see parallelism). For example, if you have a transform which outputs a number of records for each input record (“high fan-out transform”) and the pipeline is automatically optimized using Dataflow fusion optimization, the parallelism of the pipeline can be suboptimal, and may benefit from preventing fusion. Another area to watch for is “hot keys.” This blog discusses this topic in great detail.
- Are your custom transforms efficient? Job monitoring techniques like profiling your pipeline and viewing relevant metrics can help you catch and correct your inefficient use of custom transforms. For example, if your Java transform needs to check if the data matches a certain regular expression pattern, then compiling that pattern in the setup method and doing the matching using the precompiled pattern in the “process element” method is much more efficient. The simpler, but inefficient alternative is to use the String.matches() call in the “process element” method, which will have to compile the pattern every time. Another consideration regarding custom transforms is that grouping elements for external service calls can help you prepare optimal request batches for external API calls. Finally, transforms that perform multi-step operations (for example, calls to external APIs that require extensive processes for creating and closing the client for the API) can often benefit from splitting these operations, and invoking them in different methods of the ParDo (for more details, see ParDo life cycle).
- Are you doing excessive logging? Logs are great. They help with debugging, and can significantly improve developer productivity. On the other hand, excessive logging can negatively impact your pipeline’s performance.
Business requirements can also influence your pipeline’s design, and increase costs. Examples of such requirements include:
- Low end-to-end ingest latency
- Extremely high throughput
- Processing late arriving data
- Processing streaming data spikes
Considerations for developing optimized data pipelines
From our work with customers, we have compiled a set of guidelines that you can easily explore and implement to effectively optimize your Dataflow costs. Examples of these guidelines include:
Batch and streaming pipelines:
- Consider keeping your Dataflow job in the same region as your IO source and destination services.
- Consider using GPUs for specialized use cases like deep-learning inference.
- Consider specialized machine types for memory- or compute-intensive workloads.
- Consider setting the maximum number of workers for your Dataflow job.
- Consider using custom containers to pre-install pipeline dependencies, and speed up worker startup time.
- Where necessary, tune the memory of your worker VMs.
- Reduce your number of test and staging pipelines, and remember to stop pipelines that you no longer need.
Batch pipelines:
- Consider FlexRS for use cases that have start time and run time flexibility.
- Run fewer pipelines on larger datasets. Starting and stopping a pipeline incurs some cost, and processing a very small dataset in a batch pipeline can be inefficient.
- Consider defining the maximum duration for your jobs, so you can eliminate runaway jobs, which keep running without delivering value to your business.
Summary
In this post, we introduced you to a set of knowledge and techniques that can help you understand the current and near-term costs associated with your Dataflow pipelines. We also shared a series of considerations that can help you optimize your Dataflow pipelines, not just for today, but also as your business evolves over time. We shared lots of resources with you, and hope that you find them useful. We look forward to hearing about the savings and innovative solutions that you are able to deliver for your customers and your business.
DocAI Lowers Customer’s Document Processing Cost by 60 Percent. Learn How
5608
Of your peers have already read this article.
2:00 Minutes
The most insightful time you'll spend today!
Some of the most important data at your company isn’t living in databases, but in documents, and most business processes begin, involve or end with a document.
Yet most companies are still manually entering data and reliant on guesswork to make sense of it all as the volume and variety of data explodes. Organizations are also leaving heaps of value on the table in the form of new and better customer experiences that can be unlocked with artificial intelligence (AI) applied to documents.
The latest releases of Document (Doc) AI platform, Lending DocAI and Procurement DocAI, built on decades of AI innovation at Google, bring powerful and useful solutions to these challenges. Under the hood are Google’s industry-leading technologies:
- Computer vision (including OCR) and Natural Language Processing (NLP) that creates pre-trained models for high-value, high-volume documents.
- Google Knowledge Graph to validate and enhance the fields in your documents.
- Training and creation of your own custom document models.
- Human interaction with AI to ensure accuracy where needed.
Google Cloud DocAI platform, Lending DocAI and Procurement DocAI are now generally available. Thousands of customers have tried these products in the preview phase—and DocAI has already processed tens of billions of pages of documents across lending, insurance, government and other industries.
“The vast majority of enterprise content still resides in unstructured sources like documents. Google Cloud’s Document AI brings a fresh new perspective to the problem informed by the company’s decades of experience making sense of the largest unstructured corpus in the world—the world wide web.” —Ritu Jyoti, VP of AI Research, IDC
Cut document processing costs by up to 60%
Lending DocAI helps banks, mortgage brokers and other lending institutions fast track the loan application process from weeks to days, dramatically reducing the cost of issuing a loan. And Procurement DocAI enables companies to automate procurement data capture at scale, lowering processing costs by up to 60%.
These solutions are built on DocAI platform, a unified console for document processing that lets you quickly access all parsers and tools. From the platform, you can automate and validate documents to streamline workflows, reduce guesswork, and keep data accurate and compliant.
Get more value from AI with DocAI’s industry-specific solutions
According to Accenture’s AI: Built to Scale report: “Companies that scale successfully see 3x the return on their AI investments compared to those who have not fully rolled out AI capabilities.”
Core to our strategy at Google Cloud is the creation of industry-specific solutions that help companies get maximum value out of their investments in AI. We announced Lending DocAI, our first solution designed specifically for the financial services industry, at the Mortgage Bankers Association convention last year. It processes borrowers’ income and asset documents using a set of specialized machine learning (ML) models, and automates routine document reviews so that mortgage providers can focus on more important work.
Lending DocAI is now generally available and includes more specialized parsers for critical loan documents including paystubs, bank statements, and more. Our goal is to provide the right tools to help borrowers and lenders have a better experience and close home loans faster. For more, watch this video.
Procurement DocAI is also now generally available. This solution helps companies accelerate document processing for invoices, receipts, and other valuable documents in the procurement cycle.
Automating data capture is helping our customers increase accuracy and also lower their procure-to-pay processing costs. We are continually expanding the types of documents Procurement DocAI can process—the latest is a utility parser for electric, water and other bills. In addition, Procurement DocAI leverages Google Knowledge Graph to validate and enrich parsed information to make the data even more useful. Check out this overview video for more details.
One company that lives and breathes AI-enabled document management is AODocs. It uses Procurement DocAI to simplify invoice processing for enterprise customers and launched a new Gmail add-on, Invoice to Sheet, for SMB customers who just want to track their invoices in Google Sheets.
“Google Cloud’s Procurement DocAI service allows our document management platform to better automate the processing of invoices; AODocs customers who have tested our new account payables workflow estimate that the productivity of their A/P team has more than doubled, thanks to the reduction of manual data input brought by the Procurement DocAI.”—Stéphan Donzé, Founder and CEO, AODocs
The new specialized parsers for Lending and Procurement DocAI can be used alongside our existing AutoML Text & Document Classification and AutoML Document Extraction services. These technologies provide a state-of-the-art toolset for creating new document models and have been widely deployed by customers in financial services and other industries.
Partner to accelerate your AI deployment and results
Having the right partner to ease the complexity of rolling out your AI-strategy in mortgage document processing is critical to transforming your customers’ experience. We’re excited to announce a partnership with Mr. Cooper, a leader in mortgage servicing, to provide customers with more automation and workflow tools throughout their entire mortgage life cycle. As part of this agreement, both companies will collaborate on digitizing Mr. Cooper’s core mortgage platform, creating a more personal customer experience utilizing AI, and driving a broader culture of innovation to imagine and develop services and solutions that will transform the mortgage experience for American homeowners.
“Over the last few years, we have made substantial investments in our servicing technology and core mortgage platform that have revolutionized the customer experience, while providing dramatic efficiencies in operating cost. Our partnership with Google Cloud AI will build on those advances and help make these technologies available for the mortgage industry.” —Jay Bray, Chairman and CEO, Mr. Cooper Group
This builds upon the robust partner ecosystem we’re creating to help customers revolutionize the home loan experience, which includes last year’s partnership announcement with Roostify.
Integrate human review into ML predictions
Next up is the general availability of Human-in-the-Loop AI, a new DocAI feature that will help companies achieve higher document processing accuracy with the assurance of human review. Adding human review can increase accuracy and help businesses interpret predictions using purpose-built tools to enable those reviews.
Processing documents quickly and cost-effectively is important. But it’s often necessary to have a high level of assurance on data accuracy for compliance. CIOs and IT decision-makers need highly accurate ML predictions to fulfill compliance requirements, improve employee experience (e.g. less rework), and raise customer satisfaction (e.g. fewer data errors). Including human participation in ML processes allows AI and humans to work together for the best possible results.
Human-in-the-Loop AI provides the workflow to manage human review tasks and produces a percentage confidence score of how “sure” it is that the AI ingested the document correctly. Document AI extracts data from documents with ML, and when paired with Human-in-the-Loop AI, human reviewers are able to verify the data captured. This system is customizable, providing the flexibility to set different thresholds and assign individual groups of reviewers to various stages of the workflow. With Human-in-the-Loop AI, developers can choose trusted reviewers to assign to the task; these reviewers can be from within their own or partner organizations.
More Document AI resources
To learn more, check out the Document AI webpage and watch a demo of how to process sample forms in AI Platform notebooks to inspect data extraction and confidence scores. For more on how customers and partners like Workday, AODocs, and Mr. Cooper are using Document AI, listen to our fireside chat. And stay tuned for the exciting evolution of these technologies in future releases of DocAI.
5185
Of your peers have already downloaded this article
2:30 Minutes
The most insightful time you'll spend today!
For as long as business intelligence (BI) has been around, visualization tools have played an important role in helping analysts and decision-makers quickly get insights from data.
In our current era of big data analytics, that premise still holds. To provide an integrated platform for building BI dashboards on top of big data, GCP offers the combination of Google BigQuery, a cloud-native data warehouse that helps you analyze petabytes of data quickly, and Google Data Studio, a free tool that helps you quickly create beautiful reports.
First, imagine that you work for an online retailer that bases important decisions on customer interaction data in the form of large amounts (multiple TBs) of usage logs stored as date-partitioned tables in a BigQuery dataset called usage. To get business value out of that data as quickly as possible, you want to build a dashboard for analysts that will provide visualizations of trends and patterns in your data.
The good news is that a connector is available that lets you query data stored in BigQuery directly from Data Studio.
Learn how to build a BI dashboard with Data Studio as the front end, and powered by BigQuery on the back end (with some help from Google App Engine for added efficiency). Download this how-to now.
Uniting Data and Analytics in One Place: Apache Iceberg and BigLake
1138
Of your peers have already read this article.
2:30 Minutes
The most insightful time you'll spend today!
When your data is siloed data across lakes and warehouses, it can be hard to transform outcomes with your data. Apache Iceberg is an open table format that provides data management capabilities for data hosted on object stores and enables organizations to run analytics and AI use cases over a single copy of data. A growing community of data engineers, customers, and industry partners are contributing, integrating, and deploying Iceberg, making it the standard for organizations building open-format lakehouses.
To help customers on this journey, we announced support for Iceberg through BigLake in October, 2022. Since its preview, many customers have started building lakehouse workloads using Apache Iceberg as their data management layer, and this support is now generally available.
Unify analytics, streaming and AI use cases over a single copy of data
You can use open-source engines to process and ingest data into Iceberg tables, and BigQuery can query those tables. Since the preview, customers have also used Spark, Trino and Flink to process Iceberg tables and make those tables available to their BigQuery users. Then, BigLake Metastore provides shared metadata for Iceberg tables across BigQuery and open-source engines, eliminating the need to maintain multiple table definitions. Further, you can provide BigQuery datasets and table properties when creating new Iceberg tables in Spark, and those tables become automatically available for the BigQuery user to query.
When implementing Iceberg lakehouse workloads, query performance is a top priority for data warehouse users. BigQuery natively integrates with the Iceberg transaction logs and leverages its rich metadata for efficient query planning. Query plans are designed to reduce BigQuery compute consumption by lowering the amount of scanned data, by optimizing joins, and by improving data-plane parallelism. The net result is that you get better query performance and lower slot usage when querying BigLake Iceberg tables.
This GA release also adds support to provide automatic synchronization of table schema in BigQuery when the table is modified through an open-source engine.
Engine-agnostic, industry-leading security and governance built-in
Customers have been telling us that building Iceberg lakehouses in a secure and governed manner is a top priority. BigLake support for Iceberg provides fine-grained access control, including row- and column-level security as well as data masking to simplify this. These features are designed to work independently of the query engine. During the preview, we expanded BigQuery to also support differential privacy for all tables including Iceberg.
You can also define security policies on a BigLake Iceberg table using BigQuery. Security policies are then enforced regardless of the query engine used — BigQuery natively enforces these policies at runtime, and open-source engines can securely access the data using the BigQuery Storage API. The BigQuery Storage API enforces the security policies at the data-plane layer, and is offered via pre-built connectors for Spark, Trino, Presto and TensorFlow. You can also use client libraries to build connectors for custom applications.
New use cases with multi-cloud Iceberg lakehouse
The open nature of Apache Iceberg lets you build multi-cloud lakehouses with uniform management of data. With this launch, you can now create BigLake Iceberg tables on Amazon S3, and query them using BigQuery Omni. BigLake’s performance and fine-grained access control features seamlessly extend to multi-cloud Iceberg tables, so you can securely perform cross-cloud analytics with BigQuery. We’ll extend similar support to Azure data lake Gen 2 in the coming weeks.
Apache Iceberg’s format uniformity across clouds also enables new data sharing use cases to help you share data with your customers, partners and suppliers, regardless of which cloud they are using. BigLake Iceberg tables on Cloud Storage or on Amazon S3 can be shared through Analytics Hub and consumed via BigQuery or OSS engines via BigQuery Storage Read API, providing an open sharing standard and the flexibility to use multiple query engines. A notable example of this is the recently announced Salesforce Data Cloud data sharing use case, which enables bi-directional cross-cloud data sharing between Salesforce and BigQuery and which is powered by Iceberg.
Getting started
To create and query your first Iceberg table, following the documentation. or run a quick proof of concept on our analytics lakehouse with the Iceberg jump start solution.
How BigQuery’s Unique Features Support Your Data at Petabyte-scale
4779
Of your peers have already read this article.
3:00 Minutes
The most insightful time you'll spend today!
Organizations rely on data warehouses to aggregate data from disparate sources, process it, and make it available for data analysis in support of strategic decision-making. BigQuery is the Google Cloud enterprise data warehouse designed to help organizations to run large scale analytics with ease and quickly unlock actionable insights. You can ingest data into BigQuery either through batch uploading or by streaming data directly to unlock real-time insights. As a fully-managed data warehouse, BigQuery takes care of the infrastructure so you can focus on analyzing your data up to petabyte-scale. BigQuery supports SQL (Structured Query Language), which you’re likely already familiar with if you’ve worked with ANSI-compliant relational databases.
BigQuery unique features
BI Engine – BigQuery BI Engine is a fast, in-memory analysis service that provides subsecond query response times with high concurrency. BI Engine integrates with Google Data Studio and Looker for visualizing query results and enables integration with other popular business intelligence (BI) tools.
BigQuery ML: BigQuery ML is unlocking machine learning for millions of data analysts. It enables data analysts and data scientists to build and operationalize machine learning models directly within BigQuery, using simple SQL.
BigQuery Omni – BigQuery Omni is a flexible, multi-cloud analytics solution powered by Anthos that lets you cost-effectively access and securely analyze data across Google Cloud, Amazon Web Services (AWS), and Azure, without leaving the BigQuery user interface (UI). Using standard SQL and familiar BigQuery APIs, you can break down data silos and gain critical business insights from a single pane of glass.
Data QnA: Data QnA enables self-service analytics for business users on BigQuery data as well as federated data from Cloud Storage, Bigtable, Cloud SQL, or Google Drive. It uses Dialogflow and enables users to formulate free-form text analytical questions, with auto-suggested entities while users type a question.
Connected Sheets -The native integration between Sheets and BigQuery makes it possible for all business stakeholders, who are already quite familiar with spreadsheet tools, to get their own up-to-date insights at any time.
Geospatial data – BigQuery offers accurate and scalable geospatial analysis with geography data types. It supports core GIS functions – measurements, transforms, constructors, and more – using standard SQL.
How does it work?
Here’s how it works: You ingest your own data into BigQuery or use data from the public datasets. Storage and compute are decoupled and can scale independently on demand. This offers immense flexibility and cost control for your business as you don’t need to keep expensive compute resources up and running all the time. As a result, BigQuery is much more cost-effective than traditional node-based cloud data warehouse solutions or on-premises systems. BigQuery also provides automatic backup and restore of your data.
You can ingest data into BigQuery in batches or stream real-time data from web, IoT, or mobile devices via Pub/Sub. You can also use Data Transfer Service to ingest data from other clouds, on-premises systems or third-party services. BigQuery also supports ODBC and JDBC drivers to connect with existing tools and infrastructure.
Interacting with BigQuery to load data, run queries, or create ML models can be done in three different ways. You can use the UI in the Cloud Console, the BigQuery command-line tool, or the API via client libraries available in several languages.
When it comes time to visualize your data, BigQuery integrates with Looker as well as several other business intelligence tools across the Google partner ecosystem.
What about security?
BigQuery offers built-in data protection at scale. It provides security and governance tools to efficiently govern data and democratize insights within your organization.
- Within BigQuery, users can assign dataset-level and project-level permissions to help govern data access. Secure data sharing ensures you can collaborate and operate your business with trust.
- Data is automatically encrypted both while in transit and at rest, ensuring that your data is protected from intrusions, theft, and attacks.
- Cloud DLP helps you discover and classify sensitive data assets.
- Cloud IAM provides access control and visibility into security policies.
- Data Catalog helps you discover and manage data.
How much does it cost?
The BigQuery sandbox lets you explore BigQuery capabilities at no cost and confirm that BigQuery fits your needs. With BigQuery you get predictable price-performance: you pay for storing and querying data, and for streaming inserts. Loading and exporting data are free of charge. Storage costs are based on the amount of data stored, and have two rates based on how often the data is changing. Query costs can be either:
- On-demand – you are charged per query by the amount of data processed
- Flat-rate – if you prefer to purchase dedicated resources
You can start with the pay-as-you-go, on-demand option and later move to flat-rate if that better suits your usage. Or, start with flat-rate, get a better understanding of your usage and move to the pay-as-you-go models for additional workloads.
To explore BigQuery and its capabilities a bit more, check out the sandbox; and when you’re ready to modernize your data warehouse with BigQuery then check out the documentation to streamline your migration process here. https://www.youtube.com/embed/So-tVyBQt8E?enablejsapi=1&
For more #GCPSketchnote, follow the GitHub repo. For similar cloud content follow me on Twitter @pvergadia and keep an eye out on thecloudgirl.dev.
Serverless and BigQuery Together on Google Cloud: Behind the L’Oreal Beauty Tech Data Platform
5338
Of your peers have already read this article.
4:00 Minutes
The most insightful time you'll spend today!
Editor’s note: In Today’s guest post we hear from beauty leader L’Oréal about their approach to building a modern data platform on fully managed services: managing the ingest of diverse datasets into BigQuery with Cloud Run, and orchestrating transformations into relevant business domain representations for stakeholders across the organization. Learn more about how businesses have benefited from Cloud Run in Forrester’s report on Total Economic Impact.
L’Oréal was born out of science. For over 100 years, we have always shaped the future of beauty, and taken its eternal quest to new horizons. This has earned us our current position as the world’s uncontested beauty leader (~€ 32 B annual sales in 2021), present in 150 countries with over 85,000 employees.
Today, with the power of our game-changing science, multiplied by cutting-edge technologies, we continue our lifelong journey of shaping the future of beauty.
As a Beauty Tech company, we leverage our decades-long heritage of rich data assets to empower our decision-making with instant, sophisticated analysis.
Because we oversee global brands, which must adapt to local requirements, we need to maintain a deep understanding of what a brands’ data represents, while managing disparate legal and regulatory requirements for different countries. Our end goal is to run a safe, compliant and sustainable data warehouse as efficiently and effectively as possible.
We sync and aggregate internal and external data from a wide variety of sources across organizations and retail stores. This made the management of our data warehouse infrastructure used to be very complex and hard to manage before Google Cloud. L’Oréal’s footprint was so large that we once found it impossible to have a standardized method to handle data. Every process was vendor-specific, and the infrastructure was brittle. We went looking for a solution to our complex data infrastructure needs, and defined the following non-negotiable principles:
- No Ops: The job of a developer at L’Oréal is not to manage servers. We need an elastic infrastructure that scales on demand, so that our developers can focus on delivering customized and inclusive beauty experiences to all consumers, rather than focusing on managing servers.
- Secure: We have strict security and compliance requirements which vary by country, and we employ a zero-trust security strategy. We must keep both our own internal data and customer data safe and encrypted.
- Sustainable : Our data lives in multiple environments, including on-prem data centers and public cloud services. We must be able to securely access and analyze this data while minimizing the complexity and environmental impact of moving and duplicating data.
- End-to-end supervision: Because developers shouldn’t be managing servers, we need a “single pane of glass” dashboard to monitor and triage the system if something goes wrong.
- Easy-to-deploy: Deploying code safely should not compromise velocity. We are constantly developing innovations that push the boundaries of science and reinvent beauty rituals. We need integrated tools to make our code deployment process seamless and safe.
- Event-driven architecture: Our data is used globally by research, product, business and engineering teams with high expectations on data quality and timeliness. Many of our internal processes and analysis are based on near real-time data.
- Data products delivered “as a service”: We want to empower our employees to drive business value at record speed. To that end, we need solutions that enable us to remove the developers from the critical path of solution delivery as much as possible.
- Extract-load-transform (ELT): Our goal is to implement the pattern to load data as soon as possible into the data warehouse to take advantage of SQL transformations.
After considering multiple vendors on the market, with these principles in mind, we landed on end-to-end Google Cloud serverless and data tooling. We were already using Google Cloud for a few processes, including BigQuery, and loved the experience.
We’ve now expanded our use of Google Cloud to fully support the L’Oréal Beauty Tech Data Platform.
L’Oréal’s Beauty Tech Data Platform incorporates data from two types of sources: directly via API, which is data that adapts easily to our schema and is inserted directly into BigQuery, and bulk data from integrations, which require event-driven transformations using Eventarc mechanisms. These transformations are performed in Cloud Run and Cloud Functions (2nd gen), or directly in SQL. With Google Cloud, we can adapt very quickly.
Today, we currently have 8500 flows for ~5000 users using the native zero-trust capabilities offered by Google Cloud. Indeed, the flows come from Google Cloud and other third-party services.
BigQuery enabled us to adopt standard SQL as our universal language in our data warehouse and meet all expectations for queries and reporting. We were also able to load original data using features like federated queries, and efficiently transitioned from ETL to ELT data ingestion by handling semi-structured data with SQL. This approach of loading original data from sources into BigQuery with non-destructive transformations allows us to reprocess data for new use-cases easily, directly within BigQuery.
Our applications are hosted on multiple environments – on-premises, in Google Cloud, and in other public clouds. This made it difficult for our data engineers and analysts to natively analyze data across clouds until we started using BigQuery Omni. This capability of BigQuery allowed us to globally access and analyze data across clouds through a single pane of glass using the native BigQuery user interface itself. Without BigQuery Omni, it would’ve been impossible for our teams to natively do cross-cloud analytics. Moreover, it eliminated the need for us to move sensitive data, which is not only expensive because of local tax and subsea transport, but also incredibly risky – sometimes even forbidden – because of local regulations.
Today Google Cloud powers our Beauty Tech Data Platform, which stores 100TB of production data in BigQuery and processes 20TB of data each month. We have more than 8000 governed datasets, and 2 millions of BigQuery tables coming from multiple data sources such as Salesforce, SAP, Microsoft, and Google Ads.
For more complex transformations where custom and specific libraries are required, Cloud Workflows help us to manage the complexity very efficiently by orchestrating steps in containers through Cloud Run, Cloud Functions and even BigQuery jobs — the most used way to transform and add value to the L’Oréal data.
Additionally, by using BigQuery and Google Cloud’s serverless compute for API ingestion, bulk data loading, and post-loading transformations, we can keep the entire system in a single boundary of trust at a fraction of the cost. With ingest, queries, and transformations all being fully elastic and on-demand, we no longer have to perform capacity planning for either the compute or analytics components of the system. And of course these services’ pay-as-you-go model perfectly aligns with L’Oréal’s strategy of only paying for something when you use it.
Google Cloud fulfilled the requirements of our Beauty Tech Data Platform. And as if offering us a no-ops, secure, easy-to-deploy, custom-development free, event-based platform with end-to-end supervision wasn’t enough, Google Cloud also helped us with our sustainability efforts.
Being able to measure and understand the environmental footprint of our public cloud usage is also a key part of our sustainable tech roadmap. With Google Cloud Carbon Footprint, we can easily see the impact of our sustainable infrastructure approach and architecture principles. Our Beauty Tech platform is a strategic ambition for L’Oréal: inventing the beauty products of the future while becoming the company of the future.
Sustainable tech is an imperative and a very important step towards this ambition of creating responsible beauty for our consumers, and sustainable-by-design tech services for our employees. We all have a role to play, and by joining forces, we can have a positive impact.
Google Cloud’s data ecosystem and serverless tools are highly complementary, and made it possible to build a next-generation data analytics platform that met all our needs.
Get started using serverless and BigQuery together on Google Cloud today.
More Relevant Stories for Your Company
Getting Started with BigQuery: 10 Quick Steps
BigQuery is Google's fully managed, petabyte scale, low cost analytics data warehouse. BigQuery is NoOps—there is no infrastructure to manage and you don't need a database administrator—so you can focus on analyzing data to find meaningful insights, use familiar SQL, and take advantage of our pay-as-you-go model. That's great, right? So
Moving Your Data Warehouse to the Cloud? Here’s What You Need to Know
Modernizing your data warehouse is one way to keep up with evolving business requirements and harness new technology. For many companies, cloud data warehousing offers a fast, flexible, and cost-effective alternative to traditional on-premises solutions. In a report sponsored by Google Cloud, TDWI examines the rise of cloud-based data warehouses
Cloud Dataprep: The Easiest Way to Cleanse Data
Anyone who has worked with data has felt the pain of data preparation. It’s a struggle—if you are working with the wrong tools—to massage data or cleanse data of anomalies, outliers, and just plain old dirty data. Eric Anderson, a Product Manager at Google working on Cloud Dataprep and a
Largest Beauty Retailer in the US Powers Digital Transformation with Google Cloud Smart Analytics
Digital technology offers increasing flexibility and choice to consumers. As a result, the retail industry is dramatically shifting toward more tailored and personalized experiences for shoppers, and businesses are rethinking how they deliver value to customers. This couldn’t be more true for the beauty retailing industry where leading companies are
