A Comprehensive Guide for Understanding and Optimizing Your Dataflow Costs
1181
Of your peers have already read this article.
3:30 Minutes
The most insightful time you'll spend today!
Dataflow is the industry-leading platform that provides unified batch and streaming data processing capabilities, and supports a wide variety of analytics and machine learning use cases. It’s a fully managed service that comes with flexible development options (from Flex Templates and Notebooks to Apache Beam SDKs for Java, Python and Go), and a rich set of built-in management tools. It seamlessly integrates with not just Google Cloud products like Pub/Sub, BigQuery, Vertex AI, GCS, Spanner, and BigTable, but also third-party services like Kafka and AWS S3, to best meet your analytics and machine learning needs.
Dataflow’s adoption has ramped up in the past couple of years, and numerous customers now rely on Dataflow for everything from designing small proof of concepts, to large-scale production deployments. As customers are always trying to optimize their spend and do more with less, we naturally get questions related to cost optimization for Dataflow.
In this post, we’ve put together a comprehensive list of actions you can take to help you understand and optimize your Dataflow costs and business outcomes, based on our real-world experiences and product knowledge. We’ll start by helping you understand your current and near-term costs. We’ll then share how you can continuously evaluate and optimize your Dataflow pipelines over time. Let’s dive in!
Understand your current and near-term costs
The first step in most cost optimization projects is to understand your current state. For Dataflow, this involves the ability to effectively:
- Understand Dataflow’s cost components
- Predict the cost of potential jobs
- Monitor the cost of submitted jobs
Understanding Dataflow’s cost components
Your Dataflow job will have direct and indirect costs. Direct costs reflect resources consumed by your job, while indirect costs are incurred by the broader analytics/machine learning solution that your Dataflow pipeline enables. Examples of indirect costs include usage of different APIs invoked from your pipeline, such as:
- BigQuery Storage Read and Write APIs
- Cloud Storage APIs
- BigQuery queries
- Pub/Sub subscriptions and publishing, and
- Network egress, if any
Direct and indirect costs influence the total cost of your Dataflow pipeline. Therefore, it’s important to develop an intuitive understanding of both components, and use that knowledge to adopt strategies and techniques that help you arrive at a truly optimized architecture and cost for your entire analytics or machine learning solution. For more information about Dataflow pricing, see the Dataflow pricing page.
Predict the cost of potential jobs
You can predict the cost of a potential Dataflow job by initially running the job on a small scale. Once the small job is successfully completed, you can use its results to extrapolate the resource consumption of your production pipeline. Plugging those estimates into the Google Cloud Pricing Calculator should give you the predicted cost of your production pipeline. For more details about predicting the cost of potential jobs, see this (old, but very applicable) blog post on predicting the cost of a Dataflow job.
Monitor the cost of submitted jobs
You can monitor the overall cost of your Dataflow jobs using a few simple tools that Dataflow provides out of the box. Also, as you optimize your existing pipelines, possibly using recommendations from this blog post, you can monitor the impact of your changes on performance, cost, or other aspects of your pipeline that you care about. Handy techniques for monitoring your Dataflow pipelines include:
- Use metrics within the Dataflow UI to monitor key aspects of your pipeline.
- For CPU intensive pipelines, profile the pipeline to gain insight into how CPU resources are being used throughout your pipeline.
- Experience real-time cost control by creating monitoring alerts on Dataflow job metrics which ship out of the box, and that can be good proxies for the cost of your job. These alerts send real-time notifications once the metrics associated with a running pipeline exceeds a predetermined threshold. We recommend that you only do this for your critical pipelines, so you can avoid notification fatigue.
- Enable Billing Export into BiqQuery, and perform deep, ad-hoc analyses of your Dataflow costs that help you understand not just the key drivers of your costs, but also how these drivers are trending over time.
- Create a labeling taxonomy, and add labels to your Dataflow jobs that help facilitate cost attribution during the ad-hoc analyses of your Dataflow cost data using BigQuery. Check out this blog post for some great examples of how to do this.
- Run your Dataflow jobs using a custom Service Account. While this is great from a security perspective, it also has the added benefit of enabling the easy identification of APIs used by your Dataflow job.
Optimize your Dataflow costs
Once you have a good understanding of your Dataflow costs, the next step is to explore opportunities for optimization. Topics to be considered during this phase of your cost optimization journey include:
- Your optimization goals
- Key factors driving the cost of your pipeline
- Considerations for developing optimized batch and streaming pipelines
Let’s look at each of these topics in detail.
Goals
The goal of a cost optimization effort may seem obvious: “reduce my pipeline’s cost.” However, your business may have other priorities that have to be carefully considered and balanced with the cost reduction goal. From our conversations with customers, we have found that most Dataflow cost optimization programs have two main goals:
1. Reduce the pipeline’s cost
2. Continue meeting the service level agreements (SLAs) required by the business
Cost factors
Opportunities for optimizing the cost of your Dataflow pipeline will be based on the key factors driving your costs. While most of these factors will be identified through the process of understanding your current and near term costs, we have identified a set of frequently recurring cost factors, which we have grouped into three buckets: Dataflow configuration, performance, and business requirements.
Dataflow configuration includes factors like:
- Should your pipeline be streaming or batch?
- Are you using Dataflow services like Streaming Engine, or FlexRS?
- Are you using a suitable machine type and disk size?
- Should your pipeline be using GPUs?
- Do you have the right number of initial and maximum workers?
Performance includes factors like:
- Are your SDK versions (Java, Python, Go) up to date?
- Are you using IO connectors efficiently? One of the strengths of Apache Beam is a large library of IO connectors to different storage and queueing systems. Apache Beam IO connectors are already optimized for maximum performance. However, there may be cases where there are trade-offs between cost and performance. For example, BigQueryIO supports several write methods, each of them with somewhat different performance and cost characteristics. For more details, see slides 19 – 22 of the Beam Summit session on cost optimization.
- Are you using efficient coders? Coders affect the size of the data that needs to be saved to disk or transferred to a dedicated shuffle service in the intermediate pipeline stages, and some coders are more efficient than others. Metrics like total shuffle data processed and total streaming data processed can help you identify opportunities for using more efficient coders. As a general rule, you should consider whether the data that appears in your pipeline contains redundant data that can be eliminated by both filtering out unused columns as early as possible, and using efficient coders such as AvroCoder or RowCoder. Also, remember that if stages of your pipeline are fused, the coders in the intermediate steps become irrelevant.
- Do you have sufficient parallelism in your pipeline? The execution details tab, and metrics like processing parallelism keys can help you determine whether your pipeline code is taking full advantage of Apache Beam’s ability to do massively parallel processing (for more details, see parallelism). For example, if you have a transform which outputs a number of records for each input record (“high fan-out transform”) and the pipeline is automatically optimized using Dataflow fusion optimization, the parallelism of the pipeline can be suboptimal, and may benefit from preventing fusion. Another area to watch for is “hot keys.” This blog discusses this topic in great detail.
- Are your custom transforms efficient? Job monitoring techniques like profiling your pipeline and viewing relevant metrics can help you catch and correct your inefficient use of custom transforms. For example, if your Java transform needs to check if the data matches a certain regular expression pattern, then compiling that pattern in the setup method and doing the matching using the precompiled pattern in the “process element” method is much more efficient. The simpler, but inefficient alternative is to use the String.matches() call in the “process element” method, which will have to compile the pattern every time. Another consideration regarding custom transforms is that grouping elements for external service calls can help you prepare optimal request batches for external API calls. Finally, transforms that perform multi-step operations (for example, calls to external APIs that require extensive processes for creating and closing the client for the API) can often benefit from splitting these operations, and invoking them in different methods of the ParDo (for more details, see ParDo life cycle).
- Are you doing excessive logging? Logs are great. They help with debugging, and can significantly improve developer productivity. On the other hand, excessive logging can negatively impact your pipeline’s performance.
Business requirements can also influence your pipeline’s design, and increase costs. Examples of such requirements include:
- Low end-to-end ingest latency
- Extremely high throughput
- Processing late arriving data
- Processing streaming data spikes
Considerations for developing optimized data pipelines
From our work with customers, we have compiled a set of guidelines that you can easily explore and implement to effectively optimize your Dataflow costs. Examples of these guidelines include:
Batch and streaming pipelines:
- Consider keeping your Dataflow job in the same region as your IO source and destination services.
- Consider using GPUs for specialized use cases like deep-learning inference.
- Consider specialized machine types for memory- or compute-intensive workloads.
- Consider setting the maximum number of workers for your Dataflow job.
- Consider using custom containers to pre-install pipeline dependencies, and speed up worker startup time.
- Where necessary, tune the memory of your worker VMs.
- Reduce your number of test and staging pipelines, and remember to stop pipelines that you no longer need.
Batch pipelines:
- Consider FlexRS for use cases that have start time and run time flexibility.
- Run fewer pipelines on larger datasets. Starting and stopping a pipeline incurs some cost, and processing a very small dataset in a batch pipeline can be inefficient.
- Consider defining the maximum duration for your jobs, so you can eliminate runaway jobs, which keep running without delivering value to your business.
Summary
In this post, we introduced you to a set of knowledge and techniques that can help you understand the current and near-term costs associated with your Dataflow pipelines. We also shared a series of considerations that can help you optimize your Dataflow pipelines, not just for today, but also as your business evolves over time. We shared lots of resources with you, and hope that you find them useful. We look forward to hearing about the savings and innovative solutions that you are able to deliver for your customers and your business.
How OnlineSales.ai and Google Cloud Helped TATA 1mg Increase Ad Revenue by 700%
1363
Of your peers have already read this article.
3:30 Minutes
The most insightful time you'll spend today!
Editor’s note: We invited partners from across our retail ecosystem to share stories, best practices, and tips and tricks on how they are helping retailers transform during a time that continues to see tremendous change. This original blog post was published by OnlineSales.ai. Please enjoy this updated entry from our partner.
Tata’s online healthcare entity, Tata 1mg, aims to revolutionize how a consumer finds the right healthcare professional for their needs. The platform offers e-pharmacy services, diagnostics, and health content and has more than 200,000 active brands in its rapidly expanding list of suppliers, sellers and manufacturers. Several of these brands would need regular involvement from the marketplace team in order to set up and run ad campaigns. As one of the largest platforms, Tata 1mg approached OnlineSales.ai to address their growing needs for an efficient Retail Media Monetisation suite to help scale up their existing monetisation efforts.
Tata 1mg needed more data-driven insights and customizability for their monetisation efforts
Tata 1mg saw several challenges that they needed to address.
- Manual effort: Tata 1mg marketplace team’s monetization process was largely manual and required heavy time and resource investment to activate advertisers, which subsequently ate into their overall ad revenues.
- Non-scalable framework: Given the fast-growing number of advertisers on their marketplace – well into the hundreds of thousands – it became increasingly clear that their existing monetization framework was not scalable.
- Inadequate reporting & analytics: The existing framework facilitated only simple reporting, and brands were not able to draw the deep insights they needed to make data-led decisions and refine their advertising ROIs.
- Steep learning curve: Several brands lacked dedicated advertising experts who could bring the experience and skills needed to plan successful ad campaigns and optimize budgets. The involved learning curve led to increased advertiser-churn.
- Lack of personalized offerings: The manual nature of Tata 1mg’s monetization framework left little room for customizability, and account managers could offer very little in terms of tailored ad-buying experiences to different types of advertisers.
All of these issues led to regular revenue leakage. Tata 1mg realized the need for a central platform that addressed the above-mentioned issues and would also be able to cater to new ad channels and the requirements thereof. Developing such a solution in house was evaluated, but was found to be expensive and would require a lot of time to build.
OnlineSales.ai’s AI/ML-powered solution on Google Cloud offered the automation and flexibility to support modern omnichannel advertising needs
Therefore, Tata 1mg looked to OnlineSales.ai Monetize, an AI/ML-powered retail media monetisation suite on Google Cloud, to help address their needs for a cloud-based monetisation platform that offered the flexibility to support omnichannel advertising models, had automation built in to reduce manual tasks, provided detailed intelligence and analytics, and delivered quick, reliable scalability.
- White labeled self-serve platform: OnlineSales.ai’s retail media platform was implemented within a short 4 weeks. After thorough testing and implementation, the platform was successfully launched, and enabled sellers to run ads on a fully self-serve platform. This required zero involvement from the Tata team.
- Unified omni channel buying: The team at OnlineSales.ai worked closely with the platform’s core teams to activate and mobilize various advertising products through a self-serve platform, while simultaneously addressing any unique requirements they had.
- Omnichannel analytics with AI-led suggestions: Sellers are now also able to generate detailed reports that provide them critical insights into campaigns, and help them make better use of their budgets – enhancing their experience and interest in advertising on the platform.
- Personalized buying experiences: The tailored UX offered by OnlineSales.ai’s Monetize helped Tata engage brands of all levels in technical aptitude to use the platform easily. This also allowed admins to configure what types of inventories were available to different brands or advertiser segments.
OnlineSales.ai makes use of Google Cloud Dataflow to facilitate computations on real-time streaming. Microservices are deployed on the Google Kubernetes Engine (GKE) platform due to the solution’s well-known ability to handle scale.
In addition, Dataproc is used by the company to run batch processing apps while Cloud Load Balancing helps manage traffic across different regions; all at scale. Onlinesales.ai also uses Memorystore as a database for their ad servers in order to have very low latency, and deploys BigQuery to run analytical applications and facilitate powerful reporting. The bulk of their office applications are deployed on different VMs on Compute Engine, and the company also makes use of Cloud Storage for data storage and transfers between applications.
Learn more about OnlineSales.ai available on the Google Cloud Marketplace.
About OnlineSales.ai
OnlineSales.ai, offers a fully white labeled retail media platform which helps retailers unlock additional revenue by activating advertising real estate on their platform. With its AI/ML-powered solution, OnlineSales.ai allows retailers to turn brands of all sizes into advertisers – at scale. Its self-serve platform simplifies the ad buying process, and enhances outcomes with smart automation, boosting ad retailers’ ad revenues by multiples.
5195
Of your peers have already watched this video.
5:00 Minutes
The most insightful time you'll spend today!
How to Build and Maintain Spark and Hadoop Clusters Quickly, Easily and Inexpensively
How often have you struggled to create Spark and Hadoop clusters only to have them become unstable?
Building and maintaining clusters that are the building blocks of big data analytics should not be a time-consuming, repetitive, and expensive process. But with the current set of tools, it often is.
Google Cloud Dataproc solves this. It is a fast, easy-to-use, fully-managed cloud service for running Spark and Hadoop clusters in a simple, more cost-efficient way.
Operations that used to take hours or days take now seconds or minutes instead, and you pay only for the resources you use (with per-second billing).
Watch this five-minute demo to find out how—with just a few clicks—you can set up clusters that build and heal themselves. You’ll find out how to creating a large Dataproc cluster with preemptible VMs, which help run large clusters at a low total cost.
4525
Of your peers have already watched this video.
1:30 Minutes
The most insightful time you'll spend today!
AI-powered Cameras Help You Serve Your Pets Better. Learn How
Watch the video to learn how you can leverage Google Cloud platform and AI functions to build pet-detection cameras that can capture and send image-based alert messages on your phone to notify when your pets arrive or leave.
The Fantastic Story of How BMG Enables a Micropayments Strategy So Music Artists Get Paid
7425
Of your peers have already read this article.
3:30 Minutes
The most insightful time you'll spend today!
The music industry is rapidly changing. Only 20 years ago, the availability of music and the infrastructure that was required to make an album a sales success were incredibly complex and expensive. With the decline of physical sales and a fundamental shift to digital, music streaming now accounts for more than half of all sales globally.
At the same time, technology has democratized music-making; in many ways, it has made the musical landscape more diverse. Artists can upload their music with the click of a button. But while it’s easier for creators to share their songs with audiences, getting paid has become more fragmented.
Although music is booming, people no longer buy it outright. Instead, listeners download music digitally or subscribe to streaming services to have their libraries with them at all times. To monetize digital content effectively, artists need to know when, where, and how often their songs are played on each service. To help them navigate this complicated royalties landscape and maximize their profits, Berlin-based international music company BMG provides customized, transparent, and fair services to songwriters and artists.
With publishing and recording divisions under one roof, the subsidiary of international media giant Bertelsmann works with both emerging artists and established stars, including John Legend, Kylie Minogue, Mick Jagger, and Keith Richards. With the MyBMG web and mobile application, clients can view and analyze their royalty details in real time and collect payment. When a new record is released, BMG uses data to maximize its impact and revenue for its creators.
“We make sure that everyone who uses our clients’ music knows who needs to be paid the associated royalties, then we collect these royalties and share them out quickly and transparently,” explains Sebastian Hentzschel, Chief Information Officer at BMG. “When our artists release new music, we make sure that it’s marketed and promoted effectively around the world.”
“We needed a scalable solution for our royalty workloads that was intuitive for our developers. We also wanted a partner, not a client-vendor relationship. With autoscaling via BigQuery, excellent customer support, and a clean and simple user interface, Google Cloud ticks every box for us.”
—Gaurav Mittal, Vice President Group Technology, BMG
Getting up to speed with a new way of paying artists
In this digital world, artists aren’t just paid every time a fan buys an album—they’re paid a small amount, or royalty, for each song downloaded or streamed by a listener. So, when the industry shifted to digital, the volume of data that BMG needed to handle grew exponentially. “One CD sale is equivalent to about 1,500 streamed songs or plays,” says Gaurav Mittal, Vice President Group Technology at BMG. “That means IT departments have to process 1,500 times the amount of data to calculate payments for artists, and this makes scalable micropayment processing very important.”
Until 2019, BMG’s infrastructure was entirely hosted on-premises. Hardware limitations made it challenging to scale on-demand, making it harder to handle the data peaks that royalty processing can bring. “With our on-premises infrastructure, we were going to hit a ceiling in a few years,” says Gaurav. “We still managed to process royalty payments for our clients, but it was increasingly time consuming and expensive. To keep focusing on our clients, rather than our infrastructure, we decided to migrate to Google Cloud.”
From the outset, Gaurav and his team had a clear vision for the partnership: “Most importantly, we needed a scalable solution for our royalty workloads that was intuitive for our developers. We also wanted a partner, not a client-vendor relationship,” he says. “With autoscaling via BigQuery, excellent customer support, and a clean and simple user interface, Google Cloud ticks every box for us.”
Keeping artists happy with business-as-usual payouts during migration
To move applications to the cloud while keeping payment cycles on track for its artists, BMG teamed up with Google Cloud partner Rackspace Technology. “We selected Rackspace Technology because it combines strong technical muscle and a global footprint, with the customer service of a local boutique firm,“ shares Gaurav.
BMG’s own technology team put together the outline for the Google Cloud architecture, which they passed on to Rackspace Technology for optimizations and the ultimate stamp of approval. Whenever Gaurav and his developers needed support, Rackspace Technology was ready to go. “So far, we’ve migrated 17 applications successfully, and Rackspace Technology has been 100% spot-on with each suggestion,” says Gaurav. “I can’t recall a single flaw in a Rackspace Technology-approved architecture, and that really says something.”
When BMG began its migration in August 2019, the team developed an ambitious two-year plan. Only 14 months later, however, the project is more than 75% complete. Among the solutions that BMG is using today are Cloud Storage to securely store 130 TB of data, and Cloud SQL as its standard database technology. The web applications run on Compute Engine, App Engine, and Google Kubernetes Engine.
“After successfully moving a few applications, it was clear that with the strong teamwork of BMG and Rackspace Technology, together with the ease of use of Google Cloud, we could speed up the project without sacrificing quality,” says Gaurav. “We’re set to complete our migration six months before schedule, helping us to quickly move out of our hybrid environment.”
“Our income-tracking teams are very savvy on the data, and the simplicity of Google Cloud empowers them to self-serve analytics, rather than wait for IT. Our teams are much more productive.”
—Gaurav Mittal, Vice President Group Technology at BMG
Royalty reporting and processing with BigQuery and Dataproc
So far, all of BMG’s critical workloads are up and running on Google Cloud. Royalty calculations, for example, which require incredible processing power and the collection of many micropayments to ensure full and timely payout, run entirely on Dataproc with output stored on BigQuery for downstream integration and reporting.
Enabling more harmonious workflows through self-serve analytics
As the new beating heart of BMG’s royalty reporting, BigQuery changed the rhythm of collaboration company-wide. In the past, income tracking teams had to contact IT departments if they needed deeper data insights for their work. By integrating Data Catalog with BigQuery, BMG has made the data more accessible to all teams. This helps them detect missing income and new revenue streams independently, maximizing profits for artists.
“Our income-tracking teams are very savvy on the data, and the simplicity of Google Cloud empowers them to self-serve analytics, rather than wait for IT,” says Gaurav. “Our teams are much more productive.”
“Google Cloud enables us to be more client focused and deliver better features faster. We believe it’s just the beginning. We offer rights and royalty services for music publishing, recorded music, neighboring rights, and books. Without scalability limitations, it’s absolutely conceivable to offer our platform as a service to other companies or industries, such as gaming. Google Cloud has opened a world of possibilities.”
—Sebastian Hentzschel, Chief Information Officer, BMG
With a leaner IT environment, BMG can focus its effort on the needs of its clients. Beyond improvements in royalty processing, it can concentrate on app development, releasing new features and enhancements more frequently. By hosting applications on Google Kubernetes Engine, App Engine, and Compute Engine, BMG has built a CI/CD pipeline with automated deployments and testing to significantly speed up workflows.
“In our old system, it could take several weeks to set up an environment,” says Gaurav. “With Google Kubernetes Engine, any of our developers can complete the process in a few clicks. Having that autonomy makes our developers more motivated and self-driven.”
“Google Cloud enables us to be more client focused and deliver better features faster,” adds Sebastian. “We believe it’s just the beginning. We offer rights and royalty services for music publishing, recorded music, neighboring rights, and books. Without scalability limitations, it’s absolutely conceivable to offer our platform as a service to other companies or industries, such as gaming. Google Cloud has opened a world of possibilities.”
With the migration almost complete, BMG is looking forward to its next technology project. It plans to leverage AutoML to further scale and automate royalty tracking with machine learning. On the marketing side, advanced analytics will help BMG determine the effectiveness of promotional campaigns around the world, further increasing profits for artists. By connecting Google Data Studio to BigQuery, BMG will increase the quality of its analyses, helping musicians better understand the reach of their music around the world.
In the end, Gaurav shares, helping musicians is what it all comes down to. “We’re a new kind of music company because we build our services around our artist, songwriter, and publisher clients, not the other way around,” he says. “Google Cloud is helping us maintain strong relationships with our clients, and that’s music to our ears.”
4454
Of your peers have already downloaded this article
3:00 Minutes
The most insightful time you'll spend today!
Contact centers can transform customer experience using AI-driven speech analytics to evaluate every customer interaction and use it as a ‘data point’ to identify key patterns, enquiries, pain points, reviews and feedback to enhance customer experience with real-time, personalized recommendations. Knowlarity’s AI-based cloud telephony solutions for businesses built with Google Cloud takes speech analytics to another level!
Download the article to empower your contact centers with AI-powered speech analytics to make predictive analysis, reduce call handling time and volume as well as train contact center agents to provide customers with quick and real-time feedback.
More Relevant Stories for Your Company
Hospitals Can Offer Interconnected Patient Experiences Using Google’s Natural Language Services
Machine Learning (ML) in healthcare helps extract data from conversations, medical records, forms, research reports, insurance claims and other documents across the care value-chain to help care providers have a holistic view of their patients to draw insights for diagnoses and treatments. With Natural Language Processing(NLP), healthcare organizations can program
Transform ‘Dark Data’ from Documents with Document AI, Cloud Functions and Workflows
At enterprises across industries, documents are at the center of core business processes. Documents store a treasure trove of valuable information whether it's a company's invoices, HR documents, tax forms and much more. However, the unstructured nature of documents make them difficult to work with as a data source. We
Data Leaders in 2021 and Beyond: How to Prepare
As technologies and workplace environments are quickly evolving, how people and businesses leverage data in their day-to-day workflow is also changing. Data leaders are an emerging force navigating and charting pathways forward towards new horizons for how people and organizations experience data. In this presentation, Pedro Arellano, Product Marketing Director,
STAC-M3 Tick History Analytics in Google Cloud Benchmark Results Reveals it is 18X Faster than Previous Version
The Securities Technology Analysis Center (STAC®), an organization that improves technology discovery and assessment in the finance industry through dialog and research, recently audited the STAC-M3™ benchmark suite on Google Cloud (SUT ID KDB211210). These enterprise tick-analytics benchmarks assess the ability of a solution stack such as database software, servers,
