It’s simply amazing to witness how some of our systems integrators employ Google Cloud solutions to drive innovation in ways we at Google may never have considered—especially in healthcare.  According to analyst firm MarketsandMarkets, the market for the Cloud in healthcare is projected to grow 43% between 2020 and 2025 to nearly $65 billion, fueled by the need for better technology infrastructures and faster digital transformation.  

Healthcare and life sciences communities are looking to Google Cloud and its Service engagement model partners to improve collaboration and activate the power of medical data.  These transformations deliver robust data analytics and bring a much deeper perspective into health epidemics like COVID-19 to help save lives.

In the healthcare and life sciences industry, our Google Cloud partners and customers provide constant energy and inspiration–and are the magic in some key healthcare innovations globally.  

Let me show you how they are solving real-world business challenges.

Improving collaboration in Healthcare

Cloudbakers and Comanche County Memorial Hospital transitioned 2,000 employees to Google Workspace to improve collaboration, reduce costs, and increase security. By implementing a system that requires less maintenance while enabling mobile access to data and true collaboration, Comanche County Memorial Hospital saves $175,000 annually on licenses and helps medical professionals spend more time with patients.

“We chose Google Workspace because it cost a quarter of what we were paying previously, offers the kind of modern features that healthcare facilities need, and gave us data security and peace of mind.” —James Wellman, CIO, Comanche County Memorial Hospital

Accessing previously locked down medical data

Google Cloud and Quantiphi supported advances in cloud-based machine learning services to reduce infrastructure costs, unlock new paths of treatment, and dramatically reduce the amount of time it takes to evaluate scanned imagery following a stroke.  John Hopkins University BIOS Division has been working on medical imaging to accelerate insights from scans on approximately 500 patients from 2,500 hours to 90 minutes, and lead to more accurate decision-making for brain injury patients that will ultimately improve medical outcomes.  

“We’ve aligned closely with the goal of showing that a cloud-based, AI-driven approach is robust and that it can be performed while protecting personal PHI. In terms of costs, the cloud has definitely reduced some of the traditional financial demands of our research.” —Daniel F. Hanley, Jr., M.D., Director, Johns Hopkins University BIOS Division

Improving data access with the flexibility of Google Cloud

With the help of MediaAgility, TRIARQ migrated to Google Cloud to modernize their platform, build new applications, and expand their global footprint.  With BigQuery, TRIARQ can now consolidate all transactional data, past and present, into a single location to report on specific insights and predict future data from e-prescriptions to complex revenue-cycle data and value-base analysis.

“The future of this industry will need to be a global one, and we can no longer be stuck in legacy systems if we want to survive. Having a team that is passionate about supporting us in this journey is definitely a plus point.”—Yaw Kwakye, Co-founder and Chief Architect, TRIARQ

Increasing visibility to COVID-19 outbreaks for actionable insights

As part of its response to the COVID-19 pandemic, HCA Healthcare chose to work with Google Cloud and SADA to create a national portal that increases visibility into outbreaks in 3,100 counties across the country in just 8 weeks.  Now, the portal generates 30,000 new analytical views each day that can help inform private and public sector decision making for reopenings, closures, hot spots, and many other population health management activities. 

“This project required deep knowledge of AI, and consumer-facing platforms, as well as healthcare. Google brought together the ideal combination of product, people, and partners. Google Cloud’s healthcare-specific products along with SADA’s expertise in the healthcare IT space made this partnership the perfect choice to move quickly and intelligently.”—Dr. Edmund Jackson, Chief Data Officer, HCA Healthcare

We’re committed to building the technology, resources, and services through Partner Advantage to help our partners address this opportunity.  Looking for a solution focused partner in your region who has achieved Expertise and/or Specialization in your industry?  Search our Global Partner Directory.  Not yet a Google Cloud partner? Visit Partner Advantage and learn how to become one today!

The app economy has enabled a huge range of unique business models to flourish. One such model is online food ordering and delivery services, in which apps leverage geo-location data to aggregate local food choices and offer personalized options to consumers.

A leading company in this space is Just Eat. Launched in the UK in 2001 with a vision of ‘serving the world’s greatest menu. Brilliantly.’ The company has capitalized on the popularity of online food delivery and grown its presence across 12 markets. 

Just Eat acts as an intermediary between take-out food outlets and hungry customers, giving local restaurants access to a broader base of potential diners, while providing consumers with an easy and secure way to order and pay for food from their favourite restaurants.

Today the company helps 27 million customers find food from more than 112,000 restaurants—everything from homemade Italian pasta, to Chinese noodle bowls, to fish-and-chips. 

Data is the fuel of Just Eat’s rapid growth, but it wasn’t always looked at that way. In its early days, Just Eat struggled with the deluge of information and faced fragmentation across its systems. In fact, the company realized its legacy data vendor wasn’t capable of ingesting 90 percent of the data produced by its food platform. This was incredibly frustrating for Just Eat’s analysts and data scientists, who had to waste time cleaning up sources instead of leveraging the data to create a better user experience. 

Just Eat turned to Google Cloud, and now uses machine learning (ML) to power sophisticated consumer recommendations on both its app and website. It also makes heavy use of features offered by Google Cloud Platform, including BigQuery for running analytics on its customer data set and Cloud Pub/Sub for messaging app users with relevant offers in real-time. 

Having all of Just Eat’s data in one platform has translated into real value for its customers. With Google Cloud tools, Just Eat has created its own proprietary Customer Ontology framework, which today contains 5.5 billion features that better understand consumers’ behavior and food habits, and provides insights into previous visits.

Just Eat recently created an “Adventurous Index” to map its customers according to their ordering habits, enabling them to tailor their marketing and user experiences. For example, mid-adventurous customers are shown a choice of restaurants that serve their most ordered cuisine, while adventurous customers can choose from restaurants that serve a wider variety. This not only has prompted consumers to be more adventurous with their choices, but also has led to more business at a more diverse set of restaurants.

Matt Cresswell, Director of Customer Platforms at Just Eat said that Google Cloud has become integral to its product delivery: “Consumer food choice is a hugely nuanced topic. We know that individuals have their own unique journeys when they use Just Eat. We’ve sought to create a truly one-to-one relationship with every customer. The changes we’ve made to the platform mean they can access the dishes they enjoy at the touch of a fingertip, and find inspiration to discover new dishes they’ll love. We’re grateful to Google Cloud for helping us support our customers on their culinary explorations.”

Dataflow is the industry-leading platform that provides unified batch and streaming data processing capabilities, and supports a wide variety of analytics and machine learning use cases. It’s a fully managed service that comes with flexible development options (from Flex Templates and Notebooks to Apache Beam SDKs for Java, Python and Go), and a rich set of built-in management tools. It seamlessly integrates with not just Google Cloud products like Pub/Sub, BigQuery, Vertex AI, GCS, Spanner, and BigTable, but also third-party services like Kafka and AWS S3, to best meet your analytics and machine learning needs.

Dataflow’s adoption has ramped up in the past couple of years, and numerous customers now rely on Dataflow for everything from designing small proof of concepts, to large-scale production deployments. As customers are always trying to optimize their spend and do more with less, we naturally get questions related to cost optimization for Dataflow. 

In this post, we’ve put together a comprehensive list of actions you can take to help you understand and optimize your Dataflow costs and business outcomes, based on our real-world experiences and product knowledge. We’ll start by helping you understand your current and near-term costs. We’ll then share how you can continuously evaluate and optimize your Dataflow pipelines over time. Let’s dive in!

Understand your current and near-term costs

The first step in most cost optimization projects is to understand your current state. For Dataflow, this involves the ability to effectively: 

• Understand Dataflow’s cost components
• Predict the cost of potential jobs
• Monitor the cost of submitted jobs

Understanding Dataflow’s cost components

Your Dataflow job will have direct and indirect costs. Direct costs reflect resources consumed by your job, while indirect costs are incurred by the broader analytics/machine learning solution that your Dataflow pipeline enables. Examples of indirect costs include usage of different APIs invoked from your pipeline, such as: 

• BigQuery Storage Read and Write APIs
• Cloud Storage APIs
• BigQuery queries
• Pub/Sub subscriptions and publishing, and
• Network egress, if any

Direct and indirect costs influence the total cost of your Dataflow pipeline. Therefore, it’s important to develop an intuitive understanding of both components, and use that knowledge to adopt strategies and techniques that help you arrive at a truly optimized architecture and cost for your entire analytics or machine learning solution. For more information about Dataflow pricing, see the Dataflow pricing page.

Predict the cost of potential jobs

You can predict the cost of a potential Dataflow job by initially running the job on a small scale. Once the small job is successfully completed, you can use its results to extrapolate the resource consumption of your production pipeline. Plugging those estimates into the Google Cloud Pricing Calculator should give you the predicted cost of your production pipeline. For more details about predicting the cost of potential jobs, see this (old, but very applicable) blog post on predicting the cost of a Dataflow job.

Monitor the cost of submitted jobs

You can monitor the overall cost of your Dataflow jobs using a few simple tools that Dataflow provides out of the box. Also, as you optimize your existing pipelines, possibly using recommendations from this blog post, you can monitor the impact of your changes on performance, cost, or other aspects of your pipeline that you care about. Handy techniques for monitoring your Dataflow pipelines include:

1. Use metrics within the Dataflow UI to monitor key aspects of your pipeline.
2. For CPU intensive pipelines, profile the pipeline to gain insight into how CPU resources are being used throughout your pipeline. 
3. Experience real-time cost control by creating monitoring alerts on Dataflow job metrics which ship out of the box, and that can be good proxies for the cost of your job. These alerts send real-time notifications once the metrics associated with a running pipeline exceeds a predetermined threshold. We recommend that you only do this for your critical pipelines, so you can avoid notification fatigue. 
4. Enable Billing Export into BiqQuery, and perform deep, ad-hoc analyses of your Dataflow costs that help you understand not just the key drivers of your costs, but also how these drivers are trending over time. 
5. Create a labeling taxonomy, and add labels to your Dataflow jobs that help facilitate cost attribution during the ad-hoc analyses of your Dataflow cost data using BigQuery. Check out this blog post for some great examples of how to do this.
6. Run your Dataflow jobs using a custom Service Account. While this is great from a security perspective, it also has the added benefit of enabling the easy identification of APIs used by your Dataflow job.

Optimize your Dataflow costs

Once you have a good understanding of your Dataflow costs, the next step is to explore opportunities for optimization. Topics to be considered during this phase of your cost optimization journey include: 

• Your optimization goals
• Key factors driving the cost of your pipeline
• Considerations for developing optimized batch and streaming pipelines

Let’s look at each of these topics in detail.

Goals

The goal of a cost optimization effort may seem obvious: “reduce my pipeline’s cost.” However, your business may have other priorities that have to be carefully considered and balanced with the cost reduction goal. From our conversations with customers, we have found that most Dataflow cost optimization programs have two main goals:

1. Reduce the pipeline’s cost

2. Continue meeting the service level agreements (SLAs) required by the business

Cost factors

Opportunities for optimizing the cost of your Dataflow pipeline will be based on the key factors driving your costs. While most of these factors will be identified through the process of understanding your current and near term costs, we have identified a set of frequently recurring cost factors, which we have grouped into three buckets: Dataflow configuration, performance, and business requirements.

Dataflow configuration includes factors like: 

• Should your pipeline be streaming or batch? 
• Are you using Dataflow services like Streaming Engine, or FlexRS? 
• Are you using a suitable machine type and disk size?
• Should your pipeline be using GPUs?
• Do you have the right number of initial and maximum workers? 

Performance includes factors like: 

• Are your SDK versions (Java, Python, Go) up to date? 
• Are you using IO connectors efficiently? One of the strengths of Apache Beam is a large library of IO connectors to different storage and queueing systems. Apache Beam IO connectors are already optimized for maximum performance. However, there may be cases where there are trade-offs between cost and performance. For example, BigQueryIO supports several write methods, each of them with somewhat different performance and cost characteristics. For more details, see slides 19 – 22 of the Beam Summit session on cost optimization.
• Are you using efficient coders? Coders affect the size of the data that needs to be saved to disk or transferred to a dedicated shuffle service in the intermediate pipeline stages, and some coders are more efficient than others. Metrics like total shuffle data processed and total streaming data processed can help you identify opportunities for using more efficient coders. As a general rule, you should consider whether the data that appears in your pipeline contains redundant data that can be eliminated by both filtering out unused columns as early as possible, and using efficient coders such as AvroCoder or RowCoder. Also, remember that if stages of your pipeline are fused, the coders in the intermediate steps become irrelevant. 
• Do you have sufficient parallelism in your pipeline? The execution details tab, and metrics like processing parallelism keys can help you determine whether your pipeline code is taking full advantage of Apache Beam’s ability to do massively parallel processing (for more details, see parallelism). For example, if you have a transform which outputs a number of records for each input record (“high fan-out transform”) and the pipeline is automatically optimized using Dataflow fusion optimization, the parallelism of the pipeline can be suboptimal, and may benefit from preventing fusion. Another area to watch for is “hot keys.” This blog discusses this topic in great detail. 
• Are your custom transforms efficient? Job monitoring techniques like profiling your pipeline and viewing relevant metrics can help you catch and correct your inefficient use of custom transforms. For example, if your Java transform needs to check if the data matches a certain regular expression pattern, then compiling that pattern in the setup method and doing the matching using the precompiled pattern in the “process element” method is much more efficient. The simpler, but inefficient alternative is to use the String.matches() call in the “process element” method, which will have to compile the pattern every time. Another consideration regarding custom transforms is that grouping elements for external service calls can help you prepare optimal request batches for external API calls. Finally, transforms that perform multi-step operations (for example, calls to external APIs that require extensive processes for creating and closing the client for the API) can often benefit from splitting these operations, and invoking them in different methods of the ParDo (for more details, see ParDo life cycle). 
• Are you doing excessive logging? Logs are great. They help with debugging, and can significantly improve developer productivity. On the other hand, excessive logging can negatively impact your pipeline’s performance. 

Business requirements can also influence your pipeline’s design, and increase costs. Examples of such requirements include: 

• Low end-to-end ingest latency
• Extremely high throughput
• Processing late arriving data
• Processing streaming data spikes

Considerations for developing optimized data pipelines 

From our work with customers, we have compiled a set of guidelines that you can easily explore and implement to effectively optimize your Dataflow costs. Examples of these guidelines include: 

Batch and streaming pipelines: 

• Consider keeping your Dataflow job in the same region as your IO source and destination services.
• Consider using GPUs for specialized use cases like deep-learning inference.
• Consider specialized machine types for memory- or compute-intensive workloads.
• Consider setting the maximum number of workers for your Dataflow job.
• Consider using custom containers to pre-install pipeline dependencies, and speed up worker startup time.
• Where necessary, tune the memory of your worker VMs.
• Reduce your number of test and staging pipelines, and remember to stop pipelines that you no longer need.

Batch pipelines:

• Consider FlexRS for use cases that have start time and run time flexibility.
• Run fewer pipelines on larger datasets. Starting and stopping a pipeline incurs some cost, and processing a very small dataset in a batch pipeline can be inefficient.
• Consider defining the maximum duration for your jobs, so you can eliminate runaway jobs, which keep running without delivering value to your business.

Summary

In this post, we introduced you to a set of knowledge and techniques that can help you understand the current and near-term costs associated with your Dataflow pipelines. We also shared a series of considerations that can help you optimize your Dataflow pipelines, not just for today, but also as your business evolves over time. We shared lots of resources with you, and hope that you find them useful. We look forward to hearing about the savings and innovative solutions that you are able to deliver for your customers and your business.