Volkswagen + Google Cloud: Using Machine Learning to Drive Smarter with Energy Efficient Cars
2672
Of your peers have already read this article.
2:30 Minutes
The most insightful time you'll spend today!
Volkswagen strives to design beautiful, performant, and energy efficient vehicles. This entails an iterative process where designers go through many design drafts, evaluating each, integrating the feedback, and refining.
For example, a vehicle’s drag coefficient—its resistance to air—is one of the most important factors of energy efficiency. Thus, getting estimates of the drag coefficient for several designs helps the designers experiment and converge toward more energy-efficient solutions. The cheaper and faster this feedback loop is, the more it enables the designers.
Unfortunately, estimating drag coefficient is an expensive and time-consuming operation that involves either a physical wind tunnel or a computationally intensive simulation. This can be a bottleneck in the feedback cycle.
For this reason, Volkswagen and Google Cloud decided to collaborate on a joint research project to investigate using machine learning (ML) to get fast and inexpensive estimates of the drag coefficient. In this post, we’ll explore the challenges and approaches undertaken in this project.
The core principles of the project were simple. First, we needed to collect a dataset of existing car designs and their respective drag coefficients. Then, we needed to create a representation of the various cars that would be suitable for ML. The next step was to train a deep learning model to predict the drag coefficient, and then, finally, we would use that model to efficiently estimate drag for any new design.
Representing three-dimensional car designs
Design software recreates a physical object as a three-dimensional triangle mesh made up of three types of objects—faces, edges, and vertices. Figure 1, below, shows such a mesh for an Audi S6. Faces are flat surfaces, such as the window in a car door. An edge is where two faces meet (e.g., the side of the door), and a vertex is where two or more edges meet, such as the corner of the door.
Car bodies, however, come in all shapes and sizes. A Volkswagen Golf economy model is very different from a Tiguan SUV, and a single vehicle can have both large smooth surfaces as well as areas with delicately designed features. Consequently, there can be a huge variety from one polygonal mesh to the next.
ML models need consistent representation in order to form robust generalized rules. With such a dramatic variance between each polygonal mesh, the models would be compromised and the results could have huge margins of error.
We needed to find a way to create simple meshes that capture the shape of the car but are still suited for ML models.
Representing a car with digital shrink wrapping
Rather than building a representation of each car from the ground up, we applied a “shrink wrapping” method for the 3D meshes. The principle is very similar to vacuum-sealing a cucumber. The cucumber is placed in a plastic bag and the air is then gradually removed until the bag fits tightly around it, capturing its shape.
Our approach works similarly: we start with a base mesh, a simple shape that corresponds to the plastic bag, and we deform it until it captures the shape of the target mesh. For our purposes, the base mesh is a simplified representation of a car and the target mesh is the particular car we are designing for at that moment. Such meshes can be defined, managed, and presented to ML models for training using the Tensorflow Graphics and trimesh libraries.
Our “shrink wrapping” method mainly works by iteratively minimizing a measure of distance (e.g., chamfer distance) between the two meshes. Additionally we can regularize our mesh to preserve certain qualities, like smoothness, in the resulting mesh. This iterative optimization is analogous to the vacuum pump, gradually shrinking and fitting the vertices of the mesh as closely as possible to the complex shape of the car. With shrink-wrapping, we are able to produce cleaner meshes that are more suitable to our estimation task. An example of such a procedure is shown in Figure 2.
How to train a model
Shrink-wrapping the 3D car designs was an important first step, but the work was far from over. Our next challenge was to build and test the machine learning algorithms.
We wanted our algorithms to estimate the drag coefficient as accurately and quickly as possible each time it looked at a new design. To do so, we had to train the ML models on existing data.
From publicly available datasets, we calculated the drag coefficients for 800 different car meshes, which we trained the models on. Then, we evaluated the trained models on a further 100 meshes, seeing how accurate their estimates were on new data.
As we worked through this training, we refined our approach. Initially, we tested models based on convolutional neural networks – similar to PointNet – that observed only the vertices, i.e., the fixed points in each mesh. But when we tested mesh-convolutional models – similar to FeastNet – we found a slightly different focus improved the accuracy of the estimates. Rather than focusing on vertices alone, these models looked at a mesh of vertices and how they relate to each other. These models placed each vertex in a richer context, leading to more accurate estimates when air-flow hit particularly subtle design features.
Working in parallel and at scale
To collaborate across time zones and two organizations, we’ve used the Google Cloud Vertex AI platform.
Vertex AI Workbench serves as a central hub to interact with other services and infrastructure on the Vertex AI platform. It enables quick experiments and preparation of training packages for resource-intensive ML model training jobs, all in a Python notebook environment for immediate execution of code. The notebook environments allow code-based interaction with other services on Google Cloud and ML tools such as Vertex AI Training and Vertex AI Pipelines.
The process of training a new model is a seamless one. First, a dataset is prepared and stored in Google Cloud Storage, usually with the help of Tensorflow Datasets. Then, for every ML model we want to test, we package and store the training code as a container image with Google Cloud Build and Container Registry. This ensures that every job is fully documented, including the provided parameters, training code package, logs from the training task, and resulting artifacts such as metrics and model files.
From there, we submit the model to the Vertex AI Training service, which provides easy access to large scale infrastructure and hardware accelerators, such as GPUs and TPUs, by simply defining resource needs when submitting a job. By using Vertex AI Training’s hyperparameter tuning feature, we can run experiments in parallel with multiple neural networks to find the right one for our purposes.
With Vertex AI Tensorboard, we can capture metrics and visualize the results of our experiments. These are readily available to anyone in the team, wherever they are in the world, for a wider discussion.
The first milestone
This joint research effort between Volkswagen and Google has produced promising results with the help of the Vertex AI platform. In this first milestone, the team was able to successfully bring recent AI research results a step closer to practical application for car design. This first iteration of the algorithm can produce a drag coefficient estimate with an average error of just 4%, within a second.
An average error of 4%, while not quite as accurate as a physical wind tunnel test, can be used to narrow a large selection of design candidates to a small shortlist. And given how quickly the estimates appear, we have made a substantial improvement on the existing methods that take days or weeks. With the algorithm that we have developed, designers can run more efficiency tests, submit more candidates, and iterate towards richer, more effective designs in just a small fraction of the time previously required.
Going forward, faster and more accurate estimates could even enable more automated searching for efficient designs, which would help both engineers and designers to hone in on the areas of the vehicle body where they could have the most impact. An important next step will be integrating the results into 3D design software to let designers benefit from the output and provide feedback.
As we continue, our focus is on improving the accuracy of the models. Firstly, we will build a larger, better quality dataset. Secondly, we will improve our shrink-wrapping algorithm to capture more details. Finally, we will enhance our existing models by experimenting with Vertex AI Neural Architecture Search to explore and experiment with different neural architecture options.
Moreover, we believe that our results for drag coefficient estimation is only a starting point for further exploration. There could potentially be numerous use cases in the space of physical simulations and assessments where cost and time savings could be achieved through ML-based estimators.
Acknowledgements
This work wouldn’t have been possible without the contributions from Volkswagen Data:Lab, Google Research, and Google Cloud. Thanks to Ahmed Ayyad, Dr. Andrii Kleshchonok, Dr. Daniel Weimer, Gülce Cesur, Henrik Bohlke, Andreas Müller from Volkswagen, Ameesh Makadia, Ph.D., and Carlos Esteves, Ph.D., from Google Research, and Daniel Holgate, Holger Speh, and Dr. Michael Menzel from Google Cloud.
NLP in Healthcare Can Unlock Clinical Insights beyond Typical Data Format Barriers
2897
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
Aches and pains. What, if anything, is the difference between those? And do you know a “myocardial infarction” from a “heart attack”? What about an “MI”? Is that shorthand, or part of the address of a hospital in Michigan?
For people, it can be easy to understand the distinctions and nuances between similar words, phrases, and concepts, even technical ones, like those found throughout the medical field. Yet even for the most advanced AI, the contextual clues that give humans accurate comprehension of words and images remain an elusive challenge. It’s a challenge well worth solving, though: As much as 80% of all healthcare data is said to be unstructured.
It’s the kind of complicated data management challenge that natural language processing was built to solve.
Healthcare and life sciences organizations are generating vast amounts of unstructured data as part of clinical and operational workflows, which presents an enormous opportunity to derive meaningful insights for medical research, population health, and patient care. For example, clinical notes and lab reports have useful, actionable information that, when unlocked, can help improve the overall quality of patient care, accelerate the discovery of new treatments, and increase the efficiency of healthcare delivery.
This approach is at the heart of Google Cloud’s Healthcare Natural Language API, in enabling healthcare organizations to build open, intelligent systems that unlock value from healthcare data. The open cloud approach enables our partners to innovate more easily, and scale more efficiently. We believe this approach will further advance interoperability—and ultimately lead to healthier and fuller lives.
Unlockinging value from clinical documents and research materials
Over the past two years, we have seen just how powerful AI can be in expediting drug discovery efforts for COVID-19, forecasting and modeling COVID-19 cases, and building better models for a host of public health measures. The opportunities extend well beyond battling the pandemic, too, to helping combat cancers, diabetes, and disabilities, and accelerating drug discovery.
As healthcare and life sciences organizations look to incorporate new data sources in their analytics and AI workflows, Google Cloud has been investing in providing open, flexible, and easy to use API services that customers and partners can integrate into their solutions, to accelerate their development with the power of Google’s AI technology.
The Cloud Healthcare Natural Language API is one such example, and aims to provide fully managed services that deliver the latest advances in natural language processing in an easy to use and easy to integrate manner. Healthcare organizations can then build intelligent systems to improve care and reduce cost while not having to worry about the complexities of the underlying and fast-changing technology, thus enabling more open innovation in the development of healthcare applications.
A number of healthcare innovators are exploring the potential for natural language processing.
“Patients come to Mayo Clinic with a history, and that history is well-documented, but often buried in clinical notes. Extracting information from unstructured healthcare data across thousands of patients is a complex problem,” says Vish Anantraman, M.D., Chief Technology Officer at Mayo Clinic. “Custom natural language processing solutions have a great potential to extract higher quality insights from these notes and to deliver more timely, and holistic patient care.”
The best insights can often be the unexpected ones, and that is precisely what Hackensack Meridian Health, in northern New Jersey, is looking for.
“Doctor’s notes are a rich space to create structured information from their natural workflow,” says Michael Draugelis, vice president for predictive health at Hackensack Meridian Health. “We are designing new AI-powered solutions to connect clinical teams, patients, and the community automatically from these insights—without creating cumbersome screen clicks and prompts. This automation allows our clinical teams to focus on connecting with the patient.”
Hospital leaders there are testing Google’s NLP API to gather information such as social determinants of health and behavioral health signals from large amounts of clinical notes, with approximately 35 million processed. Seeking to achieve the greatest value from natural language processing, the team at Hackensack Meridian Health have specifically focused on extracting information that is inherently not easy to capture in more traditional electronic health records.
“The extracted insights from the Google NLP API creates a foundational component to map clinical protocols, pathways, and outcomes, to better understand and improve patient care,” Draugelis says.
And at the National Institutes of Health and elsewhere, researchers are exploring how natural-language-derived variables could offer an additional predictive value over and above the Veteran Health Administrations’s structured EMR-based suicide prediction model.
To help healthcare organizations achieve goals like the ones above, we at Google draw on the expertise of tens of thousands of data scientists across the company who work every day on building better AI and decades of AI research in language understanding to power the development of services such as the Cloud Healthcare Natural Language API.
According to independent benchmarking of Cloud providers offering fully managed healthcare natural language service by tech analysts GigaOm, the Google Cloud Healthcare Natural Language API was among the most accurate in the industry, outperforming other service providers in terms of correctly classified medical entities and relationships, and with very few misclassifications.
Using AI to connect systems and enhance healthcare interoperability
As an industry, healthcare and life sciences organizations have been talking about the importance of data and data interoperability for a while. But our experiences from the past couple years have demonstrated that we cannot be fully prepared for the next global health crisis without greater connections within and between organizations.
Starting with the Healthcare Data Engine, organizations have been integrating and harmonizing data securely across many of their sources—patients, members, operations, research, and public databases—so they can quickly analyze it to get insights, and then make smarter, faster decisions.
This is the start of a broader vision for a new kind of healthcare and life sciences connected world where enterprises, institutions, and startups will securely collaborate to deliver on the next generation of care. Such a future relies on cloud-based solutions that are as open and flexible as they are user-friendly, compliance-ready, and secure.
We envision a future where healthcare organizations can seamlessly connect data from various systems, unlock the value from data regardless of source or format, and break down barriers in healthcare interoperability and AI to improve healthcare and save lives.
With these goals in mind, we continue to enhance the hybrid data clouds that customers are building to organize and analyze their information, and we and our partners continue to build on our data capabilities. Given the complexities both within the field and within each organization, we believe the greatest value comes from having partners and tools available to build the AI and NLP technologies most relevant to your unique needs.
5166
Of your peers have already watched this video.
24:00 Minutes
The most insightful time you'll spend today!
BK Medical’s Lift and Shift of SAP Environs on Google Cloud Results in Real-time Success
BK Medical is known for designing active imaging systems to help care providers visualize anatomy and provide real-time guidance to aid surgical interventions. After becoming an entity independent of the parent company, Analogic corporation, BK Medical decided to separate its SAP environs from the latter, and move into cloud. After assessing cloud vendors with Managecore, BK Medical chose Google Cloud as its managed service provider that could serve as a single source of truth and also help walk through the lift and shift of SAP ECC systems to the cloud. Watch the video to learn how the migration impacted BK Medical’s goals.
How a Mid-Sized Firm is Shrinking Inventory Carryovers 50% with AI
7457
Of your peers have already read this article.
5:30 Minutes
The most insightful time you'll spend today!
For more than 10 years, NMK Textile Mills has manufactured bed linens for major retailers in the United States and Canada. As e-commerce exploded, the company’s co-founder saw an opportunity to grow the business. So, in addition to manufacturing bed linens wholesale for retail customers, NMK Textile Mills reworked its complete supply chain to manufacture products for California Design Den, which became an e-commerce retailer selling its own fashion-forward products directly to consumers online.
With California Design Den’s push into e-commerce, it became apparent the SMB company (with about 250 global employees) faced the same tough supply chain questions as large retail customers, including maintaining enough inventory to meet customer demand in a timely, efficient way.
California Design Den depended upon a myriad of systems to track its complex forecasting and reordering processes. Team members typically planned inventory manually using desktop spreadsheet software, which could lead to excess inventory. Accurately forecasting demand and supply was essential to the company’s financial success—but it was also a challenge.
A couple years ago, California Design Den partnered with Pluto7, a technology solutions provider that offers a software as a service (SaaS) called Planning In A Box. Leveraging Google Cloud Platform machine learning and artificial intelligence, Planning In A Box intelligently helps predict demand and balances it with supply.
But that was just the beginning. “Along the way, we realized that to compete with larger retailers, make quicker decisions, and move faster, we needed to go further,” says Deepak Mehrotra, Co-founder and Chief Adventurer at California Design Den.
With guidance from Pluto7, California Design Den began migrating its database to Google Cloud Platform. Using Google BigQuery, Google Compute Engine, Google Cloud SQL, and Google Cloud Storage, and experimenting with Google Cloud Vision and Google Cloud AutoML, the company is reducing inventory carryovers by more than 50%, improving the accuracy of demand planning quarter over quarter, and gaining granular insights into how individual SKUs are performing.
“We would need an army of data scientists to make faster decisions on pricing and inventory levels. With Google Cloud Platform machine learning and artificial intelligence, we don’t need that. We can make much faster pricing decisions to optimize profitability and move inventory.”
—Deepak Mehrotra, Co-founder and Chief Adventurer, California Design Den
No need for army of data scientists
“Using Google Cloud Platform machine learning and AI was essential for California Design Den if it was to compete successfully with larger retailers,” Deepak says.
For example, tastes and fashions in bed linens can change quickly and consumer prices fluctuate. With more than 2,500 SKUs, it wasn’t possible for California Design Den’s team to continuously monitor product demand and experiment with competitive pricing in real time.
“We would need an army of data scientists to make faster decisions on pricing and inventory levels,” says Deepak. “With Google Cloud Platform machine learning and artificial intelligence, we don’t need that. We can make much faster pricing decisions to optimize profitability and move inventory.”
By integrating all its data onto Google Cloud Platform, California Design Den’s team gains deeper insights into product sales over time, which in turn helps the company improve demand planning by better determining which styles to manufacture and sell in the future.
“When experienced employees leave, their knowledge leaves with them. By having all data in one place, and with machine learning and AI, California Design Den can go back in its history, look at products made or sold years ago, and analyze product performance.”
—Deepak Mehrotra, Co-founder and Chief Adventurer, California Design Den
Merging visuals with data
Before Google Cloud Platform, team members had to dig through spreadsheets and run scenarios to get a sense of how particular products had sold. The next step was to perform keyword searches across the company’s photo library in the cloud to find each product’s image. From there, a team member would insert the product images into a presentation, along with relevant data points, to provide a report for stakeholders on how particular styles performed.
Today, California Design Den, with the help of Pluto7, is integrating its entire product image library with its database on Google Cloud Platform. Experimenting with Google Cloud Vision and Google Cloud AutoML, California Design Den is moving towards a day when team members can run sales scenarios and get deep background data on individual product performance while viewing images of the relevant products.
Merging product visuals with data will help designers and team members better understand sales patterns over time and in context. In the past, making correlations between things like which sheet colors sold well in California, compared to how the same sheet colors performed on the East Coast, was something that California Design Den employees primarily did in their heads.
“When experienced employees leave, their knowledge goes with them,” says Manjunath Devadas, Founder and CEO at Pluto7. “By having all data in one place, and with machine learning and AI, California Design Den can go back in its history, look at products made or sold years ago, and analyze product performance.”
“We are literally growing the complexity of our business on all levels, including designing, manufacturing, selling, reordering, inventory holding—everything.”
—Deepak Mehrotra, Co-founder and Chief Adventurer, California Design Den
Reimagining supply-demand balancing
Pluto7’s mission statement is to democratize supply demand balancing with machine learning and AI. California Design Den is a case in point, as the combination of Planning In A Box and Google Cloud Platform gives the company greater control over its destiny.
“Big retailers used to tell us what to manufacture and how much they would pay for it,” says Deepak. “That was our primary business, and if we didn’t accept the terms, a competitor would.” Today, in addition to continuing to make products for retailers, California Design Den can design, make, and sell a variety of designs for itself, including custom and limited-edition products, thanks to Google Cloud Platform and Pluto7 software offerings. The payoff is not only in having a more diversified business. California Design Den also receives more favorable profit margins by selling its own products.
“We are literally growing the complexity of our business on all levels, including designing, manufacturing, selling, reordering, inventory holding—everything,” Deepak says. “We can make smaller batches. We can connect directly with consumers. We can identify the missing pieces—what should we produce next, when, and how much? We otherwise couldn’t afford the level of talent it would take to do this.”
Cutting through the noise
Google and Pluto7 software helped California Design Den reduce inventory carryovers by more than 50%. Inventory tracking and distribution, along with insights and visibility into product sales, are faster, more efficient, and accurate. Google Cloud Platform flexible pricing, speed, reliability, security, and scalability enable California Design Den to stay relevant and be more competitive.
In addition to benefiting from Google Cloud Platform, California Design Den relies on G Suite—also part of Google Cloud—to enhance collaboration among its global teams. Previously, the company’s email server would sometimes crash, due to the heavy load of sharing product photos and other data. “Gmail and Google Drive handle the everyday demands on the business effortlessly and reliably,” Deepak says.
“Google machine learning and AI enable us to cut through all the noise from raw data, so we can see what’s important. We can focus on analytics to guide us to success today and in the future.”
—Deepak Mehrotra, Co-founder and Chief Adventurer, California Design Den
The company is exploring additional ways to leverage Google Cloud Platform in the near future. For example, one possibility is to import customer reviews from sites where products are sold into Google BigQuery, and to use that data to perform sentiment analysis via Google Cloud Natural Language. It could provide another valuable data source to help California Design Den’s team decide where to focus future designs.
“Google machine learning and AI enable us to cut through all the noise from raw data, so we can see what’s important,” Deepak says. “We can focus on analytics to guide us to success today and in the future.”
Google Cloud Next 21 for Data Analytics Unplugged
4954
Of your peers have already read this article.
4:30 Minutes
The most insightful time you'll spend today!
October 23rd (this past Saturday!) was my 4th Googlevarsery and we are wrapping an incredible Google Next 2021!
When I started in 2017, we had a dream of making BigQuery Intelligent Data Warehouse that would power every organization’s data driven digital transformation.
This year at Next, It was amazing to see Google Cloud’s CEO, Thomas Kurian, kick off his keynote with CTO of WalMart, Suresh Kumar , talking about how his organization is giving its data the “BigQuery treatment”.
AS I recap Next 2021 and reflect on our amazing journey over the past 4 years, I’m so proud of the opportunity I’ve had to work with some of the world’s most innovative companies from Twitter to Walmart to Home Depot, Snap, Paypal and many others.
So much of what we announced at Next is the result of years of hard work, persistence and commitment to delivering the best analytics experience for customers.
I believe that one of the reasons why customers choose Google for data is because we have shown a strong alignment between our strategy and theirs and because we’ve been relentlessly delivering innovation at the speed they require.
Unified Smart Analytics Platform
Over the past 4 years our focus has been to build industries leading unified smart analytics platforms. BigQuery is at the heart of this vision and seamlessly integrates with all our other services. Customers can use BigQuery to query data in BigQuery Storage, Google Cloud Storage, AWS S3, Azure Blobstore, various databases like BigTable, Spanner, Cloud SQL etc. They can also use any engine like Spark, Dataflow, Vertex AI with BigQuery. BigQuery automatically syncs all its metadata with Data Catalog and users can then run a Data Loss Prevention service to identify sensitive data and tag it. These tags can then be used to create access policies.
In addition to Google services, all our partner products also integrate with BigQuery seamlessly. Some of the key partners highlighted at Next 21 included Data Ingestion (Fivetran, Informatica & Confluent), Data preparation (Trifacta, DBT), Data Governance (Colibra), Data Science (Databricks, Dataiku) and BI (Tableau, PowerBI, Qlik etc).
Planet Scale analytics with BigQuery
BigQuery is an amazing platform and over the past 11 years we have continued to innovate in various aspects. Scalability has always been a huge differentiator for BigQuery. BigQuery has many customers with more than 100 petabytes of data and our largest customer is now approaching an exabyte of data. Our large customers have run queries over trillions of rows.
But scale for us is not just about storing or processing a lot of data. Scale is also how we can reach every organization in the world. This is the reason we launched BigQuery Sandbox which enables organizations to get started with BigQuery without a credit card. This has enabled us to reach tens of thousands of customers. Additionally to make it easy to get started with BigQuery we have built integrations with various Google tools like Firebase, Google Ads, Google Analytics 360, etc.
Finally, to simplify adoption we now provide options for customers to choose whether they would like to pay per query, buy flat rate subscriptions or buy per second capacity. With our autoscaling capabilities we can provide customers best value by mixing flat rate subscription discounts with auto scaling with flex slots.
Intelligent Data Warehouse to empower every data analyst to become a data scientist
BigQuery ML is one of the biggest innovations that we have brought to market over the past few years. Our vision is to make every data analyst a data scientist by democratizing Machine learning. 80% of time is spent in moving, prepping and transforming data for the ML platform. This also causes a huge data governance problem as now every data scientist has a copy of your most valuable data. Our approach was very simple. We asked:”what if we could bring ML to data rather than taking data to an ML engine?”
That is how BigQuery ML was born. Simply write 2 lines of SQL code and create ML models.
Over the past 4 years we have launched many models like regression, matrix factorization, anomaly detection, time series, XGboost, DNN etc. These models are used by customers to solve complex business problems simply from segmentation, recommendations, time series forecasting, package delivery estimation etc. The service is very popular: 80%+ of our top customers are using BigQueryML today. When you consider that the average adoption rate of ML/AI is in the low 30%, 80% is a pretty good result!
We announced tighter integration of BQML with Vertex AI. Model explainability will provide the ability to explain the results of predictive ML classification and regression models by understanding how each feature contributes to the predicted result. Also users will be able to manage, compare and deploy BigQuery ML models in Vertex; leverage Vertex Pipelines to train and predict BigQuery ML models.
Real-time streaming analytics with BigQuery
Customer expectations are changing and everyone wants everything in an instant: according to Gartner, by the end of 2024, 75% of enterprises will shift from piloting to operationalizing AI, driving a 5X increase in streaming data and analytics infrastructures.
The BigQuery’s storage engine is optimized for real-time streaming. BigQuery supports streaming ingestion of 10s of millions of events in real-time and there is no impact on query performance. Additionally customers can use materialized views and BI Engine (which is now GA) on top of streaming data. We guarantee always fast, always fresh data. Our system automatically updates MVs and BI Engine.
Many customers also use our PubSub service to collect real-time events and process these through Dataflow prior to ingesting into BigQuery. This is a streaming ETL pattern which is very popular. Last year,we announced PubSub Lite to provide customers with a 90% lower price point and aTCO that is lower than any DIY Kafka deployment.
We also announced Dataflow Prime, it is our next generation platform for Dataflow. Big Data processing platforms have only focused on horizontal scaling to optimize workloads. But we have seen new patterns and use cases like streaming AI where you may have a few steps in pipelines that perform data prep and then customers have to run a GPU based model. Customers want to use different sizes and shapes of machines to run these pipelines in the most optimum manner. This is exactly what Dataflow Prime does. It delivers vertical auto scaling with the right fitting for your pipelines. We believe this should lower costs for pipelines significantly.
With Datastream as our change data capture service (built on Alooma technology), we have solved the last key problem space for customers. We can automatically detect changes in your operational databases like MySQL, Postgres, Oracle etc and sync them in BigQuery.
Most importantly, all these products work seamlessly with each other through a set of templates. Our goal is to make this even more seamless over next year.
Open Data Analytics with BigQuery
Google has always been a big believer in Open Source initiatives. Our customers love using various open source offerings like Spark, Flink, Presto, Airflow etc. With Dataproc & Composer our customers have been able to run various of these open source frameworks on GCP and leverage our scale, speed and security. Dataproc is a great service and delivers massive savings to customers moving from on-prem Hadoop environments. But customers want to focus on jobs and not clusters.
That’s why we launched Dataproc Serverless Spark (GA) offering at Next 2021. This new service adheres to one of our key design principles we started with: make data simple.
Just like with BigQuery, you can simply RUN QUERY. With Spark on Google Cloud, you simply RUN JOB. ZDNet did a great piece on this. I invite you to check it out!
Many of our customers are moving to Kubernetes and wanted to use that as the platform for Spark. Our upcoming Spark on GKE offering will give the ability to deploy spark workloads on existing Kubernetes clusters.
But for me the most exciting capability we have is, the ability to run Spark directly on BigQuery Storage. BigQuery storage is highly optimized analytical storage. By running Spark directly on it, we again bring compute to data and avoid moving data to compute.
BigSearch to power Log Analytics
We are bringing the power of Search to BigQuery. Customers already ingest massive amounts of log data into BigQuery and perform analytics on it. Our customers have been asking us for better support for native JSON and Search. At Next 21 we announced the upcoming availability of both these capabilities.
Fast cross column search will provide efficient indexing of structured, semi-structured and unstructured data. User friendly SQL functions let customers rapidly find data points without having to scan all the text in your table or even know which column the data resides in.
This will be tightly integrated with native JSON, allowing customers to get BigQuery performance and storage optimizations on JSON as well as search on unstructured or constantly changing data structures.
Multi & Cross Cloud Analytics
Research on multi cloud adoption is unequivocal — 92% of businesses in 2021 report having a multi cloud strategy. We have always believed in providing customers choice to our customers and meeting them where they are. It was clear that all our customers wanted us to take our gems like BigQuery to other clouds as their data was distributed on different clouds.
Additionally it was clear that customers wanted cross cloud analytics not multi-cloud solutions that can just run in different clouds. In short, see all their data with a single pane of glass, perform analysis on top of any data without worrying about where it is located, avoid egress costs and finally perform cross cloud analysis across datasets on different clouds.
With BigQuery Omni, we deliver on this vision, with a new way of analyzing data stored in multiple public clouds. Unlike competitors, BigQuery Omni does not create silos across different clouds. BigQUery provides a single control plane that shows an analyst all data they have access to across all clouds. Analyst just writes the query and we send it to the right cloud across AWS, Azure or GCP to execute it locally. Hence no egress costs are incurred.
We announced BQ Omni GA for both AWS and Azure at Google Next 21 and I’m really proud of the team for delivering on this vision. Check out Vidya’s session and learn from Johnson and Johnson how they innovate in a multi-cloud world.
Geospatial Analytics with BigQuery and Earth Engine
We have partnered with our Google Geospatial team to deliver GIS functionality inside BigQuery over the years. At Next we announced that customers will be able to integrate Earth Engine with BigQuery, Google Cloud’s ML technologies, and Google Maps Platform.
Think about all the scenarios and use-cases your team’s going to be able to enable sustainable sourcing, saving energy or understanding business risks.
We’re integrating the best of Google and Google Cloud together to – again – make it easier to work with data to create a sustainable future for our planet.
BigQuery as a Data Exchange & Sharing Platform
BigQuery was built to be a sharing platform. Today we have 3000+ organizations sharing more than 250 petabytes of data across organizations. Google also brings more than 150 public datasets to be used across various use cases. In addition to this, we are also bringing some of the most unique datasets like Google Trends to BigQuery. This will enable organizations to understand in real-time trends and apply to their business problems.
I am super excited about the Analytics Hub Preview announcement. Analytics Hub will provide the ability for organizations to build private and public analytics exchanges. This will include data, insights, ML Models and visualizations. This is built on top of the industry leading security capabilities of BigQuery.
Breaking Data Silos
Data is distributed across various systems in the organization and making it easy to break the data silo and make all this data accessible to all is critical. I’m also particularly excited about the Migration Factory we’re building with Informatica and the work we are doing for data movement, intelligent data wrangling with players like Trifacta and FiveTran, with whom we share over 1,000 customers (and growing!). Additionally we continue to deliver native Google service to help our customers.
We acquired Cask in 2018 and launched our self service Data Integration service in Data Fusion. Now Fusion allows customers to create complex pipelines with just simple drag and drop. This year we focused on unlocking SAP data for our customers. We have launched various SAP connectors and accelerators to achieve this.
At GCP Next we also announced our BigQuery Migration service in preview. Many of our customers are migrating their legacy data warehouses and data lakes to BigQuery. BigQuery Migration Service provides end-to-end tools to simplify migrations for these customers.
And today, to make migrations to BigQuery easier for even more customers, I am super excited to announce the acquisition of CompilerWorks. CompilerWorks’ Transpiler is designed from the ground up to facilitate SQL migration in the real world and will help our customers accelerate their migrations. It supports migrations from over 10 legacy enterprises data warehouses and we will be making it available as part of our BigQuery Migration service in the coming months.
Data Democratization with BigQuery
Over the past 4 years we have focused a lot on making it very easy to derive actionable insights from data in BigQuery. Our priority has been to provide a strong ecosystem of partners that can provide you with great tools to achieve this but also deliver native Google capabilities.
With our BI engine GA announcement which we introduced in 2019, previewed earlier this year and showcased with tools like Microsoft PowerBI and Tableau, is now available for all to play with.
BigQuery + Data Studio are like peanut butter and Jelly. They just work well together. We launched BI Engine first with Data Studio and scaled it to all the users. More than 40% of our BigQuery customers use Data Studio. Once we knew BI Engine works extremely well we now have made it an integral part of BigQuery API and launched it for all our internal and partner BI tools.
We announced GA for BI Engine at Next 2021 but we were already GA with Data Studio for the past 2 years. We recently moved the Data Studio team back into Google Cloud making the partnership even stronger. If you have not used Data Studio, I encourage you to take a look and get started for free today here!!
Connected Sheets for BigQuery is one of my favorite combinations. You can give every business user in your organization the ability to analyze billions of records using standard Google Sheets experience. I personally use it everyday to analyze all our product data.
We acquired Looker in Feb 2020 with a vision of providing a semantic modeling layer to our customers with a governed BI solution. Looker is tightly integrated with BigQuery including BigQuery ML. Our latest partnership with Tableau where Tableau customers will soon be able to leverage Looker’s semantic model, enabling new levels of data governance while democratizing access to data.
Finally, I have a dream that one day we will bring Google Assistant to your enterprise data. This is the vision of Data QnA. We are in early innings on this and we will continue to work hard to make this vision a reality.
Intelligent Data Fabric to unify the platform
Another important trend that shaped our market is the Data Mesh. Earlier this year, Starburst invited me to talk about this very topic. We have been working for years on this concept, and although we would love for all data to be neatly organized in one place, we know that our customers’ reality is that it is not (If you want to know more about this, read about my debate on this topic with Fivetran’s George Fraser, a16z’s Martin Casado and Databricks’ Ali Ghodsi).
Everything I’ve learned from customers over my years in this field is that they don’t just need a data catalog or a set of data quality and governance tools, they need an intelligent data fabric. That is why we created Dataplex, whose general availability we announced at Next.
Dataplex enables customers to centrally manage, monitor, and govern data across data lakes, data warehouses, and data marts, while also ensuring data is securely accessible to a variety of analytics and data science tools. It lets customers organize and manage data in a way that makes sense for their business, without data movement or duplication. It provides logical constructs – lakes, data zones, and assets – which enable customers to abstract away the underlying storage systems to build a foundation for setting policies around data access, security, lifecycle management, and so on. Check out Prajakta Damle’s session and learn from Deutsche Bank how they are thinking about a unified data mesh across distributed data.
Closing Thoughts
Analysts have recognized our momentum and, as I look back at this year, I couldn’t thank our customers and partners enough for the support they provided my team and I across our large Data Analytics portfolio: in March, Google BigQuery was named a Leader in The Forrester Wave™: Cloud Data Warehouse, Q1 2021. And in June, Dataflow was named a Leader in The Forrester Wave™: Streaming Analytics, Q2 2021 report.
If you want to get a taste for why customers choose us over other hyperscalers or cloud data warehousing, I suggest you watch the Data Journey series we’ve just launched, which documents the stories of organizations modernizing to the cloud with us.
The Google Cloud Data Analytics portfolio has become a leading force in the industry and I couldn’t be more excited to have been part of it. I do miss you, my customers and partners, and I’m frankly bummed that we didn’t get to meet in person like we’ve done so many times before (see a photo of my last in-person talk before the pandemic), but this Google Next was extra special, so let’s dive into the product innovation and their themes.
I hope that I will get to see you in person next time we run Google Next!
ML Models Built on Google Cloud Solutions Help You Virtually Participate in National Muffin Day!
2958
Of your peers have already read this article.
2:00 Minutes
The most insightful time you'll spend today!
If you’re here you’re probably wondering: what on Earth is the connection between muffins and machine learning, and what is National Muffin Day? To understand this, let’s start with National Muffin Day: an annual holiday co-founded by Jacob and his friend Julia Levy in 2015 to bake muffins and raise money for homelessness. National Muffin Day will occur on Sunday, February 20 this year. For more information on how to participate in National Muffin Day (which involves delicious baked goods and donations to people in need), please see the information at the bottom of this post. Last year, a colleague connected Jacob with Sara, who had done several baking projects that used machine learning to generate new recipes. They decided to collaborate for this year’s National Muffin Day, adding a new muffin recipe created with the help of machine learning.
In this post, we’ll explain how Sara used Google Cloud to develop a new muffin recipe, show you how you can participate virtually in National Muffin Day, and of course—share the recipe.
Machine learning for muffins
At its core, machine learning is the process of finding patterns in data and using those patterns to make predictions on new data. After a lot of baking over the past few years, Sara learned that baking is also based on patterns. For example, the ratio of flour, fat, liquid, and sugar that make up a cookie is very different from the ratio of those ingredients for a bread, a pie crust, or a muffin. She used that discovery to create a recipe for a hybrid cake + cookie, and a cake filled with Maltesers. Next up: muffins!
The first step was figuring out how to translate the task of generating a new muffin recipe into a machine learning task. To solve this, she planned to use numerical data on the amounts of different ingredients in a muffin recipe to train the model. Sara considered two types of models for this task: classification and regression. A classification model would categorize muffin recipes into different muffin types based on their ingredient amounts, and a regression model would do the reverse: take a type of muffin and return the amount of each ingredient needed to make it. She decided to build a regression model, since it would be more fun for the model to return ingredient amounts, rather than tell you which type of muffin recipe you’re already making.
Implementing this first required identifying a few muffin categories and collecting recipe data. This presented a new challenge, since her previous baking models used categories for distinct baked goods (i.e. cakes, cookies, breads). After scouring through quite a few recipes, Sara discovered that many muffins fall into two types: those that use only traditional ingredients as their base (flour, sugar, butter, milk, etc.), and those that include an alternative ingredient, most commonly a pureed fruit, to make the base (like bananas, applesauce, or pumpkin). Using those two categories, the model would take the type of muffin as input and return the amounts of base ingredients required to make that recipe. Here, the inputs can be any values adding up to 100%:
The next step in the ML process was collecting recipe data to use for model training and narrowing down the ingredients used to train the model. Sara wanted the model to learn the combination of core ingredients that make up a muffin batter, rather than flavorings and additions like blueberries, vanilla extract, or chocolate chips. These tasty additions could be added after the model helped create the muffin batter. Once she gathered enough recipes, she removed extra ingredients for training purposes and converted ingredients from different recipes into the same unit (grams, milliliters, and teaspoons).
Building a muffin model with Vertex AI
Sara uploaded the muffin ingredient data into BigQuery, and then created a notebook instance in Vertex AI Workbench to analyze the data. With the new Workbench managed instances, you can interactively query BigQuery tables directly from your instance and copy the code to download your data to a notebook as a Pandas DataFrame:
From her notebook instance, Sara experimented with different ML frameworks and model types. She landed on a Scikit-learn regression model to solve this task, and to mimic a real-world production environment, decided to convert this workflow into a ML pipeline. Using the Kubeflow Pipelines SDK, she ran the following on Vertex Pipelines:
The first component reads the ingredient data from BigQuery and converts it into a Pandas DataFrame which is passed to the next pipeline step. In this step, we train a custom Scikit-learn model on the recipe data. Finally, this model is deployed to an endpoint in Vertex AI. To put it all together, Sara built a web app that allowed her to easily generate ingredient amounts for different muffin types. The web app uses the Vertex AI SDK to call the deployed model endpoint and return ingredient amounts.
The recipe
With a deployed recipe generation model, the only thing left to do was test recipes in the kitchen! Because the model only returns ingredient amounts, there were still many key human elements to complete the baking process: adding yummy additions to the core muffin batter, making adjustments to optimize taste, determining the method for adding ingredients, baking time, and more. After testing a few recipes generated by the model, we landed on a favorite which we’re very excited to share with you here.
Berry ML Muffins
Makes 12 muffins
Flour 285 grams (2 cups)
Granulated sugar 250 grams (1 cup)
Baking powder 2 teaspoons
Baking soda ¼ teaspoon
Salt ½ teaspoon
Cinnamon ½ teaspoon
Milk 170 ml (⅔ cup), room temperature
Butter 55 grams (¼ cup), melted and slightly cooled
Eggs 1 egg plus 1 egg white, room temperature
Canola or vegetable oil 50 grams (¼ cup)
Sour cream 50 grams (3 tablespoons + ¾ teaspoon), room temperature
Vanilla extract 1 ½ teaspoons
Blueberries or raspberries 240 grams (1 ½ cups)
Coarse sugar, like demerara or turbinado (optional for topping) 1 tablespoon
- Measure your three cold ingredients and allow them to come to room temperature: 1 egg + 1 egg white, sour cream, and milk.
- Preheat the oven to 375 F / 190 C. Line a 12-muffin tin with cupcake liners or lightly grease with baking spray.
- In a large bowl, whisk together flour, baking powder, baking soda, salt, and cinnamon. Set aside.
- Melt your butter in a medium heat proof bowl, and allow it to cool slightly for a few minutes. Whisk in sugar until combined. Then add egg, oil, and vanilla, milk, and sour cream and whisk until fully incorporated.
- Pour the wet ingredients into the dry ingredients, mixing with a spatula until just combined. Be careful not to overmix, it’s ok if there are a few lumps in your batter.
- Prepare your fruit. If you can’t decide whether to use blueberries or raspberries, divide your batter into two bowls and do both! Crush half of your fruit and fold it into the batter. Then mix in the remaining whole berries.
- Divide the mixture evenly into the muffin tin. Optionally (but extra tasty), sprinkle the tops of each muffin with about ⅛ teaspoon of coarse sugar. Turbinado or demerara sugar work well for this. This will caramelize and add a nice texture to the tops of your muffins.
- Bake at 375 for 22 – 24 minutes, or until a toothpick inserted in the center comes out clean. For best results, do a toothpick test in a few muffins since not all ovens have an even temperature throughout. Let the muffins cool in the muffin tin for a few minutes, then transfer to a wire rack to cool completely.
- Enjoy!
How can you participate in National Muffin Day?
Participation in National Muffin Day is as easy as 1-2-3!
- On February 20, Bake Muffins. It’s time to dust those muffin tins, grab your blueberries, chocolate chips, rhubarb, and favorite ingredients, and create some magical scrumdiddlyumptiousness! If you want to join Jacob in a virtual baking party, you can register here.
- Then, Give. In non-pandemic years, we asked our bakers to personally hand muffins to hungry folks in their cities. While this is a valuable and rewarding experience, the current state of Covid means this practice is still unsafe, so we request that you refrain from doing this. Instead, if it feels safe, we encourage you to take your delicious baked goods and donate them to local homeless shelters, which can distribute them to those in need. Alternatively, you can share your muffins with friends and families and then make a donation to an organization that benefits people experiencing homelessness, like the ones listed below in step 3.
- Share Your Muffin Pics on Social Media. We’d love to see your muffins! Share your pictures on Twitter or Instagram with the hashtags #givemuffins, or share them to our official Facebook Event page. For each individual baker who participates, we will make donations to Project Homeless Connect, which provides much needed resources to people experiencing homelessness in San Francisco, Family promise, which supports unhoused families nationwide, and Pine Street Inn which provides resources for people experiencing homelessness in Boston. Donations will be on a per-baker basis (with up to $80 donated per baker!), so please feel free to loop in your significant others, kids, nieces and nephews, roommates, friends, and anybody else with a giving spirit who loves deliciousness!
More Relevant Stories for Your Company
The True Story of How HotStar Broke a World-Record–Thanks to Firebase and Google BigQuery
Hotstar, India’s largest video streaming platform with 150 million monthly active users around the world, provides live-streaming of TV shows, movies, sports, and news on the go. By using a combination of Firebase products together, Hotstar safely rolled out new features to its watch screen during a major live-streaming event
Southwire Completes SAP Migration to Google Cloud as a First Step of its Tech Evolution
“Talk about tough times, right?” That’s how Dan Stuart, Senior Vice President of IT Services at Southwire Company, refers to the months following a December 2019 ransomware event, and the COVID crisis that began in spring of 2020. Those events hit just as the company was preparing for an overhaul
TPUs Can Cut Deep Learning Costs by upto 80%—and Other Things You Didn’t Know About TPUs
The Tensor Processing Unit (TPU) is a custom ASIC chip—designed from the ground up by Google for machine learning workloads—that powers several of Google's major products including Translate, Photos, Search Assistant and Gmail. Cloud TPU provides the benefit of the TPU as a scalable and easy-to-use cloud computing resource to
Google Cloud CCAI’s Support for the Public Sector Soars during the Pandemic
Scaling Virtual Support in the Pandemic Era: The AI Connection Since the early days of the pandemic, we’ve partnered with government organizations and academic institutions to serve communities at scale with Contact Center Artificial Intelligence (CCAI). I sat down with Bill MacKenzie, IT liaison for the Upper Grand School District
