Today, ML-driven innovation is fundamentally transforming computing, enabling entirely new classes of internet services. For example, recent state-of-the-art lage models such as PaLM and Chinchilla herald a coming paradigm shift where ML services will augment human creativity. All indications are that we are still in the early stages of what will be the next qualitative step function in computing. Realizing this transformation will require democratized and affordable access through cloud computing where the best of compute, networking, storage, and ML can be brought to bear seamlessly on ever larger-scale problem domains.

Today’s release of MLPerf™ 2.0 results from the MLCommons® Association highlights the public availability of the most powerful and efficient ML infrastructure anywhere. Google’s TPU v4 ML supercomputers set performance records on five benchmarks, with an average speedup of 1.42x over the next fastest non-Google submission, and 1.5x vs our MLPerf 1.0 submission. Even more compelling — four of these record runs were conducted on the publicly available Google Cloud ML hub that we announced at Google I/O. ML Hub runs out of our Oklahoma data center, which uses over 90% carbon-free energy.

Let’s take a closer look at the results.

Performance at scale…and in the public cloud

Our 2.0 submissions1, all running on TensorFlow, demonstrated leading performance across all five benchmarks. We scaled two of our submissions to run on full TPU v4 Pods. Each Cloud TPU v4 Pod consists of 4096 chips connected together via an ultra-fast interconnect network with an industry-leading 6 terabits per second (Tbps) of bandwidth per host, enabling rapid training for the largest models.

Hardware aside, these benchmark results were made possible in no small part by our work to improve the TPU software stack. Scalability and performance optimizations in the TPU compiler and runtime, including faster embedding lookups and improved model weight distribution across the TPU pod, enabled much of these improvements, and are now widely available to TPU users. For example, we made a number of performance improvements to the virtualization stack to fully utilize the compute power of both CPU hosts and TPU chips to achieve peak performance on image and recommendation models. These optimizations reflect lessons from Google’s cutting-edge internal ML use cases across Search, YouTube, and more. We are excited to bring the benefits of this work to all Google Cloud users as well.

Translating MLPerf wins to customer wins

Cloud TPU’s industry-leading performance at scale also translates to cost savings for customers. Based on our analysis summarized in Figure 3, Cloud TPUs on Google Cloud provide ~35-50% savings vs A100 on Microsoft Azure (see Figure 3). We employed the following methodology to calculate this result:2

We compared the end-to-end times of the largest-scale MLPerf submissions, namely ResNet and BERT, from Google and NVIDIA. These submissions make use of a similar number of chips — upwards of 4000 TPU and GPU chips. Since performance does not scale linearly with chip count, we compared two submissions with roughly the same number of chips.

To simplify the 4216-chip A100 comparison for ResNet vs our 4096-chip TPU submission, we made an assumption in favor of GPUs that 4096 A100 chips would deliver the same performance as 4216 chips.

For pricing, we compared our publicly available Cloud TPU v4 on-demand prices ($3.22 per chip-hour) to Azure’s on-demand prices for A1003 ($4.1 per chip-hour). This once again favors the A100s since we assume zero virtualization overhead in moving from on-prem (NVIDIA’s results) to Azure Cloud.

The savings are especially meaningful given that real-world models such as GPT-3 and PaLM are much larger than the BERT and ResNet models used in the MLPerf benchmark: PaLM is a 540 billion parameter model, while the BERT model used in the MLPerf benchmark has only 340 million parameters — a 1000x difference in scale. Based on our experience, the benefits of TPUs will grow significantly with scale and make the case all the more compelling for training on Cloud TPU v4.

Have your cake and eat it too — a continued focus on sustainability

Performance at scale must take environmental concerns as a primary constraint and optimization target. The Cloud TPU v4 pods powering our MLPerf results run with 90% carbon-free energy and a Power Usage Efficiency of 1.10, meaning that less than 10% of the power delivered to the data center is lost through conversion, heat, or other sources of inefficiency. The TPU v4 chip delivers 3x the peak FLOPs per watt relative to the v3 generation. This combination of carbon-free energy and extraordinary power delivery and computation efficiency makes Cloud TPUs among the most efficient in the world.4

Making the switch to Cloud TPUs

There has never been a better time for customers to adopt Cloud TPUs. Significant performance and cost savings at scale as well as a deep-rooted focus on sustainability are why customers such as Cohere, LG AI Research, Innersight Labs, and Allen Institute have made the switch. If you are ready to begin using Cloud TPUs for your workloads, please fill out this form. We are excited to partner with ML practitioners around the world to further accelerate the incredible rate of ML breakthroughs and innovation with Google Cloud’s TPU offerings.

1. MLPerf™ v2.0 Training Closed. Retrieved from https://mlcommons.org/en/training-normal-20/ 29 June 2022, results 2.0-2010, 2.0-2012, 2.0-2098, 2.0-2099, 2.0-2103, 2.0-2106, 2.0-2107, 2.0-2120. The MLPerf name and logo are trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use is strictly prohibited. See www.mlcommons.org for more information.
2. MLPerf v1.0 and v2.0 Training Closed. Retrieved from https://mlcommons.org/en/training-normal-20/ 29 June 2022, results 1.0-1088, 1.0-1090, 1.0-1092, 2.0-2010, 2.0-2012, 2.0-2120.
3. ND96amsr A100 v4 Azure VMs, powered by eight 80 GB NVIDIA Ampere A100 GPUs (Azure’s flagship Deep Learning and Tightly Coupled HPC GPU offering with CentOS or Ubuntu Linux) is used for this benchmarking
4. Cost to train is not an official MLPerf metric and is not verified by MLCommons Association. Azure performance is a favorable estimate as described in the text, not an MLPerf result. Computations are based on results from MLPerf v2.0 Training Closed. Retrieved from https://mlcommons.org/en/training-normal-20/ 29 June 2022, results 2.0-2012, 2.0-2106, 2.0-2107.

With the rise of technologies like smartphones, retailers have felt the pressure to meet evolving consumer needs and expectations. Seven-Eleven Japan(“SEJ”) has long been on the forefront of this thanks to the way they develop and invest in IT. However, in recent years, Japan’s leading convenience store chain has struggled to maintain its complex legacy systems at the rate needed to keep up with today’s rapid digitization, spurred on by the increasing proliferation of smartphones and an IT vendor-dependent structure.  

Legacy systems limiting real-time responsiveness and innovation 

Since its early days, SEJ has been proactive in adopting information technology, mainly relying on technology solutions from Japan’s leading vendors. But as the systems have grown, key business issues have been resolved using a vendor-dependent structure rather than being driven by SEJ’s own needs.  

Datasets and business logic were combined and built into legacy environments, gradually leading to data silos. As a result, data was distributed across multiple systems, causing a variety of problems, including the inability to efficiently retrieve data when needed, delays in accessing data collected in individual stores, and difficulties taking measurements at the right time in business operations that require real-time responsiveness.

Connecting different systems also takes time and money, and the lead time for introducing new services—from planning to development and launch—has been longer than expected. 

To solve these problems, SEJ’s IT department built “Seven Central”—a new platform for practical data use launched in 2020 to support the company’s future IT strategies and digital transformation initiatives.

At its core, Seven Central’s ultimate purpose is to allow real-time data views. Versatile, real-time datasets—such as point-of-sale (POS) data from 7-Eleven stores—are consolidated into a centralized location in the cloud. They created a simple data mart that provides data via an API to enable them to respond more quickly to requests from individual departments. 

“In such uncertain times, it’s vital to use data to make quick decisions,” says Izuru Nishimura, Executive Officer and Head of ICT Department. “Each department across the entire company will be able to gain an immediate understanding of the situation based on the most up-to-date data and respond accordingly. This is why we built Seven Central.”

Google Cloud selected to help SEJ build and grow their data cloud

Today’s rapidly changing business environment has also highlighted the risk of IT support becoming a bottleneck. The long-term strategy is to gradually expand the datasets managed and collected in Seven Central according to business needs. 

In the first phase, SEJ collected POS data from all 21,000+ stores to enable real-time analysis. Moving forward, they would like to collect other relevant data—for example, unstructured data, such as images and videos, or master datasets that are currently stored externally. 

Google Cloud was already a top contender when SEJ started developing Seven Central in 2019. They compared various public cloud services besides Google Cloud, focusing on three main capabilities.

“We placed particular emphasis on service scalability to drive future digital transformation; security when handling data, which is the lifeline of our company; and finally, openness,” says Nishimura. He emphasizes that openness was perhaps the most important factor for choosing Google Cloud. Breaking away from the negative aspects of an entirely vendor-dependent system enabled them to build an agile development system with multiple vendors. 

Google Cloud technologies including BigQuery and API management platform, Apigee, play a vital role in Seven Central. BigQuery’s high-speed processing at petabyte scale and fully managed infrastructure helped keep costs low during development and verification.   

“Data is stored in a way that allows you to share it easily across organizations, which helps solve the issue of data silos from the perspective of scalability. I also like the fact there are some interesting features that could be used in the future—like BigQuery ML, which enables machine learning on BigQuery,” says Nishimura. 

Apigee allows SEJ to separate datasets and business logic, which is one of the key points of Seven Central. While the trend these days is to standardize interfaces using an API, the reality tends to involve many different APIs rather than the introduction of one unified API. With Apigee, SEJ provides a single unified API for all of its data cloud, and they can now understand what data is used thanks to Apigee’s API usage visualizations.  

“Right now, we collect data from all 21,000+ stores,” says Nishimura. “But in anticipation of a future expansion in business operations, we have designed a system that can scale up and run without issue, even if we were to have 30,000 stores, with 1,000 customers per store per day, purchasing five items per person.”

Real-time insights with BigQuery and Cloud Spanner

Google Cloud partner Cloud Ace came on board early in the planning phases. Based on their recommendations, SEJ decided to continue making full use of BigQuery to analyze data collected from all 21,000+ stores throughout Japan, while also using Cloud Spanner’s availability, near-unlimited scalability and transactional consistency to help achieve the real-time results needed for the project.

“Given that both the data and the regularity with which it is accessed are expected to steadily increase in the future, we chose Cloud Spanner as backend storage for data delivery via API. We consider it a good choice,” says Shota Kikuchi, General Manager, Consulting Department, Technology Division, Cloud Ace Co., Ltd.

Finally, they chose to use Google Cloud’s Stream Analytics Solutions messaging service for collecting POS data in real time, which can then be put to immediate use with Cloud Spanner and BigQuery. 

High-speed responses exceed targets and create new value 

Seven Central went live in September 2020 with surprising results. 

They initially set a target time of one hour from when a customer makes a purchase to the point when Seven Central can use that data. But when the final system was first tried—it took barely a minute. Moving forward they estimate that the latest inventory data from the service side will become available within a few minutes of being added to the system.  

“This is real innovation, and I must admit that I am quite surprised. As well as being able to solve existing issues, we also hope it will lead to new improvements and services that have been unimaginable up until now,” says Nishimura.

The team hopes to roll out the Seven Central platform in all companies affiliated with Seven & i Holdings—not just SEJ. They also plan to explore Google Cloud AI and machine learning technologies to take on challenges in new areas. For example, they are investigating the idea of clustering individual stores using BigQuery ML.

Seven Central has already attracted attention from many departments and received a lot of requests. Nishimura and his team say they hope to continue to grow Seven Central while still observing their fundamental principles—not including business logic, maintaining real-time results, and staying true to the uniqueness of SEJ.

Learn more about Google Cloud smart analytics solutions.

Documents like payslips and W2s are crucial to processes such as employment and income verification for mortgage loans, personal loans, personal finance, and benefits processing. Unfortunately, efficiently extracting data from these documents at scale can be challenging and time-consuming, with many organizations relying on manual examination of documents or automated approaches that don’t adequately capture the document data needed for given tasks. Google Cloud built Document AI to remove these barriers, empowering customers to deploy powerful machine learning models to more quickly process documents, save money, and discover insights. We’re excited to expand Document AI’s capabilities with the recent release of improved pre-trained models for W2s and payslips, built on Document AI Workbench.

Pre-trained models let developers focus on core application logic and leave the complex task of information extraction from the documents to Google’s AI technology. In many cases, the primary driver for automated data extraction is operational efficiency and cost savings, but Document AI can also open new possibilities. For example, a financial services company might use Document AI to enable fully self-serve loan applications on mobile devices, helping the organization to differentiate itself with simple, fast customer experiences.

We’ve heard from customers that more granular entity extraction from W2 and payslip documents is particularly important, with organizations requiring support for a wider variety of layouts and formats. The recent launch of the stable release of these pretrained models addresses these requests.

Here is what is new with W2 parser:

• The parser improves accuracy and entity specificity thanks to the ability to break down long entities such as addresses into fine-grained sub-entities like StreetAddressOrPostalBox, AdditionalStreetAddressOrPostalBox, City, State, and ZIP code.
• It can handle a wider variation of W2 forms, including multi-copies (2,3,4-ups) issued by various payroll vendors. The model is not limited to specific tax years, which means it should be able to process W2 for 2022 or beyond provided there are not significant changes to the format.
• It introduces eight new entities for Box 12 that represent both codes and values, enriching understanding of the various taxable and non-taxable components of the W2 recipient’s income.

Here is what is new with Payslip parser:

• Bonus, commissions, holiday, overtime, regular pay, and vacation are now part of earning_item/earning_this_period and earning_item/earning_ytd. The parser captures types of earnings beyond those categories, and maps them to their respective earning rates, hours, and pay (both for the period and year-to-date). This helps in building a more detailed understanding of the components of the payslip recipient’s income
• The parser now returns year-to-date and current-period taxes and deductions.
• Direct deposits are linked to corresponding bank account numbers.
• The parser now returns page numbers, state and federal tax exemptions, and filing statuses.

While these parsers have become more useful out of the box, with this release, the ability to uptrain makes them easy to modify as new needs arise. Uptraining lets developers further improve the accuracy of these models and extract additional fields with minimal development work. It also lets developers customize existing parsers to support new document types that are similar. For example, the parser is trained on U.S. data and could be uptrained to create a payslip parser for the U.K.

We’re pleased that parsers are already making a difference for customers. Bryan Jackson, CTO at lending automation firm Gateless, said, “High accuracy data extraction is critical to the success of our Smart Underwrite solution, and Document AI provided better results than competitors. Using the latest W2 & Payslip pretrained parsers, we saw a 48% increase in performance on pay stubs and a 15% performance improvement in W2s. The ability to easily uptrain models as new document variations are introduced ensures we continue to deliver optimal outcomes for our customers.”

Additional pre-trained models available as release candidates include parsers for 1040, 1099R, 1120, and 1120S documents. Check for details here. To learn more, talk to a Google Cloud sales executive about how Document AI can help your business, and check out our Document AI breakout session from Google Cloud Next ’22.

Retailers are becoming more skilled at making individual customers feel heard and valued. This is a necessity given the fact that 66% of respondents to a McKinsey survey stated that they expect email marketing messages to be tailored to their needs. While marketing personalization expertise is growing, it’s still difficult to manage, especially at scale.

Ocado Retail, one of the world’s largest dedicated online grocery retailers, delivering to over 645,000 customers in the U.K., has made personalization integral to its success. 

Let’s look at how Ocado retail worked with Google Cloud and partner Cognizant to develop a new data platform to power its personalization efforts from the ground up. 

Unifying data for more holistic, powerful personalization

To achieve its goals of personalization at scale, Ocado Retail needed a central data warehouse that could turn all forms of merchandising, advertising, business, and customer intelligence data into actionable insights. It wanted a means to accelerate customer segment identification, as well as the ideation and launch of relevant campaigns. 

“We standardized on Google Cloud, including BigQuery, as the foundation for our data platform because we knew it was the right solution for now and the future,” says Kieren Johnson, Head of IT at Ocado Retail. “We have an incredibly lean team and we needed a partner with exceptional expertise to help build an ambitious enterprise data warehouse to provide powerful insights. Cognizant was also the clear choice to help us get there.”

Cognizant worked closely with Ocado Retail to make sure its expertise in Google Cloud and other technologies aligned with Ocado Retail’s vision to drive more advanced personalization at higher scales using machine learning. The partner helped build the foundation on BigQuery, and then incorporated other Google Cloud tools such as Cloud Run, Vertex AI, and Vertex AI Natural Language to provide no-ops, all-code warehousing, and analytics capabilities.

Cognizant also took advantage of the Google Cloud Partner Success Services (PSS) program to ensure best practices were being followed throughout the project. PSS provided advisory services that guided Cognizant through the highly complex process of building the new data warehouse for Ocado Retail on Google Cloud.

Building the enterprise data platform in this way allows Ocado Retail to leverage the full power of cloud-based analytics while maintaining a lean team. It also allows the company to greatly scale up its personalization efforts.

Making customers feel valued at every touch

The work Ocado Retail has done with Google Cloud and Cognizant has positioned it to make its growing customer base feel valued, understood, and supported in every interaction. Before launching the project with Cognizant and Google Cloud, Ocado Retail was only able to run a couple of campaigns per week and knew it lacked optimal insight into each campaign’s efficacy. 

“We now run 10 times the number of campaigns we used to with the help of the data platform Cognizant built on Google Cloud,” says Kieren. “We run multiple campaigns every day for different customer segments, and all of that increased activity is entirely driven by data insights. The positive impacts on our marketing and customer service performance have been clear. We’re now working to expand what we do.”

Ocado Retail has enjoyed solid growth since the new data platform went live, including a 13% rise in active customers during fiscal year 2022, and has also seen a reduction in churn. It attributed these improvements to being better able to tailor products and communications to specific customer preferences.

Throughout the project, Cognizant supported data clean up while maximizing the scalable, flexible, and future-proofed data analytics infrastructure offered by Google Cloud.

Increasing data-driven actions

Ocado Retail plans to provide more data-driven insights to its commercial suppliers through a product called Beet Insights. So far, Beet Insights offers suppliers with intelligence about how their products are performing on the shelves. The result has been improving the role data plays throughout the supply chain, from production to purchase and beyond.

“By putting real-time insights about costs, marketing spend, and supply-funded activities into the hands of our commercial team and buyers, we are better positioned to improve our profits,” says Kieren. “At the same time, building data analytics into every part of our business will allow us to build on our personalization efforts.”

From the project’s inception, Ocado Retail ensured that the platform would be scalable and dynamic. Now, it is working to feed more data sources and streams into the warehouse to accelerate time to insights. Ocado Retail believes this next step in the evolution of the platform will unlock even more opportunities to initiate high-impact programs that transform personalization and every customer interaction.

Learn more about what Google Cloud and partners like Cognizant can do for your customer intelligence and personalization.

Editor’s note: Since launching its Spaces feature, Twitter has demonstrated that hearing people’s voices can bring conversations on Twitter to life in a completely new way. Next, it aimed to make it easier for customers to join and listen to live conversations they personally care about. In this blog, we learn how the Twitter Spaces Engineering team is bringing this vision to life with AutoML, powering a new ML heuristic which serves personalized recommendations to Twitter customers. The authors would like to thank Chuan Lu, Joe Balistreri, Chen-Rui Chou, Pablo Jablonski, Alberto Parrella, Pradip Thachile and Sam Lee from Twitter, as well as Helin Wang from Google, for contributions to this blog.

Since Twitter introduced Spaces in 2020 to enable live audio conversations on its platform, the Twitter Spaces Engineering team has been continually testing, building, and updating this feature in the open. Today, anyone can join, listen, and speak in a Space on Twitter, and the feature’s popularity has taken off. But this success also poses a challenge: with millions of people creating and joining Spaces at any time, how can they find the Spaces to engage with while they’re happening? Taking this as an opportunity to further improve the experience of its customers, Twitter has turned to machine learning (ML) and cloud technology for answers.

“ML fits into the natural progression of Twitter consumer and revenue product building, especially for a product feature such as Spaces,” explains Diem Nguyen, Senior Machine Learning Engineer and Data Scientist at Twitter. “We launched Spaces with a base-line algorithm using the ‘most popular’ heuristic which assumes that if a Space is popular, there’s a good chance you’d like it too. But our aim is to leverage ML to surface the most interesting and relevant Spaces to a particular Twitter customer, making it easier for them to find and join the conversations they personally care about. This is a complex functionality that Google Cloud ML capabilities help us to enable.”

Setting the stage for building new features with limited ML resources

While looking for the right tools to power this vision, Nguyen and her team started evaluating in December 2021 whether the Vertex AI platform and AutoML in particular could solve challenges observed when they first started building Spaces. These included a lack of dedicated ML resources to build and deploy the product feature, and the need to work on a multi-cloud environment.

“We had three key questions in mind during our assessment,” Nguyen explains. “Can we realistically deploy the AutoML model off-platform? Once deployed, can it solve for the request load that we get from the service we’re serving (in this case, the Spaces tab)? And finally, can we develop and maintain such a solution without a dedicated team of ML experts for this project?” The answer to all three questions was yes.

Positive answers motivated the Spaces Engineering team to take the solution to production in February 2022. “We started using AutoML Tables to train high-accuracy models with minimal ML expertise or effort, alleviating our resource constraint,” says Nguyen of the results. “Soon AutoML also stood out for its high performance and for supporting easy deployment beyond the Google Cloud Platform, making it ideal for this project hosted in a multi-cloud environment.”

Increasing customer engagement at speed with accurate ML predictions

With a classification model in place to predict the probability of user engagement in a particular Space, Twitter now aims to optimize its model with aggregated data around Twitter features that can help it better understand customer preferences. For example, if a customer has historically engaged with a particular topic and a new Space matches that topic, the ML model increases the score of that Space being served to that user on the Spaces tab.

Because Spaces are live audio conversations, the Spaces tab needs to be ranked to customers in near real time so they don’t miss out. With this in mind, Twitter’s model currently performs 900 queries per second on the Spaces tab, and evaluates 50,000 candidates per second. Meanwhile, 99% of these requests are faster than 100 milliseconds, and 90% of requests are faster than 50 milliseconds.

To measure the success of this project, Nguyen’s team conducted A/B experiments around key customer engagement metrics–A stands for the ‘most popular’ heuristic previously in production, and B is the new AutoML model which seeks to personalize Spaces recommendations to the interests of individual Twitter users. Three months into the project, the numbers were encouraging. “After deploying our AutoML Tables solution we saw an increase of 1.96% in Spaces daily active customers, which is one of our key metrics. We also noticed an increase of 1.99% in Spaces join in rates, and an increase of 8.42% in user clicks to explore a Space,” Nguyen shares. “These are positive signals that users are now engaging more with the Spaces tab service on the Twitter app, which is exactly what we set out to do with this project.”

Powering new use cases with hands-off ML frameworks

With this first solution running in production to improve the performance of the Spaces tab, Nguyen starts to ask how else it might support the experience of Twitter users moving forward. “The Spaces tab is a small surface on the Twitter app. With our current ML solution we’re some distance away from serving our home tab traffic, which is where a lot of our traffic happens and therefore would involve a much bigger-scale operation. Getting there will take some work but we’re evaluating the possibility of optimizing our model performance for this in collaboration with Google Cloud,” says Nguyen.

“As a product-led company, we focus on continually improving the customer experience and we want to iterate faster to get to that point. AutoML brings that value to our product teams because it is so hands-off. You don’t need to write any model code in order to reap the benefits from this machine learning framework; AutoML automatically experiments with many different model architectures and comes up with a state-of-the-art model that addresses your needs. So while it is not a one-size-fits-all solution, it is a great solution with the potential to power many more Twitter use cases,” she concludes.