The latest round of MLPerf benchmark results have been released, and Google’s TPU v4 supercomputers demonstrated record-breaking performance at scale. This is a timely milestone since large-scale machine learning training has enabled many of the recent breakthroughs in AI, with the latest models encompassing billions or even trillions of parameters (T5, Meena, GShard, Switch Transformer, and GPT-3). 

Google’s TPU v4 Pod was designed, in part, to meet these expansive training needs, and TPU v4 Pods set performance records in four of the six MLPerf benchmarks Google entered using TensorFlow and JAX. These scores are a significant improvement over our winning submission from last year and demonstrate that Google once again has the world’s fastest machine learning supercomputers. These TPU v4 Pods are already widely deployed throughout Google data centers for our internal machine learning workloads and will be available via Google Cloud later this year.

Figure 1: Speedup of Google’s best MLPerf Training v1.0 TPU v4 submission over the fastest non-Google submission in any availability category – in this case, all baseline submissions came from NVIDIA. Comparisons are normalized by overall training time regardless of system size. Taller bars are better.¹

Let’s take a closer look at some of the innovations that delivered these ground-breaking results and what this means for large model training at Google and beyond. 

Google’s continued performance leadership

Google’s submissions for the most recent MLPerf demonstrated leading top-line performance (fastest time to reach target quality), setting new performance records in four benchmarks. We achieved this by scaling up to 3,456 of our next-gen TPU v4 ASICs with hundreds of CPU hosts for the multiple benchmarks. We achieved an average of 1.7x improvement in our top-line submissions compared to last year’s results. This means we can now train some of the most common machine learning models in a matter of seconds.

Figure 2: Speedup of Google’s MLPerf Training v1.0 TPU v4 submission over Google’s MLPerf Training v0.7 TPU v3 submission (exception: DLRM results in MLPerf v0.7 were obtained using TPU v4). Comparisons are normalized by overall training time regardless of system size. Taller bars are better. Unet3D not shown since it is a new benchmark for MLPerf v1.0.²

We achieved these performance improvements through continued investment in both our hardware and software stacks. Part of the speedup comes from using Google’s fourth-generation TPU ASIC, which offers a significant boost in raw processing power over the previous generation, TPU v3. 4,096 of these TPU v4 chips are networked together to create a TPU v4 Pod, with each pod delivering 1.1 exaflop/s of peak performance.

Figure 3: A visual representation of 1 exaflop/s of computing power. If 10 million laptops were running simultaneously, then all that computing power would almost match the computing power of 1 exaflop/s.

In parallel, we introduced a number of new features into the XLA compiler to improve the performance of any ML model running on TPU v4. One of these features provides the ability to operate two (or potentially more) TPU cores as a single logical device using a shared uniform memory access system. This memory space unification allows the cores to easily share input and output data – allowing for a more performant allocation of work across cores. A second feature improves performance through a fine-grained overlap of compute and communication. Finally, we introduced a technique to automatically transform convolution operations such that space dimensions are converted into additional batch dimensions. This technique improves performance at the low batch sizes that are common at very large scales.

Enabling large model research using carbon-free energy

Though the margin of difference in topline MLPerf benchmarks can be measured in mere seconds, this can translate to many days worth of training time on the state-of-the-art models that comprise billions or trillions of parameters. To give an example, today we can train a 4 trillion parameter dense Transformer with GSPMD on 2048 TPU cores. For context, this is over 20 times larger than the GPT-3 model published by OpenAI last year. We are already using TPU v4 Pods extensively within Google to develop research breakthroughs such as MUM and LaMDA, and improve our core products such as Search, Assistant and Translate. The faster training times from TPUs result in efficiency savings and improved research and development velocity. Many of these TPU v4 Pods will be operating at or near 90% carbon free energy. Furthermore, cloud datacenters can be ~1.4-2X more energy efficient than typical datacenters, and the ML-oriented accelerators – like TPUs – running inside them can be ~2-5X more effective than off-the-shelf systems. 

We are also excited to soon offer TPU v4 Pods on Google Cloud, making the world’s fastest machine learning training supercomputers available to customers around the world. Cloud TPUs support leading frameworks such as TensorFlow, PyTorch, and Jax, and we recently released an all-new Cloud TPU system architecture that provides direct access to TPU host machines, greatly improving the user experience. 

Want to learn more? 

Please contact your Google Cloud sales representative to request early access to Cloud TPU v4 Pods. We are excited to see how you will expand the machine learning frontier with access to exaflops of TPU computing power!

^{1. All results retrieved from www.mlperf.org on June 30, 2021. MLPerf name and logo are trademarks. See www.mlperf.org for more information. Chart uses results 1.0-1067, 1.0-1070, 1.0-1071, 1.0-1072, 1.0-1073, 1.0-1074, 1.0-1075, 1.0-1076, 1.0-1077, 1.0-1088, 1.0-1089, 1.0-1090, 1.0-1091, 1.0-1092.
2. All results retrieved from www.mlperf.org on June 30, 2021. MLPerf name and logo are trademarks. See www.mlperf.org for more information. Chart uses results 0.7-65, 0.7-66, 0.7-67, 1.0-1088, 1.0-1090, 1.0-1091, 1.0-1092.}

Contact Center AI (CCAI) brings Google’s innovation in conversational AI to solve the most challenging customer service needs while lowering operational costs. More than a thousand customers have deployed CCAI and are steadily turning it on to power their production contact centers.

Today, we’re excited to announce that we’ve made CCAI even stronger with Agent Assist for Chat, now in public preview.

Agent Assist provides your human agents with continuous support during their calls and now chats by identifying the customers’ intent and providing them with real-time recommendations such as articles and FAQs as well as responses to customer messages to more effectively resolve the conversation.

Customers using Agent Assist for Chat have been able to manage up to 28% more conversations concurrently, while also driving up customer satisfaction by 10%. Additionally, we’ve seen them respond up to 15% faster to chats, reducing chat abandonment rates and solving more customer problems.

Agent Assist provides two key components to help agents manage conversations better: 

• Smart Reply provides response suggestions to agents so they can quickly and appropriately respond to customer messages. These suggestions can be taken from your top performing agents as well as modified even further to ensure suggestions properly reflect the tone and voice of your brand. Agent Assist learns when and what recommendations to make by building a custom model that’s trained on your (and only your) data.
• Knowledge Assist unlocks the power of your knowledge base to provide articles and FAQ suggestions to agents in real-time as the conversation progresses. When using Knowledge Assist, agents no longer need to make the customer wait while they navigate multiple applications and data to find the resolution to the customer’s issue — the answer is delivered right to them.  

“We’ve been very impressed by the chat capabilities of Agent Assist,” said Chris Smith, Vice President of Digital Service at Optus, one of the largest telecommunications companies in Australia

Optus has been using CCAI Dialogflow CX to send queries to virtual agents and sees great potential to use Agent Assist to provide recommendations to their customer support representatives. They expect Agent Assist to help minimize repetitive tasks by providing response and typeahead suggestions, helping improve the efficiency of their agents and the quality and consistency of service they provide.

Another customer, LoveHolidays, is using Agent Assist to support their agents and customers in the travel industry. 

“Agent Assist has been a beneficial aid to agents and our customers alike… It gives us the power to flex our contact center staff levels in hours not weeks,” said Eugene Neale, Director of CX Engineering & Business IT at LoveHolidays

Analysts say online chat is becoming one of the most popular ways to reach out to businesses for customer support. IDC research finds that single-function contact centers worldwide are increasingly rare — in 2020, although phone/voice is still responsible for most interactions (at around 18%); email is responsible for around 13% of interactions, and live chat (without automation) is responsible for around 8% of interactions, according to IDC, Toward the AI-Powered Contact Center, Doc # EUR147017320, December 2020.

Deploying CCAI with Agent Assist for Chat

As part of Google’s Contact Center AI suite, Agent Assist provides a seamless handoff from chats managed by your Dialogflow CX virtual agents. If a conversation or customer requires a live agent, Agent Assist will help your team pick it up quickly and drive it to a satisfying resolution. 

Historically, when managers saw contact center volumes increase they had two choices: allow customers to wait longer to speak to someone (lowering customer satisfaction) or bring on more agents (increasing cost to serve).  Deploying CCAI provides contact center leaders with a third choice: equip agents with tools like, Agent Assist for Chat, to efficiently manage customer interactions while maintaining high quality service.

Global CCAI partners support Agent Assist for Chat

Agent Assist for Chat is a set of public APIs that your engineering team can integrate directly into an agent desktop to control the agent experience from end-to-end. For a more out-of-the-box solution, we have partnered with LivePerson and 247.ai to build Agent Assist directly into their agent desktops.

“Integrating our Conversational Cloud directly with Agent Assist means agents can leverage cutting-edge productivity AI to build even further on the massive ROI of conversational commerce, from reduced agent effort and time-to-respond to increased customer satisfaction and revenue,” said Alex Spinelli, CTO of LivePerson. 

More Agent Assist resources

To learn more, check out the Agent Assist webpage. Give Agent Assist a try by training a model and then testing it using the Agent Assist simulator.

2021 has been a high-stakes year for call centers, with many organizations forced to rapidly scale up their call center operations in response to ongoing pandemic disruptions. We’re proud that 2021 has also been an amazing year for Google Cloud’s Contact Center AI (CCAI), which has helped our customers adapt and thrive, despite the challenging conditions. 

Beginning in January, we launched Dialogflow CX in GA. Agent Assist preview was released in May. Most recently, CCAI Insights GA was announced at Google Cloud NEXT in October. During NEXT, we shared lots of great content on how you can use CCAI to improve your customer experience with these breakout sessions:

• Using CCAI Insights to Better Understand Your Customers 
• Customer Impact with Conversational AI 
• Drive Results by Transforming the Customer Experience with AI-Powered Business Messages

But don’t just take our word for it. We also got a chance to hear how some companies are using CCAI to better reach their own customers, including The Home Depot, TELUS, and Love Holidays. We partnered with CDW to discuss  transforming the contact center with AI and with Quantiphi on how to migrate from Dialogflow EX to CX. Our integration with Looker Block also makes CCAI Insights even more powerful by visualizing contact center metrics. Over the summer, we hosted a Dialogflow CX competition with more than 1,100 participants. Just last month, we showed how we’ve enabled businesses to use AI in their interactions using Google Business Messages. Looking to the future, we talked about the future in our article,“Reimagining your Customer Experience with Conversational AI.”  

Amwell,  a U.S.-based telehealth company that is launching CCAI, including the recently launched CCAI Insights, is among the enterprises harnessing AI to transform its call centers. “With Contact Center AI, we aim to digitize our support for improved operational efficiency and elevated analytics capabilities, while enhancing the customer experience for patients, providers, and staff,” says Paul Johnson, SVP Client Services at Amwell. “Contact Center AI Insights will allow Amwell to better understand why our platform users are reaching out to support and how they feel about the overall experience – valuable insights for our support organization.”

As we recap the momentum of CCAI for 2021, it’s also a good time to review exactly how CCAI works.

What is CCAI?

As the volume of customer calls increases, it’s becoming even more important to make the most of human agents’ time to lower costs and improve customer experiences. CCAI enables you to do just that: it frees human agents to concentrate on more complex calls by providing them with real-time information to better handle those calls. 

Single source of intelligence: Contact Center AI provides a consistent, high-quality conversational experience across all channels and platforms, both human and virtual. Because the “brains” of CCAI are centralized in the cloud, you can apply consistent intelligence across every application in the customer journey. 

Ability to go off-script: Huge cost savings can be realized by having a virtual agent handle voice calls. This is easier said than done, however,  because conversations rarely are completely linear; instead, they meander from topic to topic, which is difficult to handle programmatically in a fixed-path Interactive Voice Response (IVR) system.   

Contact Center AI has the ability to go “off script” — to let callers go down tangents or side paths to the main conversation, while still tracking towards the main objective of the call.  With CCAI, your virtual agents can answer complex questions and complete complicated tasks, including allowing for unexpected stops and starts, unusual word choices, or implied meanings.  Developers can define supplemental questions, and CCAI can easily retain the context, answer the supplemental question, and come back to the main flow.

Versatile fulfillment: CCAI has the ability to handle multiple use cases for the customer with the same virtual agent, which enables you to fully automate routine tasks and deflect calls.  The same virtual agent can take a payment, update information like a phone number, give a customer information on their balance, and process information for other tasks, all within the same conversational flow.

How does CCAI work?

CCAI has three key components:

• Conversation Core: This is the central AI brain that underpins CCAI and its ability to understand, talk, and interact.  It enables and orchestrates high-quality conversational experiences at scale making it possible for customers to have  conversations with a virtual agent that are as good as conversations with a human agent.
  • Understand – Speech-to-text speech recognition understands what customers are saying regardless of how they phrase things, what vocabulary they use, what accent they have, and so on. 
  • Talk – Text-to-speech enables virtual agents to respond to customers in a natural, human-like manner that pushes the conversation along, rather than frustrate them.
  • Interact – Dialogflow identifies customer intent and determines the appropriate next step. You can build conversational flows in a point-and-click interface, and generate automated ML models for human-like conversational experiences.
• Virtual agents with Dialogflow: This component automates interactions with customers, using natural conversation to identify and address their issues. Virtual agents enable customers to get immediate help anytime, day or night. Agent Assist: This component brings AI to human agents to increase the quality of their work, while decreasing their average handling time.  Agent Assist shares initial context and provides real-time, turn-by-turn guidance to coach agents through business processes, as well as full call transcriptions that agents can edit and file quickly. 
• CCAI Insights: CCAI Insights aids your contact center management team in making better data driven decisions for their business by breaking down conversations using natural language processing and machine learning. Having this information allows your business to reduce manual analysis and focus on decision making like which conversations need your attention, where to deploy virtual agent automation to have the biggest impact and how to address your customer needs.  

How does CCAI create experiences for agents and customers? 

When a user initiates a chat or voice call and the contact center provider connects them with CCAI, a virtual agent engages with the user, understands their intent, and fulfills the request by connecting to the backend. If necessary, the call can be handed off to a human agent, who sees the transcript of the interaction with the virtual agent, gets feedback from the knowledge base to respond to queries in real time, and receives a summary of the call at the end. Insights help you understand what happened during the virtual agent and live agent sessions. The result is improved customer experiences and CSAT scores, lower agent handling times, and more time for human agents to spend on more complicated customer issues.

And there you have it: a quick overview of CCAI and its progress in 2021. For more details, check out the documentation or our CCAI solutions page.https://www.youtube.com/embed/6_Gilug2QYw?enablejsapi=1&

For more #GCPSketchnote, follow the GitHub repo. For similar cloud content follow me on Twitter @pvergadia and keep an eye out on thecloudgirl.dev.