Google Cloud’s AI-optimized infrastructure makes it possible for businesses to train, fine-tune, and run inference on state-of-the-art AI models faster, at greater scale, and at lower cost. We are excited to announce the preview of inference on Cloud TPUs. The new Cloud TPU v5e enables high-performance and cost-effective inference for a broad range AI workloads, including the latest state-of-the-art large language models (LLMs) and generative AI models.

As new models are released and AI becomes more sophisticated, businesses require more powerful and cost efficient compute options. Google is an AI-first company, so our AI-optimized infrastructure is built to deliver the global scale and performance demanded by Google products like YouTube, Gmail, Google Maps, Google Play, and Android that serve billions of users — as well as our cloud customers. 

LLM and generative AI breakthroughs require vast amounts of computation to train and serve AI models. We’ve custom-designed, built, and deployed Cloud TPU v5e to cost-efficiently meet this growing computational demand.

Cloud TPU v5e is a great choice for accelerating your AI inference workloads: 

• Cost Efficient: Up to 2.5x more performance per dollar and up to 1.7x lower latency for inference compared to TPU v4.
• Scalable: Eight TPU shapes support the full range of LLM and generative AI model sizes, up to 2 trillion parameters. 
• Versatile: Robust AI framework and orchestration support. 

In this blog, we’ll dive deeper into how you can leverage TPU v5e effectively for AI inference.

Up to 2.5x more performance per dollar and up to 1.7x lower latency for inference

Each TPU v5e chip provides up to 393 trillion int8 operations per second (TOPS), allowing complex models to make fast predictions. A TPU v5e pod consists of 256 chips networked over ultra-fast links. Each TPU v5e pod delivers up to 100 quadrillion int8 operations per second, or 100 PetaOps, of compute power.

We optimized the Cloud TPU inference software stack to take full advantage of this powerful hardware. The inference stack leverages XLA, Google’s AI compiler, which generates highly-efficient code for TPUs to maximize performance and efficiency.

The combined hardware and software optimizations, including int8 quantization, enable Cloud TPU v5e to achieve up to 2.5x greater inference performance per dollar than Cloud TPU v4 on state-of-the-art LLM and generative AI models, including Llama 2, GPT-3, and Stable Diffusion 2.1:

Google Internal Data. August 2023. Normalized to single-chip throughput. Precision: Llama 2 7B, 13B, 70B, GPT-J 6B: int8; GPT-J 175B, Stable Diffusion 2.1: bf16.

On latency, Cloud TPU v5e achieves up to 1.7x speedup compared to TPU v4:

Google Internal Data. August 2023. Precision: Llama 2 7B, 13B and 70B: int8; GPT-3 175B: bf16.

Google Cloud customers have been running inference on Cloud TPU v5e, and some have seen even greater speedups on their particular workloads.

AssemblyAI offers dozens of AI models to their customers for speech recognition and understanding with over 25 million inference calls on a daily basis. 

“Cloud TPU v5e consistently delivered up to 4X greater performance per dollar than comparable solutions in the market for running inference on our production model. The Google Cloud software stack is optimized for peak performance and efficiency, taking full advantage of the TPU v5e hardware that was purpose-built for accelerating the most advanced AI and ML models. This powerful and versatile combination of hardware and software dramatically accelerated our time to solution: instead of spending weeks hand-tuning custom kernels, within hours we optimized our model to meet and exceed our inference performance targets.” – Domenic Donato, VP of Technology, AssemblyAI

Scale to the full range of LLM and Generative AI model sizes 

LLMs and generative AI models continue to grow in size and computational cost. The largest models require the combined compute and memory of hundreds of hardware accelerators. Cloud TPU v5e enables inference for a wide range of model sizes. A single v5e chip can run models with up to 13B parameters. From there, you can scale up to hundreds of chips and run models with up to 2 trillion parameters.

Google Internal Data. August 2023. Batch size = 1. Multi-head attention based decoder only language models: prefix length = 2048, decode steps = 256, beam size = 32 for sampling.

Gridspace leverages Google Cloud TPU infrastructure to power its full-stack conversational AI platform – building and integrating real-time conversational ASR, LLMs, semantic search, and neural TTS.

“We’re a huge fan of Google Cloud TPUs. Our benchmarks are demonstrating a 5X increase in the speed of AI models when training and running on Google Cloud TPU v5e. We are also seeing a 6x improvement in the scale of our inference metrics. We’ve scaled our AI models to billions of conversations per year across financial services, capital markets, and healthcare with Google Cloud’s AI infrastructure. Our Grace bots are powered by models trained using Cloud TPUs and served at scale on GKE with support for PCI, HITRUST, and SOC 2 compliance.” – Wonkyum Lee, Head of Machine Learning, Gridspace 

Robust AI framework and orchestration support

Leading AI frameworks, including PyTorch, JAX, and TensorFlow, provide robust support for inference on Cloud TPU v5e. This means you can now train and serve models end-to-end on Cloud TPUs: what you train is what you serve.

Google Cloud offers you many choices to run inference on Cloud TPUs easily and reliably. From GKE and Vertex AI, to popular open-source frameworks such as Ray and Slurm, you can leverage Google Cloud TPUs in your preferred way to fit your development process.

Try Cloud TPU v5e for inference today

Cloud TPU v5e provides a high-performance, cost-efficient, scalable, and reliable inference platform for LLMs and generative AI models. Leading AI companies are leveraging the power of Cloud TPU v5e to serve AI models at scale:

To get started with inference on Cloud TPU, reach out to your Google Cloud account manager or contact Google Cloud sales.

Environmental sustainability is a key priority for TELUS, a world-leading communications technology company. It continues to rank in the top 100 most sustainably managed companies in the world, and seeks to make a healthier planet for all by leveraging its global-leading technology, compassion to drive social change and reduce our collective carbon footprint through innovative technologies and sustainable business practices.

TELUS surpassed its sustainability objectives in 2019 and is now on a journey to procure all of its electricity from renewable or low-emitting sources by 2025. Next, it aims to achieve net carbon neutrality for its operations by 2030. TELUS has also been named to the Dow Jones Sustainability Index for 21 consecutive years, a feat unmatched by any other North American telecom or cable company. In 2021, it became the first company in Canada to release a Sustainability-linked bond (SLB) framework and complete an SLB offering, formally linking TELUS financing to its environmental performance.

“We’ve spent the last decade becoming a global leader in sustainability, helping make the planet healthier by ensuring that our operations are as environmentally responsible as possible,” said Geoff Pegg, Head of Sustainability and Environment at TELUS.

In part, TELUS’ strategy is focused on three key areas:

1. Seek the best renewable energy options available
2. Focus on migrating workloads to the cloud
3. Embrace a multiplier effect through the use of sustainable partners

Renewable energy impact

Part of this environmental responsibility involves investing heavily in renewable energy sources through power purchase agreements (PPAs) that help renewable energy providers like wind farms and solar companies develop their infrastructure. TELUS executed PPAs with four Alberta-based solar and wind facilities to provide 100 per cent of its electricity load demand in a province where one-third of the grid is powered by coal.

As a technology company, electricity represents a large portion of TELUS’ energy needs: 80 percent of the operational carbon footprint comes from the power requirements for TELUS’ network and administrative buildings, Pegg explains. While TELUS is using renewable energy sources and low-emitting energy grids to power its buildings and network, there’s also the often-forgotten part of the carbon emissions equation: the energy it takes to power data centers. As the International Energy Agency recently reported, data centers represent 1 percent of the global electricity demand and that figure is expected to keep rising as the world increases usage of data-heavy technologies.

“It’s probably no surprise that everyone, whether you’re a business or a consumer, is concerned about reducing carbon emissions,” said Chris Talbott, the Google Cloud Sustainability Lead. “A lot of us think about the carbon emissions associated with our cars or with the electricity that powers our homes, but oftentimes we forget about the carbon emissions that come from the digital services that we use or the networks required to deliver that data.”

As a leader in sustainability, how can TELUS meet the energy demands of its customers while also protecting the environment? One way is through the company’s previously announced collaboration with Google Cloud. The two companies are working together to build a more sustainable world through technology and reduce TELUS’ carbon footprint, create value along the entire supply chain, and optimize industry solutions for social impact through data analytics and machine learning.

Taking a cloud first approach — reducing carbon emissions with green cloud computing

Google became carbon neutral in 2007 and has achieved 100 per cent renewable energy matching every year since 2017. Google has invested in renewable energy to match the electricity we use across our entire operations, including Google Cloud, meaning every workload that TELUS runs on Google Cloud has been matched with renewable energy purchases.

“The operational carbon footprint of running anything on Google Cloud is zero,” Talbott said. Also, by working with Google, TELUS gets the benefit of economies of scale using less electricity. Not only is TELUS leveraging Google data centers, it’s also relying on the digital collaboration made possible by Google Workspace to reduce the amount of travel required by employees attending meetings in different offices. Collaboration tools like Google Meet can reduce the carbon footprint of in-person conferences by 94 percent.

Google compensates for the environmental footprint of any electricity used in the data center and out to the edge network. “You can feel pretty good about using Google Meet because it’s carbon-neutral,” Talbott said.

Multiplier through sustainable partnerships — green cloud computing radiates out

By supporting TELUS in its environmental sustainability efforts, Google Cloud is also enabling TELUS to do the same for its various partnerships. For example, powered by Google Cloud’s infrastructure and data analytics capabilities, TELUS is partnering with Picacity (formerly NXN Digital) and Google Cloud to deliver an ecosystem of integrated smart technologies that enable cities to improve the lives of their residents.

From dynamic traffic signaling that reduces congestion and emissions, to data analytics that create smarter, more efficient city planning, the partnership is transforming the way municipalities operate in our increasingly digital world.The partnership is built on four foundational pillars of infrastructure and environmental sustainability, intelligent transportation, public safety and security, and health. In the case of intelligent transportation, this means sensors, cameras, and other devices are built into or near roads, sidewalks, and bike paths to provide data for innovative software to improve traffic flow in real time. The data can then foster informed decisions about infrastructure, city planning, fleet optimization, and public safety.

All of these environmental measures may seem small when compared with the enormity of the problem that is climate change, but as Talbott said, “Change begins with the small decisions we make every day such as paying attention to the practices of companies that we’ve come to rely on daily in the modern world. They may seem small and in the margins, but at scale, this is how we can make a real impact.”

When we launched Partner Advantage, we committed to making it predictable and easy for partners to drive business with us. Since launch, those commitments have been validated by channel experts like CRN, which gave Partner Advantage a 5-Star award for 2022, and by the fact that our partners have seen impressive growth* across virtually every facet of their business.

I am pleased to announce that our commitments endure today with the launch of Consumption Packs for Deal Acceleration Funds and Partner Services Funds (DAF and PSF for partners). Inspired by partner feedback, these new packages are designed to accelerate all stages of a customer’s journey to the cloud, and make it even easier and faster for partners to do business with us.

Consumption Packs are purpose-built so that partners can plan and initiate customer projects much more quickly, with predictable funding. They include assets and templates that allow partners to deliver Google Cloud designed and validated infrastructure, application migration and modernization plans to customers faster than ever–particularly for customers beginning their journey to the cloud. Based on learnings gathered from thousands of customer deployments, these turnkey packs have been designed by our partners and Google Cloud Partner Engineering and Professional Services’ teams.

Here’s a brief look at Consumption Packs in action:

• Consumption Packs offer pre-approved, curated templates and assets to simplify and shorten the process for most common projects.
• For Deal Acceleration Funds (DAF), packages include everything partners need to conduct assessments, workshops and proofs-of-concept so they can quickly meet customers where they are on their journey to the cloud.
• For Partner Services Funds (PSF), packages are structured so that partners can develop cloud ready foundation and migration plans that align with Google Cloud priority solution areas.
• Packages have pre-determined funding levels to enable faster deployments.

Partners still have the option to engage with Google Cloud Partner Advantage and their customers through customized requests, as they always have. This is ideally suited for projects that require a tailored approach to meet unique customer requirements.

We are launching nine consumption packs today focused on key enterprise workloads, with a vision toward introducing additional packages to cover more solutions. Partners can explore Consumption Packs now by visiting the Partner Advantage portal.

We welcome your continued feedback and suggestions, and look forward to helping our customers achieve new levels of growth and success, together.

See you in the cloud.

• The Google Cloud Business Opportunity For Partners, a commissioned Total Economic Impact™ study conducted by Forrester Consulting, October 2021