Google Cloud’s AI-optimized infrastructure makes it possible for businesses to train, fine-tune, and run inference on state-of-the-art AI models faster, at greater scale, and at lower cost. We are excited to announce the preview of inference on Cloud TPUs. The new Cloud TPU v5e enables high-performance and cost-effective inference for a broad range AI workloads, including the latest state-of-the-art large language models (LLMs) and generative AI models.

As new models are released and AI becomes more sophisticated, businesses require more powerful and cost efficient compute options. Google is an AI-first company, so our AI-optimized infrastructure is built to deliver the global scale and performance demanded by Google products like YouTube, Gmail, Google Maps, Google Play, and Android that serve billions of users — as well as our cloud customers. 

LLM and generative AI breakthroughs require vast amounts of computation to train and serve AI models. We’ve custom-designed, built, and deployed Cloud TPU v5e to cost-efficiently meet this growing computational demand.

Cloud TPU v5e is a great choice for accelerating your AI inference workloads: 

• Cost Efficient: Up to 2.5x more performance per dollar and up to 1.7x lower latency for inference compared to TPU v4.
• Scalable: Eight TPU shapes support the full range of LLM and generative AI model sizes, up to 2 trillion parameters. 
• Versatile: Robust AI framework and orchestration support. 

In this blog, we’ll dive deeper into how you can leverage TPU v5e effectively for AI inference.

Up to 2.5x more performance per dollar and up to 1.7x lower latency for inference

Each TPU v5e chip provides up to 393 trillion int8 operations per second (TOPS), allowing complex models to make fast predictions. A TPU v5e pod consists of 256 chips networked over ultra-fast links. Each TPU v5e pod delivers up to 100 quadrillion int8 operations per second, or 100 PetaOps, of compute power.

We optimized the Cloud TPU inference software stack to take full advantage of this powerful hardware. The inference stack leverages XLA, Google’s AI compiler, which generates highly-efficient code for TPUs to maximize performance and efficiency.

The combined hardware and software optimizations, including int8 quantization, enable Cloud TPU v5e to achieve up to 2.5x greater inference performance per dollar than Cloud TPU v4 on state-of-the-art LLM and generative AI models, including Llama 2, GPT-3, and Stable Diffusion 2.1:

Google Internal Data. August 2023. Normalized to single-chip throughput. Precision: Llama 2 7B, 13B, 70B, GPT-J 6B: int8; GPT-J 175B, Stable Diffusion 2.1: bf16.

On latency, Cloud TPU v5e achieves up to 1.7x speedup compared to TPU v4:

Google Internal Data. August 2023. Precision: Llama 2 7B, 13B and 70B: int8; GPT-3 175B: bf16.

Google Cloud customers have been running inference on Cloud TPU v5e, and some have seen even greater speedups on their particular workloads.

AssemblyAI offers dozens of AI models to their customers for speech recognition and understanding with over 25 million inference calls on a daily basis. 

“Cloud TPU v5e consistently delivered up to 4X greater performance per dollar than comparable solutions in the market for running inference on our production model. The Google Cloud software stack is optimized for peak performance and efficiency, taking full advantage of the TPU v5e hardware that was purpose-built for accelerating the most advanced AI and ML models. This powerful and versatile combination of hardware and software dramatically accelerated our time to solution: instead of spending weeks hand-tuning custom kernels, within hours we optimized our model to meet and exceed our inference performance targets.” – Domenic Donato, VP of Technology, AssemblyAI

Scale to the full range of LLM and Generative AI model sizes 

LLMs and generative AI models continue to grow in size and computational cost. The largest models require the combined compute and memory of hundreds of hardware accelerators. Cloud TPU v5e enables inference for a wide range of model sizes. A single v5e chip can run models with up to 13B parameters. From there, you can scale up to hundreds of chips and run models with up to 2 trillion parameters.

Google Internal Data. August 2023. Batch size = 1. Multi-head attention based decoder only language models: prefix length = 2048, decode steps = 256, beam size = 32 for sampling.

Gridspace leverages Google Cloud TPU infrastructure to power its full-stack conversational AI platform – building and integrating real-time conversational ASR, LLMs, semantic search, and neural TTS.

“We’re a huge fan of Google Cloud TPUs. Our benchmarks are demonstrating a 5X increase in the speed of AI models when training and running on Google Cloud TPU v5e. We are also seeing a 6x improvement in the scale of our inference metrics. We’ve scaled our AI models to billions of conversations per year across financial services, capital markets, and healthcare with Google Cloud’s AI infrastructure. Our Grace bots are powered by models trained using Cloud TPUs and served at scale on GKE with support for PCI, HITRUST, and SOC 2 compliance.” – Wonkyum Lee, Head of Machine Learning, Gridspace 

Robust AI framework and orchestration support

Leading AI frameworks, including PyTorch, JAX, and TensorFlow, provide robust support for inference on Cloud TPU v5e. This means you can now train and serve models end-to-end on Cloud TPUs: what you train is what you serve.

Google Cloud offers you many choices to run inference on Cloud TPUs easily and reliably. From GKE and Vertex AI, to popular open-source frameworks such as Ray and Slurm, you can leverage Google Cloud TPUs in your preferred way to fit your development process.

Try Cloud TPU v5e for inference today

Cloud TPU v5e provides a high-performance, cost-efficient, scalable, and reliable inference platform for LLMs and generative AI models. Leading AI companies are leveraging the power of Cloud TPU v5e to serve AI models at scale:

To get started with inference on Cloud TPU, reach out to your Google Cloud account manager or contact Google Cloud sales.

Visualizing trends in your logs is critical when troubleshooting an issue with your application. Using the histogram in Logs Explorer, you can quickly visualize log volumes over time to help spot anomalies, detect when errors started and see a breakdown of log volumes. But static visualizations are not as helpful as having more options for customization during your investigations. 

That’s why we’re excited to announce that we recently added three new query controls along with separate colors for log severity to the histogram. These new features make it even easier to refine and analyze your logs by time range. The new histogram controls help find logs before or after the current period, jump to a specific time range represented in a histogram bar and zoom in/out of the current time window in the histogram.

Histogram colors

The histogram now makes it easier to view the breakdown of logs by severity with the introduction of color coding. For example, the severity colors make it easy to spot an increasing number of errors even when the volume of requests is relatively constant. Looking at the histogram below, the red vs blue shading makes it clear that there has been an increase in overall log volume and provides a visual breakdown of errors within that log volume.

Pan left/right to scroll through time

Sometimes in your troubleshooting journey, you may want to look at the logs directly before or after the current set of logs. Perhaps there was an unexpected spike in errors at the beginning of the time range and you need to see the logs in the time period directly preceding the current time range. Pressing the left arrow on the left side of the histogram shifts the time range earlier while the arrow on the right side of the histogram shifts the time range ahead. Either arrow will refine the time range in the query and rerun the query to return the logs in the new time range.

Zooming in or out 

Zooming in or out from a given time range may be useful to visualize fine-grained details or a broader trend Clicking the zoom in or out icons in the upper right corner of the histogram refines the time range in the query and then reruns the query, returning the logs in the newly defined time range.

Scrolling to time 

If you see a large spike in logs volume in the histogram, it’s useful to quickly review the logs generated during that spike. Clicking on the histogram bar that contains the spike now scrolls you to the logs generated during that time period.

Where to find the histogram 

The histogram is a panel in Logs Explorer that can be displayed or hidden using the controls in the Page Layout menu. When you no longer want to display the histogram, click the “X” button in the upper right corner to quickly close it. To open it again, use the same Page Layout menu to enable the histogram display.

Get started with the histogram

These improvements move the histogram from a utility for visualization to an integral part of the troubleshooting journey. We are continuously working to launch new features that make Cloud Logging the best place to troubleshoot your Google Cloud logs. If you are not already a Cloud Logging user, review this getting started documentation or watch a quick video on troubleshooting services on Google Kubernetes Engine (GKE) to learn more. If you have specific questions or feedback, please join the discussion on our Google Cloud Community, Cloud Operations page.

It’s no question that the cloud computing industry is booming and cloud experts are in high demand.  In 2020, 67 percent of organizations surveyed in the IDG Cloud Computing Report added new cloud roles and functions (source). 

So there are plenty of cloud jobs out there, but if you don’t have years of experience, you might find it difficult to get one. This is especially true for members of underrepresented communities, who face other kinds of systemic barriers in education and employment.

To help meet this demand and democratize opportunity, Google Cloud is introducing its first virtual Certification Journey Learning program for underrepresented communities across the United States—Cloud Career Jump Start. The program is free of charge for participants, and includes technical and professional skill development and resources with an aim to provide a readiness path to certification via the Google Cloud Associate Cloud Engineer Exam. 

Google Cloud’s Certification Journey Learning Program

The program offers free access to the Google Cloud Associate Cloud Engineer training, which provides support preparations for the Certification Exam. This industry-recognized certification helps job applicants validate their cloud expertise, elevate their career, and transform businesses with Google Cloud technology. Learn more about the exam by watching the Certification Prep: Associate Cloud Engineer webinar. 

Designed specifically for Computer Science or Information Systems-related majors (or relevant experience), the 12-week program offers:

• Exclusive free access to Google Cloud Associate Cloud Engineer-related training to support preparations for the Certification Exam. 
• Guided progress through the certification prep training on Qwiklabs. 
• Office hours hosted by Google Cloud technical experts. 
• Technical workshops, including sessions with Googlers who earned cloud industry certifications and have built impactful careers in cloud tech, such as Google Cloud’s Kelsey Hightower (a Principal Developer Advocate) and Jewel Langevine (a Solution Engineer). 

Earn your Associate Cloud Engineer Certification 

Curious if you qualify to apply for the program? We’ve put together a guide below to help: 

• What does an associate cloud engineer do? Associate Cloud Engineers deploy applications, monitor operations, and manage enterprise solutions. They also use the Google Cloud Console and the command-line interface to perform common platform-based tasks, to maintain one or more deployed solutions that leverage Google-managed or self-managed services on Google Cloud.
• Who’s eligible? We are encouraging applicants from groups that have been historically underrepresented in tech including Black, Latinx, Native American & Indigenous communities. Applicants should have six or more months of hands-on experience in a Computer Information Systems-related major, and/or relevant experience through online courses, boot camps, hackathons or internships. Although the ability to program is not required, familiarity with the following IT concepts is highly recommended: virtual machines, operating systems, storage and file systems, networking, databases, programming, and working with Linux at the command line.

How do you know if you are ready for the program? 

To get an idea of what the program will cover:

• Watch the Certification Prep: Associate Cloud Engineer webinar
• Complete the Skill IQ Assessment via Pluralsight
• Review the Associate Cloud Engineer Certification Exam Guide 
• Take the Associate Cloud Engineer Practice Exam

Jump start a career in cloud infrastructure and technology 

Following the hands-on training, Google Cloud will offer an additional 9 months of career development resources and activities. This includes an online support community, mentoring with Googlers & partners including SADA, EPAM Systems Inc., and Slalom, resume and interviewing support from Google Recruiting & Staffing, and additional career workshops. We are also partnering with a number of Black-owned and -operated, nonprofit and cloud training organizations like Kura Labs. 

Cloud certifications open doors 

Want to hear more about this program from two Googlers who completed it? We asked Kelsey and Jewel to share more on how certifications helped them launch their careers in the cloud industry: 

• “IT certifications introduced me to the game; opportunities and hard work helped me change it.” – Kelsey Hightower, Staff Developer Advocate, Google Cloud Platform 
• “Becoming certified as a cloud computing professional combined with prayer, networking, and practice has kept me moving on my purposeful and rewarding career path as an engineer.” – Jewel Langevine, Solution Engineer, Google Cloud Solutions Studio

Kelsey and Jewel’s wisdom doesn’t end there. They play a central role in the program, sharing more about how they navigated certifications and leveraged them for success. 

Apply today to Cloud Career Jump Start

The program is now live in the United States, with plans to expand to other regions in the coming months. It is completely virtual, and all training is on-demand so that participants can access their coursework anytime, anywhere via the web or mobile device. To determine whether you (or someone you know) would be a great fit for the Cloud Career Jump Start, check out our guidelines and apply.