Organizations across industries are looking to AI to turn troves of data into intelligence, powered by the latest advances in generative AI. Yet for many organizations, there is a barrier to adopting the latest models because they can be costly to train or serve. A new class of cloud GPUs is needed to lower the cost of entry for businesses that want to tap the power of AI. 

Today, we’re introducing G2, the newest addition to the Compute Engine GPU family in Google Cloud. G2 is the industry’s first cloud VM powered by the newly announced NVIDIA L4 Tensor Core GPU, and is purpose-built for large inference AI workloads like generative AI. G2 delivers cutting-edge performance-per-dollar for AI inference workloads that run on GPUs in the cloud. By switching from NVIDIA A10G GPUs to G2 instances with L4 GPUs, organizations can lower their production infrastructure costs up to 40%. We also found that customers switching from NVIDIA T4 GPUs to L4 GPUs can achieve 2x-4x better performance. As a universal GPU offering, G2 instances also help accelerate other workloads, offering significant performance improvements on HPC, graphics, and video transcoding. Currently in private preview, G2 VMs are both powerful and flexible, and scale easily from one up to eight GPUs. 

Currently organizations require end-to-end enterprise ready infrastructure that will future proof their AI and HPC initiatives for a new era. G2s will be ready to be deployed on Vertex AI, GKE, and GCE, giving customers the freedom to architect their own custom software stack to meet their performance requirements and budget. With optimized Vertex AI support for G2 VMs, AI users can tap the latest generative AI models and technologies. With an easy to use UI and automated workflows, customers can access, tune and serve modern models for video, text, images, and audio without the toil of manual optimizations. The combination of these services with the power of G2 will help customers harness the power of complex machine models for their business.

NVIDIA L4 GPUs with Ada Lovelace Architecture

G2 machine families enable machine learning customers to run their production infrastructure in the cloud for a variety of applications such as language models, image classification, object detection, automated speech recognition, and language translation. Built on the Ada Lovelace architecture with fourth-generation Tensor Cores, the NVIDIA L4 GPU provides up to 30 TFLOPS of performance for FP32, and 242 TFLOPs for FP16. Newly added FP8 support, on top of existing INT8, BFLOAT16 and TF32 capabilities, makes the L4 ideal for ML inference. 

With the latest third-generation RT Cores and DLSS 3.0 technology, G2 instances are also great for graphics-intensive workloads such as rendering and remote workstations when paired with NVIDIA RTX Virtual Workstation. NVIDIA L4 provides 3x video encoding and decoding performance, and adds new AV1 hardware-encoding capabilities. For example, G2 can enable gaming customers running game engines such as Unreal and Unity with modern graphics cards to run real-time applications. Likewise, media and entertainment customers that need GPU-enabled virtual workstations can use the L4 to create photo-realistic, high-resolution 3D content for movies, games, and AR/VR experiences using applications such as Autodesk Maya or 3D Studio Max.

What customers are saying

A handful of early customers have been testing G2 and have seen great results in real-world applications. Here are what some of them have to say about the benefits that G2 with NVIDIA L4 GPUs bring:

AppLovin

AppLovin enables developers and marketers to grow with market leading technologies. Businesses rely on AppLovin to solve their mission-critical functions with a powerful, full stack solution including user acquisition, retention, monetization and measurement.

“AppLovin serves billions of AI powered recommendations per day, so scalability and value are essential to our business,” said Omer Hasan, Vice President, Operations at AppLovin. “With Google Cloud’s G2 we’re seeing that NVIDIA L4 GPUs offer a significant increase in the scalability of our business, giving us the power to grow faster than ever before.”

WOMBO

WOMBO aims to unleash everyone’s creativity through the magic of AI, transforming the way content is created, consumed, and distributed. 

“WOMBO relies upon the latest AI technology for people to create immersive digital artwork from users’ prompts, letting them create high-quality, realistic art in any style with just an idea,” said Ben-Zion Benkhin, Co-Founder and CEO of WOMBO. “Google Cloud’s G2 instances powered by NVIDIA’s L4 GPUs will enable us to offer a better, more efficient image-generation experience for users seeking to create and share unique artwork.”

Descript

Descript’s AI-powered features and intuitive interface fuel YouTube and TikTok channels, top podcasts, and businesses using video for marketing, sales, and internal training and collaboration. Descript aims to make video a staple of every communicator’s toolkit, alongside docs and slides. 

“G2 with L4’s AI Video capabilities allow us to deploy new features augmented by natural-language processing and generative AI to create studio-quality media with excellent performance and energy efficiency” said Kundan Kumar, Head of Artificial Intelligence at Descript.

Workspot

Workspot believes that the software-as-a-service (SaaS) model is the most secure, accessible and cost-effective way to deliver an enterprise desktop and should be central to accelerating the digital transformation of the modern enterprise.

“The Workspot team looks forward to continuing to evolve our partnership with Google Cloud and NVIDIA. Our customers have been seeing incredible performance leveraging NVIDIA’s T4 GPUs. The new G2 instances with L4 GPUS through Workspot’s remote Cloud PC workstations provide 2x and higher frame rates at 1280×711 and higher resolutions” said Jimmy Chang, Chief Product Officer at Workspot.

Pricing and availability

G2 instances are currently in private preview in the following regions: us-central1, asia-southeast1 and europe-west4. Submit your request here to join the private preview, or to receive a notification as when the public preview begins. Support will be coming to Google Kubernetes Engine (GKE), Vertex AI, and other Google Cloud services as well. We’ll share G2 public availability and pricing information later in the year.

Google Cloud is built on world-class technical infrastructure that supports services that are loved and relied on by billions of people across the globe: Google Search, YouTube, Gmail, Google Maps and more. A core tenet at Google Cloud is to leverage Google’s experience building and operating highly available and highly reliable planetary-scale compute, storage and networking systems and data centers. 

Google takes a workload-optimized approach to building its infrastructure, employing a combination of dedicated hardware and software components to meet its workloads’ ever-growing demands. Underpinning this infrastructure is Titanium, a system of purpose-built, custom silicon and multiple tiers of scale-out offloads that together power improvements in the performance, reliability, and security of our customers’ workloads (for example, 25% faster block storage IOPS/instance compared to the other two leading hyperscalers). Unveiled today at Google Cloud Next, you’ll find Titanium technology in many of Google Cloud’s recent infrastructure offerings.

10x demands of tomorrow 

Meeting the growing performance, reliability, and security demands of both legacy and emerging workloads is a constant challenge for cloud infrastructure providers. And now, these demands are multiplying with the heightened adoption of generative AI across almost every industry. Meanwhile, the benefits of Moore’s law have been declining in recent years. We can’t rely on silicon advances alone to meet tomorrow’s needs.

As just one example, this chart shows the exponential computing demands of large language models.

It was clear to us a long time ago that we needed to rethink our infrastructure designs to meet these demands. This is why, for several years, we’ve adopted workload-optimization and intentional design as central principles for our infrastructure platform. We engineer golden paths from silicon to the customer workload, using a combination of purpose-built infrastructure, prescriptive architectures, and an open ecosystem to deliver workload-optimized infrastructure. 

Offloads play a pivotal role

Central to this strategy are offload technologies. Traditionally, the CPU wears many hats: It runs the hypervisor, the virtualization stack to enable your workloads, and manages storage and networking I/O; it’s responsible for security isolation for virtual interfaces and physical hardware, etc. In this model, customer workloads running on the CPU contend for resources with these platform tasks.

Offloads on dedicated hardware perform behind-the-scenes security, networking, and storage functions that were previously performed by the host CPU, allowing the CPU to focus on maximizing performance for customer workloads.

A recent example of an on-host offload or accelerator is the Infrastructure Processing Unit (IPU), a system-on-chip that we co-designed with Intel to enable better security isolation and performance on our 3rd gen compute instances. The IPU enables:

• Predictable and efficient compute
• Programmable packet processing for low latency, 200 Gbps networking with 3x the packets per second compared to our previous-generation compute instances
• In-transit encryption with the PSP protocol

Another important example of Google’s on-host hardware is Titan, a secure, low-power microcontroller that helps ensure that every machine in Google Cloud boots from a trusted state.

But we did not stop there. To meet tomorrow’s demands, we knew we needed to go past the performance that could be achieved using the host’s dedicated offload hardware.

A tiered system of offloads

A key component of Titanium is its modern offload architecture, which combines capabilities whose scale and performance are well-established within Google, as well as new capabilities tailored for cloud use cases. 

Just as modern workloads scale out horizontally in the cloud, with Titanium, we’ve extended the architecture to augment on-host offloads with an additional tier of scale-out offloads that run outside the host. This system of offloads is deployed fleet-wide and dynamically adjusts to changing workload needs to continually deliver the best performance.

Example 1: Block storage
Titanium scale-out offload enables Hyperdisk block storage to deliver stellar I/O performance. Hyperdisk’s Titanium offload on the host IPU works in tandem with the Titanium scale-out offload tier that distributes I/O across Google’s massive cluster-level filesystem, Colossus.

With traditional offload architectures, higher block storage IOPS requires purchasing larger compute instances. For example, you may need to deploy a data-intensive workload on a compute instance with many more vCPUs than the workload needs just to get sufficient storage performance. This tight coupling results in wasted resources and higher costs for customers. Further, even with large instances, storage performance in the cloud may be inadequate relative to what customers are used to with on-prem storage systems.     

With our new block storage, Hyperdisk powered by Titanium, we have decoupled compute-instance size from storage performance. Hyperdisk uses a tier of offloads in our cloud fabric to offload storage I/O from the customer hosts to achieve higher storage performance even with a general-purpose VM.   

In fact, today we are announcing that Titanium-powered C3 VMs with Hyperdisk Extreme now support 500K IOPS per compute instance in preview to meet the needs of the most demanding workloads. This is 25% faster IOPS/instance compared to the other two leading hyperscalers, courtesy of the Titanium system. 

Example 2: Network routing
Virtual network routing is another example of using a second tier of scale-out offloads (“hoverboards”). With Titanium, Google’s Andromeda virtual networking stack on the IPU offload device sends all packets for which it does not have a route to Hoverboard gateways, which have forwarding information for all virtual networks. Hoverboards are standalone software switches that act as default routers for some flows.

Unlike the traditional gateway model, the control plane dynamically detects flows that exceed a specified usage threshold and programs them to be direct host-to-host flows, bypassing the hoverboards allowing hoverboards to focus on the long tail of less frequent flows. Typically, only a small subset of possible VM pairs in a network communicate with one another, so the VMs only have to store and process a small fraction of the usual network configuration on an individual VM host, improving per-server memory utilization and control-plane CPU scalability.

Titanium already powers your workloads

The Titanium journey began years ago with the component technologies described above. Many of our products already benefit from this architecture, and the newest elements of this architecture are now available with our 3rd gen Compute Engine instances such as C3 and the new Hyperdisk block storage. 

Going forward, look for the Titanium architecture to underpin all future generations of our infrastructure offerings, in the process enabling new classes of infrastructure capabilities that move well beyond the confines of a single server.

Google Cloud Results

• Improves application performance by 8x to 12x; customers can create increasingly complex optimized driving routes in single-digit seconds
• Improves customer satisfaction via increased reliability and greater application performance
• Focuses on adding value to customers by improving software and algorithms, not infrastructure management
• Saves 5x in infrastructure costs

In 2009, Dan Khasis needed to rent an apartment. His search had him driving around the greater New York City area in unfamiliar areas, scattershot-style, often ending up where he started. The frustrating experience led the serial entrepreneur to launch Route4Me, a smartphone navigation app to help consumers create driving routes optimized for multiple stops.

Soon, business users recognized Route4Me’s value and requested enhancements specifically for them. While route optimization apps for big businesses already existed, they were almost exclusively offline desktop programs that were expensive to purchase, deploy, and get trained on. Recognizing the opportunity, Route4Me developed an affordable route optimization solution across various devices, such as smartphones, smartwatches, and telematics devices. The software was tailored to logistics-intensive businesses such as last-mile delivery services and business units conducting field sales, field service, and field marketing functions.

As Route4Me grew its user base, it became clear that its infrastructure of rented, dedicated servers from various providers wasn’t sustainable. “The hardware costs seemed low, but there were many risks and hidden costs,” says Dan Khasis, Co-founder and CEO at Route4Me. For example, “Multi-zone disaster recovery, high availability, automated failover, and on-demand surging of many nodes was simply impossible,“ he adds.

Because under the hood Route4Me’s routing optimization platform requires complex computations, the company needed a globally scalable infrastructure capable of delivering low latency and high throughput. Route4Me also needed to stay competitive by developing and delivering new services as quickly and efficiently as possible.

For these and other reasons, Route4Me moved 100% into the cloud. “Like many entrepreneurial software companies, we test all the latest technologies we can find before upgrading. Typically we go with the fastest technology, with a strong bias towards open source and open standards,” Khasis says. Based on extensive testing, Route4Me selected Google Cloud Platform (GCP). Along with the scalability, flexibility, reliability, and low-cost structure of GCP, Route4Me had already migrated its entire platform to containerized microservices, which Khasis says “are extremely stable and reliable” on Google Kubernetes Engine. While Route4Me has proprietary routing and route optimization engines, it uses Google Maps for high-precision geocoding and as the frontend.

With GCP, Route4Me has reduced its IT infrastructure costs while delivering faster route optimizations and more reliable service to customers. Because of GCP, the company is also planning to add services that will deliver the fastest possible routing simulations and calculations to customers at a price that Khasis says is “impossible without a mature cloud-based platform like GCP.”

Unexpected savings, pleasant surprises

The migration to GCP and Kubernetes Engine required Route4Me to revamp its Service-Oriented Architecture (SOA) and convert millions of lines of code into containerized microservices running on Kubernetes Engine. With more than 150 microservices and thousands of add-on modules and features offered on the Route4Me platform, the migration took several months. But the transition, which began in May 2017 and concluded toward year’s end, went smoothly. “Thanks to the reliability and open source portability of Google Kubernetes Engine, Route4Me experienced one-tenth of the problems that we’ve had when onboarding to other cloud providers,” says Khasis.

Halfway into the migration, Route4Me engineers discovered an unexpected cost savings. The ability to run preemptible virtual machine (VM) instances with Kubernetes Engine resulted in a 90% savings in infrastructure costs, according to Khasis.

The engineering team was also pleasantly surprised by the improved intra-system latency and performance between the Google network and those of third-party systems and other data centers that Route4Me connects to. Overall latency dropped from 8x to 12x. “Where it used to take 8 to 14 seconds to plan a complicated route, now it takes as little as 2 seconds,” Khasis says. Route4Me is also running most of its transactional and operational data through Google BigQuery for a variety of business use cases, including complex machine learning tasks such as geospatial analytics, geospatial pattern detection, and synthetic density.

Scaling while delivering great performance

Route4Me algorithms take into account such data as driving distance, driving time, who’s driving, the day of the week, the vehicle being used, weather conditions, and dozens of other attributes. “All those scenarios and data have to be run in near real time,” Khasis explains. The Route4Me system must access multiple internal and external databases, aggregate all the information in parallel, and deliver it using a high-speed infrastructure platform.

“Our core services and algorithms work much faster on a Google architecture, bringing the total time to solve a complex route problem down to single-digit seconds.” “Many of those steps are resource-intensive,” Khasis adds. “With Kubernetes Engine clusters, we can do much more, scaling up and down as needed, and still deliver great performance to customers around the world.”

Because of its scale, Route4Me built its own automation system for marketing, support, and communications with its customers. “Since we moved our proprietary marketing automation system to GCP, we began delivering our omni-channel marketing communications more reliably, and the correct message reached customers faster and at just the right moment,” says Khasis. “That’s translated to happier customers and increased revenue.”

Customer satisfaction has increased, too, because Route4Me’s users experience far fewer slowdowns than before due to the reliability of GCP. The reliability also means the company spends less time worrying about certain clusters or servers going down for extended periods of time. “We have zero sysadmins, which was the Achilles heel of some of my previous startups,” says Khasis. “So we can focus on software development rather than infrastructure management.”

In order to scale as needed and develop new features, Khasis had expected the company would need to hire more SysAdmin, DevOps, and SecOps staff. “But once we migrated to the modern GCP environment, we didn’t have to make those hires. We saved a lot of money by not having to hire, train, and manage more people,” explains Khasis.

Flexible GCP pricing, in which customers only pay for what they use, has saved Route4Me money on its IT infrastructure. “Preemptible server pricing on GCP is so aggressive,” Khasis says. “If servers are automatically shut off for a certain time period, we don’t pay for them for that period. And if servers are on for a certain amount of time, we get an automatic 30% discount. We’re saving money on the platform with fixed and dynamic workloads.”

Per-second billing with GCP also helps Route4Me cut costs. “If it only takes 25 seconds to do something, we only pay for those 25 seconds,” Khasis says. For the same 25 seconds, other cloud providers might charge for 10 minutes usage or even an hour.”

Road map for the future

In the coming year, Route4Me plans to offer additional add-ons as part of its self-service marketplace, providing customers with transparent pricing on highly complex route optimizations. The service will be extremely valuable to heavy users. For instance, if an organization has to visit 50,000 locations by a certain time, it might wonder if it needs to add 20 people to make that happen and how much it’s going to cost. “Because we’re on GCP, our customer can run a variety of complicated routing scenarios to see which one is the most efficient in seconds instead of minutes,” says Khasis. “As far as I know, none of our competitors can offer that kind of service, giving us an edge as well as a new revenue stream.”

Going forward, Route4Me will begin migrating a huge portion of its core routing optimization platform to Google Google Cloud Spanner. “We want to take further advantage of Cloud Spanner, which comes closest to the CAP theorem and permits us to operate an infinitely scalable and nearly indestructible platform,” Khasis says.

As one example, Route4Me receives telematics data, such as GPS coordinates, from Internet of Things (IoT) devices in smartphones and vehicles, and performs complex algorithmic analysis running on Cloud Spanner. This provides real-time return on investment (ROI) information, so customers can see how much money they’re saving by using Route4Me routing optimization services.

“In order to help as many logistics-intensive businesses as possible, we intend to migrate our proprietary mapping, routing, and route optimization services to Cloud Spanner to take advantage of its extreme reliability and redundancy, and the multi-availability zones of Google Cloud Platform,” says Khasis.

Route4Me also plans to leverage Google machine learning technology, in part to make its routing solution available for use in autonomous and drone vehicles, as well as decentralized edge computing deployments. In addition, Google security and encryption technology will help the company expand its offerings to the heavily regulated medical industry.

Over 60 Route4Me team members use G Suite for almost everything. ”We’re interested in using everything possible with G Suite. We get inspiration from G Suite, too. A lot of thinking and effort went into improving G Suite, and we use that as inspiration to improve own products.”