It’s incredible to see how startups across industries are using cloud technology to help address some of our most pressing, important, and life-altering challenges. Startups and high-growth technology companies are choosing Google Cloud and using technologies like Google Compute Engine (GCE), BigQuery, Looker, Firebase and more to help businesses reduce energy consumption, build more inclusive and sustainable workforces, and in the case of high-growth biotech company Freenome, are creating diagnostic tests that will help detect life-threatening diseases like cancer in the earliest, most-treatable stages.

Freenome is driven by its mission to develop high-quality diagnostic tests to detect and treat diseases like cancer from a simple blood draw. In 2022, Freenome significantly accelerated its growth on Google Cloud to support the business as it began the clinical trials of its diagnostic blood testing technology. Today, the high-growth biotech is further deepening its partnership with Google Cloud in order to support its rapid growth and scale as it concludes clinical validation, takes its tests through FDA approvals, and prepares to bring its product to market.

When detected early, data shows that there is a higher probability for cancer to be beaten, yet not everyone has access to early detection measures. By creating a way to detect the earliest warning signs of cancer with a standard blood test, Freenome is helping bridge the gap between accessibility and early cancer detection. To do this, Freenome built a multiomics platform capable of analyzing and detecting disease-associated patterns in the blood using molecular biology, advanced biology, and machine learning. By applying machine learning models trained to scan for tumor and non-tumor biomarkers to the diagnostic process, Freenome’s tests can identify suspicious molecular patterns in a patient’s blood, which will ultimately help more people detect cancer at its earliest stages in the body.

The amount of molecular data extracted from blood samples can quickly add up to hundreds of terabytes worth of data, so it was clear early on that Freenome would need infrastructure that could support the fast sequencing and processing of large amounts of data. In addition, Freenome’s collaboration technology needed to provide flexibility, security, and proper identity management safeguards, given the nature of its business. To meet these needs and support the company’s plans for growth and innovation, Freenome selected Google Cloud as its primary cloud provider, utilizing services like Google Cloud Storage and Google Kubernetes Engine (GKE), along with Google Workspace as its collaboration platform.

Using Cloud Storage alongside GKE gives Freenome the computing power needed to sequence and process mass amounts of blood sample data with high-performance, speed, and at scale. Cloud Storage also makes it easy for Freenome to leverage other Google Cloud capabilities like BigQuery for analytics with built-in query acceleration. Additionally, the built-in cluster management capabilities of GKE make it easy for Freenome’s engineering and IT teams to manage and deploy new workflows to the high-performance computing clusters used by the machine learning components of its multiomics platform to speed up cancer detection. Freenome also uses Google Cloud technologies like Artifact Registry and Cloud SQL, which help the company ensure a managed and secure software supply chain of containers and other artifacts.

Today, as a part of its expanded partnership with Google Cloud, Freenome is significantly increasing its use of Cloud Storage and GKE as it works to complete the clinical trials related to its diagnostic test. By expanding its use of GKE and Cloud Storage, Freenome will be equipped to perform the compute-intensive analytical work required for running its research workflow and diagnostic classifier algorithms. In addition, Freenome teams will continue leveraging Google Workspace products across the company so they can securely manage and collaborate on business-critical content.

Besides using Cloud Storage, GKE, and Google Workspace to support the company’s rapid growth, Freenome plans to leverage Google Cloud technologies like BigQuery to support the research and development of new products. The company is also testing security technologies like BeyondCorp to keep its growing workforce secure and productive at scale.

As the future of disease detection continues to evolve, Google Cloud is proud to support the growth of innovative companies like Freenome with infrastructure and cloud technologies so they can help empower more people with early disease detection solutions and ultimately, change more lives for the better.

When Dassana co-founders Gaurav Kumar and Parth Shah, formerly founder and founding engineer at RedLock (now Prisma Cloud by Palo Alto Networks), set out on a new startup journey in 2020, they knew exactly where to start: sign up for Google Workspace.

“Every startup I’ve been at, we used Google Workspace,” said Kumar. “We’ve been using it for so long, and we’re all used to it. It’s like drinking water—you don’t think about it.”

“We’re big on user experience,” explained Shah. “Google is one of the few companies out there that is all about building the right kind of user experience that’s easy to follow. The sharing capabilities are amazing and, of course, easy to use. Docs, Sheets, Slides—we use all of it.”

Dassana, which emerged from stealth with $5 million in seed funding earlier this year, is a next-generation security data lake. It provides a holistic picture of security risk across an organization and its business units by ingesting large volumes of structured data in a schema-less fashion. Their success is a great testament to why startups are choosing not only Google Workspace, but a range of Google Cloud products.

Though the Dassana team was comfortable with Workspace from the start, not all their early technology choices were the best fit for the company, and Google Cloud services became more crucial as the startup evolved. For example, the team opted to use Amazon Web Services (AWS) to start their cloud journey, but ultimately started to explore other cloud options when they decided to run their technology platform on Kubernetes.

“We started looking into which cloud platforms provide the best Kubernetes experience,” said Kumar. “Hands-down, Google Kubernetes Engine (GKE) had the best experience. If you look at product velocity and how GKE has evolved over time, from its early days to GKE Autopilot and all the features and other native integrations—nothing even comes close.”

In particular, Kumar noted that the native integration between Google Workspace and GKE was particularly unique and useful. “When I go to Google Workspace and then go to GKE, my identity is already there,” he said. “I don’t have to integrate or manage anything. If I disable an account in Workspace, it’s also disabled on GKE.”

Another advantage is that GKE allows you to set up Google Groups to work with Kubernetes role-based access control (RBAC) for GKE clusters. This lets administrators maintain users and groups outside of GKE and assign RBAC permissions directly to Groups in Workspace without any extra engineering work or overhead management.

“I can actually use my Workspace identity to give granular controls to my GKE workloads. The integration of Google Groups in GKE and Kubernetes is a lifesaver. It’s saved us a lot of hassles,” said Kumar. Kumar also noted that the platform delivers better performance compared to other solutions thanks to the low latency of Google Cloud’s global network.

Like many startups, Dassana has found a powerful combination in Google Workspace and GKE on Google Cloud that lets them connect technologies, easily collaborate with their teams, and rapidly build their product. The team also recognizes the necessity of continued innovation, and is exploring additional Google Cloud products to help them accelerate their momentum. For example, Dassana plans to use the performance and scale of Cloud Storage buckets to store the company’s data. The team is also investigating how to save time by using Pub/Sub to integrate data directly from Google Workspace and other sources for security analytics.

To learn more about why startups like Dassana are choosing Google Workspace and Google Cloud to accelerate their growth and reach their goals, visit our startup solutions pages for Google Workspace and Google Cloud.

Google Cloud’s AI-optimized infrastructure makes it possible for businesses to train, fine-tune, and run inference on state-of-the-art AI models faster, at greater scale, and at lower cost. We are excited to announce the preview of inference on Cloud TPUs. The new Cloud TPU v5e enables high-performance and cost-effective inference for a broad range AI workloads, including the latest state-of-the-art large language models (LLMs) and generative AI models.

As new models are released and AI becomes more sophisticated, businesses require more powerful and cost efficient compute options. Google is an AI-first company, so our AI-optimized infrastructure is built to deliver the global scale and performance demanded by Google products like YouTube, Gmail, Google Maps, Google Play, and Android that serve billions of users — as well as our cloud customers. 

LLM and generative AI breakthroughs require vast amounts of computation to train and serve AI models. We’ve custom-designed, built, and deployed Cloud TPU v5e to cost-efficiently meet this growing computational demand.

Cloud TPU v5e is a great choice for accelerating your AI inference workloads: 

• Cost Efficient: Up to 2.5x more performance per dollar and up to 1.7x lower latency for inference compared to TPU v4.
• Scalable: Eight TPU shapes support the full range of LLM and generative AI model sizes, up to 2 trillion parameters. 
• Versatile: Robust AI framework and orchestration support. 

In this blog, we’ll dive deeper into how you can leverage TPU v5e effectively for AI inference.

Up to 2.5x more performance per dollar and up to 1.7x lower latency for inference

Each TPU v5e chip provides up to 393 trillion int8 operations per second (TOPS), allowing complex models to make fast predictions. A TPU v5e pod consists of 256 chips networked over ultra-fast links. Each TPU v5e pod delivers up to 100 quadrillion int8 operations per second, or 100 PetaOps, of compute power.

We optimized the Cloud TPU inference software stack to take full advantage of this powerful hardware. The inference stack leverages XLA, Google’s AI compiler, which generates highly-efficient code for TPUs to maximize performance and efficiency.

The combined hardware and software optimizations, including int8 quantization, enable Cloud TPU v5e to achieve up to 2.5x greater inference performance per dollar than Cloud TPU v4 on state-of-the-art LLM and generative AI models, including Llama 2, GPT-3, and Stable Diffusion 2.1:

Google Internal Data. August 2023. Normalized to single-chip throughput. Precision: Llama 2 7B, 13B, 70B, GPT-J 6B: int8; GPT-J 175B, Stable Diffusion 2.1: bf16.

On latency, Cloud TPU v5e achieves up to 1.7x speedup compared to TPU v4:

Google Internal Data. August 2023. Precision: Llama 2 7B, 13B and 70B: int8; GPT-3 175B: bf16.

Google Cloud customers have been running inference on Cloud TPU v5e, and some have seen even greater speedups on their particular workloads.

AssemblyAI offers dozens of AI models to their customers for speech recognition and understanding with over 25 million inference calls on a daily basis. 

“Cloud TPU v5e consistently delivered up to 4X greater performance per dollar than comparable solutions in the market for running inference on our production model. The Google Cloud software stack is optimized for peak performance and efficiency, taking full advantage of the TPU v5e hardware that was purpose-built for accelerating the most advanced AI and ML models. This powerful and versatile combination of hardware and software dramatically accelerated our time to solution: instead of spending weeks hand-tuning custom kernels, within hours we optimized our model to meet and exceed our inference performance targets.” – Domenic Donato, VP of Technology, AssemblyAI

Scale to the full range of LLM and Generative AI model sizes 

LLMs and generative AI models continue to grow in size and computational cost. The largest models require the combined compute and memory of hundreds of hardware accelerators. Cloud TPU v5e enables inference for a wide range of model sizes. A single v5e chip can run models with up to 13B parameters. From there, you can scale up to hundreds of chips and run models with up to 2 trillion parameters.

Google Internal Data. August 2023. Batch size = 1. Multi-head attention based decoder only language models: prefix length = 2048, decode steps = 256, beam size = 32 for sampling.

Gridspace leverages Google Cloud TPU infrastructure to power its full-stack conversational AI platform – building and integrating real-time conversational ASR, LLMs, semantic search, and neural TTS.

“We’re a huge fan of Google Cloud TPUs. Our benchmarks are demonstrating a 5X increase in the speed of AI models when training and running on Google Cloud TPU v5e. We are also seeing a 6x improvement in the scale of our inference metrics. We’ve scaled our AI models to billions of conversations per year across financial services, capital markets, and healthcare with Google Cloud’s AI infrastructure. Our Grace bots are powered by models trained using Cloud TPUs and served at scale on GKE with support for PCI, HITRUST, and SOC 2 compliance.” – Wonkyum Lee, Head of Machine Learning, Gridspace 

Robust AI framework and orchestration support

Leading AI frameworks, including PyTorch, JAX, and TensorFlow, provide robust support for inference on Cloud TPU v5e. This means you can now train and serve models end-to-end on Cloud TPUs: what you train is what you serve.

Google Cloud offers you many choices to run inference on Cloud TPUs easily and reliably. From GKE and Vertex AI, to popular open-source frameworks such as Ray and Slurm, you can leverage Google Cloud TPUs in your preferred way to fit your development process.

Try Cloud TPU v5e for inference today

Cloud TPU v5e provides a high-performance, cost-efficient, scalable, and reliable inference platform for LLMs and generative AI models. Leading AI companies are leveraging the power of Cloud TPU v5e to serve AI models at scale:

To get started with inference on Cloud TPU, reach out to your Google Cloud account manager or contact Google Cloud sales.