Editor’s note: Today’s post highlights five takeaways from our session at the first ever Google Cloud Public Sector Summit. To watch the full session, check out All the Right Moves: Prioritizing Investments in Technology.

Now more than ever, government agencies need to invest in digital services to fulfill their missions and better serve communities. Yet modernization isn’t a one-and-done approach; it’s a sustained effort, with multi-year implications, and requires careful consideration of how to integrate existing investments to optimize costs. Digital transformation requires coordination between different programs and agencies, including all of their many competing considerations. In short, maximizing technology investments requires careful planning, strategic thinking, and industry partners that can provide flexibility and security and meet agencies where they are. 

I sat down with Suzette Kent, former U.S. federal chief information officer, and Dominic Sale, assistant commissioner for Technology Transformation Services at the General Services Administration (GSA) for a conversation to unpack this important topic and discuss industry best practices. 

The panelists had five tips for government employees who are making technology purchasing decisions for their agency.

1.   Put the agency’s mission first.Avoid getting distracted by exciting new trends and focus on long-term goals that can impact which procurement strategies or funds could be used. The discussion started with how government agencies could cut through the noise about technology and prioritize which technology is best for their needs. Sale and Kent agreed that an agency should focus on its core mission outcomes and let its technology needs flow from that. Sale also emphasized the importance of having technology design respond to humans’ needs, which has historically been a challenge for government agencies.

For agencies to stay focused on their mission, final decisions about technology need to be made by the program manager who best understands each program’s mission. The government CIO’s role is to be the enabler for the technology and leverage it at the enterprise level, particularly when it comes to sharing infrastructure.  The takeaway: enable mission programs by empowering your teams and providing access to authorized, compliant, innovative data platforms that programs can move confidently and quickly with.

2.   Invest in interoperability. Agency employees often struggle to balance the need for a positive return on investment (ROI) with requirements for meeting mission objectives. While the panelists agreed that the total cost of ownership was important, they also emphasized taking an expanded view of ROI, including future-proofing and investing in functionality that may not realize its return for many years based on the initiative. Saving money isn’t particularly valuable if the solution doesn’t meet an agency’s needs. When choosing a technology partner, government employees should understand its long-term vision to ensure that the partner fits agency priorities. Partners’ technologies should also integrate seamlessly with existing systems so agencies don’t duplicate investment costs.

For example, Google Anthos extends Google Cloud services and engineering practices into an organization’s existing environment, establishing operational consistency across apps and modernization. With Anthos, agencies can simply and securely build and deploy applications anywhere, integrating cloud services across platforms. This allows them to enjoy a consistent DevOps experience for hybrid and multi-cloud environments and enables new innovation. Most importantly, this enables an enterprise data platform, one of the largest catalysts for mission transformation and applied AI.

3.   Take advantage of artificial intelligence (AI) benefits. Over the course of the pandemic, the rapid application of AI has improved government productivity, efficiency, and the ability to deliver critical new services to the public at scale. This has further cemented AI’s role as an essential government technology for the present and future. In fact, Nextgov reports that “46% of government IT specialists plan to use AI and machine learning (ML) for embedded systems in the near future.”

As we’ve seen over the course of the pandemic, government programs can start small with AI  pilots before moving into broad deployment. This can help agencies understand AI’s potential before moving to full production. People always supervise AI technologies, and the possibilities are endless. Google Cloud’s Contact Center AI (CCAI) has helped government agencies improve the customer experience, by allowing citizens to schedule vaccine appointments via a platform of their choice with up to 28 languages and dialects, and manage vaccine deployment. The U.S. Navy spends billions annually to fight rust and corrosion on its ships. Inspections of ships, aircraft and vehicles are a time-consuming and critical part of keeping the U.S. Navy at top performance so Google Cloud and Simple Technology Solutions (STS) rapidly built an AI-based corrosion-detection and analysis system. The system detected and analyzed corrosion on vessels with 90% accuracy and will eventually be used to automate inspections of vessels, aircraft, and vehicles—saving billions of dollars. Document AI helps a variety of government agencies scale their document processing, reducing the time it typically takes to process enormous amounts of data and related citizen claims.

Successful adoption of AI also depends on the quality of the data. Ultimately, agencies need high-quality enterprise data pipes so that employees and the community trust the system and public sector agencies. Sale described a GSA project that used AI bots to read legal contracts and look for particular phrases that would indicate a specific use case. Previously, an employee would have had to read through the contracts and search for the information. In this way, AI is saving the government both money and time.

4.   Creating better experiences for the public. Sale observed that, “trust is the government’s currency and profit motive.” And trust comes when the public can be served with the same modern tools and technology they’re used to – in real-time and with transparency in mind. For example, agencies can provide transparency in public-facing dashboards for programs and supply services that deliver information in real-time through solutions like CCAI.

Trust also requires that constituents feel that government agencies will keep their data safe and secure. The need for a globally secure infrastructure with systems that are up-to-date and designed with security at every level, underpinned by zero-trust enterprise-wide remains paramount – particularly after the series of recent cyberattacks targeting government IT infrastructure.

5.   Finding the right technology partner. Government leaders need technology partners who  provide a flexible and interoperable platform to integrate existing investments and maximize technical value. Historically, public sector agencies have largely been forced to adopt private clouds, which has reduced their access to richer features, and hindered their ability to adopt a full range of security and product capabilities. The right partner won’t require government leaders to compromise on functionality or service availability to achieve compliance. The right partner can harness the power of emerging technology to make it Government-ready and the true promise of cloud– the access and integration of open data– to make missions more powerful and impactful for the constituencies they serve.

Google Cloud’s AI-optimized infrastructure makes it possible for businesses to train, fine-tune, and run inference on state-of-the-art AI models faster, at greater scale, and at lower cost. We are excited to announce the preview of inference on Cloud TPUs. The new Cloud TPU v5e enables high-performance and cost-effective inference for a broad range AI workloads, including the latest state-of-the-art large language models (LLMs) and generative AI models.

As new models are released and AI becomes more sophisticated, businesses require more powerful and cost efficient compute options. Google is an AI-first company, so our AI-optimized infrastructure is built to deliver the global scale and performance demanded by Google products like YouTube, Gmail, Google Maps, Google Play, and Android that serve billions of users — as well as our cloud customers. 

LLM and generative AI breakthroughs require vast amounts of computation to train and serve AI models. We’ve custom-designed, built, and deployed Cloud TPU v5e to cost-efficiently meet this growing computational demand.

Cloud TPU v5e is a great choice for accelerating your AI inference workloads: 

• Cost Efficient: Up to 2.5x more performance per dollar and up to 1.7x lower latency for inference compared to TPU v4.
• Scalable: Eight TPU shapes support the full range of LLM and generative AI model sizes, up to 2 trillion parameters. 
• Versatile: Robust AI framework and orchestration support. 

In this blog, we’ll dive deeper into how you can leverage TPU v5e effectively for AI inference.

Up to 2.5x more performance per dollar and up to 1.7x lower latency for inference

Each TPU v5e chip provides up to 393 trillion int8 operations per second (TOPS), allowing complex models to make fast predictions. A TPU v5e pod consists of 256 chips networked over ultra-fast links. Each TPU v5e pod delivers up to 100 quadrillion int8 operations per second, or 100 PetaOps, of compute power.

We optimized the Cloud TPU inference software stack to take full advantage of this powerful hardware. The inference stack leverages XLA, Google’s AI compiler, which generates highly-efficient code for TPUs to maximize performance and efficiency.

The combined hardware and software optimizations, including int8 quantization, enable Cloud TPU v5e to achieve up to 2.5x greater inference performance per dollar than Cloud TPU v4 on state-of-the-art LLM and generative AI models, including Llama 2, GPT-3, and Stable Diffusion 2.1:

Google Internal Data. August 2023. Normalized to single-chip throughput. Precision: Llama 2 7B, 13B, 70B, GPT-J 6B: int8; GPT-J 175B, Stable Diffusion 2.1: bf16.

On latency, Cloud TPU v5e achieves up to 1.7x speedup compared to TPU v4:

Google Internal Data. August 2023. Precision: Llama 2 7B, 13B and 70B: int8; GPT-3 175B: bf16.

Google Cloud customers have been running inference on Cloud TPU v5e, and some have seen even greater speedups on their particular workloads.

AssemblyAI offers dozens of AI models to their customers for speech recognition and understanding with over 25 million inference calls on a daily basis. 

“Cloud TPU v5e consistently delivered up to 4X greater performance per dollar than comparable solutions in the market for running inference on our production model. The Google Cloud software stack is optimized for peak performance and efficiency, taking full advantage of the TPU v5e hardware that was purpose-built for accelerating the most advanced AI and ML models. This powerful and versatile combination of hardware and software dramatically accelerated our time to solution: instead of spending weeks hand-tuning custom kernels, within hours we optimized our model to meet and exceed our inference performance targets.” – Domenic Donato, VP of Technology, AssemblyAI

Scale to the full range of LLM and Generative AI model sizes 

LLMs and generative AI models continue to grow in size and computational cost. The largest models require the combined compute and memory of hundreds of hardware accelerators. Cloud TPU v5e enables inference for a wide range of model sizes. A single v5e chip can run models with up to 13B parameters. From there, you can scale up to hundreds of chips and run models with up to 2 trillion parameters.

Google Internal Data. August 2023. Batch size = 1. Multi-head attention based decoder only language models: prefix length = 2048, decode steps = 256, beam size = 32 for sampling.

Gridspace leverages Google Cloud TPU infrastructure to power its full-stack conversational AI platform – building and integrating real-time conversational ASR, LLMs, semantic search, and neural TTS.

“We’re a huge fan of Google Cloud TPUs. Our benchmarks are demonstrating a 5X increase in the speed of AI models when training and running on Google Cloud TPU v5e. We are also seeing a 6x improvement in the scale of our inference metrics. We’ve scaled our AI models to billions of conversations per year across financial services, capital markets, and healthcare with Google Cloud’s AI infrastructure. Our Grace bots are powered by models trained using Cloud TPUs and served at scale on GKE with support for PCI, HITRUST, and SOC 2 compliance.” – Wonkyum Lee, Head of Machine Learning, Gridspace 

Robust AI framework and orchestration support

Leading AI frameworks, including PyTorch, JAX, and TensorFlow, provide robust support for inference on Cloud TPU v5e. This means you can now train and serve models end-to-end on Cloud TPUs: what you train is what you serve.

Google Cloud offers you many choices to run inference on Cloud TPUs easily and reliably. From GKE and Vertex AI, to popular open-source frameworks such as Ray and Slurm, you can leverage Google Cloud TPUs in your preferred way to fit your development process.

Try Cloud TPU v5e for inference today

Cloud TPU v5e provides a high-performance, cost-efficient, scalable, and reliable inference platform for LLMs and generative AI models. Leading AI companies are leveraging the power of Cloud TPU v5e to serve AI models at scale:

To get started with inference on Cloud TPU, reach out to your Google Cloud account manager or contact Google Cloud sales.

“Talk about tough times, right?”

That’s how Dan Stuart, Senior Vice President of IT Services at Southwire Company, refers to the months following a December 2019 ransomware event, and the COVID crisis that began in spring of 2020. Those events hit just as the company was preparing for an overhaul of their SAP environment. This comprehensive plan included three key elements. First, the company wanted to upgrade their SAP ECC environment to take advantage of the latest functionality available for this critical ERP system. Second, Southwire aimed to deploy SAP Business Warehouse on SAP HANA to accelerate vital reporting for all business users. Third, the company wanted to upgrade to the latest version of SAP Process Orchestration—an essential component that touches key manufacturing interfaces in all Southwire facilities. 

Southwire had looked at multiple options for the upgrades, including remaining entirely on-premises, colocation, and full cloud migration. “Going to the cloud seemed a lot more compelling,” says Joe Schleupner, Southwire’s Senior Director of PMO & ITS planning and implementation. “We were going to the cloud eventually, so why take these intermediary steps? Let’s just get it done.”

After looking at several options, Southwire decided to migrate to Google Cloud. “We wanted to be on a platform for SAP that was flexible, scalable, and secure; that we could count on to get up and running quickly,” says Stuart. “We chose Google Cloud not only for those reasons, but also because we recognize that Google has other assets that we may be able to take advantage of down the line, such as technologies like artificial intelligence (AI).” 

More stability, less worry

As one of the leading manufacturers of wire and cable used in the transmission and distribution of electricity, Southwire aids the delivery of power to millions of people worldwide. They have more than 30 manufacturing facilities across the United States running 24/7. Any downtime directly affects productivity and revenue. With help from Google Cloud and their implementation partner NIMBL, Southwire completed the SAP migration to Google Cloud over a planned maintenance weekend on July 4th.

The migration itself, while complex, went quickly and smoothly. “Just moving to the cloud was quite a feat because we were dealing with so much data, but in total the SAP system was down for only ~16 hours,” says Schleupner.

“As a project manager, I always felt that Google Cloud had my back” Schleupner says. The Process Orchestration (PO) migration was of particular concern, considering that it controlled all of Southwire’s manufacturing interfaces across the entire company. “Every critical piece of information that goes from SAP down to the manufacturing system goes through that system,” says Schleupner.

Even after migrating, Southwire discovered that making changes to the system was fast, easy, and resulted in no downtime. Normally, certain types of changes would have involved taking down SAP for at least an hour.

The Southwire team also appreciates the fact that the modern cloud architecture means spending less time on routine infrastructure maintenance. “It’s one less thing for me to worry about,” Stuart says, “I can focus on the business side of the house and move the technology and responsibilities to what we do within the Google Cloud Platform.”

What comes next?

While the cloud migration will increase stability, uptime, performance, and security, there is much more to come. Southwire is currently working on a disaster recovery implementation for their SAP environment on Google Cloud. Stuart and Schleupner are excited about where Google Cloud can further take Southwire. They are considering an SAP Hybris e-commerce implementation as well as connected factory and/or factory automation initiatives that can take advantage of artificial intelligence and machine learning. 

To Stuart and Schleupner, the migration of Southwire’s SAP environment to Google Cloud, as important as it was, really represents the first step in the company’s tech evolution. Now that much of the heavy lifting is complete, Southwire’s digital transformation can begin in earnest. “There’s no shortage of areas where I think Google Cloud will come into play,” Stuart says, “and we intend to look at these things with an open mind to understand how we can leverage current investments to take our organization where we want to go.”

Learn more about Southwire’s SAP on Google Cloud deployment and how Google Cloud can transform the way you work with your SAP enterprise applications. Visit cloud.google.com/solutions/sap.

About Vimeo

Vimeo gives video creators the tools to host, share, and sell videos of the highest quality possible. It reaches viewers in over 150 countries who can watch content anytime, on nearly every Internet-connected device.

Industries: Media & Entertainment

Location: United States

About Fastly

Fastly helps the world’s most popular digital businesses keep pace with their customer expectations by delivering fast, secure, and scalable online experiences. Businesses trust the Fastly edge cloud platform to accelerate the pace of technical innovation, mitigate evolving threats, and scale on demand.

Industries:

Location:

Vimeo is a video-sharing platform that’s home to imaginative video creators and hundreds of millions of viewers. Sixty million people create, host, and sell high-quality videos on Vimeo, including more than 800,000 who subscribe to the service’s premium tools. Over 240 million people in more than 150 countries watch videos monthly.

Vimeo was using its own servers to allow users to upload videos to its service, a cloud storage platform for storing videos, and as an alternative solution for streaming. It was looking for a solution that would do away with its own servers for uploading. Vimeo built a new adaptive video-delivery service on Google Cloud Platform and the Fastly edge cloud that can scale on demand to meet Vimeo’s growing needs for video streaming.

“Our business is dependent on delivering high-quality video; that’s our competitive edge,” says Naren Venkataraman, Senior Director of Engineering at Vimeo. “Thanks to Fastly and Google Cloud Platform, we’re delivering more high-quality videos than ever at less cost, leading to our continuing success and growth.”

Tuning video delivery

Building a great video experience begins with a fast, reliable upload service. Vimeo replaced its servers for accepting video uploads with Google Cloud Storage, fronted by the Fastly edge cloud to help ensure regional routing and low-latency, high-throughput connections for Vimeo’s publishers. Multi-regional Google Cloud Storage offers fast, resumable upload capability that helps make for better user experience.

The video delivery service transcodes videos and streams them to users—videos are customized depending on network traffic and the devices to which the videos are delivered. The goal is to deliver the highest-quality, smooth playback experience across all platforms over varying network conditions and device capabilities.

Google Compute Engine packages the videos, which are stored on Google Cloud Storage. Google Compute Engine can automatically scale to allow Vimeo to deliver videos on the fly, even when demand spikes and many users stream videos simultaneously across a very diverse library. The low latency of Google Cloud Storage helps with fast startup times, while providing scalable storage to host millions of videos from Vimeo’s loyal community of content creators.

“Fastly and Google Cloud Platform enabled us to build a low-latency, highly scalable, on-the-fly adaptive video streaming packager in a short period of time with a small team,” says Naren.

High-quality video means more users

With Fastly and Google Cloud Platform, Vimeo is delivering more and higher-quality videos to its users because of the platform’s low latency, high bandwidth, and ability to scale. Because of the system’s reliability, fewer users stop watching videos because of delays and glitches. Vimeo engineers do not have to spend their time managing infrastructure and now focus on improving the video delivery service, leading to improved customer satisfaction. Costs are reduced because Vimeo does not have to manage the infrastructure in-house.

“We’ve chosen Google for Fastly’s Cloud Accelerator because at Google innovation happens faster, and Google Cloud Platform is driving cloud computing and cloud storage in the right direction,” says Lee Chen, Head of Strategic Partnerships at Fastly.

The Home Depot (THD) is the world’s largest home-improvement chain, growing to more than 2,200 stores and 700,000 products in four decades. Much of that success was driven through the analysis of data. This included developing sales forecasts, replenishing inventory through the supply chain network, and providing timely performance scorecards.

However, to compete in today’s business world, THD has taken this data-driven approach to an entirely new level of success on Google Cloud, providing capabilities not practical on legacy technologies.

The pressures of contemporary growth that drove much of the work are familiar to many businesses. In addition to everything it was doing, THD needed to better integrate the complexities in its related businesses, like tool rental and home services. It needed to better empower teams, including a fast-growing data analysis staff and store associates with mobile computing devices. It wanted to better use online commerce and artificial intelligence to meet customer needs, while maintaining better security.

Even before addressing these new challenges, THD’s existing on-premises data warehouse was under stress as more data was required for analytics and data analysts were utilizing the data with increasingly complex use cases. This drove rapid growth of the data warehouse, but also created constant challenges for the team in managing priorities, performance, and cost.

In order to add capacity to the environment, it was a major planning, architecture, and testing effort. In one case, adding on-premises capacity took six months of planning and a three-day service outage. Within a year, capacity was again scarce, impacting performance and ability to execute all the reporting and analytics workloads required. The capacity refresh cycles were shrinking, and the expecations for data were growing. There had to be a better way.

Still, THD did not take its move to the cloud lightly. A large-scale enterprise data warehouse migration involves tremendous effort among people, process, and technology. After careful consideration, THD chose Google Cloud’s BigQuery for its cloud enterprise data warehouse.

BigQuery, a scalable serverless data warehouse, was better on cost, infrastructure agility, and analytics capability, driving better insights with improved performance. There are no service interruptions when capacity is added, and that capacity can be added within a week (and soon same day). It doesn’t require complex system administration, and its standard SQL support means people can easily ramp up quickly. Valuable BigQuery products like Identity and Access Management meant THD could create many separate Google Cloud projects, while ensuring that different teams weren’t interfering with each other or accessing protected data.

THD also utilizes BigQuery’s flat-rate monthly pricing model that allows teams to budget their capacity based on need and provides billing predictability. The capacity not being used by a given project is available for enterprise use. This ensures no surprises when the monthly bill arrives and provides all analytical users access to significant computing power.

While THD’s legacy data warehouse contained 450 terabytes of data, the BigQuery enterprise data warehouse has over 15 petabytes. That means better decision-making by utilizing new datasets like website clickstream data and by analyzing additional years of data.

As for performance, look at this chart:

With the cloud EDW migration complete, and the legacy on-premises data warehouse retired, analysts now execute more complex and demanding workloads that they would not have been able to complete before, such as utilizing Datalab for orchestrating analytics through Python Notebooks, utilizing BigQuery ML for machine learning directly against the BigQuery data (no movement of large datasets), and AutoML to help determine the best model for predictions.

Additionally, engineers at THD have adapted BigQuery to monitor, analyze, and act on application performance data across all its stores and warehouses in real time, something that was not practical in the on-premises system.

With over 600 projects that THD now has on Google Cloud, the BigQuery story is just one of the many ways that Google Cloud is working with THD to deliver meaningful business results, every day.