Powered by Vertex AI (Google Cloud’s platform for accelerating development and deployment of machine learning models into production), SAVI (Semi Automated Vision Inspection)1 is transforming surgical instrument identification and cataloging, leading to fewer canceled surgeries and easing pressure on surgery waitlists.

Max Kelsen, an analytics and software agency that specializes in machine learning, has worked closely with Google Cloud and Johnson & Johnson MedTech to create a system that can manage tens of thousands of individual devices, their characteristics, and how they apply to each set or tray used by a surgeon. SAVI does this while delivering a one in 10,000 real-world error rate, much faster and more accurately than manual processes currently in use across the industry. Implementing SAVI can also unlock end-to-end visibility and traceability across the surgical set supply chain and provide advanced analytics and insights.

Eliminating time-consuming manual processes

Surgeons need a large number of specialist instruments and devices to complete complex, delicate procedures. Because each tray of these instruments can typically cost more than $350,000, and having every type of set on shelf at every surgical facility is not feasible, manufacturers generally loan them to hospitals for procedures, such as inserting one of the manufacturers’ implants into a patient’s knee. Once a procedure is complete, the hospital returns the instrument tray to the manufacturer for storage and re-distribution to other hospitals as needed.

Each time a hospital returns a tray, the manufacturer needs to check that each instrument is there, correctly placed, cleaned, and fit for the purpose of the next procedure. As each set may hold more than 400 instruments, completing this process manually is complex and time-consuming. While each tray is checked before and after surgery at the hospital, and again when it arrives and leaves the manufacturer’s facility, Max Kelsen finds that 5% of surgeries can still be affected by missing, broken or bent instruments. This has a severe downstream impact on private hospitals in particular, directly affecting patient safety and outcomes; in Australia, for example, around 60% of surgeries are performed in private hospitals.

Johnson & Johnson MedTech has 60,000 surgical trays across the Asia-Pacific, and loans these trays out about 100,000 times per month. The manufacturer approached Max Kelsen to help design and develop a solution to make the supply chain more efficient, and to give more visibility into asset movement. As a Google Cloud Partner specializing in applying machine learning at scale in healthcare contexts, Max Kelsen had the expertise and track record to meet Johnson & Johnson MedTech’s need for a globally scalable solution that was engineered for quality and performance.

The first step was to establish a baseline for the project by determining how long the manufacturer’s team took to process each tray, and to set an efficiency number. We then spent six months determining and evaluating how to deliver a robust, accurate solution that outperformed current manual and labor-intensive methods in processing instruments and trays, globally. Our work included extensive technical feasibility research involving a representative sample for the variety and complexity of sets, trays, and devices needed for different types of surgery, including orthopedics, spinal trauma, and maxillofacial groups.

Working with Google Cloud to accelerate and de-risk the project

This is a familiar problem that is industry-wide. The issue has been widely explored and tried with a number of technologies over several years without producing the scalability and performance results required to make this an appropriate and feasible solution. Google Cloud partnered with Max Kelsen to accelerate and de-risk this large and strategic project for a mutual customer.

Technical feasibility took four months, prior to a year-long production pilot of SAVl in a distribution center in Queensland that services over 100 hospitals. After obtaining enough real-world data and experience to validate that the solution was as scalable and as accurate as needed, an Asia-Pacific rollout of the system commenced. SAVI is now live across Johnson & Johnson MedTech’s operations in Australia, New Zealand, and Japan, garnering recognition with a JAISA excellence award.

Google Cloud machine learning is integral to SAVI. Google Cloud’s technologies were a big differentiator for Max Kelsen’s engineering team in delivering the breakthroughs needed at scale, and in production, to meet Johnson & Johnson MedTech’s needs.

Reducing checking and documentation time

Running SAVI in Google Cloud has reduced the time Johnson & Johnson MedTech needs to check and document inspections of these surgical instrument sets by over 40%. The application also delivers consistent measurable quality that is often hard to measure at scale when using manual processes. During the pandemic, the application enabled Johnson & Johnson MedTech to operate with a lower headcount for the same volume output, enabling the organization to quickly service a backlog of waiting list surgeries.

In addition, the automation delivered with SAVI has reduced the time required to bring technicians up to speed on quality control processes, from eight to 12 months down to just three months, enhancing productivity and performance while delivering a more robust workforce.

So how does SAVI work in a real-world context? SAVI is deployed via a tablet and a web-based application incorporates an API to photograph the medical device trays, as shown below. Max Kelsen captures the photograph and sends it to a range of different services, via an API endpoint hosted on Google Cloud:

• Image information is stored in Cloud Storage
• Data relating to the trays is stored in Cloud SQL for PostgreSQL
• APIs and web UI components run in CloudRun
• Analytics data is stored within BigQuery

Once this tray and device onboarding stage is completed, the next step is to perform inferences from the images and data. By hosting online models with Kubeflow model serving on GKE, we enable a model to identify all the instruments in a tray at low latency.

Vertex AI Workbench notebooks are used for data exploration and modeling. Kubeflow training pipelines hosted on GKE are executed to produce machine learning models for specific surgical instrument sets. Several hundred machine learning models are then hosted with Kubeflow model serving on GKE, with state and analytics managed using Firebase. Using machine learning to infer from images whether any devices are incorrectly placed, dirty, or otherwise not fit for purpose, the data is then returned to the tablet for the user to respond accordingly.

Based on our success to date with SAVI, it is now available on Google Cloud Marketplace to help healthcare organizations achieve machine learning-powered efficiencies across a range of use cases, and ultimately improve patient safety and outcomes.

Not to be confused with the usage of Visual Inspection Model (Assembly) available in Vertex AI Vision

Trend 2: Continued Efficiency Improvements for ML
Improvements in efficiency — arising from advances in computer hardware design as well as ML algorithms and meta-learning research — are driving greater capabilities in ML models. Many aspects of the ML pipeline, from the hardware on which a model is trained and executed to individual components of the ML architecture, can be optimized for efficiency while maintaining or improving on state-of-the-art performance overall. Each of these different threads can improve efficiency by a significant multiplicative factor, and taken together, can reduce computational costs, including CO2 equivalent emissions (CO2e), by orders of magnitude compared to just a few years ago. This greater efficiency has enabled a number of critical advances that will continue to dramatically improve the efficiency of machine learning, enabling larger, higher quality ML models to be developed cost effectively and further democratizing access. I’m very excited about these directions of research!

Continued Improvements in ML Accelerator Performance
Each generation of ML accelerator improves on previous generations, enabling faster performance per chip, and often increasing the scale of the overall systems. Last year, we announced our TPUv4 systems, the fourth generation of Google’s Tensor Processing Unit, which demonstrated a 2.7x improvement over comparable TPUv3 results in the MLPerf benchmarks. Each TPUv4 chip has ~2x the peak performance per chip versus the TPUv3 chip, and the scale of each TPUv4 pod is 4096 chips (4x that of TPUv3 pods), yielding a performance of approximately 1.1 exaflops per pod (versus ~100 petaflops per TPUv3 pod). Having pods with larger numbers of chips that are connected together with high speed networks improves efficiency for larger models.

ML capabilities on mobile devices are also increasing significantly. The Pixel 6 phone features a brand new Google Tensor processor that integrates a powerful ML accelerator to better support important on-device features.

Our use of ML to accelerate the design of computer chips of all kinds (more on this below) is also paying dividends, particularly to produce better ML accelerators.

Continued Improvements in ML Compilation and Optimization of ML Workloads
Even when the hardware is unchanged, improvements in compilers and other optimizations in system software for machine learning accelerators can lead to significant improvements in efficiency. For example, “A Flexible Approach to Autotuning Multi-pass Machine Learning Compilers” shows how to use machine learning to perform auto-tuning of compilation settings to get across-the-board performance improvements of 5-15% (and sometimes as much as 2.4x improvement) for a suite of ML programs on the same underlying hardware. GSPMD describes an automatic parallelization system based on the XLA compiler that is capable of scaling most deep learning network architectures beyond the memory capacity of an accelerator and has been applied to many large models, such as GShard-M4, LaMDA, BigSSL, ViT, MetNet-2, and GLaM, leading to state-of-the-art results across several domains.

Human-Creativity–Driven Discovery of More Efficient Model Architectures
Continued improvements in model architectures give substantial reductions in the amount of computation needed to achieve a given level of accuracy for many problems. For example, the Transformer architecture, which we developed in 2017, was able to improve the state of the art on several NLP and translation benchmarks while simultaneously using 10x to 100x less computation to achieve these results than a variety of other prevalent methods, such as LSTMs and other recurrent architectures. Similarly, the Vision Transformer was able to show improved state-of-the-art results on a number of different image classification tasks despite using 4x to 10x less computation than convolutional neural networks.

Machine-Driven Discovery of More Efficient Model Architectures
Neural architecture search (NAS) can automatically discover new ML architectures that are more efficient for a given problem domain. A primary advantage of NAS is that it can greatly reduce the effort needed for algorithm development, because NAS requires only a one-time effort per search space and problem domain combination. In addition, while the initial effort to perform NAS can be computationally expensive, the resulting models can greatly reduce computation in downstream research and production settings, resulting in greatly reduced resource requirements overall. For example, the one-time search to discover the Evolved Transformer generated only 3.2 tons of CO2e (much less than the 284t CO2e reported elsewhere; see Appendix C and D in this joint Google/UC Berkeley preprint), but yielded a model for use by anyone in the NLP community that is 15-20% more efficient than the plain Transformer model. A more recent use of NAS discovered an even more efficient architecture called Primer (that has also been open-sourced), which reduces training costs by 4x compared to a plain Transformer model. In this way, the discovery costs of NAS searches are often recouped from the use of the more-efficient model architectures that are discovered, even if they are applied to only a handful of downstream uses (and many NAS results are reused thousands of times).

NAS has also been used to discover more efficient models in the vision domain. The EfficientNetV2 model architecture is the result of a neural architecture search that jointly optimizes for model accuracy, model size, and training speed. On the ImageNet benchmark, EfficientNetV2 improves training speed by 5–11x while substantially reducing model size over previous state-of-the-art models. The CoAtNet model architecture was created with an architecture search that uses ideas from the Vision Transformer and convolutional networks to create a hybrid model architecture that trains 4x faster than the Vision Transformer and achieves a new ImageNet state of the art.

The broad use of search to help improve ML model architectures and algorithms, including the use of reinforcement learning and evolutionary techniques, has inspired other researchers to apply this approach to different domains. To aid others in creating their own model searches, we have open-sourced Model Search, a platform that enables others to explore model search for their domains of interest. In addition to model architectures, automated search can also be used to find new, more efficient reinforcement learning algorithms, building on the earlier AutoML-Zero work that demonstrated this approach for automating supervised learning algorithm discovery.

Use of Sparsity
Sparsity, where a model has a very large capacity, but only some parts of the model are activated for a given task, example or token, is another important algorithmic advance that can greatly improve efficiency. In 2017, we introduced the sparsely-gated mixture-of-experts layer, which demonstrated better results on a variety of translation benchmarks while using 10x less computation than previous state-of-the-art dense LSTM models. More recently, Switch Transformers, which pair a mixture-of-experts–style architecture with the Transformer model architecture, demonstrated a 7x speedup in training time and efficiency over the dense T5-Base Transformer model. The GLaM model showed that transformers and mixture-of-expert–style layers can be combined to produce a model that exceeds the accuracy of the GPT-3 model on average across 29 benchmarks using 3x less energy for training and 2x less computation for inference. The notion of sparsity can also be applied to reduce the cost of the attention mechanism in the core Transformer architecture.

The use of sparsity in models is clearly an approach with very high potential payoff in terms of computational efficiency, and we are only scratching the surface in terms of research ideas to be tried in this direction.

Each of these approaches for improved efficiency can be combined together so that equivalent-accuracy language models trained today in efficient data centers are ~100 times more energy efficient and produce ~650 times less CO2e emissions, compared to a baseline Transformer model trained using P100 GPUs in an average U.S. datacenter using an average U.S. energy mix. And this doesn’t even account for Google’s carbon-neutral, 100% renewable energy offsets. We’ll have a more detailed blog post analyzing the carbon emissions trends of NLP models soon.

There are a number of terms and concepts in cloud computing, and not everyone is familiar with all of them. To help, we’ve put together a list of common questions, and the meanings of a few of those acronyms. You can find all these, and many more, in our learning resources.

What are containers?

Containers are packages of software that contain all of the necessary elements to run in any environment. In this way, containers virtualize the operating system and run anywhere, from a private data center to the public cloud or even on a developer’s personal laptop. Containerization allows development teams to move fast, deploy software efficiently, and operate at an unprecedented scale. Read more.

Containers vs. VMs: What’s the difference?

You might already be familiar with VMs: a guest operating system such as Linux or Windows runs on top of a host operating system with access to the underlying hardware. Containers are often compared to virtual machines (VMs). Like virtual machines, containers allow you to package your application together with libraries and other dependencies, providing isolated environments for running your software services. However, the similarities end here as containers offer a far more lightweight unit for developers and IT Ops teams to work with, carrying a myriad of benefits. Containers are much more lightweight than VMs, virtualize at the OS level while VMs virtualize at the hardware level, and share the OS kernel and use a fraction of the memory VMs require. Read more.

What is Kubernetes?

With the widespread adoption of containers among organizations, Kubernetes, the container-centric management software, has become the de facto standard to deploy and operate containerized applications. Google Cloud is the birthplace of Kubernetes—originally developed at Google and released as open source in 2014. Kubernetes builds on 15 years of running Google’s containerized workloads and the valuable contributions from the open source community. Inspired by Google’s internal cluster management system, Borg, Kubernetes makes everything associated with deploying and managing your application easier. Providing automated container orchestration, Kubernetes improves your reliability and reduces the time and resources attributed to daily operations. Read more.

What is microservices architecture?

Microservices architecture (often shortened to microservices) refers to an architectural style for developing applications. Microservices allow a large application to be separated into smaller independent parts, with each part having its own realm of responsibility. To serve a single user request, a microservices-based application can call on many internal microservices to compose its response. Containers are a well-suited microservices architecture example, since they let you focus on developing the services without worrying about the dependencies. Modern cloud-native applications are usually built as microservices using containers. Read more.

What is ETL?

ETL stands for extract, transform, and load and is a traditionally accepted way for organizations to combine data from multiple systems into a single database, data store, data warehouse, or data lake. ETL can be used to store legacy data, or—as is more typical today—aggregate data to analyze and drive business decisions. Organizations have been using ETL for decades. But what’s new is that both the sources of data, as well as the target databases, are now moving to the cloud. Additionally, we’re seeing the emergence of streaming ETL pipelines, which are now unified alongside batch pipelines—that is, pipelines handling continuous streams of data in real time versus data handled in aggregate batches. Some enterprises run continuous streaming processes with batch backfill or reprocessing pipelines woven into the mix. Read more.

What is a data lake?

A data lake is a centralized repository designed to store, process, and secure large amounts of structured, semistructured, and unstructured data. It can store data in its native format and process any variety of it, ignoring size limits. Read more.

What is a data warehouse?

Data-driven companies require robust solutions for managing and analyzing large quantities of data across their organizations. These systems must be scalable, reliable, and secure enough for regulated industries, as well as flexible enough to support a wide variety of data types and use cases. The requirements go way beyond the capabilities of any traditional database. That’s where the data warehouse comes in. A data warehouse is an enterprise system used for the analysis and reporting of structured and semi-structured data from multiple sources, such as point-of-sale transactions, marketing automation, customer relationship management, and more. A data warehouse is suited for ad hoc analysis as well custom reporting and can store both current and historical data in one place. It is designed to give a long-range view of data over time, making it a primary component of business intelligence. Read more.

What is streaming analytics?

Streaming analytics is the processing and analyzing of data records continuously rather than in batches. Generally, streaming analytics is useful for the types of data sources that send data in small sizes (often in kilobytes) in a continuous flow as the data is generated. Read more.

What is machine learning (ML)?

Today’s enterprises are bombarded with data. To drive better business decisions, they have to make sense of it. But the sheer volume coupled with complexity makes data difficult to analyze using traditional tools. Building, testing, iterating, and deploying analytical models for identifying patterns and insights in data eats up employees’ time. Then after being deployed, such models also have to be monitored and continually adjusted as the market situation or the data itself changes. Machine learning is the solution. Machine learning allows businesses to enable the data to teach the system how to solve the problem at hand with machine learning algorithms—and how to get better over time. Read more.

What is natural language processing (NLP)?

Natural language processing (NLP) uses machine learning to reveal the structure and meaning of text. With natural language processing applications, organizations can analyze text and extract information about people, places, and events to better understand social media sentiment and customer conversations. Read more.

Learn more

This is just a sampling of frequently asked questions about cloud computing. To learn more, visit our resources page at cloud.google.com/learn.

Over the last two years, we’ve seen a significant uptick in the number of people using messaging to connect with businesses. Whether it was checking hours of operation, verifying what was in stock, or scheduling a pick-up, the pandemic caused a significant shift in consumer behavior.

• 45% of users are spending more time on messaging services because of the pandemic 
• 85% of consumers would like to message brands directly

With the rise in demand for messaging, consumers expect communication with businesses to be  speedy, simple, and convenient. For businesses, keeping up with customer inquiries can be a labor-intensive process, and offering 24/7 support outside of store hours can be costly.  

To help businesses seamlessly deliver helpful, timely, and engaging conversations with customers when and where they need help, we introduced AI-powered Business Messages.

https://youtube.com/watch?v=fcgP3RHjBLY%3Fenablejsapi%3D1%26

What is AI-powered Business Messages?

With AI-powered Business Messages, you can connect with your customers in their moment of need, in the places they’re looking for answers—such as Google Search, Google Maps, or any brand-owned channel. For instance, check out how Walmart customers in the US are able to receive real-time information on product availability, straight from a search results page.

People turn to Google when they are searching for answers to their questions, looking to buy something, or trying to accomplish a particular task with one of our many tools. In fact, 68% of all online experiences begin with a search engine.

At Google, we know how important it is for interactions with a brand to be personalized, helpful, and simple. With AI-powered Business Messages, customers are able to chat with virtual agents that understand, interact, and respond in natural ways.

We are also combining smart automation with the ability for customers to chat with live agents when it’s really needed. This approach saves your customers precious time, while also saving you money. And with Business Messages automatically handling many customer inquiries in the background, businesses have the option to distribute their human customer service agents to address other needs. 

Getting started with conversational AI is easy with Bot-in-a-Box

We know it can be difficult to get started with AI. That’s why we are utilizing existing Google AI tools like Dialogflow—part of Google Cloud Contact Center AI—to create the capability within Google’s Business Messages called Bot-in-a-Box, which makes getting started with Conversational AI easy. Bot-in-a-Box allows for fast and effective adoption of automation for businesses of all sizes. 

Enabling Business Messages with Bot-in-a-Box can be as simple as leveraging an existing customer FAQ document you already have, whether it’s from a web page or an internal document. And since the conversational AI is powered by Business Messages and Dialogflow working together, your chat bot is able to understand and respond to customer questions automatically without the need to write code. 

Bot-in-a-Box also supports other critical journeys like “Custom Intents.” That means that your bot is able to understand the different ways customers express a similar question and respond accurately by using machine learning capabilities.

Finding success 

In April 2021, Wake County courthouses in North Carolina partnered with Tango Technology to implement Business Messages when it became apparent that being able to provide the public and attorneys with around-the-clock access to information would significantly reduce the pressure on courthouse staff. Using Bot-in-a-Box, Tango Technology was able to customize a solution for Wake County Courthouse, Justice Center, and Clerk of Superior Court in just four days.

“With the combination of Google’s Business Messages, GCP, and Dialogflow, we were able to spin up an AI-driven bot for the courts in days. And the technology stack allows us to continually improve by adding functionality in an agile process.”— Mike Lotz, Co-Founder, Tango Technology

With Business Messages, North Carolina courthouses saw a 37% decrease in the call volume handled by courthouse staff.  With 398,298 fewer phone calls during the first year of operation, the AI-based messages helped Wake County Courthouse work more efficiently and productively.

We’ve seen many brands benefiting from AI-powered Business Messages. For instance, Levi’s saw a 30% increase in off-hours shoppers and surpassed 85% customer satisfaction scores after implementing Business Messages. They also drove 30x more store-related questions than Levi’s website chat. 

Bring Google’s conversational AI to your storefront with Business Messages

Google’s Business Messages makes it easier for businesses of all sizes to engage their existing or potential customers in a virtual conversation, when and where they need it. 

To learn more, watch our Cloud Next session here or visit us at g.co/businessmessages. We have specialized services to help you get started  and can share the wisdom of our channel partners and dedicated experts who specialize in unleashing the potential of conversational AI.