Hackensack Meridian Health (HMH) executive Mark Eimer explains how an ambitiously-timed rollout of a comprehensive suite of Google products helped the entire organization—from doctors to IT staff—achieve better security, cultivate a more equitable work environment, and ultimately, improve patient outcomes.

How does a recently merged, 17-hospital healthcare system fast-track a platform migration and hardware rollout securely and in a way that improves work for everyone, regardless of location or role? These are the questions that kept me up at night in early 2020, when the pandemic demanded a “big bang”—something our legacy laptops and operating systems couldn’t handle.

We began our work with Google in 2020 with the adoption of Chrome as our default browser. As we migrated platforms, keeping patient data safe was of the utmost importance to us, along with providing every staff member with the tools they needed to work virtually. Our staff often experienced issues accessing our web-based applications using Internet Explorer or Edge Browser, a problem that went away when we switched to Chrome. Chrome’s versatile compatibility also made it easier for my team to migrate all of our web-based operations, and Chrome’s security and manageability were key components to making this switch a huge win for the organization.

The success of this migration led us to extend our Google partnership to patient care applications—where Google’s expertise in AI and ML helps scale the use of diagnostics tools and improve other aspects of the patient journey.

Achieving security at every step 

Like so many other healthcare organizations, we’ve been concerned about ransomware attacks. This is part of why we moved to Google Workspace and distributed over 3,000 Chrome OS devices in kiosk mode in March of 2020, when many of us went remote due to the pandemic. We were very concerned about team members accessing corporate applications through home devices that were running EOL operating systems (WIN7), as well as a general lack of antivirus and encryption measures.

We were protected by the fact that Google’s software and hardware both had built-in security features that we needed to stave off sophisticated attackers. For example, Chrome OS automatically updates to the latest security update and encrypts data living outside the cloud on the hardware. These features protected us from security-related disruptions, letting us securely move a huge library of file shares and emails across thousands of accounts to Google Chrome OS in just four months.

A year later, in March 2021, we migrated the enterprise over to Google Workspace and saw an immediate reduction in spam by 30% from the inherent built-in AI/ML. This meant staff were less likely to receive (and click through) phishing attempts. My team could connect, create, and collaborate easily and securely—even as more of us were working from home and needed to access sensitive data remotely. 

Leveling the playing field

As an organization, we were surprised by how many team members didn’t have personal computers at home. We quickly decided that if we needed team members to work from home, the health network would have to supply hardware. Chromebooks’ lower price tag compared to PCs—on top of their built-in security controls—allowed us to purchase, deploy, and support that initial distribution of 3,000 Chromebooks to team members in less than three weeks, providing devices to every eligible remote employee instead of just a select few. This was vital to reaching our equitable technology goal as part of our diversity and inclusion initiative: everybody has the same tools to do good work.

When all employees have what they need to do their jobs well, we get better patient outcomes. Before we began this cloud adoption journey, patient and staff experiences were different within the hospitals and outside of them. 

Now it’s the same wherever our staff is, and we’ve seen efficiency and accessibility benefits extend to the patient side. For example, we built a web-based contact center that supports 80 locations that use Workspace and Chrome OS devices. Since customer service, admin, and providers are all on the same system, it has become a one-stop shop for patients.

Furthermore, through the Grow with Google program, we were able to provide another benefit to employees that drove our equity goals. Google trained 50 non-IT staff members—from environmental services, food and nutrition, and other non-tech areas who were interested in making a career change to IT—on the Google products we were using. They may not have thought about switching to a career in IT before the Grow with Google program came to our organization, but through this partnership, they now have that opportunity.

A strategic, long-term partner

With any large-scale rollout, the work doesn’t end once laptops are in employee hands. Google has shown their commitment to long-term collaboration as they continuously optimize their products for the unique needs of healthcare providers and go the extra mile in tailoring tools to our staff’s workflows. 

For example, on the Chrome OS side, the Google team has helped our registration desks and document centers with device integration for hardware like credit card readers and e-signature pads. They’ve also helped us meet security and privacy requirements mandated by state and federal governments around HIPAA, Medicaid, and Medicare reimbursements. Over this next year, we’ll look at a feature roadmap with Google Cloud to deliver further enhancements, iterating on the product itself to meet our needs for the present and the future.

Delivering the future of healthcare

The benefits we’ve seen around security and usability—and the ability to provide all staff with equal access to Google’s technology—are why we’re expanding our partnership with Google to both the administrative and clinical sides of HMH. In addition to further Google rollouts with corporate, next year we’re distributing Chromebooks to all 350 of our ambulatory clinics.

We’re also working with the Google professional services team to create a custom AI model that analyzes 3D mammogram images. This AI model will enable two providers to read mammograms—which adheres to international best practices but is currently rare in the US—without requiring additional time. Conducting double readings of mammograms will yield better health outcomes for our patients, such as a lower patient recall rate and an increased accuracy in detecting breast cancer.

We’re currently building the model using a variety of Google Cloud products, including Cloud Healthcare API. Once complete, this model is expected to be trained, deployed, and maintained in Google’s Vertex AI, allowing our providers to be more productive as they make clinical decisions with AI support. As the model is proven over time, we plan to make the predictive services accessible to other healthcare organizations.

With Google, we’re able to achieve a unified architecture for storing data as well as training and deploying AI models, which enable our staff to work more efficiently and securely from anywhere. While I may not be able to predict the future as accurately as AI can, I foresee our continued partnership with Google as a key part of HMH’s improved provider and patient outcomes.

One of India’s best-known healthcare brands, Manipal Group prides itself on clinical excellence and a patient-centric approach. From humble beginnings in 1953 as a single teaching hospital—the Kasturba Medical College—in a university town in Karnataka, India, Manipal Group has grown to a presence in seven cities in India and operations in Malaysia and Nigeria. “Our founder, Dr T. M. A. Pai, established Kasturba Medical College just six years after India gained independence,” says C. G. Muthana, Chief Operating Officer, Teaching Hospitals, Manipal Health Enterprises Pvt. Ltd, part of Manipal Group.

Manipal Health Enterprises Pvt. Ltd operates 11 corporate, or for-profit, hospitals and five teaching hospitals. “We have about 7,000 beds India-wide, and on that measure, we are the third-largest private-sector healthcare provider in India today,” says Muthana. The organization now employs about 6,000 people in its corporate hospitals and 5,000 people in its teaching hospitals.

Manipal Health Enterprises Pvt. Ltd acquires and operates world-class medical technologies in its teaching and corporate hospitals. However, with 75% of corporate hospital wards allocated to fee-paying private patients—compared to just 25% of the wards in teaching hospitals—the differences in information technology budgets are significant. “In the general wards that comprise most of the wards in teaching hospitals, patients are typically treated for free or at heavily subsidized rates,” explains Muthana. “So revenues and costs of delivery vary between hospitals, while employee costs remain similar.”

Rostering nurses a critical task

Rostering nurses to work shifts is one of the most important tasks at Manipal Group corporate and teaching hospitals. As at all hospitals, nurses administer medications, monitor patients, maintain records, manage intravenous lines, and work with doctors to heal patients. They can also provide advice and support to patients and loved ones, including teaching them how to administer medication outside a hospital setting. If too few nurses are rostered for a particular shift, the quality of patient care may suffer.

Google Cloud results

• Enabled the hospital to correctly size its nursing workforce
• Lowers stress on nurses by delivering more equitable rostering
• Presents opportunity to better manage nurses’ leave and other administration tasks

However, rostering at the group’s hospitals was a time-consuming exercise. Senior nurses on each ward would have to spend up to 45 minutes per day manually amending paper-based rosters to accommodate requested changes to duty shifts for personal or other circumstances. The organization began receiving complaints from patients, doctors, and other hospital staff that, on occasion, too few nurses were rostered on for certain shifts—particularly at its flagship hospital in Bangalore. “We had based our rostering calculations on the number of beds occupied by patients and, when we investigated, we found at a macro level, we were rostering on the correct number of nurses,” says Muthana. “However, on some days, on wards optimally staffed by, say, 10 nurses per shift, we might have 14 nurses rostered on for one shift and seven rostered on for another shift. Those times we had seven, we had a clear shortage.”

In 2012, Muthana asked a consultant to develop algorithms to help automate the rostering. “Unfortunately, there were so many variables, the consultant failed to solve the problem,” he says. The Chief Operating Officer’s next step was to ask the founder and Chief Executive Officer of predictive analytics business Retigence Technologies—already working with the business on a materials management project—to develop an application to manage rostering.

Removing the daily drudgery

“Our plan with the automation project was to relieve our senior nurses of the daily drudgery of amending the rosters and to deliver rosters that were as fair as possible to all our staff,” says Muthana. “We also saw an opportunity to reduce our costs by reducing the overall number of nurses needed to look after our patients.”

Retigence Technologies’ team members then worked with the group’s nurses to capture the variables and requirements for the project. For example, a minimum number of nurses with one year experience or more needs to be rostered on for each shift. Retigence Technologies then started building an application using compute resources available through Compute Engine, a Google Cloud product. “We selected Google primarily because of its pioneering work in artificial intelligence (AI) and machine learning,” says Srinibas Behera, founder and Chief Executive Officer of Retigence Technologies.

With the nurses rostering application developed, Manipal Health Enterprises Pvt. Ltd undertook several pilots to build user acceptance. “Some of our senior nurses took time to accept the fact automation removed their control over rostering assignments, but were finally convinced by the better transparency the product offered,” says Muthana. The organization deployed the application to a smaller hospital and secured user support before rolling out the application to its Bangalore flagship.

Eliminating stress

Deploying the application has enabled Manipal Health Enterprises Pvt. Ltd to remove a buffer of about 100 nurses retained to accommodate the variations in number of nurses rostered for individual shifts. “The savings on those salaries more than paid for the cost of developing the application,” says Muthana.

The application also enabled the organization to reduce the stress on nurses—both the nurses in charge of the rosters and the nurses subject to the rosters. “Night shifts were more equitably distributed among the nurses, while we have been able to reduce the 45 minutes per day required to amend rosters to just 10 minutes,” says Muthana. “In Bangalore alone, we have 51 nurses in charge of rostering—so the combined saving there equates to nearly 30 hours per day.” This is freeing up these senior nurses to complete more important tasks.

The application was subsequently implemented at Kasturba Hospital, Manipal, again reducing the time needed to generate complete nursing rosters to less than 10 minutes.

Managing leave and training

The organization now plans to extend the application to manage leave and training for its nurses and other employees. “I would like to see every employee given an annual leave plan that is added to the roster at the start of the year,” says Muthana. “The flexibility and control afforded by the application would enable us to address challenges such as managing leave across a workforce with high attrition rates.” The organization would also be able to create a calendar to ensure nurses receive all their required training.

“We also plan to keep fine-tuning the application to deploy nurses more efficiently and continue to reduce their stress levels,” adds Muthana. “We also want to create a nursing load indicator tailored to patients’ specific circumstances. For example, a sedated patient may not require much nursing care, whereas a patient who comes in with a broken leg and may be on a ventilator may require assistance from three nurses at once.” The business plans to use Google’s AI and machine learning APIs in the future to improve the value and user experience of the product.

The latest round of MLPerf benchmark results have been released, and Google’s TPU v4 supercomputers demonstrated record-breaking performance at scale. This is a timely milestone since large-scale machine learning training has enabled many of the recent breakthroughs in AI, with the latest models encompassing billions or even trillions of parameters (T5, Meena, GShard, Switch Transformer, and GPT-3). 

Google’s TPU v4 Pod was designed, in part, to meet these expansive training needs, and TPU v4 Pods set performance records in four of the six MLPerf benchmarks Google entered using TensorFlow and JAX. These scores are a significant improvement over our winning submission from last year and demonstrate that Google once again has the world’s fastest machine learning supercomputers. These TPU v4 Pods are already widely deployed throughout Google data centers for our internal machine learning workloads and will be available via Google Cloud later this year.

Figure 1: Speedup of Google’s best MLPerf Training v1.0 TPU v4 submission over the fastest non-Google submission in any availability category – in this case, all baseline submissions came from NVIDIA. Comparisons are normalized by overall training time regardless of system size. Taller bars are better.¹

Let’s take a closer look at some of the innovations that delivered these ground-breaking results and what this means for large model training at Google and beyond. 

Google’s continued performance leadership

Google’s submissions for the most recent MLPerf demonstrated leading top-line performance (fastest time to reach target quality), setting new performance records in four benchmarks. We achieved this by scaling up to 3,456 of our next-gen TPU v4 ASICs with hundreds of CPU hosts for the multiple benchmarks. We achieved an average of 1.7x improvement in our top-line submissions compared to last year’s results. This means we can now train some of the most common machine learning models in a matter of seconds.

Figure 2: Speedup of Google’s MLPerf Training v1.0 TPU v4 submission over Google’s MLPerf Training v0.7 TPU v3 submission (exception: DLRM results in MLPerf v0.7 were obtained using TPU v4). Comparisons are normalized by overall training time regardless of system size. Taller bars are better. Unet3D not shown since it is a new benchmark for MLPerf v1.0.²

We achieved these performance improvements through continued investment in both our hardware and software stacks. Part of the speedup comes from using Google’s fourth-generation TPU ASIC, which offers a significant boost in raw processing power over the previous generation, TPU v3. 4,096 of these TPU v4 chips are networked together to create a TPU v4 Pod, with each pod delivering 1.1 exaflop/s of peak performance.

Figure 3: A visual representation of 1 exaflop/s of computing power. If 10 million laptops were running simultaneously, then all that computing power would almost match the computing power of 1 exaflop/s.

In parallel, we introduced a number of new features into the XLA compiler to improve the performance of any ML model running on TPU v4. One of these features provides the ability to operate two (or potentially more) TPU cores as a single logical device using a shared uniform memory access system. This memory space unification allows the cores to easily share input and output data – allowing for a more performant allocation of work across cores. A second feature improves performance through a fine-grained overlap of compute and communication. Finally, we introduced a technique to automatically transform convolution operations such that space dimensions are converted into additional batch dimensions. This technique improves performance at the low batch sizes that are common at very large scales.

Enabling large model research using carbon-free energy

Though the margin of difference in topline MLPerf benchmarks can be measured in mere seconds, this can translate to many days worth of training time on the state-of-the-art models that comprise billions or trillions of parameters. To give an example, today we can train a 4 trillion parameter dense Transformer with GSPMD on 2048 TPU cores. For context, this is over 20 times larger than the GPT-3 model published by OpenAI last year. We are already using TPU v4 Pods extensively within Google to develop research breakthroughs such as MUM and LaMDA, and improve our core products such as Search, Assistant and Translate. The faster training times from TPUs result in efficiency savings and improved research and development velocity. Many of these TPU v4 Pods will be operating at or near 90% carbon free energy. Furthermore, cloud datacenters can be ~1.4-2X more energy efficient than typical datacenters, and the ML-oriented accelerators – like TPUs – running inside them can be ~2-5X more effective than off-the-shelf systems. 

We are also excited to soon offer TPU v4 Pods on Google Cloud, making the world’s fastest machine learning training supercomputers available to customers around the world. Cloud TPUs support leading frameworks such as TensorFlow, PyTorch, and Jax, and we recently released an all-new Cloud TPU system architecture that provides direct access to TPU host machines, greatly improving the user experience. 

Want to learn more? 

Please contact your Google Cloud sales representative to request early access to Cloud TPU v4 Pods. We are excited to see how you will expand the machine learning frontier with access to exaflops of TPU computing power!

^{1. All results retrieved from www.mlperf.org on June 30, 2021. MLPerf name and logo are trademarks. See www.mlperf.org for more information. Chart uses results 1.0-1067, 1.0-1070, 1.0-1071, 1.0-1072, 1.0-1073, 1.0-1074, 1.0-1075, 1.0-1076, 1.0-1077, 1.0-1088, 1.0-1089, 1.0-1090, 1.0-1091, 1.0-1092.
2. All results retrieved from www.mlperf.org on June 30, 2021. MLPerf name and logo are trademarks. See www.mlperf.org for more information. Chart uses results 0.7-65, 0.7-66, 0.7-67, 1.0-1088, 1.0-1090, 1.0-1091, 1.0-1092.}