Clinical Decision Support System (CDSS) is an important technology for the healthcare industry that analyzes data to help healthcare professionals make decisions related to patient care. The market size for the global clinical decision support system appears poised for expansion, with one study predicting a compound annual growth rate (CAGR) of 10.4%, from 2022 to 2030, to $10.7 billion.

For any health organization that wants to build a CDSS system, one key block is to locate and extract the medical entities that are present in the clinical notes, medical journals, discharge summaries, etc. Along with entity extraction, the other key components of the CDSS system are capturing the temporal relationships, subjects, and certainty assessments.

At Google Cloud, we know how critical it is for the healthcare industry to build CDSS systems, so we worked with Apollo 24|7, the largest multi-channel digital healthcare platform in India, to build the key blocks of their CDSS solution.

We helped them to parse the discharge summaries and prescriptions to extract the medical entities. These entities can then be used to build a recommendation engine that would help doctors with the “Next Best Action” recommendation for medicines, lab tests, etc.

Let’s take a sneak peek at Apollo 24|7’s entity extraction solutions, and the various Google AI technologies that were tested to form the technology stack.

Datasets Used

To perform our experiments on entity extraction, we used two types of datasets.

1. i2b2 Dataset – i2b2 is an open-source clinical data warehousing and analytics research platform that provides annotated deidentified patient discharge summaries made available to the community for research purposes. This dataset was primarily used for training and validation of the models.
2. Apollo 24|7’s Dataset – De-identified doctor’s notes from Apollo24|7 were used for testing. Doctors annotated them to label the entities and offset values.

Experimentation and choosing the right approach — Four models put to test

For entity extraction, both Google Cloud products and open-source approaches were explored. Below are the details:

1. Healthcare Natural Language API: This is a no-code approach that provides machine learning solutions for deriving insights from medical text. Using this, we parsed unstructured medical text and then generated a structured data representation of the medical knowledge entities stored in the data for downstream analysis and automation. The process includes:

• Extract information about medical concepts like diseases, medications, medical devices, procedures, and their clinically relevant attributes;
• Map medical concepts to standard medical vocabularies such as RxNorm, ICD-10, MeSH, and SNOMED CT (US users only);
• Derive medical insights from text and integrate them with data analytics products in Google Cloud.

The advantage of using this approach is that it not only extracts a wide range of entity types like MED_DOSE, MED_DURATION, LAB_UNIT, LAB_VALUE, etc, but also captures functional features such as temporal relationships, subjects, and certainty assessments, along with the confidence scores. Since it is available on Google Cloud, this offers long-term product support. It is also the only fully-managed NLP service among all the approaches tested and hence, it requires the least effort to implement and manage.

But one thing to keep in mind is that since the Healthcare NL API offers natural language models that are pre-trained, it currently cannot be used for custom entity extraction models trained using custom annotated medical text or to extract custom entities. This has to be done via AutoML Entity Extraction for Healthcare, another Google Cloud service for custom model development. Custom model development is important for adapting the pre-trained models to new languages or region-specific natural language processing, such as medical terms whose use may be more prevalent in India than in other regions

2. Vertex AutoML Entity Extraction for Healthcare: This is a low-code approach that’s already available on Google Cloud. We used AutoML Entity Extraction to build and deploy custom machine learning models that analyzed documents, categorized them, and identified entities within them. This custom machine learning model was trained on the annotated dataset provided by the Apollo 24|7 team.

The advantage of AutoML Entity Extraction is that it gives the option to train on a new dataset. However, one of the prerequisites to keep in mind is that it needs a little pre-processing to capture the input data in the required JSONL format. Since this is an AutoML model just for Entity Extraction, it does not extract relationships, certainty assessments, etc.

3. BERT-based Models on Vertex AI: Vertex AI is Google Cloud’s fully managed unified AI platform to build, deploy, and scale ML models faster, with pre-trained and custom tooling. We experimented with multiple custom approaches based on pre-trained BERT-based models, which have shown state-of-the-art performance in many natural language tasks. To gain better contextual understanding of medical terms and procedures, these BERT-based approaches are explicitly trained on medical domain data. Our experiments were based on BioClinical BERT, BioLink BERT, Blue BERT trained on Pubmed dataset, and Blue BERT trained on Pubmed + MIMIC datasets.

The major advantage of these BERT-based models is that they can be finetuned on any Entity Recognition task with minimal efforts.

However, since this is a custom approach, it requires some technical expertise. Additionally, it does not extract relationships, certainty assessments, etc. This is one of the main limitations of using BERT-based models.

4. ScispaCy on Vertex AI: We used Vertex AI to perform experiments based on ScispaCy, which is a Python package containing spaCy models for processing biomedical, scientific or clinical text.

Along with Entity Extraction, Scispacy on Vertex AI provides additional components like Abbreviation Detector, Entity Linking, etc. However, when compared to other models, it was less precise, with too many junk phrases, like “Admission Date,” captured as entities.

“Exploring multiple approaches and understanding the pros/cons of each approach helped us to decide the one that would fit our business requirements.” according to Abdussamad M, Engineering Lead at Apollo 24|7.

Evaluation Strategy

In order to match the parsed entity with the test data labels, we used extensive matching logic that comprised of the below four methods:

1. Exact Match – Exact match captures entities where the model output and the entities in the test dataset match. Here, the offset values of the entities have also been considered. For example, the entity “gastrointestinal infection” that is present as-is in both the model output and the test label will be considered an “Exact Match.”
2. Match-Score Logic – We used a scoring logic for matching the entities. For each word in the test data labels, every word in the model output is matched along with the offset. A score is calculated between the entities and based on the threshold, it is considered as a match.
3. Partial Match – In this matching logic, entities like “hypertension” and “hypertensive” are matched based on the Fuzzy logic.
4. UMLS Abbreviation Lookup – We also observed that the medical text had some abbreviations, like AP meaning abdominal pain. These were first expanded by doing a lookup on the respective UMLS (Unified Medical Language System) tables and then passed to the individual entity extraction models.

Performance Metrics

We used precision and recall metrics to compare the outcomes of different models/experiments.

Precision (also called positive predictive value) is the fraction of relevant instances among the retrieved instances, while recall (also known as sensitivity) is the fraction of relevant instances that were retrieved.

The below example shows how to calculate these metrics for a given sample.

Example sample: “Krish has fever, headache and feels uncomfortable”

Expected Entities: [“fever”, “headache”]

Model Output: [“fever”, “feels”, “uncomfortable”]

Thus,

Experimentation Results
The following table captures the results of the above experiments on Apollo24|7’s internal datasets.

Finally, the Blue BERT model trained on the Pubmed dataset had the best performance metrics with a 81% improvement on Apollo 24|7’s baseline mode with the Healthcare Natural Language API providing the context, relationships, and codes. This performance could be further improved by implementing an ensemble of these two models.

“With the Blue BERT model giving the best performance for entity extraction on Vertex AI and the Healthcare NL API being able to extract the relationships, certainty assessments etc, we finally decided to go with an ensemble of these 2 approaches,“ Abdussamad added.

Fast track end-to-end deployment with Google Cloud AI Services (AIS)

Google AIS (Professional Services Organization) helped Apollo24|7 to build the key blocks of the CDSS system.

The partnership between Google Cloud and Apollo 24|7 is just one of the latest examples of how we’re providing AI-powered solutions to solve complex problems to help organizations drive the desired outcomes. To learn more about Google Cloud’s AI services, visit our AI & ML Products page, and to learn more about Google Cloud solutions for health care, explore our Google Cloud Healthcare Data Engine page.

Acknowledgements

We’d like to give special thanks to Nitin Aggarwal, Gopala Dhar and Kartik Chaudhary for their support and guidance throughout the project. We are also thankful to Manisha Yadav, Santosh Gadgei and Vasantha Kumar for implementing the GCP infrastructure. We are grateful to the Apollo team (Chaitanya Bharadwaj, Abdussamad GM, Lavish M, Dinesh Singamsetty, Anmol Singh and Prithwiraj) and our partner team from HCL/Wipro (Durga Tulluru and Praful Turanur) who partnered with us in delivering this successful project. Special thanks to the Cloud Healthcare NLP API team (Donny Cheung, Amirhossein Simjour, and Kalyan Pamarthy).

Customers reach out to contact centers for help in moments of urgent need, but due to increasing demands, new channels, peak times, and operational pressures, contact centers often struggle to provide timely help. To bridge this gap, enterprises are increasingly investing in AI-driven solutions that balance addressing customer expectations with operational efficiency.

But building and generating value from such solutions can be complicated and challenging. Google Cloud built Contact Center AI (CCAI) to streamline and shorten this time to value, and CCAI Platform, our newest addition, takes a crucial step in this effort by introducing end-to-end call center capabilities. After debuting these new capabilities in March, we are excited to announce that CCAI Platform is now generally available across the US, Canada, UK, Germany, France, Italy and Spain—with more markets soon to come.

Delivering world-class customer experiences and accelerating time-to-value with CCAI

CCAI encompasses a comprehensive set of offerings to address the top pain points of the three main user groups in the contact center: contact center owners, their agents, and the customers they serve.

Dialogflow lets the contact center manager scale their operations while balancing cost and customer satisfaction, including reducing painful, long waiting times endured by end users. Using Dialogflow, contact center managers can build complex chat and voice virtual agents—a proven, cost-effective way to scale contact centers while continuing to provide great customer experiences. Available 24/7, without any waiting queue, these virtual agents can converse naturally with customers, identify their issues, and address them effectively.

Agent Assist reduces overall handling time and coaches human agents to become more effective and helpful. The service uses AI to “listen” to the voice and chat conversations between the human representative and the customer, then provides real-time guidance and recommendations to the agent, based on historical conversations, knowledge bases, and best practices of experienced agents. It also automates post-call actions such as transcription and call summarization, saving significant time and overhead at the end of every call.

CCAI Insights stores and analyzes all the customer conversations in the contact center, whether with human or virtual agents, to provide leaders with real-time, actionable data points on customer queries, agent performance, sentiment trends, and opportunities for automation.

At the heart of these technologies is our conversational AI brain. It uses Google Research’s technology to talk, understand, and interact, enabling and orchestrating high-quality conversational experiences at scale.

CCAI Platform: a modern CCaaS and the shortest path to CCAI value

While the value of the CCAI offerings is clear to our customers, we also hear from them that integrating these solutions with legacy infrastructure takes too long.

To minimize these integration difficulties, accelerate time-to-value using the CCAI offerings, and help businesses provide outstanding customer experiences, we’re pleased to announce the general availability of CCAI Platform, the Contact Center as a Service (CCaaS) solution from Google Cloud built in partnership with UJET.

CCAI Platform is a modern, turnkey Contact Center as a service solution, designed with user-first, AI-first, and mobile-first principles. It offers:

• Turnkey core Contact Center capabilities out-of-the-box, for faster time to production, lower implementation overhead, and custom development needed
• AI-powered experiences, from routing to better handling customer interactions
• Deep integration with CCAI’s offerings, to provide a unified end-to-end experience for contact center transformation
• Mobile-first design that enables interactions in line with the way people expect to communicate across channels
• CRM-centered design with automated updates, so agents can focus on the customer
• Deployment flexibility, with customer data residing in their CRM and the flexibility to bring their own telephony carrier to minimize cost

All of this is available without the typical need to integrate complex technologies from multiple providers.

“With Google Cloud and CCAI Platform, we will quickly move our contact center to the cloud, supporting both our customers and agents with industry-leading CX innovations, all while streamlining operations through more efficient customer care operations,” said Dean Kontul, Division CIO of KeyBank.

For customers looking to change platforms for a cloud-native CCaaS with deep Google AI integrations, CCAI Platform offers end-to-end capabilities that accelerate call center transformations. We also remain strongly committed to customer choice, and customers will continue to have the option to integrate our latest and greatest CCAI offerings through our existing OEM partners.

The Contact Center conversation is just beginning

This launch is part of a broader effort to deliver more value, faster, to more CCAI customers. As companies replace interactive voice response (IVR) with intelligent virtual agents (IVA) and begin to collect and analyze data, use cases are likely to grow more sophisticated—which is one reason Google Cloud is continuing to invest in technologies to make our CCAI offerings even more useful, as well as best practices like the following:

• CCAI Agent Assist and Insights are a great first step in AI transformation. They let contact center owners enable call transcription and use Topic Modeling to identify conversation themes that demand attention. Human agents can automatically generate high-quality conversation summaries to reduce call wrap-up time, and the associated costs, while improving business insights. We are working to make these features available both in CCAI Platform and with our partner ISVs.
• Chat and call steering are the first step for IVA automation. Another area of broad impact is conversational chat or call steering, in which friction is reduced by routing customers to the correct virtual or live agent experience. Many call centers rely on IVR systems in which customers have to use a keypad to select an option. Enterprise leaders tell us that attrition is very high throughout this process: some customers angrily hang up without resolution and, just as bad, many simply pound a single key in hopes of reaching a human agent, leading to the customer reaching the wrong person because their issue was never correctly identified or routed. Using Dialogflow’s natural language understanding (NLU) capabilities can sweep away such problems, with the customer more likely to not only reach the appropriate resources, but also share conversational data from which insights can be gleaned. It’s an approach that can pay dividends right away, and a quick first step to IVA automation.

In coming months, we will continue to work on these and other capabilities that are targeted to deliver higher and quicker value to our customers. We plan to release pre-built components to help companies tackle call center use cases in specific industries, for example. We’ll also continue to partner with companies that share our vision of transforming customer experiences with AI, such as TTEC, a provider of customer experience technology and software.

“TTEC Digital and Google Cloud have a shared vision for transforming global CX delivery through artificial intelligence, digital innovation, and operational excellence,” said Sam Thepvongs, VP of TTEC Digital. “With CCAI Platform, we can offer our largest enterprise customers a strategic blueprint for moving to the cloud while adopting a leading, AI-powered contact center platform. We couldn’t be more excited about this evolution of Google Cloud’s groundbreaking CCAI portfolio, and the opportunity to help our customers digitally transform their CX through this partnership.”

To get started with CCAI Platform, visit our solutions page or checkout our new omni channel demo video—and don’t forget to join us at Google Cloud Next ’22, where I’ll be sharing exciting new updates for CCAI in my session, “Delight customers in every interaction with Contact Center AI.”

GOALS

• Develop smarter, more effective media buying models through data
• Drive relevant advertising
• Scale predictive modelling across all touchpoints in the customer journey

APPROACH

Combined contextual data to create a predictive model with granular layers

Activated data in real-time

RESULTS

• 40% lower cost per booking
• More than twice as many bookings at same spend
• 1.4 times higher click-through rate for test group than control

KLM partnered with Relay42, whose data management platform (DMP) empowers marketers to achieve data-driven personalisation at scale.

The Relay42 DMP works by stitching together all touchpoints and data sources (including all Google solutions), orchestrating customer journeys in real-time and then activating the unified data to drive business and user experience results.

Relay42 and KLM created a data flow setup with the Relay42 DMP at the core. Thanks to Relay42’s tag management system, all customer interactions on KLM’s website and app, as well as relevant indicators from other channels and data sources such as email, social, CRM, call center and affiliates can be tracked and synced in Google Analytics 360.

Data gathered via Relay42’s own tags and DoubleClick Floodlight tags can also be sent from the DMP to the DoubleClick platform so that relevant ads can be targeted to appropriate audiences.

In order to perform deep-dive analyses, KLM simply exports raw-level data (from DoubleClick using Data Transfer and from Google Analytics 360 through seamless integration) to BigQuery. BigQuery can then correlate site behaviour with ad impressions, which enables Relay42 to activate the data and inform the build of predictive models.

“We believe that data in combination with technological innovation is bound to make advertising smarter and more relevant on every touchpoint. By combining data sources, leveraging first-party data and activating this in real-time, advertising can turn into a personal dialogue, a rewarding one-to-one interaction instead of one-to-many push messaging.”

– Kevin Duijndam, Cross Channel Marketing Manager, KLM

A Predictive Model to Improve Display Remarketing

KLM decided to test a new approach to remarketing.

“Our goal was to get rid of irrelevant ads, as they are simply annoying”, explains Kevin Duijndam, the airline’s Cross Channel Marketing Manager.

“Our assumption was that people who fly with us often already know us, so it would be irrelevant to tell them about flying with us again. However, we were wondering when exactly someone is a ‘frequent flyer’. The more we thought about it, the more complex the set of business rules became, so in the end we realized we couldn’t just focus on frequent flyers, but instead should use machine learning to understand when ads are irrelevant.”

KLM developed a real-time buying setup to include predictive modelling. In this setup, website and app interactions are tracked in the DMP thanks to the Relay42 tag management system.

Relevant consumer behaviour can be streamed in real-time to a prediction engine developed by KLM in the Google Cloud Platform, with the outputs then streamed straight back to the DMP. From here, the DMP can activate rule-based segments based on the prediction outcome. And by syncing this with DoubleClick, ads can be served and targeted to maximise relevance.

The team tested their new predictive model to assess any gains in performance. KLM deliberately chose to measure the results in an A/B setup within a defined period rather than measuring the differences year over year or month over month. Such comparisons are less reliable due to rapid changes relating to seasonality, internal capacities and external factors caused by competitors.

Business Gains and Customer Experience Wins

In the test, KLM linked the Relay42 DMP customer interaction data to their predictive model to predict how relevant their ads would be. The setup enabled a decision to be made in real-time whether or not to serve a specific ad to a user.

Through the A/B tests it became clear that the new model in fact did generate a significant uplift in bookings. With the cost per booking 40% lower during the test period, KLM was able to achieve more than twice as many bookings at the same spend.

 The test produced wins in terms of customer experience, too. The click-through rate for the test group was more than 1.4 times higher than for the control group, indicating that the new model was successfully reaching users with messages they found to be relevant rather than annoying.

 The success goes beyond improving KLM’s display remarketing efforts, though.

“Even more importantly, we’ve laid the IT data flow foundation in such a way that KLM is now able to execute on our data through all of our digital marketing channels and apply our prediction models at scale”, Kevin says. “So we can be flexible to plug in other models but can also now scale through other online media channels like search or video.”