Healthcare breakthroughs change the world and bring hope to humanity through scientific rigor, human insight, and compassion. We believe AI can contribute to this, with thoughtful collaboration between researchers, healthcare organizations and the broader ecosystem. 

Today, we’re sharing exciting progress on these initiatives, with the announcement of limited access to Google’s medical large language model, or LLM, called Med-PaLM 2. It will be available in coming weeks to a select group of Google Cloud customers for limited testing, to explore use cases and share feedback as we investigate safe, responsible, and meaningful ways to use this technology. 

Med-PaLM 2 harnesses the power of Google’s LLMs, aligned to the medical domain to more accurately and safely answer medical questions. As a result, Med-PaLM 2 was the first LLM to perform at an “expert” test-taker level performance on the MedQA dataset of US Medical Licensing Examination (USMLE)-style questions, reaching 85%+ accuracy, and it was the first AI system to reach a passing score on the MedMCQA dataset comprising Indian AIIMS and NEET medical examination questions, scoring 72.3%.

Industry-tailored LLMs like Med-PaLM 2 are part of a burgeoning family of generative AI technologies that have the potential to significantly enhance healthcare experiences. We’re looking forward to working with our customers to understand how Med-PaLM 2 might be used to facilitate rich, informative discussions, answer complex medical questions, and find insights in complicated and unstructured medical texts. They might also explore its utility to help draft short- and long-form responses and summarize documentation and insights from internal data sets and bodies of scientific knowledge.

Innovating responsibly with AI

Since last year, we’ve been researching and evaluating Med-PaLM and Med-PaLM 2, assessing it against multiple criteria — including scientific consensus, medical reasoning, knowledge recall, bias, and likelihood of possible harm — which were evaluated by clinicians and non-clinicians from a range of backgrounds and countries. 

Med-PaLM 2’s impressive performance on medical exam-style questions is a promising development, but we need to learn how this can be harnessed to benefit healthcare workers, researchers, administrators, and patients. In building Med-PaLM 2, we’ve been focused on safety, equity, and evaluations of unfair bias. Our limited access for select Google Cloud customers will be an important step in furthering these efforts, bringing in additional expertise across the healthcare and life sciences ecosystem. 

What’s more, when Google Cloud brings new AI advances to our products, our commitment is two-fold: to not only deliver transformative capabilities, but also ensure our technologies include proper protections for our organizations, their users, and society. To this end, our AI Principles, established in 2017, form a living constitution that guides our approach to building advanced technologies, conducting research, and drafting our product development policies. 

From AI to generative AI 

Google’s deep history in AI informs our work in generative AI technologies, which can find complex relationships in large sets of training data, then generalize from what they learn to create new data. Breakthroughs such as the Transformer have enabled LLMs and other large models to scale to billions of parameters, letting generative AI move beyond the limited pattern-spotting of earlier AIs and into the creation of novel expressions of content, from speech to scientific modeling. 

Google Cloud is committed to bringing to market products that are informed by our research efforts across Alphabet. In 2022, we introduced a deep integration between Google Cloud and Alphabet’s AI research organizations, which allows Vertex AI to run DeepMind’s groundbreaking protein structure prediction system, AlphaFold.

Much more is on the way. In one sense, generative AI is revolutionary. In another, it’s the familiar technology story of more and better computing creating new industries, from desktop publishing to the internet, social networks, mobile apps, and now, generative AI.

Building on AI leadership

Additionally, today we’re announcing a new AI-enabled Claims Acceleration Suite, designed to streamline processes for health insurance prior authorization and claims processing. The Claims Acceleration Suite helps both providers of insurance plans and healthcare to create operational efficiencies and reduce administrative burdens and costs by converting unstructured data into structured data that help experts make faster decisions and improve access to timely patient care. 

On the clinical side, last year we announced Medical Imaging Suite, an AI-assisted diagnosis technology being used by Hologic to improve cervical cancer diagnoses and Hackensack Meridian Health to predict metastasis in patients with prostate cancer. Elsewhere, Mayo Clinic and Google have collaborated on an AI algorithm to improve the care of head and neck cancers, and Google Health recently partnered with iCAD to improve breast cancer screening with AI.

From these examples and more, it’s clear that the healthcare industry has moved from testing AI to deploying it to improve workflows, solve business problems, and speed healing. With this in mind, we expect rapid interest in and uptake of generative AI technologies. Healthcare organizations are eager to learn about generative AI and how they can use it to make a real difference.

Looking ahead

The power of AI has reinforced Google Cloud’s commitment to privacy, security, and transparency. Our platforms are designed to be flexible, including data and model lineage capabilities, integrated security and identity management services, support for third-party models, choice and transparency on models and costs, integrated billing and entitlement support, and support across many languages. 

While we’ll have some innovations like Med-PaLM 2 that are tuned for healthcare, we also have products that are relevant across industries. Last month, we announced several generative AI capabilities coming to Google Cloud, including Generative AI support in Vertex AI and Generative AI App Builder, which are already being tested by a number of customers. Developers and businesses already use Vertex AI to build and deploy machine learning models and AI applications at scale, and we recently added Generative AI support in Vertex AI. This gives customers foundation models they can fine-tune with their own data, and the ability to deploy applications with this powerful new technology. We also launched Generative AI App Builder to help organizations build their own AI-powered chat interfaces and digital assistants in minutes or hours by connecting conversational AI flows with out-of-the-box search experiences and foundation models.

As AI proves its value, it’s likely there will be increased focus on high-quality data collection and curation in healthcare and life sciences. Improving the flow and unification of data across health care systems, referred to as data interoperability, is one of the most important building blocks to leveraging AI, and it helps organizations run more effectively, improve patient care, and helps people live healthier lives. We expect to continue our investments in technology, infrastructure, and data governance.

We’re committed to realizing the potential of this technology in healthcare. By working with a handful of trusted healthcare organizations early on, we’ll learn more about what can be achieved, and how this technology can safely advance. For all of us, the prospects are inspiring, humbling, and exciting. 

If you’re interested in exploring generative AI on Cloud, you can sign-up for our Trusted Tester program or reach out to your Google Cloud sales representative.

Many machine learning (ML) use cases, like fraud detection, ad targeting, and recommendation engines, require near real-time predictions. The performance of these predictions is heavily dependent on access to the most up-to-date data, with delays of even a few seconds making all the difference. But it’s difficult to set up the infrastructure needed to support high-throughput updates and low-latency retrieval of data.

Starting this month, Vertex AI Matching Engine and Feature Store will support real-time Streaming Ingestion as Preview features. With Streaming Ingestion for Matching Engine, a fully managed vector database for vector similarity search, items in an index are updated continuously and reflected in similarity search results immediately. With Streaming Ingestion for Feature Store, you can retrieve the latest feature values with low latency for highly accurate predictions, and extract real-time datasets for training.

For example, Digits is taking advantage of Vertex AI Matching Engine Streaming Ingestion to help power their product, Boost, a tool that saves accountants time by automating manual quality control work.“Vertex AI Matching Engine Streaming Ingestion has been key to Digits Boost being able to deliver features and analysis in real-time. Before Matching Engine, transactions were classified on a 24 hour batch schedule, but now with Matching Engine Streaming Ingestion, we can perform near real time incremental indexing – activities like inserting, updating or deleting embeddings on an existing index, which helped us speed up the process. Now feedback to customers is immediate, and we can handle more transactions, more quickly,” said Hannes Hapke, Machine Learning Engineer at Digits.

This blog post covers how these new features can improve predictions and enable near real-time use cases, such as recommendations, content personalization, and cybersecurity monitoring.

Streaming Ingestion enables real-time AI

As organizations recognize the potential business impact of better predictions based on up-to-date data, more real-time AI use cases are being implemented. Here are some examples:

• Real-time recommendations and a real-time marketplace: By adding Streaming Ingestion to their existing Matching Engine-based product recommendations, Mercari is creating a real-time marketplace where users can browse products based on their specific interests, and where results are updated instantly when sellers add new products. Once it’s fully implemented, the experience will be like visiting an early-morning farmer’s market, with fresh food being brought in as you shop. By combining Streaming Ingestion with Matching Engine’s filtering capability, Mercari can specify whether or not an item should be included in the search results, based on tags such as “online/offline” or “instock/nostock.”

• Large-scale personalized content streaming: For any stream of content representable with feature vectors (including text, images, or documents), you can design pub-sub channels to pick up valuable content for each subscriber’s specific interests. Because Matching Engine is scalable (i.e., it can process millions of queries each second), you can support millions of online subscribers for content streaming, serving a wide variety of topics that are changing dynamically. With Matching Engine’s filtering capability, you also have real-time control over what content should be included, by assigning tags such as “explicit” or “spam” to each object. You can use Feature Store as a central repository for storing and serving the feature vectors of the contents in near real time.

• Monitoring: Content streaming can also be used for monitoring events or signals from IT infrastructure, IoT devices, manufacturing production lines, and security systems, among other commercial use cases. For example, you can extract signals from millions of sensors and devices and represent them as feature vectors. Matching Engine can be used to continuously update a list of “the top 100 devices with possible defective signals,” or “top 100 sensor events with outliers,” all in near real time.

• Threat/spam detection: If you are monitoring signals from security threat signatures or spam activity patterns, you can use Matching Engine to instantly identify possible attacks from millions of monitoring points. In contrast, security threat identification based on batch processing often involves potentially significant lag, leaving the company vulnerable. With real-time data, your models are better able to catch threats or spams as they happen in your enterprise network, web services, online games, etc.

Implementing streaming use cases

Let’s take a closer look at how you can implement some of these use cases.

Real-time recommendations for retail

Mercari built a feature extraction pipeline with Streaming Ingestion.

The feature extraction pipeline is defined with Vertex AI Pipelines, and is periodically invoked by Cloud Scheduler and Cloud Functions to initiate the following process:

1. Get item data: The pipeline issues a query to fetch the updated item data from BigQuery.
2. Extract feature vector: The pipeline runs predictions on the data with the word2vec model to extract feature vectors.
3. Update index: The pipeline calls Matching Engine APIs to add the feature vectors to the vector index. The vectors are also saved to Cloud Bigtable (and can be replaced with Feature Store in the future).

“We have been evaluating the Matching Engine Streaming Ingestion and couldn’t believe the super short latency of the index update for the first time. We would like to introduce the functionality to our production service as soon as it becomes GA, ” said Nogami Wakana, Software Engineer at Souzoh (a Mercari group company).

This architecture design can be also applied to any retail businesses that need real-time updates for product recommendations.

Ad targeting

Ad recommender systems benefit significantly from real-time features and item matching with the most up-to-date information. Let’s see how Vertex AI can help build a real-time ad targeting system.

The first step is generating a set of candidates from the ad corpus. This is challenging because you must generate relevant candidates in milliseconds and ensure they are up to date. Here you can use Vertex AI Matching Engine to perform low-latency vector similarity matching, generate suitable candidates, and use Streaming Ingestion to ensure that your index is up-to-date with the latest ads.

Next is reranking the candidate selection using a machine learning model to ensure that you have a relevant order of ad candidates. For the model to use the latest data, you can use Feature Store Streaming Ingestion to import the latest features and use online serving to serve feature values at low latency to improve accuracy.

After reranking the ads candidates, you can apply final optimizations, such as applying the latest business logic. You can implement the optimization step using a Cloud Function or Cloud Run.

What’s Next?

Interested? The documents for Streaming Ingestion are available and you can try it out now. Using the new feature is easy: For example, when you create an index on Matching Engine with the REST API, you can specify the indexUpdateMethod attribute as STREAM_UPDATE.

{
    displayName: "'${DISPLAY_NAME}'", 
    description: "'${DISPLAY_NAME}'",
    metadata: {
       contentsDeltaUri: "'${INPUT_GCS_DIR}'", 
       config: {
          dimensions: "'${DIMENSIONS}'",
          approximateNeighborsCount: 150,
          distanceMeasureType: "DOT_PRODUCT_DISTANCE",
          algorithmConfig: {treeAhConfig: {leafNodeEmbeddingCount: 10000, leafNodesToSearchPercent: 20}}
       },
    },
    indexUpdateMethod: "STREAM_UPDATE"
}

After deploying the index, you can update or rebuild the index (feature vectors) with the following format. If the data point ID exists in the index, the data point is updated, otherwise, a new data point is inserted.

{
    
datapoints: [
        
{datapoint_id: "'${DATAPOINT_ID_1}'", feature_vector: [...]}, 
        {datapoint_id: "'${DATAPOINT_ID_2}'", feature_vector: [...]}
    
]
}

It can handle the data point insertion/update at high throughput with low latency. The new data point values will be applied in any new queries within a few seconds or milliseconds (the latency varies depending on the various conditions).

The Streaming Ingestion is a powerful functionality and very easy to use. No need to build and operate your own streaming data pipeline for real-time indexing and storage. Yet, it adds significant value to your business with its real-time responsiveness.

To learn more, take a look at the following blog posts for learning Matching Engine and Feature Store concepts and use cases:

• Vertex Matching Engine: Blazing fast and massively scalable nearest neighbor search
• Find anything blazingly fast with Google’s vector search technology
• Kickstart your organization’s ML application development flywheel with the Vertex Feature Store
• Meet AI’s multitool: Vector embeddings

Documents like payslips and W2s are crucial to processes such as employment and income verification for mortgage loans, personal loans, personal finance, and benefits processing. Unfortunately, efficiently extracting data from these documents at scale can be challenging and time-consuming, with many organizations relying on manual examination of documents or automated approaches that don’t adequately capture the document data needed for given tasks. Google Cloud built Document AI to remove these barriers, empowering customers to deploy powerful machine learning models to more quickly process documents, save money, and discover insights. We’re excited to expand Document AI’s capabilities with the recent release of improved pre-trained models for W2s and payslips, built on Document AI Workbench.

Pre-trained models let developers focus on core application logic and leave the complex task of information extraction from the documents to Google’s AI technology. In many cases, the primary driver for automated data extraction is operational efficiency and cost savings, but Document AI can also open new possibilities. For example, a financial services company might use Document AI to enable fully self-serve loan applications on mobile devices, helping the organization to differentiate itself with simple, fast customer experiences.

We’ve heard from customers that more granular entity extraction from W2 and payslip documents is particularly important, with organizations requiring support for a wider variety of layouts and formats. The recent launch of the stable release of these pretrained models addresses these requests.

Here is what is new with W2 parser:

• The parser improves accuracy and entity specificity thanks to the ability to break down long entities such as addresses into fine-grained sub-entities like StreetAddressOrPostalBox, AdditionalStreetAddressOrPostalBox, City, State, and ZIP code.
• It can handle a wider variation of W2 forms, including multi-copies (2,3,4-ups) issued by various payroll vendors. The model is not limited to specific tax years, which means it should be able to process W2 for 2022 or beyond provided there are not significant changes to the format.
• It introduces eight new entities for Box 12 that represent both codes and values, enriching understanding of the various taxable and non-taxable components of the W2 recipient’s income.

Here is what is new with Payslip parser:

• Bonus, commissions, holiday, overtime, regular pay, and vacation are now part of earning_item/earning_this_period and earning_item/earning_ytd. The parser captures types of earnings beyond those categories, and maps them to their respective earning rates, hours, and pay (both for the period and year-to-date). This helps in building a more detailed understanding of the components of the payslip recipient’s income
• The parser now returns year-to-date and current-period taxes and deductions.
• Direct deposits are linked to corresponding bank account numbers.
• The parser now returns page numbers, state and federal tax exemptions, and filing statuses.

While these parsers have become more useful out of the box, with this release, the ability to uptrain makes them easy to modify as new needs arise. Uptraining lets developers further improve the accuracy of these models and extract additional fields with minimal development work. It also lets developers customize existing parsers to support new document types that are similar. For example, the parser is trained on U.S. data and could be uptrained to create a payslip parser for the U.K.

We’re pleased that parsers are already making a difference for customers. Bryan Jackson, CTO at lending automation firm Gateless, said, “High accuracy data extraction is critical to the success of our Smart Underwrite solution, and Document AI provided better results than competitors. Using the latest W2 & Payslip pretrained parsers, we saw a 48% increase in performance on pay stubs and a 15% performance improvement in W2s. The ability to easily uptrain models as new document variations are introduced ensures we continue to deliver optimal outcomes for our customers.”

Additional pre-trained models available as release candidates include parsers for 1040, 1099R, 1120, and 1120S documents. Check for details here. To learn more, talk to a Google Cloud sales executive about how Document AI can help your business, and check out our Document AI breakout session from Google Cloud Next ’22.