Documents like payslips and W2s are crucial to processes such as employment and income verification for mortgage loans, personal loans, personal finance, and benefits processing. Unfortunately, efficiently extracting data from these documents at scale can be challenging and time-consuming, with many organizations relying on manual examination of documents or automated approaches that don’t adequately capture the document data needed for given tasks. Google Cloud built Document AI to remove these barriers, empowering customers to deploy powerful machine learning models to more quickly process documents, save money, and discover insights. We’re excited to expand Document AI’s capabilities with the recent release of improved pre-trained models for W2s and payslips, built on Document AI Workbench.

Pre-trained models let developers focus on core application logic and leave the complex task of information extraction from the documents to Google’s AI technology. In many cases, the primary driver for automated data extraction is operational efficiency and cost savings, but Document AI can also open new possibilities. For example, a financial services company might use Document AI to enable fully self-serve loan applications on mobile devices, helping the organization to differentiate itself with simple, fast customer experiences.

We’ve heard from customers that more granular entity extraction from W2 and payslip documents is particularly important, with organizations requiring support for a wider variety of layouts and formats. The recent launch of the stable release of these pretrained models addresses these requests.

Here is what is new with W2 parser:

• The parser improves accuracy and entity specificity thanks to the ability to break down long entities such as addresses into fine-grained sub-entities like StreetAddressOrPostalBox, AdditionalStreetAddressOrPostalBox, City, State, and ZIP code.
• It can handle a wider variation of W2 forms, including multi-copies (2,3,4-ups) issued by various payroll vendors. The model is not limited to specific tax years, which means it should be able to process W2 for 2022 or beyond provided there are not significant changes to the format.
• It introduces eight new entities for Box 12 that represent both codes and values, enriching understanding of the various taxable and non-taxable components of the W2 recipient’s income.

Here is what is new with Payslip parser:

• Bonus, commissions, holiday, overtime, regular pay, and vacation are now part of earning_item/earning_this_period and earning_item/earning_ytd. The parser captures types of earnings beyond those categories, and maps them to their respective earning rates, hours, and pay (both for the period and year-to-date). This helps in building a more detailed understanding of the components of the payslip recipient’s income
• The parser now returns year-to-date and current-period taxes and deductions.
• Direct deposits are linked to corresponding bank account numbers.
• The parser now returns page numbers, state and federal tax exemptions, and filing statuses.

While these parsers have become more useful out of the box, with this release, the ability to uptrain makes them easy to modify as new needs arise. Uptraining lets developers further improve the accuracy of these models and extract additional fields with minimal development work. It also lets developers customize existing parsers to support new document types that are similar. For example, the parser is trained on U.S. data and could be uptrained to create a payslip parser for the U.K.

We’re pleased that parsers are already making a difference for customers. Bryan Jackson, CTO at lending automation firm Gateless, said, “High accuracy data extraction is critical to the success of our Smart Underwrite solution, and Document AI provided better results than competitors. Using the latest W2 & Payslip pretrained parsers, we saw a 48% increase in performance on pay stubs and a 15% performance improvement in W2s. The ability to easily uptrain models as new document variations are introduced ensures we continue to deliver optimal outcomes for our customers.”

Additional pre-trained models available as release candidates include parsers for 1040, 1099R, 1120, and 1120S documents. Check for details here. To learn more, talk to a Google Cloud sales executive about how Document AI can help your business, and check out our Document AI breakout session from Google Cloud Next ’22.

Healthcare breakthroughs change the world and bring hope to humanity through scientific rigor, human insight, and compassion. We believe AI can contribute to this, with thoughtful collaboration between researchers, healthcare organizations and the broader ecosystem. 

Today, we’re sharing exciting progress on these initiatives, with the announcement of limited access to Google’s medical large language model, or LLM, called Med-PaLM 2. It will be available in coming weeks to a select group of Google Cloud customers for limited testing, to explore use cases and share feedback as we investigate safe, responsible, and meaningful ways to use this technology. 

Med-PaLM 2 harnesses the power of Google’s LLMs, aligned to the medical domain to more accurately and safely answer medical questions. As a result, Med-PaLM 2 was the first LLM to perform at an “expert” test-taker level performance on the MedQA dataset of US Medical Licensing Examination (USMLE)-style questions, reaching 85%+ accuracy, and it was the first AI system to reach a passing score on the MedMCQA dataset comprising Indian AIIMS and NEET medical examination questions, scoring 72.3%.

Industry-tailored LLMs like Med-PaLM 2 are part of a burgeoning family of generative AI technologies that have the potential to significantly enhance healthcare experiences. We’re looking forward to working with our customers to understand how Med-PaLM 2 might be used to facilitate rich, informative discussions, answer complex medical questions, and find insights in complicated and unstructured medical texts. They might also explore its utility to help draft short- and long-form responses and summarize documentation and insights from internal data sets and bodies of scientific knowledge.

Innovating responsibly with AI

Since last year, we’ve been researching and evaluating Med-PaLM and Med-PaLM 2, assessing it against multiple criteria — including scientific consensus, medical reasoning, knowledge recall, bias, and likelihood of possible harm — which were evaluated by clinicians and non-clinicians from a range of backgrounds and countries. 

Med-PaLM 2’s impressive performance on medical exam-style questions is a promising development, but we need to learn how this can be harnessed to benefit healthcare workers, researchers, administrators, and patients. In building Med-PaLM 2, we’ve been focused on safety, equity, and evaluations of unfair bias. Our limited access for select Google Cloud customers will be an important step in furthering these efforts, bringing in additional expertise across the healthcare and life sciences ecosystem. 

What’s more, when Google Cloud brings new AI advances to our products, our commitment is two-fold: to not only deliver transformative capabilities, but also ensure our technologies include proper protections for our organizations, their users, and society. To this end, our AI Principles, established in 2017, form a living constitution that guides our approach to building advanced technologies, conducting research, and drafting our product development policies. 

From AI to generative AI 

Google’s deep history in AI informs our work in generative AI technologies, which can find complex relationships in large sets of training data, then generalize from what they learn to create new data. Breakthroughs such as the Transformer have enabled LLMs and other large models to scale to billions of parameters, letting generative AI move beyond the limited pattern-spotting of earlier AIs and into the creation of novel expressions of content, from speech to scientific modeling. 

Google Cloud is committed to bringing to market products that are informed by our research efforts across Alphabet. In 2022, we introduced a deep integration between Google Cloud and Alphabet’s AI research organizations, which allows Vertex AI to run DeepMind’s groundbreaking protein structure prediction system, AlphaFold.

Much more is on the way. In one sense, generative AI is revolutionary. In another, it’s the familiar technology story of more and better computing creating new industries, from desktop publishing to the internet, social networks, mobile apps, and now, generative AI.

Building on AI leadership

Additionally, today we’re announcing a new AI-enabled Claims Acceleration Suite, designed to streamline processes for health insurance prior authorization and claims processing. The Claims Acceleration Suite helps both providers of insurance plans and healthcare to create operational efficiencies and reduce administrative burdens and costs by converting unstructured data into structured data that help experts make faster decisions and improve access to timely patient care. 

On the clinical side, last year we announced Medical Imaging Suite, an AI-assisted diagnosis technology being used by Hologic to improve cervical cancer diagnoses and Hackensack Meridian Health to predict metastasis in patients with prostate cancer. Elsewhere, Mayo Clinic and Google have collaborated on an AI algorithm to improve the care of head and neck cancers, and Google Health recently partnered with iCAD to improve breast cancer screening with AI.

From these examples and more, it’s clear that the healthcare industry has moved from testing AI to deploying it to improve workflows, solve business problems, and speed healing. With this in mind, we expect rapid interest in and uptake of generative AI technologies. Healthcare organizations are eager to learn about generative AI and how they can use it to make a real difference.

Looking ahead

The power of AI has reinforced Google Cloud’s commitment to privacy, security, and transparency. Our platforms are designed to be flexible, including data and model lineage capabilities, integrated security and identity management services, support for third-party models, choice and transparency on models and costs, integrated billing and entitlement support, and support across many languages. 

While we’ll have some innovations like Med-PaLM 2 that are tuned for healthcare, we also have products that are relevant across industries. Last month, we announced several generative AI capabilities coming to Google Cloud, including Generative AI support in Vertex AI and Generative AI App Builder, which are already being tested by a number of customers. Developers and businesses already use Vertex AI to build and deploy machine learning models and AI applications at scale, and we recently added Generative AI support in Vertex AI. This gives customers foundation models they can fine-tune with their own data, and the ability to deploy applications with this powerful new technology. We also launched Generative AI App Builder to help organizations build their own AI-powered chat interfaces and digital assistants in minutes or hours by connecting conversational AI flows with out-of-the-box search experiences and foundation models.

As AI proves its value, it’s likely there will be increased focus on high-quality data collection and curation in healthcare and life sciences. Improving the flow and unification of data across health care systems, referred to as data interoperability, is one of the most important building blocks to leveraging AI, and it helps organizations run more effectively, improve patient care, and helps people live healthier lives. We expect to continue our investments in technology, infrastructure, and data governance.

We’re committed to realizing the potential of this technology in healthcare. By working with a handful of trusted healthcare organizations early on, we’ll learn more about what can be achieved, and how this technology can safely advance. For all of us, the prospects are inspiring, humbling, and exciting. 

If you’re interested in exploring generative AI on Cloud, you can sign-up for our Trusted Tester program or reach out to your Google Cloud sales representative.

Each day, more documents are created and used across companies to make decisions. However, the value in these documents is primarily expressed as unstructured data, which makes the value difficult and manually intensive to extract and use for business processes. 

As the number and variety of documents used by businesses proliferate, machine learning (ML) solutions need to be more flexible to handle the broader set of use cases. That’s why we introduced the first Document AI Workbench model, Custom Document Extractor (CDE), in Public Preview at Google Cloud Next ‘22. CDE makes it fast and easy to apply ML to virtually any document-based workflow to extract structured data from unstructured document types, to automate business processes. 

CDE lets developers and analysts use their own data to train models and extract fields from documents needed for the business. CDE lets organizations build models faster and with less data — thus accelerating time-to-value for processing and analysis of data in documents.

Today, we announce that Document AI Workbench is Generally Available (GA), open to all customers, ready for production use through APIs and the Google Cloud Console. Document AI Workbench is covered by the Document AI SLA — online and batch document prediction is supported with >=99.9% uptime. Furthermore, Document AI Workbench is now covered by Google Cloud’s GA product terms. For example, Google will notify customers at least 12 months before significantly modifying a customer-facing Google API in a backwards-incompatible manner.

In this blog post, we’ll explore ways customers are already using CDE and Document AI Workbench’s updated capabilities.

What users are saying about Workbench

Deliver higher model accuracy with Workbench

Users leverage Workbench to ultimately save time and money. A third party evaluated Document AI Workbench and concluded that it extracts data more accurately¹ than several competing products for document types with variable layouts (e.g. invoice, receipt, bank statements, paystubs). Better accuracy drives higher automation rates, helping Workbench users save time and money.  

Chris Jangareddy, managing director for Artificial Intelligence & Data at Deloitte Consulting LLP said, “Google Cloud Document AI is a leading document processing solution packed with rich features like multi-step classify and text extraction to automate sorting, classification, extraction, and quality assurance. By combining Document AI with Workbench, Google Cloud has created a forward-thinking and powerful AI platform for intelligent document processing that will allow for process transformation at an enterprise scale with predictable outcomes that can benefit businesses.”

Mansoor Khan, CEO of OneClinic said, “We help medical professionals scale their clinics through automation. We used Google’s Document AI Workbench to create a model to automatically extract data from patients’ insurance cards as part of our patient check-in software. Workbench is easy to use and we are really happy with the model accuracy — it extracts data more accurately than what we would expect from human data entry.”

Rajnish Palande, VP, Google Business Unit for BFSI, TCS said, “The Google Cloud Document AI Workbench leverages artificial intelligence (AI) to manage and glean insights from unstructured data. The Workbench brings together the power of classification, auto-annotation, page-number identification and multi-language support to help organizations rapidly deliver enhanced accuracy, improved operational efficiency, higher confidence in the information extract, and increased return on investment.” 

Build production ready models faster with Workbench

Document AI Workbench helps users create machine learning models faster. For example, a third-party evaluation shows Document AI Workbench trains machine learning models up to 3x faster than a leading competitor. This is an important improvement which lowers total cost of ownership and increases value.

Dallas Dolen, Partner, Google Alliance Leader at PwC said, “Google Document AI Workbench helps to accelerate our custom parser models training as well as improves accuracy and performance using a custom document extractor with human in the loop. It helps us solve complex business problems for our clients in the financial services and healthcare industries.”

Ziang Jia, Senior DocAl Development Lead at Resultant, said, “Document AI Workbench has unlocked a brand-new machine learning development experience for information extraction solutions. Its simplicity and robustness enabled us to build models and deliver a highly accurate outcome in an agile way for a large government agency. We couldn’t be more impressed by its simplicity and robustness and are excited to see how the product will evolve in the future.”

Sean Earley, VP of Delivery Services of Zencore said, “Document AI Workbench allows us to develop highly accurate document parsing models in a matter of days. Our customers have automated tasks that formerly required significant human labor. For example, using Document AI Workbench, a team of two trained a model to split, classify and extract data from 15 document types to automate Home Mortgage Disclosure Act reporting. The mean trained model accuracy was 94%, drastically reducing the operational cost of our customer’s compliance reporting procedures.”

What’s new with Document AI Workbench 

The latest Workbench capabilities make it even easier to train and deploy an extraction model: 

• With Workbench’s public APIs, you can programmatically create, delete, train, evaluate and deploy models.
• Our updated dataset management tools automatically detect and create existing schema labels from your pre-annotated documents. They also provide you more flexibility when creating and managing schema.
• Our new DocAI Toolkit includes a labeled document converter so that you can easily convert your labeled documents to DocAI’s format and start training faster.
• We’ve reduced the cognitive load for labelers with efficiency enhancements to our Labeling UI.    
• The revamped Processor Gallery helps you quickly identify the best model for your use case.

 What’s next for Document AI Workbench

We continue to invest in Document AI Workbench to help you automate document processing. Here are a few things we’re working on that we’re excited about:

• Classify document types with the Custom Document Classifier (CDC), coming soon in public preview
• Copy processor versions across projects and processors to streamline managing development and production environments
• Support larger documents (e.g., longer than 50 pages) so you can process a wider array of documents
• Broader (non-latin) language support–equivalent to Document AI OCR
• And many more investments, using state of the art technology, to help you build world class models faster to automate document processing

Document AI Workbench is in GA and ready for production workloads. Learn more via Document AI Workbench documentation or try it out in the Google Cloud Console.

^{Acknowledgements: Tomas Moreno, Outbound Product Manager, Lukas Rutishauser, Software Engineering Manager, Michael Kwong, Software Engineering Manager, Rajagopal Janani, Software Engineering Manager, Michael Lanning, UX Designer.}

^{1. When trained with 200+ documents}

For years, retailers have been told that they must embrace a litany of new technologies, trends, and imperatives like online shopping, mobile apps, omnichannel, and digital transformation. In search of growth and stability, retailers adopted many of these, only to realize that for every box they ticked, there was another one waiting.

And then the pandemic hit, along with rising social movements and increasingly harsh weather. Some retailers were more prepared to take on these disruptions than others, which crystallized a new universal truth across the industry: the ability to adapt on the fly became the most important trait to survive and thrive.

Today’s retail landscape has surfaced both existing and new challenges for specialty and department store retailers. Approximately 88% of purchases previously occurred within a store environment. Now, it’s closer to 59%, with the remainder done online or through other omni methods. 

With such constant change and upheaval, it can feel like the mantra now is ABP: always be pivoting. 

The big question isn’t just how to maintain constant momentum and agility—it’s how to do it without sapping your workforce, your inventory, or your profits in the process. The pivot is now a given. What matters is how you do it.

Adapting requires a flexible base of technology that allows retailers to shift and scale seamlessly with the needs of the moment. 

They need to be able to leverage real-time insights and enhance customer experiences rapidly, online and in the real world (not to mention the growing hybridization that’s AR and VR). They need to modernize their stores to power engaging consumer and associate experiences. They need to enhance operations for rapid scaling between full operations and digital-only offerings.

To help retailers achieve these goals and more, Google Cloud is honing a trio of essential innovations: demand forecasting that harnesses the power of data analytics and artificial intelligence; enhanced product discovery to improve conversion across channels; and the tools to help create the modern store experience.

In other words, here’s some of the biggest ways we’re ready to help you pivot.

Pivot point 1: Harnessing data and AI for demand forecasting with Vertex AI

One of the greatest challenges for retailers when building organizational flexibility is managing inventory and the supply chain. 

We are in the midst of one of the worst global supply chain crises, stemming from soaring demand and logistics issues brought on by the pandemic. This crisis has only heightened the challenge retailers face when assessing demand and product availability. Even in normal times, mismanagement of inventory can add up to a trillion-dollar problem, according to IHL Group (costing $634 billion in lost sales worldwide each year, while overstocks result in $472 billion in lost revenues due to markdowns). 

On the flipside, optimizing your supply chain can lead to greater profits. For instance, McKinsey predicts that a 10% to 20% improvement in retail supply chain forecasting accuracy is likely to produce a 5% reduction in inventory costs and a 2% to 3% increase in revenues.

Some of the challenges related to demand forecasting include:

• Low accuracy leads to excess inventory, missed sales, and pressure on fragile supply chains.
• Real drivers of product demand are not included, because large datasets are hard to model using traditional methods.
• Poor accuracy for new product launches and products that have sparse or intermittent demand.
• Complex models are hard to understand, leading to poor product allocation and low return on investment on promotions.
• Different departments use different methods, leading to miscommunication and costly reconciliation errors.

AI-based demand forecasting techniques can help. Vertex AI Forecast supports retailers in maintaining greater inventory flexibility by infusing machine learning into their existing systems. Machine learning and AI-based forecasting models like Vertex AI are able to digest large sets of disparate data, drive analytics and automatically adjust when provided with new information. 

With these machine learning models, retailers can not only incorporate historical sales data, but also use close to real-time data such as marketing campaigns, web actions like a customer clicking the “add to cart” button on a website, local weather forecasts, and much more. 

Pivot point 2: Enhanced product discovery through AI-powered search and recommendations

If customers can’t easily find what they are looking for, whether online or at the store, they will turn to someone else. That’s a simple statement, but one with profound impacts.

In research conducted by The Harris Poll and Google Cloud, we found that over a six month period, 95% of consumers received search results that were not relevant to what they were searching for on a retail website. And roughly 85% of consumers view a brand differently after an unsuccessful search, while 74% say they avoid websites where they’ve experienced search difficulties in the past.

Each year, retailers lose more than $300 billion dollars from search abandonment, or when a consumer searches for a product on a retailer’s website but does not find what they are looking for. Our product discovery solutions help you surface the right products, to the right customers, at the right time. These solutions include: 

• Vision Product Search, which is like bringing the augmented reality of Google Lens to a retailer’s own branded mobile app experience. Both shoppers and retail store associates can search for products using an image they’ve photographed or found online and receive a ranked list of similar items.
• Recommendations AI, which enables retailers to deliver highly personalized recommendations at scale across channels.
• Retail Search, which provides Google-quality search results on a retailer’s own website and mobile applications.

All three are powered by Google Cloud, leveraging Google’s advanced understanding of user context and intent, utilizing technology to deliver a seamless experience to every shopper. With these combined capabilities, retailers are able to reduce search abandonment and improve conversions across their digital properties. 

Pivot point 3: Building the modern store

Stores are no longer places for just browsing and buying. They must be flexible operation centers, ready to pivot to address changing circumstances. The modern store must be multiple things at once: a mini-fulfillment and return center, a recommendation engine, a shopping destination, a fun place to work, and more. 

Just as retail companies had to embrace omnichannel, stores are now becoming omnichannel centers on their own, mixing the digital and physical into a single location. Retailers can use physical stores as a vehicle to deliver superior customer experiences. This will demand heightened levels of collaboration and cooperation between stores, digital, and tech infrastructure teams, building on the agile ways they have worked together.

In many ways, it’s about allowing our physical spaces to function more like digital ones. Google Cloud can help by bringing the scalability, security, and reliability of the cloud to the store, allowing physical locations to upgrade infrastructure and modernize their internal and customer-facing applications. 

Think of it as when a new OS gets released for your phone. It’s the same small, hard box, but the experience can feel radically different. Now, extend that same idea to a digitally enabled store. With the right displays, interfaces, and tools at a given retail location, the team only needs to send an over-the-air update to create radically fresh experiences, ranging from sales displays to fulfillment or employee engagement.

Such an approach can enable streamlined experiences for both customers and store associates. For instance, when it comes to the modern store’s evolving role as a fulfillment or return center, cloud solutions can help drive efficiency in stores through automation of ordering, replenishment, and fulfillment of omnichannel order selection.  

Similar tools for personalized product discovery online can be applied to customers in the store, helping them to browse and explore, or even create a tailored shopping experience. 

The impact of store associates can be maximized by equipping them with technology to provide expertise that drives value-added customer service, as well as increasing productivity in stores by streamlining operations, thus lowering overhead cost. At the register, customers should be able to enjoy frictionless checkout while ensuring reliable, accurate, secure transactions.

Google Cloud can help retailers transform

The ability to adapt and pivot to meet today’s changing consumer needs requires that retailers rely on modern tools to obtain operational flexibility. We believe that every company can be a tech company. That every decision is data driven. That every store is physical and digital all at once. That every worker is a tech worker. 

Google Cloud works with retailers to help them solve their most challenging problems. We have the unique ability to handle massive amounts of unstructured data, in addition to advanced capabilities in AI and ML. Our products and solutions help retailers focus on what’s most important—from improving operations to capturing digital and omnichannel revenue.

The app economy has enabled a huge range of unique business models to flourish. One such model is online food ordering and delivery services, in which apps leverage geo-location data to aggregate local food choices and offer personalized options to consumers.

A leading company in this space is Just Eat. Launched in the UK in 2001 with a vision of ‘serving the world’s greatest menu. Brilliantly.’ The company has capitalized on the popularity of online food delivery and grown its presence across 12 markets. 

Just Eat acts as an intermediary between take-out food outlets and hungry customers, giving local restaurants access to a broader base of potential diners, while providing consumers with an easy and secure way to order and pay for food from their favourite restaurants.

Today the company helps 27 million customers find food from more than 112,000 restaurants—everything from homemade Italian pasta, to Chinese noodle bowls, to fish-and-chips. 

Data is the fuel of Just Eat’s rapid growth, but it wasn’t always looked at that way. In its early days, Just Eat struggled with the deluge of information and faced fragmentation across its systems. In fact, the company realized its legacy data vendor wasn’t capable of ingesting 90 percent of the data produced by its food platform. This was incredibly frustrating for Just Eat’s analysts and data scientists, who had to waste time cleaning up sources instead of leveraging the data to create a better user experience. 

Just Eat turned to Google Cloud, and now uses machine learning (ML) to power sophisticated consumer recommendations on both its app and website. It also makes heavy use of features offered by Google Cloud Platform, including BigQuery for running analytics on its customer data set and Cloud Pub/Sub for messaging app users with relevant offers in real-time. 

Having all of Just Eat’s data in one platform has translated into real value for its customers. With Google Cloud tools, Just Eat has created its own proprietary Customer Ontology framework, which today contains 5.5 billion features that better understand consumers’ behavior and food habits, and provides insights into previous visits.

Just Eat recently created an “Adventurous Index” to map its customers according to their ordering habits, enabling them to tailor their marketing and user experiences. For example, mid-adventurous customers are shown a choice of restaurants that serve their most ordered cuisine, while adventurous customers can choose from restaurants that serve a wider variety. This not only has prompted consumers to be more adventurous with their choices, but also has led to more business at a more diverse set of restaurants.

Matt Cresswell, Director of Customer Platforms at Just Eat said that Google Cloud has become integral to its product delivery: “Consumer food choice is a hugely nuanced topic. We know that individuals have their own unique journeys when they use Just Eat. We’ve sought to create a truly one-to-one relationship with every customer. The changes we’ve made to the platform mean they can access the dishes they enjoy at the touch of a fingertip, and find inspiration to discover new dishes they’ll love. We’re grateful to Google Cloud for helping us support our customers on their culinary explorations.”