Editor’s note: In this post, I’ll be showing some amazing ways Document AI can help you extract meaning from your documents  – keep reading, or jump directly into a tutorial using the Cloud Console!

Documents are a crucial part of most businesses, used to store and communicate important information. The variety is vast: invoices, contracts, receipts, applications, plus documents unique within industries and geographies. Unfortunately, making the information contained in these documents accessible can be a time-consuming and manual process.

Document AI is a document understanding platform in Google Cloud that takes unstructured data from documents and transforms it into structured data, making them easier to understand, analyze, and consume. By using this technology, you can streamline your document processing workflows, reduce errors, and unlock insights that were previously buried in mountains of paperwork. 

Whether you’re a small business owner or an enterprise looking to bring efficiency to your operations, Document AI has something to offer. So let’s take a look and see what it can do!

Understanding documents with Document AI

When we say that Document AI can understand documents, we mean that it is able to analyze the content within documents and derive meaningful insights from it. This goes beyond simply recognizing the characters and words within a document (which is what traditional OCR technology does) – Document AI can actually comprehend the meaning behind the text.

For example, let’s say you have a contract that needs to be processed. Traditional OCR technology might be able to extract the text from the document, but it would not be able to understand the legal terms and clauses within it. Document AI, on the other hand, can actually interpret the meaning of the text and extract key information such as parties involved, terms and conditions, dates, and signatures.

Document AI offers several pre-built models and processors that are specifically designed to extract different types of data from various document types. Within the specific document types, the processors can perform several tasks such as Optical Character Recognition (OCR), form parsing, splitting, classification or entity extraction. These processors can be customized and combined to create powerful document processing workflows that are tailored to a business’s unique needs.

Let’s look closer at a few of the processors available in Document AI, including the Form Parser, Invoice Parser, Expense Parser, Identity Document Proofing Parser, and Intelligent Document Quality Processor.

Form Parser

This general processor is designed to extract structured data from forms such as application forms, surveys, and questionnaires. It automatically identifies and extracts data from form fields (key-value pairs), such as names, addresses, dates, and other types of structured data; even checkboxes and tables. This processor also leverages deep learning models to extract generic entities that are common in various document types, meaning it can identify if something is an email address, phone number, datetime, organization, quantity, price, person, and more.

In this visualization of the API response, you can see that Document AI has identified several key value pairs that correspond to the form’s fields and the responder’s answers.

Also of interest is that the form parser recognized certain generic entities including: several dates, an address, phone numbers, email, and two people (the responder and their listed emergency contact).

Invoice Parser

This parser is designed to identify and extract relevant information from invoices including a large number of typical invoice fields, but can also be customized (uptrained) to recognize different invoice layouts, languages, and data fields.  Invoices are a critical part of the accounts payable process, making this functionality valuable across industries and companies building A/P features into their products.

In this visualization of the API response, you can see that Document AI has extracted a large number of key-value pairs and even provided normalized values for several of the fields.

Expense Parser

This specialized processor is designed to extract data from receipts and invoices, such as vendor name, date, and total amount paid. It can also identify line items within an invoice and categorize them based on the type of expense (e.g. meals, travel, office supplies). The expense parser makes it easier for you to process expense reports and other financial documents, and it can integrate with other tools and systems to completely automate the entire expense reporting process.

In this visualization of the API response, you can see that Document AI has extracted the text from the receipt and identified several typical entities such as purchase date and time, payment type, and total amount.

Identity Document Proofing Parser

This processor is designed to help predict the validity of ID documents with four different signals.

1. is_identity_document detection: Predicts whether an image contains a recognized identity document.
2. suspicious_words detection: Predicts whether words are present that aren’t typical on IDs.
3. image_manipulation detection: Predicts whether the image was altered or tampered via an image editing tool.
4. online_duplicate detection: Predicts whether the image can be found online.

If suspicious words are detected or the image can be found online, additional information is provided to explain these signals. 

This can be particularly useful for businesses that need to verify the identity of customers or employees as part of their operations. This processor could be used in conjunction with other processors that extract key information such as name, date of birth, ID number, and expiration date from specific identity documents (US Driver License Parser, US Passport, France National ID Parser, etc.).

In this visualization of the API response, you can see that Document AI has passed the document on the first detection point (is_identity_document), but failed the document for the other three items and provided additional information in the evidence fields.

Intelligent Document Quality Processor

This general use processor is designed to detect a variety of document quality issues, such as missing pages, blurry images, low contrast, inconsistent formatting, and incorrect data, and flag these potential issues which could affect their usability, accuracy, or compliance. It can also identify sensitive information that has not been redacted or missing information required by regulatory standards.

This quality assessment is returned as a quality score from 0 to 1, where 1 means perfect quality. If the quality score detected is lower than 0.5, a list of negative quality reasons (sorted by the likelihood) is also returned.

In this visualization of the API response, you can see that Document AI has determined a quality_score of 0.006 and also provided a list of several reasons including document and text cutoff, blurriness, and glare (among others).

Next steps

These are just a few examples of the types of processors that Document AI offers. Perusing the current documentation, you can find it has more than 40 processors, with each providing several functions, to explore. Document AI also offers you the ability to uptrain certain processors and supports the option to build your own custom processor. Across the board, if a document contains structured or unstructured text, Document AI has the capability to extract valuable data from it. 

Get started and learn more by heading to our tutorials in the Cloud Console:

• Get started with Document AI in the Google Cloud console
• Build a document processing pipeline with Workflows
• Uptrain a specialized processor
• Create a Custom Document Extractor in the Google Cloud console
• Create a Custom Document Classifier in the Google Cloud console
• Get started with Human-in-the-Loop

Financial institutions across the globe rely on real-time indices to inform real-time portfolio valuations, to provide benchmarks for other investments, and as a basis for passive investment instruments including exchange-traded products (ETPs). This reliance is growing—the index industry dramatically expanded in 2020, reaching revenues of $4.08 billion.

Today, indices are calculated and distributed by index providers with proximity and access to underlying asset data, and with differentiating real-time data processing capabilities. These providers offer subscriptions to real-time feeds of index prices and publish the constituents, calculation methodology, and update frequency for each index.

But as new assets, markets, and data sources have proliferated, financial institutions have developed new requirements. Financial institutions will need to quickly create bespoke and frequently updating indices that represent a specific actual or theoretical portfolio, with its unique constituents and weightings.

In other words, existing index providers and other financial institutions alike will need mechanisms for rapid creation of real-time indices. This blog post’s focus—an index publication pipeline collaboratively developed by CME Group and Google Cloud—is an example of such a mechanism. 

The pipeline closely approximates a particular CME Group index benchmark, but with far greater frequency (in near real time vs. daily) than its official counterpart. It does so by leveraging open-source models such as Apache Beam and cloud-based technologies such as Dataflow, which automatically scales pipelines based on inbound data volume.

Machine learning’s production problem

In the past decade, advances in AI toolchains have enabled faster ML model training—and yet a majority of ML models are still not making it into production. As organizations endeavor to develop their ML capabilities, they soon realize that a real-world ML system is comprised of a small amount of ML code embedded in a network of complex and large ancillary components. Each component brings its own development and operational challenges, which are met by bringing a DevOps methodology to the ML system, commonly referred to as MLOps (Machine Learning Operations). To apply ML to business problems, a firm must develop continuous delivery and automation pipelines for ML.

This index publication collaboration is instructive because it demonstrates MLOps best practices for just such a pipeline. One Apache Beam pipeline, suited for operating on both batch and streaming data, extracts insights and packages them for downstream consumers. These consumers may include ML pipelines that, thanks to Apache Beam, require only one code path for inference across batch and real-time data sources. The pipeline is run inside Google Cloud’s Dataflow execution engine, greatly simplifying management of underlying compute resources. 

But the collaboration’s value is not constrained to the ML and data science realm. The project shows that consumers of the Apache Beam pipeline’s insights may also include traditional business intelligence dashboards and reporting tools. It also demonstrates the simplicity and economy of cloud-based time series data such as CME Smart Stream, which is metered by the hour, quickly and automatically provisioned, and consumable at a per-product-code (not per-feed) level.

A focus on real-time processing for financial services

To illustrate the above points, the collaboration applies data engineering and MLOps best practices to a financial services problem. We chose the financial services domain because many financial institutions do not yet have real-time market data processing or MLOps capabilities today, owing to a significant gap on either side of their ML/AI objectives.

Upstream from ML/AI models, financial institutions often experience a data engineering gap. For many financial institutions, batch processes have sufficiently addressed business requirements. As a result, the temporal nature of the time series data underlying these processes is deemphasized. For example, the original purpose of most trade booking systems was to capture a trade and ensure that it found its way to the middle and back office for settlement. It was not built with ML/AI in mind, and its underlying data therefore has not been packaged for consumption by ML/AI processes. 

And downstream from ML/AI models, financial institutions often encounter the aforementioned “ML production problem.”

As ML/AI becomes ever more strategic, these two gaps have left many financial institutions in a conundrum—unable to train ML models for lack of properly packaged time series data, and unmotivated to package time series data for lack of ML models. By recreating a key energy market index using open-source libraries and cloud-based tools, this collaboration demonstrates that for the financial services domain a solution to this conundrum is more accessible today than ever. 

Creating a new index

We modeled our new index after one of CME Group’s many index benchmarks. The particular index expresses the value of a basket of three New York Mercantile Exchange—listed energy futures as a single price. Today, CME Group publishes the index at the end of the day by calculating the settlement price of each underlying futures contract, and then weighing and summing these values. 

While CME Group does not currently publish this index in real time, this collaboration aims to create a near real-time solution leveraging Google Cloud capabilities and CME Group market data delivered via CME Smart Stream. However, in order to publish the value so frequently—every five seconds, with 40-second publish latency—this collaboration’s pipeline has to solve a number of challenges in near-real time.

First, the pipeline must process sparse data from three separate trades feeds in memory to create open-high-low-close (OHLC) bars. More specifically, for five-second windows for each of the three front-month (and sometimes second-month) energy contracts, a bar must be produced. This is solved by using the Apache Beam library to implement functions which, when executed on Dataflow, automatically scale out as input load increases. The bars must be time-aligned across the underlying feeds, which is greatly simplified by Beam’s watermark feature. And for intervals in which no tick data is observed, the Beam library is used to pull forward the last value received, yielding perfect gap-free bars for downstream processors.

Second, the pipeline must calculate volume-weighted average price (VWAP) in near real-time for each front-month contract. The VWAP calculations are also written using the Beam API and executed on Dataflow. Each of these functions requires visibility of each element in the time window, so the functions cannot be arbitrarily scaled out. Nonetheless, this is tractable because their input—OHLC bars—is manageably small.

Third, the pipeline must replicate CME Group’s specific settlement price methodology for each contract. The rules specify whether to use VWAP or another source as price, depending on certain conditions. They also specify how to weigh combinations of monthly contracts during a roll period. The pipeline again encapsulates these requirements as an Apache Beam class, and joins the separate price streams at the correct time boundary.

The end result is a new stream publishing bespoke index data to a Google Cloud Pub/Sub topic thousands of times daily, enabling AI models as well as traditional industry index usage, dashboards, and other tools to assist real-time decision making. The stream’s pipeline uses open source libraries that solve common time series problems out-of-the box, and cloud-based services to reduce the user’s operational and scaling burden.

The importance of cloud-based data

The promise of cloud-based pipeline execution services cannot be realized using legacy data access patterns, which often require market data users to colocate and configure servers and network gear. Such patterns inject expense and scaling complexity into the pipeline’s overall operation, diverting resources from the adoption of MLOps best practices. Instead, a newer, cloud-based access pattern—in which resources subscribe to data streams inexpensively, rapidly and programatically—is necessary.

In 2018, CME Group identified the customer need for accessible futures and options market data. CME Group collaborated with Google Cloud to launch CME Smart Stream, which distributes CME Group’s real-time market data across Google Cloud’s global infrastructure with sub-second latency. Any customer with a CME Group data usage license and a Google Cloud project can consume this data for an hourly usage fee, without purchasing and configuring servers and network gear.

CME Smart Stream met this index pipeline’s requirements for cost-effective, cloud-based streaming data, but this is just one use case. Since the launch of a CME Smart Stream offering on Google Cloud, globally dispersed firms have adopted the solution. For example, Coin Metrics has been using the offering to better inform its customers in the crypto markets. According to CME Group, Smart Stream has become popular with new customers as the fastest, simplest way to access CME Group’s market data from anywhere in the world. 

Adapt the design pattern to your needs

By combining cloud-based data, open-source libraries, and cloud-based pipeline execution services, we created a real-time index using the same constituents as its end-of-day counterpart. Additionally, financial institutions will find this approach addresses many other challenges—real-time valuation of a large set of portfolios; benchmark creation for new ETPs; or external publication of new indices.

Give it a try

This approach is available to help you meet your organization’s needs. Please review our user guide, whose Tutorials section provides a step-by-step guide to constructing a simple Apache Beam pipeline to generate metrics on streaming data in real-time, and connecting a new data source to the pipeline. We’ll be discussing this topic in CME Group’s webinar End-to-End Market Data Solutions in the Cloud at 10:30 am ET on June 16th.

Over the last year, COVID-19 presented unforeseen challenges for practically every type of business and organization—including schools, colleges, and universities. For educational institutions, the pandemic was an unapologetic agent of acceleration, shifting one billion learners from in-person to online learning within two months. 

The rapid transition to online learning exposed many schools’ lack of readiness for the new online learning environment. It also widened the learning equity gap for students, with fewer than 40% of students from low-income families having access to the tools required for remote learning.

For those who do have online access, today’s students expect everything from engaging and collaborative digital learning experiences to skills-based training for their roles in the future workforce. Expectations are also high for 24×7 multi-channel tech support across all learning devices, applications, and platforms.  

In these remarkable times, education technology companies have an important role to play in supporting academic institutions and students. Indeed, this is already happening, as the EdTech (Educational Technology) market is nearly tripling, with total global expenditures expected to reach $404 billion by 2025. However, the success of these EdTech companies depends on their performance in a number of areas, including:

• Content and products: How quickly can they generate new content and react to learner needs with new products to additional markets for broader adoption?
• Personalization: How effectively can they leverage artificial intelligence (AI) to provide a personalized experience to all types of learners?
• Trust and security: How trusted and secure are their services when educational organizations are suffering the highest number of data breaches since 2005?

Here are a few examples of how EdTech companies are successfully using AI and analytics to capture this opportunity and transform their businesses:

• Build better products: iSchoolConnect is an online platform that lets students explore schools, courses, and countries where they might study, and makes higher education admissions accessible to students around the globe. The company leverages AI services to help educational institutions optimize their academic operations by accelerating admission processing by greater than 90%, while saving significant costs.
• Launch in new markets faster: Classroom creativity tools provider Book Creator uses AI APIs to enhance accessibility and improve the learner experience. “The broad suite of intelligent APIs enables us to deliver richer experiences, faster and more easily, without having to be experts in machine learning, drawing recognition, map embeds, or other areas,” says VP of engineering Thom Leggett.
• Scale businesses securely: Using DevOps and CDN [content delivery network] services, Chrome browser recording extension creator Screencastify was able to support eight times growth in users overnight amid the COVID-19 pandemic, while maintaining consistent total cost of ownership. These technologies helped the company rapidly scale operations in response to the overnight increase in demand from consumers and assure student data privacy and security on a budget. “We know this is just the beginning, as more educators rely on technology to deliver richer, more interactive curricula to students,” says CEO James Francis. 
• Provide personalized learning and support: Smart analytics and AI can provide personalized support and recommendations for students, forecast demand, and predict shifts in learners’ preferences. Online learning platform Mindvalley uses cloud-based tools to understand and make decisions based on user activity and leverage machine learning to predict behavior. 

Google Cloud is partnering with many of these leading EdTech companies, as well as industry-leading consortiums like Ed-Fi and Unizin, to standardize educational common data models and best practices for more agile and cost-effective integration of EdTech into existing environments.

The education landscape is changing rapidly, and EdTech has a major role to play as institutions adapt to the massive shift in learners’ preferences and expectations. We’re committed to empowering EdTech companies with the tools and services they need to help expand learning for everyone, anywhere. 

Watch our Spotlight session with EdTechX to learn more.

If you’ve opened an account at a bank, applied for a government benefit, or provided a proof of age document on an ecommerce website, chances are you’ve had to share a physical or digital copy of a Driver’s License or a passport as proof of your identity. For businesses or public sector organizations that need this information to provide services, processing images of identity documents has long been a time- and resource-intensive process that requires extensive human intervention. Solutions exist to help digitally capture the data, but they require extensive human intervention that impacts the speed and cost of processing and ultimately the time to service customers.

The Google Cloud Document AI family of solutions has been designed to help solve some of the hardest problems for data capture at scale by extracting structured data from unstructured documents to help reduce processing costs and improve business speed and efficiency. Today, we’re announcing the general availability of identity parsers that bring the power of Document AI to customer verification, KYC, and other identity-based workflows.

With Document AI for Identity, businesses can leverage automation to extract information from identity documents with a high degree of accuracy, without having to bear the cost and turnaround time of manual tasks by a service provider. Document AI for Identity leverages artificial Intelligence to provide a set of pre-trained models that can parse identity and supports US driver’s licenses (generally available), US passports (generally available), French driver’s licenses (preview) and French National ID cards (preview), with more documents to be added from around the world over the coming months.

When our customers process high-volume workloads or complex workflows, they need a high degree of accuracy, since getting the first step wrong can derail the entire workflow. The introduction of special parsers for Identity processing can help solve one of the most commonly required document processing needs that our financial services and public sector customers face.

Along with the identity parsers, Google Cloud is also offering its “Human in the Loop” service, in which verification for a subset of identity documents can be automatically assigned to a pool of humans (internal or external) for manual review, based on confidence scores.

While there are multiple industries and applications that could benefit from Document AI for Identity, we’ve seen two main kinds of applications being adopted during the solution’s preview. One is around processing ID cards uploaded as unstructured images at scale, so that enterprises can have IDs on file. The second use case is to perform advanced checks on identity documents to validate their authenticity and / or to detect fraud. Google Cloud’s fraud detector API (which is currently in preview) can complement Document AI for Identity and apply an extra layer of normalization to help validate the identity as a government-issued ID by checking for suspicious words, image manipulation, and other common issues with forged identity documents. With new versions of driver’s licenses being frequently released, Document AI for identity uses specialized models and constantly-updated training data to help make sure the parsers can offer a high degree of accuracy. For all use cases, Document AI does not retain any customer data after completing the processing request (successfully or with an error).

Check out this demo and visit the Document AI for Identity landing page for more information on how Document AI can help solve your identity processing needs, and ask your Google Cloud account team to help you integrate Identity Document AI into your workflows.

For practitioners who’re interested in trying out Identity DocAI, check out our companion practitioner blog for step by step instructions on how to get started.