Investing in Artificial Intelligence (AI) can bring a competitive advantage to your organization. If you’re in charge of an AI or Data Science team, you’ll want to measure and maximize the value that you’re providing. Here is some advice from our years of experience in the field. 

A checklist to embark on a project: 

As you embark on projects we’ve found it’s good to have the following areas covered: 

• Have a customer. It’s important to have a customer for your work, and that they  agree with what you’re trying to achieve. Be sure to know what value you’re delivering to them. 
• Have a business case.  This will rely on estimates and assumptions, and may take no more than a few minute’s work.  You should revise this, but always know what justifies your team’s effort, and what you (and your customer) expect to get in return. 
• Know what process you will change or create. You’ll want to put your work in production, so you have to be clear about what business operations are changing or created around your work and who needs to be involved to make it happen
• Have a measurement plan. You’ll want to show that ongoing work is impacting some relevant business indicator. Measure and show incremental value. The goal of these measurements is to establish what has changed because of your project that would otherwise not have changed. Be sure to account for other factors like seasonality or other business changes that may affect your measurements.
• Use all the above to get your organization’s support for your team and your work. 

What measures to use?

As you start the work, what measures and indicators can you use to show that your team’s work is useful for your organization?

How many decisions you make. A major function of ML is to automate and optimize decisions: which product to recommend, which route to follow, etc. Use logs to track how many decisions your systems are making. 

Changes to revenue or costs. Better and quicker decisions often lead to increased revenue or savings. If possible, measure it directly, otherwise estimate it (for example fuel costs saved from less distance traveled, or increased purchases from personalized offers). 

As an example, the Illinois Department of Employment Security is using Contact Center AI to rapidly deploy virtual agents to help more than 1 million citizens file unemployment claims. To measure success the team tracked the two outcomes:  (1) the number of web inquiries and voice calls they were able to handle, and (2) the overall cost of the call center after the implementation. Post implementation, they were able to observe more than 140,000 phone and web inquiries per day and over 40,000 after-hours calls per night. They  also anticipate an estimated annual cost savings of $100M based on an initial analysis of IDES’s virtual agent data (see more in the link to case study).

Implementation costs. The other side of increased revenue or savings, is to put your achievements in the context of how much they cost. Show the technology costs that your team incurs and, ideally, how you can deliver more value, more efficiently. 

How much time was saved.  If the team built a routing system then it saved travel time, if it built an email classifier then it saved reading time, etc. Quantify how many hours were given back to the organization thanks to the efficiency of your system. 

In the medical field, quicker diagnostics matter. Johns Hopkins University’s Brain Injury Outcomes (BIOS) Division has focused on studying brain hemorrhage aiming to improve medical outcomes. The team identified the time to insights as a key metric in measuring business success. They experimented with a range of cloud computing solutions like Dataflow, Cloud Healthcare API, Compute Engine, and AI Platform for distributed training to accelerate iterations. As a result, in their recent work they were able to accelerate insights from scans from approximately 500 patients from 2,500 hours to 90 minutes.

How many applications your team supports. Some of your organization’s operations don’t use ML (say reconciling financial ledgers) but others do. Know how many parts of your organization benefit from the optimization and automation your team builds.

User experience. You may be able to measure your customer’s experience: fewer complaints, better reviews, reduced latency, more interactions, etc. This is valid both for internal and external stakeholders. At Google we measure usage and regularly ask for feedback on any internal system or process.

One of our customers, The City of Memphis, is using VisionAI and ML to tackle a common but very challenging issue: identifying and addressing potholes.  The implementation team identified the percentage increase of potholes identified as one of the key metrics along with accuracy and cost savings. The solution captures video footage from it’s public vehicles and leverages Google Cloud capabilities like Compute Engine, AI Platform, and BigQuery to automate the review of videos.  The project increased  pothole detection by 75% with over 90% accuracy. By measuring and demonstrating these outcomes, the team proved the viability of a cost-effective, cloud-based machine learning model and is looking into new applications of AI and ML that will further improve city services and help it build a better future for its 652,000 residents. 

Acknowledgements

Filipe and Payam would like to thank our colleague and co-author Mona Mona (AI/ML Customer Engineer, Healthcare and lifesciences) who contributed equally to the writing.

Financial markets were among the first to adopt new technologies, and that has certainly been true of the derivatives markets, which were early adopters of electronic trading. Going forward, new capabilities will transform the way industry participants communicate, analyze, and trade.

I sat down with Google Cloud’s Phil Moyer and former SEC Commissioner, Troy Paredes, for a fireside chat at FIA Boca 2022 to discuss the future of markets and policy, the new technologies that are already paving the way for greater speed and transparency, and how cloud can help promote greater resiliency, performance, and security to enable the long-term vision for the market. The following is a summary of our discussion.

The current state of cloud technology

When it comes to technology adoption, we’re seeing the market and participants adopt cloud technologies, and increasingly, machine learning (ML) on a wider scale. Cloud technology allows for easier, faster, and much more secure experimentation with large datasets and ML.

A recent Google sponsored study by Coalition Greenwich (September, 2021) showed that more than 93% of trading systems, exchanges, and data providers are in some way providing services on the cloud. The same study, revealed that about 72% of the financial industry across the buy side and sell side, intend to consume public cloud-data based market data within the next 12 months.

Data-driven decision-making and risk management have always been, and continue to remain, the cornerstones of the financial markets. Over time, technology innovation has facilitated access to better insights from data, and therefore, better decision-making and the ability to manage risk. That expectation is now mainstream, and will continue to grow in sophistication.

The multi-phased technology trajectory

The movement of exchanges to the cloud will occur in a “crawl-walk-run” fashion, with low-hanging fruits the first to be picked in the near term while bigger, paradigmatic changes will occur over the medium and long term. Some organizations are starting all three stages simultaneously, understanding that each will move at an independent cadence.

The “crawl” phase is one in which foundations are built, starting with organizations moving data to the cloud and experimenting with some degree of analytics. It’s one of the most important phases because it’s where the opportunity to increase transparency and risk management takes shape.

In moving to the cloud, the infrastructure – which in the past relied on a combination of people, processes, and some technology – becomes the code that runs applications. This early phase is key to empowering organizations to shift to a cloud-based, agile-first operating model that makes it easier and more seamless to launch new products in the future, including by freeing up people and resources from IT management to more mission-focused work.

Establishing the cloud operating model simplifies the “walk” and “run” phases where compliance is more automated, latency-sensitive applications are more readily available, and the next generation of exchanges, market participants, and regulators is better prepared to meet future challenges.

The “walk” phase is where much of the innovation happens. Exchanges are making significant progress in leveraging foundational data decisions in the “crawl” phase and innovations in the cloud to improve settlement, clearing, risk management, collateral management, and compliance, and launch new products.

And finally, the “run” phase is where organizations will start to move the latency-sensitive markets to the cloud, as the markets increasingly will demand low-latency and high performance along with transparency and analytics to solve historical obstacles to market access.

Opportunities for both regulators and market participants

Any time significant technological change takes place, regulators explore its implications, particularly with respect to their ability to meet their regulatory objectives.

Increasingly, we are seeing technological change driving more opportunities for regulators and market participants alike. Such changes may also allow better protection of the marketplace, with greater integrity and transparency.

Over time, regulatory regimes – rules, regulations, statutes, interpretations, and guidance – will also adjust to new technologies, both benefiting the marketplace and advancing regulatory goals.

As one example, the cloud is increasing the ability to meet compliance obligations by allowing compliance to be built into transactions. Moreover, predicated on the vision of real-time regulatory reporting, and given the pace of technological change in the marketplace over the last several years, various regulators have been using more advanced analytics. This trend will continue to help them more effectively and efficiently meet their objectives, and monitor and meet the expectations they have for the entire market.

Machine learning’s role in the financial markets

Google Cloud’s head of AI and Industry Solutions, Andrew Moore, said that ML will be doing three key things for us in the next 10 years: giving us meaning, providing concierge services, and serving as a guardian. Extracting information that is critical to investor decision-making can be extremely important. With more data than ever, ML can increase the ability to process it while also becoming more accessible in the cloud and better supporting regulatory objectives.

The technology will likely manifest in trading and anti-money laundering activities as they relate market functions, as well as managing a wide variety of risks – supporting the interests of both investors and regulators in terms of decision-making, surveillance, and protections.

Rather than taking individuals out of the equation, the digitization of markets, assets, and guard rails combined with ML will allow people to focus their expertise in different ways to achieve key objectives.

Building the market foundation for the future

The goals of operational resiliency, security, and privacy will continue to be critical for building the market foundation for both participants and regulators. While technology promises to create advantages in concrete, tangible ways, it will be important to scrutinize potential risks and concerns.

Priority one for technology providers is to build an environment of trustless security, including encryption at motion and encryption at rest, ensuring that markets are operationally resilient while instilling confidence for any exchange that runs on top of that infrastructure. Multicloud architectures and approaches are likely also to be part of the solution for operational resilience.

Throughout time, liquidity has been the outcome of improved access, transparency, and security. Technology providers are responding by sharing both the responsibility for, and fate of, the markets of the future to build an efficient, faster, and more transparent and secure financial industry.

You can learn more about our approach in our newest white paper, Building the financial markets foundation for the future.

In the United States, Tax Season descends upon the country every April, requiring millions of Americans to spend hours deciphering cryptic documents and performing complex math just to figure out what they owe. Wouldn’t it be grand if there was a way for a computer to take all the relevant documents and extract out exactly what the IRS is looking for? Lending Document AI from Google Cloud supports common document types used for Income Tax Filing, such as W-2s and 1099s. These advancements in machine learning technology now makes it possible to alleviate some anxiety leading up to April 15th.

Lending Document AI is a Document Understanding solution that allows for classification and parsing of documents commonly used in the mortgage lending industry. The data in these unstructured files is then converted into a structured format, which can be stored in a database or used for analysis and calculations. You can read more about the product in the announcement blog post. For this tax filing use case, we will focus on automatically classifying and parsing the 2020 editions of the following forms:

W-2

1099-DIV

1099-INT

1099-MISC

1099-NEC

This sample application creates an automated pipeline where the user can bulk upload a collection of PDFs, the Lending Document Splitter & Classifier will classify each document and send each PDF to the appropriate specialized parser to extract the data, which can then be used to calculate an individual tax return and fill out a 1040 Form.

Overview

Let’s explore how this application works. You can check out the sample code in this GitHub Repository.

Here is an outline of the architecture of this application. As you can see, it utilizes Cloud Run and Firestore in Native Mode for the web application in addition to Document AI.

The User uploads multiple PDF files to the web application, hosted on Cloud Run.

An API call is made to the Lending Document Splitter & Classifier for each PDF file.

The output of the classifier (e.g. W-2, 1099-MISC, etc.) is then mapped to an appropriate specialized parser in the Google Cloud Project.

Each document file is sent to the appropriate specialized parser that matches the document type.

The entities are extracted by the parser processor and the data is written to Firestore.

The raw data is now retrieved from Firestore and displayed to the User showing the file classification and extracted values from each form.

The data values from all the forms are used together to calculate an individual income tax return.

The Calculated Tax Rates/Incomes/Deductions are displayed to the User in a Tabular Format matching the IRS Form 1040. The app also displays which form data was used for each field. (Some output fields use values from multiple forms, such as line 25b.)

Step-by-Step directions

Want to try this out for yourself? Here’s how you can deploy and run this application using a Google Cloud Project. You can run this in Cloud Shell (Quickstart) or on your local machine.

NOTE: The Lending Processors in this Demo are in Limited GA as of March 2022. If you have a business use case for these processors, you can fill out and submit the Document AI limited access customer request form.

Install dependencies

1. Clone the GitHub Repository to get the sample code.

git clone https://github.com/GoogleCloudPlatform/document-ai-samples.git

2. Enter the directory for the tax pipeline demo

cd document-ai-samples/tax-processing-pipeline-python

3. Install Python and the Google Cloud SDK if they aren’t already installed.
4. Install the python libraries:

pip install -r requirements.txt

5. Create a new Google Cloud project, and enable billing if you don’t already have one.
6. Enable the Document AI API:

gcloud services enable documentai.googleapis.com

7. Setup application default credentials:
   gcloud auth application-default login

Deploy demo application

1. Edit the config.yaml file, adding your own Project Details docai_processor_location: us # Document AI Processor Location (us OR eu)
   docai_project_id: YOUR_PROJECT_ID # Project ID for Document AI Processors
   firestore:
   collection: tax_demo_documents # Set with your preferred Firestore Collection Name
   project_id: YOUR_PROJECT_ID # Project ID for Firestore Database
2. Run setup scripts to create the processors and Cloud Run app in your project. python3 setup.py
   gcloud run deploy tax-demo –source .
3. Visit the Deployed Web Page (You should get a link from the deployment command)

Calculate Tax Returns Homepage

4. Upload Documents. I created some sample documents you can download from the sample-docs folder of the repository.

This demo currently supports the following Document Types (2020 Editions)

W-2
1099-DIV
1099-INT
1099-MISC
1099-NEC

5. Click “Upload” Button, wait for processing to complete.
   The page will display the steps completed for each document file. These are also written to stdout for troubleshooting purposes.

6. View the extracted values from each file.

7. Click “Calculate Taxes” to see the tax calculation output

Conclusion

Warning: This is NOT financial advice, for educational purposes only.

Congratulations! You now have a fully functional tax processing application that can also be modified for use with other workflows that require data from multiple specialized documents.

The Document AI API is flexible and modular enough that most of the code in this example can be reused for any specialized processor.

Now tax returns can be filed with minimal manual effort!

If you want to learn more about Document AI, check out the Cloud Documentation and these videos:

• Getting started with the Document AI platform
• Process billions of pages and cut operational costs with DocAI

And if you want more hands-on experience, I recommend following these step-by-step codelabs to get started with the key features of Document AI:

• Optical Character Recognition (OCR) with Document AI (Python)
• Form Parsing with Document AI (Python)
• Specialized Processors with Document AI (Python)
• Managing Document AI processors with Python