Leave the Tax Filing to ML: Lending Doc AI’s Automation Classifying and Parsing IT Documents

2992
Of your peers have already read this article.
3:00 Minutes
The most insightful time you'll spend today!
In the United States, Tax Season descends upon the country every April, requiring millions of Americans to spend hours deciphering cryptic documents and performing complex math just to figure out what they owe. Wouldn’t it be grand if there was a way for a computer to take all the relevant documents and extract out exactly what the IRS is looking for? Lending Document AI from Google Cloud supports common document types used for Income Tax Filing, such as W-2s and 1099s. These advancements in machine learning technology now makes it possible to alleviate some anxiety leading up to April 15th.
Lending Document AI is a Document Understanding solution that allows for classification and parsing of documents commonly used in the mortgage lending industry. The data in these unstructured files is then converted into a structured format, which can be stored in a database or used for analysis and calculations. You can read more about the product in the announcement blog post. For this tax filing use case, we will focus on automatically classifying and parsing the 2020 editions of the following forms:
W-2
1099-DIV
1099-INT
1099-MISC
1099-NEC
This sample application creates an automated pipeline where the user can bulk upload a collection of PDFs, the Lending Document Splitter & Classifier will classify each document and send each PDF to the appropriate specialized parser to extract the data, which can then be used to calculate an individual tax return and fill out a 1040 Form.
Overview
Let’s explore how this application works. You can check out the sample code in this GitHub Repository.
Here is an outline of the architecture of this application. As you can see, it utilizes Cloud Run and Firestore in Native Mode for the web application in addition to Document AI.

The User uploads multiple PDF files to the web application, hosted on Cloud Run.
An API call is made to the Lending Document Splitter & Classifier for each PDF file.
The output of the classifier (e.g. W-2, 1099-MISC, etc.) is then mapped to an appropriate specialized parser in the Google Cloud Project.
Each document file is sent to the appropriate specialized parser that matches the document type.
The entities are extracted by the parser processor and the data is written to Firestore.
The raw data is now retrieved from Firestore and displayed to the User showing the file classification and extracted values from each form.
The data values from all the forms are used together to calculate an individual income tax return.
The Calculated Tax Rates/Incomes/Deductions are displayed to the User in a Tabular Format matching the IRS Form 1040. The app also displays which form data was used for each field. (Some output fields use values from multiple forms, such as line 25b.)
Step-by-Step directions
Want to try this out for yourself? Here’s how you can deploy and run this application using a Google Cloud Project. You can run this in Cloud Shell (Quickstart) or on your local machine.
NOTE: The Lending Processors in this Demo are in Limited GA as of March 2022. If you have a business use case for these processors, you can fill out and submit the Document AI limited access customer request form.
Install dependencies
- Clone the GitHub Repository to get the sample code.
git clone https://github.com/GoogleCloudPlatform/document-ai-samples.git
- Enter the directory for the tax pipeline demo
cd document-ai-samples/tax-processing-pipeline-python
- Install Python and the Google Cloud SDK if they aren’t already installed.
- Install the python libraries:
pip install -r requirements.txt
- Create a new Google Cloud project, and enable billing if you don’t already have one.
- Enable the Document AI API:
gcloud services enable documentai.googleapis.com
- Setup application default credentials:
gcloud auth application-default login
Deploy demo application
- Edit the config.yaml file, adding your own Project Details docai_processor_location: us # Document AI Processor Location (us OR eu)
docai_project_id: YOUR_PROJECT_ID # Project ID for Document AI Processors
firestore:
collection: tax_demo_documents # Set with your preferred Firestore Collection Name
project_id: YOUR_PROJECT_ID # Project ID for Firestore Database - Run setup scripts to create the processors and Cloud Run app in your project. python3 setup.py
gcloud run deploy tax-demo –source . - Visit the Deployed Web Page (You should get a link from the deployment command)
Calculate Tax Returns Homepage

- Upload Documents. I created some sample documents you can download from the sample-docs folder of the repository.
This demo currently supports the following Document Types (2020 Editions)
W-2
1099-DIV
1099-INT
1099-MISC
1099-NEC
- Click “Upload” Button, wait for processing to complete.
The page will display the steps completed for each document file. These are also written to stdout for troubleshooting purposes.

6. View the extracted values from each file.

7. Click “Calculate Taxes” to see the tax calculation output

Conclusion
Warning: This is NOT financial advice, for educational purposes only.
Congratulations! You now have a fully functional tax processing application that can also be modified for use with other workflows that require data from multiple specialized documents.
The Document AI API is flexible and modular enough that most of the code in this example can be reused for any specialized processor.
Now tax returns can be filed with minimal manual effort!
If you want to learn more about Document AI, check out the Cloud Documentation and these videos:
- Getting started with the Document AI platform
- Process billions of pages and cut operational costs with DocAI
And if you want more hands-on experience, I recommend following these step-by-step codelabs to get started with the key features of Document AI:
Digital Maturity in Higher Ed Tied to Improvements in Students’ Journey: Study

5133
Of your peers have already read this article.
3:00 Minutes
The most insightful time you'll spend today!
Why Higher Ed Needs to Go All-in on Digital
In the wake of the COVID-19 pandemic, the majority of students within the 18-24-year-old demographic now expect hybrid learning environments–even once we are beyond the pandemic. And a vast number of adult learners are seeking options that accommodate their work and family lives now that it’s clear that effective learning can indeed occur virtually. Implementing cloud technologies and achieving digital maturity within higher education will enable institutions to be innovative and responsive to evolving student preferences, while being prepared for future disruptions.

The state of digital maturity
In February and March 2021, Boston Consulting Group (BCG), in partnership with Google, surveyed U.S. higher education leaders on their views of the state of digital maturity in the higher education sector. This survey found that institutional and technology leaders strongly agreed that moving legacy IT systems to the cloud, centralizing and integrating data, and increasing the use of advanced analytics is necessary to make a successful digital transformation, and ultimately achieve digital maturity.
But what is digital maturity? Digital maturity—a measure of an organization’s ability to create value through digital delivery—focuses on three areas of technological advancement that drive large-scale innovation:
- Using cloud infrastructure
- Expanding access to data
- Using that data to improve processes through advanced analytics, such as Artificial Intelligence and Machine Learning (AI/ML)
Although university leaders agree on prioritizing digital maturity, more than 55% said they considered their schools to be “digital performers” or “digital leaders.” However, only 25% of tech leaders at these universities stated that their schools regularly use data analytics. As with corporations and governments, higher education institutions face barriers to technological innovation, such as:
- Competing priorities to meet step-change goals and decentralized decision making
- Budget constraints
- Cultural resistance to change
- Tech staff skillset gaps
Still, leaders understand that the way to overcome institutional inertia is with a strong, goal-oriented vision of what is best for the institution overall. Although only a handful of schools have reached digital maturity as we define it, others can learn a great deal from their examples. Here are the top takeaways from higher education leaders who successfully transformed their institutions:
Digital solutions can improve the student journey in many ways

As digital capabilities hold the key to dealing effectively with declining enrollment and rising costs, higher ed leaders identified four goals that are critical to improving performance:
- Improve the student journey
- Increase operational efficiency
- Scale computing power in advanced research
- Innovate education delivery
The research found that technology investments can help enhance the student journey in the recruiting and retention of students, improving digital education delivery, government funding, and donations from alumni. Digital maturity can make institutions more agile and efficient in delivering education that aligns with the changing societal norms, evolving student preferences, and future disruptions. Survey participants shared that they plan to increase the use of the cloud by more than 50% over the next three years. By shifting legacy IT systems to the cloud, institutions can increase scalability, lower the cost of ownership, and improve operational agility, while offering a more secure, long-term data storage solution.
Cloud-native software-as-a-service (SaaS) solutions provide an excellent platform for centralizing data. However, institutions that attempt to “lift and shift” their legacy systems to the cloud may encounter challenges to achieving measurable improvements in data integration and cost reduction. Higher ed leaders must realize that centralizing data and transitioning to the cloud do not happen simultaneously.
Leaders who are able to articulate a strong vision and commitment will experience a more successful technology transformation. By linking their vision to specific needs, such as more effective recruiting, leaders will find their technology investments will have a more substantial return. University presidents should base their decisions about which systems to move, when, and how on desired performance outcomes.
Big visions become a reality with small steps. Small pilot projects are an excellent way to start the journey toward digital maturity. Small steps toward a significant transformation can reduce resistance to change, build positive momentum, and produce better student outcomes. Read the full report here. If you’d like to talk to a Google Cloud expert, get in touch.

3130
Of your peers have already downloaded this article
22:30 Minutes
The most insightful time you'll spend today!
Companies everywhere are seeking to leverage the power of AI. And rightly so. The smart applications of AI enable organizations to improve, to scale, and to accelerate the decision-making process across most business functions, so as to work both more efficiently and more effectively. It can also open up new avenues and new revenue streams, providing the organization with an additional competitive edge.
In short, many believe (as we do) that the enterprises that invest in building industry-specific AI solutions today are positioning themselves to be the global economic leaders of tomorrow. But the path to building an effective AI capability is not an easy one. There are many challenges to overcome. Challenges with the technology to develop platforms and solutions. With the people who will implement and manage that technology. With the data that fuels the technology. And with the processes that govern the whole of it. How do you harness the power inherent in AI, while avoiding any potential missteps?
That’s where Google Cloud comes in. Our framework for AI adoption provides a guide to technology leaders who want to build an effective AI capability, one that enables them to leverage the power of AI to enhance and streamline their business, smoothly and smartly. The framework is informed by Google’s own evolution, innovation, and leadership in AI, including experience deploying AI in production through products such as Gmail and Google Photos. It is also inspired by many years of experience helping cloud customers — from startups to enterprises, in various industries — to solve complex challenges.
With Google Cloud’s AI Adoption Framework, you’ll be able to create and evolve your own transformative AI capability. You’ll have a map for assessing where you are in the journey and where, at the end of it, you’d like to be. You’ll have a structure for building scalable AI capabilities to create better insights from big data with powerful algorithms across the entire business.
With Google Cloud as your guide, the path to AI is considerably smoother.
Download this whitepaper to find out:
- A map for assessing where you are in your AI journey and where you want to be
- A comprehensive structure for building an effective AI capability across your entire organisation to create actionable insights from data
- A technical deep dive for technology leaders
Smooth AI Adoption for Companies involves Three Principled Phases!

4088
Of your peers have already read this article.
4:00 Minutes
The most insightful time you'll spend today!
Business leaders see much in Artificial Intelligence, including new ways to save money, serve customers, and figure out what to build. What’s often tougher is deciding how to engage. That’s understandable, since getting into AI often raises questions about costs, data integrity, project length, and similar issues of planning and execution.
For CIOs concerned with the deployment and use of IT at their companies, these are critical issues. Here is one way to think about this complexity, by breaking it down into three different areas; Early Automation, Learning and working, and System Views.
First, a bit of good news and myth dispelling for readers who think AI is some far off promise, still in the labs and not safe for work. The reality is that AI is very much here, and almost everyone in your workplace is using it every day. People work with AI when they use Google Search, Photos, the autocomplete feature in Docs, live translation in Pixel phones, and many other areas. It is in the products of many other companies too.
“IDC reports that the market for AI software, hardware and services is expected to break the $500 billion mark in 2023.”
Additionally, it’s increasingly clear that AI can be adapted as a virtuous process, not just a one-off purchase (though there is much to recommend that.) During a recent Alphabet earnings call, CEO Sundar Pichai noted that “investments in AI will be key” to its near-term strategy, with new techniques that make it faster and easier to train and build AI for a number of uses. Additionally, the company is offering AI-driven “insights, new tools, and automation” to its advertising clients. The striking thing in this was the way that developing AI in one area could lead to growth in many others.
So, how does an IT leader foster a growth process like this for their stakeholders? By leading people through the well-established stages of Awareness, Learning, and Extension. Here’s what I mean.
Awareness: Early Automation
Consumer-facing AI is particularly strong in communications functions like voice recognition, translation, and writing tips. It’s similar in business uses: One of the most effective early instances of AI in the workplace has been Contact Center AI (CCAI), which manages basic customer communications, automatically answering common questions and prioritizing calls that require human assistance. It is doing what automation has always done best, automating the rote stuff and leaving the higher-value imaginative activity to people. It has been used by governments, retailers, telecommunications companies, and others, in a wide variety of use cases.
These and similar language-centric products, like DocAI for extracting information from things like invoices, receipts, or AI that extracts information from business contracts, have a number of benefits. For one, the investment is relatively easy to control, unlike with a research project leading to a formal launch. The payoff is also clearer. In the case of contact centers, in particular, the automation relieves stress, wins loyalty and slows disaffection in a high-turnover area. In both cases, successful results build allies in the business, who can testify to the earlier benefits when it’s time to take on something more complex.
Perhaps best of all, creating interest in basic AI services for business, right now, means people become engaged in learning more, since they see the early benefits and wonder what else might be done.
Learning: The Human Factor
The Natural Language Processing (NLP) that goes into these ready made, “out of the box” AI products can, not surprisingly, be used on much more sophisticated levels. Twitter, for example, processes 400 billion different events in real time, and its staff queries this trove using advanced NLP, answering questions and improving customer experiences.
There is clearly an enormous gap between Call Center AI and processing Twitter’s 400 billion events per day, but it’s not noticed enough how quickly that gap is closing. Look at how many products, partners, and training resources have emerged in the past few years. The gap makes sense, insofar as both the means of AI (like large data sets, good algorithms, and sufficient computing) and the value of AI, are new.
Increasingly, as AI is incorporated into standard enterprise tools like spreadsheets and analytic tools, easier to use AI becomes a skill within reach for many (even as the advanced end becomes more complex, meaning this ease of use process will continue for some time.)
It’s so new that AI skills demand isn’t met by conventional education means, creating lots of good opportunities for both nonstandard skills training, and in-house learning in the workforce. Companies offering AI skills training could well gain a competitive advantage and retain staff better.
Extension: Building System Views
When a new technology lands and gains in popularity, people seek to find new uses for it, or build connections among its different uses. Networked computing is one example, but think also of the way cars were soon followed by trucks and fire engines, or the way the data services on wireless phones soon morphed into the App Economy. If something is useful, people look for ways to grow it.
How will AI grow? My colleague Dominik Wee recently wrote about ways that AI will soon change supply chains, change product design, and improve sustainability. Most interestingly, he talked about how customers in manufacturing were realizing savings and gaining insights when once separate quality control data was combined with system wide views of the quality process.
There are several reasons to think AI will promote many such system views. For one thing, successful AI promotes the collection of data from more places, at greater frequency, since that leads to insight (and the cost of data collection is dropping.) Additionally, AI is good at spotting patterns and interactions that are not currently known. As well, AI is used in prediction and scenario planning, which leads to better understanding of how large-scale systems interact.
This comes at a time when we have more ways of seeing the world, from satellites, sensors, social media, and much more. We have more awareness of interactions, and a demand to understand them, in everything from the supply chain crisis, to human rights and sourcing regulations, or in the business realities of partnering, and serving customers in all sorts of ways, online and in the physical world.
Whether by coincidence or design, the Age of AI is also an age when organizations see themselves more accurately with a rich web of connections, with their choices and actions having more resonance than ever. That awareness is both a competitive tool, and a call to greater responsibility, potentially affording more customer loyalty and a more satisfied workforce for those who get it right.
That transformation won’t happen everywhere overnight, but it seems to be happening at all sorts of companies. And the trend for AI to assist, to be studied and grown, and to provide a richer understanding of the world, is happening every time someone touches this technology, at whatever level they need.
3243
Of your peers have already watched this video.
38:09 Minutes
The most insightful time you'll spend today!
TensorFlow: The Show-and-Tell Data Scientists Have Been Asking For
TensorFlow is among the most popular, if not, the most popular deep learning libraries today. According to one ranking, “TensorFlow is at least two standard deviations above the mean on all calculated metrics.”
Watch as Lak Lakshmanan, Technical Lead, Machine Learning and Big Data, Google Cloud, walks through a development workflow that will make operationalization easier to execute, including the process of building a complete machine learning pipeline covering ingest, exploration, training, evaluation, deployment, and prediction.
He also talks about the need for distributed training. But what’s the benefit of distributed TensorFlow? Many machine learning frameworks can only handle “toy problems”, or problems that can be solved by input data that fits into memory. These are small data sets.
But to build effective machine learning you need big data, feature engineering, and model architectures. With large amounts of data batching and distribution are very important. That’s where distributed training comes in.
6449
Of your peers have already watched this video.
1:30 Minutes
The most insightful time you'll spend today!
Value Realization with Google Cloud for Retail SAP Data
Retail engagements have changed drastically over the last few years, and by the virtue of COVID-19 pandemic, retail data and customers’ expectations plummeted. To drive value and transformation across the entire value-chain retailers can make most of Google Cloud’s secure, reliable IaaS by migrating their SAP systems and taking advantage of integrations, insights and innovations. In times of change retail companies can gain maximum visibility of SAP data unlocking Google Cloud’s infrastructure modernization and Big Data and analytics capabilities. Watch the video to understand how Google Cloud and SAP partnership is a golden handshake for retail businesses’ future.
More Relevant Stories for Your Company

How adidas’s Marketing Team Benefits from Google Cloud Marketing Platform
We live in a world where the consumer journey is no longer linear, making it difficult to know when to deliver a brand message or a product message. “We talk a lot about the consumer journey no longer being linear. It’s very difficult to know when the consumer is going
New Technology: The Projected Total Economic Impact™ Of Google Cloud Contact Center AI
According to Forrester Research, customers expect easy and effective customer service that builds positive emotional connections every time they interact with a brand or organization. Additionally, 40% of surveyed business leaders say that improving their organization’s customer experience (CX) is a high priority, ahead of initiatives like improving products and

AL/ML and Data Products Delivered through Google Cloud Makes them Leader of Gartner 2022 Magic Quadrant
Gartner® named Google as a Leader in the 2022 Magic Quadrant™ for Cloud AI Developer Services report. This evaluation covered Google’s language, vision and structured data products including AutoML, all of which we deliver through Google Cloud. We believe this recognition is a reflection of the confidence and satisfaction that

An Introduction to MLOps on Google Cloud
The enterprise machine learning life cycle is expanding as firms increasingly look to automate their production ML systems. MLOps is an ML engineering culture and practice that aims at unifying ML system development and ML system operation enabling shorter development cycles, increased deployment velocity, and more dependable releases in close







