How a Mid-Sized Firm is Shrinking Inventory Carryovers 50% with AI
7465
Of your peers have already read this article.
5:30 Minutes
The most insightful time you'll spend today!
For more than 10 years, NMK Textile Mills has manufactured bed linens for major retailers in the United States and Canada. As e-commerce exploded, the company’s co-founder saw an opportunity to grow the business. So, in addition to manufacturing bed linens wholesale for retail customers, NMK Textile Mills reworked its complete supply chain to manufacture products for California Design Den, which became an e-commerce retailer selling its own fashion-forward products directly to consumers online.
With California Design Den’s push into e-commerce, it became apparent the SMB company (with about 250 global employees) faced the same tough supply chain questions as large retail customers, including maintaining enough inventory to meet customer demand in a timely, efficient way.
California Design Den depended upon a myriad of systems to track its complex forecasting and reordering processes. Team members typically planned inventory manually using desktop spreadsheet software, which could lead to excess inventory. Accurately forecasting demand and supply was essential to the company’s financial success—but it was also a challenge.
A couple years ago, California Design Den partnered with Pluto7, a technology solutions provider that offers a software as a service (SaaS) called Planning In A Box. Leveraging Google Cloud Platform machine learning and artificial intelligence, Planning In A Box intelligently helps predict demand and balances it with supply.
But that was just the beginning. “Along the way, we realized that to compete with larger retailers, make quicker decisions, and move faster, we needed to go further,” says Deepak Mehrotra, Co-founder and Chief Adventurer at California Design Den.
With guidance from Pluto7, California Design Den began migrating its database to Google Cloud Platform. Using Google BigQuery, Google Compute Engine, Google Cloud SQL, and Google Cloud Storage, and experimenting with Google Cloud Vision and Google Cloud AutoML, the company is reducing inventory carryovers by more than 50%, improving the accuracy of demand planning quarter over quarter, and gaining granular insights into how individual SKUs are performing.
“We would need an army of data scientists to make faster decisions on pricing and inventory levels. With Google Cloud Platform machine learning and artificial intelligence, we don’t need that. We can make much faster pricing decisions to optimize profitability and move inventory.”
—Deepak Mehrotra, Co-founder and Chief Adventurer, California Design Den
No need for army of data scientists
“Using Google Cloud Platform machine learning and AI was essential for California Design Den if it was to compete successfully with larger retailers,” Deepak says.
For example, tastes and fashions in bed linens can change quickly and consumer prices fluctuate. With more than 2,500 SKUs, it wasn’t possible for California Design Den’s team to continuously monitor product demand and experiment with competitive pricing in real time.
“We would need an army of data scientists to make faster decisions on pricing and inventory levels,” says Deepak. “With Google Cloud Platform machine learning and artificial intelligence, we don’t need that. We can make much faster pricing decisions to optimize profitability and move inventory.”
By integrating all its data onto Google Cloud Platform, California Design Den’s team gains deeper insights into product sales over time, which in turn helps the company improve demand planning by better determining which styles to manufacture and sell in the future.
“When experienced employees leave, their knowledge leaves with them. By having all data in one place, and with machine learning and AI, California Design Den can go back in its history, look at products made or sold years ago, and analyze product performance.”
—Deepak Mehrotra, Co-founder and Chief Adventurer, California Design Den
Merging visuals with data
Before Google Cloud Platform, team members had to dig through spreadsheets and run scenarios to get a sense of how particular products had sold. The next step was to perform keyword searches across the company’s photo library in the cloud to find each product’s image. From there, a team member would insert the product images into a presentation, along with relevant data points, to provide a report for stakeholders on how particular styles performed.
Today, California Design Den, with the help of Pluto7, is integrating its entire product image library with its database on Google Cloud Platform. Experimenting with Google Cloud Vision and Google Cloud AutoML, California Design Den is moving towards a day when team members can run sales scenarios and get deep background data on individual product performance while viewing images of the relevant products.
Merging product visuals with data will help designers and team members better understand sales patterns over time and in context. In the past, making correlations between things like which sheet colors sold well in California, compared to how the same sheet colors performed on the East Coast, was something that California Design Den employees primarily did in their heads.
“When experienced employees leave, their knowledge goes with them,” says Manjunath Devadas, Founder and CEO at Pluto7. “By having all data in one place, and with machine learning and AI, California Design Den can go back in its history, look at products made or sold years ago, and analyze product performance.”
“We are literally growing the complexity of our business on all levels, including designing, manufacturing, selling, reordering, inventory holding—everything.”
—Deepak Mehrotra, Co-founder and Chief Adventurer, California Design Den
Reimagining supply-demand balancing
Pluto7’s mission statement is to democratize supply demand balancing with machine learning and AI. California Design Den is a case in point, as the combination of Planning In A Box and Google Cloud Platform gives the company greater control over its destiny.
“Big retailers used to tell us what to manufacture and how much they would pay for it,” says Deepak. “That was our primary business, and if we didn’t accept the terms, a competitor would.” Today, in addition to continuing to make products for retailers, California Design Den can design, make, and sell a variety of designs for itself, including custom and limited-edition products, thanks to Google Cloud Platform and Pluto7 software offerings. The payoff is not only in having a more diversified business. California Design Den also receives more favorable profit margins by selling its own products.
“We are literally growing the complexity of our business on all levels, including designing, manufacturing, selling, reordering, inventory holding—everything,” Deepak says. “We can make smaller batches. We can connect directly with consumers. We can identify the missing pieces—what should we produce next, when, and how much? We otherwise couldn’t afford the level of talent it would take to do this.”
Cutting through the noise
Google and Pluto7 software helped California Design Den reduce inventory carryovers by more than 50%. Inventory tracking and distribution, along with insights and visibility into product sales, are faster, more efficient, and accurate. Google Cloud Platform flexible pricing, speed, reliability, security, and scalability enable California Design Den to stay relevant and be more competitive.
In addition to benefiting from Google Cloud Platform, California Design Den relies on G Suite—also part of Google Cloud—to enhance collaboration among its global teams. Previously, the company’s email server would sometimes crash, due to the heavy load of sharing product photos and other data. “Gmail and Google Drive handle the everyday demands on the business effortlessly and reliably,” Deepak says.
“Google machine learning and AI enable us to cut through all the noise from raw data, so we can see what’s important. We can focus on analytics to guide us to success today and in the future.”
—Deepak Mehrotra, Co-founder and Chief Adventurer, California Design Den
The company is exploring additional ways to leverage Google Cloud Platform in the near future. For example, one possibility is to import customer reviews from sites where products are sold into Google BigQuery, and to use that data to perform sentiment analysis via Google Cloud Natural Language. It could provide another valuable data source to help California Design Den’s team decide where to focus future designs.
“Google machine learning and AI enable us to cut through all the noise from raw data, so we can see what’s important,” Deepak says. “We can focus on analytics to guide us to success today and in the future.”
3141
Of your peers have already watched this video.
21:30 Minutes
The most insightful time you'll spend today!
Document AI
Most business transactions begin, involve, or end with a document. But working with documents can be tricky, as leaders across industries seeking digital transformation can attest to.
These enterprises face similar challenges as they seek to extract information from documents. The process can be costly, time consuming, and prone to errors with manual data entry.
Learn how to use machine learning to organize, process, and extract data within documents. Also, learn about some examples of how various customers have found success using Google Cloud Document AI.
In this video Sudheera Vanguri, Product Manager, Google Cloud AI, highlights new Document AI capabilities. She walks you through of the building blocks of Document AI and demonstrates the new UI. She also highlights specialized Document AI models pre-trained for invoice and healthcare document processing as well as shows customer examples and live demos.
How AI and ML Helps Interpret Baseball Fandom during this MLB Season
4558
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
The game of baseball has no shortage of statistics — from batting average to exit velocity, strikeouts to wins above replacement. Among all sports, Major League Baseball (MLB) arguably contains the most analytical and data-driven participants and fan base. Subconsciously or viscerally, players and managers on the field and those following from anywhere are constantly assessing and making decisions based off of game play trends and expectations — whether a batter will come through with a hit in an important situation, when a pitcher should be pulled. Less analyzed, however, is what leads fans to become engaged with certain players or teams, and what factors drive their love of the game. This is the motivation behind the problem being posed by Major League Baseball in their Kaggle competition for Player Digital Engagement Forecasting. Can you use machine learning to deconstruct baseball fandom?
This competition asks you to predict measures of digital engagement for each active player on a daily basis during the MLB season. So, how large was the surge in fan interest after Joe Musgrove threw the first no-hitter in Padres history? Is Shohei Ohtani’s engagement higher when he pitches well, when he hits a monster home run…or when he does both? You’re provided a wealth of game, team and player information – detailed stats, awards, rosters, and transaction information – as well as social and digital engagement data as your inputs. Data scientists will recognize this as an exciting forecasting problem with both traditional regression and time series components, where having this input data just prior to the prediction date is critical to determining which players will receive the most engagement.
With so many variables in the game, there are an endless number of vectors which could possibly influence fan engagement. Eleven-time All-Star Miguel Cabrera delighted fans by hitting the first home run of the season – in the snow! Occasionally a lesser-known player like Musgrove or Carlos Rodón “wins the day” with an unlikely no-hitter. And sometimes just getting traded to an iconic franchise like the Yankees generates a ton of fan interest, like it did for Rougned Odor in early April.
As these examples show, a player’s digital engagement can be pretty dynamic during the season, with many different potential contributors to who is “trending” on a given day. How can you use data to uncover which factors are the most influential of engagement with each player’s digital content?
Ready to play ball? Check out the competition on Kaggle for all the details. $50,000 in prizes is up for grabs in two prize categories. The code competition puts your machine learning skills to the test, to see who can build the most accurate forecasting models to predict daily digital engagement for every active player. You’ll have until July 31st to build your models and then be evaluated on a future time frame, which will determine the winners. For data visualization and exploration experts out there, the explainability prizes give you an opportunity to analyze more broadly which factors, even those outside of what we’re providing directly, most influence digital engagement. You’ll be evaluated on how well you can use what the data is telling you to support your findings.
And if you’re looking to get started, we’ve provided an introductory video and some notebook tutorials, including a starting point for harnessing the power of Vertex AI through tools including Cloud Notebooks, Explainable AI, and Vizier.
With the second half of the season upon us, it’s an exciting time to be an MLB fan. With this Kaggle competition, it’s also a perfect opportunity to use data science to help understand baseball fandom and potentially earn some of your own accolades in the process. Step up to the plate!
Major League Baseball trademarks and copyrights are used with permission of Major League Baseball. Visit MLB.com.
2617
Of your peers have already downloaded this article
15:00 Minutes
The most insightful time you'll spend today!
Across industries, DevOps and DataOps have been widely adopted as methodologies to improve quality and reduce the time to market of software engineering and data engineering initiatives.
With the rapid growth in machine learning (ML) systems, similar approaches need to be developed in the context of ML engineering, which handle the unique complexities of the practical applications of ML. This is the domain of MLOps.
MLOps is a set of standardized processes and technology capabilities for building, deploying, and operationalizing ML systems rapidly and reliably.
Inside, you will find an outline of an MLOps framework that defines core processes and technical capabilities. Organizations can use this framework to help establish mature MLOps practices for building and operationalizing ML systems.
Adopting the framework can help organizations improve collaboration between teams, improve the reliability and scalability of ML systems, and shorten development cycle times. These benefits in turn drive innovation and help gain overall business value from investments in ML.
Google Cloud Next 21 for Data Analytics Unplugged
4961
Of your peers have already read this article.
4:30 Minutes
The most insightful time you'll spend today!
October 23rd (this past Saturday!) was my 4th Googlevarsery and we are wrapping an incredible Google Next 2021!
When I started in 2017, we had a dream of making BigQuery Intelligent Data Warehouse that would power every organization’s data driven digital transformation.
This year at Next, It was amazing to see Google Cloud’s CEO, Thomas Kurian, kick off his keynote with CTO of WalMart, Suresh Kumar , talking about how his organization is giving its data the “BigQuery treatment”.
AS I recap Next 2021 and reflect on our amazing journey over the past 4 years, I’m so proud of the opportunity I’ve had to work with some of the world’s most innovative companies from Twitter to Walmart to Home Depot, Snap, Paypal and many others.
So much of what we announced at Next is the result of years of hard work, persistence and commitment to delivering the best analytics experience for customers.
I believe that one of the reasons why customers choose Google for data is because we have shown a strong alignment between our strategy and theirs and because we’ve been relentlessly delivering innovation at the speed they require.
Unified Smart Analytics Platform
Over the past 4 years our focus has been to build industries leading unified smart analytics platforms. BigQuery is at the heart of this vision and seamlessly integrates with all our other services. Customers can use BigQuery to query data in BigQuery Storage, Google Cloud Storage, AWS S3, Azure Blobstore, various databases like BigTable, Spanner, Cloud SQL etc. They can also use any engine like Spark, Dataflow, Vertex AI with BigQuery. BigQuery automatically syncs all its metadata with Data Catalog and users can then run a Data Loss Prevention service to identify sensitive data and tag it. These tags can then be used to create access policies.
In addition to Google services, all our partner products also integrate with BigQuery seamlessly. Some of the key partners highlighted at Next 21 included Data Ingestion (Fivetran, Informatica & Confluent), Data preparation (Trifacta, DBT), Data Governance (Colibra), Data Science (Databricks, Dataiku) and BI (Tableau, PowerBI, Qlik etc).
Planet Scale analytics with BigQuery
BigQuery is an amazing platform and over the past 11 years we have continued to innovate in various aspects. Scalability has always been a huge differentiator for BigQuery. BigQuery has many customers with more than 100 petabytes of data and our largest customer is now approaching an exabyte of data. Our large customers have run queries over trillions of rows.
But scale for us is not just about storing or processing a lot of data. Scale is also how we can reach every organization in the world. This is the reason we launched BigQuery Sandbox which enables organizations to get started with BigQuery without a credit card. This has enabled us to reach tens of thousands of customers. Additionally to make it easy to get started with BigQuery we have built integrations with various Google tools like Firebase, Google Ads, Google Analytics 360, etc.
Finally, to simplify adoption we now provide options for customers to choose whether they would like to pay per query, buy flat rate subscriptions or buy per second capacity. With our autoscaling capabilities we can provide customers best value by mixing flat rate subscription discounts with auto scaling with flex slots.
Intelligent Data Warehouse to empower every data analyst to become a data scientist
BigQuery ML is one of the biggest innovations that we have brought to market over the past few years. Our vision is to make every data analyst a data scientist by democratizing Machine learning. 80% of time is spent in moving, prepping and transforming data for the ML platform. This also causes a huge data governance problem as now every data scientist has a copy of your most valuable data. Our approach was very simple. We asked:”what if we could bring ML to data rather than taking data to an ML engine?”
That is how BigQuery ML was born. Simply write 2 lines of SQL code and create ML models.
Over the past 4 years we have launched many models like regression, matrix factorization, anomaly detection, time series, XGboost, DNN etc. These models are used by customers to solve complex business problems simply from segmentation, recommendations, time series forecasting, package delivery estimation etc. The service is very popular: 80%+ of our top customers are using BigQueryML today. When you consider that the average adoption rate of ML/AI is in the low 30%, 80% is a pretty good result!
We announced tighter integration of BQML with Vertex AI. Model explainability will provide the ability to explain the results of predictive ML classification and regression models by understanding how each feature contributes to the predicted result. Also users will be able to manage, compare and deploy BigQuery ML models in Vertex; leverage Vertex Pipelines to train and predict BigQuery ML models.
Real-time streaming analytics with BigQuery
Customer expectations are changing and everyone wants everything in an instant: according to Gartner, by the end of 2024, 75% of enterprises will shift from piloting to operationalizing AI, driving a 5X increase in streaming data and analytics infrastructures.
The BigQuery’s storage engine is optimized for real-time streaming. BigQuery supports streaming ingestion of 10s of millions of events in real-time and there is no impact on query performance. Additionally customers can use materialized views and BI Engine (which is now GA) on top of streaming data. We guarantee always fast, always fresh data. Our system automatically updates MVs and BI Engine.
Many customers also use our PubSub service to collect real-time events and process these through Dataflow prior to ingesting into BigQuery. This is a streaming ETL pattern which is very popular. Last year,we announced PubSub Lite to provide customers with a 90% lower price point and aTCO that is lower than any DIY Kafka deployment.
We also announced Dataflow Prime, it is our next generation platform for Dataflow. Big Data processing platforms have only focused on horizontal scaling to optimize workloads. But we have seen new patterns and use cases like streaming AI where you may have a few steps in pipelines that perform data prep and then customers have to run a GPU based model. Customers want to use different sizes and shapes of machines to run these pipelines in the most optimum manner. This is exactly what Dataflow Prime does. It delivers vertical auto scaling with the right fitting for your pipelines. We believe this should lower costs for pipelines significantly.
With Datastream as our change data capture service (built on Alooma technology), we have solved the last key problem space for customers. We can automatically detect changes in your operational databases like MySQL, Postgres, Oracle etc and sync them in BigQuery.
Most importantly, all these products work seamlessly with each other through a set of templates. Our goal is to make this even more seamless over next year.
Open Data Analytics with BigQuery
Google has always been a big believer in Open Source initiatives. Our customers love using various open source offerings like Spark, Flink, Presto, Airflow etc. With Dataproc & Composer our customers have been able to run various of these open source frameworks on GCP and leverage our scale, speed and security. Dataproc is a great service and delivers massive savings to customers moving from on-prem Hadoop environments. But customers want to focus on jobs and not clusters.
That’s why we launched Dataproc Serverless Spark (GA) offering at Next 2021. This new service adheres to one of our key design principles we started with: make data simple.
Just like with BigQuery, you can simply RUN QUERY. With Spark on Google Cloud, you simply RUN JOB. ZDNet did a great piece on this. I invite you to check it out!
Many of our customers are moving to Kubernetes and wanted to use that as the platform for Spark. Our upcoming Spark on GKE offering will give the ability to deploy spark workloads on existing Kubernetes clusters.
But for me the most exciting capability we have is, the ability to run Spark directly on BigQuery Storage. BigQuery storage is highly optimized analytical storage. By running Spark directly on it, we again bring compute to data and avoid moving data to compute.
BigSearch to power Log Analytics
We are bringing the power of Search to BigQuery. Customers already ingest massive amounts of log data into BigQuery and perform analytics on it. Our customers have been asking us for better support for native JSON and Search. At Next 21 we announced the upcoming availability of both these capabilities.
Fast cross column search will provide efficient indexing of structured, semi-structured and unstructured data. User friendly SQL functions let customers rapidly find data points without having to scan all the text in your table or even know which column the data resides in.
This will be tightly integrated with native JSON, allowing customers to get BigQuery performance and storage optimizations on JSON as well as search on unstructured or constantly changing data structures.
Multi & Cross Cloud Analytics
Research on multi cloud adoption is unequivocal — 92% of businesses in 2021 report having a multi cloud strategy. We have always believed in providing customers choice to our customers and meeting them where they are. It was clear that all our customers wanted us to take our gems like BigQuery to other clouds as their data was distributed on different clouds.
Additionally it was clear that customers wanted cross cloud analytics not multi-cloud solutions that can just run in different clouds. In short, see all their data with a single pane of glass, perform analysis on top of any data without worrying about where it is located, avoid egress costs and finally perform cross cloud analysis across datasets on different clouds.
With BigQuery Omni, we deliver on this vision, with a new way of analyzing data stored in multiple public clouds. Unlike competitors, BigQuery Omni does not create silos across different clouds. BigQUery provides a single control plane that shows an analyst all data they have access to across all clouds. Analyst just writes the query and we send it to the right cloud across AWS, Azure or GCP to execute it locally. Hence no egress costs are incurred.
We announced BQ Omni GA for both AWS and Azure at Google Next 21 and I’m really proud of the team for delivering on this vision. Check out Vidya’s session and learn from Johnson and Johnson how they innovate in a multi-cloud world.
Geospatial Analytics with BigQuery and Earth Engine
We have partnered with our Google Geospatial team to deliver GIS functionality inside BigQuery over the years. At Next we announced that customers will be able to integrate Earth Engine with BigQuery, Google Cloud’s ML technologies, and Google Maps Platform.
Think about all the scenarios and use-cases your team’s going to be able to enable sustainable sourcing, saving energy or understanding business risks.
We’re integrating the best of Google and Google Cloud together to – again – make it easier to work with data to create a sustainable future for our planet.
BigQuery as a Data Exchange & Sharing Platform
BigQuery was built to be a sharing platform. Today we have 3000+ organizations sharing more than 250 petabytes of data across organizations. Google also brings more than 150 public datasets to be used across various use cases. In addition to this, we are also bringing some of the most unique datasets like Google Trends to BigQuery. This will enable organizations to understand in real-time trends and apply to their business problems.
I am super excited about the Analytics Hub Preview announcement. Analytics Hub will provide the ability for organizations to build private and public analytics exchanges. This will include data, insights, ML Models and visualizations. This is built on top of the industry leading security capabilities of BigQuery.
Breaking Data Silos
Data is distributed across various systems in the organization and making it easy to break the data silo and make all this data accessible to all is critical. I’m also particularly excited about the Migration Factory we’re building with Informatica and the work we are doing for data movement, intelligent data wrangling with players like Trifacta and FiveTran, with whom we share over 1,000 customers (and growing!). Additionally we continue to deliver native Google service to help our customers.
We acquired Cask in 2018 and launched our self service Data Integration service in Data Fusion. Now Fusion allows customers to create complex pipelines with just simple drag and drop. This year we focused on unlocking SAP data for our customers. We have launched various SAP connectors and accelerators to achieve this.
At GCP Next we also announced our BigQuery Migration service in preview. Many of our customers are migrating their legacy data warehouses and data lakes to BigQuery. BigQuery Migration Service provides end-to-end tools to simplify migrations for these customers.
And today, to make migrations to BigQuery easier for even more customers, I am super excited to announce the acquisition of CompilerWorks. CompilerWorks’ Transpiler is designed from the ground up to facilitate SQL migration in the real world and will help our customers accelerate their migrations. It supports migrations from over 10 legacy enterprises data warehouses and we will be making it available as part of our BigQuery Migration service in the coming months.
Data Democratization with BigQuery
Over the past 4 years we have focused a lot on making it very easy to derive actionable insights from data in BigQuery. Our priority has been to provide a strong ecosystem of partners that can provide you with great tools to achieve this but also deliver native Google capabilities.
With our BI engine GA announcement which we introduced in 2019, previewed earlier this year and showcased with tools like Microsoft PowerBI and Tableau, is now available for all to play with.
BigQuery + Data Studio are like peanut butter and Jelly. They just work well together. We launched BI Engine first with Data Studio and scaled it to all the users. More than 40% of our BigQuery customers use Data Studio. Once we knew BI Engine works extremely well we now have made it an integral part of BigQuery API and launched it for all our internal and partner BI tools.
We announced GA for BI Engine at Next 2021 but we were already GA with Data Studio for the past 2 years. We recently moved the Data Studio team back into Google Cloud making the partnership even stronger. If you have not used Data Studio, I encourage you to take a look and get started for free today here!!
Connected Sheets for BigQuery is one of my favorite combinations. You can give every business user in your organization the ability to analyze billions of records using standard Google Sheets experience. I personally use it everyday to analyze all our product data.
We acquired Looker in Feb 2020 with a vision of providing a semantic modeling layer to our customers with a governed BI solution. Looker is tightly integrated with BigQuery including BigQuery ML. Our latest partnership with Tableau where Tableau customers will soon be able to leverage Looker’s semantic model, enabling new levels of data governance while democratizing access to data.
Finally, I have a dream that one day we will bring Google Assistant to your enterprise data. This is the vision of Data QnA. We are in early innings on this and we will continue to work hard to make this vision a reality.
Intelligent Data Fabric to unify the platform
Another important trend that shaped our market is the Data Mesh. Earlier this year, Starburst invited me to talk about this very topic. We have been working for years on this concept, and although we would love for all data to be neatly organized in one place, we know that our customers’ reality is that it is not (If you want to know more about this, read about my debate on this topic with Fivetran’s George Fraser, a16z’s Martin Casado and Databricks’ Ali Ghodsi).
Everything I’ve learned from customers over my years in this field is that they don’t just need a data catalog or a set of data quality and governance tools, they need an intelligent data fabric. That is why we created Dataplex, whose general availability we announced at Next.
Dataplex enables customers to centrally manage, monitor, and govern data across data lakes, data warehouses, and data marts, while also ensuring data is securely accessible to a variety of analytics and data science tools. It lets customers organize and manage data in a way that makes sense for their business, without data movement or duplication. It provides logical constructs – lakes, data zones, and assets – which enable customers to abstract away the underlying storage systems to build a foundation for setting policies around data access, security, lifecycle management, and so on. Check out Prajakta Damle’s session and learn from Deutsche Bank how they are thinking about a unified data mesh across distributed data.
Closing Thoughts
Analysts have recognized our momentum and, as I look back at this year, I couldn’t thank our customers and partners enough for the support they provided my team and I across our large Data Analytics portfolio: in March, Google BigQuery was named a Leader in The Forrester Wave™: Cloud Data Warehouse, Q1 2021. And in June, Dataflow was named a Leader in The Forrester Wave™: Streaming Analytics, Q2 2021 report.
If you want to get a taste for why customers choose us over other hyperscalers or cloud data warehousing, I suggest you watch the Data Journey series we’ve just launched, which documents the stories of organizations modernizing to the cloud with us.
The Google Cloud Data Analytics portfolio has become a leading force in the industry and I couldn’t be more excited to have been part of it. I do miss you, my customers and partners, and I’m frankly bummed that we didn’t get to meet in person like we’ve done so many times before (see a photo of my last in-person talk before the pandemic), but this Google Next was extra special, so let’s dive into the product innovation and their themes.
I hope that I will get to see you in person next time we run Google Next!
Streamlining Business Processes with Google’s Document AI: Invoices, Contracts, and Beyond
1149
Of your peers have already read this article.
3:30 Minutes
The most insightful time you'll spend today!
Editor’s note: In this post, I’ll be showing some amazing ways Document AI can help you extract meaning from your documents – keep reading, or jump directly into a tutorial using the Cloud Console!
Documents are a crucial part of most businesses, used to store and communicate important information. The variety is vast: invoices, contracts, receipts, applications, plus documents unique within industries and geographies. Unfortunately, making the information contained in these documents accessible can be a time-consuming and manual process.
Document AI is a document understanding platform in Google Cloud that takes unstructured data from documents and transforms it into structured data, making them easier to understand, analyze, and consume. By using this technology, you can streamline your document processing workflows, reduce errors, and unlock insights that were previously buried in mountains of paperwork.
Whether you’re a small business owner or an enterprise looking to bring efficiency to your operations, Document AI has something to offer. So let’s take a look and see what it can do!
Understanding documents with Document AI
When we say that Document AI can understand documents, we mean that it is able to analyze the content within documents and derive meaningful insights from it. This goes beyond simply recognizing the characters and words within a document (which is what traditional OCR technology does) – Document AI can actually comprehend the meaning behind the text.
For example, let’s say you have a contract that needs to be processed. Traditional OCR technology might be able to extract the text from the document, but it would not be able to understand the legal terms and clauses within it. Document AI, on the other hand, can actually interpret the meaning of the text and extract key information such as parties involved, terms and conditions, dates, and signatures.
Document AI offers several pre-built models and processors that are specifically designed to extract different types of data from various document types. Within the specific document types, the processors can perform several tasks such as Optical Character Recognition (OCR), form parsing, splitting, classification or entity extraction. These processors can be customized and combined to create powerful document processing workflows that are tailored to a business’s unique needs.
Let’s look closer at a few of the processors available in Document AI, including the Form Parser, Invoice Parser, Expense Parser, Identity Document Proofing Parser, and Intelligent Document Quality Processor.
Form Parser
This general processor is designed to extract structured data from forms such as application forms, surveys, and questionnaires. It automatically identifies and extracts data from form fields (key-value pairs), such as names, addresses, dates, and other types of structured data; even checkboxes and tables. This processor also leverages deep learning models to extract generic entities that are common in various document types, meaning it can identify if something is an email address, phone number, datetime, organization, quantity, price, person, and more.
In this visualization of the API response, you can see that Document AI has identified several key value pairs that correspond to the form’s fields and the responder’s answers.
Also of interest is that the form parser recognized certain generic entities including: several dates, an address, phone numbers, email, and two people (the responder and their listed emergency contact).
Invoice Parser
This parser is designed to identify and extract relevant information from invoices including a large number of typical invoice fields, but can also be customized (uptrained) to recognize different invoice layouts, languages, and data fields. Invoices are a critical part of the accounts payable process, making this functionality valuable across industries and companies building A/P features into their products.
In this visualization of the API response, you can see that Document AI has extracted a large number of key-value pairs and even provided normalized values for several of the fields.
Expense Parser
This specialized processor is designed to extract data from receipts and invoices, such as vendor name, date, and total amount paid. It can also identify line items within an invoice and categorize them based on the type of expense (e.g. meals, travel, office supplies). The expense parser makes it easier for you to process expense reports and other financial documents, and it can integrate with other tools and systems to completely automate the entire expense reporting process.
In this visualization of the API response, you can see that Document AI has extracted the text from the receipt and identified several typical entities such as purchase date and time, payment type, and total amount.
Identity Document Proofing Parser
This processor is designed to help predict the validity of ID documents with four different signals.
- is_identity_document detection: Predicts whether an image contains a recognized identity document.
- suspicious_words detection: Predicts whether words are present that aren’t typical on IDs.
- image_manipulation detection: Predicts whether the image was altered or tampered via an image editing tool.
- online_duplicate detection: Predicts whether the image can be found online.
If suspicious words are detected or the image can be found online, additional information is provided to explain these signals.
This can be particularly useful for businesses that need to verify the identity of customers or employees as part of their operations. This processor could be used in conjunction with other processors that extract key information such as name, date of birth, ID number, and expiration date from specific identity documents (US Driver License Parser, US Passport, France National ID Parser, etc.).
In this visualization of the API response, you can see that Document AI has passed the document on the first detection point (is_identity_document), but failed the document for the other three items and provided additional information in the evidence fields.
Intelligent Document Quality Processor
This general use processor is designed to detect a variety of document quality issues, such as missing pages, blurry images, low contrast, inconsistent formatting, and incorrect data, and flag these potential issues which could affect their usability, accuracy, or compliance. It can also identify sensitive information that has not been redacted or missing information required by regulatory standards.
This quality assessment is returned as a quality score from 0 to 1, where 1 means perfect quality. If the quality score detected is lower than 0.5, a list of negative quality reasons (sorted by the likelihood) is also returned.
In this visualization of the API response, you can see that Document AI has determined a quality_score of 0.006 and also provided a list of several reasons including document and text cutoff, blurriness, and glare (among others).
Next steps
These are just a few examples of the types of processors that Document AI offers. Perusing the current documentation, you can find it has more than 40 processors, with each providing several functions, to explore. Document AI also offers you the ability to uptrain certain processors and supports the option to build your own custom processor. Across the board, if a document contains structured or unstructured text, Document AI has the capability to extract valuable data from it.
Get started and learn more by heading to our tutorials in the Cloud Console:
More Relevant Stories for Your Company
Data to Business Outcomes with Google’s Data Analytics Design Pattern
Companies today are inundated with vast amounts of data from various sources. This overwhelming amount of data is meant to benefit the company, but often leaves data teams feeling overwhelmed, which can create data bottlenecks and result in a slow time to value. In fact, only twenty seven percent of
How TeamSnap Improved Return on Ad Spend Significantly
Anyone who has ever coached or played on a sports team, or had a child involved in sports, knows how difficult scheduling and logistics can be. From game and practice schedules, to uniforms and who’s bringing the snacks, it can be a lot for coaches, administrators, parents, and players to
Transform Your Marketing Strategy with Tinyclues and Google Cloud CDP
Editor’s note: The post is part of a series highlighting our awesome partners, and their solutions, that are Built with BigQuery. What are Customer Data Platforms (CDPs) and why do we need them? Today, customers utilize a wide array of devices when interacting with a brand. As an example, think
Apollo24|7 partnered with Google Cloud to build the Clinical Decision Support System (CDSS) together
Clinical Decision Support System (CDSS) is an important technology for the healthcare industry that analyzes data to help healthcare professionals make decisions related to patient care. The market size for the global clinical decision support system appears poised for expansion, with one study predicting a compound annual growth rate (CAGR)
