Data Catalog for Data Discovery and Metadata Management - Build What's Next

3011

Of your peers have already watched this video.

25:00 Minutes

The most insightful time you'll spend today!

Explainer

Data Catalog for Data Discovery and Metadata Management

Data discovery and metadata management is a common pain point for most enterprises. Data Catalog is a managed service that addresses this pain point with a scalable and performant solution.

“Data catalog, simply put, enables users to discover, understand, and manage all their data assets,” says Shekhar Bapat, Product Manager, Google Cloud.

Data Catalog also offers integration with DLP for auto detection and tagging of sensitive data and facilitates better data governance. Data Catalog provides support for non-Google Cloud data assets and enables customers to discover, understand, and manage ALL their data.

The session is co-presented with Shruti Thaker, Head of Alt Vendor Data and Alpha Capture, BlackRock, which has invested in Data Catalog. Thaker will share how BlackRock is using Data Catalog to create an effective metadata solution for BlackRock data assets.

Blog

DocAI Lowers Customer’s Document Processing Cost by 60 Percent. Learn How

5618

Of your peers have already read this article.

2:00 Minutes

The most insightful time you'll spend today!

Document processing in most organizations require human intervention for review, data extraction and unlocking valuable insights. With Google Cloud Doc AI platform, document-intensive industries and businesses can end their guess work through automation, document processing accuracy, ML-based predictions and better UX at 60 percent lesser document processing cost.

Some of the most important data at your company isn’t living in databases, but in documents, and most business processes begin, involve or end with a document. 

Yet most companies are still manually entering data and reliant on guesswork to make sense of it all as the volume and variety of data explodes. Organizations are also leaving heaps of value on the table in the form of new and better customer experiences that can be unlocked with artificial intelligence (AI) applied to documents. 

The latest releases of Document (Doc) AI platformLending DocAI and Procurement DocAI, built on decades of AI innovation at Google, bring powerful and useful solutions to these challenges. Under the hood are Google’s industry-leading technologies:

  • Computer vision (including OCR) and Natural Language Processing (NLP) that creates pre-trained models for high-value, high-volume documents. 
  • Google Knowledge Graph to validate and enhance the fields in your documents.
  • Training and creation of your own custom document models. 
  • Human interaction with AI to ensure accuracy where needed.

Google Cloud DocAI platform, Lending DocAI and Procurement DocAI are now generally available. Thousands of customers have tried these products in the preview phase—and DocAI has already processed tens of billions of pages of documents across lending, insurance, government and other industries.

“The vast majority of enterprise content still resides in unstructured sources like documents. Google Cloud’s Document AI brings a fresh new perspective to the problem informed by the company’s decades of experience making sense of the largest unstructured corpus in the world—the world wide web.” Ritu Jyoti, VP of AI Research, IDC

Cut document processing costs by up to 60%

Lending DocAI helps banks, mortgage brokers and other lending institutions fast track the loan application process from weeks to days, dramatically reducing the cost of issuing a loan. And Procurement DocAI enables companies to automate procurement data capture at scale, lowering processing costs by up to 60%.

These solutions are built on DocAI platform, a unified console for document processing that lets you quickly access all parsers and tools. From the platform, you can automate and validate documents to streamline workflows, reduce guesswork, and keep data accurate and compliant. 

Get more value from AI with DocAI’s industry-specific solutions

According to Accenture’s AI: Built to Scale report: “Companies that scale successfully see 3x the return on their AI investments compared to those who have not fully rolled out AI capabilities.” 

Core to our strategy at Google Cloud is the creation of industry-specific solutions that help companies get maximum value out of their investments in AI. We announced Lending DocAI, our first solution designed specifically for the financial services industry, at the Mortgage Bankers Association convention last year. It processes borrowers’ income and asset documents using a set of specialized machine learning (ML) models, and automates routine document reviews so that mortgage providers can focus on more important work. 

Lending DocAI is now generally available and includes more specialized parsers for critical loan documents including paystubs, bank statements, and more. Our goal is to provide the right tools to help borrowers and lenders have a better experience and close home loans faster. For more, watch this video.

Procurement DocAI is also now generally available. This solution helps companies accelerate document processing for invoices, receipts, and other valuable documents in the procurement cycle. 

Automating data capture is helping our customers increase accuracy and also lower their procure-to-pay processing costs. We are continually expanding the types of documents Procurement DocAI can process—the latest is a utility parser for electric, water and other bills. In addition, Procurement DocAI leverages Google Knowledge Graph to validate and enrich parsed information to make the data even more useful. Check out this overview video for more details. 

One company that lives and breathes AI-enabled document management is AODocs. It uses Procurement DocAI to simplify invoice processing for enterprise customers and launched a new Gmail add-on, Invoice to Sheet, for SMB customers who just want to track their invoices in Google Sheets.

“Google Cloud’s Procurement DocAI service allows our document management platform to better automate the processing of invoices; AODocs customers who have tested our new account payables workflow estimate that the productivity of their A/P team has more than doubled, thanks to the reduction of manual data input brought by the Procurement DocAI.”Stéphan Donzé, Founder and CEO, AODocs

The new specialized parsers for Lending and Procurement DocAI can be used alongside our existing AutoML Text & Document Classification and AutoML Document Extraction services. These technologies provide a state-of-the-art toolset for creating new document models and have been widely deployed by customers in financial services and other industries. 

Partner to accelerate your AI deployment and results

Having the right partner to ease the complexity of rolling out your AI-strategy in mortgage document processing is critical to transforming your customers’ experience. We’re excited to announce a partnership with Mr. Cooper, a leader in mortgage servicing, to provide customers with more automation and workflow tools throughout their entire mortgage life cycle. As part of this agreement, both companies will collaborate on digitizing Mr. Cooper’s core mortgage platform, creating a more personal customer experience utilizing AI, and driving a broader culture of innovation to imagine and develop services and solutions that will transform the mortgage experience for American homeowners.

“Over the last few years, we have made substantial investments in our servicing technology and core mortgage platform that have revolutionized the customer experience, while providing dramatic efficiencies in operating cost. Our partnership with Google Cloud AI will build on those advances and help make these technologies available for the mortgage industry.” —Jay Bray, Chairman and CEO, Mr. Cooper Group

This builds upon the robust partner ecosystem we’re creating to help customers revolutionize the home loan experience, which includes last year’s partnership announcement with Roostify.

Integrate human review into ML predictions 

Next up is the general availability of Human-in-the-Loop AI, a new DocAI feature that will help companies achieve higher document processing accuracy with the assurance of human review. Adding human review can increase accuracy and help businesses interpret predictions using purpose-built tools to enable those reviews. 

Processing documents quickly and cost-effectively is important. But it’s often necessary to have a high level of assurance on data accuracy for compliance. CIOs and IT decision-makers need highly accurate ML predictions to fulfill compliance requirements, improve employee experience (e.g. less rework), and raise customer satisfaction (e.g. fewer data errors). Including human participation in ML processes allows AI and humans to work together for the best possible results.

gcp human-in-the-loop ai.gif

Human-in-the-Loop AI provides the workflow to manage human review tasks and produces a percentage confidence score of how “sure” it is that the AI ingested the document correctly. Document AI extracts data from documents with ML, and when paired with Human-in-the-Loop AI, human reviewers are able to verify the data captured. This system is customizable, providing the flexibility to set different thresholds and assign individual groups of reviewers to various stages of the workflow. With Human-in-the-Loop AI, developers can choose trusted reviewers to assign to the task; these reviewers can be from within their own or partner organizations.

More Document AI resources

To learn more, check out the Document AI webpage and watch a demo of how to process sample forms in AI Platform notebooks to inspect data extraction and confidence scores. For more on how customers and partners like Workday, AODocs, and Mr. Cooper are using Document AI, listen to our fireside chat. And stay tuned for the exciting evolution of these technologies in future releases of DocAI.

5203

Of your peers have already watched this video.

5:00 Minutes

The most insightful time you'll spend today!

How-to

How to Build and Maintain Spark and Hadoop Clusters Quickly, Easily and Inexpensively

How often have you struggled to create Spark and Hadoop clusters only to have them become unstable?

Building and maintaining clusters that are the building blocks of big data analytics should not be a time-consuming, repetitive, and expensive process. But with the current set of tools, it often is.

Google Cloud Dataproc solves this. It is a fast, easy-to-use, fully-managed cloud service for running Spark and Hadoop clusters in a simple, more cost-efficient way.

Operations that used to take hours or days take now seconds or minutes instead, and you pay only for the resources you use (with per-second billing).

Watch this five-minute demo to find out how—with just a few clicks—you can set up clusters that build and heal themselves. You’ll find out how to creating a large Dataproc cluster with preemptible VMs, which help run large clusters at a low total cost.

Case Study

AirAsia Flies High With Data Analytics and AI

6662

Of your peers have already read this article.

11:30 Minutes

The most insightful time you'll spend today!

How AirAsia leverages a broad suite of data prep, data streaming, data warehouse, data analytics tools, and AI to dramatically improve ease and speed, while lowering cost. In the process is it also better able to refine pricing, increase revenue, and improve customer experience.

AirAsia’s vision is simple: allow everyone to fly. Founded in 2001, the airline and sister company AirAsia X have grown to service 150+ destinations in 25 markets, using 274 aircraft to operate 11,000+ weekly flights from 23 hubs across the region. While the airline is known as a provider of low-cost airfares to locations across Asia, this is only one part of its value proposition. AirAsia also aims to deliver innovative, personalized products and services that meet the needs of each of its passengers.

Technology is key to AirAsia’s success and, with strong support from Group Chief Executive Officer Tony Fernandes, in 2016 the airline began a five-year program to become a data-first business and a digital airline.

“We wanted to better ensure we were using data correctly to become more agile, efficient, and customer oriented,” says Lye Kong Wei, Chief of Data Science, Group Head at AirAsia.

AirAsia needed technologies and services that could capture, process, analyze, and report on data, while delivering value for money and meeting its speed and availability requirements. The airline also wanted to minimize infrastructure management and system administration demands on its technology team.

“We knew data was a big part of making decisions in the future. So we needed a platform that could scale to meet our growing appetite for it. Google Cloud—in particular BigQuery—was ideal for this task.”

Lye Kong Wei, Chief of Data Science, Group Head, AirAsia

Google Cloud the best fit

AirAsia realized only a cloud service could meet its needs and began evaluating the market. The airline then conducted a proof of concept and found Google Cloud was the best fit for its business. It was already familiar with Google Cloud, having deployed G Suite collaboration and productivity applications to its workforce in all countries except China. According to Kong Wei, products such as FormsDocsSheets, and Gmail delivered a considerable improvement in collaboration between various departments, as well as streamlining and automating a range of processes.

The business was particularly excited by the potential of the BigQuery analytics data warehouse to power its digital transformation. “We knew data was a big part of making decisions in the future,” says Kong Wei. “So we needed a platform that could scale to meet our growing appetite for it. Google Cloud—in particular BigQuery—was ideal for this task.”

“With BigQuery, we could process queries and requests much faster than previously and tackle more complex problems.”

Lye Kong Wei, Chief of Data Science, Group Head, AirAsia

The AirAsia technology team was impressed by the ease and flexibility with which it could extract, transform, and load customer data from its systems, websites, and mobile applications into BigQuery for analysis. Data, reports, and dashboards were delivered and visualized through Google Data Studio.

BigQuery also scaled seamlessly to support data growth and, as a managed service, required minimal administration from the airline’s technology team. “In addition, with BigQuery, we could process queries and requests much faster than previously and tackle more complex problems,” says Kong Wei. “As a result, we could be more innovative about realizing opportunities,” he adds, citing the benefits of being able to view and understand historical measures of booking curves—a measure of how long it took customers to book before a flight. This improves the airline’s ability to manage revenues.

A broad ecosystem

BigQuery and Data Studio are just two components of a broad ecosystem—powered largely by Google Cloud services—deployed by AirAsia. Pub/Sub provides a scalable message queue that enables AirAsia developers to integrate systems hosted on Google Cloud or externally. Apache Airflow enables the business to create, schedule, and monitor workflows, while Cloud Composer manages the dependencies of PHP and related libraries. App Engine allows AirAsia personnel to develop and host web applications. “Thanks to App Engine, we’ve easily been able to create new applications, services, and APIs powered by the data we have been collecting,” says Kong Wei.

AirAsia is also running the middleware for its APIs in a managed Google Kubernetes Engine environment for increased scalability, resource optimization, and reliability, while Cloud Storage provides storage for data from a range of systems and sources. Dataflow enables the business to transform and process data in stream and batch modes from its website search page as customers look for flights.

“With a minimal number of people involved, we can very quickly transform an idea or thought process into a deliverable. Prior to Google Cloud, bringing those ideas to fruition would have been impossible.”

Lye Kong Wei, Chief of Data Science, Group Head, AirAsia

Faster deployment and testing

The stability of Google Cloud service means AirAsia has a reliable base from which to launch new products and features. “If we have consistent reliability from our core systems—and Google Cloud incorporates monitoring tools such Cloud Monitoring that enable us to identify issues quickly—developers and product engineers can focus on turning ideas into reality,” says Kong Wei. “With a minimal number of people involved, we can very quickly transform an idea or thought process into a deliverable. Prior to Google Cloud, bringing those ideas to fruition would have been impossible.”

Robust security

AirAsia is also relying on Security Health Analytics, a product that integrates with Security Command Center, to identify misconfigurations and compliance violations in its Google Cloud resources and take action. Security Health Analytics ensures the airline’s budgets go to keeping customers’ travel costs low rather than recovering from security breaches.

The product enables AirAsia to check that resources are configured properly and are compliant with CIS benchmarks as its critical workloads run in Google Kubernetes Engine and App Engine.

“Being able to go to the new Security Health Analytics dashboard eliminates the guesswork of what we have running and if it is secure,” says Muhammad Faeez Bin Azmi, Information Security and Automation Solution Architect. “Now anyone on our team, even non-security professionals, can go to this dashboard and see a list of the misconfigured assets and compliance violations across all of our Google Cloud resources. We can also see the severity of misconfigurations, which helps us prioritize our response.”

“Security Health Analytics has really helped us reduce the amount of time we spend trying to figure out what’s wrong with our resources. It’s allowed us to use our time more effectively to identify and resolve more security issues than we could before.”

A new identity solution

AirAsia had also used a legacy on-premises directory for many years. However, as the company grew and expanded to new markets and regions, it had to manage multiple servers across a number of on-premises data centers and the public cloud, which proved costly and time-consuming.

Its Allstars—the airline’s name for its employees—needed to easily access a number of legacy on-premises apps in addition to a growing number of SaaS apps. As a business, it also needed a more seamless integration between its HR system of record and its identity solution for user provisioning and life cycle management. Solving these challenges with its existing on-premises directory was simply not feasible.

AirAsia brought up its identity concerns with the Google Cloud team, and after a number of conversations, decided to deploy Cloud Identity, Google’s cloud-based Identity and Access Management solution, to help address the identity challenges it was facing.

The airline chose Cloud Identity for a number of reasons—first, it was eager to move to the cloud as quickly as possible. Moving identity management to the cloud was a key enabler of this and the airline’s broader digital transformation. Managing identities from the cloud also enabled the airline to have a single identity and set of credentials for each employee, which they could use to access all the applications they need to be productive, both in the cloud and on-premises.

In addition, deploying Cloud Identity was a key step towards enabling the zero trust security model, which the airline felt was the best approach to strengthen its security posture and fight modern threats. Cloud Identity also integrated seamlessly with its existing technologies, which include not only Google Cloud products like G Suite and Chrome OS but also third-party tools like Citrix, Papercut, and others.

And finally, Cloud Identity offered significant cost and resource savings. With Cloud Identity in place, AirAsia’s IT department could spend less time worrying about managing multiple on-premises directory servers and and could instead focus on delivering value to Allstar employees.

Machine learning employed to increase ancillary revenue

In March 2018, AirAsia established the groundwork to use machine learning to optimize pricing for a range of services and began by using AI Platform to sort and predict demand for ancillary services such as baggage, seats, and meals. “By using AI Platform, we can sort based on data about history to predict the future,” says Kong Wei.

Dialogflow in wide use

With Google Cloud well established within the business, AirAsia is using Dialogflow, a voice and conversational interface development suite (and one of the core components of Contact Center AI) that enables businesses to create engaging AI powered voice- and text-based interfaces such as chatbots and voice apps—to streamline operations and reduce costs

“Tony Fernandes, our Group Chief Executive Officer, motivated us to create a range of metrics that would enable us to respond more quickly and efficiently to customers,” says Yuashini Vellasamy, Product Manager at AirAsia.

This resulted in the initial deployment of Dialogflow in AirAsia’s operational areas, from crew scheduling to internal business tasks.

With Dialogflow, AirAsia is able to provide pilots, crew, catering, and other teams with flight times, capacity and any other relevant information about their assignments. Another bot accepts medical certificates and updates from pilots and crew members unable to make rostered assignments and switches those assignments to other pilots and crews. “This improves our on-time performance and operational efficiency, as we are able to optimize the creation of crew schedules,” says Vellasamy.

AirAsia also uses Dialogflow to power a “safety bot” for airport ground staff. “We tried a lot of tools to encourage staff to advise us when they saw a safety issue—but employee usage was low,” says Vellasamy. “However, when we set up the safety bot, which asks users just four questions about an issue they have observed, we saw an increase in employee interaction, which helps ensure proper safety measures.

Beyond crew optimization and overall safety, AirAsia’s HR department in Asia uses a Dialogflow-powered bot to run a post-orientation engagement program for new employees. The bot follows up with employees about issues from parking to AirAsia’s onboarding buddy program, saving HR employees time from scheduling one-on-one meetings. Dialogflow also powered an employee satisfaction survey of 3,000 team members, resulting in a 30% increase in response rate compared to the previous year.

One of Dialogflow’s benefits is its language coverage, which the airline’s marketing team uses to target customers and prospects across a region with diverse languages and cultures. “For weekly campaigns and promotions, our sales and marketing teams simply update the campaign materials and the changes are reflected automatically in Dialogflow and consequently to users,” says Vellasamy. “This enables us to quickly scale and reach our customers globally.”

An upward trajectory

AirAsia is poised to reap further benefits from Google Cloud as its deployment matures. “On the data side, we’re more advanced, while on the application and programming side we have a long way to go,” says Kong Wei. “We’re on a trajectory upwards—we’re looking at a lot of new ideas and how to embrace and deploy them so we can further our data-first and digital airline agendas.”

3131

Of your peers have already watched this video.

38:30 Minutes

The most insightful time you'll spend today!

How-to

Building Data Lakes on Google Cloud

A data lake is a centralized and a secure data platform for ingesting, storing, processing, and analyzing all your data–structured semi-structured and unstructured data–at scale.

“When we work with our customers we find out that the traditional approach for building data lakes, where you bring data from different sources and land them into a single central store has not worked. The traditional approach ended up creating swamps instead of lakes,” says Nitin Motgi, Product Manager, Google Cloud.

In this video, Motgi will talk about how to build data lakes correctly.

Google Cloud provides all the capabilities enterprises need to create and manage data lakes. Enterprises can use Google Cloud to aggregate their data and efficiently analyze it using cloud-native or open source tools irrespective of where the data is managed.

Also, learn how Google is making investments to simplify data lake deployment and management.

E-book

The Data Scientist Guide to Preparing and Curating Data for AI

DOWNLOAD E-BOOK

5369

Of your peers have already downloaded this article

8:15 Minutes

The most insightful time you'll spend today!

To design and implement a successful machine learning (ML) project, you often need to collaborate with multiple teams, including those in business, sales, research, and engineering. Understanding the essentials of gathering and preparing your data is crucial to align teams, and getting a project off the ground.

Building an ML model that achieves your business’ goal, requires the kind of data that a model can learn from. If the goal is to predict a target output based on some input, you’ll need some data consisting of past input-output examples. For example, to detect (that is, to predict) a fraudulent credit card transaction (the target of the prediction), each example must contain information on a past credit card transaction and whether that transaction was fraudulent or not.

So what are the best practices in preparing and curating data for a machine learning initiative? The answer to this question lies within the guide. You’ll learn:

  • To evaluate whether your data is suitable for ML
  • What types of data you need–and which ones to eliminate
  • How to prevent data leakage
  • Tricks to speed up an ML project such as Google’s human labeling service, and Cloud Dataprep
  • The importance of the granularity of data

More Relevant Stories for Your Company

Case Study

An Indian Example of How to Really Up Your Customer Experience Game and Increase Conversion Rates With AI

How about selfie analysis of users to recommend them the right lipstick color? That’s just one of the many ideas folks at Purplle.com came up with to improve the buying experience of Indian consumers. And without the power of Google Cloud, it would probably have remained just that…an idea. But

Case Study

Video: How AI is Helping Biologists Protect Wildlife

According to the World Wildlife Fund, vertebrate populations have shrunk an average of 60 percent since the 1970s. And a recent UN global assessment found that we’re at risk of losing one million species to extinction, many of which may become extinct within the next decade.  To better protect wildlife, seven organizations, led

Blog

Make Meaningful Analysis with Geo Boundary Public Datasets on BigQuery

Geospatial data is a critical component for a comprehensive analytics strategy. Whether you are trying to visualize data using geospatial parameters or do deeper analysis or modeling on customer distribution or proximity, most organizations have some type of geospatial data they would like to use - whether it be customer

Blog

A Comprehensive Guide for Understanding and Optimizing Your Dataflow Costs

Dataflow is the industry-leading platform that provides unified batch and streaming data processing capabilities, and supports a wide variety of analytics and machine learning use cases. It’s a fully managed service that comes with flexible development options (from Flex Templates and Notebooks to Apache Beam SDKs for Java, Python and Go),

SHOW MORE STORIES