At enterprises across industries, documents are at the center of core business processes. Documents store a treasure trove of valuable information whether it’s a company’s invoices, HR documents, tax forms and much more. However, the unstructured nature of documents make them difficult to work with as a data source. We call this “dark data” or unstructured data that businesses collect, process and store but do not utilize for purposes such as analytics, monetization, etc. These documents in pdf or image formats, often trigger complex processes that have historically relied on fragmented technology and manual steps. With compute solutions on Google Cloud and Document AI, you can create seamless integrations and easy to use applications for your users. Document AI is a platform and a family of solutions that help businesses to transform documents into structured data backed by machine learning. In this blog post we’ll walk you through how to use Serverless technology to process documents with Cloud Functions, and with workflows of business processes orchestrating microservices, API calls, and functions, thanks to Workflows.

At Cloud Next 2021, we presented how to build easy AI-powered applications with Google Cloud. We introduced a sample application for handling incoming expense reports, analyzing expense receipts with Procurement Document AI, a DocAI solution for automating procurement data capture from forms including invoices, utility statements and more. Then organizing the logic of a report approval process with Workflows, and used Cloud Functions as glue to invoke the workflow, and do analysis of the parsed document.

We also open sourced the code on this Github repository, if you’re interested in learning more about this application.

In the above diagram, there are two user journeys: the employee submitting an expense report where multiple receipts are processed at once, and the manager validating or rejecting the expense report. 

First, the employee goes to the website, powered by Vue.js for the frontend progressive JavaScript framework and Shoelace for the library of web components. The website is hosted via Firebase Hosting. The frontend invokes an HTTP function that triggers the execution of our business workflow, defined using the Workflows YAML syntax. 

Workflows is able to handle long-running operations without any additional code required, in our case we are asynchronously processing a set receipt files. Here, the Document AI connector directly calls the batch processing endpoint for service. This API returns a long-running operation: if you poll the API, the operation state will be “RUNNING” until it has reached a “SUCCEEDED” or “FAILED” state. You would have to wait for its completion. However, Workflows’ connectors handle such long-running operations, without you having to poll the API multiple times till the state changes. Here’s how we call the batch processing operation of the Document AI connector:

  - invoke_document_ai:
    call: googleapis.documentai.v1.projects.locations.processors.batchProcess
    args:
        name: ${"projects/" + project + "/locations/eu/processors/" + processorId}
        location: "eu"
        body:
            inputDocuments:
                gcsPrefix:
                    gcsUriPrefix: ${bucket_input + report_id}
            documentOutputConfig:
                gcsOutputConfig: 
                    gcsUri: ${bucket_output + report_id}
            skipHumanReview: true
    result: document_ai_response

Machine learning uses state of the art Vision and Natural Language Processing models to intelligently extract schematized data from documents with Document AI. As a developer, you don’t have to figure out how to fine tune or reframe the receipt pictures, or how to find the relevant field and information in the receipt. It’s Document AI’s job to help you here: it will return a JSON document whose fields are: line_item, currency, supplier_name, total_amount, etc. Document AI is capable of understanding standardized papers and forms, including invoices, lending documents, pay slips, driver licenses, and more.

A cloud function retrieves all the relevant fields of the receipts, and makes its own tallies, before submitting the expense report for approval to the manager. Another useful feature of Workflows is put to good use: Callbacks, that we introduced last year. In the workflow definition we create a callback endpoint, and the workflow execution will wait for the callback to be called to continue its flow, thanks to those two instructions:

  - create_callback:
    call: events.create_callback_endpoint
    args:
        http_callback_method: "POST"
    result: callback_details
...
- await_callback:
    try:
        call: events.await_callback
        args:
            callback: ${callback_details}
            timeout: 3600
        result: callback_request
    except:
        as: e
        steps:
            - update_status_to_error:
              ...

In this example application, we combined the intelligent capabilities of Document AI to transform complex image documents into usable structured data, with Cloud Functions for data transformation, process triggering, and callback handling logic, and Workflows enabled us to orchestrate the underlying business process and its service call logic.

Going further 

If you’re looking to make sense of your documents, turning dark data into structured information, be sure to check out what Document AI offers. You can also get your hands on a codelab to get started quickly, in which you’ll get a chance at processing handwritten forms. If you want to explore Workflows, quickstarts are available to guide you through your first steps, and likewise, another codelab explores the basics of Workflows. As mentioned earlier, for a concrete example, the source code of our smart expense application is available on Github. Don’t hesitate to reach out to us at @glaforge and @asrivas_dev to discuss smart scalable apps with us.

Demand for spoken audio content such as podcasts remains robust despite the proliferation of video services and other entertainment options for consumers. Shibin Li, Co-founder of Castbox, credits growth of the global podcast platform to the following: speed and availability, market-leading features, the proliferation of smart devices to deliver audio content, and activities — such as driving and working around the house — that make consuming video difficult.

Founded in 2016 and headquartered in Beijing, China, Castbox enables users to locate, access, and create spoken audio content. Available on iOS and Android, Castbox supports 50 million podcasts, on-demand radio programs, and audiobooks in 70 languages from 175 countries. The platform hosts about 2 million users per day and is the largest podcast platform on Android.

Castbox includes a range of features that build on its core service to provide a high-quality user experience. These features include curated podcast recommendations and in-audio search.

A combination of services

At its inception, Castbox relied on a combination of a multinational cloud services, as well as Google Cloud Platform (GCP) services, including the Google BigQuery analytics data warehouse and Cloud APIs to provide programmatic interfaces with Google Cloud services, and a range of services from the Google mobile development platform Firebase.

However, as Castbox matured and its user base expanded, the business increased its reliance on Google Cloud Platform and Firebase.

“We needed to access stable cloud services as we could not tolerate long periods of downtime that would compromise the user experience,” says Li. “Furthermore, we had to support up to 50,000 concurrent connections, and potentially more in future, without disruption.”

“Based on our analysis of the data in Google BigQuery, we can determine what type of content users are listening to, how long they like to listen to it, and when they like to listen to it. This allows us to recommend similar podcasts to each user based on the preferences he or she expressed, encouraging activity on and return visits to our platform.”

—Shibin Li, Co-Founder, Castbox

Competitive differentiation

Castbox also found machine learning-powered Google Cloud Platform APIs could help deliver features, such as in-audio search, that differentiate the podcast platform from its competitors. In addition, Firebase SDKs and Firebase A/B Testing would enable Castbox to create and analyze new applications, as well as make adjustments based on user feedback. Firebase Realtime Database would allow the business to support tens of thousands of concurrent user connections.

The diligence of the Google Cloud team in advising Li and her team about forthcoming products and services also swayed Castbox towards Google technologies. The business gained the opportunity with Google to join several programs that offered early access to Google innovations.

Signature in-audio search service

Castbox now uses Google Cloud Platform services in the Tokyo, Japan, and U.S. East regions. Cloud Speech-to-Text API plays a key role in delivering Castbox’s signature in-audio search service. This service enables users to search transcriptions of audio content on the platform for words or phrases. The search results incorporate the title of the podcast and the search term in context (for example, within the sentence or sentence excerpt in which it appears). Each use of the word or phrase is time-stamped so it can easily be found. The API enables Castbox developers to apply neural network algorithms to achieve audio-to-text conversion accuracy rates of greater than 96%, while search queries typically experience latency of just 50 milliseconds.

In addition, the latency of comparison data, converting audio to text, is only about 250 milliseconds, contributing to the processing of about 12 minutes worth of audio to text in just 10 minutes. “We can process about 20 hours of audio files in one day,” Li says. “This enables us to transcribe and index all the new episodes of a podcast in that period.”

50,000 concurrent connections

With Firebase Realtime Database, Castbox now supports up to 50,000 concurrent connections to its platform with an average latency per connection of just 10 milliseconds. “Firebase Realtime Database also allows us to continue operating in offline mode, which is extremely helpful if we experience any network disruptions,” Li explains. “When we come back online again, any data is simply synchronized with the database.”

Google BigQuery and the analytics capabilities of Firebase SDKs also enable Castbox to monitor and analyze user behaviors. “Based on our analysis of the data in Google BigQuery, we can determine what type of content users are listening to, how long they like to listen to it, and when they like to listen to it,” Li explains. “This allows us to recommend similar podcasts to each user based on the preferences he or she expressed, encouraging activity on and return visits to our platform.”

“Given our queries may span up to 40 days of data, we may be analyzing up to 1,200 GB at one time. We have no problem doing this with Google BigQuery.”

—Shibin Li, Co-Founder, Castbox

Castbox also uses its analyses of Google BigQuery data to amend banners and summaries on its platform to encourage users to listen to additional content. Furthermore, the service is prepared to make surprise recommendations of content to users based on the preferences and reactions of users with similar tastes.

“We analyze a pool of data growing at up to 30 GB per day,” Li explains. “Given our queries may span up to 40 days of data, we may be analyzing up to 1,200 GB at one time. We have no problem doing this with Google BigQuery.” These analyses also support Castbox’s decision to start creating original content, such as finance and economic news, for its platform.

Checking weekly changes

Castbox does not rely only on analyzing user data to deliver a high-quality experience. The business aggregates user feedback from emails and Google Play reviews to make weekly changes to its platform. It then uses Firebase A/B Testing to check whether these changes are met with a positive user response.

“We are extremely pleased with Google Cloud Platform and Firebase. We have been able to differentiate ourselves from our competitors and provide an attractive option for users at a time when content and entertainment options are exploding. We have a great opportunity with Google to continue to improve the value of our offering to users and build engagement and loyalty.”

—Shibin Li, Co-Founder, Castbox

Castbox’s positive experiences with Google Cloud Platform are encouraging the business to grow its use of the product. “We are trying to move some more services to Google Cloud Platform because it is very stable and scalable,” Li says. The business is keen to explore the capabilities of Cloud Pub/Sub to provide low latency messaging between applications, Cloud Spanner to deliver a distributed relational database service, and Cloud Dataflow to transform and enrich data in stream and batch modes.

“We are extremely pleased with Google Cloud Platform and Firebase,” Li concludes. “We have been able to differentiate ourselves from our competitors and provide an attractive option for users at a time when content and entertainment options are exploding. We have a great opportunity with Google to continue to improve the value of our offering to users and build engagement and loyalty.”

With Google Cloud, AirAsia is now a “data first” business that can capture, analyze, and report on rising volumes of data to address complex problems and increase revenue streams. The airline is also using AI-powered chatbots to streamline internal operations and provide a faster, more efficient service.

Google Cloud results

• Increases employee survey response rates by 30% and improved safety, scheduling, and employee orientation
• Enables movement towards a zero trust security model while reducing administration load on IT team
• Increases business agility by deploying faster and more frequently

Industries: Travel & Hospitality

Location: Malaysia

Become a digital airline powered by data and machine learning

AirAsia’s vision is simple: allow everyone to fly. Founded in 2001, the airline and sister company AirAsia X have grown to service 150+ destinations in 25 markets, using 274 aircraft to operate 11,000+ weekly flights from 23 hubs across the region. While the airline is known as a provider of low-cost airfares to locations across Asia, this is only one part of its value proposition. AirAsia also aims to deliver innovative, personalized products and services that meet the needs of each of its passengers.

Technology is key to AirAsia’s success and, with strong support from Group Chief Executive Officer Tony Fernandes, in 2016 the airline began a five-year program to become a data-first business and a digital airline.

“We wanted to better ensure we were using data correctly to become more agile, efficient, and customer oriented,” says Lye Kong Wei, Chief of Data Science, Group Head at AirAsia.

AirAsia needed technologies and services that could capture, process, analyze, and report on data, while delivering value for money and meeting its speed and availability requirements. The airline also wanted to minimize infrastructure management and system administration demands on its technology team.

Google Cloud the best fit

AirAsia realized only a cloud service could meet its needs and began evaluating the market. The airline then conducted a proof of concept and found Google Cloud was the best fit for its business. It was already familiar with Google Cloud, having deployed Google Workspace collaboration and productivity applications to its workforce in all countries except China. According to Kong Wei, products such as Forms, Docs, Sheets, and Gmail delivered a considerable improvement in collaboration between various departments, as well as streamlining and automating a range of processes.

The business was particularly excited by the potential of the BigQuery analytics data warehouse to power its digital transformation. “We knew data was a big part of making decisions in the future,” says Kong Wei. “So we needed a platform that could scale to meet our growing appetite for it. Google Cloud—in particular BigQuery—was ideal for this task.”

The AirAsia technology team was impressed by the ease and flexibility with which it could extract, transform, and load customer data from its systems, websites, and mobile applications into BigQuery for analysis. Data, reports, and dashboards were delivered and visualized through Looker Studio.

BigQuery also scaled seamlessly to support data growth and, as a managed service, required minimal administration from the airline’s technology team. “In addition, with BigQuery, we could process queries and requests much faster than previously and tackle more complex problems,” says Kong Wei. “As a result, we could be more innovative about realizing opportunities,” he adds, citing the benefits of being able to view and understand historical measures of booking curves—a measure of how long it took customers to book before a flight. This improves the airline’s ability to manage revenues.

A broad ecosystem

BigQuery and Looker Studio are just two components of a broad ecosystem—powered largely by Google Cloud services—deployed by AirAsia. Pub/Sub provides a scalable message queue that enables AirAsia developers to integrate systems hosted on Google Cloud or externally. Apache Airflow enables the business to create, schedule, and monitor workflows, while Cloud Composer manages the dependencies of PHP and related libraries. App Engine allows AirAsia personnel to develop and host web applications. “Thanks to App Engine, we’ve easily been able to create new applications, services, and APIs powered by the data we have been collecting,” says Kong Wei.

AirAsia is also running the middleware for its APIs in a managed Google Kubernetes Engine environment for increased scalability, resource optimization, and reliability, while Cloud Storage provides storage for data from a range of systems and sources. Dataflow enables the business to transform and process data in stream and batch modes from its website search page as customers look for flights.

Faster deployment and testing

The stability of Google Cloud service means AirAsia has a reliable base from which to launch new products and features. “If we have consistent reliability from our core systems—and Google Cloud incorporates monitoring tools such Cloud Monitoring that enable us to identify issues quickly—developers and product engineers can focus on turning ideas into reality,” says Kong Wei. “With a minimal number of people involved, we can very quickly transform an idea or thought process into a deliverable. Prior to Google Cloud, bringing those ideas to fruition would have been impossible.”

Robust security

AirAsia is also relying on Security Health Analytics, a product that integrates with Security Command Center, to identify misconfigurations and compliance violations in its Google Cloud resources and take action. Security Health Analytics ensures the airline’s budgets go to keeping customers’ travel costs low rather than recovering from security breaches.

The product enables AirAsia to check that resources are configured properly and are compliant with CIS benchmarks as its critical workloads run in Google Kubernetes Engine and App Engine.

“Being able to go to the new Security Health Analytics dashboard eliminates the guesswork of what we have running and if it is secure,” says Muhammad Faeez Bin Azmi, Information Security and Automation Solution Architect. “Now anyone on our team, even non-security professionals, can go to this dashboard and see a list of the misconfigured assets and compliance violations across all of our Google Cloud resources. We can also see the severity of misconfigurations, which helps us prioritize our response.”

“Security Health Analytics has really helped us reduce the amount of time we spend trying to figure out what’s wrong with our resources. It’s allowed us to use our time more effectively to identify and resolve more security issues than we could before.”

A new identity solution

AirAsia had also used a legacy on-premises directory for many years. However, as the company grew and expanded to new markets and regions, it had to manage multiple servers across a number of on-premises data centers and the public cloud, which proved costly and time-consuming.

Its Allstars—the airline’s name for its employees—needed to easily access a number of legacy on-premises apps in addition to a growing number of SaaS apps. As a business, it also needed a more seamless integration between its HR system of record and its identity solution for user provisioning and life cycle management. Solving these challenges with its existing on-premises directory was simply not feasible.

AirAsia brought up its identity concerns with the Google Cloud team, and after a number of conversations, decided to deploy Cloud Identity, Google’s cloud-based Identity and Access Management solution, to help address the identity challenges it was facing.

The airline chose Cloud Identity for a number of reasons—first, it was eager to move to the cloud as quickly as possible. Moving identity management to the cloud was a key enabler of this and the airline’s broader digital transformation. Managing identities from the cloud also enabled the airline to have a single identity and set of credentials for each employee, which they could use to access all the applications they need to be productive, both in the cloud and on-premises.

In addition, deploying Cloud Identity was a key step towards enabling the zero trust security model, which the airline felt was the best approach to strengthen its security posture and fight modern threats. Cloud Identity also integrated seamlessly with its existing technologies, which include not only Google Cloud products like Google Workspace and Chrome OS but also third-party tools like Citrix, Papercut, and others.

And finally, Cloud Identity offered significant cost and resource savings. With Cloud Identity in place, AirAsia’s IT department could spend less time worrying about managing multiple on-premises directory servers and and could instead focus on delivering value to Allstar employees.

Machine learning employed to increase ancillary revenue

In March 2018, AirAsia established the groundwork to use machine learning to optimize pricing for a range of services and began by using AI Platform to sort and predict demand for ancillary services such as baggage, seats, and meals. “By using AI Platform, we can sort based on data about history to predict the future,” says Kong Wei.

Dialogflow in wide use

With Google Cloud well established within the business, AirAsia is using Dialogflow, a voice and conversational interface development suite (and one of the core components of Contact Center AI) that enables businesses to create engaging AI powered voice- and text-based interfaces such as chatbots and voice apps—to streamline operations and reduce costs

“Tony Fernandes, our Group Chief Executive Officer, motivated us to create a range of metrics that would enable us to respond more quickly and efficiently to customers,” says Yuashini Vellasamy, Product Manager at AirAsia.

This resulted in the initial deployment of Dialogflow in AirAsia’s operational areas, from crew scheduling to internal business tasks.

With Dialogflow, AirAsia is able to provide pilots, crew, catering, and other teams with flight times, capacity and any other relevant information about their assignments. Another bot accepts medical certificates and updates from pilots and crew members unable to make rostered assignments and switches those assignments to other pilots and crews. “This improves our on-time performance and operational efficiency, as we are able to optimize the creation of crew schedules,” says Vellasamy.

AirAsia also uses Dialogflow to power a “safety bot” for airport ground staff. “We tried a lot of tools to encourage staff to advise us when they saw a safety issue—but employee usage was low,” says Vellasamy. “However, when we set up the safety bot, which asks users just four questions about an issue they have observed, we saw an increase in employee interaction, which helps ensure proper safety measures.

Beyond crew optimization and overall safety, AirAsia’s HR department in Asia uses a Dialogflow-powered bot to run a post-orientation engagement program for new employees. The bot follows up with employees about issues from parking to AirAsia’s onboarding buddy program, saving HR employees time from scheduling one-on-one meetings. Dialogflow also powered an employee satisfaction survey of 3,000 team members, resulting in a 30% increase in response rate compared to the previous year.

One of Dialogflow’s benefits is its language coverage, which the airline’s marketing team uses to target customers and prospects across a region with diverse languages and cultures. “For weekly campaigns and promotions, our sales and marketing teams simply update the campaign materials and the changes are reflected automatically in Dialogflow and consequently to users,” says Vellasamy. “This enables us to quickly scale and reach our customers globally.”

An upward trajectory

AirAsia is poised to reap further benefits from Google Cloud as its deployment matures. “On the data side, we’re more advanced, while on the application and programming side we have a long way to go,” says Kong Wei. “We’re on a trajectory upwards—we’re looking at a lot of new ideas and how to embrace and deploy them so we can further our data-first and digital airline agendas.”