Businesses around the globe are realizing the benefits of replacing legacy data silos with cloud-based enterprise data warehouses, including easier collaboration across business units and access to insights within their data that were previously unseen. However, bringing data from numerous disparate data sources into a single data warehouse requires you to develop pipelines that ingest data from these various sources into your enterprise data warehouse. Historically, this has meant that data engineering teams across the organization procure and implement various tools to do so. But this adds significant complexity to managing and maintaining all these pipelines and makes it much harder to effectively scale these efforts across the organization. Developing enterprise-grade, cloud-native pipelines to bring data into your data warehouse can alleviate many of these challenges. But, if done incorrectly, these pipelines can present new challenges that your teams will have to spend their time and energy addressing. 

Developing cloud-based data ingestion pipelines that replicate data from various sources into your cloud data warehouse can be a massive undertaking that requires significant investment of staffing resources. Such a large project can seem overwhelming and it can be difficult to identify where to begin planning such a project. We have defined the following principles for data pipeline planning to begin the process. These principles are intended to help you answer key business questions about your effort and begin to build data pipelines that address your business and technical needs. Each section below details a principle of data pipelines and certain factors your teams should consider as they begin developing their pipelines.

Principle 1: Clarify your objectives

The first principle to consider for pipeline development is clarify your objectives. This can be broadly defined as taking a holistic approach to pipeline development that encompasses requirements from several perspectives: technical teams, regulatory or policy requirements, desired outcomes, business goals, key timelines, available teams and their skill sets, and downstream data users. Clarifying your objectives clearly identifies and defines requirements from each key stakeholder at the beginning of the process and continually checks development against these requirements to ensure the pipelines built will meet these requirements.This is done by first clearly defining the desired end state for each project in a way that addresses a demonstrated business need of downstream data users. Remember that data pipelines are almost always the means to accomplish your end state, rather than the end state itself. An example of an effectively defined end-state is “enabling teams to gain a better understanding of our customers by providing access to our CRM data within our cloud data warehouse” rather than “move data from our CRM to our cloud data warehouse”. This may seem like a merely semantic difference, but framing the problem in terms of business needs helps your teams make technical decisions that will best meet these needs. 

After clearly defining the business problem you are trying to solve, you should facilitate requirement gathering from each stakeholder and use these requirements to guide the technical development and implementation of your ingestion pipelines. We recommend gathering stakeholders from each team, including downstream data users, prior to development to gather requirements for the technical implementation of the data pipeline. These will include critical timelines, uptime requirements, data update frequency, data transformation, DevOps needs, and security, policy, or regulatory requirements by which a data pipeline must meet.

Principle 2: Build your team

The second principle to consider for pipeline development is build your team. This means ensuring you have the right people with the right skills available in the right places to develop, deploy, and maintain your data pipelines. After you have gathered your pipeline requirements, you can begin to develop a summary architecture that will be used to build and deploy your data pipelines. This will help you identify the human talent you will need to successfully build, deploy, and manage these data pipelines and identify any potential shortfalls that would require additional support from either third-party partners or new team members.

Not only do you need to ensure you have the right people and skill sets available in aggregate, but these individuals need to be effectively structured to empower them to maximize their abilities. This means developing team structures that are optimized for each team’s responsibilities and their ability to support adjacent teams as needed.

This also means developing processes that prevent blockers to technical development whenever possible, such as ensuring that teams have all of the appropriate permissions they need to move data from the original source to your cloud data warehouse without violating the concept of least privilege. Developers need access to the original data source (depending on your requirements and architecture) in addition to the destination data warehouse. Examples of this are ensuring that developers have access to develop and/or connect to a Salesforce Connected App or read access to specific Search Ads 360 data fields.

Principle 3: Minimize time to value

The third principle to consider for pipeline development is minimize time to value. This means considering the long-term maintenance burden of a data pipeline prior to developing and deploying it in addition to being able to deploy a minimum viable pipeline as quickly as possible. Generally speaking, we recommend the following approach to building data pipelines to minimize their maintenance burden: Write as little code as possible. Functionally, this can be implemented by:

1. Leveraging interface-based data ingestion products whenever possible. These products minimize the amount of code that requires ongoing maintenance and empower users who aren’t software developers to build data pipelines. They can also reduce development time for data pipelines, allowing them to be deployed and updated more quickly. 

• Products like Google Data Transfer Service and Fivetran allow for managed data ingestion pipelines by any user to centralize data from SaaS applications, databases, file systems, and other tooling. With little to no code required, these managed services enable you to connect your data warehouse to your sources quickly and easily.
• For workloads managed by ETL developers and data engineers, tools like Google Cloud’s Data Fusion provide an easy-to-use visual interface for designing, managing and monitoring advanced pipelines with complex transformations.

2. Whenever interface-based products or data connectors are insufficient, use pre-existing code templates. Examples of this include templates available for Dataflow that allow users to define variables and run pipelines for common data ingestion use cases, and the Public Datasets pipeline architecture that our Datasets team uses for onboarding.

3. If neither of these options are sufficient, utilize managed services to deploy code for your pipelines. Managed services, such as Dataflow or Dataproc, eliminate the operational overhead of managing pipeline configuration by automatically scaling pipeline instances within predefined parameters.

Principle 4: Increase data trust and transparency

The fourth principle to consider for pipeline development is increase data trust and transparency. For the purposes of this document, we define this as the process of overseeing and managing data pipelines across all tools. Numerous data ingestion pipelines that each leverage different tools or are not developed under a coordinated management plan can result in “tech sprawl”, which significantly increases the management overhead of data ingestion pipelines as the quantity of data pipelines increases. This becomes especially cumbersome if you are subject to service-level agreements, or legal, regulatory, or policy requirements for overseeing data pipelines. Preventing tech sprawl is, by far, the best strategy for dealing with it by developing streamlined pipeline management processes that automate reporting. Although this can theoretically be achieved by building all of your data pipelines using a single cloud-based product, we do not recommend doing so because it prevents you from taking advantage of features and cost optimizations that come with choosing the best product for your use case. 

A monitoring service such as Google Cloud Monitoring Service or Splunk that automates metrics, events, and metadata collection from various products, including those hosted in on-premise and hybrid computing environments, can help you centralize reporting and monitoring of your data pipelines. A metadata management tool such as Google Cloud’s Data Catalog or Informatica’s Enterprise Data Catalog can help you better communicate the nuances of your data so users better understand which data resources are best fit for a given use case. This significantly reduces your pipeline’s governance burden by eliminating manual reporting processes that often result in inaccuracies or lagging updates.

Principle 5: Manage costs

The fifth principle to consider for pipeline development is manage costs. This encompasses both the cost of cloud resources and the staffing costs necessary to design, develop, deploy, and maintain your cloud resources. We believe that your goal should not necessarily be to minimize cost, but rather maximizing the value of your investment. This means maximizing the impact of every dollar spent by minimizing waste in cloud resource utilization and human time. There are several factors to consider when it comes to managing costs:

• Use the right tool for the job – Different data ingestion pipelines will have different requirements for latency, uptime, transformations, etc. Similarly, different data pipeline tools have different strengths and weaknesses. Choosing the right tool for each data pipeline can help your pipelines operate significantly more efficiently. This can reduce your overall cost, free up staffing time to focus on the most impactful projects, and make your pipelines much more efficient.
• Standardize resource labeling –  Implement and utilize a consistent labeling schema across all tools and platforms to have the most comprehensive view of your organization’s spending. One example is requiring all resources to be labeled by the cost center or team at time of creation. Consistent labeling allows you to monitor your spend across different teams and calculate the overall value of your cloud spending.
• Implement cost controls – If available, leverage cost controls to prevent errors that result in unexpectedly large bills. 
• Capture cloud spend – Capture your spend on all cloud resource utilization for internal analysis using a cloud data warehouse and a data visualization tool. Without it, you won’t understand the context of changes in cloud spend and how they correlate with changes in business.
• Make cost management everyone’s job – Managing costs should be part of the responsibilities of everyone who can create or utilize cloud resources. To do this well, we recommend making cloud spend reporting more transparent internally and/or implementing chargebacks to internal cost centers based on utilization.

Long-term, the increased granularity in cost reporting available within Google Cloud can help you better measure your key performance indicators. You can shift from cost-based reporting (i.e. – “We spent $X on BigQuery storage last month”) to value-based reporting (i.e. – “It costs $X to serve customers who bring in $Y revenue”). 

To learn more about managing costs, check out Google Cloud’s “Understanding the principles of cost optimization” white paper.

Principle 6: Leverage continually improving services

The sixth principle is leverage continually improving services. Cloud services are consistently improving their performance and stability, even if some of these improvements are not obvious to users. These improvements can help your pipelines run faster, cheaper, and more consistently over time. You can take advantage of the benefits of these improvements by:

• Automating both your pipelines and pipeline management: Not only should data pipelines be automated, but almost all aspects of managing your pipelines can also be automated. This includes pipeline/data lineage tracking, monitoring, cost management, scheduling, access management and more. This helps reduce long-term operational costs of each data pipeline that can significantly alter your value proposition and prevent any manual configurations from negating the benefits of later product improvements.
• Minimizing pipeline complexity whenever possible: While ingestion pipelines are relatively easy to develop using UI-based or managed services, they also require continued maintenance as long as they are in use. The most easily maintained data ingestion pipelines are typically the ones that minimize complexity and leverage automatic optimization capabilities. Any transformation in a data ingestion pipeline is a manual optimization of the pipeline that may struggle to adapt or scale as the underlying services improve. You can minimize the need for such transformations by building ELT (extract, load, transform) pipelines rather than ETL (extract, transform, load) pipelines. This pushes transformations down to the data warehouse that is use a specifically optimized query engine to transform your data rather than manually configured pipelines.

Next steps

If you’re looking for more information about developing your cloud-based data platform, check out our Build a modern, unified analytics data platform whitepaper. You can also visit our data integration site to learn more and find ways to get started with your data integration journey.

Once you’re ready to begin building your data ingestion pipelines, learn more about how Cloud Data Fusion and Fivetran can help you make sure your pipelines address these principles.

We are excited to announce that Google has been named a Leader in The Forrester Wave™: Streaming Analytics, Q2 2021 report. Thank you to our strong community of customers and partners for working with us to deliver a customer focused product. We believe Forrester’s recognition is an acknowledgement of our leadership across an integrated set of capabilities that rely on data to drive transformation. We were also honored to be named a leader in The Forrester Wave™: Cloud Data Warehouse, Q1 2021. 

Forrester gave Dataflow a score of 5 out of 5 across 12 different criteria and according to the report: “Google Cloud Dataflow has strengths in data sequencing, advanced analytics, performance, and high-availability. Google Dataflow’s sweet spot is for enterprises that have a preponderance of real-time data generated on Google Cloud Platform or wish to simplify all data processing by using a single platform that unifies both streaming and batch jobs.” 

Harnessing the power of real-time data

The speed with which businesses are able to respond to change is the difference between those that successfully navigate the future and those that get left behind. In order to accelerate their digital transformation, reimagine their business and leverage the power of real-time data,  today’s data leaders require a streaming analytics platform that provides both depth and breadth.  

Cloud Pub/Sub and Cloud Dataflow, based on more than a decade of experience in internet scale systems for Google’s own needs, provide customers with a reliable, scalable, performant platform. In addition, we’ve designed these products for ease of use to make streaming analytics accessible to more users, which is why customers such as Sky and others from across all industries use Dataflow to run streaming analytics workloads.

5 out of 5 across key streaming analytics criteria

While Forrester gave Dataflow a score of 5 out of 5 in 12 criteria, the product achieved the highest possible scores in areas that are top of mind for our customers.

We continue to be focused on solving problems that matter to you. For example, just in the last month we announced Dataflow Prime and  Auto Sharding for BigQuery – two new auto tuning capabilities that bring efficiency and simplicity to your streaming pipelines. 

Dataflow achieves highest score possible in strategy 

With Google, organizations gain an industry leading product and a partner that has the vision and strategy to help you tackle new business challenges and provide delightful experiences to your customers.

In summary, we are honored to be a Leader in The Forrester Wave™, Streaming Analytics, and look forward to continuing to innovate and partner with you on your digital transformation journey. 

Download the full report: The Forrester Wave™: Streaming Analytics, Q2 2021 and check out these smart analytics reference patterns. To learn more about Dataflow, visit our website and get to know the product by taking an interactive tutorial. You can also watch recordings from the Data Cloud Summit event (May 2021), where we provided an in-depth view of new product innovations in Dataflow and other data analytics products.

As data environments become more complex, companies are turning to streaming analytics solutions that analyze data as it’s ingested and deliver immediate, high-value insights into what is happening now. These insights enable decision makers to act in real time to take advantage of opportunities or respond to issues as they occur.

While understanding what is happening now has great business value, forward-thinking companies are taking things a step further, using real-time analytics integrated with artificial intelligence (AI) and business intelligence (BI) to answer the question, “what might happen in the future?” Arkansas-based Tyson Foods has embraced AI/BI analytics to enable predictive insights that unlock new opportunities and drive future growth.

Creating a digital twin for connected intelligence company wide

Before using AI/BI, Tyson’s analytics capabilities consisted of traditional BI solutions focused on KPIs and simplifying data so that humans could understand it. Tyson wanted to leverage its data to uncover ways to improve current processes and grow its business. But with BI alone, Tyson struggled to use data to run the simulations and scenarios essential to make educated decisions. To keep growing, it had to embrace the complexity of its data, building ways to analyze it and use it to inform decision making. 

Tyson’s on-premises analytics solutions limited its ability to be aggressive and make intelligent, timely, prescriptive decisions. The solution was to create a digital twin to scale optimizations within business processes, moving from local optimizations to system-wide connected optimizations. Doing so meant shifting entirely to cloud computing, with an initial focus on building the ingestion component of the digital twin platform.

Investing in a digital twin enabled Tyson to accelerate new capabilities like supply chain simulation “what-if” scenarios, prescriptive price elasticity recommendations, and improvement of customer intimacy. 

Solving the ingestion problem for faster time to insights

Before its migration to Google Cloud, analytics projects that Tyson suffered from uncertainty over how to obtain the data. This problem was prolific and caused project times to be extended for weeks or even months due to the need to write and support one-off data ingestion processes at the front end. This problem also prevented the IT team from delivering analytics solutions fast enough for the business to take full advantage of them. 

To solve this analytics problem, the team created Data Ingestion Compute Engine (DICE). DICE is a Google Cloud-hosted, open-source, cloud-native ingestion platform developed to provide configuration-based, no-ops, code-free ingestion from disparate enterprise data systems, both internal and external. It is centered on three high-level goals:

1. Accelerate the speed of delivery of IT analytics solutions
2. Enable growth of IT capabilities to produce meaningful insight
3. Reduce long-term total cost of ownership for ingestion solutions

Creating DICE ingestion platform with Google Cloud services

Teams use DICE to set up secure data ingestion jobs in minutes without having to manage complex connections or write, deploy, and support their own code. DICE enables unbound scale, highly parallel processing, DevSecOps, open source, and the implementation of Lambda Data Architecture.

A DICE job is the logical unit of work in the DICE platform, consisting of immutable and mutable configurations persisted as JSON documents stored in Firestore. The job exists as an instruction set for the DICE data engine, which is Apache Beam running Dataflow to instruct which data to pull, how to pull it, how often to pull it, how to process it, when it changes, and where to direct it.

Two of DICE’s primary layers include the metadata engine and the data engine. The metadata engine is responsible for the creation and management of DICE job configuration and orchestration. It is made up of many microservices that interact with multiple Google Cloud services, including the job configuration creation API, job build configuration helper API, and job execution scheduler API.

The data engine is responsible for the physical ingestion of data, the change detection processing of that data, and the delivery of that data to specified targets. The data engine is Java code that uses the Apache Beam unified programming model and runs in Dataflow. It is comprised of streaming, jobs, and Dataflow flex template batch jobs. Logically, the data engine is segmented across three layers: the inbound processing layer, the DICE file system layer, and the target processing layer, which takes the data from the DICE file system and moves it to targets.

Rolling DICE for thousands of ingestion jobs each day

DICE was first deployed to a production environment in November 2019, and just two years later, it has more than 3,000 data ingestion jobs from more than a hundred disparate data systems, both internal and external to Tyson Foods. Most of these jobs run multiple times a day. On a daily basis the DICE environment sees more than 25,000 Dataflow jobs running and an average of 3.25 terabytes of new data being ingested.

DICE supports ingestion from many different types of technologies, including BigQuery, SQL Server, SAP HANA, Postgres, Oracle, MySQL, Db2, various types of file systems, and FTP servers. Additionally, DICE supports target platform technologies for ingestion jobs that include multiple JDBC targets, multiple file system targets, and BigQuery and queue-based store and forward technologies. 

The platform continues to see linear growth of DICE jobs, all while keeping platform costs relatively flat. With increasing demand for the platform, Tyson’s IT team is constantly enhancing DICE to support new sources and targets.

This intelligent platform keeps adding new value and makes it simple for Tyson to take advantage of its data. This innovation is a necessity in this fast-changing world of digital business in which companies must transform a high volume of complex data into actionable insight.

AirAsia’s vision is simple: allow everyone to fly. Founded in 2001, the airline and sister company AirAsia X have grown to service 150+ destinations in 25 markets, using 274 aircraft to operate 11,000+ weekly flights from 23 hubs across the region. While the airline is known as a provider of low-cost airfares to locations across Asia, this is only one part of its value proposition. AirAsia also aims to deliver innovative, personalized products and services that meet the needs of each of its passengers.

Technology is key to AirAsia’s success and, with strong support from Group Chief Executive Officer Tony Fernandes, in 2016 the airline began a five-year program to become a data-first business and a digital airline.

“We wanted to better ensure we were using data correctly to become more agile, efficient, and customer oriented,” says Lye Kong Wei, Chief of Data Science, Group Head at AirAsia.

AirAsia needed technologies and services that could capture, process, analyze, and report on data, while delivering value for money and meeting its speed and availability requirements. The airline also wanted to minimize infrastructure management and system administration demands on its technology team.

“We knew data was a big part of making decisions in the future. So we needed a platform that could scale to meet our growing appetite for it. Google Cloud—in particular BigQuery—was ideal for this task.”

—Lye Kong Wei, Chief of Data Science, Group Head, AirAsia

Google Cloud the best fit

AirAsia realized only a cloud service could meet its needs and began evaluating the market. The airline then conducted a proof of concept and found Google Cloud was the best fit for its business. It was already familiar with Google Cloud, having deployed G Suite collaboration and productivity applications to its workforce in all countries except China. According to Kong Wei, products such as Forms, Docs, Sheets, and Gmail delivered a considerable improvement in collaboration between various departments, as well as streamlining and automating a range of processes.

The business was particularly excited by the potential of the BigQuery analytics data warehouse to power its digital transformation. “We knew data was a big part of making decisions in the future,” says Kong Wei. “So we needed a platform that could scale to meet our growing appetite for it. Google Cloud—in particular BigQuery—was ideal for this task.”

“With BigQuery, we could process queries and requests much faster than previously and tackle more complex problems.”

—Lye Kong Wei, Chief of Data Science, Group Head, AirAsia

The AirAsia technology team was impressed by the ease and flexibility with which it could extract, transform, and load customer data from its systems, websites, and mobile applications into BigQuery for analysis. Data, reports, and dashboards were delivered and visualized through Google Data Studio.

BigQuery also scaled seamlessly to support data growth and, as a managed service, required minimal administration from the airline’s technology team. “In addition, with BigQuery, we could process queries and requests much faster than previously and tackle more complex problems,” says Kong Wei. “As a result, we could be more innovative about realizing opportunities,” he adds, citing the benefits of being able to view and understand historical measures of booking curves—a measure of how long it took customers to book before a flight. This improves the airline’s ability to manage revenues.

A broad ecosystem

BigQuery and Data Studio are just two components of a broad ecosystem—powered largely by Google Cloud services—deployed by AirAsia. Pub/Sub provides a scalable message queue that enables AirAsia developers to integrate systems hosted on Google Cloud or externally. Apache Airflow enables the business to create, schedule, and monitor workflows, while Cloud Composer manages the dependencies of PHP and related libraries. App Engine allows AirAsia personnel to develop and host web applications. “Thanks to App Engine, we’ve easily been able to create new applications, services, and APIs powered by the data we have been collecting,” says Kong Wei.

AirAsia is also running the middleware for its APIs in a managed Google Kubernetes Engine environment for increased scalability, resource optimization, and reliability, while Cloud Storage provides storage for data from a range of systems and sources. Dataflow enables the business to transform and process data in stream and batch modes from its website search page as customers look for flights.

“With a minimal number of people involved, we can very quickly transform an idea or thought process into a deliverable. Prior to Google Cloud, bringing those ideas to fruition would have been impossible.”

—Lye Kong Wei, Chief of Data Science, Group Head, AirAsia

Faster deployment and testing

The stability of Google Cloud service means AirAsia has a reliable base from which to launch new products and features. “If we have consistent reliability from our core systems—and Google Cloud incorporates monitoring tools such Cloud Monitoring that enable us to identify issues quickly—developers and product engineers can focus on turning ideas into reality,” says Kong Wei. “With a minimal number of people involved, we can very quickly transform an idea or thought process into a deliverable. Prior to Google Cloud, bringing those ideas to fruition would have been impossible.”

Robust security

AirAsia is also relying on Security Health Analytics, a product that integrates with Security Command Center, to identify misconfigurations and compliance violations in its Google Cloud resources and take action. Security Health Analytics ensures the airline’s budgets go to keeping customers’ travel costs low rather than recovering from security breaches.

The product enables AirAsia to check that resources are configured properly and are compliant with CIS benchmarks as its critical workloads run in Google Kubernetes Engine and App Engine.

“Being able to go to the new Security Health Analytics dashboard eliminates the guesswork of what we have running and if it is secure,” says Muhammad Faeez Bin Azmi, Information Security and Automation Solution Architect. “Now anyone on our team, even non-security professionals, can go to this dashboard and see a list of the misconfigured assets and compliance violations across all of our Google Cloud resources. We can also see the severity of misconfigurations, which helps us prioritize our response.”

“Security Health Analytics has really helped us reduce the amount of time we spend trying to figure out what’s wrong with our resources. It’s allowed us to use our time more effectively to identify and resolve more security issues than we could before.”

A new identity solution

AirAsia had also used a legacy on-premises directory for many years. However, as the company grew and expanded to new markets and regions, it had to manage multiple servers across a number of on-premises data centers and the public cloud, which proved costly and time-consuming.

Its Allstars—the airline’s name for its employees—needed to easily access a number of legacy on-premises apps in addition to a growing number of SaaS apps. As a business, it also needed a more seamless integration between its HR system of record and its identity solution for user provisioning and life cycle management. Solving these challenges with its existing on-premises directory was simply not feasible.

AirAsia brought up its identity concerns with the Google Cloud team, and after a number of conversations, decided to deploy Cloud Identity, Google’s cloud-based Identity and Access Management solution, to help address the identity challenges it was facing.

The airline chose Cloud Identity for a number of reasons—first, it was eager to move to the cloud as quickly as possible. Moving identity management to the cloud was a key enabler of this and the airline’s broader digital transformation. Managing identities from the cloud also enabled the airline to have a single identity and set of credentials for each employee, which they could use to access all the applications they need to be productive, both in the cloud and on-premises.

In addition, deploying Cloud Identity was a key step towards enabling the zero trust security model, which the airline felt was the best approach to strengthen its security posture and fight modern threats. Cloud Identity also integrated seamlessly with its existing technologies, which include not only Google Cloud products like G Suite and Chrome OS but also third-party tools like Citrix, Papercut, and others.

And finally, Cloud Identity offered significant cost and resource savings. With Cloud Identity in place, AirAsia’s IT department could spend less time worrying about managing multiple on-premises directory servers and and could instead focus on delivering value to Allstar employees.

Machine learning employed to increase ancillary revenue

In March 2018, AirAsia established the groundwork to use machine learning to optimize pricing for a range of services and began by using AI Platform to sort and predict demand for ancillary services such as baggage, seats, and meals. “By using AI Platform, we can sort based on data about history to predict the future,” says Kong Wei.

Dialogflow in wide use

With Google Cloud well established within the business, AirAsia is using Dialogflow, a voice and conversational interface development suite (and one of the core components of Contact Center AI) that enables businesses to create engaging AI powered voice- and text-based interfaces such as chatbots and voice apps—to streamline operations and reduce costs

“Tony Fernandes, our Group Chief Executive Officer, motivated us to create a range of metrics that would enable us to respond more quickly and efficiently to customers,” says Yuashini Vellasamy, Product Manager at AirAsia.

This resulted in the initial deployment of Dialogflow in AirAsia’s operational areas, from crew scheduling to internal business tasks.

With Dialogflow, AirAsia is able to provide pilots, crew, catering, and other teams with flight times, capacity and any other relevant information about their assignments. Another bot accepts medical certificates and updates from pilots and crew members unable to make rostered assignments and switches those assignments to other pilots and crews. “This improves our on-time performance and operational efficiency, as we are able to optimize the creation of crew schedules,” says Vellasamy.

AirAsia also uses Dialogflow to power a “safety bot” for airport ground staff. “We tried a lot of tools to encourage staff to advise us when they saw a safety issue—but employee usage was low,” says Vellasamy. “However, when we set up the safety bot, which asks users just four questions about an issue they have observed, we saw an increase in employee interaction, which helps ensure proper safety measures.

Beyond crew optimization and overall safety, AirAsia’s HR department in Asia uses a Dialogflow-powered bot to run a post-orientation engagement program for new employees. The bot follows up with employees about issues from parking to AirAsia’s onboarding buddy program, saving HR employees time from scheduling one-on-one meetings. Dialogflow also powered an employee satisfaction survey of 3,000 team members, resulting in a 30% increase in response rate compared to the previous year.

One of Dialogflow’s benefits is its language coverage, which the airline’s marketing team uses to target customers and prospects across a region with diverse languages and cultures. “For weekly campaigns and promotions, our sales and marketing teams simply update the campaign materials and the changes are reflected automatically in Dialogflow and consequently to users,” says Vellasamy. “This enables us to quickly scale and reach our customers globally.”

An upward trajectory

AirAsia is poised to reap further benefits from Google Cloud as its deployment matures. “On the data side, we’re more advanced, while on the application and programming side we have a long way to go,” says Kong Wei. “We’re on a trajectory upwards—we’re looking at a lot of new ideas and how to embrace and deploy them so we can further our data-first and digital airline agendas.”