Mercari’s Big Leap: Supercharging Growth with Google Cloud’s BigQuery
1127
Of your peers have already read this article.
6:30 Minutes
The most insightful time you'll spend today!
When peer-to-peer marketplace Mercari came to the US in 2014, it had its work cut out for it. Surrounded by market giants like eBay, Craigslist, and Wish, Mercari needed to carve out an approach to compete for new users. Furthermore, Mercari wanted to build a network for buyers and sellers to return to, rather than a site for individual specialty purchases.
As an online marketplace that connects millions of people across the U.S. to shop and sell items of value no longer being used, Mercari is built for the everyday shopper and casual seller. Two teams, Machine Learning (ML) team and Marketing Technology specialists, both led by Masumi Nakamura, Mercari VP of Engineering, saw an opportunity to supercharge Mercari’s growth in the US by leveraging their first-party data in BigQuery and connecting predictive models built in Google Cloud directly to marketing channels, such as churn predictions and item recommendations for email campaigns, and LTV predictions to optimize paid media. Churn predictions could be used to target marketing communications, and item recommendations could be used to personalize the content of those communications at the user level. By fully utilizing cloud computing services, they could grow sustainably and flexibly, focusing their team’s efforts where they belonged — user understanding and personalized marketing.
In 2018, the Mercari US team engaged GrowthLoop, formerly Flywheel Software, experts in leveraging first-party customer data for business growth. Working exclusively in Google Cloud and BigQuery, GrowthLoop helped Masumi transform Marketing Technology at Mercari in the US.
Use cases: challenges
Masumi and the ML team’s primary goal aimed to reduce churn across buyers and sellers. Customers would make an initial purchase, but repurchase and resale rates were lower than the team hoped for. The ML team, led by Masumi, was confident that if they could get customers to make a second and third purchase, they could drive strong lifetime value (LTV).
Despite the team’s robust data science capabilities and investments in a data warehouse (BigQuery), they were missing the ability to streamline efforts for efficient audience segmentation and targeting. Like most companies looking to utilize data for marketing, the team at Mercari had to engage with engineering in order to build out customer segments for campaign launches and testing. From start to finish, launching a single campaign could take three months.
In short, Mercari needed a way to speed up the process across the teams at Mercari. How could they turn the team’s predictions into active marketing experiments with greater velocity and agility?
Solution: BigQuery and GrowthLoop supercharge growth across the customer lifecycle
With their strong data engineering foundation and BigQuery already in place, the Mercari team began addressing their needs step-by-step. First, they used predictions to identify retention features, then built out initial segment definitions based on those features. From there, the team designed and launched experiments and measured their performance, refining as they went. By providing Mercari’s marketing team with the ability to build their own customer lists that leveraged predictive models without requiring continuous support from other teams’ data engineers and business intelligence analysts, GrowthLoop enabled them to address churn and acquisition with a single, self-serve solution.
The dynamic duo: GrowthLoop and BigQuery
- Customer 360: GrowthLoop enabled Mercari to combine their data sources into a single view of their customers in BigQuery, then connected them to marketing and sales channels via GrowthLoop’s platform. Notably, Mercari is able to leverage its own complex data model, which was ideal for a two-sided marketplace. This is shown in the “Collect & Transform” stage in the architecture diagram above.
- Predictive models: GrowthLoop activated predictions that had been snapshotted by Mercari’s team in BigQuery. The Mercari ML team used Jupyter notebooks offering part of Google Vertex AI Workbench to build user churn and customer lifetime value (CLTV) prediction models, then productionized them using Cloud Composer to deploy Airflow DAGs, which wrote the predictions back to BigQuery for targeting, and triggered exports to destination channels using Pub/Sub. This is shown in the “Intelligence” stage of the architecture diagram above.
- Extensible measurement and data visualization: Since GrowthLoop writes all audience data back to BigQuery, the Mercari analytics team can conduct performance analysis on metrics from revenue to retention. They are able to use GrowthLoop’s performance visualization in-app, but they are also able to create custom data visualizations with Looker Studio. This is also shown in the “Intelligence” stage of the architecture diagram.
- Seamless routing and activation: With GrowthLoop’s audience platform connected directly to Customer 360 and the predictive model’s results in BigQuery, the marketing team is able to launch and sync audiences and their personalization attributes across all of Mercari’s major marketing, sales and product channels, such as Braze, Google Ads and other destinations. This is shown in the “Routing” and “Activate” stage of the architecture diagram.
“Being able to measure what you’re doing – that results-based orientation – is key. The thing that I like most about GrowthLoop is that you brought a really fundamental way of thinking which was very feedback-based and open to experimenting but within reason. With other products, that feedback loop isn’t so built in that it’s very easy to get lost.”– Masumi Nakamura, VP of Engineering at Mercari
Predictive modeling puts the burn on churn
In collaboration with GrowthLoop, Mercari began analyzing user data in BigQuery via Vertex AI Workbench to identify patterns across churned customers. The teams evaluated a range of attributes like the customer acquisition channel, categories browsed or purchased from, and whether or not they had any saved searches while shopping. Comparing various models and performance metrics, the teams selected the best model for accurately predicting when a buyer or seller was at risk to churn. For sellers, they evaluated audience members by the time elapsed since their last sale – for buyers, the time since their last purchase.
These churn prediction scores could then be applied to data pipelines that would feed into GrowthLoop’s audience builder. Audience members with a high likelihood to churn would be segmented into their own group and from there, Mercari could target those users with relevant paid media and email campaigns.
By partnering with GrowthLoop, Mercari was able to simultaneously bridge the gap between the data and marketing teams – and reduce the time between segmentation and campaign launch from months to just a few days.
“One of the big areas of benefit of working with GrowthLoop was the increased integration of marketing channels such as the CRM, User Acquisition, as well as more traditional marketing channels.” – Masumi Nakamura, VP of Engineering at Mercari
Creating the audience within the audience
Once the team had successfully created a model to predict churn across buyers and sellers, Mercari needed to launch retargeting campaigns to measure their ability to reduce churn. Each of their ongoing experiments features tailored segments along with automatic A/B testing. With analytics and activation all under one roof, the marketing team at Mercari could craft audiences and begin measuring the impact of their targeted campaigns. Since starting their work with GrowthLoop, the Mercari team has created over 120 audiences.
“Our marketing teams are more sophisticated with in-house knowledge, but GrowthLoop provides a more user-friendly way to build audiences for campaigns.” – Masumi Nakamura, VP of Engineering, Mercari
“GrowthLoop brings a very fundamental way of thinking about problems, including experimentation…. The ability to organize experiments and results was key. The number of variables is too high for most people without good organization.”– Masumi Nakamura, VP of Engineering at Mercari
Making segmentation smarter
Mercari’s first audiences leveraging GrowthLoop were sent to Braze to supercharge email campaigns and coupons with churn predictions and automated campaign performance evaluations. Then, Mercari shifted its focus to Facebook for paid media retargeting, using GrowthLoop’s lifecycle segmentation framework to target customers at the right step in their user journey. Lastly, Mercari moved its focus to Google Ads, where they used GrowthLoop to implement new segmentation models based on product category propensity. Mercari had long used Google Ads for product listing ads, and with GrowthLoop, Mercari was able to define more powerful product propensity segments and measure custom incremental lift metrics.
Finding new users in the haystack
Finally, in addition to preventing churn and driving retention, the Mercari team also wanted to boost user acquisition. They were having trouble measuring performance of UA campaigns due to new iOS and Facebook data privacy restrictions that made measuring campaign attribution impossible for many users. Using the familiar stack of Vertex AI Workbench for analysis, performance analysis on campaign data in BigQuery, and Airflow DAGs deployed via Cloud Composer to productionize the data pipelines, GrowthLoop enabled the team to activate targeted campaigns based on a user’s geographical location. In this way, Mercari could make decisions about their UA campaigns using incrementality analysis between geographic regions rather than attribution data, thus preserving user privacy.
The Mercari approach to customer data activation and acquisition
Other marketplace retailers can learn from Mercari’s successes activating data from BigQuery with GrowthLoop. Here are a few best practices to apply:
Identify your team’s needs and existing strengths
Mercari knew that their team had built out a strong foundation for data analysis within BigQuery. They also knew that their process was missing a key component that would allow them to activate that data. In order to achieve similar results, work to evaluate the strength of your team and your data – and define exactly what you aim to achieve with customer segmentation.
Partner with the right providers
With BigQuery, the Mercari team had all of their data centralized in one single location, simplifying the process for predictive modeling, segmentation, and activation. By partnering with GrowthLoop, this centralized data could be activated with ease across Mercari’s marketing teams. When evaluating providers for data warehousing, segmentation, and activation, be sure to partner with a provider that ensures you can get the most out of your data.
Know your audience
With a deeper understanding of their customers, Mercari was able to see nearly immediate value. By investing in the proper tools to accurately predict customer behavior, Mercari delivered impact in exactly the right areas. Using the data you’ve already compiled on your customers, consider partnering with a customer segmentation platform provider like GrowthLoop. In fact, Masumi went so far as to organize his Machine Learning team around these concepts: “We split the ML team into two areas – one to augment and work with GrowthLoop, the other team was to augment and orient around item data.”
Scale has been modified to intentionally obfuscate actual results.
How to boost growth like Mercari in three steps
Today, many leading brands leverage GrowthLoop and BigQuery to drive marketing and sales wins. Whether your company is in retail, financial services, travel, software, or another industry entirely, you can join the growing number of companies driving sustainable growth through real-time analytics by connecting BigQuery from Google Cloud to GrowthLoop. Here’s how:
If you have customer data in BigQuery…
- Book a GrowthLoop + BigQuery demo customized to your use cases.
- Link your BigQuery tables and marketing and sales destinations to the GrowthLoop platform.
- Launch your first GrowthLoop audience in less than one week.
If you are getting started with BigQuery…
- Get a Data Strategy Session with a GrowthLoop Solutions Architect at no cost.
- Use our Quick Start Program to get started with BigQuery in 4 to 8 weeks.
- Launch your first GrowthLoop audience in less than one week thereafter.
GrowthLoop and Google: Better together
The key question for many marketers today is, “How do you best leverage all you know about your customers to drive more intelligent and effective marketing engagement?” When Mercari set out to answer this question in 2019, they applied an innovative BigQuery data strategy that leveraged machine learning models. However, they achieved remarkable marketing results because they were among the first companies to discover and apply GrowthLoop to enable the marketing team to launch audiences with a first party data platform directly connected to their datasets and predictions in BigQuery. This greatly accelerated the design-launch-measure feedback loop to generate repeatable growth in customer lifetime value.
The Built with BigQuery advantage for ISVs and Data Providers
Google is helping companies like GrowthLoop build innovative applications on Google’s data cloud with simplified access to technology, helpful and dedicated engineering support, and joint go-to-market programs through the Built with BigQuery initiative. Participating companies can:
- Accelerate product design and architecture through access to designated experts who can provide insight into key use cases, architectural patterns, and best practices.
- Amplify success with joint marketing programs to drive awareness, generate demand, and increase adoption.
BigQuery gives ISVs the advantage of a powerful, highly scalable data warehouse that’s integrated with Google Cloud’s open, secure, sustainable platform. And with a huge partner ecosystem and support for multi-cloud, open source tools and APIs, Google provides technology companies the portability and extensibility they need to avoid data lock-in.
Click here to learn more about Built with BigQuery.
We thank the Mercari, GrowthLoop and Google Cloud team members who collaborated on the blog:
Mercari: Masumi Nakamura, VP of Engineering
GrowthLoop: Julia Parker, Product Marketing Manager; Alex Cuevas, Head of Analytics
Google: Sujit Khasnis, Solutions Architect
3021
Of your peers have already watched this video.
18:00 Minutes
The most insightful time you'll spend today!
Driving Business Transformation in Financial Services Using Google Cloud and AI/ML
These are challenging times with financial services institutions navigating the pandemic.
Interest rates are an all-time low, and customer expectations are changing with the shift to digital. For example, digital banking has increased from 63 in 2019 to 72 percent today. Then regulatory requirements are evolving and compliance costs are increasing. Also, the financial services industry as a whole is facing new and creative types of fraud.
In this video, Tom Shane, Product Manager – Google Cloud and Michael Baldwin, Head of Product – Financial Services, Google Cloud – Google Cloud, uncover three areas where they see AI helping to unlock business transformation in financial services.
Find out how AI can help reimagine the customer experience, how financial institutions are looking for ways to use data and analytics to transform how they detect and manage risk, and how financial institutions are asking Google Cloud experts for a best-in-class AI platforms so that they can build their own AI-powered solutions to solve their most critical business problems.
Spark on Google Cloud: How this Helps Customers with Agility, Cost Reduction and Time Spent on Spark
4879
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
Apache Spark has become a popular platform as it can serve all of data engineering, data exploration, and machine learning use cases. However, Spark still requires the on-premises way of managing clusters and tuning infrastructure for each job. Also, end to end use cases require Spark to be used along with technologies like TensorFlow, and programming languages like SQL and Python. Today, these operate in silos, with Spark on unstructured data lakes, SQL on data warehouses, and TensorFlow in completely separate machine learning platforms. This increases costs, reduces agility, and makes governance extremely hard; prohibiting enterprises from making insights available to the right users at the right time.
Announcing Spark on Google Cloud, now serverless and integrated
We are excited to announce Spark on Google Cloud, bringing industry’s first autoscaling serverless Spark, seamlessly integrated with the best of Google Cloud and open source tools, so you can effortlessly power ETL, data science, and data analytics use cases at scale. Google Cloud has been running large scale business critical Spark workloads for enterprise customers for 6+ years, using open source Spark in Dataproc. Today, we are furthering our commitment by enabling customers to:
- Eliminate time spent managing Spark clusters: With serverless Spark, users submit their Spark jobs, and let them do auto-provision, and autoscale to finish.
- Enable data users of all levels: Connect, analyze, and execute Spark jobs from the interface of users’ choice including BigQuery, Vertex AI or Dataplex, in 2 clicks, without any custom integrations.
- Retain flexibility of consumption: No one size fits all. Use Spark as serverless, deploy on Google Kubernetes Engine (GKE), or on compute clusters based on the requirements.
With Spark on Google Cloud, we are providing a way for customers to use Spark in a cloud native manner (serverless), and seamlessly with tools used by data engineers, data analysts, and data scientists for their use cases. These tools will help customers on their way to realize the data platform redesign they have embarked on.
“Deutsche Bank is using Spark for a variety of different use cases. Migrating to GCP and adopting Serverless Spark for Dataproc allows us to optimize our resource utilization and reduce manual effort so our engineering teams can focus on delivering data products for our business instead of managing infrastructure. At the same time we can retain the existing code base and knowhow of our engineers, thus boosting adoption and making the migration a seamless experience.”—Balaji Maragalla, Director Big Data Platform, Deutsche Bank
“We see serverless Spark playing a central role in our data strategy. Serverless Spark will provide an efficient, seamless solution for teams that aren’t familiar with big data technology or don’t need to bother with idiosyncrasies of Spark to solve their own processing needs. We’re excited about the serverless aspect of the offering, as well as the seamless integration with BigQuery, Vertex AI, Dataplex and other data services.” —Saral Jain, Director of Engineering, Infrastructure and Data, Snap Inc.
Dataproc Serverless for Spark
Per IDC, developers spend 40% time writing code, and 60% of the time tuning infrastructure and managing clusters. Furthermore, not all Spark developers are infrastructure experts, resulting in higher costs and productivity impact. With serverless Spark, developers can spend all their time on the code and logic. They do not need to manage clusters or tune infrastructure. They submit Spark jobs from their interface of choice, and processing is auto-scaled to match the needs of the job. Furthermore, while Spark users today pay for the time the infrastructure is running, with serverless Spark they only pay for the job duration.
Spark through BigQuery
BigQuery, the leading data warehouse, now provides a unified interface for data analysts to write SQL or PySpark. The code is executed using serverless Spark seamlessly, without the need for infrastructure provisioning. BigQuery has been the pioneer for serverless data warehousing, and now supports serverless Spark for Spark-based analytics.
Spark through Vertex AI
Data scientists no longer need to go through custom integrations to use Spark with their notebooks. Through Vertex AI Workbench, they can connect to Spark with a single click, and do interactive development. With Vertex AI, Spark can easily be used together with other ML frameworks like TensorFlow, Pytorch, Sci-kit learn, and BigQuery ML. All the Google Cloud security, compliance, and IAM are automatically applied across Vertex AI and Spark. Once you are ready to deploy the ML models, the notebook can be executed as a Spark job in Dataproc, and scheduled as part of Vertex AI Pipelines.
Spark through Dataplex
Dataplex is an intelligent data fabric that enables organizations to centrally manage, monitor, and govern their data across data lakes, data warehouses, and data marts with consistent controls, providing access to trusted data and powering analytics at scale. Now, you can use Spark on distributed data natively through Dataplex. Dataplex provides a collaborative analytics interface, with 1-click access to SparkSQL, Notebooks, or PySpark, and the ability to save, share, search notebooks and scripts alongside data.
Flexibility of consumption
We understand one size does not fit all. Spark is available for consumption in 3 different ways based on your specific needs. For customers standardizing on Kubernetes for infrastructure management, run Spark on Google Kubernetes Engine (GKE) to improve resource utilization and simplify infrastructure management. For customers looking for Hadoop style infrastructure management, run Spark on Google Compute Engine (GCE). For customers, who’re looking for no-ops Spark deployment, use serverless Spark!
ESG Senior Analyst Mike Leone commented, “Google Cloud is making Spark easier to use and more accessible to a wide range of users through a single, integrated platform. The ability to run Spark in a serverless manner, and through BigQuery and Vertex AI will create significant productivity improvement for customers. Further, Google’s focus on security and governance makes this Spark portfolio useful to all enterprises as they continue migrating to the Cloud.”
Getting started
Dataproc Serverless for Spark will be Generally Available within a few weeks. BigQuery and Dataplex integration is in Private Preview. Vertex AI workbench is available in Public Preview, you can get started here. For all capabilities, you can request for Preview access through this form.
You can work with Google Cloud partners to get started as well.
“We are excited to partner with Google Cloud as we look to provide our joint customers with the latest innovations on Spark. We see Spark being used for a variety of analytics and ML use cases. Google is taking Spark a step further by making it serverless, and available through BigQuery, Vertex AI and Dataplex for a wide spectrum of users.” —Sharad Kumar, Cloud First data and AI Lead at Accenture
For more information, visit our website or the watch announcement video and our conversation with Snap at Next 2021.
Poor Product Discovery Causes Shoppers’ Abandonment, Time to Augment ‘Search Experience’ with Google’s Retail Search!
3107
Of your peers have already read this article.
2:30 Minutes
The most insightful time you'll spend today!
How many times has a shopper searched for a product on a store’s website only to get results that aren’t relevant—or worse, provided no search results at all? While most ecommerce sites have search capabilities, few accomplish their ultimate goal: making it easy for customers to discover the products they want.
Some 94% of U.S. consumers abandoned a shopping session because they received irrelevant search results, according to a 2021 survey conducted by The Harris Poll and Google Cloud. It’s a phenomenon known as “search abandonment.” Indeed, poor product discovery experiences can stop a purchase in its tracks and leave shoppers frustrated. Retailers miss out on a staggering $300 billion each year due to search abandonment in the U.S. alone.
With the announcement today of our latest discovery solution, Retail Search, now generally available, retailers worldwide can supercharge their websites and mobile apps with Google-quality search. Built on Google’s technologies that understand user intent and context, the solution helps businesses improve the search and overall shopping experience across all of their digital touchpoints.
Already, early adopters like Lowes, Fnac Darty, and Casas Pernambucanas have been using Google Cloud’s discovery solutions to increase sales conversions, boost basket sizes, and improve customer engagement.
Understanding user intent in search is a hard problem
While we’ve come a long way from the days when search was largely based on keywords and boolean rules, shoppers still struggle to find what they’re looking for. They often have to come up with a perfectly-worded query that a retailer’s site search engine will understand, sometimes rephrasing several times before getting the results they want—if they find anything relevant at all.
Traditional search technologies don’t work in the modern age of online retail, where tens or even hundreds of thousands of items are available on a single ecommerce site.
Today, people expect search engines to understand their intent more deeply, return relevant results faster, and help them discover new products easily with personalized recommendations.
Fortunately, product discovery experiences on retail sites can now offer superior, contextual experiences for consumers, combining marketing and data science insights along with advanced search technologies based on machine learning and artificial intelligence.
Now, through the power of Retail Search, when a shopper searches for a “long black dress with short sleeves and comfortable fit” on an ecommerce site, they should immediately get results for precisely that—rather than refining their search multiple times, or worse, giving up their shopping journey.
Helping retailers solve their search experience woes
Retail Search transforms the shopping experience and makes it easier for shoppers to find relevant products by surfacing results that are intuitive and contextual.
This fully managed service is easily customizable, enabling organizations to craft shopper-focused search experiences. Our site search solution builds upon decades of Google’s experience and innovation in search indexing, retrieval, and ranking. Retailers can make product discovery even easier for shoppers, while optimizing for their business goals with advanced capabilities, including:
- Advanced query understanding that produces better results from even the broadest queries, including non-product searches.
- Semantic search to effectively match product attributes with website content for fast, relevant product discovery.
- Optimized results that leverage user interaction and ranking models to meet specific business goals.
- State-of-the-art security and privacy practices that ensure retailer data is isolated with strong access controls and is only used to deliver relevant search results on their own properties.
Retailers can now shape product discovery into a shopping experience that is timely, relevant, and personalized—all powered by Google Cloud.
Early adopters seeing immediate impact
Several leading retailers around the world have been able to quickly create better online shopping experiences for their customers and capture more digital and omnichannel growth using Google Cloud’s product discovery solutions.
“With limited customer signals and no historical data, descriptive long-tail searches are some of the most challenging queries to understand,” said Neelima Sharma, senior vice president, technology, e-commerce, marketing and merchandising at Lowe’s. “We have been partnering with Google Cloud to give our customers relevant results for long-tail searches and have seen an increase in click-through and search conversion and a drop in our ‘No Results Found’ rate since we launched.”
“Making constant improvements to our website search engines has always been a priority for us as we aim to give our customers a simpler, more customized and enhanced online shopping experience,” said Olivier Theulle, Fnac Darty’s chief ecommerce and digital officer. “As we implement Google Cloud’s site search solution on our Fnac Darty websites and are the first French retailer to do so, we expect the solution to deliver increased conversion rates while also offering greater customer satisfaction.”
“Google Cloud has helped improve the indexing and quality of search results on Pernambucanas’ digital platforms, providing a better experience to our customers and improving sales conversions,” said Fabiano Rustice, chief information officer at Pernambucanas. “On Black Friday in November 2021—the largest retail date in Brazil—we saw a 20% reduction in search refinements per user, for which Retail Search was instrumental. We are proud to be an early adopter and to have Google Cloud as a strategic innovation partner.”
In addition to Google Cloud’s efforts working directly with customers, our ecosystem of key partners, including GroupBy, Lucidworks, GridDynamics, SpringML, and others are also leveraging Google Cloud discovery solutions to provide value added services to their customers.
“In the fast paced and extremely competitive ecommerce environment, we are seeing that forward-looking and innovative retailers are prioritizing product discovery,” said Roland Gossage, CEO of GroupBy. “GroupBy has proudly partnered with Google Cloud to launch its product discovery solutions across several client environments. Already we have seen a more than 10% increase in online revenues, which equates to millions of dollars, and this is just the beginning.”
To learn more, visit Discovery Solutions for Retail or contact your Google Cloud field sales representative.
Speak Now to Book a Flight: easyJet’s Uses AI to Improve Customer Experience
3479
Of your peers have already read this article.
4:30 Minutes
The most insightful time you'll spend today!
With a growing fleet of 325 aircraft that cover more than 1,000 routes across 158 airports, easyJet is one of Europe’s most popular airlines. And easyJet serves an average of 90 million passengers each year, so a helpful mobile experience for its customers is a top priority.
Travellers today are inherently mobile-first, so finding new ways to make it easier for them to search and book flights is key. To do exactly that, easyJet partnered with technology company Travelport to develop Speak Now, a new feature on easyJet’s mobile app that interprets voice searches to deliver accurate and relevant flight information to travelers.
See how it works:
Powered by Dialogflow, Google Cloud’s natural language understanding tool for building conversational experiences, Speak Now lets customers ask questions to determine exactly what they’re looking for—from destinations, to dates and times, to airports they want to fly from.
Dialogflow, a core component of Google Cloud Contact Center AI, makes it easy to build accurate, flexible conversation interfaces that allow users to ask questions and accomplish tasks in everyday language.
How do we create conversational experiences across devices and platforms for enterprises?
It’s clear that the rise in voice search is changing the way we go about our daily lives. Twenty-seven percent of the global online population already uses voice search on mobile, and the rapid adoption of this technology is reshaping entire industries. It’s no surprise that easyJet looked to adopt this technology to positively transform experiences for their customers.
Dialogflow, a core component of Google Cloud Contact Center AI, makes it easy to build accurate, flexible conversation interfaces that allow users to ask questions and accomplish tasks in everyday language. It understands the nuances of human language and translates end-user text or audio during a conversation to structured data that apps and services can understand.
Speak Now is a great example of how we’re using cloud technologies and AI to make the experience of buying and managing travel continually better for everyone.
Daniel Young, Head of Digital Experience, easyJet
Daniel Young, Head of Digital Experience at easyJet commented: “We picked Dialogflow due to its strengths and ease with which a powerful conversational agent can be built. Speak Now is a great example of how we’re using cloud technologies and AI to make the experience of buying and managing travel continually better for everyone. This is the latest in a series of innovative features that will make booking travel as easy as it can possibly be, giving easyJet customers a helpful digital experience.”
Consumers already rely on voice assistants to play their favorite music, add items to a shopping list, and order taxis, Speak Now is a great example of how voice assistants can now make the customer experience better and more intuitive for travel.
Time-series Model on Google Cloud Allows Better Transparency on Fishing and Marine Activities
4406
Of your peers have already read this article.
3:30 Minutes
The most insightful time you'll spend today!
Who would have known that today technology would enable us with the ability to use machine learning to track vessel activity, and make pattern inferences to help address IUU (illegal, unreported, and unregulated) fishing activities. What’s even more noteworthy is that we now have the computing power to share this information publicly in order to enable fair and sustainable use of our ocean.
An amazing group of humans at the nonprofit Global Fishing Watch took on this massive big data challenge and succeeded. You can immediately access their dynamic map on their website globalfishingwatch.org/map that is bringing greater transparency to fishing activity and supporting the creation and management of marine protected areas throughout the world.
In our second episode of our People and Planet AI series we were inspired by their ML solution to this challenge, and we built a short video and sample with all the relevant code you need to get started with building a basic time-series classification model in Google Cloud, and visualize it in an interactive map.
Architecture
These are the components used to build a model for this sample:
- Global Fishing Watch GitHub: where we got the data
- Apache Beam: (open source library) runs on Dataflow.
- Dataflow: (Google’s data processing service) creates 2 datasets; 1 for training a model and the other to evaluate its results.
- TensorflowKeras: (high level API library) used to define a machine learning model, which we then train in Vertex AI.
- Vertex AI: (a platform to build, deploy, and scale ML models) we train and output the model.
Pricing and steps
The total cost to run this solution was less than $5.
There are seven steps we went through with their approximate time and cost:
Why do we use a time series classification model?
Vessels in the ocean are constantly moving, which creates distinctive patterns from a satellite view.
We can train a model to recognize the shapes of a vessel’s trajectory. Large vessels are required to use the automatic identification system, or AIS. The GPS-like transponders regularly broadcast a vessel’s maritime mobile service identity, or MMSI, and other critical information to nearby ships, as well as to terrestrial and satellite receivers. While AIS is designed to prevent collisions and boost overall safety at sea, it has turned out to be an invaluable system for monitoring vessels and detecting suspicious fishing behavior globally.
One tricky part is that the MMSI data location signal (which includes a timestamp, latitude, longitude, distance from port, and more) is not emitted at regular intervals. AIS broadcast frequency changes with vessel speed (faster at higher speeds), and not all AIS messages that are broadcast are received – terrestrial receivers require line-of-sight, satellites must be overhead, and high vessel density can cause signal interference. For example, AIS messages might be received frequently as a vessel leaves the docks and operates near shore, then less frequently as they move further offshore until satellite reception improves. This is challenging for a machine learning model to interpret. There are too many gaps in the data, which makes it hard to predict.
A way to solve this is to normalize the data and generate fixed-sized hourly windows. Then the model can predict if the vessel is fishing or not fishing for each hour.
It could be hard to know if a ship is fishing or not by just looking at its current position, speed, and direction. So we look at the data from the past as well, looking at the future could also be an option if we don’t need to do real time predictions. For this sample, it seemed reasonable to look 24 hours into the past to make a prediction. This means we need at least 25 hours of data to make a prediction for a single hour (24 hours in the past + 1 current hour). But we could predict longer time sequences as well. In general, to get hourly predictions, we need (n+24) hours of data.
Options to deploy and access the model
For this sample specifically we used Cloud Run to host the model as a web app so that other apps can call it to make predictions on an ongoing basis; this is our favorite in terms of pricing if you need to access your model from the internet over an extended period of time (charged per prediction request). You can also host it directly from Vertex AI where you trained and built the model, just note there is an hourly cost for using those VMs even if they are idle. If you do not need to access the model over the internet, you can make predictions locally or download the model onto a microcontroller if you have an IoT sensor strategy.
Want to go deeper?
If you found this project interesting and would like to dive deeper either into the specifics of the thought process behind each step of this solution or even run through the code in your own project (or test project); we invite you to check out our interactive sample hosted on Colab, which is a free Jupyter notebook. It serves as a guide with all the steps to run the sample, including visualizing the predictions on a dynamically moving map using an open source Python library called Folium.
There’s no prior experience required! Just click “open in Colab” which is linked at the bottom of GitHub.
You will need a Google Cloud Platform project. If you do not have a Google Cloud project you can create one with the free $300 Google Cloud credit, you just need to ensure you set up billing, and later delete the project after testing the desired sample.
🌏🌎🌍 We hope to inspire you to build other beautiful climate-related solutions.
More Relevant Stories for Your Company
How Ather Energy is leveraging the Cloud to build and scale smart mobility solutions for India
In 2013, long before the world was discussing clean energy and sustainable practices, two IIT Madras graduates — Swapnil Jain and Tarun Mehta — had an idea to develop India’s first-ever electrical scooter. This was at a time when auto manufacturers were still focusing on fossil-fuel-driven vehicles and ‘eco-friendly’ mobility
Google Cloud CCAI’s Support for the Public Sector Soars during the Pandemic
Scaling Virtual Support in the Pandemic Era: The AI Connection Since the early days of the pandemic, we’ve partnered with government organizations and academic institutions to serve communities at scale with Contact Center Artificial Intelligence (CCAI). I sat down with Bill MacKenzie, IT liaison for the Upper Grand School District
Tackling Real-Time Bidding Challenges: Arpeely’s Fresh Approach with Google Cloud
At Arpeely, we’ve developed some of the world’s most advanced advertising technology. Our machine learning (ML) media acquisition platform and “win-win” business model enables customers to bring highly intentful users to their offerings with precision, peace of mind and minimal overhead. Real-time bidding is a dynamic and intricate process that involves
How Apna is using data and AI to drive the gig economy in India
In this episode, Theo speaks to Ronak Shah, Head of Data at apna.co. As one of the largest gig economies in the world, India is seeing rising numbers among younger people joining the ecosystem. Driven by digital adoption and new ways of working, Apna saw an opportunity to help bridge
