redBus: Mastering Big Data with Google BigQuery
5387
Of your peers have already read this article.
2:30 Minutes
The most insightful time you'll spend today!
In 2006, online travel agency redBus introduced internet bus ticketing in India, unifying tens of thousands of bus schedules into a single booking operation. (Think of it as Expedia for bus booking.) Using BigQuery, redBus crunches terabytes of booking and inventory data in mere seconds and at a fraction of the cost of other big data services. BigQuery also helps engineers fix glitches quickly, minimize lost sales, and improve customer service.
Challenge
Executives at the Bangalore-based redBus needed a powerful tool to analyze booking and inventory data across their system of hundreds of bus operators serving more than 10,000 routes. They considered using clusters of Hadoop servers to process the data but decided the system would take too much time to set up and would require a specialized staff to maintain it. It also would not provide the lightning-fast analysis they needed.
“It would have taken at least a couple of hours to analyze anything,” says Pradeep Kumar, a technical architect at redBus. “Crunching very large data sets would have been a day’s job. We needed something more powerful to get the real-time analysis we were looking for.”
Solution
Kumar and his colleagues learned about BigQuery and realized it was the right match for their data processing needs. The web-based service, which enables companies to analyze massive datasets using Google’s data processing infrastructure, is easy to set up and manage since its simple, SQL-like query language doesn’t require complex technology or specialized personnel. It also has low overhead costs.
The redBus team uses BigQuery as part of an intricate data collection and analysis process. Applications hosted on a range of servers continually pump information related to customer searches, seat inventory, and bookings into a centralized data collection system. Engineers upload the data to BigQuery, which provides answers to complex queries within seconds. For example, BigQuery helps redBus staff:
- Learn how many times customers searched for seats and found none or very few available, indicating more seats should be added to a route
- Investigate decreases in bookings and notify engineers if a technical problem is the cause
- Identify server problems by quickly analyzing data related to server activity
Results
BigQuery provides near real-time data analysis capabilities at 20% of the cost of maintaining a complex Hadoop infrastructure. Queries that would have required a day to analyze on a Hadoop framework take less than 30 seconds using Google’s web-based service.
“We explored several data analytics solutions. Nothing comes remotely close to the sheer power of Google BigQuery,” Kumar says. “It made large-scale data collection and crunching possible with little effort, which has translated to a significant business advantage.”
Google Cloud Platform results
- Analyzes data sets as large as 2 terabytes in less than 30 seconds using a simple, SQL-like language
- Saves time analyzing technical problems and customer booking trends
- Spends 80% less than they would have on a Hadoop infrastructure and avoids setting up and maintaining a complex infrastructure in-house
- Strengthens the company by improving customer service and engineering quality
The fast insights gained through BigQuery are also making redBus a stronger company. By minimizing the time it takes staff members to solve technical problems, BigQuery has helped improve customer service and reduce lost sales.
“Getting to the root of problems used to be really time-consuming,” Kumar says. “By the time we figured it out, customers might have given up. Now if there are booking problems, BigQuery helps us understand the reason right away. Choosing Google BigQuery was the right decision for our company.”
How Kinguin Notched Up Shopping Experience with Google Recommendations AI
6445
Of your peers have already read this article.
3:00 Minutes
The most insightful time you'll spend today!
Over 2.14 billion people worldwide are expected to buy online this year, according to Statista. Online retail sales will account for 22% of all purchases by 2023. But in a competitive retail landscape, positive interactions can mean the difference between a sale and an abandoned shopping cart.
One of the leading global marketplaces – Kinguin.net is a haven for gamers. Their bustling ecommerce business conducts over 500,000 new transactions monthly. Users will encounter over 50,000 unique digital products, from video games, gift cards, in-game items to computer software and services. With over 10 million registered users, Kinguin improved their experience by helping users find items quickly and deliver service at scale.
Helping customers find what they want, fast
Because of Kinguin’s high volume of users—both buyers and sellers—and breadth of digital products, browsing and shopping can be challenging. “Customers shop online for choice and convenience, but it can sometimes be overwhelming. We want anyone who shops at Kinguin to find what they are looking for quickly and easily,” says Viktor Romaniuk Wanli, Kinguin CEO and Founder.
Today’s retailers know that creating personalized shopping experiences is crucial for establishing and maintaining customer loyalty. Kinguin discovered their users were getting a rather standard retail experience. They wondered how they could offer them a more tailored, personalized experience.
They knew product recommendations were a great way to personalize experiences because they help customers discover products that match their tastes and preferences. But it’s not that easy to recommend products. Various shifting factors make recommendations much more complex:
- Customer behavior. Understanding customers is tough. How do you recommend something to a cold start user who’s never been to your site before? What happens when their behavior changes?
- Omnichannel context. According to Harvard Business Review, 73% of all customers use many channels when they buy. What happens when they go from desktop to mobile or from social media shopping to a proprietary app?
- Product data challenges. How do you recommend new products within a large catalog of items? What if your product data has sparse labeling or unstructured metadata?
Data wasn’t a problem for Kinguin. They had data orders, history, wishlists, and could collect events based on their platform interactions. It was the machine learning model expertise they lacked. So rather than building their own solution, they determined it was more cost effective for them to find a reliable partner. It was also essential that the solution integrated easily with Kubernetes, which enabled their global network.
With these considerations in mind, they applied for the Google Recommendations AI beta program. Kinguin became the first gaming e-commerce platform in Europe to use Recommendations AI when it launched in 2020.
Pro gamer move: using a fully managed AI service
Google Recommendations AI uses algorithms to deliver highly personalized suggestions tailored to a customer’s preferences. Google Cloud based these algorithms on the same research that powers models by YouTube search and Google Shopping. Algorithms are always being tuned and adjusted to focus on individuals themselves—not just items.
Many shopping AIs rely on manually provisioning infrastructure and training machine learning models. Instead, Recommendations AI’s deep learning models use item and user metadata to gain insights. It processes Kinguin’s thousands of products at scale, iterating in real time. First, Kinguin pieces together a customer’s history and shopping journey. Then, using Recommendations AI, they can serve up personalized products—even for long-tail products and cold-start users.
By leveraging internal tools, Kinguin didn’t need to start implementation from scratch. After a few trial sessions with Google Cloud engineers, they got started right away. Due to the fast-paced nature of a marketplace—i.e., price changes, out-of-stock items—Kinguin needed their recommendations to be as close to real time as possible. They used internal event buses to stream events and their product catalog directly to the recommendations API.
Kinguin rolled out in high-traffic areas, including their home page, product page, and category pages. They analyzed heat maps and scroll maps to figure out where to test placements. They also experimented with different recommendation models such as “recently bought together” and “you may like.” Engineers also factored in where they were implementing the models. For example, the “others you might like” model would fit best on the homepage, while “frequently bought together” made sense at checkout.
Understanding how product recommendations influence financials is critical for demonstrating the impact of personalization. Using BigQuery, Kinguin could analyze different cost projection models. BigQuery helped them dig into specific financial data to understand their margins and revenue gains.
Playing to win: enhanced customer experience
Since adopting Recommendations AI, Kinguin has improved both customer experience and satisfaction. Search times have shortened by 20 seconds. Additionally, their average cart value has increased by 5 EUR. Conversion rates have quadrupled since the outset. Click-thru rates have doubled, increasing by 2.16 on product pages and 2.8 times on recommendations pages.
“Google Recommendations AI has helped us evolve our service, increase customer loyalty and satisfaction. It has also contributed to a significant rise in sales,” says Wanli. Kinguin is already thinking about other ways of enhancing user experiences with recommendations. Ideas include their checkout process, other landing pages, and email marketing.
Kinguin’s journey with Google Cloud shows how companies can leverage AI to optimize sales and deliver high-performing, low-latency recommendations to any customer touchpoint.
Learn more about Recommendations AI and Google Cloud AI and machine learning solutions.
World’s Largest Online-only Grocery Retailer Uses AI to Figure Which Customers Need Most Attention
9455
Of your peers have already read this article.
4:20 Minutes
The most insightful time you'll spend today!
In the United Kingdom, the popularity of online grocery shopping is expected to surge from about 6% of the market today to 9% by 2021, according to market research firm Mintel. One of the pioneers of online-only grocery retailing is Ocado, based in Hatfield, Hertfordshire in the U.K. Since starting commercial deliveries in 2002, the company has grown to 600,000 active customers, 260,000 weekly orders, and £1.39 billion in annual revenue.
Ocado takes supermarket trips out of the equation by enabling shoppers to purchase items online through its convenient web and mobile applications. Items are then picked and packed in automated warehouses and shipped directly to customers in a one-hour time slot of their choosing. Ocado’s delivery punctuality is 95%, order accuracy is 99%, and its service footprint now reaches more than 70% of the U.K. population.
“Google Cloud Platform gives us the flexibility and performance to tackle the large and complex data challenges unique to our business.”
—Paul Clarke, Chief Technology Officer, Ocado
The company achieved its success by building in-house almost all the technology and automation that powers its end-to-end e-commerce, fulfillment, and logistics platform. Ocado also developed a new platform, the Ocado Smart Platform (OSP), which offers large brick-and-mortar grocery retailers around the world access to a best-in-class solution for online grocery.
Democratizing machine learning
The shopping journey for online grocery retailing differs significantly from other e-businesses. Customers often buy dozens of products at once, a single household may have multiple buyers using multiple devices, and product shelf life may only be a couple of days.
“We often say that having built an end-to-end platform that can do online grocery scalably and profitably, we can do other forms of online retail; but the reverse does not necessarily follow,” says Paul Clarke, Chief Technology Officer at Ocado. “Google Cloud Platform gives us the flexibility and performance to tackle the large and complex data challenges unique to our business.”
The Ocado business model takes advantage of consumers’ shifting preferences and the links between digital technology and shopping experiences.
“Google Cloud Machine Learning Engine gives us the agility we need. Our developers were able to try out TensorFlow and see firsthand the benefits of machine learning in the cloud.”
—Paul Clarke, Chief Technology Officer, Ocado
The company has been building machine learning into its systems for over five years. Until recently, Ocado machine learning applications required specialist data scientists, typically with PhDs in machine learning, who would build these solutions from the ground up. It also required the specialist who set up the system and costly on-premises infrastructure to train and run these systems.
However, working with Google as a private alpha testing site for Google Cloud Machine Learning Engine accelerated its adoption of artificial intelligence (AI).
“We’ve been talking about how the cloud could democratize AI for some time,” says Paul. “Google Cloud Machine Learning Engine gives us the agility we need. Our developers were able to try out TensorFlow and see firsthand the benefits of machine learning in the cloud.”
TensorFlow is an open source software library for machine learning developed by the Google Brain team. Ocado developers, engineers, and data scientists now use TensorFlow for many of their machine learning projects. They deploy the models they build on Google Cloud Machine Learning Engine, which lets them train models faster across servers, desktop computers, and mobile devices through a single application program interface (API). Additionally, Google Cloud Machine Learning Engine integrates easily with the other Google Cloud Platform products used widely at Ocado.
What do customers really want?
One of the first TensorFlow models Ocado created was a machine learning algorithm that tags and categorizes customer emails and then prioritizes them for response.
The contact center receives thousands of emails each day and Ocado wanted to automate determining which ones needed to be answered immediately and which ones could wait.
For example, a first-time customer expressing their delight in using Ocado doesn’t need to be responded to with the same urgency as a customer who is missing an item from their order or who won’t be home to receive the delivery.
“Enabling agents to respond without having to sort through less-urgent emails improves Ocado’s responsiveness and customer service.”
—James Donkin, General Manager, Ocado
“We get a lot of emails from customers saying, ‘Our service was great,’ or ‘The driver was very courteous,'” says James Donkin, General Manager, Ocado. “But when issues like weather or road conditions potentially affect delivery, we often get surges of urgent questions. Enabling agents to respond without having to sort through less-urgent emails improves Ocado’s responsiveness and customer service.”
Using Google Cloud Machine Learning Engine, TensorFlow, and a large data set culled from several years’ worth of manually categorized customer emails, Ocado experimented on which kind of neural network architecture would best prioritize emails. After testing its models, Ocado implemented the highest-performing one and has been able to respond to urgent messages four times faster. The company also discovered that 7% of its emails don’t require a response at all, which means call center representatives now have more time to devote to higher priority messages.
“Without Google Cloud Machine Learning Engine, it would have been a lot harder to succeed on a project like email classification,” says Roland Plaszowski, who has recently managed several big data projects and initiatives at Ocado.
“Even if we invested significantly in infrastructure, it would be difficult to manage because of the computational intensity. It’s challenging and expensive to run machine learning projects at the same time without infrastructure that you can scale easily.”
Ocado also uses machine learning to predict customer behavior and improve experiences. By analyzing order data, Ocado makes shopping as frictionless as possible. For example, the ordering system can pre-populate customers’ shopping carts with items they are most likely to purchase, remind customers about items they may have forgotten, and notify them of multi-buy offers they haven’t completed, for example, only buying one of a buy one, get one free offer. Based on machine learning from previous purchase data, the Ocado system can also offer new products that are likely to delight customers.
“You will regularly see items that are more personally relevant to you instead of items that are being promoted more generally,” says James. “I’m a vegetarian, so I’m offered specials for vegetarian products that I normally buy and new ones that I’ve never bought. I’m also less likely to see things that I’m not interested in.”
Machines and machine learning
Within the Internet of Things (IoT), Ocado is looking to enhance its warehouse robots with machine learning. An integral part of the OSP, thousands of robots continually stream data into Google Cloud Storage and Google BigQuery.
Ocado data scientists apply machine learning to create a type of swarm intelligence that enables warehouse robots to work cooperatively to achieve a common goal. Projects include modules to search robot telemetry data, such as whether a battery pack is operating within standard tolerances or whether firmware has been successfully loaded, and use it to optimize maintenance schedules or detect patterns in wear and tear.
“Another challenge we’re looking at is how to embed machine learning directly into robots so they become smarter in terms of self-testing, exception handling, and error recovery,” says Paul. “This is a challenging combination of IoT, data analytics, and machine learning that we believe Google BigQuery and Google Cloud Machine Learning are particularly well suited to helping Ocado achieve.”
The company also discovered that 7% of its emails don’t require a response at all, which means call center representatives now have more time to devote to higher priority messages.
Scaling for new business
Scalability is also a major reason behind some of Ocado’s cloud initiatives, including the migration of all its on-premises data to the cloud. Ocado wanted to improve customer experiences, empower business teams with greater insight, and reduce IT overhead, so it consolidated onto Google Cloud Platform.
“The old databases just weren’t fast enough,” says Paul. “We needed a solution that could scale with the amount of data we generate and how we use it. Google Cloud Storage and Google BigQuery now provide the backbone, from a data point of view, for the Ocado Smart Platform.”
Ocado estimates its business, product, and transaction data is approaching two petabytes. Combining customer and supply chain data helps both internal Ocado operations and the company’s ambitions to commercialize OSP.
“When compared with other options for expansion internationally, selling OSP as a managed service lets us turn companies that could have been competitors into customers,” says Paul. “We want to build OSP once and then turn it on for multiple business-to-business customers.”
Each time Ocado adds a new hosting customer to OSP, it will launch a customized instance to fit that customer’s requirements. The capacity and performance of each new OSP instance must be able to scale quickly as the backend platform for established retailers with large numbers of products, customers, and transactions.
Ocado’s first OSP customer, Morrisons, is already benefiting from this first-of-a kind solution. Morrisons is one of the UK’s four largest supermarkets and uses OSP to power its online retail business. Using Google Cloud Platform, Ocado has stored, processed, and analyzed terabytes of Morrisons’ data using a dedicated data lake and Google BigQuery.
In addition to using Google Cloud Platform for OSP, Ocado also adopted it for its own online grocery retail business operation. Ocado originally used the Apache Spark and Apache Hadoop open-source frameworks on Google Compute Engine for its data platform. Moving to Google BigQuery frees Ocado business analysts from the complex query setup and workflows associated with Spark and Hadoop. Plus, it lets Ocado share data analytics with suppliers and partners.
Google BigQuery is well integrated with TensorFlow on Google Cloud Machine Learning Engine and Google Cloud Dataproc, the Apache Spark and Apache Hadoop service that lets Ocado use open source data tools for batch processing, querying, streaming, and machine learning. Google Cloud Dataflow and Google Cloud Dataproc handle cluster management, and provide an easy-to-use framework so developers can spend less time and money on administration and more time on delivering valuable business features.
Switching from Hadoop to Google BigQuery revealed a series of cost and performance improvements. For example, Ocado no longer needed to decide how many instances to bring up in a cluster or wait for the instances to spin up. Google handled everything.
“We simply ran our queries and paid for the resources that we use,” adds Roland. “One big win with Google BigQuery is we don’t have to do maintenance. Best of all, we saw Google BigQuery outperform our Hadoop cluster by over 80 times on our largest dataset, and for only two-thirds the cost.”
11218
Of your peers have already watched this video.
1:30 Minutes
The most insightful time you'll spend today!
Indian Retailer Figures Optimizes Hyperlocal Delivery to Increase Customer Experience
Anyone who follows the Indian e-commerce scene knows that one of the largest challenges these companies face is hyperlocal delivery.
That was a problem facing Wellness Forever, a retail chain of pharmacies with 150-plus stores across India.
“Exactly a year ago, we started our journey of hyperlocal deliveries. This optimization was a big time challenge for us to understand how to optimize this,” Palani Subbiah, CTO, Wellness Forever.
The problem in front of Wellness Forever was to identify which customer could can be sold from which store, so that a delivery could be made within 90 minutes.
“We handle a large amount of customer data and we wanted to use insights to help and improve the customer satisfaction index,” says Subbiah.
To do that Wellness Forever leveraged Google Big Query to run massive amount of data to come up with the operational insights. They also used Firebase and Google Maps.
“By 2021, we are going to have about 450 stores. Those stores are going to be not only a physical store, which is a digital store.
5123
Of your peers have already watched this video.
1:47 Minutes
The most insightful time you'll spend today!
How adidas’s Marketing Team Benefits from Google Cloud Marketing Platform
We live in a world where the consumer journey is no longer linear, making it difficult to know when to deliver a brand message or a product message.
“We talk a lot about the consumer journey no longer being linear. It’s very difficult to know when the consumer is going to come in contact with your message. And because of that it makes it much harder to deliver the right message at the right time,” says Chris Murphy, Head of Digital Experience, adidas.
To solve the problem, adidas turned to the Google Cloud Marketing Platform to use audience insights to inform the art of storytelling and help teams, from brand marketing to retail to e-commerce, deliver relevant messages across channels.
“In a singular environment like Google Cloud Marketing Platform, when you’re able to see audience insights and information…we know when someone should be receiving a brand message versus a product message — when they’re thinking about buying or when they’re actually ready to purchase,” says Murphy.
It wasn’t easy at the start. “Being in brand marketing for so long, there was this initial pushback on data and the science behind it, because it was an art. But what we’re seeing now is the ability for data and insights to really inform the art of storytelling. It allows us to understand the impact that we’re having in real time,” says Kelly Olmstead, VP, Brand Activation for North America, adidas.
Today adidas’ marketing team depends more heavily on data and analysis to guide its strategy. “Starting every meeting with data and insights, that’s something that we want to take outside of just the marketing organization, and really infuse it across our culture here at adidas,” says Olmstead.
Watch how adidas benefits from using the Google Cloud Marketing Platform.
3008
Of your peers have already watched this video.
36:30 Minutes
The most insightful time you'll spend today!
Architecting and building a data lake on GCP with open source tools
Creating data lakes is often the first step towards maximizing value from data by generating insights for the business.
Hadoop data leaks, are today, the most common ones that are found on-premise and many of Google’s customers are moving these to the Google Cloud Platform. And the trend is accelerating.
In this video, Google Cloud Strategic Cloud Engineer, Roderick Yao, will teach you about the growing challenges in managing on-prem data lakes and what is driving the growth of open source implementations on the cloud.
He will walked you through how to architect, migrate, and secure your own open source data lake on Google Cloud using a mix of managed services and open source tools.
During the course of this presentation, Yao will go over:
- Why run data lakes on Google Cloud
- Designing and migrating data lakes
- Security and governance
More Relevant Stories for Your Company
Serverless and BigQuery Together on Google Cloud: Behind the L’Oreal Beauty Tech Data Platform
Editor's note: In Today's guest post we hear from beauty leader L'Oréal about their approach to building a modern data platform on fully managed services: managing the ingest of diverse datasets into BigQuery with Cloud Run, and orchestrating transformations into relevant business domain representations for stakeholders across the organization. Learn
New Capabilities in Google Cloud Dataprep: A Must-Read for Data Teams
In this video, Bertrand Cariou, Sr. Director Product Marketing, Trifacta and Sean Ma, Sr. Director, Product Management, Trifacta take a deep dive into new capabilities for Cloud Dataprep. They walk us through several new features to enable a wider range of use cases with Cloud Dataprep; and, major enhancements to
IT Team Figures Out Easiest Way to Build Data Pipelines and Create ML Models
Building a strong brand in today's hyper-competitive business environment takes vision. It also requires a flexible, easily managed approach to digital asset management (DAM), so marketing professionals and other stakeholders can easily share, store, track, and manipulate assets to build the brand. Many of today's leading companies, including JetBlue, Slack,
AirAsia Flies High With Data Analytics and AI
AirAsia’s vision is simple: allow everyone to fly. Founded in 2001, the airline and sister company AirAsia X have grown to service 150+ destinations in 25 markets, using 274 aircraft to operate 11,000+ weekly flights from 23 hubs across the region. While the airline is known as a provider of
