Bigtable’s High Performance and Low Opex is Great for Building Personalization
3754
Of your peers have already read this article.
3:00 Minutes
The most insightful time you'll spend today!
Customer expectations have shifted as a result of evolving needs. Across industries, customers expect that you treat them as individuals, and demonstrate how well you understand and serve their unique needs. This concept—personalization—is the idea that you’re delivering a tailored experience to each customer corresponding to their needs and preferences; you’re setting up a process to create individualized interactions that improves the customer’s experience. According to Salesforce, 84% of consumers say being treated like a person, not a number, is very important to winning their business.
Entire industries are undergoing digital transformation to better serve their customers through personalized experiences. For example, retailers are improving engagement and conversion with personalized content, offers, and product recommendations. Advertising technology organizations are increasing the relevance and effectiveness of their ads using customer insights like their specific interests, purchasing intent, and buying behavior. Digital music services are helping their customers discover and enjoy new music, playlists, and podcasts based on their listening behavior and interests.
As customers want more and more personalization, modern technology is making it possible for many more businesses to achieve this. In this post, we’ll look at some common challenges to implementing personalization capabilities and how to solve them with transformative database technologies like Google Cloud’s Bigtable. Bigtable powers core Google services such as Google Maps that supports more than a billion users, and its petabyte scale, high availability, high throughput, and price-performance advantages help you deliver personalization at scale.
Challenges with personalization
Data is at the heart of personalization. To deliver personalization at scale, an application needs to store, manage, and access large volumes of data (a combination of customer-specific data and anonymized aggregate data across customers) to develop a deep understanding of the behavior, needs, and preferences of each customer. Your database needs to very quickly write large volumes of data concurrently for all active customers. You need to continuously capture data on customer behavior because each step potentially informs the next, e.g., adding an item to a shopping cart can be used to trigger new recommendations for related or complementary products. Much of this data needed for personalization is semi-structured and sparse, and therefore requires a database with a flexible data model.
Personalization at scale requires large volumes of data to be read in near real-time so that it can be in the critical serving path to deliver a seamless user experience, often with a total application latency of less than 100ms. This means your requests to the database need to return results with latencies of single-digit milliseconds. You need to ensure that application latencies do not degrade as you onboard more customers. Data needs to be organized efficiently and integrated with other tools so that you can run deep analytical queries and use machine learning (ML) models to develop personalized recommendations, and store the aggregates in your operational database for serving your customers. You also need the ability to run large batch reads for analytics without affecting the serving performance of your application.
In addition, you need to ensure that your database costs do not explode with the popularity of your application. Your database needs to consistently deliver low total cost of ownership (TCO), and high price-performance as your data volumes and throughput needs grow. Your database needs to scale seamlessly and linearly to deliver consistent, predictable performance to all users around the world. Additionally, your database needs to be easy to manage, so that you can focus on your application instead of managing the complexity of your database.
Why a NoSQL database is the right fit for personalization
Every database reflects a set of engineering tradeoffs. When relational databases were designed 40 years ago, storage, compute, and memory were thousands of times more expensive than they are today. Databases were deployed on a single server to a relatively small number of concurrent users, whose access to the systems tended to be during normal business hours when users had network access. Relational databases were designed with these resources, costs, and use in mind. They work very hard to be storage and memory efficient, and assume a single server for deployments.
As the costs of storage, memory, and compute decreased, and as data and workloads grew to exceed the capacity of commodity hardware, engineers began to reconsider these tradeoffs with different goals in mind. New types of databases later emerged that assumed distributed architectures so they could be easier to scale, especially with cloud infrastructure. With this approach the tradeoff in turn was to forego the sophistication of SQL and much of the data integrity and transactional capabilities developed in relational systems. These systems are commonly called NoSQL databases.
Traditional relational databases assume a fixed schema that will change infrequently over time. While this predictability of data structure allows for many optimizations, it also makes it difficult and cumbersome to add new and varying data elements in your application. NoSQL databases, such as key value stores and document databases, relax the rigidity of the schema and allow for data structures to evolve much more easily over time. Flexible data models speed the pace of innovation in your application, and increase your ability to iterate on your ML models, which is essential for personalization. In addition, the scalability of systems like Cloud Bigtable allow you to deliver personalization to millions of concurrent users while you continue to evolve how you personalize experiences for your customers.
How Cloud Bigtable enables personalization at scale
Cloud Bigtable supports personalization at scale with its ability to handle millions of requests per second, cost-effectively store petabytes of data1, and deliver consistent single-digit millisecond latencies for reads and writes. Bigtable delivers a unique mix of high performance and low operating cost to reduce your TCO.
We’ve heard from Spotify, Segment, and Algolia about how they’ve built personalized experiences for their customers with Bigtable. Check out this presentation to hear Peter Sobot of Spotify describe how they use Bigtable for personalization.
Let’s imagine a scenario where your application takes off like a rocketship, and grows to 250 million users. Let’s assume a peak 1.75 million concurrent users of your application2, with each user sending two requests per minute to your database. This will drive 3.5 million requests per minute to your database, or approximately 58.3K requests per second. Pricing for Bigtable to run this workload will start at under $400 per day3.
Bigtable scales throughput linearly with additional nodes. With separation of compute and storage, Bigtable automatically configures throughput by adjusting the association of nodes and data to provide consistent performance. When a node is experiencing heavy load, Bigtable automatically moves some of the traffic to a node with lower load to improve the overall performance. Bigtable also supports cross-region replication, with local writes in each region. This allows you to manage your data near your customers’ geographic locations, reducing network latency and bringing predictable, low-latency reads and writes to your customers in different regions around the world.
Bigtable is a NoSQL database developed and operated by Google Cloud. Bigtable provides a column family data model that allows you to flexibly store varying data elements for customers associated with their behavior and preferences, store a very large number of such data elements across your customers, and quickly iterate on your application. Bigtable supports trillions of rows with millions of columns. Each row in Bigtable supports up to 256 MB of data, so that you can easily store all personalized data for a customer in a single row. Bigtable tables are sparse, and there is no storage penalty for a column that is not used in a row; you only pay for the columns that store values.
BigQuery ML allows you to create and run ML models directly in BigQuery to develop personalization recommendations that you can bring back to Bigtable. You can easily pipe Bigtable data into BigQuery to run deep analytical queries and develop recommendations. These aggregates, like computed recommendations, are brought back to Bigtable so your application can serve those recommendations to users with low latency and massive scale.
Bigtable integrates with the Apache Beam ecosystem and Dataflow to make it easier for you to process and analyze your data. With application profiles and replication in Bigtable, you can isolate your workloads so that batch reads do not slow down your serving workload that has a mix of reads and writes. This enables your application to perform near real-time reads at scale to develop and train machine learning models in TensorFlow for personalization. Bigtable gives you the right operational data platform to develop personalization recommendations offline or in real-time, and serve them to your customers.
Here’s a look at conceptual schema examples for personalization in ecommerce:
And here’s a quick overview of what personalization use cases require, and how Bigtable addresses them.
Bigtable is fully managed to free you from the complexity of managing your database, so that you can focus on delivering a deeply personalized experience to your customers. Learn more about Bigtable.
1. Storage pricing (HDD) starts at $0.026 per GB/mo (us-central1)
2. Assumes application is used 24 hours a day, average user session is 5 minutes (Android app average), and daily peak is 2x the average. (250 million / (24 hours / 5 minutes) *2 = 1,736,111 peak concurrent users (us-central1 region)
3. Cloud Bigtable pricing for us-central1 region. Assumes 25 TB SSD storage (100 KB per user, for 250 million users) per month, 10 compute nodes per month (with no replication), includes data backup. Bigtable pricing details.
Rebel Foods Improves Accuracy of Forecast Time by 60% by Using Google Cloud
5321
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
Google Cloud Results
- Enables accurate allocation of marketing spend to underserved areas
- Supports expansion into international markets
- Helps ensure accurate forecasting of inventory levels
- Improves accuracy of forecasted delivery times by at least 60%
Operating in India since 2011, Rebel Foods has grown from a brick-and-mortar business that provided wraps to customers to a cloud kitchen that delivers cuisine to about one million consumers per month. “We started with the Faasos food brand and now we have scaled up to 10 brands,” says Soumyadeep Barman, Chief Technology Officer at Rebel Foods. “We have doubled our revenue every year from 2014 until now, and we operate kitchens in 15 cities across India. Each kitchen offers at least seven of our brands to customers.”
Barman attributes Rebel Foods’ success to the fact that it is a full stack company. “We procure, we have our own inventory, we prepare the food, we deliver the food to customers, and we make sure customers are delighted every time they order,” he says.
The rapid emergence and adoption of mobile technologies and services in India gave the business its opportunity to expand quickly. “The boom in applications and the web really got going in India in about 2014,” Barman says. “The subsequent emergence of smart devices and mobile applications opened up new markets, including older people who had not really used a computer until then.”
The business released the first iteration of its mobile application in 2013 on servers in an on-premises data center. “However, we experienced breakages because our infrastructure was not scalable or dependable enough, and we decided to move to another solution,” Barman says.
Google Maps Platform delivers opportunity
In 2014, Rebel Foods decided to move to the cloud and selected Google Cloud because of its stability, reliability, and scalability.
The business also wanted to take advantage of the opportunities Google Maps Platform presented to improve the efficiency and effectiveness of its delivery service. With 175 kitchens delivering to about 900 locations across India, Rebel Foods needs to provide estimated delivery times and meet delivery guarantees, while accounting for all the factors that might affect how quickly a rider can reach a customer’s doorstep.
The business turned to Google Maps Platform Premier Partner Searce for support in leveraging Google Maps Platform APIs to deliver a compelling customer experience and improve its efficiency. “Searce helped us determine the Google Maps Platform APIs we should use across our mobile applications and websites, and how many licenses we needed to conduct activities like calculating estimated delivery time and reviewing order heat maps,” Barman says. “Thanks to the firm’s support, Google Maps Platform APIs were a game changer for us.”
Mapping customer locations
Customers accurately pinpoint their location in a map through functionality made available through the Places API and Geocoding API, in conjunction with the JavaScript API. Drivers use the Directions API to identify the quickest route to customers.
Customers can also track the progress of delivery and estimated time of arrival using an Android or iOS application, or the brand websites.
Deploying Google Maps Platform APIs enabled Rebel Foods to improve by up to 60 percent the accuracy of forecasted delivery times. “Rather than tell a customer we can reach them in, say, 45 minutes, based on previous experience and gut feeling, we can retrieve an accurate traffic scenario and calculate delivery times based on traffic congestion levels and likely average speeds,” Barman says
Allocating budget effectively
Google Maps Platform also allows the business to combine mapping of customers to individual kitchens and to how often customers place orders – and for what value. This enabled the business to understand where to allocate budget for local marketing to stimulate demand in underserved areas.
Google Maps Platform technologies complement Rebel Foods’ use of Google Cloud Platform services such as the BigQuery analytics data warehouse to process data used to forecast inventory levels and provide recommendations to customers based on previous usage and behaviors. The business also runs its key applications in Kubernetes Engine to achieve cost-effective scalability, so it can expand to international markets.
“We are targeting growth into a range of international markets in January 2019, including Australia, the Middle East, and Southeast Asia,” Barman says. “With the user data and experience provided by Google Maps Platform in particular, we are poised for success.”
5583
Of your peers have already watched this video.
1:30 Minutes
The most insightful time you'll spend today!
A Quick Guide to Cloud Monitoring
Cloud Monitoring is a tool that allows you to gain visibility into the performance, availability, and health of your applications and infrastructure. In this video, we show you what Cloud Monitoring is and how you can use it to custom define service-level objectives (SLOs), monitor application metrics, and the overall health of your applications infrastructure. Watch to learn how you can use Cloud Monitoring!
3478
Of your peers have already watched this video.
1:30 Minutes
The most insightful time you'll spend today!
Pharma Firm Drives 80% Improvement in Speed with SAP on Google Cloud
FFF Enterprises is a leading supplier of critical-care biopharmaceuticals, plasma products, and vaccines. Their passion for patient safety and product efficacy drives their mission of Helping Healthcare Care.
For FFF Enterprises if they have to focus on ERP infrastructure, that takes away from getting products to patents. Learn why FFF Enterprises chose to deploy SAP on Google Cloud and drove an 80% improvement in speed for their SAP environment at a lower cost.
Google Cloud Accelerates Financial Organizations’ Journey towards Digital Transformation
10499
Of your peers have already read this article.
2:00 Minutes
The most insightful time you'll spend today!
When I reflect back on the past year and the pandemic, I’m struck by how the reliance on remote work and operations has changed the fundamentals of business forever. For the financial services industry, this rings particularly true. Many conversations I’m having right now with organizations revolve around embracing a transformation cloud, and thinking of cloud computing not just as an infrastructure decision, but also as the locus for transformation throughout the company.
Today, as we welcome the industry to our Financial Services Summit, we’ll demonstrate just how Google Cloud accelerates a financial organization’s digital transformation through app and infrastructure modernization, data democratization, people connections, and trusted transactions. We hope you’ll join us.
How we’re helping financial services firms build their transformation clouds
At Google Cloud, we continue to focus on areas where we can bring the best of our capabilities to banking, capital markets, insurance, and payments customers around the world. Our work with financial services industry customers has given us a deep understanding of the real-world, specific use cases that matter most to them. This groundwork led us to engineer products and solutions that are open and flexible, not ones that force them to rip out existing investments in ERP or other early IaaS cloud implementations.
It’s why we’ve engineered solutions such as Lending DocAI, Open Banking with Apigee, and Datashare for financial services to help transform the industry. These solutions were created with our customers’ security and compliance top-of-mind and are built at the intersection of user-optimized experience, technology, and sovereignty.
At their core, financial institutions want to drive growth, reduce costs, mitigate risk, stay compliant, and increase efficiency. As a result, when we partner with them on their transformation journeys, we consider three essential focus areas:
- Enabling the human experience and connected interactions
- Building an open and intelligent data foundation for better insights
- Providing the most trusted and secure cloud in the industry
Enabling humans and connected interactions
A company’s transformation is about more than technology; people and culture ultimately drive change. HSBC, for example, recognized its business users would benefit from guided answers to common questions around risk policy compliance, and turned to Google Cloud to leverage AI and machine learning bots to assist employees, ease the burden on policy experts, and improve the user experience. Using Dialogflow, a core component of Contact Center AI, HSBC was able to build a conversational platform that quickly and accurately addresses user needs at scale.
Another example is Equifax, which used Google Workspace to support collaboration not only internally between employees, but also externally with customers. Customers can use Google Cloud solutions for financial services to build these sorts of technology-enabled human interactions quickly and easily—supporting organizational change at scale.
Building an open and intelligent data foundation for smarter, faster insights
The real impact of Google Cloud solutions for financial services comes when the whole company has access to the right information at the right time, and can act more intelligently on that data. Our solutions help businesses safely leverage their data and get a complete 360-degree view of their customers’ information, which can often be scattered across multiple systems (CRM, lending, credit, etc.). This helps financial institutions improve the overall customer experience—and sometimes even develop new products quickly.
Indeed, all financial institutions are looking for ways to grow revenue and reduce expenses, and data can be a critical ingredient to doing both effectively. As daily transactions rise, so does the volume and complexity of data. But to implement new customer experience innovations (and new revenue streams), financial institutions must first capture data effectively. This is why AXA Switzerland, for example, uses real-time analytics on Google Cloud to gain cross-industry insights about future trends and customer preferences.
Financial services organizations also need the confidence of building on a platform that provides choice, flexibility, and agility to move fast. It’s why we have an open cloud approach that allows Google Cloud services to run in different physical locations such as on-premises, other public clouds, and the edge. Customers can also harness the power of data and AI through our open APIs, machine-learning services, and analytics engines on any major cloud platform. This is why companies like Macquarie Bank are taking advantage of Google Cloud’s open, hybrid architecture to modernize and empower its developers.
Compliant and secure to address risk and regulatory needs
As a highly regulated industry, financial services is focused on security and compliance, risk and regulations, and fraud detection and prevention. Google Cloud offers unique capabilities to earn customers’ trust as part of our continuing work to be the most trusted cloud in the industry. Google Cloud provides a secure foundation that you can verify and independently control. Our cloud technology reduces risk and data loss, because it is built on comprehensive zero-trust architecture. Finally, we offer a shared-fate model built on best practices in risk management via automation, guidance, and insurance. This is why customers like BBVA have confidence anywhere their systems may operate.
On the regulatory front, global legislators and regulators continue to focus on the stability of the industry that only a decade ago went through one of the biggest liquidity crises in history. With this oversight comes strong expectations of risk mitigation. Google Cloud offers a single, global set of controls, reviewed by financial institutions and regulators around the world, and verified in collaborative audits—making compliance simpler and less costly for our customers.
Finally, Google Cloud allows financial services firms to operate confidently with advanced security tools that help protect data, applications, and infrastructure, as well as their customers from fraudulent activity, spam, and abuse. We help protect your data against threats, using the same infrastructure and security services we use for our own operations, ensuring you never have to trade-off between ease of use and security. Google Cloud encrypts data at-rest and in-transit. And we now also offer the ability to encrypt data-in use, while it’s being processed for customer VM and container workloads.
Let us help you with your transformation cloud journey
We’ve seen leading financial services companies embrace Google Cloud to help them move beyond infrastructure toward the next phase of their cloud evolution. This is an era where no company is better positioned to lead than Google Cloud, and we’re excited to help you with your journey.
Learn more about Google Cloud for financial services.
Cloud Bigtable brings database stability and performance to Precognitive
5734
Of your peers have already read this article.
3:40 Minutes
The most insightful time you'll spend today!
At Precognitive, we were able to start with a blank technology slate to support our fraud detection software products. When we started building the initial version of our platform in 2017, we had some decisions to make: What coding language to use? What cloud infrastructure provider to choose? What database to use? The majority of the decisions were straightforward, but we struggled to decide upon a database. We had plenty of collective experience with relational databases, but not with a wide-column database like Cloud Bigtable—which we knew we’d need to scale our behavior and device workloads. At launch, our products were supported by a self-managed database, but we quickly migrated to Cloud Bigtable, and we love it.
To efficiently support our bursty, real-time fraud detection workloads, we needed a cloud database that could satisfy the following key requirements:
- Stability to keep up with increased adoption of our products
- Intelligent scaling that avoids bottlenecks
- Native integrations with BigQuery and Cloud Dataproc
- Managed services that free up our engineers’ time to work on our products
Adding Cloud Bigtable as our performance database
As we scaled our services and added customers, our data collection services for our Device Intelligence and Behavioral Analytics products were seeing thousands of events per second. Cloud Bigtable provided a stable managed database that could handle the volume we were receiving during peak hours. We weren’t always able to handle this scale, as an early version of our product utilized a self-managed database.
Every month, two or three engineers spent hours managing the database instances. Whenever the instances crashed, it would cost at least one engineer a day or two of productivity attempting to restore the instances and recovering any data from our backup database. Managing this database internally was taking precious time away from product development.
We circled back to Cloud Bigtable. After two weeks of R&D, we decided to switch the Device Intelligence and Behavioral Analytics services to Cloud Bigtable.
Cloud Bigtable solved our scaling issues. Cloud Bigtable had been attractive to us from the start because it was fully managed, and offered regional replication and other features we were lacking in our own managed instances. Cloud Bigtable provides horizontal scaling and automatically rebalances row keys (equivalent to a shard key) over time to prevent “hot” nodes. In addition, Cloud Bigtable provides a connector to BigQuery and Cloud Dataproc that allows us to analyze the terabytes of data we are processing and use that data for unsupervised machine learning.
The perks of using Cloud Bigtable
After the migration to Cloud Bigtable, we noticed a number of additional benefits: improved I/O performance, a significant cost reduction, and a sizable decrease in hours spent on database maintenance.
We measured some of our typical metrics before and after implementing Cloud Bigtable. Our request latency dropped by about 30 ms on average (to sub-10 ms) for API requests. Prior to the change, we were seeing latencies of 40+ ms on average. This latency drop on our Behavioral Analytics and Device Intelligence products allowed us to trim about an additional 10 to 15 ms off our average response time across all dependent services.
Before we moved to Cloud Bigtable, we had to scale our database instances every time a new customer was onboarded. We were over-scaling in an attempt to avoid constantly resizing our database servers. By sunsetting our self-managed database and switching to Cloud Bigtable, we cut database infrastructure costs by approximately 35% and can now scale as needed, with a couple of clicks, during onboarding.
We have spent zero hours managing a Cloud Bigtable database since launch, and we put the time we are saving every month toward product development.
Moving forward with Cloud Bigtable
As an engineering team, we love working with Cloud Bigtable. We are not only seeing improved developer experience and reduced latency, which keeps the engineers happy, but also reduced costs, which keeps the business happy. We’re able to build more product, too, with the time we’ve saved by switching to Cloud Bigtable. Stay tuned to our engineering blog for more on the lessons we’ve learned and our contributions to the wider Cloud Bigtable community.
More Relevant Stories for Your Company
Twitter Charts #HybridCloud Journey With Google Cloud
Social media giant Twitter needs no introduction. The 24/7 live platform, which crunches massive volumes of data every second, was using its data centers for a lot of its infrastructure and used the cloud for some of what it does. However, it needed ever more storage and compute resources and
Casper on Google Cloud: Revolutionizing Web3 Development with Flexibility & Security
Casper Labs announced a collaboration with Google Cloud that will allow developers to launch public and/or private Casper nodes directly from Google Cloud. This enables a much more seamless and highly secure process for the millions of developers who want to build in blockchain environments without having to learn new,
8 Must-Have Google Cloud Products for Startups
Startups worldwide turn to Google Cloud tools to build fast on a strong and easy to use platform that helps them get to market and launch products faster, all while building on the cleanest cloud in the industry. Startups leverage Google Cloud and our Google for Startups Cloud Program to
How Carrefour is Building the Future of Retail on Google Cloud
Multinational retailer Carrefour was facing the challenge of meeting increasing customer expectations and realized that it needed to innovate faster. That's when the company decided to move its SAP workloads to Gooogle Cloud. As a result, Carrefour transformed over 1,000 stores and redesigned its back office management with SAP on
