As organizations expand their use of cloud computing services, more of their sensitive data inevitably moves to and lives in the cloud. Much of this sensitive data is unstructured and can be challenging to secure. Despite this potential challenge, the usefulness of cloud for data storage and processing is too big for most organizations to ignore and has in turn led to data sprawl, where their sensitive data is spread over many resources, both in the cloud and on-premise. Addressing data sprawl requires solutions that can discover, manage, and secure sensitive data, especially unstructured data, as it spreads.

To help organizations confidently move their sensitive data to the cloud, Google Cloud works diligently to earn and maintain customer trust. Control and transparency are pillars of our approach to offering a trusted cloud. Therefore, we’ve been expanding our capabilities to act on unstructured data as sprawl increases.

Given the importance of these capabilities to our strategy, we are happy to announce today that Forrester Research has named Google Cloud a Leader in The Forrester Wave™: Unstructured Data Security Platforms, Q2 2021 report, and rated Google Cloud highest in the current offering category among the providers evaluated.

The report evaluates the 11 most significant providers with platform solutions to secure and protect unstructured data, spanning from cloud providers to data security-focused vendors. The report notes that “Google offers breadth and depth with built-in data security in the cloud. Google Cloud Platform, Google Workspace, and BeyondCorp Enterprise have underlying data security products and features for protecting customer data.”

Google Cloud tools focused on protecting unstructured data were developed and battle-tested internally at Google to alleviate some of our own data security challenges. This brings the best of Google security to the organizations utilizing Google Cloud and our security tools. The report highlights that “Google productizes capabilities originally developed to secure its own business, and brings a disciplined approach to product enhancements for enterprise requirements. It serves a wide range of enterprise and mid-market, with a focus on emphasizing data protection needs by industry. ”

Google Cloud’s data security strategy focuses on meeting customers wherever they are in their cloud migration journey. The report highlights that “Google further enables a Zero Trust approach with third-party integrations through its BeyondCorp Alliance of partners in device management, endpoint security and gateways.”

Google Cloud received the highest possible score in sixteen criteria, in total receiving the most 5 out of 5 ratings among all vendors assessed. These criteria include: Data Intelligence, Access Control, Deletion, Obfuscation-Scope, Obfuscation-Key Management, Deployment, Security and Risk, APIs and Integration, Data Security Platform Vision, Data Security Execution Roadmap, Performance, Planned Enhancements, Zero Trust Enabling Partner Ecosystem, Diversity, Equity and Inclusion, Installed Base, and Revenue. 

Notably, Google Cloud received the highest possible score in the Obfuscation criteria. Obfuscation can help protect sensitive data, like personally identifiable information (PII), which is critical to many enterprise workflows. Cloud DLP helps customers inspect and mask this sensitive data with techniques like redaction, bucketing, and tokenization, which help strike the balance between risk and utility. This is especially crucial when dealing with unstructured or free-text workloads, in which it can be challenging to know what data to redact. More than 150 detectors combine to power Cloud DLP’s masking, which can be deployed in data migrations and business workloads like real-time data collection and processing. For Obfuscation specifically, the report mentioned that Google “takes a broad view of DLP, which includes in-line redaction of sensitive elements in unstructured data and DLP APIs that extend support to additional data types like images or other media.”

We are honored to be a Leader in The Forrester Wave™ Unstructured Data Security Platforms Q2 2021 report, and look forward to continuing to innovate and partner with you on ways to make your digital transformation journey safer as we work to become your most trusted Cloud.

A copy of the full report can be viewed here.

Education data can inform strategy, enhance both teaching and learning, and help us better understand customer needs. But that data is only as useful as an organization’s ability to access, analyze, and act upon it. Education Technology (EdTech) companies with a strong data warehouse management policy can help to make the most of educational data at all levels.

Creating a data team

When an EdTech doesn’t have a dedicated data team, databases will likely be poorly maintained and data can be hard to find, it can lack critical security, or fail to meet a customer’s specific requirements. A dedicated data team can prevent this and better support an organization’s data needs.  There’s a correlation between the maturity of a data team and the success of an organization’s data utilization. When considering the three primary components of team, outlook, and results, we can better understand an organization’s maturity. By using this matrix from Looker’s Analytical Maturity Playbook, you can assess the maturity of your organization’s data team. Most would agree, the goal is for your operation to move from data chaos to data-driven. But, how can you do that? Think about your data this way: the intelligence you gain from your data is not only an asset…it’s a product. When your data team views your business intelligence as a product—one that’s just as important as the services you offer—they can set the stage for creating a successful data-driven company. Mature data organizations:

• Report functionality regularly
• Constantly assess potential improvements to existing systems
• Manage content rollout to internal employees

Redivis: Making data accessible, connected & secure

Redivis is a platform that connects academic researchers with data and the tools to understand education intelligence. Their customers, who are researchers, expressed how difficult it is to find new datasets for their studies. Once they found a dataset, they often had to access and work with it before they could tell if the information would be useful for their studies. Redivis Data administrators were also concerned about data security. Redivis built their platform on top of Google Cloud’s security infrastructure, making it a highly scalable data processing environment. 

Redivis developed a transparent, tiered access system for datasets that allows researchers to request separate access to a dataset’s documentation to determine whether they can use the dataset. Detailed audit logs (supported by Google Cloud Logging) and robust application-level security controls tighten data access. To offer more insights for their researchers, they securely connected their private datasets with public datasets hosted in BigQuery allowing researchers to run queries on billions of records in seconds, not hours.

With a data team in place, it’s time to visualize where to store and organize your data. Data is an asset. You want to invest in collecting, operationalizing, and storing it properly. “Inadequate data aggregation, harmonization, and processing impedes teams from making the right data-driven decisions and pivoting when needed,” according to Jesus Trujillo Gomez, a strategic business executive at Google Cloud. You can determine how mature your warehouse is by using this matrix to help you identify where your organization falls on the spectrum (least mature on the left, most mature on the right). A company’s data warehouse can be assessed in four areas: Technology, Pipeline, Quality, and Performance. Here are some suggested steps to bring your EdTech company one step closer to a data powerhouse. 

Classcraft: Using a data-driven approach to make learning fun

Launched in 2014, learning company Classcraft is reimagining the classroom, creating better learning environments with gamification. To support their mission, their team has created a complete data platform that uses data analytics to find new ways to engage students in the classroom. Classcraft became a Google for Education Build Partner to keep up with rising demand. They expanded their footprint to reach 6 million students and educators in 165 countries. 

They also migrated to Google Cloud, building on several integrations with Google Workspace for Education, Google Classroom, and Chrome for Education, allowing them to support customers ranging from individual teachers to entire school districts.  Classcraft migrated its database and analytics infrastructure to BigQuery and Cloud Storage. “We capture about 10 million new data points every month related to school culture and student performance,” says Chief Technology Officer and VP of Product Stéphane Guillemette. “With BigQuery and other Google Cloud tools, we can quickly deliver valuable insights to educators.”

Every hour of every day, Google moves a lot of data around the world. But how? With the Effingo (Latin for ‘to duplicate’ or ‘copy’) data copying service, a Google-only service that moves data internally and on Google Cloud customers’ behalf.

As a Google Cloud customer perhaps you move data with Storage Transfer Service, a managed transfer option that provides out-of-the-box security, reliability, and performance, that eliminates the need to optimize and maintain scripts, and handle retries. Storage Transfer Service is useful for consolidating data from separate projects, to move data into a backup location, or to change the location of your data.

Likewise, Google’s internal services need very similar functionality to manage their data. This includes data that is in charge of the ecosystem of Google Cloud services, such as BigQuery and Cloud Spanner.

Effingo solves a challenging problem of data movement at a global scale. It supports a wide range of workloads with different traffic characteristics and different requirements for data replication, durability, and latency. In this article we will explore the main motivations behind data movement and solutions to core infrastructure problems that we face when copying exabytes of data on a daily basis.

Why move data in the first place?

But first, let’s think about why you even want to move or geographically distribute your data.

For one thing, you want to replicate your data for durability and reliability. You definitely don’t want to keep a single copy of your data. In the end, we’re talking about a file that is stored on a hard drive. Any hard drive can fail at any point. For this reason, keeping copies of your data in different locations helps ensure that at any point of time there is a copy of your file that you can read from. Moreover, these locations should be geographically distant from each other to avoid even temporary data loss due to natural disasters, network cuts, and other incidents that can impact a local area.

Second, you want your data to be close to your users to reduce latency. You want to serve data with minimal delay to give a great user experience. In many cases, delays caused by data transfers from remote locations can spoil the usability of your application or create the perception that your service doesn’t work. According to Google research, 53% of mobile users abandon sites that take over 3 seconds to load. There are other consequences of lack of data proximity: for example, transferring data from distant locations wastes network resources, which can have a negative impact on the environment. Google has been carbon-neutral for our operations since 2007, and we have matched 100% of the electricity consumption of our operations with renewable energy purchases annually since 2017, but we aim higher: Our goal is to run all our data centers and offices on carbon-free energy, 24/7, by 2030.

The third reason to geographically distribute your data is to balance storage capacity between clusters. This reduces the cost of storing data because it enables us to increase the overall capacity of our data centers and use computation power in less loaded clusters. We can then limit the amount of wasted resources and use otherwise idle hardware. This is especially relevant for batch processing and storing cold data where the exact storage location is not that important.

What are the challenges?

Operating at such a large scale as Google makes data movement a challenging problem for several reasons. Here are a few examples.

Data must be secure and consistent at its destination. Achieving this is complex. For example, to copy a file from one location to another, you need the following: throughput to read and write to disks, network capacity to transfer bytes between data centers, and compute resources to instrument the whole operation. 

Considering the volume of data that Google operates on, high and predictable throughput is one of the key features of a copy service at scale. It is typical to transfer terabytes of data per second, and so it’s essential to have a scalable service that can handle traffic with sometimes rapidly changing patterns. It poses a further question: how to run such a service and still be resource efficient?

Finally, it’s crucial to ensure high system availability. Google infrastructure is dynamic and copying data between any two clusters may behave differently at different points of time. Effingo needs to be resilient to cluster turndowns, network bottlenecks, rapid changes in resource demands and capacity, and so on.

How do we solve (some) of these problems?

One of the core use cases for data movement is replication. If you want to ensure that your data is reliable, you want to keep more than one copy, ideally in different locations, to be less prone to data loss.

Let’s consider the following example. You have a file in cluster A, and you want to make copies in destinations B – G.

In the simplistic approach, you would start copying to all destinations in parallel.

Unfortunately, it is highly inefficient to replicate your file this way. This solution doesn’t scale and requires significant investment in infrastructure. 

If you consider the cost of network infrastructure, moving data across the ocean is an expensive operation. Effingo creates data transfer plans that reduce the volume of data to transfer across the ocean. If possible, it reuses already replicated data as a new source.

To create data transfer plans Effingo needs to know alternative data sources and be aware of the network topology. 

If it goes to the alternative sources, Effingo stores a recent history of transferred files together with their metadata and time-bound permissions to access the files. Whenever a new copy is issued, Effingo checks whether there were copies of the original file that could serve as the same source but in the alternative location. Worth noting is that Effingo is very strict in verifying that both files — the requested source file and the alternative transfer file — are indeed the same to follow high security standards. Effingo not only checks whether the same user issued a copy from the same source but also if file properties including checksum, ciphertext checksum, mtime and several others match.

Once Effingo knows the source files and their alternative sources, it creates a transfer plan over the Google network. Effingo has a model of the network in the form of a graph where each location is a node and each edge is a weighted link between each node. For each copy Effingo creates a Minimum Spanning Tree over such a graph, which serves as an input to the transfer engine. Thanks to this approach we can select a plan that is optimal for each copy.  

In the next example, we show a more efficient approach. Effingo first makes an expensive copy to a remote location and then uses it as a new source. For instance, Effingo first copies a file from the US (source A) to Europe (destination C) and then uses the Europe-based file as a copy source for copies on the same continent. Note that Effingo never stores files in temporary locations for optimization purposes – the service only uses locations explicitly requested by our users.

In addition to resource efficiency, the main challenge is to ensure high throughput of data transfer. We solve this problem by applying several approaches that address different problems that may arise.

As described above, our customers’ workloads exhibit a range of different traffic patterns, which results in dynamic changes in capacity on each network link. For this reason, Effingo not only needs to respond to altering resource availability, but also to ensure user fairness in such conditions. In other words, Effingo wants all copy operations to progress while using resources in the best possible way. To achieve this, it uses sophisticated parallelism controls. These controls scale the service up to meet increased demand and limit service capacity if there is service overload, performance degradation, or resource waste.

Further, Effingo must be very resilient and react quickly to errors. On the file level, the service extensively uses retries for transient failures and aborts transfers quickly if it detects non-retryable errors. Effingo uses metrics, logs, and other observability signals to adjust data transfers when needed. For example, it can detect that copies from a specific source are slow and there is another file replica available. Effingo then reconfigures the copy operation to use the other replica to complete the data move.

Data movement at global scale is hard

Hopefully at this point you have a better understanding of why moving data is a challenging problem for infrastructure that operates at Google’s global scale. Effingo supports a range of services that run on Google infrastructure, including Google Cloud services and many internal Google services. While moving data at large scale requires a lot of attention to resource usage and resilience to support high throughput, there is good news: We keep working on this problem and make our infrastructure better every day, so you can run your business on Google Cloud and all data movement is transparent to you.

Japan is home to millions of vending machines installed on streets and in buildings, sports stadiums and other facilities. Vending machine owners and operators, including beverage manufacturers, stock these machines with different product combinations depending on location and demand. For example, they primarily display coffee and energy drinks in machines placed in offices and sports drinks and mineral water in machines at sports facilities. The combinations also vary by season: for example, owners and operators may display cold beverages in summer and hot beverages in winter. 

Traditionally, vending machine operators have relied on the intuition and experience of sales managers to determine the optimum product mix for each vending machine. However, in recent years, manufacturers such as Coca-Cola Bottlers Japan (CCBJ) have turned to data to analyze and make strategic decisions about when and where to locate products in machines.

CCBJ is the number one Coca-Cola bottler in Asia and vending machines comprise the bulk of its business. The organization operates about 700,000 machines across Tokyo, Osaka, Kyoto, and 35 prefectures. Minori Matsuda, Google Developer Expert and also Data Science Manager at CCBJ, says “The billions of data records collected from 700,000 physical devices are a great asset and a treasure trove we can take advantage of.”

Minori points out that when considering the mix of products in vending machines in sporting facilities, the managers naturally assume sports drinks would generally sell well. However, analysis of purchase data – including hot drinks and hot drinks plus sports drinks – found many parents purchased sweet drinks such as milk tea when they attended games or sessions involving their children.  “Analyzing data gives us new discoveries and, by using catchy storytelling techniques from exploratory data analysis, we are instilling a data culture within our company,” he says. “It’s worth creating by looking at facts rather than making assumptions!”

Minori believes that to analyze the vast amount of data collected from more than 700,000 vending machines, the business needs a powerful analytical platform. However, until recently, CCBJ had to extract data for analysis from its core systems, load this data into a warehouse it created and perform the required analyses.  The billions of records of data generated across the fleet – including transaction data – exposed some challenges for traditional analysis platforms. They could not efficiently process data at a considerable scale: it could take a day to return results and required extensive maintenance due to the size.

CCBJ considered building a machine learning (ML) platform as a layer on top of existing systems in August 2020 and opted for Google Cloud the following month.  “I feel that Google Cloud has an edge in all products and is very well thought out,“ says Minori, noting the scalability and cost of the platform allow the business to take a ‘trial and error’ approach to achieve the best outcomes from ML. Google Cloud also delivered the required visibility and flexibility to help the business deliver change every day against key performance indicators. 

MLOps platform streamlines ML pipeline development

CCBJ built its analysis platform using Vertex AI (formerly AI Platform) centered on a BigQuery analytics data warehouse, and partly using AutoML for tabular data. “We have created a prediction model of where to place vending machines, what products are lined up in the machines and at what price, how much they will sell, and implemented a mechanism that can be analyzed on a map,” says Minori, adding that building the platform with Google Cloud was not difficult. “We were able to realize it in a short period of time with a sense of speed, from platform examination to introduction, prediction model training, on-site proof of concept to rollout.”

The new data analytics platform of CCBJ consists of the following parts:

Data Sources

• The data collected from the vending machines are all stored on BigQuery.

Data Discovery and Feature Engineering

• Minori and other data scientists at CCBJ are using Vertex Notebooks, where they access the data on BigQuery by executing SQL queries directly from the Notebooks. This environment is used for the data discovery process and feature engineering. 

ML Training

• For ML training, CCBJ uses AutoML for Tabular data, Custom model training on Vertex AI, and BigQuery ML. AutoML gives model performance with AUC curves and also feature importance graphs.

ML Prediction and Serving

• For ML prediction, CCBJ uses Online Prediction for AutoML models and Online Prediction for custom models for real-time prediction when the salesperson finds the interesting point
• Batch Prediction is used for generating a large prediction map that covers the whole country
• The prediction results are distributed to sales managers’ tablets

CCBJ started constructing the platform in September 2020, and completed it within a month. The business has conducted proofs of concept at its base in Kyoto since February 2021, and since April, has rolled out the platform to sales managers in 35 prefectures in one metropolitan area. “Data analysis is built into the day-to-day routines of sales managers with 100% utilization,” says Minori. “They can utilize the prediction results on tablets that were able to achieve pretty high accuracy from the start.”

The hardest part was the education of sales managers in the field; having them understand the reasoning behind the ML prediction results for particular outcomes, so they could be convinced to make use of the results. “For example, regarding a new installation location predicted by the model, it seemed that there was no effective information for installation from the map information, but when I actually went there, there was a motorcycle shop and it was a place where young people who like motorcycles gathered,” says Minori. “Or there is a small meeting place where the elderly in the neighborhood are active. 

“In many cases, new discoveries that cannot be understood from map information alone can be derived from the data.”

Minori also points to a phenomenon whereby humans pursued and confirmed factors inferred by the model – meaning that once they experienced analysis and it worked effectively, they asked why the same type of analysis or prediction could not be undertaken next time. The resulting cycle of more inquiries generated, more information gathered and more data captured for analysis meant the accuracy of results was improved.

Minori describes Vertex AI as having a number of strengths in helping CCBJ build a ML data analysis platform. “One of the major merits of Vertex AI was that we were able to realize MLOps that streamlines the entire development life cycle from construction of the ML pipeline to its execution,” he says.

With near real-time data analysis through Google Cloud, CCBJ teams can spend time developing strategies rather than waiting for data requested from the IT systems department. Exploratory data analysis is also considerably easier as repeated trial and error has greatly improved the accuracy of analyses. Before we used Machine Learning, most machine placement processes were done by human senses, by looking at a map to find the suggestion points. By using Machine Learning to generate a massive number of placement point suggestions, the efficiency of routing of salespeople has been dramatically improved. 

In the future, CCBJ aims to automate the continuous training pipeline with Vertex AI. “CCBJ is a tech company that operates in the food industry,” says Minori. With the organization operating a vending machine network of 700,000 units, it would like to create new businesses based on utilization and analyzing data. Some of these businesses may be based on Sustainable Development Goals (SDGs) initiatives such as the utilization of recycled PET bottles, measures to prevent food loss and ways of using vending machines to contribute to local communities, which we have been working on for some time. It would be interesting if we could collaborate with Google Cloud on these in the future.”

Customer expectations have shifted as a result of evolving needs. Across industries, customers expect that you treat them as individuals, and demonstrate how well you understand and serve their unique needs. This concept—personalization—is the idea that you’re delivering a tailored experience to each customer corresponding to their needs and preferences; you’re setting up a process to create individualized interactions that improves the customer’s experience. According to Salesforce, 84% of consumers say being treated like a person, not a number, is very important to winning their business. 

Entire industries are undergoing digital transformation to better serve their customers through personalized experiences. For example, retailers are improving engagement and conversion with personalized content, offers, and product recommendations. Advertising technology organizations are increasing the relevance and effectiveness of their ads using customer insights like their specific interests, purchasing intent, and buying behavior. Digital music services are helping their customers discover and enjoy new music, playlists, and podcasts based on their listening behavior and interests. 

As customers want more and more personalization, modern technology is making it possible for many more businesses to achieve this. In this post, we’ll look at some common challenges to implementing personalization capabilities and how to solve them with transformative database technologies like Google Cloud’s Bigtable. Bigtable powers core Google services such as Google Maps that supports more than a billion users, and its petabyte scale, high availability, high throughput, and price-performance advantages help you deliver personalization at scale.  

Challenges with personalization 

Data is at the heart of personalization. To deliver personalization at scale, an application needs to store, manage, and access large volumes of data (a combination of customer-specific data and anonymized aggregate data across customers) to develop a deep understanding of the behavior, needs, and preferences of each customer. Your database needs to very quickly write large volumes of data concurrently for all active customers. You need to continuously capture data on customer behavior because each step potentially informs the next, e.g., adding an item to a shopping cart can be used to trigger new recommendations for related or complementary products. Much of this data needed for personalization is semi-structured and sparse, and therefore requires a database with a flexible data model. 

Personalization at scale requires large volumes of data to be read in near real-time so that it can be in the critical serving path to deliver a seamless user experience, often with a total application latency of less than 100ms. This means your requests to the database need to return results with latencies of single-digit milliseconds. You need to ensure that application latencies do not degrade as you onboard more customers. Data needs to be organized efficiently and integrated with other tools so that you can run deep analytical queries and use machine learning (ML) models to develop personalized recommendations, and store the aggregates in your operational database for serving your customers. You also need the ability to run large batch reads for analytics without affecting the serving performance of your application. 

In addition, you need to ensure that your database costs do not explode with the popularity of your application. Your database needs to consistently deliver low total cost of ownership (TCO), and high price-performance as your data volumes and throughput needs grow. Your database needs to scale seamlessly and linearly to deliver consistent, predictable performance to all users around the world. Additionally, your database needs to be easy to manage, so that you can focus on your application instead of managing the complexity of your database.

Why a NoSQL database is the right fit for personalization 

Every database reflects a set of engineering tradeoffs. When relational databases were designed 40 years ago, storage, compute, and memory were thousands of times more expensive than they are today. Databases were deployed on a single server to a relatively small number of concurrent users, whose access to the systems tended to be during normal business hours when users had network access. Relational databases were designed with these resources, costs, and use in mind. They work very hard to be storage and memory efficient, and assume a single server for deployments. 

As the costs of storage, memory, and compute decreased, and as data and workloads grew to exceed the capacity of commodity hardware, engineers began to reconsider these tradeoffs with different goals in mind. New types of databases later emerged that assumed distributed architectures so they could be easier to scale, especially with cloud infrastructure. With this approach the tradeoff in turn was to forego the sophistication of SQL and much of the data integrity and transactional capabilities developed in relational systems. These systems are commonly called NoSQL databases.

Traditional relational databases assume a fixed schema that will change infrequently over time. While this predictability of data structure allows for many optimizations, it also makes it difficult and cumbersome to add new and varying data elements in your application. NoSQL databases, such as key value stores and document databases, relax the rigidity of the schema and allow for data structures to evolve much more easily over time. Flexible data models speed the pace of innovation in your application, and increase your ability to iterate on your ML models, which is essential for personalization. In addition, the scalability of systems like Cloud Bigtable allow you to deliver personalization to millions of concurrent users while you continue to evolve how you personalize experiences for your customers.

How Cloud Bigtable enables personalization at scale

Cloud Bigtable supports personalization at scale with its ability to handle millions of requests per second, cost-effectively store petabytes of data¹, and deliver consistent single-digit millisecond latencies for reads and writes. Bigtable delivers a unique mix of high performance and low operating cost to reduce your TCO. 

We’ve heard from Spotify, Segment, and Algolia about how they’ve built personalized experiences for their customers with Bigtable. Check out this presentation to hear Peter Sobot of Spotify describe how they use Bigtable for personalization. 

Let’s imagine a scenario where your application takes off like a rocketship, and grows to 250 million users. Let’s assume a peak 1.75 million concurrent users of your application², with each user sending two requests per minute to your database. This will drive 3.5 million requests per minute to your database, or approximately 58.3K requests per second. Pricing for Bigtable to run this workload will start at under $400 per day³.

Bigtable scales throughput linearly with additional nodes. With separation of compute and storage, Bigtable automatically configures throughput by adjusting the association of nodes and data to provide consistent performance. When a node is experiencing heavy load, Bigtable automatically moves some of the traffic to a node with lower load to improve the overall performance. Bigtable also supports cross-region replication, with local writes in each region. This allows you to manage your data near your customers’ geographic locations, reducing network latency and bringing predictable, low-latency reads and writes to your customers in different regions around the world. 

Bigtable is a NoSQL database developed and operated by Google Cloud. Bigtable provides a column family data model that allows you to flexibly store varying data elements for customers associated with their behavior and preferences, store a very large number of such data elements across your customers, and quickly iterate on your application. Bigtable supports trillions of rows with millions of columns. Each row in Bigtable supports up to 256 MB of data, so that you can easily store all personalized data for a customer in a single row. Bigtable tables are sparse, and there is no storage penalty for a column that is not used in a row; you only pay for the columns that store values.

BigQuery ML allows you to create and run ML models directly in BigQuery to develop personalization recommendations that you can bring back to Bigtable. You can easily pipe Bigtable data into BigQuery to run deep analytical queries and develop recommendations. These aggregates, like computed recommendations, are brought back to Bigtable so your application can serve those recommendations to users with low latency and massive scale. 

Bigtable integrates with the Apache Beam ecosystem and Dataflow to make it easier for you to process and analyze your data. With application profiles and replication in Bigtable, you can isolate your workloads so that batch reads do not slow down your serving workload that has a mix of reads and writes. This enables your application to perform near real-time reads at scale to develop and train machine learning models in TensorFlow for personalization. Bigtable gives you the right operational data platform to develop personalization recommendations offline or in real-time, and serve them to your customers.

Here’s a look at conceptual schema examples for personalization in ecommerce:

And here’s a quick overview of what personalization use cases require, and how Bigtable addresses them.

Bigtable is fully managed to free you from the complexity of managing your database, so that you can focus on delivering a deeply personalized experience to your customers. Learn more about Bigtable.

<sup>1. Storage pricing (HDD) starts at $0.026 per GB/mo (us-central1)
2. Assumes application is used 24 hours a day, average user session is 5 minutes (Android app average), and daily peak is 2x the average. (250 million / (24 hours / 5 minutes) *2 = 1,736,111 peak concurrent users (us-central1 region)
3. Cloud Bigtable pricing for us-central1 region. Assumes 25 TB SSD storage (100 KB per user, for 250 million users) per month, 10 compute nodes per month (with no replication), includes data backup. Bigtable pricing details.</sup>