Cloud Spanner & Bigtable Helps Sabre Build Consistency & Scalability to Serve 1 Billion Travellers - Build What's Next
Blog

Cloud Spanner & Bigtable Helps Sabre Build Consistency & Scalability to Serve 1 Billion Travellers

3628

Of your peers have already read this article.

1:30 Minutes

The most insightful time you'll spend today!

Sabre, an innovative tech company leverages databases and AI to power travel companies across the globe. The firm took its 10-year old association with Google a step further by leveraging Spanner and Bigtable to vest its focus on innovation.

Sabre is an innovative software and technology company that leverages highly scalable databases and artificial intelligence (AI) to power travel industry partners around the globe. Our company processes more than 12 billion shopping requests and serves over 1 billion travelers every year.

The companies that partner with us include airlines, travel websites, agencies, and hotels. We are distinguished by a long history of integrating cutting edge technology into our operations. Our earliest innovations include building one of the first transaction processing systems in existence and a travel distribution network that predates the Internet! Through our next generation of AI-powered solutions, we are focused on optimizing the retailing, distribution, and fulfillment experience of the travel industry.

In my role as chief architect for Sabre Labs, I spearhead the long-term technology choices we are making to enhance the development, deployment, and operation of our software. We’ve embarked on a multi-year strategy to transform our technology and solutions. Rearchitecting our infrastructure to become fully cloud native is a central tenet of this transformation. In these early iterations of our plan, we have seen the significant impact of machine learning and its potential in bringing truly personalized travel experiences to customers. As part of this effort, we’ve established a 10-year partnership with Google to help accelerate our transformation and bring innovation to the travel industry.

Database choices require tradeoffs

The complexity and scale of the travel industry places high demands on the cloud services we utilize. We tend to place more emphasis on how a particular cloud service will impact an application’s reliability, performance, or development time, rather than choosing a service purely for its functionality. When it comes to databases, this can often mean making a tradeoff between latency and consistency.

The tradeoff exists for any database that serves multiple copies of its data in different availability zones or geographic regions for reliability. A database designed to ensure that everyone sees a consistent view of the latest data might update those copies synchronously using a consensus algorithm, which affects how quickly the data can be served. On the other hand, a database that’s optimized for faster data serving might update each copy asynchronously and not guarantee consistent reads across records.

Cloud Spanner and Bigtable–two of Google Cloud’s managed databases–are both highly effective services, and each one could support many of our travel applications. But as you will see, the latency vs. consistency tradeoff made it clear which one was best suited for two of our most critical cases.

Google Cloud Spanner facilitates strong, global consistency for Sabre

An airline’s reservation database stores a passenger’s booking information, seat selection, tickets, special requests, and other critical information about their trip. As a result, this data sits at the consistency end of that consistency/latency spectrum. Sabre typically processes thousands of reservation updates per second on behalf of our carrier customers. An airline’s reservation database must be served from many availability zones (and data is replicated across these availability zones) so that it remains available in the event of an outage. It also requires ACID properties for transactional updates across records since airlines often make changes to multiple passengers and multiple flights at the same time.

We needed a system that can handle bursts of concurrent updates, as would occur during a snowstorm when hundreds of thousands of passengers might be automatically moved to alternate flights. Spanner is a great fit for the reservations case because of its unique consistency guarantees. It processes over 1 billion requests per second at peak and provides five 9s SLA (99.999 percent) to support our applications. Spanner also helps us maintain compliance, business continuity, redundancy, and reliability using the same secure-by-design infrastructure, built-in data protection and replication, and multi-layered security that are essential to our Google Cloud workloads.

Spanner’s client libraries also provide built-in mechanisms to handle retrying in the event of write conflicts with another transaction and allow developers to choose stale reads in read-only transactions for improved performance. Of course, consistency across multiple zones or regions doesn’t come for free. It means higher write latencies than if we wrote to a comparable database running in a single availability zone, but for an application that manages flight reservations, it’s a tradeoff that makes sense.

Bigtable provides predictable, low latency at scale

Our flight shopping systems sit at the other end of the latency/consistency spectrum. Sabre’s shopping engine generates millions of itineraries per second on behalf of travelers using mobile apps, third-party travel websites, and airline call centers. Each itinerary requires significant compute resources to calculate: We need to find which combinations of flights make sense and evaluate complex rules about their availability and pricing. Users are typically less patient while searching for a flight than they are when booking it, so we needed a low latency solution.  But we can cache many of these shopping results to reduce our compute usage. For instance we can decide how long to cache results based on factors like how far the flight results are from the departure. 

Bigtable makes an excellent choice for this shopping cache. It’s a NoSQL database service built to handle high-throughput, low-latency applications, with more than 10 exabytes of data under management. Bigtable’s unique latency properties—such as predictability and single-digit millisecond response time even for multi-petabyte tables—enable us to serve large volumes of shopping results cost effectively, while providing low response times to travelers.

Google Cloud supports a focus on innovation

Managed databases like Bigtable and Spanner are a significant part of Sabre’s cloud strategy. The combination of unique tools like Key Visualizer for Bigtable and Spanner as well as integration with other Google Cloud services like Cloud IAM, Cloud Monitoring, and now Datastream, make the experience of operating managed databases with Google Cloud much easier than it is with several of their self-hosted counterparts. As a result of their granular pricing models and ability to be deployed and automated by SREs, the managed databases we use also end up with a lower total cost of ownership.

We’re particularly excited about a few recent database-related announcements from Google Cloud. Bigtable’s SLA update gives us more concrete expectations in terms of multi-cluster, multi-region uptime. Spanner’s change to provisioning in Processing Units increases its cost efficiency when deploying in non-production environments where we may need many isolated instances, but won’t come close to the limits of a single node. In those cases, Spanner instances may now be configured in one-tenth of a node increments.

Our cloud transformation depends on having a choice of databases for different use cases, trade offs, and migration schedules. In addition to managed databases, we expect to use self-hosted databases, database solutions available in Cloud Marketplace, and transitional services like Cloud SQL for a few more years. In an industry as demanding as travel, accelerating our most critical applications using technology unique to Google Cloud means less time spent optimizing latency and consistency, and more time spent innovating.

Learn more about how your organization can use Bigtable and Spanner.

Blog

Built with BigQuery: Retailers Unlock the Value of Data with SoundCommerce

3237

Of your peers have already read this article.

3:30 Minutes

The most insightful time you'll spend today!

Google Cloud has teamed up with SoundCommerce to help retailers make the most of their data. This partnership allows retailers to better analyze and understand their data, leading to improved profitability. Read more!

As economic conditions change, retail brands’ reliance on ever-growing customer demand puts these companies at financial and even existential risk. Top-line revenue and active customer growth do not equal profitable growth.

Despite multi-billion dollar valuations for some brands, especially those operating the direct-to-consumer model, the rising costs of meeting shoppers’ high expectations (i.e. free shipping, free returns) along with the escalating cost of goods, fulfillment operations, and delivery costs, create pressure for brands to turn a profit. The ONLY way brands drive profitable growth is by managing variable costs in real-time. This in turn mandates adopting modern data cloud infrastructure including Google Cloud services such as BigQuery.

To unlock the value of data, Google Cloud has partnered with SoundCommerce, a retail data and analytics platform that offers a unique way of connecting marketing, merchandising, and operations data and modeling it within a retail context – all so brands can optimize profitability across the business.

Profitability can be measured per order through short-term metrics like contribution profit or long-term metrics like Customer Lifetime Value (CLV). Often, retailers calculate CLV as a measure of revenue with no consideration for the variable costs of serving that customer, for instance: the costs of marketing, discounting, delivering orders to the doorstep, or post-conversion operational exceptions (e.g. cancellations, returns).

What may first appear to be a high lifetime value customer through revenue-based CLV models, may not be profitable at all. By connecting marketing, merchandising, and operations data together, brands can understand their most profitable programs, channels, and products through the lens of actual customer value – and optimize accordingly.

The journey for brands starts with the awareness and data enablement of a more complex data set containing all variable revenue, cost, and margin inputs. What does a retailer need to do to achieve this?

  1. All data together in one place
  2. Matched and deduplicated disparate data
  3. Data is organized into entities and concepts that business decision makers can understand
  4. A common definition of key business metrics (this is especially important yet challenging for retailers because systems are siloed by the department and common KPIs like contribution profit per order may be defined differently across a company)
  5. Branched outputs for actionability: BI dashboards vs. data activation to improve marginal profit.

Once brands understand these requirements, up next is execution. This responsibility may fall within a ‘build’ strategy on the shoulders of technical IT/data leadership and their team(s) within a brand. This offers maximum control but at maximum cost and time-to-value. Retail data sources are complicated and change often. Technical teams within brands can spend too much time building and maintaining the tactical data ingestion process, which means they are spending less time deriving business value from the data.

But it doesn’t have to be this hard. There are other options in the market that brands can consider, such as a tool like SoundCommerce which provides a library of data connectors, pre-built and customizable date mappings, and outbound data orchestrations all tailor-made and ready-to-go for retail brands.

SoundCommerce empowers retail technology leaders to:

  • Maintain data ownership to allow users to send modeled data to external data warehouses or layer-on business analytics tools for greater flexibility
  • Provide universal access to data across the organization so every employee can have access to and participate in a low-code or no-code experience
  • Expand and democratize data exploration and activation among both technical and non-technical users

For retail business and marketing decision-makers, SoundCommerce makes it easy to:

  • Calculate individual customer lifetime value through the lens of profitability – not just revenue.
  • Evaluate and predict lifetime shopper performance – identify which programs drive the highest CLV impact
  • Set CAC and retention cost thresholds – determine optimal Customer Acquisition Costs (CAC) and retention costs that ensure marketing efforts are profitable through the lens of total lifetime transactions

Below is a sample data flow that illustrates how SoundCommerce connects to all the tools a Retailer is using and ingests the first-party data, agnostic of the platform. SoundCommerce then models and transforms the data within the retail context for brands to take immediate action on the insights they gain from the modeled data.

SoundCommerce built on Google Cloud Platform Services

SoundCommerce selected the Google Cloud Platform and its services to achieve what they set out to do – drive profitability for retailers and brands. SoundCommerce perfected this very need of retailers to centralize and harmonize data, map to business users’ needs to infer key metrics and insights by visualizing the data, or reuse the produced datasets to build upon other use cases specific to retailers. SoundCommerce built a cloud-native solution on Google Cloud leveraging the data cloud platform. Data from various sources are ingested in raw format, parsed, and processed as messages in Cloud Storage buckets using Google Kubernetes Engine (GKE) and stored as individual events in Cloud BigTable. A mapping engine maps the data to proprietary data models stored in BigQuery to store the produced data as datasets. Customers use visualization dashboards in Looker to access the data exposed as materialized views from within BigQuery. In many cases, these views are directly accessible by the customer per their use case.

Power Retail Profitable Growth with Analytics Hub

SoundCommerce adopted BigQuery and the recent release of Analytics Hub – a data exchange that enables BigQuery users to efficiently and securely share data. This feature ensures a more scalable direct access experience for SoundCommerce’s current and future customers. It meets brands where they are in their data maturity by giving them the keys to own their data and control their analytics-driven business outcomes. With this feature, retailers can customize their analysis with additional data they own and manage.

“Retail Brands need flexible data models to make key business decisions in real-time to optimize contribution profit and shopper lifetime value,” said SoundCommerce CEO Eric Best. “GCP and BigQuery with Analytics Hub make it easy for SoundCommerce to land and maintain complex data sets, so brands and retailers can drive profitable shopper experiences with every decision.”

SoundCommerce uses Analytics Hub to increase the pace of innovation by sharing datasets with its customers in real time by using the streaming functionality of BigQuery. Customers subscribe to specific datasets through a data exchange as data is generated from external data sources and published into BigQuery. This leads to a natural flow of data that scales easily to hundreds of exchanges and thousands of listings. From the Customer’s viewpoint, Analytics Hub enables them to search listings and coalesce data from other software vendors to produce richer insights. All of the benefits are an add-on to the BigQuery features such as separation of compute and storage, petabyte-scale serverless data warehouse, and tighter integration with several Google Cloud products.

The below diagram shows a view of SoundCommerce sharing datasets with one of its customers:

  1. A SoundCommerce GCP project that hosts the BigQuery instance contains one or more Source Datasets that are composed into a Shared Dataset for a specific customer. The dataset is wrapped around Materialized views but can include other BigQuery objects such as Tables, Views, Authorized views and datasets, BigQuery ML models, external Tables, etc.
  2. The same SoundCommerce GCP project contains the data exchange that acts as a container regarding the shared datasets. The exchange is made private to securely share the curated dataset relevant to the customer. The shared dataset is published into the data exchange as a private listing. The listing inherits the security permissions that are configured on the exchange.
  3. SoundCommerce shares a direct link to the Exchange to the Customer, which they can add as a Linked dataset into their project in their Google Cloud Organization. The shareable link can be pointed to a private listing. From here on, the dataset is visible in the Customer project like any other dataset and immediately available to accept queries and return results. Alternatively, the customer can also view the listing in their own project under Analytics Hub and subscribe to it by adding it as a linked dataset.

SoundCommerce is incrementally onboarding customers to use Analytics Hub for all data sharing use cases. This enables brands to get business insights faster and gain an understanding of their profitable growth quickly. Plus, it gives them the ownership to own their own data and manage it how they see fit for their business. From a technical standpoint, the adoption of Analytics Hub has led to leveraging an inherent capability in BigQuery for data sharing, faster scaling, and reducing operational overhead to onboard customers.

The Built with BigQuery advantage for ISVs

Through Built with BigQuery launched in April as part of the Google Data Cloud Summit, Google is helping tech companies like SoundCommerce build innovative applications on Google’s data cloud with simplified access to technology, helpful and dedicated engineering support, and joint go-to-market programs. Participating companies can:

  • Get started fast with a Google-funded, pre-configured sandbox.
  • Accelerate product design and architecture through access to designated experts from the ISV Center of Excellence who can provide insight into key use cases, architectural patterns, and best practices.
  • Amplify success with joint marketing programs to drive awareness, generate demand, and increase adoption.

BigQuery gives ISVs the advantage of a powerful, highly scalable data warehouse that’s integrated with Google Cloud’s open, secure, sustainable platform. And with a huge partner ecosystem and support for multi-cloud, open source tools, and APIs, Google provides technology companies the portability and extensibility they need to avoid data lock-in.

Click here to learn more about Built with BigQuery.

We thank the many Google Cloud team members including Yemi Falokun in Partner Engineering who contributed to close collaboration with the SoundCommerce team.

5920

Of your peers have already watched this video.

33:30 Minutes

The most insightful time you'll spend today!

Case Study

How Spotify Serves Personalized Music Recommendations

Music for everyone. That’s Spotify’s promise. And it lives up to it by serving over 140 million music lovers across the world.  Every day it renders personalized music recommendations to hundreds of millions of happy customers.

“Spotify is a global business with a global user base. Being able to provide a great audio experience to our users is a key priority for us,” said Niklas Gustavsson, chief architect at Spotify. 

To be able to do that, Spotify needed a system that would scale and enable personalization and provide a seamless customer experience. That’s why it moved its infrastructure to Google Cloud and Cloud Bigtable. This allows Spotify to deliver recommendations at scale, roll out experiments quickly, and ingest terabytes every day via Cloud Dataflow. 

“With Cloud Bigtable clusters in Asia, Europe, and the United States, we’re able to get low-latency data access all over the world, enabling Spotify to provide a seamless experience for our users. We’re also pleased with the continuous collaboration and deep engagement with Google’s product teams to accelerate both our businesses, ” says Gustavsson.

In this video, Peter Sobot, Senior Engineer, Personalization, Spotify talks about how Cloud Bigtable is helping Cloud Bigtable personalize recommendations, including tips about how to design a good schema, how to avoid latency when ingesting new data, and effective caching strategies to scale to tens of millions of data points per second.

Blog

A Recap on Google Cloud Databases and Storage Options- Part 2

3950

Of your peers have already read this article.

6:00 Minutes

The most insightful time you'll spend today!

Refresh your learning on Google Cloud databases and storage options. Make your way to the end of the blogpost as a fun challenge if you are a database practitioner, data enthusiast, cloud geek or just anyone interested in Google's data solutions!

Isaac Newton said, “If I have seen farther than others it is because I have stood on the shoulders of giants.” He meant that in order to explain the Law of Gravity, he used the work of major thinkers who came before him in order to make intellectual progress (as-in giving credit). In the 1640’s (yes, the time of the English Civil War and the start of the mini Ice Age), we can say the Age of Reason began and the word “data” was also (re)born in a way.

Why? Hard to assert a specific reason, it could be apropos of all those events of that decade, and somewhere amidst all that, the scientists and Churchmen (rather Churchman, Henry Hammond who really coined the term data) started to pen down their credit in books and there was a continued proliferation of data to reason their finding (and most times to reason why their work was better than that of the others). And thus comes to us the word datum from the Latin verb dare (it means “to give”, not the English dare).

Dare to Recap History?


Data is history captured through language (it has become the future as well, but that is for another day). Now we all like history (well, most of us). But it is highly likely the context gets lost in the complexity and style of definition. One way to mitigate that risk is to have a clear set of definitions (language), sustained hold of events (history), a clean process of capture (extract) and a scalable process for translation and aggregation (transform). If we want our data to be successful and rise to the occasion, then we need to keep these ways to mitigate the risk of complexity in mind.

And this is exactly what we discussed in the Part 1 of this blog series, Data Modeling Basics—the various business attributes, technical aspects, design questions, and considerations for designing your database model.

In this blog…


We will look into the different databases and storage options in Google Cloud, a brief note on each one of them, when to choose one over the other, interesting alternatives, exceptions and if you make it to the end of the blog, a fun challenge to make sure we put this little tech nugget to an ACID test (see what I did there?). If you are a cloud enthusiast, a database practitioner, a data geek, or a general wonderer of life with computing, you may find this engaging…

Google Cloud Storage Options


We at Google Cloud, have realized how hard it is to go through these laundry list assessment aspects and have made it simpler for you with a Decision Tree. (Of course, It ain’t Christmas if not for the tree):

If only the world was always “Structured”


In a structured world, you will know all the attributes on a first-name basis (I mean to say that you will have a well defined fixed set of attributes that can be modeled in a table of rows and columns), and the applications are transactional or analytical in orientation. Transactional Structured Data operate one row at a time generally and they need to adhere to ACID compliance. (Ah. Now you connect the dots, if not already.) ACID properties are Atomicity, Consistency, Isolation, and Durability. Cloud SQL and Cloud Spanner are our Google Cloud choices for Transactional Structured Data use cases.

Let’s look at the below aspects for each type and structure of data:

  • Why that option? (highlights and key features)
  • When to choose?
  • When not to choose?
  • Security aspects

Cloud SQL

  • Fully Managed, cloud-native RDBMS (Relational DataBase Management System) that offers both MySQL, PostgreSQL, SQL Server engines
  • Cloud SQL is accessible from apps running on App Engine, GKE, or Compute Engine

Note: A managed database is one that does not require as much administration and operational support (creating databases, performing backups, updating the operating system of database instances) as an unmanaged database.

When to use Cloud SQL?

  • Typical online transaction processing (OLTP) workloads
  • Lift and shift of on-premise SQL databases (or from anywhere else) to cloud
  • Regional applications that do not need to store > 30 TB of data in a single instance

When not to use Cloud SQL?


Cloud SQL is not an appropriate storage system for online analytical processing (OLAP) workloads or data that requires dynamic schemas on a per-object basis.

Security


Data stored is encrypted both in transit and at rest. Have built-in support for access control, using network firewalls to manage database access.

Cloud Spanner


  • Relational, horizontally scalable, global database with strong consistency
  • Supports schemas, ACID transactions, and SQL queries (ANSI 2011)
  • Scales horizontally in regions, but can also scale across regions for workloads that have more stringent availability requirements

When to use Cloud Spanner?

  • For large amounts of data and when you require high transactional consistency
  • When you require sharding for higher throughput, access and low latency

When not to use Cloud Spanner?


Cloud Spanner is not an appropriate storage system for online analytical processing (OLAP) workloads

Security


Security features in Spanner include data-layer encryption, audit logging, and Identity and Access Management (IAM) integration.

Analytical Structure is when we want the data to tell us an aggregated or enhanced story, for which we use limited columns and multiple rows and hence mostly use a Column-Oriented storage mechanism. Column-oriented storage is if we want to store the data in the tables by columns instead of by rows, and this column-oriented storage is done to efficiently access only a subset of columns for querying. BigQuery is the data warehouse option for analytics needs.

BigQuery


  • BigQuery is a fully managed Data Warehouse for analytics with built-in data transfer service
  • Peta-byte scale, low-cost warehouse that supports loading data through the web interface, command line tools, and REST API calls
  • Incorporates features for machine learning, business intelligence, and geospatial analysis that are provided through BigQuery ML, BI Engine, and GIS.

Note: A data warehouse stores large quantities of data for query and analysis instead of transactional processing.

When to use BigQuery?


For use cases that cover process analytics and optimization, big data (Petabyte scale) processing and analytics, data warehouse modernization, machine learning-based behavioral analytics, and predictions

When not to use BigQuery?


BigQuery is not a Transactional database and is oriented on running analytical queries, not for simple CRUD operations and queries.

Security


BigQuery provides encryption at rest and in transit. Cloud Data Loss Prevention (Cloud DLP) can be used to scan the BigQuery tables and to protect sensitive data and meet compliance requirements. BigQuery supports access control of datasets and tables using Identity and Access Management (IAM).

And then we have the Semi-structured and the Unstructured world of data that we will address in the below sections.

Cloud Firestore (Cloud Datastore)


Firestore is the next major version of Datastore and a re-branding of the product. Taking the best of Datastore and the Firebase Realtime Database, Firestore is a NoSQL document database built for automatic scaling, high performance, and ease of application development.

  • A fully managed, serverless NoSQL Google Cloud database designed for the development of serverless apps that stores JSON data
  • Can be used to store, sync, and query data for web, mobile, and IoT applications
  • Automatically handles sharding and replication making it highly available, durable, and scalable
  • Provides ACID transactions, SQL-like queries, indexes, and more
  • If a client does not have network connectivity, the Firestore API lets your app persist data to a local disk and synchronizes itself with the current server state once connectivity is reestablished

When to use?


For use cases of app development, live synchronization, offline support, multi-user collaborative applications, leader board, etc.

When not to use?


Not a relational database so not meant for relational structured data use cases.

Security


Firestore Security Rules support serverless authentication and authorization for the mobile and web client libraries. Identity and Access Management (IAM) manages database access.

Cloud Bigtable

  • Bigtable is a wide-column, fully managed, high-performance NoSQL database service designed for terabyte- to petabyte-scale workloads
  • Bigtable is battle tested on Google internal Bigtable database infrastructure that powers Google Search, Google Analytics, Google Maps, and Gmail
  • Provides consistent, low-latency, and high-throughput storage for large-scale NoSQL data

When to use?

  • For large amounts of single key data and is preferable for low-latency, high throughput workloads
  • For real-time app serving workloads and large-scale analytical workloads

When not to use?


While Bigtable is considered an OLTP system, it doesn’t support multi-row transactions, SQL queries or joins. For those use cases, consider either Cloud SQL or Datastore.

Security

  • All the data at rest in Cloud Bigtable is encrypted using Google’s default encryption, by default.
  • Instead of Google managing the encryption keys that protect your data, your Bigtable instance can also be protected using a key that you manage (customer-managed encryption keys (CMEK)) in Cloud Key Management Service (Cloud KMS).

Cloud Storage

  • Google Cloud Storage is an object storage system that is durable and highly available, persists unstructured data like images, videos, data files, videos, backup, and other data
  • It is unstructured and so the files in the cloud storage are atomic that you read the entire file but you cannot access specific blocks in the files
  • Cloud Storage is available in multiple classes, depending on the availability and performance required for apps and services
  • Standard – Offers the highest levels of availability and is appropriate for storing data that requires low-latency access
    Nearline – Low-cost, highly durable, fast-access storage service for storing data that you access less than once per month
    Coldline – Very-low-cost, highly durable, fast-access storage service for storing data that you intend to access less than once per quarter
    Archive – Lowest-cost, highly durable, fast-access storage service for storing data that you intend to access less than once per year

Security


Files in Cloud Storage are organized by project into individual buckets. These buckets can support either custom access control lists (ACLs) or centralized identity and access management (IAM) controls.

Firebase Realtime Database

  • Firebase is a realtime, NoSQL, Google Cloud database that is a part of the Firebase platform that allows you to store and sync data in real-time and includes caching capabilities for offline use
  • Data is stored as JSON and synchronized in real-time to every connected client and remains available when app goes offline

When to use?


For mobile and web app development, development of apps that work across devices

When not to use?


Not in relational dataset use cases. The Realtime Database is a NoSQL database and as such has different optimizations and functionality compared to a relational database. The Realtime Database API is designed to only allow operations that can be executed quickly.

Security


The Realtime Database provides a flexible, expression-based rules language, called Firebase Realtime Database Security Rules, to define how your data should be structured and when data can be read from or written to. When integrated with Firebase Authentication, developers can define who has access to what data, and how they can access it.

That’s a rather packed read. But I hope you find this useful to understand comprehensively the basics of data, storage options and databases in Google Cloud Platform.

Next Steps, before I go…


In the blog part 1 of the series, I ended with an action item – “How would you model a NoSQL solution for an application that needs to query the lineage between individual entities that are represented in pairs?”.

Well, my answer is Firestore. As part of this episode, why don’t you take some time to go over the options and key aspects that attribute to this.

Trend Analysis

Four Key Takeaways from Google and Quantum Metric’s Back-to-School Retail Benchmark Study

4775

Of your peers have already read this article.

2:00 Minutes

The most insightful time you'll spend today!

Google research and Quantum Metric's Back-to-School study on the U.S. retail data helps identify pain points and improve search and personalization for shoppers as they prep up for post-holiday season. Read this blogpost for more insights!

Is it September yet? Hardly! School is barely out for the summer. But according to Google and Quantum Metric research, the back-to-school and off-to-college shopping season – which in the U.S. is second only to the holidays in terms of purchasing volume1 – has already begun. For retailers, that means planning for this peak season has kicked off as well.

We’d like to share four key trends that emerged from Google research and Quantum Metric’s Back-to-School Retail Benchmarks study of U.S. retail data, explore the reasons behind them, and outline the key takeaways.

  1. Out-of-stock and inflation concerns are changing the way consumers shop. Back-to-school shoppers are starting earlier every year, with 41% beginning even before school is out – even more so when buying for college1. Why? The behavior is driven in large part by consumers’ concerns that they won’t be able to get what they need if they wait too long. 29% of shoppers start looking a full month before they need something1.

Back-to-school purchasing volume is quite high, with the majority spending up to $500 and 21% spending more than $1,0001. In fact, looking at year-over-year data, we see that average cart values have not only doubled since November 2021, but increased since the holidays1. And keep in mind that back-to-school spending is a key indicator leading into the holiday season.

That said, as people are reacting to inflation, they are comparing prices, hunting for bargains, and generally taking more time to plan. This is borne out by the fact that 76% of online shoppers are adding items to their carts and waiting to see if they go on sale before making the purchase1. And, to help stay on budget and reduce shipping costs, 74% plan to make multiple purchases in one checkout1. That carries over to in-store shopping, when consumers are buying more in one visit to reduce trips and save on gas.

2. The omnichannel theme continues. Consumers continue to use multiple channels in their shopping experience. As the pandemic has abated, some 82% expect that their back-to-school buying will be in-store, and 60% plan to purchase online. But in any case, 45% of consumers report that they will use both channels; more than 50% research online first before ever setting foot in a store2. Some use as many as five channels, including video and social media, and these 54% of consumers spend 1.5 times more compared to those who use only two channels4.

And mobile is a big part of the journey. Shoppers are using their phones to make purchases, especially for deadline-driven, last-minute needs, and often check prices on other retailers’ websites while shopping in-store. Anecdotally, mobile is a big part of how we ourselves shop with our children, who like to swipe on the phone through different options for colors and styles. We use our desktops when shopping on our own, especially for items that require research and represent a larger investment – and our study shows that’s quite common.

3. Consumers are making frequent use of wish lists. One trend we have observed is a higher abandonment rate, especially for apparel and general home and school supplies, compared to bigger-ticket items that require more research. But that can be attributed in part to the increasing use of wish lists. Online shoppers are picking a few things that look appealing or items on sale, saving them in wish lists, and then choosing just a few to purchase. Our research shows that 39% of consumers build one or two wish lists per month, while 28% said they build one or two each week, often using their lists to help with budgeting1.

4. Frustration rates have dropped significantly. Abandonment rates aside, shopper annoyance rates are down by 41%, year over year1. This is despite out-of-stock concerns and higher prices. But one key finding showed that both cart abandonment and “rage clicks” are more frequent on desktops, possibly because people investing time on search also have more time to complain to customer service.

And frustration does still exist. Some $300 billion is lost each year in the U.S. from bad search experiences5. Data collected internationally shows that 80% of consumers view a brand differently after experiencing search difficulties, and 97% favor websites where they can quickly find what they are looking for5.

Lessons to Learn


What are the key takeaways for retailers? In general, consider the sources of customer pain points and find ways to erase friction. Improve search and personalization. And focus on improving the customer experience and building loyalty. Specifically:

80% of shoppers want personalization6. Think about how you can drive personalized promotions or experiences that will drive higher engagement with your brand.

46% of consumers want more time to research1. Drive toward providing more robust research and product information points, like comparison charts, images, and specific product details.

43% of consumers want a discount1, but given current economic trends, retailers may not be offering discounts. In order to appease budget-conscious shoppers, retailers can consider other retention strategies such as driving loyalty using points, rewards, or faster-shipping perks.

Be sure to keep returns as simple as possible so consumers feel confident when making a purchase, and reduce possible friction points if a consumer decides to make a return. 43% of shoppers return at least a quarter of the products they buy and do not want to pay for shipping or jump through hoops1.

How We Can Help


Google-sponsored research shows that price, deals, and promotions are important to 68% of back-to-school shoppers.7 In addition, shoppers want certainty that they will get what they want. Google Cloud can make it easier for retailers to enable customers to find the right products with discovery solutions. These solutions provide Google-quality search and recommendations on a retailer’s own digital properties, helping to increase conversions and reduce search abandonment. In addition, Quantum Metric solutions, available on the Google Cloud Marketplace, are built with BigQuery, which helps retailers consolidate and unlock the power of their raw data to identify areas of friction and deliver improved digital shopping experiences.

We invite you to watch the Total Retail webinar “4 ways retailers can get ready for back-to-school, off-to college” on demand and to view the full Back-to-School Retail Benchmarks report from Quantum Metric.

Sources:
1. Back-to-School Retail Benchmarks report from Quantum Metric
2. Google/Ipsos,Moments 2021, Jun 2021, Online survey, US, n=335 Back to School shoppers
3. Google/Ipsos, Moments 2021, Jun 2021, Online survey, US, n=2,006 American general population 18+
4. Google/Ipsos, Holiday Shopping Study, Oct 2021 – Jan 2022, Online survey, US, n=7,253, Americans 18+ who conducted holiday shopping activities in past two days
5. Google Cloud Blog, Nov 2021, “Research: Search abandonment has a lasting impact on brand loyalty”
6. McKinsey & Company, “Personalizing the customer experience: Driving differentiation in retail”
7. Think with Google, July 2021, “What to expect from shoppers this back-to-school season”

Blog

Google Cloud and Climate Engine Collaborate to Support Climate Action in Public Sector

4813

Of your peers have already read this article.

1:30 Minutes

The most insightful time you'll spend today!

Google Cloud and Climate Engine partner to build climate resilience with Google Earth Engine's world-class geospatial capacities, combining AI and ML to provide a centralized system to gather, process and analyse earth based data.

While there is uncertainty about how much the climate will change in the future, we know it won’t look like the past. Extreme weather events will increase in frequency and severity; the world will continue to warm, and the cost of climate change will increase.

Government plays a vital role in understanding and responding to these changes quickly. Achieving this improved response time will require data insights to ensure informed decision-making—from local to global scales. The challenge is not only urgent; it’s one of the world’s biggest “big data” problems.

Fortunately, new technologies to help us monitor the Earth are proliferating. Thousands of satellites take millions of images of the planet every day. Sensors generate data about temperature, precipitation, wind, soil conditions, and more—as frequently as every second. We have more information about the planet’s systems than at any other time in history. And the data will only continue to grow. The problem is not the lack of data–it is harnessing this data to drive insights for decision makers to tackle climate change. That’s why Google Cloud has partnered with Climate Engine.

How Climate Engine and Google Cloud enable greater climate resilience

Climate Engine is a scientist-led company that works with Google to accelerate and scale the use of Google Earth Engine’s world-class geospatial capacities (in addition to those of Google Cloud Storage and BigQuery, among other tools) in support of climate action in the public sector. Powered by Google Cloud’s infrastructure, Google Earth Engine (GEE) combines a multi-petabyte catalog of satellite imagery and geospatial datasets with planetary-scale analysis capabilities, enabling scientists, researchers, and developers to detect changes, map trends, and quantify differences on the Earth’s surface. 

With cloud-based technologies, we can leverage massive computing at a scale that generates actionable insights from Earth-based data. These insights help us better manage resources, understand risks, predict changes, and respond to disasters as we meet the challenge of climate change. Geospatial AI combines the power of artificial intelligence (AI) and machine learning (ML) with geospatial analysis. Google Cloud’s Geospatial AI solutions provide departments and agencies with a centralized system to collect, process, and deliver Earth-based data into decision-making contexts.

Climate Engine and Google Cloud provide specialists with the opportunity to go back in time and see how our landscapes have changed due to changes in climate and other human activities over the past few decades. Years of data can now be quantitatively analyzed and visualized in a matter of a few seconds, enabling government agencies to fulfill their mandates by drawing invaluable insights into how landscapes are changing, what physical and natural assets are at risk, and where the opportunities are for reducing emissions and increasing carbon sequestration. 

“This is game changing for natural resource managers and scientists at public institutions at all levels of government,” says Dr. Daniel McEvoy, regional climatologist, at the Desert Research Institute & Western Regional Climate Center, Nevada System of Higher Education.https://www.youtube.com/embed/aPGsi8bd_Zk?enablejsapi=1&

Use cases for geospatial climate information systems

The use cases for this technology are as varied as the climate challenges themselves. These include monitoring, predicting, and analyzing the risks of extreme weather events like floods, wildfire, drought, extreme heat, wind, and other climate hazards. Use cases also include tracking changes in ecosystems, disease vectors, water availability and quality, soil health, growing seasons, air pollution, and more. These use cases are some of the ways that Google Cloud and Climate Engine can help the public sector deliver on government mandates. These provide insights that are helpful for a wide range of departments, and that can be applied in spatial and temporal scales that are meaningful for governments to take action.

“Our planet is changing at a rate that we have never experienced,” says Forrest Melton of the NASA Western Water Applications Office. To respond to these changes, we must understand what is happening across a wide range of environmental variables and at geospatial scales that range from local to global. We now have access to more data about the planet than ever before. The big challenge is converting data into actionable insights and then rapidly integrating these insights into decision-making systems. Climate Engine and Google Cloud help resolve this problem through innovative analytical tools and effective use of cloud computing.” 

Climate change carries an existential risk to our current and future stability and security. Together, we are working to provide transformational technologies that help meet that risk and build a safer, more resilient future for all of us. 

Learn more about Google Cloud’s environmental initiatives here and here.

More Relevant Stories for Your Company

Blog

Discover Latest Resources on Google Cloud’s Datasets Solution

Editor’s note:  With Google Cloud’s datasets solution, you can access an ever-expanding resource of the newest datasets to support and empower your analyses and ML models, as well as frequently updated best practices on how to get the most out of any of our datasets. We will be regularly updating this

How-to

How to Enhance Incremental Pipeline Performance while Ingesting Data into BigQuery

When you build a data warehouse, the important question is how to ingest data from the source system to the data warehouse. If the table is small you can fully reload a table on a regular basis, however, if the table is large a common technique is to perform incremental

How-to

How to Decide Whether to Run a Database on Kubernetes

Today, more and more applications are being deployed in containers on Kubernetes—so much so that we’ve heard Kubernetes called the Linux of the cloud. Despite all that growth on the application layer, the data layer hasn’t gotten as much traction with containerization. That’s not surprising, since containerized workloads inherently have

Case Study

What Swiggy and You Can Learn From This Company’s Use of ML to Engage Customers

The app economy has enabled a huge range of unique business models to flourish. One such model is online food ordering and delivery services, in which apps leverage geo-location data to aggregate local food choices and offer personalized options to consumers. A leading company in this space is Just Eat.

SHOW MORE STORIES