In recent years, the grocery industry has had to shift to facilitate a wider variety of checkout journeys for customers. This has meant ensuring a richer transaction mix, including mobile shopping, online shopping, in-store checkout, cashierless checkout or any combination thereof like buy online, pickup in store (BOPIS).  

What’s more, in the past year and a half alone grocers have had to enable consumers new ways to shop for essentials. This has included needing to rapidly integrate or build on-demand delivery apps, offer curbside pickup with near-instant fulfillment as well as support touchless and cashless checkout experiences. Searches on Google Maps for retailers in the US with curbside pickup options have increased by 9000% since March 2020, and we believe these trends from 2020 will continue to define the future of grocery shopping.

The future of grocery will require agility and openness

Firstly, the need to rapidly adapt to changing consumer habits will be the new normal. Grocers will increasingly look to digitally transform legacy retail systems and modernize point of sale (POS) platforms to deliver and scale omnichannel experiences as quickly as possible. This necessitates a more agile and open architectural approach to technology – one built on microservices and leverages APIs so that new applications and experiences can be built, integrated and delivered faster.

Automation and data-driven retailing will be table stakes

In order for retailers to blend what they’re offering in the store with digital experiences more efficiently, they will also need to automate more. For example, with automation and business intelligence, grocers can take labor that might have been tied up with tender operations and checkout and redistribute those resources to restocking shelves, curbside pick-up or improving customer experiences. 

Automation and access to real-time in-store inventory & supply chain data can also help grocers avoid the supply chain challenges seen in the early days of COVID-19. Grocers will need to find ways to leverage automation to ingest, organize, and analyze data from physical store networks, digital channels, distribution centers to better forecast demand and manage future fluctuations.

How NCR and Google Cloud are helping grocers adapt to disruption with operational agility

Helping grocers improve operational agility to address changing consumer shopping habits and to thrive during times of disruption is something that NCR and Google Cloud have teamed up to do. NCR has over 135 years of experience in retail, having invented the cash register and are continuing to help grocers innovate. NCR Emerald builds upon the company’s leadership in POS software and has turned it into a unified platform that helps grocers operate the entire store from front to back. The solution supports cashier-led checkout, self-checkout, integrated payments, merchandising, and enables regional managers and corporate employees access to the analytics and tools needed to optimize loyalty programs and promotions.

NCR has invested in a comprehensive, agile, and API-led retail architecture that lets grocers continually innovate and design new experiences as customers and the industry evolve. By running Emerald on Google Cloud, NCR can offer the solution on a subscription basis, helping grocers lower upfront capital expenditures and ensuring scalability. What’s more, NCR can tap into Google Cloud’s strength in data, analytics, and openness to deliver three key imperatives. Let’s take a look at each of these below.

Run the way grocers need to while leveraging Google Cloud as a single source of logic

Traditionally the POS system lived in the store. If disaster strikes, people still need access to food and essentials so the grocery store still needs to operate. It hardly gets more mission-critical than that. NCR Emerald is built on microservices, leveraging Kubernetes for front-of-house compute, and VMs (See graphic 1 below). This makes it easy to support lightweight clients accessible by store employees via any range of mobile devices, computer terminals, self-service kiosks, peripheral devices like receipt printers as well as legacy applications.

What’s unique is that because Emerald runs on Google Cloud, it supports all those in-store and digital touchpoints mentioned above, but also allows grocers to run lean. Emerald leverages Google Cloud as a single source of truth and operates a lot of what it does out of logic. Every sales transaction coming from every channel, including e-commerce, can be logged via NCR’s Hosted Service and centralized in BigQuery and Bigtable as a transaction data master. This enables the grocer to manage any transactional use case very consistently, whether it e supporting customers who want to purchase in one store and return in another, offering digital receipts or the ability to exchange online purchases in store. Emerald on Google Cloud can help retailers extend capabilities through the power of the cloud but not need to live exclusively in the cloud. In other words, the solution allows grocers the ability to run the way they need to.

Enable data-driven and real-time decision making for grocers

Store managers, regional managers, category managers, and others all require different cuts of the data to do their jobs effectively. However, data silos persist and how data is formatted and arranged can still remain pretty static. Therefore allowing users with different roles the ability to view and analyze that data quickly and in different ways continues to be a challenge. 

As mentioned above, Emerald leverages Google Cloud data management solutions as the central repository for transactional, behavioral, and merchandising data. Every transaction from every store and every channel can be stored via NCR Hosted Service on BigQuery and Bigtable. NCR Analytics then harnesses the advanced analytical and data visualization capabilities of Looker to help grocers get a consolidated view of their business across all channels and then allow employees to slice and dice the data they way they need to. NCR Analytics also leverages the power of Google Cloud AI and machine learning to add another level of intelligence to the retailer’s data. For example, store managers can visualize how well they’re using their real estate and see how productive lanes 1-3 are compared with 7-10 or compare self-service versus manned lanes. By mapping to the retailer’s own catalog, they can also break down category-level performance and trends.

NCR Analytics takes advantage of Google Cloud’s data pipeline to reduce processing time, with scaling and resource management provided out of the box. By letting the cloud store and process the data, NCR is providing the ability for retailers to analyze their data in near real-time across all platforms – a real game changer in the grocery business.

Open APIs let grocers continually enrich the retail experience

Finally, Emerald is built on an API-first architecture managed through Apigee. It uses the power of Apigee as an open API platform to expose how Emerald can work with other NCR applications like loyalty and promotions, and third party applications like mobile ordering and order delivery to enrich the grocery experience for employees and customers. Every API that Emerald uses is available on Apigee, allowing them to share code samples and giving developers the ability to run scripts. This approach can allow retailers the ability to innovate in a fraction of the time and cost, speeding up 3rd party integrations up front and as businesses grow. 

Take, for example, Northgate Market, a chain of 40 stores in California, that were able to transform its digital operations and enable experiences that set it apart from competitors – quickly and simply with Emerald. It took less than 6 months to go from contract to live deployment in the first store. Since then, Northgate Market has been able to extend their intelligence by leveraging the power of Looker and NCR Analytics.

Learn more about how NCR has been able to leverage an open, cloud-enabled architecture to help customers innovate across the retail, hospitality, and banking industries on the webinar “Role of APIs in Digital Transformation”. You can also learn more about how Northgate uses e-commerce to transform customer experience and gain consumer insights.

Cloud Bigtable is a popular and widely used key-value database available on Google Cloud. The service provides scale elasticity, cost efficiency, excellent performance characteristics, and 99.999% availability SLA. This has led to massive adoption with thousands of customers trusting Bigtable to run a variety of their mission-critical workloads.

Bigtable has been in continuous production usage at Google for more than 15 years now. It processes more than 5 billion requests per second at peak and has more than 10 exabytes of data under management. It’s one of the largest semi-structured data storage services at Google.

One of the key use cases for Bigtable at Google is ad personalization. This post describes the central role that Bigtable plays within ad personalization.

Ad personalization

Ad personalization aims to improve user experience by presenting topical and relevant ad content. For example, I often watch bread-making videos on YouTube. If ads personalization is enabled in my ad settings, my viewing history could indicate to YouTube that I’m interested in baking as a topic and would potentially be interested in ad content related to baking products.

Ad personalization requires large-scale data processing in near real-time for timely personalization with strict controls for user data handling and retention. System availability needs to be high, and serving latencies need to be low due to the narrow window within which decisions need to be made on what ad content to retrieve and serve. Sub-optimal serving decisions (e.g. falling back to generic ad content) could potentially impact user experience. Ad economics requires infrastructure costs to be kept as low as possible.

Google’s ad personalization platform provides frameworks to develop and deploy machine learning models for relevance and ranking of ad content. The platform supports both real-time and batch personalization. The platform is built using Bigtable, allowing Google products to access data sources for ads personalization in a secure manner that is both privacy and policy compliant, all while honoring users’ decisions about what data they want to provide to Google

The output from personalization pipelines, such as advertising profiles are stored back in Bigtable for further consumption. The ad serving stack retrieves these advertising profiles to drive the next set of ad serving decisions.

Some of the storage requirements of the personalization platform include:

• Very high throughput access for batch and near real-time personalization
• Low latency (<20 ms at p99) lookup for reads on the critical path for ad serving
• Fast (i.e. in the order of seconds) incremental update of advertising models in order to reduce personalization delay

Bigtable

Bigtable’s versatility in supporting both low-cost, high-throughput access to data for offline personalization as well as consistent low-latency access for online data serving makes it an excellent fit for the ads workloads.

Personalization at Google-scale requires a very large storage footprint. Bigtable’s scalability, performance consistency and low cost required to meet a given performance curve are key differentiators for these workloads.

Data model

The personalization platform stores objects in Bigtable as serialized protobufs keyed by Object ids. Typical data sizes are less than 1 MB and serving latency is less than 20 ms at p99.

Data is organized as corpora, which correspond to distinct categories of data. A corpus maps to a replicated Bigtable.

Within a corpus, data is organized as DataTypes, logical groupings of data. Features, embeddings, and different flavors of advertising profiles are stored as DataTypes, which map to Bigtable column families. DataTypes are defined in schemas which describe the proto structure of the data and additional metadata indicating ownership and provenance. SubTypes map to Bigtable columns and are free-form.

Each row of data is uniquely identified by a RowID, which is based on the Object ID. The personalization API identifies individual values by RowID (row key), DataType (column family), SubType (column part), and Timestamp.

Consistency

The default consistency mode for operations is eventual. In this mode, data from the Bigtable replica nearest to the user is retrieved, providing the lowest median and tail latency.

Reads and writes to a single Bigtable replica are consistent. If there are multiple replicas of Bigtable in a region, traffic spillover across regions is more likely. To improve the likelihood of read-after-write consistency, the personalization platform uses a notion of row affinity. If there are multiple replicas in a region, one replica is preferentially selected for any given row, based on a hash of the Row ID.

For lookups with stricter consistency requirements, the platform first attempts to read from the nearest replica and requests that Bigtable return the current low watermark (LWM) for each replica. If the nearest replica happens to be the replica where the writes originated, or if the LWMs indicate that replication has caught up to the necessary timestamp, then the service returns a consistent response. If replication has not caught up, then the service issues a second lookup—this one targeted at the Bigtable replica where writes originated. That replica could be distant and the request could be slow. While waiting for a response, the platform may issue failover lookups to other replicas in case replication has caught up at those replicas.

Bigtable replication

The Ads personalization workloads use a Bigtable replication topology with more than 20 replicas, spread across four continents.

Replication helps address the high availability needs for ad serving. Bigtable’s zonal monthly uptime percentage is in excess of 99.9%, and replication coupled with a multi-cluster routing policy allows for availability in excess of 99.999%.

A globe-spanning topology allows for data placement that is close to users, minimizing serving latencies. However, it also comes with challenges such as variability in network link costs and throughputs. Bigtable uses Minimum Spanning Tree-based routing algorithms and bandwidth-conserving proxy replicas to help reduce network costs.

For ads personalization, reducing Bigtable replication delay is key to lowering the personalization delay (the time between a user’ action and when that action has been incorporated into advertising models to show more relevant ads to the user). Faster replication is preferred but we also need to balance serving traffic against replication traffic and make sure low-latency user-data serving is not disrupted due to incoming or outgoing replication traffic flows. Under the hood, Bigtable implements complex flow control and priority boost mechanisms to manage global traffic flows and to balance serving and replication traffic priorities.

Workload Isolation

Ad personalization batch workloads are isolated from serving workloads by pinning a given set of workloads onto certain replicas; some Bigtable replicas exclusively drive personalization pipelines while others drive user-data serving. This model allows for a continuous and near real-time feedback loop between serving systems and offline personalization pipelines, while protecting the two workloads from contending with each other.

For Cloud Bigtable users, AppProfiles and cluster-routing policies provide a way to confine and pin workloads to specific replicas to achieve coarse-grained isolation.

Data residency

By default, data is replicated to every replica—often spread out globally—which is wasteful for data that is only accessed regionally. Regionalization saves on storage and replication costs by confining data to the region where it is most likely to be accessed. Compliance with regulations mandating that data pertaining to certain subjects are physically stored within a given geographical area is also vital.

The location of data can be either implicitly determined by the access location of requests or through location metadata and other product signals. Once the location for a user is determined, it is stored in a location metadata table which points to the Bigtable replicas that read requests should be routed to. Migration of data based on row-placement policies happens in the background, without downtime or serving performance regressions.

Conclusion

In this blog post, we looked at how Bigtable is used within Google to support an important use case—modeling user intent for ad personalization.

Over the past decade, Bigtable has scaled as Google’s personalization needs have scaled by orders of magnitude. For large-scale personalization workloads, Bigtable offers low cost storage with excellent performance characteristics. It seamlessly handles global traffic flows with simple user configurations. Its ease at handling both low-latency serving and high-throughput batch computations make it an excellent option for lambda-style data processing pipelines.

We continue to drive high levels of investment to further lower costs, improve performance, and bring new features to make Bigtable an even better choice for personalization workloads.

Learn more

To get started with Bigtable, try it out with a Qwiklab and learn more about the product here.

Acknowledgements
We’d like to thank Ashish Awasthi, Ashish Chopra, Jay Wylie, Phaneendhar Vemuru, Bora Beran, Elijah Lawal, Sean Rhee and other Googlers for their valuable feedback and suggestions.

How will your organization manage this year’s data growth and business requirements? Your actions and strategies involving data and AI will improve or undermine your organization’s competitiveness in the months and years to come. Our teams at Google Cloud have an eye on the future as we evolve our strategies to protect technology choice, simplify data integration, increase AI adoption, deliver needed information on demand, and meet security requirements. 

Google Cloud worked with* IDC on multiple studies involving global organizations across industries in order to explore how data leaders are successfully addressing key data and AI challenges. We compiled the results in our 2023 Data and AI Trends report. In it, you’ll find the metrics-rich research behind the top five data and AI trends, along with tips and customer examples for incorporating them into your plans. 

1: Show data silos the door

Given the increasing volumes of data we’re all managing, it’s no surprise that siloed transactional databases and warehousing strategies can’t meet modern demands. Organizations want to improve how they store, manage, analyze, and govern all their data, while reducing costs. They also want to eliminate conflicting insights from replicated data and empower everyone with fresh data.

A unified data cloud enables the integration of data and insights into transformative digital experiences and better decision making.

Andi Gutmans, GM and VP of Engineering for Databases, Google Cloud

Tweet this quote

In the report, you can learn how to adopt a unified data cloud that supports every stage of the data lifecycle so that you can improve data usage, accessibility, and governance. Inform your strategy by drawing on organizations’ examples such as a data fabric that improves customer experiences by connecting more than 80 data silos, as well as other unified data clouds that save money and simplify growth. 

2: Usher in the age of the open data ecosystem

Data is the key to unlocking AI, speeding up development cycles, and increasing ROI. To protect against data and technology lock-in, more organizations are adopting open source software and open APIs.

Tweet this quote

Understand how you can simplify data integration, facilitate multicloud analytics, and use the technologies you want with an open data ecosystem, as described in the report. Learn from metrics about global open source adoption and public dataset usage. And explore how global companies adopted open data ecosystems to improve patient outcomes, increase website traffic by 25%, and cut operating costs by 90%.

3: Embrace the AI tipping point

Pulling useful information out of data is easier with AI and ML. Not only can you identify patterns and answer questions faster but the technologies also make it easier to solve problems at scale.

We’ve reached the AI tipping point. Whether people realize it or not, we’re already using applications powered by AI—every day. Social media platforms, voice assistants, and driving services are easy examples.

June Yang, VP, Cloud AI and Industry Solutions, Google Cloud

Tweet this quote

Organizations share how they’re reaching their goals using AI and ML by empowering “citizen data scientists” and having them focus on small wins first. Gain tips from Yang and other experts for developing your AI strategy. And read how organizations achieve outcomes such as a reduction of 7,400 tons per year in carbon emissions and a more than 200% increase in ROI from ad spend by using pattern recognition and other AI capabilities. 

4: Infuse insights everywhere

Yesterday’s BI solutions have led to outdated insights and user fatigue with the status quo, based on generic metrics and old information. Research shows that as new tools come online, expectations for BI are changing, with companies revising their strategies to improve decision making, speed up the development of new revenue streams, and increase customer acquisition and retention by providing individuals with needed information on demand.

Organizations are equipping business decision-makers with the tools they need to incorporate required insights into their everyday workflows.

Kate Wright, Senior Director, Product Management, Google Cloud

Tweet this quote

In the report, you’ll discover why and how data leaders are rethinking their BI analytics strategies and applications to improve users’ trust and use of data in automated workflows, customizable dashboards, and on-demand reports. Global companies also share how they improve decision making with self-service BI, customer experiences with IoT analysis, and threat mitigation with embedded analytics.

5: Get to know your unknown data

Increasing data volumes can make it harder to know where and what data they store, which may create risk. Case in point: If a customer unexpectedly shares personally identifiable information during a recorded customer support call or chat session, that data might require specialized governance, which the standardized storage process may not provide.

If you don’t know what data you have, you cannot know that it’s accurately secured. You also don’t know what security risks you are incurring, or what security measures you need to take.

Anton Chuvakin, Senior Staff Security Consultant, Google Cloud

Tweet this quote

Check out the report to learn about data security risks that are often overlooked and how to develop proactive governance strategies for your sensitive data. You can also read how global organizations have increased customer trust and productivity by improving how they discover, classify, and manage their structured and unstructured data.  

Be ready for what’s next

What’s exciting about these trends is that they’re enabling organizations across industries to realize very different goals using their choice of technologies. And although all the trends depend on each other, research shows you can realize measurable benefits whether you adopt one or all five.

Review the report yourself and learn how you can refine your organization’s data and AI strategies by drawing on the collective insights, experiences, and successes of more than 800 global organizations. 

As Google Cloud’s fully managed relational database that offers near unlimited scale, strong consistency, and availability up to 99.999%, Cloud Spanner powers applications at any scale in industries such as financial services, games, retail, and healthcare. 

When you set up a Spanner instance, you can choose from two different kinds of configurations: regional and multi-regional. Both configuration types offer high availability, near unlimited scale, and strong consistency. Regional configurations offer 99.99% availability and can survive zone outages. Multi-regional configurations offer 99.999% availability and can survive two zone outages and entire regional outages.

Today, we’re announcing a number of significant enhancements to Spanner’s regional and multi-regional capabilities: 

• Configurable read-only replicas let you add read-only replicas to any regional or multi-regional Spanner instance to deliver low latency reads to clients in any geography
• Spanner’s zero-downtime instance move service gives you the freedom to move your production Spanner instances from any configuration to another on the fly, with zero downtime, whether it’s regional, multi-regional, or a custom configuration with configurable read-only replicas 
• We’re also dropping the list prices of our nine-replica global multi-regional configurations nam-eur-asia1 and nam-eur-asia3 to make them even more affordable for global workloads

Let’s take a look at each of these enhancements in a bit more detail. 

Configurable read-only replicas

One of Spanner’s most powerful capabilities is its ability to deliver high performance across vast geographic territories. Spanner achieves this performance with read-only replicas. As its name suggests, a read-only replica contains an entire copy of the database and it can serve stale reads without requiring a round trip back to the leader region. In doing so, read-only replicas deliver low latency stale reads to nearby clients and help increase a node’s overall read scalability.

For example, a global online retailer would likely want to ensure that its customers worldwide can search and view products from its catalog efficiently. This product catalog would be ideally suited for Spanner’s nam-eur-asia1 multi-region configuration, which has read/write replicas in the United States and read-only replicas in Belgium and Taiwan. This would ensure that customers can view the product catalog with low latency around the globe.

Until today, read-only replicas were available in several multi-region configurations: nam6, nam9, nam12, nam-eur-asia1, and nam-eur-asia3. But now, with configurable read-only replicas, you can add read-only replicas to any regional or multi-regional Spanner instance so that you can deliver low-latency stale reads to clients everywhere. 

To add read-only replicas to a configuration, go to the Create Instance page in the Google Cloud console. You’ll now see a “Configure read-only replicas” section. In this section, select the region for the read-only replica, along with the number of replicas you want per node, and create the instance. It’s as simple as that! 

The following snapshot shows how to add a read-only replica in us-west2 (Los Angeles) to the nam3 multi-regional configuration.

As we roll out configurable read-only replicas, we do not yet offer read-only replicas in every configuration/region pair. If you find that your desired read-only replica region is not yet listed, simply fill out this request form.

Configurable read-only replicas are available today for $1/replica/node-hour plus storage costs. Full details on pricing are available at Cloud Spanner pricing. 

Also announcing: Spanner’s zero-downtime instance move service

Now that you can use configurable read-only replicas to create new instance configurations that are tailored to your specific needs, how can you migrate your current Spanner instances to these new configurations without any downtime? Spanner database instances are mission critical and can scale to many petabytes and millions of queries per second. So you can imagine that moving a Spanner instance from one configuration to another — say us-central1 in Iowa to nam3 with a read-only replica in us-west2 — is no small feat. Factor in Spanner’s stringent availability of up to 99.999% while serving traffic at extreme scale, and it might seem impossible to move a Spanner instance from us-central1 to nam3 with zero downtime.

However, that’s exactly what we’re announcing today! With the instance move service, now generally available, you can request a zero-downtime, live migration of your Spanner instances from any configuration to any other configuration — whether they are regional, multi-regional, or custom configurations with configurable read-only replicas. 

To request an instance move, select “contact Google” in the Edit Instance of the Google Cloud Console and fill out the instance move request form. Once you make a move request, we’ll contact you to let you know the start date of your instance configuration move, and then move your configuration with zero downtime and no code changes while preserving the SLA guarantees of your configuration. 

When moving an instance, both the source and destination instance configurations are subject to hourly compute and storage charges, as outlined in Cloud Spanner pricing. Depending on your environment, instance moves can take anywhere from a few hours to a few days to complete. Most importantly, during the instance move, your Spanner instance continues to run without any downtime, and can continue to rely on Spanner’s high availability, near unlimited scale, and strong consistency to serve your mission-critical production workloads. 

Price drops for global 9-replica Spanner multi-regional configurations

Finally, we’re also pleased to announce that we’re making it even more compelling to use Spanner’s global configurations of nam-eur-asia1 and nam-eur-asia3 by dropping the compute list price of these configurations from $9/node/hour to $7/node/hour. With write quorums in North America and read-only replicas in both Europe and Asia, these configurations are perfectly suited for global applications with strict performance requirements and 99.999% availability. And now, they’re even more cost-effective to use!

Learn more 

• If you are new to Spanner, try Spanner at no charge with a 90-day free trial instance.
• Learn more about multi-regional Spanner configurations by reading Demystifying Cloud Spanner multi-region configurations