Editor’s note: We’re hearing here how martech provider Wunderkind easily met the scaling demands of their growing customer base on multiple use cases with Cloud Bigtable and other Google Cloud data solutions.

Wunderkind is a performance marketing channel and we mostly have two kinds of customers: online retailers, and publishers like Gizmodo Media Group, Reader’s Digest, The New York Post and more. We help  retailers boost their e-commerce revenue through real-time messaging solutions designed for email, SMS, onsite, and advertising. Brands want to provide a one-to-one experience to more of their customers, and we use our extensive history with best practices in email marketing and technology to help brands reach more customers through targeted messaging and personalized shopping experiences. With  publishers, it’s a different value proposition, we use the same platform to provide a non disruptive and personalized ad experience for their website. For example, if you are on their site and then you left, we might show an ad tailored to you when you come back later – depending on the campaign. 

After running into limitations with our legacy database system, we turned to Cloud Bigtable and Google Cloud, which helped us be more flexible and easily scale for high traffic demand – which can be a stable 40,000 requests per second, and meet the needs of our growing number of data use cases. 

Three different databases power our core product

In our core offering, companies send us user events from their websites. We store these events and later decide (using our secret sauce) if and how to reach out to those users on behalf of our customers. Because many of our customers are retailers, Black Friday and Cyber Monday are big traffic days for us as. On such days, we can get 31 billion events, sometimes as many as  200K events per second. We show 1.6 billion impressions that have seen close to 1 billion pageviews. And at the end of all this, we securely send about 100 million emails. We noticed the same thing for election time; traffic reached the same high volume. We need scalable solutions to support this level of traffic as well as the elasticity to let us pay only for what we use, and that’s where Google Cloud comes in.

So how does this work? Our externally facing APIs, which are running on Google Kubernetes Engine, receive those user events—up to hundreds of thousand per second. All the components in our architecture need to be able to handle this demand. So from our APIs, those events go to Pub/Sub, Dataflow and from there they are written to Bigtable and BigQuery, Google Cloud’s serverless, and highly scalable data warehouse. This business user activity data underpins almost all our products. Events can be things like product views or additions to shopping carts. When we store this data in Bigtable, we use a combination of email address and the customer ID as the Bigtable key and we record the event details in that record. 

What do we do with this information next? It’s important to mention that we also mark the last time we received an event about a user in Memorystore for Redis, Google Cloud’s fully managed Redis service. This is important because we have another service that is periodically checking Memorystore for users that have not been active for a campaign-specific period of time (it can be 30 minutes, for example), then deciding whether to reach out to them.

How we decide when we reach out is an intelligent part of our product offering, based on the channel, message, product, etc. When we do reach out, we use Memorystore for Redis as a rate limiter or token bucket. In order not to overwhelm the email or texting providers we send API requests to, we throttle those requests using Memorystore. (We prefer to preemptively throttle the outgoing API requests as opposed to handling errors later.)

When we do reach out, often we will need details for a specific product—let’s say if the website belongs to a retailer. We usually get that information from the retailer through various channels and we store product information in Cloud SQL for MySQL. We pull that information when we need to send an email with product information, and we use Memorystore for Redis to cache that information, since many of the products are repeatedly called. Our Cloud SQL instance has 16 vCPUs, 60GBs of memory and 0.5TB of disk space and when we perform those product information updates, we have about a thousand write transactions per second. We are also in the process of migrating some tables from a self managed MySQL instance, and we keep those tables synchronized with Cloud SQL using Datastream. 

Our user history database was originally stored in AWS DynamoDB, but we were running into problems with how they structured the data, and we’d often get hot shards but with no way to determine how or why. That led to our decision to migrate to Bigtable. We set up the migration first by writing the data to two locations from Pub/Sub, performed some backfill of data until that was up and running, and then started working on the reading. We performed this over a few short months, then switched everything to Bigtable. 

So, as mentioned, we are using Bigtable for multiple databases. The instance that stores our user events has about 30 TB with about 50 nodes.

Profile management

A second use case for Bigtable is for user profile management, where we track, for example, user attributes based on subscription activity, whether they’ve opted in or out of various lists, and where we apply list-specific rules that determine which targeted emails we send out to users. 

Our very own URL shortener

Our third use case for Bigtable is our URL shortener. When our customers build out campaigns and choose a URL, we append tracking information to the query string of the URLs and they become long. Many times, we are sending them via SMS texts, so the URLs need to be short. We originally used an external solution, but made the determination that they couldn’t support our future demands. Our calls tend to be very bursty in nature, and we needed to plan for a future state of supporting higher throughput. We use a separate table in Bigtable for this shortened URL. We generate the short slug that is 62 bit-encoded and use it as the rowkey. We use the long slug as a Protobuf-encoded data structure in one of the row cells and we also have a cell for counting how many times it was used. We use Bigtable’s atomic increment to increase that counter to track how many times the short slug was used. 

When the user receives a text message on their phone, they click the short URL, which goes through to us, and we expand it to the long slug (from Bigtable) and redirect them to the appropriate site location. Obviously, for the URL shortener use case, we need to make the conversion very quickly. Bigtable’s low latency helps us meet that demand and we can scale it up to meet higher throughput demands.

Meeting the future with Google Cloud

Our business has grown considerably, and as we keep signing up new clients, we need to scale up accordingly, and Bigtable has met our scaling demands easily. With Bigtable and other Google Cloud products powering our data architecture, we’ve met the demand of incredibly high traffic days in the last year, including Black Friday and Cyber Monday. Traffic for these events went much higher than expected, and Bigtable was there, helping us easily scale on demand. 

We are working on leveraging a more cloud native approach and using Google Cloud managed services like GKE, Dataflow, pub/sub, Cloud SQL , Memorystore, BigQuery and more. Google has those 1st party products and we don’t see the value in rolling out or self managing such solutions ourselves..

Thanks to Google Cloud, we now have reliable and flexible data solutions that will help us meet the needs of our growing customer base, and delight their users with fast, responsive, personalized shopping messaging and experiences. 

Learn more about Wunderkind and Cloud Bigtable. Or check out our recent blog exploring the differences between Bigtable and BigQuery.

We’re excited to announce that Spring Cloud GCP version 4.0 is now generally available! In this post, we’ll be describing what the new major version has to offer, and demonstrating the process of using the migration guide on one of our reference architectures, Bank of Anthos.

What’s new?

With this release, Spring Cloud GCP officially supports Spring Boot 3.x. However, this migration involves a number of breaking changes as outlined in the migration guide. The full list of changes made is available on Github, but the one of the most significant differences is that Java 17 is now a minimum requirement.

Another notable feature of this release is the addition of starter artifacts – Spring Boot starters for Google Cloud – that provide dependencies and auto-configurations for 80+ Google Client libraries. Just as the name suggests, these starters can serve as helpful starting points when working with a new client library. For now, they’re in preview. 

These starters are not included in the BOM by default and need to be added as a dependency to your project before they can be used. For example, if you wanted to get started with Cloud Text-to-Speech, you would include the following:

<dependency>
    <groupId>com.google.cloud</groupId>
    <artifactId>google-cloud-texttospeech-spring-starter</artifactId>
    <version>4.0.0-preview</version>
  </dependency>

The upgrade process

We prepared a migration guide to help answer any questions involved with moving from 3.x to 4.x.  Let’s follow those instructions to migrate Bank of Anthos.

We’ll start by cloning and building the application before the upgrade, according to the quickstart and development guide:

PROJECT_ID=<YOUR-PROJECT-ID>
gcloud services enable container --project ${PROJECT_ID}

git clone https://github.com/GoogleCloudPlatform/bank-of-anthos.git
cd bank-of-anthos/

gcloud services enable container.googleapis.com monitoring.googleapis.com \
  --project ${PROJECT_ID}

REGION=us-central1
gcloud container clusters create-auto bank-of-anthos \
  --project=${PROJECT_ID} --region=${REGION}

gcloud container clusters get-credentials bank-of-anthos \
  --project=${PROJECT_ID} --region=${REGION}

skaffold run --default-repo=gcr.io/${PROJECT_ID}/bank-of-anthos

At the end of this, you should see a “deployment stabilized” message:

Deployments stabilized in 9.657 seconds

With the environment set up for development, we’re ready to move on to the actual migration. The migration is already complete at time of writing, but here’s the full list of changes made:

• We began by checking the current Java, Spring Boot and Spring Cloud GCP versions.
  • Our Java version is okay, but the other two are out of date, and will need to be updated.
    We can also add the Spring Cloud GCP BOM to manage our versions going forward while we’re here.
• Then, in no particular order, we can make the following code changes:
  • Remove the explicit type declarations from our ResponseEntity instantiations.
  • Replace all javax.* imports with Jakarta.* counterparts, since Spring 3.0 makes use of Jakarta EE 9.0

With those changes in place, the app is upgraded and ready to re-deploy!

Live example

For a finished example of this migration, check out the Bank of Anthos repository on Github. It’s an excellent sample application that showcases a polyglot Java & Python app, served on Kubernetes and Google Cloud. All CI/CD and configurations are open source, so it may be instructive as you approach this migration.

Thanks for reading, and feel free to provide comments or feedback on Twitter, or in the issues section of the repository.

As part of our ongoing series on cost management for observability data in Google Cloud, we’re going to share four steps for getting the most out of your logs while on a budget. While we’ll focus on optimizing your costs within Google Cloud, we’ve found that this works with customers with infrastructure and logs on prem and in other clouds as well.

Step 1: Analyze your current spending on logging tools

To get started, create an itemized list of what volume of data is going where and what it costs. We’ll start with the billing report and the obvious line items including those under Operations Tools/Cloud Logging:

• Log Volume – the cost to write log data to disk once (see our previous blog post for an explanation)
• Log Storage Volume – the cost to retain logs for more than 30 days 

If you’re using tools outside Cloud Logging, you’ll also need to include any costs related to these solutions. Here’s a list to get you started:

• Log vendor and hardware costs — what are you paying to observability vendors? If you’re running your own logging solution, you’ll want to include the cost of compute and disk.
• If you export logs within Google Cloud, include Cloud Storage and BigQuery costs
• Processing costs — consider the costs for Kafka, Pub/Sub or Dataflow to process logs. Network egress charges may apply if you’re moving logs outside Google Cloud.
• Engineering resources dedicated to managing your logging tools across your enterprise often are significant too!

Step 2: Eliminate waste — don’t pay for logs you don’t need

While not all costs scale directly with volume, optimizing your log volume is often the best way to reduce spend. Even if you are using a vendor with a contract that locks you into a fixed price for a period of time, you may still have costs in your pipeline that can be reduced by avoiding wasteful logs such as Kafka, Pub/Sub or Dataflow costs. 

Finding chatty logs in Google Cloud

The easiest way to understand which sources are generating the highest volume of logs within Google Cloud is to start with our pre-built dashboards in Cloud Monitoring. To access the available dashboards:

1. Go to Monitoring -> Dashboards
2. Select “Sample Library” -> “Logging”

This blog post has some specific recommendations for optimizing logs for GKE and GCE using prebuilt dashboards.

As a second option, you can use Metrics Explorer and system metrics to analyze the volume of logs. For example, type “log bytes ingested” into the filter. This specific metric corresponds to the Cloud Logging “Log Volume” charge. There are many ways to filter this data. To get a big picture, we often start with grouping by both “resource_type” and “project_id”. 

To narrow down the resource type in a particular project, add a “project_id” filter. Select “sum” under the Advanced Options -> Click on Aligner and select “sum”. Sort by volume to see the resources with the highest log volume.

While these rich metrics are great for understanding volumes, you’ll probably want to eventually look at the logs to see whether they’re critical to your observability strategy. In Logs Explorer, the log fields on the left side help you understand volumes and filter logs from a resource type.

Reducing log volume with the Logs Router 

Now that we understand what types of logs are expensive, we can use the Log Router and our sink definitions to reduce these volumes. Your strategy will depend on your observability goals, but here are some general tools we’ve found to work well.

The most obvious way to reduce your log volume is not to send the same logs to multiple storage destinations. One common example of this is when a central security team uses an aggregated log sink to centralize their audit logs but individual projects still ingest these logs. Instead, use exclusion filters on the _Default log sink and any other log sinks in each project to avoid these logs. Exclusion filters also work on log sinks to BigQuery, Pub/Sub, or Cloud Storage.

Similarly, if you’re paying to store logs in an external log management tool, you don’t have to save these same logs to Cloud Logging. We recommend keeping a small set of system logs from GCP services such as GKE in Cloud Logging in case you need assistance from GCP support but what you store is up to you, and you can still export them to the destination of your choice!

Another powerful tool to reduce log volume is to sample a percentage of chatty logs. This can be particularly useful with 2XX log balancer logs, for example. This can be a powerful tool, but we recommend you design a sampling strategy based on your usage, security and compliance requirements and document it clearly.

Step 3: Optimize costs over the lifecycle of your logs

Another option to reduce costs is to avoid storing logs for more time than you need them. Cloud Logging charges based on the monthly log volume retained per month. There’s no need to switch between hot and cold storage in Cloud Logging; doubling the default amount of retention only increases the cost by 2%. You can change your custom log retention at any time.

If you are storing your logs outside of Cloud Logging, it is a good idea to compare the cost to retain logs and make a decision. 

Step 4: Setup alerts to avoid surprise bills

Once you are confident that the volume of logs being routed through log sinks fit in your budget, set up alerts so that you can detect any spikes before you get a large bill. To alert based on the volume of logs ingested into Cloud Logging:

1. Go to the Logs-based metrics page. Scroll down to the bottom of the page and click the three dots on “billing/bytes_ingested” under System-defined metrics. 
2. Click “ Create alert from metric”
3. Add filters (For example: use resource_id or project_id. This is optional). 
4. Select the logs based metric for the alert policy.

You can also set up similar alerts on the volume for log sinks to Pub/Sub, BigQuery or Cloud Storage.

Conclusion

One final way to stretch your observability budget is to use more Cloud Operations. We’re always working to bring our customers the most value possible for their budget such as our latest feature, Log Analytics, which adds querying capabilities but also makes the same data available for analytics, reducing the need for data silos. Many small customers can operate entirely on our free tier. Larger customers have expressed their appreciation for the scalable Log Router functionality available at no extra charge that would otherwise require an expensive event store to process data. So it’s no surprise that a 2022 IDC report showed that more than half of respondents surveyed stated that managing and monitoring tools from public cloud platforms provide more value compared to third-party tools. Get started with Cloud Logging and Monitoring today.

Product teams often serve as the nexus between all organizational units within a company, ensuring that a specific line of business follows a clear roadmap and that cross-functional alignment drives success. At their most empowered, product owners are like miniature CEOs for each offering within a company.

Unfortunately, in today’s complicated IT landscape, not everything that should be thought of as a product receives the designation. In my time as a digital consultant, I’ve seen one particular technology too often dismissed as back-office minutiae — application programming interfaces, or APIs.

APIs can be the glue that binds a company’s resources together and the mechanisms through which an enterprise’s data, functionality, and other digital assets are accessed and distributed. They enable software systems to communicate and let developers leverage software for new applications and digital experiences, abstracting underlying complexity into a consistent interface that allows data and functionality to be combined and reused with unprecedented flexibility.

Unfortunately, many business leaders see APIs as middleware and thus as undeserving of the resources of a full-fledged digital product. Their organizations create APIs as one-off projects, with engineers moving on to other projects when they are done and no one looking after an API’s maintenance or whether it offers utility beyond the bespoke use case for which it was created.

This is a problem because APIs can unlock digital assets that otherwise would be captive within enterprise silos. An API that exposes a customer database can likely be used and reused for numerous purposes, for example, and treating developers as customers of this API can help increase adoption of and the amount of innovation around it. But such a powerful outcome is unlikely if an API has been created only as a system integration detail rather than as a product with an owner who oversees a full lifecycle, from ideation to adoption to iteration. Put bluntly, a project-oriented attitude toward APIs misses the boat.

Pitney Bowes, a company I’ve worked with at Google Cloud, offers a high-profile example in which a notable enterprise has treated APIs as products and given its API program the same treatment it would give to any go-to-market offering. After selling traditional mailing and shipping solutions for nearly 100 years, the company transformed itself into a provider of digital logistics and e-commerce solutions by building out 160 public APIs and 400 internal APIs. This enabled its developers and partners to easily consume the company’s location intelligence and shipping services in new, agile ways. It also accelerated the company’s internal project timelines, shortening the average project lifecycle from over a year to just a few months. What’s more, APIs not only introduced efficiencies and reduced costs, but also opened new revenue streams. Because developers can more easily leverage individual Pitney Bowes business capabilities for their applications, the company’s services are manifesting in new ways — such as when Pitney Bowes technology is harnessed to let customers print USPS shipping labels from within the context of an online marketplace.

Manage APIs as products

Traditionally, but wrongly, APIs have been considered pieces of other products. They’ve been built by the IT team at the request of product owners every time a project came up that required one. When the funding spigot is flowing, APIs generally receive support. But once the project runs its course, the APIs are neglected and quickly grow outdated.

Without consistent resources, APIs can’t be adapted to fluctuations in the enterprise and market dynamics. Worst of all, project-oriented APIs often end up siloed within the business unit that gave birth to them, which defies the benefit of modern APIs: to not only allow enterprise assets to be reused by other business units and partners but also accelerate the business’s innovation by increasing the number of developers interacting with its capabilities.

In contrast, if an API is treated as a product in its own right with its own owner lobbying for consistent funding, the API can be iterated according to shifts in internal and market trends. Ideally, the product owner should continually monitor an API’s engagement and collect and synthesize analytics to improve quality of service. Product-minded companies frequently launch APIs as minimum viable products in order to get ideas to market faster, then use API consumption analytics to understand developer engagement and refine APIs to be even more attractive, powerful, and intuitive.

With the help of management tools that facilitate dynamic changes, an API product owner can also help ensure that developers benefit from the latest and greatest build with minimal interruption.

Virtually all traditional product ownership responsibilities can be easily applied to an API. An API product owner is responsible for galvanizing the entire company around their API’s value. Their team can answer any questions users may have about the API’s value proposition or how to use it. It may sound trivial but having dedicated experts available to assist with any question that may arise can be a boon to an API’s velocity.

Additionally, if a dedicated product owner is assigned to pilot a company’s aggregate API program, they can use comprehensive monitoring and analysis of all API traffic to derive insight into which APIs drive value and which don’t. This can save considerable time and money from being wasted on the wrong bets. The product owner can also create API bundles segmented according to each developer community the program wants to serve. Here, the goal is not just to expose enterprise assets, but also to encourage repeat consumption. When it comes to third-party engagement, these bundles should seek to improve partner stickiness to cultivate and maintain new revenue channels. Just as any product is only useful if it has users adopting it and sticking with it, APIs are typically valuable when they attract communities of repeat users.

Tools and processes that turn APIs into strategic assets

To drive an API as a strategic asset, an API product manager should support the API with management tools and processes.

First off, it’s necessary to establish comprehensive security layers for an API so cross-functional internal groups and third-party partners are comfortable using it. Mechanisms that apply protection with introducing user friction — such as OAuth, API key verification, XML/JSON threat protection — are strongly encouraged. It’s also often prudent to integrate machine learning functionality that can detect bot attacks and other adversaries before they breach an API and affect either its developer users or the end users consuming those developers’ apps.

Enterprise software products are typically afforded visualization tools, dashboards, reports, and self-service functionality to drive adoption — and APIs should be no different. Comprehensive portals that highlight API features (including knowledge centers, interactive documentation, blogs, and forums) should be crafted to promote developer consumption. Anything that is likely to magnetize collaborators and expedite registration should be provided for an API. A primary goal of API-first development is to increase velocity, so offering developers a self-service experience while maintaining visibility and control over API usage is crucial.

Manage with an “outside-in” perspective

Perhaps the most important role an API product owner must fulfill is maintaining a staunch outside-in approach.

In an inside-out perspective, enterprise leaders look internally for gaps and fill them with existing IT capabilities. Internal intuition untested by the market can define planning, and the resources that IT already has — rather than those that customers demand — typically restrict options. In contrast, an outside-in approach requires the company to listen to what the market is asking for and to respond appropriately. Outside-in thinkers start by asking what the customer needs, then build strategies from there. Because customer preferences are moving targets, an outside-in perspective is constantly evolving and seeking new data. In the case of API programs, product managers should find ways to align API user data with business KPIs, which can be essential to organizing stakeholders and achieving buy-in for the program throughout the organization.

Because both consumers and developers will abandon products that do not meet their needs, an outside-in perspective is increasingly non-negotiable. The product owner needs to analyze rapidly-changing market trends, understand the competitive landscape, keep a pulse on customer and stakeholder needs, and facilitate a backlog of changes that can help the company satisfy the market. Essentially, API product owners are the end-user advocate — and that’s for both for developers using the APIs and the people touching the applications powered by those APIs.

None of this means product owners need to be entirely risk averse, however. They should have the freedom to experiment with hypotheses and to make adjustments — but to make experimentation viable over the long run, they also need to have the foresight to kill a project immediately if the early end-user response is poor. If the product owner is not learning and iterating, they probably are not pushing the API program forward.

Calling all product professionals

If an enterprise plans to pursue an API-first mandate, it’s crucial that the API program is a well-funded, autonomous business unit. Without a visionary owner, APIs will not be able to usher in the transformation experienced by companies such as Pitney Bowes. The API-first philosophy requires that APIs are built prior to applications and that the API provide a value proposition of its own — which means that if an enterprise wants to accelerate innovation and take advantage of APIs, its first step should be to hire API product managers.

While many organizations are driving digital transformation by migrating to the cloud, there are some industries, geographies, and use cases that require a different approach to cloud modernization. Regulated industries such as healthcare, insurance, pharmaceutical, energy, telecommunication, and banking have stringent data residency and sovereignty requirements. Other industries need to meet local data processing requirements, while others require real-time data processing with sub-millisecond latencies, for example to detect defects on manufacturing lines. These use cases demand a combination of edge, on-premises and cloud services for their infrastructure.

With these requirements in mind, Google Cloud launched Google Distributed Cloud powered by Anthos to extend the power of Google Cloud infrastructure and services to the edge (or closer). The underlying infrastructure for this service comes in two variants: a 42U rack filled with compute, storage, and networking devices called Google Distributed Cloud Edge Rack and a 1U appliance called Google Distributed Cloud Edge Appliance.

In this blog post, we discuss the Google Distributed Cloud Edge Appliance and how manufacturing, retail, and automotive industry verticals can use it to address common use cases.

How the appliance works

But first, let’s talk about the Google Distributed Cloud Edge Appliance itself.

Google Distributed Cloud Edge Appliances comprises two components: (1) Distributed Cloud Edge infrastructure and (2) the Distributed Cloud Edge service.

The Google Distributed Cloud Edge service runs on Google Cloud and serves as a control plane for the nodes and clusters running on your appliance. In order to perform remote management of the appliance and to collect metrics, the Distributed Cloud Edge service must be connected to Google Cloud at all times, allowing you to manage your workloads on the edge hardware through the Google Cloud Console. For customers who can’t be connected at all times for data residency or sovereignty reasons, we highly recommend that the appliance be connected to the cloud at least once a month to allow for needed security patches and updates.

Google Distributed Cloud Edge Appliances come with built-in network ports that provide connectivity to the control plane via the internet, Cloud VPN, or Dedicated Interconnect, and to your on-prem network. Each Google Distributed Cloud Edge Appliance is homed to a specific Google Cloud region but it is designed to also use any public Google Cloud endpoint to communicate with the control plane in Google Cloud, allowing you to move these appliances between different geographic locations.

There are two NFS shares on each appliance; one is offline, meaning it does not transfer data to Google Cloud, and the other is online, meaning data saved to that share is synced to Cloud Storage on Google Cloud for further processing. The appliance supports Server Message Block (SMB) and Secure File Transfer Protocols (SFTP) for communication.

Each Google Distributed Cloud Edge Appliance runs Google Distributed Cloud Virtual, enabling you to build a single-node Kubernetes cluster with access to the underlying file system of the appliance. This allows you to build containerized applications on the underlying appliance hardware to address use cases in the following verticals.

Vertical use cases

Now that you understand how Google Cloud Edge Appliance is configured, let’s consider some of the industry use cases where it can provide unique value.

Manufacturing

In the manufacturing industry, quality control and safety is a crucial factor. Businesses need to ensure products are manufactured to the highest standards to remain competitive in their markets, to retain customers, and to keep factory workers safe. To do this, manufacturers need real-time data about the products being manufactured on the production lines, ensuring quality control and gaining a real-time view of where people are on the factory floor.

In manufacturing environments, Google Distributed Cloud Edge Appliance can be used to detect hazards or manufacturing defects in real-time. Figure 1 is a reference architecture for a hazard detection solution running off a Google Distributed Cloud Edge Appliance on a factory floor.

In this architecture, cameras on the factory floor stream live video into the Google Distributed Cloud Edge Appliance. Depending on the number of cameras and appliances, cameras could be split or mapped to different appliances. This architecture makes it possible to initially transfer video data to Google Cloud using an online NFS share. Once in Google Cloud, you can use AutoML to train and build models that can be used as part of the hazard detection solution.

With these trained models, the cameras can stream video data into the appliance using the real-time streaming protocol (RTSP). You can then use AutoML inference to analyze the real-time video streaming data.

For example, in this reference architecture, if an individual comes too close to the fork lift, a function is triggered by the microservices running on the edge appliance that pushes a notification either to a messaging service, or to an enterprise resource planning tool. This alerts factory managers to factory floor hazards in real time so they can take corrective action.

You can also review messages and videos later on for preventive planning purposes, or push streamed videos to Cloud Storage for archive, to use the appliance’s storage space more efficiently.

Data transfers to Google Cloud can be done over Google Cloud Dedicated Interconnect, or VPN between the region and your site. This connectivity also allows you to send the appliance’s control-plane network traffic to the region.

You could also use the reference architecture in figure 2 for a product anomaly detection solution running off a Google Distributed Cloud Edge Appliance on a factory floor or manufacturing line. In this instance, machine learning models are trained to detect anomalies on finished products before final packaging.

Retail

In the retail industry, the Google Distributed Cloud Edge Appliance reference architecture in Figure 2 enables a number of transformative capabilities for retail operations, including:

• contactless checkout
• product scans
• mobile-scan-bag
• cashierless checkout
• unattended retail shops
• visual check-out monitoring

It does all this within a retailer’s facilities with the low latency and high throughput you need to process data locally, so you can obtain actionable insights from your data.

Or, you could use Google Distributed Cloud Edge Appliance at the edge to overhaul store management operations, for example, monitoring store occupancy, queue depth and wait times, detecting slips and falls and out-of-stock items, or monitoring inventory compliance.

Automotive

Advanced Driver Assistance Systems (ADAS) are becoming standard in modern automobiles. To successfully build and roll out continued improvements around ADAS, the automotive industry continues to run extensive tests on ADAS systems that are built into the vehicles they manufacture. Automotive companies can use Google Distributed Cloud Edge Appliance to modernize and transform how they collect data for the ADAS systems they’re developing. For example, test vehicles contain several different sensors that generate data, which can be quickly offloaded to an in-vehicle edge appliance.

Then, within the appliance, you can deploy containerized workloads to transform sensor data, infer videos and images and detect events. This alleviates the need for operators to label all events and allows development teams to quickly gather insights from the tests.

If you want to focus on a subset of information, you can transfer specific data or the entire data payload into Transfer Appliances when vehicles return to the development center. All these systems, i.e., transfer appliances and edge appliances, work in tandem to reduce local system administration and operational costs through a cloud-based control plane.

This approach allows you to deploy, track, monitor and configure services that are running in data centers or at edge locations from the cloud. From the factories, the data can be moved offline or online into Google Cloud where you can use different storage classes and processing capabilities to further process or store the data. You can also deploy newly trained models and business rules back to the edge appliances. In all this, data transfers between the cloud and the appliance are performed using end-to-end encryption, to give you control over your data.

The reference architecture in Figure 3 shows an ADAS implementation where Google Distributed Cloud Edge Appliance is being used to gather, process and transform data at the edge in the automotive industry. It could also be applied to data capture and processing use cases in manned and unmanned vehicles. Notice how the Distributed Edge Appliance extends to the cloud by sending data there, or using other cloud-based services.

We’re just getting started

These are just a few of the use cases where organizations in the manufacturing, retail and automotive industries are using Google Distributed Cloud Edge Appliance with modern and containerized applications that are powered by Google Cloud. If you’re interested in bringing the power of Google Cloud to the edge using Google Distributed Cloud Edge Appliances to transform your business, reach out to us or any of our accredited partners.