Learn to Easily Administer Multi-cluster Kubernetes Environs: Part 4 KRM Series

5574
Of your peers have already read this article.
4:00 Minutes
The most insightful time you'll spend today!
This is part 4 in a multi-part series about the Kubernetes Resource Model. See parts 1, 2, and 3 to learn more.
Kubernetes clusters can scale. Open-source Kubernetes supports up to 5,000 Nodes, and GKE supports up to 15,000 Nodes. But scaling out a single cluster can only get you so far: if your cluster’s control plane goes down, your entire platform goes down; if the Cloud region running your cluster has a service interruption, so does your app.
Many organizations choose, instead, to operate multiple Kubernetes clusters. Besides availability, there are lots of reasons to consider multi-cluster, such as allocating a cluster to each development team, splitting workloads between cloud and on-prem, or providing burst capability for traffic spikes.
But operating a multi-cluster platform comes with its own challenges. How to consistently administer many clusters at once? How to keep the clusters secure? How to deploy and monitor applications running across multiple clusters? How to seamlessly fail over from one region to another?
This post introduces a few tools that can help platform teams more easily administer a multi-cluster Kubernetes environment.
The platform base layer, with Config Sync
In the last post, we explored how thoughtful platform abstractions can help reduce toil for app developers- including for a multi-cluster environment, where automation such as CI/CD handles all interactions with the staging and production clusters. But equally important is the platform base layer, the Kubernetes resources and configuration that are shared across services. Your platform base layer might consist of Namespaces, role-based access control, and shared workloads like Prometheus.
Platform abstractions depend on the existence of these base-layer resources. And so does the security and stability of your platform as a whole. It’s important that these resources not only get deployed, but also stay put. CI/CD is great for deploying resources, but what about making sure resources stay deployed? What if a Kubernetes Namespace gets deleted? Or a Prometheus StatefulSet is modified?
Kubernetes’ job is to ensure that the cluster’s actual state matches the desired state. But sometimes, the “desired” state isn’t desired at all – it’s a developer who mistakenly modified a resource, or a bad actor that’s gained access into the system. For this reason, a platform base layer needs more than a one-and-done CI/CD pipeline. A tool called Config Sync can help with this.
Config Sync is a Google Cloud product that can sync Kubernetes resources from a Git repository to one or more GKE or Anthos clusters. Unlike CI/CD tools like Cloud Build, Config Sync watches your clusters constantly, making sure that the intended resource state in the cluster always matches what’s in Git. Config Sync is designed primarily for base-layer resources like namespaces and RBAC. In this way, Config Sync is complementary to, not a replacement for, CI/CD.

Config Sync runs in a Pod inside your Kubernetes cluster, watching your Git config repo for changes, and also watching the cluster itself for any divergence from your desired state in Git. If any configuration drift is detected from what’s stored in Git, Config Sync will update the API Server accordingly.
You can point multiple Config Sync deployments at the same Git repo, allowing you to manage the base-layer platform resources for multiple clusters using the same source of truth. And by using Git as the landing zone for config, you can benefit from some of the GitOps principles we discussed in part 2, including the ability to audit and roll back configuration changes.
Let’s walk through an example of how to manage base-layer resources with Config Sync.
The Cymbal Bank platform consists of four GKE clusters: admin, dev, staging, and prod. We can install Config Sync on all four clusters using the gcloud tool or the Google Cloud Console, pointing all four clusters at a single Git repository, called cymbalbank-policy. Note that this repo is separate from the application source and config repos, and is managed by the platform team. From the Console, we can see that all four clusters are synced to the same commit of the cymbalbank-policy repo.

Now, let’s say that the Cymbal Bank platform team wants to limit the amount of CPU and memory resources each application team can request for their service. Kubernetes ResourceQuotas help impose these limits, and prevent unexpected Pod evictions.

The platform team can define a set of ResourceQuotas for each application namespace. They can also scope the resources to only be applied to a subset of clusters – for instance, to the production cluster only. (If no cluster name selector is specified, Config Sync will deploy the resource to all clusters by default.)
apiVersion: v1kind: ResourceQuotametadata:name: production-quotanamespace: frontendannotations:configsync.gke.io/cluster-name-selector: cymbal-prodspec:hard:cpu: 700mmemory: 512Mi
From here, the platform team can commit the resources to the cymbalbank-policy repo, and Config Sync, always watching the policy repo, will deploy the resources to the production cluster:
NAMESPACE NAME AGE REQUEST LIMITbalancereader production-quota 6m56s cpu: 300m/700m, memory: 612Mi/512Mi
If a developer tries to delete one of the ResourceQuotas, Config Sync will block the request, helping to ensure that these base-layer resources stay put.
error: You must be logged in to the server (admission webhook "v1.admission-webhook.configsync.gke.io" denied the request: requester is not authorized to delete managed resources)
In this way, Config Sync can help platform teams ensure the stability of that platform base-layer, as well as ensure resource consistency across multiple clusters at once. This, in turn, can help organizations mitigate the complexity of adding new clusters to their environment.
Enforce policies on Kubernetes resources
Config Sync is a powerful tool on its own, and can work with any Kubernetes resource that your cluster recognizes. This includes Custom Resource Definitions (CRDs) installed with add-ons like Anthos Service Mesh.
But Config Sync, by default, doesn’t have an idea of “good or bad” Kubernetes resources. It will deploy whatever resources land in Git, even resources that might pose a security risk to your organization. Security is an essential feature of any developer platform, and when it comes to Kubernetes, it’s important to think about security from the initial software design stages, and set up your clusters with security best-practices in mind.
But it’s just as important to think about security at deploy-time. Who and what can access your clusters? What kinds of Kubernetes resources – and fields within those resources- are allowed? These decisions will depend on lots of factors, including the kinds of data your application deals with, and any industry-specific regulations.
One common security use case for KRM is the need to monitor incoming Kubernetes resources, whether they’re coming in through kubectl, CI/CD, or Config Sync. But if you have multiple clusters, your Kubernetes environment has multiple API Servers, and therefore multiple entry points.
A Google tool called Policy Controller can help automate resource monitoring across multiple clusters. Policy Controller is a Kubernetes admission controller that can accept or reject incoming resources based on custom policies you define. Policy Controller is based on the OpenPolicyAgent Gatekeeper project, and it allows you to define policies, or “Constraints,” as KRM. This means you can deploy them using Config Sync, via Git. Once deployed, Policy Controller uses your Constraints as a set of rules to evaluate all incoming KRM, rejecting resources that fall out of compliance.
Let’s walk through an example. Say that the Cymbal Bank security team wants to ensure that no code in development is accessible to the public. Kubernetes Services of type LoadBalancer expose public IP addresses by default, so the platform team wants to define a PolicyController constraint that blocks Services of that type on the development GKE cluster.

To do this, the platform team can define a Policy Controller Constraint as KRM. This Constraint uses a Constraint Template, provided through the pre-installed Constraint Template library. The ConstraintTemplate defines the logic of the policy itself, and the Constraint makes the template concrete, populating any variables needed to execute the policy logic. Here, we’re also adding a Config Sync cluster name annotation, to scope this resource to apply only to the development cluster.
apiVersion: constraints.gatekeeper.sh/v1beta1kind: K8sNoExternalServicesmetadata:name: dev-no-ext-servicesannotations:configsync.gke.io/cluster-name-selector: cymbal-devspec:internalCIDRs: []
The platform team can then commit the resource to the cymbalbank-policy repo, and Config Sync will deploy the resource to the development cluster.
From here, if an app developer tries to create an externally-accessible Kubernetes Service, Policy Controller will block the resource from being created.
for: "constraint-ext-services/contacts-svc-lb.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [denied by dev-no-ext-services] Creating services of type `LoadBalancer` without Internal annotation is not allowed
The platform team can define as many of these Constraints as they want, each defining a separate policy.
Writing custom policies
The Policy Controller Constraint Template library provides a lot of functionality, from blocking privileged containers, to requiring certain resource labels, to preventing app teams from deploying into certain namespaces. But if you want to enforce custom logic on your organization’s KRM, you can do so by writing a custom Constraint Template.
Constraint Templates are written in a query language called Rego. Rego was designed for policy rule evaluation, and it can introspect Kubernetes resource fields to make a conclusion as to whether the resource is allowed or not.
For instance, let’s say that the platform team wants to limit the number of containers allowed inside a single application Pod. Too many containers per Pod can cause outage risks— when one container crashes, the entire Pod crashes.

To enforce this policy, the platform team can define a Constraint Template, using the Rego language, that looks inside a resource to ensure that the number of containers per Pod is within the allowed limit:
apiVersion: templates.gatekeeper.sh/v1beta1kind: ConstraintTemplatemetadata:name: k8slimitcontainersperpodspec:crd:spec:names:kind: K8sLimitContainersPerPodvalidation:openAPIV3Schema:properties:allowedNumContainers:type: integertargets:- target: admission.k8s.gatekeeper.shrego: |package k8slimitcontainersperpodnumTemplateContainers := count(input.review.object.spec.template.spec.containers)numRunningContainers := count(input.review.object.spec.containers)containerLimit := input.parameters.allowedNumContainerstemplate_containers_over_limit = true {numTemplateContainers > containerLimit}running_containers_over_limit = true {numRunningContainers > containerLimit}violation[{"msg": msg}] {template_containers_over_limitmsg := sprintf("Number of containers in template (%v) exceeds the allowed limit (%v)", [numTemplateContainers, containerLimit])}violation[{"msg": msg}] {running_containers_over_limitmsg := sprintf("Number of running containers (%v) exceeds the allowed limit (%v)", [numRunningContainers, containerLimit])}Then, the platform team can define a concrete Constraint, using this Constraint Template, to set the number of allowed containers per Pod to 3:apiVersion: constraints.gatekeeper.sh/v1beta1kind: K8sLimitContainersPerPodmetadata:name: limit-three-containersspec:parameters:allowedNumContainers: 3
Finally, the platform team can push these resources to the cymbalbank-policy repo, and Config Sync will deploy the policy to all four clusters. If a developer tries to define a Kubernetes Deployment containing more containers per pod than what’s allowed, the resource will be blocked at deploy time:
Error from server ([limit-three-containers] Number of containers in template (4) exceeds the allowed limit (3)): error when creating "constraint-limit-containers/test-workload.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [limit-three-containers] Number of containers in template (4) exceeds the allowed limit (3)
Custom Constraint Templates can give platform teams lots of flexibility in the types of policies they define and enforce in a Kubernetes environment.
Integrating policy checks into CI/CD
As we explored earlier, Config Sync and CI/CD are complementary tools. Config Sync works great for base-layer platform resources and policies, whereas CI/CD works well for application tests and deployment.
But one pitfall of having two separate KRM deployment mechanisms is that app developers may not know that their resources are out of policy until they try to deploy them into production. This is especially true if some policies are scoped only to production, as we saw with the ResourceQuota example. Ideally, the platform team has a way to empower developers and code reviewers to know ahead of time whether new or modified resources are still in compliance. We can enable this use case by integrating policy checks into the existing Cymbal Bank CI/CD.

Policy Controller operates, by default, as a Kubernetes Admission Controller running inside the cluster. But Policy Controller also provides a “standalone” mode, running inside a container, that can be used outside of a cluster, such as from inside a Cloud Build pipeline.
In the example below, Cloud Build executes Policy Controller checks by getting the cymbalbank-app-config manifests, cloning the cymbalbank-policy resources, and using the “policy-controller-validate” container image to evaluate the app manifests against the policies.
steps:- id: 'Render prod manifests'name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'entrypoint: '/bin/sh'args: ['-c', 'mkdir hydrated-manifests && kubectl kustomize overlays/prod > hydrated-manifests/prod.yaml']- id: 'Clone cymbalbank-policy repo'name: 'gcr.io/kpt-dev/kpt'entrypoint: '/bin/sh'args: ['-c', 'kpt pkg get https://github.com/$$GITHUB_USERNAME/cymbalbank-policy.git@main constraints&& kpt fn source constraints/ hydrated-manifests/ > hydrated-manifests/kpt-manifests.yaml']secretEnv: ['GITHUB_USERNAME']- id: 'Validate prod manifests against policies'name: 'gcr.io/config-management-release/policy-controller-validate'args: ['--input', 'hydrated-manifests/kpt-manifests.yaml']availableSecrets:secretManager:- versionName: projects/${PROJECT_ID}/secrets/github-username/versions/1env: 'GITHUB_USERNAME'timeout: '1200s' #timeout - 20 minutes
From here, an app developer or operator can know if their resources violate org-wide policies, by looking at the Cloud Build output for their Pull Request:
Status: Downloaded newer image for gcr.io/config-management-release/policy-controller-validate:latestError: Found 1 violations:[1] Number of containers in template (4) exceeds the allowed limit (3)
By integrating policy checks into CI/CD, app development teams can understand whether their resources are in compliance, and platform teams add an additional layer of policy checks to the platform.
Overall, Config Sync and Policy Controller can provide a powerful toolchain for standardizing base-layer config across a multi-cluster environment. Check out the Part 4 demo to try out each of these examples.
And stay tuned for Part 5, where we’ll learn how to use KRM to manage cloud-hosted resources.
A Record Breaking Calculation: 100 Trillion Digits of π on Google Cloud!

3577
Of your peers have already read this article.
4:00 Minutes
The most insightful time you'll spend today!
Records are made to be broken. In 2019, we calculated 31.4 trillion digits of π — a world record at the time. Then, in 2021, scientists at the University of Applied Sciences of the Grisons calculated another 31.4 trillion digits of the constant, bringing the total up to 62.8 trillion decimal places. Today we’re announcing yet another record: 100 trillion digits of π.
This is the second time we’ve used Google Cloud to calculate a record number1 of digits for the mathematical constant, tripling the number of digits in just three years.
This achievement is a testament to how much faster Google Cloud infrastructure gets, year in, year out. The underlying technology that made this possible is Compute Engine, Google Cloud’s secure and customizable compute service, and its several recent additions and improvements: the Compute Engine N2 machine family, 100 Gbps egress bandwidth, Google Virtual NIC, and balanced Persistent Disks. It’s a long list, but we’ll explain each feature one by one.
Before we dive into the tech, here’s an overview of the job we ran to calculate our 100 trillion digits of π.
- Program: y-cruncher v0.7.8, by Alexander J. Yee
- Algorithm: Chudnovsky algorithm
- Compute node: n2-highmem-128 with 128 vCPUs and 864 GB RAM
- Start time: Thu Oct 14 04:45:44 2021 UTC
- End time: Mon Mar 21 04:16:52 2022 UTC
- Total elapsed time: 157 days, 23 hours, 31 minutes and 7.651 seconds
- Total storage size: 663 TB available, 515 TB used
- Total I/O: 43.5 PB read, 38.5 PB written, 82 PB total

Architecture overview
Calculating π is compute-, storage-, and network-intensive. Here’s how we configured our Compute Engine environment for the challenge.
For storage, we estimated the size of the temporary storage required for the calculation to be around 554 TB. The maximum persistent disk capacity that you can attach to a single virtual machine is 257 TB, which is often enough for traditional single node applications, but not in this case. We designed a cluster of one computational node and 32 storage nodes, for a total of 64 iSCSI block storage targets.

The main compute node is a n2-highmem-128 machine running Debian Linux 11, with 128 vCPUs and 864 GB of memory, and 100 Gbps egress bandwidth support. The higher bandwidth support is a critical requirement for the system as we adopted a network-based shared storage architecture.
Each storage server is a n2-highcpu-16 machine configured with two 10,359 GB zonal balanced persistent disks. The N2 machine series provides balanced price/performance, and when configured with 16 vCPUs it provides a network bandwidth of 32 Gbps, with an option to use the latest Intel Ice Lake CPU platform, which makes it a good choice for high-performance storage servers.
Automating the solution
We used Terraform to set up and manage the cluster. We also wrote a couple of shell scripts to automate critical tasks such as deleting old snapshots, and restarting from snapshots (we didn’t need to use this though). The Terraform scripts created OS guest policies to help ensure that the required software packages were automatically installed. Part of the guest OS setup process was handled by startup scripts. In this way, we were able to recreate the entire cluster with just a few commands.
We knew the calculation would run for several months and even a small performance difference could change the runtime by days or possibly weeks. There are also a number of combinations of parameters in the operating system, infrastructure, and application itself. Terraform helped us test dozens of different infrastructure options in a short time. We also developed a small program that runs y-cruncher with different parameters and automated a significant portion of the measurement. Overall, the final design for this calculation was about twice as fast as our first design. In other words, the calculation could’ve taken 300 days instead of 157 days!
The scripts we used are available on GitHub if you want to look at the actual code that we used to calculate the 100 trillion digits.
Choosing the right machine type for the job
Compute Engine offers machine types that support compute- and I/O-intensive workloads. The amount of available memory and network bandwidth were the two most important factors, so we selected n2-highmem-128 (Intel Xeon, 128 vCPUs and 864 GB RAM). It satisfied our requirements: high-performance CPU, large memory, and 100 Gbps egress bandwidth. This VM shape is part of the most popular general purpose VM family in Google Cloud.
100 Gbps networking
The n2-highmem-128 machine type’s support for up to 100 Gbps of egress throughput was also critical. Back in 2019 when we did our 31.4-trillion digit calculation, egress throughput was only 16 Gbps, meaning that bandwidth has increased by 600% in just three years. This increase was a big factor that made this 100-trillion experiment possible, allowing us to move 82.0 PB of data for the calculation, up from 19.1 PB in 2019.
We also changed the network driver from virtio to the new Google Virtual NIC (gVNIC). gVNIC is a new device driver and tightly integrates with Google’s Andromeda virtual network stack to help achieve higher throughput and lower latency. It is also a requirement for 100 Gbps egress bandwidth.
Storage design
Our choice of storage was crucial to the success of this cluster – in terms of capacity, performance, reliability, cost and more. Because the dataset doesn’t fit into main memory, the speed of the storage system was the bottleneck of the calculation. We needed a robust, durable storage system that could handle petabytes of data without any loss or corruption, while fully utilizing the 100 Gbps bandwidth.
Persistent Disk (PD) is a durable high-performance storage option for Compute Engine virtual machines. For this job we decided to use balanced PD, a new type of persistent disk that offers up to 1,200 MB/s read and write throughput and 15-80k IOPS, for about 60% of the cost of SSD PDs. This storage profile is a sweet spot for y-cruncher, which needs high throughput and medium IOPS.
Using Terraform, we tested different combinations of storage node counts, iSCSI targets per node, machine types, and disk size. From those tests, we determined that 32 nodes and 64 disks would likely achieve the best performance for this particular workload.
We scheduled backups automatically every two days using a shell script that checks the time since the last snapshots, runs the fstrim command to discard all unused blocks, and runs the gcloud compute disks snapshot command to create PD snapshots. The gcloud command returns and y-cruncher resumes calculations after a few seconds while the Compute Engine infrastructure copies the data blocks asynchronously in the background, minimizing downtime for the backups.
To store the final results, we attached two 50 TB disks directly to the compute node. Those disks weren’t used until the very last moment, so we didn’t allocate the full capacity until y-cruncher reached the final steps of the calculation, saving four months worth of storage costs for 100 TB.
Results
All this fine tuning and benchmarking got us to the one-hundred trillionth digit of π — 0. We verified the final numbers with another algorithm (Bailey–Borwein–Plouffe formula) when the calculation was completed. This verification was the scariest moment of the entire process because there is no sure way of knowing whether or not the calculation was successful until it finished, five months after it began. Happily, the Bailey-Borwein-Plouffe formula found that our results were valid. Woo-hoo! Here are the last 100 digits of the result:
4658718895 1242883556 4671544483 9873493812 1206904813
2656719174 5255431487 2142102057 7077336434 3095295560
You can also access the entire sequence of numbers on our demo site.
So what?
You may not need to calculate trillions of decimals of π, but this massive calculation demonstrates how Google Cloud’s flexible infrastructure lets teams around the world push the boundaries of scientific experimentation. It’s also an example of the reliability of our products – the program ran for more than five months without node failures, and handled every bit in the 82 PB of disk I/O correctly. The improvements to our infrastructure and products over the last three years made this calculation possible.
Running this calculation was great fun, and we hope that this blog post has given you some ideas about how to use Google Cloud’s scalable compute, networking, and storage infrastructure for your own high performance computing workloads. To get started, we’ve created a codelab where you can create and calculate pi on a Compute Engine virtual machine with step-by-step instructions. And for more on the history of calculating pi, check out this post on The Keyword. Here’s to breaking the next record!
- We are actively working with Guinness World Records to secure their official validation of this feat as a “World Record”, but we couldn’t wait to share it with the world. This record has been reviewed and validated by Alexander J. Yee, the author of y-cruncher.
Sainsbury’s Uses AI to Figure Out How the World Eats

8987
Of your peers have already read this article.
4:45 Minutes
The most insightful time you'll spend today!
Retail will forever be an industry that must constantly reinvent itself in response to, and anticipation of, ever-changing consumer demands.
Digital transformation is fueling these changes and we’ve previously spoken about how businesses including Ulta Beauty and Kohl’s are taking advantage of Google Cloud to put data at the center of what they do and deliver the best possible shopping experience and product offerings for their customers.
Leveraging Google Cloud machine learning platform, Sainsbury is able to develop predictive analytics models to spot trends and adjust inventory, providing shoppers with a better experience.
Sainsbury’s, one of Britain’s best-known supermarkets, is another great example of a business transforming the way it engages with its customers with the cloud.
With over 150 years of service, Sainsbury’s vision is to be the most trusted retailer, where people love to work and shop. It makes customers’ lives easier, by offering great quality and service at fair prices.
The food industry and the way that customers shop is rapidly changing. From foodie hashtags on Instagram, to the latest cooking fads, customers want to stay connected to the latest trends and Sainsbury’s is empowering them do that.
To help Sainsbury’s achieve this goal, its Commercial and Technology teams, in partnership with Accenture, are building cutting-edge machine learning solutions on Google Cloud Platform (GCP) to provide new insights on what customers want and the trends driving their eating habits.
With the help of Google Cloud Platform, we are generating new insights into how the world eats and lives, to help us stay ahead of market trends and provide an even better shopping experience for our customers.
–Phil Jordan, Group CIO, Sainsbury’s
Sainsbury’s solution relies on data from multiple structured and unstructured sources. Using Google Cloud’s powerful cloud-based analytics tools to ingest, clean and classify that data, and a custom-built front-end interface for internal users to seamlessly navigate through a variety of filters and categories, Sainsbury’s is able to gain advanced insights in real time.
As a result, Sainsbury’s has been able to develop predictive analytics models to spot trends and adjust inventory, providing shoppers with a better experience.
Phil Jordan, Group CIO of Sainsbury’s believes this project will have a big impact.
“The grocery market continues to change rapidly. We know our customers want high quality at great value and that finding innovative and distinctive products is increasingly important to them. With the help of Google Cloud Platform, we are generating new insights into how the world eats and lives, to help us stay ahead of market trends and provide an even better shopping experience for our customers.”
This project is also a great example of the successes Google Cloud customers have when they work with the company’s partners.
“We’re delighted to partner with Google Cloud to help the Sainsbury’s Commercial team apply predictive analytics to the identification of new and emerging trends in grocery,” says Adrian Bertschinger, Managing Director for Retail, Accenture.
“The food sector is experiencing significant, rapid disruption, and this new, cloud-based insights platform will help Sainsbury’s identify trends much earlier and adapt their product assortment in a faster, more informed way—all for the benefit of customers.”
Whatever the next food or shopping trend may be, Sainsbury’s is looking to the cloud to help them stay a step ahead.
AgroStar: Small farms in India getting big help from the cloud

13242
Of your peers have already read this article.
3:30 Minutes
The most insightful time you'll spend today!
AgroStar has launched a cloud-based mobile app that is helping to boost crop yields and encourage best practices for small farmers in India. Launched as an on-premises ecommerce platform selling farm tools in 2008, the firm turned to Google Cloud Platform (GCP) to expand its offering. It now uses cloud-based analytics and is deploying ML models to provide timely advice in five languages on everything from seed optimization, crop rotation, and soil nutrition to pest control.
A 2018 survey underscored the demand for agricultural planning for Indian farmers. While farming remains a dominant sector in India, employing half of its labor force, 70 percent of small farmers – those cultivating fewer than three acres – said their crops are damaged by unforeseen weather and pests. An even higher number – 74 percent – say they lack access to farming-related information.
Widening that gap is the relative lack of access to new, higher yield seeds and improved soil analyses for small farmers, who must otherwise rely on traditional methods. “It could take a few years for innovative information to trickle down from universities to small, grassroots farmers,” says Pritesh Gudge, AgroStar Software Engineer. “Today, just by clicking through our Android application, farmers learn about new, effective farming practices and receive advice customized to their crop and soil.”
Connecting a million farmers in the cloud
Operating in the Indian states of Gujarat, Maharashtra, Rajasthan, Orissa, Bihar, and Karnataka, AgroStar is closing the knowledge gap with a full-service, cloud-based SaaS solution – the only one of its kind in India. It combines agronomy, data science, and analytics to help farmers by providing a variety of resources.
AgroStar has reached over a million farmers through its Android app, the AgroStar Agri-Doctor. The mobile client is available as a web-based or full-featured native app. Both provide access to the firm’s knowledge base hosted on GCP, a Q&A forum that connects farmers to each other to help understand and better solve problems and to learn about innovative practices and products. Farmers can also click through to follow local and national market trends that help forecast crop prices.
In addition to the self-service knowledge base, AgroStar provides access to agronomy experts who use cloud-based analytics tools and historical data to provide season-and locale-specific advice to each farmer. “We are now tracking thousands of calls in 5 languages each day,” says Pritesh.
The AgroStar app also provides links to purchase and then track the delivery of farm tools and supplies such as cultivators and fertilizers. An in-house platform manages fulfillment centers and a doorstep delivery network simplifies the supply chain while giving farmers what they need, when they need it. By procuring directly from the manufacturers and primary distributors of farm supplies, Agrostar is achieving cost savings, which it passes on to farmers.
Build fast, pivot faster
From the start, the human and environmental variables of farming in India, not to mention the volume of AgroStar’s few hundred thousand monthly active users, made a highly scalable cloud-based solution inevitable. Farmers rely on the firm’s Agri-Doctor app to provide advice in multiple languages on topics that range widely throughout three growing seasons, each with distinct crop nutrition and rotation cycles and farm implementation requirements.
“For farmers, the focus keeps changing every month, and every season,” says Pritesh. “To serve our growing community, we needed a platform that could process images at high volume, fulfill tools and seed orders across thousands of miles, and respond to multilingual queries. We quickly moved away from spreadsheets and server-based solutions – we needed to build fast and pivot faster.”
Ending late-night deployments
The firm’s first cloud experience was with an AWS solution. At the time, AWS was the only cloud provider in India, but AgroStar wanted to find a solution that was easier to use and offered better integration with Android devices. “Deployment and processing costs were very high, and the developer tools and documentation were not as intuitive as we needed,” says Pritesh.
When GCP service arrived in India in October 2017, AgroStar embarked on a platform re-implementation that made possible dramatic changes in the way it developed and deployed its solution. Using Google Kubernetes Engine (GKE) for crop advice management and Compute Engine for its production application services, the firm built the backend for the Agri-Doctor discussion forum in only three weeks. The platform’s microservice architecture is implemented in Python and Golang and deployed on GCP.
AgroStar began to realize significant efficiencies in its build, deploy, and test cycles. “We previously needed to work overnight to deploy to production,” says Pritesh. “Now using Google for Kubernetes containers and a rolling update strategy, we can deploy during the day without any problems or interruptions to service.”
The move to GCP streamlined AgroStar’s stack. “We were running 12 independent instances on AWS,” says Pritesh. “With Google Kubernetes Engine, we are deployed on a single cluster at a cost savings of $1,300 per month and growing.”
Improving customer response times by 85 percent
With a managed deployment capability, AgroStar can devote more time and resources to executing on its platform and Agri-Doctor app development plan. A strategic goal was managing customer response times as the firm grew its base. GCP has helped the firm meet that goal, achieving an 85 percent improvement in customer response times even as traffic grew significantly.
“With our on-premises solution, we could handle around 100 customers daily, which took 30 to 50 minutes for each customer,” says Pritesh. “We now handle thousands of customers daily, taking only 4 to 5 minutes for each one.”
AgroStar used Firebase to implement its Agri-Doctor app. A real-time cloud database, Firebase provides an API that enables the Agri-Doctor advice forum to be synchronized across all its far-flung mobile clients, effectively sharing knowledge base updates with one million users in near real time.
Using cloud tools to manage and monitor
Cloud Pub/Sub, Kafka, and Cloud Dataflow manage data ingestion and queueing of event and transaction data to the analytics layer. BigQuery fetches and persists data to Cloud Storage. Cloud SQL and dashboards powered by Tableau deliver farmer crop and soil profiles within minutes.
Cloud IAM helps AgroStar control access to all its cloud resources. And Stackdriver, the integrated logging aggregation capability for GCP, helps monitor and speed debugging on every tier of the AgroStar solution.
Machine learning to enhance yields
AgroStar is developing a variety of ML components to improve responsiveness and extend its platform offerings.
To speed up the diagnosis of and treatment for crop blight, AgroStar is building a deep learning pipeline using TensorFlow. The pipeline relies on GoogLeNet models that use multi-layered convolutional visual pattern recognition. It will assess uploaded images to support a disease-detection capability on the mobile app. Based on the commercially successful AI algorithms that automated postal code processing, GoogLeNet offers improved performance and computational efficiencies by using a creative layering technique that distinguishes them from older, sequential recognition engines.
To improve its customer search experience, AgroStar is developing an ML pipeline that shrinks fetch times by suggesting tags mapped to stored data. Processed using TPUs, Cloud Natural Language and Video AI, the tags provide a metadata layer that supports queries in any of the ten natural languages that AgroStar farmers can use.
The AgroStar search pipeline consists of Long Short-Term Memory (LSTM) models of Recurrent Neural Networks. Recurrent networks exhibit “memory” through iterative processing and are distinguished from feedforward networks by a feedback loop connected to their past decisions, ingesting their own outputs moment after moment as input.
Implementing a recommendation engine
The firm is also adapting the Random Forests TensorFlow AI model to develop a crop and product recommendation engine. The model is trained by consuming numerical (rainfall, humidity, water availability per acre) and categorical (soil type, water sources) parameters to suggest appropriate products by season, region, and locale.
To simplify the product suggestion experience, AgroStar developers are testing Cloud Dialogflow, the Google Cloud conversational interface, to build a chatbot capability into its mobile app. The bot will track a farmer’s crop schedules and answer simple questions by linking to the recommendation engine.
AgroStar is also extending its analytics platform with AI-powered sales planning and forecasting. Using linear regression models implemented in TensorFlow and powered by Cloud ML Engine, the capability will enhance supply chain logistics as the company scales its operations across India.
To provide a credit on-demand offering for a range of seed-to-harvest cycle products, AgroStar is attempting to use Vision API to create an AI model that will convert uploaded photos of customer application records into standard data formats. The firm’s credit policy features a grace period in which farmers begin paying back loans after harvested crops go to market.
A versatile and friendly development ecosystem
AgroStar credits the convivial tools and documentation that GCP offers and its incremental, pay-as-you-go pricing model for both the firm’s success and its ability to manage growth.
“What Google Cloud offers is extremely good documentation and extremely simple-to-use tools and interfaces across all services,” says Pritesh. “It helped us initially deploy our platform and at every scale that we have required since then, and its cost effectiveness enabled us to staff up to meet new feature milestones.”
Google Products Helps HMH’s Healthcare Staff Work from Anywhere Efficiently and Securely!

10226
Of your peers have already read this article.
3:00 Minutes
The most insightful time you'll spend today!
Hackensack Meridian Health (HMH) executive Mark Eimer explains how an ambitiously-timed rollout of a comprehensive suite of Google products helped the entire organization—from doctors to IT staff—achieve better security, cultivate a more equitable work environment, and ultimately, improve patient outcomes.
How does a recently merged, 17-hospital healthcare system fast-track a platform migration and hardware rollout securely and in a way that improves work for everyone, regardless of location or role? These are the questions that kept me up at night in early 2020, when the pandemic demanded a “big bang”—something our legacy laptops and operating systems couldn’t handle.
We began our work with Google in 2020 with the adoption of Chrome as our default browser. As we migrated platforms, keeping patient data safe was of the utmost importance to us, along with providing every staff member with the tools they needed to work virtually. Our staff often experienced issues accessing our web-based applications using Internet Explorer or Edge Browser, a problem that went away when we switched to Chrome. Chrome’s versatile compatibility also made it easier for my team to migrate all of our web-based operations, and Chrome’s security and manageability were key components to making this switch a huge win for the organization.
The success of this migration led us to extend our Google partnership to patient care applications—where Google’s expertise in AI and ML helps scale the use of diagnostics tools and improve other aspects of the patient journey.

Achieving security at every step
Like so many other healthcare organizations, we’ve been concerned about ransomware attacks. This is part of why we moved to Google Workspace and distributed over 3,000 Chrome OS devices in kiosk mode in March of 2020, when many of us went remote due to the pandemic. We were very concerned about team members accessing corporate applications through home devices that were running EOL operating systems (WIN7), as well as a general lack of antivirus and encryption measures.
We were protected by the fact that Google’s software and hardware both had built-in security features that we needed to stave off sophisticated attackers. For example, Chrome OS automatically updates to the latest security update and encrypts data living outside the cloud on the hardware. These features protected us from security-related disruptions, letting us securely move a huge library of file shares and emails across thousands of accounts to Google Chrome OS in just four months.
A year later, in March 2021, we migrated the enterprise over to Google Workspace and saw an immediate reduction in spam by 30% from the inherent built-in AI/ML. This meant staff were less likely to receive (and click through) phishing attempts. My team could connect, create, and collaborate easily and securely—even as more of us were working from home and needed to access sensitive data remotely.
Leveling the playing field
As an organization, we were surprised by how many team members didn’t have personal computers at home. We quickly decided that if we needed team members to work from home, the health network would have to supply hardware. Chromebooks’ lower price tag compared to PCs—on top of their built-in security controls—allowed us to purchase, deploy, and support that initial distribution of 3,000 Chromebooks to team members in less than three weeks, providing devices to every eligible remote employee instead of just a select few. This was vital to reaching our equitable technology goal as part of our diversity and inclusion initiative: everybody has the same tools to do good work.
When all employees have what they need to do their jobs well, we get better patient outcomes. Before we began this cloud adoption journey, patient and staff experiences were different within the hospitals and outside of them.
Now it’s the same wherever our staff is, and we’ve seen efficiency and accessibility benefits extend to the patient side. For example, we built a web-based contact center that supports 80 locations that use Workspace and Chrome OS devices. Since customer service, admin, and providers are all on the same system, it has become a one-stop shop for patients.
Furthermore, through the Grow with Google program, we were able to provide another benefit to employees that drove our equity goals. Google trained 50 non-IT staff members—from environmental services, food and nutrition, and other non-tech areas who were interested in making a career change to IT—on the Google products we were using. They may not have thought about switching to a career in IT before the Grow with Google program came to our organization, but through this partnership, they now have that opportunity.
A strategic, long-term partner
With any large-scale rollout, the work doesn’t end once laptops are in employee hands. Google has shown their commitment to long-term collaboration as they continuously optimize their products for the unique needs of healthcare providers and go the extra mile in tailoring tools to our staff’s workflows.
For example, on the Chrome OS side, the Google team has helped our registration desks and document centers with device integration for hardware like credit card readers and e-signature pads. They’ve also helped us meet security and privacy requirements mandated by state and federal governments around HIPAA, Medicaid, and Medicare reimbursements. Over this next year, we’ll look at a feature roadmap with Google Cloud to deliver further enhancements, iterating on the product itself to meet our needs for the present and the future.

Delivering the future of healthcare
The benefits we’ve seen around security and usability—and the ability to provide all staff with equal access to Google’s technology—are why we’re expanding our partnership with Google to both the administrative and clinical sides of HMH. In addition to further Google rollouts with corporate, next year we’re distributing Chromebooks to all 350 of our ambulatory clinics.
We’re also working with the Google professional services team to create a custom AI model that analyzes 3D mammogram images. This AI model will enable two providers to read mammograms—which adheres to international best practices but is currently rare in the US—without requiring additional time. Conducting double readings of mammograms will yield better health outcomes for our patients, such as a lower patient recall rate and an increased accuracy in detecting breast cancer.
We’re currently building the model using a variety of Google Cloud products, including Cloud Healthcare API. Once complete, this model is expected to be trained, deployed, and maintained in Google’s Vertex AI, allowing our providers to be more productive as they make clinical decisions with AI support. As the model is proven over time, we plan to make the predictive services accessible to other healthcare organizations.
With Google, we’re able to achieve a unified architecture for storing data as well as training and deploying AI models, which enable our staff to work more efficiently and securely from anywhere. While I may not be able to predict the future as accurately as AI can, I foresee our continued partnership with Google as a key part of HMH’s improved provider and patient outcomes.
Google Cloud and Ericsson to Deliver 5G and Edge Cloud Solutions

3420
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
Editor’s note: 5G is much more than a network—it’s a platform for innovation with the ability to provide immediate global scale and enable use cases that we haven’t even dreamed of yet. But to achieve this, industry players must come together to drive this growing ecosystem. Today, Erik Ekudden, CTO of Ericsson, and Bikash Koley, VP of Global Networking at Google Cloud, share their insights into the potential of edge and 5G.
Experts consider 2021 to be the year that serves as the inflection point between network readiness and 5G availability. However, communications service providers (CSPs) are still faced with the task of modernizing their networks, systems, and infrastructure to maximize the potential of 5G for themselves and for the enterprise customers they serve. As you weigh whether to tackle this challenge, let’s first examine what’s different about 5G and how CSPs can best leverage 5G and the edge together as an even stronger platform for innovation than just 5G alone.
With 5G, applications and mobile networks are no longer isolated
Let’s first address what 5G brings to the table. Faster speeds and lower latency are expected of course, and throughout the evolution from 2G to 3G to 4G, there’s been a step function in performance improvements with each of these. However, with every generation, applications and networks have been like ships passing in the night. Networks have been unaware of what the applications have been doing and applications have been guessing what the network was capable of.
What’s different this time around is that there is some real innovation happening with the development and rollout of 5G itself. The network is on a path to become more accessible via APIs so that applications can call on and consume what they need from the networks in a more programmable way. This sets us up for more open architectures and ecosystems.
5G leverages a more open architecture
One key difference with 5G as compared to prior generations is that it’s the most open and flexible network architecture the CSP industry has seen. This is thanks to its service-based approach and the decoupling of hardware and software components. CSPs are now running core network elements in the public cloud, private cloud, hybrid cloud, and even multi-cloud, which was unthinkable even five years ago.

The flexibility of this more open architecture allows us to push the cloud to the edge while still being able to manage it from a single pane of glass. This is a huge leap forward from traditional networks, which have been domain-specific, managed in silos, and with slow service creation and delivery. Now, hyperscale cloud vendors and network equipment providers are offering solutions that help CSPs break down these silos to enable more flexible, automated networks with improved orchestration, visibility, and control across multi-vendor, multi-cloud and hyperscale cloud-provider environments. In addition, the separation of hardware and software provides a much more flexible and cost-effective way to upgrade from one network generation to the next.
To provide solutions that are relevant to enterprises, CSPs must offer capabilities beyond connectivity. Enterprise service orchestration, including exposure of network assets and network slicing, are foundational capabilities to provide value to the application ecosystem and be in control of the network and the delivered services.
Combining 5G and edge to help industries reimagine user experiences
This convergence of compute, storage and networking at the edge coming together for the first time will enable CSPs and enterprises to offer their customers completely reimagined user experiences. Consider, for example, how the automotive industry might enhance how customers shop for a car. As part of Fiat Chrysler Automobiles’ Virtual Showroom at the recent CES 2021 event, consumers were able to experience the innovative new 2021 Jeep Wrangler 4xe by scanning a QR code with their phones, and then see an Augmented Reality (AR) model of the Wrangler right in front of them—virtually placed on their own driveway or in any open space.
By rendering the model in Google Cloud, then streaming it to mobile devices, visitors could also see what the car looked like from any angle, in different colors, and even step inside to see the interior in incredible detail. That is the true digitization of an industry segment and highlights the device-to-network-to-edge-to-cloud application relationship and how it can impact the user experience.
A programmable network unlocks more application use cases
The programmability of the 5G network will truly enable application developers to utilize all the benefits of the underlying network. Programmability supports ease of use and enables CSPs, integrated software vendors (ISVs) and the ecosystem to have the right network-level APIs exposed so that applications can be optimized based on the network behavior and vice versa. Imagine, for instance, automatically pushing applications from a cloud region to the edge based on network latency and performance metrics.
Finally, 5G’s programmability is also about having the right tools available for developers to build and integrate applications on the network with zero-touch onboarding and validation. With this, we’ve come to the point where the network is now a “platform” for application innovation.
5G and edge will be all about the ecosystem
One thing is for certain: the shift to 5G will place tremendous focus on the ecosystem, and it needs to be an ecosystem that includes CSPs, public cloud providers, application developers and technology providers, all coming together to optimize the user experiences across industry applications. For instance, Google Cloud and Ericsson recently announced our partnership to deliver 5G and edge cloud solutions for CSPs and enterprise. In addition, Google Cloud is also teaming up with popular ISVs to deliver more than 200 edge applications from 30-plus partners, all running on our cloud.
With collaboration across ISVs, cloud providers and network equipment providers, we are enabling the rapid delivery and deployment of new vertical services and applications, leveraging capabilities like Anthos, artificial intelligence (AI), and machine learning (ML), as well as multi-vendor, multi-cloud and hyperscale cloud provider service orchestration and global edge networks such as those provided by Google and telecom service providers.
As members of the technology ecosystem, we talk about compute, storage and networking, but when it comes down to it, it’s about optimally placing these resources—whether in the cloud, at the provider core, at the edge or anywhere in between—to maximize the end-user experience. The openness and programmability of 5G lends itself to collaboration like never before. We predict that in 2022 and beyond, it will be all about the ecosystem coming together to leverage 5G and the edge to build innovations that we can’t yet imagine.
To learn more, watch the full Ericsson 5G Things CTO Focus fireside chat, where we discuss these topics and more.
More Relevant Stories for Your Company
Maximizing the Value of Your Cloud Migration
Technology leaders have shifted many apps and workloads to public cloud, and they are not finished; 63% plan further expansion in the next 12 months. According to this newly resealed Forrester report, enterprises are migrating all types of apps and workloads to cloud and the migration provides firms with a

Efficient, Safe and Dynamic Gaming Experience: Aristocrat’s Digital Journey on Google Cloud
Since Aristocrat’s founding in 1953, technology has constantly transformed gaming and the digital demands on our gaming business are a far cry from challenges we faced when we started. As we continue to expand globally, security and compliance are top priorities. Managing IT security for several gaming subsidiaries and our
State Of Public Cloud Migration 2020
Although most companies have migrated some infrastructure to public cloud, most enterprise workloads still run in on-premises data centers. Technology leaders want to migrate more workloads and infrastructure to public cloud, but they are frustrated by the process of assessing their existing workloads, identifying a migration strategy that aligns IT

How Google Cloud and SAP Address Global Supply Chain Initiatives
With SAP Sapphire kicking off today in Orlando, we’re looking forward to seeing our customers and discussing how they can make core processes more efficient and improve how they serve their customers. One thing is certain to be top of mind – the global supply chain challenges facing the world








