This is part 4 in a multi-part series about the Kubernetes Resource Model. See parts 1, 2, and 3 to learn more. 

Kubernetes clusters can scale. Open-source Kubernetes supports up to 5,000 Nodes, and GKE supports up to 15,000 Nodes. But scaling out a single cluster can only get you so far: if your cluster’s control plane goes down, your entire platform goes down; if the Cloud region running your cluster has a service interruption, so does your app. 

Many organizations choose, instead, to operate multiple Kubernetes clusters. Besides availability, there are lots of reasons to consider multi-cluster, such as allocating a cluster to each development team, splitting workloads between cloud and on-prem, or providing burst capability for traffic spikes. 

But operating a multi-cluster platform comes with its own challenges. How to consistently administer many clusters at once? How to keep the clusters secure? How to deploy and monitor applications running across multiple clusters? How to seamlessly fail over from one region to another?

This post introduces a few tools that can help platform teams more easily administer a multi-cluster Kubernetes environment. 

The platform base layer, with Config Sync 

In the last post, we explored how thoughtful platform abstractions can help reduce toil for app developers- including for a multi-cluster environment, where automation such as CI/CD handles all interactions with the staging and production clusters. But equally important is the platform base layer, the Kubernetes resources and configuration that are shared across services. Your platform base layer might consist of Namespaces, role-based access control, and shared workloads like Prometheus. 

Platform abstractions depend on the existence of these base-layer resources. And so does the security and stability of your platform as a whole. It’s important that these resources not only get deployed, but also stay put. CI/CD is great for deploying resources, but what about making sure resources stay deployed? What if a Kubernetes Namespace gets deleted? Or a Prometheus StatefulSet is modified? 

Kubernetes’ job is to ensure that the cluster’s actual state matches the desired state. But sometimes, the “desired” state isn’t desired at all – it’s a developer who mistakenly modified a resource, or a bad actor that’s gained access into the system. For this reason, a platform base layer needs more than a one-and-done CI/CD pipeline. A tool called Config Sync can help with this. 

Config Sync is a Google Cloud product that can sync Kubernetes resources from a Git repository to one or more GKE or Anthos clusters. Unlike CI/CD tools like Cloud Build, Config Sync watches your clusters constantly, making sure that the intended resource state in the cluster always matches what’s in Git. Config Sync is designed primarily for base-layer resources like namespaces and RBAC. In this way, Config Sync is complementary to, not a replacement for, CI/CD. 

Config Sync runs in a Pod inside your Kubernetes cluster, watching your Git config repo for changes, and also watching the cluster itself for any divergence from your desired state in Git. If any configuration drift is detected from what’s stored in Git, Config Sync will update the API Server accordingly. 

You can point multiple Config Sync deployments at the same Git repo, allowing you to manage the base-layer platform resources for multiple clusters using the same source of truth. And by using Git as the landing zone for config, you can benefit from some of the GitOps principles we discussed in part 2, including the ability to audit and roll back configuration changes.

Let’s walk through an example of how to manage base-layer resources with Config Sync. 

The Cymbal Bank platform consists of four GKE clusters: admin, dev, staging, and prod. We can install Config Sync on all four clusters using the gcloud tool or the Google Cloud Console, pointing all four clusters at a single Git repository, called cymbalbank-policy. Note that this repo is separate from the application source and config repos, and is managed by the platform team. From the Console, we can see that all four clusters are synced to the same commit of the cymbalbank-policy repo. 

Now, let’s say that the Cymbal Bank platform team wants to limit the amount of CPU and memory resources each application team can request for their service. Kubernetes ResourceQuotas help impose these limits, and prevent unexpected Pod evictions.

The platform team can define a set of ResourceQuotas for each application namespace. They can also scope the resources to only be applied to a subset of clusters – for instance, to the production cluster only. (If no cluster name selector is specified, Config Sync will deploy the resource to all clusters by default.)

  apiVersion: v1
kind: ResourceQuota
metadata:
  name: production-quota
  namespace: frontend
  annotations:
    configsync.gke.io/cluster-name-selector: cymbal-prod
spec:
  hard:
    cpu: 700m
    memory: 512Mi

From here, the platform team can commit the resources to the cymbalbank-policy repo, and Config Sync, always watching the policy repo, will deploy the resources to the production cluster:

  NAMESPACE                      NAME                  AGE     REQUEST                                                                                                                               LIMIT
balancereader                  production-quota      6m56s   cpu: 300m/700m, memory: 612Mi/512Mi

If a developer tries to delete one of the ResourceQuotas, Config Sync will block the request, helping to ensure that these base-layer resources stay put.  

  error: You must be logged in to the server (admission webhook "v1.admission-webhook.configsync.gke.io" denied the request: requester is not authorized to delete managed resources)

In this way, Config Sync can help platform teams ensure the stability of that platform base-layer, as well as ensure resource consistency across multiple clusters at once. This, in turn, can help organizations mitigate the complexity of adding new clusters to their environment.  

Enforce policies on Kubernetes resources

Config Sync is a powerful tool on its own, and can work with any Kubernetes resource that your cluster recognizes. This includes Custom Resource Definitions (CRDs) installed with add-ons like Anthos Service Mesh. 

But Config Sync, by default, doesn’t have an idea of “good or bad” Kubernetes resources. It will deploy whatever resources land in Git, even resources that might pose a security risk to your organization. Security is an essential feature of any developer platform, and when it comes to Kubernetes, it’s important to think about security from the initial software design stages, and set up your clusters with security best-practices in mind.   

But it’s just as important to think about security at deploy-time. Who and what can access your clusters? What kinds of Kubernetes resources – and fields within those resources- are allowed? These decisions will depend on lots of factors, including the kinds of data your application deals with, and any industry-specific regulations. 

One common security use case for KRM is the need to monitor incoming Kubernetes resources, whether they’re coming in through kubectl, CI/CD, or Config Sync. But if you have multiple clusters, your Kubernetes environment has multiple API Servers, and therefore multiple entry points.  

A Google tool called Policy Controller can help automate resource monitoring across multiple clusters. Policy Controller is a Kubernetes admission controller that can accept or reject incoming resources based on custom policies you define. Policy Controller is based on the OpenPolicyAgent Gatekeeper project, and it allows you to define policies, or “Constraints,” as KRM. This means you can deploy them using Config Sync, via Git. Once deployed, Policy Controller uses your Constraints as a set of rules to evaluate all incoming KRM, rejecting resources that fall out of compliance.  

Let’s walk through an example. Say that the Cymbal Bank security team wants to ensure that no code in development is accessible to the public. Kubernetes Services of type LoadBalancer expose public IP addresses by default, so the platform team wants to define a PolicyController constraint that blocks Services of that type on the development GKE cluster.

To do this, the platform team can define a Policy Controller Constraint as KRM. This Constraint uses a Constraint Template, provided through the pre-installed Constraint Template library. The ConstraintTemplate defines the logic of the policy itself, and the Constraint makes the template concrete, populating any variables needed to execute the policy logic. Here, we’re also adding a Config Sync cluster name annotation, to scope this resource to apply only to the development cluster.

  apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sNoExternalServices
metadata:
  name: dev-no-ext-services
  annotations:
    configsync.gke.io/cluster-name-selector: cymbal-dev
spec:
  internalCIDRs: []

The platform team can then commit the resource to the cymbalbank-policy repo, and Config Sync will deploy the resource to the development cluster. 

From here, if an app developer tries to create an externally-accessible Kubernetes Service, Policy Controller will block the resource from being created.

  for: "constraint-ext-services/contacts-svc-lb.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [denied by dev-no-ext-services] Creating services of type `LoadBalancer` without Internal annotation is not allowed

The platform team can define as many of these Constraints as they want, each defining a separate policy.

Writing custom policies 

The Policy Controller Constraint Template library provides a lot of functionality, from blocking privileged containers, to requiring certain resource labels, to preventing app teams from deploying into certain namespaces. But if you want to enforce custom logic on your organization’s KRM, you can do so by writing a custom Constraint Template. 

Constraint Templates are written in a query language called Rego. Rego was designed for policy rule evaluation, and it can introspect Kubernetes resource fields to make a conclusion as to whether the resource is allowed or not. 

For instance, let’s say that the platform team wants to limit the number of containers allowed inside a single application Pod. Too many containers per Pod can cause outage risks— when one container crashes, the entire Pod crashes.

To enforce this policy, the platform team can define a Constraint Template, using the Rego language, that looks inside a resource to ensure that the number of containers per Pod is within the allowed limit: 

  apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8slimitcontainersperpod
spec:
  crd:
    spec:
      names:
        kind: K8sLimitContainersPerPod
      validation:
        openAPIV3Schema:
          properties:
            allowedNumContainers:
              type: integer
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8slimitcontainersperpod
        numTemplateContainers := count(input.review.object.spec.template.spec.containers)
        numRunningContainers := count(input.review.object.spec.containers)
        containerLimit := input.parameters.allowedNumContainers
        template_containers_over_limit = true {
          numTemplateContainers > containerLimit
        }
        running_containers_over_limit = true {
          numRunningContainers > containerLimit
        }
        violation[{"msg": msg}] {
          template_containers_over_limit
          msg := sprintf("Number of containers in template (%v) exceeds the allowed limit (%v)", [numTemplateContainers, containerLimit])
        }
        violation[{"msg": msg}] {
          running_containers_over_limit
          msg := sprintf("Number of running containers (%v) exceeds the allowed limit (%v)", [numRunningContainers, containerLimit])
        }
Then, the platform team can define a concrete Constraint, using this Constraint Template, to set the number of allowed containers per Pod to 3: 
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sLimitContainersPerPod
metadata:
  name: limit-three-containers
spec:
  parameters:
    allowedNumContainers: 3

Finally, the platform team can push these resources to the cymbalbank-policy repo, and Config Sync will deploy the policy to all four clusters. If a developer tries to define a Kubernetes Deployment containing more containers per pod than what’s allowed, the resource will be blocked at deploy time: 

  Error from server ([limit-three-containers] Number of containers in template (4) exceeds the allowed limit (3)): error when creating "constraint-limit-containers/test-workload.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [limit-three-containers] Number of containers in template (4) exceeds the allowed limit (3)

Custom Constraint Templates can give platform teams lots of flexibility in the types of policies they define and enforce in a Kubernetes environment. 

Integrating policy checks into CI/CD 

As we explored earlier, Config Sync and CI/CD are complementary tools. Config Sync works great for base-layer platform resources and policies, whereas CI/CD works well for application tests and deployment. 

But one pitfall of having two separate KRM deployment mechanisms is that app developers may not know that their resources are out of policy until they try to deploy them into production. This is especially true if some policies are scoped only to production, as we saw with the ResourceQuota example. Ideally, the platform team has a way to empower developers and code reviewers to know ahead of time whether new or modified resources are still in compliance. We can enable this use case by integrating policy checks into the existing Cymbal Bank CI/CD.

Policy Controller operates, by default, as a Kubernetes Admission Controller running inside the cluster. But Policy Controller also provides a “standalone” mode, running inside a container, that can be used outside of a cluster, such as from inside a Cloud Build pipeline. 

In the example below, Cloud Build executes Policy Controller checks by getting the cymbalbank-app-config manifests, cloning the cymbalbank-policy resources, and using the “policy-controller-validate” container image to evaluate the app manifests against the policies. 

  steps:
- id: 'Render prod manifests'
  name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
  entrypoint: '/bin/sh'
  args: ['-c', 'mkdir hydrated-manifests && kubectl kustomize overlays/prod > hydrated-manifests/prod.yaml']
- id: 'Clone cymbalbank-policy repo'
  name: 'gcr.io/kpt-dev/kpt'
  entrypoint: '/bin/sh'
  args: ['-c', 'kpt pkg get https://github.com/$$GITHUB_USERNAME/cymbalbank-policy.git@main constraints
                  && kpt fn source constraints/ hydrated-manifests/ > hydrated-manifests/kpt-manifests.yaml']
  secretEnv: ['GITHUB_USERNAME']
- id: 'Validate prod manifests against policies'
  name: 'gcr.io/config-management-release/policy-controller-validate'
  args: ['--input', 'hydrated-manifests/kpt-manifests.yaml']
availableSecrets:
  secretManager:
  - versionName: projects/${PROJECT_ID}/secrets/github-username/versions/1 
    env: 'GITHUB_USERNAME'
timeout: '1200s' #timeout - 20 minutes

From here, an app developer or operator can know if their resources violate org-wide policies, by looking at the Cloud Build output for their Pull Request:  

  Status: Downloaded newer image for gcr.io/config-management-release/policy-controller-validate:latest
Error: Found 1 violations:
[1] Number of containers in template (4) exceeds the allowed limit (3)

By integrating policy checks into CI/CD, app development teams can understand whether their resources are in compliance, and platform teams add an additional layer of policy checks to the platform. 

Overall, Config Sync and Policy Controller can provide a powerful toolchain for standardizing base-layer config across a multi-cluster environment. Check out the Part 4 demo to try out each of these examples. 

And stay tuned for Part 5, where we’ll learn how to use KRM to manage cloud-hosted resources. 

Editor’s note: We invited partners from across our retail ecosystem to share stories, best practices, and tips and tricks on how they are helping retailers transform during a time that continues to see tremendous change. This original blog post was published by OnlineSales.ai. Please enjoy this updated entry from our partner.

Tata’s online healthcare entity, Tata 1mg, aims to revolutionize how a consumer finds the right healthcare professional for their needs. The platform offers e-pharmacy services, diagnostics, and health content and has more than 200,000 active brands in its rapidly expanding list of suppliers, sellers and manufacturers. Several of these brands would need regular involvement from the marketplace team in order to set up and run ad campaigns. As one of the largest platforms, Tata 1mg approached OnlineSales.ai to address their growing needs for an efficient Retail Media Monetisation suite to help scale up their existing monetisation efforts. 

Tata 1mg needed more data-driven insights and customizability for their monetisation efforts

Tata 1mg saw several challenges that they needed to address.

1. Manual effort: Tata 1mg marketplace team’s monetization process was largely manual and required heavy time and resource investment to activate advertisers, which subsequently ate into their overall ad revenues.
2. Non-scalable framework: Given the fast-growing number of advertisers on their marketplace – well into the hundreds of thousands – it became increasingly clear that their existing monetization framework was not scalable.
3. Inadequate reporting & analytics: The existing framework facilitated only simple reporting, and brands were not able to draw the deep insights they needed to make data-led decisions and refine their advertising ROIs.
4. Steep learning curve: Several brands lacked dedicated advertising experts who could bring the experience and skills needed to plan successful ad campaigns and optimize budgets. The involved learning curve led to increased advertiser-churn.
5. Lack of personalized offerings: The manual nature of Tata 1mg’s monetization framework left little room for customizability, and account managers could offer very little in terms of tailored ad-buying experiences to different types of advertisers.

All of these issues led to regular revenue leakage. Tata 1mg realized the need for a central platform that addressed the above-mentioned issues and would also be able to cater to new ad channels and the requirements thereof. Developing such a solution in house was evaluated, but was found to be expensive and would require a lot of time to build.

OnlineSales.ai’s AI/ML-powered solution on Google Cloud offered the automation and flexibility to support modern omnichannel advertising needs

Therefore, Tata 1mg looked to OnlineSales.ai Monetize, an AI/ML-powered retail media monetisation suite on Google Cloud, to help address their needs for a cloud-based monetisation platform that offered the flexibility to support omnichannel advertising models, had automation built in to reduce manual tasks, provided detailed intelligence and analytics, and delivered quick, reliable scalability.

1. White labeled self-serve platform: OnlineSales.ai’s retail media platform was implemented within a short 4 weeks. After thorough testing and implementation, the platform was successfully launched, and enabled sellers to run ads on a fully self-serve platform. This required zero involvement from the Tata team.
2. Unified omni channel buying: The team at OnlineSales.ai worked closely with the platform’s core teams to activate and mobilize various advertising products through a self-serve platform, while simultaneously addressing any unique requirements they had.
3. Omnichannel analytics with AI-led suggestions: Sellers are now also able to generate detailed reports that provide them critical insights into campaigns, and help them make better use of their budgets – enhancing their experience and interest in advertising on the platform.
4. Personalized buying experiences: The tailored UX offered by OnlineSales.ai’s Monetize helped Tata engage brands of all levels in technical aptitude to use the platform easily. This also allowed admins to configure what types of inventories were available to different brands or advertiser segments.

OnlineSales.ai makes use of Google Cloud Dataflow to facilitate computations on real-time streaming. Microservices are deployed on the Google Kubernetes Engine (GKE) platform due to the solution’s well-known ability to handle scale.

In addition, Dataproc is used by the company to run batch processing apps while Cloud Load Balancing helps manage traffic across different regions; all at scale. Onlinesales.ai also uses Memorystore as a database for their ad servers in order to have very low latency, and deploys BigQuery to run analytical applications and facilitate powerful reporting. The bulk of their office applications are deployed on different VMs on Compute Engine, and the company also makes use of Cloud Storage for data storage and transfers between applications.

Learn more about OnlineSales.ai available on the Google Cloud Marketplace.

^{About OnlineSales.ai}

^{OnlineSales.ai, offers a fully white labeled retail media platform which helps retailers unlock additional revenue by activating advertising real estate on their platform. With its AI/ML-powered solution, OnlineSales.ai allows retailers to turn brands of all sizes into advertisers – at scale. Its self-serve platform simplifies the ad buying process, and enhances outcomes with smart automation, boosting ad retailers’ ad revenues by multiples.}

The pandemic continues to deeply affect our lives around the globe. In some places, new cases are surging and returning to work is the last thing on people’s minds. In other areas, conditions are improving and companies are starting to think about transitioning their workforce back to the office. 

Exactly when and how to do this remains complex and varies by country, industry, and company.  What’s certain is that hybrid work will become an essential part of the business world moving forward. And finding solutions that bridge the gap between “in-person” and “somewhere else” are crucial.  

At Google, we’ve been focused on what the hybrid transformation means for us. To prepare for hybrid work, using modern solutions is a key enabler.

Chrome OS: Supporting Google’s return to office strategy 

At Google, we’ll move to a hybrid work week where most Googlers spend approximately three days in the office and two days wherever they work best. It’s no surprise that as the modern, secure, cloud-first platform, Chrome OS is playing a key role in our transition to a hybrid work model. Because Chrome OS devices can easily be shared, more flexible working models and spaces are now possible. And with user profiles stored in the cloud and collaboration solutions like Google Workspace, employees can log in to any Chrome OS device, access what they need and pick up where they left off.

Here are just a few ways we are using Chrome OS and its tools to support the return to the office:

• In select office locations, Googlers can reserve desks through an internal booking tool set up with a high-performance Chromebox, keyboard, mouse, and monitor. Employees can log in to the Chromebox which syncs their cloud profile, and start working with the same environment they have on all their Chrome OS devices.
• We’re announcing new docking stations that are designed for Chrome OS devices and allow employees to bring in their Chromebook from home, connect to the dock with one USB-C cable, and use a monitor, keyboard, and mouse for a full desktop experience. 
• Every Chrome OS device enables a zero-trust security working model with BeyondCorp Enterprise providing our workforce with simple and secure access to applications while providing additional security controls for IT.
• We’ve deployed the new Chrome OS Readiness Tool to our extended workforce to identify employees that are able to switch to Chrome OS. This allows us to expand the latest security, deployment, and manageability benefits to more of the workforce.

Additional ways Chrome OS is helping organizations with return to office

We aren’t the only ones supporting our return to office and hybrid work strategy with Chrome OS. We’ve heard more ways our customers are using Chrome OS to make the transition as smooth as possible. These include:

• Streamlining deployment and management of Chrome OS devices using zero-touch enrollment which allows devices to automatically enroll into a corporate domain without IT configuration.
  Grab & Go Chromebooks being used for frontline and hybrid information workers, allowing employees to grab a Chromebook from a cart and get to work right away.
• Parallels Desktop for Chrome OS being deployed to allow employees to access Windows or legacy apps locally on their Chrome OS devices.
• Existing Windows and Mac devices being modernized and repurposed to run a Chrome OS experience using CloudReady. (Google is currently offering a CloudReady promotion. Learn more here.)

While the Chrome OS team has been working towards making remote working as seamless as possible for IT, we’ve also made advancements in supporting traditional technology that’s required in the office.

• In October, we announced Chrome Enterprise Recommended: a collection of identity, printing, productivity, communications, and virtualization solutions that are verified to run great on Chrome OS.
• With the increased usage of video conferencing on Chrome OS devices, we’ve made improvements to Google Meet and Zoom performance including camera and video improvements to reduce any unnecessary processing and features that intelligently adapt to your device, your network, and what you are working on.
• For improved access to Windows and legacy apps, VMware Horizon introduced multi-monitor support and USB redirection and Citrix Workspace released a tech preview with webcam enhancements and Microsoft Teams optimizations. 
• We integrated with the Okta Workflows platform, so IT administrators can include Chrome OS in it’s access logic and deploy quickly without code. Recently, we’ve added the ability to require users to reauthenticate on their Chrome OS device once Okta has detected that the user has changed their password. Try out this new capability by signing up for our Trusted Tester program.
• We’ve increased our support for direct IP printing with additional printer models since the beginning of 2020. In addition, we have made improvements to management features, including support for multiple print servers and launched policy APIs to provide a better IT admin experience. 

Join us for a digital event with Modern Computing Alliance and its newest member HP 

Last year we announced the launch of the Modern Computing Alliance—a collaboration of industry leaders including Box, Chrome Enterprise, Citrix, Dell, Google Workspace, Imprivata, Intel, Okta, RingCentral, Slack, VMware, and Zoom aim to create the pioneering solutions that businesses need. We are thrilled to introduce HP, who brings their innovative hardware and enterprise hardware expertise to the Modern Computing Alliance and has been working closely with the alliance to ensure true silicon-to-cloud innovation.

As an alliance, we’ve been deeply engaged in the hybrid work shift. We invite you to join us for a digital event where we discuss questions about returning to work. Like how product design can encourage participation and collaboration regardless of where employees are, important security issues to keep in mind, and what factors businesses should consider to support a safe, working environment.

Home. Heading back. Hybrid.
Hear from the experts on hybrid work and return to office.
Date: May 20th, 2021

Register here. 

If you are ready to try Chrome OS today, it’s easy to get started. You can contact us to get connected to a partner, sign up for a free 30-day trial of Chrome Enterprise Upgrade to start managing Chrome OS devices, or deploy the Chrome OS Readiness Tool to identify employees that are ready to switch to Chrome OS.POSTED IN:

• CHROME ENTERPRISE
• PRODUCTIVITY & COLLABORATION
• RETURN TO OFFICE