This is part 4 in a multi-part series about the Kubernetes Resource Model. See parts 1, 2, and 3 to learn more. 

Kubernetes clusters can scale. Open-source Kubernetes supports up to 5,000 Nodes, and GKE supports up to 15,000 Nodes. But scaling out a single cluster can only get you so far: if your cluster’s control plane goes down, your entire platform goes down; if the Cloud region running your cluster has a service interruption, so does your app. 

Many organizations choose, instead, to operate multiple Kubernetes clusters. Besides availability, there are lots of reasons to consider multi-cluster, such as allocating a cluster to each development team, splitting workloads between cloud and on-prem, or providing burst capability for traffic spikes. 

But operating a multi-cluster platform comes with its own challenges. How to consistently administer many clusters at once? How to keep the clusters secure? How to deploy and monitor applications running across multiple clusters? How to seamlessly fail over from one region to another?

This post introduces a few tools that can help platform teams more easily administer a multi-cluster Kubernetes environment. 

The platform base layer, with Config Sync 

In the last post, we explored how thoughtful platform abstractions can help reduce toil for app developers- including for a multi-cluster environment, where automation such as CI/CD handles all interactions with the staging and production clusters. But equally important is the platform base layer, the Kubernetes resources and configuration that are shared across services. Your platform base layer might consist of Namespaces, role-based access control, and shared workloads like Prometheus. 

Platform abstractions depend on the existence of these base-layer resources. And so does the security and stability of your platform as a whole. It’s important that these resources not only get deployed, but also stay put. CI/CD is great for deploying resources, but what about making sure resources stay deployed? What if a Kubernetes Namespace gets deleted? Or a Prometheus StatefulSet is modified? 

Kubernetes’ job is to ensure that the cluster’s actual state matches the desired state. But sometimes, the “desired” state isn’t desired at all – it’s a developer who mistakenly modified a resource, or a bad actor that’s gained access into the system. For this reason, a platform base layer needs more than a one-and-done CI/CD pipeline. A tool called Config Sync can help with this. 

Config Sync is a Google Cloud product that can sync Kubernetes resources from a Git repository to one or more GKE or Anthos clusters. Unlike CI/CD tools like Cloud Build, Config Sync watches your clusters constantly, making sure that the intended resource state in the cluster always matches what’s in Git. Config Sync is designed primarily for base-layer resources like namespaces and RBAC. In this way, Config Sync is complementary to, not a replacement for, CI/CD. 

Config Sync runs in a Pod inside your Kubernetes cluster, watching your Git config repo for changes, and also watching the cluster itself for any divergence from your desired state in Git. If any configuration drift is detected from what’s stored in Git, Config Sync will update the API Server accordingly. 

You can point multiple Config Sync deployments at the same Git repo, allowing you to manage the base-layer platform resources for multiple clusters using the same source of truth. And by using Git as the landing zone for config, you can benefit from some of the GitOps principles we discussed in part 2, including the ability to audit and roll back configuration changes.

Let’s walk through an example of how to manage base-layer resources with Config Sync. 

The Cymbal Bank platform consists of four GKE clusters: admin, dev, staging, and prod. We can install Config Sync on all four clusters using the gcloud tool or the Google Cloud Console, pointing all four clusters at a single Git repository, called cymbalbank-policy. Note that this repo is separate from the application source and config repos, and is managed by the platform team. From the Console, we can see that all four clusters are synced to the same commit of the cymbalbank-policy repo. 

Now, let’s say that the Cymbal Bank platform team wants to limit the amount of CPU and memory resources each application team can request for their service. Kubernetes ResourceQuotas help impose these limits, and prevent unexpected Pod evictions.

The platform team can define a set of ResourceQuotas for each application namespace. They can also scope the resources to only be applied to a subset of clusters – for instance, to the production cluster only. (If no cluster name selector is specified, Config Sync will deploy the resource to all clusters by default.)

  apiVersion: v1
kind: ResourceQuota
metadata:
  name: production-quota
  namespace: frontend
  annotations:
    configsync.gke.io/cluster-name-selector: cymbal-prod
spec:
  hard:
    cpu: 700m
    memory: 512Mi

From here, the platform team can commit the resources to the cymbalbank-policy repo, and Config Sync, always watching the policy repo, will deploy the resources to the production cluster:

  NAMESPACE                      NAME                  AGE     REQUEST                                                                                                                               LIMIT
balancereader                  production-quota      6m56s   cpu: 300m/700m, memory: 612Mi/512Mi

If a developer tries to delete one of the ResourceQuotas, Config Sync will block the request, helping to ensure that these base-layer resources stay put.  

  error: You must be logged in to the server (admission webhook "v1.admission-webhook.configsync.gke.io" denied the request: requester is not authorized to delete managed resources)

In this way, Config Sync can help platform teams ensure the stability of that platform base-layer, as well as ensure resource consistency across multiple clusters at once. This, in turn, can help organizations mitigate the complexity of adding new clusters to their environment.  

Enforce policies on Kubernetes resources

Config Sync is a powerful tool on its own, and can work with any Kubernetes resource that your cluster recognizes. This includes Custom Resource Definitions (CRDs) installed with add-ons like Anthos Service Mesh. 

But Config Sync, by default, doesn’t have an idea of “good or bad” Kubernetes resources. It will deploy whatever resources land in Git, even resources that might pose a security risk to your organization. Security is an essential feature of any developer platform, and when it comes to Kubernetes, it’s important to think about security from the initial software design stages, and set up your clusters with security best-practices in mind.   

But it’s just as important to think about security at deploy-time. Who and what can access your clusters? What kinds of Kubernetes resources – and fields within those resources- are allowed? These decisions will depend on lots of factors, including the kinds of data your application deals with, and any industry-specific regulations. 

One common security use case for KRM is the need to monitor incoming Kubernetes resources, whether they’re coming in through kubectl, CI/CD, or Config Sync. But if you have multiple clusters, your Kubernetes environment has multiple API Servers, and therefore multiple entry points.  

A Google tool called Policy Controller can help automate resource monitoring across multiple clusters. Policy Controller is a Kubernetes admission controller that can accept or reject incoming resources based on custom policies you define. Policy Controller is based on the OpenPolicyAgent Gatekeeper project, and it allows you to define policies, or “Constraints,” as KRM. This means you can deploy them using Config Sync, via Git. Once deployed, Policy Controller uses your Constraints as a set of rules to evaluate all incoming KRM, rejecting resources that fall out of compliance.  

Let’s walk through an example. Say that the Cymbal Bank security team wants to ensure that no code in development is accessible to the public. Kubernetes Services of type LoadBalancer expose public IP addresses by default, so the platform team wants to define a PolicyController constraint that blocks Services of that type on the development GKE cluster.

To do this, the platform team can define a Policy Controller Constraint as KRM. This Constraint uses a Constraint Template, provided through the pre-installed Constraint Template library. The ConstraintTemplate defines the logic of the policy itself, and the Constraint makes the template concrete, populating any variables needed to execute the policy logic. Here, we’re also adding a Config Sync cluster name annotation, to scope this resource to apply only to the development cluster.

  apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sNoExternalServices
metadata:
  name: dev-no-ext-services
  annotations:
    configsync.gke.io/cluster-name-selector: cymbal-dev
spec:
  internalCIDRs: []

The platform team can then commit the resource to the cymbalbank-policy repo, and Config Sync will deploy the resource to the development cluster. 

From here, if an app developer tries to create an externally-accessible Kubernetes Service, Policy Controller will block the resource from being created.

  for: "constraint-ext-services/contacts-svc-lb.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [denied by dev-no-ext-services] Creating services of type `LoadBalancer` without Internal annotation is not allowed

The platform team can define as many of these Constraints as they want, each defining a separate policy.

Writing custom policies 

The Policy Controller Constraint Template library provides a lot of functionality, from blocking privileged containers, to requiring certain resource labels, to preventing app teams from deploying into certain namespaces. But if you want to enforce custom logic on your organization’s KRM, you can do so by writing a custom Constraint Template. 

Constraint Templates are written in a query language called Rego. Rego was designed for policy rule evaluation, and it can introspect Kubernetes resource fields to make a conclusion as to whether the resource is allowed or not. 

For instance, let’s say that the platform team wants to limit the number of containers allowed inside a single application Pod. Too many containers per Pod can cause outage risks— when one container crashes, the entire Pod crashes.

To enforce this policy, the platform team can define a Constraint Template, using the Rego language, that looks inside a resource to ensure that the number of containers per Pod is within the allowed limit: 

  apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8slimitcontainersperpod
spec:
  crd:
    spec:
      names:
        kind: K8sLimitContainersPerPod
      validation:
        openAPIV3Schema:
          properties:
            allowedNumContainers:
              type: integer
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8slimitcontainersperpod
        numTemplateContainers := count(input.review.object.spec.template.spec.containers)
        numRunningContainers := count(input.review.object.spec.containers)
        containerLimit := input.parameters.allowedNumContainers
        template_containers_over_limit = true {
          numTemplateContainers > containerLimit
        }
        running_containers_over_limit = true {
          numRunningContainers > containerLimit
        }
        violation[{"msg": msg}] {
          template_containers_over_limit
          msg := sprintf("Number of containers in template (%v) exceeds the allowed limit (%v)", [numTemplateContainers, containerLimit])
        }
        violation[{"msg": msg}] {
          running_containers_over_limit
          msg := sprintf("Number of running containers (%v) exceeds the allowed limit (%v)", [numRunningContainers, containerLimit])
        }
Then, the platform team can define a concrete Constraint, using this Constraint Template, to set the number of allowed containers per Pod to 3: 
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sLimitContainersPerPod
metadata:
  name: limit-three-containers
spec:
  parameters:
    allowedNumContainers: 3

Finally, the platform team can push these resources to the cymbalbank-policy repo, and Config Sync will deploy the policy to all four clusters. If a developer tries to define a Kubernetes Deployment containing more containers per pod than what’s allowed, the resource will be blocked at deploy time: 

  Error from server ([limit-three-containers] Number of containers in template (4) exceeds the allowed limit (3)): error when creating "constraint-limit-containers/test-workload.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [limit-three-containers] Number of containers in template (4) exceeds the allowed limit (3)

Custom Constraint Templates can give platform teams lots of flexibility in the types of policies they define and enforce in a Kubernetes environment. 

Integrating policy checks into CI/CD 

As we explored earlier, Config Sync and CI/CD are complementary tools. Config Sync works great for base-layer platform resources and policies, whereas CI/CD works well for application tests and deployment. 

But one pitfall of having two separate KRM deployment mechanisms is that app developers may not know that their resources are out of policy until they try to deploy them into production. This is especially true if some policies are scoped only to production, as we saw with the ResourceQuota example. Ideally, the platform team has a way to empower developers and code reviewers to know ahead of time whether new or modified resources are still in compliance. We can enable this use case by integrating policy checks into the existing Cymbal Bank CI/CD.

Policy Controller operates, by default, as a Kubernetes Admission Controller running inside the cluster. But Policy Controller also provides a “standalone” mode, running inside a container, that can be used outside of a cluster, such as from inside a Cloud Build pipeline. 

In the example below, Cloud Build executes Policy Controller checks by getting the cymbalbank-app-config manifests, cloning the cymbalbank-policy resources, and using the “policy-controller-validate” container image to evaluate the app manifests against the policies. 

  steps:
- id: 'Render prod manifests'
  name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
  entrypoint: '/bin/sh'
  args: ['-c', 'mkdir hydrated-manifests && kubectl kustomize overlays/prod > hydrated-manifests/prod.yaml']
- id: 'Clone cymbalbank-policy repo'
  name: 'gcr.io/kpt-dev/kpt'
  entrypoint: '/bin/sh'
  args: ['-c', 'kpt pkg get https://github.com/$$GITHUB_USERNAME/cymbalbank-policy.git@main constraints
                  && kpt fn source constraints/ hydrated-manifests/ > hydrated-manifests/kpt-manifests.yaml']
  secretEnv: ['GITHUB_USERNAME']
- id: 'Validate prod manifests against policies'
  name: 'gcr.io/config-management-release/policy-controller-validate'
  args: ['--input', 'hydrated-manifests/kpt-manifests.yaml']
availableSecrets:
  secretManager:
  - versionName: projects/${PROJECT_ID}/secrets/github-username/versions/1 
    env: 'GITHUB_USERNAME'
timeout: '1200s' #timeout - 20 minutes

From here, an app developer or operator can know if their resources violate org-wide policies, by looking at the Cloud Build output for their Pull Request:  

  Status: Downloaded newer image for gcr.io/config-management-release/policy-controller-validate:latest
Error: Found 1 violations:
[1] Number of containers in template (4) exceeds the allowed limit (3)

By integrating policy checks into CI/CD, app development teams can understand whether their resources are in compliance, and platform teams add an additional layer of policy checks to the platform. 

Overall, Config Sync and Policy Controller can provide a powerful toolchain for standardizing base-layer config across a multi-cluster environment. Check out the Part 4 demo to try out each of these examples. 

And stay tuned for Part 5, where we’ll learn how to use KRM to manage cloud-hosted resources. 

Today, we’re announcing that we’re working with AT&T to establish Messages by Google as the default messaging application for all AT&T customers in the United States using Android phones. The collaboration aims to help accelerate the industry toward global Rich Communication Services (RCS) coverage and interoperability to offer a consistent, secure, and enhanced messaging experience for all Android users around the world.

“Many AT&T customers have enjoyed the advantages of RCS for years when texting with friends and family,” said David Christopher, executive vice president and general manager – AT&T Mobility. “We look forward to working closely with Google to extend these benefits to even more of our customers as they enjoy richer conversations with others around the world.”

Working together, AT&T and Google will continue the momentum to upgrade SMS with enhanced messaging features offered in Messages, which includes the support of chat features based on the open RCS standard. With Messages as the default messaging application, all AT&T customers using Android devices will get enhanced features so they can:

• Share full-resolution pictures from a recent event or vacation 
• Send a higher-quality video of that soccer goal and the celebration that followed
• Know when someone is replying to a text
• Send and receive messages over Wi-Fi or data
• Participate in group chats where it’s easy to add someone else to the conversation, or let someone leave, without starting a brand new thread

In addition to these features, we’re also rolling out end-to-end encryption for one-on-one RCS conversations between people using Messages and people who have chat features enabled.

For years, we’ve been working with the mobile industry and device makers to bring enhanced and secure messaging to everyone on Android. Today’s announcement—that AT&T customers using Android devices will soon be able to enjoy these features by default—is a major step forward.

SAP Build Process Automation is designed to optimize business processes and boost efficiency. The platform helps both business users and developers alike digitize core workflows and incorporate artificial intelligence (AI) into time consuming and error-prone manual tasks.

All digital paths can benefit from automation. The pandemic, supply chain shortages, and other disruptive events have upped the pressure on businesses and their workers to perform in more efficient and flexible ways.

Google Cloud and SAP have responded by providing an integrated toolbox that can fundamentally change the way businesses operate — all while creating value for their customers.

With a focus on taking process automation to an even more advanced level and removing inefficiencies from workflows, SAP has introduced integrations with Google Cloud Document AI, and Google Workspace for SAP Build Process Automation customers. This integration can reduce or eliminate many repetitive and error-prone tasks, so that companies can help save money, operate more productively, and scale more easily.

AI unleashes innovation

Continuous advances in digital systems and advanced technology introduce new opportunities to rethink workflows. SAP recognizes the role that AI-powered automation can play in transforming workflows, and that the benefits of doing so extend beyond basic time savings and cost cutting. By plugging Google Cloud Document AI into the application, SAP Build Process Automation’s low-code, no-code platform enables SAP to help its customers in lines of business and IT integrate multiple applications while democratizing access to governed machine learning technology.

Machine learning and process automation can drive efficiency with SAP S/4HANA

Customers using SAP S/4HANA can build workflow improvements into all major core processes, such as order entry, invoice creation, asset posting, and many others, helping them to be faster and more efficient in the process. Google Cloud Document AI extracts key elements — including addresses, article numbers, price, quantity, and more — within emails, PDFs, handwritten notes, and other formats.

Integrated automation can drive results for invoice and purchase order processing

An example of the improvements these integrations have made to SAP Build Process Automation is the use of AI, productivity tools, and automation for processing purchase orders. In the past, workers had to manually select relevant orders in their Gmail accounts and extract key data from large PDF files, including the order date, supplier details, article numbers, quantity, unit price, and the total amount. Then, workers would have to enter all of the individual line items one by one into the SAP S/4HANA system.

Today, through SAP’s integrated automation platform, customers can automatically extract and organize data by Google Workspace (Google Sheets, Google Drive, and Gmail) and Document AI. This works by extracting data contained in Gmail attachments, downloading it into Google Drive and extracting fields using Document AI’s pretrained models. The tool then enters the consolidated order information from Google Sheets into SAP S/4HANA. For example, the Canton of Zurich in Switzerland experienced a significant reduction of workload to process compensation forms once the organization implemented this automation. Furthermore, implementing the automation can avoid audit and compliance issues, and improve data quality.

The screenshot below shows how a workflow operates within the SAP Build Process Automation software.

Sales, procurement, finance, and other functions are also able to handle more strategic work that delivers greater value to customers with these SAP and Google Cloud integrations. They’ve also boosted both security and regulatory compliance for customers, including those in the financial services space.

For example, the Google Cloud Document AI technology can process thousands of supply chain invoices, validates and systematically approves them. And, over time, the machine learning and AI components improve the analysis process and ensure that data processed with SAP Build Process Automation is adhering to a company’s best practices and essential regulatory requirements.

SAP and Google Cloud put automation to work

Achieving the most accurate, efficient business processes is possible with an automation framework that embeds collaboration and productivity applications while giving lines of business and IT users access to machine learning technology. SAP Build Process Automation combined with Google Cloud Document AI, and Google Workspace has proven to be a catalyst in driving innovation and business transformation for customers across multiple industries, leading to an average of 22% to 30% faster time to market, and significant financial gains, including up to 20% accounts payable savings potential. An example of these improvements includes those experienced by TasNetworks, which had a 25% reduction in back-office processing efforts after implementing these technologies.

To learn more about how Google Cloud and SAP are building solutions for accelerating business value, visit cloud.google.com/solutions/sap. You can find more information about SAP Build Process Automation at sap.com/build-automation.

To start your transformation journey today, choose the SAP Business Technology Platform region that’s best for you. We’re also happy to announce that Google Cloud offers the first and only option to run BTP in the cloud in India — learn more here: Google Cloud’s newest SAP Business Technology Platform Region.

Many organizations, including Google, have moved quickly to embrace working from home. With this widespread remote work, it’s never been more important for IT admins to be certain that every device in their organization is secure, even when that device isn’t company-owned. 

We pioneered zero-trust security through our BeyondCorp strategy and leverage it to offer advanced security for G Suite* users to protect secure access for all devices. Admins can enforce these controls across G Suite and other corporate applications and data, ensuring consistent security and user experience across your organization.

Today, we’re laying out six key controls that IT admins can use within G Suite to help keep their organizations safe when using the bring your own device approach (BYOD). You can also review our detailed security checklist here, and learn more from our course on Managing G Suite here. 

Secure mobile and desktop devices with endpoint management

BYOD devices can differ widely across an organization, with a range of OS versions, hardware modes, patch versions, and more, so it’s impossible to rely on a one-size-fits-all approach to device management. With Google endpoint management, IT admins can easily support a variety of mobile and desktop devices by enforcing measures like minimum software versions and blocking jailbroken or rooted devices, in many cases without requiring full device rights for employee privacy. 

When it comes to managing mobile devices, G Suite offers basic and advanced mobile device management:

• With basic mobile device management, BYOD devices are secured with baseline security features with no end user friction. Admins can enforce a passcode, get a device inventory, wipe Google accounts remotely, and even remotely install applications on Android devices. 
• With advanced mobile device management, admins can apply more policy controls over BYOD devices, and Android users can keep their personal data private and separate from their work data with Android Work Profiles. You can also allow and manage work apps on iOS and Android devices.

Admins can also manage and secure desktop devices with fundamental device management and enhanced desktop security for Windows. With fundamental device management, when a user logs into G Suite through any browser on a Windows, Mac, Chrome, or Linux device, that device will be automatically enrolled with endpoint management. This provides a base level of security to every desktop device that accesses G Suite data. With enhanced desktop security for Windows, admins can easily manage and secure Windows 10 devices through the admin console.

Enable secure connections without a corporate VPN using context-aware access

Context-aware access offers protection from unwanted access to G Suite services without the need for a VPN, and allows admins to set up different access levels based on a user’s identity and the context of the request, taking into account factors such as the country, device security status, and IP address of the request. For example, you can require BYOD devices accessing G Suite to meet encryption and password requirements, or restrict contractors from accessing G Suite from company managed Chromebooks.

Control data access with app access control

It’s important to protect all devices in your organization—corporate or BYOD—from malicious apps trying to gain access to corporate data. Using app access control, admins can take steps to prevent these apps from tricking users into mistakenly granting access to corporate data. With this feature, admins can choose which third-party apps are allowed to access users’ G Suite data by explicitly trusting, limiting, or blocking access for apps.

Enforce 2-Step Verification

With 2-Step Verification, admins can reduce the risk of unauthorized access by asking users for additional proof of identity when signing in. And you can now use the Advanced Protection Program—our strongest protection for users at risk of targeted attacks. With the Advanced Protection Program for the enterprise, we’ll enforce a specific set of policies for enrolled users including security key enforcement, blocking access to untrusted apps and enhanced scanning for email threats

If you choose not to use security keys for any reason, you have multiple other options to enforce 2-Step Verification on BYOD devices. For Android and iOS, you can use Google prompt, Google Authenticator, text message, or phone call options for a second verification step.

Prevent data loss and leakage with data loss prevention

We know that as an admin, one of your highest priorities is to keep internal information safe and secure. That’s why we developed data loss prevention (DLP) policies to help protect sensitive information in Drive, Docs, Sheets, Slides, and Gmail from loss, misuse, or being accessed by unauthorized users. With G Suite DLP, admins can choose which types of data are sensitive and exactly how to protect them. Our controls enable easy detection of a wide variety of common info types, and administrators can supplement this with custom content detectors to meet their organization’s needs. You can also classify files in Drive automatically using DLP rules (beta) to categorize your data by sensitivity levels. DLP works on all the devices in your organization, including BYOD ones, since the protection is at the data and application level. 

In addition to DLP, you can use DXP for iOS devices to restrict the copy/pasting of G Suite data to other accounts, personal or otherwise. DXP for iOS can also restrict users’ ability to drag and drop files from specific apps within their G Suite account. Similarly, you can use Google endpoint management to configure Android devices to prevent data sharing between personal and work profiles.

Make retention and eDiscovery possible on all your devices with Vault

To support your organization’s retention and eDiscovery needs, Vault enables corporate data that’s stored in G Suite and accessed by BYOD devices to be available for all your information governance needs. No matter the owner of the device, your organization’s data stored in Gmail, Drive, Chat, Groups, Voice, and Meet are accessible to Vault.

Using the zero-trust security model, the G Suite features above work together to keep your data protected and organization secure across all devices, whether they’re corporate-owned or BYOD.

(*Google Workspace was previously G Suite)