Oxford Economics and Google Cloud surveyed 1,000 CIOs across seven industries around the world to understand their approaches to developing strong business partnerships that support innovation and drive business results—and to find out what the most successful companies are doing differently. Download to read more.

This is part 4 in a multi-part series about the Kubernetes Resource Model. See parts 1, 2, and 3 to learn more. 

Kubernetes clusters can scale. Open-source Kubernetes supports up to 5,000 Nodes, and GKE supports up to 15,000 Nodes. But scaling out a single cluster can only get you so far: if your cluster’s control plane goes down, your entire platform goes down; if the Cloud region running your cluster has a service interruption, so does your app. 

Many organizations choose, instead, to operate multiple Kubernetes clusters. Besides availability, there are lots of reasons to consider multi-cluster, such as allocating a cluster to each development team, splitting workloads between cloud and on-prem, or providing burst capability for traffic spikes. 

But operating a multi-cluster platform comes with its own challenges. How to consistently administer many clusters at once? How to keep the clusters secure? How to deploy and monitor applications running across multiple clusters? How to seamlessly fail over from one region to another?

This post introduces a few tools that can help platform teams more easily administer a multi-cluster Kubernetes environment. 

The platform base layer, with Config Sync 

In the last post, we explored how thoughtful platform abstractions can help reduce toil for app developers- including for a multi-cluster environment, where automation such as CI/CD handles all interactions with the staging and production clusters. But equally important is the platform base layer, the Kubernetes resources and configuration that are shared across services. Your platform base layer might consist of Namespaces, role-based access control, and shared workloads like Prometheus. 

Platform abstractions depend on the existence of these base-layer resources. And so does the security and stability of your platform as a whole. It’s important that these resources not only get deployed, but also stay put. CI/CD is great for deploying resources, but what about making sure resources stay deployed? What if a Kubernetes Namespace gets deleted? Or a Prometheus StatefulSet is modified? 

Kubernetes’ job is to ensure that the cluster’s actual state matches the desired state. But sometimes, the “desired” state isn’t desired at all – it’s a developer who mistakenly modified a resource, or a bad actor that’s gained access into the system. For this reason, a platform base layer needs more than a one-and-done CI/CD pipeline. A tool called Config Sync can help with this. 

Config Sync is a Google Cloud product that can sync Kubernetes resources from a Git repository to one or more GKE or Anthos clusters. Unlike CI/CD tools like Cloud Build, Config Sync watches your clusters constantly, making sure that the intended resource state in the cluster always matches what’s in Git. Config Sync is designed primarily for base-layer resources like namespaces and RBAC. In this way, Config Sync is complementary to, not a replacement for, CI/CD. 

Config Sync runs in a Pod inside your Kubernetes cluster, watching your Git config repo for changes, and also watching the cluster itself for any divergence from your desired state in Git. If any configuration drift is detected from what’s stored in Git, Config Sync will update the API Server accordingly. 

You can point multiple Config Sync deployments at the same Git repo, allowing you to manage the base-layer platform resources for multiple clusters using the same source of truth. And by using Git as the landing zone for config, you can benefit from some of the GitOps principles we discussed in part 2, including the ability to audit and roll back configuration changes.

Let’s walk through an example of how to manage base-layer resources with Config Sync. 

The Cymbal Bank platform consists of four GKE clusters: admin, dev, staging, and prod. We can install Config Sync on all four clusters using the gcloud tool or the Google Cloud Console, pointing all four clusters at a single Git repository, called cymbalbank-policy. Note that this repo is separate from the application source and config repos, and is managed by the platform team. From the Console, we can see that all four clusters are synced to the same commit of the cymbalbank-policy repo. 

Now, let’s say that the Cymbal Bank platform team wants to limit the amount of CPU and memory resources each application team can request for their service. Kubernetes ResourceQuotas help impose these limits, and prevent unexpected Pod evictions.

The platform team can define a set of ResourceQuotas for each application namespace. They can also scope the resources to only be applied to a subset of clusters – for instance, to the production cluster only. (If no cluster name selector is specified, Config Sync will deploy the resource to all clusters by default.)

  apiVersion: v1
kind: ResourceQuota
metadata:
  name: production-quota
  namespace: frontend
  annotations:
    configsync.gke.io/cluster-name-selector: cymbal-prod
spec:
  hard:
    cpu: 700m
    memory: 512Mi

From here, the platform team can commit the resources to the cymbalbank-policy repo, and Config Sync, always watching the policy repo, will deploy the resources to the production cluster:

  NAMESPACE                      NAME                  AGE     REQUEST                                                                                                                               LIMIT
balancereader                  production-quota      6m56s   cpu: 300m/700m, memory: 612Mi/512Mi

If a developer tries to delete one of the ResourceQuotas, Config Sync will block the request, helping to ensure that these base-layer resources stay put.  

  error: You must be logged in to the server (admission webhook "v1.admission-webhook.configsync.gke.io" denied the request: requester is not authorized to delete managed resources)

In this way, Config Sync can help platform teams ensure the stability of that platform base-layer, as well as ensure resource consistency across multiple clusters at once. This, in turn, can help organizations mitigate the complexity of adding new clusters to their environment.  

Enforce policies on Kubernetes resources

Config Sync is a powerful tool on its own, and can work with any Kubernetes resource that your cluster recognizes. This includes Custom Resource Definitions (CRDs) installed with add-ons like Anthos Service Mesh. 

But Config Sync, by default, doesn’t have an idea of “good or bad” Kubernetes resources. It will deploy whatever resources land in Git, even resources that might pose a security risk to your organization. Security is an essential feature of any developer platform, and when it comes to Kubernetes, it’s important to think about security from the initial software design stages, and set up your clusters with security best-practices in mind.   

But it’s just as important to think about security at deploy-time. Who and what can access your clusters? What kinds of Kubernetes resources – and fields within those resources- are allowed? These decisions will depend on lots of factors, including the kinds of data your application deals with, and any industry-specific regulations. 

One common security use case for KRM is the need to monitor incoming Kubernetes resources, whether they’re coming in through kubectl, CI/CD, or Config Sync. But if you have multiple clusters, your Kubernetes environment has multiple API Servers, and therefore multiple entry points.  

A Google tool called Policy Controller can help automate resource monitoring across multiple clusters. Policy Controller is a Kubernetes admission controller that can accept or reject incoming resources based on custom policies you define. Policy Controller is based on the OpenPolicyAgent Gatekeeper project, and it allows you to define policies, or “Constraints,” as KRM. This means you can deploy them using Config Sync, via Git. Once deployed, Policy Controller uses your Constraints as a set of rules to evaluate all incoming KRM, rejecting resources that fall out of compliance.  

Let’s walk through an example. Say that the Cymbal Bank security team wants to ensure that no code in development is accessible to the public. Kubernetes Services of type LoadBalancer expose public IP addresses by default, so the platform team wants to define a PolicyController constraint that blocks Services of that type on the development GKE cluster.

To do this, the platform team can define a Policy Controller Constraint as KRM. This Constraint uses a Constraint Template, provided through the pre-installed Constraint Template library. The ConstraintTemplate defines the logic of the policy itself, and the Constraint makes the template concrete, populating any variables needed to execute the policy logic. Here, we’re also adding a Config Sync cluster name annotation, to scope this resource to apply only to the development cluster.

  apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sNoExternalServices
metadata:
  name: dev-no-ext-services
  annotations:
    configsync.gke.io/cluster-name-selector: cymbal-dev
spec:
  internalCIDRs: []

The platform team can then commit the resource to the cymbalbank-policy repo, and Config Sync will deploy the resource to the development cluster. 

From here, if an app developer tries to create an externally-accessible Kubernetes Service, Policy Controller will block the resource from being created.

  for: "constraint-ext-services/contacts-svc-lb.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [denied by dev-no-ext-services] Creating services of type `LoadBalancer` without Internal annotation is not allowed

The platform team can define as many of these Constraints as they want, each defining a separate policy.

Writing custom policies 

The Policy Controller Constraint Template library provides a lot of functionality, from blocking privileged containers, to requiring certain resource labels, to preventing app teams from deploying into certain namespaces. But if you want to enforce custom logic on your organization’s KRM, you can do so by writing a custom Constraint Template. 

Constraint Templates are written in a query language called Rego. Rego was designed for policy rule evaluation, and it can introspect Kubernetes resource fields to make a conclusion as to whether the resource is allowed or not. 

For instance, let’s say that the platform team wants to limit the number of containers allowed inside a single application Pod. Too many containers per Pod can cause outage risks— when one container crashes, the entire Pod crashes.

To enforce this policy, the platform team can define a Constraint Template, using the Rego language, that looks inside a resource to ensure that the number of containers per Pod is within the allowed limit: 

  apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8slimitcontainersperpod
spec:
  crd:
    spec:
      names:
        kind: K8sLimitContainersPerPod
      validation:
        openAPIV3Schema:
          properties:
            allowedNumContainers:
              type: integer
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8slimitcontainersperpod
        numTemplateContainers := count(input.review.object.spec.template.spec.containers)
        numRunningContainers := count(input.review.object.spec.containers)
        containerLimit := input.parameters.allowedNumContainers
        template_containers_over_limit = true {
          numTemplateContainers > containerLimit
        }
        running_containers_over_limit = true {
          numRunningContainers > containerLimit
        }
        violation[{"msg": msg}] {
          template_containers_over_limit
          msg := sprintf("Number of containers in template (%v) exceeds the allowed limit (%v)", [numTemplateContainers, containerLimit])
        }
        violation[{"msg": msg}] {
          running_containers_over_limit
          msg := sprintf("Number of running containers (%v) exceeds the allowed limit (%v)", [numRunningContainers, containerLimit])
        }
Then, the platform team can define a concrete Constraint, using this Constraint Template, to set the number of allowed containers per Pod to 3: 
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sLimitContainersPerPod
metadata:
  name: limit-three-containers
spec:
  parameters:
    allowedNumContainers: 3

Finally, the platform team can push these resources to the cymbalbank-policy repo, and Config Sync will deploy the policy to all four clusters. If a developer tries to define a Kubernetes Deployment containing more containers per pod than what’s allowed, the resource will be blocked at deploy time: 

  Error from server ([limit-three-containers] Number of containers in template (4) exceeds the allowed limit (3)): error when creating "constraint-limit-containers/test-workload.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [limit-three-containers] Number of containers in template (4) exceeds the allowed limit (3)

Custom Constraint Templates can give platform teams lots of flexibility in the types of policies they define and enforce in a Kubernetes environment. 

Integrating policy checks into CI/CD 

As we explored earlier, Config Sync and CI/CD are complementary tools. Config Sync works great for base-layer platform resources and policies, whereas CI/CD works well for application tests and deployment. 

But one pitfall of having two separate KRM deployment mechanisms is that app developers may not know that their resources are out of policy until they try to deploy them into production. This is especially true if some policies are scoped only to production, as we saw with the ResourceQuota example. Ideally, the platform team has a way to empower developers and code reviewers to know ahead of time whether new or modified resources are still in compliance. We can enable this use case by integrating policy checks into the existing Cymbal Bank CI/CD.

Policy Controller operates, by default, as a Kubernetes Admission Controller running inside the cluster. But Policy Controller also provides a “standalone” mode, running inside a container, that can be used outside of a cluster, such as from inside a Cloud Build pipeline. 

In the example below, Cloud Build executes Policy Controller checks by getting the cymbalbank-app-config manifests, cloning the cymbalbank-policy resources, and using the “policy-controller-validate” container image to evaluate the app manifests against the policies. 

  steps:
- id: 'Render prod manifests'
  name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
  entrypoint: '/bin/sh'
  args: ['-c', 'mkdir hydrated-manifests && kubectl kustomize overlays/prod > hydrated-manifests/prod.yaml']
- id: 'Clone cymbalbank-policy repo'
  name: 'gcr.io/kpt-dev/kpt'
  entrypoint: '/bin/sh'
  args: ['-c', 'kpt pkg get https://github.com/$$GITHUB_USERNAME/cymbalbank-policy.git@main constraints
                  && kpt fn source constraints/ hydrated-manifests/ > hydrated-manifests/kpt-manifests.yaml']
  secretEnv: ['GITHUB_USERNAME']
- id: 'Validate prod manifests against policies'
  name: 'gcr.io/config-management-release/policy-controller-validate'
  args: ['--input', 'hydrated-manifests/kpt-manifests.yaml']
availableSecrets:
  secretManager:
  - versionName: projects/${PROJECT_ID}/secrets/github-username/versions/1 
    env: 'GITHUB_USERNAME'
timeout: '1200s' #timeout - 20 minutes

From here, an app developer or operator can know if their resources violate org-wide policies, by looking at the Cloud Build output for their Pull Request:  

  Status: Downloaded newer image for gcr.io/config-management-release/policy-controller-validate:latest
Error: Found 1 violations:
[1] Number of containers in template (4) exceeds the allowed limit (3)

By integrating policy checks into CI/CD, app development teams can understand whether their resources are in compliance, and platform teams add an additional layer of policy checks to the platform. 

Overall, Config Sync and Policy Controller can provide a powerful toolchain for standardizing base-layer config across a multi-cluster environment. Check out the Part 4 demo to try out each of these examples. 

And stay tuned for Part 5, where we’ll learn how to use KRM to manage cloud-hosted resources. 

In the past six months, Mynd Property Management has doubled in size—in part by hiring, but also through acquisitions that help build our business. It’s an exciting time to be at Mynd—every week brings new people who need to be set up with email and network access, which can also be a bit hectic. To speed onboarding, we use G Suite Enterprise to get our new people working as soon as they walk through the door.

Streamlined onboarding
We started using G Suite when we launched the business because it fit our overall approach to technology—that is, deploying cloud tools that are flexible for our employees and easy for IT to manage. Good thing we picked G Suite, because given the pace of our acquisitions—13 at last count—we have our hands full with combining business systems and keeping people productive. Onboarding is literally 80 percent of our daily work! 

Some people on our team don’t have technical backgrounds, so we are careful to only introduce new software tools once people are prepared to use them. With G Suite, people need very little ramp-up time because it is an intuitive, integrated solution. In one of our in-person training sessions, we showed an accountant how to open up a Microsoft Excel file in Google Sheets, and he started working on it right away, without much help.

We’re a remote-first company, so working in the cloud is second nature to us. About 75 percent of our total workforce are remote full-timers. G Suite tools are optimized for mobile, so they’re especially useful for the 50 full-time property managers who work solely on mobile devices in the field. 

Better meetings, easier file storage
In Mynd’s early days, we used G Suite Basic, but upgraded to G Suite Enterprise after Google Cloud Premier Partner Suitebriar showed us the benefits of using tools like Hangouts Meet and shared drives in Google Drive. They’ve really helped us scale the business, so we can stay productive even in the midst of acquisitions—and of course, onboard people more quickly. 

Many G Suite tools have become things we can’t live without. Our executives got first crack at using Hangouts Meet Hardware, and loved it so much—especially the recording feature—that now all of our conference rooms have it. 

Google Drive got the same love from everyone. It’s all we need for file sharing and storage, which means we don’t have to pay for another storage solution. Drive also helps with onboarding; as soon we assign an employee a Google Account, she or he automatically gets access to the shared drives (in Drive) that they need to do their jobs. 

Customizing the tech stack
As much as we love G Suite, we still like to use some non-G Suite tools. It helps that we can tailor G Suite to fit in our technology stack any way we want. We’d never get that control and customization from other vendors in the productivity space.

For example, we love using Slack for internal chat and messaging, so we turned off Hangouts Chat. We like that we can control these channels, and that G Suite is not an all-or-nothing product. That means we’re not afraid to try out new G Suite features, since we can roll them back if they’re not right for us. 

The result of our day-to-day G Suite use is that we don’t suffer much down time even during acquisitions. We can pull relevant data from users’ old systems and easily automate user creation from start to finish. When we merge with companies that already use G Suite, we can move their data over to the business in less than a week.

When we recently moved over 80 new employees and 50 different web domains to G Suite, it was much less challenging than we expected. Since we don’t plan to slow down on acquisitions, we breathe easier knowing that G Suite keeps everyone working in the middle of so much change.

Many executives across the globe are feeling the effects of a recent pivot to hybrid work models. This massive transition represents a significant departure from existing office culture. As business leaders look to operationalize their flexible work model infrastructure, they’re also evaluating what it takes to attract and retain employees in this new landscape. Instead of seeing these as separate initiatives, business leaders have a unique opportunity to collectively overhaul corporate culture, focusing on hybrid work and diversity, equity, and inclusion (DEI) strategies in tandem.

Combining DEI and hybrid work strategies provides a more inclusive culture

Focusing on diversity, equity, and inclusion doesn’t mean business leaders are offering an underrepresented individual a specific role or platform because they fit a certain demographic category. Lack of job quality criteria leads to animosity and cultural inequality. Here, diversity means creating a meritocracy with equal pay and equal opportunity for all.

Organizations that thoughtfully combine DEI and hybrid work strategies will produce cultures that improve employee experience and business agility. Integrated DEI and hybrid workplace support better business outcomes by:

Increasing the total available talent pool. A business must cast a wide net for candidates by expanding how and where it searches. Adding a flexible work schedule policy expands potential talent by allowing parents, caregivers, and others to select the hours that work best for their family obligations. Remote work also holds a higher priority in job seekers’ desires. For example, a FlexJobs survey revealed that 24% of workers say the ability to work from home is so important to them that they are willing to take a 10-20% pay cut to work remotely, and 21% would give up some vacation time.

Creating a more inclusive culture. Job seekers want to work for companies that prioritize DEI. According to a recent CNBC and SurveyMonkey Workforce survey, nearly 80% of workers say that they want to work for a company that values diversity, equity, and inclusion. A DEI-centric work culture allows job seekers to feel like there’s a place for them.

Fostering innovation and profitability with broader perspectives. In general, diversity and inclusion promotes the expression of ideas from people with different backgrounds, leading to breakthroughs in creativity, innovation, and attracting new customers. Workplace DEI programs create an employee base that closely mirrors a customer base. A diverse team is less likely to miss potential market opportunities if representative of a broad range of ethnicity, race, age, assigned sex, and socioeconomic backgrounds. A 2020 McKinsey analysis found that companies in the top quartile for ethnic and cultural diversity on executive teams were 36% more likely to have above-average profitability than companies in the fourth quartile.

Hybrid work should not derail DEI efforts

While there are numerous benefits to integrating DEI and hybrid work strategies, it would be naive to think it won’t bring challenges. Businesses run the risk of creating new inequities and exacerbating existing ones as they move to hybrid models. One potential issue lies in which employees can work remotely. An HBR article shared how pre-Covid less than 30% of all workers could work from home; only 16% of Latinx workers and 19% of Black workers had remote flexibility, compared to 37% of Asian workers and 30% of white workers.

Even for employees who can work remotely, it’s essential to eliminate “proximity bias.” A study by SHRM found that 67% of surveyed supervisors admitted to considering remote workers more easily replaceable than onsite workers at their organization. Additionally, 72% said they would prefer all of their teams to be working in the office.

A successful hybrid environment strikes a balance between flexibility and inclusivity where individuals feel valued, and everyone has equal opportunities for advancement. In another study, researchers concluded that remote workers and office workers were promoted at the same rate, but found that remote workers’ salaries grew more slowly. Leading companies recognize that physically being at the office full-time isn’t necessary to produce great results.

8 steps to creating a diverse hybrid work culture

As DEI becomes a new business imperative, organizations must work to minimize unconscious bias, creating market-based salaries for all workers, and introducing educational programs that support talent development. Here are some concrete steps you can take to get started.

• Educate current and future leaders on DEI and hybrid work issues. As you build hybrid work strategies, it’s crucial to discover and understand unconscious biases. According to Deloitte, companies need a holistic DEI learning strategy, including unconscious bias training and developing the conditions for long-term behavior changes.
• Embrace tools that support DEI and hybrid workplaces. Organizations can leverage technology and AI-enabled HR software to support their efforts. Collaboration tools provide everyone with a voice regardless of where they are (for example, using Spaces for team chats in Google Workspace so that everyone has access to the same information). Video call features such as transcriptions and translation can help the hearing-and-vision-impaired and employees who speak different languages. It’s also possible to purchase recruiting solutions with AI technology to reduce the chance of bias or discrimination when reviewing candidates.
• Strengthen culture by marrying DEI with meritocracy. There should also be explicit guidelines to ensure the systems and processes for hiring and promoting individuals are both transparent and equitable. One method for accomplishing this goal is to define detailed job requirements and metrics per role to understand what skills are associated with specific functions. It’s also essential to ensure you’ve designed these processes in a way that screens out bias. Leading companies will deploy analytics tools to evaluate the systems’ fairness and track if the systems are progressing against diversity targets.
• Create upskilling programs to train employees. Many organizations struggle to find talent with the right skill sets. One solution to this problem is to invest in educating existing employees. Companies must develop diverse talent at every level to create a pipeline of candidates that progresses into senior management. Similar to the way companies offered management training programs to college graduates, organizations should design programs that provide skills enhancement for entry and mid-level employees. Almost every job will require a certain level of technical skill—organizations must upskill existing talent.
• Restructure compensation before employees leave. It’s easier to retain an employee than to onboard and train a new hire. According to the Atlanta Federal Reserve Bank’s wage growth tracker, wages for people who have switched jobs have outpaced those who’ve stayed at one employer since 2011. One way to retain talent is to right-size salaries to a competitive market rate.
• Remember that money isn’t everything. While individuals may switch jobs for money, they often stay in a position for the culture. Creating a leading employee experience helps organizations keep talent. Work from home programs, flexible work schedules, and advanced programs such as childcare benefits help build the foundation. But increasingly, individuals are looking for a sense of belonging. According to research from McKinsey, employees who feel included are three times more likely to report being excited about working for their organization and committed to its success. Another McKinsey report focusing on attrition shows that employees primarily leave companies because they don’t feel valued or don’t have a sense of belonging.
• Don’t forget to commit the resources. Most organizations built their cultures before DEI and hybrid work were requirements. These companies must invest in new work tools and educational programs, and create new hiring and talent management practices. A business can make a more significant impact with fewer resources by ensuring purchases and processes support flexible and inclusive work cultures.
• Make sure actions are backed up with DEI transparency measurements. The pressure to support DEI could lead to situations where activities are celebrated over results. While it’s not easy to be vulnerable and transparent about where the problems exist, employers should take stock of where they are today. They should present this authentically, create goals for the future, and design a measurable action plan to track its progress. With so many variables and unknowns, it’s crucial for talent leaders to track the way that hybrid work and DEI efforts advance.

Diversity should be part of your efforts because it’s the culture you want to create. However, ignoring diversity efforts can also come with ramifications. For example, Goldman Sachs announced in 2021 that the firm would only underwrite IPOs in the United States and Europe for companies with at least two diverse board members. Meanwhile, the U.S. Securities and Exchange Commission adopted new rules this summer that require Nasdaq-listed companies to have at least two diverse directors – including one woman and at least one member of an underrepresented community.

In order for the benefits of diversity to emerge in the workplace, companies need to enact cultural and behavioral change. As stated in the ICSM report, “Never become too comfortable with your efforts, as there is always room for improvement with diversity at your organization, whether you see it or not.” It’s clear that what organizations have done to attract talent in the past isn’t resonating the same way today. A diverse workforce delivers better ideas and stronger communities, building deeper relationships with customers along the way. Moving forward, it’s expected that a company’s investment in a diverse, inclusive, and hybrid work culture will set it apart from its competition, leading to increased business agility and better financial outcomes.

Founded in 1930 and headquartered in Jordan, Arab Bank is one of the oldest banks in the Middle East. Operating out of 28 countries, we’ve earned our customers’ trust with a prudent approach to operations and respect for the cultures and customs in the region. 

With a few exceptions where cloud providers have hosted their datacenter in a Middle Eastern or North African country, the banking sector, in general, in the region has been slow to adopt cloud technology for a number of reasons, including concern about data security, maturity and security controls of cloud services (PaaS and SaaS), and regulations in place. But on the other hand, we saw the opportunity to accelerate our development and testing using the cloud, as well as to partner with the fintech community and digital service providers to integrate their solution in the banking ecosystem. We needed more flexibility to connect with the outside world, and a more open architecture to help us drive our internal innovation at a faster rate with the help of the fintech industry. By collaborating with Google Cloud, we reached those goals and accelerated app development and testing through products like Apigee and Anthos. We’re now offering innovative apps and services to our customers and employees that leverage new technological capabilities to give more agility and flexibility, and to optimize our workloads.

Embracing the cloud in a regulated industry 

To get started with the cloud, we needed to create internal awareness about cloud technology, the API layer, containers and their benefits amongst our leaders and staff. Google helped us educate and get buy-in from key functions by organizing open technology demonstration sessions and discussion panels. When considering potential cloud providers we had four decision criteria: maturity of security controls, ease of use, cost, and scalability / agility for new deployments and continuous innovation. This last factor was critical, and we were impressed by Google Cloud’s innovation roadmap, both via direct conversations and at Google’s Next conference, where we met a lot of people passionate about technology, innovation and building something new.  

Going back to our journey, given the above-mentioned regional limitations, we started to develop a hybrid cloud approach. This helped us continue to operate on-premises for a number of services in production, particularly those that have personally identifiable information (PII) or other sensitive data attached, and to leverage the cloud for development, testing and production workloads that don’t contain customer data. 

In the short term, we didn’t anticipate that our many jurisdictions would allow data to be transported to other countries. But cloud tools will allow us to tokenize or anonymize customer data while maintaining customer data on-premises. This applies to many digital journeys such as customer onboarding, credit facility online applications, or marketplace navigation. In the coming years, we predict our API integration with partners will accelerate and enrich the overall digital value proposition of our business segments, namely consumer banking, small and medium-size businesses and large corporate and institutional clients. 

Building connections and cornerstones in the cloud

The first move in our digital transformation was to implement Apigee, Google Cloud’s API management platform, to connect to the world’s digital banking ecosystem. Apigee provides the security, sharing, mediation policies, and developer portal capabilities for us to successfully meet Open Banking standards while focusing on innovation.  

On the back of the Apigee implementation, we created an accelerator program to incubate Fintech ideas that can, in turn, integrate into our digital platforms and be offered to our customers. We also developed various banking APIs, all designed and documented in accordance with PSD2 and Open Banking regulations, and made them available to our partners. These APIs exposed on our API development portal offer the needed code structure for fintech companies to design creative solutions around them.

Next, we adopted Anthos, Google Cloud’s managed application platform. Anthos has become a cornerstone of our operations because it works across hybrid cloud, offering integration of microservice containers and fueling collaborative opportunities with external parties. Our current Anthos infrastructure includes several hundreds of  microservices now running on containers in Google Kubernetes Engine (GKE) and on-premises. We now use the cloud for collaboration, development and testing, but not for production, which is done on-premises. 

Along the way, Google Cloud’s Professional Services Organization (PSO) helped us through the entire cloud setup process, and with the adoption of Anthos. We originally built on the cloud tools through an iterative process, learning from our successes and errors along the way. Now that we have a better sense of how Anthos operates, we’re building a fresh infrastructure atop a sound, stable, and resilient foundation that will let us scale easily as we work to transform Arab Bank into a digital-first enterprise, that is our ambition. 

Currently products running on Anthos include customer acquisition and onboarding via mobile apps, and our Arabi-Pay app, which allows customers to instantly pay each other via WhatsApp or other messaging platforms. Leveraging Anthos, our instant loan service for Arab Bank salaried employees can grant and disburse loans up to $7,000 in less than seven minutes.

In addition, we’ve built a number of digital journeys for our Small and Medium Enterprise (SME) customers, such as our SME client digital onboarding process and paperless SME lending platform.

While some may think that digital adoption in this part of the world can be slow, as customer contact remains anchored in our customs, the recent COVID-19 pandemic has accelerated the adoption of digital banking services and electronic payments, inspiring more confidence to buy and pay online. Thanks to our rich and user-friendly banking app that relies on Apigee and Anthos for critical customer journeys, over 90% of new-to-bank customers are using our mobile apps. Within the next 18 months, we predict that number will be closer to 100%. 

Of course, with higher customer adoption comes the challenge of potential service interruptions. A single moment of downtime can be highly visible to many digital customers. But Google Cloud’s Anthos and Apigee give us the flexibility to resume processes at a fast rate, so any interruptions are almost invisible to our customers. In fact, when the COVID-19 pandemic hit, though our branches could be open only for limited hours each day, our consumer clients in particular were able to take advantage of our digital services in a very self-sufficient manner. Being well positioned with Google Cloud, we could also keep our internal teams and external partners connected and productive. Without Google Cloud, continuing the digital transformation of the bank at the pace we wanted would have been a big challenge. 

Collaborating across borders and time zones

With Google Cloud, our ability to collaborate and partner has transformed significantly. We operate 24/7 now because our developers are scattered across multiple geographies and different time zones. Because testing and deployment can run around the clock, including on weekends, we currently deploy a new digital journey in a few weeks, faster than ever before. This has given our organization a spirited mindset that prioritizes innovation, and raised the bar in terms of our operating model.  

Another consideration about Google Cloud tools is the elimination of inefficient processes typically seen in a software development lifecycle. We now build in a completely agile manner, from design squads until deployment in production. Compared to where we were two years ago, when we had an annual maximum of two systems releases in production, we now have close to monthly releases of our digital packages. In addition, we also  supplement those monthly releases with additional ad-hoc releases and fixes in between. As a result, we have removed internal silos and improved tremendously the collaboration between the product and sales teams, operations, IT Dev Factory and Infrastructure, as well as our supporting functions.

Through the agile process facilitated by APIs, we introduced Design Thinking workshops involving external customers and prospects early on to understand their true pain points better and emotions during the existing journeys and how to make new digital journeys frictionless. As a result, the relevance of our products for various customer personas has improved tremendously. 

Transcending banking 

With Google Cloud, we can offer our customers so much more than just banking. We’ve become more digitally relevant to their lives. For example, we recently launched a mortgage app that helps customers all the way from home selection through mortgage negotiations and closing, and even to getting the home decorated. It’s an end-to-end journey in which API integration with key regional players was a cornerstone to our success. 

For other digital products, we have an extensive roadmap of lifestyle-based solutions relevant to each segment and age group. We’ve only scratched the surface of the services we can provide, and we see the cloud as the future for everything we want to do.

Read more about Google Cloud’s Open Banking solution to learn how you can simplify and accelerate the process of delivering open banking as required by PSD2. You can also view our video on Open Banking, powered by Apigee API Management.