Hi there! I’m David Challoner from Access Site Reliability Engineering (SRE), here with Anthony Bushong from Developer Relations to talk about how Corp Eng is adopting Anthos Service Mesh internally at Google.

Corp Eng is Google’s take on “Enterprise IT”. A big part of the Corp Eng mission is running the first and third party software that powers internal business processes – from legal and finance to floor planning and even the app hosting our cafe menus – all with the same security or production standards as any of Google’s first party applications.

Googlers need to access these applications, which sometimes then need to access other applications or other Google Cloud services. This traffic can cross different trust boundaries which can trigger different policies.

Access SRE runs the systems that mediate this access, and we implemented Anthos Service Mesh as part of our solution to secure the way Googlers access these applications.

But why?

You can probably tell, but the applications Corp Eng is responsible for have disparate requirements. This often means that certain applications are tied to disparate infrastructure due to legal, business or technical reasons – which can be challenging when those infrastructures work and operate differently.

Enter Anthos. Google Cloud built Anthos to provide a consistent platform interface unifying the experience of working with apps on these varying underlying infrastructures, with the Kubernetes API at its foundation.

So when searching for the right tool to build a common authorization framework to mediate access to CorpEng services, we turned to Anthos – specifically Anthos Service Mesh, powered by the open-source project, Istio. Whether these services were deployed in Google Cloud, in Corp Eng data centers, or at the edge onsite at actual Google campuses, Anthos Service Mesh delivered a consistent means for us to program secure connectivity.

To frame the impact ASM had on our organization, it’s helpful to introduce the roles of the folks who manage and use it:

For security stakeholders, ASM provides an extensible policy enforcement point running next to each application capable of provisioning a certificate based on the identity of the workload and enforcing mandatory fine-grained application-aware access controls.

For platform operators, ASM is delivered as a managed product, which reduces operational overhead by providing out-of-the-box release channels, maintenance windows, and a published Service Level Objective(SLO).

For service owners, ASM enables the decoupling of their applications from networking concerns, while also providing features like rate limiting, load shedding, request tracing, monitoring, and more. Features like these were typically only available for applications that ran on Borg, Google’s first-party cluster manager that ultimately inspired the creation of Kubernetes.

In sum, we were able to secure access to a plethora of different services with minimal operational overhead, all while providing service owners granular traffic control.

Let’s see what this looks like in practice!

The architecture

In this flow, user access first reaches the Google Cloud Global Load Balancer [1], configured with Identity Aware Proxy (IAP) and Cloud Armor. IAP is the publicly available implementation of Google’s internal philosophy of BeyondCorp, providing an authentication layer that works from untrusted networks without the need for a VPN.

Once a user is authenticated, their request then flows to the Ingress Gateway provided by Anthos Service Mesh [2]. This provides additional checks that traffic flows to services only when the request has come through IAP, while also enforcing mutual TLS (mTLS) between the Anthos Service Mesh Gateway to the Corp services owned by various teams.

Finally, additional policies are enforced by the sidecar running in every single service Pod [3]. Policies are pulled from source control using Anthos Config Management[4], and are propagated to all sidecars by the managed control plane provided by Anthos Service Mesh[5].

Managing the mesh

If you’re not familiar with how Istio works, it follows the pattern of a control plane and a data plane. We talked a little bit about the data plane – it is made up of the sidecar containers running alongside all of our service Pods. The control plane, however, is what’s responsible for updating these sidecars with the policies we want to enforce:

Thus, it is critical for us to ensure that the control plane is healthy. This is where Anthos Service Mesh gives our platform owners a huge advantage with its support for a fully-managed control plane. To provision cloud resources, like many other companies, our organization uses Terraform, the popular open-source infrastructure as code project. This gave us a declarative and familiar means for provisioning the Anthos Service Mesh control plane.

First, you enable the managed control plane feature for GKE by creating the google_gke_hub_feature resource below using Terraform.

resource "google_gke_hub_feature" "feature_asm" {
name = "servicemesh"
location = "global"
provider = google-beta
}

Keep in mind that at publication time, this is only available via the google-bet provider in Terraform.

Once created, we then provision a ControlPlaneRevision custom resource in a GKE cluster to spin up a managed control plane for ASM in that cluster:

apiVersion: mesh.cloud.google.com/v1alpha1
kind: ControlPlaneRevision
metadata:
  
name: asm-managed
  
namespace: istio-system
spec:
  
type: managed_service
  
channel: regular

Using this custom resource, we are able to set the release channel for the ASM managed control plane. This allows for our platform team to define the pace of upgrades in accordance with our team’s needs.

In addition to managing the control plane, ASM also provides management functionality around the data plane to ensure each sidecar Envoy is kept up to date with the latest security updates and is compatible with the control plane – one less thing for service operators to worry about. It does this using Kubernetes Mutating Admission Webhooks and Namespace labels to modify our Pod workload definitions to inject the appropriate sidecar proxy version.

Syncing mandatory access policies

With the core Anthos Service Mesh components in place, our security practitioners can define consistent, mandatory security policies for every single GKE cluster, using Istio APIs.

For example, one policy is enforcing strict mTLS between Pods using automatically provisioned workload identity certificates. Earlier, we talked about how this is enforced between the Istio Gateway; that same policy enforces mTLS between all Pods in our cluster.

Another policy we implement is denying all egress traffic by default, requiring service teams to explicitly declare their outbound dependencies. The following is an example of using an Istio Service Entry to allow granular access to a specific external service – in this case, Google. This helps prevent unintended access to external services.

apiVersion: networking.istio.io/v1alpha3

kind: ServiceEntry
metadata:
  
name: google
spec:
  
hosts:
  - www.google.com
  
ports:
  - number: 443
    
name: https
    
protocol: HTTPS
  
resolution: DNS
  
location: MESH_EXTERNAL
EOF

These policies are automatically synced to all service mesh namespaces in each cluster using Anthos Config Management. By using our internal source control system as a source of truth, Anthos Config Management can sync and reconcile policies across all of our GKE clusters, ensuring that these policies are in place for every single one of our services. You can find more details about our implementation of Anthos Config Management here.

With this in place, our team plans on eventually migrating away from security automation that operates solely based on explicit IP, port and protocol policies.

Integration with Identity-aware Proxy

The publicly available version of the BeyondCorp proxy used by CorpEng is called Identity-aware Proxy (IAP), which offers an integration with Anthos Service Mesh. IAP allows you to authenticate users trying to access your services and apply Context-Aware-Access policies. This integration comes with two main benefits:

• Ensuring that user traffic to services in the service mesh only come through Identity-aware Proxy
• Enforcing Context-aware access (CAA) trust levels for devices, defined by multiple device signals we collect

Identity-aware Proxy allows us to capture this information in a Request Context Token (RCToken), which is a JSON Web Token (JWT) created by Identity-aware Proxy that can be verified by ASM. IAP inserts this JWT into the Ingress-Authorization header. Using Istio Authorization Policies similar to the following policy, any requests without this JWT are denied:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: iap-gateway-require-jwt
namespace: istio-system
spec:
selector:
matchLabels:
app: istio-iap-ingressgateway
action: DENY
rules:
- from:
- source:
notRequestPrincipals: ["*"]

Here is an example policy that requires a fullyTrustedDevice access level – this might be a device in your organization that is known to be corporate-owned, fully-updated, and running an IT-approved configuration :

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: require-fully-trusted-device
  namespace: fooService
spec:
  selector:
    matchLabels:
      app: fooService
  action: ALLOW
  rules:
  - from:
    - source:
       requestPrincipals: ["*"]
    when:
    - key: request.auth.claims[google.access_levels]
      values: ["accessPolicies/$orgId/accessLevels/fullyTrustedDevice"]

This allows our security team to not only secure service to service communications, or outbound calls from services, but also specifically require incoming requests come from trusted devices and authenticated users using a trusted device.

Enabling service teams

As an SRE, one of our priorities is ensuring Service-level indicators (SLIs), SLOs, and Service-level agreements (SLAs) exist for services. Anthos Service Mesh helps us empower service owners to do this for their services, as it exposes horizontal request metrics like latency and availability to all services in the mesh.

Before Anthos Service Mesh, each application had to export these separately (if at all). With ASM service owners can easily define their Service’s SLOs in the cloud console or via terraform using these horizontally exported metrics. This then allows us to integrate SLOs into our higher-level service definitions so we can enable SLO monitoring and alerting by default. You can see the SRE book for more details on SLOs and Error budgets.

The takeaway

ASM is a powerful tool that enterprises can use to modernize their IT infrastructure. It provides:

• A shared environment-agnostic enforcement point to manage security policy
• A unified way to provision identities, describe application dependencies

This also enables previously unheard of operational capabilities such as distributed tracing or incremental canary rollouts – which were difficult to find in the typical enterprise application landscape.

Because it can be incrementally adopted and composed with existing authorization systems to close gaps – barriers to adoption are low and we recommend you start evaluating it today!

Cloud infrastructure offers so many advantages: on-demand scalability, built-in security, and a bevy of tooling to scale your business at the speed of the Internet.

It enables companies to pursue “blitzscaling,” as Reid Hoffman calls it. Capital expenses that take years or decades to pay off are no longer required to establish global networking, compute capacity, storage resources, and application enablement tooling. By renting these assets from cloud providers, you can translate capital costs to operational costs, help your company to manage resources efficiently, keep expenses aligned with growth trajectory, and develop a portfolio of business-driving technology assets faster than would have previously been possible.

Certainly, this is the message from many founders and venture capitalists: move to the cloud, build a ton of software, scale like crazy, and win. Sounds great, right?

But history has shown us that the process is not quite this simple. Scale can be great and is a prerequisite to many of today’s most exciting business opportunities — but scale also frequently produces unintended consequences. You need to be able not only to achieve scale but also to manage it.

When scale produces bloat

To understand how unforeseen impacts can ripple out from a rapidly-scaled technology, consider the first mass-produced vehicle, the Model T. The first production model was produced in 1908, and less than two decades later, Ford had produced 15 million. This rapid growth profoundly affected urban living for decades.

Thanks to cars, fewer workers needed to live near cities or along major public transportation lines. The ease-of-access to personal transportation led to suburban population centers and, ultimately, urban sprawl. “Sub-cities” extricated homeowners from the density of urban population centers but also created a complex web of unintended consequences: new and often duplicative administrative bodies, new taxes, new zoning laws, and more intricate infrastructure projects. We are arguably still dealing with this fallout today as communities grapple with antiquated zoning laws and, in their attempts to find ways forward, often produce only more sprawl.

If your technology is in the cloud, you may be challenged with similar issues. For example, when developers build software in the cloud, many of the barriers to building software are removed. As a result, developers build a lot of software — but often without a lot of intentional design. This in turn results in companies building a large number of disparate systems and overwhelming app sprawl.

The cloud can help you proliferate technologies so quickly, in other words, that effective management and re-use of resources becomes incredibly tough.

Managing scale

Software assets are often seen as comprising the “brains” of a company — but to effectively grow, you should consider not only brains but also the digital nervous system. You need systems that connect the brains to all of the other important limbs that have to coordinate in order for your company to drive value.

One way of creating this nervous system and managing this complexity is to leverage the facade design pattern: applying an API layer that abstracts the underlying complexity of multiple systems into an elegant, reliable interface that encourages discovery and re-use of applications, functions, and other technology artifacts such as build and deployment pipelines.

For example, too many enterprises build a new digital “road” for each application that needs to authenticate or authorize users. Instead, you can leverage a facade pattern to help establish one “main road” for these purposes, encourage reuse of the road for new projects, and — with API management — monitor and control all traffic along the road. For tasks such as aligning risk and compliance operations or unifying developer onboarding functions, the distinction between reusing elegant roads and continually building new complicated ones could not be more important.

API management tools mean you can establish a single-pane-of-glass view into your network of digital roads and destinations or, if you prefer the biology metaphor, into the nervous system routes connecting your company’s software brains. This view becomes a point at which suspicious API usage patterns can be detected, reported, and handled and through which business-driving insights from legitimate traffic can be gleaned. Rather than dealing with IT sprawl, you can maintain visibility over your assets, control how they are used, roll out experimental digital products and get immediate feedback on adoption, and generate analytics to help you effectively divest from and invest in opportunities as the market demands.

More is not always better

More is not always better, and, in fact, it is sometimes worse.

It’s a time-worn sales axiom that would-be customers are more likely to make a choice when presented with two or three options rather than fifty, for example, and virtually all of us can relate to moments of “analysis paralysis” triggered by too many choices. These dynamics of choice are such that the diminishing marginal utility of each choice can detract from each option: with each choice comes a little stress, and as these stresses accumulate, customer satisfaction suffers. IT systems are no different; if developers building new connected experiences are left to their own designs, rather than encouraged with standardized resources and best practices, their work may add to complexity and customer dissatisfaction.

One need look only at the various open air markets around the world to see this point in action. They may offer many things to see but the experience is anything but efficient. The multitude of shops selling similar and duplicate items, the zigzag layout, and the expected friction from bargaining with each vendor all mean concepts such as market-wide product discovery and product inventory go out the window. If your developer experience mirrors these marketplaces, your efforts to scale are more likely to tangle up your operations than to satisfy customers.

Contrast this experience with luxury retail experiences where product areas are clearly demarcated in different retail spaces, product explanations accompany showcase items, inventory is available locally or ready to ship, clear pricing is readily available, and similar stores are intentionally anchored to strategic physical areas to encourage customer flow among them. The developer programs that cloud efforts are often meant to enable require a similar focus on luxurious experiences.

If your growth strategy creates bloat, internal developers will not use resources efficiently and your ability to share resources with external partners will likely be hamstrung. Applying API facades helps to ensure that your growing software portfolio is not a complicated maze to be navigated but rather a series of technology products for developers to leverage.

Indeed, you should think of the API itself as a product, not just a way of surfacing or connecting technology. The better the product, the more easily it can be managed, the better the experience developers will have using it, and the more control you’ll have harnessing the cloud to extend your business’s footprint.

Application modernization is quickly becoming one of the pillars of successful digital transformation and cloud migration initiatives. Many organizations are becoming aware of the dramatic benefits that can be achieved by moving legacy, on-premises apps and databases into cloud native infrastructure and services, such as reduced Total Cost of Ownership (TCO), elimination of expensive commercial software licenses, and improved performance, scalability, security and availability.

The complexity of applications and databases to a cloud-centric architecture requires a rapid, accurate, and customized assessment of modernization potential and identification of challenges. Addressing business and functional drivers, TCO calculations, uncovering technological challenges and cross-platform incompatibilities, preparation of migration, and rollback plans can be essential to the success and outcome of the migration. 

These cloud migration initiatives are often divided into three high-level phases: 

1. Discovery: identifying and cataloging the source inventory. Output is usually an inventory of source apps, databases, servers, networking, storage, etc. The discovery of existing assets within a data center is usually straightforward and can often be highly automated. 
2. Pre-migration readiness: the planning phase. This includes the analysis of the current portfolio of the databases and applications for migration readiness, determining the target architecture, identifying technological challenges or incompatibilities, calculating TCO, and preparing detailed migration plans. 
3. Migration execution: where the rubber hits the road. During this phase of the migration process, database schemas are actively converted, the application data access layer is refactored, data is replicated from source to target, often in real-time, and the application is deployed in its determined compute platform(s). 

Successful evaluation and planning phase as part of the pre-migration readiness phase can bolster confidence in investment towards modernization. Skipping or inaccurately completing the pre-migration phase can lead to a costly and sub-optimal result. Relying on manual pre-migration assessments can lead to long migration timelines, reduced success rates and poor confidence in the post-migration state, increased risk and total migration cost.

Some of the commonly asked question during pre-migration include:

1. How compatible are my source databases, which are often commercial and proprietary in nature, with their open-source cloud-native alternatives? For example, how compatible are my Oracle workloads and usage patterns with Cloud SQL for PostgreSQL?
2. What’s my degree of vendor lock-in with my current technology stack? Are proprietary features and capabilities being used that are incompatible with open-source database technologies?
3. How tightly-coupled are my applications with my current database engine technology? Can my applications be deployed as-is, refactored for cloud readiness with ease, or will it be a big undertaking?
4. How much effort will my migration require? How expensive will it be? What will be my run-rate in Google Cloud post-migration and my ROI?
5. Can we identify quick-win applications and databases to start with?

There is a direct association between the accuracy and speed of the pre-migration phase and the outcome of the migration itself. The faster and more accurately organizations complete the required pre-migration analysis, the more cost efficient and successful the migration itself will usually be.  

EPAM Systems, Inc., a leader in digital transformation, worked with Google Cloud as a preferred partner to accelerate cloud migrations beginning with pre-migration assessments. Leveraging EPAM’s migVisor for Google Cloud—a unique pre-migration accelerator that automates the pre-migration process—and EPAM’s consulting and support services, organizations can quickly generate a cloud migration roadmap for rapid and systematic pre-migration analysis. This approach has resulted in the completion of thousands of database assessments for hundreds of customers.

migVisor is agentless, non-intrusive, and hosted in the EPAM cloud. migVisor seamlessly connects to your source databases and runs SQL queries to ascertain the database configuration, code, schema objects and infrastructure setup. Scanning of source databases is done rapidly and without interruption to production workloads.

migVisor prepares customers to land applications in Google Cloud and its managed suite of databases services and platforms such as Cloud SQL, bare metal hosting, Spanner and Cloud Bigtable. migVisor supports re-hosting (lift-and-shift), re-platforming, and re-factoring.  

“EPAM’s recent application assessment update to its migration tooling system, migVisor, will bring a new level of transparency to the entire application and database modernization process”,  said Dan Sandlin, Google Cloud Data GTM Director at Google Cloud. “This enables organizations to make the most of digital technologies and provides a clear IT ecosystem transformation that allows our customers to build a flexible foundation for future innovation.”

Previously, migVisor focused on assessments of the source databases and the compatibility of customers’ existing database portfolio with cloud-centric database technologies. Coming this quarter, migVisor adds support for application assessments, augmenting its existing and class-leading capabilities in the database space. 

The addition of application modernization assessment functionality in migVisor, combined with EPAM’s certification and specialization in Google Cloud Data Management and hands-on engineering experience, strengthens EPAM’s position as a leader for large-scale digital transformation projects and migVisor as a trusted product for cloud migration assessments to Google Cloud customers. EPAM provides customers an end-to-end solution for faster and more cost-effective migrations.  Assessments that used to take weeks can now be completed in mere days. 

Within minutes of registering for an account, anyone can start using migVisor by EPAM to automatically assess applications and application code. Visit the migVisor page to learn more and sign up for your account.