When it comes to creating a business that can thrive in the digital age, the benefits of DevOps are clear. In a recent survey sponsored by Google Cloud, Harvard Business Review Analytic Services found that about two-thirds of the respondents who use DevOps have seen benefits that impact their bottom line, including increased speed to market (identified by 70% of the respondents), productivity (67%), customer relevance (67%), innovation (66%) and product/service quality (64%). Not only do these factors deliver profitability, but they’re precisely what cements a winning reputation among customers – the ultimate payoff.

We’ve seen these benefits within Google where DevOps helped us build secure products used and loved by billions of people across the world. However, as head of DevOps practices at Google, I know firsthand how daunting implementing a new model can be, even if it’s worthwhile. Intrigued but unsure of where to begin, customers constantly ask me: “How can I make DevOps a reality?”

Implementing DevOps can be really hard. Getting people to work differently doesn’t happen overnight. From our own journey toward embracing what are now known as DevOps practices, we learned seven critical lessons essential to adopting a DevOps model:

1. Pilot a small project. This provides a low-stakes opportunity to master key DevOps capabilities, such as building small, diverse teams with shared goal. A few small wins will provide evidence to the rest of organization that DevOps works. Soon others will want to follow suit.

2. Be an open-source player. Leveraging open-source tools and engaging in the community keeps you up-to-date on the best solutions and practices and attracts top talent. It also flattens your company’s learning curve and speeds up release cycles. According to a recent DORA study, 58% of businesses made extensive use of open source.

3. Embed security within software development process. By addressing potential security issues as early as possible, you’ll avoid pushing those issues out to production. Over half of participants in the Harvard Business Review Analytic Services survey look for holistic approaches to improve security while automating the DevOps toolchain. In addition, the recent DORA study also found that top performers who build security into software development conduct security reviews and complete changes in just days.

4. Apply DevOps best practices. Use Site Reliability Engineering (SRE) principles to help build collaboration, reduce waste, and increase efficiency. Also look for ways to implement end-to-end automation. Not only does automation enable higher productivity, it frees organizations up to focus on what really matters:  delivering value and driving performance.

5. Provide immersive training. People will only commit to organizational change when they understand its premise and are given the resources and opportunity to put new tech to work. That’s why three-quarters of the top-performing DevOps teams in the Harvard Business Review Analytic Services survey, as well as Google, provide immersive, hands-on DevOps coaching and training, such as code labs and quick-start projects.

6. Establish a no-blame culture. By running blameless post-mortem meetings in a safe environment built on trust, we learn from our mistakes. Because let’s face it, defects and coding errors happen when building software. By presenting mistakes as opportunities, you enable people to relate to one another and solve problems together, while ensuring that the same mistake won’t happen again. That’s how the DevOps model can evolve faster.

7. Build a culture that supports DevOps.  I’m underlining this because the rest is worthless without it. When people feel like they have each other’s backs, they’re more likely to take smart risks; more likely to create; more likely to move faster. Trust comes down to these principles:

• Data-driven decisions: Look at data from code, logs, and traces, and use that data to arrive at decisions.
  
• Transparency: Choose sharing over secrecy and siloing. Everyone sees the same data means everyone feels comfortable and confident.
  
• Shared goals: Constantly collaborate so developers and operators are working toward a common goal.
  

Those are the basics. Reading the Harvard Business Review Analytic Services survey in its entirety will help flesh out the details. The report is full of proven tactics used by the most successful DevOps-based businesses, as well as statistics that demonstrate why it’s a worthy investment. After you’ve digested those facts and figures, consider my own intangible observation: there’s something magical about understanding what makes people productive, collaborating with them, and then empowering them to deliver value. Hope you get as much out of this transformation as we did.

Hi there! I’m David Challoner from Access Site Reliability Engineering (SRE), here with Anthony Bushong from Developer Relations to talk about how Corp Eng is adopting Anthos Service Mesh internally at Google.

Corp Eng is Google’s take on “Enterprise IT”. A big part of the Corp Eng mission is running the first and third party software that powers internal business processes – from legal and finance to floor planning and even the app hosting our cafe menus – all with the same security or production standards as any of Google’s first party applications.

Googlers need to access these applications, which sometimes then need to access other applications or other Google Cloud services. This traffic can cross different trust boundaries which can trigger different policies.

Access SRE runs the systems that mediate this access, and we implemented Anthos Service Mesh as part of our solution to secure the way Googlers access these applications.

But why?

You can probably tell, but the applications Corp Eng is responsible for have disparate requirements. This often means that certain applications are tied to disparate infrastructure due to legal, business or technical reasons – which can be challenging when those infrastructures work and operate differently.

Enter Anthos. Google Cloud built Anthos to provide a consistent platform interface unifying the experience of working with apps on these varying underlying infrastructures, with the Kubernetes API at its foundation.

So when searching for the right tool to build a common authorization framework to mediate access to CorpEng services, we turned to Anthos – specifically Anthos Service Mesh, powered by the open-source project, Istio. Whether these services were deployed in Google Cloud, in Corp Eng data centers, or at the edge onsite at actual Google campuses, Anthos Service Mesh delivered a consistent means for us to program secure connectivity.

To frame the impact ASM had on our organization, it’s helpful to introduce the roles of the folks who manage and use it:

For security stakeholders, ASM provides an extensible policy enforcement point running next to each application capable of provisioning a certificate based on the identity of the workload and enforcing mandatory fine-grained application-aware access controls.

For platform operators, ASM is delivered as a managed product, which reduces operational overhead by providing out-of-the-box release channels, maintenance windows, and a published Service Level Objective(SLO).

For service owners, ASM enables the decoupling of their applications from networking concerns, while also providing features like rate limiting, load shedding, request tracing, monitoring, and more. Features like these were typically only available for applications that ran on Borg, Google’s first-party cluster manager that ultimately inspired the creation of Kubernetes.

In sum, we were able to secure access to a plethora of different services with minimal operational overhead, all while providing service owners granular traffic control.

Let’s see what this looks like in practice!

The architecture

In this flow, user access first reaches the Google Cloud Global Load Balancer [1], configured with Identity Aware Proxy (IAP) and Cloud Armor. IAP is the publicly available implementation of Google’s internal philosophy of BeyondCorp, providing an authentication layer that works from untrusted networks without the need for a VPN.

Once a user is authenticated, their request then flows to the Ingress Gateway provided by Anthos Service Mesh [2]. This provides additional checks that traffic flows to services only when the request has come through IAP, while also enforcing mutual TLS (mTLS) between the Anthos Service Mesh Gateway to the Corp services owned by various teams.

Finally, additional policies are enforced by the sidecar running in every single service Pod [3]. Policies are pulled from source control using Anthos Config Management[4], and are propagated to all sidecars by the managed control plane provided by Anthos Service Mesh[5].

Managing the mesh

If you’re not familiar with how Istio works, it follows the pattern of a control plane and a data plane. We talked a little bit about the data plane – it is made up of the sidecar containers running alongside all of our service Pods. The control plane, however, is what’s responsible for updating these sidecars with the policies we want to enforce:

Thus, it is critical for us to ensure that the control plane is healthy. This is where Anthos Service Mesh gives our platform owners a huge advantage with its support for a fully-managed control plane. To provision cloud resources, like many other companies, our organization uses Terraform, the popular open-source infrastructure as code project. This gave us a declarative and familiar means for provisioning the Anthos Service Mesh control plane.

First, you enable the managed control plane feature for GKE by creating the google_gke_hub_feature resource below using Terraform.

resource "google_gke_hub_feature" "feature_asm" {
name = "servicemesh"
location = "global"
provider = google-beta
}

Keep in mind that at publication time, this is only available via the google-bet provider in Terraform.

Once created, we then provision a ControlPlaneRevision custom resource in a GKE cluster to spin up a managed control plane for ASM in that cluster:

apiVersion: mesh.cloud.google.com/v1alpha1
kind: ControlPlaneRevision
metadata:
  
name: asm-managed
  
namespace: istio-system
spec:
  
type: managed_service
  
channel: regular

Using this custom resource, we are able to set the release channel for the ASM managed control plane. This allows for our platform team to define the pace of upgrades in accordance with our team’s needs.

In addition to managing the control plane, ASM also provides management functionality around the data plane to ensure each sidecar Envoy is kept up to date with the latest security updates and is compatible with the control plane – one less thing for service operators to worry about. It does this using Kubernetes Mutating Admission Webhooks and Namespace labels to modify our Pod workload definitions to inject the appropriate sidecar proxy version.

Syncing mandatory access policies

With the core Anthos Service Mesh components in place, our security practitioners can define consistent, mandatory security policies for every single GKE cluster, using Istio APIs.

For example, one policy is enforcing strict mTLS between Pods using automatically provisioned workload identity certificates. Earlier, we talked about how this is enforced between the Istio Gateway; that same policy enforces mTLS between all Pods in our cluster.

Another policy we implement is denying all egress traffic by default, requiring service teams to explicitly declare their outbound dependencies. The following is an example of using an Istio Service Entry to allow granular access to a specific external service – in this case, Google. This helps prevent unintended access to external services.

apiVersion: networking.istio.io/v1alpha3

kind: ServiceEntry
metadata:
  
name: google
spec:
  
hosts:
  - www.google.com
  
ports:
  - number: 443
    
name: https
    
protocol: HTTPS
  
resolution: DNS
  
location: MESH_EXTERNAL
EOF

These policies are automatically synced to all service mesh namespaces in each cluster using Anthos Config Management. By using our internal source control system as a source of truth, Anthos Config Management can sync and reconcile policies across all of our GKE clusters, ensuring that these policies are in place for every single one of our services. You can find more details about our implementation of Anthos Config Management here.

With this in place, our team plans on eventually migrating away from security automation that operates solely based on explicit IP, port and protocol policies.

Integration with Identity-aware Proxy

The publicly available version of the BeyondCorp proxy used by CorpEng is called Identity-aware Proxy (IAP), which offers an integration with Anthos Service Mesh. IAP allows you to authenticate users trying to access your services and apply Context-Aware-Access policies. This integration comes with two main benefits:

• Ensuring that user traffic to services in the service mesh only come through Identity-aware Proxy
• Enforcing Context-aware access (CAA) trust levels for devices, defined by multiple device signals we collect

Identity-aware Proxy allows us to capture this information in a Request Context Token (RCToken), which is a JSON Web Token (JWT) created by Identity-aware Proxy that can be verified by ASM. IAP inserts this JWT into the Ingress-Authorization header. Using Istio Authorization Policies similar to the following policy, any requests without this JWT are denied:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: iap-gateway-require-jwt
namespace: istio-system
spec:
selector:
matchLabels:
app: istio-iap-ingressgateway
action: DENY
rules:
- from:
- source:
notRequestPrincipals: ["*"]

Here is an example policy that requires a fullyTrustedDevice access level – this might be a device in your organization that is known to be corporate-owned, fully-updated, and running an IT-approved configuration :

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: require-fully-trusted-device
  namespace: fooService
spec:
  selector:
    matchLabels:
      app: fooService
  action: ALLOW
  rules:
  - from:
    - source:
       requestPrincipals: ["*"]
    when:
    - key: request.auth.claims[google.access_levels]
      values: ["accessPolicies/$orgId/accessLevels/fullyTrustedDevice"]

This allows our security team to not only secure service to service communications, or outbound calls from services, but also specifically require incoming requests come from trusted devices and authenticated users using a trusted device.

Enabling service teams

As an SRE, one of our priorities is ensuring Service-level indicators (SLIs), SLOs, and Service-level agreements (SLAs) exist for services. Anthos Service Mesh helps us empower service owners to do this for their services, as it exposes horizontal request metrics like latency and availability to all services in the mesh.

Before Anthos Service Mesh, each application had to export these separately (if at all). With ASM service owners can easily define their Service’s SLOs in the cloud console or via terraform using these horizontally exported metrics. This then allows us to integrate SLOs into our higher-level service definitions so we can enable SLO monitoring and alerting by default. You can see the SRE book for more details on SLOs and Error budgets.

The takeaway

ASM is a powerful tool that enterprises can use to modernize their IT infrastructure. It provides:

• A shared environment-agnostic enforcement point to manage security policy
• A unified way to provision identities, describe application dependencies

This also enables previously unheard of operational capabilities such as distributed tracing or incremental canary rollouts – which were difficult to find in the typical enterprise application landscape.

Because it can be incrementally adopted and composed with existing authorization systems to close gaps – barriers to adoption are low and we recommend you start evaluating it today!

Founded in 2006, PaGaLGuY started as a forum that enabled students, typically aged between 20 and 30 to discuss and seek advice on academic issues. By 2011, PaGaLGuY had increased its traffic to about 250,000 page views per month. The business is now one of India’s largest education networks and provides an app that users can download to their Android and iOS devices.

Over the past six years, PaGaLGuY has extended its service to include video advice from experts on education topics and grown the number of views of its pages to 1.5 million per day. As Head of Technology for the business, Sandeep Kalidindi has played a key role in ensuring PaGaLGuY is as engaging as possible to users. “Because the product is advertising based, the greater the user engagement, the greater the advertising revenue,” Kalidindi explains.

Google Cloud Platform Results

• Supported growth to 1.5 million page views per day and demand spikes that see requests increase from about 90 per second to about 1,200 per second
• Reduced API latency from about 1 second to about 40 milliseconds
• Reduced system administration time from three to four days per week to 30 minutes every two weeks

In 2015, PaGaLGuY’s senior management team decided to deliver an even more relevant experience for users of the education network. “The core thing we had to do was personalise the experience for each and every student that visited PaGaLGuY,” Kalidindi says. “So we had to capture each student’s data to customise what they see when they open the site.”

The business also found traffic to the network was straining its infrastructure. During demand peaks, created by exams involving as many as 5 million students, PaGaLGuY would be inaccessible for periods of 30 minutes to one hour. Furthermore, average API latency had climbed to an unacceptable 1 second, compromising performance.

PaGaLGuY needed to access extensive compute resources to undertake its planned change. Had the business relied on a physical technology architecture to undertake the transformation, it would have had to purchase capacity equivalent to 16 new servers. “There was no way with a small team we could grow to that extent in a short time,” Kalidindi says. “This was the right moment for us to explore cloud services.”

The business established two primary requirements the selected cloud service needed to meet. First, PaGaLGuY had to be able to scale the platform with costs rising only in proportion to the increase in resources consumed. Accordingly, the business would have to minimise the number of employees required to manage the cloud environment. Second, the platform had to give PaGaLGuY easy access to student data and the ability to undertake prompt, granular analysis.

PaGaLGuY reviewed available public cloud services and determined that Google Cloud Platform (GCP) was the best fit for its business. “Google Cloud Platform was considerably more mature than the alternatives, with a high degree of automation and a suite of managed services,” Kalidindi says. PaGaLGuY management then discussed with Google how to optimise cost, performance and availability of its personalised education network on GCP.

With assistance from Google and business transformation specialists Searce, PaGaLGuY was able to deliver the platform into production on GCP in 10 months. “Searce was very proactive in ensuring the environment met our needs and allowing us to gain priority access to Google services in development,” Kalidindi says. “Their team was integral to the success of the migration.”

PaGaLGuY has been running in production in GCP for two years. The education network’s GCP architecture comprises a scalable back-end built on Google App Engine; a managed environment for its containerised applications in Google Kubernetes Engine; messaging-oriented middleware through Google Cloud Pub/Sub; a relational database in Google Cloud SQL; a managed data analytics warehouse running in Google BigQuery; stream and batch data processing through Google Cloud Dataflow; and object storage in Google Cloud Storage.

PaGaLGuY has leveraged GCP services to break down its platform application from a monolithic build to a series of microservices running in Google App Engine that enable independent deployment cycles, minimise test and quality assurance overheads and provide clearer monitoring and logging.

Running on GCP has enabled PaGaLGuY to add new personalisation features and grow fourfold without having to add any new engineers or administrators to accommodate the increased traffic. The business has also used the platform to seamlessly collect and aggregate students’ data for analysis, reporting and delivering a more targeted user experience. Furthermore, PaGaLGuY has been able to provide its management team with direct access to Google BigQuery to scrutinise data rather than require them to wait at least a day to view reports created by the product or technology teams.

Support demand peaks of 1,200 requests per second

“Thanks to Google Cloud Platform, we can easily support demand peaks that see requests per second rise from an average 90 per second to about 1,200 per second for as long as 45 minutes,” Kalidindi says. Due to GCP’s scalability, PaGaLGuY can ensure its education network remains available and performance remains consistent during those periods.

Latency cut to 40 milliseconds

The business has also reduced average API latency from 1 second to about 40 milliseconds. Furthermore, using GCP has enabled PaGaLGuY to automate most of its processes and reduce system administration requirements from three to four days a week across its team members to about half an hour per week.

The performance of GCP has transformed PaGaLGuY’s culture and processes. “Once our team was exposed to Google Cloud Platform and understood the superiority of the platform, our mindset changed from ‘let us do everything on our own’ to ‘let us do what we do best’ and delegate the remainder,” Kalidindi says. The quality of the service provided by GCP means PaGaLGuY effectively considers the cloud provider as part of its team. “We are always eager to see what new services are being launched and are extremely excited about what Google Cloud Platform can provide as part of its roadmap.” he concludes.