Editor’s Note: In a previous blog, we discussed how home improvement retailer Lowe’s was able to increase the number of releases it supports by adopting Google’s Site Reliability Engineering (SRE) framework on Google Cloud. Lowe’s went from one release every two weeks to 20+ releases daily, helping meet its customer needs faster and more effectively. Today, the Lowe’s SRE team shares how they used SRE principles to decrease their mean-time-to-recovery (MTTR) by over 80 percent.

The stakes of managing Lowes.com have never been higher, and that means spotting, troubleshooting and recovering from incidents as quickly as possible, so that customers can continue to do business on our site. 

To do that, it’s crucial to have solid incident engineering practices in place. Resolving an incident means mitigating the impact and/or restoring the service to its previous condition. The average time it takes to do this is called mean time to recovery (MTTR). Tracking this metric helps us stay on top of the overall reliability of our systems at Lowe’s, while simultaneously improving the speed with which we recover. Our goal is to keep the MTTR metric as low as possible, so that failures don’t negatively impact our business. Here are the four areas we addressed to drive holistic improvement in our MTTR.

Lowe’s incident reporting process

To reduce MTTR, we created a seamless incident reporting process following SRE principles. Our incident reporting process is a workflow that starts at the time an incident occurs, and ends with an SRE captain who closes the action items after a postmortem report. With this approach, we are able to limit the number of critical incidents. The reporting process involves three core components: monitoring, alerting, and blameless postmortems.

Monitoring and alerting

Having proper monitoring and alerting in place is crucial when it comes to incident management. Monitoring and alerting tools let you detect issues as soon as they occur, and notify the right person in the shortest possible time to take action. From a measurement standpoint, we track this as our mean time to acknowledge (MTTA). This is the average time it takes from when an alert is triggered, to when work on the issue begins.

At the time of an incident, our monitoring and alerting tools notify the on-call SRE first responder via PagerDuty in the form of a phone call, text message and email. Our SRE software engineering team has done a lot of automation to enable various Service Level Indicator (SLI) alerts and Service Level Agreement (SLA) notifications. The on-call SRE then initiates a triage call with our service/domain stakeholders to resolve the incident. As a result, we reduced our MTTA from 30 minutes in 2019, to one minute – a 97 percent decrease. 

Blameless postmortems: learning from incidents

A postmortem is a written record of an incident, its impact, the actions taken to resolve it, the root cause and the follow-up actions to prevent the incident from recurring (see example here). A blameless postmortem builds on that and is a core part of an SRE culture, and our culture at Lowe’s. We ensure that individuals are not singled out, and the outcome for all postmortems are directed toward learnings and process improvement.

For us, the postmortem process is the biggest part of our incident workflow. When an SRE creates a new postmortem report, the first step is to conduct a postmortem session with domain stakeholders to review the report. The postmortem then goes into the review stage and gets reviewed by more stakeholders in our weekly postmortem meeting. In the final stage of this process, the SRE captain will close the report once everyone in the weekly meeting agrees that the report is complete.

To conduct a successful postmortem, it is critical to keep the focus on identifying gaps and issues with the system and operations processes, rather than an individual, and generate concrete actions to address the problems we’ve identified. To ensure this, we follow a couple of best practices:

1. We start by gathering the facts from the person who identified the problem, and each SLI owner has to identify a gap or the next SLI upstream owner who created the impact for them.
2. Every SLI owner is provided full opportunity to present their case, and identifying the issue is done as a community exercise. 
3. Once action items and process changes are identified, an owner is nominated to complete the actions, or they will volunteer. 
4. For easy reference, we publish and store postmortems in our incident knowledge base. This process helps SREs continuously improve as future incidents arise. 

Continuous Improvement 

Encouraging a culture of honest, transparent and direct feedback that you need for blameless postmortems is often an iterative process that needs sponsorship from executives, empowering incident captains to lead the entirety of the discussion and outcomes. Running successful postmortems, and completing action items from them, needs to be recognized and accounted for in SRE performance objective assessment. As shared in Google’s SRE book, the best practice is to ensure that writing effective postmortems is a rewarded and celebrated practice, with leadership’s acknowledgement and participation. This is possibly the hardest part to accomplish in an effective postmortem during a cultural transformation unless you have full buy-in from leadership.

However, it’s all well worth it. This process is a key part of how we were able to improve our MTTR over time—from two hours in 2019 to just 17 minutes! 

Our SRE incident reporting process has also transformed how our company solves issues. By streamlining this workflow from alerting, to solving an issue, to blameless postmortems, we have reduced our MTTR by 82 percent and our MTTA by 97 percent. Most importantly, our team is learning from every incident and becoming better engineers as a result. Visit the SRE Google Cloud website to learn more about implementing SRE best practices in the cloud.

Acknowledgement

Special thanks to Rahul Mohan Kola Kandy, Vivek Balivada, and the Digital SRE team at Lowe’s for contributing to this blog post.

Digital tools such as cloud computing are fueling economic transformation across the US, including Ohio. Google continues to invest across cities and communities in Ohio, bringing over 200 jobs to the state, and helping provide $12.85 billion of economic activity for tens of thousands of Ohio businesses, nonprofits, publishers, creators and developers. To further accelerate the transformation of all Ohioan businesses and technologists, we’re thrilled to announce our newest Google Cloud region in Columbus, Ohio is open. The Columbus cloud region brings a second region to the Midwest, the 10th region to North America, and grows our global cloud region count to 33.

A region for the Buckeye State

Now open to Google Cloud customers, the Columbus region (us-east5) provides you with the speed and availability you need to innovate faster, build high-performing applications, and serve local customers — all on the cleanest cloud in the industry. Additionally, the region gives you added flexibility to distribute your workloads across the central, midwest, and eastern US.

The Columbus region offers immediate access to three zones, for high availability workloads, and our standard set of products, including Compute Engine, Google Kubernetes Engine, Cloud Storage, Persistent Disk, CloudSQL, and Cloud Identity. Our private backbone connects Columbus to our global network more quickly and securely. In addition, you can integrate your on-premises workloads with our new region using Cloud Interconnect. This means that Columbus-based customers can expand globally from their front door, and those based outside the region can more easily reach their users in the Midwest.

What customers are saying

Industries including retail, financial services, and IT are investing in Columbus. Organizations across these verticals have turned to the Google Cloud to innovate faster and help solve their most complex challenges

“As Wendy’s continues to innovate in new ways to create fast, frictionless, and fun interactions that redefine the way customers visit and enjoy our restaurants, our partnership with Google Cloud is a key enabler to delivering on our AI/ML and data analytics strategies. The proximity of the new Google Cloud region to Wendy’s headquarters provides the ability for us to move and scale quickly as business needs evolve. Additionally, Google Cloud’s investment in Columbus positions central Ohio as a true technology hub, which further boosts Wendy’s and other regional employers’ ability to recruit innovative talent,” said Kevin Vasconi, Chief Information Officer, Wendy’s.

“Huntington National Bank’s API Architecture is a central component to our growth and technology strategy. As our business segments grow from an offering and geographic perspective, we must evolve our technology to provide the optimal experience for our customers and our partners. Collaborating directly with Google Cloud on the build out of their cloud region in Central Ohio, provides the access our technology teams need to innovatively scale our infrastructure to meet the demands of our business with increased availability, lower latency, and greater resiliency,” said Geoff Preston, Chief Architect, Huntington National Bank.

“Google Cloud has been instrumental in our ability to scale and optimize data management and compute resources. We prioritize scale, elasticity and resilience in cloud services and Google Cloud delivers all three globally and locally. With Google Cloud security, we can efficiently process the quantities of application data required to accelerate alert detection and reduce response times for the critical infrastructure our customers depend on to enable the continuity of their vital applications.” said Sheryl Haislet, Chief Information Officer at Vertiv, a global provider of critical digital infrastructure and continuity systems headquartered in Columbus, Ohio, that leverages Google Cloud solutions to provide resilience for its operations and to better support customers.

“The addition of the new cloud region in Ohio continues to demonstrate Google Cloud’s commitment to the enterprise space and their presence in the region,” said Chris Delong, Chief Technology Officer, Designer Brands Inc. / DSW

What’s next

We are thrilled to welcome you to our new cloud region in Columbus, and eagerly await to see what you build with our platform. Register here for our Cloud Study Jam in June – an event for local developers to get hands-on training with Google Cloud. Stay tuned for more region announcements and launches this year, including our next U.S. region in Dallas, TX. And for more information, contact sales to get started with Google Cloud today.

Editor’s note: 5G is much more than a network—it’s a platform for innovation with the ability to provide immediate global scale and enable use cases that we haven’t even dreamed of yet. But to achieve this, industry players must come together to drive this growing ecosystem. Today, Erik Ekudden, CTO of Ericsson, and Bikash Koley, VP of Global Networking at Google Cloud, share their insights into the potential of edge and 5G.

Experts consider 2021 to be the year that serves as the inflection point between network readiness and 5G availability. However, communications service providers (CSPs) are still faced with the task of modernizing their networks, systems, and infrastructure to maximize the potential of 5G for themselves and for the enterprise customers they serve. As you weigh whether to tackle this challenge, let’s first examine what’s different about 5G and how CSPs can best leverage 5G and the edge together as an even stronger platform for innovation than just 5G alone.

With 5G, applications and mobile networks are no longer isolated

Let’s first address what 5G brings to the table. Faster speeds and lower latency are expected of course, and throughout the evolution from 2G to 3G to 4G, there’s been a step function in performance improvements with each of these. However, with every generation, applications and networks have been like ships passing in the night. Networks have been unaware of what the applications have been doing and applications have been guessing what the network was capable of. 

What’s different this time around is that there is some real innovation happening with the development and rollout of 5G itself. The network is on a path to become more accessible via APIs so that applications can call on and consume what they need from the networks in a more programmable way. This sets us up for more open architectures and ecosystems.

5G leverages a more open architecture 

One key difference with 5G as compared to prior generations is that it’s the most open and flexible network architecture the CSP industry has seen. This is thanks to its service-based approach and the decoupling of hardware and software components. CSPs are now running core network elements in the public cloud, private cloud, hybrid cloud, and even multi-cloud, which was unthinkable even five years ago.

The flexibility of this more open architecture allows us to push the cloud to the edge while still being able to manage it from a single pane of glass. This is a huge leap forward from traditional networks, which have been domain-specific, managed in silos, and with slow service creation and delivery. Now, hyperscale cloud vendors and network equipment providers are offering solutions that help CSPs break down these silos to enable more flexible, automated networks with improved orchestration, visibility, and control across multi-vendor, multi-cloud and hyperscale cloud-provider environments. In addition, the separation of hardware and software provides a much more flexible and cost-effective way to upgrade from one network generation to the next.

To provide solutions that are relevant to enterprises, CSPs must offer capabilities beyond connectivity. Enterprise service orchestration, including exposure of network assets and network slicing, are foundational capabilities to provide value to the application ecosystem and be in control of the network and the delivered services.

Combining 5G and edge to help industries reimagine user experiences

This convergence of compute, storage and networking at the edge coming together for the first time will enable CSPs and enterprises to offer their customers completely reimagined user experiences. Consider, for example, how the automotive industry might enhance how customers shop for a car. As part of Fiat Chrysler Automobiles’ Virtual Showroom at the recent CES 2021 event, consumers were able to experience the innovative new 2021 Jeep Wrangler 4xe by scanning a QR code with their phones, and then see an Augmented Reality (AR) model of the Wrangler right in front of them—virtually placed on their own driveway or in any open space.

By rendering the model in Google Cloud, then streaming it to mobile devices, visitors could also see what the car looked like from any angle, in different colors, and even step inside to see the interior in incredible detail. That is the true digitization of an industry segment and highlights the device-to-network-to-edge-to-cloud application relationship and how it can impact the user experience.

A programmable network unlocks more application use cases

The programmability of the 5G network will truly enable application developers to utilize all the benefits of the underlying network. Programmability supports ease of use and enables CSPs, integrated software vendors (ISVs) and the ecosystem to have the right network-level APIs exposed so that applications can be optimized based on the network behavior and vice versa. Imagine, for instance, automatically pushing applications from a cloud region to the edge based on network latency and performance metrics.

Finally, 5G’s programmability is also about having the right tools available for developers to build and integrate applications on the network with zero-touch onboarding and validation. With this, we’ve come to the point where the network is now a “platform” for application innovation. 

5G and edge will be all about the ecosystem

One thing is for certain: the shift to 5G will place tremendous focus on the ecosystem, and it needs to be an ecosystem that includes CSPs, public cloud providers, application developers and technology providers, all coming together to optimize the user experiences across industry applications. For instance, Google Cloud and Ericsson recently announced our partnership to deliver 5G and edge cloud solutions for CSPs and enterprise.  In addition, Google Cloud is also teaming up with popular ISVs to deliver more than 200 edge applications from 30-plus partners, all running on our cloud.  

With collaboration across ISVs, cloud providers and network equipment providers, we are enabling the rapid delivery and deployment of new vertical services and applications, leveraging capabilities like Anthos, artificial intelligence (AI), and machine learning (ML), as well as multi-vendor, multi-cloud and hyperscale cloud provider service orchestration and global edge networks such as those provided by Google and telecom service providers.

As members of the technology ecosystem, we talk about compute, storage and networking, but when it comes down to it, it’s about optimally placing these resources—whether in the cloud, at the provider core, at the edge or anywhere in between—to maximize the end-user experience. The openness and programmability of 5G lends itself to collaboration like never before. We predict that in 2022 and beyond, it will be all about the ecosystem coming together to leverage 5G and the edge to build innovations that we can’t yet imagine.

To learn more, watch the full Ericsson 5G Things CTO Focus fireside chat, where we discuss these topics and more.

To fuel the breakthroughs of the future, today’s researchers need easy access to the most innovative cloud technologies. That’s why we are delighted to announce the latest expansion of Google Cloud research credits beyond its previous scope of government and academic research institutions to researchers at nonprofit institutions. This initiative is part of our ongoing efforts to make the advantages of cloud research available to more people everywhere–and to drive more discoveries faster. 

To serve researchers outside of academia, we are now offering nonprofit institutions access to Google’s research credit program. Leading centers of scientific knowledge like the New York Genome Center, Scripps Research Institute, the Vector Institute in Toronto, the Ontario Institute for Cancer Research, and the Allen Institute for AI (AI2) have already benefited from the efficiencies of managing and analyzing their datasets on Google Cloud.

Michael Schmitz, Director of Engineering at AI2, reports that “Google Cloud has been essential for our research program. While we have an on-premise installation for custom hardware, the ability to use Google Cloud for isolated workstations (where researchers always have guaranteed access to a development environment) and bursting with increased need (such as our summer intern surge or a hyperparameter tuning just before our deadline) has been a critical part of conducting our research.”

Also, to better support Ph.D. students, we are updating the terms of Google Cloud credits for use toward advanced research. Now, Ph.D. students can apply for up to $1,000 grants that are renewable over the five years of their programs. We know that the researchers of the future need the best resources now.

Hassaan Maan, Ph.D. student at the University of Toronto/Vector Institute, says “when the pandemic started, I began working on a web-based platform to help researchers analyze genome sequences of the COVID-19 virus in real time and compare with data from labs around the world. Deploying the Covid Genotyping Tool (CGT) platform was challenging, as I had minimal experience hosting web applications and we required a scalable and fine-tuned deployment. Google Cloud’s comprehensive user interface and thorough documentation made for a seamless deployment, and I was able to get the platform up and running in less than two weeks. Google Cloud was a huge factor in the success of this project, and I hope to be able to use it further during my Ph.D. studies.”

To ramp up your own research project with Google Cloud, apply now for free credits in selected countries. To supplement your own learning on Google Cloud, sign up to earn four Google Cloud skills badges that you can share on your social platforms.