How Lowe’s SRE Team Decreases Mean-time-to-recovery (MTTR)

3367
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
Editor’s Note: In a previous blog, we discussed how home improvement retailer Lowe’s was able to increase the number of releases it supports by adopting Google’s Site Reliability Engineering (SRE) framework on Google Cloud. Lowe’s went from one release every two weeks to 20+ releases daily, helping meet its customer needs faster and more effectively. Today, the Lowe’s SRE team shares how they used SRE principles to decrease their mean-time-to-recovery (MTTR) by over 80 percent.
The stakes of managing Lowes.com have never been higher, and that means spotting, troubleshooting and recovering from incidents as quickly as possible, so that customers can continue to do business on our site.
To do that, it’s crucial to have solid incident engineering practices in place. Resolving an incident means mitigating the impact and/or restoring the service to its previous condition. The average time it takes to do this is called mean time to recovery (MTTR). Tracking this metric helps us stay on top of the overall reliability of our systems at Lowe’s, while simultaneously improving the speed with which we recover. Our goal is to keep the MTTR metric as low as possible, so that failures don’t negatively impact our business. Here are the four areas we addressed to drive holistic improvement in our MTTR.
Lowe’s incident reporting process
To reduce MTTR, we created a seamless incident reporting process following SRE principles. Our incident reporting process is a workflow that starts at the time an incident occurs, and ends with an SRE captain who closes the action items after a postmortem report. With this approach, we are able to limit the number of critical incidents. The reporting process involves three core components: monitoring, alerting, and blameless postmortems.
Monitoring and alerting
Having proper monitoring and alerting in place is crucial when it comes to incident management. Monitoring and alerting tools let you detect issues as soon as they occur, and notify the right person in the shortest possible time to take action. From a measurement standpoint, we track this as our mean time to acknowledge (MTTA). This is the average time it takes from when an alert is triggered, to when work on the issue begins.
At the time of an incident, our monitoring and alerting tools notify the on-call SRE first responder via PagerDuty in the form of a phone call, text message and email. Our SRE software engineering team has done a lot of automation to enable various Service Level Indicator (SLI) alerts and Service Level Agreement (SLA) notifications. The on-call SRE then initiates a triage call with our service/domain stakeholders to resolve the incident. As a result, we reduced our MTTA from 30 minutes in 2019, to one minute – a 97 percent decrease.
Blameless postmortems: learning from incidents
A postmortem is a written record of an incident, its impact, the actions taken to resolve it, the root cause and the follow-up actions to prevent the incident from recurring (see example here). A blameless postmortem builds on that and is a core part of an SRE culture, and our culture at Lowe’s. We ensure that individuals are not singled out, and the outcome for all postmortems are directed toward learnings and process improvement.
For us, the postmortem process is the biggest part of our incident workflow. When an SRE creates a new postmortem report, the first step is to conduct a postmortem session with domain stakeholders to review the report. The postmortem then goes into the review stage and gets reviewed by more stakeholders in our weekly postmortem meeting. In the final stage of this process, the SRE captain will close the report once everyone in the weekly meeting agrees that the report is complete.
To conduct a successful postmortem, it is critical to keep the focus on identifying gaps and issues with the system and operations processes, rather than an individual, and generate concrete actions to address the problems we’ve identified. To ensure this, we follow a couple of best practices:
- We start by gathering the facts from the person who identified the problem, and each SLI owner has to identify a gap or the next SLI upstream owner who created the impact for them.
- Every SLI owner is provided full opportunity to present their case, and identifying the issue is done as a community exercise.
- Once action items and process changes are identified, an owner is nominated to complete the actions, or they will volunteer.
- For easy reference, we publish and store postmortems in our incident knowledge base. This process helps SREs continuously improve as future incidents arise.
Continuous Improvement
Encouraging a culture of honest, transparent and direct feedback that you need for blameless postmortems is often an iterative process that needs sponsorship from executives, empowering incident captains to lead the entirety of the discussion and outcomes. Running successful postmortems, and completing action items from them, needs to be recognized and accounted for in SRE performance objective assessment. As shared in Google’s SRE book, the best practice is to ensure that writing effective postmortems is a rewarded and celebrated practice, with leadership’s acknowledgement and participation. This is possibly the hardest part to accomplish in an effective postmortem during a cultural transformation unless you have full buy-in from leadership.
However, it’s all well worth it. This process is a key part of how we were able to improve our MTTR over time—from two hours in 2019 to just 17 minutes!
Our SRE incident reporting process has also transformed how our company solves issues. By streamlining this workflow from alerting, to solving an issue, to blameless postmortems, we have reduced our MTTR by 82 percent and our MTTA by 97 percent. Most importantly, our team is learning from every incident and becoming better engineers as a result. Visit the SRE Google Cloud website to learn more about implementing SRE best practices in the cloud.
Acknowledgement
Special thanks to Rahul Mohan Kola Kandy, Vivek Balivada, and the Digital SRE team at Lowe’s for contributing to this blog post.
Intel-Google Collaboration Brings Edge Computing on Factory Floors: Hannover Messe 2022

3225
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
The typical smart factory is said to produce around 5 petabytes of data per week. That’s equivalent to 5 million gigabytes, or roughly 20,000 smartphones.
Managing such vast amounts of data in one facility, let alone a global organization, would be challenging enough. Doing so on the factory floor, in near-real-time, to drive insights, enhancements, and particularly safety, is a big dream for leading manufacturers. And for many, it’s becoming a reality, thanks to the possibilities unlocked with edge computing.
Edge computing brings computation, connectivity, and data closer to where the information is generated, enabling better data control, faster insights, and actions. Taking advantage of edge computing requires the hardware and software to collect, process, and analyze data locally to enable better decisions and improve operations.
At Hannover Messe 2022, Intel and Google Cloud will demonstrate a new technology implementation that combines the latest generation of Intel processors with Google Cloud’s data and AI expertise to optimize production operations from edge to cloud. This proof-of-concept project is powered by the Edge Insights for Industrial platform (EII), an industry-specific platform from Intel; and a pair of Google Cloud solutions: Anthos, Google Cloud’s managed applications platform, and the newly-launched Manufacturing Data Engine.
Edge computing exploits the untapped gold mine of data sitting on-site and is expected to grow rapidly. The Linux Foundation’s “2021 State of the Edge” predicts that by 2025, edge-related devices will produce roughly 90 zettabytes of data. Edge computing can help provide greater data privacy and security, and can accomodate the reduced bandwidth needs between local storage and the cloud.
Imagine a world in which the power of big data and AI-driven data analytics is available at the point where the data is gathered to inform, make, and implement decisions in near real-time.
This could be anywhere on the factory floor, from a welding station to a painting operation or more. Data would be collected by monitoring robotic welders, for example, and analyzed by industrial PCs (IPCs) located at the factory edge. These edge IPCs would detect when the welders are starting to go off spec, predicting increased defect rates even before they appear, and adding preventive maintenance to correct the errors without any direct intervention. Real time, predictive analytics using AI could substantially prevent defects before they happen. Or the same IPCs could use digital cameras for visual inspection to monitor and identify defects in real-time, allowing them to be addressed quickly.
Edge computing has powerful potential applications in assisting with data gathering, processing, storage and analysis in many manufacturing sectors, including automotive, semiconductor and electronics manufacturing, and consumer packaged goods. Whether modeling and analysis is done and stored locally or in the cloud, or is predictive, simultaneous, or lagged, technology providers are aligning to meet these needs. This is the new world of edge computing.
The joint Intel and Google Cloud proof of concept aims to extend the Google Cloud capabilities and solutions to the edge. Intel’s full breadth of industrial solutions, hardware and software, are coming together in this edge-ready solution, encompassing Google Cloud industry-leading tools. The concept shortens the time to insights, streamlining data analytics and AI at the edge.

The Intel-Google Cloud proof of concept demonstrates how manufacturers can gather and analyze data from over 250 factory devices using Manufacturing Connect from Google Cloud, providing a powerful platform to run data ingestion and AI analytics at the edge.
In this demonstration in Hannover, Intel and Google Cloud show how manufacturers can capture time-series data from robotic welders to inspect welding quality and show how predictive analytics can benefit the factory operators. In addition, the video and image data is captured from a factory camera to show how visual inspection can highlight anomalies on plastic chips with model scoring. The demo also features zero-touch device onboarding using FIDO Device Onboard (FDO) to illustrate the ease with which additional computers could be added to the existing Anthos cluster.
By combining Google Cloud’s expertise in data, AI/ML and Intel’s Edge Insight’s for Industrial platform that was optimized to run on Google Anthos, manufacturers can run and manage their containerized applications at the edge, in on-premise data center, or in public clouds using an efficient and secure connection to the Manufacturing Data Engine from Google Cloud. It forges a complete edge-to-cloud solution.
Simplified device onboarding is available using Fido Device Onboard (FDO)—an open IoT protocol that brings fast, secure, and scalable zero-touch onboarding of new IoT devices to the edge. FDO allows factories to easily deploy automation and intelligence in their environment without introducing complexity into their OT infrastructure.
The Intel-Google Cloud implementation can analyze that data using localized Intel or third-party AI and machine learning algorithms. Applications can be layered on the Intel hardware and Anthos ecosystem, allowing customized data monitoring and ingestion, data management and storage, modeling, and analytics. This joint PoC facilitates and support improved decision making and operations, whether automated or triggered by the engineers on the front lines.
Intel collaborates with a vibrant ecosystem of leading hardware partners to develop solutions for the industrial market by using the latest generation of Intel processors. These processors can run data intensive workloads at the edge with ease.

Putting data and AI directly into the hands of manufacturing engineers can improve quality inspection loops, customer satisfaction, and ultimately the bottom line.
The new manufacturing solutions will be demonstrated in person for the first time at Hannover Messe 2022, May 30–June 2, 2022. Visit us at Stand E68, Hall 004, or schedule a meeting for an onsite demonstration with our experts.
Simplifying Payments for SMBs: Helcim’s Transformational Approach

1459
Of your peers have already read this article.
6:00 Minutes
The most insightful time you'll spend today!
Small and medium-sized businesses and enterprises are the backbone of the US economy generating more than 44% of GDP1. Yet these organizations are still underserved when it comes to online financial tools—and dealing with payments is no exception. Complaints include hidden fees, limited capabilities, long-term lease agreements, and poor customer service.
These are the issues that we wanted to solve when we launched Helcim in 2020 in Calgary (Alberta, Canada). We provide a payment service that offers low rates through our interchange plus pricing model, no monthly fee for the core payments offering, and numerous payment options, as well as simple, affordable hardware such as card readers.
Our digital-first approach makes it easy for owners of small and medium-sized businesses to get started. Online sign-up means no paperwork and near instant access to Helcim’s software and all-in-one platform experience. Merchants can choose from a range of payment solutions from the Helcim app including in-person payments, SMS payment requests, online invoices with pay now buttons, and more.
To achieve our goals, we built most of the business systems and processes in-house including our technology stack, financial partnerships, marketing, and everything in between. This is something that few startups would dare to do, but it enabled us to build a payments platform offering the rich capabilities and performance SMBs really need.
Taking control with the cloud
Before migrating our infrastructure to Google Cloud, it was hosted at two colocation data centers in Calgary. This model served us well, but as we grew, most of our hardware needed to be replaced to maintain service security and performance.
A successful round of Series A funding also impacted our trajectory. Giving us the fuel we needed to scale and innovate faster. When we considered the choice between making a large capital investment in our existing environment, or to transition to the cloud, the decision was clear: The cloud was the way to go.
We looked at other big names in cloud hosting and tested another platform. But Google Cloud is by far the best environment for us. It’s much easier for a lean technology team to manage, as we embark on our first cloud strategy. It also offers all the tools and advanced machine learning capabilities we need.
Google Cloud also comes with the backing of Alphabet, a business that in the past five years has spent more on research and development than any other organization in the S&P 5002. The ability to engage the Google Workspace account team for guidance to improve our everyday processes was another bonus.
A variety of investment and training programs from Google Cloud also influenced our decision. We participated in Google for Startups Accelerator Canada which gave us access to Google Cloud experts across all our technology domains. This helped accelerate our infrastructure migration while ensuring we optimized every service from day one. We were also eligible for $100,000 USD of Google Cloud credits, covering our Google Cloud costs which helped us get everything up and running cost-effectively. The partnership with the wider Google team has been extremely impactful as we stand everything up for our business.
But ultimately, it’s the sheer depth and breadth of the Google Cloud environment that makes the difference. Here are the Google Cloud tools we currently have at Helcim.
We chose BigQuery because it is easy to aggregate new data and apply the right access controls. It also delivers outstanding performance running our analytics workloads.
Vertex AI makes the deployment of new models exponentially faster, and with Google Cloud, we can do most of the work through containers instead of proprietary tooling.
GKE has enabled us to migrate to a fully managed cloud service and avoid a lift-and-shift exercise that would have prevented us from getting the full benefits of a containerized infrastructure. Running a new GKE environment also enabled us to significantly reduce platform latency by 20%-50%. We can now deploy projects in hours instead of days.
We updated our centralized file system to use Cloud Storage, which is faster and more scalable.
All of our software today is built on top of MySQL, so CloudSQL was the natural choice for cloud database management.
Cloud Run is more flexible than other serverless tools and was an easy way to reduce the burden of managing some of our services.
Boosting performance across the business
By migrating to Google Cloud, we transformed our application performance. We were able to decouple the infrastructure between systems and use modern server hardware for our compute and database instances. With very little change to our code, we saw a 50%+ increase in the speed of our entire platform.
Giving developers more control of the technology running their systems via containers means that they can enhance systems through more frequent language updates and by deploying new technologies to optimize workloads.
We can more closely monitor our systems to diagnose issues and fix their root cause faster. Being able to quickly add resources gives us further options if we experience platform latency or increased traffic.

Maintaining momentum with machine learning
Leveraging Vertex AI has significantly reduced the time it takes the team to build and deploy new machine learning models. Greater agility in our data stack has also freed up time for exploratory work in the data team. For instance, by creating low fidelity proxy data for human behavior in the application process, we created a new model that will reduce the number of manually reviewed batches by more than 10%.
We’ve also been able to improve our deployment process and the time to rollback. As a result, breaking changes in production have been reduced from more than five minutes to less than 30 seconds.
Thanks to BigQuery, we can make better use of data to support key business decisions. Previously it was hard to aggregate data from different sources and while maintaining our strict requirements for customer data confidentiality. We also needed specialist SQL knowledge to consume it. By investing in a more modern data stack, the availability of trusted data across the organization has increased exponentially.
We’ve also overcome the constraints imposed by static hardware environments especially when maintaining a high-availability configuration between two locations. With Google Cloud, we’re no longer constrained by such a rigid arrangement and the deployment velocity of new infrastructure tooling has been reduced from months to days.
Security is another area where Google Cloud excels. From hackers and fraudsters to bots and web attacks, it protects our users, applications, and data, while facilitating compliance with local and regional authorities. We can also integrate more easily with our security partners ensuring that we can empower our team to stay ahead of cyber criminals and other external threats.
Building the payments platform for the future, today
When we look to the future, Google Cloud opens the door to dozens of opportunities to widen our appeal to SMBs while remaining competitive. Its advanced infrastructure for cloud computing, data analytics and ML supports our roadmap to profitability and will help us attract future rounds of funding.
Above all it provides a foundation for growth. We grew 400% in 2021 and raised more capital in the spring of 2022 to grow even faster. In 2022 we were also listed as one of the top payments processors by industry publications such as Nerdwallet and Merchant Maverick. With Google Cloud, we can build on this success, continue to innovate, and help our SMB customers take their payments and e-commerce strategies to the next level.
If you want to learn more about how Google Cloud can help your startup, visit our page here to get more information about our program, and sign up for our communications to get a look at our community activities, digital events, special offers, and more.
To learn more about Google for Startups Accelerators and to apply to a program in your region, visit the website here.
1. Small Businesses Generate 44 Percent Of U.S. Economic Activity
2. Alphabet: Big Value In Big Tech
Start-up Paves Way for More Inclusive Clinical Research: Honoring Black Founders of Acclinate with Google Cloud

8414
Of your peers have already read this article.
4:00 Minutes
The most insightful time you'll spend today!
Editor’s note: February is Black History Month—a time for us to come together to celebrate the diverse set of experiences, perspectives and identities that make up the Black experience. Over the next few weeks, we will highlight Black-led startups and how they use Google Cloud to grow their businesses. Today’s feature highlights Acclinate and its founders, Del and Tiffany.
As patients, as caregivers, and as parents taking our own children to the doctor, we want recommended medications to be safe and effective. It’s a right everyone deserves.
It’s known that certain medications don’t work in the same way in all populations. For example, Albuterol, a medication often prescribed for asthma, is less effective in 67% of all Puerto Ricans and 47% of Black Americans. These problems—which can have deadly consequences—result from historically limited diversity in pharmaceutical clinical trials.
We founded our startup Acclinate to integrate culture and technology to achieve more inclusive clinical research. Help pharmaceutical companies and healthcare organizations access and engage communities of color so research is more inclusive.

Bridging the health equity gap by building trust
It’s important that health research organizations access and engage communities of color so their efforts reflect all the people they serve. Take a disease like diabetes, which affects a significantly higher proportion of Black Americans. When you look at even recent clinical trials for diabetic drugs, the representation of Black Americans among participants is only in the low single digits, despite comprising 13 percent of the U.S. population and more than 40 percent of diabetes patients in this country. Industry leaders have been aware of the lack of diversity issue, but some have chosen to ignore it or brush it aside. The biggest problem, in our opinion, is that there have been no penalties for not achieving higher diversity figures in clinical trials, and only minor financial repercussions to pharma/biotech companies when their treatments either do not work across all groups once approved, or there is a lack of uptake by all groups due to the lack of testing in those groups. The lack of clinical trial diversity has adversely impacted the reputation of the industry and the ability to recruit diverse populations in the future.
Acclinate integrates culture and technology to promote diverse patient representation in medical research. Our approach is not transactional. We build trust through our #NOWINCLUDED community, which is an ongoing, ever-expanding digital platform that educates and engages with communities of color on health issues.
#NOWINCLUDED includes a website app, and social media presence where members can learn information about diseases, particularly those with greater negative impacts on people of color, such as cancer, diabetes, and cardiovascular diseases. Members can share stories and ask questions. By providing access to trusted resources about these health issues and the latest clinical research, we empower Black people to take control of their health and consider participating in research that is shaping the future of healthcare.
For healthcare-related organizations, we offer the opportunity to better understand the attitudes, aspirations, and unmet needs of underrepresented minority communities. Data from #NOWINCLUDED feeds our HIPAA-compliant SaaS platform, e-DICT™ (Enhanced Diversity in Clinical Trials), which uses predictive analytics and machine learning to identify individuals matching the requirements and most likely to be receptive to participation in a particular clinical trial.
Acclinate scales its platform with Google Cloud
We rely on Google Cloud services, including Vertex AI, to know whom to ask, when to ask, and how to ask for clinical trial participation. With Vertex AI, we enjoy a unified platform for developing our artificial intelligence models, including tools for preparing and storing our datasets. We can easily train and compare models using AutoML, which requires minimal ML expertise or effort with its intuitive graphical interface. This allows us to leverage more than ten ordinal and categorical data points to determine in real time a community member’s likelihood to enroll, which we call our Participation Probability Index (PPI). Our models evolve in an iterative process the more we interact with, and learn about, our community members.
We follow the pay-per-use Google Cloud Platform architecture model using serverless technology, which helps reduce infrastructure management costs and lets us focus on product development and engaging with communities across the U.S.
CloudSQL, a fully managed relational database service, integrates easily with BigQuery so we can glean insights for our clients in real time, all with Google Cloud’s robust security, governance, and reliability controls. Virtual Private Cloud (VPC) gives us scalable and flexible networking for our cloud-based resources and services. We also use Identity and Access Management (IAM) to simplify oversight of Google Cloud resource permissions for different user groups and roles, with appropriate security protections.
API Gateway manages our APIs using Cloud Functions, which both use consumption-based pricing, plus give our developers consistent and highly secure access to our services through a well-defined REST API. We use Memorystore for Redis to reduce platform latency. This is done with a fully managed service powered by the Redis in-memory data store, which builds application caches for fast data access. All of this comes together to provide an outstanding experience for our platform’s users and contributors.
- DSC00049.JPG
- DSC00010 (2).JPG
- pitch del tiff.jpg
Expanding influence with the Google for Startups Accelerator for Black Founders
Three months of one-on-one Google support and mentorship as part of the most recent Google for Startups Accelerator : Black Founders cohort not only helped us build our product, but also helped us earn external credibility. People use Google every day, so whether we’re trying to engage in conversations with industry experts or with somebody in a rural community, it is helpful to have the buy-in of a globally-recognized brand as we take on a historically difficult, systemic issue with challenges around trust. Getting access to the products, best practices, and people we need to build and grow through the Accelerator program has been priceless. For example, working with the Google AdWords team helped us generate important traffic from people interested in learning more about #NOWINCLUDED or sharing their story with us. Jason Scott, who leads the Google for Startups Accelerator: Black Founders program, is still connecting us to people in his network and identifying key opportunities for us months after the program wrapped. He continues to demonstrate that he is invested in seeing us succeed.
Our company has made great progress against our goals, in part thanks to receiving capital from the Google for Startups Black Founders Fund. We received $100K in non-dilutive funding along with Google Cloud credits, Google.org Ads grants, and hands-on support. We used the funds to pay for the transition and development costs associated with moving to Google Cloud. The Google support and accountability has been incredible. After receiving the Google for Startups Black Founders Fund award, we’ve gone on to raise another $1M and moved our cloud from Salesforce to Google Cloud.
We also had the amazing opportunity to be selected as one of six companies to take part in a face-to-face web conference with Sundar Pichai, Google’s CEO. We were thrilled to hear him explain his vision around health equity and the role Google plays. Ultimately, for us, it’s not just about the funding we get, but we are also gratified to receive support from an entity that truly believes in addressing this issue. We know Google is aligned with our mission of health and racial equity.

Championing diversity in clinical trials
Our 2022 looks bright. We expanded our presence to Washington, D.C. as part of the Johnson & Johnson Innovation JLABS ecosystem. We were also selected to take part in the BLUE KNIGHT initiative created between Johnson & Johnson Innovation and the Biomedical Advanced Research and Development Authority (BARDA) under the U.S. Department of Health and Human Services.
Acclinate is also on track to have contracts with five of the top 25 largest biopharmaceutical companies in the U.S this year. They’ve taken note, as has the Food and Drug Administration (FDA), that the lack of diversity in clinical trials represents a significant health concern—to the extent that the FDA has provided strong guidance for pharmaceutical companies to diversify their clinical trials. At the same time, the industry is also responding to pressure from communities of people of color to make equitable representation a priority.
Today, we are in the fortunate but challenging position to have significant inbound opportunities coming our way. In response, we continue to recruit and hire talented people to join our team. On the technology side, we are happy to be aligned with Google Cloud to have powerful cloud infrastructure that will scale with us, as well as high-caliber champions united in partnership. With people’s lives at stake, we are passionate in our commitment to helping ensure medications do what they are supposed to do: heal and improve the quality of life for everyone who takes them.
Hear Acclinate cofounders Del Smith and Tiffany Whitlow chat with Google’s Head of Startup Developer Ecosystem Jason Scott and fellow Black Founders Fund recipient Bobby Bryant about building on Google Cloud in a recent Google for Startups Instagram Live.
If you want to learn more about how Google Cloud can help your startup, visit our page here to get more information about our program, and sign up for our communications to get a look at our community activities, digital events, special offers, and more
4035
Of your peers have already watched this video.
15:00 Minutes
The most insightful time you'll spend today!
Modernize your Windows Workloads by Migrating them to Google Cloud
Google has plenty to offer when it comes to migrating and modernizing traditional enterprise Windows workloads to the cloud. Explore different approaches for re-hosting, modernizing, and transforming Windows applications, and the benefits of moving to Google Cloud.
Learn from demos on some of the cutting-edge technologies that can offload some of the operational IT burden into optimized managed services.

3459
Of your peers have already listened to this podcast
1:30 Minutes
The most insightful time you'll spend today!
Rethinking Financial Services with Google Cloud
Editor’s Note: In a previous blog, we discussed how home improvement retailer Lowe’s was able to increase the number of releases it supports by adopting Google’s Site Reliability Engineering (SRE) framework on Google Cloud. Lowe’s went from one release every two weeks to 20+ releases daily, helping meet its customer needs faster and more effectively. Today, the Lowe’s SRE team shares how they used SRE principles to decrease their mean-time-to-recovery (MTTR) by over 80 percent.
The stakes of managing Lowes.com have never been higher, and that means spotting, troubleshooting and recovering from incidents as quickly as possible, so that customers can continue to do business on our site.
To do that, it’s crucial to have solid incident engineering practices in place. Resolving an incident means mitigating the impact and/or restoring the service to its previous condition. The average time it takes to do this is called mean time to recovery (MTTR). Tracking this metric helps us stay on top of the overall reliability of our systems at Lowe’s, while simultaneously improving the speed with which we recover. Our goal is to keep the MTTR metric as low as possible, so that failures don’t negatively impact our business. Here are the four areas we addressed to drive holistic improvement in our MTTR.
Lowe’s incident reporting process
To reduce MTTR, we created a seamless incident reporting process following SRE principles. Our incident reporting process is a workflow that starts at the time an incident occurs, and ends with an SRE captain who closes the action items after a postmortem report. With this approach, we are able to limit the number of critical incidents. The reporting process involves three core components: monitoring, alerting, and blameless postmortems.
Monitoring and alerting
Having proper monitoring and alerting in place is crucial when it comes to incident management. Monitoring and alerting tools let you detect issues as soon as they occur, and notify the right person in the shortest possible time to take action. From a measurement standpoint, we track this as our mean time to acknowledge (MTTA). This is the average time it takes from when an alert is triggered, to when work on the issue begins.
At the time of an incident, our monitoring and alerting tools notify the on-call SRE first responder via PagerDuty in the form of a phone call, text message and email. Our SRE software engineering team has done a lot of automation to enable various Service Level Indicator (SLI) alerts and Service Level Agreement (SLA) notifications. The on-call SRE then initiates a triage call with our service/domain stakeholders to resolve the incident. As a result, we reduced our MTTA from 30 minutes in 2019, to one minute – a 97 percent decrease.
Blameless postmortems: learning from incidents
A postmortem is a written record of an incident, its impact, the actions taken to resolve it, the root cause and the follow-up actions to prevent the incident from recurring (see example here). A blameless postmortem builds on that and is a core part of an SRE culture, and our culture at Lowe’s. We ensure that individuals are not singled out, and the outcome for all postmortems are directed toward learnings and process improvement.
For us, the postmortem process is the biggest part of our incident workflow. When an SRE creates a new postmortem report, the first step is to conduct a postmortem session with domain stakeholders to review the report. The postmortem then goes into the review stage and gets reviewed by more stakeholders in our weekly postmortem meeting. In the final stage of this process, the SRE captain will close the report once everyone in the weekly meeting agrees that the report is complete.
To conduct a successful postmortem, it is critical to keep the focus on identifying gaps and issues with the system and operations processes, rather than an individual, and generate concrete actions to address the problems we’ve identified. To ensure this, we follow a couple of best practices:
- We start by gathering the facts from the person who identified the problem, and each SLI owner has to identify a gap or the next SLI upstream owner who created the impact for them.
- Every SLI owner is provided full opportunity to present their case, and identifying the issue is done as a community exercise.
- Once action items and process changes are identified, an owner is nominated to complete the actions, or they will volunteer.
- For easy reference, we publish and store postmortems in our incident knowledge base. This process helps SREs continuously improve as future incidents arise.
Continuous Improvement
Encouraging a culture of honest, transparent and direct feedback that you need for blameless postmortems is often an iterative process that needs sponsorship from executives, empowering incident captains to lead the entirety of the discussion and outcomes. Running successful postmortems, and completing action items from them, needs to be recognized and accounted for in SRE performance objective assessment. As shared in Google’s SRE book, the best practice is to ensure that writing effective postmortems is a rewarded and celebrated practice, with leadership’s acknowledgement and participation. This is possibly the hardest part to accomplish in an effective postmortem during a cultural transformation unless you have full buy-in from leadership.
However, it’s all well worth it. This process is a key part of how we were able to improve our MTTR over time—from two hours in 2019 to just 17 minutes!
Our SRE incident reporting process has also transformed how our company solves issues. By streamlining this workflow from alerting, to solving an issue, to blameless postmortems, we have reduced our MTTR by 82 percent and our MTTA by 97 percent. Most importantly, our team is learning from every incident and becoming better engineers as a result. Visit the SRE Google Cloud website to learn more about implementing SRE best practices in the cloud.
Acknowledgement
Special thanks to Rahul Mohan Kola Kandy, Vivek Balivada, and the Digital SRE team at Lowe’s for contributing to this blog post.
More Relevant Stories for Your Company

Take a Look at 30 Eventrac Locations!
New locations in Eventarc Back in August, we announced more Eventarc locations (17 new regions, as well as 6 new dual-region and multi-region locations to be precise). This takes the total number of locations in Eventarc to more than 30. You can see the full list in the Eventarc locations page or by running gcloud

The Fantastic Story of How BMG Enables a Micropayments Strategy So Music Artists Get Paid
The music industry is rapidly changing. Only 20 years ago, the availability of music and the infrastructure that was required to make an album a sales success were incredibly complex and expensive. With the decline of physical sales and a fundamental shift to digital, music streaming now accounts for more

FFF Enterprises See 80% Improvements in Speed at Lower Cost by Moving SAP Data to Google Cloud
FFF Enterprises, a pharmaceutical distributor of lifesaving biopharma products, vaccines and plasma products deployed SAP on Google Cloud to leverage its ability to scale server demands, reduce costs and improve speed and performance. Learn how the migration helps FFF empower healthcare to care!

10 Reasons that Make Google Cloud the Champion of IaaS
When you choose to run your business on Google Cloud you benefit from the same planet-scale infrastructure that powers Google’s products such as Maps, YouTube, and Workspace. We have picked 10 ways in which Google Cloud Infrastructure services outshine alternatives in the market in how they simplify your operations, save






