Bootstrap your Startup with our technical guided series

At Google Cloud, we want to provide you with the access to all the tools you need to grow your business. Through the Google Cloud Technical Guides for Startups, leverage industry leading solutions with how-to video guides and resource handbooks curated for startups.

This multi-series contains 3 chapters: Start, Build and Grow, which matches your startup’s stage of growth:

• The Start Series: Begin by building, deploying and managing new applications on Google Cloud from start to finish.
• The Build Series: Optimize and scale existing deployments to reach your target audiences.
• The Grow Series: Grow and attain scale with deployments on Google Cloud.

Kick off with The Start Series

The Start Series is designed to help your startup begin building, deploying and managing new applications on Google Cloud from start to finish. The series contains 12 videos and is dedicated to those who are starting out their cloud journey with Google Cloud. From setting up your project, to choosing the right compute option, to configuring your networking to managing your databases, and understanding support and billing – the Start Series guides you at every step of the journey.

Check out our website and our Google Cloud Technical Guides for Startups full playlist.

Coming up next – The Build Series

Launch into the next part of the journey continuing from the Start Series, with the upcoming Build Series, where we will be focusing on the optimization and scaling of existing deployments to help your startups reach your target audiences.

Join us by checking out the video series on the Google Cloud Tech channel, and subscribe to stay up to date.

See you in the cloud!

Editor’s Note: In a previous blog, we discussed how home improvement retailer Lowe’s was able to increase the number of releases it supports by adopting Google’s Site Reliability Engineering (SRE) framework on Google Cloud. Lowe’s went from one release every two weeks to 20+ releases daily, helping meet its customer needs faster and more effectively. Today, the Lowe’s SRE team shares how they used SRE principles to decrease their mean-time-to-recovery (MTTR) by over 80 percent.

The stakes of managing Lowes.com have never been higher, and that means spotting, troubleshooting and recovering from incidents as quickly as possible, so that customers can continue to do business on our site. 

To do that, it’s crucial to have solid incident engineering practices in place. Resolving an incident means mitigating the impact and/or restoring the service to its previous condition. The average time it takes to do this is called mean time to recovery (MTTR). Tracking this metric helps us stay on top of the overall reliability of our systems at Lowe’s, while simultaneously improving the speed with which we recover. Our goal is to keep the MTTR metric as low as possible, so that failures don’t negatively impact our business. Here are the four areas we addressed to drive holistic improvement in our MTTR.

Lowe’s incident reporting process

To reduce MTTR, we created a seamless incident reporting process following SRE principles. Our incident reporting process is a workflow that starts at the time an incident occurs, and ends with an SRE captain who closes the action items after a postmortem report. With this approach, we are able to limit the number of critical incidents. The reporting process involves three core components: monitoring, alerting, and blameless postmortems.

Monitoring and alerting

Having proper monitoring and alerting in place is crucial when it comes to incident management. Monitoring and alerting tools let you detect issues as soon as they occur, and notify the right person in the shortest possible time to take action. From a measurement standpoint, we track this as our mean time to acknowledge (MTTA). This is the average time it takes from when an alert is triggered, to when work on the issue begins.

At the time of an incident, our monitoring and alerting tools notify the on-call SRE first responder via PagerDuty in the form of a phone call, text message and email. Our SRE software engineering team has done a lot of automation to enable various Service Level Indicator (SLI) alerts and Service Level Agreement (SLA) notifications. The on-call SRE then initiates a triage call with our service/domain stakeholders to resolve the incident. As a result, we reduced our MTTA from 30 minutes in 2019, to one minute – a 97 percent decrease. 

Blameless postmortems: learning from incidents

A postmortem is a written record of an incident, its impact, the actions taken to resolve it, the root cause and the follow-up actions to prevent the incident from recurring (see example here). A blameless postmortem builds on that and is a core part of an SRE culture, and our culture at Lowe’s. We ensure that individuals are not singled out, and the outcome for all postmortems are directed toward learnings and process improvement.

For us, the postmortem process is the biggest part of our incident workflow. When an SRE creates a new postmortem report, the first step is to conduct a postmortem session with domain stakeholders to review the report. The postmortem then goes into the review stage and gets reviewed by more stakeholders in our weekly postmortem meeting. In the final stage of this process, the SRE captain will close the report once everyone in the weekly meeting agrees that the report is complete.

To conduct a successful postmortem, it is critical to keep the focus on identifying gaps and issues with the system and operations processes, rather than an individual, and generate concrete actions to address the problems we’ve identified. To ensure this, we follow a couple of best practices:

1. We start by gathering the facts from the person who identified the problem, and each SLI owner has to identify a gap or the next SLI upstream owner who created the impact for them.
2. Every SLI owner is provided full opportunity to present their case, and identifying the issue is done as a community exercise. 
3. Once action items and process changes are identified, an owner is nominated to complete the actions, or they will volunteer. 
4. For easy reference, we publish and store postmortems in our incident knowledge base. This process helps SREs continuously improve as future incidents arise. 

Continuous Improvement 

Encouraging a culture of honest, transparent and direct feedback that you need for blameless postmortems is often an iterative process that needs sponsorship from executives, empowering incident captains to lead the entirety of the discussion and outcomes. Running successful postmortems, and completing action items from them, needs to be recognized and accounted for in SRE performance objective assessment. As shared in Google’s SRE book, the best practice is to ensure that writing effective postmortems is a rewarded and celebrated practice, with leadership’s acknowledgement and participation. This is possibly the hardest part to accomplish in an effective postmortem during a cultural transformation unless you have full buy-in from leadership.

However, it’s all well worth it. This process is a key part of how we were able to improve our MTTR over time—from two hours in 2019 to just 17 minutes! 

Our SRE incident reporting process has also transformed how our company solves issues. By streamlining this workflow from alerting, to solving an issue, to blameless postmortems, we have reduced our MTTR by 82 percent and our MTTA by 97 percent. Most importantly, our team is learning from every incident and becoming better engineers as a result. Visit the SRE Google Cloud website to learn more about implementing SRE best practices in the cloud.

Acknowledgement

Special thanks to Rahul Mohan Kola Kandy, Vivek Balivada, and the Digital SRE team at Lowe’s for contributing to this blog post.

We’re excited to announce that Google Cloud Next returns on October 11–13, 2022. 

Join us for keynotes from industry luminaries and engage live with Google developers. Explore dynamic content across various learning levels, and dive deep into technologies and solutions spanning the Google Cloud and Google Workspace portfolios. Participate in breakout sessions, demos, and hands-on training. Hear from the world’s leading companies about their digital transformation journeys. You’ll have opportunities to connect with experts, get inspired, and boost your skills. We can’t wait to see you at Next ’22!

It’s too early to determine how the event experience will span the digital and physical worlds, so please stay tuned for updates as we plan with the health and safety of the attendees in mind. In the meantime, mark October 11–13 in your calendar, and visit our event site for updates. For more inspiration, rediscover Next ’21, now available on demand.