Editor’s Note: In a previous blog, we discussed how home improvement retailer Lowe’s was able to increase the number of releases it supports by adopting Google’s Site Reliability Engineering (SRE) framework on Google Cloud. Lowe’s went from one release every two weeks to 20+ releases daily, helping meet its customer needs faster and more effectively. Today, the Lowe’s SRE team shares how they used SRE principles to decrease their mean-time-to-recovery (MTTR) by over 80 percent.

The stakes of managing Lowes.com have never been higher, and that means spotting, troubleshooting and recovering from incidents as quickly as possible, so that customers can continue to do business on our site. 

To do that, it’s crucial to have solid incident engineering practices in place. Resolving an incident means mitigating the impact and/or restoring the service to its previous condition. The average time it takes to do this is called mean time to recovery (MTTR). Tracking this metric helps us stay on top of the overall reliability of our systems at Lowe’s, while simultaneously improving the speed with which we recover. Our goal is to keep the MTTR metric as low as possible, so that failures don’t negatively impact our business. Here are the four areas we addressed to drive holistic improvement in our MTTR.

Lowe’s incident reporting process

To reduce MTTR, we created a seamless incident reporting process following SRE principles. Our incident reporting process is a workflow that starts at the time an incident occurs, and ends with an SRE captain who closes the action items after a postmortem report. With this approach, we are able to limit the number of critical incidents. The reporting process involves three core components: monitoring, alerting, and blameless postmortems.

Monitoring and alerting

Having proper monitoring and alerting in place is crucial when it comes to incident management. Monitoring and alerting tools let you detect issues as soon as they occur, and notify the right person in the shortest possible time to take action. From a measurement standpoint, we track this as our mean time to acknowledge (MTTA). This is the average time it takes from when an alert is triggered, to when work on the issue begins.

At the time of an incident, our monitoring and alerting tools notify the on-call SRE first responder via PagerDuty in the form of a phone call, text message and email. Our SRE software engineering team has done a lot of automation to enable various Service Level Indicator (SLI) alerts and Service Level Agreement (SLA) notifications. The on-call SRE then initiates a triage call with our service/domain stakeholders to resolve the incident. As a result, we reduced our MTTA from 30 minutes in 2019, to one minute – a 97 percent decrease. 

Blameless postmortems: learning from incidents

A postmortem is a written record of an incident, its impact, the actions taken to resolve it, the root cause and the follow-up actions to prevent the incident from recurring (see example here). A blameless postmortem builds on that and is a core part of an SRE culture, and our culture at Lowe’s. We ensure that individuals are not singled out, and the outcome for all postmortems are directed toward learnings and process improvement.

For us, the postmortem process is the biggest part of our incident workflow. When an SRE creates a new postmortem report, the first step is to conduct a postmortem session with domain stakeholders to review the report. The postmortem then goes into the review stage and gets reviewed by more stakeholders in our weekly postmortem meeting. In the final stage of this process, the SRE captain will close the report once everyone in the weekly meeting agrees that the report is complete.

To conduct a successful postmortem, it is critical to keep the focus on identifying gaps and issues with the system and operations processes, rather than an individual, and generate concrete actions to address the problems we’ve identified. To ensure this, we follow a couple of best practices:

1. We start by gathering the facts from the person who identified the problem, and each SLI owner has to identify a gap or the next SLI upstream owner who created the impact for them.
2. Every SLI owner is provided full opportunity to present their case, and identifying the issue is done as a community exercise. 
3. Once action items and process changes are identified, an owner is nominated to complete the actions, or they will volunteer. 
4. For easy reference, we publish and store postmortems in our incident knowledge base. This process helps SREs continuously improve as future incidents arise. 

Continuous Improvement 

Encouraging a culture of honest, transparent and direct feedback that you need for blameless postmortems is often an iterative process that needs sponsorship from executives, empowering incident captains to lead the entirety of the discussion and outcomes. Running successful postmortems, and completing action items from them, needs to be recognized and accounted for in SRE performance objective assessment. As shared in Google’s SRE book, the best practice is to ensure that writing effective postmortems is a rewarded and celebrated practice, with leadership’s acknowledgement and participation. This is possibly the hardest part to accomplish in an effective postmortem during a cultural transformation unless you have full buy-in from leadership.

However, it’s all well worth it. This process is a key part of how we were able to improve our MTTR over time—from two hours in 2019 to just 17 minutes! 

Our SRE incident reporting process has also transformed how our company solves issues. By streamlining this workflow from alerting, to solving an issue, to blameless postmortems, we have reduced our MTTR by 82 percent and our MTTA by 97 percent. Most importantly, our team is learning from every incident and becoming better engineers as a result. Visit the SRE Google Cloud website to learn more about implementing SRE best practices in the cloud.

Acknowledgement

Special thanks to Rahul Mohan Kola Kandy, Vivek Balivada, and the Digital SRE team at Lowe’s for contributing to this blog post.

Built for a CapEx world, most organizations’ finance systems aren’t set up to take advantage of cloud’s dynamic, OpEx-driven consumption patterns.

It may not be a household name yet, but chances are you’ve crossed paths with OpenX today. OpenX, a leader in programmatic advertising, operates one of the world’s largest ad exchanges, serving over 250 billion ad requests per day, connecting more than 30,000 brands and reaching nearly one billion consumers. To make it happen, in 2019, OpenX migrated entirely out of its data centers and became the first major ad-exchange platform to move completely to the cloud.

The OpenX CTO, Paul Ryan, knew that this cloud transformation initiative had the potential to increase costs faster than its revenues. To be successful, he needed his engineering, finance, and business teams to forge a new “cost-aware” culture, complete with effective cost visibility and controls. In other words, he needed Cloud FinOps — an operational framework and cultural shift that brings technology, finance, and business together to drive financial accountability and realize business benefits through cloud transformation.

Ryan laid out a cloud migration roadmap that included cost governance and controls around project ownership, established cost responsibility with engineering teams to accurately forecast cloud consumption, and challenged developers to lower per-unit costs — while at the same time improving performance, scalability, speed and global reach.

It worked! In just 9 months, OpenX reduced their per-unit cost by over 60%. The framework allowed OpenX to launch new regions in a matter of days, reduce their time to market for new features by over 50%, and complete their migration in record time — seven months! “We are now able to stop worrying about legacy infrastructure and focus more on our growth categories,” said Ryan. “Our tech stack is getting smarter and more sophisticated by the day, and we have the flexibility to scale our infrastructure in real-time as the business scales and evolves.”

Unblocking Cloud’s Potential

Cloud holds the key to a successful digital transformation. In fact, McKinsey forecasts that by 2030, the Fortune 500 alone may realize over $1 trillion of EBITDA value drivers associated with public cloud enablement. But unlike OpenX, many companies struggle to achieve near-term value objectives from their cloud investments. Surveys reflect that more than 30% of cloud spend in 2021 was wasted or inefficient, while upwards of 80% of CIOs have yet to achieve the business benefits of migrating to the cloud.

Traditional IT finance processes are ill-suited for cloud infrastructure: Traditional planning and budgeting processes are challenged to address dynamic consumption patterns and complex migrations. Centralized IT budgets using traditional allocations fail to provide the necessary visibility into sources of cost overruns. CapEx-focused cost controls have little ability to manage largely OpEx-driven spend. Trend-based forecasting often inaccurately predicts cloud costs. And developer teams lack access to cost-aware architecture patterns to deploy the applications more efficiently.

Enter Cloud FinOps

At Google we’ve worked with many companies, like OpenX, to help organizations realize the transformational benefits of the cloud by cultivating a culture of transparency and embedding agile processes to manage costs. We’ve distilled these learnings into a Cloud FinOps operational framework that gives organizations the financial governance and accountability they need to grow their business sustainably.

At a high level, a Cloud FinOps approach depends on five key areas:

1. Accountability and Enablement

Accountability and enablement aim at instilling a cost-conscious culture across the organization. Oftentimes, this means standing up a cross-functional and dedicated team with members from technology, finance and engineering to establish cloud financial best practices and governance. In various organizations, we’ve seen this through an extension of a Cloud Center of Excellence, a Cloud Business Office or simply a Cloud FinOps team. Enablement focuses on empowering IT, finance and business leaders through training to help them better understand the economics of cloud services and the strategies to efficiently deploy and manage them. Cloud financial training guides teams on how to design cost-effective cloud environments, for example, embracing ”cloud-native” design principles such as auto-scaling/elasticity and Infrastructure as a Code.

2. Measurement and Business Value Realization

Effective measurements not only create awareness and enable agile processes, but also support a culture that celebrates success and rewards teams for achieving business objectives. As such, measurement in the service of business value realization is about developing a comprehensive set of long-term benefits and cost KPIs to quantify the total net value of the return on digital transformation. Organizations often start with cost-related KPIs and eventually evolve those KPIs into business value metrics that are mapped to targeted business outcomes.

3. Cloud Cost Optimization

Cloud cost optimization is an iterative and continuous process that provides a consistent methodology to manage cloud consumption cost-effectively. There are three key areas of optimization:

Resource optimization – Model cost-effective cloud usage based on utilization and consumption patterns.
Pricing optimization – Manage cloud spend through a continuous analysis of various pricing models. In a Google Cloud context, that might mean Committed Use Discounts, BigQuery flat rate reservations, etc.
Architecture optimization – Build applications with a cost-aware architecture by leveraging newer generation compute instances (like Tau VMs, which offer an industry-leading 42% better price-performance versus comparable offerings), or using managed services and serverless technology to offload operational overhead.
For example, video hosting, sharing and services platform provider Vimeo built transcoding pipelines by using Google Cloud Spot VMs to optimize their infrastructure spend. To do so, they created fault-tolerant workloads that could withstand preemptions, and in exchange, got up to a 91% discount compared to using regular on-demand instances.

4. Planning and Forecasting

In the cloud, accurately forecasting your finances requires rethinking of traditional approaches to depreciation and trend-based forecasting of maintenance and licensing costs. One way to improve the accuracy of your dynamic cloud needs is to use workload-specific forecasting models that leverage a combination of trend-based models for steady-state workloads, driver-based models for scaling applications, as well as monthly variance analysis. In other words, you can define cloud budgets and forecasts by monitoring cloud consumption trends, allocating cloud cost pools with a proper tagging strategy that’s mapped to a chart of accounts in a general ledger, and conducting a cost-benefit analysis based on cloud infrastructure, implementation, and support costs.

5. Tools and Accelerators

Without the proper tools and processes in place, understanding and managing cloud costs can be complex — and this especially true as organizations scale their business in the cloud. By deploying proper cloud cost management tools and accelerators such as Looker Cloud Cost Management and automation scripts to set guardrails and enforce cost control policies, organizations can effectively manage and track cloud spend with access to near-real-time billing and cost data to make better informed business decisions.

The key objectives of Google Cloud Cost Management tools are to make it as simple as possible for organizations to get visibility into their current and forecasted costs with built-in reporting and customizable dashboards; help drive greater accountability for cloud spending across the organization by providing flexible ways to organize cloud resources and allocate costs; provide strong financial governance controls to reduce the risk of overspending; and offer intelligent recommendations for optimizing cloud costs and usage.

Start Saving with Cloud

Businesses are continuously seeking to better operate and manage their cloud environments and the need is ever increasing to transparently manage cloud spend, optimize costs, and obtain their desired business agility. By enhancing your Cloud FinOps capabilities and adopting principles of continuous cost optimization, you too can accelerate the business value of cloud computing.

Special thanks to Bruce Warner, Daniel Petibone, Nihar Jhawar and FinOps Foundation community for their contributions and sharing their domain expertise to this important Cloud FinOps topic.

In 2013, long before the world was discussing clean energy and sustainable practices, two IIT Madras graduates — Swapnil Jain and Tarun Mehta — had an idea to develop India’s first-ever electrical scooter.

This was at a time when auto manufacturers were still focusing on fossil-fuel-driven vehicles and ‘eco-friendly’ mobility solutions were more a trendy alternative catering to a niche market.

The duo founded Ather Energy in 2013 and launched their first fully-electric scooter, the Ather S340, in Bengaluru in 2016. Since then, the company has released several new models into the market and is planning to expand to eight more cities by the end of the year.

To support the smooth running of their vehicles, lower costs, improve time to market, and create great customer experience, Ather turned to Google Cloud.

Read the Full Story on YourStory