How Lowe’s SRE Team Decreases Mean-time-to-recovery (MTTR)
3373
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
Editor’s Note: In a previous blog, we discussed how home improvement retailer Lowe’s was able to increase the number of releases it supports by adopting Google’s Site Reliability Engineering (SRE) framework on Google Cloud. Lowe’s went from one release every two weeks to 20+ releases daily, helping meet its customer needs faster and more effectively. Today, the Lowe’s SRE team shares how they used SRE principles to decrease their mean-time-to-recovery (MTTR) by over 80 percent.
The stakes of managing Lowes.com have never been higher, and that means spotting, troubleshooting and recovering from incidents as quickly as possible, so that customers can continue to do business on our site.
To do that, it’s crucial to have solid incident engineering practices in place. Resolving an incident means mitigating the impact and/or restoring the service to its previous condition. The average time it takes to do this is called mean time to recovery (MTTR). Tracking this metric helps us stay on top of the overall reliability of our systems at Lowe’s, while simultaneously improving the speed with which we recover. Our goal is to keep the MTTR metric as low as possible, so that failures don’t negatively impact our business. Here are the four areas we addressed to drive holistic improvement in our MTTR.
Lowe’s incident reporting process
To reduce MTTR, we created a seamless incident reporting process following SRE principles. Our incident reporting process is a workflow that starts at the time an incident occurs, and ends with an SRE captain who closes the action items after a postmortem report. With this approach, we are able to limit the number of critical incidents. The reporting process involves three core components: monitoring, alerting, and blameless postmortems.
Monitoring and alerting
Having proper monitoring and alerting in place is crucial when it comes to incident management. Monitoring and alerting tools let you detect issues as soon as they occur, and notify the right person in the shortest possible time to take action. From a measurement standpoint, we track this as our mean time to acknowledge (MTTA). This is the average time it takes from when an alert is triggered, to when work on the issue begins.
At the time of an incident, our monitoring and alerting tools notify the on-call SRE first responder via PagerDuty in the form of a phone call, text message and email. Our SRE software engineering team has done a lot of automation to enable various Service Level Indicator (SLI) alerts and Service Level Agreement (SLA) notifications. The on-call SRE then initiates a triage call with our service/domain stakeholders to resolve the incident. As a result, we reduced our MTTA from 30 minutes in 2019, to one minute – a 97 percent decrease.
Blameless postmortems: learning from incidents
A postmortem is a written record of an incident, its impact, the actions taken to resolve it, the root cause and the follow-up actions to prevent the incident from recurring (see example here). A blameless postmortem builds on that and is a core part of an SRE culture, and our culture at Lowe’s. We ensure that individuals are not singled out, and the outcome for all postmortems are directed toward learnings and process improvement.
For us, the postmortem process is the biggest part of our incident workflow. When an SRE creates a new postmortem report, the first step is to conduct a postmortem session with domain stakeholders to review the report. The postmortem then goes into the review stage and gets reviewed by more stakeholders in our weekly postmortem meeting. In the final stage of this process, the SRE captain will close the report once everyone in the weekly meeting agrees that the report is complete.
To conduct a successful postmortem, it is critical to keep the focus on identifying gaps and issues with the system and operations processes, rather than an individual, and generate concrete actions to address the problems we’ve identified. To ensure this, we follow a couple of best practices:
- We start by gathering the facts from the person who identified the problem, and each SLI owner has to identify a gap or the next SLI upstream owner who created the impact for them.
- Every SLI owner is provided full opportunity to present their case, and identifying the issue is done as a community exercise.
- Once action items and process changes are identified, an owner is nominated to complete the actions, or they will volunteer.
- For easy reference, we publish and store postmortems in our incident knowledge base. This process helps SREs continuously improve as future incidents arise.
Continuous Improvement
Encouraging a culture of honest, transparent and direct feedback that you need for blameless postmortems is often an iterative process that needs sponsorship from executives, empowering incident captains to lead the entirety of the discussion and outcomes. Running successful postmortems, and completing action items from them, needs to be recognized and accounted for in SRE performance objective assessment. As shared in Google’s SRE book, the best practice is to ensure that writing effective postmortems is a rewarded and celebrated practice, with leadership’s acknowledgement and participation. This is possibly the hardest part to accomplish in an effective postmortem during a cultural transformation unless you have full buy-in from leadership.
However, it’s all well worth it. This process is a key part of how we were able to improve our MTTR over time—from two hours in 2019 to just 17 minutes!
Our SRE incident reporting process has also transformed how our company solves issues. By streamlining this workflow from alerting, to solving an issue, to blameless postmortems, we have reduced our MTTR by 82 percent and our MTTA by 97 percent. Most importantly, our team is learning from every incident and becoming better engineers as a result. Visit the SRE Google Cloud website to learn more about implementing SRE best practices in the cloud.
Acknowledgement
Special thanks to Rahul Mohan Kola Kandy, Vivek Balivada, and the Digital SRE team at Lowe’s for contributing to this blog post.
Learning Should be Your Resolution for 2022: Register for Google Cloud Skills Boost
3285
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
Start your 2022 New Year’s resolutions by learning at no cost how to use Google Cloud with the following training opportunities:
30 day access to Google Cloud Skills Boost
Register by January 31, 2022 and claim 30 days free access to Google Cloud Skills Boost to complete the Getting Started with Google Cloud learning path.
Google Cloud Skills Boost is the definitive destination for skills development where you can personalize learning paths, track progress, and validate your newly-earned expertise with skill badges.
The Getting Started with Google Cloud learning path will give you the opportunity to earn three skill badges after you complete hands-on labs and courses designed for aspiring cloud engineers and architects. It covers the fundamentals of Google Cloud including core infrastructure, big data and ML, writing gcloud commands, using Cloud Shell, deploying virtual machines, and running containerized applications on GKE.
Cloud OnBoard: half day training on getting started with Google Cloud fundamentals
Attend the Getting Started Cloud OnBoard on January 20 for a comprehensive Google Cloud orientation. Google Cloud experts will show you how to execute your compute, available storage options, how to secure your data, and available Google Cloud managed services.
Cloud Study Jam: expert-guided hands-on lab
Google Cloud experts will walk you through a hands-on lab included in Google Cloud Skill Boost’s Getting Started with Google Cloud learning path when you join our Cloud Study Jam on January 27. Google Cloud experts will also answer questions live via chat during this event.
STAC-M3 Tick History Analytics in Google Cloud Benchmark Results Reveals it is 18X Faster than Previous Version
4769
Of your peers have already read this article.
5:00 Minutes
The most insightful time you'll spend today!
The Securities Technology Analysis Center (STAC®), an organization that improves technology discovery and assessment in the finance industry through dialog and research, recently audited the STAC-M3™ benchmark suite on Google Cloud (SUT ID KDB211210). These enterprise tick-analytics benchmarks assess the ability of a solution stack such as database software, servers, and storage, to perform a variety of I/O-intensive and compute-intensive operations on historical market data.
Following up on our previous STAC-M3 benchmark audit (SUT ID KDB181001), a redesigned Google Cloud architecture leveraged the most recent version of kdb+ 4.0, the time-series database from KX, and achieved significant improvements: 35 out of 41 benchmarks ran faster in the new cluster – by up to 18x faster than Google Cloud’s prior results. Key highlights include the following:
Compared to the previous STAC-M3 Antuco suite results on Google Cloud:
- Was faster in 13 of 17 mean response-time benchmarks
- Was 18x faster – a 94% reduction in run time – in the version of Year-High Bid that allows caching (STAC-M3.ß1.1T.YRHIBID-2.TIME), which also set an overall record for all published results
- Had 9x higher throughput in Year-High Bid (STAC-M3.ß1.1T.YRHIBID.MBPS)
Compared to the previous STAC-M3 Kanaga suite results on Google Cloud:
- Was faster in 22 of 24 mean response-time benchmarks
- Was over 10x faster in all four Market Snapshot workloads (STAC-M3.ß1.10T.YR[2,3,4,5]-MKTSNAP.TIME)
- Had 5x the throughput in Year-High Bid involving 2 years of data (STAC-M3.ß1.1T.2YRHIBID.MBPS)
“The STAC-M3 standard was designed by financial firms to reveal the performance of tick analytics stacks. Generational improvements like those exhibited by Google Cloud’s most recent STAC-M3 audit, are important data points for firms evaluating new architectures for performance and scale,” said Peter Nabicht, President of STAC.
These performance results may translate to real-world advantages that may be difficult for investment firms to achieve in static and costly on-premises environments: immediate answers in high data velocity markets, more thoroughly explored research theories by adding data or new quantitative approaches, and reduced costs by releasing cloud resources more quickly.
STAC-M3: High-speed tick analytics
Designing for record-breaking results
In our STAC-M3 audit, the stack under test (SUT) was designed to take advantage of horizontal scalability in the cloud by sharding data across independent compute nodes. The cluster of 12 Google Compute Engine N2 instances was powered by Intel Cascade Lake, with each node using 32 vCPUs, 160GiB of memory, and 9TiB of local NVMe SSDs. The full STAC-M3 Antuco and Kanaga data set was split across the cluster and kdb+ scripts distributed queries between nodes.
This configuration was the sweet spot for this particular workload, but this architecture does not need to be limited to 12 nodes for other workloads – the data sharding algorithm could scale to any number of nodes as required by workload demands. Since scaling out the cluster in this manner increases the total pool of available storage, this architecture can continue scaling out to petabytes of storage across hundreds of nodes.
The ability to spawn large clusters with hundreds of thousands of processors on demand at low cost, and to delete the resources when jobs complete, not only changes the economics of running computations on large financial data sets, it also opens up opportunities to explore solutions to new types of problems that were previously overlooked due to the constraints of fixed hardware on-premises. You can check the pricing of this VM configuration using the Google Cloud Pricing Calculator. The costs can be reduced even further by using preemptible VMs.
While the new cluster used a similar number of nodes, cores, and total memory as the previously-audited cluster, the redesigned architecture allowed us to harness the low latency and high throughput of Local NVMe SSDs.
Resources on demand
The cluster was created on demand using Terraform and Ansible during testing and auditing. The use of infrastructure as code (IaC) techniques ensured that the cluster, fully loaded with the STAC-M3 data set, could be created when needed and then removed when benchmarking was complete. It also meant that the cluster configuration was enforced by code on each deployment, eliminating configuration variance and drift. The full IaC definition to create the cluster can be retrieved from the report in the STAC Vault.
Each time the cluster was created, data was streamed to Local SSDs from Google Cloud Storage, our reliable and secure object storage, at up to the line rate of 32Gbps per node. The entire 57TiB STAC-M3 Antuco and Kanaga data was replicated from Cloud Storage to local storage in approximately 20 minutes.
Since each node was independent and responsible for its own shard of data, doubling the cluster size would cut the synchronization time in half, or copy twice as much data in the same amount of time. Using higher bandwidth options of up to 100Gbps would triple the possible throughput for a relatively small incremental cost, trading an approximately 11%-23% price increase at current list prices for a 200% data synchronization performance increase. Taking advantage of fast networking to cache sharded data in parallel to a large cluster makes storing bulk data in Cloud Storage viable for even the largest workloads.
For quants working on vast data sets in sprawling compute clusters, the ability to fully describe infrastructure as declarative code, create elastic resources on demand, cache data quickly from cheap bulk storage, and turn resources off when computations complete is a dramatic change compared to waiting months to grow on-premises clusters – and a compelling reason to use cloud infrastructure.
To see how we designed and optimized the cluster for API-driven cloud resources, read our new whitepaper.
STAC-A2™: Calculating derivatives risk
In 2018, we showed that cloud instances can outperform bare metal when analyzing large tick history data sets in the demanding suite of STAC-M3 benchmarks. Last year, Google Cloud’s partner Appsbroker showed that the same was true for calculating derivatives risk in STAC-A2 on Google Cloud. You can read about how Appsbroker built its record-breaking STAC-A2 compute cluster on Google Cloud in its blog post, or access the STAC Report directly. Here are the highlights:
Compared to all other publicly reported solutions, this solution, based on a cluster of 10 virtual machines, had:
- The highest throughput (STAC-A2.β2.HPORTFOLIO.SPEED)
- The fastest cold time in the large problem size (STAC-A2.β2.GREEKS.10-100k-1260.TIME.COLD)
Compared to a solution involving an 8-node, on-premises cluster (SUT ID INTC181012), this 10-node, cloud-based solution:
- Had 5 times the maximum paths (STAC-A2.β2.GREEKS.MAX_PATHS)
- Had 10% greater throughput (STAC-A2.β2.HPORTFOLIO.SPEED)
- Was 18% faster in cold runs of the large problem size (STAC-A2.β2.GREEKS.10-100k-1260.TIME)
- Was 9% faster in cold runs of the baseline problem size (STAC-A2.β2.GREEKS.TIME.COLD)
Finding market advantages with Google Cloud
Across the investment management industry, every firm is seeking many of the same competitive advantages. However, finding unique opportunities and managing larger and larger data sets is becoming a major strain. Cloud is fundamentally changing how quants tackle the problem while empowering them to manage risk and generate higher returns.
Building on-premises computing clusters with tens or hundreds of thousands of cores and petabytes of storage requires huge up-front investments and lead time measured in months or years. Google Cloud makes the same scale available to its customers, provisioned on demand and paid per use. More importantly, the elasticity of cloud resources enables agility that is simply not available in a fixed data center cluster – the agility to explore, experiment, iterate, and respond to markets faster than before.
Scaling out to tens of thousands of cores in minutes and then removing the resources immediately not only changes the speed at which questions can be answered; it encourages different and more frequent questions, asked simultaneously on many independent clusters, free from the constraints of fixed on-premises hardware.
It is this flexibility and power that enables financial services firms to leverage larger data sets and get results, backtest, research, and analyze large amounts of data, faster and whenever they need it.
Download our whitepaper to learn more about our latest STAC-M3 tick history analytics benchmark results and how to optimize cloud infrastructure for high-speed market data analysis.
APAC’s Retail Digital Pulse by Google Cloud and IDC Retail Index
DOWNLOAD TREND ANALYSIS3432
Of your peers have already downloaded this article
15:00 Minutes
The most insightful time you'll spend today!
In the post COVID-19 pandemic world, some retail businesses have aced digital transformation while others are still lagging behind. The Google Cloud commissioned IDC Retail Insights analyzed over 1, 108 retailers across seven nations in the Asia Pacific region and across eight different segments (online, drugstores, speciality shops, convenience stores, department stores, restaurants, supermarkets and hypermarkets/big box stores) to develop Google Cloud Retail Digital Index. The digital maturity was reviewed in five dimensions that define the digital pulse like strategy, people, data, technology and process.
Apart from empowering retail brands to live up to customer expectations, technology is the backbone to back-office operations and still ranks low in the digital pulse. Download the Google-ODC InfoBrief to catch up on the latest, detailed insights on Asia Pacific’s Retail Digital Index along with essential guidance on driving digital transformation, use cases and more!
Gartner Names Google Cloud a Leader in the 2019 Cloud Infrastructure as a Service Market
DOWNLOAD WHITEPAPER3547
Of your peers have already downloaded this article
2:30 Minutes
The most insightful time you'll spend today!
As increasing amount of cloud IaaS is being bought for traditional IT, with an emphasis on cost reduction, safety and security, service providers are offering a high-quality service, with excellent availability, good performance, high security and good customer support.
In its Magic Quadrant for Cloud Infrastructure as a Service (IaaS), Gartner notes that the distinctions among providers are apparent in the market for cloud IaaS in terms of worldwide enterprise adoption, capabilities and service availability. Infrastructure and operations leaders should evaluate providers with broad capabilities and a positive track record for customer success.
Download the Gartner Magic Quadrant to understand why Google is a leader in the market.
5547
Of your peers have already downloaded this article
1:30 Minutes
The most insightful time you'll spend today!
Security in public clouds differs intrinsically from customer-owned infrastructure because there is shared responsibility for security between the customer and the cloud provider.
Cloud Security is different from on-premises security because of the combination of the following:
- Differences in security primitives, visibility, and control points within the infrastructure, products
- and services.
- New cloud-native development methodologies like containerization and DevSecOps.
- The continued velocity and variety of new cloud products and services and how they can be
- consumed.
- Cultural shifts in how organizations deploy, manage, and operate systems.
Here is a Google Cloud Security Foundations Guide to help you develop a security blueprint for your cloud deployments.
More Relevant Stories for Your Company
Google Cloud’s 101 Illustrated References for Cloud Engineers and Architects
Many people are visual learners; I am definitely one of them, and judging by the tremendous response you all showed to Google Cloud Sketchnotes on LinkedIn and Twitter, you are too. -Priyanka Vergadia, Lead Developer Advocate, Google "A picture is worth a thousand words" Shortly after I started creating and
The Pathway to Innovation: Migrating SAP to the Cloud
While innovation remains a top priority across all industries, many companies are not achieving the results they want from their innovation efforts. As companies migrate their SAP enterprise resource planning (ERP) systems to the cloud, technology leaders see this as an opportunity to reimagine business processes and propel their innovation
Three German Retail Firms Choose Google Cloud to Migrate SAP Workloads for Business Transformation
The retail industry is rapidly evolving, with customers demanding exceptional digital experiences, and retailers adjusting their businesses to be more efficient. With many retailers relying on SAP for critical business functions like digital transactions, finance, supply chains and more, this shift has led them to look for ways to modernize
Intel-Google Collaboration Brings Edge Computing on Factory Floors: Hannover Messe 2022
The typical smart factory is said to produce around 5 petabytes of data per week. That’s equivalent to 5 million gigabytes, or roughly 20,000 smartphones. Managing such vast amounts of data in one facility, let alone a global organization, would be challenging enough. Doing so on the factory floor, in
