How Lowe’s SRE Team Decreases Mean-time-to-recovery (MTTR)

3372
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
Editor’s Note: In a previous blog, we discussed how home improvement retailer Lowe’s was able to increase the number of releases it supports by adopting Google’s Site Reliability Engineering (SRE) framework on Google Cloud. Lowe’s went from one release every two weeks to 20+ releases daily, helping meet its customer needs faster and more effectively. Today, the Lowe’s SRE team shares how they used SRE principles to decrease their mean-time-to-recovery (MTTR) by over 80 percent.
The stakes of managing Lowes.com have never been higher, and that means spotting, troubleshooting and recovering from incidents as quickly as possible, so that customers can continue to do business on our site.
To do that, it’s crucial to have solid incident engineering practices in place. Resolving an incident means mitigating the impact and/or restoring the service to its previous condition. The average time it takes to do this is called mean time to recovery (MTTR). Tracking this metric helps us stay on top of the overall reliability of our systems at Lowe’s, while simultaneously improving the speed with which we recover. Our goal is to keep the MTTR metric as low as possible, so that failures don’t negatively impact our business. Here are the four areas we addressed to drive holistic improvement in our MTTR.
Lowe’s incident reporting process
To reduce MTTR, we created a seamless incident reporting process following SRE principles. Our incident reporting process is a workflow that starts at the time an incident occurs, and ends with an SRE captain who closes the action items after a postmortem report. With this approach, we are able to limit the number of critical incidents. The reporting process involves three core components: monitoring, alerting, and blameless postmortems.
Monitoring and alerting
Having proper monitoring and alerting in place is crucial when it comes to incident management. Monitoring and alerting tools let you detect issues as soon as they occur, and notify the right person in the shortest possible time to take action. From a measurement standpoint, we track this as our mean time to acknowledge (MTTA). This is the average time it takes from when an alert is triggered, to when work on the issue begins.
At the time of an incident, our monitoring and alerting tools notify the on-call SRE first responder via PagerDuty in the form of a phone call, text message and email. Our SRE software engineering team has done a lot of automation to enable various Service Level Indicator (SLI) alerts and Service Level Agreement (SLA) notifications. The on-call SRE then initiates a triage call with our service/domain stakeholders to resolve the incident. As a result, we reduced our MTTA from 30 minutes in 2019, to one minute – a 97 percent decrease.
Blameless postmortems: learning from incidents
A postmortem is a written record of an incident, its impact, the actions taken to resolve it, the root cause and the follow-up actions to prevent the incident from recurring (see example here). A blameless postmortem builds on that and is a core part of an SRE culture, and our culture at Lowe’s. We ensure that individuals are not singled out, and the outcome for all postmortems are directed toward learnings and process improvement.
For us, the postmortem process is the biggest part of our incident workflow. When an SRE creates a new postmortem report, the first step is to conduct a postmortem session with domain stakeholders to review the report. The postmortem then goes into the review stage and gets reviewed by more stakeholders in our weekly postmortem meeting. In the final stage of this process, the SRE captain will close the report once everyone in the weekly meeting agrees that the report is complete.
To conduct a successful postmortem, it is critical to keep the focus on identifying gaps and issues with the system and operations processes, rather than an individual, and generate concrete actions to address the problems we’ve identified. To ensure this, we follow a couple of best practices:
- We start by gathering the facts from the person who identified the problem, and each SLI owner has to identify a gap or the next SLI upstream owner who created the impact for them.
- Every SLI owner is provided full opportunity to present their case, and identifying the issue is done as a community exercise.
- Once action items and process changes are identified, an owner is nominated to complete the actions, or they will volunteer.
- For easy reference, we publish and store postmortems in our incident knowledge base. This process helps SREs continuously improve as future incidents arise.
Continuous Improvement
Encouraging a culture of honest, transparent and direct feedback that you need for blameless postmortems is often an iterative process that needs sponsorship from executives, empowering incident captains to lead the entirety of the discussion and outcomes. Running successful postmortems, and completing action items from them, needs to be recognized and accounted for in SRE performance objective assessment. As shared in Google’s SRE book, the best practice is to ensure that writing effective postmortems is a rewarded and celebrated practice, with leadership’s acknowledgement and participation. This is possibly the hardest part to accomplish in an effective postmortem during a cultural transformation unless you have full buy-in from leadership.
However, it’s all well worth it. This process is a key part of how we were able to improve our MTTR over time—from two hours in 2019 to just 17 minutes!
Our SRE incident reporting process has also transformed how our company solves issues. By streamlining this workflow from alerting, to solving an issue, to blameless postmortems, we have reduced our MTTR by 82 percent and our MTTA by 97 percent. Most importantly, our team is learning from every incident and becoming better engineers as a result. Visit the SRE Google Cloud website to learn more about implementing SRE best practices in the cloud.
Acknowledgement
Special thanks to Rahul Mohan Kola Kandy, Vivek Balivada, and the Digital SRE team at Lowe’s for contributing to this blog post.

3495
Of your peers have already downloaded this article
1:30 Minutes
The most insightful time you'll spend today!
Reduce risks, increase uptime, and free up working capital with Google Cloud’s SAP Business Continuity Program. Working closely with our experienced partner community, we will ease the lift and shift of your on premises or hosted SAP environments with simple, no-cost migrations.
Gain a future-proof digital business platform with integrated artificial intelligence, machine learning, and advanced analytics that deliver deep insights for all who need them.
Get the performance, compute power, flexibility, scalability, availability, and security you need for your current and future business needs with SAP on Google Cloud. Download this factsheet to learn more.
Synopsys & Google Cloud: Helping Semiconductor Companies Drive Electronic Design Automation Innovation
2863
Of your peers have already read this article.
2:30 Minutes
The most insightful time you'll spend today!
Google Cloud and Synopsys Inc are partnering to help semiconductor companies drive Electronic Design Automation (EDA) innovation in the cloud to accelerate time to market, and lower costs across the entire semiconductor product life cycle. Synopsys is the industry’s largest provider of EDA technology used in the design and verification of integrated circuits, or semiconductor chips. Bringing Synopsys technology onto Google Cloud enables customers to benefit from scalable cloud bursting and complementary licensing models to help them quickly deploy and scale Synopsys’ EDA tools. Semiconductor companies can significantly accelerate chip design and production, increase designer productivity, and empower innovation in power-performance-area optimization.
Google Cloud Platform has enabled high performance computing workloads, such as those used by today’s semiconductor chip designers, to deliver exceptionally fast results and reduce prototyping from years to months. Google Cloud’s HPC services have delivered highly scalable solutions across multiple industries – from scientific research to autonomous vehicle testing and simulation.
EDA software is a large consumer of high performance computing capacity in the cloud. With the release of Synopsys Cloud bring-your-own-cloud (BYOC) solution on Google Cloud, chip designers can now scale their Google Cloud infrastructure with Synopsys’s leading EDA tools under the flexible FlexEDA pay-per-use model and access unlimited EDA software license availability on-demand by the hour or minute.
The Synopsys Cloud BYOC deployment architecture on Google Cloud is enabled through their unique cloud metering service which uses Google Cloud regional MIGs (Managed Instance Groups) for autoscaling and multi-zone deployment. This enables the service to scale up to meet customer workload demands and scale down to optimize costs. Secrets used by Synopsys Cloud are securely stored in Google Secret Manager and usage data is encrypted using Google Cloud key management, providing customers with a highly secure design environment.
Synopsys Cloud also leverages Google Cloud’s Ops Agent and Operations Suite Dashboards are used to show metric data, alerting policies, and log entries, providing customers with detailed analytics visibility to make better chip design project lifecycle management decisions. The Synopsys Cloud BYOC solution has been validated with EDA workloads scaling out to thousands of cores on Google Cloud, using Google Filestore network file storage during validation to provide the highest throughput performance.

Vikram Bhatia, Head, Synopsys Cloud Product Management, Synopsys said, “With the release of Synopsys Cloud BYOC solution on Google Cloud, we are transforming the way our mutual semiconductor customers can design the chips of the future. Google Cloud has been leading the innovation wave for semiconductors in the cloud and we are excited about being an early adopter in leveraging those innovations for our unique EDA offerings on the cloud.” Customers can evaluate a full featured Synopsys Cloud BYOC environment on Google Cloud for free by signing up at: synopsys.com/cloud.
“Combined with GCP’s unique platform services for AI, security, and shared storage, the Synopsys Cloud BYOC solution creates a compelling package for semiconductor designers who will create the next generation of chips for the world’s insatiable needs” says Simon Floyd, Industry Director, Manufacturing & Transportation, Google Cloud.
Google Cloud provides everything you need, including free Google Cloud credits to get you up and running. Click here to learn more about Semiconductors on Google Cloud.
Your DW Need Scaling Up? Try What This Company Did: It Can Run 25,000 Events a Second

5077
Of your peers have already read this article.
7:30 Minutes
The most insightful time you'll spend today!
With access to more data than ever before, companies have never been better positioned to adopt precision marketing methods and target the right customers at the right time. Emarsys, a digital marketing platform, enables its clients to collect, analyze, and act on a wide variety of data. From websites to mobile apps to emails, Emarsys’ customers can handle data from all its digital channels on a single, easy-to-use platform. Emarsys also makes sure that customers receive the highest quality data possible, making for smarter decisions and better business practices.
“We were close to the limits of our internal data warehouse, scalability-wise. We didn’t want to get to the point where we’d have to delete data or cancel projects. In Google Cloud we saw a platform that could scale with our ambitions and be optimized for AI and real-time solutions.”
—Levente Otti, Head of Data, Emarsys
Since launching as an email solutions provider in 2000, Emarsys has grown into the world’s largest independent digital marketing platform, with more than 2,500 clients worldwide and reaching more than 1.4 billion people. By 2016, the company felt that its existing data warehouse platform was close to its limits, affecting not just day-to-day operations but also important strategic goals.
“We were close to the limits of our internal data warehouse, scalability-wise. We didn’t want to get to the point where we’d have to delete data or cancel projects,” says Levente Otti, Head of Data at Emarsys. “In Google Cloud, we saw a platform that could scale with our ambitions and be optimized for AI and real-time solutions.”
Minimal maintenance, unlimited scale with Google Cloud
Digital marketing is a highly competitive environment. Emarsys works alongside big players with a huge market share on the one hand and smaller, specialist companies on the other. It has thrived by successfully combining the all-inclusive offerings of the former with the agility of the latter, constantly looking for ways to innovate and improve. In recent years, the company had started to feel that the ability to handle large quantities of data was no longer enough. The next challenge was speed. “We truly believe that in the future, everything will be done in real time, including data processing, analytics, and AI predictive models,” Levente says.
At the start of 2016, Emarsys’ existing data warehouse was a software-as-a-service solution running on-premises, which required hardware and software maintenance in order to keep up with the company’s growing appetite for data-heavy use cases such as prediction and analytics. The existing platform had proven its worth processing large amounts of data in batches, but its real-time capabilities were limited. Moreover, Emarsys had begun to experiment with AI technology, but found that its data warehouse couldn’t scale to accommodate some of the more resource-intensive processes, such as training the predictive models. The company decided that it needed a new, cloud-based data platform.
After evaluating some of the leading cloud providers, Emarsys chose Google Cloud for its mature AI capabilities and its ease of use. “With the other solutions, we still had to rent virtual machines and hardware and be responsible for maintenance. At the time, Google Cloud was the only provider that could take that management overhead away from us, while keeping customers accounted for on every query level,” Levente says.
To implement its new data platform, Emarsys teamed up with Google Cloud Partner Aliz. Over a series of meetings, workshops, and architecture reviews, Aliz helped Emarsys navigate the Google Cloud ecosystem to find the right products for the solution it was looking for. “Aliz really helped us set off in the right direction,” explains Levente.
With Google BigQuery, we can run queries which process terabytes of data, in seconds. We can also develop our own user-defined functions, incorporating Bayesian statistics into our predictive algorithms. That means we can take into account historical data, resulting in much more accurate predictions in a scalable way within seconds.”
—Levente Otti, Head of Data, Emarsys
Emarsys’ new data platform would actually be two: one platform for batch processing data and one for real-time analysis and interactions. Firstly, a proprietary publishing component gathered all the data points from Emarsys’ various channels including the website, mobile, emails, and custom events. With Cloud Pub/Sub and Cloud Dataflow, Emarsys transported and processed the data into BigQuery, which allows for further work and reviews that take into account errors or delayed events. After this, the data was exported to the main batch processing platform, which ran on BigQuery. For the real-time analytics, Emarsys used Cloud Bigtable to access data and Cloud Dataflow to pipeline it into the real-time platform, which could communicate with AI components or interaction components via an API to deliver real-time interactions with customers.
On top of the overall data infrastructure, Emarsys built a new AI platform with Google Cloud components. Training the predictive models had been an issue in the past due to the large number of resources required, so Emarsys chose to use Google Kubernetes Engine clusters, which can scale up and down on demand, without the need for hardware configuration or management. The trained models were held securely in Cloud Storage. From here, they were integrated with BigQuery for power and flexibility, allowing Emarsys to improve not just the speed of its AI predictions but also the quality.
“With Google BigQuery, we can run queries which process terabytes of data, in seconds,” shares Levente. “We can also develop our own user-defined functions incorporating Bayesian statistics into our predictive algorithms. That means we can take into account historical data, resulting in much more accurate predictions in a scalable way within seconds.”
Real-time insight, long-term satisfaction
Google Cloud enabled Emarsys to build a scalable data and AI platform that delivers powerful, actionable insights in real time. According to Levente, the company wanted to spend less time managing overload and more time considering how it should handle data. An immediate result of the new platform has been that data is now available in a scalable way, without hardware additions and management.
“With Google Cloud, we’ve been able to build a truly real-time data platform. The norm used to be daily batch processing of data. Now, if an event happens, marketing actions can be executed within seconds, and customers can react immediately. That makes us very competitive in our market.”
—Levente Otti, Head of Data, Emarsys
The clear and innovative pricing schemes of Google Cloud have also brought a new level of accountability to Emarsys’ costs in a way that wasn’t possible with its on-premises infrastructure. “Now that we only pay for what we use, we can assign costs to specific customers or queries, which has a huge impact on our pricing and product development strategies,” says Levente.
Thanks to the power of BigQuery and the scale at which it can handle data, Emarsys can now apply its analytics and AI tools to their full potential. “It’s very important to enable our clients to create the best possible experience for customers,” says Levente. At the same time, the company has cut its AI platform costs by 70% with Kubernetes while increasing scalability compared to the previous solution. The whole data platform was built to be scalable, and its first big test came during the retail peak of Black Friday, when it comfortably handled 250,000 events per second. “Perhaps the biggest impact on the business came with the real-time nature of the new platform,” says Levente.
“With Google Cloud, we’ve been able to build a truly real-time data platform,” he explains. “The norm used to be daily batch processing of data. Now, if an event happens, marketing actions can be executed within seconds, and customers can react immediately. That makes us very competitive in our market.”
Since implementing the new platform, Emarsys has continued to innovate with it and is about to release a new Real-Time Decision Framework, which will provide customers with even more real-time products and tools. The company continues to work with Aliz and Google Cloud, exploring other products such as Google BigQuery ML and TensorFlow to improve its AI processes. “We had a problem that we wanted to tackle now, and for us, Google Cloud was the best way of doing that,” says Levente. “But it was also about looking ahead. We felt that Google Cloud offered us the best way of future-proofing our platform.”
Google Cloud & Optiva Partnership Cements the Future of Telecom for Driving Strong Customer Experience

3186
Of your peers have already read this article.
3:00 Minutes
The most insightful time you'll spend today!
Editor’s note: Coming out of Mobile World Congress 2022, we are excited to share key learnings from our partner ecosystem on how to leverage advancements in 5G technologies to power customer experiences that seamlessly blend our physical and virtual world into one. The original version of this blog was published by Optiva, Inc. Please enjoy this updated entry from our partner.
Telecom operators and communications service providers (CSPs) are elevating customer experiences (CX) across the customer lifecycle. Today’s digital customers’ expectations, needs, usage behaviors and choices are growing and evolving exponentially. Therefore, it is critical to deliver superior and personalized digital customer experiences at each customer lifecycle touchpoint.
To succeed, you need to deliver a dynamic customer experience. Operators are embracing the mantra — next-level CX is the new currency — from onboarding and instant offer provisioning to delivering enhanced self service and supporting subscription renewals, queries and billing actions, and in-session improved experiences. There are commercial benefits to adapting to this paradigm, too, as key customer segments may see added value in enhanced experiences. With 5G, the Ericsson “5G consumer potential” report found that half of early adopters would be willing to pay 32% more for 5G services. Consider the advanced experiences that 5G is able to support, such as:
- Low-latency connectivity and real-time network slicing capabilities, delivering an immersive yet reliable augmented and virtual reality (AR/VR) experience. Imagine a soccer match with a rich 360-degree stadium experience from the customer’s choice of location that immerses a fan in the game excitement.
- AI-driven insights enabling operators to predict customer behavior patterns in real time and leveraging available network capacity to provide customers with personalized, just-in-time discounts and offers that can increase ARPU, enhance the customer experience and reduce churn.
- Proactive action, such as instantly optimizing 5G connectivity network slice bandwidth when the quality of service does not meet its promised level. This could include delivering assured, lag-free network performance to an online gamer for next-level gaming intensity or it could be about providing proactive, transparent reimbursements to end customers on the fly, preventing dissatisfaction complaints. Further, 5G with dedicated network slices also enables business applications that exceed an end-to-end SLA essential to a business-to-business (B2B) subscriber.
- Enabling immersive experiences across all services and touchpoints by redefining how consumers interact with offerings from anywhere, whether that’s a smartphone or tablet. What’s more, it’s about feeding intelligence gleaned from those customer interactions into a single view across all ecosystems in real time to gain a full picture of the customer.
How cloud shapes the future of telecom customer experience
Through cloud technology and cloud-native architectures, telecom operators and service providers have the opportunity to deliver such 5G use cases and differentiated offerings. Cloud maximizes the benefits and enables delivery, thereby dramatically improving and reimagining the possibilities for CX. This includes how customers connect, consume and buy services, and it strengthens customer affinity and loyalty.
Cloud and the technologies it maximizes, such as 5G, also present a wide range of innovative monetization opportunities beyond traditional telecom revenue streams and beyond connectivity. Thus, the highest priority for telecom must be on effectively and efficiently harnessing the potential capabilities for launching personalized service offerings for consumer and enterprise at a high velocity. As a result, the new customer engagement model requires agility, responsiveness and reliability to deliver these services across all touchpoints of the customer’s journey.
As such, Optiva and Google Cloud are engaging in a multi-year partnership to help CSPs enable faster time to innovation, flexible 5G monetization and operational cost savings, while driving strong customer experience. Leveraging the Google Cloud platform enabled by Anthos, which supports the deployment and operation of business support system (BSS) applications across public clouds, on-premises data centers and at the network edge, Optiva’s distributed solution deployment offers telecom operators new ways to monetize 5G networks through use cases such as private 5G, IoT and ultra-low latency edge solutions.
Gaining a competitive edge on the new playing field
The solution to these new BSS and monetization requirements lies in the cloud’s unique advantages. For example, to achieve agility, an essential cloud tool, the sandbox, allows operators to accelerate iterations to find optimal solutions. The sandbox shortens product cycles to a fraction of traditional timelines and empowers operators to reinvent their functionalities and service capabilities — lowering business risk and driving dramatic cost savings.
As a result, operators can increasingly explore, experiment, learn, launch and relaunch rapidly. This allows for the fast introduction of new and differentiated offerings, increased service velocity and cost-effective go-to-market opportunities. For that reason, a new competitive playing field is emerging and making the days of traditional and full digital transformations a thing of the past.
Instead, by leveraging cloud technologies, customer lifecycle opportunities and possibilities are born, such as:
- End-to-end digital onboarding experiences: Hassle-free digital customer onboarding in little time by leveraging next-gen BSS with embedded automation across the different modules. This digitizes the customer registration and ordering process, including customer verification, SIM allocation, and the selection and activation of a user’s choice of plans and more.
- Real-time offer optimization based on customer insights: On-the-fly optimization of offers based on AI-driven real-time insights to predict usage behavior.
- Handling ultra-low latency service quality with distributed systems: Delivering and charging for ultra-fast services from the edge rather than sending and processing them at a central cloud. This enables new business opportunities by leveraging new private 5G offerings.
- Assured service quality and complaint reduction through automated real-time network configuration: By consistently monitoring the network quality and application and taking corrective actions to match the SLA requirements, we can boost the user experience (e.g., if a user subscribes to an 8K video plan). Thus, if the bandwidth level drops below the agreed-upon resolution level, the service can push an update to the user and potentially offer them a complimentary added data bundle leveraging analytics, churn prediction models and insights.
- Maintaining a real-time single source of truth for customer data: Having a single, distributed repository of real-time updated customer data allows CSPs to deliver customer services more smoothly across all touchpoints.
- Expanding product catalog with a partner ecosystem: Leveraging open APIs to build and expand partner ecosystems to launch new products and services that enable CSPs to expand the services they provide customers and help increase their market relevance.
Cloud momentum accelerates, enabling revolutionized BSS and revenue models
Service providers are forging their paths and investing in and adopting cloud technologies. Cloud empowers operators beyond connectivity and volume offerings on data, text and voice. The technology offers more and unlocks the operator’s ability to meet specific user segment experience requirements in real time and differentiate offerings based on latency, capacity, throughput, speed and device type.
As a result, operators can shift to new product-driven monetization capabilities, allowing them to configure their BSS without heavy customizations or necessitating the expertise of their IT teams. Instead, they can now empower, for example, marketing teams — with minimal steps and product-specific expertise needed — to optimize rate plans in real time based on usage and experience analytics, roll out promotions and satisfy customer demand for a delightful experience.
The new currency across the customer lifecycle — next-level telecom BSS and CX
Operators need the capability to learn fast, fail fast, launch, and relaunch in quick cycles. This capability is growing more critical as Capex and Opex become challenged and protecting ARPU and increasing subscribers becomes harder in a cloud economy. Telecom operators are picking up speed for cloudification and reimagining the potential of their BSS systems. And with 5G, innovation driven by cloud-native capabilities and automation via machine learning, operators have a genuine opportunity to revolutionize customer engagement and deliver a next-level hyper-personalized CX — the new currency of 5G cloud.
3492
Of your peers have already watched this video.
1:30 Minutes
The most insightful time you'll spend today!
How Carrefour is Building the Future of Retail on Google Cloud
Multinational retailer Carrefour was facing the challenge of meeting increasing customer expectations and realized that it needed to innovate faster. That’s when the company decided to move its SAP workloads to Gooogle Cloud.
As a result, Carrefour transformed over 1,000 stores and redesigned its back office management with SAP on Google Cloud. Not just that, it also realized the added benefits of flexibility, agility, and disaster recovery.
Watch this video as Carrefour executives explain why they decided to move SAP to Google Cloud and the benefits the company derived.
More Relevant Stories for Your Company

Google Unveils New Cloud Region in Delhi NCR to Power India’s Digitization
In the past year, Google has worked to surface timely and reliable health information, amplify public health campaigns, and help nonprofits get urgent support to Indians in need. Now, we are continuing to focus on helping India’s businesses accelerate their digital transformation, deepening our commitment to India’s digitization and economic recovery.
Google Leads the Pack in Cloud Data Warehouse: Forrester Research
Cloud data warehouse solutions are changing the way firms build and support data platforms for insights. From provisioning a cloud data warehouse in minutes without requiring any technical expertise to allowing business analysts and other nontechnical users to access, store, and process large amounts of data for insights, they allow

Trading and Investment Companies will Increase Consumption of Cloud Services: Study Confirms
While some traditional financial services companies have more slowly transitioned to the cloud, capital markets firms have embraced cloud computing across their entire value chains — front-, middle-, and back-office. We wanted to understand the dynamics behind this rapid adoption, the most common use cases, and the types of technology

PaGaLGuY Turns to Google Cloud Platform to Power Leading India Education Network
Founded in 2006, PaGaLGuY started as a forum that enabled students, typically aged between 20 and 30 to discuss and seek advice on academic issues. By 2011, PaGaLGuY had increased its traffic to about 250,000 page views per month. The business is now one of India’s largest education networks and







