Editor’s note: Today we hear from the Lowe’s SRE team. They share about how they have been able to increase the number of releases they can support by adopting Google’s Site Reliability Engineering (SRE) framework and leveraging their partnership with Google Cloud. 

At Lowe’s, we’ve made significant progress in our multiyear technology transformation. To modernize our systems and build new capabilities for our customers and associates, we leverage Google’s SRE framework and Google Cloud, which helps us meet their needs faster and more effectively. With these efforts, we’ve been able to go from one release every two weeks to 20+ releases daily—about 20X more releases per month. 

Our SRE transformation didn’t happen overnight, though. Every step along the way brought some challenges. But looking back, we are excited to see how much we have accomplished for our customers as a result. 

Back in 2018, before adopting SRE practices, we were more reactive than proactive, following an “eyes on glass” approach. On-call structures and incident management efficiency were not at optimal levels with too many repetitive and manual tasks, resulting in operational toil. Production concerns were not surfaced into the product roadmap, which resulted in delays in making fixes.

Bootstrapping SRE at Lowe’s

As we moved from on-prem to Google Cloud, we decided to move from a monolithic- to microservices-based architecture. And to better manage this new architecture, we embarked on an SRE journey. 

Then as COVID-19 hit, we really had to accelerate this journey as customers increasingly moved to online ordering and delivery to meet their Total Home Improvement needs. To do so, we followed four key principles that allowed us to meet changing customer needs quickly and release fast and reliably.

1. Automate away toil 
   As we moved from traditional Ops to an SRE ecosystem, our biggest opportunity was reducing toil, so that engineers can spend time on activities that drive business impact and customer outcomes. We think of toil as work that is manual, repetitive,  tactical, devoid of enduring value—but automatable. So, to tackle toil, we focused on automating away the need for manual intervention. As an example, we made sure engineers were not the first point of contact for any alert. Any triage or resolution that an engineer can perform, a machine can be trained to do the same. We used supervised and unsupervised learning techniques to automate our toil. With a long-term goal of “no toil,” our SREs work on identifying and reducing toil to a manageable level across the organization.
2. Engineer alignment through roadmaps
   Our goal is to maximize the engineering velocity of developer teams while keeping products reliable. We want an engagement model where product, SRE and development teams are closely aligned. A key way we’ve been able to create this alignment is by having our SREs embedded into domain and product teams. Each domain has an SRE, who is  involved at the beginning stages of product development to ensure that the domain stakeholders are in alignment with the SRE initiatives. As such, SREs are able to improve the reliability, performance, scalability and launch velocity of the services throughout all phases of the service lifecycle. 
3. Adopt one-touch releases
   Our path to production used to contain many manual steps and validations, slowing the rate at which we released features. Additionally, we used to bulk all our releases together to deploy at once, which increased the risk of failure and created a longer feedback loop from production. To tackle this with an SRE mindset, we created a one-touch release process in which SREs review the product team’s pull requests. When approved, this triggers a DevSecOps pipeline that deploys the approved changes to production securely. This process created a safe, reliable and sustainable continuous delivery pipeline with quick feedback loops. Striking the right balance between speed, innovation and stability, we were able to increase our releases exponentially for the year, taking less than 30 minutes per release to deliver quality code, including various automated quality checks and processes, all in just one click. 
4. Embrace capacity planning
   To ensure our services have enough spare capacity to handle any surge in traffic patterns, our SREs emphasize capacity planning, making recommended capacity changes in the continuous delivery (CD) pipeline. They constantly monitor performance to make sure the service is robust, stable and available. And when there’s a sudden surge beyond the forecasted volume, SREs change the capacity on demand and document changes for the performance and domain teams. 

Capacity planning is especially important for us during peak holiday times such as Black Friday and Cyber Monday (BFCM). We lay out our SRE stability plan three months in advance and surface into the domain team’s product roadmap. This way development teams are able to allocate sufficient engineering time to reliability. We do performance testing to ensure the environment is able to sustain increased load over long periods of time and also handle sudden surges in traffic. We also do region failover testing at a global scale to validate the automatic failover duration of service level agreements  (SLAs), SRE and domain readiness. Additionally, we conduct Black Friday and Cyber Monday-specific destructive testing to validate customer experience, reliability and more.

Google Cloud’s Black Friday and Cyber Monday white-glove service played a key role in ensuring our success in both BFCM 2019 and BFCM 2020. This service included on-site visits from Google’s Customer Reliability Engineering (CRE) team who reviewed Lowe’s web architecture, capacity planning, operations practices for event risks, and presented workshops on topics such as incident response best practices. 

Looking ahead

There is always room for improvement, and at Lowe’s we aim to continuously improve our SRE practices. One thing that has worked well for us, which we plan to continue, has been our road shows, where senior SRE leads present to other SREs and application domain teams on the latest SRE principles and best practices, and to get input in real-time from them. 

Google’s tools and methodology have played an instrumental role in helping reshape our SRE practices and better serve our customers. We look forward to building on the momentum and partnership as we continue our SRE journey at Lowe’s. 

If you want to learn more about how to adopt SRE best practices on Google Cloud, check out our documentation. If you want to learn more about Google SRE, visit our website. Stay tuned for the next blogs with Lowe’s discussing how they trained their engineering talent to adopt SRE practices and tooling, and how they improved MTTR using SRE principles.

In previous blog posts, we’ve discussed the benefits that organizations can reap by measuring and then optimizing their overall ESG impact. In this post, we’ll look at the endgame: sustainable transformation into an organization that continually adapts to maximize its resilience while reducing risk. Where will my customers be with the evolving global community landscape? How can I design communities and products with sustainability in mind? Are my assets exposed to physical climate risk — drought, flood, wildfire? Adaptation and resilience are the board and legislative focus areas that the ambition loops created by Google Cloud Ready – Sustainability seek to accelerate. 

This is especially important due to the scrutiny of governments and financial institutions as they evaluate the viability of physical asset investments. These organizations want assurances that their investments will not be under undue environmental risk. Similarly, companies embarking on net-zero transitions can realize interest expense savings by providing the measurable, accountable assurances their financiers demand.

Leveraging analytics for climate and business resilience

We’ve said it before, but it bears repeating: The Google Cloud Ready – Sustainability program challenges the notion that sustainability is just a nice-to-have or merely good PR. Rather, we designed the program to help organizations adapt to a net-zero future with solutions that reduce risk, support growth, provide competitive advantages, and positively impact the bottom line. The following partners provide data and analytics that offer deep insights into the environment, natural resources, and social trends that point the way toward sustainable, profitable growth. 

Atlas AI has built a geospatial artificial intelligence platform that helps every organization anticipate changing societal conditions — where people live, where wealth and poverty are concentrated, how the physical makeup of communities is evolving, and more — to determine where to invest today to prepare for the world of tomorrow. Atlas AI customers, which range from the World Bank and global NGOs to multinational corporations, use the platform to accelerate growth, future-proof supply chains, target resources to vulnerable populations, and build greater resilience to climate change.  

Climate Engine’s SpatiaFi solution, powered by Google Cloud, helps financial services organizations connect economic assets to the world of Earth observation data supporting regulatory reporting, climate risk reduction, and new innovative sustainable finance offerings. Combining billions of planetary observations with peer-reviewed scientific methods, SpatiaFi is connecting Earth science to operational systems offering new levels of accuracy, transparency, scalability, and trust in climate and sustainability data in the context of financial decision-making.

Kumi Analytics‘ KACSAT solution was approved by OxCarbon as a methodology for the calculation and verification of voluntary carbon offsets in support of nature-based solutions. KACSAT leverages multiple satellite platforms and Google Earth Engine‘s state-of-the-art processing tools to enable science-based carbon assessment aligned to the Oxford Offsetting Principles of environmental integrity, additionality, permanence, and achieving net-zero emissions. Environmental organizations, financial institutions, and corporations can now offset their carbon footprints through the reforestation and conservation of millions of hectares of forests — something that was previously limited due to the cost and time of validating high-quality forest projects.

Tensorflight revolutionizes property underwriting with AI. Its cutting-edge tool provides underwriters and insurers with access to rich and highly accurate datasets for commercial and residential properties, enabling them to create superior insurance products. As pioneers of the first property-inspection platform based on convolutional neural networks, Tensorflight’s advanced technology connects seamlessly via API and utilizes ground-level imagery, satellite, and aerial data. This unique approach enables the company to generate precise replacement costs and conduct more comprehensive risk assessments. By automating property inspections and enhancing underwriting accuracy, Tensorflight empowers insurance providers to plan for the future with pragmatic decision-making. With Tensorflight, the future of property underwriting is transformed, offering unparalleled insights, efficiency, and sustainability.

Are you ready for sustainability transformation?

Transforming into a net-zero organization means much more than building resilience to climate change, as important as that is. It also means finding the customers of the future, achieving ongoing energy savings, more sustainable resource usage, and improved transparency to regulatory institutions and consumers. Google Cloud Ready – Sustainability partners offer certified solutions that can help organizations get there.

Look for Google Cloud Ready – Sustainability validated solutions on the Google Cloud Partner Directory Listing, Google Cloud Ready Sustainability Partner Advantage page, and — if applicable — via the Google Cloud Marketplace. We hope to help customers better understand how these technologies can help them meet their ESG goals, find the right solutions for their particular challenges, and implement solutions faster. 

Learn more about the Google Cloud Ready – Sustainability validation initiative and explore all our partners in sustainability transformation.

The mobile industry is extremely fast-paced. New smartphones arrive at rapid speed. Smartphone OS updates regularly add new features and capabilities. And by the end of 2018, 5G wireless, the game-changing successor to 4G, will begin to be available in some markets.

With more than 125 T-Mobile premium retail stores across the United States, MobileOne is at the forefront of the mobile industry’s constant, rapid-fire changes. The company is growing quickly, too, expanding its number of stores by 340% in just 3 years.

In this high-growth, non-stop environment, MobileOne’s mission is to keep frontline employees trained, informed, and updated at all times so that they, in turn, can serve as trusted experts to customers. Ultimately, the end game is to stay competitive by delivering the kind of positive experience that encourages customers to return, and to recommend MobileOne to friends.

But fulfilling this mission had become a major operational challenge for MobileOne. Information was typically created in the company’s productivity software suite and shared in emailed file attachments through the company’s on-premises email system. The method was slow and inefficient, and it could introduce errors if an outdated file was emailed.

In addition, there were no central, shared, easily accessible repositories of company knowledge. Thus, to find information shared a few months ago, team members might have to search their email archives for the appropriate file, and the search capabilities of MobileOne’s legacy email system were far from robust. Worse, a file that one employee suddenly needed may exist only on the hard drive of another employee who might be away on vacation.

“Without G Suite, I can’t imagine how we could keep up with everything.”

– Revel Stark, Director of Recruiting and Marketing, MobileOne LLC

MobileOne leaders recognized that a digital workforce transformation was needed to help the company stay focused on delivering excellent customer service. In response, the company began moving all employees to G Suite with the help of Suitebriar, a Google Cloud – G Suite Premier partner.

Sharing docs in seconds

G Suite applications help MobileOne team members collaborate, share information, and receive the latest training. G Suite also helps MobileOne move at an accelerated pace. For example, on a recent Sunday, Revel Stark, Director of Recruiting and Marketing at MobileOne, was on his way to the beach with his family when someone in the company needed a document from him right away.

“It took just seconds to launch the Google Drive smartphone app, do a quick search, and share the requested document,” says Revel. “Before G Suite, I’d have to start up my laptop, assuming I even had it with me, and search for the file. If the file wasn’t on my laptop’s drive, it might be Monday before I could have found and shared it. That just doesn’t work for us. Without G Suite, I can’t imagine how we could keep up with everything.”

Creating central repositories

“Now, with Drive and Sheets, everyone has access to the most recent, business-critical information at the same time. Having live spreadsheets helps us ensure that no important information gets lost, despite the rapid pace in which we work.”

– Revel Stark, Director of Recruiting and Marketing, MobileOne LLC

Drive, coupled with G Suite apps such as Docs and Sheets, streamlines document sharing at MobileOne by enabling the company to create central repositories of information that are easily accessible and, with Google search technology, easily searchable.

For example, the company used to track operations, growth metrics, and performance in its legacy productivity suite’s spreadsheet application. The spreadsheet files were shared via email. But given MobileOne’s rapid growth, the information in those spreadsheet files could become outdated quickly. And there was confusion at times about which file was the most recent version.

“Now, with Drive and Sheets, everyone has access to the most recent, business-critical information at the same time,” Revel says. “Having live spreadsheets helps us ensure that no important information gets lost, despite the rapid pace in which we work.”

MobileOne has set up a Team Drive for each of the company’s departments, too. “With Team Drives, department heads can make sure their department’s documents are well organized, up to date, and easy to find,” Revel says. “And they can give access only to the people who need it.”

Streamlining interview schedules

The thousands of apps in the G Suite Marketplace enable businesses to add even more functionality to the G Suite apps that their employees use every day. MobileOne uses a third-party scheduling app that integrates with Calendar to streamline the scheduling of job applicant interviews. “Job applicants can look at our calendar and pick an open slot to schedule their interview,” says Revel. “It speeds up the process by not having to go back and forth in email, trying to find a time that’s mutually agreeable.”

Simplifying admin duties

“Before G Suite, it was time-consuming to add a new user to our email system,” Revel says. “I’d have to ask our service provider, then wait for them to do it. But with the easy-to-use G Suite admin panel, we can add someone to Gmail ourselves in a minute or two.”

Receiving personalized service

To support the rollout, MobileOne worked with Suitebriar, a Google Cloud – G Suite Premier Partner. Suitebriar efficiently set up MobileOne on G Suite and now provides webinars and on-going support to help MobileOne users get more out of G Suite. “Suitebriar is incredibly responsive,” says Revel. “They know our business, they give us personalized service, and they get back to me quickly when I have questions, sometimes in as soon as 30 seconds.”

Driving new business with Forms

MobileOne uses Forms to help it organize public events and, in combination with Sheets, to measure the success of those events.

Last year MobileOne held 200 events across the company’s 125 stores and is planning to hold up to 400 events this year. The events are designed to connect MobileOne associates with local residents and bring people into its stores who might not otherwise visit.

“We developed a pre-event form that stores use to communicate with our corporate office about what’s needed to set up their event,” Revel explains. “And we have a post-event form that stores can fill out to capture what worked and what didn’t work at the event. The information in that form gets dropped into a Sheets spreadsheet, so we can track the success of events across all stores. It helps us focus on the events that drive the most traffic into stores, where we can connect with locals and show them how we can help them.”

Using Hangouts to train — and applaud

“When a team member in the field does something fantastic, we can gather a bunch of people in a conference room, get that team member into Hangouts, and give him or her a round of applause. It’s a great way to give someone a pat on the back.”

– Revel Stark, Director of Recruiting and Marketing, MobileOne LLC

At MobileOne, Hangouts is among the most used G Suite app. The company uses Hangouts to bring executives together in different locations for a weekly briefing. “Hangouts breaks down the barriers that geographical distances create, so that everyone gets quality time together,” Revel says.

Training is another key Hangouts use case. “Because keeping our people well informed is so important, we use Hangouts frequently to train team members,” says Revel. “We’re currently using Hangouts with a Hangouts Meet hardware in our corporate office for training, and we’re considering expanding that by adding 20 to 30 Hangouts Meet hardware in training rooms across all seven states where we operate.”

MobileOne uses Slides for training presentations, which are often scheduled using Calendar and presented in Hangouts. “Calendar makes it easy to set up training events and to see the availability of people you want to invite,” Revel says. When people find a time that works, they simply add a Hangouts link to the calendar event. Then during the Hangouts session, a presenter can launch a Slides presentation, and attendees can ask questions with Hangouts chat. All of this is easy to set up and the training happens in real-time, which gives it more of an impact.

The company is also leveraging Hangouts in a less conventional way. “When a team member in the field does something fantastic, we can gather a bunch of people in a conference room, get that team member into Hangouts, and give him or her a round of applause,” says Revel. “It’s a great way to give someone a pat on the back.

It’s simply amazing to witness how some of our systems integrators employ Google Cloud solutions to drive innovation in ways we at Google may never have considered—especially in healthcare.  According to analyst firm MarketsandMarkets, the market for the Cloud in healthcare is projected to grow 43% between 2020 and 2025 to nearly $65 billion, fueled by the need for better technology infrastructures and faster digital transformation.  

Healthcare and life sciences communities are looking to Google Cloud and its Service engagement model partners to improve collaboration and activate the power of medical data.  These transformations deliver robust data analytics and bring a much deeper perspective into health epidemics like COVID-19 to help save lives.

In the healthcare and life sciences industry, our Google Cloud partners and customers provide constant energy and inspiration–and are the magic in some key healthcare innovations globally.  

Let me show you how they are solving real-world business challenges.

Improving collaboration in Healthcare

Cloudbakers and Comanche County Memorial Hospital transitioned 2,000 employees to Google Workspace to improve collaboration, reduce costs, and increase security. By implementing a system that requires less maintenance while enabling mobile access to data and true collaboration, Comanche County Memorial Hospital saves $175,000 annually on licenses and helps medical professionals spend more time with patients.

“We chose Google Workspace because it cost a quarter of what we were paying previously, offers the kind of modern features that healthcare facilities need, and gave us data security and peace of mind.” —James Wellman, CIO, Comanche County Memorial Hospital

Accessing previously locked down medical data

Google Cloud and Quantiphi supported advances in cloud-based machine learning services to reduce infrastructure costs, unlock new paths of treatment, and dramatically reduce the amount of time it takes to evaluate scanned imagery following a stroke.  John Hopkins University BIOS Division has been working on medical imaging to accelerate insights from scans on approximately 500 patients from 2,500 hours to 90 minutes, and lead to more accurate decision-making for brain injury patients that will ultimately improve medical outcomes.  

“We’ve aligned closely with the goal of showing that a cloud-based, AI-driven approach is robust and that it can be performed while protecting personal PHI. In terms of costs, the cloud has definitely reduced some of the traditional financial demands of our research.” —Daniel F. Hanley, Jr., M.D., Director, Johns Hopkins University BIOS Division

Improving data access with the flexibility of Google Cloud

With the help of MediaAgility, TRIARQ migrated to Google Cloud to modernize their platform, build new applications, and expand their global footprint.  With BigQuery, TRIARQ can now consolidate all transactional data, past and present, into a single location to report on specific insights and predict future data from e-prescriptions to complex revenue-cycle data and value-base analysis.

“The future of this industry will need to be a global one, and we can no longer be stuck in legacy systems if we want to survive. Having a team that is passionate about supporting us in this journey is definitely a plus point.”—Yaw Kwakye, Co-founder and Chief Architect, TRIARQ

Increasing visibility to COVID-19 outbreaks for actionable insights

As part of its response to the COVID-19 pandemic, HCA Healthcare chose to work with Google Cloud and SADA to create a national portal that increases visibility into outbreaks in 3,100 counties across the country in just 8 weeks.  Now, the portal generates 30,000 new analytical views each day that can help inform private and public sector decision making for reopenings, closures, hot spots, and many other population health management activities. 

“This project required deep knowledge of AI, and consumer-facing platforms, as well as healthcare. Google brought together the ideal combination of product, people, and partners. Google Cloud’s healthcare-specific products along with SADA’s expertise in the healthcare IT space made this partnership the perfect choice to move quickly and intelligently.”—Dr. Edmund Jackson, Chief Data Officer, HCA Healthcare

We’re committed to building the technology, resources, and services through Partner Advantage to help our partners address this opportunity.  Looking for a solution focused partner in your region who has achieved Expertise and/or Specialization in your industry?  Search our Global Partner Directory.  Not yet a Google Cloud partner? Visit Partner Advantage and learn how to become one today!