Google Cloud Migration Speeds Up The New York Times' Journey to New Normal - Build What's Next
Case Study

Google Cloud Migration Speeds Up The New York Times’ Journey to New Normal

5802

Of your peers have already read this article.

2:30 Minutes

The most insightful time you'll spend today!

The New York Times' Google Cloud and BigQuery journey transformed their data architecture. It helped build and leverage comprehensive datasets to allow newsrooms to operate at speed and quickly adapt to the new normal. Learn how.

Like virtually every business across the globe, The New York Times had to quickly adapt to the challenges of the coronavirus pandemic last year. Fortunately, our data system with Google Cloud positioned us to perform quickly and efficiently in the new normal. 

How we use data

We have an end-to-end type of data platform; on one side we work very closely with our product teams to collect the right level of data that they’re interested in, such as which articles people are reading, and how long they’re staying onsite. We frequently measure our audience to understand our user segments, and how they come onsite or use our apps. We then provide that data to analysts for end-to-end analytics. 

On the other side, the newsroom is also focused on audience, and we build tools to help them understand how Google Search or different social promotions play a role in a person’s decision to read The New York Times, and also to get a better sense of their behavior on our pages. With this data, the newsroom can make decisions about information that should be displayed on our homepage or in push notifications. 

Ultimately, we’re interested in behavioral analytics—how people engage with our site and our apps. We want to understand different behavioral patterns, and which factors or features will encourage users to register and subscribe with us. 

We also use data to create or curate preferences around personalization, to ensure we’re delivering to our users fresh content, or content that they may not have normally read. Likewise, our data also gets used in our targeting system, so that we can send out the right messaging about our various subscription packages to the right users.

Choosing to migrate to Google Cloud

When I came to The New York Times over five years ago, our data architecture was not working for us. Our infrastructure was gathering data that proved harder for analysts to crunch on a daily basis. We were also hitting hang ups with how that data was streaming into our system and environment. Back then we’d run a query and then go grab some coffee, hoping that the query would finish or give us the right data by the time we came back to our desks. Sometimes it would, sometimes it wouldn’t.

We realized that Hadoop was definitely not going to be the on-premises solution for us, and that’s when we started talking with the Google Cloud team. We began our digital transformation with a migration to BigQuery, their fully managed, serverless database warehouse. We were under a pretty aggressive migration timeline, focusing first on moving over analytics. We made sure our analysts got a top-of-the-line system that treated them the way that they themselves would want to treat the data. 

One significant prominent requirement in our data architecture choice was to enable analysts to be able to work as quickly as they needed to provide high-quality deliverables for their business partners. For our analysts, the transition to BigQuery was night and day. I still remember when my manager ran his very first query on BigQuery and was ready to go grab his coffee, but the query finished by the time he got up from his chair. Our analysts talk about that to this day.

While we were doing the BigQuery transition, we did have concerns about our other systems not scaling correctly. Two years ago, we weren’t sure we’d be able to scale up to the audience we expected on that election day. We were able to band-aid a solution back then, but we knew we only had two more years to figure out a real, dependable solution. 

During that time, we moved our streaming pipeline over to Google Cloud, primarily using App Engine, which has been a flexible environment that enabled quick scaling changes and requirements as needed. Dataflow and Pub/Sub also played significant roles in managing the data. In Q4 of 2020 we had our most significant traffic ever recorded, at 273 million global readers, and four straight days of the highest traffic we’ve had compared to other election weeks. We were proud to see that there was no data loss.

A couple of years ago, on our legacy system, I was up until three in the morning one night trying to keep data running for their needs. This year, for election night, I relaxed and ate a pint of ice cream because I was able to more easily manage our data environment, allowing us to set and meet higher expectations for data ingestion, analysis and insight among our partners in the newsroom.

How COVID-19 changed our 2020 roadmap 

The coronavirus pandemic definitely wasn’t on my team’s roadmap for 2020, and it’s important to mention here that The New York Times is not fundamentally a data company. Our job is to get the news out to our users every single day in paper, on apps, and onsite. Our newsroom didn’t expect the need to build out a giant coronavirus database that would enrich the news they share every day. 

Our newsroom moves quickly, and our engineers have built one of the most comprehensive datasets on COVID-19 in the U.S. With Google, The New York Times decided to make our data publicly available on BigQuery Google’s COVID-19 public dataset. Check out this webinar for more details on our evolution architecture:https://www.youtube.com/embed/mtNlrFpschU?enablejsapi=1&

Flexible approach

We have many different teams that work within Google Cloud, and they’ve been able to pick from the range of available services and tailor project requirements keeping those tools available in mind.

One challenge we think about with the data platform at The New York Times is determining the priorities of what we build. Our ability to engage with product teams at Google though the Data Analytics Customer Council allows us to see into the BigQuery roadmap, or the data analytics roadmap, and plays a significant role in determining where we focus our own development. For example, we’ve built tools like our Data Reporting API, which reads data directly from BigQuery, in order to take advantage of tools like BigQuery BI Engine. This approach encourages our analysts to be better managers of their domains around dimensions and metrics, but not have to focus on building caching mechanisms of their data. Getting that kind of clarity helps us plan how to build The New York Times in the new normal and beyond.

If you are interested to learn more about the data teams at the New York Times, take a look at our open tech roles here and you’ll find many interesting articles at NYT data blog.

Blog

The Journey ahead for Google Cloud and SAP

3425

Of your peers have already read this article.

1:30 Minutes

The most insightful time you'll spend today!

Many customers choose Google Cloud for SAP implementation owing to its superior data and analytics services. To help customers with freedom, flexibility and ROI in cloud investments, the duo announced expanded partnership. Learn now!

The partnership between Google Cloud and SAP is entering a new chapter. Over the years, our close partnership with SAP and our mutual dedication to customers has inspired us to do some of our most unique and innovative work—such as building Fast Restart and Memory Poisoning Recovery capabilities for S/4HANA workloads, offering real-time access to our advanced machine learning (ML) capabilities, and developing integrations with the industry’s best cloud security and network infrastructure.

With an expanded, strategic partnership, we are capitalizing on one of the cloud’s most valuable benefits: enabling customer choice without driving up cost or complexity. Customers have shared that they appreciate what Google Cloud has done over the years to support their ability to choose—from our support for multi-cloud environments to our track record with open-source technologies like Kubernetes, to name a couple of examples. 

“We chose Google Cloud to support our SAP implementation. Our decision had a lot to do with the relationship between Google Cloud and SAP and also for the applications and services that are offered by Google Cloud, like BigQuery, which are helping to enable data and analytics within our organization.”— Sam Moses, Vice President of Corporate Systems, The Home Depot

Now, we’re placing this same emphasis on customer choice, flexibility, and freedom to help SAP customers get greater ROI from their cloud investments in three ways:

  • Execute seamless business transformations that deliver all of the benefits of modern, cloud-native applications and services—quickly, economically, and without disrupting day-to-day operations
  • Migrate critical business systems to the cloud, in order to lock in OpEx savings and refocus their IT teams on strategic business technology initiatives
  • Augment existing business systems with cloud capabilities in AI, ML, and analytics

The cornerstone of our expanded partnership is the RISE with SAP program—a comprehensive set of tools, technology, and expert support for a fast, predictable, cost-effective path to business transformation with SAP in the cloud. RISE is also designed to meet every SAP customer on their own terms, whether they’re undertaking an enterprise-wide cloud migration or pursuing an incremental or hybrid-cloud approach. 

To support customers taking advantage of RISE with SAP, Google Cloud will work even more closely with SAP to accelerate customers’ cloud migrations and business process modernization across several key areas. This includes ensuring global availability of multiple SAP services and products—such as SAP Business Technology Platform and SAP HANA Cloud—on Google Cloud’s reliable, scalable infrastructure and high-speed network. Google Cloud is also committed to bringing SAP’s solution for RPA in manufacturing to the Google Cloud Marketplace to support business process modernization. 

Additionally, integrating our AI, ML, and analytics capabilities with key SAP solutions will help companies augment their business systems with sophisticated capabilities and technologies so they can draw more value out of the data stored with them.

“We can deliver our 160 key business insights two or three orders of magnitude faster with BigQuery. When you think about our marketing campaigns and month-end finance processes, we’re talking about jobs that used to take five to eight hours to complete that are now running in seconds.”— Dan Semmens, Head of Data and AI, ATB Financial

Join us on the journey ahead

There’s so much more to come from SAP and Google Cloud. Learn more about our commitment to enabling digital transformation and hear more from customers about their SAP on Google Cloud deployments.

Case Study

BURGER KING Germany: Serving Up Marketing Insights and Supply Chain Visibility Easily

13957

Of your peers have already read this article.

8:30 Minutes

The most insightful time you'll spend today!

BURGER KING Germany built an ETL pipeline that channels ticket data for every sale into BigQuery allowing the marketing team to easily see exactly which products are selling well so that they can tweak promos. The data is also helps monitor the supply chain to make sure enough produce is delivered to restaurants in response to changes in demand.

Do hamburgers really come from Hamburg? This may still be a matter of debate, but the popularity of American-style burger joints not just in Hamburg but all over Germany, is clear. Germany’s top two fast food companies are both burger chains. One of them is BURGER KING®, a global brand that welcomes more than 11 million customers worldwide every day. The company arrived in Germany in 1976, when its first restaurant opened in Berlin. It now operates more than 100 restaurants across Germany, with franchisees operating more than 600 restaurants of their own.

“In the fast food industry, being able to move quickly is very important. That means delivering the right promotion to our app or launching a viral campaign within days. To do that across more than 700 restaurants, we need the support of the right technology.”

Oliver Mielentz, IT Manager, BURGER KING® Deutschland GmbH

Previously a subsidiary of the U.S. business, BURGER KING® Germany became an independent company in 2015. As a result, it needed to develop its own IT infrastructure, and the changeover needed to happen fast. “We had to put in place systems that would work for the entire network of franchisees and enable us to easily roll out campaigns,” explains Oliver Mielentz, IT Manager at BURGER KING® Deutschland GmbH.

With the help of Google Cloud Premier partner Cloudwürdig, BURGER KING® Germany chose Google Cloud and G Suite as the right combination to suit its needs.

“In the fast food industry, being able to move quickly is very important,” says Oliver. “That means delivering the right promotion to our app or launching a viral campaign within days. To do that across more than 700 restaurants, we need the support of the right technology.”

Building a franchisee platform in just three months

When a business has multiple franchisees, it’s important to make sure everyone is on the same page, especially in the fast-paced fast food environment. “We have to collate data from all our franchisees and produce reports quickly in order to react to changes in customer behavior,” explains Oliver. “That means processing every transaction that takes place in our restaurants.” Following the restructure, BURGER KING® Germany also needed to build a secure invoicing system with data storage and optimize its communication channels.

“Using Tableau with BigQuery, we’re able to produce reports very quickly. Previously, it took much longer, as data had to be fetched manually. Our reaction time is now almost a business day faster.”

Oliver Mielentz, IT Manager, BURGER KING® Deutschland GmbH

With support from Witter-IT, BURGER KING® Germany chose Cloudwürdig to build its BKD Connect internal platform on Google Cloud. Thanks to the ready-to-go tools on Google Cloud, it was able to put its invoicing system and data warehouse in place in just three months.

For the BURGER KING® Germany data warehouse, Cloudwürdig built an ETL pipeline that channels ticket data for every sale into BigQuery. “Data is gathered from the restaurants,” says Oliver, “and using Tableau with BigQuery, we’re able to produce reports very quickly. Previously, it took much longer, as data had to be fetched manually. Our reaction time is now almost a business day faster.”

As the ticket data for every transaction is stored in BigQuery, the marketing team can easily see exactly which products are selling well. That’s crucial for tweaking promotions as well as monitoring the supply chain to make sure enough produce is delivered to restaurants in response to changes in demand.

“Thanks to BigQuery, we have a speedy data pipeline that enables us to react on the same day to changes in the market and eliminate bottlenecks in production,” says Oliver.

Switching to G Suite to improve communication

To enable franchisees to sign in to its BKD Connect Platform, BURGER KING® Germany needed a secure authentication system. To solve that problem, it chose to provide franchisees with G Suite accounts. “It’s really easy to set up a new franchisee on the platform. I just create a new G Suite account and Drive folder for it, and it’s ready to go,” says Oliver. G Suite also helps the franchise network to run efficiently, as daily reports are automatically saved to Drive and shared to the appropriate regional network. “Thanks to that system, it’s much easier for any team at headquarters to access the information it needs,” Oliver explains.

BURGER KING® Germany also recently extended its use of G Suite across the whole company. “Following an evaluation of our previous email and productivity software, I made the decision to switch solely to G Suite,” says Oliver. BURGER KING® Germany employees now use GmailCalendar, and Drive for their day-to-day productivity needs. “We only just completed the migration, but already, everyone’s happy,” says Oliver. “It’s so easy to share a file using Drive or set up a meeting on Calendar.”

“We’re big fans of Hangouts Meet, and we have two rooms here at our Hanover headquarters equipped with Hangouts Meet hardware,” Oliver adds. “The speech quality is good, and it’s helpful to be able to see every participant, especially when you’re running a meeting with multiple franchisees.”

Optimizing infrastructure to power innovative campaigns

The BURGER KING® app, available for iOS and Android, helps the company to deliver a great customer experience. Through their MyBK accounts, guests can access coupons and special promotions. “We had a really interesting campaign for Easter: guests used the app to hunt for virtual Easter eggs,” explains Oliver. “We knew it was going to be big, and our previous back end wouldn’t have been able to handle the traffic.”

To enable the marketing campaign to go ahead, BURGER KING® Germany moved the back end of the app, along with its website, to Google Cloud. For developing and running its web and app back ends, it now uses App Engine and virtual machines on Compute Engine, as well as Memorystore and Cloud Functions. For monitoring and logging, it uses Stackdriver, and Cloud CDN and Cloud DNS to easily handle its traffic.

“We ran the campaign without any performance issues, even though we were receiving several million hits a day,” says Oliver. Since migrating the back end to Google Cloud, the marketing team also launched the popular “Escape the Clown” campaign. “That campaign blew our minds!” says Oliver. “It wouldn’t have been possible without Google Cloud, because it required a lot of back end capacity.”

To develop the app infrastructure it needs, BURGER KING® Germany relies on Cloudwürdig. “Working with Cloudwürdig is great because the team has the same agile mindset as us,” says Oliver. “When we have a new idea, we just set up a meeting, and in a couple of days the new infrastructure is in place. For Escape the Clown, it only took a few weeks to get everything ready to launch.”

Leveraging integrated tools to grow the business

Using Google Cloud together with G Suite enables BURGER KING® Germany to run its franchise network efficiently, while keeping its IT team lean. “Google Cloud and G Suite are the perfect fit for the way of working at BURGER KING® Germany,” says Oliver. “Many of the company’s operatives are often on the road, visiting restaurants and franchisees. With these tools, they can work flexibly and react quickly to the situation on the ground.”

“In order to grow the business, we need to use the data we receive every day to understand exactly what is happening in our restaurants. With the tools provided by Google Cloud, we can get more guests through the door and offer them a better experience.”

Oliver Mielentz, IT Manager, BURGER KING® Deutschland GmbH

It also helps to keep infrastructure costs under control. “With Google Cloud, we only pay for what we use, which is really important for us,” Oliver explains. “It means we can scale up quickly if we see an opportunity to react to a trend in customer behavior and launch a new marketing campaign that resonates with the moment. When it’s finished, we can then scale down again, and that definitely saves us money.”

BURGER KING® is now working with Cloudwürdig to add more functionality to the BURGER KING® app using Google Kubernetes Engine. “We like to work with customers long-term to support their digital transformation. BURGER KING® Germany is a great example of how one project can develop into a great collaboration,” says Benny Woletz, Managing Director of Cloudwürdig.

BURGER KING® also plans to expand its presence in Germany and gain a greater market share by further tailoring both its marketing and the way it runs its restaurants to answer its guests’ needs. “In order to grow the business, we need to use the data we receive every day to understand exactly what is happening in our restaurants,” says Oliver. “With the tools provided by Google Cloud, we can get more guests through the door and offer them a better experience.”

Blog

STAC-M3 Tick History Analytics in Google Cloud Benchmark Results Reveals it is 18X Faster than Previous Version

4763

Of your peers have already read this article.

5:00 Minutes

The most insightful time you'll spend today!

Google Cloud's redesigned STAC -M3™ benchmark suite was recently audited to assess performance of tick analytics stack and other data points that are relevant for financial firms to perform I/O intensive and compute-intensive tasks with market data.

The Securities Technology Analysis Center (STAC®), an organization that improves technology discovery and assessment in the finance industry through dialog and research, recently audited the STAC-M3™ benchmark suite on Google Cloud (SUT ID KDB211210). These enterprise tick-analytics benchmarks assess the ability of a solution stack such as database software, servers, and storage, to perform a variety of I/O-intensive and compute-intensive operations on historical market data.

Following up on our previous STAC-M3 benchmark audit (SUT ID KDB181001), a redesigned Google Cloud architecture leveraged the most recent version of kdb+ 4.0, the time-series database from KX, and achieved significant improvements: 35 out of 41 benchmarks ran faster in the new cluster – by up to 18x faster than Google Cloud’s prior results. Key highlights include the following:

Compared to the previous STAC-M3 Antuco suite results on Google Cloud:

  • Was faster in 13 of 17 mean response-time benchmarks
  • Was 18x faster – a 94% reduction in run time – in the version of Year-High Bid that allows caching (STAC-M3.ß1.1T.YRHIBID-2.TIME), which also set an overall record for all published results
  • Had 9x higher throughput in Year-High Bid (STAC-M3.ß1.1T.YRHIBID.MBPS)

Compared to the previous STAC-M3 Kanaga suite results on Google Cloud:

  • Was faster in 22 of 24 mean response-time benchmarks
  • Was over 10x faster in all four Market Snapshot workloads (STAC-M3.ß1.10T.YR[2,3,4,5]-MKTSNAP.TIME)
  • Had 5x the throughput in Year-High Bid involving 2 years of data (STAC-M3.ß1.1T.2YRHIBID.MBPS)

“The STAC-M3 standard was designed by financial firms to reveal the performance of tick analytics stacks. Generational improvements like those exhibited by Google Cloud’s most recent STAC-M3 audit, are important data points for firms evaluating new architectures for performance and scale,” said Peter Nabicht, President of STAC.

These performance results may translate to real-world advantages that may be difficult for investment firms to achieve in static and costly on-premises environments: immediate answers in high data velocity markets, more thoroughly explored research theories by adding data or new quantitative approaches, and reduced costs by releasing cloud resources more quickly.

STAC-M3: High-speed tick analytics


Designing for record-breaking results

In our STAC-M3 audit, the stack under test (SUT) was designed to take advantage of horizontal scalability in the cloud by sharding data across independent compute nodes. The cluster of 12 Google Compute Engine N2 instances was powered by Intel Cascade Lake, with each node using 32 vCPUs, 160GiB of memory, and 9TiB of local NVMe SSDs. The full STAC-M3 Antuco and Kanaga data set was split across the cluster and kdb+ scripts distributed queries between nodes.

Figure 1.  Sharded kdb+ 4.0 STAC-M3 architecture on Google Cloud.

This configuration was the sweet spot for this particular workload, but this architecture does not need to be limited to 12 nodes for other workloads – the data sharding algorithm could scale to any number of nodes as required by workload demands. Since scaling out the cluster in this manner increases the total pool of available storage, this architecture can continue scaling out to petabytes of storage across hundreds of nodes.

The ability to spawn large clusters with hundreds of thousands of processors on demand at low cost, and to delete the resources when jobs complete, not only changes the economics of running computations on large financial data sets, it also opens up opportunities to explore solutions to new types of problems that were previously overlooked due to the constraints of fixed hardware on-premises. You can check the pricing of this VM configuration using the Google Cloud Pricing Calculator. The costs can be reduced even further by using preemptible VMs.

While the new cluster used a similar number of nodes, cores, and total memory as the previously-audited cluster, the redesigned architecture allowed us to harness the low latency and high throughput of Local NVMe SSDs.

Resources on demand

The cluster was created on demand using Terraform and Ansible during testing and auditing. The use of infrastructure as code (IaC) techniques ensured that the cluster, fully loaded with the STAC-M3 data set, could be created when needed and then removed when benchmarking was complete. It also meant that the cluster configuration was enforced by code on each deployment, eliminating configuration variance and drift. The full IaC definition to create the cluster can be retrieved from the report in the STAC Vault.

Each time the cluster was created, data was streamed to Local SSDs from Google Cloud Storage, our reliable and secure object storage, at up to the line rate of 32Gbps per node. The entire 57TiB STAC-M3 Antuco and Kanaga data was replicated from Cloud Storage to local storage in approximately 20 minutes.

Figure 2. STAC-M3 sharded data set.

Since each node was independent and responsible for its own shard of data, doubling the cluster size would cut the synchronization time in half, or copy twice as much data in the same amount of time. Using higher bandwidth options of up to 100Gbps would triple the possible throughput for a relatively small incremental cost, trading an approximately 11%-23% price increase at current list prices for a 200% data synchronization performance increase. Taking advantage of fast networking to cache sharded data in parallel to a large cluster makes storing bulk data in Cloud Storage viable for even the largest workloads.

For quants working on vast data sets in sprawling compute clusters, the ability to fully describe infrastructure as declarative code, create elastic resources on demand, cache data quickly from cheap bulk storage, and turn resources off when computations complete is a dramatic change compared to waiting months to grow on-premises clusters – and a compelling reason to use cloud infrastructure.

To see how we designed and optimized the cluster for API-driven cloud resources, read our new whitepaper.

STAC-A2™: Calculating derivatives risk


In 2018, we showed that cloud instances can outperform bare metal when analyzing large tick history data sets in the demanding suite of STAC-M3 benchmarks. Last year, Google Cloud’s partner Appsbroker showed that the same was true for calculating derivatives risk in STAC-A2 on Google Cloud. You can read about how Appsbroker built its record-breaking STAC-A2 compute cluster on Google Cloud in its blog post, or access the STAC Report directly. Here are the highlights:

Compared to all other publicly reported solutions, this solution, based on a cluster of 10 virtual machines, had:

  • The highest throughput (STAC-A2.β2.HPORTFOLIO.SPEED)
  • The fastest cold time in the large problem size (STAC-A2.β2.GREEKS.10-100k-1260.TIME.COLD)

Compared to a solution involving an 8-node, on-premises cluster (SUT ID INTC181012), this 10-node, cloud-based solution:

  • Had 5 times the maximum paths (STAC-A2.β2.GREEKS.MAX_PATHS)
  • Had 10% greater throughput (STAC-A2.β2.HPORTFOLIO.SPEED)
  • Was 18% faster in cold runs of the large problem size (STAC-A2.β2.GREEKS.10-100k-1260.TIME)
  • Was 9% faster in cold runs of the baseline problem size (STAC-A2.β2.GREEKS.TIME.COLD)

Finding market advantages with Google Cloud


Across the investment management industry, every firm is seeking many of the same competitive advantages. However, finding unique opportunities and managing larger and larger data sets is becoming a major strain. Cloud is fundamentally changing how quants tackle the problem while empowering them to manage risk and generate higher returns.

Building on-premises computing clusters with tens or hundreds of thousands of cores and petabytes of storage requires huge up-front investments and lead time measured in months or years. Google Cloud makes the same scale available to its customers, provisioned on demand and paid per use. More importantly, the elasticity of cloud resources enables agility that is simply not available in a fixed data center cluster – the agility to explore, experiment, iterate, and respond to markets faster than before.

Scaling out to tens of thousands of cores in minutes and then removing the resources immediately not only changes the speed at which questions can be answered; it encourages different and more frequent questions, asked simultaneously on many independent clusters, free from the constraints of fixed on-premises hardware.

It is this flexibility and power that enables financial services firms to leverage larger data sets and get results, backtest, research, and analyze large amounts of data, faster and whenever they need it.

Download our whitepaper to learn more about our latest STAC-M3 tick history analytics benchmark results and how to optimize cloud infrastructure for high-speed market data analysis.

How-to

What You Can Learn from ‘Up or Out’ Framework for Cloud Adoption

3232

Of your peers have already read this article.

1:30 Minutes

The most insightful time you'll spend today!

When it comes to cloud adoption, there can be no 'one-size-fits-all' approach. Google Cloud's detailed white paper explores the up, our and both framework to help your organization decide the most palatable way to benefit from cloud migration.

In times of significant disruption, organizations are faced with three choices: Retrench within legacy solutions, pause and do nothing while waiting for more data or different circumstances, or press ahead, potentially even accelerating to realize the desired outcome. In such an environment, it is critical to ensure you’re delivering the greatest possible impact to the business. 

In Google Cloud’s Office of the CTO, or OCTO, we have the privilege of co-innovating with customers to explore what’s possible and how we can re-imagine and solve their most strategic challenges. These collaborative innovation engagements are often core to critical transformational projects, which often include the rehosting, evolution, and at times re-architecture of existing business solutions. 

We are happy to offer this brand new white paper, where we have distilled the conversations we’ve had with CIOs, CTOs, and their technical staff into several frameworks that can help cut through the hype and the technical complexity, to help devise the strategy that empowers both the business and IT. We called one such framework “up or out.” (And we don’t mean some consulting firm’s hard-nosed career philosophy.) 

One model that we found can help enterprises chart their cloud adoption journey delineates cloud migration along two axes—up and out, and we’ll cover this in much greater detail in the white paper itself.

cloud migration.jpg
Click to enlarge

As you can see, there isn’t a single path to the cloud—not for individual enterprises and not even for individual applications. The up or out framework can help an IT organization and its leadership characterize how they can best benefit from migrating their services or workloads. The framework acts as a general pattern that highlights the continuum of approaches to explore, and you can learn all about it by downloading this detailed white paper.

Or, if you’re really ready to jump start your migration today, you can take advantage of our current offer by signing up for a free discovery and assessment.

Case Study

How Lowe’s SRE Team Decreases Mean-time-to-recovery (MTTR)

3370

Of your peers have already read this article.

1:30 Minutes

The most insightful time you'll spend today!

After increasing the number of releases with the adoption of Google's SRE framework on Google Cloud. Lowe's SRE team managed to decrease their mean-time-to-recovery (MTTR) by over 80 percent. Learn how!

Editor’s Note: In a previous blog, we discussed how home improvement retailer Lowe’s was able to increase the number of releases it supports by adopting Google’s Site Reliability Engineering (SRE) framework on Google Cloud. Lowe’s went from one release every two weeks to 20+ releases daily, helping meet its customer needs faster and more effectively. Today, the Lowe’s SRE team shares how they used SRE principles to decrease their mean-time-to-recovery (MTTR) by over 80 percent.

The stakes of managing Lowes.com have never been higher, and that means spotting, troubleshooting and recovering from incidents as quickly as possible, so that customers can continue to do business on our site. 

To do that, it’s crucial to have solid incident engineering practices in place. Resolving an incident means mitigating the impact and/or restoring the service to its previous condition. The average time it takes to do this is called mean time to recovery (MTTR). Tracking this metric helps us stay on top of the overall reliability of our systems at Lowe’s, while simultaneously improving the speed with which we recover. Our goal is to keep the MTTR metric as low as possible, so that failures don’t negatively impact our business. Here are the four areas we addressed to drive holistic improvement in our MTTR.

Lowe’s incident reporting process

To reduce MTTR, we created a seamless incident reporting process following SRE principles. Our incident reporting process is a workflow that starts at the time an incident occurs, and ends with an SRE captain who closes the action items after a postmortem report. With this approach, we are able to limit the number of critical incidents. The reporting process involves three core components: monitoring, alerting, and blameless postmortems.

Monitoring and alerting

Having proper monitoring and alerting in place is crucial when it comes to incident management. Monitoring and alerting tools let you detect issues as soon as they occur, and notify the right person in the shortest possible time to take action. From a measurement standpoint, we track this as our mean time to acknowledge (MTTA). This is the average time it takes from when an alert is triggered, to when work on the issue begins.

At the time of an incident, our monitoring and alerting tools notify the on-call SRE first responder via PagerDuty in the form of a phone call, text message and email. Our SRE software engineering team has done a lot of automation to enable various Service Level Indicator (SLI) alerts and Service Level Agreement (SLA) notifications. The on-call SRE then initiates a triage call with our service/domain stakeholders to resolve the incident. As a result, we reduced our MTTA from 30 minutes in 2019, to one minute – a 97 percent decrease. 

Blameless postmortems: learning from incidents

A postmortem is a written record of an incident, its impact, the actions taken to resolve it, the root cause and the follow-up actions to prevent the incident from recurring (see example here). A blameless postmortem builds on that and is a core part of an SRE culture, and our culture at Lowe’s. We ensure that individuals are not singled out, and the outcome for all postmortems are directed toward learnings and process improvement.

For us, the postmortem process is the biggest part of our incident workflow. When an SRE creates a new postmortem report, the first step is to conduct a postmortem session with domain stakeholders to review the report. The postmortem then goes into the review stage and gets reviewed by more stakeholders in our weekly postmortem meeting. In the final stage of this process, the SRE captain will close the report once everyone in the weekly meeting agrees that the report is complete.

To conduct a successful postmortem, it is critical to keep the focus on identifying gaps and issues with the system and operations processes, rather than an individual, and generate concrete actions to address the problems we’ve identified. To ensure this, we follow a couple of best practices:

  1. We start by gathering the facts from the person who identified the problem, and each SLI owner has to identify a gap or the next SLI upstream owner who created the impact for them.
  2. Every SLI owner is provided full opportunity to present their case, and identifying the issue is done as a community exercise. 
  3. Once action items and process changes are identified, an owner is nominated to complete the actions, or they will volunteer. 
  4. For easy reference, we publish and store postmortems in our incident knowledge base. This process helps SREs continuously improve as future incidents arise. 

Continuous Improvement 

Encouraging a culture of honest, transparent and direct feedback that you need for blameless postmortems is often an iterative process that needs sponsorship from executives, empowering incident captains to lead the entirety of the discussion and outcomes. Running successful postmortems, and completing action items from them, needs to be recognized and accounted for in SRE performance objective assessment. As shared in Google’s SRE book, the best practice is to ensure that writing effective postmortems is a rewarded and celebrated practice, with leadership’s acknowledgement and participation. This is possibly the hardest part to accomplish in an effective postmortem during a cultural transformation unless you have full buy-in from leadership.

However, it’s all well worth it. This process is a key part of how we were able to improve our MTTR over time—from two hours in 2019 to just 17 minutes! 

Our SRE incident reporting process has also transformed how our company solves issues. By streamlining this workflow from alerting, to solving an issue, to blameless postmortems, we have reduced our MTTR by 82 percent and our MTTA by 97 percent. Most importantly, our team is learning from every incident and becoming better engineers as a result. Visit the SRE Google Cloud website to learn more about implementing SRE best practices in the cloud.


Acknowledgement

Special thanks to Rahul Mohan Kola Kandy, Vivek Balivada, and the Digital SRE team at Lowe’s for contributing to this blog post.

More Relevant Stories for Your Company

Case Study

How Companies can Improve Scalability, Flexibility, and Reliability While Reducing Costs: Tips from Route4Me

Google Cloud Results Improves application performance by 8x to 12x; customers can create increasingly complex optimized driving routes in single-digit secondsImproves customer satisfaction via increased reliability and greater application performanceFocuses on adding value to customers by improving software and algorithms, not infrastructure managementSaves 5x in infrastructure costs In 2009, Dan

Trend Analysis

2022’s First Cloud CISO Perspectives: Recap of the Megatrends, Releases and News

I’m excited to share our first Cloud CISO Perspectives post of 2022. It's already shaping up to be an eventful year for our industry and we’re only in month one. There’s a lot to recap in this post, including the U.S. government’s recent efforts to address critical security issues, like

Blog

Accelerating business growth with a seamless migration to Google Cloud

In today’s hybrid office environments, it can be difficult to know where your most valuable, sensitive content is, who’s accessing it, and how people are using it. That’s why Egnyte focuses on making it simple for IT teams to manage and control a full spectrum of content risks, from accidental

Blog

Prepare for the Unknown in Supply Chain with SAP IBP and Google Cloud

Responding to multiple, simultaneous disruptive forces has become a daily routine for most demand planners. To effectively forecast demand, they need to be able to predict the unpredictable while accounting for diverse and sometimes competing factors, including: Labor and materials shortagesGlobal health crisesShifting cross-border restrictionsUnprecedented weather impactsA deepening focus on

SHOW MORE STORIES