Cloud transformation and the adoption of modern DevOps technology presents both opportunities and challenges for IT compliance functions. With DevOps style application development, the feedback loop for developers and engineers is much tighter than with traditional application development pipelines, enabling speed and agility of application release cycles. While speedy CI/CD is a critical advantage of DevOps, it also shifts compliance left in the development timeline, and therefore puts pressure on the IT risk & compliance organization to modernize their approach to regulatory compliance as well. With the ongoing shift towards cloud technologies and DevOps, modernization of regulatory compliance is no longer optional for an IT compliance function

Compliance modernization is a broad mandate that spans the way the function is governed; the tools, technology, and analytics it uses; the number and nature of its connections to other parts of the business; verifiability and auditability of the controls’ evidence, the expectations assigned to it; and more.

Public cloud technology is becoming a core part of many industries today, and with this comes some potential risks such as cloud misconfigurations exposing intellectual property, loss of physical control of assets, skillset scarcity around cloud based security and compliance. 

Given the constantly changing risk landscape, it is critical that regulations more closely align to address these risks. As regulations and risks evolve, the aim of a modern compliance function is to help an organization stay compliant as it goes through a digital transformation. As organizations go through digital transformation, IT compliance also needs to transform — via upgrading the technology stack, modifying the business processes and most importantly re-skilling people to become cloud aware.

Today we are releasing the new paper by Google Cloud’s Office of the CISO. In the paper we reveal a new approach for modernizing your compliance approach using modern approaches and Google Cloud toolsets. Your team can leverage the paper to add value to enterprises, both by charting a course to the safe use of cloud technology and by reducing risk through the use of the public cloud.

Read the paper “Assuring Compliance in the Cloud.”

Also, review these related resources:

• “Risk Governance of Digital Transformation in the Cloud“ paper
• “Making Compliance Cloud-native” (episode 14) with Zeal Somani
• Google Cloud Compliance resource center
• Our compliance blueprints: PCI DSS on GKE and GCP FedRAMP Blueprint
• Google Cloud security best practices center

Editor’s note: Today we hear from the Lowe’s SRE team. They share about how they have been able to increase the number of releases they can support by adopting Google’s Site Reliability Engineering (SRE) framework and leveraging their partnership with Google Cloud. 

At Lowe’s, we’ve made significant progress in our multiyear technology transformation. To modernize our systems and build new capabilities for our customers and associates, we leverage Google’s SRE framework and Google Cloud, which helps us meet their needs faster and more effectively. With these efforts, we’ve been able to go from one release every two weeks to 20+ releases daily—about 20X more releases per month. 

Our SRE transformation didn’t happen overnight, though. Every step along the way brought some challenges. But looking back, we are excited to see how much we have accomplished for our customers as a result. 

Back in 2018, before adopting SRE practices, we were more reactive than proactive, following an “eyes on glass” approach. On-call structures and incident management efficiency were not at optimal levels with too many repetitive and manual tasks, resulting in operational toil. Production concerns were not surfaced into the product roadmap, which resulted in delays in making fixes.

Bootstrapping SRE at Lowe’s

As we moved from on-prem to Google Cloud, we decided to move from a monolithic- to microservices-based architecture. And to better manage this new architecture, we embarked on an SRE journey. 

Then as COVID-19 hit, we really had to accelerate this journey as customers increasingly moved to online ordering and delivery to meet their Total Home Improvement needs. To do so, we followed four key principles that allowed us to meet changing customer needs quickly and release fast and reliably.

1. Automate away toil 
   As we moved from traditional Ops to an SRE ecosystem, our biggest opportunity was reducing toil, so that engineers can spend time on activities that drive business impact and customer outcomes. We think of toil as work that is manual, repetitive,  tactical, devoid of enduring value—but automatable. So, to tackle toil, we focused on automating away the need for manual intervention. As an example, we made sure engineers were not the first point of contact for any alert. Any triage or resolution that an engineer can perform, a machine can be trained to do the same. We used supervised and unsupervised learning techniques to automate our toil. With a long-term goal of “no toil,” our SREs work on identifying and reducing toil to a manageable level across the organization.
2. Engineer alignment through roadmaps
   Our goal is to maximize the engineering velocity of developer teams while keeping products reliable. We want an engagement model where product, SRE and development teams are closely aligned. A key way we’ve been able to create this alignment is by having our SREs embedded into domain and product teams. Each domain has an SRE, who is  involved at the beginning stages of product development to ensure that the domain stakeholders are in alignment with the SRE initiatives. As such, SREs are able to improve the reliability, performance, scalability and launch velocity of the services throughout all phases of the service lifecycle. 
3. Adopt one-touch releases
   Our path to production used to contain many manual steps and validations, slowing the rate at which we released features. Additionally, we used to bulk all our releases together to deploy at once, which increased the risk of failure and created a longer feedback loop from production. To tackle this with an SRE mindset, we created a one-touch release process in which SREs review the product team’s pull requests. When approved, this triggers a DevSecOps pipeline that deploys the approved changes to production securely. This process created a safe, reliable and sustainable continuous delivery pipeline with quick feedback loops. Striking the right balance between speed, innovation and stability, we were able to increase our releases exponentially for the year, taking less than 30 minutes per release to deliver quality code, including various automated quality checks and processes, all in just one click. 
4. Embrace capacity planning
   To ensure our services have enough spare capacity to handle any surge in traffic patterns, our SREs emphasize capacity planning, making recommended capacity changes in the continuous delivery (CD) pipeline. They constantly monitor performance to make sure the service is robust, stable and available. And when there’s a sudden surge beyond the forecasted volume, SREs change the capacity on demand and document changes for the performance and domain teams. 

Capacity planning is especially important for us during peak holiday times such as Black Friday and Cyber Monday (BFCM). We lay out our SRE stability plan three months in advance and surface into the domain team’s product roadmap. This way development teams are able to allocate sufficient engineering time to reliability. We do performance testing to ensure the environment is able to sustain increased load over long periods of time and also handle sudden surges in traffic. We also do region failover testing at a global scale to validate the automatic failover duration of service level agreements  (SLAs), SRE and domain readiness. Additionally, we conduct Black Friday and Cyber Monday-specific destructive testing to validate customer experience, reliability and more.

Google Cloud’s Black Friday and Cyber Monday white-glove service played a key role in ensuring our success in both BFCM 2019 and BFCM 2020. This service included on-site visits from Google’s Customer Reliability Engineering (CRE) team who reviewed Lowe’s web architecture, capacity planning, operations practices for event risks, and presented workshops on topics such as incident response best practices. 

Looking ahead

There is always room for improvement, and at Lowe’s we aim to continuously improve our SRE practices. One thing that has worked well for us, which we plan to continue, has been our road shows, where senior SRE leads present to other SREs and application domain teams on the latest SRE principles and best practices, and to get input in real-time from them. 

Google’s tools and methodology have played an instrumental role in helping reshape our SRE practices and better serve our customers. We look forward to building on the momentum and partnership as we continue our SRE journey at Lowe’s. 

If you want to learn more about how to adopt SRE best practices on Google Cloud, check out our documentation. If you want to learn more about Google SRE, visit our website. Stay tuned for the next blogs with Lowe’s discussing how they trained their engineering talent to adopt SRE practices and tooling, and how they improved MTTR using SRE principles.

Many of our customers want to know how to choose a technology stack for solving problems with machine learning (ML). There are many choices for these solutions available, some that you can build and some that you can buy. We’ll be focusing on the build side here, exploring the various options and the problems they solve, along with our recommendations. 

The best ML applications are trained with the largest amount of data

But first, keep in mind an important concept: the quality of your ML model improves with the size of your data. Dramatic ML performance and accuracy are driven by improvements in data size, as shown in the graph below. This is a text model, but the same principles hold for all kinds of ML models.

The X axis represents the size of the data set and the Y axis is the error rate. As the size of the data set increases, the error rate drops. But notice something critical about the size of the data set — the x-axis is2^20, 2^21, 2^ 22, etc. In other words, each new tic here is a doubling of the data set size. To get a linear decrease in your error rate you need to exponentially increase the size of your data set. 

The blue curve in the graph represents a slightly more sophisticated ML model than the orange curve. Suppose you are deciding between two choices: create a better model or double the data set size. Assuming that these two choices cost the same, it’s better to keep gathering more data. It’s only when improvements due to data size increases start to plateau that it becomes necessary to build a better model. 

Secondly, ML systems need to be retrained for new situations. For example, if you have a recommendation system in YouTube and you want to provide recommendations in Google Now, you can’t use the same recommendations model. You have to train it in the second instance on the recommendations you want to make in Google Now. So even though the model, the code, and the principles are the same, you have to retrain the model with new data for new situations. 

Now, let’s combine these two concepts: you get a better ML model when you have more data, and an ML model typically needs to be retrained for a new situation. You have a choice of either spending your time building an ML model or buying a vendor’s off-the-shelf model. 

To answer the question of whether to buy or whether to build, first determine if the buyable model is solving the same problem that you want to solve. Has it been trained on the same input and on similar labels? Let’s say you’re trying to do a product search, and the model has been trained on catalog images as inputs. But you want to do a product search based on users’ mobile phone photographs of the products. The model that was trained on catalog images won’t work on your mobile phone photographs, and you’d have to build a new model. 

But let’s say you’re considering a vendor’s translation model that’s been trained on speeches in the European Parliament. If you want to translate similar speeches, the model works well as it uses the same kind of data. 

The next question to ask: does the vendor have more data than you do? If the vendor has trained their model on speeches in the European Parliament but you have access to more speech data than they have, you should build. If they have more data, then we recommend buying their model. 

Bottom line: buy the vendor’s solution if it’s trained on the same problem and has access to more data than you do. 

Technology stack for common ML use cases

If you need to build, what is the technology stack you need? What are the skills your people need to develop? This depends on the type of problem you are solving.  There are four broad categories of ML applications: predictive analytics, unstructured data, automation, and personalization. The recommended technology stack for each is slightly different. 

Predictive analytics

Predictive analytics includes detecting fraud, predicting click-through rates, and forecasting demand. 

Step one: build an enterprise data warehouse
Here, your data set is primarily structured data, so our recommended first step is to store your data in an enterprise data warehouse (EDW). Your EDW is a source of training examples and product histories tracked over time, and can break down silos and gather data from throughout your organization.

Step two: get good at data analytics
Next, you’d build a data culture, get skilled at data analytics, start to build dashboards, and enable data-driven decisions. At this point, you have all of the data and you know which pieces are trustworthy. 

Step three: build ML
From your EDW, you can build your models using SQL pipelines. We recommend using BigQuery ML when doing ML with the data in your EDW. If you want to build a more sophisticated model, you can train TensorFlow/Keras models on BigQuery data. A third option is AutoML tables for state-of-the-art accuracy and for building online microservices.

Unstructured data

Examples of how our customers use ML to gain insights from unstructured data include annotating videos, identifying eye diseases, and triaging emails. Unstructured data can include videos, images, natural language, and text. Deep learning has revolutionized the way we do ML on unstructured data, whether you’re looking at language understanding, image classification, or speech-to-text. 

For unstructured data, the models you use will  employ deep learning. Here, the ROI heavily favors using AutoML. The amount of time that you’d spend trying to create a new ML model from scratch is almost never worth it. You can spend your money more effectively collecting more data than trying to get a slightly better model. Regardless of the type of unstructured data, our recommendation is to use AutoML for small and medium size data sizes.

But AutoML has a limit to scale. At some point, the size of your data set is going to be so large that architecture search is going to get really expensive. At that point, you may want to go to a best-of-breed model with custom retraining from TensorFlow Hub, for example.  If you have data sets that are in the millions of examples, you can build your own custom neural network (NN) architectures. But determine if your data set size has started to plateau, by plotting a graph similar to the one at the top of this post. Build a custom NN architecture only after you’ve plateaued, where increasing amounts of data won’t give you a better model. 

Automation

Some examples of how customers are using ML for automation include scheduling maintenance, counting retail footfall, and scanning medical forms. The key thing to keep in mind as you pick a technology stack for these problems is that you’re not building just one ML model. If you want to schedule maintenance orwant to reject transactions, for example, you’ll need to train multiple linked models. 

Instead of individual models, think in terms of ML pipelines, which you can orchestrate using all of the technologies already mentioned. Then you have three choices for operationalizing, with three levels of sophistication.

1. Vertex AI has turnkey serverless training and batch/online predictions. This is what is recommended for a team of data scientists. .
2. Deep Learning VM Image, Cloud Run, Cloud Functions or Dataflow feature customized training and batch/online predictions. This is what is recommended if the team consists of  data engineers and  scientists.
3. Vertex AI Pipelines are fully customizable and recommended for organizations with separate ML engineering and data science teams.

When doing automation, the individual models that you chain together into a pipeline will be a mix – some will be prebuilt, some will be customized, and others will be built from scratch. Vertex AI, by providing a unified interface for all these model types, simplifies the operationalization of these models.

Personalization

ML application examples of personalization include customer segmentation, customer targeting, and product recommendations. For personalization, we again recommend using an EDW, because customer segmentation uses structured marketing data. For product recommendations, you will similarly have prior purchases and web logs in your EDW., You can power clustering applications, or recommendation systems like matrix factorization, and create embeddings directly from your EDW for sophisticated recommendation systems.

For specific use cases, choose the technology stack based on your data size and scope. Start with BigQuery ML for its quick, easy matrix factorization approach. Once your application proves viable and you want a slightly better accuracy, then try AutoML recommendations. But once your data set grows beyond the capabilities of AutoML recommendations, consider training your own custom TensorFlow and Keras models. 

To summarize, successful ML starts with the question, “Do I build or do I buy?” If an off-the-shelf solution exists that was trained with similar data and with access to more data than you have, then buy it. Otherwise build it, using the technology stack recommended above for the four categories of ML applications.

Learn more about our artificial intelligence (AI) and ML solutions and check out sessions from our Applied ML Summit on-demand.

Do hamburgers really come from Hamburg? This may still be a matter of debate, but the popularity of American-style burger joints not just in Hamburg but all over Germany, is clear. Germany’s top two fast food companies are both burger chains. One of them is BURGER KING®, a global brand that welcomes more than 11 million customers worldwide every day. The company arrived in Germany in 1976, when its first restaurant opened in Berlin. It now operates more than 100 restaurants across Germany, with franchisees operating more than 600 restaurants of their own.

“In the fast food industry, being able to move quickly is very important. That means delivering the right promotion to our app or launching a viral campaign within days. To do that across more than 700 restaurants, we need the support of the right technology.”

—Oliver Mielentz, IT Manager, BURGER KING® Deutschland GmbH

Previously a subsidiary of the U.S. business, BURGER KING® Germany became an independent company in 2015. As a result, it needed to develop its own IT infrastructure, and the changeover needed to happen fast. “We had to put in place systems that would work for the entire network of franchisees and enable us to easily roll out campaigns,” explains Oliver Mielentz, IT Manager at BURGER KING® Deutschland GmbH.

With the help of Google Cloud Premier partner Cloudwürdig, BURGER KING® Germany chose Google Cloud and G Suite as the right combination to suit its needs.

“In the fast food industry, being able to move quickly is very important,” says Oliver. “That means delivering the right promotion to our app or launching a viral campaign within days. To do that across more than 700 restaurants, we need the support of the right technology.”

Building a franchisee platform in just three months

When a business has multiple franchisees, it’s important to make sure everyone is on the same page, especially in the fast-paced fast food environment. “We have to collate data from all our franchisees and produce reports quickly in order to react to changes in customer behavior,” explains Oliver. “That means processing every transaction that takes place in our restaurants.” Following the restructure, BURGER KING® Germany also needed to build a secure invoicing system with data storage and optimize its communication channels.

“Using Tableau with BigQuery, we’re able to produce reports very quickly. Previously, it took much longer, as data had to be fetched manually. Our reaction time is now almost a business day faster.”

—Oliver Mielentz, IT Manager, BURGER KING® Deutschland GmbH

With support from Witter-IT, BURGER KING® Germany chose Cloudwürdig to build its BKD Connect internal platform on Google Cloud. Thanks to the ready-to-go tools on Google Cloud, it was able to put its invoicing system and data warehouse in place in just three months.

For the BURGER KING® Germany data warehouse, Cloudwürdig built an ETL pipeline that channels ticket data for every sale into BigQuery. “Data is gathered from the restaurants,” says Oliver, “and using Tableau with BigQuery, we’re able to produce reports very quickly. Previously, it took much longer, as data had to be fetched manually. Our reaction time is now almost a business day faster.”

As the ticket data for every transaction is stored in BigQuery, the marketing team can easily see exactly which products are selling well. That’s crucial for tweaking promotions as well as monitoring the supply chain to make sure enough produce is delivered to restaurants in response to changes in demand.

“Thanks to BigQuery, we have a speedy data pipeline that enables us to react on the same day to changes in the market and eliminate bottlenecks in production,” says Oliver.

Switching to G Suite to improve communication

To enable franchisees to sign in to its BKD Connect Platform, BURGER KING® Germany needed a secure authentication system. To solve that problem, it chose to provide franchisees with G Suite accounts. “It’s really easy to set up a new franchisee on the platform. I just create a new G Suite account and Drive folder for it, and it’s ready to go,” says Oliver. G Suite also helps the franchise network to run efficiently, as daily reports are automatically saved to Drive and shared to the appropriate regional network. “Thanks to that system, it’s much easier for any team at headquarters to access the information it needs,” Oliver explains.

BURGER KING® Germany also recently extended its use of G Suite across the whole company. “Following an evaluation of our previous email and productivity software, I made the decision to switch solely to G Suite,” says Oliver. BURGER KING® Germany employees now use Gmail, Calendar, and Drive for their day-to-day productivity needs. “We only just completed the migration, but already, everyone’s happy,” says Oliver. “It’s so easy to share a file using Drive or set up a meeting on Calendar.”

“We’re big fans of Hangouts Meet, and we have two rooms here at our Hanover headquarters equipped with Hangouts Meet hardware,” Oliver adds. “The speech quality is good, and it’s helpful to be able to see every participant, especially when you’re running a meeting with multiple franchisees.”

Optimizing infrastructure to power innovative campaigns

The BURGER KING® app, available for iOS and Android, helps the company to deliver a great customer experience. Through their MyBK accounts, guests can access coupons and special promotions. “We had a really interesting campaign for Easter: guests used the app to hunt for virtual Easter eggs,” explains Oliver. “We knew it was going to be big, and our previous back end wouldn’t have been able to handle the traffic.”

To enable the marketing campaign to go ahead, BURGER KING® Germany moved the back end of the app, along with its website, to Google Cloud. For developing and running its web and app back ends, it now uses App Engine and virtual machines on Compute Engine, as well as Memorystore and Cloud Functions. For monitoring and logging, it uses Stackdriver, and Cloud CDN and Cloud DNS to easily handle its traffic.

“We ran the campaign without any performance issues, even though we were receiving several million hits a day,” says Oliver. Since migrating the back end to Google Cloud, the marketing team also launched the popular “Escape the Clown” campaign. “That campaign blew our minds!” says Oliver. “It wouldn’t have been possible without Google Cloud, because it required a lot of back end capacity.”

To develop the app infrastructure it needs, BURGER KING® Germany relies on Cloudwürdig. “Working with Cloudwürdig is great because the team has the same agile mindset as us,” says Oliver. “When we have a new idea, we just set up a meeting, and in a couple of days the new infrastructure is in place. For Escape the Clown, it only took a few weeks to get everything ready to launch.”

Leveraging integrated tools to grow the business

Using Google Cloud together with G Suite enables BURGER KING® Germany to run its franchise network efficiently, while keeping its IT team lean. “Google Cloud and G Suite are the perfect fit for the way of working at BURGER KING® Germany,” says Oliver. “Many of the company’s operatives are often on the road, visiting restaurants and franchisees. With these tools, they can work flexibly and react quickly to the situation on the ground.”

“In order to grow the business, we need to use the data we receive every day to understand exactly what is happening in our restaurants. With the tools provided by Google Cloud, we can get more guests through the door and offer them a better experience.”

—Oliver Mielentz, IT Manager, BURGER KING® Deutschland GmbH

It also helps to keep infrastructure costs under control. “With Google Cloud, we only pay for what we use, which is really important for us,” Oliver explains. “It means we can scale up quickly if we see an opportunity to react to a trend in customer behavior and launch a new marketing campaign that resonates with the moment. When it’s finished, we can then scale down again, and that definitely saves us money.”

BURGER KING® is now working with Cloudwürdig to add more functionality to the BURGER KING® app using Google Kubernetes Engine. “We like to work with customers long-term to support their digital transformation. BURGER KING® Germany is a great example of how one project can develop into a great collaboration,” says Benny Woletz, Managing Director of Cloudwürdig.

BURGER KING® also plans to expand its presence in Germany and gain a greater market share by further tailoring both its marketing and the way it runs its restaurants to answer its guests’ needs. “In order to grow the business, we need to use the data we receive every day to understand exactly what is happening in our restaurants,” says Oliver. “With the tools provided by Google Cloud, we can get more guests through the door and offer them a better experience.”