What’s the most difficult question a security operations team can face? For some, is it, “Who is trying to attacks us?” Or perhaps, “Which cyberattacks can we detect?” How do teams know when they have enough information to make the “right” decision? Metrics can help inform our responses to those questions and more, but how can we tell which metrics are the best ones to rely on during mission-critical or business-critical crises?

As we discussed in our blogs, “Achieving Autonomic Security Operations: Reducing toil” and “Achieving Autonomic Security Operations: Automation as a Force Multiplier,” your Security Operations Center (SOC) can learn a lot from what IT operations discovered during the Site Reliability Engineering (SRE) revolution. In this post, we discuss how those lessons apply to your SOC, and center them on another SRE principle—Service Level Objectives (SLOs).

Even though industry definitions can vary for these terms, SLI, SLO, and SLA have specific meanings, wrote the authors of the Service Level Objectives chapter in our e-book, “Site Reliability Engineering: How Google runs production systems.” (All subsequent quotes come from the SLO chapter of the book, which we’ll refer to as the “SRE book.”)

• SLI: “An SLI is a service level indicator—a carefully defined quantitative measure of some aspect of the level of service that is provided.”
• SLO: “An SLO is a service level objective: a target value or range of values for a service level that is measured by an SLI.”
• SLA: An SLA is a Service Level Agreement about the above: “an explicit or implicit contract with your users that includes consequences of meeting (or missing) the SLOs they contain.”

In practice, we measure something (SLI) and we set the target value (SLO); we may also have an agreement about it (SLA).

This is not about cliches like “what gets measured gets done” here, but metrics and SLIs/SLOs will to a large extent determine the fate of your SOC. For example, SOCs (including at some Managed Security Service Providers) that obsessively focus on “time to address the alert” end up reducing their security effectiveness while making things go “whoosh” fast. If you equate mean time to detect or discover (MTTD) with “time to address the alert” and then push the analyst to shorten this time, attackers gain an advantage while defenders miss things and lose.

How to choose which metrics to track

One view of metrics would be that “whatever sounds bad” (such as attacks per second or incidents per employee) needs to be minimized, while “whatever sounds good” (such as successes, reliability, or uptime) needs to be maximized.

But the SRE experience is that sometimes good metrics have an optimum level, and yes, even reliability (and maybe even security). The book’s authors, Chris Jones, John Wilkes, and Niall Murphy with Cody Smith, cite an example of a service that defied common wisdom and was too reliable.

“Its high reliability provided a false sense of security because the services could not function appropriately when the service was unavailable, however rarely that occurred… SRE makes sure that global service meets, but does not significantly exceed, its service level objective,” they wrote.

The SOC lesson here is that some security metrics have optimum value. The above-mentioned time to detect has an optimum for your organization. Another example is the number of phishing incidents, which may in fact have an optimum value. If nobody phishes you, it’s probably because they already have credentialed access to many of your systems – so in your SOC, think of SLI optimums, and don’t automatically assume zero or infinite targets for metrics.

Three specific quotes from the SRE book remind us that “good metrics” may need to be balanced with other metrics, rather than blindly pushed up:

• “User-facing serving systems generally care about availability, latency, and throughput.”
• “Storage systems often emphasize latency, availability, and durability.”
• “Big data systems, such as data processing pipelines, tend to care about throughput and end-to-end latency.”

In a SOC, this may mean that you can detect threats quickly, review all context related to an incident, and perform deep threat research—but the results may differ for various threats. A fourth guidepost explains why your SOC should care even about this: “Whether or not a particular service has an SLA, it’s valuable to define SLIs and SLOs and use them to manage the service.” Indeed, we agree that SLIs and SLOs matter more for your SOC than any SLAs or other agreements.

Metrics matter, but so does flexibility

When considering the list of most difficult questions a security operations team can face, it’s vital to understand how to evaluate metrics to reach accurate answers. Consider another insight from the book: “Most metrics are better thought of as distributions rather than averages.”

If the average alert response is 20 minutes, does that mean that “all alerts are addressed in 18 to 22 minutes,” or that “all alerts are addressed in five minutes, while one alert is addressed in six hours?” Those different answers point to very different operational environments.

What we’ve seen before in SOCs is that a single outlier event is probably the one that matters most. As the authors put it, “The higher the variance in response times, the more the typical user experience is affected by long-tail behavior.” So, in security land, that one alert that took six hours to respond to was likely related to the most dangerous activity detected.

To address this, the book advises, “Using percentiles for indicators allows you to consider the shape of the distribution.” Google detection teams track the 5% and 95% values, not just averages.

Another useful concept from SRE is the “error budget,” a rate at which the SLOs can be missed, and tracked on a daily or weekly basis. It’s a SLO for meeting other SLOs.

The SOC value here may not be immediately obvious, but it’s vital to understanding the unique role security occupies in technology. In security, metrics can be a distraction because the real game is about preventing the threat actor from achieving their objectives. Based on our own experiences, most blue teams would rather miss the SLO and catch the threat in their environment. The defenders win when the attacker loses, not when the defenders “comply with a SLA.” The concept of the error budget might be your best friend here.

The SRE book takes that line of thinking even further. “It’s both unrealistic and undesirable to insist that SLOs will be met 100% of the time: doing so can reduce the rate of innovation and deployment.”

More broadly, and as we said in our recent paper with Deloitte on SOCs, rigid obeisance is its own vulnerability to exploit. “This adherence to process and lack of ability for the SOC to think critically and creativity provides potential attackers with another opportunity to successfully exploit a vulnerability within the environment, no matter how well planned the supporting processes are.”

To be successful at defending their organizations, SOCs must be less like the unbending oak and more like the pliant but resilient willow.

Track metrics but stay focused on threats

A third interesting puzzle from our SRE brethren: “Don’t pick a target based on current performance.”

We all want to get better at what we do, so choosing a target goal for improvement based on our existing performance can’t be bad, right? It turns out, however, that choosing a goal that sets up unrealistic or otherwise unhelpful, or woefully insufficient, expectations can do more harm than good.

Here is an example: An analyst handles 30 alerts a day (per their SLI), and their manager wants to improve by 15% so they set the SLO to 35 alerts a day. But how many alerts are there? Leaving aside the question of whether it is the right SLI for your SOC, what if you have 5,000 alerts, and you drop 4,970 of them on the floor. When you “improve,” you still drop 4,965 on the floor. Is this a good SLO? No, you need to hire, automate, filter, tune, or change other things in your SOC, not set better SLO targets that seemingly improve upon today’s numbers.

To this, our SRE peers say: “As a result, we’ve sometimes found that working from desired objectives backward to specific indicators works better than choosing indicators and then coming up with targets… Start by thinking about (or finding out!) what your users care about, not what you can measure.”

In the SOC, this probably means start with threat models and use cases, not the current alert pipeline performance.

SOC guidance can sometimes be more cryptic than we’ve let on. One challenging question is determining how many metrics we really need in a typical SOC. SREs wax philosophical here: “Choose just enough SLOs to provide good coverage of your system’s attributes.”

In our experience, we haven’t seen teams succeed with more than 10 metrics, and we haven’t seen people describe and optimize SOC performance with fewer than 3. However, SREs offer a helpful, succinct test: “If you can’t ever win a conversation about priorities by quoting a particular SLO, it’s probably not worth having that SLO.”

SLOs will get to define your SOC, so define them the way you want your SOC to be, the book advises. “It’s better to start with a loose target that you tighten than to choose an overly strict target that has to be relaxed when you discover it’s unattainable. SLOs can—and should—be a major driver in prioritizing work for SREs and product developers, because they reflect what users care about.”

Importantly, make SLOs for your SOC transparent within the company. As the SREs say, “Publishing SLOs sets expectations for system behavior.” The benefit is that nobody can blame you for non-performance if you perform to those agreed upon SLOs.

Finally, here are some examples of metrics from our teams at Google. In addition to reviewing all escalated alerts, they collect and review weekly:

• event volume
• event source counts
• pipeline latency
• triage time median
• triage time at 95%

Analyzing these metrics can reveal useful guidance for applying SRE principles and ideas with their detection and response teams.

Event volume: What we need to know here is what is driving the volume. Is the event volume normal, high, or low—and why? Was there a flood of messages? New data source causing high volume? What caused it? Any bad signals? Or is there a problematic area of the business that needs strategic follow-up to implement additional controls?

Event source count: Are there signals or automation that’s behaving abnormally? Is there new automation that’s misbehaving? Counting events for each source call makes for a decent SLI.

Pipeline latency: Here at Google, we aim for a confirmed detection within an hour of an event being generated. The aspirational time is 5 minutes. This means that the event pipeline latency is something that must be tracked very diligently. This also means that we must scrutinize automation latency. To achieve this, we try to remove self-caused latency so that we’re not hiding the pain of bad signals or bad automation.

We triage median and 95p time: We track the response time to events. As the SRE book points out, tracking only a single average number can get you in trouble very quickly. Note that triage time is not the same as time to resolution, but more of a dwell time for an attacker before they are discovered.

Incident resolution times: When you have a SLI but not a SLO, this can be the proverbial elephant in the room and create all sorts of bad incentives to “go fast” instead of “go good.” Specifically, SLO without SLI causes harm from encouraging the analysis to resolve quickly and potentially increase the risk of missing serious security incidents, especially when subtle signals are involved.

When reviewing alert escalations, we look to determine if the analysis is deep enough, if handoffs contain the right information for our response teams, and to get a sense of analyst fatigue. If analysts are phoning in their notes, it’s a sign that they’re over a particular signal or that there are a ton of duplicate incidents and we need to drive the business in some way.

By measuring these and other factors, metrics allow us to drive down the cost of each detection. Ultimately, this can help our detection and response operation scale faster than the threats.

Related posts:

• “Achieving Autonomic Security Operations: Automation as a Force Multiplier”
• “Achieving Autonomic Security Operations: Reducing toil”
• “Taking an autonomic approach to security operations” video
• “New Paper: “Future Of The SOC: Process Consistency and Creativity: a Delicate Balance” (Paper 3 of 4)”
• “New Paper: “Autonomic Security Operations — 10X Transformation of the Security Operations Center””
• “EP75 How We Scale Detection and Response at Google: Automation, Metrics, Toil” podcast episode

For a look ahead at the trends that will animate the retail industry this year, let’s take a look back at the 2022 National Retail Federation (NRF) “Big Show” in NYC.

Attendees at January’s event were treated to tangible examples of how retail challenges are being solved today, including new solutions to help them parse customer expectations and buying patterns, adapt stores into omni-channel experience hubs, and improve data visibility and actionability.

NRF 2022 also took the “omni-channel everything” theme of last year’s show to the logical next level: Enabling the best hybrid experiences. The message came through loud and clear of the importance of integration and interoperability in this new hybrid world – making everything work well together.

The need for modern digital infrastructure to enable this blending of physical and digital retail smoothly is paramount. To that end, technology vendors demonstrated how digital transformation initiatives, such as contactless and real time IoT and mobile applications, need to be built on cloud, edge, and secure connectivity to allow retailers to achieve the modern seamless hybrid retail that today’s consumer wants.

Other prominent themes and technologies highlighted at NRF included: extending engagement in the metaverse, sustainability, physical and digital security, and the agility and adaptability imperative.

The Metaverse and Hybrid (Omni-channel) Experiences

Today, the metaverse is an extension of our lives, enhanced by technology, which exists as a series of virtual worlds. In the future, the metaverse will be an interconnected, endless world where digital and physical lives fully converge. Imagine waiting for an appointment at a real booth on the NRF show floor while your avatar roams a fully fleshed-out digital NRF, meeting other virtual attendees, stopping for coffee at the digital Starbucks, and paying for a coffee that an in-the-flesh Starbucks employee brings to them. Digital and physical selves merge seamlessly in the metaverse, as the worlds draw closer together.

In the metaverse, brands have a digital presence, too. Nike filed seven trademarks late last year, including those for “Nike,” “Just Do It,” and its swoosh logo, and posted openings for virtual designer roles, indicating its intent to make and sell virtual branded sneakers and apparel. It subsequently purchased RTFKT Studios, a company that already makes and sells NFTs and digital sneakers. (In one collaboration with teenage artist FEWOCiOUS, the company sold 600 pair/NFTs of sneakers in just six minutes to the tune of more than $3.1 million.)

The metaverse also opens possibilities for gathering data about consumers and product demand. Imagine a sneaker drop in the virtual world. Certain styles of new kicks sell like gangbusters, giving the brand insight into what might sell IRL, intelligence that leads to trend-right production and less inventory headed for markdown or landfills. The metaverse can be a vehicle for more sustainable operations.

The metaverse further bridges the narrowing gap between digital worlds and physical worlds. Most consumers aren’t outfitting an avatar, but they are moving between online and offline and expect retailers to accommodate those hybrid omni-channel journeys seamlessly. Those demands have accelerated around last-mile delivery and experiences such as buying online and picking up in store (BOPIS) or at curbside, shopping in store and returning merchandise online, adding items to a BOPIS purchase when at the store, or communicating a substitution to the third-party grocery delivery service

Hybrid experiences open opportunities to please the consumer in new ways, but they also add expense and complexity. The need to meet this demand while enabling profitability was a major theme behind many of the technologies discussed at NRF. These included artificial intelligence (AI) for recommending the right product, return logistics software for defining and guiding product-specific reverse logistics workflows, order orchestration and fulfillment applications for omni-channel shopping, and last-mile delivery visibility for optimizing customer experience, to name a few. Also on display were task management applications help to improve and optimize in-store employee engagement, as well as touch-free applications to allow for faster payments and customer self-service checkout. RFID continues to improve inventory accuracy and inventory locating on the shelf, throughout the store, and the supply chain.

Sustainability

NRF 2022 saw a strong focus on sustainability. An NRF/IBV study released at the show highlighted the significant embrace of sustainable shopping by consumers. According to the survey, 62% of shoppers are “willing to change their purchasing habits to reduce environmental impacts.” About half indicated a willingness to pay a premium – on average a 70% premium – for sustainable products and brands.

Retailers are working to improve sustainability and reduce carbon footprint across operations by using sustainable sourcing through the supply chain, the store, and even returns. Tech vendors unveiled a variety of solutions enabled by cloud/edge, AI, computer vision, and IoT/RFID to allow retailers to effectively measure and record their environmental efforts, with the goal of reducing their impact.

Several cloud and digital infrastructure providers showcased sustainability clouds and other technology aimed at asset management with the goal of reducing energy consumption, water usage, waste. Examples included using IoT sensors to reduce water usage, optimizing re-use of store assets, and dashboards that allow retailers to accurately monitor and measure carbon output. However, such sustainability solutions can be most successful when running on the next-generation digital infrastructure that helps retailers better compete and differentiate in today’s omni-channel world.

Physical and Digital Security

According to a 2021 NRF survey, 57% of U.S. retailers reported the pandemic led to an increase in organized retail crime, while 50% reported an increase in shoplifting. When IDC’s Future Enterprise Resiliency & Spending Survey, Wave 10 (November 2021) asked retailers which digital infrastructure investments would provide the greatest strategic advantage in 2022, their #1 response was “cybersecurity and recovery investments.”

A wide range of technology vendors acknowledged retailer concerns with regards to security, fraud, and loss prevention:

• Networking, connectivity, and edge vendors highlighted multilayer security solutions that promise to protect data from a range of IoT applications that utilize customer and associate data. Many offer security consulting services to address varied threats including ransomware, retail crime, and loss prevention.
• Security and e-commerce security vendors showcased solutions to prevent fraud and abuse in e-commerce applications as well as omni-channel applications such as BOPIS and curbside pickup, using AI-based analysis for identifying “bad”/risky customers and mitigating risk.
• Cloud vendors highlighted how retail clouds provide consistent, reliable identity management and data security.
• POS/payments/store technology vendors emphasized their ability to handle payments securely from any platform with multifactor tokenization, improved identity techniques such as biometrics and voice authentication, as well as AI-enabled and computer vision solutions for loss prevention at checkout and at the door.

The Agility and Adaptability Imperative

On display at the show were multiple flavors of the digital infrastructure technology that retailers need to achieve agile, personalized, data-driven, integrated seamless operations across the many channels of today’s retail landscape. The emphasis was apt. More than half of retailers plan to boost investment in business agility and operational agility over the next 12 months, according to IDC’s Future Enterprise Resiliency & Spending Survey, Wave 10 (November 2021).

Technology vendors highlighted their connectivity investments to enable business and operational agility and their technology investments for better ease of integration, scalability, and the ability to more easily swap out or mix and match applications with integrated platforms, open systems, hybrid cloud, and retail industry clouds.

Vendors also showed off infrastructure to better harness data while enabling its visibility, maximizing its value, and providing the data-driven personalization essential for competitive advantage and differentiation. Highlights included fast, secure connectivity, 5G and Wifi-6, and edge- and cloud-enabled data and AI platforms to generate real-time insights – all designed to enable today’s omni-channel retail.

Advice for the technology buyer

Retailers should consider these key themes from NRF 2022 when making technology investment decisions for 2022 and beyond. To avoid lagging behind those retailers already moving toward thriving into the future, take action to:

• Enable the seamless, contactless omni-channel approach that today’s consumers want and expect.
• Replace legacy infrastructure that was not built to handle the modern retail environment that requires the agility and adaptability to seamlessly connect rapidly increasing volumes of data securely and more quickly than ever.

Whether sustainability, adaptability, the metaverse, or security are top concerns, addressing business needs holistically and strategically should be job #1.

Continue the conversation by downloading our Transforming retail and CPG markets whitepaper today.

At Google Cloud, we’re focused on making it easy for organizations to build solutions quickly and securely. Identity and Access Management (IAM) is the core security control for establishing who has access to which cloud resources and making sure access permissions are aligned to your company’s business and security policies.

We are excited to announce the general availability of IAM Deny policies. This new capability helps you easily create access guardrails for Google Cloud resources. With IAM Deny policies, you can create rules that broadly restrict resource access. It provides a powerful, coarse-grained access control to help implement security policies at scale.

Control with IAM Policy – Allow and Deny

IAM Deny policies complement IAM Allow policies as you define access to your resources. Google Cloud’s IAM Allow policy lets you grant granular access to Google Cloud resources. The more coarse-grained Deny policies let you explicitly prohibit access to certain resources regardless of existing Allow rules. IAM Deny policies always supersede IAM Allow policies and override conflicting IAM Allow rules.

IAM Deny policies can be applied across many Google Cloud resources, such as Compute Engine, Cloud Storage, and Google Cloud Kubernetes Engine. These policies can help reduce toil on administrators as they can set up deny rules that will be enforced at scale without requiring reviews and changes of existing access rules. This makes resource governance simpler.

How to strengthen your security posture with IAM Deny

There are multiple use cases where IAM Deny policies can be used to help strengthen posture. Some of these are:

• Establish a default security baseline: IAM Deny can be used to set base policies at the organization level, folder level, and project level to deny access to resources. For example, you can attach a deny policy at the organization level to deny all users access to sensitive data storage buckets, making an exception for a specific user group.
• Prevent backdoors: IAM Deny rules override any IAM Allow rules. This can help you ensure that no “backdoor” access can be granted. For example, an organization that wants to ensure only central-admin can create projects in a folder can add an IAM Deny policy that restricts all users from creating projects except central-admin. This can help ensure no backdoor users will be added using allow policy rules.
• Prevent data exfiltration: Controlling access to data is a key measure to prevent exfiltration. For example, an organization that wants to restrict access to their personally identifiable information (PII) can set an IAM Deny policy on resources containing PII that denies access to all users except those in a group called PII-admin.
• Demonstrate compliance: Many industry regulations and compliance frameworks require organizations to demonstrate least-privilege access to sensitive resources. IAM Deny policies can create a baseline access restriction that always takes precedence when applied to resources.

Simplify IAM administration

IAM Deny can also help simplify common administrative tasks. It provides a list of permissions for Google Cloud services that you can readily use in your deny policies. Here are a few common situations where you can use IAM Deny to help streamline your access management.

• Centralize administrative privileges: You can use deny policies to restrict certain types of administrative activities to specific users or user groups. For example, if you want to limit custom role management to a single central team, you can create a deny rule that denies the permissions required for custom role management to all users, except users in the central-admin group. This will only let members of the central-admin group manage custom roles, even if other users have the required permissions.
• Create exceptions to access grants: You can use deny policies to deny inherited permissions. For example, you can grant a role at a high level in the resource hierarchy, and then deny the role’s permissions on individual lower-level resources if necessary.
• Block access based on tags: Google Cloud supports key-value based tags and you can use deny policies to deny permissions based on tags without adding an IAM Condition to every role grant. For example, you can create a rule to deny delete permissions for resources tagged as “production” for everyone except project-admins.

IAM Deny in action

The following example illustrates how a deny policy can be used to block all users from deleting projects unless the user is a member of a project-admins group or the project being deleted is tagged as a “test” project. Without a deny policy, you would have to manually track all members who have the permission to delete projects, and ensure that no undesired user has access to this permission. The deny policy makes it easy for the administrator to build this guardrail.

The below deny rule denies permission to everyone except project-admins@example.com for projects not tagged as test. Add this deny rule to a deny policy and attach the policy at the org level. That’s it!

(Note that project-admins@example.com is a security group. Using security groups is a best practice that should be followed to help keep your resources secure.)

{
"name": "policies/cloudresourcemanager.googleapis.com%2Fprojects%2F253519172624/denypolicies/limit-project-deletion",
"uid": "06ccd2eb-d2a5-5dd1-a746-eaf4c6g3f816",
"kind": "DenyPolicy",
"displayName": "Only project admins can delete projects.",
"etag": "MTc1MTkzMjY0MjUyMTExODMxMDQ=",
"createTime": "2021-09-07T23:15:35.258319Z",
"updateTime": "2021-09-07T23:15:35.258319Z",
"rules": [
{
"denyRule": {
"deniedPrincipals": [
"principalSet://goog/public:all"
],
"exceptionPrincipals": [
"principalSet://goog/group/project-admins@example.com"
],
"deniedPermissions": [
"cloudresourcemanager.googleapis.com/projects.delete"
],
"denialCondition": {
"title": "Only for non-test projects",
"expression": "!resource.matchTag('12345678/env', 'test')"
}
}
}
]
}

Getting started with IAM Deny Policies

With IAM Deny, you now have a powerful new capability to build security guardrails and more effectively control access to your Google Cloud resources. IAM Deny is now generally available for all customers through gCloud and APIs. We offer IAM Deny to Google Cloud customers at no additional cost. You can learn more about IAM Deny by visiting our documentation page.

Together we can solve for the future of cloud security. Join us to learn how you can stay ahead of the next generation of threats with Google Cloud – whether you need to keep your organization secure in the cloud, on-premises, or in a hybrid environment. Get fresh insights from industry leaders and engage in interactive sessions that can help you solve your most critical security challenges. 

Our digital event has ended, but you can still explore our sessions on demand.
The Security Summit is part of our digital Google Cloud Summit series. Check out the other events in the series to explore Google Cloud technology applications in various industries and dive into our latest digital innovations.Original air date:20 July 2021 21:30 Register to watch on demand.