New Study Suggests Public Sector Firms Must Move Over Legacy Productivity Tools

5028
Of your peers have already read this article.
6:00 Minutes
The most insightful time you'll spend today!
Prior to joining Google Cloud, I spent 20 years in the public sector serving in various security roles, most recently as the head of the cybersecurity division at the newly established Cybersecurity and Infrastructure Security Agency (CISA). I was responsible for delivering services and capabilities to about 100 civilian agencies, as well as our critical infrastructure for state and local partners across the country.
Throughout my time serving in the U.S. government, I spent a great deal of energy working to keep bad actors out of our systems. What I eventually realized was that we were at a disadvantage. We didn’t just have legacy systems to deal with; we also had a legacy mindset. Our policies and capabilities were overly reliant on a perimeter-based approach to defense and introduced layers of friction with tools like VPNs, which inhibited productivity and increased frustration.
I spent the last few years of my time in the government doing my best to modernize this approach to security and supporting the many people who shared this vision. While most of them are still there, doing excellent work, repeated cybersecurity breaches of U.S. government systems have interrupted vital work and cost taxpayers billions of dollars. The SolarWinds breach in 2020, for example, may have cost governments and businesses more than $100 billion and the loss of vital national security information. Yet many government agencies continue to rely on the same legacy productivity software.
Today, a new survey conducted by Public Opinion Strategies and commissioned by Google Cloud shows that many government workers echo these concerns. The survey—which polled 2,600 working Americans, includes 600 workers from the D.C. metro area and 338 workers employed by federal, state, or local governments across the country—found that the majority of government employees surveyed reported being “very” concerned about cyberattacks striking their employers in the coming years. And nearly one-third of surveyed government employees in the D.C. metro area say they have experienced a disruption at work because of a cyberattack.
Results from the survey also showed a lack of satisfaction with legacy software, with more than 50% of government workers nationally responding that there are other products and services that could help them do their jobs better.
These new findings not only speak to the challenges our government employees face, but also outline an opportunity for improved innovation and security that can help government employees better achieve their missions.
Concerns about cybersecurity

The majority of respondents said they believed it was likely that the federal government would be the victim of a cyberattack in the next few years. This percentage was even higher in the D.C. metro area and among government employees, with 40% of government workers living in metro D.C. saying it was “very likely.”

Respondents in all groups—national, D.C. metro, and government employees—were more likely to be concerned about cyberattacks as the perceived threat moved closer to home. For example, 80% of those surveyed said that recent attacks have them concerned about their personal data and privacy, and that of their family members. And for those in the D.C. area or in government jobs, the numbers were consistently higher than the national average.
One likely reason for such broad concern about cyberattacks is that many respondents reported experiencing a cyberattack at work. And while more than one-in-ten workers nationally have experienced disruptions from a cyberattack, the number increased to nearly one-in-three for D.C.-based government employees—nearly three times higher than the national average.

The problem with IT ‘monoculture’
According to survey respondents, 84% of D.C. metro government employees primarily use Microsoft products at work, including Word, Outlook, Teams, and OneDrive. This is confirmed by another recent study by Omdia that found 85% of government employees use Microsoft productivity software, far and away the largest IT productivity vendor by market share.

This reliance on a single software suite might suggest that these products are safe and secure, but the Public Opinion Strategies survey found that more than half of all respondents said that the government’s reliance on these Microsoft products actually made the federal government more vulnerable to hacking or cyberattacks.

Given these vulnerabilities, why does government IT continue to rely on the same set of productivity tools in the workplace? The reason, according to survey respondents, has more to do with inertia than innovation. When asked why their employers used Microsoft services, around half said that the reason their employer continues to choose legacy, incumbent vendors was more about not wanting to change than wanting the most effective tool for the job.

Choice and shadow IT
Coupled with these concerns about vulnerabilities comes a noted sentiment among those surveyed that their current IT solutions are not best for their needs. About half of those surveyed who primarily use Microsoft at work said that they would prefer to have a choice to use products and services from other companies. And among those who use Microsoft at work, 43% believe there are other products and services that would allow them to do their job better.

This may be leading workers to adopt “shadow IT,” or using products and services that are not officially approved or endorsed by their IT departments. In fact, the Public Opinion Strategies survey found that 35% of D.C. metro government workers have used shadow IT to get their jobs done. And among workers aged 20 to 34, that number jumps to 41%.
Rethinking IT purchasing priorities
With so many survey respondents reporting that they are dissatisfied with their legacy IT solutions, it may be time for the government to rethink its approach to procurement. In a separate research survey from Omdia in December 2021, 250 people responsible for technology purchasing decisions in the U.S. federal, state, and local governments said that government technology and procurement practices often are more about making things easier for IT vs. choosing what employees feel would be the best solution. In fact, only 27% of officials surveyed in that research cited “user demand” as a factor affecting their purchasing decisions.
As governments work to meet the demands and preferences of their constituents—and their employees—it’s clear that there’s an overreliance on legacy solutions, despite a track record of cybersecurity vulnerabilities and poor user perception.
At Google Cloud, we believe it’s time for more diversity and choice in the tools available for our civil servants across the nation—70% of whom use Gmail outside of work, according to our survey. Government workers have the right to benefit from the same flexible, secure-by-design tools at the office that they use in their personal lives. You can learn more about Google Workspace for Government here.
Assuring Compliance in the Cloud: Paper by Google Cloud’s Office of the CISO

4926
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
Cloud transformation and the adoption of modern DevOps technology presents both opportunities and challenges for IT compliance functions. With DevOps style application development, the feedback loop for developers and engineers is much tighter than with traditional application development pipelines, enabling speed and agility of application release cycles. While speedy CI/CD is a critical advantage of DevOps, it also shifts compliance left in the development timeline, and therefore puts pressure on the IT risk & compliance organization to modernize their approach to regulatory compliance as well. With the ongoing shift towards cloud technologies and DevOps, modernization of regulatory compliance is no longer optional for an IT compliance function
Compliance modernization is a broad mandate that spans the way the function is governed; the tools, technology, and analytics it uses; the number and nature of its connections to other parts of the business; verifiability and auditability of the controls’ evidence, the expectations assigned to it; and more.
Public cloud technology is becoming a core part of many industries today, and with this comes some potential risks such as cloud misconfigurations exposing intellectual property, loss of physical control of assets, skillset scarcity around cloud based security and compliance.
Given the constantly changing risk landscape, it is critical that regulations more closely align to address these risks. As regulations and risks evolve, the aim of a modern compliance function is to help an organization stay compliant as it goes through a digital transformation. As organizations go through digital transformation, IT compliance also needs to transform — via upgrading the technology stack, modifying the business processes and most importantly re-skilling people to become cloud aware.
Today we are releasing the new paper by Google Cloud’s Office of the CISO. In the paper we reveal a new approach for modernizing your compliance approach using modern approaches and Google Cloud toolsets. Your team can leverage the paper to add value to enterprises, both by charting a course to the safe use of cloud technology and by reducing risk through the use of the public cloud.
Read the paper “Assuring Compliance in the Cloud.”
Also, review these related resources:
- “Risk Governance of Digital Transformation in the Cloud“ paper
- “Making Compliance Cloud-native” (episode 14) with Zeal Somani
- Google Cloud Compliance resource center
- Our compliance blueprints: PCI DSS on GKE and GCP FedRAMP Blueprint
- Google Cloud security best practices center
One Goal for Google: Get PQC Ready

2904
Of your peers have already read this article.
3:30 Minutes
The most insightful time you'll spend today!
The National Institute of Standards and Technology (NIST) on Tuesday announced the completion of the third round of the Post-Quantum Cryptography (PQC) standardization process, and we are pleased to share that a submission (SPHINCS+) with Google’s involvement was selected for standardization. Two submissions (Classic McEliece, BIKE) are being considered for the next round.
We want to congratulate the Googlers involved in the submissions (Stefan Kölbl, Rafael Misoczki, and Christiane Peters) and thank Sophie Schmieg for moving PQC efforts forward at Google. We would also like to congratulate all the participants and thank NIST for their dedication to advancing these important issues for the entire ecosystem.
This work is incredibly important as we continue to advance quantum computing. Large-scale quantum computers will be powerful enough to break most public-key cryptosystems currently in use and compromise digital communications on the Internet and elsewhere. The goal of PQC is to develop cryptographic systems that safeguard against these potential threats, and NIST’s announcement is a critical step toward that goal. Governments in particular are in a race to secure information because foreign adversaries can harvest sensitive information now and decrypt it later.
At Google, our work on PQC is focused on four areas: 1) driving industry contributions to standards bodies; 2) moving the ecosystem beyond theory and into practice (primarily through testing PQC algorithms); 3) taking action to ensure that Google is PQC ready; and 4) helping customers manage the transition to PQC.
Driving industry contributions to a range of standards bodies
In addition to our work with NIST, we continue to drive industry contributions to international standards bodies to help advance PQC standards. This includes ISO 14888-4, where Googlers are the editors for a standard on stateful hash-based signatures. More recently, we also contributed to the IETF proposal on data formats, which will define JSON and CBOR serialization formats for PQC digital signature schemes. These standards, collectively, will enable large organizations to build PQC solutions that are compatible and ease the transition globally.
Moving the ecosystem beyond theory and into practice: Testing PQC algorithms
We’ve been working with the security community for over a decade to explore options for PQC algorithms beyond theoretical implementations. We announced in 2016 an experiment in Chrome where a small fraction of connections between desktop Chrome and Google’s servers used a post-quantum key-exchange algorithm, in addition to the elliptic-curve key-exchange algorithm that would typically be used. By adding a post-quantum algorithm in a hybrid mode with the existing key-exchange, we were able to test its implementation without affecting user security.
We took this work further in 2019 and announced a wide-scale post-quantum experiment with Cloudflare. We worked together to implement two post-quantum key exchanges, integrated them into Cloudflare’s TLS stack, and deployed the implementation on edge servers and in Chrome Canary clients. Through this work, we learned more about the performance and feasibility of deployment in TLS of two post-quantum key agreements, and have continued to integrate these learnings into our technology roadmap.
In 2021, we tested broader deployment of post-quantum confidentiality in TLS and discovered a range of network products that were incompatible with post-quantum TLS. We were able to work with the vendor so that the issue was fixed in future firmware updates. By experimenting early, we resolved this issue for future deployments.
Taking action to ensure that Google is PQC ready
At Google, we’re well into a multi-year effort to migrate to post-quantum cryptography that is designed to address both immediate and long-term risks to protect sensitive information. We have one goal: ensure that Google is PQC ready. Internally, this effort has several key priorities, including securing asymmetric encryption, in particular encryption in transit. This means using ALTS, for which we are using a hybrid key-exchange, to secure internal traffic; and using TLS (consistent with NIST standards) for external traffic. A second priority is securing signatures in the case of hard-to-change public keys or keys with a long lifetime, in particular focusing on hardware, especially hardware deployed outside of Google’s control.
We’re also focused on sharing the information we learn to help others address PQC challenges. For example, we recently published a paper that includes PQC transition timelines, leading strategies to protect systems against quantum attacks, and approaches for combining pre-quantum cryptography with PQC to minimize transition risks. The paper also suggests standards to start experimenting with now and provides a series of other recommendations to allow organizations to achieve a smooth and timely PQC transition.
Helping customers manage the transition to PQC
At Google Cloud, we are working with many large enterprises to ensure they are crypto-agile and to help them prepare for the PQC transition. We fully expect customers to turn to us for post-quantum cloud capabilities, and we will be ready. We are committed to supporting their PQC transition with a range of Google products, services, and infrastructure. As we make progress, we will continue to provide more PQC updates on Google core, cloud, and other services, and updates will also come from Android, Chrome and other teams. We will further support our customers with Google Cloud transformation partners like the Google Cybersecurity Action Team to help provide deep technical expertise on PQC topics.
Google’s ASO to Help U.S. Public Sector Achieve M-21-31 and EO 14028

3153
Of your peers have already read this article.
3:00 Minutes
The most insightful time you'll spend today!
As sophisticated cyberattack campaigns increasingly target the U.S. public and private sectors during the COVID era, the White House and federal agencies have taken steps to protect critical infrastructure and remote-work infrastructure. These include Executive Order 14028 and the Office of Management and Budget’s Memorandum M-21-31, which recommend adopting Zero Trust policies, and span software supply chain security, cybersecurity threat management, and strengthening cyberattack detection and response.
However, implementation can be a challenge for many agencies due to cost, scalability, engineering, and a lack of resources. Meeting the requirements of the EO and OMB guidance may require technology modernization and transformational changes around workforce and business processes.
Today we are announcing Autonomic Security Operations (ASO) for the U.S. public sector, a solution framework to modernize cybersecurity analytics and threat management that’s aligned with the objectives of EO 14028 and OMB M-21-31. Powered by Google’s Chronicle and Siemplify, ASO helps agencies to comprehensively manage cybersecurity telemetry across an organization, meet the Event Logging Tier requirements of the White House guidance, and transform the scale and speed of threat detection and response. ASO can support government agencies in achieving continuous detection and continuous response so that security teams can increase their productivity, reduce detection and response time, and keep pace with – or ideally, stay ahead of – attackers.
While the focus of OMB M-21-31 is on the implementation of technical capabilities, transforming security operations will require more than just technology. Transforming processes and people in the security organization is also important for long-term success. ASO provides a more comprehensive lens through which to view the OMB event logging capability tiers, which can help drive a parallel transformation of security-operations processes and personnel.

Modern Cybersecurity Threat Detection and Response
Google provides powerful technical capabilities to help your organization achieve the requirements of M-21-31 and EO 14028:
Security Information & Event Management (SIEM) – Chronicle provides high-speed petabyte-scale analysis, and is capable of consuming log types outlined in the Event Logging (EL) tiers in a highly cost-effective manner.
Security Orchestration, Analytics, and Response (SOAR) – Siemplify offers dozens of out-of-box playbooks to deliver agile cybersecurity response and drive mission impact, including instances of automating 98% of Tier-1 alerts and driving an 80% reduction in caseload.
User and Entity Behavior Analytics (UEBA) – For agencies that want to develop their own behavioral analytics, agencies can use BigQuery, Google’s petabyte scale data lake, to store, manage, and analyze diverse data types from many sources. Telemetry can be exported out of Chronicle, and custom data pipelines can be built to import other relevant data from disparate tools and systems, such as IT Ops, HR and personnel data, and physical security data. From there, users can leverage BQML to readily generate machine learning models without needing to move the data out of BigQuery. For Google Cloud workloads, our Security Command Center Premium product offers native, turnkey UEBA across GCP workloads.
Endpoint Detection and Response (EDR) – For most agencies, EDR is a heavily adopted technology that has broad applicability in Security Operations. We offer integrations to many EDR vendors. Take a look at our broad list of Chronicle integrations here.
Threat intelligence – Our solution offers a native integration with VirusTotal, has the ability to operationalize threat intelligence feeds natively in Chronicle, and integrates with various TI and TIP solutions.
Community Security Analytics
To increase collaboration across public-sector and private-sector organizations, we recently launched our Community Security Analytics (CSA) repository, where we’ve partnered with the MITRE Engenuity Center for Threat-Informed Defense, CYDERES, and others to develop open-source queries and rules that support self-service security analytics for detecting common cloud-based security threats. CSA queries are mapped to the MITRE ATT&CK® framework of tactics, techniques and procedures (TTPs) to help you evaluate their applicability in your environment and include them in your threat model coverage.
“Deloitte is excited to collaborate with Google Cloud on their transformational public sector Autonomic Security Operations (ASO) solution offering. Deloitte has been recognized as Google Cloud’s Global Services Partner of the Year for four consecutive years, and also as their inaugural Public Sector Partner of the Year in 2020,” said Chris Weggeman, managing director of GPS Cyber and Strategic Risk, Google Cloud Cyber Alliance Leader, Deloitte & Touche LLP. “Our deep bench of more than 1,000 Google Cloud certifications, capabilities spanning the Google Cloud security portfolio, and decades of delivery experience in the government and public sector makes us well-positioned to help our clients undertake critical Security Operations Center transformation efforts with Google Cloud ASO.”
Cost-effective for government agencies
To help Federal Agencies meet the requirements of M-21-31 and the broader EO, Google’s ASO solutions can drive efficiencies and help manage the overall costs of the transformation. ASO can make petabyte-scale data ingestion and management more viable and cost-effective. This is critical at a time when M-21-31 is requiring many agencies to ingest and manage dramatically higher volumes of data that had not been previously budgeted for.
Partners
We’re investing in key partners who can help support U.S. government agencies on this journey. Deloitte and CYDERES both have deep expertise to help transform agencies’ Security Operations capabilities, and we continue to expand our partners to support the needs of our clients. A prototypical journey can be seen below.

“Cyderes shares Google Cloud’s mission to transform security operations, and we are honored to deliver the Autonomic Security Operations solution to the U.S. public sector. As the number one MSSP in the world (according to Cyber Defense Magazine’s 2021 Top MSSPs List) with decades of advisory and technology experience detecting and responding to the world’s biggest cybersecurity threats, Cyderes is uniquely positioned to equip federal agencies and departments to go far beyond the requirements of the executive order to transform their security programs entirely via Google’s unique ASO approach,” said Robert Herjavec, CEO of CYDERES. “As an original launch partner of Google Cloud’s Chronicle, our deep expertise will propel our joint offering to modernize security operations in the public sector, all with significant cost efficiency compared to competing solutions.” said Eric Foster, President of CYDERES.
Embracing ASO
Autonomic Security Operations can help U.S. government agencies advance their event logging capabilities in alignment with OMB maturity tiers. More broadly, ASO can help the U.S. government undertake a larger transformation of technology, process, and people, toward a model of continuous threat detection and response. As such, we believe that ASO can help address a number of challenges presently facing cybersecurity teams, from the global shortage of skilled workers, to the overproliferation of security tools, to poor cybersecurity situational awareness and analyst burnout caused by an increase of data without sufficient context or tools to automate and scale detection and response.
We believe that by embracing ASO, agencies can help agencies achieve:
10x technology, through the use of cloud-native tools that help agencies meet event logging requirements in the near term, while powering a longer-term transformation in threat management;
10x process, by redesigning workflows and using automation to achieve Continuous Detection and Continuous Response in security operations;
10x people, by transforming the productivity and effectiveness of security teams and expanding their diversity; and
10x influence across the enterprise through a more collaborative and data-driven approach to solving security problems between security teams and non-security stakeholders.
To learn more about Google’s Autonomic Security Operations solution for the U.S. public sector, please read our whitepaper. More broadly, Google Cloud continues to provide leadership and support for a wide range of critical public-sector initiatives, including our work with the MITRE Engenuity Center for Threat-Informed Defense, the membership of Google executives on the President’s Council of Advisors on Science and Technology and the newly established Cyber Safety Review Board; Google’s White House commitment to invest $10 billion in Zero Trust and software supply chain security, and Google Cloud’s introduction of a framework for software supply chain integrity. We look forward to working with the U.S. government to make the nation more secure.
Visit our Google Cloud for U.S. federal cybersecurity webpage.
Related posts:
Autonomic Security Operations for the U.S. Public Sector Whitepaper
“Achieving Autonomic Security Operations: Reducing toil”
“Achieving Autonomic Security Operations: Automation as a Force Multiplier”
“Advancing Autonomic Security Operations: New resources for your modernization journey”
Track metrics and stay focused on threats: Achieving Autonomic Security Operations

2842
Of your peers have already read this article.
2:30 Minutes
The most insightful time you'll spend today!
What’s the most difficult question a security operations team can face? For some, is it, “Who is trying to attacks us?” Or perhaps, “Which cyberattacks can we detect?” How do teams know when they have enough information to make the “right” decision? Metrics can help inform our responses to those questions and more, but how can we tell which metrics are the best ones to rely on during mission-critical or business-critical crises?
As we discussed in our blogs, “Achieving Autonomic Security Operations: Reducing toil” and “Achieving Autonomic Security Operations: Automation as a Force Multiplier,” your Security Operations Center (SOC) can learn a lot from what IT operations discovered during the Site Reliability Engineering (SRE) revolution. In this post, we discuss how those lessons apply to your SOC, and center them on another SRE principle—Service Level Objectives (SLOs).
Even though industry definitions can vary for these terms, SLI, SLO, and SLA have specific meanings, wrote the authors of the Service Level Objectives chapter in our e-book, “Site Reliability Engineering: How Google runs production systems.” (All subsequent quotes come from the SLO chapter of the book, which we’ll refer to as the “SRE book.”)
- SLI: “An SLI is a service level indicator—a carefully defined quantitative measure of some aspect of the level of service that is provided.”
- SLO: “An SLO is a service level objective: a target value or range of values for a service level that is measured by an SLI.”
- SLA: An SLA is a Service Level Agreement about the above: “an explicit or implicit contract with your users that includes consequences of meeting (or missing) the SLOs they contain.”
In practice, we measure something (SLI) and we set the target value (SLO); we may also have an agreement about it (SLA).
This is not about cliches like “what gets measured gets done” here, but metrics and SLIs/SLOs will to a large extent determine the fate of your SOC. For example, SOCs (including at some Managed Security Service Providers) that obsessively focus on “time to address the alert” end up reducing their security effectiveness while making things go “whoosh” fast. If you equate mean time to detect or discover (MTTD) with “time to address the alert” and then push the analyst to shorten this time, attackers gain an advantage while defenders miss things and lose.
How to choose which metrics to track
One view of metrics would be that “whatever sounds bad” (such as attacks per second or incidents per employee) needs to be minimized, while “whatever sounds good” (such as successes, reliability, or uptime) needs to be maximized.
But the SRE experience is that sometimes good metrics have an optimum level, and yes, even reliability (and maybe even security). The book’s authors, Chris Jones, John Wilkes, and Niall Murphy with Cody Smith, cite an example of a service that defied common wisdom and was too reliable.
“Its high reliability provided a false sense of security because the services could not function appropriately when the service was unavailable, however rarely that occurred… SRE makes sure that global service meets, but does not significantly exceed, its service level objective,” they wrote.
The SOC lesson here is that some security metrics have optimum value. The above-mentioned time to detect has an optimum for your organization. Another example is the number of phishing incidents, which may in fact have an optimum value. If nobody phishes you, it’s probably because they already have credentialed access to many of your systems – so in your SOC, think of SLI optimums, and don’t automatically assume zero or infinite targets for metrics.
Three specific quotes from the SRE book remind us that “good metrics” may need to be balanced with other metrics, rather than blindly pushed up:
- “User-facing serving systems generally care about availability, latency, and throughput.”
- “Storage systems often emphasize latency, availability, and durability.”
- “Big data systems, such as data processing pipelines, tend to care about throughput and end-to-end latency.”
In a SOC, this may mean that you can detect threats quickly, review all context related to an incident, and perform deep threat research—but the results may differ for various threats. A fourth guidepost explains why your SOC should care even about this: “Whether or not a particular service has an SLA, it’s valuable to define SLIs and SLOs and use them to manage the service.” Indeed, we agree that SLIs and SLOs matter more for your SOC than any SLAs or other agreements.
Metrics matter, but so does flexibility
When considering the list of most difficult questions a security operations team can face, it’s vital to understand how to evaluate metrics to reach accurate answers. Consider another insight from the book: “Most metrics are better thought of as distributions rather than averages.”
If the average alert response is 20 minutes, does that mean that “all alerts are addressed in 18 to 22 minutes,” or that “all alerts are addressed in five minutes, while one alert is addressed in six hours?” Those different answers point to very different operational environments.
What we’ve seen before in SOCs is that a single outlier event is probably the one that matters most. As the authors put it, “The higher the variance in response times, the more the typical user experience is affected by long-tail behavior.” So, in security land, that one alert that took six hours to respond to was likely related to the most dangerous activity detected.
To address this, the book advises, “Using percentiles for indicators allows you to consider the shape of the distribution.” Google detection teams track the 5% and 95% values, not just averages.
Another useful concept from SRE is the “error budget,” a rate at which the SLOs can be missed, and tracked on a daily or weekly basis. It’s a SLO for meeting other SLOs.
The SOC value here may not be immediately obvious, but it’s vital to understanding the unique role security occupies in technology. In security, metrics can be a distraction because the real game is about preventing the threat actor from achieving their objectives. Based on our own experiences, most blue teams would rather miss the SLO and catch the threat in their environment. The defenders win when the attacker loses, not when the defenders “comply with a SLA.” The concept of the error budget might be your best friend here.
The SRE book takes that line of thinking even further. “It’s both unrealistic and undesirable to insist that SLOs will be met 100% of the time: doing so can reduce the rate of innovation and deployment.”
More broadly, and as we said in our recent paper with Deloitte on SOCs, rigid obeisance is its own vulnerability to exploit. “This adherence to process and lack of ability for the SOC to think critically and creativity provides potential attackers with another opportunity to successfully exploit a vulnerability within the environment, no matter how well planned the supporting processes are.”
To be successful at defending their organizations, SOCs must be less like the unbending oak and more like the pliant but resilient willow.
Track metrics but stay focused on threats
A third interesting puzzle from our SRE brethren: “Don’t pick a target based on current performance.”
We all want to get better at what we do, so choosing a target goal for improvement based on our existing performance can’t be bad, right? It turns out, however, that choosing a goal that sets up unrealistic or otherwise unhelpful, or woefully insufficient, expectations can do more harm than good.
Here is an example: An analyst handles 30 alerts a day (per their SLI), and their manager wants to improve by 15% so they set the SLO to 35 alerts a day. But how many alerts are there? Leaving aside the question of whether it is the right SLI for your SOC, what if you have 5,000 alerts, and you drop 4,970 of them on the floor. When you “improve,” you still drop 4,965 on the floor. Is this a good SLO? No, you need to hire, automate, filter, tune, or change other things in your SOC, not set better SLO targets that seemingly improve upon today’s numbers.
To this, our SRE peers say: “As a result, we’ve sometimes found that working from desired objectives backward to specific indicators works better than choosing indicators and then coming up with targets… Start by thinking about (or finding out!) what your users care about, not what you can measure.”
In the SOC, this probably means start with threat models and use cases, not the current alert pipeline performance.
SOC guidance can sometimes be more cryptic than we’ve let on. One challenging question is determining how many metrics we really need in a typical SOC. SREs wax philosophical here: “Choose just enough SLOs to provide good coverage of your system’s attributes.”
In our experience, we haven’t seen teams succeed with more than 10 metrics, and we haven’t seen people describe and optimize SOC performance with fewer than 3. However, SREs offer a helpful, succinct test: “If you can’t ever win a conversation about priorities by quoting a particular SLO, it’s probably not worth having that SLO.”
SLOs will get to define your SOC, so define them the way you want your SOC to be, the book advises. “It’s better to start with a loose target that you tighten than to choose an overly strict target that has to be relaxed when you discover it’s unattainable. SLOs can—and should—be a major driver in prioritizing work for SREs and product developers, because they reflect what users care about.”
Importantly, make SLOs for your SOC transparent within the company. As the SREs say, “Publishing SLOs sets expectations for system behavior.” The benefit is that nobody can blame you for non-performance if you perform to those agreed upon SLOs.
Finally, here are some examples of metrics from our teams at Google. In addition to reviewing all escalated alerts, they collect and review weekly:
- event volume
- event source counts
- pipeline latency
- triage time median
- triage time at 95%
Analyzing these metrics can reveal useful guidance for applying SRE principles and ideas with their detection and response teams.
Event volume: What we need to know here is what is driving the volume. Is the event volume normal, high, or low—and why? Was there a flood of messages? New data source causing high volume? What caused it? Any bad signals? Or is there a problematic area of the business that needs strategic follow-up to implement additional controls?
Event source count: Are there signals or automation that’s behaving abnormally? Is there new automation that’s misbehaving? Counting events for each source call makes for a decent SLI.
Pipeline latency: Here at Google, we aim for a confirmed detection within an hour of an event being generated. The aspirational time is 5 minutes. This means that the event pipeline latency is something that must be tracked very diligently. This also means that we must scrutinize automation latency. To achieve this, we try to remove self-caused latency so that we’re not hiding the pain of bad signals or bad automation.
We triage median and 95p time: We track the response time to events. As the SRE book points out, tracking only a single average number can get you in trouble very quickly. Note that triage time is not the same as time to resolution, but more of a dwell time for an attacker before they are discovered.
Incident resolution times: When you have a SLI but not a SLO, this can be the proverbial elephant in the room and create all sorts of bad incentives to “go fast” instead of “go good.” Specifically, SLO without SLI causes harm from encouraging the analysis to resolve quickly and potentially increase the risk of missing serious security incidents, especially when subtle signals are involved.
When reviewing alert escalations, we look to determine if the analysis is deep enough, if handoffs contain the right information for our response teams, and to get a sense of analyst fatigue. If analysts are phoning in their notes, it’s a sign that they’re over a particular signal or that there are a ton of duplicate incidents and we need to drive the business in some way.
By measuring these and other factors, metrics allow us to drive down the cost of each detection. Ultimately, this can help our detection and response operation scale faster than the threats.
Related posts:
- “Achieving Autonomic Security Operations: Automation as a Force Multiplier”
- “Achieving Autonomic Security Operations: Reducing toil”
- “Taking an autonomic approach to security operations” video
- “New Paper: “Future Of The SOC: Process Consistency and Creativity: a Delicate Balance” (Paper 3 of 4)”
- “New Paper: “Autonomic Security Operations — 10X Transformation of the Security Operations Center””
- “EP75 How We Scale Detection and Response at Google: Automation, Metrics, Toil” podcast episode
Conversations on Government Security with Dmitri Alperovitch, CrowdStrike’s Founder and Former CTO

4848
Of your peers have already read this article.
2:00 Minutes
The most insightful time you'll spend today!
As I was preparing for my fireside chat with Dmitri Alperovitch, Founder and Former CTO of CrowdStrike and Executive Chairman of the Silverado Policy Accelerator, for the Google Cloud Government Security Summit taking place on Tuesday, July 20th, I was reflecting about recent developments in cybersecurity, Zero Trust best practices – including aspects of Google’s journey – and the benefits that this approach to cybersecurity brings to the public sector.
As Dmitri and I discussed what we wanted to share during our session at the Government Security Summit, I quickly realized how pertinent the information was to our public sector audience. For that reason, ahead of Tuesday’s event, I wanted to give you a glimpse of what is to come in our fireside chat. Read below for an exclusive preview of what we plan to discuss. If you are planning to attend on Tuesday, we encourage you to ask questions through our interactive chat function and sign up for 1:1 time with our Ask the Expert program.
Check out my chat with Dmitri below…
Heather: What do you see as the most significant changes in both the attack surface and cybercriminal tactics in recent years? And why do these changes present unprecedented challenges for public sector?
Dmitri: In terms of the threat landscape, the more things change, the more they stay the same. The adversaries haven’t changed that much. It’s actually striking. Not all threats are emanating from governments or state actors – either directly or by providing safe harbor to criminals – but a large portion are. That has not changed. The volume and scale of attacks have grown, however, and become massive. And now, the greatest challenge is that everyone is a target. Before, only top-end institutions and government organizations were facing nation state threats. That has changed either through targeted attacks or accidental ones, such as WannaCry. The question now is “How do we protect the most vulnerable and extend much-needed protection to smaller organizations that do not have deep security expertise or resources?”
Heather: Earlier this year, the Biden Administration released its Cybersecurity Executive Order (EO). What do you see as the most important takeaways and potential impact of the EO?
Dmitri: The Cybersecurity EO is set to have a significant impact. The biggest change is a shift in strategy. For many decades, we all had the mentality that we have to keep attackers out of the network. That is great in theory, but we all know that it is a virtual impossibility with a network of any significant size. Adversaries will find a way in. And, we used to think it was game over once they were in the network.
Then about 10 years ago, we began to see a change. Google led the way with its BeyondCorp strategy as did Lockheed Martin with its sentinel Kill Chain framework. Now, when an adversary penetrates a network, it’s not the end of the game, but the beginning. That is the case if the organization is prepared with a network architecture built around Zero Trust principles. The adversary now has to move laterally, steal credentials, and elevate privileges to get to the resources they want. A Zero Trust architecture can slow them down and give organizations time to detect and eject adversaries from the network. The EO recognizes the power of this approach and has made Zero Trust its fundamental tenet.
Heather: Federal agencies have spent the last 60 days looking at Zero Trust and developing their plans. What do you hope to see? What is critical to accelerating implementation?
Dmitri: The EO establishes a very ambitious timeline, and the US government is not known to move rapidly due to a number of restrictions and considerations. That said, the EO recognizes the idea that logging has to be at the center of modern architecture. Agencies need full visibility into what is going on at endpoints and across the network, have to hunt continuously across networks for adversaries, and must work to rapidly eject them. That is at the core of the EO, and it is a great shift. The private sector should be watching and learning as well.
Heather: What pitfalls should the Federal government look out for in moving forward to implement the EO?
Dmitri: One of the main challenges is the limited authority of the President. Congress is the only branch with the power to change the laws. While the establishment of the Cybersecurity and Infrastructure Security Agency (CISA) a few years ago allows for hunting across agency networks, CISA still has limited authority. It cannot manage the cybersecurity of more than 120 civilian agencies.
Some agencies are performing well when it comes to security, but we have to recognize that this is not the norm. Not all agencies have the capacity to recruit the best cybersecurity talent, but their needs may be just as great. We need to centralize capabilities and leverage the cloud, like Google, to provide all agencies with the best cyber talent and resources.
Join us at the Google Cloud Government Security Summit to hear the rest of our conversation. The event is complimentary. Register today to reserve your spot on July 20th. If you have any questions about the event, please reach out to us at cloudsummitsupport@google.com.
More Relevant Stories for Your Company

Launches and Stories on Google Cloud Security from Q1: Fresh off the Boat!
The security world keeps changing, with new tools and new threats in the ever-evolving arms race that is cybersecurity. To keep you up to speed on all that Google Cloud is doing to help safeguard your data and your applications, welcome to the first installment of the Security Roundup. In

Stop Cribbing About Shadow IT and Start Taking Charge Now
Employees use tools at their disposal to get work done, but if these tools (often legacy) hamper collaboration or are inflexible, they’ll turn to less secure options for the sake of convenience. According to Gartner, a third of successful attacks experienced by enterprises will come from Shadow IT usage by 2020.

How to Retrace the Steps of a Potential Phishing Attack with Threat Analytics
The majority of cyber attacks begin with phishing emails and websites. Attackers use many tricks, including by leveraging enterprise brand assets, such as company names and logos, to develop phishing websites that appear authentic and lure internet users to enter valuable information such as user names and passwords. Experts say

How Firewall Insights can Simplify Corporate Firewall Rules
Corporate firewalls typically include a massive number of rules, which accumulate over time as new workloads are added. When rules stack up piecemeal like this, misconfigurations occur that, at best, create headaches for security administrators, and at worst, create vulnerabilities that lead to security breaches. To address this, we have






