Quanta Services is the leading specialty contractor with the largest and highly-skilled trained workforce in North America. It provides fully-integrated solutions for the electric power

pipeline industrial and telecommunications industries the company’s geographic footprint which includes North America Latin America and Australia.

It’s network of companies ensures world-class execution with local delivery and has over 40,000 employees.

“The exciting thing about Backstory is it allows us to land massive amounts of data from all of our different security tools and then Chronicle worries about how to correlate and aggregate this information for us. We can then focus on the highest priority threats.”

Richard Breaux, Manager Security Operations Quanta Services

Because Quanta Services’ customers provide energy and telecommunications to their customers, they have world-class cybersecurity requirements.

“We must ensure that Quanta leads the industry given the fact that our business is building the core infrastructure that powers people’s lives. It’s crucial that we meet their cybersecurity requirements,” says James Stinson, VP-IT, Quanta Services.

To support this goal Quanta implemented a number of security tools that generate terabytes of security telemetry. Over time, however, these systems generated an information overload for the company’s limited pool of qualified security resources.

“Our logging tools also couldn’t keep up with the rapidly growing information. With limited resources we need to focus our security analysts time on high quality work instead of digging through mountains of data,” says Stinson.

They achieve this by implementing Backstory and Chronicle, which is on Google Cloud.

“The exciting thing about Backstory is it allows us to land massive amounts of data from all of our different security tools and then Chronicle worries about how to correlate and aggregate this information for us. We can then focus on the highest priority threats,” says Richard Breaux, Manager Security Operations Quanta Services.

As a result, the company’s security  team spends less time getting to the core information they need to address these incidents.

“What used to take us 15 minutes or more, we can now accomplish in seconds,” says Breaux.

Forensics is the application of science to criminal and civil laws. It is a proven approach for gathering and processing evidence at a crime scene. An integral step in the forensics process is the isolation of the scene without contaminating or modifying the evidence. The isolation step prevents any further contamination or tampering with possible evidence. The same philosophy can be applied to the investigation of digital events.

In this post we will review methods, tactics and architecture designs to isolate an infected VM while still making it accessible to forensic tools. The goal is to allow access so that data and evidence can be captured while protecting other assets. There are many forensic tools for networking that can be used to analyze the captured traffic. This post does not cover these tools but rather how to configure GCP to capture live traffic in the most efficient and secured way. Once traffic is captured, customers can use whatever tools they prefer to run the analysis. More details about these tools and required agents can be found here and details about open source tooling that Google and others are developing are available here.

In cloud security context, when a VM shows signs of compromise, the most common immediate reaction is to take a snapshot, shut down the instance and relocate the image snapshot to an isolated environment, a method known as “dead analysis”. However, shutting down the instance will impede an important step in the investigation and digital forensics, as some important information in a buffer or the RAM may be lost. 

The other forensic approach is “live analysis”, in which the VM is kept on and evidence is gathered from the VM directly. Live forensics enables the imaging of RAM, bypasses most hard drives and software encryption, determines the cause of abnormal traffic, and is extremely useful when dealing with active network intrusions. This process is usually performed by forensic analysts. For example, if there is a good chance the malware resides only in memory then live forensics is, in some cases, the only way to capture and analyze the malware. In this method, in addition to disk and memory evidence, a forensic analysis can also capture live-network from data sent over the compromised VM network interfaces. Some of the benefits of collecting live networks are reconstruction and visualizing traffic flow in real-time, in particular during active network intrusions or attacks. 

In the cloud, a VM must be isolated when it becomes apparent that an incident has happened, in order to protect other VMs from being infected. Our Cloud Forensics 101 session covers the process and required artifacts, such as logs, that need to be collected for cloud forensics. 

What happens when your image is compromised

Let’s now assume that one of the VMs in your infrastructure has been compromised and alarms are coming from products such as GCP’s Cloud Security and Command Center, Chronicle backstory or your SIEM. 

An incident response plan consists of 3 phases: preparation (actions taken before an attack), detection (actions taken during an attack) and response (actions taken after an attack). During the detection phase, the Computer Security Incident Response Team (CSIRT) or threat analysts decide whether live acquisition analysis is required. If live forensics is required, for example when it is vital to acquire a VM’s RAM, then one of the first courses of action is to isolate and contain the VM from the rest of the world and connect the Forensics VPC to the VM for investigation. The forensics VPC resides in a forensics GCP project, it includes digital forensics tools to capture evidence from the VM such as SANS Investigative Forensics Toolkit – SIFT, The Sleuth Kit, Autopsy, Encase, FTK and alike. These tools are already installed, configured, tested and ready to use. The forensics project will also save and preserve evidence such as disk and memory images for forensic review.

We’ll cover two scenarios in this post, the first scenario is to isolate the image and connect the forensics VPC to the image for live acquisition. 

In the second scenario we will also capture live traffic from the isolated image for live network digital forensics. To capture live traffic from the infected VM, we will leverage the GCP Packet Mirroring service to duplicate all traffic going in and out of the VM and send it to a Forensics VPC for analysis. Network forensics analysis tools such as Palo Alto VM-Series for IDS, ExtraHop Reveal(x), CheckPoint CloudGuard, Arkime (formerly Moloch), Corelight are installed, configured and ready for deployment in the Forensics VPC, these tools will be used to analyze the duplicate network traffic. 

Isolating the infected VM from other resources and connecting the forensics VPC

As part of the Incident Response plan preparation phase, the CSIRT created a Google Cloud Forensics Project. Since the Forensics project will be used only when needed, it’s better to automate the creation of the project and its resources with a tool such as Terraform. It is important to grant access to this project only to individuals and groups who deal with incident response and forensics, such as CSIRT. As shown in figure 1, the Forensics project on the right includes its own VPC, non-overlapped subnet and VM images with pre-installed and pre-configured forensics tools. Internal load-balancer and instance-groups are also configured, we will use these resources to capture live traffic, as described later in this post.

In order to contain the spread of any malware or network activity, such as data exfiltration, we’ll isolate the VM with VPC firewall rules. The GCP VPC firewall is a distributed firewall that always enforces its rules, protecting the instances regardless of their configuration and operating systems. In other words, the compromised VM cannot override the firewall enforcement if its policies follow the principle of least privilege . Rules can be applied to all instances in the network, target network tags or service accounts.

Step 1 in the diagram above shows how an infected VM is isolated from the rest of the network by firewall rules that deny any ingress and egress traffic from any CIDR beside the forensics subnet CIDR. The infected VM is tagged with a unique network tag, for example “<image-name>_InfectedVM”, then firewalls rules are applied on the network tag. This ensures that the infected VM is isolated from the project and the Internet while enabling access to the VM via VPC peering which we’ll configure in step-2. You can learn more about VPC firewalls rules here.

In step 2, the VPC from the forensics project is peered with the VPC in the production project. When VPC peering is established routes are exchanged between the VPCs. By default, VPC peering exchanges all subnet routes, however, custom routes can also be filtered if required. At this point, the VM from the forensics project can communicate with the infected VM and start the live forensics analysis job using the pre-installed and pre-configured forensics tools.

Shared VPC is a network construct that allows you to connect resources from multiple projects, called service-projects, to a common VPC in a host-project. VPCs from different projects can securely communicate with each other via the hosted project network while centralizing the network administration. Figure 2 depicts Shared VPC topology, rather than using VPC peering, during step 2 the Forensics project is simply attached to the host project. After the attachment, the Shared VPC allows the forensics tools to communicate with the infected VMs.

Capturing live network traffic with Google Traffic Mirroring

If live network forensics is required, for example during active network intrusions, then the incoming and outgoing traffic needs to be duplicated and captured. While VPC Flow logs capture the networking metadata telemetry, this is not enough for live network forensics analysis. GCP Packet Mirroring clones the traffic of a specified instance in a VPC and forwards it to a specified internal load balancer which collects the mirrored traffic and sends it to an attached instance group. Packet mirroring captures all the traffic from the specified subnet, network tags, or instance name. 

Figure 3 depicts the steps that allow the compromised VM to communicate with the rest of the world (for example beaconing with C&C) while capturing all traffic for investigation in a peered VPC deployment.

Figure 4 depicts the steps that allow the compromised VM to communicate with the rest of the world while capturing all traffic for investigation in a shared VPC deployment.

We will use the Forensics’ project internal load balancer and the instance group VMs which include packet capture and analysis tools. Note that production and forensics networks must be in the same region. Detailed steps to configure packet mirroring are available on this page. 

If you are using a Shared VPC then check the Packet Mirroring configuration for Shared VPC for configuration details. Figure 4 depicts the packet mirroring flow in a shared VPC topology.

It is recommended to automate and periodically test the process to make sure that in case of an incident, the entire setup and Forensics toolchain can be quickly deployed. If after initial investigation a suspicious destination, such as a Command and Control [C&C] Server, has been identified, then the Packet Mirroring policy can be adjusted with a policy filter that only mirrors traffic from that C&C server IP address.

An incident management plan must be in place for companies using cloud services, and this plan should also include the option of using live acquisition when necessary. design and preparation for forensics acquisition allows the company to build the infrastructure that can be deployed and connected to the appropriate VM automatically. The architectures described in this post can help the process of collecting and preserving vital evidence for the forensic process, while the incident response team resolves the incident.

As sophisticated cyberattack campaigns increasingly target the U.S. public and private sectors during the COVID era, the White House and federal agencies have taken steps to protect critical infrastructure and remote-work infrastructure. These include Executive Order 14028 and the Office of Management and Budget’s Memorandum M-21-31, which recommend adopting Zero Trust policies, and span software supply chain security, cybersecurity threat management, and strengthening cyberattack detection and response.

However, implementation can be a challenge for many agencies due to cost, scalability, engineering, and a lack of resources. Meeting the requirements of the EO and OMB guidance may require technology modernization and transformational changes around workforce and business processes.

Today we are announcing Autonomic Security Operations (ASO) for the U.S. public sector, a solution framework to modernize cybersecurity analytics and threat management that’s aligned with the objectives of EO 14028 and OMB M-21-31. Powered by Google’s Chronicle and Siemplify, ASO helps agencies to comprehensively manage cybersecurity telemetry across an organization, meet the Event Logging Tier requirements of the White House guidance, and transform the scale and speed of threat detection and response. ASO can support government agencies in achieving continuous detection and continuous response so that security teams can increase their productivity, reduce detection and response time, and keep pace with – or ideally, stay ahead of – attackers.

While the focus of OMB M-21-31 is on the implementation of technical capabilities, transforming security operations will require more than just technology. Transforming processes and people in the security organization is also important for long-term success. ASO provides a more comprehensive lens through which to view the OMB event logging capability tiers, which can help drive a parallel transformation of security-operations processes and personnel.

Modern Cybersecurity Threat Detection and Response

Google provides powerful technical capabilities to help your organization achieve the requirements of M-21-31 and EO 14028:

Security Information & Event Management (SIEM) – Chronicle provides high-speed petabyte-scale analysis, and is capable of consuming log types outlined in the Event Logging (EL) tiers in a highly cost-effective manner.

Security Orchestration, Analytics, and Response (SOAR) – Siemplify offers dozens of out-of-box playbooks to deliver agile cybersecurity response and drive mission impact, including instances of automating 98% of Tier-1 alerts and driving an 80% reduction in caseload.

User and Entity Behavior Analytics (UEBA) – For agencies that want to develop their own behavioral analytics, agencies can use BigQuery, Google’s petabyte scale data lake, to store, manage, and analyze diverse data types from many sources. Telemetry can be exported out of Chronicle, and custom data pipelines can be built to import other relevant data from disparate tools and systems, such as IT Ops, HR and personnel data, and physical security data. From there, users can leverage BQML to readily generate machine learning models without needing to move the data out of BigQuery. For Google Cloud workloads, our Security Command Center Premium product offers native, turnkey UEBA across GCP workloads.

Endpoint Detection and Response (EDR) – For most agencies, EDR is a heavily adopted technology that has broad applicability in Security Operations. We offer integrations to many EDR vendors. Take a look at our broad list of Chronicle integrations here.

Threat intelligence – Our solution offers a native integration with VirusTotal, has the ability to operationalize threat intelligence feeds natively in Chronicle, and integrates with various TI and TIP solutions.

Community Security Analytics

To increase collaboration across public-sector and private-sector organizations, we recently launched our Community Security Analytics (CSA) repository, where we’ve partnered with the MITRE Engenuity Center for Threat-Informed Defense, CYDERES, and others to develop open-source queries and rules that support self-service security analytics for detecting common cloud-based security threats. CSA queries are mapped to the MITRE ATT&CK® framework of tactics, techniques and procedures (TTPs) to help you evaluate their applicability in your environment and include them in your threat model coverage.

“Deloitte is excited to collaborate with Google Cloud on their transformational public sector Autonomic Security Operations (ASO) solution offering. Deloitte has been recognized as Google Cloud’s Global Services Partner of the Year for four consecutive years, and also as their inaugural Public Sector Partner of the Year in 2020,” said Chris Weggeman, managing director of GPS Cyber and Strategic Risk, Google Cloud Cyber Alliance Leader, Deloitte & Touche LLP. “Our deep bench of more than 1,000 Google Cloud certifications, capabilities spanning the Google Cloud security portfolio, and decades of delivery experience in the government and public sector makes us well-positioned to help our clients undertake critical Security Operations Center transformation efforts with Google Cloud ASO.”

Cost-effective for government agencies

To help Federal Agencies meet the requirements of M-21-31 and the broader EO, Google’s ASO solutions can drive efficiencies and help manage the overall costs of the transformation. ASO can make petabyte-scale data ingestion and management more viable and cost-effective. This is critical at a time when M-21-31 is requiring many agencies to ingest and manage dramatically higher volumes of data that had not been previously budgeted for.

Partners

We’re investing in key partners who can help support U.S. government agencies on this journey. Deloitte and CYDERES both have deep expertise to help transform agencies’ Security Operations capabilities, and we continue to expand our partners to support the needs of our clients. A prototypical journey can be seen below.

“Cyderes shares Google Cloud’s mission to transform security operations, and we are honored to deliver the Autonomic Security Operations solution to the U.S. public sector. As the number one MSSP in the world (according to Cyber Defense Magazine’s 2021 Top MSSPs List) with decades of advisory and technology experience detecting and responding to the world’s biggest cybersecurity threats, Cyderes is uniquely positioned to equip federal agencies and departments to go far beyond the requirements of the executive order to transform their security programs entirely via Google’s unique ASO approach,” said Robert Herjavec, CEO of CYDERES. “As an original launch partner of Google Cloud’s Chronicle, our deep expertise will propel our joint offering to modernize security operations in the public sector, all with significant cost efficiency compared to competing solutions.” said Eric Foster, President of CYDERES.

Embracing ASO

Autonomic Security Operations can help U.S. government agencies advance their event logging capabilities in alignment with OMB maturity tiers. More broadly, ASO can help the U.S. government undertake a larger transformation of technology, process, and people, toward a model of continuous threat detection and response. As such, we believe that ASO can help address a number of challenges presently facing cybersecurity teams, from the global shortage of skilled workers, to the overproliferation of security tools, to poor cybersecurity situational awareness and analyst burnout caused by an increase of data without sufficient context or tools to automate and scale detection and response.

We believe that by embracing ASO, agencies can help agencies achieve:

10x technology, through the use of cloud-native tools that help agencies meet event logging requirements in the near term, while powering a longer-term transformation in threat management;

10x process, by redesigning workflows and using automation to achieve Continuous Detection and Continuous Response in security operations;

10x people, by transforming the productivity and effectiveness of security teams and expanding their diversity; and

10x influence across the enterprise through a more collaborative and data-driven approach to solving security problems between security teams and non-security stakeholders.

To learn more about Google’s Autonomic Security Operations solution for the U.S. public sector, please read our whitepaper. More broadly, Google Cloud continues to provide leadership and support for a wide range of critical public-sector initiatives, including our work with the MITRE Engenuity Center for Threat-Informed Defense, the membership of Google executives on the President’s Council of Advisors on Science and Technology and the newly established Cyber Safety Review Board; Google’s White House commitment to invest $10 billion in Zero Trust and software supply chain security, and Google Cloud’s introduction of a framework for software supply chain integrity. We look forward to working with the U.S. government to make the nation more secure.

Visit our Google Cloud for U.S. federal cybersecurity webpage.

Related posts:

Autonomic Security Operations for the U.S. Public Sector Whitepaper

“Achieving Autonomic Security Operations: Reducing toil”

“Achieving Autonomic Security Operations: Automation as a Force Multiplier”

“Advancing Autonomic Security Operations: New resources for your modernization journey”

Yesterday we held our final Google Cloud Security Talks event of 2021. Our event focused on zero trust and covered everything from Google’s history with BeyondCorp to our strategic thinking when it comes to applying zero trust principles to production environments. We shared product updates across the portfolio and talked about how zero trust fits into our invisible security vision.

In case you missed the event, we’ve put together a brief recap below. Of course, we’d also encourage you to check out the sessions on-demand for all the details!

Our opening keynote highlighted many recent security launches, including:

• the general availability of Cloud IDS, which helps detect network-based threats, including attempts to exploit the Apache Log4j vulnerability; 
• the new Mute Findings feature for Security Command Center which helps organizations more effectively manage their security posture on GCP based on their policies and requirements;
• and the Threat Horizons report, a new report by the Google Cybersecurity Action Team (GCAT) describing recent threat intelligence observations from teams across Google.

We then brought together security leaders across Google for a panel discussion of our zero trust vision. Our panelists emphasized the need to embrace the fact that security is not static which makes adoption of a zero trust mindset where trust is established via multiple mechanisms and verified continuously an imperative. The session also highlighted how even taking small steps – such as prioritizing specific applications as you begin your zero trust journey – can still have a large impact on improving your security posture.

Next, we covered some recent product announcements from the BeyondCorp Enterprise team, including the new Policy Troubleshooter feature. This session gave users a closer look at new product features across access, signal integrations, and threat and data protection capabilities leveraging machine-learning in real-time.

We then pivoted the conversation and two of our speakers shed more light on Google’s BeyondProd approach. First, we shared our learnings from building a cloud-native security model, which provides significant benefits to both application development and security teams. We outlined different principles you can apply to your infrastructure design and operations in order to strengthen the deployment of your workloads, better secure their communications, and limit their exposure to other workloads, in ways that can ultimately reduce the burden on individual developers.

We discussed how customers can bring BeyondProd to life in their own environments and offer zero trust protection for workloads with Google Cloud capabilities. These sessions featured customers who shared their experiences with VPC-Service Controls and Certificate Authority (CA) Service. We also announced some exciting news: CA Service now supports third-party identity federation and can issue certificates to attest to on-premises third-party identities using Google Cloud workload identity federation. Certificates can be issued for both users and workloads. 

Next, in light of the White House effort to deploy a zero trust model across the US federal government, we spent some time sharing how Google Cloud can support government guidance. Google recently announced a $10 billion investment to strengthen cybersecurity, including expanding zero-trust programs, helping secure the software supply chain, and enhancing open-source security. This session gave an overview of how different Google Cloud services align to federal cybersecurity priorities.

Our last session provided an overview of how we are strengthening security and privacy across the Google Workspace platform applying zero trust principles to the applications that allow customers to collaborate effectively with features such as Trust Rules for Drive.

These sessions will be available on-demand on our Google Cloud Security Talks page for you to reference. If you’d like to speak with a representative about Google Cloud’s zero trust access solutions, please fill out this form. Lastly, if you are planning to attend the 2022 RSA Conference in San Francisco (February 7-10, 2022), check out goo.gle/rsa-2022 to arrange a meeting with our team at the event.

Organizations in every region and industry are developing APIs to enable easier and more standardized delivery of services and data for digital experiences. This increasing shift to digital experiences has grown API usage and traffic volumes. However, as malicious API attacks also have grown, API security has become an important battleground over business risk.

To help customers more easily address their growing API security needs, Google Cloud is announcing today the Preview of Advanced API Security, a comprehensive set of API security capabilities built on Apigee, our API management platform. Advanced API Security enables organizations to more easily detect security threats. Here’s a closer look at the two key functionality included in this launch: identifying API misconfigurations and detecting bots.

Identify API misconfigurations

Misconfigured APIs are one of the leading reasons for API security incidents. In 2017, Gartner® predicted that by 2022 API abuses will be the most frequent attack vector resulting in data breaches for enterprise web applications. Today, our customers tell us application API security is one of their top concerns, which is supported by an independent study from 2021 by Fugue and Sonatype. The report found that misconfigurations are the number one cause of data breaches, and that “too many cloud APIs and interfaces to adequately govern” are frequently the main point of attack in cyberattacks.

While identifying and resolving API misconfigurations is a top priority for many organizations, the configuration management process can be time consuming and require considerable resources.

Advanced API Security can make it easier for API teams to identify API proxies that do not conform to security standards. To help identify APIs that are misconfigured or experiencing abuse, Advanced API Security regularly assesses managed APIs and provides API teams with a recommended action when configuration issues are detected.

Advanced API Security identifies misconfigured API proxies, including the missing CORS policy.
APIs form an integral part of the digital connective tissue that make modern medicine run smoothly for patients and healthcare staff. One common healthcare API use case occurs when a healthcare organization inputs a patient’s medical coverage information into a system that works with insurance companies. Almost instantly, that system determines the patient’s coverage for a specific medication or procedure, a process which is enabled by APIs. Because of the often-sensitive personal healthcare data being transmitted, it is important that the required authentication and authorization policies are implemented so that only authorized users, such as an insurance company, can access the API.

Advanced API Security can detect if those required policies have not been applied, an alert which can help reduce the surface area of API security risks. By leveraging Advanced API Security, API teams at healthcare organizations can more easily detect misconfiguration issues and can reduce security risks to sensitive information.

Detect Bots

Because of the increasing volume of API traffic, there is also an increase in cybercrime in the form of API bot attacks—the automated software programs deployed over the Internet for malicious purposes like identity theft.

Advanced API Security uses pre-configured rules to help provide API teams an easier way to identify malicious bots within API traffic. Each rule represents a different type of unusual traffic from a single IP address. If an API traffic pattern meets any of the rules, Advanced API Security reports it as a bot.

Additionally, Advanced API Security can speed up the process of identifying data breaches by identifying bots that successfully resulted in the HTTP 200 OK success status response code.

Financial services APIs are frequently the target of malicious bot attacks due to the high-value data that is processed. A bank that has adopted open banking standards by making APIs accessible to customers and partners can use Advanced API Security to make it easier to analyze traffic patterns and identify the sources of malicious traffic. You may experience this when your bank allows you to access your data with a third-party application. While a malicious hacker could try to use a bot to access this information, Advanced API Security can help the bank’s API team to identify and stop malicious bot activity in API traffic.

API Security at Equinix

Equinix powers the world’s digital leaders, bringing together and interconnecting infrastructure to fast-track digital advantage. Operating a global network of more than 240 data centers with a 99.999% or greater uptime, Equinix simplifies global interconnections for organizations, saving customers time and effort with the Apigee API management platform.

“A key enabler of our success is Google’s Apigee, delivering digital infrastructure services securely and quickly to our customers and partners,” said Yun Freund, senior vice president of Platform at Equinix. “Security is a key pillar to our API-first strategy and Apigee has been instrumental in enabling our customers to securely bridge the connections they need for their businesses to easily identify potential security risks and mitigate threats in a timely fashion. As our API traffic has grown, so has the amount of time and effort required to secure our APIs. Having a bundled solution in one managed platform gives us a differentiated high-performing solution.”

Getting started

To learn more, check out the documentation or contact us to request access to get started with Advanced API Security.

To learn more about API security best practices, please register to attend our Cloud OnAir webcast on Thursday, July 28th, 2:00 pm PT.