Forensics is the application of science to criminal and civil laws. It is a proven approach for gathering and processing evidence at a crime scene. An integral step in the forensics process is the isolation of the scene without contaminating or modifying the evidence. The isolation step prevents any further contamination or tampering with possible evidence. The same philosophy can be applied to the investigation of digital events.

In this post we will review methods, tactics and architecture designs to isolate an infected VM while still making it accessible to forensic tools. The goal is to allow access so that data and evidence can be captured while protecting other assets. There are many forensic tools for networking that can be used to analyze the captured traffic. This post does not cover these tools but rather how to configure GCP to capture live traffic in the most efficient and secured way. Once traffic is captured, customers can use whatever tools they prefer to run the analysis. More details about these tools and required agents can be found here and details about open source tooling that Google and others are developing are available here.

In cloud security context, when a VM shows signs of compromise, the most common immediate reaction is to take a snapshot, shut down the instance and relocate the image snapshot to an isolated environment, a method known as “dead analysis”. However, shutting down the instance will impede an important step in the investigation and digital forensics, as some important information in a buffer or the RAM may be lost. 

The other forensic approach is “live analysis”, in which the VM is kept on and evidence is gathered from the VM directly. Live forensics enables the imaging of RAM, bypasses most hard drives and software encryption, determines the cause of abnormal traffic, and is extremely useful when dealing with active network intrusions. This process is usually performed by forensic analysts. For example, if there is a good chance the malware resides only in memory then live forensics is, in some cases, the only way to capture and analyze the malware. In this method, in addition to disk and memory evidence, a forensic analysis can also capture live-network from data sent over the compromised VM network interfaces. Some of the benefits of collecting live networks are reconstruction and visualizing traffic flow in real-time, in particular during active network intrusions or attacks. 

In the cloud, a VM must be isolated when it becomes apparent that an incident has happened, in order to protect other VMs from being infected. Our Cloud Forensics 101 session covers the process and required artifacts, such as logs, that need to be collected for cloud forensics. 

What happens when your image is compromised

Let’s now assume that one of the VMs in your infrastructure has been compromised and alarms are coming from products such as GCP’s Cloud Security and Command Center, Chronicle backstory or your SIEM. 

An incident response plan consists of 3 phases: preparation (actions taken before an attack), detection (actions taken during an attack) and response (actions taken after an attack). During the detection phase, the Computer Security Incident Response Team (CSIRT) or threat analysts decide whether live acquisition analysis is required. If live forensics is required, for example when it is vital to acquire a VM’s RAM, then one of the first courses of action is to isolate and contain the VM from the rest of the world and connect the Forensics VPC to the VM for investigation. The forensics VPC resides in a forensics GCP project, it includes digital forensics tools to capture evidence from the VM such as SANS Investigative Forensics Toolkit – SIFT, The Sleuth Kit, Autopsy, Encase, FTK and alike. These tools are already installed, configured, tested and ready to use. The forensics project will also save and preserve evidence such as disk and memory images for forensic review.

We’ll cover two scenarios in this post, the first scenario is to isolate the image and connect the forensics VPC to the image for live acquisition. 

In the second scenario we will also capture live traffic from the isolated image for live network digital forensics. To capture live traffic from the infected VM, we will leverage the GCP Packet Mirroring service to duplicate all traffic going in and out of the VM and send it to a Forensics VPC for analysis. Network forensics analysis tools such as Palo Alto VM-Series for IDS, ExtraHop Reveal(x), CheckPoint CloudGuard, Arkime (formerly Moloch), Corelight are installed, configured and ready for deployment in the Forensics VPC, these tools will be used to analyze the duplicate network traffic. 

Isolating the infected VM from other resources and connecting the forensics VPC

As part of the Incident Response plan preparation phase, the CSIRT created a Google Cloud Forensics Project. Since the Forensics project will be used only when needed, it’s better to automate the creation of the project and its resources with a tool such as Terraform. It is important to grant access to this project only to individuals and groups who deal with incident response and forensics, such as CSIRT. As shown in figure 1, the Forensics project on the right includes its own VPC, non-overlapped subnet and VM images with pre-installed and pre-configured forensics tools. Internal load-balancer and instance-groups are also configured, we will use these resources to capture live traffic, as described later in this post.

In order to contain the spread of any malware or network activity, such as data exfiltration, we’ll isolate the VM with VPC firewall rules. The GCP VPC firewall is a distributed firewall that always enforces its rules, protecting the instances regardless of their configuration and operating systems. In other words, the compromised VM cannot override the firewall enforcement if its policies follow the principle of least privilege . Rules can be applied to all instances in the network, target network tags or service accounts.

Step 1 in the diagram above shows how an infected VM is isolated from the rest of the network by firewall rules that deny any ingress and egress traffic from any CIDR beside the forensics subnet CIDR. The infected VM is tagged with a unique network tag, for example “<image-name>_InfectedVM”, then firewalls rules are applied on the network tag. This ensures that the infected VM is isolated from the project and the Internet while enabling access to the VM via VPC peering which we’ll configure in step-2. You can learn more about VPC firewalls rules here.

In step 2, the VPC from the forensics project is peered with the VPC in the production project. When VPC peering is established routes are exchanged between the VPCs. By default, VPC peering exchanges all subnet routes, however, custom routes can also be filtered if required. At this point, the VM from the forensics project can communicate with the infected VM and start the live forensics analysis job using the pre-installed and pre-configured forensics tools.

Shared VPC is a network construct that allows you to connect resources from multiple projects, called service-projects, to a common VPC in a host-project. VPCs from different projects can securely communicate with each other via the hosted project network while centralizing the network administration. Figure 2 depicts Shared VPC topology, rather than using VPC peering, during step 2 the Forensics project is simply attached to the host project. After the attachment, the Shared VPC allows the forensics tools to communicate with the infected VMs.

Capturing live network traffic with Google Traffic Mirroring

If live network forensics is required, for example during active network intrusions, then the incoming and outgoing traffic needs to be duplicated and captured. While VPC Flow logs capture the networking metadata telemetry, this is not enough for live network forensics analysis. GCP Packet Mirroring clones the traffic of a specified instance in a VPC and forwards it to a specified internal load balancer which collects the mirrored traffic and sends it to an attached instance group. Packet mirroring captures all the traffic from the specified subnet, network tags, or instance name. 

Figure 3 depicts the steps that allow the compromised VM to communicate with the rest of the world (for example beaconing with C&C) while capturing all traffic for investigation in a peered VPC deployment.

Figure 4 depicts the steps that allow the compromised VM to communicate with the rest of the world while capturing all traffic for investigation in a shared VPC deployment.

We will use the Forensics’ project internal load balancer and the instance group VMs which include packet capture and analysis tools. Note that production and forensics networks must be in the same region. Detailed steps to configure packet mirroring are available on this page. 

If you are using a Shared VPC then check the Packet Mirroring configuration for Shared VPC for configuration details. Figure 4 depicts the packet mirroring flow in a shared VPC topology.

It is recommended to automate and periodically test the process to make sure that in case of an incident, the entire setup and Forensics toolchain can be quickly deployed. If after initial investigation a suspicious destination, such as a Command and Control [C&C] Server, has been identified, then the Packet Mirroring policy can be adjusted with a policy filter that only mirrors traffic from that C&C server IP address.

An incident management plan must be in place for companies using cloud services, and this plan should also include the option of using live acquisition when necessary. design and preparation for forensics acquisition allows the company to build the infrastructure that can be deployed and connected to the appropriate VM automatically. The architectures described in this post can help the process of collecting and preserving vital evidence for the forensic process, while the incident response team resolves the incident.

In cybersecurity, speed matters. Whether a security analyst is trying to understand the details of an alert that was triggered by an indicator of compromise (IoC), or find additional context for a suspicious asset, speed is often the critical factor that will help thwart a cyberattack before threat actors are able to inflict damage.

With speed in mind, we are pleased to announce the general availability of our new investigative experience in Chronicle Security Operations. We are continuing to deliver on our mission to bring the power of Google to security operations and are raising the bar for search and the investigative experience in the SOC.

With this release, SecOps teams will be able to harness Chronicle’s lightning-fast search across any form of structured data. Additionally, the new investigation experience can provide greater flexibility to pivot and drill-down when conducting complex, open-ended threat investigations and surface insights quickly and easily.

Our Unified Data Model (UDM) schema, with its built-in flexibility that can effectively handle a wide variety of security related events, is at the heart of Chronicle’s powerful search. We have scaled this capability by optimizing query responses across structured data. Additionally, analysts can investigate large datasets and build complex queries with a new and intuitive user experience. With user personalization enhancements, analysts can quickly access saved searches and top queries in their environment to improve routine SOC workflows.

With the new investigative experience, security teams can:

• Drive faster threat understanding with an interactive event results timeline that helps eliminate unnecessary long wait times by streaming results as they are processed to quickly begin threat analysis
• Use enhanced context and operationalize relevant data for threat analysis with one-click filter-to-query conversion
• Personalize the analyst experience with saved search and search history functions for quicker analyst knowledge recall
• Power threat investigation and hunting with a new, improved, highly performant UDM search

Let’s look at an example of how to use the reimagined investigative experience and our new broader, faster search.

In our use-case, a security analyst is investigating a curated detection alert for potentially suspicious behavior on a Windows environment. Additionally, there is a low prevalence domain from host “win-dc-01” with ip “10.166.0.3”. To investigate further, let’s open the UDM Search page and construct a query containing the host and IP information along with destination information of the domain the host was contacting (edge.microsoft.com). Over 70,000 events stream into the investigation interface providing the analyst an immediate picture of data surrounding their alert.

The new interactive events timeline can provide a clear picture of event trends over time with key statistical data which can be easily filtered. Additionally, with the new quick filters, the analyst is able to easily filter out hosts that are known-good to get pertinent information about the domain.

Analyzing value aggregations present in the filter panel, automatically generated by Chronicle, enables analysts to domain values of highest interest, and quickly determine that approximately 800 events were present in the last week that related to edge.microsoft.com URL. Since Microsoft Edge is disallowed in the organization, there should not be any outbound traffic to this type of destination.

As a final step before orchestrating a response in Chronicle SOAR’s case management, the analyst can save their search to quickly recall the steps they took in the future for related investigations.

We have already seen our customers use these new capabilities in preview to build new use cases, accelerate existing threat hunting workflows, and drive faster threat response. We will continue our mission to bring Google speed, scale, and intelligence to the investigative experience, expediting “time to aha” for security analysts, and driving better, faster responses.

Ready to put Chronicle to work in your Security Operations Center? Contact Google Cloud sales or your customer success CSM team. You can also learn more about all these new capabilities in Google Chronicle in our product documentation.

As one of the most trusted cloud platform providers, Google is committed to providing our clients secure and reliable environments for their workloads. Google believes the future of computing will increasingly shift to private, encrypted services where users can be confident that their data is not being exposed to cloud providers or their own insiders. Confidential Computing helps make this future possible by keeping data encrypted in memory, and elsewhere outside the CPU, while it is being processed.

Since our first Confidential Computing offering in 2018, Google has been a pioneer in making the technology widely available through our cloud, while also constantly adding new features that allow a wide range of organizations to benefit from it. 

Expanding the Confidential Computing portfolio

Today at the RSA Conference 2023, we are announcing several exciting updates to our Confidential Computing portfolio.

We are expanding our portfolio with the next generation of our Confidential Computing VM instances that utilize AMD Infinity Guard technology, specifically AMD SEV-SNP (Secure Encrypted Virtualization-Secure Nested Paging). SEV-SNP adds additional new hardware-based security protections such as strong memory integrity protection and hardware rooted attestation. Confidential VMs on general purpose N2D VMs with SEV-SNP enabled are now in Private Preview. Learn more in our deep dive here.

“Advanced, hardware-based security features – such as AMD Infinity Guard built into AMD EPYC processors – are a crucial part of cloud offerings and the further expansion of cloud computing,” said Ram Peddibhotla, corporate vice president, Cloud Business, AMD. “Our work with Google Cloud delivers on the leadership security features our joint customers demand with nearly effortless implementation and minimal performance impact as more mission critical and business critical workloads move to the cloud.”

Google is committed to ensuring Confidential Computing technology is as secure as possible before releasing products to customers. We constantly evaluate various attack vectors to make certain that Google Cloud Confidential Computing environments are protected against a wide range of attacks. 

Throughout last year, Google and Intel collaborated on a research project to identify potential security vulnerabilities in Intel’s new Confidential Computing technology, Trust Domain Extensions (TDX). We’ve recently released the full report and made it available in our blog here.

“We want to make it such that people don’t worry about the security and trustworthiness of their data,” said Anil Rao, vice president and general manager of systems architecture and engineering in the Office of the CTO at Intel. “Organizations use confidential computing to control their data and provide access to trusted parties in a manner that is verifiable, revocable and time sensitive – we have an obligation to make sure the technology is secure. Our early effort with Google solidifies our commitment to perform thorough analysis to address all potential vulnerabilities.”

Confidential Space is now in general availability. Confidential Space builds on Confidential Computing and provides a secure enclave, also known as a Trusted Execution Environment (TEE). Google Cloud customers can leverage the TEE for various privacy-focused use cases such as joint data analysis and machine learning (ML) model training, with trust guarantees that data can stay protected — including hardened protection against cloud service provider access. 

Confidential Space allows for new privacy-preserving technologies to take shape, and one such technology is The Privacy Sandbox. This is a crucial step forward as Privacy Sandbox for the Web will phase out third-party cookies and limit covert tracking. By supporting a choice of trusted execution environments, including Confidential Space, the Privacy Sandbox will provide ad-techs with safer alternatives to existing technology, so they can continue building digital businesses while keeping user data private.

“Trusted Execution Environments (TEEs) are already in use across industries such as digital advertising to enable privacy preserving analytics, and we are excited to leverage Google Cloud’s built-in TEE called Confidential Space,” said Gaurav Bhaya, vice president and general manager, Google Ads Measurement.

Building on our history of innovation with Confidential Computing

Google has worked diligently with internal and external partners to validate, audit and publish the security assessment of the hardware, firmware, and software that underpins Confidential Computing. 

Last year, a joint security effort between Google Project Zero, Google Cloud, and AMD led to more secure technology for the entire industry, regardless of what service provider you choose for your Confidential Computing deployment. You can learn more in our full report here.

Since then, we have continued to expand the portfolio of Confidential Computing products and services, including Confidential Virtual Machines (CVMs), Confidential GKE, Confidential Dataproc, and Confidential Space. These offerings give our customers more choice in their deployments. 

Currently, our Confidential Computing services are widely available across 80% of Google Cloud regions and growing, and we recently added support for Compute-Optimized C2D VMs for Confidential GKE.

Organizations are now leveraging Confidential Computing for joint data analysis and machine learning (ML) model training with trust guarantees that the data they own stays protected and across various other industries by organizations, including AstraZeneca, Bullish, HashiCorp, Matrixx Software, MonetaGo and Yellowdog.

Confidential Computing has also proven to be a helpful additional control for organizations implementing digital sovereignty strategies, providing an encryption capability, and protection for data-in-use where encryption keys are not accessible by the cloud provider.

We can’t wait to see the possibilities this technology will open up for your organization. Check out our Confidential Computing page to learn more.

With unemployment application submissions reaching record numbers over the past year, state and local agencies in the United States have faced the challenge of processing unprecedented numbers of claims per week. The digital infrastructure most agencies have in place is unable to handle this volume, resulting in constituents waiting longer, and bad actors taking advantage of vulnerable systems. The Department of Labor Inspector General estimates that $63 billion in claims distributed is either an improper payment or fraud. 

Validating claims also requires secure data sharing with other agencies for document and identity verification. Government leaders need a way to allow case adjudicators to quickly and confidently release backlogged claims, integrate with existing systems, and segment legitimate claims from potentially fraudulent ones — all within limited government budgets — securely and at scale.

Implementing a fraud detection solution on Google Cloud

States were under pressure to release payments, while also filtering out potentially fraudulent claims. SpringML and Google Cloud developed a framework to give adjudicators a reliable verification process that quickly filters potentially fraudulent claims, while processing the remaining claims so benefits reach citizens in a timely manner. SpringML and Google Cloud, applied AI-powered machine learning models to detect anomalous patterns in large datasets. Using Google Cloud tools, SpringML implemented a solution to streamline workflows, improve efficiencies, automate processes and identify potentially fraudulent claims.

SpringML used a variety of Google Cloud products to deliver a fraud detection solution, including:

• Google Cloud Storage to store and manage data
• BigQuery to store tabular data and BigQuery Machine Learning (BQML) to conduct machine learning on that data
• AutoML solutions to build predictive models and risk scoring
• Visualization tools such as Looker and Data Studio to present data and help government leaders make informed decisions.

Implementing machine learning to detect improper payments allows agencies to classify claims as “fraud” or “not fraud” based on the number of flags, as well as prioritize the most urgent claims. Deploying intelligent virtual agents to handle frequently asked questions meant that live agents could focus their time on more challenging cases. 

Even once the pandemic is behind us, there will be bad actors trying to take advantage of overwhelmed or legacy systems. We’ve identified a few best practices for agencies managing enormous case loads and looking to improve improper payment analytics: 

• Move your systems to the cloud. Many on-premises legacy systems can’t update their applications and scale to meet the volume of claims. Moving to a cloud environment enables rapid solution deployment and ingestion of large amounts of data without fear of overloading the system. The cloud scales with you–cost-effectively and securely. 
• Understand patterns in the data. The answer is always in the data — we used deep analysis to help uncover suspicious patterns in large data sets. We implemented unsupervised machine learning to learn behaviors and create configurable rules that adjust to new information that comes into the system. We can uncover patterns that are likely associated with fraud – ones that a human might have missed. 
• Use AI/ML tools to automate your existing systems and teams. These tools enable humans to work smarter and more efficiently. We automate anomaly detection and create dashboards for adjudicators to rapidly process claims. We are enabling the Wisconsin Department of Workforce Development by implementing automatic calculations and processing of recharge amounts, resulting in faster processing times and fewer human errors. Proactive fraud detection and timely calculation of recharge payment allowed DWD to ensure the benefits reached the right individuals.
• Build flexibility into your systems. We discovered that fraud patterns change over time. For instance,flags for fraud during March-May 2020 were vastly different from those we found in June-July 2020. Google Cloud tools make it easy to continually update algorithms to detect patterns and integrate external data sources.

Using Google Cloud tools, we can update digital infrastructure and incorporate machine learning best practices to help organizations efficiently process large volumes of claims and identify high probability fraudulent ones. SpringML provides consulting and implementation services and industry-specific analytics solutions that deliver high-impact business value to accelerate data-driven digital transformation. Learn more about fraud detection and how to improve improper payments analytics by watching our webinar.