Our quarterly Security Talks series brings together experts from Google Cloud Security teams and the industry to share information on our latest security products, innovations and best practices. Below is an introduction to Mandiant’s Security Talks presentation on Dec. 7, Mandiant Tales from the Front Lines: Activate Cyber Defenses Against Supply Chain Compromise.

The discovery in December 2020 of the SolarWinds supply chain compromise, a global computer intrusion campaign, caused organizations and security leaders to completely reassess their Cyber Threat Profiles, risk assessments, and defensive postures. It was a rare, watershed event that savvy security leaders and IT decision makers have been using to improve how security teams protect systems, networks, and data across industries and sectors, especially critical infrastructure. Two years later, we’re looking back at the security incident, what we’ve learned, and how it continues to drive change across security teams and organizations.

The SolarWinds compromise leveraged access to the software maker’s Orion business software in order to distribute a malicious update containing malware. Mandiant has since attributed this operation to APT29, a Russia-based espionage group assessed to be sponsored by the Russian Foreign Intelligence Service (SVR). This operation afforded APT29 with unparalleled access to an estimated 18,000 global organizations for follow-on targeting based on the attacker’s discretion and collection priorities.

While the security industry has yet to see anything that rivals the scale and scope of the SolarWinds compromise since it was discovered, a recent Google Cloud research report on software supply chain security found that there’s been a sharp increase in software supply chain attacks across nearly every sector.

Mandiant analysis identified supply chain compromise as the second-most prevalent initial infection vector in 2021. When the initial infection vector was identified, supply chain compromise accounted for 17% of intrusions in 2021 compared to less than 1% of intrusions in 2020. Further, 86% of supply chain compromise intrusions in 2021 were related to the SolarWinds breach.

Since 2020, Mandiant has also observed an increase in financially-motivated threat actors targeting the software supply chain. These actors compromised popular software packages and even mobile applications in order to deploy ransomware, cryptocurrency miners, and banking trojans. In one case, malicious code was inserted into a popular package, prompting the U.S. Cybersecurity and Infrastructure Agency (CISA) to issue an alert about the compromise.

Mitigation recommendations

The discovery of the SolarWinds incident resulted in increased global attention around supply chain compromise threats. Government authorities and technology organizations have responded with several initiatives, legislation, and studies seeking to reduce software supply chain and software dependency risk. Although these initiatives may make it more difficult for threat actors to conduct these operations, organizations should also take steps to mitigate supply chain threats.

Mandiant suggests that organizations contemplate applying multiple layers of security policies, plans, and solutions to maximize opportunities to provide early threat detection and prevent an anomaly or compromise stemming from the supply chain. This guidance is aligned with best practices Google Cloud has recommended for organizations to improve their software supply chain security.

Mandiant has also provided the following additional recommendations for organizations to consider more generally:

• Establish a vendor vetting process to evaluate vendor security practices before deploying hardware or software.
• Create a change control process and board for all enterprise hardware and software changes, which could include a centralized IT or IT security managed process for downloading, testing, and pushing updates to users.
• Use an advanced endpoint security solution, such as an endpoint detection and response (EDR), to detect malicious behavior when a tainted software package is downloaded and executed.
• Ensure proper logging and monitoring is in place between the software, hardware, and the internet.
• Implement the software and hardware using network segmentation with minimal access to the internet.
• Enact employee training programs and policies to encourage security best practices, including security audits of code in development.

Next steps

You can watch the Google Cloud Security Talks presentation on SolarWinds and the current state of supply chain threats here. You can learn more about Google Cloud’s solutions to help improve software supply chain security here, or for more general inquiries, please contact us or reach out to your sales representative to learn how we can assist.

Editor’s note: Today we are hearing from Jono MacDougall , Principal Software Engineer at Ravelin. Ravelin delivers market-leading online fraud detection and payment acceptance solutions for online retailers. To help us meet the scaling, throughput, and latency demands of our growing roster of large-scale clients, we migrated to Google Cloud and its suite of managed services, including Cloud Bigtable, the scalable NoSQL database for large workloads.

As a fraud detection company for online retailers, each new client brings new data that must be kept in a secure manner and new financial transactions to analyze. This means our data infrastructure must be highly scalable and constantly maintain low latency. Our goal is to bring these new organizations on quickly without interrupting their business. We help our clients with checkout flows, so we need latencies that won’t interrupt that process—a critical concern in the booming online retail sector. 

We like Cloud Bigtable because it can quickly and securely ingest and process a high volume of data. Our software accesses data in Bigtable every time it makes a fraud decision. When a client’s customer places an order, we need to process their full history and as much data as possible about that customer in order to detect fraud, all while keeping their data secure. Bigtable excels at accessing and processing that data in a short time window. With a customer key, we can quickly access data, bring it into our feature extraction process, and generate features for our models and rules. The data stays encrypted at rest in Bigtable, which keeps us and our customers safe.

Bigtable also lets us present customer profiles in our dashboard to our client, so that if we make a fraud decision, our clients can confirm the fraud using the same data source we use.

We have configured our bigtable clusters to only be accessible within our private network and have restricted our pods access to it using targeted service accounts. This way the majority of our code does not have access to bigtable and only the bits that do the reading and writing have those privileges.

We also use Bigtable for debugging, logging, and tracing, because we have spare capacity and it’s a fast, convenient location. 

We conduct load testings against Bigtable.  We started at a low rate of ~10 Bigtable requests per second and we peaked at ~167000 mixed read and write requests per second  at absolute peak. The only intervention that was done to achieve this was pressing a single button to increase the number of nodes in the database. No other changes were made.

In terms of real traffic to our production system, we have seen ~22,000 req/s (combined read/write) on Bigtable in our live environment as a peak within the last 6 weeks.

Migrating seamlessly to Google Cloud 

Like many startups, we started with Postgres, since it was easy and it was what we knew, but we quickly realized that scaling would be a challenge, and we didn’t want to manage enormous Postgres instances. We looked for a kind of key value store, because we weren’t doing crazy JOINS or complex WHERE clauses. We wanted to provide a customer ID and get everything we knew about it, and that’s where key value really shines.  

I used Cassandra at a previous company, but we had to hire several people just for that chore. At Ravelin we wanted to move to managed services and save ourselves that headache. We were already heavy users and fans of BigQuery, Google Cloud’s serverless, scalable data warehouse, and we also wanted to start using Kubernetes. This was five years ago, and though quite a few providers offer Kubernetes services now, we still see Google Cloud at the top of that stack with Google Kubernetes Engine (GKE). We also like Bigtable’s versioning capability that helped with a use case involving upserts. All of these features helped us choose Bigtable.

Migrations can be intimidating, especially in retail where downtime isn’t an option. We were migrating not just from Postgres to Bigtable, but also from AWS to Google Cloud. To prepare, we ran in AWS like always, but at the same time we set up a queue at our API level to mirror every request over to Google Cloud. We looked at those requests to see if any were failing, and confirmed if the results and response times were the same as in AWS. We did that for a month, fine tuning along the way. 

Then we took the big step and flipped a config flag and it was 100% over to Google Cloud. At the exact same time, we flipped the queue over to AWS so that we could still send traffic into our legacy environment. That way, if anything went wrong, we could fail back without missing data. We ran like that for about a month, and we never had to fail back. In the end, we pulled off a seamless, issue-free online migration to Google Cloud.

Flexing Bigtable’s features

For our database structure, we originally had everything spread across rows, and we’d use a hash of a customer ID as a prefix. Then we could scan each record of history, such as orders or transactions. But eventually we got customers that were too big, where the scanning wasn’t fast enough. So we switched and put all of the customer data into one row and the history into columns. Then each cell was a different record, order, payment method, or transaction. Now, we can quickly look up the one row and get all the necessary details of that customer. Some of our clients send us test customers who place an order, say, every minute, and that quickly becomes problematic if you want to pull out enormous amounts of data without any limits on your row size. The garbage collection feature makes it easy to clean up big customers.  

We also use Bigtable replication to increase reliability, atomicity, and consistency. We need strong consistency guarantees within the context of a single request to our API since we make multiple bigtable requests within that scope. So within a request we always hit the same replica of Bigtable and if we have a failure, we retry the whole request. That allows us to make use of the replica and some of the consistency guarantees, a nice little trade-off where we can choose where we want our consistency to live.https://www.youtube.com/embed/0-eH5u7rrQQ?enablejsapi=1&

We also use BigQuery with Bigtable for training on customer records or queries with complicated WHERE clauses. We put the data in Bigtable, and also asynchronously in BigQuery using streaming inserts, which allows our data scientists to query it in every way you can imagine, build models, and investigate patterns and not worry about query engine limitations. Since our Bigtable production cluster is completely separate, doing a query on BigQuery has no impact on our response times. When we were on Postgres many years ago, it was used for both analysis and real time traffic and it was not the optimal solution for us. We also use Elasticsearch for powering text searches for our dashboard.

If you’re using Bigtable, we recommend three features:

• Key visualizer. If we get latency or errors coming back from Bigtable, we look at the key visualizer first. We may have a hotkey or a wide row, and the visualizer will alert us and provide the exact key range where the key lives, or the row in question. Then we can go in and fix it at that level. It’s useful to know how your data is hitting Bigtable and if you’re using any anti-patterns or if your clients have changed their traffic pattern that exacerbated some issue.
• Garbage collection. We can prevent big row issues by putting size limits in place with the garbage collection policies.  
• Cell versioning. Bigtable has a 3d array, with rows, columns, and cells, which are all the different versions. You can make use of the versioning to get history of a particular value or to build a time series within one row. Getting a single row is very fast in Bigtable so as long as you can keep the data volume in check for that row, making use of cell versions is a very powerful and fast option. There are patterns in the docs that are quite useful and not immediately obvious. For example, one trick is to reverse your timestamps (MAXINT64 – now) so instead of the latest version, you can get the oldest version effectively reversing the cell version sorting if you need it.

Google Cloud and Bigtable help us meet the low-latency demands of the growing online retail sector, with speed and easy integration with other Google Cloud services like BigQuery. With their managed services, we freed up time to focus on innovations and meet the needs of bigger and bigger customers. 

Learn more about Ravelin and Bigtable, and check out our recent blog, How BIG is Cloud Bigtable?

While cloud security skeptics might believe that data in the cloud is just one access configuration mistake away from a breach, the reality is that a well-designed set of defense in depth controls can help minimize the risk of configuration mistakes and other security issues. Our Virtual Private Cloud (VPC) Service Controls can play a vital role in creating an additional layer of security while also making it easier to manage your data in a way that most cloud services can’t do today.

Organizations across industries and business models use cloud services for activities such as processing their data, performing analytics, and deploying systems. VPC Service Controls can empower an organization when deciding how users and data can cross the perimeter of the supported cloud services, if at all. While VPC Service Controls are designed to help stop attackers, they can also enable contextual trusted data sharing (similar to how Zero Trust allows contextual access).

What are VPC Service Controls

VPC Service Controls help administrators define a security perimeter around Google-managed services, which can control communication to and between those services. The Service Controls isolate your Google Cloud resources from unauthorized networks, including the internet. For example, this can help you keep a clear separation between services that are allowed to run in production and services that are not.

VPC Service Controls can help you prevent mistakes that lead to costly data breaches because they control access to your data at a granular level. They add context-aware access controls on these services, and can help you achieve your organization’s Zero Trust access goals.

Like wearing two layers of clothing made from different fabrics to protect you from winter weather, VPC Service Controls may appear similar to Identity and Access Management (IAM) but they come from a different approach to implementing security. IAM enables granular identity based access control; VPC Service Controls create a security perimeter that protects your cloud resources and sets up private connectivity to Google Cloud’s APIs and services. While it’s recommended to use both, VPC Service Controls have an added bonus: They can support blocking data theft during a breach.

The additional layer of security that VPC Service Controls offer customers is challenging to achieve with on-premise systems or even with other cloud providers. You can think of it as an firewall for APIs that also adds a logical security control around three paths that data can take:

• From the public internet to your resources
• Inside your VPC and the cloud service perimeter
• For service-to-service communication (for example, denying access to someone who wants to load data to BigQuery or exfiltrate data from a BigQuery instance.)

How VPC Service Controls can help stop attackers

VPC Service Controls are used to enforce a security perimeter. They can help isolate resources of multi-tenant Google Cloud services, which can help reduce the risk of data exfiltration or a data breach.

For example, a bank that migrated financial data processing to Google Cloud can use VPC Service Controls to isolate their processing pipeline from public access (or any unauthorized access) by defining a trusted service perimeter.

How VPC Service Controls can enable trusted sharing

VPC Service Controls are used to securely share data across service perimeters with full control over what resource can connect to other resources, or outside the perimeter. This can help mitigate data exfiltration risks stemming from stolen identities, IAM policy misconfigurations, some insider threats, and compromised virtual machines.

Returning to our bank example, that same bank using VPC Service Controls may securely share or access data across Service Perimeters and Organizations. They may allow access to specific partners and for specific operations.

How VPC Service Controls support Zero Trust access

VPC Service Controls deliver Zero Trust access to multi-tenant Google Cloud services. Clients can restrict access to authorized IPs, client context, user identity, and device parameters while connecting to multi-tenant services from the internet and other services.

A bank can use moving its services to the public cloud as an opportunity to abandon outdated access management approaches and adopt Zero Trust access. VPC Service Controls let them create granular access control policies in Access Context Manager based on attributes such as user location and IP address. For example, it would allow an analyst to only access Big Query from a corporate device on the corporate network during business hours. These policies can help ensure the appropriate security controls are in place when granting access to cloud resources from the Internet.

Next steps with VPC

Check out these pages to learn more about VPC Service Controls for your sensitive cloud deployments, especially for regulated workloads. This blog is the third in our Best Kept Security Secrets series, which includes how to tap into the power of Organization Policy Service and how Cloud EKM can help resolve the cloud trust paradox.

In cybersecurity, speed matters. Whether a security analyst is trying to understand the details of an alert that was triggered by an indicator of compromise (IoC), or find additional context for a suspicious asset, speed is often the critical factor that will help thwart a cyberattack before threat actors are able to inflict damage.

With speed in mind, we are pleased to announce the general availability of our new investigative experience in Chronicle Security Operations. We are continuing to deliver on our mission to bring the power of Google to security operations and are raising the bar for search and the investigative experience in the SOC.

With this release, SecOps teams will be able to harness Chronicle’s lightning-fast search across any form of structured data. Additionally, the new investigation experience can provide greater flexibility to pivot and drill-down when conducting complex, open-ended threat investigations and surface insights quickly and easily.

Our Unified Data Model (UDM) schema, with its built-in flexibility that can effectively handle a wide variety of security related events, is at the heart of Chronicle’s powerful search. We have scaled this capability by optimizing query responses across structured data. Additionally, analysts can investigate large datasets and build complex queries with a new and intuitive user experience. With user personalization enhancements, analysts can quickly access saved searches and top queries in their environment to improve routine SOC workflows.

With the new investigative experience, security teams can:

• Drive faster threat understanding with an interactive event results timeline that helps eliminate unnecessary long wait times by streaming results as they are processed to quickly begin threat analysis
• Use enhanced context and operationalize relevant data for threat analysis with one-click filter-to-query conversion
• Personalize the analyst experience with saved search and search history functions for quicker analyst knowledge recall
• Power threat investigation and hunting with a new, improved, highly performant UDM search

Let’s look at an example of how to use the reimagined investigative experience and our new broader, faster search.

In our use-case, a security analyst is investigating a curated detection alert for potentially suspicious behavior on a Windows environment. Additionally, there is a low prevalence domain from host “win-dc-01” with ip “10.166.0.3”. To investigate further, let’s open the UDM Search page and construct a query containing the host and IP information along with destination information of the domain the host was contacting (edge.microsoft.com). Over 70,000 events stream into the investigation interface providing the analyst an immediate picture of data surrounding their alert.

The new interactive events timeline can provide a clear picture of event trends over time with key statistical data which can be easily filtered. Additionally, with the new quick filters, the analyst is able to easily filter out hosts that are known-good to get pertinent information about the domain.

Analyzing value aggregations present in the filter panel, automatically generated by Chronicle, enables analysts to domain values of highest interest, and quickly determine that approximately 800 events were present in the last week that related to edge.microsoft.com URL. Since Microsoft Edge is disallowed in the organization, there should not be any outbound traffic to this type of destination.

As a final step before orchestrating a response in Chronicle SOAR’s case management, the analyst can save their search to quickly recall the steps they took in the future for related investigations.

We have already seen our customers use these new capabilities in preview to build new use cases, accelerate existing threat hunting workflows, and drive faster threat response. We will continue our mission to bring Google speed, scale, and intelligence to the investigative experience, expediting “time to aha” for security analysts, and driving better, faster responses.

Ready to put Chronicle to work in your Security Operations Center? Contact Google Cloud sales or your customer success CSM team. You can also learn more about all these new capabilities in Google Chronicle in our product documentation.