Twenty years ago, organizations discovered the magic of server virtualization. Among the many benefits of virtualization, it heralded the end of the need for one physical server per application and the beginning of managing software-defined infrastructure. It helped automate building and installing systems, and it helped lay foundations for today’s Continuous Integration/Continuous Delivery mechanisms and DevOps practices. However, the adoption of virtualization was not always smooth and several hurdles stood in its path. Today’s organizations are facing similar obstacles in their journey to the cloud.

Organizations that wanted to virtualize their server environment two decades ago faced a significant challenge that was neither technical nor budgetary, but organizational, stretching across IT teams and beyond. The first team to be concerned about virtualization was the one in charge of installing the physical servers in the racks of the datacenter, managing the physical network connections and operating environmental controls, such as HVAC and power systems. These tasks were drastically reduced, if not outright eliminated.

The second team affected by this shift in technology was the one responsible for installing the operating system and running the post-install procedures. Their job almost completely disappeared as most virtual machines were automatically instantiated from templates.

As a result, successful adoption relied on the ability to smooth these organizational changes. Along the way, training the staff and making sure they acquire the new skills necessary to operate the virtualization infrastructure was critical. And usually, the affected teams eventually found that their new responsibilities were different but still essential to the organization, and perhaps even more appealing.

Virtualizations lessons for cloud transformation regulators

Discussions on virtualization didn’t end with these internal changes when organizations were operating in a regulated environment. They needed to engage closely with their regulators. Regulated organizations often considered the hypervisor solely as an additional layer of software with potential vulnerabilities.

This viewpoint raised risks (such as virtual machine escape) that didn’t exist with physical servers. Even if this is taken into account and the risks mitigated, it should have been considered in balance with all the benefits of machine virtualization, including security benefits such as asset identification, harmonization, and smoother patch management.

While the relevance and benefits of virtualization are now widely accepted, history has repeated itself as similar issues have arisen for organizations facing cloud migrations. When large cloud service providers (CSP) began supporting critical workloads for their customers, regulators increased their oversight over CSPs and issued guidance concerning cloud adoption. This happened in the European financial sector, with guidelines from the European Supervisory Authorities (European Banking Authority, European Securities and Market Authority and European Insurance and Occupational Pensions Authority,) and the coming Digital Operational Resilience Act regulation.

These regulations can help establish trust between providers and customers. As regulator decisions are vital for organizations, there is an opportunity now with cloud technology to balance the risks with the benefits and take into consideration all the ways CSPs can help organizations improve their overall security and compliance levels.

Cultural transformation can drive cloud transformation

Another similarity between cloud transformation and the journey to virtualization is the required internal transformation. When an organization decides to start its journey to the cloud, training its staff with the new technology and tools is a necessary step, but probably not the first nor the most arduous to make. A change of mindset is paramount to fully incorporate all the cloud benefits, in particular regarding security.

This new approach begins with a deep transformation at the organizational level. Organizations should avoid viewing the cloud as a datacenter because there’s so much more potential to cloud technology. This new approach and its accompanying organizational transformation is driven by two factors: The burdens that the CSP removes from the customer, and the new flexibility that software-defined infrastructure brings to the table.

The CSP directly manages several tasks and services, and their security, which means that they’re no longer a direct concern for the customer. These responsibilities include data center management, security-hardened hardware, default encryption for data at rest and data in transit, and resilient network management.

Since all the infrastructure used by the organization to manage its workloads is software defined, it brings much more flexibility to what can be done with it. Software-defined infrastructure can more easily enable security guardrails and continuous compliance that weren’t possible before.

While there’s a temptation to approach an organization’s cloud transformation as merely a “lift and shift” operation from on-premises infrastructure, the reality is that approach reduces the potential gains a cloud transformation can bring to an organization. Moving systems to the cloud accounts for a small portion of the potential impact, while updating the mindset can be truly transformational. Development, architecture and security processes need to be rebuilt; this inevitably requires a deeper transformation of the organization.

The next step is to transform the operations, to align them with the overall organization and the new defined processes, operationalize CI/CD and build DevOps practices. After that, the technological gap will be adjusted, moving the new tools and teaching teams how to best use them.

The important point here is the order of the transformation to manage: organization, operations and technologies (O-O-T) and not the other way round (T-O-O). Of course, these three steps should not be considered completely sequential, as technology plays an important role in the organization and the operations. However, it is essential to support the development of personnel and frameworks to take full advantage of the transformation opportunity, rather than primarily focusing on the tooling.

In addition, at Google Cloud, we believe in a shared fate that goes far beyond the usual shared responsibility model. We think it is part of our duty to help our customers to achieve their goals in their scope of responsibility. We prepare secure landing zones and guide the customers, we bring transparency to security controls and help them with cyber-insurance.

Our shared objective with our customers is to continuously improve security. Our Google Cybersecurity Action Team initiative helps us to be engaged alongside our customers, provide them with necessary guidance, support them in their security and compliance strategies, and assist their organizational and operational transformation in their cloud journey.

• Listen to a related podcast “How to Apply Lessons from Virtualization Transition to Make Cloud Transformation Better”
• Listen to “Preparing for Cloud Migrations from a CISO Perspective, Part 1”
• Read “The journey to the cloud mitigates enterprise risk”
• Also, review “Megatrends drive cloud adoption—and improve security for all”
• Visit Google Cybersecurity Action Team website

Part of being a security leader whose organization is taking on a digital transformation is preparing for hard questions – and complex answers – on how to implement a transformation strategy.

In our previous CISO Survival Guide blog, we discussed how financial services organizations can more securely move to the cloud. We examined how to organize and think about the digital transformation challenges facing the highly-regulated financial services industry, including the benefits of the Organization, Operation, and Technology (OOT) approach, as well as embracing new processes like continuous delivery and required cultural shifts.

As part of Google Cloud’s commitment to shared fate, today we offer tips on how to ask the right questions that can help create the conversations that lead to better transformation outcomes for your organization. While there often is more than one right answer, a thoughtful, methodical approach to asking targeted questions and maintaining an open mind about the answers you hear back can help achieve your desired result. These questions are designed to help you figure out where to start and where to end your organization’s security transformation. By asking the following questions, CISOs and business leaders can develop a constructive, focused dialogue which can help determine the proper balance between implementing security controls and fine-tuning the risk tolerance set by the executive management and the board of directors.

To start the conversation, begin by asking:

• What defines our organization’s culture?
• How can we best integrate the culture with our security goals?

CISOs should ask business leaders:

• What makes a successful transformation?
• What are the key goals of the transformation?
• What data is (most) valuable?
• What data can be retired, reclassified, or migrated?
• What losses can we afford to take and still function?
• What is the real risk that the organization is willing to accept?

Business leaders should ask CISOs and the security team:

• What are the best practices for protecting our valuable data?
• What is the business impact of implementing those controls?
• What are the top threats that we need to address?

CISOs and business leaders should ask:

• Which threats are no longer as important?
• Where could we potentially use spending for more cost-effective controls such as firewalls and antivirus software?
• What benefits do we get from refactoring our applications?
• Are we really transforming, or lifting and shifting?
• How should we perform identity and access management to meet our business objectives?
• What are the core controls needed to ensure enterprise-level performance for the first workloads?

CISOs and risk teams should ask:

• How can we use the restructuring of an existing body of code to streamline security functions?
• How should we monitor our security posture to ensure we are aligned with our risk appetite?

Business and technical teams should ask:

• What’s our backup plan?
• What do we do if that fails?

Practical advice and the realities of operational transformation

Some organizations have been working in the cloud for more than a decade and have already addressed many operational procedures, sometimes with painful lessons learned along the way. If you’ve been operating in the cloud securely for that long, we recognize that there’s a lot to be gained from understanding your approaches to culture, operational expertise, and technology.

However, there are still many organizations that have not thought through how they will operate in a cloud environment until it’s almost ready – and at that point, it might be too late. If you can’t detail how a cloud environment will operate before its launch, how will you know who should be responsible for maintaining it?

Who are the critical stakeholders, along with those responsible for engineering and maintaining specific systems, who should be identified at the start of the transformation? There are likely several groups of stakeholders, such as those aligned with operations for transformation, and those focused on control design for cloud aligned with operations.

If you don’t have the operators involved in the design phase, you’re destined to create clever security controls with very little practical value because those tasked with day-to-day maintenance most likely won’t have the expertise or training to effectively operate these controls.

This is complicated by the fact that many organizations are struggling to recruit and retain resources with the right skills to operate in the cloud. We believe that training current employees to learn new cloud skills, and giving them the time away from other responsibilities, can help build skilled, diverse cloud security teams.

If your organization continually experiences high turnover in security leadership and skilled staff, it’s up to you to navigate your culture to ensure greater consistency. You can, of course, choose to supplement internal knowledge with trusted partners – however, that’s an expensive strategy for ongoing operational cost.

We met recently with a security organization that turns over skilled staff and leadership every two to three years. This rate of churn results in a continual resetting of security goals. This particular team joked that it’s like “Groundhog Day” as they constantly re-evaluate their best security approaches yet make no meaningful progress. This is not a model to emulate.

Many security controls fail not because they are improperly engineered, but because the people who use them – your security team – are improperly trained and insufficiently motivated. This is especially true for teams with high turnover rates and other organizational misalignments. A security control that blocks 100% of attacks might be engineered correctly, but if you can’t efficiently operate it, the effectiveness of the control will plummet to zero over time. Worse, it then becomes a liability because you incorrectly assume you have a functioning control.

In our next blog, we will highlight several proven approaches that we believe can help guide your security team through your organization’s digital transformation.

To learn more now, check out:

• Previous blog
• Podcast: CISO walks into the cloud: Frustrations, successes, lessons… and does the risk change?
• Report: CISO’s Guide to Cloud Security Transformation

Whether driven by government policy, industry regulation, or geo-political considerations, the evolution of cloud computing has led organizations to want even more control over their data and more transparency from their cloud services. At Google Cloud, one of the best tools for achieving that level of control and transparency is a bit of technological magic we call Cloud External Key Manager (EKM).

Cloud EKM can help you protect your cloud data at rest with encryption keys which are stored and managed in a third-party key management system that’s outside Google Cloud’s infrastructure, and ultimately outside Google’s control. This can help you achieve full separation between your encryption keys and your data stored in the cloud. Cloud EKM works with symmetric and asymmetric encryption keys, and offers organization policies that allow for fine-grained control over what types of keys are used. Via Key Access Justification (KAJ) it also offers a way for clients to control each key use.

At their core, many cloud security and cloud computing discussions are about the kinds of trust that Cloud EKM specifically and encryption more broadly can help create. While the concept of digital trust is much bigger than cybersecurity and its tripartite components of security, privacy, and compliance, one of the most crucial themes of cloud computing is the cloud trust paradox. In order to trust the cloud more, you must be able to trust it less, and external control of keys and their use can help reduce concerns over unauthorized access to sensitive data.

How it works

From our Cloud EKM documentation, you can use keys that you manage within a supported external key management partner to protect data within Google Cloud. You can protect data at rest in services that support CMEK, or by calling the Cloud Key Management Service API directly.

Cloud EKM provides several benefits:

• Key provenance: You control the location and distribution of your externally-managed keys. Externally-managed keys are never cached or stored within Google Cloud. Google cannot see them. Instead, Cloud EKM communicates directly with the external key management device for each request.
• Access control: You manage access to your externally-managed keys. Before you can use an externally-managed key to encrypt or decrypt data in Google Cloud, you must grant the Google Cloud project access to use the key. You can revoke this access at any time.
• Centralized key management: You can manage your keys and access policies from a single location and user interface, whether the data they protect resides in the cloud or on your premises. The system that managed the keys is entirely outside Google control.

In all cases, the key resides on the external system, and is never sent to Google.

Here’s how it works:

1. Create or use an existing key in a supported external key management partner system. This key has a unique URI.
2. Grant your Google Cloud project access to use the key, in the external key management partner system.
3. Create a Cloud EKM key in your Google Cloud project, using the URI for the externally-managed key.

The Cloud EKM key and the external key management partner key work together to protect your data. The external key is never exposed to Google and cannot be accessed by Google employees. Furthermore, Cloud EKM can be combined with Key Access Justifications (KAJ) to establish cryptographic control over data access.

KAJ with Cloud EKM can give customers the ability to deny Google Cloud administrators access to their data at rest for any reason, even in situations typically exempted from customer control, such as outages or responses to third-party data requests. KAJ does this by providing customers a clear reason why data is being decrypted, which they can use to programmatically decide whether to permit decryption and thus allow access to their data.

Previously, we’ve discussed three patterns where keeping the keys off the cloud may in fact be truly necessary or outweighs the benefits of cloud-based key management. Here’s a brief summary of those three scenarios where Cloud EKM can help solve these Hold Your Own Key dilemmas.

Scenario 1: The last data to go to the cloud

As organizations complete their digital transformations by migrating data processing workloads to the cloud, there is often a pool of data that can not be moved to the cloud. Perhaps it’s the most sensitive data, the most regulated data, or the data with the toughest internal security control requirements.

Finance, healthcare, manufacturing and other heavily-regulated organizations face myriad risk, compliance, and policy reasons that may make it challenging to send some of their data to a public cloud provider. However, the organization may be willing to migrate this data set to the cloud as long as it is encrypted and they have sole possession of the encryption keys.

Scenario 2: Regional regulations and concerns

Regional requirements are playing a larger role in how organizations migrate to and operate workloads in the public cloud.

Some organizations are already facing situations where they are based in one country and want to use a cloud provider based in a different country, but they aren’t comfortable with or legally allowed to give the provider access to encryption keys for their stored data. Here the situations are more varied, and can include an organization’s desire to stay ahead of evolving regulatory demands or industry-specific mandates.

Ultimately, this scenario allows organizations to utilize Google Cloud while keeping their encryption keys in the location of their choice, and under their physical and administrative control.

Scenario 3: Centralized encryption key control

The focus here is on operational efficiency. Keeping all the keys within one system to cover multiple cloud and on-premise environments can help reduce overhead and attack surface, thus helping to improve security. As Gartner researchers concluded in their report, “Develop an Enterprisewide Encryption Key Management Strategy or Lose the Data1,” organizations are motivated to reduce the number of key management tools.

“By minimizing the number of third-party encryption solutions being deployed within an environment, organizations can focus on establishing a cryptographic center of excellence,” Gartner researchers said

Given that few organizations are 100% cloud-based today for workloads that require encryption, keeping keys on-prem can streamline key management. Centralizing key management can give the cloud user a central location to enforce policies around access to keys and access to data-at-rest, while a single set of keys can help reduce management complexity. A properly implemented system with adequate security and redundancy outweighs the need to have multiple systems.

Do I need Cloud EKM?

Whether protecting highly sensitive data, retaining key control to address geopolitical and regional concerns, or supporting hybrid and multi-cloud architectures, Cloud EKM is best suited for those Google Cloud customers who must keep their encryption keys off of the cloud and always under their full control.

To learn more about Cloud EKM, please review these resources:

• Our research explaining why Google Cloud users can benefit from Cloud EKM
• The most recent updates to Cloud EKM
• Take a deeper dive into the cloud trust paradox

1. Gartner, Develop an Enterprisewide Encryption Key Management Strategy or Lose the Data, David Mahdi, Brian Lowans, March 2022.

In fast-moving organizations, it’s not uncommon for cloud resources, including entire projects, to occasionally be forgotten about. Not only such unattended resources can be difficult to identify, but they also tend to create a lot of headaches for product teams down the road, including unnecessary waste and security risks. 

To help you prune your idle cloud resources, we’re excited to introduce Unattended Project Recommender. It’s a new feature of Active Assist that provides you with a one-stop shop for discovering, reclaiming, and shutting down unattended projects. With actionable and automatic recommendations, you no longer have to worry about wasting money or mitigating security risks presented by your idle resources. Unattended Project Recommender uses machine learning to identify, with a high degree of confidence, projects that are likely abandoned based on API and networking activity, billing, usage of cloud services, and other signals. This feature is available via the Recommender API today, making it easy for you to integrate with your company’s existing workflow management and communication tools, or export results to a BigQuery table for custom analysis.

Thousands of projects can be unattended in large organizations, presenting major security risks

Your cloud projects can go abandoned or unattended for a number of reasons — ranging from a test environment that’s no longer needed, to project cancellation, to project owner switching jobs, and more. Not only can such projects contribute to your cloud bill (waste) but they may contain security issues such as open firewalls or privileged service account keys that attackers can exploit to get a hold of your cloud resources for cryptocurrency mining or, worse, compromise your company’s sensitive data. These security risks tend to grow over time because the latest best practices and patches are usually not applied to unattended projects. 

We experience this issue here at Google, too… In fact, it has been on Google’s internal security team’s radar for some time now, so we joined forces and looked into this problem together, starting with our very own “google.com” organization cloud projects. We quickly found some projects that were unattended, but remediating this issue was easier said than done due to challenges in several areas:

• Detection: With lots of signals available to you via sources like Cloud Monitoring, what are the right ones you should look at (e.g. API, networking, user activity)? How can you tell the difference between an unattended project and a project that has a low level of activity by design (e.g. a “shell” project that holds an auth token)?
• Remediation: Once you have identified a project that seems abandoned, how do you go about ensuring that it’s indeed an unattended project? How do you reduce the risk of deleting something that might be essential to a production workload, causing irreversible data loss? How do you solve this at the scale of your entire organization, beyond a one-time cleanup? 

Over the course of 2021 we built and tested a Google-internal prototype first, cleaning up many of our internal unattended projects, and then worked with a number of Google Cloud customers to build and tune this feature based on real-life data (thank you to all of our early adopters for working with us and your generous feedback that helped us shape this feature!) It was not uncommon for us to come across organizations with thousands of unattended projects, and we’re very excited to bring Unattended Project Recommender to all customers, in public preview.

Discovering and acting on unattended project recommendations

Unattended Project Recommender analyzes usage activity across all projects under your organization, including the following data:

• API activity (e.g. service accounts with authentication activity, API calls consumed)
• Networking activity (ingress and egress)
• Billing activity (e.g. services with billable usage)
• User activity (e.g. active project owners)
• Cloud services usage (e.g. number of active VMs, BigQuery jobs, storage requests)

Based on these signals, it can generate recommendations to clean up projects that have low usage activity (where “low usage” is defined using a machine learning model that ranks projects in your organization by level of usage), or recommendations to reclaim projects that have high usage activity but no active project owners. Here’s what an example post-processed summary list of recommendations can look like for the “foobar” organization that has 3 projects:

  Project ID: demo-project-307815
Recommendation: CLEANUP_PROJECT

Project ID: new-project
Recommendation: N/A

Project ID: bobs-playground-project
Recommendation: RECLAIM_PROJECT

In addition to the recommendations, you can also examine the underlying project activity insights that the recommendations are based upon. The insights provide additional information that can be useful for integration with your organization’s existing workflows and automation (e.g. send an auto-generated email or chat message to project owners based on the list provided by the owners field). Here’s an example insight payload:

  content:
  activeAppengineInstanceDailyCount: 0
  activeCloudsqlInstanceDailyCount: 0
  activeGceInstanceDailyCount: 3
  activeServiceAccountDailyCount: 1
  apiClientDailyCount: 18922           // Daily average API calls produced
  bigqueryInflightJobDailyCount: 0
  bigqueryInflightQueryDailyCount: 0
  bigqueryStorageDailyBytes: 0
  bigqueryTableDailyCount: 0
  consumedApiDailyCount: 0             // Daily average API calls consumed
  datastoreApiDailyCount: 0
  gcsObjectDailyCount: 11
  gcsRequestDailyCount: 0
  gcsStorageDailyBytes: 2663548
  hasActiveOauthTokens: false          // OAuth tokens used in the last 180 days
  hasBillingAccount: true
  numActiveUserOwners: 1
  owners:                              // List of project owners
  - activeOnProject: false
    member: user:user1@example.com
  - activeOnProject: true
    member: user:user2@example.com
  serviceWithBillableUsage:
  – Cloud Storage
  - Compute Engine
  vpcEgressDailyBytes: 264456938       // Daily average VPC egress bytes
  vpcIngressDailyBytes: 392435047      // Daily average VPC ingress bytes
  usagePercentile: 20                  // Level of usage relative to other projects

GCP projects are used in many different ways and for many different purposes. In case you get a recommendation to delete a project that’s being used in a way that’s out of the scope for this feature, you can dismiss the recommendation and it will stop showing up for the given project. 

Restoring deleted projects

When you choose to shut down a project using the projects.delete() method, it gets marked for deletion. After a project is marked for deletion, it becomes unusable, all resources within that project are shut down, and a 30-day wait period for the project and all of its data to get fully deleted begins.

In case a useful project is accidentally shut down, you have the option to restore the project within that 30-day wait period. Since restoring allows you to recover most but not necessarily all of your project data and resources, we recommend carefully examining the utilization insights associated with a project and considering any additional utilization signals that may not be captured by the Unattended Project Recommender before taking the cleanup action.

Early customer success stories

A number of enterprise customers are already using Unattended Project Recommender to keep their organizations clean of unattended projects and resources.

Decathlon, a French sporting goods retailer, is excited for the insight Unattended Project Recommender will bring to their environment, and are already deploying it as a part of their latest cloud security initiatives.

“After a thorough test of this feature and the validation of our CISO, we ended up deleting our first 775 projects, and no one complained! A great help to improve our security. The next step for us will be to operationalize it at scale, and implement a company wide policy for unattended resource management.” —Adeline Villette, Cloud Security Officer

For Veolia, one of the world’s largest water, waste and energy management companies, not only does this feature reduce security risks and waste, but also helps drive cultural shift and alignment with its ecological transformation strategy.

“This feature allows us to reduce our costs and security debt on assets that are no longer in use, and is also fully in line with Veolia’s philosophy of limiting its carbon footprint. After having tested Unattended Project Recommender on more than 3,000 projects throughout our organization, we are looking to bring it as proactive alerts to our project owners at scale.”—Thomas Meriadec, Product Manager

Box, a secure cloud content management provider, views it as a foundation for building a repeatable process to remediate unused resources.

“Unattended Project Recommender is a great fit for us. It gives us a unified view of project usage across our entire organization and enables us to address security risks of legacy projects in a systematic and organized manner, ensuring an even safer environment.” —Matt Bowes, Staff Security Engineer

Getting started with the Unattended Project Recommender

To help you get started, we’ve prepared a Cloud Shell tutorial (source code) that you can use to find unattended project recommendations within your own Projects/Folders/Organization. Click this button to clone the tutorial from GitHub and run in your Cloud Shell environment:

As you can see, listing recommendations for your projects only takes a few clicks with the tutorial (special thanks to Lanre Ogunmola, Security & Compliance Specialist, for making this look so easy)! For additional detail on using the gcloud CLI or API to discover unattended project recommendations, please refer to the documentation page.

You can also automatically export all recommendations from your Organization to BigQuery and then investigate the recommendations with DataStudio or Looker, or use Connected Sheets that let you use Google Workspace Sheets to interact with the data stored in BigQuery without having to write SQL queries.

As with any other Recommender, you can choose to opt out of data processing at any time by disabling the appropriate data groups in the Transparency & control tab under Privacy & Security settings.

We hope that you can leverage Unattended Project Recommender to improve your cloud security posture and reduce cost, and can’t wait to hear your feedback and thoughts about this feature! Please feel free to reach us at active-assist-feedback@google.com and we also invite you to sign up for our Active Assist Trusted Tester Group if you would like to get early access to the newest features as they are developed.