“Ninety percent of all data today was created in the last two years—that’s 2.5 quintillion bytes of data per day,” according to business data analytics company Domo. That would be a mind-bending statistic, except that it’s already five years old.

As data usage has undergone drastic expansion and changes in the past five years, so have your business needs for data. Technology such as cloud computing and AI have changed how we use data, derive value from data, and glean insights from data. Your organization is no longer just crunching and re-crunching the same data sets. Data moves, shifts, and replicates, as you mingle data sets and gain new value in the process, as we say in our Data Cloud story. All the while, your data resides in—and is being created in—new places.

Data lives in a myriad of locations now and requires access from different locations and mediums, yet many of today’s security models are not geared towards this. In short, your data has fallen out of love with your security model, but attackers have not. So, how do we realign data and security so they are once again in a healthy relationship?

Google Cloud, as a leader in cloud data management and cloud security, is positioned uniquely to define and lead this effort. We’ve identified some challenges around the classic approach to data security and the changes triggered by the near-ubiquity of the cloud. The case is compelling for adopting a modern approach to data security. We contend that the optimal way forward is with autonomic data security.

A relatively new concept, autonomic data security is security that’s been integrated with data throughout its lifecycle. It can make things easier on users by freeing them from defining and redefining myriad rules about who can do what, when, where. It’s an approach that keeps pace with constantly evolving cyberthreats and business changes.

Autonomic data security can help you keep your IT assets more secure and can make your business and IT processes speedier. For example, data sharing with partners and data access decisions simultaneously becomes faster and more secure. This may sound like magic, but in fact relies on a constant willingness to change and adapt to both business changes and threat evolution.

Taking the precepts, concepts, and forward-looking solutions presented in this paper into consideration, we strongly believe that now is the right time to assess where you and your business are when it comes to data security. Cloud also brings an incredible scale of computing. Where gigabytes once roamed, petabytes are now common. This means that many data security approaches, especially the manual ones, are no longer practical.

To prepare for the future of data security, we recommend you challenge your current model and assumptions and ask critical questions, evaluate where you are, and then start to put a plan in place of how you could start incorporating the autonomic data security pillars into your data security model.

There are two sets of questions organizations need to discover the answers to as they start this journey. The first set of questions will help you identify the nature and status of your data, and inform the answers to the second set.

• What data do I have?
• Who owns it?
• Is it sensitive?
• How is it used?
• What is the value in storing the data?

The second set focuses on higher-level problems:

• What is my current approach to data security?
• Where does it fail to support the business and counter the threats?
• Does it support my business?
• Should I consider making a change? And if yes, in what direction?

The path to improved data security starts by asking the right questions. You can read the full Autonomic Data Security paper for a more in-depth exploration here and learn more about the idea in this podcast episode.

Not too long ago, McKinsey released a report titled “Making a secure transition to the public cloud,” the result of interviews with IT security experts at nearly 100 enterprises around the world.

Leveraging the expertise of Google Cloud and McKinsey security experts, the research presents a strategic framework for IT security in cloud and hybrid environments, and provides recommendations on how to migrate to the cloud while keeping security top of mind.

The research shows what many already know: that public cloud adoption is accelerating thanks to increased technical flexibility, simpler scaling, and lower operating costs.

What’s exciting is that the research also reveals that many Chief Information Security Officers (CISOs) no longer view security as an inhibitor to adoption but instead an opportunity.

“In many cases [CISOs] acknowledge that cloud service providers’ security resources dwarf their own,” the authors write—and now these companies are focused on how to best adopt and configure cloud services for increased security.

The research identifies three common archetypes for perimeter security: backhauling, cleansheeting, and adopting cloud provider controls by default.

• Backhauling allows companies to continue managing IT security on-prem, with an external gateway connecting the data center to the public cloud. Approximately half of the companies surveyed currently use this model, but only 11% plan to continue doing so, since it can keep companies from realizing certain cloud benefits, such as agility.
• Cleansheeting requires greater investment and expertise, as it calls for redesigning IT security around a “virtual perimeter” and leveraging multiple cloud-native tools and services.
• Using cloud provider controls is the most cost-effective solution, but—depending on the cloud provider—can limit autonomy and may offer limited capabilities.

McKinsey uses these three models, along with the decision to re-architect applications for the cloud, to identify six “archetypes” for cloud security. Each archetype has its own tradeoffs.

The report also includes a tactical 10-step plan for successful cloud migration. Download it now.

In fast-moving organizations, it’s not uncommon for cloud resources, including entire projects, to occasionally be forgotten about. Not only such unattended resources can be difficult to identify, but they also tend to create a lot of headaches for product teams down the road, including unnecessary waste and security risks. 

To help you prune your idle cloud resources, we’re excited to introduce Unattended Project Recommender. It’s a new feature of Active Assist that provides you with a one-stop shop for discovering, reclaiming, and shutting down unattended projects. With actionable and automatic recommendations, you no longer have to worry about wasting money or mitigating security risks presented by your idle resources. Unattended Project Recommender uses machine learning to identify, with a high degree of confidence, projects that are likely abandoned based on API and networking activity, billing, usage of cloud services, and other signals. This feature is available via the Recommender API today, making it easy for you to integrate with your company’s existing workflow management and communication tools, or export results to a BigQuery table for custom analysis.

Thousands of projects can be unattended in large organizations, presenting major security risks

Your cloud projects can go abandoned or unattended for a number of reasons — ranging from a test environment that’s no longer needed, to project cancellation, to project owner switching jobs, and more. Not only can such projects contribute to your cloud bill (waste) but they may contain security issues such as open firewalls or privileged service account keys that attackers can exploit to get a hold of your cloud resources for cryptocurrency mining or, worse, compromise your company’s sensitive data. These security risks tend to grow over time because the latest best practices and patches are usually not applied to unattended projects. 

We experience this issue here at Google, too… In fact, it has been on Google’s internal security team’s radar for some time now, so we joined forces and looked into this problem together, starting with our very own “google.com” organization cloud projects. We quickly found some projects that were unattended, but remediating this issue was easier said than done due to challenges in several areas:

• Detection: With lots of signals available to you via sources like Cloud Monitoring, what are the right ones you should look at (e.g. API, networking, user activity)? How can you tell the difference between an unattended project and a project that has a low level of activity by design (e.g. a “shell” project that holds an auth token)?
• Remediation: Once you have identified a project that seems abandoned, how do you go about ensuring that it’s indeed an unattended project? How do you reduce the risk of deleting something that might be essential to a production workload, causing irreversible data loss? How do you solve this at the scale of your entire organization, beyond a one-time cleanup? 

Over the course of 2021 we built and tested a Google-internal prototype first, cleaning up many of our internal unattended projects, and then worked with a number of Google Cloud customers to build and tune this feature based on real-life data (thank you to all of our early adopters for working with us and your generous feedback that helped us shape this feature!) It was not uncommon for us to come across organizations with thousands of unattended projects, and we’re very excited to bring Unattended Project Recommender to all customers, in public preview.

Discovering and acting on unattended project recommendations

Unattended Project Recommender analyzes usage activity across all projects under your organization, including the following data:

• API activity (e.g. service accounts with authentication activity, API calls consumed)
• Networking activity (ingress and egress)
• Billing activity (e.g. services with billable usage)
• User activity (e.g. active project owners)
• Cloud services usage (e.g. number of active VMs, BigQuery jobs, storage requests)

Based on these signals, it can generate recommendations to clean up projects that have low usage activity (where “low usage” is defined using a machine learning model that ranks projects in your organization by level of usage), or recommendations to reclaim projects that have high usage activity but no active project owners. Here’s what an example post-processed summary list of recommendations can look like for the “foobar” organization that has 3 projects:

  Project ID: demo-project-307815
Recommendation: CLEANUP_PROJECT

Project ID: new-project
Recommendation: N/A

Project ID: bobs-playground-project
Recommendation: RECLAIM_PROJECT

In addition to the recommendations, you can also examine the underlying project activity insights that the recommendations are based upon. The insights provide additional information that can be useful for integration with your organization’s existing workflows and automation (e.g. send an auto-generated email or chat message to project owners based on the list provided by the owners field). Here’s an example insight payload:

  content:
  activeAppengineInstanceDailyCount: 0
  activeCloudsqlInstanceDailyCount: 0
  activeGceInstanceDailyCount: 3
  activeServiceAccountDailyCount: 1
  apiClientDailyCount: 18922           // Daily average API calls produced
  bigqueryInflightJobDailyCount: 0
  bigqueryInflightQueryDailyCount: 0
  bigqueryStorageDailyBytes: 0
  bigqueryTableDailyCount: 0
  consumedApiDailyCount: 0             // Daily average API calls consumed
  datastoreApiDailyCount: 0
  gcsObjectDailyCount: 11
  gcsRequestDailyCount: 0
  gcsStorageDailyBytes: 2663548
  hasActiveOauthTokens: false          // OAuth tokens used in the last 180 days
  hasBillingAccount: true
  numActiveUserOwners: 1
  owners:                              // List of project owners
  - activeOnProject: false
    member: user:user1@example.com
  - activeOnProject: true
    member: user:user2@example.com
  serviceWithBillableUsage:
  – Cloud Storage
  - Compute Engine
  vpcEgressDailyBytes: 264456938       // Daily average VPC egress bytes
  vpcIngressDailyBytes: 392435047      // Daily average VPC ingress bytes
  usagePercentile: 20                  // Level of usage relative to other projects

GCP projects are used in many different ways and for many different purposes. In case you get a recommendation to delete a project that’s being used in a way that’s out of the scope for this feature, you can dismiss the recommendation and it will stop showing up for the given project. 

Restoring deleted projects

When you choose to shut down a project using the projects.delete() method, it gets marked for deletion. After a project is marked for deletion, it becomes unusable, all resources within that project are shut down, and a 30-day wait period for the project and all of its data to get fully deleted begins.

In case a useful project is accidentally shut down, you have the option to restore the project within that 30-day wait period. Since restoring allows you to recover most but not necessarily all of your project data and resources, we recommend carefully examining the utilization insights associated with a project and considering any additional utilization signals that may not be captured by the Unattended Project Recommender before taking the cleanup action.

Early customer success stories

A number of enterprise customers are already using Unattended Project Recommender to keep their organizations clean of unattended projects and resources.

Decathlon, a French sporting goods retailer, is excited for the insight Unattended Project Recommender will bring to their environment, and are already deploying it as a part of their latest cloud security initiatives.

“After a thorough test of this feature and the validation of our CISO, we ended up deleting our first 775 projects, and no one complained! A great help to improve our security. The next step for us will be to operationalize it at scale, and implement a company wide policy for unattended resource management.” —Adeline Villette, Cloud Security Officer

For Veolia, one of the world’s largest water, waste and energy management companies, not only does this feature reduce security risks and waste, but also helps drive cultural shift and alignment with its ecological transformation strategy.

“This feature allows us to reduce our costs and security debt on assets that are no longer in use, and is also fully in line with Veolia’s philosophy of limiting its carbon footprint. After having tested Unattended Project Recommender on more than 3,000 projects throughout our organization, we are looking to bring it as proactive alerts to our project owners at scale.”—Thomas Meriadec, Product Manager

Box, a secure cloud content management provider, views it as a foundation for building a repeatable process to remediate unused resources.

“Unattended Project Recommender is a great fit for us. It gives us a unified view of project usage across our entire organization and enables us to address security risks of legacy projects in a systematic and organized manner, ensuring an even safer environment.” —Matt Bowes, Staff Security Engineer

Getting started with the Unattended Project Recommender

To help you get started, we’ve prepared a Cloud Shell tutorial (source code) that you can use to find unattended project recommendations within your own Projects/Folders/Organization. Click this button to clone the tutorial from GitHub and run in your Cloud Shell environment:

As you can see, listing recommendations for your projects only takes a few clicks with the tutorial (special thanks to Lanre Ogunmola, Security & Compliance Specialist, for making this look so easy)! For additional detail on using the gcloud CLI or API to discover unattended project recommendations, please refer to the documentation page.

You can also automatically export all recommendations from your Organization to BigQuery and then investigate the recommendations with DataStudio or Looker, or use Connected Sheets that let you use Google Workspace Sheets to interact with the data stored in BigQuery without having to write SQL queries.

As with any other Recommender, you can choose to opt out of data processing at any time by disabling the appropriate data groups in the Transparency & control tab under Privacy & Security settings.

We hope that you can leverage Unattended Project Recommender to improve your cloud security posture and reduce cost, and can’t wait to hear your feedback and thoughts about this feature! Please feel free to reach us at active-assist-feedback@google.com and we also invite you to sign up for our Active Assist Trusted Tester Group if you would like to get early access to the newest features as they are developed.