In fast-moving organizations, it’s not uncommon for cloud resources, including entire projects, to occasionally be forgotten about. Not only such unattended resources can be difficult to identify, but they also tend to create a lot of headaches for product teams down the road, including unnecessary waste and security risks. 

To help you prune your idle cloud resources, we’re excited to introduce Unattended Project Recommender. It’s a new feature of Active Assist that provides you with a one-stop shop for discovering, reclaiming, and shutting down unattended projects. With actionable and automatic recommendations, you no longer have to worry about wasting money or mitigating security risks presented by your idle resources. Unattended Project Recommender uses machine learning to identify, with a high degree of confidence, projects that are likely abandoned based on API and networking activity, billing, usage of cloud services, and other signals. This feature is available via the Recommender API today, making it easy for you to integrate with your company’s existing workflow management and communication tools, or export results to a BigQuery table for custom analysis.

Thousands of projects can be unattended in large organizations, presenting major security risks

Your cloud projects can go abandoned or unattended for a number of reasons — ranging from a test environment that’s no longer needed, to project cancellation, to project owner switching jobs, and more. Not only can such projects contribute to your cloud bill (waste) but they may contain security issues such as open firewalls or privileged service account keys that attackers can exploit to get a hold of your cloud resources for cryptocurrency mining or, worse, compromise your company’s sensitive data. These security risks tend to grow over time because the latest best practices and patches are usually not applied to unattended projects. 

We experience this issue here at Google, too… In fact, it has been on Google’s internal security team’s radar for some time now, so we joined forces and looked into this problem together, starting with our very own “google.com” organization cloud projects. We quickly found some projects that were unattended, but remediating this issue was easier said than done due to challenges in several areas:

• Detection: With lots of signals available to you via sources like Cloud Monitoring, what are the right ones you should look at (e.g. API, networking, user activity)? How can you tell the difference between an unattended project and a project that has a low level of activity by design (e.g. a “shell” project that holds an auth token)?
• Remediation: Once you have identified a project that seems abandoned, how do you go about ensuring that it’s indeed an unattended project? How do you reduce the risk of deleting something that might be essential to a production workload, causing irreversible data loss? How do you solve this at the scale of your entire organization, beyond a one-time cleanup? 

Over the course of 2021 we built and tested a Google-internal prototype first, cleaning up many of our internal unattended projects, and then worked with a number of Google Cloud customers to build and tune this feature based on real-life data (thank you to all of our early adopters for working with us and your generous feedback that helped us shape this feature!) It was not uncommon for us to come across organizations with thousands of unattended projects, and we’re very excited to bring Unattended Project Recommender to all customers, in public preview.

Discovering and acting on unattended project recommendations

Unattended Project Recommender analyzes usage activity across all projects under your organization, including the following data:

• API activity (e.g. service accounts with authentication activity, API calls consumed)
• Networking activity (ingress and egress)
• Billing activity (e.g. services with billable usage)
• User activity (e.g. active project owners)
• Cloud services usage (e.g. number of active VMs, BigQuery jobs, storage requests)

Based on these signals, it can generate recommendations to clean up projects that have low usage activity (where “low usage” is defined using a machine learning model that ranks projects in your organization by level of usage), or recommendations to reclaim projects that have high usage activity but no active project owners. Here’s what an example post-processed summary list of recommendations can look like for the “foobar” organization that has 3 projects:

  Project ID: demo-project-307815
Recommendation: CLEANUP_PROJECT

Project ID: new-project
Recommendation: N/A

Project ID: bobs-playground-project
Recommendation: RECLAIM_PROJECT

In addition to the recommendations, you can also examine the underlying project activity insights that the recommendations are based upon. The insights provide additional information that can be useful for integration with your organization’s existing workflows and automation (e.g. send an auto-generated email or chat message to project owners based on the list provided by the owners field). Here’s an example insight payload:

  content:
  activeAppengineInstanceDailyCount: 0
  activeCloudsqlInstanceDailyCount: 0
  activeGceInstanceDailyCount: 3
  activeServiceAccountDailyCount: 1
  apiClientDailyCount: 18922           // Daily average API calls produced
  bigqueryInflightJobDailyCount: 0
  bigqueryInflightQueryDailyCount: 0
  bigqueryStorageDailyBytes: 0
  bigqueryTableDailyCount: 0
  consumedApiDailyCount: 0             // Daily average API calls consumed
  datastoreApiDailyCount: 0
  gcsObjectDailyCount: 11
  gcsRequestDailyCount: 0
  gcsStorageDailyBytes: 2663548
  hasActiveOauthTokens: false          // OAuth tokens used in the last 180 days
  hasBillingAccount: true
  numActiveUserOwners: 1
  owners:                              // List of project owners
  - activeOnProject: false
    member: user:user1@example.com
  - activeOnProject: true
    member: user:user2@example.com
  serviceWithBillableUsage:
  – Cloud Storage
  - Compute Engine
  vpcEgressDailyBytes: 264456938       // Daily average VPC egress bytes
  vpcIngressDailyBytes: 392435047      // Daily average VPC ingress bytes
  usagePercentile: 20                  // Level of usage relative to other projects

GCP projects are used in many different ways and for many different purposes. In case you get a recommendation to delete a project that’s being used in a way that’s out of the scope for this feature, you can dismiss the recommendation and it will stop showing up for the given project. 

Restoring deleted projects

When you choose to shut down a project using the projects.delete() method, it gets marked for deletion. After a project is marked for deletion, it becomes unusable, all resources within that project are shut down, and a 30-day wait period for the project and all of its data to get fully deleted begins.

In case a useful project is accidentally shut down, you have the option to restore the project within that 30-day wait period. Since restoring allows you to recover most but not necessarily all of your project data and resources, we recommend carefully examining the utilization insights associated with a project and considering any additional utilization signals that may not be captured by the Unattended Project Recommender before taking the cleanup action.

Early customer success stories

A number of enterprise customers are already using Unattended Project Recommender to keep their organizations clean of unattended projects and resources.

Decathlon, a French sporting goods retailer, is excited for the insight Unattended Project Recommender will bring to their environment, and are already deploying it as a part of their latest cloud security initiatives.

“After a thorough test of this feature and the validation of our CISO, we ended up deleting our first 775 projects, and no one complained! A great help to improve our security. The next step for us will be to operationalize it at scale, and implement a company wide policy for unattended resource management.” —Adeline Villette, Cloud Security Officer

For Veolia, one of the world’s largest water, waste and energy management companies, not only does this feature reduce security risks and waste, but also helps drive cultural shift and alignment with its ecological transformation strategy.

“This feature allows us to reduce our costs and security debt on assets that are no longer in use, and is also fully in line with Veolia’s philosophy of limiting its carbon footprint. After having tested Unattended Project Recommender on more than 3,000 projects throughout our organization, we are looking to bring it as proactive alerts to our project owners at scale.”—Thomas Meriadec, Product Manager

Box, a secure cloud content management provider, views it as a foundation for building a repeatable process to remediate unused resources.

“Unattended Project Recommender is a great fit for us. It gives us a unified view of project usage across our entire organization and enables us to address security risks of legacy projects in a systematic and organized manner, ensuring an even safer environment.” —Matt Bowes, Staff Security Engineer

Getting started with the Unattended Project Recommender

To help you get started, we’ve prepared a Cloud Shell tutorial (source code) that you can use to find unattended project recommendations within your own Projects/Folders/Organization. Click this button to clone the tutorial from GitHub and run in your Cloud Shell environment:

As you can see, listing recommendations for your projects only takes a few clicks with the tutorial (special thanks to Lanre Ogunmola, Security & Compliance Specialist, for making this look so easy)! For additional detail on using the gcloud CLI or API to discover unattended project recommendations, please refer to the documentation page.

You can also automatically export all recommendations from your Organization to BigQuery and then investigate the recommendations with DataStudio or Looker, or use Connected Sheets that let you use Google Workspace Sheets to interact with the data stored in BigQuery without having to write SQL queries.

As with any other Recommender, you can choose to opt out of data processing at any time by disabling the appropriate data groups in the Transparency & control tab under Privacy & Security settings.

We hope that you can leverage Unattended Project Recommender to improve your cloud security posture and reduce cost, and can’t wait to hear your feedback and thoughts about this feature! Please feel free to reach us at active-assist-feedback@google.com and we also invite you to sign up for our Active Assist Trusted Tester Group if you would like to get early access to the newest features as they are developed.

It’s no question that the cloud computing industry is booming and cloud experts are in high demand.  In 2020, 67 percent of organizations surveyed in the IDG Cloud Computing Report added new cloud roles and functions (source). 

So there are plenty of cloud jobs out there, but if you don’t have years of experience, you might find it difficult to get one. This is especially true for members of underrepresented communities, who face other kinds of systemic barriers in education and employment.

To help meet this demand and democratize opportunity, Google Cloud is introducing its first virtual Certification Journey Learning program for underrepresented communities across the United States—Cloud Career Jump Start. The program is free of charge for participants, and includes technical and professional skill development and resources with an aim to provide a readiness path to certification via the Google Cloud Associate Cloud Engineer Exam. 

Google Cloud’s Certification Journey Learning Program

The program offers free access to the Google Cloud Associate Cloud Engineer training, which provides support preparations for the Certification Exam. This industry-recognized certification helps job applicants validate their cloud expertise, elevate their career, and transform businesses with Google Cloud technology. Learn more about the exam by watching the Certification Prep: Associate Cloud Engineer webinar. 

Designed specifically for Computer Science or Information Systems-related majors (or relevant experience), the 12-week program offers:

• Exclusive free access to Google Cloud Associate Cloud Engineer-related training to support preparations for the Certification Exam. 
• Guided progress through the certification prep training on Qwiklabs. 
• Office hours hosted by Google Cloud technical experts. 
• Technical workshops, including sessions with Googlers who earned cloud industry certifications and have built impactful careers in cloud tech, such as Google Cloud’s Kelsey Hightower (a Principal Developer Advocate) and Jewel Langevine (a Solution Engineer). 

Earn your Associate Cloud Engineer Certification 

Curious if you qualify to apply for the program? We’ve put together a guide below to help: 

• What does an associate cloud engineer do? Associate Cloud Engineers deploy applications, monitor operations, and manage enterprise solutions. They also use the Google Cloud Console and the command-line interface to perform common platform-based tasks, to maintain one or more deployed solutions that leverage Google-managed or self-managed services on Google Cloud.
• Who’s eligible? We are encouraging applicants from groups that have been historically underrepresented in tech including Black, Latinx, Native American & Indigenous communities. Applicants should have six or more months of hands-on experience in a Computer Information Systems-related major, and/or relevant experience through online courses, boot camps, hackathons or internships. Although the ability to program is not required, familiarity with the following IT concepts is highly recommended: virtual machines, operating systems, storage and file systems, networking, databases, programming, and working with Linux at the command line.

How do you know if you are ready for the program? 

To get an idea of what the program will cover:

• Watch the Certification Prep: Associate Cloud Engineer webinar
• Complete the Skill IQ Assessment via Pluralsight
• Review the Associate Cloud Engineer Certification Exam Guide 
• Take the Associate Cloud Engineer Practice Exam

Jump start a career in cloud infrastructure and technology 

Following the hands-on training, Google Cloud will offer an additional 9 months of career development resources and activities. This includes an online support community, mentoring with Googlers & partners including SADA, EPAM Systems Inc., and Slalom, resume and interviewing support from Google Recruiting & Staffing, and additional career workshops. We are also partnering with a number of Black-owned and -operated, nonprofit and cloud training organizations like Kura Labs. 

Cloud certifications open doors 

Want to hear more about this program from two Googlers who completed it? We asked Kelsey and Jewel to share more on how certifications helped them launch their careers in the cloud industry: 

• “IT certifications introduced me to the game; opportunities and hard work helped me change it.” – Kelsey Hightower, Staff Developer Advocate, Google Cloud Platform 
• “Becoming certified as a cloud computing professional combined with prayer, networking, and practice has kept me moving on my purposeful and rewarding career path as an engineer.” – Jewel Langevine, Solution Engineer, Google Cloud Solutions Studio

Kelsey and Jewel’s wisdom doesn’t end there. They play a central role in the program, sharing more about how they navigated certifications and leveraged them for success. 

Apply today to Cloud Career Jump Start

The program is now live in the United States, with plans to expand to other regions in the coming months. It is completely virtual, and all training is on-demand so that participants can access their coursework anytime, anywhere via the web or mobile device. To determine whether you (or someone you know) would be a great fit for the Cloud Career Jump Start, check out our guidelines and apply.

Today’s Google Doodle reminds us of the enormous changes our planet is experiencing due to climate change. Everyone, from businesses to governments to technologists, has the opportunity to meet this challenge — transforming themselves and their organizations to be more sustainable. For this Earth Day 2022, and indeed Earth Week, Google Cloud celebrates the organizations and individuals who are fighting climate change with innovative technology. We don’t want you to miss a thing, so here’s a recap of all our news in one handy location.

We asked global CEOs: what is it going to take to make progress on sustainability in your org?
In a survey of 1,500 CXOs across 16 countries, many executives say they are willing to do what it takes to have more sustainable practices. But despite their ambition, real measures of impact are lacking. To see what will change that, check out the blog.

We announced a new innovation challenge supporting climate science and research…
Our blog on Monday announced the Climate Innovation Challenge Research Credits program, to support researchers as they work to better understand climate change, increase climate resilience and develop new, promising solutions to urgent climate challenges. You can apply for research credits here.

…and shared stories of researchers making a difference
We interviewed Dr. Richard Fernandes from Natural Resources Canada, who built the LEAF toolbox that maps and assesses vegetation with satellite data from Google Earth Engine. You can read our Q&A here.

Canada has approximately 10 million square kilometers of land and the annual data volume of these maps is equivalent to streaming HD movies for over 750 hours non-stop. Cloud computing allows us to manage all this data in a useful and accessible way.

Dr. Fernandes, Research Scientist

We also published a story about the U.S. Department of Agriculture’s Forest Service, and how they use Google Cloud processing and analysis tools to help sustainably manage 193 million acres of land.

We turned the lights on at new clean energy projects in four countries…
We shared details of our battery project in Belgium, solar projects in Denmark, and wind projects in Chile and Finland. Our battery project in Belgium is the first of its kind, enabling us to switch from diesel generators to a cleaner backup solution that will keep the internet up and running in the event of a power disruption. These will all help us continue to operate the cleanest cloud in the industry.

…and made it easier to learn how to build applications more sustainably
We launched a new lab that walks users through our Carbon Sense suite of products. From using our region picker app to make low-carbon architecture decisions, to analyzing the carbon footprint of your Google Cloud app with Carbon Footprint, we’re building sustainability into the tools you use every day. You can also find Carbon Footprint training in the new Data Warehouse Cloud On-Board.

We formed an ecosystem of partners to help accelerate sustainability projects…
The Google Cloud partner ecosystem is critical to helping our customers act sustainably today. A new whitepaper produced in partnership with Enterprise Strategy Group shares real-world solutions that could make an immediate impact — not in the next decade, but right now.

…and shared stories of innovative startups changing the game with Google Cloud.
Take Enexor and its partners, who are producing clean and sustainable energy from discarded plastics and agro-waste. The blog from Lee Jestings, Enexor Founder & CEO, shares how Google for Startups got them started, and which Google Cloud tools help them build predictive models. Check out their story.

Or Nuuly, the rental and resale business created by the URBN portfolio, which also includes Urban Outfitters, Anthropologie, and Free People. In the blog you can read how Nuuly is using technology to provide a sustainable experience to employees and customers — from upcycling clothing, to recyclable and reusable packaging.

Whether you’re a startup, scientist, executive or developer, at Google Cloud we’ll continue to work hard to help make your digital transformation a sustainable one.

Learn more about our sustainability work here, and don’t miss the inaugural Cloud Sustainability Summit this June. Register now.