Can you keep a secret? Many people struggle to keep secrets. So do organizations. An organization’s digital credentials — its passwords, API keys, tokens, and encryption keys baked into its software code — are its vital secrets, used to authenticate access to protected resources and services.    

The security of these secrets represent one of the biggest risks that enterprises must manage when securing application infrastructure and environments. Fortunately, we have an important tool to help our customers better manage their secrets: Google Cloud Secret Manager. 

Secret Manager is a secure and efficient tool to centrally store, access, manage, and audit your organization’s secrets. As we noted in our April 2023 Threat Horizons Report, insufficient management of credentials and keys is now one of the top security issues facing cloud customers today. In particular, we have observed service account keys being used for everything from insider threats to cross-cloud breaches to intellectual property espionage.

Along with the increasing popularity of DevOps, secrets are used throughout development. In some cases, secrets are coded as plaintext into configuration files or directly into functions, where they can end up exposed in code repositories. If compromised, secrets can be used to gain unauthorized privileged access to enterprise data, infiltrate critical systems, and launch attacks.

Secret Manager can help organizations better manage their secrets even as concerns over potential security breaches grow. Organizations must consider the following four points during their secrets management risk analysis: 

1. Secrets sprawl: As organizations move more workloads to the cloud, the number of secrets they need to manage increases exponentially. This can lead to secrets sprawl, where secrets are stored in a variety of different locations, making it difficult to track and manage them.
2. Human error: Humans are fallible, and this is especially true when it comes to managing secrets. Mistakes, such as accidentally sharing a secret with the wrong person or the misconfiguration of a secret management tool, can have serious consequences.
3. Compliance violations: Organizations that handle sensitive data are subject to a variety of compliance regulations, including PCI-DSS, SOC 2, HIPAA, and GDPR. Without proper management, secrets could lead to potential compliance violations. 
4. Downtime: If secrets are not properly managed or rotated, it could lead to downtime for applications and services that rely on them.

Secret Manager can enable IT teams to keep track of secrets by providing a centralized place to store and manage them. You can access secrets directly from the Secret Manager console, or use the Secret Manager API or SDKs. Secret Manager is also designed to scale with your application. You can store as many secrets as you need without performance issues, and its API-first design lets you extend and integrate into other existing systems. 

Secret Manager integrations in Google Cloud

In many cases, developers save secret configuration details as plaintext in code for ease of use. There are two main risks associated with this approach:

• Anyone with access to secrets can see and copy them. There is no way to enforce any auditing or access controls practices here.
• It becomes difficult to move code between dev, test, and production environments.

Cloud-centered integrations supported by Secret Manager with other Google Cloud services in the software supply chain make it easier and safer to store and access sensitive information. 

The tool is about the secret information being made securely available in build or runtime.

Secret Manager integrations also can play a role in securing sensitive information in the software supply chain:

1. Integration with CI/CD systems used to build and deploy your software

Cloud Code integration can help developers build more secure applications by preventing hard-coding of sensitive configuration data such as passwords, credentials, and tokens in the codebase. It replaces them with secrets which can be programmatically fetched as required. You can create, access, and modify secrets directly from within your favorite IDE without having to navigate away from it. Integration is available for VS Code, IntelliJ, and Cloud Shell Editor.

Cloud Functions integration lets you access and expose secrets as environment variables or as a volume via the filesystem.

Cloud Build: To handle sensitive data in builds, you can store them in Secret Manager and then configure the build steps to access the information directly. In the build steps, you can associate the value of a secret to an environment and you can access this value via an environment variable from scripts or processes. Two common uses for fetching sensitive dependencies during build time are:

• Storing Docker authentication credentials for pull/push of images to Docker hub.
• Creating GitHub pull requests in response to builds.

2. Integration with runtime engines

Cloud Run: Native integration makes it easy to mount secrets in cloud run services. Using this integration, secrets can be made available to containers either as volume mounts or as env variables. Sensitive dependencies can be stored in environment variables, by referencing a secret name stored in Secret Manager instead of storing it as a plain text environment variable. This secret access operation can be done directly from the Cloud Run console without changing a single line of source code. 

If code reads secrets from a file instead of an env variable, the “mounted as volume” option can be used to make secrets look like files to the env vars. During service deployment, the service account used to run the containers are checked to see if they have the required permissions or not, preventing any unwanted access.

Kubernetes and GKE: The CSI driver–provider solution is an open-source work that provides integration with K8s. The driver lets customers centrally store their secrets in Google Secret Manager and consume them in a kubernetes native manner, as files mounted in K8s pods. The solution also comes with additional syncing capabilities. This method of consuming secrets also allows users to take advantage of SM’s capabilities like versioning, managed rotation, default/CMEK encryption and cloud-focused integrations with other Google Cloud services. Work is currently ongoing to provide official support to the driver solution as a GKE component.

Is Secret Manager right for my organization?

Developers carry a big responsibility to protect the sensitive data they are entrusted to work with, and keep secrets out of their code. However, organizations also have a responsibility to provide the tools and solutions needed to help their teams deliver on these expectations. Secret Manager makes it straightforward for developers to store and manage secrets while improving security. 

To learn more about Secret Manager, please review these resources: 

• Overview of Secret Manager
• Manage your Cloud Run secrets securely with Secret Manager (video)

Read the previous Best Kept Security Secrets blogs here:

• How VPC Service Controls can help build defense in depth
• Tap into the power of Organization Policy Service
• How Cloud EKM can help resolve the cloud trust paradox
• How Assured Workloads accelerates security and compliance

With unemployment application submissions reaching record numbers over the past year, state and local agencies in the United States have faced the challenge of processing unprecedented numbers of claims per week. The digital infrastructure most agencies have in place is unable to handle this volume, resulting in constituents waiting longer, and bad actors taking advantage of vulnerable systems. The Department of Labor Inspector General estimates that $63 billion in claims distributed is either an improper payment or fraud. 

Validating claims also requires secure data sharing with other agencies for document and identity verification. Government leaders need a way to allow case adjudicators to quickly and confidently release backlogged claims, integrate with existing systems, and segment legitimate claims from potentially fraudulent ones — all within limited government budgets — securely and at scale.

Implementing a fraud detection solution on Google Cloud

States were under pressure to release payments, while also filtering out potentially fraudulent claims. SpringML and Google Cloud developed a framework to give adjudicators a reliable verification process that quickly filters potentially fraudulent claims, while processing the remaining claims so benefits reach citizens in a timely manner. SpringML and Google Cloud, applied AI-powered machine learning models to detect anomalous patterns in large datasets. Using Google Cloud tools, SpringML implemented a solution to streamline workflows, improve efficiencies, automate processes and identify potentially fraudulent claims.

SpringML used a variety of Google Cloud products to deliver a fraud detection solution, including:

• Google Cloud Storage to store and manage data
• BigQuery to store tabular data and BigQuery Machine Learning (BQML) to conduct machine learning on that data
• AutoML solutions to build predictive models and risk scoring
• Visualization tools such as Looker and Data Studio to present data and help government leaders make informed decisions.

Implementing machine learning to detect improper payments allows agencies to classify claims as “fraud” or “not fraud” based on the number of flags, as well as prioritize the most urgent claims. Deploying intelligent virtual agents to handle frequently asked questions meant that live agents could focus their time on more challenging cases. 

Even once the pandemic is behind us, there will be bad actors trying to take advantage of overwhelmed or legacy systems. We’ve identified a few best practices for agencies managing enormous case loads and looking to improve improper payment analytics: 

• Move your systems to the cloud. Many on-premises legacy systems can’t update their applications and scale to meet the volume of claims. Moving to a cloud environment enables rapid solution deployment and ingestion of large amounts of data without fear of overloading the system. The cloud scales with you–cost-effectively and securely. 
• Understand patterns in the data. The answer is always in the data — we used deep analysis to help uncover suspicious patterns in large data sets. We implemented unsupervised machine learning to learn behaviors and create configurable rules that adjust to new information that comes into the system. We can uncover patterns that are likely associated with fraud – ones that a human might have missed. 
• Use AI/ML tools to automate your existing systems and teams. These tools enable humans to work smarter and more efficiently. We automate anomaly detection and create dashboards for adjudicators to rapidly process claims. We are enabling the Wisconsin Department of Workforce Development by implementing automatic calculations and processing of recharge amounts, resulting in faster processing times and fewer human errors. Proactive fraud detection and timely calculation of recharge payment allowed DWD to ensure the benefits reached the right individuals.
• Build flexibility into your systems. We discovered that fraud patterns change over time. For instance,flags for fraud during March-May 2020 were vastly different from those we found in June-July 2020. Google Cloud tools make it easy to continually update algorithms to detect patterns and integrate external data sources.

Using Google Cloud tools, we can update digital infrastructure and incorporate machine learning best practices to help organizations efficiently process large volumes of claims and identify high probability fraudulent ones. SpringML provides consulting and implementation services and industry-specific analytics solutions that deliver high-impact business value to accelerate data-driven digital transformation. Learn more about fraud detection and how to improve improper payments analytics by watching our webinar. 

Twenty years ago, organizations discovered the magic of server virtualization. Among the many benefits of virtualization, it heralded the end of the need for one physical server per application and the beginning of managing software-defined infrastructure. It helped automate building and installing systems, and it helped lay foundations for today’s Continuous Integration/Continuous Delivery mechanisms and DevOps practices. However, the adoption of virtualization was not always smooth and several hurdles stood in its path. Today’s organizations are facing similar obstacles in their journey to the cloud.

Organizations that wanted to virtualize their server environment two decades ago faced a significant challenge that was neither technical nor budgetary, but organizational, stretching across IT teams and beyond. The first team to be concerned about virtualization was the one in charge of installing the physical servers in the racks of the datacenter, managing the physical network connections and operating environmental controls, such as HVAC and power systems. These tasks were drastically reduced, if not outright eliminated.

The second team affected by this shift in technology was the one responsible for installing the operating system and running the post-install procedures. Their job almost completely disappeared as most virtual machines were automatically instantiated from templates.

As a result, successful adoption relied on the ability to smooth these organizational changes. Along the way, training the staff and making sure they acquire the new skills necessary to operate the virtualization infrastructure was critical. And usually, the affected teams eventually found that their new responsibilities were different but still essential to the organization, and perhaps even more appealing.

Virtualizations lessons for cloud transformation regulators

Discussions on virtualization didn’t end with these internal changes when organizations were operating in a regulated environment. They needed to engage closely with their regulators. Regulated organizations often considered the hypervisor solely as an additional layer of software with potential vulnerabilities.

This viewpoint raised risks (such as virtual machine escape) that didn’t exist with physical servers. Even if this is taken into account and the risks mitigated, it should have been considered in balance with all the benefits of machine virtualization, including security benefits such as asset identification, harmonization, and smoother patch management.

While the relevance and benefits of virtualization are now widely accepted, history has repeated itself as similar issues have arisen for organizations facing cloud migrations. When large cloud service providers (CSP) began supporting critical workloads for their customers, regulators increased their oversight over CSPs and issued guidance concerning cloud adoption. This happened in the European financial sector, with guidelines from the European Supervisory Authorities (European Banking Authority, European Securities and Market Authority and European Insurance and Occupational Pensions Authority,) and the coming Digital Operational Resilience Act regulation.

These regulations can help establish trust between providers and customers. As regulator decisions are vital for organizations, there is an opportunity now with cloud technology to balance the risks with the benefits and take into consideration all the ways CSPs can help organizations improve their overall security and compliance levels.

Cultural transformation can drive cloud transformation

Another similarity between cloud transformation and the journey to virtualization is the required internal transformation. When an organization decides to start its journey to the cloud, training its staff with the new technology and tools is a necessary step, but probably not the first nor the most arduous to make. A change of mindset is paramount to fully incorporate all the cloud benefits, in particular regarding security.

This new approach begins with a deep transformation at the organizational level. Organizations should avoid viewing the cloud as a datacenter because there’s so much more potential to cloud technology. This new approach and its accompanying organizational transformation is driven by two factors: The burdens that the CSP removes from the customer, and the new flexibility that software-defined infrastructure brings to the table.

The CSP directly manages several tasks and services, and their security, which means that they’re no longer a direct concern for the customer. These responsibilities include data center management, security-hardened hardware, default encryption for data at rest and data in transit, and resilient network management.

Since all the infrastructure used by the organization to manage its workloads is software defined, it brings much more flexibility to what can be done with it. Software-defined infrastructure can more easily enable security guardrails and continuous compliance that weren’t possible before.

While there’s a temptation to approach an organization’s cloud transformation as merely a “lift and shift” operation from on-premises infrastructure, the reality is that approach reduces the potential gains a cloud transformation can bring to an organization. Moving systems to the cloud accounts for a small portion of the potential impact, while updating the mindset can be truly transformational. Development, architecture and security processes need to be rebuilt; this inevitably requires a deeper transformation of the organization.

The next step is to transform the operations, to align them with the overall organization and the new defined processes, operationalize CI/CD and build DevOps practices. After that, the technological gap will be adjusted, moving the new tools and teaching teams how to best use them.

The important point here is the order of the transformation to manage: organization, operations and technologies (O-O-T) and not the other way round (T-O-O). Of course, these three steps should not be considered completely sequential, as technology plays an important role in the organization and the operations. However, it is essential to support the development of personnel and frameworks to take full advantage of the transformation opportunity, rather than primarily focusing on the tooling.

In addition, at Google Cloud, we believe in a shared fate that goes far beyond the usual shared responsibility model. We think it is part of our duty to help our customers to achieve their goals in their scope of responsibility. We prepare secure landing zones and guide the customers, we bring transparency to security controls and help them with cyber-insurance.

Our shared objective with our customers is to continuously improve security. Our Google Cybersecurity Action Team initiative helps us to be engaged alongside our customers, provide them with necessary guidance, support them in their security and compliance strategies, and assist their organizational and operational transformation in their cloud journey.

• Listen to a related podcast “How to Apply Lessons from Virtualization Transition to Make Cloud Transformation Better”
• Listen to “Preparing for Cloud Migrations from a CISO Perspective, Part 1”
• Read “The journey to the cloud mitigates enterprise risk”
• Also, review “Megatrends drive cloud adoption—and improve security for all”
• Visit Google Cybersecurity Action Team website

The success of the modern security operations center, despite the infusion of automation, machine learning, and artificial intelligence, remains heavily dependent on people. This is largely due to the vast amounts of data a security operations center must ingest—a product of an ever-expanding attack surface and the borderless enterprise brought on by the rapid rise of cloud adoption.

All those alerts coming in mean proactive and reactive human decision making remains critical.

Perhaps it should come as no surprise that the information security analyst now ranks as No. 1 News’ 100 Best Jobs Rankings, “determined by identifying careers with the largest projected number and percentage of openings through 2030, according to the U.S. Bureau of Labor Statistics.” Security, and specifically detection and response, is not only a business imperative—it is arguably the top worry on the minds of CEOs.

However, the security analyst is also one of the most likely professionals to want to leave their jobs, according to a newly released “Voice of the SOC Analyst” study conducted by Tines.

What gives? Turnover woes are attributable to several key SecOps challenges that never seem to budge.

1) Alert fatigue and false positives: Have you ever received so much spam or junk mail that you end up ignoring your new messages entirely, leading you to miss an important one? The same can happen for alerts. Too much noise is unsustainable and can lead to the real threats being missed, especially as perimeters expand and cloud adoption increases.

2) Disparate tools: Already in the company of too many point-detection tools, security operations professionals are saying hello to a few more in the era of remote work and increased cloud demands. The latest count is north of 75 security tools that need to be managed by the average enterprise.

3) Manual processes: Use case procedures that result in inconsistent, unrepeatable processes can bottleneck response times and frustrate SecOps teams. Not everything in the SOC needs to—or should be—automated, but much can be, which then frees up analysts and engineers to concentrate on higher-order tasks and be able to more easily train new employees.

4) Talent shortage: Death, taxes, and the cybersecurity skills shortage. As sure as the sun will rise tomorrow, so will the need for skilled individuals to wage the cybersecurity fight. But what happens when not enough talent is filling the seats? Teams must compensate to fill the gap.

5) Lack of visibility: Security operations metrics are critical for improving productivity and attracting executive buy-in and support, but SecOps success can be difficult to track, as reports can require a significant amount of work to pull together.

The caveat of course is that it would be rare to find a SecOps team working without the above challenges. As such, some of the immediate steps you can take to push back against these constraints focus on people-powered processes and technologies to remedy the issues.

According to a recent paper co-authored by Google Cloud and Deloitte:

Humans are—and will be—needed to both perform final triage on the most obtuse security signals (similar to conventional SOC Level 3+) and to conduct a form of threat hunting (i.e. looking for what didn’t trigger that alert).

Machines will be needed to deliver better data to humans, both in a more organized form (stories made of alerts) and in improved quality detections using rules and algorithms— all while covering more emerging IT environments.

Both humans and machines will need to work together on mixed manual and automated workflows.

So, what does this ultimately mean you must do to improve your security operations? Here are five practical suggestions:

Detect Threats More Efficiently

Efficiencies within the SOC can be realized from a SIEM solution that automatically detects threats in real-time and at scale. The right platform will support massive data ingestion and storage, relieve traditional cost and scaling limitations, and broaden the lens for anomaly and machine learning/AI-based detection. With data stored and analyzed in one place, security teams can investigate and detect threats more effectively.

Respond to Threats Automatically

SOAR can be a game-changer in terms of caseload reduction and faster (and smarter, especially when integrated with threat intelligence) response times. But before rushing headfirst into automation, you should consider your processes, review outcomes you are trying to achieve (such as reduced MTTD)–and then decide exactly what you want to automate (which can be a lot with SOAR). Once clear processes are determined where automation can contribute, SOC personnel are freed up to be more creative in their work.

Prioritize Logs

Many teams lack a strategy for collecting, analyzing and prioritizing logs, despite the fact that these sources of insight often hold the clues of an ongoing attack. To help, here are two cheat sheets featuring essential logs to monitor.

Outsource What You Can’t Do Yourself

Process improvements may help you compensate for perceived personnel shortages (for example, perhaps fixing a misconfigured monitoring tool will reduce alert noise). Of course, many organizations need additional human hands to help them perform tasks like round-the-clock monitoring and more specialized functions like threat hunting. Here is where a managed security services provider or managed detection provider can be helpful. Be realistic about your budget, however, as you may be able to introduce some solutions in-house.

Institute Career Models

Lack of management support is cited as the fourth-biggest obstacle to a fully functioning SOC model, according to the 2022 SANS Security Operations Center Survey. To overcome this, SecOps leaders must help improve workflow processes, protect innovation, keep teams absorbed in inspiring and impactful work versus mundane tasks, remain flexible with staff, and endorse training and career development. Because at the end of the day, the SOC is still distinctly human–and that is who will be the difference maker between success and failure.