Access to the cloud has advanced the distribution and consumption of financial information on a global scale. In parallel, the global financial data landscape has been transformed by an influx of alternative data sources, including social media, meteorological data, satellite imagery, and other data. Exchanges and market data providers now find they need to include these new datasets to enrich their products and compete, which has meant they now must consider cloud-based models to keep up with the demands of their customers who expect easy, quick, flexible and cost-efficient ways to consume market data.

To address these needs, today we’re announcing the general availability of Datashare for financial services, a new Google Cloud solution that brings together the entire capital markets ecosystem—data publishers, and data consumers—to exchange market data securely and easily.

Datashare helps organize third-party financial information, making it accessible and useful to market data publishers and data consumers. We open-sourced the entire Datashare solution so market data publishers can now onboard their licensed datasets to Google Cloud securely, quickly and easily, while data consumers can consume that data as a service in tools of their preference, such as BigQuery.

Three ways to distribute and consume your data

Batch data delivery

Datashare provides a batch data delivery mechanism for data publishers to deliver their reference data, historical tick data, alternative market data sources and more via BigQuery, reducing the administrative burden on data consumers to extract insights from data. 

Real-time data streaming delivery

By using this event-based data delivery channel for rapidly changing instrument prices, tick data, orders, news and others via Pub/Sub, data consumers can reliably process individual messages or rewind to a point in time to replay a prior market scenario and test model changes.

Monetizing licensed datasets

Market data publishers can onboard their licensed datasets to Google Cloud and make them available via a one-stop-shop on Google Cloud Marketplace, enabling a new sales channel to expand market reach.

Reference architecture

Check out the diagram below to see how you can share your batch and real-time data directly to your Google Cloud customers with BigQuery and Pub/Sub.

As you can see in the above reference architecture, both publishers and consumers can derive several benefits from the solution:

Benefits for data publishers

• You no longer have to maintain your own delivery and licensing infrastructure.
• You can easily package and deliver granular data products and experiments with SQL.
• You can have a solution that scales with your business as data volumes and number of customers grow.

Benefits for data consumers

• Your data is ready for analysis and machine learning (ML)— you no longer have to maintain extract, transform, and load (ETL) pipelines to load files and transform data.
• You can avoid the expense and burden of maintaining multiple copies of large data files.
• You can be more targeted with consumption of data using BigQuery queries, improving performance, and compliance, and reducing cost.

Accessing the datasets

Google Cloud has been working with multiple industry firms on innovating in the market data space. By using Datashare for publishing, data publishers can make their entire datasets available on Google Cloud. Early adopters of Datashare include firms such as OneTick and Accern. OneTick’s datasets include reference and historical futures data (that can be accessed in our console with your login). Accern’s datasets include alternative data such as market sentiment and credit analysis data (that can be accessed in our console with your login).

To make it more helpful, we partnered with Accern to create a hypothetical scenario to describe the data acquisition and analytics process step-by-step.

Accern use case 

As a sustainability analyst, you require an economic, social and governance (ESG) dataset to determine which sector is the most widely covered ESG sector by analysts, and to also identify the sector with the lowest ESG sentiment score. Now, you can discover and acquire an ESG dataset in Google Cloud.

Step 1. Navigate to the Financial Services solutions page in the Google Cloud console:

Step 2. Click a dataset, for example Accern AI-Generated ESG Insights, then review the overview details, plans and pricing, documentation and support information. To view the available pricing tiers, click ‘View All Plans’. Once you’ve decided on a tier that you would like to subscribe to, click ‘Select’, choose a billing account and review and accept the terms of service to complete the subscription. Once the steps are complete, click ‘Subscribe’ at the bottom. An overlay window will appear, click ‘Register with Accern’ to activate and complete the subscription.

Step 3. Once activation is complete, you’ll be directed to the Datashare ‘My Products’ screen. Voila! You are now subscribed to Accern’s ESG Scores dataset and can access it in your Google Cloud instance using BigQuery. To access the data, click the hour glass icon on the corresponding ‘My Products’ record that you just purchased. An overlay will present you with the details on the dataset and/or table. Click the ‘Navigate to Table’ button to navigate through to the BigQuery console.

Step 4. Now that you have access and are in the BigQuery console, it’s time to generate data insights.

^{For this example, we’ve eliminated the company identifying information that is included as part of the subscription and aggregated company ESG in a view where each row represents a day, an industry sector, a specific identified ‘ESG Issues’ (event_group and event) and the respective ‘ESG Sentiment’ per issue.}

For example, row 1 indicates that within the ‘Healthcare’ sector, there was a ‘Social – Civil Society’ issue identified and it had a negative ESG sentiment score of -15.35.

Step 5. Generate a report by exporting it to Data Studio to build visualizations and conduct additional analysis on the ESG data.

Select ‘Export’ and ‘Explore with Data Studio’.

Step 6. Build a simple/basic report.

Now that the ESG data appears in Data Studio, you can start by building a simple chart to help you understand which industry sectors have the highest volume of discussions around ESG and the overall ESG Sentiment per industry sector.

To build the chart:

• Select the chart type ‘Table’.

• Include Entity_Sector as your dimension to aggregate results by ‘Industry Sector.’
• Include Signal_ID as a measure to count the number of ESG passages identified per ‘Industry Sector.’
• Include AVG(Event_Sentiment) as a measure to display the overall ESG Sentiment per ‘Industry Sector’ across ESG Issues.

You can see sectors that are  most discussed when it comes to ESG related topics and their corresponding ‘ESG Sentiment’ scores.

Step 7. Build your final report in Data Studio.

As a next step you can further drill into the data to understand ESG data specific to each ‘Industry Sector’ and identify positive and negative ESG practices. 

Accern has built a more complex sample dashboard and made it available publicly here. You can interact with this report and play around with the data. The dashboard can help to identify material ESG insights for each sector to inform your investment and risk processes. If you have additional questions, you can reach out to Accern directly.

Discovering, accessing and analyzing licensed datasets is quick and easy. Stay tuned for more updates on new licensed datasets.

Publishing your data via Datashare

If you are a publisher of market data, alternative, or exotic data, you can use Datashare to get it published on Google Cloud Marketplace.

Start by joining the Partner Advantage program by registering for the Partner Advantage Portal and applying for the Partner Advantage Build Model engagement. Visit our getting started guide for information to get started on publishing licensed datasets in the Marketplace. Stay tuned for a future blog post about using Datashare to publish datasets in the Marketplace.

More solutions for capital markets

Check out other Google Cloud solutions for capital markets.

Many organizations require effective processes and techniques for removing or obfuscating certain sensitive information in the data they store. An important tool to achieve this goal is de-identification. Defined by NIST as a technique that “removes identifying information from a dataset so that individual data cannot be linked with specific individuals. De-identification can reduce the privacy risk associated with collecting, processing, archiving, distributing or publishing information.

Always striving to make data security easier, today we are happy to announce the availability of a de-identification action for our Cloud Storage inspection jobs. Now, you can de-identify Cloud Storage objects, folders, and buckets without needing to run your own pipeline or custom code. Additionally, we have enhanced our transforms by adding a new dictionary replacement method that can help you achieve stronger privacy protection – especially with unstructured data you might store like customer support chat logs.

The “De-identify findings” Action

The “de-identify findings” action for Cloud DLP inspection jobs is a fully managed feature that creates a de-identified copy of the data objects that are inspected. This means that you can inspect a Cloud Storage bucket for sensitive data like Personal Identifiable Information (PII) and then create a redacted copy of these objects all with a few clicks in the Console UI. No need to write custom code or manage complex pipelines and since it’s fully managed, it will auto-scale for you without you needing to manage quota.

This new action supports the following data types:

• Text files
• Comma- or tab-separated values
• Images (see regional limitations)

Once enabled, the DLP job will perform an inspection of the data and produce a de-identified copy of all supported files into the output bucket or folder.

You can also use the new de-identify action on Job Triggers to automatically de-identify new content as it appears on a recurring schedule. This is useful for creating a workflow with a safe drop zone for incoming files that need to be de-identified before being made accessible.

What can automatic De-identification do?

Cloud DLP provides a set of transformation techniques to de-identify sensitive data while attempting to make the data still useful for your business. These techniques include:

• Redaction: Deletes all or part of a detected sensitive value.
• Replacement: Replaces a detected sensitive value with a specified surrogate value.
• Masking: Replaces a number of characters of a sensitive value with a specified surrogate character, such as a hash (#) or asterisk (*).
• Crypto-based tokenization: Encrypts the original sensitive data value using a cryptographic key. Cloud DLP supports several types of tokenization, including transformations that can be reversed, or “re-identified.”
• Bucketing: “Generalizes” a sensitive value by replacing it with a range of values. (For example, replacing a specific age with an age range, or temperatures with ranges corresponding to “Hot,” “Medium,” and “Cold.”)
• Date shifting: Shifts sensitive date values by a random amount of time.
• Time extraction: Extracts or preserves specified portions of date and time values.

New Dictionary Replace method

When a sensitive data element is found, dictionary replacement replaces it with a randomly selected value from a list of words that you provide. This transformation method is especially useful if you want the redacted output to have more realistic surrogate values.

Consider the following example: You collect customer support chat logs as part of providing service to your customers. These support chat logs contain various types of Personal Identifiable Information (PII) including people’s names and email addresses. Cloud DLP can find and de-identify the sensitive elements with static replacements such as “[REDACTED]” to help prevent someone from seeing this sensitive data.

With the new dictionary replacement method you can instead replace these findings with a randomly selected value from a dictionary. This dictionary replacement provides two key benefits over static replacement:

1. The resulting output can look more realistic
2. Because the output looks more realistic, it can help conceal any residual names (a privacy de-identification technique sometimes referred to as “hiding in plain sight”)

An example of this:

Input:

[Agent] Hi, my name is Jason, can I have your name?

[Customer] My name is Valeria

[Agent] In case we need to contact you, what is your email address?

[Customer] My email is v.racer@example.org

[Agent] Thank you. How can I help you?

De-identified Output:

[Agent] Hi, my name is Gavaia, can I have your name?

[Customer] My name is Bijal

[Agent] In case we need to contact you, what is your email address?

[Customer] My email is happy.elephant44@example.org

[Agent] Thank you. How can I help you?

As you can see in the output, the names and email addresses have been replaced with a random value that both protects the original sensitive information but also makes the output look more realistic. This can make the data more useful and help “hide” any residual PII.

Next Steps:

To learn more about De-Identification check out our Technical Docs, try De-identification of Storage in the Cloud Console and Watch a recent Google I/O talk on De-identification of data.

Creating a secure-by-default experience based on robust defenses has always been a core design principle for Gmail. That’s why we’ve established a strong baseline of security in Gmail, with built-in protections to help automatically filter out potentially malicious messages. While these defenses help keep Gmail users safe, email functions as part of a large, complex, interconnected ecosystem that we continually invest in and work to protect. After first announcing Gmail’s Brand Indicators for Message Identification (BIMI) pilot last year, today we’re announcing that over the coming weeks we’re rolling out Gmail’s general support of BIMI, an industry standard that aims to drive adoption of strong sender authentication for the entire email ecosystem. BIMI provides email recipients and email security systems increased confidence in the source of emails, and enables senders to provide their audience with a more immersive experience.

“Bank of America has a wide range of security measures in place to support our customers, and we constantly evolve our program to deliver best in class protection. Part of this effort is our partnership with Google on BIMI, which provides an easy way to validate if correspondence is from us.” — Bank of America

BIMI enables organizations that authenticate their emails using Domain-based Message Authentication, Reporting, and Conformance (DMARC)—a standard for providing strong sender authentication that allows security systems to perform better filtering, separating legitimate messages from potentially spoofed ones—to validate ownership of their logos and securely transmit them to Google. BIMI is designed to be easy: for organizations with DMARC in place, validated logos display on authenticated emails from their domains and subdomains.

Here’s how it works: Organizations who authenticate their emails using Sender Policy Framework (SPF) or Domain Keys Identified Mail (DKIM) and deploy DMARC can provide their validated trademarked logos to Google via a Verified Mark Certificate (VMC). BIMI leverages Mark Verifying Authorities, like Certification Authorities, to verify logo ownership and provide proof of verification in a VMC. Once these authenticated emails pass our other anti-abuse checks, Gmail will start displaying the logo in the existing avatar slot.  

“Gmail’s support of BIMI is a win for email authentication, brand trust, and consumers alike. BIMI gives organizations the opportunity to provide their customers with a more immersive email experience, strengthening email sender authentication across the entire email ecosystem.” — Seth Blank, Chair of the AuthIndicators Working Group

This is just the start for BIMI. The standard expects to expand support across logo types and validators. For logo validation, BIMI is starting by supporting the validation of trademarked logos, since they are a common target of impersonation. Today, Entrust and DigiCert support BIMI as Certification Authorities, and in the future the BIMI working group expects this list of supporting validation authorities to expand further. To learn more about BIMI and see the latest news, visit the working group’s website.

To take advantage of BIMI, ensure that your organization has adopted DMARC, and that you have validated your logo with a VMC. For Gmail users, no action is required. We’re proud to be one of the leading members in both establishing and supporting the BIMI standard and will continue to support efforts that contribute to security for the entire email ecosystem.

In the US alone, 24.5 million school records have been leaked across 1,327 data breaches since 2005. And when +1.3 billion students moved to remote learning, the pandemic became an accelerant for cyber attacks with the number of attacks in education spiking 30% YoY during July and August 2020. In our experience working with education customers, we’ve seen three areas impacted by cyber attacks:

• Financial: Institutions experience financial loss from operational disruption and demands from cyber criminals who stole data. For example, The University of California, San Francisco confirmed it paid a ransom of $1.14 million to the criminals behind a cyber attack on its School of Medicine. To protect sensitive student data, the federal government released a statement encouraging all postsecondary institutions to implement NIST 800-171 controls, a policy designed to protect sensitive student data. 
• Operational:Cyber attacks can prevent students, faculty and staff from accessing systems that are necessary to continue teaching, learning, and research initiatives, bringing operations to a halt. Hartford Public Schools in Connecticut postponed its first day of classes following a ransomware attack that shut down the district’s system.
• Reputational: Institutions who have financial or student data breaches can suffer from lower enrollment rates and a loss of grant funding. A study conducted by the Ponemon Institute pointed out that higher education institutions are judged largely on their reputation. A single data breach can significantly impact the reputation of the institution, in addition to substantial financial implications.

How is Google Cloud helping educational institutions put security in place before a data breach happens?

Google Cloud’s security model model, global infrastructure, and unique capability to innovate is helping academic institutions keep their organizations secure and in compliance. For example, Brown University leverages Google Workspace for Education to protect information sharing among faculty, students, and staff, while reducing IT maintenance costs.

Google Workspace helps establish each user’s identity in Google Cloud and this feature has become a core component of Google’s Zero Trust security model, implemented through BeyondCorp. By shifting access controls from the network perimeter to individual users, BeyondCorp enables secure work from virtually any location without the need for a traditional VPN. 

Most academic institutions look to minimize their risk in a shared responsibility model when running infrastructure-as-a-service or platform-as-a-service workloads. Google Cloud has partnered with industry-leading security consortia like RHEDCloud to understand the specific needs of educational and academic research institutions, and follow applicable regulations and grant requirements. We have co-developed solutions with partners, like Burwood Group, to meet those needs and help our customers automate secure and compliant environments for their researchers, students and staff. Burwood Group has partnered with 27 top-tier research universities (R-1) to deploy security solutions. Here are a few of the products we’ve seen the most interest in:

• Blueprint scripts uses Google Cloud built-in services to automate the provisioning and management of enterprise applications and research environments to be in compliance with HIPAA, FedRAMP, CUI, NIST 800-53, NIST CSF, and GDPR regulations.
• Security Command Center manages assets in your organization, uncovers vulnerabilities and threats, and reviews your organization’s compliance.  The solution generates security alerts and audit events from logs that integrate with existing enterprise systems like Security Information and Event Management (SIEMs), Endpoint Detection & Response (EDR’s) and Security, Orchestration, Automation, & Response  (SOAR) tools.
• Chronicle, Google Cloud’s next generation SIEM platform, is changing the game. It is a cloud-native platform designed to ingest your organization’s logs for a flat rate, abandoning the tiered model most SIEM providers offer. It can sift through petabytes of data in seconds and is useful for organizations that are both cost and security conscious.
• VirusTotal Premium provides access to the world’s largest corpus of threat data to protect your organization proactively. The VT API can easily integrate into your commonly used SIEM, EDR and SOAR tools, alerting you to threats that would otherwise go unnoticed.
• reCAPTCHA, helps defend against common attack patterns such as scraping or credential stuffing in web applications.
• Cloud Armor defends against distributed denial-of-service (DDoS) attacks.

For research workloads, these tools provide controls for data ingestion, the export of research data, and, sharing permissions and auditability, allowing researchers to collaborate with other institutions. These tools also automate the provisioning of environments to provide secure execution of Jupyter notebooks and other applications commonly used in research, like Matlab, SAS and Python/R environments.

A customized approach that aligns with education-specific needs

Google Cloud has partnered with academic institutions and research groups to understand their specific needs, and co-develop solutions that help customers govern, control, and audit security and compliance for all types of workloads in a shared responsibility framework. These solutions integrate with most common enterprise and security systems and permit customization to accommodate specific needs around security or data sharing (e.g grant requirements).

To learn more, watch our latest on-demand webinar, Cloud Security Best Practices for Higher Education, hosted by Google Cloud and our partners, Burwood Group and Carahsoft.

Can you keep a secret? Many people struggle to keep secrets. So do organizations. An organization’s digital credentials — its passwords, API keys, tokens, and encryption keys baked into its software code — are its vital secrets, used to authenticate access to protected resources and services.    

The security of these secrets represent one of the biggest risks that enterprises must manage when securing application infrastructure and environments. Fortunately, we have an important tool to help our customers better manage their secrets: Google Cloud Secret Manager. 

Secret Manager is a secure and efficient tool to centrally store, access, manage, and audit your organization’s secrets. As we noted in our April 2023 Threat Horizons Report, insufficient management of credentials and keys is now one of the top security issues facing cloud customers today. In particular, we have observed service account keys being used for everything from insider threats to cross-cloud breaches to intellectual property espionage.

Along with the increasing popularity of DevOps, secrets are used throughout development. In some cases, secrets are coded as plaintext into configuration files or directly into functions, where they can end up exposed in code repositories. If compromised, secrets can be used to gain unauthorized privileged access to enterprise data, infiltrate critical systems, and launch attacks.

Secret Manager can help organizations better manage their secrets even as concerns over potential security breaches grow. Organizations must consider the following four points during their secrets management risk analysis: 

1. Secrets sprawl: As organizations move more workloads to the cloud, the number of secrets they need to manage increases exponentially. This can lead to secrets sprawl, where secrets are stored in a variety of different locations, making it difficult to track and manage them.
2. Human error: Humans are fallible, and this is especially true when it comes to managing secrets. Mistakes, such as accidentally sharing a secret with the wrong person or the misconfiguration of a secret management tool, can have serious consequences.
3. Compliance violations: Organizations that handle sensitive data are subject to a variety of compliance regulations, including PCI-DSS, SOC 2, HIPAA, and GDPR. Without proper management, secrets could lead to potential compliance violations. 
4. Downtime: If secrets are not properly managed or rotated, it could lead to downtime for applications and services that rely on them.

Secret Manager can enable IT teams to keep track of secrets by providing a centralized place to store and manage them. You can access secrets directly from the Secret Manager console, or use the Secret Manager API or SDKs. Secret Manager is also designed to scale with your application. You can store as many secrets as you need without performance issues, and its API-first design lets you extend and integrate into other existing systems. 

Secret Manager integrations in Google Cloud

In many cases, developers save secret configuration details as plaintext in code for ease of use. There are two main risks associated with this approach:

• Anyone with access to secrets can see and copy them. There is no way to enforce any auditing or access controls practices here.
• It becomes difficult to move code between dev, test, and production environments.

Cloud-centered integrations supported by Secret Manager with other Google Cloud services in the software supply chain make it easier and safer to store and access sensitive information. 

The tool is about the secret information being made securely available in build or runtime.

Secret Manager integrations also can play a role in securing sensitive information in the software supply chain:

1. Integration with CI/CD systems used to build and deploy your software

Cloud Code integration can help developers build more secure applications by preventing hard-coding of sensitive configuration data such as passwords, credentials, and tokens in the codebase. It replaces them with secrets which can be programmatically fetched as required. You can create, access, and modify secrets directly from within your favorite IDE without having to navigate away from it. Integration is available for VS Code, IntelliJ, and Cloud Shell Editor.

Cloud Functions integration lets you access and expose secrets as environment variables or as a volume via the filesystem.

Cloud Build: To handle sensitive data in builds, you can store them in Secret Manager and then configure the build steps to access the information directly. In the build steps, you can associate the value of a secret to an environment and you can access this value via an environment variable from scripts or processes. Two common uses for fetching sensitive dependencies during build time are:

• Storing Docker authentication credentials for pull/push of images to Docker hub.
• Creating GitHub pull requests in response to builds.

2. Integration with runtime engines

Cloud Run: Native integration makes it easy to mount secrets in cloud run services. Using this integration, secrets can be made available to containers either as volume mounts or as env variables. Sensitive dependencies can be stored in environment variables, by referencing a secret name stored in Secret Manager instead of storing it as a plain text environment variable. This secret access operation can be done directly from the Cloud Run console without changing a single line of source code. 

If code reads secrets from a file instead of an env variable, the “mounted as volume” option can be used to make secrets look like files to the env vars. During service deployment, the service account used to run the containers are checked to see if they have the required permissions or not, preventing any unwanted access.

Kubernetes and GKE: The CSI driver–provider solution is an open-source work that provides integration with K8s. The driver lets customers centrally store their secrets in Google Secret Manager and consume them in a kubernetes native manner, as files mounted in K8s pods. The solution also comes with additional syncing capabilities. This method of consuming secrets also allows users to take advantage of SM’s capabilities like versioning, managed rotation, default/CMEK encryption and cloud-focused integrations with other Google Cloud services. Work is currently ongoing to provide official support to the driver solution as a GKE component.

Is Secret Manager right for my organization?

Developers carry a big responsibility to protect the sensitive data they are entrusted to work with, and keep secrets out of their code. However, organizations also have a responsibility to provide the tools and solutions needed to help their teams deliver on these expectations. Secret Manager makes it straightforward for developers to store and manage secrets while improving security. 

To learn more about Secret Manager, please review these resources: 

• Overview of Secret Manager
• Manage your Cloud Run secrets securely with Secret Manager (video)

Read the previous Best Kept Security Secrets blogs here:

• How VPC Service Controls can help build defense in depth
• Tap into the power of Organization Policy Service
• How Cloud EKM can help resolve the cloud trust paradox
• How Assured Workloads accelerates security and compliance