Since 2014, tech companies have relied on metrics to identify trouble spots, establish baselines, and measure meaningful progress in diversity, including publishing DEI data directly through diversity annual reports. However, as our understanding of DEI has evolved over time each company’s report has diverged, creating a fragmented landscape of industry-wide data. This separation is problematic as no single company’s diversity dataset can solve tech’s DEI challenges. Instead, we need to build industry-wide systemic solutions to create sustainable change, and those start with establishing a common language for DEI data and standardizing and sharing DEI data across tech companies.

The challenge is most companies publish diversity data in a way that makes it difficult to analyze, such as using bar graphs and pie charts. Researchers, as a result, are unable to easily pull the real, detailed data or aggregate and analyze it for their unique needs (e.g. hiring trends for tech vs. nontechnical roles, representation of Asian women in leadership, and more). As we shared in our most recent Diversity Annual Report, external research shows that sustainable change will only come from solutions that encompass the entire tech industry and data transparency is a critical step in this work.

Google’s Diversity Annual Report public dataset & BigQuery

In May, we released our 2022 Diversity Annual Report, which includes demographic data on workforce representation, hiring, and attrition of employees at Google, including leadership. You can see our hiring data by race/ethnicity, gender, and intersectional hiring over time, by region, and more. In an effort to make this data more transparent and accessible for analysis, we have added it as a public dataset in BigQuery, Google Cloud’s powerful data warehousing tool. Our data is now even easier for researchers, community organizations, and industry groups to leverage and compare against external benchmarks to help contextualize our progress.

As one of the first in the industry, we are proud to have published our diversity data on BigQuery for the second year in a row. Our dataset, among others, is public, stored and paid for by Google so those who are interested can use BigQuery’s advanced analytic capabilities through the Google Cloud Public Dataset Program for free*.

Contextualizing diversity data is necessary to make meaningful conclusions

While Google’s diversity dataset can help users compare their own datasets to Google’s current and historical trends, DEI data is only useful when analyzed in the context of other relevant datasets. For example, concluding that Black+ hiring has increased from 8.8% to 9.4% has little meaning unless there is a point of reference, like US general census data, labor force participation rates or graduation rates. It’s why Google also includes other public datasets in BigQuery such as related industry DEI data, talent and graduation pools. By doing so, users can run a sample query that then compares Google’s hiring and representation to related industries (software publishers, data process services, etc.). Users can then better understand and contextualize areas of progress and opportunity, and they can more objectively identify where organizations can take a proactive role in addressing not only DEI in their companies, but also in the communities in which they call home.

Intersectional data and disaggregated baseline metrics

In the workplace, intersectional data is key to understanding the layers of exclusion and inequity that may exist for certain groups. This includes those with social identity overlap, like race, gender, and LGBTQ+, that create multiple levels of inequality or discrimination. It’s critical for DEI data to be disaggregated in a meaningful way to diagnose the true health of a system and to better understand how intersectionality contributes to the greater DEI landscape .

Google’s Diversity Annual Report public dataset includes Google’s intersectional hiring and representation data also broken down by tech, non-tech, and leadership roles. BigQuery’s friendly interface makes it easy to select the relevant parameters and join this data with other public or private datasets to help meaningfully contextualize Google’s data against the broader industry. Anyone from data scientists to DEI stewards can launch public datasets from the Google Cloud Marketplace and start querying them right away. Findings can be visualized through tools like Looker, Data Studio, or Tableau.

This is just the beginning

As DEI work continues to evolve into an industry-wide approach, we must encourage a standard of practice for collecting and reporting data across the board. At scale, data has the power to enable the tech industry to make real improvements collectively, in addition to inside our individual companies. We hope our dataset and sample queries give researchers and individuals a launchpad to become DEI practitioners of the tech industry, and ensure we have the right data to solve the right problems.

Footnote:

*The only time anyone would need to pay is for queries performed on the data after BigQuery’s 1TB/mo of free tier processing. This means that each user receives 1TB of free BigQuery processing every month, which can be used to run queries on this public dataset.

At Redisconf 2018 in San Francisco last month, we announced the public beta of Cloud Memorystore for Redis, a fully-managed in-memory data store service. Today, the public beta is available for everyone to try. Cloud Memorystore provides a scalable, more secure and highly available Redis service fully managed by Google. It’s fully compatible with open source Redis, letting you migrate your applications to Google Cloud Platform (GCP) with zero code changes.

As more and more applications need to process data in real-time, you may want a caching layer in your infrastructure to reduce latency for your applications. Redis delivers fast in-memory caching, support for powerful data structures and features like persistence, replication and pub-sub. For example, data structures like sorted sets make it easy to maintain counters and are widely used to implement gaming leaderboards. Whether it’s simple session caching, developing games played by millions of users or building fast analytical pipelines, developers want to leverage the power of Redis without having to worry about VMs, patches, upgrades, firewall rules, etc.

Early adopters of Cloud Memorystore have been using the service for the last few months and they are thrilled with the service.

Simple and flexible provisioning
How you choose to deploy Cloud Memorystore for Redis depends on the availability and performance needs of your application. You can deploy Redis as a standalone instance or with a replica to provide high availability. But while replicating a Redis instance provides only data redundancy, you still need to do the heavy lifting of health checking, electing of a primary, client connections on failover, etc. The Cloud Memorystore service takes away all this complexity and makes it easy for you deploy a Redis instance that meets your application’s needs.

Cloud Memorystore provides two tiers of service, Basic and Standard, each with different availability characteristics. Regardless of the tier of service, you can provision a Redis instance as small as 1 GB up to 300 GB. With network throughput up to 12 Gbps, Cloud Memorystore supports applications with very high bandwidth needs.

Here is a summary of the capabilities of each tier:

Provisioning a Cloud Memorystore instance is simple: just choose a tier, the size you need to support the instance availability and performance needs, and the region. Your Redis instance will be up and running within a few minutes.

“Lift and shift” applications
Once provisioned, using Cloud Memorystore is a breeze. You can connect to the Redis instance using any of the tools and libraries you commonly use in your environment. Cloud Memorystore clients makes use of IP addresses to connect to the instance. Applications always connect to one IP address and Cloud Memorystore ensures the traffic is directed to the primary in case there is a failover.

Other key features
Whether it’s provisioning, monitoring or scaling memory, Cloud Memorystore simplifies common management tasks.

Security
Open-source Redis has very minimal security, and as a developer or administrator, it can be challenging to ensure all Redis instances in your organization are protected. With Cloud Memorystore, Redis instances are deployed using a private IP address, which prevents the instance from being accessed from the internet. You can also use Cloud Identity & Access Management (IAM) roles to ensure granular access for managing the instance. Additionally, authorized networks ensure that the Redis instance is accessible only when connected to the authorized VPC network.

Stackdriver integration
Cloud Memorystore instances publish all the key metrics into Stackdriver, Google Cloud’s monitoring and management suite. You can monitor all of your instances from the Stackdriver dashboard, and use Stackdriver Logging to get more insights about the Redis instances

Seamless memory scaling
When a mobile application goes viral, it may be necessary to provision a larger Redis instance to meet latency and throughput needs. With Cloud Memorystore you can scale up the instance with a few clicks, and the Standard High Availability tier lets you scale the instance with minimal disruption to the application.

On-demand pricing
Cloud Memorystore provides on-demand pricing with no upfront cost and has per second billing. Moreover, there is no charge for network traffic coming in and out of a Cloud Memorystore instance. For more information, refer to Cloud Memorystore pricing.

Coming soon to Cloud Memorystore
This Cloud Memorystore public beta release is just a starting point for us. Here is a preview of some of the features that are coming soon.

• New region support
• Import-Export of RDB files into a Google Cloud Storage bucket
• Support for Redis 4.0
• Support for Redis Cluster

We are excited about what is upcoming for Cloud Memorystore and we would love to hear your feedback! If you have any requests or suggestions, please let us know through Issue Tracker. You can also join the conversation at Cloud Memorystore discussion group.

Sign up for a $300 credit to try Cloud Memorystore and the rest of GCP. Start with a small Redis instance for testing and development, and then when you’re ready, scale up to serve performance-intensive applications.

Want to learn more? Register for the upcoming webinar on Tuesday, June 26th 9:00 am PT to hear all about Cloud Memorystore for Redis.

Over one third of organizations believe that data analytics and machine learning have the most potential to significantly alter the way they run business over the next 3 to 5 years. However, only 26% of organizations are data driven. One of the biggest reasons for this gap is that a major portion of the data generated today is unstructured, which includes images, documents, and videos. It is estimated to cover roughly up to 80% of all data, which has so far remained untapped by organizations.

One of the goals of Google’s data cloud is to help customers realize value from data of all types and formats. Earlier this year, we announced BigLake, which unifies data lakes and warehouses under a single management framework, enabling you to analyze, search, secure, govern and share unstructured data using BigQuery.

At Next ‘22, we announced the preview of object tables, a new table type in BigQuery that provides a structured record interface for unstructured data stored in Google Cloud Storage. This enables you to directly run analytics and machine learning on images, audio, documents and other file types using existing frameworks like SQL and remote functions natively in BigQuery itself. Object tables also extend our best practices of securing, sharing and governing structured data to unstructured, without needing to learn or deploy new tools.

Directly process unstructured data using BigQuery ML

Object tables contain metadata such as URI (Uniform Resource Identifier), content type, and size that can be queried just like other BigQuery tables. You can then derive inferences using machine learning models on unstructured data with BigQuery ML. As part of preview, you can import open source TensorFlow Hub image models, or your own custom models to annotate the images. Very soon, we plan to enable this for audio, video, text and many other formats, and pre-trained models to enable out-of-the box analysis. Check out this video to learn more and watch a demo.

Create an object table

CREATE EXTERNAL TABLE my_dataset.object_table
WITH CONNECTION us.my_connection
OPTIONS(uris=["gs://mybucket/images/*.jpg"],
object_metadata="SIMPLE", metadata_cache_mode="AUTOMATIC");
​
# Generate inferences with BQML
SELECT * FROM ML.PREDICT(
MODEL my_dataset.vision_model,
(SELECT ML.DECODE_IMAGE(data) AS img FROM my_dataset.object_table)
);

By analyzing unstructured data natively in BigQuery, businesses can

• Eliminate manual effort as pre-processing steps such as tuning image sizes to model requirements are automated
• Leverage the simple and familiar SQL interface to quickly gain insights
• Save costs by utilizing existing BigQuery slots without needing to provision new forms of compute

Adswerve is a leading Google Marketing, Analytics and Cloud partner on a mission to humanize data. Twiddy & Co. is Adswerve’s client – a vacation rental company in North Carolina. By combining structured and unstructured data, Twiddy and Adswerve used BigQuery ML to analyze images of rental listings and predict the click-through rate, enabling data-driven photo editorial decisions.

“Twiddy now has the capability to use advanced image analysis to stay competitive in an ever changing landscape of vacation rental providers – and can do this using their in-house SQL skills.” said Pat Grady, Technology Evangelist, Adswerve

Process unstructured data using remote functions

Customers today use remote functions (UDFs) to process structured data for languages and libraries that are not supported in BigQuery. We are extending this capability to process unstructured data using object tables.

Object tables provide signed URLs to allow remote UDFs running on Cloud Functions or Cloud Run to process the object table content. This is particularly useful for running Google’s pre-trained AI models, including Vision AI, Speech-to-Text, Document AI, open source libraries such as Apache Tika, or deploying your own custom models where performance SLAs are important.

Here’s an example of an object table being created over PDF files that are parsed using an open source library running as a remote UDF.

SELECT uri, extract_title(samples.parse_tika(signed_url)) AS title<br>FROM EXTERNAL_OBJECT_TRANSFORM(TABLE pdf_files_object_table,<br>["SIGNED_URL"]);

Extending more BigQuery capabilities to unstructured data

Business intelligence – The results of analyzing unstructured data either directly in BigQuery ML or via UDFs can be combined with your structured data to build unified reports using Looker Studio (at no charge), Looker or any of your preferred BI solutions. This allows you to gain more comprehensive business insights. For example, online retailers can analyze product return rates by correlating them with the images of defective products. Similarly, digital advertisers can correlate ad performance with various attributes of ad creatives to make more informed decisions.

BigQuery search index – Customers are increasingly using the search functionality of BigQuery to power search use cases. These capabilities now extend to unstructured data analytics as well. Whether you use BigQueryML to produce inference on images or use remote UDFs with Doc AI to produce document extraction, the results can now be search indexed and used to support search access patterns.

Here’s an example of search index on data that is parsed from PDF files:

CREATE SEARCH INDEX my_index ON pdf_text_extract(ALL COLUMNS);
​
SELECT * FROM pdf_text_extract WHERE SEARCH(pdf_text, "Google");

Security and governance – We are extending BigQuery’s row-level security capabilities to help you secure objects in Google Cloud Storage. By securing specific rows in an object table, you can restrict the ability of end users to retrieve the signed URLs of corresponding URIs present in the table. This is a shared responsibility security model, for which administrators need to ensure that end users don’t have direct access to Google Cloud Storage, and use signed URLs from object tables as the only access mechanism.

Here’s an example of a policy for PII images that are secured to be first processed through a blur pipeline:

CREATE ROW ACCESS POLICY pii_data ON object_table_images
GRANT TO ("group:admin@example.com")
FILTER USING (ARRAY_LENGTH(metadata)=1 AND
metadata[OFFSET(0)].name="face_detected")

Soon, Dataplex will support object tables, allowing you to automatically create object tables in BigQuery and manage and govern unstructured data at scale.

Data sharing – You can now use Analytics Hub to share unstructured data with partners, customers and suppliers while not compromising on security and governance. Subscribers can consume the rows of object tables that are shared with them, and use signed URLs for unstructured data objects.

Getting Started

Submit this form to try these new capabilities that unlock the power of your unstructured data in BigQuery. Watch this demo to learn more about these new capabilities.

Special thanks to engineering leaders Amir Hormati, Justin Levandoski and Yuri Volobuev for contributing to this post.

Access to the cloud has advanced the distribution and consumption of financial information on a global scale. In parallel, the global financial data landscape has been transformed by an influx of alternative data sources, including social media, meteorological data, satellite imagery, and other data. Exchanges and market data providers now find they need to include these new datasets to enrich their products and compete, which has meant they now must consider cloud-based models to keep up with the demands of their customers who expect easy, quick, flexible and cost-efficient ways to consume market data.

To address these needs, today we’re announcing the general availability of Datashare for financial services, a new Google Cloud solution that brings together the entire capital markets ecosystem—data publishers, and data consumers—to exchange market data securely and easily.

Datashare helps organize third-party financial information, making it accessible and useful to market data publishers and data consumers. We open-sourced the entire Datashare solution so market data publishers can now onboard their licensed datasets to Google Cloud securely, quickly and easily, while data consumers can consume that data as a service in tools of their preference, such as BigQuery.

Three ways to distribute and consume your data

Batch data delivery

Datashare provides a batch data delivery mechanism for data publishers to deliver their reference data, historical tick data, alternative market data sources and more via BigQuery, reducing the administrative burden on data consumers to extract insights from data. 

Real-time data streaming delivery

By using this event-based data delivery channel for rapidly changing instrument prices, tick data, orders, news and others via Pub/Sub, data consumers can reliably process individual messages or rewind to a point in time to replay a prior market scenario and test model changes.

Monetizing licensed datasets

Market data publishers can onboard their licensed datasets to Google Cloud and make them available via a one-stop-shop on Google Cloud Marketplace, enabling a new sales channel to expand market reach.

Reference architecture

Check out the diagram below to see how you can share your batch and real-time data directly to your Google Cloud customers with BigQuery and Pub/Sub.

As you can see in the above reference architecture, both publishers and consumers can derive several benefits from the solution:

Benefits for data publishers

• You no longer have to maintain your own delivery and licensing infrastructure.
• You can easily package and deliver granular data products and experiments with SQL.
• You can have a solution that scales with your business as data volumes and number of customers grow.

Benefits for data consumers

• Your data is ready for analysis and machine learning (ML)— you no longer have to maintain extract, transform, and load (ETL) pipelines to load files and transform data.
• You can avoid the expense and burden of maintaining multiple copies of large data files.
• You can be more targeted with consumption of data using BigQuery queries, improving performance, and compliance, and reducing cost.

Accessing the datasets

Google Cloud has been working with multiple industry firms on innovating in the market data space. By using Datashare for publishing, data publishers can make their entire datasets available on Google Cloud. Early adopters of Datashare include firms such as OneTick and Accern. OneTick’s datasets include reference and historical futures data (that can be accessed in our console with your login). Accern’s datasets include alternative data such as market sentiment and credit analysis data (that can be accessed in our console with your login).

To make it more helpful, we partnered with Accern to create a hypothetical scenario to describe the data acquisition and analytics process step-by-step.

Accern use case 

As a sustainability analyst, you require an economic, social and governance (ESG) dataset to determine which sector is the most widely covered ESG sector by analysts, and to also identify the sector with the lowest ESG sentiment score. Now, you can discover and acquire an ESG dataset in Google Cloud.

Step 1. Navigate to the Financial Services solutions page in the Google Cloud console:

Step 2. Click a dataset, for example Accern AI-Generated ESG Insights, then review the overview details, plans and pricing, documentation and support information. To view the available pricing tiers, click ‘View All Plans’. Once you’ve decided on a tier that you would like to subscribe to, click ‘Select’, choose a billing account and review and accept the terms of service to complete the subscription. Once the steps are complete, click ‘Subscribe’ at the bottom. An overlay window will appear, click ‘Register with Accern’ to activate and complete the subscription.

Step 3. Once activation is complete, you’ll be directed to the Datashare ‘My Products’ screen. Voila! You are now subscribed to Accern’s ESG Scores dataset and can access it in your Google Cloud instance using BigQuery. To access the data, click the hour glass icon on the corresponding ‘My Products’ record that you just purchased. An overlay will present you with the details on the dataset and/or table. Click the ‘Navigate to Table’ button to navigate through to the BigQuery console.

Step 4. Now that you have access and are in the BigQuery console, it’s time to generate data insights.

^{For this example, we’ve eliminated the company identifying information that is included as part of the subscription and aggregated company ESG in a view where each row represents a day, an industry sector, a specific identified ‘ESG Issues’ (event_group and event) and the respective ‘ESG Sentiment’ per issue.}

For example, row 1 indicates that within the ‘Healthcare’ sector, there was a ‘Social – Civil Society’ issue identified and it had a negative ESG sentiment score of -15.35.

Step 5. Generate a report by exporting it to Data Studio to build visualizations and conduct additional analysis on the ESG data.

Select ‘Export’ and ‘Explore with Data Studio’.

Step 6. Build a simple/basic report.

Now that the ESG data appears in Data Studio, you can start by building a simple chart to help you understand which industry sectors have the highest volume of discussions around ESG and the overall ESG Sentiment per industry sector.

To build the chart:

• Select the chart type ‘Table’.

• Include Entity_Sector as your dimension to aggregate results by ‘Industry Sector.’
• Include Signal_ID as a measure to count the number of ESG passages identified per ‘Industry Sector.’
• Include AVG(Event_Sentiment) as a measure to display the overall ESG Sentiment per ‘Industry Sector’ across ESG Issues.

You can see sectors that are  most discussed when it comes to ESG related topics and their corresponding ‘ESG Sentiment’ scores.

Step 7. Build your final report in Data Studio.

As a next step you can further drill into the data to understand ESG data specific to each ‘Industry Sector’ and identify positive and negative ESG practices. 

Accern has built a more complex sample dashboard and made it available publicly here. You can interact with this report and play around with the data. The dashboard can help to identify material ESG insights for each sector to inform your investment and risk processes. If you have additional questions, you can reach out to Accern directly.

Discovering, accessing and analyzing licensed datasets is quick and easy. Stay tuned for more updates on new licensed datasets.

Publishing your data via Datashare

If you are a publisher of market data, alternative, or exotic data, you can use Datashare to get it published on Google Cloud Marketplace.

Start by joining the Partner Advantage program by registering for the Partner Advantage Portal and applying for the Partner Advantage Build Model engagement. Visit our getting started guide for information to get started on publishing licensed datasets in the Marketplace. Stay tuned for a future blog post about using Datashare to publish datasets in the Marketplace.

More solutions for capital markets

Check out other Google Cloud solutions for capital markets.