Customers tell us that sharing and exchanging data with other organizations is a critical element of their analytics strategy, but it’s hamstrung by unreliable data and processes, and only getting harder with security threats and privacy regulations on the rise. 

Furthermore, traditional data sharing techniques use batch data pipelines that are expensive to run, create late arriving data, and can break with any changes to the source data. They also create multiple copies of data, which brings unnecessary costs and can bypass data governance processes. These techniques do not offer features for data monetization, such as managing subscriptions and entitlements. Altogether, these challenges mean that organizations are unable to realize the full potential of transforming their business with shared data.

To address these limitations, we are introducing Analytics Hub, a new fully managed service, available in Q3, in preview, that helps you unlock the value of data sharing, leading to new insights and increased business value. With Analytics Hub you get:

• A rich data ecosystem by publishing and subscribing to analytics-ready datasets. 
• Control and monitoring over how your data is being used, because data is shared in one place.
• A self-service way to access valuable and trusted data assets, including data provided by Google. For example, a unique dataset from Google Search Trends will be available, that you can query and combine with your own data.
• An easy way to monetize your data assets without the overhead of building and managing the infrastructure. 

Built on a decade of cross-organizational sharing

While Analytics Hub is a new service, it builds on BigQuery, Google’s petabyte-scale, serverless cloud data warehouse. BigQuery’s unique architecture provides separation between compute and storage, enabling data publishers to share data with as many subscribers as you want without having to make multiple copies of your data. With BigQuery, there are no servers to deploy or manage, which means that data consumers get immediate value from shared data. Data can be provided and consumed in real-time using the streaming capabilities of BigQuery and you can leverage the built in machine learning, geospatial, and natural language capabilities of BigQuery or take advantage of the native business intelligence support with tools like Looker, Google Sheets, and Data Studio.

BigQuery has had cross-organizational, in-place data sharing capabilities since it was introduced in 2010. We took a look at usage metrics in BigQuery and found that over a 7 day period in April, we had over 3,000 different organizations sharing over 200 petabytes of data. These numbers don’t include data sharing between departments within the same organization.

As you can see, data sharing in BigQuery is already popular. But we want to make it easier and even more scalable.

Raising the bar on data sharing 

To make data sharing easier and more scalable in BigQuery, Analytics Hub introduces the  concepts of shared datasets and exchanges. As a data publisher, you create shared datasets that contain the views of data that you want to deliver to your subscribers. Next, you create exchanges, which are used to organize and secure shared datasets. By default, exchanges are completely private, which means that only the users and groups that you give access to can view or subscribe to the data. You can also create internal exchanges or leverage public exchanges provided by Google. Finally, you publish shared datasets into an exchange to make them available to subscribers. 

Data subscribers search through the datasets that are available across all exchanges for which they have access and subscribe to relevant datasets. This creates a linked dataset in their project that they can query and join with their own data. Subscribers pay for the queries that they run against the data while the publisher pays for the storage of the data. Data providers can add new data, new tables, or new columns to the shared dataset and these will be immediately available to subscribers. In addition, the publisher can track subscribers, disable subscriptions, and see aggregated usage information for the shared data. 

Analytics Hub makes it easy for you to publish, discover, and subscribe to valuable datasets that you can combine with your own data to derive unique insights. Here are some types of data that will be available through Analytics Hub:

• Public datasets: Easy access to the existing repository of over 200 public datasets, including data about weather and climate, cryptocurrency, healthcare and life sciences, and transportation. 
• Google datasets: Unique, freely-available datasets from Google. One example of this is the COVID-19 community mobility dataset. Another example is the forthcoming Google Trends dataset, which will provide the top 25 search terms and top 25 rising search terms over a 5 year window in 210 distinct locations in the US. Trends data can be used by everyone in the organization to gain insights into what customers care about.
• Commercial (paid for) datasets: We are working with leading commercial data providers to bring their data products to Analytics Hub. If you are interested in delivering your data via Analytics Hub, we’re also introducing Data Gravity, an initiative that provides storage benefits and new distribution paths for data published through Analytics Hub. 
• Internal datasets: We know that data sharing can be challenging in larger organizations. Analytics Hub can be used for internal data, for example, to share standardized customer demographics with your sales engineering and data science teams.

Customers and partners using Analytics Hub

“Google Search Trends data has always been an important tool for our WPP agency data teams. At WPP we believe that data variety is a superpower which is why we are excited to use the new Trends dataset availability within BigQuery, plus the launch of Analytics Hub. The best creativity in the world is informed by data insights, and influenced by what people search for, so the operational efficiencies we’ll gain via the Analytics Hub and the insights we can drive with Trends data are just phenomenal.”
—Di Mayze Global Head of Data and AI, WPP

“Equifax Ignite is our shared data analytics environment within our Equifax data fabric. We are excited to partner with Google to leverage Analytics Hub and BigQuery to deliver data to over 400 statisticians and data modelers as well as securely sharing data with our partner financial institutions.”
—Kumar Menon, SVP Data Fabric and Decision Science, Equifax

“The flow of data and insights between our teams at Deloitte and our clients is paramount for building truly transformational data cultures. With its purpose-built architecture for secure data exchanges and sharing analytics resources, Google Cloud’s Analytics Hub can help provide significant operational efficiencies for how Deloitte teams support our clients’ data-driven initiatives within their industry ecosystems. It will also help minimize the worries about scale, privacy and security, or the administrative burden associated with each.”
—Navin Warerkar, Managing Director, Deloitte Consulting LLP, and US Google Cloud Data & Analytics GTM Lead

“Crux Informatics is proud to partner with Google to support the launch of Analytics Hub, removing friction for those who need access to analytics-ready data. With thousands of datasets from over 140 sources, Crux Informatics will accelerate access to data on Analytics Hub and together provide a more efficient and cost effective solution to deliver datasets in Google Cloud’s ecosystem.”
—Will Freiberg, CEO, Crux Informatics

Next steps for Analytics Hub

This is just the beginning for Analytics Hub. As we get to preview and general availability, we will be adding additional capabilities, including workflows for publishing and subscribing, publishing analytics assets (Looker Blocks, Data Studio reports, Connected Google Sheets) along with the shared data, the ability for data publishers to specify query restrictions on the usage of their data, and making it easy for data publishers to create sandbox environments for subscribers to work with their data, even if they are not yet on Google Cloud. We will provide features in Analytics Hub for monetization of data, including managing subscriptions, data entitlements, and billing.

Please sign up for the preview, which is scheduled to be available in the third quarter of 2021. In the meantime, you can learn more about BigQuery and how to leverage its built-in data sharing capabilities. Please go to g.co/cloud/analytics-hub to register your interest in Analytics Hub.

Large Machine Learning (ML) models – such as large language models, generative AI, and vision models – are dramatically increasing the number of trainable parameters and are achieving state-of-the-art results. Increasing the number of parameters results in the model being too large to fit on a single VM instance thus demands distributed compute to spread the model across multiple nodes. Google Kubernetes Engine (GKE) has built-in support for NCCL Fast Socket, to help improve the time to train large ML models with distributed, multi-node clusters.

Enterprises are looking for faster and cheaper performance to train their ML models. With distributed training, communicating gradients across nodes is a performance bottleneck. Optimizing inter-node latency is critical to reduce training time and costs. Distributed training uses collective communication as a transport layer over the network between the multiple hosts. Collective communication primitives such as all-gather, all-reduce, broadcast, reduce, reduce-scatter, and point-to-point send and receive are used in distributed training in Machine Learning. 

The NVIDIA Collective Communication Library (NCCL) is commonly used by popular ML frameworks such as TensorFlow and PyTorch. It is a highly optimized implementation for high bandwidth and low latency between NVIDIA GPUs. Google developed a proprietary version of NCCL called NCCL Fast Socket to optimize performance for deep learning on Google Cloud.

NCCL Fast Socket uses a number of techniques to achieve better and more consistent NCCL performance.

• Use of multiple network flows to attain maximum throughput. NCCL Fast Socket introduces additional optimizations over NCCL’s built-in multi-stream support, including better overlapping of multiple communication requests.
• Dynamic load balancing of multiple network flows. NCCL can adapt to changing network and host conditions. With this optimization, straggler network flows will not significantly slow down the entire NCCL collective operation.
• Integration with Google Cloud’s Andromeda virtual network stack.This increases overall network throughput by avoiding contentions in virtual machines (VMs).

We tested (NVIDIA NCCL tests) the performance of NCCL Fast Socket vs NCCL on various machine shapes with 2 node GKE clusters.

The following chart shows the results. For each machine shape, the NCCL performance without Fast Socket is normalized to 1. In each case, using NCCL Fast Socket demonstrated increased performance in a range of 1.3 to 2.6 times faster internetwork communication speed.

As a built-in feature, GKE users can take advantage of NCCL Fast Socket without changing or recompiling their applications, ML frameworks (such as TensorFlow or PyTorch), or even the NCCL library itself. To start using NCCL Fast Socket, create a node pool that uses the plugin with the --enable-fast-socket and --enable-gvnic flags. You can also update an existing node pool using gcloud container node-pools update.

gcloud container node-pools create NODEPOOL_NAME \
    --accelerator type=ACCELERATOR_TYPE, count=ACCELERATOR_COUNT \
    --machine-type=MACHINE_TYPE \
    --cluster=CLUSTER_NAME \
    --enable-fast-socket \
    --enable-gvnic

To achieve better network throughput with NCCL, Google Virtual NICs (gVNICs) must be enabled when creating VM instances. For detailed instructions on how to use gVNICs, please refer to the gVNIC guide. 

To verify that NCCL Fast Socket has been enabled, view the kube-system pods:

kubectl get pods -n kube-system

And the output should b similar to:

NAME                         READY   STATUS    RESTARTS   AGE
fast-socket-installer-qvfdw  2/2     Running   0          10m
fast-socket-installer-rtjs4  2/2     Running   0          10m
fast-socket-installer-tm294  2/2     Running   0          10m

To learn more visit GKE NCCL Fast Socket documentation. We look forward to hearing how NCCL Fast Socket improves your ML Training experience on GKE.

NVIDIA GPU-powered instances on Google Cloud provide an optimal platform for organizations to develop their AI applications on the latest hardware and software stack, then seamlessly deploy those applications at scale in production.

Simplifying Workflows to Speedup AI Developments

NVIDIA recently announced the One Click Deploy feature on the NVIDIA NGC catalog, the hub for GPU-optimized AI software. Developed in collaboration with Google Cloud, this feature simplifies the deployment of AI software, to a single click from the NGC catalog.

This allows data scientists to deploy frameworks, software development kits and Jupyter Notebooks directly to Google Cloud’s Vertex AI Workbench, a new managed Jupyter Notebook service on top of Vertex AI, Google’s service for machine learning operations.

Under the hood, this feature launches the JupyterLab instance on Google Cloud Vertex AI Workbench with optimal instance configuration, preloads the software dependencies, and downloads the NGC notebook in one go.

NGC Catalog – One Stop for AI Software

NVIDIA is expanding the rich trove of NVIDIA AI software in the catalog to ensure AI practitioners have everything they need to get started — from frameworks to models.

All of the AI models in the catalog come with credentials. They’re like resumes that show the model’s skills, the dataset that trained it, how to use the model and how it’s expected to perform.

These model credentials provide transparency, which gives developers the confidence in picking the right model for their use case.

The NGC catalog also hosts Jupyter Notebooks tailored for the most popular AI/ML applications. Examples include:

Computer Vision – A collection of models for detecting human actions, gestures and more.

Automatic Speech Recognition – An end-to-end workflow for text-to-speech training.

Recommendation – A collection of example notebooks to help build end-to-end recommendation services.

Serve Robotics uses Vertex AI and the simple easy-to-use interface that the NGC One Click Deploy delivers.

“NGC catalog allows our ML research engineers to launch environments for experiments on Vertex AI with a single click. This saves us the efforts on ML infra setup and lets researchers focus on the ML problem in the computer vision and robotics space more efficiently.”—Kaiwen Yuan, Director of ML/Head of Perceptions & Predictions at Serve Robotics

Accelerate ML Deployments

Explore hundreds of Jupyter notebook examples for speech, computer vision and recommenders and, if you’re just getting started with AI, browse NVIDIA’s collection of Jupyter notebook examples and run it using the One Click Deploy feature on Google Cloud Vertex AI.

From the historic Gold Country to the rugged heights of the Sierra Nevada, Placer County encompasses more than 1,500 square miles—and provides services to nearly 400,000 residents. Across the county, residents access resources in person, over the phone, and through the county website. In 2018, the county piloted a suite of eServices through its Community Development Resource Agency (CDRA) with the goal of helping its geographically dispersed population more easily apply for permits, make appointments, and get immediate answers to specific questions.

Now, with the help of Google Cloud Dialogflow and Speech-to-Text API, the county has created a virtual agent that enables anyone, anywhere, to simply start a conversation by saying “Ask Placer County” into a variety of voice assistant hardware. This virtual agent builds on the success of eServices and helps the public access information through their smartphones or smart home devices, which is especially helpful for individuals without immediate access to a computer—such as the traveling public or a contractor working on-site, for example.

The pilot program has been an opportunity to explore new technology and create another communication channel to our citizens. “Our ambitious and long-term goal is that ‘Ask Placer County’ will be like having a personal assistant to everything related to Placer County. While the virtual agent is currently limited to a few departments, we plan to expand it countywide,” says Ben Palacio, senior IT analyst for Placer County.

“Our ambitious and long-term goal is that ‘Ask Placer County’ will be like having a personal assistant to everything related to Placer County. While the virtual agent is currently limited to a few departments, we plan to expand it countywide.”

—Ben Palacio, Senior IT Analyst, Placer County

New eServices get a boost from AI virtual agent

The CDRA, one of many departments and agencies within Placer County, deployed interactive eServices for residents. As part of this initiative, the CDRA made a special request to the information technology (IT) department: Build an AI-based virtual agent that would spotlight the eServices on the website and make them easily available through a conversational interface.

“Having a specific technology requirement voiced by an agency was visionary and very welcome. It meant they were engaged and excited about the possibilities that these new technologies had to offer,” says Mike Spak, IT Manager for Placer County.

Today, the CDRA’s eServices help residents answer important questions, such as: “Is my land zoned for adding a garage?” “What historical engineering work has been done on my property?” And, “How can I make an appointment at a permitting office?”

“Having a specific technology requirement voiced by an agency was visionary and very welcome. It meant they were engaged and excited about the possibilities that these new technologies had to offer.”

—Mike Spak, IT Manager, Placer County

Google Cloud fit easily into Placer County’s multivendor environment

With the help of consultants from Dito, a Google Cloud Premier Partner, Placer County chose Google Cloud to help with “Ask Placer County.”

Placer County runs a multivendor IT environment. “Especially with the cloud, the easier it is to integrate a vendor’s capabilities with others, the better for everybody,” says Palacio. “Google Cloud was able to integrate with other vendors’ capabilities for this project.”

For example, Placer County uses the Google Cloud operations suite (formerly Stackdriver) to monitor, troubleshoot, and improve its cloud infrastructure and application performance. And it uses Dialogflow, which powers the natural language processing (NLP) interface of the “Ask Placer County” virtual agent. Both integrate easily with tools from other vendors.

“Especially with the cloud, the easier it is to integrate a vendor’s capabilities with others, the better for everybody. Google Cloud was able to integrate with other vendors’ capabilities for this project.”

—Ben Palacio, Senior IT Analyst, Placer County

Real-time conversations powered by “Ask Placer County”

Dito got “Ask Placer County” up and running using App Engine and Datastore. Today, the “Ask Placer County” virtual agent provides information ranging from how to adopt a pet to short-term-rental compliance.

Users can say “Ask Placer County” into their smartphones or their computer microphones and access targeted information from the pilot departments. IT staff reviews the asked questions and are constantly updating the virtual agent to provide more accurate and responsive information.

“You could be in your backyard with a contractor trying to get a project started, and if the contractor has a question about permitting or zoning, they can use their smartphone to get an answer right then and there, rather than having to interrupt the meeting, get back in their car, and drive to a county office,” says Palacio.

The county hopes “Ask Placer County” will improve the efficiency of in-person interactions between the public and county employees as well. Because the public can access common questions and information online or through the virtual agent, employees’ interactions with the public can be more focused and productive.

“We are providing more effective and efficient service to our customers through 24/7 access to information and by reducing the proportion of staff’s time in responding to emails and voicemails to answer our customer questions,” says Shawna Purvines, Principal Planner for Placer County.

According to Placer County, the virtual agent currently answers a monthly average of around 200 questions for the CDRA, and the county continues to develop more and more complete answers across participating departments.

“We are providing more effective and efficient service to our customers through 24/7 access to information and by reducing the proportion of staff’s time in responding to emails and voicemails to answer our customer questions.”

—Shawna Purvines, Principal Planner, Placer County

Constantly evolving based on users’ needs

To best serve the needs of the CDRA, and potentially other departments over time, “Ask Placer County” continues to evolve. The county can add and retire questions based on the needs of constituents. This technology can be tailored to address current needs, such as responding to COVID-19. “Our hope is that, in quickly evolving situations, the virtual agent can be a resource for the public to access real-time information about public services,” says Palacio.

As a component of the eServices group, “Ask Placer County” has kept construction and development in Placer County operational during the COVID-19 pandemic, and it continues to provide opportunities for the public to access county resources without coming in to the counters, reducing the need for in-person visits. In fact, according to Placer County data, permit applications have increased by 17%.

Placer County also performs analytics using the Google Cloud operations suite to gain insights into the most popular questions and, just as importantly, the questions that are not being answered. “We get insight into what we’re missing—for those times when the virtual agent can’t find a response in our database,” says Palacio. Placer County uses these insights to improve answers and ultimately improve the efficacy of the virtual agent over time.

“This continual, real-time learning is the exciting part of the application that lets us help constituents in entirely new ways. It’s really cool,” says Palacio.

County IT workers actively analyze CDRA webpages to determine which are the most visited and to craft questions and answers to add to “Ask Placer County.” According to Placer County, the number of questions the virtual agent can answer is now more than 600. But the CDRA pages are just a small subset of the 5,000 pages that make up the Placer County website. A future goal is to provide a backstop for customer questions that can’t be answered, transferring those customers to county staff for resolution.

“Looking ahead, this technology has the ability to provide answers to thousands and thousands of questions,” says Palacio.

“This continual, real-time learning is the exciting part of the application that lets us help constituents in entirely new ways. It’s really cool.”

—Ben Palacio, Senior IT Analyst, Placer County

Scaling “Ask Placer County” countywide

The limited rollout of the virtual agent technology allowed IT to monitor its effectiveness and make initial adjustments. As the virtual agent evolves, the Placer County chief information officer sees opportunities for it to engage the public countywide.

The “Ask Placer County” virtual agent can truly be a concierge, helping the public navigate county resources. It can help citizens become more engaged with their elected officials, it can provide information, and it can more efficiently connect the public with the services they seek.

Some of the future possibilities include providing current information on board meetings and individual elected officials, checking on permit status and burn days, and getting the latest county news. “There are so many ways we can use this solution to tackle issues within the county that we’re going to have to somehow prioritize them,” says Palacio.

In the past year, Google has worked to surface timely and reliable health information, amplify public health campaigns, and help nonprofits get urgent support to Indians in need. Now, we are continuing to focus on helping India’s businesses accelerate their digital transformation, deepening our commitment to India’s digitization and economic recovery. To support customers and the public sector in India and across Asia Pacific, we’re excited to announce that our new Google Cloud region in Delhi National Capital Region (NCR) is now open. 

Designed to help both Indian and global companies alike build highly available applications for their customers, the Delhi NCR region is our second Google Cloud region in India and 10th to open in Asia Pacific. 

What customers and partners are saying

Navigating this past year has been a challenge for companies as they grapple with changing customers demands and economic uncertainty. Technology has played a critical role, and we’ve been fortunate to partner with and serve people, companies, and government institutions around the world to help them adapt. The Google Cloud region in Delhi NCR will help our customers adapt to new requirements, new opportunities and new ways of working, like we’ve helped so many companies do in the region: 

• InMobi scaled a personalized AI platform to support 120+ million active users. “With the arrival of the Google Cloud Delhi NCR, InMobi Group sees the opportunity to continue closing the gap between our users and products,” says Mohit Saxena, Co-founder and Group CTO of Inmobi. “Glance, especially, has been serving AI-powered personalised content to over 120 million active users. We can’t wait to continue giving them truly meaningful experiences that are speedy, scale well, and are relevant to them, by expanding the use of our current tools working on Google Cloud with the opening of a new region.”
• Groww now supports a sizable user base. “Google Cloud provides great technology that enables us to build and scale infrastructure to millions of users, and the new Google Cloud region in Delhi NCR will continue to help more businesses and startups in India access powerful cloud-based infrastructure, products and services,” says Neeraj Singh, Co-founder and Chief Technology Officer, Groww.
• HDFC Bank is positioned for the future. “At HDFC Bank, we are harnessing technology platforms to both run and build the bank. As we progress to be future ready, the objective is to invest in future technologies that give us scale, efficiency and resiliency. Towards this the Google Cloud region in Delhi NCR will enable us to enhance our resiliency and help us in building an active-active design framework for our new generation applications on cloud,” says Ramesh Lakshminarayanan, CIO, HDFC Bank.  
• Dr. Reddy’s Lab built a modern data platform with Google Cloud. “At Dr Reddy’s, we pride ourselves in helping patients regain good health, acting quickly to provide innovative solutions to address patients’ unmet needs and in accelerating access to medicines to people worldwide. Our Google Cloud-powered data platform is helping us realize these objectives and we welcome Google’s investment in the new Delhi NCR region as helping us and other businesses in India make further contributions to our social and economic future,” says Mukesh Rathi, Senior Vice President & CIO, Dr. Reddy’s Laboratories.
• “To survive the disruption caused by the pandemic and to succeed in the long term, organizations need to become digital natives, so they can be more agile, explore new business models and build new capabilities that boost resilience. A cloud-first strategy plays a key role in enabling businesses to do this,” said Piyush N. Singh, Lead – India market unit & lead – Growth and Strategic Client Relationships, Asia Pacific and Latin America, Accenture. “Harnessing the potential of cloud requires the right data infrastructure and this expansion by Google Cloud will undoubtedly help Indian enterprises in their digital transformation journeys.”

A global network of regions

Delhi NCR joins 25 existing Google Cloud regions connected via our high-performance network, helping customers better serve their users and customers throughout the globe. As the second region in India, customers benefit from improved business continuity planning with distributed, secure infrastructure needed to meet IT and business requirements for disaster recovery, while maintaining data sovereignty.

With this new region, Google Cloud customers operating in India also benefit from low latency and high performance of their cloud-based workloads and data. Designed for high availability, the region opens with three availability zones to protect against service disruptions, and offers a portfolio of key products, including Compute Engine, App Engine, Google Kubernetes Engine, Cloud Bigtable, Cloud Spanner, and BigQuery. 

Supporting India’s recovery with training and education

Google and Google Cloud will also continue to support our customers with people and education programs. We’re investing in local talent and the local developer community to help enterprises digitally transform and support economic recovery. 

Through the India Digitization Fund, we expanded our efforts to support India’s recovery from COVID-19—in particular, through programs to support education and small businesses. In addition to expanding internet access, and investments to help start-ups accelerate India’s digital transformation, we’ve grown our Grow with Google efforts. Businesses can access digital tools to maintain business continuity, find resources like quick help videos, and learn digital skills—in both English and in Hindi.

Helping customers build their transformation clouds

Google Cloud is here to support businesses, helping them get smarter with data, deploy faster, connect more easily with people and customers throughout the globe, and protect everything that matters to their businesses. The cloud region in Delhi NCR offers new technology and tools that can be a catalyst for this change. To learn more, visit the Google Cloud locations page, and be sure to watch the region launch event here.