Editor’s note:  With Google Cloud’s datasets solution, you can access an ever-expanding resource of the newest datasets to support and empower your analyses and ML models, as well as frequently updated best practices on how to get the most out of any of our datasets. We will be regularly updating this blog with new datasets and announcements, so be sure to bookmark this link and check back often.

August 2021

New dataset: Google Cloud Release Notes

• Looking for a new way to access Google Cloud Release Notes besides the docs, the XML feed, and Cloud Console? Check out the Google Cloud Release Notes dataset. With up-to-date release notes for all generally available Google Cloud products, this dataset allows you to use BigQuery to programmatically analyze release notes across all products, exploring security bulletin updates, fixes, changes, and the latest feature releases. 
• Access the dataset

July 2021

Best practice: Use Google Trends data for common business needs

• The Google Trends dataset represents the first time we’re adding Google-owned Search data into Datasets for Google Cloud. The Trends data allows users to measure interest in a particular topic or search term across Google Search, from around the United States, down to the city-level. You can learn more about the dataset here, and check out the Looker dashboard here! These tables are super valuable in their own right, but when you blend them with other actionable data you can unlock whole new areas of opportunity for your team. To learn how to make informed decisions with Google Trends data, keep reading.
• Access the dataset

New dataset: COVID-19 Vaccination Search Insights

• With COVID-19 vaccinations being a topic of interest around the United States, this dataset shows aggregated, anonymized trends in searches related to COVID-19 vaccination and is intended to help public health officials design, target, and evaluate public education campaigns. Check out this interactive dashboard to explore searches for COVID-19 vaccination topics by region.
• Access the dataset

June 2021

New dataset: Google Diversity Annual Report 2021

• Since 2014, Google has disclosed data on the diversity of its workforce in an effort to bring candid transparency to the challenges technology companies like Google face in recruitment and retention of underrepresented communities. In an effort to make this data more accessible and useful, we’ve loaded it into BigQuery for the first time ever. To view Google’s Diversity Annual Report and learn more, check it out.
• Access the dataset

New dataset: Google Trends Top 25 Search terms

• The most popular and surging Google Search terms are now available in BigQuery as a public dataset. View the Top 25 and Top 25 rising queries from Google Trends from the past 30-days, including 5 years of historical data across the 210 Designated Market Areas (DMAs) in the US. Keep reading.
• Access the dataset

New dataset: COVID-19 Vaccination Access

• With metrics quantifying travel times to COVID-19 vaccination sites, this dataset is intended to help Public Health officials, researchers, and Healthcare Providers to identify areas with insufficient access, deploy interventions, and research these issues. Check out how this data is being used in a number of new tools.
• Access the dataset

Best practice: Leveraging BigQuery Public Boundaries datasets for geospatial analytics 

• Geospatial data is a critical component for a comprehensive analytics strategy. Whether you are trying to visualize data using geospatial parameters or do deeper analysis or modeling on customer distribution or proximity, most organizations have some type of geospatial data they would like to use – whether it be customer zipcodes, store locations, or shipping addresses. However, converting geographic data into the correct format for analysis and aggregation at different levels can be difficult. In this post, we’ll walk through some examples of how you can leverage the Google Cloud platform alongside Google Cloud Public Datasets to perform robust analytics on geographic data. Keep reading.
• Access the dataset

Get the metadata and try BigQuery sandbox 

When you’ve learned about many of our datasets and pre-built solutions from across Google, you may be ready to start querying them. Check out the full dataset directory and read all the metadata at g.co/cloud/marketplace-datasets, then dig into the data with our free-to-use BigQuery sandbox account, or $300 in credits with our Google Cloud free trial.

San José is one of the most diverse U.S. cities, with residents speaking more than 100 languages. Several years ago, we set out to improve city community interactions through more equitable service management and delivery. This demanded a new approach to automating the intake of requests from a majority population whose first language is not English.

We first created the 311 portal and app, which was an important step as we effectively separated resident service requests from emergencies. The way we describe it to our citizens, you call 311 for a burning question, and 911 for a burning building. In the last fiscal year, San José 311 received nearly 210,000 contacts by phone and an additional 211,000 service requests through the SJ 311 app. 

Through the portal and app, we gave citizens an omnichannel experience enabling them to interact with the city to request improvements, access useful information, and get emergency help when they need it. 

In order to truly serve our diverse communities, we recognized language translation services would be required to offer truly equitable services to everyone. That’s when we started working closely with SpringML, Google Cloud, and other partners with involvement from our Mayor and City CIO.

Building public services with community engagement in mind

When we first rolled out the My San José website and mobile app, we used an out-of-the-box translation service that ended up not working. It had poor accuracy and did not meet our needs to provide all citizens with coherent services. After looking at many other options, we decided to partner with SpringML and Google Cloud to leverage the AutoML Translation with other technologies such as our virtual agent.

SpringML was selected through an open RFP process, and helped us to build and optimize our integrations, interfaces, and more between several systems, making the app and website more intuitive to manage. SpringML delivered the product we needed on time and up to specifications, and additional value came from the training sessions they provided to our team. This enabled us to understand everything we could do with AutoML and opened the door to other enhancements such as simplifying the vernacular used with our residents, making government access easier to navigate regardless of natural language spoken. 

After establishing the My San José app’s translation capabilities using AutoML, SpringML also helped us incorporate Dialogflow virtual agents. Dialogflow also positions us to make modifications with our own staffing practices – something that has become increasingly important amid the frequent changes in service levels from COVID-19 response in the past year.

Responding to community needs

With the app up-and-running, our next step was to bring in community members to help with testing, improvements, and more. We wanted the app and the website to not just be something we provided to the community, but rather something they helped us build so they would readily adopt it.

Thanks to the greater accuracy of translation supported by Google Cloud services, we were able to leverage the expertise of a small pool of community members to evaluate translations. AutoML Translation and Glossary proved to be a powerful combination that pushed us closer to our goals. 

Our primary targets were Spanish and Vietnamese translations. We are now seeing 90 percent accuracy in automated Spanish translations while Vietnamese translations continue to improve. We continue to work to simplify the language used in these services, which makes a big difference in terms of ensuring optimal language accessibility.

This work includes best serving our community members who primarily use phones to get in touch with us through 311 services. Using Google Cloud Contact Center AI, we have been able to effectively manage the calls we receive 24×7 and communicate with residents who speak Spanish as well as English. No matter which channel one of our residents choose to use to reach out, we can serve them efficiently. 

A well-timed release

We’re proud of the work we’ve done. We’ve made many government services available to our community 24 hours a day, 7 days a week — accessible through many channels. Regardless of a person’s native language, the consistency of experiences enjoyed by everyone is improving every day thanks to AI. We’re also actively incorporating more language translation capabilities to better serve more people.

While we began this process several years ago, the recent integration of machine learning language translation with our customer relationship management system in late 2020 was very well-timed because we were able to incorporate this into our COVID-19 pandemic response. 

We’re also beginning to work with other municipalities across the U.S. to share some of the lessons we’ve learned and success we’ve seen in hopes of furthering more equitable citizen services far beyond our City limits. 

We are excited to continue working with SpringML, Google Cloud, and other partners to improve our city and the equity and quality of services that our residents enjoy.

Learn more about how you can work with a Google Cloud Partner here.

Editor’s note: The post is part of a series highlighting our awesome partners, and their solutions, that are Built with BigQuery.

What are Customer Data Platforms (CDPs) and why do we need them?

Today, customers utilize a wide array of devices when interacting with a brand. As an example, think about the last time you bought a shirt. You may start with a search on your phone as you take the subway to work. During that 20 minute ride, you narrow down the type of shirt . Later, as you take your lunch break, you spend a few more minutes refining your search on your work laptop and you are able to find two shirt models of interest. Pressed for time, you add both to your shopping cart at an online retailer to review at a later point. Finally, after you arrive back home and as you are checking your physical mail, you stumble across a sales advertisement for the type of shirt that you are looking for, available at your local brick and mortar store. The next day you visit that store during your lunch break and purchase the shirt.

Many marketers face the challenge of creating a consistent 360 customer view that captures the customer lifecycle, as illustrated in the example above – including their online/offline journey, interacting with multiple data points across multiple data sources.

The evolution of managing customer data reached a turning point in the late 90’s with CRM software that sought to match current and potential customers with their interactions. Later as a backbone of data-driven marketing, Data Management Platforms (DMPs) expanded the reach of data management to include second and third party datasets including anonymous IDs. A Customer Data Platform combines these two types of systems, creating a unified, persistent customer view across channels (mobile, web etc) that provide data visibility and granularity at individual level.

A new approach to empowering marketing heroes

Tinyclues is a company that specializes in empowering marketers to drive sustainable engagement from their customers and generate additional revenue, without damaging customer equity. The company was founded in 2010 on a simple hunch: B2C marketing databases contain sufficient amounts of implicit information (data unrelated to explicit actions) to transform the way marketers interact with customers, and a new class of algorithms based on Deep Learning (sophisticated machine learning that mimics the way humans learn) holds the power to unlock this data’s potential. Where other players in the space have historically relied – and continue to rely – on a handful of explicit past behaviors and more than a handful of assumptions, Tinyclues’ predictive engine uses all of the customer data that marketers have available in order to formulate deeply precise models, down even to the SKU level. Tinyclues’ algorithms are designed to detect changes in consumption patterns in real-time, and adapt predictions accordingly.

This technology allows marketers to find precisely the right audiences for any offer during any timeframe, increasing engagement with those offers and, ultimately, revenue; additionally, marketers are able to increase campaign volume while decreasing customer fatigue and opt-outs, knowing that audiences are receiving only the most relevant messages. Tinyclues’ technology also reduces time spent building and planning campaigns by upwards of 80%, as valuable internal resources can be diverted away from manual audience-building.

Google Cloud’s Data Platform, spearheaded by BigQuery, provides a serverless, highly scalable, and cost-effective foundation to build this next generation of CDPs.

Tinyclues Architecture:

To enable this scalable solution for clients, Tinyclues receives purchase and interaction logs from clients in addition to product and user tables. In most cases, this data is already in the client’s BigQuery instance, in which case they can be easily shared with Tinyclues utilizing BigQuery authorized views.

In cases where the data is not in BigQuery, flat files are sent to Tinyclues via GCS and are ingested in the client’s data set via a lightweight Cloud Function. The orchestration of all pipelines is implemented via Cloud Composer (Google’s managed Airflow). The transformation of data is accomplished by utilizing simple select statements in the Data Built Tool (DBT), which is wrapped inside an airflow DAG that powers all data normalization and transformations. There are several other DAGs to fulfill more functionalities, including:

• Indexing the product catalog on Elastic Cloud (Elasticsearch managed service) on GCP to provide auto-complete search capabilities to TCs clients as shown below:

• The export of Tinyclues-powered audiences to the clients’ activation channels, whether they are using SFMC, Braze, Adobe, GMP, or Meta.

Tinyclues AI/ML Pipeline powered by Google Vertex AI

TCs ML Training pipelines are used to train models that calculate propensity scores. They are composed using Airflow DAGs, powered by Tensorflow & Vertex AI Pipelines. BigQuery is used natively, without data movement, to perform as much feature engineering as possible in-place.

TC uses the TFX library to run ML Pipelines in Vertex AI. Building on top of Tensorflow as their main deep learning framework of choice due to its maturity, open source platform, scalability and support for complex data structures (Ragged and Sparse Tensors).

Below is a partial example of TC’s Vertex AI Pipeline graph, illustrating the workflow steps in the training pipeline. This pipeline allows for the modularization & standardization of functionality into easily manageable building blocks. These blocks are composed of TFX components (TC reuses most of the standard components in addition to customizing some such as a proprietary implementation of the Evaluator to compute both ML Metrics (which is part of the standard implementation) but also more Business Metrics like Overlap of clickers etc. The individual components/steps are chained with DSL to form a pipeline that is modular and easily orchestrated or updated as needed.

With the trained Tensorflow models available in GCS, TCs exposes these in BigQuery ML (BQML) to enable their clients to score millions of users for their propensity to buy X or Y within minutes. This would not be possible without the power of BigQuery and also frees TC from previously experienced scalability issues.

As an illustration, TC has the need to score thousands of topics among millions of users. This used to take north of 20 hours on their previous stack, and now takes less than 20 minutes thanks to the optimization work that TC has implemented in their custom algorithm and the sheer power of BQ to scale to any workload accordingly.

Data Gravity: Breaking the Paradigm – Bringing the Model to your Data

BQML enables TC to call pre-trained TensorFlow models within an SQL environment, thus avoiding exporting data in and out of BQ using already provisioned BQ serverless processing power. Using BQML removes the layers between the models and the data warehouse and allows them to express the entire inference pipe as a number of SQL requests. TC no longer has to export data to load it into their models. Instead, they are bringing their models to the data.

Avoiding the export of data in and out of BQ and the serverless provisioning and start of machines saves significant time. As an example, exporting an 11M lines campaign for a large client previously took 15 min or more to process. Deployed on BQML it now takes minutes with more than half of the processing time attributed to network transfers to our client system.

Inference times in BQML compared to TCs legacy stack:

As can be seen, using this approach enabled by BQML, the reduction in the number of steps leads to a 50% decrease in overall inference time, improving upon each step of the prediction.

The Proof is in the pudding

Tinyclues has consistently delivered on its promises of increased autonomy for CRM teams, rapid audience building, superior performance against in-house segmentation, identification of untapped messaging and revenue opportunities, fatigue management, and more, working with partners like Tiffany & Co, Rakuten, and Samsung, among many others.

Conclusion

Google’s data cloud provides a complete platform for building data-driven applications like the headless CDP solution developed by Tinyclues — from simplified data ingestion, processing, and storage to powerful analytics, AI, ML, and data sharing capabilities — all integrated with the open, secure, and sustainable Google Cloud platform. With a diverse partner ecosystem, open-source tools, and APIs, Google Cloud can provide technology companies the portability and differentiators they need to serve the next generation of marketing customers.

To learn more about Tinyclues on Google Cloud, visit Tinyclues. Click here to learn more about Google Cloud’s Built with BigQuery initiative.

We thank the many Google Cloud team members who contributed to this ongoing data platform collaboration and review, especially Dr. Ali Arsanjani in Partner Engineering.

Verve Group is an ecosystem of demand and supply technologies fusing data, media, and technology to deliver results and growth to both advertisers and publishers – no matter the screen or location, no matter who, what, or where a customer is. Classifying massive amounts of this unstructured data at scale is the first step in helping to surface relevant, high-quality content to users—and that’s where natural language processing (NLP) comes in.

Verve Group uses the NLP API from Google Cloud’s Vertex AI to fetch data for their internal content classification quality verification and as an additional source for building categorization models. By leveraging the NLP API’s Content Classification models, which are now generally available and offer Google’s latest large language model (LLM) technology, Verve Group powers classification through an updated and expanded training data set with over 1,000 labels and support for 11 languages (Chinese, French, German, Italian, Japanese, Korean, Portuguese, Russia, Spanish, and Dutch join previously-available English). 

Verve Group has been using Google Cloud’s NLP API since day one, because of both the ease of implementation and the quality compared to competing NLP products. With documentation that is “comprehensive and self-explanatory,” the NLP API “allows for fast adoption and implementation, from test models all the way to production,” said Rami Alanko, GM of Verve Group.

Leveraging the NLP API has facilitated Verve Group’s fast go-to-market motions by enabling its customers to quickly discover and classify new content. “Operating on a global level with tens of different regions and languages, we have still been able to maintain high quality for our product and high retention rates with our clients,” Rami shared. “In a recent client case, we achieved 82% improvement in CTR when optimized with content quality measurements enabled by the API. In another client case, we drove brand safety risk down to 0.16% from 4% thanks to classification quality. Along with the new functionalities of the Google NLP, I can only see this trend continuing to strengthen.”

Verve Group is excited to further expand their NLP use cases by leveraging the new Content Classification models, which have already helped them expand their classification inventory, improve the quality and performance of their quality verification for customers, and unlock new use cases for NLP. “Our classification model accuracy improved 41% using Google NLP as a verification partner,” said Rami.  

Additionally, Verve Group is now using the API for metadata analysis on a  large image database. “We browse the database and run the image metadata via our classification. This flow enables us to classify images reliably aligned with our standard classification.  We pretty much use the same data flow for our runtime in-app textual content analysis, therefore allowing for close to real-time consumer engagement,” Rami added.

To learn more about how companies are leveraging NLP API from Google Cloud Vertex AI, click here, and to learn more about Google Cloud’s work with foundation models and generative AI, read The Prompt on Transform with Google Cloud.

Conversational AI is opening up a new world of possibilities in areas like customer experience, user engagement, and access to content. 

In Cloud AI, we’ve taken Google’s groundbreaking machine learning models in speech and natural language processing and applied them to the contact center space, radically improving the customer experience while also driving down operational costs. It’s shifting the focus of contact centers from the backroom, out of sight (and mind) to the boardroom and the strategic heart of the business.

Our solution, called Contact Center AI (CCAI), is an accelerator of digital transformation as organizations all over the world figure out how to support their customers during these challenging times. One customer that’s redefined the possibilities of AI-powered conversation using CCAI, is Verizon.

Whether through voice calls or chat, Verizon customers will no longer need to go through menu prompts or option trees; they simply say or type their question, and CCAI’s natural-language recognition feature finds the best way to assist them. No stilted speech or robot-like commands.

“These customer service enhancements, powered by the Verizon collaboration with Google Cloud, offer a faster and more personalized digital experience for our customers while empowering our customer support agents to provide a higher level of service,” said Shankar Arumugavelu, SVP & CIO at Verizon. 

CCAI is also driving cost savings without cutting corners on customer service. In the past, to improve customer satisfaction (CSAT), you had to hire more agents, increasing operating costs. Conversely, if you reduced the number of agents, your CSAT scores went down. Contact Center AI (CCAI) lets you have both.  

CCAI does this by expediting customer requests using virtual agents, assisting live agents, and providing insights on customer interactions to drive improvements back into the system. The result is a better experience for your customers—they get answers faster—and you get lower operating costs. It allows human agents to focus on higher value tasks in their customer interactions.

New products and features add richer customer experience 

We are continuing to invest in CCAI and today we are excited to announce the following new capabilities:

Dialogflow CX is the latest version of Dialogflow, a development suite used by over a million developers for building natural, rich conversational experiences into mobile and web applications, smart devices, bots, interactive voice response systems, popular messaging platforms and more. This new version of Dialogflow is optimized for large contact centers that deal with complex (multi-turn) conversations and it is truly omnichannel – you build it once and deploy it everywhere – in your contact centers and digital channels. Dialogflow CX features a new visual builder to create, build and manage virtual agents. It’s available now, in beta.

Dialogflow CX brings conversation state management to a whole new level

Lukasz Rewerenda
Principal Solutions Architect, Randstad (Netherlands)

Agent Assist for Chat is a new module for Agent Assist that provides agents with continuous support over “chat” in addition to voice calls, by identifying intent and providing real-time, step-by-step assistance. Agent Assist enables agents to be more agile and efficient and spend more time on difficult conversations, giving both the customer and the agent a better experience. It transcribes calls in real time, identifies customer intent, provides real-time, step by step assistance (recommended articles, workflows, etc.), and automates call dispositions.

For customers in regulated industries, Agent Assist can remove the risk of agents providing inaccurate information (which can happen due to high agent turnover and limited training). Agent Assist can also surface the latest discount information, deals and special offers, which can be hard for agents to keep track of as this information changes frequently. 

Custom Voice, available in beta, is a new capability for CCAI and our Text-to-Speech API that lets you create a unique voice to represent your brand across all your customer touchpoints, instead of using a common voice shared with other organizations. By taking advantage of the custom Text-to-Speech model created with Custom Voice, you can define and choose the voice profile that suits your business and adjust to changes without scheduling studio time with voice actors to record new phrases.

At Google Cloud, we are committed to ensuring that our products and features are in alignment with our AI Principles. If you are interested in Custom Voice, there is a review process to ensure that your use case is aligned with our AI principles. Learn more about Custom Voice and how to sign up here. 

Bring Google’s groundbreaking speech models on-prem

For many customers, an all-in approach to public cloud is not an option, which is why we’re extending our AI capabilities to run on-prem. Last week we announced Speech-to-Text On-Prem, the first of our hybrid AI offerings, now generally available. Speech-to-Text On-Prem gives you full control over speech data and since it runs in your own data center, it’s easy to comply with your data residency and compliance requirements. At the same time, Speech-to-Text On-Prem uses state-of-the-art speech models from Google researchers that are more accurate, smaller, and require less computing resources to run than existing solutions.

Take a look at how businesses saved costs and improved customer experience with CCAI in the new commissioned study conducted by Forrester Consulting: “New Technology: The Projected Total Economic Impact™ Of Google Cloud Contact Center AI.” To learn more about CCAI and conversational AI, see how our customers are using the solution and check out these sessions at Next OnAir:

• CCAI State of the Union (keynote)
• Measuring and Improving Speech-to-Text Accuracy (breakout)
• Transform customer service with CCAI (demo)
• See the next generation of 3-1-1 in action (demo)