Editor’s note: When Europe’s largest mobile communications company, Vodafone, was asked by the European Commission to help understand population movement across the European Union and the UK to help fight COVID-19, it was able to provide anonymized mobile network-based insights to answer the call. Here’s how Vodafone, with the support of Google Cloud, rapidly mobilized the COVID-19 frontline, while respecting its customers’ privacy.

With the emergence of COVID-19 in early 2020, the European Commission—the executive branch of the European Union (EU)—knew that technology would be instrumental in its fight to control the pandemic. With various lockdowns imposed across its member states, the Commission was keen to predict and prevent the spread of COVID-19 and to manage the related social, political and financial impacts. 

Mobile network data helps track COVID-19 across the EU

Mobile networks produce location data, which can be turned into useful anonymous insights to understand population movement within a geographic area. The European Commission, working with mobile industry association GSMA (Groupe Speciale Mobile Association), asked Europe’s major mobile phone operators for help in producing insights to support the fight against COVID-19. As the largest mobile network operator within the EU, Vodafone saw this as a critical opportunity to participate. 

Vodafone had previous experience of using mobile network data to support pandemic research. For example, in 2019, Vodafone provided mobility pattern analysis to help track the spread of Malaria in Mozambique. And, during the early stages of the COVID-19 pandemic (prior to working with the European Commission), Vodafone assisted the Italian and Spanish governments in understanding their citizens’ mobility patterns. Vodafone had also previously offered anonymized and aggregated population mobility insights to support public transport and tourism authorities and retail organizations in a number of countries. Consequently, Vodafone was perfectly placed to play a greater role in supporting the European Commission’s response to the pandemic. 

When asked to assist the European Commission, Vodafone first considered how it could safely share its data with the governing body without providing details on the individual movements of its customers. It realized it could achieve this through an elaborate set of anonymization and aggregation techniques. Insights are aggregated from a minimum of 50 users and Vodafone only shared these anonymous insights and never the actual raw data with the Commission. As specified by the EU, these insights are then presented onto a large geographical region, typically a city or a county with thousands of people living in that area.

These insights illustrate how people move, helping to determine how lockdowns and self-isolation measures were impacting behaviors.

Using Google Cloud to collate and store population mobility data

In April 2020, Vodafone began migrating its operations, including its mobile data, to Google Cloud on servers in Europe and the UK with elaborate security safeguards, including encryption, building on a previous partnership. 

With the data residing in EU and UK data centers and not the United States, Vodafone could then retrieve anonymous insights from Google Cloud Storage instantaneously. Before supplying any information to the European Commission, however, Vodafone used Dataflow to validate the data and run a series of tests to ensure the database had accurate data, before ingesting and archiving the relevant metrics. For instant access, the data was then made available to the European Commission using a Redis database on Google Kubernetes Engine.

To ensure aggregate Vodafone customer data was always safe, secure, and anonymous, all entry points to the front-end were protected behind Google Cloud Armor, where only specific IP addresses were allowed. Using these tools, seamless data pipelines fed in predefined key performance indicators from each specified European market. While data quality measures ensured the definitions for metrics across markets were consistent and could be accurately compared.

The architecture (pictured below) shows how Vodafone integrated and anonymized its data on Google Cloud.

Live interactive dashboard shows population mobility in real-time

With its data integrated on Google Cloud, Vodafone created a live, interactive dashboard to track mobility patterns and share relevant information with the European Commission in real-time. 

The European Commission Joint Research Center (JRC) was able to gather valuable information from these insights, which enabled them to see where population mobility was aiding the spread of the disease, when cross-referenced with health data. It could also assess the implications of lockdowns on different populations and forecast cross-country spreading.

Mobile data aids disease modeling for multiple stakeholders

The Vodafone data became instrumental in modeling the likely course of the disease too. For example, the University of Southampton in the UK used it to predict the outcome of different coordinated COVID-19 exit strategies across Europe. This research was published in Science Magazine in September 2020. 

The Vodafone data dashboard continues to be used by individual governments, NGOs and organizations to further investigate the impacts of the pandemic and to measure the effectiveness of response strategies alongside the rollout of vaccination programs. The project also helped Vodafone win a DataIQ award for most effective stakeholder engagement. 

Using the learnings from this project, Vodafone has been able to adapt its own B2B solution, called Vodafone Analytics, by adaptIng and migrating the code to work in Google Cloud Platform. This solution has been rolled out across Germany, Greece, Portugal and South Africa, and new countries are being onboarded every day. Vodafone Analytics already has more than 100 customers leveraging it for a variety of use cases—Italian fashion retailer OVS, uses it for its smart retail operation, while global real estate company, JLL, uses it to understand the footfall passing through its properties. 

Working together, Vodafone and Google Cloud continue to help a range of organizations, governments, and NGOs navigate through the ongoing pandemic,  optimize their operations, and help the greater good, without infringing individuals’ fundamental rights to privacy.

To learn more about Google Cloud and Vodafone, watch our full interview here.

With the rise of technologies like smartphones, retailers have felt the pressure to meet evolving consumer needs and expectations. Seven-Eleven Japan(“SEJ”) has long been on the forefront of this thanks to the way they develop and invest in IT. However, in recent years, Japan’s leading convenience store chain has struggled to maintain its complex legacy systems at the rate needed to keep up with today’s rapid digitization, spurred on by the increasing proliferation of smartphones and an IT vendor-dependent structure.  

Legacy systems limiting real-time responsiveness and innovation 

Since its early days, SEJ has been proactive in adopting information technology, mainly relying on technology solutions from Japan’s leading vendors. But as the systems have grown, key business issues have been resolved using a vendor-dependent structure rather than being driven by SEJ’s own needs.  

Datasets and business logic were combined and built into legacy environments, gradually leading to data silos. As a result, data was distributed across multiple systems, causing a variety of problems, including the inability to efficiently retrieve data when needed, delays in accessing data collected in individual stores, and difficulties taking measurements at the right time in business operations that require real-time responsiveness.

Connecting different systems also takes time and money, and the lead time for introducing new services—from planning to development and launch—has been longer than expected. 

To solve these problems, SEJ’s IT department built “Seven Central”—a new platform for practical data use launched in 2020 to support the company’s future IT strategies and digital transformation initiatives.

At its core, Seven Central’s ultimate purpose is to allow real-time data views. Versatile, real-time datasets—such as point-of-sale (POS) data from 7-Eleven stores—are consolidated into a centralized location in the cloud. They created a simple data mart that provides data via an API to enable them to respond more quickly to requests from individual departments. 

“In such uncertain times, it’s vital to use data to make quick decisions,” says Izuru Nishimura, Executive Officer and Head of ICT Department. “Each department across the entire company will be able to gain an immediate understanding of the situation based on the most up-to-date data and respond accordingly. This is why we built Seven Central.”

Google Cloud selected to help SEJ build and grow their data cloud

Today’s rapidly changing business environment has also highlighted the risk of IT support becoming a bottleneck. The long-term strategy is to gradually expand the datasets managed and collected in Seven Central according to business needs. 

In the first phase, SEJ collected POS data from all 21,000+ stores to enable real-time analysis. Moving forward, they would like to collect other relevant data—for example, unstructured data, such as images and videos, or master datasets that are currently stored externally. 

Google Cloud was already a top contender when SEJ started developing Seven Central in 2019. They compared various public cloud services besides Google Cloud, focusing on three main capabilities.

“We placed particular emphasis on service scalability to drive future digital transformation; security when handling data, which is the lifeline of our company; and finally, openness,” says Nishimura. He emphasizes that openness was perhaps the most important factor for choosing Google Cloud. Breaking away from the negative aspects of an entirely vendor-dependent system enabled them to build an agile development system with multiple vendors. 

Google Cloud technologies including BigQuery and API management platform, Apigee, play a vital role in Seven Central. BigQuery’s high-speed processing at petabyte scale and fully managed infrastructure helped keep costs low during development and verification.   

“Data is stored in a way that allows you to share it easily across organizations, which helps solve the issue of data silos from the perspective of scalability. I also like the fact there are some interesting features that could be used in the future—like BigQuery ML, which enables machine learning on BigQuery,” says Nishimura. 

Apigee allows SEJ to separate datasets and business logic, which is one of the key points of Seven Central. While the trend these days is to standardize interfaces using an API, the reality tends to involve many different APIs rather than the introduction of one unified API. With Apigee, SEJ provides a single unified API for all of its data cloud, and they can now understand what data is used thanks to Apigee’s API usage visualizations.  

“Right now, we collect data from all 21,000+ stores,” says Nishimura. “But in anticipation of a future expansion in business operations, we have designed a system that can scale up and run without issue, even if we were to have 30,000 stores, with 1,000 customers per store per day, purchasing five items per person.”

Real-time insights with BigQuery and Cloud Spanner

Google Cloud partner Cloud Ace came on board early in the planning phases. Based on their recommendations, SEJ decided to continue making full use of BigQuery to analyze data collected from all 21,000+ stores throughout Japan, while also using Cloud Spanner’s availability, near-unlimited scalability and transactional consistency to help achieve the real-time results needed for the project.

“Given that both the data and the regularity with which it is accessed are expected to steadily increase in the future, we chose Cloud Spanner as backend storage for data delivery via API. We consider it a good choice,” says Shota Kikuchi, General Manager, Consulting Department, Technology Division, Cloud Ace Co., Ltd.

Finally, they chose to use Google Cloud’s Stream Analytics Solutions messaging service for collecting POS data in real time, which can then be put to immediate use with Cloud Spanner and BigQuery. 

High-speed responses exceed targets and create new value 

Seven Central went live in September 2020 with surprising results. 

They initially set a target time of one hour from when a customer makes a purchase to the point when Seven Central can use that data. But when the final system was first tried—it took barely a minute. Moving forward they estimate that the latest inventory data from the service side will become available within a few minutes of being added to the system.  

“This is real innovation, and I must admit that I am quite surprised. As well as being able to solve existing issues, we also hope it will lead to new improvements and services that have been unimaginable up until now,” says Nishimura.

The team hopes to roll out the Seven Central platform in all companies affiliated with Seven & i Holdings—not just SEJ. They also plan to explore Google Cloud AI and machine learning technologies to take on challenges in new areas. For example, they are investigating the idea of clustering individual stores using BigQuery ML.

Seven Central has already attracted attention from many departments and received a lot of requests. Nishimura and his team say they hope to continue to grow Seven Central while still observing their fundamental principles—not including business logic, maintaining real-time results, and staying true to the uniqueness of SEJ.

Learn more about Google Cloud smart analytics solutions.

Editor’s note:  With Google Cloud’s datasets solution, you can access an ever-expanding resource of the newest datasets to support and empower your analyses and ML models, as well as frequently updated best practices on how to get the most out of any of our datasets. We will be regularly updating this blog with new datasets and announcements, so be sure to bookmark this link and check back often.

August 2021

New dataset: Google Cloud Release Notes

• Looking for a new way to access Google Cloud Release Notes besides the docs, the XML feed, and Cloud Console? Check out the Google Cloud Release Notes dataset. With up-to-date release notes for all generally available Google Cloud products, this dataset allows you to use BigQuery to programmatically analyze release notes across all products, exploring security bulletin updates, fixes, changes, and the latest feature releases. 
• Access the dataset

July 2021

Best practice: Use Google Trends data for common business needs

• The Google Trends dataset represents the first time we’re adding Google-owned Search data into Datasets for Google Cloud. The Trends data allows users to measure interest in a particular topic or search term across Google Search, from around the United States, down to the city-level. You can learn more about the dataset here, and check out the Looker dashboard here! These tables are super valuable in their own right, but when you blend them with other actionable data you can unlock whole new areas of opportunity for your team. To learn how to make informed decisions with Google Trends data, keep reading.
• Access the dataset

New dataset: COVID-19 Vaccination Search Insights

• With COVID-19 vaccinations being a topic of interest around the United States, this dataset shows aggregated, anonymized trends in searches related to COVID-19 vaccination and is intended to help public health officials design, target, and evaluate public education campaigns. Check out this interactive dashboard to explore searches for COVID-19 vaccination topics by region.
• Access the dataset

June 2021

New dataset: Google Diversity Annual Report 2021

• Since 2014, Google has disclosed data on the diversity of its workforce in an effort to bring candid transparency to the challenges technology companies like Google face in recruitment and retention of underrepresented communities. In an effort to make this data more accessible and useful, we’ve loaded it into BigQuery for the first time ever. To view Google’s Diversity Annual Report and learn more, check it out.
• Access the dataset

New dataset: Google Trends Top 25 Search terms

• The most popular and surging Google Search terms are now available in BigQuery as a public dataset. View the Top 25 and Top 25 rising queries from Google Trends from the past 30-days, including 5 years of historical data across the 210 Designated Market Areas (DMAs) in the US. Keep reading.
• Access the dataset

New dataset: COVID-19 Vaccination Access

• With metrics quantifying travel times to COVID-19 vaccination sites, this dataset is intended to help Public Health officials, researchers, and Healthcare Providers to identify areas with insufficient access, deploy interventions, and research these issues. Check out how this data is being used in a number of new tools.
• Access the dataset

Best practice: Leveraging BigQuery Public Boundaries datasets for geospatial analytics 

• Geospatial data is a critical component for a comprehensive analytics strategy. Whether you are trying to visualize data using geospatial parameters or do deeper analysis or modeling on customer distribution or proximity, most organizations have some type of geospatial data they would like to use – whether it be customer zipcodes, store locations, or shipping addresses. However, converting geographic data into the correct format for analysis and aggregation at different levels can be difficult. In this post, we’ll walk through some examples of how you can leverage the Google Cloud platform alongside Google Cloud Public Datasets to perform robust analytics on geographic data. Keep reading.
• Access the dataset

Get the metadata and try BigQuery sandbox 

When you’ve learned about many of our datasets and pre-built solutions from across Google, you may be ready to start querying them. Check out the full dataset directory and read all the metadata at g.co/cloud/marketplace-datasets, then dig into the data with our free-to-use BigQuery sandbox account, or $300 in credits with our Google Cloud free trial.

Demand for spoken audio content such as podcasts remains robust despite the proliferation of video services and other entertainment options for consumers. Shibin Li, Co-founder of Castbox, credits growth of the global podcast platform to the following: speed and availability, market-leading features, the proliferation of smart devices to deliver audio content, and activities — such as driving and working around the house — that make consuming video difficult.

Founded in 2016 and headquartered in Beijing, China, Castbox enables users to locate, access, and create spoken audio content. Available on iOS and Android, Castbox supports 50 million podcasts, on-demand radio programs, and audiobooks in 70 languages from 175 countries. The platform hosts about 2 million users per day and is the largest podcast platform on Android.

Castbox includes a range of features that build on its core service to provide a high-quality user experience. These features include curated podcast recommendations and in-audio search.

A combination of services

At its inception, Castbox relied on a combination of a multinational cloud services, as well as Google Cloud Platform (GCP) services, including the Google BigQuery analytics data warehouse and Cloud APIs to provide programmatic interfaces with Google Cloud services, and a range of services from the Google mobile development platform Firebase.

However, as Castbox matured and its user base expanded, the business increased its reliance on Google Cloud Platform and Firebase.

“We needed to access stable cloud services as we could not tolerate long periods of downtime that would compromise the user experience,” says Li. “Furthermore, we had to support up to 50,000 concurrent connections, and potentially more in future, without disruption.”

“Based on our analysis of the data in Google BigQuery, we can determine what type of content users are listening to, how long they like to listen to it, and when they like to listen to it. This allows us to recommend similar podcasts to each user based on the preferences he or she expressed, encouraging activity on and return visits to our platform.”

—Shibin Li, Co-Founder, Castbox

Competitive differentiation

Castbox also found machine learning-powered Google Cloud Platform APIs could help deliver features, such as in-audio search, that differentiate the podcast platform from its competitors. In addition, Firebase SDKs and Firebase A/B Testing would enable Castbox to create and analyze new applications, as well as make adjustments based on user feedback. Firebase Realtime Database would allow the business to support tens of thousands of concurrent user connections.

The diligence of the Google Cloud team in advising Li and her team about forthcoming products and services also swayed Castbox towards Google technologies. The business gained the opportunity with Google to join several programs that offered early access to Google innovations.

Signature in-audio search service

Castbox now uses Google Cloud Platform services in the Tokyo, Japan, and U.S. East regions. Cloud Speech-to-Text API plays a key role in delivering Castbox’s signature in-audio search service. This service enables users to search transcriptions of audio content on the platform for words or phrases. The search results incorporate the title of the podcast and the search term in context (for example, within the sentence or sentence excerpt in which it appears). Each use of the word or phrase is time-stamped so it can easily be found. The API enables Castbox developers to apply neural network algorithms to achieve audio-to-text conversion accuracy rates of greater than 96%, while search queries typically experience latency of just 50 milliseconds.

In addition, the latency of comparison data, converting audio to text, is only about 250 milliseconds, contributing to the processing of about 12 minutes worth of audio to text in just 10 minutes. “We can process about 20 hours of audio files in one day,” Li says. “This enables us to transcribe and index all the new episodes of a podcast in that period.”

50,000 concurrent connections

With Firebase Realtime Database, Castbox now supports up to 50,000 concurrent connections to its platform with an average latency per connection of just 10 milliseconds. “Firebase Realtime Database also allows us to continue operating in offline mode, which is extremely helpful if we experience any network disruptions,” Li explains. “When we come back online again, any data is simply synchronized with the database.”

Google BigQuery and the analytics capabilities of Firebase SDKs also enable Castbox to monitor and analyze user behaviors. “Based on our analysis of the data in Google BigQuery, we can determine what type of content users are listening to, how long they like to listen to it, and when they like to listen to it,” Li explains. “This allows us to recommend similar podcasts to each user based on the preferences he or she expressed, encouraging activity on and return visits to our platform.”

“Given our queries may span up to 40 days of data, we may be analyzing up to 1,200 GB at one time. We have no problem doing this with Google BigQuery.”

—Shibin Li, Co-Founder, Castbox

Castbox also uses its analyses of Google BigQuery data to amend banners and summaries on its platform to encourage users to listen to additional content. Furthermore, the service is prepared to make surprise recommendations of content to users based on the preferences and reactions of users with similar tastes.

“We analyze a pool of data growing at up to 30 GB per day,” Li explains. “Given our queries may span up to 40 days of data, we may be analyzing up to 1,200 GB at one time. We have no problem doing this with Google BigQuery.” These analyses also support Castbox’s decision to start creating original content, such as finance and economic news, for its platform.

Checking weekly changes

Castbox does not rely only on analyzing user data to deliver a high-quality experience. The business aggregates user feedback from emails and Google Play reviews to make weekly changes to its platform. It then uses Firebase A/B Testing to check whether these changes are met with a positive user response.

“We are extremely pleased with Google Cloud Platform and Firebase. We have been able to differentiate ourselves from our competitors and provide an attractive option for users at a time when content and entertainment options are exploding. We have a great opportunity with Google to continue to improve the value of our offering to users and build engagement and loyalty.”

—Shibin Li, Co-Founder, Castbox

Castbox’s positive experiences with Google Cloud Platform are encouraging the business to grow its use of the product. “We are trying to move some more services to Google Cloud Platform because it is very stable and scalable,” Li says. The business is keen to explore the capabilities of Cloud Pub/Sub to provide low latency messaging between applications, Cloud Spanner to deliver a distributed relational database service, and Cloud Dataflow to transform and enrich data in stream and batch modes.

“We are extremely pleased with Google Cloud Platform and Firebase,” Li concludes. “We have been able to differentiate ourselves from our competitors and provide an attractive option for users at a time when content and entertainment options are exploding. We have a great opportunity with Google to continue to improve the value of our offering to users and build engagement and loyalty.”