New Enterprise Capabilities and Changes to Cloud Spanner Reduces Cost of Running Workloads by 90 Percent

5389
Of your peers have already read this article.
3:00 Minutes
The most insightful time you'll spend today!
Customers love Cloud Spanner because it gives them the benefits of relational semantics and SQL while also delivering the scale and availability of non-relational databases. Many of these customers want to move even more of their work to Spanner, and have requested smaller instance sizes to support development, testing and small production workloads. We’re happy to announce that they’ll soon get what they asked for: more granular instance sizing is coming to Spanner.
Granular instance sizing will be available in Public Preview soon. With this feature, you can run workloads on Spanner at as low as 1/10th the cost of regular instances, equating to approximately $65/month.
In addition, we are launching new enterprise capabilities to break down operational silos for real-time insights and provide greater database observability to developers:
- Datastream, now in public preview, is a change data capture (CDC) service that allows enterprises to synchronize data across heterogeneous databases and applications. With Spanner support in Datastream, users will be able to stream data from MySQL or Oracle to Spanner reliably and with minimal latency.
- BigQuery federation to Spanner (coming soon) lets users query transactional data residing in Spanner, from BigQuery, without moving or copying data.
- Key Visualizer, available now in public preview, provides interactive monitoring so developers can quickly identify trends and usage patterns in Spanner.
Democratizing access with more granular instance sizing
Today, customers provision Spanner instances by specifying the number of nodes they need to run their workloads. Each node can provide up to 10,000 queries per second (QPS) of reads or 2,000 QPS of writes (writing single rows at 1 KB of data per row) and 2 TB of storage. A Spanner node is replicated across 3 zones for regional instances, and 5 or more zones for multi-regional instances. The choice of node count determines the amount of serving and storage resources that are available to the databases in a given instance.
Historically, the most granular unit for provisioning resources on Spanner has been one node. To enable more granular control, we are introducing Processing Units (PUs); one Spanner node is equal to 1,000 PUs. Customers can now provision in batches of 100 PUs, and get a proportionate amount of compute and storage resources. This will allow teams to run smaller workloads on Spanner at much lower cost. With this feature, customers can start at 100 PUs and scale up as needed in batches of 100 PUs, to up to 1,000 PUs (1 node), all with zero downtime. Subsequently, customers can continue to scale up by adding more nodes, just like what they do today. Customers do have the choice of using either PUs or nodes to provision resources within workloads, when those workloads occupy multiple nodes of capacity.
To illustrate with an example, let’s say a game developer creates a Spanner instance called “baseball-game” in us-central1, with 100 PU compute capacity at $65/month price.

As you can see below, proportional maximum storage of 205 GB is assigned to the 100 PU instance.

Once the game starts becoming popular and resource demands increase, the user edits the instance to increase the compute capacity to 500 PUs with proportional maximum 1 TB of storage.

The game grows in popularity, and the gaming company prepares to launch a much-awaited capability in the game. Anticipating a sharp increase in usage at launch day, the game developers increase the compute capacity to 3,000 PUs. Compute capacity assigned to the instance over a period of time can be viewed in Cloud Monitoring graphs:

Request Access
You can request early access to granular instance sizing feature by filling this form.
Breaking down operational silos with Datastream CDC and BigQuery federation
BigQuery Federation. BigQuery has made analytics easy by bringing together data from multiple sources for seamless analysis. Soon you’ll also be able to analyze data in Spanner directly in BigQuery. With Spanner’s BigQuery federation, you’ll be able to instantly query data residing in Spanner in real-time without moving or copying the data. Simply set up the Spanner as an external data source in BigQuery, as shown below.

Change Data Capture ingest. Now in public preview, Datastream lets you stream change data into Google Cloud from MySQL and Oracle databases. As of today, you can ingest this change data directly into Spanner using a built-in Dataflow template. This lets you migrate data from MySQL and Oracle databases into Spanner in near-real time.
Understanding performance and resource usage with Key Visualizer
Key Visualizer is a new interactive monitoring tool that lets developers and administrators analyze usage patterns in Spanner. It reveals trends and outliers in key performance and resource metrics for databases of any size, helping to optimize queries and reduce infrastructure costs. Designed for performance tuning and instance sizing, Key Visualizer is available today in public preview in the web-based Cloud Console for all Spanner databases at no additional cost. Learn more in the Key Visualizer blog.
Learn more
- To request early access to granular instance sizing in Spanner, fill out this form.
- To get started with Spanner today, create an instance or try it out with a Spanner Qwiklab.
Architecting Data Pipelines Directly Improves Customer Experience at Universe.com

6094
Of your peers have already read this article.
3:30 Minutes
The most insightful time you'll spend today!
At Universe, we serve customers day and night and are always working to make sure they have a great experience, whether online or at one of our live events.
Our technology has to make that possible, and our legacy systems weren’t cutting it anymore. What we needed was a consistent, reliable infrastructure that would help our internal teams provide a fast and innovative ticket-buying experience to customers.
With our data well-managed, we could free up time for our developers to bring new web features to customers, like tailored add-ons at checkout.
Our team of about 20 software engineers needed more flexibility and agility in our infrastructure; we were using various data processing tools, and it wasn’t easy to share data across teams so that everyone saw the same information.
We also needed to incorporate streaming data into the data warehouse to ensure the consistency and integrity of data that’s read in a particular window of time from multiple sources. Our developer teams needed to be able to ship new features faster, and the data back ends were getting in the way.
In addition, when GDPR regulations went into effect, we needed to make sure all our data was anonymized, and we couldn’t do that with our legacy tools.
Finding the right data tools for the job
To make sure our customers were getting a top-notch online experience, we had to make the right technology choices.
Our first step was to centralize multiple data sources and create a single data warehouse that could serve as the foundation for all of our reporting requirements, both internal and external.
The new technology infrastructure we built had to let us move and analyze data easily, so our teams could focus on using that data and insights to better serve our customers.
Previously, we had lots of siloed systems and applications running in AWS. We did a trial using Redshift, but we needed more flexibility than it offered in how we loaded historical data into our cloud data warehouse. Though we were using MongoDB Atlas for our transactional database, it was important to continue using SQL for querying data.
The trial task that really sold us on BigQuery was when we wanted to alter a small table that had about 20 million rows, used for internal reporting.
We needed to add a value, but our PostgreSQL system wouldn’t allow it. Using Apache Beam, we set up a simple pipeline that moved data from the original source into BigQuery to see if we could add the column there.
BigQuery ingested the data and let us add the new value in seconds. That was a significant moment that led us to start looking at how we could build end-to-end solutions on Google Cloud. BigQuery gave us multiple options to load our historical data in batches and build powerful pipelines.
We also explored Google Cloud’s migration tools and data pipeline options. Once we saw how Cloud Dataflow worked, with its Apache Beam back-end, we never looked back. Google Cloud provided us with the data tools we needed to build our data infrastructure.
Cloud for data, and for users
Introducing new technologies isn’t always simple—companies sometimes avoid it altogether because it’s so hard. But our Google Cloud onboarding process has been easy.
It took us less than two months to fully deploy our BigQuery data warehouse using the Cloud Dataflow-Apache Beam combination.
Moving to Google Cloud brought us a lot of technology advantages, but it’s also been hugely helpful for our internal users and customers. The data analytics capabilities that we’re now able to offer users has really impressed our internal teams, including developers and DevOps, even those who haven’t used this type of technology before.
Some internal clients are already entirely self-service. We’ve hosted frequent demos, and also hosted some “hack days,” where we share knowledge with our internal teams to show them what’s possible.
We quickly found that BigQuery helped us solve scale and speed problems.
One of our main pain points had been adding upsell opportunities for customers during the checkout process. The legacy technology hadn’t allowed us to quickly reflect those changes in the data warehouse. With BigQuery, we’re able to do that, and devote fewer resources to making it happen. We’ve also eliminated the time we were spending tuning memory and availability, since BigQuery handles that. Database administration and tuning required specialized knowledge and experience and took up time. With BigQuery, we don’t have to worry about configuring that hardware and software. It just works.
We’ve also eliminated the time we were spending tuning memory and availability, since BigQuery handles that. Database administration and tuning required specialized knowledge and experience and took up time. With BigQuery, we don’t have to worry about configuring that hardware and software. It just works.
Two features in particular that we implemented using BigQuery have helped us improve the performance of our core transactional database. First, using Cloud Dataflow to convert raw MongoDB logs to structured rows under a BigQuery table, which we can then query using SQL to identify slow or underperforming queries. Second, we can now query multiple logging tables using wildcards, since we load Fastly logs to BigQuery.
Along with MongoDB Atlas as our main transactional database, much of our infrastructure now runs as Google Cloud microservices using Google Kubernetes Engine (GKE), including the home page and our payment system. Kubernetes cron jobs power background scheduled jobs, and we also use Cloud Pub/Sub. Cloud Storage handles any data storage if any space constraints emerge.
Our overall performance has increased by about 10x with BigQuery. Both our customers and internal clients, like our sales and finance teams, benefit from the new low-latency reporting. Reports that used to be weekly or monthly are now available in near-real time. It’s not only faster to read records, but faster to move the data, too.
We have Cloud Dataflow pipelines that write to multiple places, and the speed of moving and processing data is incredibly helpful. We stream in financial data using Cloud Dataflow in streaming mode, and plan to have different streaming pipelines as we grow. We have several batch pipelines that run every day. We can move terabytes of data without performance issues, and process more than 100,000 rows of data per second from the underlying database. It used to take us a month to move that volume of data into our data warehouse. With BigQuery, it takes two days.
We’re also enjoying how easy and productive these tools are. They make our life as software engineers easier, so we can focus on the problem at hand, not fighting with our tools.
What’s next for Universe
Our team will continue to push even more into Google Cloud’s data platform. We have plans to explore Cloud Datastore next. We’re also moving our databases to PostgreSQL on GCP, using Cloud Dataflow and Beam. BigQuery’s machine learning tools may also come into play as Universe’s cloud journey evolves, so we can start doing predictive analytics based on our data. We’re looking forward to gaining even more speed and agility to meet our business goals and customer needs.
UKG Ready: Meeting the Needs of Complex Machine Learning Models and Distributed Data Sets

3964
Of your peers have already read this article.
2:30 Minutes
The most insightful time you'll spend today!
Business Problem
UKG Ready primarily operates in the Small and Medium Business (SMB) space, so inherently many customers are forced to operate and make key business decisions with less Workforce Management (WFM) / Human Capital Management (HCM) data. In addition to volume, SMB lacks the variety of data needed to create a dynamic and agile organization. This puts SMB at a major disadvantage compared to larger segments.
Project Goals
People Insights module is committed to surfacing insights to customers in the context of their day-to-day duties and aid in decision making. With the SMB customer data limitations mentioned above, the goal of this project was to create a global dataset that augments individual customer data to bring light to less obvious, yet important information.
Challenges
UKG Ready is a highly configurable application that gives customers the opportunity to build solutions on a platform that meets their specific business needs. High configurability gives high flexibility to customers in their usage of the software. However, it becomes nearly impossible to create a global dataset for machine learning and data insights. UKG Ready manages just under 4 million of the US workforce and some 30,000+ customers. Despite the large employee dataset size, machine learning models that are specific to customers are starved for data because the individual customers have a relatively small employee population. Does that mean we cannot support our SMB customers’ decision making with ML?
Result
Partnering with Google, we were able to develop an approach that allowed us to standardize various domain entities (pay categories, time off codes, job titles, etc.) so that we could build a global dataset to augment SMB customer data. Using machine learning we were able to build a common vocabulary across our customer base. This common vocabulary encapsulates the nuances of how our customers manage their business and yet is generalized and standardized such that the data can be aggregated over the variety of customer configurations. This allows us to serve up practical insights to customers through various use cases. Our partnership allowed us to leverage Google Cloud Services to meet the needs of our complex machine learning models, distributed data sets and CI/CD processes.
How
UKG Ready decided to partner with Google for an end-to-end solution for the analytics offering. This allowed us to focus on our core business logic without having to worry about the platform, environment configurations, performance and scalability of the entire solution. We make use of various Google Cloud services such as Cloud Triggers, Cloud Storage, Cloud Functions, Cloud Composer, Cloud Dataflow, Big Query, Vertex AI, Cloud Pub/Sub… to host our analytics solution. Jenkins manages the entire CI/CD pipelines and cloud environments are configured and deployed using Terraform.
The standardization of business entities problem was solved in three distinct steps:

Step 1: Collecting aggregated data
We needed an approach to collect aggregated data from our highly distributed, sharded, multi-tenant data sources. We developed a custom solution that allows us to extract data aggregated at source for PII and GDPR considerations and transfer to Google Cloud Storage in the fastest manner possible. Data is then transformed and stored in Big Query. Services used: GCS, Cloud Functions, DataFlow, Cloud Composer and Big Query. All processes are orchestrated using Cloud Composer and detailed logging is available in Cloud Logging (Stackdriver).
Step 2: Applying NLP (Natural Language Processing)
Once we had the variety of customer configurations or the business entities available, we then applied NLP algorithms to categorize and standardize these in buckets. This approach assumes that customers use natural language for configurations like job titles, pay codes etc.

String Preparation
The input data for string preparation process is an entity string or several strings, that describe one entity object (like name-description pair or code-name pair). The output represents set of tokens that may be used to run a classification/clustering model. The process of string preparation tokenizes strings, replaces shortcuts, handles abbreviations, translates tokens, handles grammatical errors and mistypes
ML Models
Statistical
The idea of the model is to use defined target classes (clusters) and assign several tokens (anchors) to each of them an entity that has any of those tokens would be “attracted” to appropriate class. All other tokens are weighted according to frequencies of usage of theses tokens in the entities with anchor tokens:
Using anchor tokens, we are building kind-of Word2Vec - dimensionality of vector is equal to number of target classes. The higher the specific dimension (cluster) value, the higher the probability of entity to be included in appropriate cluster. Final prediction for entity tokens list for specific class is sum of weights of all the tokens included. Predicted cluster is a cluster that has maximal prediction score.
Lexical Model
We managed to generate reasonable amount of labeled data during statistical model implementation and testing. That opens a possibility to build “classical” NLP model that uses labeled data to train classification neural network using pretrained layers to produce token embeddings or even string embeddings. We started experimentation with pre-trained models like GloVe and got good results with single words and bi-grams but started getting issues in handling of n-grams. Our Google account team came to our rescue and recommended some white papers that helped formulate our strategy. We now use Tensorflow nnlm-en-dim128 model to produce string embeddings – it was trained on 200B records English Google News corpus and produces for each input string 128-dimensional vector. After that we use several Dense and Dropout layers to build a classification model.
Ensembling
To perform ensembling all the model results for each class are cast to probabilities using softmax transformation with scale normalization. Final predicted probability is maximal average score of both models among all the classes scores – appropriate class is predicted class.
The machine learning models are deployed on Vertex AI and are used in batch predictions. Model performance is captured at every prediction boundary and monitored for quality in production.
Step 3: Making available common vocabulary
Having the standardized vocabulary, we then needed a mechanism to have the results be available in UKG Ready reports and customer specific models like Flight Risk and Fatigue. For this we again used Google Services for orchestration, data transformation and data storage.
Once the modeling is complete, we made the customer specific models leveraging the above architecture be available in Reports. We utilized our proven existing technology choices in GCP for orchestration, data transformation and data storage
Results
We are able to build a common vocabulary of our customers’ business entities with good confidence. And be an expert advisor to our SMB customers in their decision-making using machine learning. With the advice of our Google account team and using Google services we can add value to our product in a relatively short amount of time. And we are not done! We continue to use this platform for new use cases, complex business problems and innovative machine learning solutions.
Sample result:

Special thanks to Kanchana Patlolla , AI Specialist, Google for the collaboration in bringing this to light
Unlocking the Potential of Advanced Analytics with BigQuery and Connected Vehicle Data

2867
Of your peers have already read this article.
2:30 Minutes
The most insightful time you'll spend today!
As software-defined vehicles continue to advance and the quantity of digital services grows to meet consumer demand, the data required to provide these services continue to grow as well. This makes automotive manufacturers and suppliers look for capabilities to log and analyze data, update applications, and extend commands to in-vehicle software.
The challenges the automotive sector faces can be quantified. A modern vehicle contains upwards of 70 electronic control units (ECUs), most of which are connected to one or more sensors. Not only is it now possible to exactly measure many aspects of vehicle performance, but new options become available. Using LIDAR (light detection and ranging), for example, vehicles are achieving higher levels of autonomy; this leads to a data stream from such demanding applications that may reach 25 GB per hour. For the in-vehicle processing of data, 100 million lines of software code may be present — more than a fighter jet. This in-vehicle code will have to be maintained with updates and new functionalities.
Access to the data will allow manufacturers to gain valuable insights into operational details of their vehicles. The use of this data can help to reduce costs and risks, increase ROI, support ESG initiatives, and provide valuable insights to develop innovative solutions and shorten the time to value for Electric Vehicle innovations.
Sibros’ Deep Connected Platform (DCP) makes it possible for these manufacturers to build and launch new connected vehicle use cases from production to post-sale at scale by connecting and managing all software and data throughout every life cycle stage. A key component of this platform is the Sibros Deep Logger that provides capabilities like the following:
- Full configurability of what to record, when to record it, and how fast to record it.
- High resolution timestamps of all Controller Area Network (CAN) messages.
- Dynamic application of live log configurations to receive new data points without deploying new software.
For example, properly analyzed engine data enables true predictive maintenance for the first time, which creates the option to repair or replace components before failure happens. Another example would be the evaluation of data regarding the use of certain in-car features with the goal to redesign its interior.
Two other components of the DCP are software updates and remote commands to ECUs. The DCP on Google Cloud enables seamless integration with any vehicle architecture and provides OEMs and suppliers with the platform to manage connected vehicle data at rest and in transit using a proven and secure way on a global scale.

OEMs can pull data through APIs provided by Sibros into Google Data Cloud (including BigQuery) to gain access to the rich information data sets provided by the DCP within their environment and blend this data with their first party data sets to provide value insights for their business. Some of the Connected Vehicle insights that DCP information enables are:
- Damage prevention, improved operation, or development of the next generation of engines with insights from complex analyses that could consider parameters like model, engine type, mileage, overall speed, temperature, air pressure, load, services, and more.
- The combination of electric vehicle battery usage data like charging cycles, engine performance, and battery age with contributing factors as the use of the air conditioning to determine if such factors contribute to hazardous battery conditions and for improved battery development.
- Cross-organization collaboration in R&D by the provision of information on all these metrics and more from real-world driving, like engine knock data and even tire pressure.
- Google Cloud’s unified data cloud offering provides a complete platform for building data-driven applications like those from Sibros — from simplified data ingestion, processing, and storage to powerful analytics, AI, ML, and data sharing capabilities — integrated with Google Cloud. With a diverse partner ecosystem and support for multi-cloud, open-source tools and APIs, Google Cloud provides Sibros the portability and the extensibility they need to avoid data lock-in.
“Software has an ever increasing importance in the automotive world, even more so with electric vehicles and new mobility services. Google Cloud is partnering with Sibros to bring their award winning Deep Connected Platform to deliver high frequency, low latency over-the-air software updates, data logging & diagnostics capabilities to our automotive customers, leveraging the security and scale of Google Cloud. This is revolutionizing everything from development cycles to business models and customer relationships.” — Matthias Breunig, Director, Global Automotive Solutions, Google Cloud
Through Built with BigQuery, Google Cloud is helping tech companies like Sibros build innovative applications on Google’s Data Cloud with simplified access to technology, helpful and dedicated engineering support, and joint go-to-market programs.
“Sibros is looking forward to partnering with Google Cloud, which will enable vehicle manufacturers and suppliers to reach the next level in their use of data. Sibros solutions for Deep Data Logging and Updating on the Google Data Cloud, combined with Google BigQuery, will help them to mitigate risks, reduce costs, add innovative products, and introduce value-added use cases.” — Xiaojian Huang, Chief Digital Officer, Software, Sibros
Sibros and Google Cloud are driving Connected Mobility transformation to help our customers accelerate R&D innovation, power efficient operations, and unlock software-defined vehicle use cases with a full stack connected vehicle platform. Click here to learn more about Sibros on Google Cloud.
Data Cloud Skills to Pick Up in 2022: Google Experts Recommended

4961
Of your peers have already read this article.
3:00 Minutes
The most insightful time you'll spend today!
It’s 2022 and nanosatellites, NFTs, and autonomous cars that deliver your pizza are in full force. In a world where people rely on simple technology to untangle complex problems, companies must deliver simple experiences to be successful in today’s landscape. For many cloud providers this means enabling tightly integrated data offerings that simplify the data delivery process without losing sight of the sophisticated needs of the modern data consumer.
But while the name of the game is helping companies reach informed decisions from their data simpler and faster, what about the data practitioners – data analysts, data engineers, database administrators, developers, etc – who use these cloud data tools and technologies everyday? To proactively stay ahead of data cloud market trends in 2022 should data practitioners invest their time in specializing their data cloud skill sets (e.g. go deep in, say, data pipelining skills) or instead invest their time generalizing their data cloud skill sets (e.g. growing proficiencies in a mix of data analytics, databases, AI/ML, and more domains)?
Skill deep or wide with data – that is the question
For Abdul Razack, VP, Solutions Engineering, Technology Solutions and Strategy at Google Cloud, the answer is a bit of both.
“Data practitioners need to be broad in terms of their technology skills, but specialized with respect to the domain or domains in which they apply them. The reason why is because many things that used to be separate skill sets are now converging – like business analytics, streaming, machine learning, data pipelines, and data warehousing. Data practitioners need to be able to implement end-to-end workflows that solve specific business problems using skills from each category.”
It’s true, thousands of customers are choosing Google’s data cloud because it offers a unified and open approach to cloud that enables their practitioners to break down silos, begin and end projects without leaving the data platform, and innovate faster across their organization.
The data practitioners who mirror Google data cloud’s frame of mind of being smart and agile across data domains in their skilling and learning will reap the benefits of solving more nuanced problems – building out internet-scale applications, fine tuning smart processes with analytics and AI, constructing data meshes that make product building simple, etc – at a larger scale than they would if they specialized in just one or two areas alone.
“Of course at the end of the day it depends on what tools a data practitioner is using to complete their workflows. There’s only so much you can learn and skills you can develop when you’re using limited tools. Growing data proficiencies across the board is made a lot easier when you’re using a data platform like BigQuery to address all these needs. BigQuery eliminates the choices you have to make – for instance you don’t have to choose between streaming data and data at rest, batch and realtime, or business intelligence and data science. This freedom gives data professionals a huge advantage when they’re building their skill sets and taking on more complex projects.” -Abdul Razack – VP, Solutions Engineering, Technology Solutions and Strategy, Google Cloud
Knowing your value is half the battle when upskilling
While some experts think technology is the limiting factor of whether or not you can even go wide or go deep in the first place, others like Google Cloud’s Head of Data and Analytics Bruno Aziza purport that it also depends on who you are, who you wish to be, and what investments your company is making to ensure you can become that person.
“If you wish to set yourself up to be a Chief Data Officer, then you’ll want to understand how technologies fit together across your data estate first” said Aziza. “Only after you feel like you’re the go-to ‘data person’ can you then decide which part of the technology stack you want to double-down on.”
But technology isn’t everything. Aziza notes, “Make sure you focus on the business impact that your data work provides. You want to spend as much time as you can with your business counterparts to understand their business goals and challenges. The Harvard Business Review provides great guidance on how to succeed as a Chief Data Officer.”
Even if you don’t have your sights set on a C-suite role, both Aziza and Razack contend that the number one skill data practitioners should tackle in 2022 is actually a broad and perhaps abstract one: develop and exercise the curiosity to solve problems with a data-driven strategy.
That is, today’s data practitioners should always be interested in educating themselves in the industry and continually upskilling in something. And their employers should also be invested in helping practitioners develop those interests, most likely through exposure to learning materials, engaging in career conversations, subsidized courses, or incentives attached to pursuing a new certification or skill.
“Every industry is going through a digital transformation and the ability to identify what data to collect, how to prepare the data, and how to derive insights from it is critical. Therefore, the ability to find business challenges and formulate a data-driven approach to address those problems is the most important skill to have.” Abdul Razack – VP, Solutions Engineering, Technology Solutions and Strategy, Google Cloud.
Take the example of the “Data Mesh” I just wrote about in VentureBeat. You’ll find 3 types of attitudes towards this new concept. There are Disciples who encourage continued learning only from the source – like the author of a new book or the creator of a theory. There are Distractors who tell you that new skills, trends, and technologies are fake news. And there are Distorters like vendors who will sell you one easy fix solution. But it’s the data practitioner who needs to proceed with caution when interacting with all three types and forge their own path to discovering the truth when they’re learning and building skills. And for better or worse, this comes with trial and error, experimentation, and an eagerness to grow relative to where they began.”
Ready to start data upskilling? Start here.
For those interested in keeping up their data curiosities, check out our Data Journeys video series. Each week Bruno Aziza investigates a new authentic customer’s data journey – from migrating to cloud or building a data platform to carrying out new data for good initiatives. Learn how they did it, their data dos and don’ts, and what’s next for them on their journey. These videos include a flavor of both specializing your data competencies and broadening your data competencies.
For those interested in deepskilling, connect with Google’s data community at our upcoming virtual event: Latest Google Cloud data analytics innovations. Register and save your spot now to get your data questions answered live by GCP’s top data leaders and watch demos from our latest products and features including BigQuery, Dataproc, Dataplex, Dataflow, and more.
If you have any questions or need support along your learning journey – we’re here for you! Sign up to be a Google Cloud Innovator, and join the Google Cloud Data Analytics Community.
Google Cloud’s Firebase Realtime Database and BigQuery AllowsCastbox to Ramp Up Customer Experience

8148
Of your peers have already read this article.
3:30 Minutes
The most insightful time you'll spend today!
Demand for spoken audio content such as podcasts remains robust despite the proliferation of video services and other entertainment options for consumers. Shibin Li, Co-founder of Castbox, credits growth of the global podcast platform to the following: speed and availability, market-leading features, the proliferation of smart devices to deliver audio content, and activities — such as driving and working around the house — that make consuming video difficult.
Founded in 2016 and headquartered in Beijing, China, Castbox enables users to locate, access, and create spoken audio content. Available on iOS and Android, Castbox supports 50 million podcasts, on-demand radio programs, and audiobooks in 70 languages from 175 countries. The platform hosts about 2 million users per day and is the largest podcast platform on Android.
Castbox includes a range of features that build on its core service to provide a high-quality user experience. These features include curated podcast recommendations and in-audio search.
A combination of services
At its inception, Castbox relied on a combination of a multinational cloud services, as well as Google Cloud Platform (GCP) services, including the Google BigQuery analytics data warehouse and Cloud APIs to provide programmatic interfaces with Google Cloud services, and a range of services from the Google mobile development platform Firebase.
However, as Castbox matured and its user base expanded, the business increased its reliance on Google Cloud Platform and Firebase.
“We needed to access stable cloud services as we could not tolerate long periods of downtime that would compromise the user experience,” says Li. “Furthermore, we had to support up to 50,000 concurrent connections, and potentially more in future, without disruption.”
“Based on our analysis of the data in Google BigQuery, we can determine what type of content users are listening to, how long they like to listen to it, and when they like to listen to it. This allows us to recommend similar podcasts to each user based on the preferences he or she expressed, encouraging activity on and return visits to our platform.”
—Shibin Li, Co-Founder, Castbox
Competitive differentiation
Castbox also found machine learning-powered Google Cloud Platform APIs could help deliver features, such as in-audio search, that differentiate the podcast platform from its competitors. In addition, Firebase SDKs and Firebase A/B Testing would enable Castbox to create and analyze new applications, as well as make adjustments based on user feedback. Firebase Realtime Database would allow the business to support tens of thousands of concurrent user connections.
The diligence of the Google Cloud team in advising Li and her team about forthcoming products and services also swayed Castbox towards Google technologies. The business gained the opportunity with Google to join several programs that offered early access to Google innovations.
Signature in-audio search service
Castbox now uses Google Cloud Platform services in the Tokyo, Japan, and U.S. East regions. Cloud Speech-to-Text API plays a key role in delivering Castbox’s signature in-audio search service. This service enables users to search transcriptions of audio content on the platform for words or phrases. The search results incorporate the title of the podcast and the search term in context (for example, within the sentence or sentence excerpt in which it appears). Each use of the word or phrase is time-stamped so it can easily be found. The API enables Castbox developers to apply neural network algorithms to achieve audio-to-text conversion accuracy rates of greater than 96%, while search queries typically experience latency of just 50 milliseconds.
In addition, the latency of comparison data, converting audio to text, is only about 250 milliseconds, contributing to the processing of about 12 minutes worth of audio to text in just 10 minutes. “We can process about 20 hours of audio files in one day,” Li says. “This enables us to transcribe and index all the new episodes of a podcast in that period.”
50,000 concurrent connections
With Firebase Realtime Database, Castbox now supports up to 50,000 concurrent connections to its platform with an average latency per connection of just 10 milliseconds. “Firebase Realtime Database also allows us to continue operating in offline mode, which is extremely helpful if we experience any network disruptions,” Li explains. “When we come back online again, any data is simply synchronized with the database.”
Google BigQuery and the analytics capabilities of Firebase SDKs also enable Castbox to monitor and analyze user behaviors. “Based on our analysis of the data in Google BigQuery, we can determine what type of content users are listening to, how long they like to listen to it, and when they like to listen to it,” Li explains. “This allows us to recommend similar podcasts to each user based on the preferences he or she expressed, encouraging activity on and return visits to our platform.”
“Given our queries may span up to 40 days of data, we may be analyzing up to 1,200 GB at one time. We have no problem doing this with Google BigQuery.”
—Shibin Li, Co-Founder, Castbox
Castbox also uses its analyses of Google BigQuery data to amend banners and summaries on its platform to encourage users to listen to additional content. Furthermore, the service is prepared to make surprise recommendations of content to users based on the preferences and reactions of users with similar tastes.
“We analyze a pool of data growing at up to 30 GB per day,” Li explains. “Given our queries may span up to 40 days of data, we may be analyzing up to 1,200 GB at one time. We have no problem doing this with Google BigQuery.” These analyses also support Castbox’s decision to start creating original content, such as finance and economic news, for its platform.
Checking weekly changes
Castbox does not rely only on analyzing user data to deliver a high-quality experience. The business aggregates user feedback from emails and Google Play reviews to make weekly changes to its platform. It then uses Firebase A/B Testing to check whether these changes are met with a positive user response.
“We are extremely pleased with Google Cloud Platform and Firebase. We have been able to differentiate ourselves from our competitors and provide an attractive option for users at a time when content and entertainment options are exploding. We have a great opportunity with Google to continue to improve the value of our offering to users and build engagement and loyalty.”
—Shibin Li, Co-Founder, Castbox
Castbox’s positive experiences with Google Cloud Platform are encouraging the business to grow its use of the product. “We are trying to move some more services to Google Cloud Platform because it is very stable and scalable,” Li says. The business is keen to explore the capabilities of Cloud Pub/Sub to provide low latency messaging between applications, Cloud Spanner to deliver a distributed relational database service, and Cloud Dataflow to transform and enrich data in stream and batch modes.
“We are extremely pleased with Google Cloud Platform and Firebase,” Li concludes. “We have been able to differentiate ourselves from our competitors and provide an attractive option for users at a time when content and entertainment options are exploding. We have a great opportunity with Google to continue to improve the value of our offering to users and build engagement and loyalty.”
More Relevant Stories for Your Company

Home Depot Leverages Google Cloud’s BigQuery and DataFlow to Break Data Silos and Craft Personalized CX
The Home Depot, Inc., is the world’s largest home improvement retailer with annual revenue of over $151B. Delighting our customers—whether do-it-yourselfers or professionals—by providing the home improvement products, services, and equipment rentals they need, when they need them, is key to our success. We operate more than 2,300 stores throughout

The Fantastic Story of How BMG Enables a Micropayments Strategy So Music Artists Get Paid
The music industry is rapidly changing. Only 20 years ago, the availability of music and the infrastructure that was required to make an album a sales success were incredibly complex and expensive. With the decline of physical sales and a fundamental shift to digital, music streaming now accounts for more

Quick Recap on Google Cloud: Latest News, Launches, Updates, Events and More
Want to know the latest from Google Cloud? Find it here in one handy location. Check back regularly for our newest updates, announcements, resources, events, learning opportunities, and more. Tip: Not sure where to find what you’re looking for on the Google Cloud blog? Start here: Google Cloud blog 101: Full list

Fairygoodboss and Google Cloud Tied to Advancing Diversity and Female Leadership at Workplaces
May is Asian American Pacific Islander Heritage Month — a time for us to come together to celebrate and remember the important people and history of Asian and Pacific Island heritage. This feature highlights founder Georgene Huang and how her startup Fairygodboss uses easy to use Google Cloud tools to






