3275
Of your peers have already watched this video.
5:00 Minutes
The most insightful time you'll spend today!
What is Dataflow?
What is Dataflow, and how can you use it for your data processing needs?
In this episode of Google Cloud Drawing Board, Priyanka Vergadia walks you through Dataflow, a serverless system for processing and enriching data, supporting both streaming and batch models.
Here’s what’s inside:
0:00 – 0:14 Video snapshot
0:15 – 0:57 What is Dataflow
0:58 – 2:23 How does Dataflow work
2:24 – 3:34 How to use Dataflow
3:35 – 3:58 Dataflow security
3:58 – 4:24 What does it cost to use Dataflow
4:25 – 4:55 Dataflow use cases
6809
Of your peers have already watched this video.
2:30 Minutes
The most insightful time you'll spend today!
What is BigQuery?
BigQuery is Google Cloud’s enterprise data warehouse designed to help you ingest, store, analyze, and visualize big data with ease.
Organizations rely on data warehouses to aggregate data from disparate sources, process it, and make it readily available for data analysis that supports their strategic decision-making.
You can ingest data into BigQuery either through batch uploading or streaming data directly to deliver real-time insights.
As a fully-managed data warehouse, Google takes care of the infrastructure so you can focus on analyzing your data up to petabyte scale.
BigQuery supports the same Structured Query Language, or SQL, for analyzing your data, which you may be familiar with if you’ve worked with ANSI-compliant relational databases in the past.
If you’re looking to create machine learning models using your enterprise data, you can do so with BigQuery ML.
With only a few lines of SQL, you can train and execute models on your BigQuery data without needing to move it around.
When it comes time to visualize your data, BigQuery integrates with Looker, as well as several other business intelligence tools across our partner ecosystem.
Now, how do you use BigQuery?
Luckily, it’s straightforward to get up and running with BigQuery.
After creating a GCP project, you can immediately start querying public data sets, which Google Cloud hosts and makes available to all BigQuery users, or you can load your own data into BigQuery to analyze.
Interacting with BigQuery to load data, run queries, or even create ML models can be done in three different ways.
First is by using the UI and the Cloud Console. Second is by using the BigQuery command line tool. And third is by making calls to the BigQuery API, using client libraries available in several languages.
BigQuery is integrated with Google Cloud’s Identity and Access Management Service so you can securely share your data and analytical insights across the organization.
What does it cost to use BigQuery?
With BigQuery, you pay for storing and querying data and streaming inserts.
Loading and exporting data are free of charge.
Storage costs are based on the amount of data stored and have two rates based on how often the data is changing.
Query costs can be either on demand, meaning you are charged per query by the amount of data processed, or flat rate for customers who want to purchase dedicated resources.
6535
Of your peers have already watched this video.
24:00 Minutes
The most insightful time you'll spend today!
Swarovski’s Journey towards Online and Offline Conversion with Predictive Analytics
Luxury brand and leader in crystals and glass production, Swarovski has charmed customers with its exquisite collections for over 125 years. To understand their customers better and map their online behaviors, Swarovski had to overcome prediction hurdles as majority of the purchases are not frequent or habitual. They are mostly impulse buys or have no rational behind the purchase in order for the brand to accurately map customers’ interest and delight them with relevant personalization or website customization strategy.
Swarovski used a machine learning (ML) model to predict the most performing SKUs and list of products based on both online and offline indicators to target buyers. A score was assigned to each product in the list and was personalized at the country level that delivered relevant insights. Swarovski is aiming to expand the product listing page to personalize at customer level. Watch the video to dive deep into Swarovski’s data analytics efforts to answer complex questions, reporting and prediction using both online and offline data.
Google Cloud’s Firebase Realtime Database and BigQuery AllowsCastbox to Ramp Up Customer Experience
8164
Of your peers have already read this article.
3:30 Minutes
The most insightful time you'll spend today!
Demand for spoken audio content such as podcasts remains robust despite the proliferation of video services and other entertainment options for consumers. Shibin Li, Co-founder of Castbox, credits growth of the global podcast platform to the following: speed and availability, market-leading features, the proliferation of smart devices to deliver audio content, and activities — such as driving and working around the house — that make consuming video difficult.
Founded in 2016 and headquartered in Beijing, China, Castbox enables users to locate, access, and create spoken audio content. Available on iOS and Android, Castbox supports 50 million podcasts, on-demand radio programs, and audiobooks in 70 languages from 175 countries. The platform hosts about 2 million users per day and is the largest podcast platform on Android.
Castbox includes a range of features that build on its core service to provide a high-quality user experience. These features include curated podcast recommendations and in-audio search.
A combination of services
At its inception, Castbox relied on a combination of a multinational cloud services, as well as Google Cloud Platform (GCP) services, including the Google BigQuery analytics data warehouse and Cloud APIs to provide programmatic interfaces with Google Cloud services, and a range of services from the Google mobile development platform Firebase.
However, as Castbox matured and its user base expanded, the business increased its reliance on Google Cloud Platform and Firebase.
“We needed to access stable cloud services as we could not tolerate long periods of downtime that would compromise the user experience,” says Li. “Furthermore, we had to support up to 50,000 concurrent connections, and potentially more in future, without disruption.”
“Based on our analysis of the data in Google BigQuery, we can determine what type of content users are listening to, how long they like to listen to it, and when they like to listen to it. This allows us to recommend similar podcasts to each user based on the preferences he or she expressed, encouraging activity on and return visits to our platform.”
—Shibin Li, Co-Founder, Castbox
Competitive differentiation
Castbox also found machine learning-powered Google Cloud Platform APIs could help deliver features, such as in-audio search, that differentiate the podcast platform from its competitors. In addition, Firebase SDKs and Firebase A/B Testing would enable Castbox to create and analyze new applications, as well as make adjustments based on user feedback. Firebase Realtime Database would allow the business to support tens of thousands of concurrent user connections.
The diligence of the Google Cloud team in advising Li and her team about forthcoming products and services also swayed Castbox towards Google technologies. The business gained the opportunity with Google to join several programs that offered early access to Google innovations.
Signature in-audio search service
Castbox now uses Google Cloud Platform services in the Tokyo, Japan, and U.S. East regions. Cloud Speech-to-Text API plays a key role in delivering Castbox’s signature in-audio search service. This service enables users to search transcriptions of audio content on the platform for words or phrases. The search results incorporate the title of the podcast and the search term in context (for example, within the sentence or sentence excerpt in which it appears). Each use of the word or phrase is time-stamped so it can easily be found. The API enables Castbox developers to apply neural network algorithms to achieve audio-to-text conversion accuracy rates of greater than 96%, while search queries typically experience latency of just 50 milliseconds.
In addition, the latency of comparison data, converting audio to text, is only about 250 milliseconds, contributing to the processing of about 12 minutes worth of audio to text in just 10 minutes. “We can process about 20 hours of audio files in one day,” Li says. “This enables us to transcribe and index all the new episodes of a podcast in that period.”
50,000 concurrent connections
With Firebase Realtime Database, Castbox now supports up to 50,000 concurrent connections to its platform with an average latency per connection of just 10 milliseconds. “Firebase Realtime Database also allows us to continue operating in offline mode, which is extremely helpful if we experience any network disruptions,” Li explains. “When we come back online again, any data is simply synchronized with the database.”
Google BigQuery and the analytics capabilities of Firebase SDKs also enable Castbox to monitor and analyze user behaviors. “Based on our analysis of the data in Google BigQuery, we can determine what type of content users are listening to, how long they like to listen to it, and when they like to listen to it,” Li explains. “This allows us to recommend similar podcasts to each user based on the preferences he or she expressed, encouraging activity on and return visits to our platform.”
“Given our queries may span up to 40 days of data, we may be analyzing up to 1,200 GB at one time. We have no problem doing this with Google BigQuery.”
—Shibin Li, Co-Founder, Castbox
Castbox also uses its analyses of Google BigQuery data to amend banners and summaries on its platform to encourage users to listen to additional content. Furthermore, the service is prepared to make surprise recommendations of content to users based on the preferences and reactions of users with similar tastes.
“We analyze a pool of data growing at up to 30 GB per day,” Li explains. “Given our queries may span up to 40 days of data, we may be analyzing up to 1,200 GB at one time. We have no problem doing this with Google BigQuery.” These analyses also support Castbox’s decision to start creating original content, such as finance and economic news, for its platform.
Checking weekly changes
Castbox does not rely only on analyzing user data to deliver a high-quality experience. The business aggregates user feedback from emails and Google Play reviews to make weekly changes to its platform. It then uses Firebase A/B Testing to check whether these changes are met with a positive user response.
“We are extremely pleased with Google Cloud Platform and Firebase. We have been able to differentiate ourselves from our competitors and provide an attractive option for users at a time when content and entertainment options are exploding. We have a great opportunity with Google to continue to improve the value of our offering to users and build engagement and loyalty.”
—Shibin Li, Co-Founder, Castbox
Castbox’s positive experiences with Google Cloud Platform are encouraging the business to grow its use of the product. “We are trying to move some more services to Google Cloud Platform because it is very stable and scalable,” Li says. The business is keen to explore the capabilities of Cloud Pub/Sub to provide low latency messaging between applications, Cloud Spanner to deliver a distributed relational database service, and Cloud Dataflow to transform and enrich data in stream and batch modes.
“We are extremely pleased with Google Cloud Platform and Firebase,” Li concludes. “We have been able to differentiate ourselves from our competitors and provide an attractive option for users at a time when content and entertainment options are exploding. We have a great opportunity with Google to continue to improve the value of our offering to users and build engagement and loyalty.”
3036
Of your peers have already watched this video.
24:00 Minutes
The most insightful time you'll spend today!
How to Modernize Your Data infrastructure and Analytics: Case Study
Data modernization is key to digital transformation. Legacy systems are slow, expensive and cannot keep up with changing requirements. Data teams often spend more time on data pipelines than they do on data analysis or data science.
in this video, Harish Ramachandraiah, Director Eng. & Analytics, Sunrun, speaks with Joel Mckelvey, Product Marketing Manager, Google Cloud. They discuss how Sunrun leveraged Looker and BigQuery to reduce the complexity of legacy extract-transform-load processes, improve database performance, and adapt quickly to changes in data.
Now Sunrun makes data accessible where it’s needed, in near-real time.
Mckelvey shows how a modern data stack helps companies simplify the data pipeline, consolidate vital data, and accelerate analytics performance while improving agility, efficiency, and data governance.
3633
Of your peers have already watched this video.
5:00 Minutes
The most insightful time you'll spend today!
Quick Demo: How to Push Data to PubSub, Dataflow, and BigQuery to Improve Customer Experience
How can you use big data to raise your business’ customer experience levels? It’s a question many
Imagine for
Here’s an example: Customers taking a Monday morning flights from certain hubs would indicate that they are business users. These customers would appreciate being offered in-flight Wifi. Such a targetted offer would increase both customer experience and revenues.
In this five-minute demo, you’ll learn how easy it is to query data sets to reveal, for example, which passengers have connecting flights—and also have checked-in luggage—and would
Watch it now!
More Relevant Stories for Your Company
Google Cloud Data Heroes Series: Honoring Data Practitioner’s Journey and Learning with GCP
Google Cloud Data Heroes is a series where we share stories of the everyday heroes who use our data analytics tools to do amazing things. Like any good superhero tale, we explore our Google Cloud Data Heroes’ origin stories, how they moved from data chaos to a data-driven environment, what
Don’t sweat the big stuff. Make it Google’s problem
Need to interpolate new time series data values over 5 billion rows? Don’t reach for python. Make that Google’s problem and do it in BigQuery. Need to aggregate petabytes of geospatial data across arbitrary polygons and put it on a map for analysis? Make that Google’s problem and use BQ-GIS.
How TeamSnap Improved Return on Ad Spend Significantly
Anyone who has ever coached or played on a sports team, or had a child involved in sports, knows how difficult scheduling and logistics can be. From game and practice schedules, to uniforms and who’s bringing the snacks, it can be a lot for coaches, administrators, parents, and players to
Looking for a Cloud Data Warehouse? Find out Why Forrester Thinks Google BigQuery is a Leader
We are thrilled to announce that Google has been named a Leader in The Forrester Wave™: Cloud Data Warehouse, Q1 2021 report. For more than a decade, BigQuery, our petabyte-scale cloud data warehouse, has been in a class of its own. We're excited to share this recognition and we want to thank
