How Connected-Stories Uses BigQuery and AI/ML to Craft Personalized Ad Experiences

4017
Of your peers have already read this article.
4:00 Minutes
The most insightful time you'll spend today!
Editor’s note: The post is part of a series highlighting our awesome partners, and their solutions, that are Built with BigQuery
In the field of producing engaging video content such as ads, many marketers ignore the power of data to improve their creative efforts to meet the consumers’ need for personalized messages. The demand for creative tech to efficiently personalize is real as marketers need personalized video Ads to reach their audience with the right message at the right time. Data, Insights and Technology are the main ingredients to deliver this value while ensuring security and privacy requirements are met. The Connected-Stories team partnered with Google Cloud to build a platform for Ad personalization. Google Data Cloud and BigQuery are at the forefront to assimilate data, leverage ML models, create personalized ads, and capitalize on real-time intelligence as the core features of the Connected-Stories NEXT platform.
Connected-Stories NEXT is an end-to-end creative management platform to develop, serve, and optimize interactive video and display ads that scale across any channel. The platform ingests first-party data to create custom ML models, measure numerous third-party data points to help brands develop unique customer journeys and create videos that their data signals can drive. An intelligent feedback loop passes real-time data back, enabling brands to make data-driven and actionable video ads that take the brand’s campaigns to the next level.

The core use case of the NEXT platform revolves around collecting user’s interaction data and optimizing for precision and speed to create an actionable Ad experience that is personalized for each user. The platform processes complex data points to create interactive data visualizations that allow for accurate analysis. The platform uses Vertex AI to access managed tools, workflows, and infrastructure to build, deploy, and scale ML models that have improved the accuracy to identify segments for further analysis.
The platform ingests 200M data events with peaks and valleys of activity. These events are processed to generate dashboards that enable users to visualize metrics based on filters in real-time. These dashboards have high performance requirements in terms of a responsive user interface under constantly changing data dimensions.
Google Cloud’s serverless stack coupled with limitless data cloud infrastructure has been the core to the NEXT platform’s data-driven innovation. The growing volume of data ingested, streamed and processed were scaled uniformly across the compute, storage and analytical layers of solution. A lean development team at Connected-Stories were able to focus all-in on the solution, while the serverless stack scaled, lowered attack service in terms of security and optimized the cost footprint through pay-as-you-go features.
BigQuery has been the backbone to support the vast amounts of data spreading over multiple geos resulting in workloads running at petabyte scale. BigQuery’s fully managed serverless architecture, real-time streaming, built-in machine learning and rich business intelligence capabilities distinguishes itself from a cloud data warehouse. It is the foundation needed to approach data and serve users in an unlimited number of ways. For an application with zero tolerance for failure, given its fully managed nature, BigQuery handles replication, recovery, data distributed optimization and management.
The platform’s requirements include the need for low maintenance, constantly ingesting and refreshing data and smart-tuning of aggregated data. These capabilities can be implemented by BigQuery’s materialized views feature. Materialized views are useful for precomputed views that regularly cache query results for better performance. These views possess the innate feature to read only the delta change from base tables and calculate the up-to-date aggregations. Materialized views impart faster outputs and consume fewer resources while reducing the cost footprint.
Some key considerations in using Google cloud and focusing on the Serverless stack include: quick onboarding to development, prototyping in short sprints and ease of preparing data in a rapidly changing environment. Typical considerations around low code / no code include data transformation, aggregation and reduced deployment time. These considerations are fulfilled through using serverless capabilities within Google Cloud such as PubSub, Cloud Storage, Cloud Run, Cloud Composer, Dataflow and BigQuery as described in the Architecture diagram below. The use of each of these components and services are described below.

- Input/Ingest: At a high-level, microservices hosted in Cloud Run collect and aggregate incoming Ads events.
- Enrichment: The output of this stage is a Pub-Sub message enriched with more attributes based on a pre-configured campaign.
- Store: a Cloud Dataflow streaming job to create text files in Cloud Storage buckets.
- Trigger: Cloud Composer triggers the spark jobs based on text files to process and group them to produce desired output as one record per impression, a logical group of events.
- Deploy: Cloud Build is then used to automate all deployments.
Thus far, all Google cloud managed services work together to ingest, store and trigger the orchestration, all of which are scalable based on configurations including autoscaling capabilities.
- Visualization: A visualization tool reads data from BigQuery to compute pre-aggregations required for each dashboard.
- Data Model Evolution considerations: Though the solution served the purpose of creating pre-aggregations, as the data model evolved by adding a column or creating a new table, it led to recreating pre-aggregations and querying the data again. Alternatively, creating aggregate tables as an extra output of current ETLs seemed like a viable option. However, this would increase the cost and complexity of jobs. A similar situation to reprocess or update aggregated tables would occur as data is updated.
Precomputed views of data that is periodically cached are critical to reach the audience with the right message at the right time.
- Performance: In order to increase the performance of the platform, we need to have regularly precomputed views of the data, cached .
- Materialized Views: Consumers of these views needed faster response times, to consume fewer resources and output only the changes in comparison to a base table. BigQuery Materialized views were used to solve this very requirement. Materialized views have been highly leveraged to optimize the design resulting in lesser maintenance and access to fresh data with high performance with a relatively low technical investment in creating and maintaining SQL code.
- Dashboards: Application dashboards pointing to the Materialized views are highly performant and provide a view into fresh data.
- Custom Reports with Vertex AI Notebooks: Vertex AI notebooks directly read data from BigQuery to produce custom reports for a subset of customers. Vertex AI has been hugely beneficial to data analysts, where an environment with pre-installed libraries simplifies the readiness to use. Vertex AI Workbench notebooks are used to share these reports within the team allowing them to work always on the cloud without having the need to download data at any time. Besides, it increases the velocity to develop and test ML models faster.
The NEXT platform has yielded benefits such as customers having the ability to create unique consumer journeys powered by AI / ML personalization triggers, using first-party data and business intelligence tools to capitalize on real-time creative intelligence, which is a dashboard to measure campaign performance for cross-functional teams to analyze the impact of Ad content experience at a granular level. All of these while ensuring controlled access to data to enrich data without moving across clouds. The NEXT platform can keep up with increased demands for agility, scalability and reliability through the underlying usage of Google Cloud.
Partnering with Google, in the context of the Google Built with BigQuery program has surfaced the differentiated value in areas of creating interactive personalized Ads by using real-time data. In addition, by sharing this data across organizations as assets, ML models have fueled higher levels of innovation. Connected-Stories plan to deepen the penetration into the entire spectrum of services offered in the AI/ML area to enhance core functionality and provide newer capabilities to the platform.
Click here to learn more about Connected-Stories NEXT Platform capabilities.
The Built with BigQuery Advantage for ISVs
Through Built with BigQuery, launched in April ‘22 as part of Google Data Cloud Summit, Google is helping tech companies like Connected-Stories co-innovate in building applications that leverage Google’s data cloud with simplified access to technology, helpful and dedicated engineering support, and joint go-to-market programs. Participating companies can:
- Get started fast with a Google-funded, pre-configured sandbox.
- Accelerate product design and architecture through access to designated technical experts from the ISV Center of Excellence who can share insights from key use cases, architectural patterns, and best practices encountered in the field.
- Amplify success with joint marketing programs to drive awareness, generate demand, and increase adoption.
The Google Data Cloud spectrum of products and specifically BigQuery give ISVs the advantage of a powerful, highly scalable data warehouse that’s integrated with Google Cloud’s open, secure, sustainable platform. And with a huge and expanding partner ecosystem and support for multi-cloud, open source tools and APIs, Google provides technology companies the portability and extensibility they need to avoid data lock-in and exercise choice.
We thank the Google Cloud and Connected-Stories team members who co-authored the blog: Connected-Stories: Luna Catini, Marketing Director, Google: Sujit Khasnis, Cloud Partner Engineering
Home Depot Leverages Google Cloud’s BigQuery and DataFlow to Break Data Silos and Craft Personalized CX

5011
Of your peers have already read this article.
4:00 Minutes
The most insightful time you'll spend today!
The Home Depot, Inc., is the world’s largest home improvement retailer with annual revenue of over $151B. Delighting our customers—whether do-it-yourselfers or professionals—by providing the home improvement products, services, and equipment rentals they need, when they need them, is key to our success.
We operate more than 2,300 stores throughout the United States, Canada, and Mexico. We also have a substantial online presence through HomeDepot.com, which is one of the largest e-commerce platforms in the world in terms of revenue. The site has experienced significant growth both in traffic and revenue since the onset of Covid-19.
Because many of our customers shop at both our brick-and-mortar stores and online, we’ve embarked on a multi-year strategy to offer a shopping experience that seamlessly bridges the physical and digital worlds. To maximize value for the increasing number of online shoppers, we’ve shifted our focus from event marketing to personalized marketing, as we found it to be far more effective in improving the customer experience throughout the sales journey. This led to changing our approach to marketing content, email communications, product recommendations, and the overall website experience.
Challenge: launching a modern marketing strategy using legacy IT
For personalized marketing to be successful, we had to improve our ability to recognize a customer at the point of transaction so we could—among other things—suspend irrelevant and unnecessary advertising. Most of us have experienced the annoyance of receiving ads for something we’ve already purchased, which can degrade our perception of the brand itself. While many online retailers can identify 100% of their customer transactions due to the rich information captured during checkout, most of our transactions flow through physical stores, making this a more difficult problem to solve.
Our old legacy IT system, which ran in an on-premises data center and leveraged Hadoop, also challenged us since maintaining both the hardware and software stack required significant resources. When that system was built, personalized marketing was not a priority, so it took several days to process customer transaction data and several weeks to roll out any system changes. Further, managing and maintaining the large Hadoop cluster base presented its own set of issues in terms of quality control and reliability, as did keeping up with open-source community updates for each data processing layer.
Adopting a hybrid approach
As we worked through the challenges of our legacy system, we started thinking about what we wanted our future system to look like. Like many companies, we began with a “build vs. buy” analysis. We looked at several products on the market and determined that while each of them had their strengths, none was able to offer the complete set of features we needed.
Our project team didn’t think it made sense to build a solution from scratch, nor did we have access to the third-party data we needed. After much consideration, we decided to adopt a solution that combined a complete rewrite of the legacy system with the support of a partner to help with the customer transaction matching process.
Building the foundation on Google Cloud
We chose Google Cloud’s data platform, specifically BigQuery, Dataflow, DataProc, Cloud Storage, and Cloud Composer. Google Cloud platform empowered us to break down data silos and unify each stage of the data lifecycle from ingestion, storage, and processing to analysis and insights. Google Cloud offered best-in-class integration with open-source standards and provided the portability and extensibility we needed to make our hybrid solution work well. The open standards of BigQuery’s BQ Storage API allowed us to leverage fast BQ storage layers to be utilized with other compute platforms, e.g., DataProc.
We used BigQuery combined with Dataflow to integrate our first- and third-party data into an enterprise data and analytics data lake architecture. The system then combined previously siloed data and used BigQuery ML to create complete customer profiles spanning the entire shopping experience, both in-store and online.
Understanding the customer journey with the help of Dataflow and BigQuery
The process of developing customer profiles involves aggregating a number of first- and third-party data sources to create a 360-degree view of the customer based on both their history and intent. It starts with creating a single historical customer profile through data aggregation, deduplication, and enrichment. We used several vendors to help with customer resolution and NCOA (Change of Address) updates, which allows the profile to be house-holded and transactions to be properly reconciled to both the individual and the household. This output is then matched to different customer signals to help create an understanding of where the customer is in their journey—and how we can help.
The initial implementation used Google Dataflow, Google’s streaming analytics solution, to load data from Google Cloud Storage into BigQuery and perform all necessary transformations. The Dataflow process was converted into BQML (BigQuery Machine Learning) since this significantly reduced costs and increased visibility into data jobs. We used Google Cloud Composer, a fully managed workflow orchestration service, to help orchestrate all data operations and DataProc and Google Kubernetes Engine to enable special case data integration so we could quickly pivot and test new campaigns. The architecture diagram below shows the overall structure of our solution.

Taking full advantage of cloud-native technology
In our initial migration to Google Cloud, we moved most of our legacy processes in their original form. However, we quickly learned that this approach didn’t take full advantage of the cloud-native and more improved features Google Cloud offered such as auto scaling of resources, flexibility to decouple storage from the compute layer, and a wide variety of options to choose the best tool for the job. We refactored our Hadoop-based data pipelines written in Java-based Map Reduce and our Pig Latin jobs to Dataflow and BigQuery jobs. This dramatically reduced processing time and made our data pipeline code concise and efficient.
Previously, our legacy system processes ran longer than intended, and data was not used efficiently. Optimizing our code to be cloud-native and leveraging all the capabilities of Google Cloud services resulted in reduced run times. We decreased our data processing window from 3 days to 24 hours, improved resource usage by dramatically reducing the amount of compute we used to possess this data, and built a more streamlined system. This in turn reduced cloud costs and provided better insight. For example, DataFlow offers powerful native features to monitor data pipelines, enabling us to be more agile.
Leveraging the flexibility and speed of the cloud to improve outcomes
Today, using a continuous integration/continuous delivery (CI/CD) approach, we can deploy multiple system changes each week to further improve our ability to recognize in-store transactions. Leveraging the combined capabilities of various Google Cloud systems—BigQuery, DataFlow, Cloud Composer, Dataproc, and Cloud Storage–we drastically increased our ability to recognize transactions and can now connect over 75% of all transactions to an existing household. Further, the flexible Google Cloud environment coupled with our cloud-native application makes our team more nimble and better able to respond to emerging problems or new opportunities.
Increased speed has led to better outcomes in our ability to match transactions across all sales channels to a customer and thereby improve their experience. Before moving to Google Cloud, it took 48 to 72 hours to match customers to their transactions, but now we can do it in less than 24 hours.
Making marketing more personal—and more efficient
The ability to quickly match customers to transactions has huge implications for our downstream marketing efforts in terms of both cost and effectiveness. By knowing what a customer has purchased, we can turn off ads for products they’ve already bought or offer ads for things that support what they’ve bought recently. This helps us use our marketing dollars much more efficiently and offer an improved customer experience.
Additionally, we can now apply the analytical models developed using BQML and Vertex AI to sort customers into audiences. This allows us to more quickly identify a customer’s current project, such as remodeling a kitchen or finishing a basement, and then personalize their journey by offering them information on products and services that matter most at a given point through our various marketing channels. This provides customers with a more relevant and customized shopping journey that mirrors their individual needs.
Protecting a customer’s privacy
With this ability to better understand our customers, we also have the responsibility to ensure we have good oversight and maintain their data privacy. Google’s cloud solutions provide us the security needed to help protect our customers’ data, while also being flexible enough to allow us to support state and federal regulations, like the California Customer Privacy Act. This way we can provide a customer the personalized experience they desire without having to fear how their data is being used.
With flexible Google Cloud technology in place, The Home Depot is well positioned to compete in an industry where customers have many choices. By putting our customers’ needs first, we can stay top of mind whenever the next project comes up.
Pay-Go Option for MongoDB on Google Cloud Helps Customers Scale as Required

3357
Of your peers have already read this article.
3:00 Minutes
The most insightful time you'll spend today!
As the volume and velocity of data grows daily, business success depends on the ability to manage it effectively and transform it into actionable insights. Since 2019, Google Cloud and MongoDB have worked together to give businesses the secure, global, and highly performant infrastructure, the sophisticated data intelligence, and the developer-centric tools they need to power modern, data-driven cloud applications. Our partnership with MongoDB continues to deliver richer yet simpler ways to lead with software.
With MongoDB Atlas on Google Cloud, developers can build upon a solid foundation that enables them to work with data the way they want in support of global-scale applications. For example, Forbes, a large business media brand, migrated its platform to Google Cloud and MongoDB Atlas in just six months. The company’s new cloud infrastructure helped the website scale to accommodate record-breaking growth even while making development more nimble. MongoDB’s document model meant developers could build new features quickly, easily incorporate changes, and better handle a growing diversity of data types. It also allowed for more powerful tools, such a machine-language trending story recommendation engine for journalists. Results have been impressive, including:
- 58% faster build time for new products and fixes
- Accelerated release cycle by 4x
- Reduced total cost of ownership by 25%
- 28% increase in subscriptions from new newsletters
The Forbes team is already looking ahead, with plans for improved personalization, loyalty, and management of first-party data. “Our decision to migrate to Google Cloud was made based on the toolset that it offers, scalability, and developer friendliness,” says Vadim Supitsky, Forbes CTO.
Over the course of the partnership between MongoDB and Google Cloud, we have continually added new benefits for our mutual customers, such as the ability to to integrate MongoDB Atlas with Google Cloud products; leveraging BigQuery to create a managed, serverless, scalable architecture; rich data connectivity; and flexible scaling. We’ve also made it easier to migrate MongoDB on-premises instances to MongoDB Atlas on Google Cloud.
Making MongoDB on Google Cloud even more flexible with Pay-Go
That’s why we’re excited to see yet another reason for companies to choose MongoDB Atlas on Google Cloud: A new pay-as-you-go option, available on the Google Cloud Marketplace. With this new offering, developers now have a simplified subscription experience, and enterprises have a simplified way to procure MongoDB in addition to privately negotiated offers already supported on the Google Cloud Marketplace. There are no up-front commitments required to use MongoDB Atlas on Google Cloud, and customers pay only for the resources they use and scale based on their needs. Here are just a few of the benefits of this new service:
- Spin up a MongoDB Atlas cluster on the Google Cloud Console within a few minutes
- No need for a separate credit card payment: You can use your Google Cloud Billing account for your MongoDB Atlas environment
- Receive a single bill for Google Cloud and MongoDB Atlas
- Apply Google Cloud committed spend to MongoDB transactions through Google Cloud Marketplace
- Get started with MongoDB Atlas with 512 MB of storage for free. Atlas free tier clusters are perfect for learning MongoDB or prototyping applications
We’re excited to see even further growth in our continuing partnership with three new developments: First, because resellers are such an important force, we are pleased to share that reseller partners can now make MongoDB Atlas available via the Google Cloud Marketplace. Second, MongoDB and Google Cloud have expanded our joint reach to 28 global regions after adding Toronto, Canada, and Santiago, Chile. And third, we’ve made a joint commitment to early stage companies through MongoDB for Startups and the Google for Startups Cloud Program. Pay-Go is great for startups because it’s easy to scale and you only pay for what you use. Similarly, MongoDB and Google are helping startups get off the ground with our respective programs, which both include credits and support.
Better together
Your company’s transformation hinges on new cloud database capabilities. The first step to building all-new digital experiences is to select the operational database that will power your application and, in essence, run your business.The partnership between MongoDB and Google Cloud gives you the benefits of a modern database service in a tightly integrated, cloud-native way. Since the beginning of our collaborative journey, we have made significant enhancements to the experience for our mutual customers, and we continue to work toward providing a rich developer experience for MongoDB Atlas on Google Cloud.
Discover how customers choose MongoDB and Google Cloud to power the future of customer innovation. For more information on how to get started, visit MongoDB Atlas on Google Cloud Marketplace.
Become Truly Data-driven with Unified Analytics Platform Built on Google Cloud

3671
Of your peers have already read this article.
2:00 Minutes
The most insightful time you'll spend today!
Every company these days is becoming a data company whether they know it or not. This results in preparing the company to create an ecosystem for data processing. Traditionally, organisations’ data ecosystems consisted of point solutions that used to provide data services. For example, one of the most common questions we get from customers is, “Do I need a data lake, or should I consider a data warehouse? Do you recommend I consider both?” Traditionally, these two architectures have been viewed as separate systems, applicable to specific data types and user skill sets. Increasingly, we are seeing a blurring of lines between data warehouses and data lakes, which provide customers with an opportunity to create a more comprehensive platform that gives them the best of both worlds.
What if we don’t need to compromise? And we create an end-to-end solution compromising the entire data management and processing stages, from data collection to serving. Thus, Data platforms are typically used to store vast amounts of data in varying formats and doing so without compromising on latency. At the same time, providing a platform for all users throughout the data lifecycle.
Emerging Trends
There are data solutions and architectures we’re seeing and anticipating. Emerging concepts include data lakehouses, data meshes, and data vaults. Some are not new and have been around in different shapes and formats, however, all of them work naturally within a Google Cloud environment. Lets look into both ends of the spectrum of enabling data and enabling teams.
Data mesh facilitates a decentralized approach to data ownership, allowing individual lines of business to publish and subscribe to data in a standardized manner, instead of forcing data access and stewardship through a single, centralized team. On the other hand, a data lake house brings raw and processed data closer together, allowing for a more streamlined and centralized repository of data needed throughout the organization. Processing can be done in transit via ELT, reducing the need to copy datasets across systems. This allows for easier data exploration and easier governance. Data vault is designed to separate data driven and model driven parts, so the way data is integrated in the raw vault enables parallel loading so that large implementations can scale out easily.
In Google Cloud, there is no need to keep them separate. In fact, with interoperability among our portfolio of data analytics products, you can easily provide access to data residing in different places, effectively bringing your data lake and data warehouse together on a single platform.
Let’s look at some of the technological innovations that make this reality. BigQuery’s storage API allows treating a data warehouse like a data lake, letting you access the data residing in BigQuery. For example, you can use Spark to access data residing in the data warehouse without it affecting performance of any other jobs accessing it. This is all made possible by the underlying architecture, which separates compute and storage.
We continue to offer specialized products and solutions around data lake and data warehouse functionality but over time we expect to see a significant enough convergence of the two systems that the terminology will change. At Google Cloud, we consider this combination an “analytics data platform”.
Tactical or Strategical
Key differentiators of Google Cloud’s data analytics platform are being open, intelligent, flexible, and tightly integrated. There are many technologies in the market which provide tactical solutions that may feel comfortable and familiar. However, this can be a rather short-term approach that simply lifts and shifts a siloed solution into the cloud. In contrast, an analytics data platform built on Google Cloud provides modern data warehousing and data lake capabilities with close integration to our AI Platform. It also provides built-in streaming, ML, and geospatial capabilities and an in-memory solution for BI use cases. Depending on your organizational data needs, Google Cloud has the set of products, tools, and services to create the right data platform for you.
To become a truly data-driven organization, the first step is to design and implement an analytics data platform that meets your technical and business needs. Whether you want to empower teams to own, publish, and share their data across the organization, or you want to create a streamlined store of raw and processed data for easier discovery, there is a solution that best meets the needs of your company.
To learn more about the elements of a unified analytics data platform built on Google Cloud, and the differences in platform architectures and organizational structures, read our Unified Analytics Platform paper.
PaGaLGuY Turns to Google Cloud Platform to Power Leading India Education Network

8150
Of your peers have already read this article.
2:30 Minutes
The most insightful time you'll spend today!
Founded in 2006, PaGaLGuY started as a forum that enabled students, typically aged between 20 and 30 to discuss and seek advice on academic issues. By 2011, PaGaLGuY had increased its traffic to about 250,000 page views per month. The business is now one of India’s largest education networks and provides an app that users can download to their Android and iOS devices.
Over the past six years, PaGaLGuY has extended its service to include video advice from experts on education topics and grown the number of views of its pages to 1.5 million per day. As Head of Technology for the business, Sandeep Kalidindi has played a key role in ensuring PaGaLGuY is as engaging as possible to users. “Because the product is advertising based, the greater the user engagement, the greater the advertising revenue,” Kalidindi explains.
Google Cloud Platform Results
- Supported growth to 1.5 million page views per day and demand spikes that see requests increase from about 90 per second to about 1,200 per second
- Reduced API latency from about 1 second to about 40 milliseconds
- Reduced system administration time from three to four days per week to 30 minutes every two weeks
In 2015, PaGaLGuY’s senior management team decided to deliver an even more relevant experience for users of the education network. “The core thing we had to do was personalise the experience for each and every student that visited PaGaLGuY,” Kalidindi says. “So we had to capture each student’s data to customise what they see when they open the site.”
The business also found traffic to the network was straining its infrastructure. During demand peaks, created by exams involving as many as 5 million students, PaGaLGuY would be inaccessible for periods of 30 minutes to one hour. Furthermore, average API latency had climbed to an unacceptable 1 second, compromising performance.
PaGaLGuY needed to access extensive compute resources to undertake its planned change. Had the business relied on a physical technology architecture to undertake the transformation, it would have had to purchase capacity equivalent to 16 new servers. “There was no way with a small team we could grow to that extent in a short time,” Kalidindi says. “This was the right moment for us to explore cloud services.”
The business established two primary requirements the selected cloud service needed to meet. First, PaGaLGuY had to be able to scale the platform with costs rising only in proportion to the increase in resources consumed. Accordingly, the business would have to minimise the number of employees required to manage the cloud environment. Second, the platform had to give PaGaLGuY easy access to student data and the ability to undertake prompt, granular analysis.
PaGaLGuY reviewed available public cloud services and determined that Google Cloud Platform (GCP) was the best fit for its business. “Google Cloud Platform was considerably more mature than the alternatives, with a high degree of automation and a suite of managed services,” Kalidindi says. PaGaLGuY management then discussed with Google how to optimise cost, performance and availability of its personalised education network on GCP.
With assistance from Google and business transformation specialists Searce, PaGaLGuY was able to deliver the platform into production on GCP in 10 months. “Searce was very proactive in ensuring the environment met our needs and allowing us to gain priority access to Google services in development,” Kalidindi says. “Their team was integral to the success of the migration.”
PaGaLGuY has been running in production in GCP for two years. The education network’s GCP architecture comprises a scalable back-end built on Google App Engine; a managed environment for its containerised applications in Google Kubernetes Engine; messaging-oriented middleware through Google Cloud Pub/Sub; a relational database in Google Cloud SQL; a managed data analytics warehouse running in Google BigQuery; stream and batch data processing through Google Cloud Dataflow; and object storage in Google Cloud Storage.
PaGaLGuY has leveraged GCP services to break down its platform application from a monolithic build to a series of microservices running in Google App Engine that enable independent deployment cycles, minimise test and quality assurance overheads and provide clearer monitoring and logging.
Running on GCP has enabled PaGaLGuY to add new personalisation features and grow fourfold without having to add any new engineers or administrators to accommodate the increased traffic. The business has also used the platform to seamlessly collect and aggregate students’ data for analysis, reporting and delivering a more targeted user experience. Furthermore, PaGaLGuY has been able to provide its management team with direct access to Google BigQuery to scrutinise data rather than require them to wait at least a day to view reports created by the product or technology teams.
Support demand peaks of 1,200 requests per second
“Thanks to Google Cloud Platform, we can easily support demand peaks that see requests per second rise from an average 90 per second to about 1,200 per second for as long as 45 minutes,” Kalidindi says. Due to GCP’s scalability, PaGaLGuY can ensure its education network remains available and performance remains consistent during those periods.
Latency cut to 40 milliseconds
The business has also reduced average API latency from 1 second to about 40 milliseconds. Furthermore, using GCP has enabled PaGaLGuY to automate most of its processes and reduce system administration requirements from three to four days a week across its team members to about half an hour per week.
The performance of GCP has transformed PaGaLGuY’s culture and processes. “Once our team was exposed to Google Cloud Platform and understood the superiority of the platform, our mindset changed from ‘let us do everything on our own’ to ‘let us do what we do best’ and delegate the remainder,” Kalidindi says. The quality of the service provided by GCP means PaGaLGuY effectively considers the cloud provider as part of its team. “We are always eager to see what new services are being launched and are extremely excited about what Google Cloud Platform can provide as part of its roadmap.” he concludes.
Soundtrack Your Brand: Delivering Sound That Stands Out From the Crowd with BigQuery

2839
Of your peers have already read this article.
6:00 Minutes
The most insightful time you'll spend today!
Editor’s note: Soundtrack Your Brand is an award-winning streaming service with the world’s largest licensed music catalog built just for businesses, backed by Spotify. Today, we hear how BigQuery has been a foundational component in helping them transform big data into music.
Soundtrack Your Brand is a music company at its heart, but big data is our soul. Playing the right music at the right time has a huge influence on the emotions a brand inspires, the overall customer experience, and sales. We have a catalog of over 58 million songs and their associated metadata from our music providers and a vast amount of user data that helps us deliver personalized recommendations, curate playlists and stations, and even generate listening schedules. As an example, through our Schedules feature our customers can set up what to play during the week. Taking that one step further, we provide suggestions on what to use in different time slots and recommend entire schedules.
Using BigQuery, we built a data lake to empower our employees to access all this content and metadata in a structured way. Ensuring that our data is easily discoverable and accessible allows us to build any type of analytics or machine learning (ML) use case and run queries reliably and consistently across the complete data set. Today, our users are benefiting from this advanced analytics through the personalized recommendations we offer across our core features: Home, Search, Playlists, Stations, and Schedules.
Fine-tuning developer productivity
The biggest business value that comes from BigQuery is how much it speeds up our development capabilities and allows us to ship features faster. In the past 3 years, we have built more than 150 pipelines and more than 30 new APIs within our ML and data teams that total about 10 people. That is an impressive rate of a new pipeline every week and a new API every month. With everything in BigQuery, it’s easy to simply write SQL and have it be orchestrated within a CI/CD toolchain to automate our data processing pipelines. An in-house tool built as a github template, in many ways very similar to Dataform, helps us build very complex ETL processes in minutes, significantly reducing the time spent on data wrangling.
BigQuery acts as a cornerstone for our entire data ecosystem, a place to anchor all our data and be our single source of truth. This single source of truth has expanded the limits of what we can do with our data. Most of our pipelines start from a data lake, or end at a data lake, increasing re-usability of data and collaboration. For example, one of our interns built an entire churn prediction pipeline in a couple of days on top of existing tables that are produced daily. Nearly a year later, this pipeline is still running without failure largely due to its simplicity. The pipeline is BigQuery queries chained together into a BigQuery ML model running on a schedule with Kubeflow Pipelines.
Once we made BigQuery the anchor for our data operations, we discovered we could apply it to use cases that you might not expect, such as maintaining our configurations or supporting our content management system. For instance, we created a Google Sheet where our music experts are able to correct genre classification mistakes for songs by simply adding a row to a Google Sheet. Instead of hours or days to create a bespoke tool, we were able to set everything up in a few minutes.
BigQuery’s ability to consume Excel spreadsheets allows business users who play key roles in improving our recommendations engine and curating our music, such as our content managers and DJs, to contribute to the data pipeline.
Another example is our use of BigQuery as an index for some of our large Cloud Storage buckets. By using cloud functions to subscribe to read/write events for a bucket, and writing those events to partitioned tables, our pipelines can easily and in a natural way quickly search and access files, such as downloading and processing the audio of new track releases. We also make use of Log Events when a table is added to a dataset to trigger pipelines that process data on demand, such as JSON/CSV files from some of our data providers that are newly imported into BQ. Being the place for all file integration and processing, BQ allows new data to be quickly available to our entire data ecosystem in a timely and cost effective manner while allowing for data retention, ETL, ACL and easy introspection.
BigQuery makes everything simple. We can make a quick partitioned table and run queries that use thousands of CPU hours to sift through a massive volume of data in seconds — and only pay a few dollars for the service. The result? Very quick, cost-effective ETL pipelines.
In addition, centralizing all of our data in BigQuery makes it possible to easily establish connections between pipelines providing developers with a clear understanding of what specific type of data a pipeline will produce. If a developer wants a different outcome, she can copy the github template and change some settings to create a new, independent pipeline.
Another benefit is that developers don’t have to coordinate schedules or sync with each other’s pipelines: they just need to know that a table that is updated daily exists and can be relied on as a data source for an application. Each developer can progress their work independently without worrying about interfering with other developers’ use of the platform.
Making iteration our forte
Out of the box, BigQuery met and exceeded our performance expectations, but ML performance was the area that really took us by surprise. Suddenly, we found ourselves going through millions of rows in a few seconds, where the previous method might have taken an hour. This performance boost ultimately led to us improving our artist clustering workload from more than 24 hours on a job running 100 CPU workers to 10 minutes on a BigQuery pipeline running inference queries in a loop until convergence. This more than 140x performance improvement also came at 3% of the cost.
Currently we have more than 100 Neural Network ML models being trained and run regularly in batch in BQML. This setup has become our favorite method for both fast prototyping and creating production ready models. Not only is it fast and easy to hypertune in BQML, but our benchmarks show comparable performance metrics to using our own Tensorflow code. We now use Tensorflow sparingly. Differences in input data can have an even greater impact on the experience of the end user than individual tweaks to the models.
BigQuery’s performance makes it easy to iterate with the domain experts who help shape our recommendations engine or who are concerned about churn, as we are able to show them the outcome on our recommendations from changes to input data in real-time. One of our favorite things to do is to build a Data Studio report that has the ML.predict query as part of its data source query. This report shows examples of good/bad predictions in the report along with bias/variance summaries and a series of drop-downs, thresholds and toggles to control the input features and the output threshold. We give that report to our team of domain experts to help manually tune the models, putting the model tuning right in the hands of the domain experts. Having humans in the loop has become trivial for our team. In addition to fast iteration, the BigQuery ML approach is also very low maintenance. You don’t need to write a lot of Python or Scala code or maintain and update multiple frameworks—everything can be written as SQL queries run against the data store.
Helping brands to beat the band—and the competition
BigQuery has allowed us to establish a single source of truth for our company that our developers and domain experts can build on to create new and innovative applications that help our customers find the sound that fits their brand.
Instead of cobbling together data from arbitrary sources, our developers now always start with a data set from BigQuery and build forward. This guarantees the stability of our data pipeline and makes it possible to build outward into new applications with confidence. Moreover, the performance of BigQuery means domain experts can interact with the analytics and applications that developers create more easily and see the results of their recommended improvements to ML models or data inputs quickly. This rapid iteration drives better business results, keeps our developers and domain experts aligned, and ensures Soundtrack Your Brand keeps delivering sound that stands out from the crowd.
More Relevant Stories for Your Company

How Pantheon Improved Performance and Reliability by Moving to Google Cloud
Google Cloud Results Improves performance and reliability, enabling Pantheon to serve larger customersSupports 99.95% uptime across 200,000+ websitesReduces cloud infrastructure costs by 40%Enables future analytics offerings based on machine learning and big data analytics Nearly every business needs a web presence—but the vast majority of companies don’t want to be

Google Cloud and Climate Engine Collaborate to Support Climate Action in Public Sector
While there is uncertainty about how much the climate will change in the future, we know it won’t look like the past. Extreme weather events will increase in frequency and severity; the world will continue to warm, and the cost of climate change will increase. Government plays a vital role

Cart.com to Transform e-Commerce for Brands Globally
The ecommerce playing field has been hard to navigate for most retailers, and Cart.com is on a mission to change that. Traditionally, retailers needing to run their online store, order fulfillment, customer service, marketing, and other essential activities have had to cobble together systems to get the capabilities they need

Two Ad Agencies Leverage BigQuery to Support Next-gen Ad Campaigns
Advertising agencies are faced with the challenge of providing the precision data that marketers require to make better decisions at a time when customers’ digital footprints are rapidly changing. They need to transform customer information and real-time data into actionable insights to inform clients what to execute to ensure the






