Updating Twitter’s Ad Engagement Analytics Platform for the Modern Age

3103
Of your peers have already read this article.
3:30 Minutes
The most insightful time you'll spend today!
As part of the daily business operations on its advertising platform, Twitter serves billions of ad engagement events, each of which potentially affects hundreds of downstream aggregate metrics. To enable its advertisers to measure user engagement and track ad campaign efficiency, Twitter offers a variety of analytics tools, APIs, and dashboards that can aggregate millions of metrics per second in near-real time.
In this post, you’ll get details on how the Twitter Revenue Data Platform engineering team, led by Steve Niemitz, migrated their on-prem architecture to Google Cloud to boost the reliability and accuracy of Twitter’s ad analytics platform.
Deciding to migrate
Over the past decade, Twitter has developed powerful data transformation pipelines to handle the load of its ever-growing user base worldwide. The first deployments for those pipelines were initially all running in Twitter’s own data centers. The input data streamed from various sources into Hadoop Distributed File System (HDFS) as LZO-compressed Thrift files in an Elephant Bird container format. The data was then processed and aggregated in batches by Scalding data transformation pipelines. Then, aggregation results were output into Manhattan, Twitter’s homegrown distributed key-value store, for serving. Additionally, a streaming system using Twitter’s homegrown systems Eventbus (a messaging tool built on top of DistributedLog), Heron (a stream processing engine), and Nighthawk (a sharded Redis deployment) powered the real-time analytics that Twitter had to provide, filling the gap between the current time and the last batch run.
While this system consistently sustained massive scale, its original design and implementation was starting to reach some limits. In particular, some parts of the system that had grown organically over the years were difficult to configure and extend with new features. Some intricate, long-running jobs were also unreliable, leading to sporadic failures. The legacy end-user serving system was very expensive to run and couldn’t support large queries.
To accommodate for the projected growth in user engagement over the next few years and streamline the development of new features, the Twitter Revenue Data Platform engineering team decided to rethink the architecture and deploy a more flexible and scalable system in Google Cloud.
Platform modernization: First iteration
In the middle of 2017, Steve and his team tackled the first redesign iteration of its advertising data platform modernization, leading to Twitter’s collaboration with Google Cloud.
At first, the team left the data aggregation legacy Scalding pipelines unchanged and continued to run them in Twitter’s data centers. But the batch layer’s output was switched from Manhattan to two separate storage locations in Google Cloud:
- BigQuery—Google’s serverless and highly scalable data warehouse, to support ad-hoc and batch queries.
- Cloud Bigtable—Google’s low-latency, fully managed NoSQL database, to serve as a back end for online dashboards and consumer APIs.
The output aggregations from the Scalding pipelines were first transcoded from Hadoop sequence files to Avro on-prem, staged in four-hour batches to Cloud Storage, and then loaded into BigQuery. A simple pipeline deployed on Dataflow, Google Cloud’s fully managed streaming and batch analytics service, then read the data from BigQuery and applied some light transformations. Finally, the Dataflow pipeline wrote the results into Bigtable.
The team built a new query service to fetch aggregated values from Bigtable and process end-user queries. They deployed this query service in a Google Kubernetes Engine (GKE) cluster in the same region as the Bigtable instance to optimize for data access latency.
Here’s a look at the architecture:

This first iteration already brought many important benefits:
- It de-risked the overall migration effort, letting Twitter avoid migrating both the aggregation business logic and storage at the same time.
- The end-user serving system’s performance improved substantially. Thanks to Bigtable’s linear scalability and extremely low latency for data access, the serving system’s P99 latencies decreased from 2+ seconds to 300ms.
- Reliability increased significantly. The team now rarely, if ever, gets paged for the serving system anymore.
Platform modernization: second iteration
With the new serving system in place, in 2019 the Twitter team began to redesign the rest of the data analytics pipeline using Google Cloud technologies. The redesign sought to solve several existing pain points:
- Because the batch and streaming layers ran on different systems, much of the logic was duplicated between systems.
- While the serving system had been moved into the cloud, the existing pain points of the Hadoop aggregation process still existed.
- The real-time layer was expensive to run and required significant operational attention.
With these pain points in mind, the team began evaluating technologies that could help solve them. They considered several open-source stream processing frameworks initially: Apache Flink, Apache Kafka Streams, and Apache Beam. After evaluating all possible options, the team chose Apache Beam for a few key reasons:
- Beam’s built-in support for exactly-once operations at extremely large scale across multiple clusters.
- Deep integration with other Google Cloud products, such as Bigtable, BigQuery, and Pub/Sub, Google Cloud’s fully managed, real-time messaging service.
- Beam’s programming model, which unifies batch and streaming and lets a single job operate on either batch inputs (Cloud Storage), or streaming inputs (Pub/Sub).
- The ability to deploy Beam pipelines on Dataflow’s fully managed service.
The combination of Dataflow’s fully managed approach and Beam’s comprehensive feature set let Twitter simplify the structure of its data transformation pipeline, as well as increase overall data processing capacity and reliability.
Here’s what the architecture looks like after the second iteration:

In this second iteration, the Twitter team re-implemented the batch layer as follows: Data is first staged from on-prem HDFS to Cloud Storage. A batch Dataflow job then regularly loads the data from Cloud Storage, processes the aggregations, and dual-writes the results to BigQuery for ad-hoc analysis and Bigtable for the serving system.
The Twitter team also deployed an entirely new streaming layer in Google Cloud. For data ingestion, an on-prem service now pushes two different streams of Avro-formatted messages to Pub/Sub. Each message contains a bundle of multiple raw events and affects between 100 and 1,000 aggregations. This leads to more than 3 million aggregations per second performed by four Dataflow jobs (J0-3 in the diagram above). All Dataflow jobs share the same topology, although each job consumes messages from different streams or topics.
One stream, which contains critical data, enters the system at a rate of 200,000 messages per second and is partitioned in two separate Pub/Sub topics. A Dataflow job (J3 in the diagram) consumes those two streams, performs 400,000 aggregations per second, and outputs the results to a table in Bigtable.
The other stream, which contains less critical but higher volume data, enters the system at a rate of around 80,000 messages per second and is partitioned into six separate topics. Three Dataflow jobs (J0, J1, and J2) share the processing of this larger stream, with each of them handling two of the available six topics in parallel, then also outputting the results to a table in Bigtable. In total, those three jobs process over 2 million aggregations/second.
Partitioning the high-volume stream into multiple topics offers a number of advantages:
- The partitioning is organized by applying a hash function on the aggregation key and then dividing the function’s result by the number of available partitions (in this case, six). This guarantees that any per-key grouping operation in downstream pipelines is scoped to a single partition, which is required for consistent aggregation results.
- When deploying updates to the Dataflow jobs, admins can drain and relaunch each job individually in sequence, allowing the remaining pipelines to continue uninterrupted and minimizing impact on the end users.
- The three jobs can each handle two topics without issue currently, and there is still room to scale horizontally up to six jobs if needed. The number of topics (six) is arbitrary, but is a good balance at the moment based on current needs and potential spikes in traffic.
To assist with job configuration, Twitter initially considered using Dataflow’s template system, a powerful feature that enables the encapsulation of Dataflow pipelines into repeatable templates that can be configured at runtime. However, since Twitter needed to deploy jobs with topologies that might change over time, the team decided instead to implement a custom declarative system where developers can specify different parameters for their jobs in a pystachio DSL: tuning parameters, data sources to operate on, sink tables for aggregation outputs, and the jobs’ source code location. A new major version of Dataflow templates, called Flex Templates, will remove some of the previous limitations with the template architecture and allow any Dataflow job to be templatized.
For job orchestration, the Twitter team built a custom command line tool that processes the configuration files to call the Dataflow API and submit jobs. The tool also allows developers to submit a job update by automatically performing a multi-step process, like this:
- Drain the old job:
- Call the Dataflow API to identify which data sources are used in the job (for example, a Pub/Sub topic reader).
- Initiate a drain request.
- Poll the Dataflow API for the watermark of the identified sources until the maximum watermark is hit, which indicates that the draining operation is complete.
- Launch the new job with the updated code.
This simple, flexible, and powerful system allows developers to focus on their data transformation code without having to be concerned about job orchestration or the underlying infrastructure details.
Looking ahead
Six months after fully transitioning its ad analytics data platform to Google Cloud, Twitter has already seen huge benefits. Twitter’s developers have gained in agility as they can more easily configure existing data pipelines and build new features much faster. The real-time data pipeline has also greatly improved its reliability and accuracy, thanks to Beam’s exactly-once semantics and the increased processing speed and ingestion capacity enabled by Pub/Sub, Dataflow, and Bigtable.
Twitter engineers have enjoyed working with Dataflow and Beam for several years now, since version 2.2, and plan to continue expanding their usage. Most importantly, they’ll soon merge the batch and streaming layers into a single, authoritative streaming layer.
Throughout this project, the Twitter team collaborated very closely with Google engineers to exchange feedback and discuss product enhancements. We look forward to continuing this joint technical effort on several ongoing large-scale cloud migration projects at Twitter. Stay tuned for more updates!
How AI and ML Helps Interpret Baseball Fandom during this MLB Season

4554
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
The game of baseball has no shortage of statistics — from batting average to exit velocity, strikeouts to wins above replacement. Among all sports, Major League Baseball (MLB) arguably contains the most analytical and data-driven participants and fan base. Subconsciously or viscerally, players and managers on the field and those following from anywhere are constantly assessing and making decisions based off of game play trends and expectations — whether a batter will come through with a hit in an important situation, when a pitcher should be pulled. Less analyzed, however, is what leads fans to become engaged with certain players or teams, and what factors drive their love of the game. This is the motivation behind the problem being posed by Major League Baseball in their Kaggle competition for Player Digital Engagement Forecasting. Can you use machine learning to deconstruct baseball fandom?
This competition asks you to predict measures of digital engagement for each active player on a daily basis during the MLB season. So, how large was the surge in fan interest after Joe Musgrove threw the first no-hitter in Padres history? Is Shohei Ohtani’s engagement higher when he pitches well, when he hits a monster home run…or when he does both? You’re provided a wealth of game, team and player information – detailed stats, awards, rosters, and transaction information – as well as social and digital engagement data as your inputs. Data scientists will recognize this as an exciting forecasting problem with both traditional regression and time series components, where having this input data just prior to the prediction date is critical to determining which players will receive the most engagement.
With so many variables in the game, there are an endless number of vectors which could possibly influence fan engagement. Eleven-time All-Star Miguel Cabrera delighted fans by hitting the first home run of the season – in the snow! Occasionally a lesser-known player like Musgrove or Carlos Rodón “wins the day” with an unlikely no-hitter. And sometimes just getting traded to an iconic franchise like the Yankees generates a ton of fan interest, like it did for Rougned Odor in early April.

As these examples show, a player’s digital engagement can be pretty dynamic during the season, with many different potential contributors to who is “trending” on a given day. How can you use data to uncover which factors are the most influential of engagement with each player’s digital content?
Ready to play ball? Check out the competition on Kaggle for all the details. $50,000 in prizes is up for grabs in two prize categories. The code competition puts your machine learning skills to the test, to see who can build the most accurate forecasting models to predict daily digital engagement for every active player. You’ll have until July 31st to build your models and then be evaluated on a future time frame, which will determine the winners. For data visualization and exploration experts out there, the explainability prizes give you an opportunity to analyze more broadly which factors, even those outside of what we’re providing directly, most influence digital engagement. You’ll be evaluated on how well you can use what the data is telling you to support your findings.
And if you’re looking to get started, we’ve provided an introductory video and some notebook tutorials, including a starting point for harnessing the power of Vertex AI through tools including Cloud Notebooks, Explainable AI, and Vizier.
With the second half of the season upon us, it’s an exciting time to be an MLB fan. With this Kaggle competition, it’s also a perfect opportunity to use data science to help understand baseball fandom and potentially earn some of your own accolades in the process. Step up to the plate!
Major League Baseball trademarks and copyrights are used with permission of Major League Baseball. Visit MLB.com.
The Fantastic Story of How BMG Enables a Micropayments Strategy So Music Artists Get Paid

7446
Of your peers have already read this article.
3:30 Minutes
The most insightful time you'll spend today!
The music industry is rapidly changing. Only 20 years ago, the availability of music and the infrastructure that was required to make an album a sales success were incredibly complex and expensive. With the decline of physical sales and a fundamental shift to digital, music streaming now accounts for more than half of all sales globally.
At the same time, technology has democratized music-making; in many ways, it has made the musical landscape more diverse. Artists can upload their music with the click of a button. But while it’s easier for creators to share their songs with audiences, getting paid has become more fragmented.
Although music is booming, people no longer buy it outright. Instead, listeners download music digitally or subscribe to streaming services to have their libraries with them at all times. To monetize digital content effectively, artists need to know when, where, and how often their songs are played on each service. To help them navigate this complicated royalties landscape and maximize their profits, Berlin-based international music company BMG provides customized, transparent, and fair services to songwriters and artists.
With publishing and recording divisions under one roof, the subsidiary of international media giant Bertelsmann works with both emerging artists and established stars, including John Legend, Kylie Minogue, Mick Jagger, and Keith Richards. With the MyBMG web and mobile application, clients can view and analyze their royalty details in real time and collect payment. When a new record is released, BMG uses data to maximize its impact and revenue for its creators.
“We make sure that everyone who uses our clients’ music knows who needs to be paid the associated royalties, then we collect these royalties and share them out quickly and transparently,” explains Sebastian Hentzschel, Chief Information Officer at BMG. “When our artists release new music, we make sure that it’s marketed and promoted effectively around the world.”
“We needed a scalable solution for our royalty workloads that was intuitive for our developers. We also wanted a partner, not a client-vendor relationship. With autoscaling via BigQuery, excellent customer support, and a clean and simple user interface, Google Cloud ticks every box for us.”
—Gaurav Mittal, Vice President Group Technology, BMG
Getting up to speed with a new way of paying artists
In this digital world, artists aren’t just paid every time a fan buys an album—they’re paid a small amount, or royalty, for each song downloaded or streamed by a listener. So, when the industry shifted to digital, the volume of data that BMG needed to handle grew exponentially. “One CD sale is equivalent to about 1,500 streamed songs or plays,” says Gaurav Mittal, Vice President Group Technology at BMG. “That means IT departments have to process 1,500 times the amount of data to calculate payments for artists, and this makes scalable micropayment processing very important.”
Until 2019, BMG’s infrastructure was entirely hosted on-premises. Hardware limitations made it challenging to scale on-demand, making it harder to handle the data peaks that royalty processing can bring. “With our on-premises infrastructure, we were going to hit a ceiling in a few years,” says Gaurav. “We still managed to process royalty payments for our clients, but it was increasingly time consuming and expensive. To keep focusing on our clients, rather than our infrastructure, we decided to migrate to Google Cloud.”
From the outset, Gaurav and his team had a clear vision for the partnership: “Most importantly, we needed a scalable solution for our royalty workloads that was intuitive for our developers. We also wanted a partner, not a client-vendor relationship,” he says. “With autoscaling via BigQuery, excellent customer support, and a clean and simple user interface, Google Cloud ticks every box for us.”
Keeping artists happy with business-as-usual payouts during migration
To move applications to the cloud while keeping payment cycles on track for its artists, BMG teamed up with Google Cloud partner Rackspace Technology. “We selected Rackspace Technology because it combines strong technical muscle and a global footprint, with the customer service of a local boutique firm,“ shares Gaurav.
BMG’s own technology team put together the outline for the Google Cloud architecture, which they passed on to Rackspace Technology for optimizations and the ultimate stamp of approval. Whenever Gaurav and his developers needed support, Rackspace Technology was ready to go. “So far, we’ve migrated 17 applications successfully, and Rackspace Technology has been 100% spot-on with each suggestion,” says Gaurav. “I can’t recall a single flaw in a Rackspace Technology-approved architecture, and that really says something.”
When BMG began its migration in August 2019, the team developed an ambitious two-year plan. Only 14 months later, however, the project is more than 75% complete. Among the solutions that BMG is using today are Cloud Storage to securely store 130 TB of data, and Cloud SQL as its standard database technology. The web applications run on Compute Engine, App Engine, and Google Kubernetes Engine.
“After successfully moving a few applications, it was clear that with the strong teamwork of BMG and Rackspace Technology, together with the ease of use of Google Cloud, we could speed up the project without sacrificing quality,” says Gaurav. “We’re set to complete our migration six months before schedule, helping us to quickly move out of our hybrid environment.”
“Our income-tracking teams are very savvy on the data, and the simplicity of Google Cloud empowers them to self-serve analytics, rather than wait for IT. Our teams are much more productive.”
—Gaurav Mittal, Vice President Group Technology at BMG
Royalty reporting and processing with BigQuery and Dataproc
So far, all of BMG’s critical workloads are up and running on Google Cloud. Royalty calculations, for example, which require incredible processing power and the collection of many micropayments to ensure full and timely payout, run entirely on Dataproc with output stored on BigQuery for downstream integration and reporting.
Enabling more harmonious workflows through self-serve analytics
As the new beating heart of BMG’s royalty reporting, BigQuery changed the rhythm of collaboration company-wide. In the past, income tracking teams had to contact IT departments if they needed deeper data insights for their work. By integrating Data Catalog with BigQuery, BMG has made the data more accessible to all teams. This helps them detect missing income and new revenue streams independently, maximizing profits for artists.
“Our income-tracking teams are very savvy on the data, and the simplicity of Google Cloud empowers them to self-serve analytics, rather than wait for IT,” says Gaurav. “Our teams are much more productive.”
“Google Cloud enables us to be more client focused and deliver better features faster. We believe it’s just the beginning. We offer rights and royalty services for music publishing, recorded music, neighboring rights, and books. Without scalability limitations, it’s absolutely conceivable to offer our platform as a service to other companies or industries, such as gaming. Google Cloud has opened a world of possibilities.”
—Sebastian Hentzschel, Chief Information Officer, BMG
With a leaner IT environment, BMG can focus its effort on the needs of its clients. Beyond improvements in royalty processing, it can concentrate on app development, releasing new features and enhancements more frequently. By hosting applications on Google Kubernetes Engine, App Engine, and Compute Engine, BMG has built a CI/CD pipeline with automated deployments and testing to significantly speed up workflows.
“In our old system, it could take several weeks to set up an environment,” says Gaurav. “With Google Kubernetes Engine, any of our developers can complete the process in a few clicks. Having that autonomy makes our developers more motivated and self-driven.”
“Google Cloud enables us to be more client focused and deliver better features faster,” adds Sebastian. “We believe it’s just the beginning. We offer rights and royalty services for music publishing, recorded music, neighboring rights, and books. Without scalability limitations, it’s absolutely conceivable to offer our platform as a service to other companies or industries, such as gaming. Google Cloud has opened a world of possibilities.”
With the migration almost complete, BMG is looking forward to its next technology project. It plans to leverage AutoML to further scale and automate royalty tracking with machine learning. On the marketing side, advanced analytics will help BMG determine the effectiveness of promotional campaigns around the world, further increasing profits for artists. By connecting Google Data Studio to BigQuery, BMG will increase the quality of its analyses, helping musicians better understand the reach of their music around the world.
In the end, Gaurav shares, helping musicians is what it all comes down to. “We’re a new kind of music company because we build our services around our artist, songwriter, and publisher clients, not the other way around,” he says. “Google Cloud is helping us maintain strong relationships with our clients, and that’s music to our ears.”
How Ather Energy is leveraging the Cloud to build and scale smart mobility solutions for India

8398
Of your peers have already read this article.
8:30 Minutes
The most insightful time you'll spend today!
In 2013, long before the world was discussing clean energy and sustainable practices, two IIT Madras graduates — Swapnil Jain and Tarun Mehta — had an idea to develop India’s first-ever electrical scooter.
This was at a time when auto manufacturers were still focusing on fossil-fuel-driven vehicles and ‘eco-friendly’ mobility solutions were more a trendy alternative catering to a niche market.
The duo founded Ather Energy in 2013 and launched their first fully-electric scooter, the Ather S340, in Bengaluru in 2016. Since then, the company has released several new models into the market and is planning to expand to eight more cities by the end of the year.
To support the smooth running of their vehicles, lower costs, improve time to market, and create great customer experience, Ather turned to Google Cloud.
Read the Full Story on YourStory

ESG Report: Economic Advantages of Google BigQuery OnDemand Serverless Analytics
DOWNLOAD WHITEPAPER5906
Of your peers have already downloaded this article
10:30 Minutes
The most insightful time you'll spend today!
Traditional big data solutions require a significant upfront investment, ongoing maintenance of hardware and software, and manual provisioning to match compute and storage resources with demand. Google’s serverless analytics warehouse does away with all that extra work, helping businesses focus on what matters most: getting value from their data.
Enterprise Strategy Group (ESG), which examined the economic value propositions of Google BigQuery and alternative big data solutions, reports that BigQuery enables organizations to:
- Save up to 88 percent on data warehousing over a three-year period
- Achieve a faster time to value by getting up and running quickly
- Empower more employees to become citizen data scientists
Eliminate maintenance tasks so teams can spend more time gaining insights
Download the complete report to learn more.
2916
Of your peers have already watched this video.
36:00 Minutes
The most insightful time you'll spend today!
What’s New in BigQuery, Google Cloud’s Modern Data Warehouse
The demands that today’s enterprise has from data go far beyond the capabilities of traditional data warehousing. For many leaders, the need to digitally transform their businesses is a key driver for data analytics spending.
Businesses want to make real-time decisions from fresh information as well as make predictions from their data in order to remain competitive.
In this video, watch Sudhir Hasbe, Director of Product Management, Data Analytics, Google Cloud and Tino Tereshko, Product Manager, BigQuery, Google Cloud demonstrate what’s new in BigQuery, Google Cloud’s enterprise data warehouse, and hear about all the latest feature innovations and see some amazing demos.

More Relevant Stories for Your Company
3 AI Tools You Can Deploy Immediately
AI has the power to revolutionize every industry—from retail to agriculture, and education to healthcare. Yet many businesses still haven’t begun to adopt AI. There are a number of factors, including the need for specialized talent and hardware, the right types and quantities of data for training and refining machine
How AI is Revolutionizing Retail: Lessons for Marketers
Reach the Right Customers How can AI help you target customers who are looking for products like yours? The challenge: This customer wants to buy the best cat food for her pets. How AI Works The Result A targeted offer for a discount on luxury cat food is shown to

What Swiggy and You Can Learn From This Company’s Use of ML to Engage Customers
The app economy has enabled a huge range of unique business models to flourish. One such model is online food ordering and delivery services, in which apps leverage geo-location data to aggregate local food choices and offer personalized options to consumers. A leading company in this space is Just Eat.

How Vertex AI Helps Coca-Cola Bottlers Japan Analyze Billions of Data Records
Japan is home to millions of vending machines installed on streets and in buildings, sports stadiums and other facilities. Vending machine owners and operators, including beverage manufacturers, stock these machines with different product combinations depending on location and demand. For example, they primarily display coffee and energy drinks in machines








