Transform Your Marketing Strategy with Tinyclues and Google Cloud CDP
2619
Of your peers have already read this article.
4:00 Minutes
The most insightful time you'll spend today!
Editor’s note: The post is part of a series highlighting our awesome partners, and their solutions, that are Built with BigQuery.
What are Customer Data Platforms (CDPs) and why do we need them?
Today, customers utilize a wide array of devices when interacting with a brand. As an example, think about the last time you bought a shirt. You may start with a search on your phone as you take the subway to work. During that 20 minute ride, you narrow down the type of shirt . Later, as you take your lunch break, you spend a few more minutes refining your search on your work laptop and you are able to find two shirt models of interest. Pressed for time, you add both to your shopping cart at an online retailer to review at a later point. Finally, after you arrive back home and as you are checking your physical mail, you stumble across a sales advertisement for the type of shirt that you are looking for, available at your local brick and mortar store. The next day you visit that store during your lunch break and purchase the shirt.
Many marketers face the challenge of creating a consistent 360 customer view that captures the customer lifecycle, as illustrated in the example above – including their online/offline journey, interacting with multiple data points across multiple data sources.
The evolution of managing customer data reached a turning point in the late 90’s with CRM software that sought to match current and potential customers with their interactions. Later as a backbone of data-driven marketing, Data Management Platforms (DMPs) expanded the reach of data management to include second and third party datasets including anonymous IDs. A Customer Data Platform combines these two types of systems, creating a unified, persistent customer view across channels (mobile, web etc) that provide data visibility and granularity at individual level.
A new approach to empowering marketing heroes
Tinyclues is a company that specializes in empowering marketers to drive sustainable engagement from their customers and generate additional revenue, without damaging customer equity. The company was founded in 2010 on a simple hunch: B2C marketing databases contain sufficient amounts of implicit information (data unrelated to explicit actions) to transform the way marketers interact with customers, and a new class of algorithms based on Deep Learning (sophisticated machine learning that mimics the way humans learn) holds the power to unlock this data’s potential. Where other players in the space have historically relied – and continue to rely – on a handful of explicit past behaviors and more than a handful of assumptions, Tinyclues’ predictive engine uses all of the customer data that marketers have available in order to formulate deeply precise models, down even to the SKU level. Tinyclues’ algorithms are designed to detect changes in consumption patterns in real-time, and adapt predictions accordingly.
This technology allows marketers to find precisely the right audiences for any offer during any timeframe, increasing engagement with those offers and, ultimately, revenue; additionally, marketers are able to increase campaign volume while decreasing customer fatigue and opt-outs, knowing that audiences are receiving only the most relevant messages. Tinyclues’ technology also reduces time spent building and planning campaigns by upwards of 80%, as valuable internal resources can be diverted away from manual audience-building.
Google Cloud’s Data Platform, spearheaded by BigQuery, provides a serverless, highly scalable, and cost-effective foundation to build this next generation of CDPs.
Tinyclues Architecture:
To enable this scalable solution for clients, Tinyclues receives purchase and interaction logs from clients in addition to product and user tables. In most cases, this data is already in the client’s BigQuery instance, in which case they can be easily shared with Tinyclues utilizing BigQuery authorized views.
In cases where the data is not in BigQuery, flat files are sent to Tinyclues via GCS and are ingested in the client’s data set via a lightweight Cloud Function. The orchestration of all pipelines is implemented via Cloud Composer (Google’s managed Airflow). The transformation of data is accomplished by utilizing simple select statements in the Data Built Tool (DBT), which is wrapped inside an airflow DAG that powers all data normalization and transformations. There are several other DAGs to fulfill more functionalities, including:
- Indexing the product catalog on Elastic Cloud (Elasticsearch managed service) on GCP to provide auto-complete search capabilities to TCs clients as shown below:
- The export of Tinyclues-powered audiences to the clients’ activation channels, whether they are using SFMC, Braze, Adobe, GMP, or Meta.
Tinyclues AI/ML Pipeline powered by Google Vertex AI
TCs ML Training pipelines are used to train models that calculate propensity scores. They are composed using Airflow DAGs, powered by Tensorflow & Vertex AI Pipelines. BigQuery is used natively, without data movement, to perform as much feature engineering as possible in-place.
TC uses the TFX library to run ML Pipelines in Vertex AI. Building on top of Tensorflow as their main deep learning framework of choice due to its maturity, open source platform, scalability and support for complex data structures (Ragged and Sparse Tensors).
Below is a partial example of TC’s Vertex AI Pipeline graph, illustrating the workflow steps in the training pipeline. This pipeline allows for the modularization & standardization of functionality into easily manageable building blocks. These blocks are composed of TFX components (TC reuses most of the standard components in addition to customizing some such as a proprietary implementation of the Evaluator to compute both ML Metrics (which is part of the standard implementation) but also more Business Metrics like Overlap of clickers etc. The individual components/steps are chained with DSL to form a pipeline that is modular and easily orchestrated or updated as needed.
With the trained Tensorflow models available in GCS, TCs exposes these in BigQuery ML (BQML) to enable their clients to score millions of users for their propensity to buy X or Y within minutes. This would not be possible without the power of BigQuery and also frees TC from previously experienced scalability issues.
As an illustration, TC has the need to score thousands of topics among millions of users. This used to take north of 20 hours on their previous stack, and now takes less than 20 minutes thanks to the optimization work that TC has implemented in their custom algorithm and the sheer power of BQ to scale to any workload accordingly.
Data Gravity: Breaking the Paradigm – Bringing the Model to your Data
BQML enables TC to call pre-trained TensorFlow models within an SQL environment, thus avoiding exporting data in and out of BQ using already provisioned BQ serverless processing power. Using BQML removes the layers between the models and the data warehouse and allows them to express the entire inference pipe as a number of SQL requests. TC no longer has to export data to load it into their models. Instead, they are bringing their models to the data.
Avoiding the export of data in and out of BQ and the serverless provisioning and start of machines saves significant time. As an example, exporting an 11M lines campaign for a large client previously took 15 min or more to process. Deployed on BQML it now takes minutes with more than half of the processing time attributed to network transfers to our client system.
Inference times in BQML compared to TCs legacy stack:
As can be seen, using this approach enabled by BQML, the reduction in the number of steps leads to a 50% decrease in overall inference time, improving upon each step of the prediction.
The Proof is in the pudding
Tinyclues has consistently delivered on its promises of increased autonomy for CRM teams, rapid audience building, superior performance against in-house segmentation, identification of untapped messaging and revenue opportunities, fatigue management, and more, working with partners like Tiffany & Co, Rakuten, and Samsung, among many others.
Conclusion
Google’s data cloud provides a complete platform for building data-driven applications like the headless CDP solution developed by Tinyclues — from simplified data ingestion, processing, and storage to powerful analytics, AI, ML, and data sharing capabilities — all integrated with the open, secure, and sustainable Google Cloud platform. With a diverse partner ecosystem, open-source tools, and APIs, Google Cloud can provide technology companies the portability and differentiators they need to serve the next generation of marketing customers.
To learn more about Tinyclues on Google Cloud, visit Tinyclues. Click here to learn more about Google Cloud’s Built with BigQuery initiative.
We thank the many Google Cloud team members who contributed to this ongoing data platform collaboration and review, especially Dr. Ali Arsanjani in Partner Engineering.
Quick Watch on Six Database Trends
3483
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
In a data-driven, global, always-on world, databases are the engines that let businesses innovate and transform. As databases get more sophisticated and more organizations look for managed database services to handle infrastructure needs, there are a few key trends we’re seeing. Here’s what to watch.
6173
Of your peers have already watched this video.
18:00 Minutes
The most insightful time you'll spend today!
An Overview of Google’s Data Cloud
Data access, management and privacy has been at the center of priorities for enterprises that are aiming to be more agile, reliable and data-driven. Google Cloud’s technology innovations spanning products like BigQuery, Spanner, Looker and VertexAI help organizations navigate the complexities related to siloed data in large volumes sprawled across databases, data lakes, data warehouses, and data marts in multiple clouds and on-premises. Watch the video to learn how companies are building data on Google Cloud for better analysis, security and management to achieve bottomline!
6078
Of your peers have already watched this video.
48:30 Minutes
The most insightful time you'll spend today!
Can Your Data Warehouse Handle a 100-Trillion Row Query?
Today’s enterprise demands from data go far beyond the capabilities of traditional data warehousing and for many leaders, the need to digitally transform their businesses is a key driver for data analytics spending.
Businesses want to make real-time decisions from fresh information as well as make future predictions from their data in order to remain competitive.
In this video, Jordan Tigani, Director of Product Management, Google BigQuery reveals the power of Google Cloud’s modern data warehouse, BigQuery, that helps businesses make informed decisions quickly.
In addition, he talks about how big Google BigQuery can get. He shares examples of how one customer ran a query against a giant table of 100 trillion rows. “I think it was something like 19 petabytes of data scanned. It took about took about 20 minutes. It used 39,000 slots, which is about 20,000 cores,” says Tigani.
He also shares examples of how businesses, such as online retailer, Zulily generate real business benefits from being able to query large datasets faster, and more easily than ever–without having to invest time managing infrastructure.
Finally, Amir Aryanpour, Technical Architect, Channel 4, talks abouut how connecting connecting Google BigQuery to other solutions with the Google Cloud Platform, including storage, data visualisation, and a sentiment analysis engine, among others, helped the company.
Cloud Bigtable Helps Fraud-detection Company Meet Scalability Demands and Secure Customer Data
7115
Of your peers have already read this article.
3:00 Minutes
The most insightful time you'll spend today!
Editor’s note: Today we are hearing from Jono MacDougall , Principal Software Engineer at Ravelin. Ravelin delivers market-leading online fraud detection and payment acceptance solutions for online retailers. To help us meet the scaling, throughput, and latency demands of our growing roster of large-scale clients, we migrated to Google Cloud and its suite of managed services, including Cloud Bigtable, the scalable NoSQL database for large workloads.
As a fraud detection company for online retailers, each new client brings new data that must be kept in a secure manner and new financial transactions to analyze. This means our data infrastructure must be highly scalable and constantly maintain low latency. Our goal is to bring these new organizations on quickly without interrupting their business. We help our clients with checkout flows, so we need latencies that won’t interrupt that process—a critical concern in the booming online retail sector.
We like Cloud Bigtable because it can quickly and securely ingest and process a high volume of data. Our software accesses data in Bigtable every time it makes a fraud decision. When a client’s customer places an order, we need to process their full history and as much data as possible about that customer in order to detect fraud, all while keeping their data secure. Bigtable excels at accessing and processing that data in a short time window. With a customer key, we can quickly access data, bring it into our feature extraction process, and generate features for our models and rules. The data stays encrypted at rest in Bigtable, which keeps us and our customers safe.
Bigtable also lets us present customer profiles in our dashboard to our client, so that if we make a fraud decision, our clients can confirm the fraud using the same data source we use.
We have configured our bigtable clusters to only be accessible within our private network and have restricted our pods access to it using targeted service accounts. This way the majority of our code does not have access to bigtable and only the bits that do the reading and writing have those privileges.
We also use Bigtable for debugging, logging, and tracing, because we have spare capacity and it’s a fast, convenient location.
We conduct load testings against Bigtable. We started at a low rate of ~10 Bigtable requests per second and we peaked at ~167000 mixed read and write requests per second at absolute peak. The only intervention that was done to achieve this was pressing a single button to increase the number of nodes in the database. No other changes were made.
In terms of real traffic to our production system, we have seen ~22,000 req/s (combined read/write) on Bigtable in our live environment as a peak within the last 6 weeks.
Migrating seamlessly to Google Cloud
Like many startups, we started with Postgres, since it was easy and it was what we knew, but we quickly realized that scaling would be a challenge, and we didn’t want to manage enormous Postgres instances. We looked for a kind of key value store, because we weren’t doing crazy JOINS or complex WHERE clauses. We wanted to provide a customer ID and get everything we knew about it, and that’s where key value really shines.
I used Cassandra at a previous company, but we had to hire several people just for that chore. At Ravelin we wanted to move to managed services and save ourselves that headache. We were already heavy users and fans of BigQuery, Google Cloud’s serverless, scalable data warehouse, and we also wanted to start using Kubernetes. This was five years ago, and though quite a few providers offer Kubernetes services now, we still see Google Cloud at the top of that stack with Google Kubernetes Engine (GKE). We also like Bigtable’s versioning capability that helped with a use case involving upserts. All of these features helped us choose Bigtable.
Migrations can be intimidating, especially in retail where downtime isn’t an option. We were migrating not just from Postgres to Bigtable, but also from AWS to Google Cloud. To prepare, we ran in AWS like always, but at the same time we set up a queue at our API level to mirror every request over to Google Cloud. We looked at those requests to see if any were failing, and confirmed if the results and response times were the same as in AWS. We did that for a month, fine tuning along the way.
Then we took the big step and flipped a config flag and it was 100% over to Google Cloud. At the exact same time, we flipped the queue over to AWS so that we could still send traffic into our legacy environment. That way, if anything went wrong, we could fail back without missing data. We ran like that for about a month, and we never had to fail back. In the end, we pulled off a seamless, issue-free online migration to Google Cloud.
Flexing Bigtable’s features
For our database structure, we originally had everything spread across rows, and we’d use a hash of a customer ID as a prefix. Then we could scan each record of history, such as orders or transactions. But eventually we got customers that were too big, where the scanning wasn’t fast enough. So we switched and put all of the customer data into one row and the history into columns. Then each cell was a different record, order, payment method, or transaction. Now, we can quickly look up the one row and get all the necessary details of that customer. Some of our clients send us test customers who place an order, say, every minute, and that quickly becomes problematic if you want to pull out enormous amounts of data without any limits on your row size. The garbage collection feature makes it easy to clean up big customers.
We also use Bigtable replication to increase reliability, atomicity, and consistency. We need strong consistency guarantees within the context of a single request to our API since we make multiple bigtable requests within that scope. So within a request we always hit the same replica of Bigtable and if we have a failure, we retry the whole request. That allows us to make use of the replica and some of the consistency guarantees, a nice little trade-off where we can choose where we want our consistency to live.https://www.youtube.com/embed/0-eH5u7rrQQ?enablejsapi=1&
We also use BigQuery with Bigtable for training on customer records or queries with complicated WHERE clauses. We put the data in Bigtable, and also asynchronously in BigQuery using streaming inserts, which allows our data scientists to query it in every way you can imagine, build models, and investigate patterns and not worry about query engine limitations. Since our Bigtable production cluster is completely separate, doing a query on BigQuery has no impact on our response times. When we were on Postgres many years ago, it was used for both analysis and real time traffic and it was not the optimal solution for us. We also use Elasticsearch for powering text searches for our dashboard.
If you’re using Bigtable, we recommend three features:
- Key visualizer. If we get latency or errors coming back from Bigtable, we look at the key visualizer first. We may have a hotkey or a wide row, and the visualizer will alert us and provide the exact key range where the key lives, or the row in question. Then we can go in and fix it at that level. It’s useful to know how your data is hitting Bigtable and if you’re using any anti-patterns or if your clients have changed their traffic pattern that exacerbated some issue.
- Garbage collection. We can prevent big row issues by putting size limits in place with the garbage collection policies.
- Cell versioning. Bigtable has a 3d array, with rows, columns, and cells, which are all the different versions. You can make use of the versioning to get history of a particular value or to build a time series within one row. Getting a single row is very fast in Bigtable so as long as you can keep the data volume in check for that row, making use of cell versions is a very powerful and fast option. There are patterns in the docs that are quite useful and not immediately obvious. For example, one trick is to reverse your timestamps (MAXINT64 – now) so instead of the latest version, you can get the oldest version effectively reversing the cell version sorting if you need it.
Google Cloud and Bigtable help us meet the low-latency demands of the growing online retail sector, with speed and easy integration with other Google Cloud services like BigQuery. With their managed services, we freed up time to focus on innovations and meet the needs of bigger and bigger customers.
Learn more about Ravelin and Bigtable, and check out our recent blog, How BIG is Cloud Bigtable?
Gartner Magic Quadrant: Google Cloud a Leader in Operational DB Management Systems
4042
Of your peers have already read this article.
2:15 Minutes
The most insightful time you'll spend today!
We’re pleased to announce that Gartner has named Google Cloud a Leader in its 2019 Magic Quadrant report for Operational Database Management Systems (OPDBMS). This news reflects what we hear from our customers: that Google Cloud databases are flexible, open, and easy to use.
These include our fully compatible managed services for popular database engines like MySQL, PostgreSQL, SQL Server and Redis, and scalable cloud-native relational and non-relational databases like Cloud Spanner, Cloud Bigtable, and Cloud Firestore, plus fully managed partner services like MongoDB Atlas, Elastic, and Redis Enterprise. You can also run proprietary database workloads on Google Compute Engine or via our Bare Metal Solution.
Enterprises databases in production
We’ve heard great stories from our customers about their use of Google Cloud databases to run their businesses with ease and flexibility. Our database products meet varying needs for scalability and power.
Gaming company Bandai Namco Entertainment needed fast scalability, a global network, and real-time analytics to serve users its Dragon Ball Legends game. They were initially considering sharded MySQL to handle the scale, but opted for Cloud Spanner. Because it’s strongly consistent, fully managed, and scales seamlessly, Cloud Spanner supported the game’s rollout and allowed millions of worldwide players to compete without downtime.
And media leader The New York Times found our Cloud Firestore database service as they built a truly real-time collaboration tool that lets multiple writers and editors make changes in docs at the same time, keeping track of what’s newest. Cloud Firestore is designed for just this type of task, since it supports offline and real-time sync.
E-commerce brand analytics and protection company 3PM Solutions empowers global brands to manage, protect, and grow revenue by using Google Cloud Platform services such as Cloud Bigtable. Using Google Cloud, they’ve been able to analyze 160 million customer reviews of more than 2 million sellers in less than four hours.
Leanplum built its cloud-native marketing platform on Google Cloud to help marketers personalize and orchestrate cross-channel customer engagement campaigns at scale. “
We were looking for cost-effective cloud services that can reduce engineers’ overhead associated with managing infrastructure,” says Athena Koutsonikolos, Leanplum’s VP of marketing. “Using fully managed services like Cloud SQL, developers can now focus on building what matters to our customers.”
Gartner 2019 Magic Quadrant for Operational Database Management Systems – November 25, 2019, Merv Adrian, Donald Feinberg, Henry Cook.
Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, express or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.
More Relevant Stories for Your Company
Easy Access to Stream Analytics with Google Cloud
By 2025, more than a quarter of the data created in the global datasphere will be real-time in nature. “This is important because in the real-time world, “the window of opportunity diminishes and goes away really fast. You want to be able to respond to your customer needs, their asks,
1 Developer. 5 Months. A Revenue Generating App With 100K Users With Firebase
This is a guest post authored by Firebase customer, Anton Ivanov, Founder & CEO of DealCheck Real estate investing is a fantastic way to build a stream of passive income and grow your wealth. Numerous studies have pointed out that real estate investing has created more millionaires throughout history than
Gartner Identifies Critical Capabilities for Data Management Solutions
Data management solutions for analytics offerings are consolidating, with major vendors able to address a range of use cases and smaller vendors addressing a subset of use cases. Data and analytics leaders can use this research to guide evaluation and initial vendor selection for DMSA offerings. For data and analytics
Three New Features in Cloud SQL for SQL Server Extends its Functionality
As a product with a long history in the database ecosystem, SQL Server offers numerous native capabilities that help provide scalability and security to its users. However, it can be time consuming and complex to take advantage of these features. Google Cloud SQL for SQL Server saves your team time
