Experts’ Guideline for Personalizing Platforms with the Right Recommendation System on Google Cloud
5021
Of your peers have already read this article.
5:00 Minutes
The most insightful time you'll spend today!
Over the past two decades, consumers have become accustomed to receiving personalized recommendations in all facets of their online life. Whether that be recommended products while shopping on Amazon, a curated list of apps in the Google Play store, or relevant videos to watch next on YouTube. In fact, in a Verge article “How YouTube perfected the feed: Google Brain gave YouTube new life,” the Google Brain team reveals how their recommendation engine has impacted the platform with “more than 70 percent of the time people spend watching videos on the site being driven by YouTube’s algorithmic recommendations” thereby increasing time spent on the platform by 20X in three years.
It’s become clear that personalized recommendations are no longer a differentiator for an organization but rather something consumers have come to expect in their day-to-day experiences online. So what should you do if you are behind the curve and want to get started or simply want to improve upon what you already have? While there are all sorts of techniques, from content-based systems to deep learning methods, our goal in this recommender-focused blog series is to demystify three available approaches to building recommendation systems on Google Cloud: Matrix Factorization in BigQuery Machine Learning (BQML), Recommendations AI, and deep retrieval techniques available via the Two-Tower built-in algorithm.
One of these approaches can be used to meet you where you are in your personalization journey, no matter if you are just starting or if you are well into it. This first blog post will introduce our three approaches and when to use them.
What is Matrix Factorization and how does it work?
Collaborative filtering is a foundational model for building a recommendation system as the input dataset is simple and the embeddings are learned for you. How does Matrix factorization fit into the mix you might be wondering? Matrix factorization is simply the model that applies collaborative filtering. BQML enables users to create and execute a matrix factorization model by using standard SQL directly in the data warehouse.
Collaborative filtering begins by creating an interaction matrix. The interaction matrix represents users as a row and items as columns in your dataset. This interaction matrix often is sparse in nature as not all users will have interacted with many items in your catalog. This is where embeddings come into play. Generating embeddings for users and items not only allows you to collapse many sparse features into a lower dimensional space but they also allow you to derive a similarity measure so that similar users/items fall nearby in the embedding space. These similarity measures are key as collaborative filtering uses similarities between users and items to make the end recommendations. The underlying assumption being that similar users will like similar items whether that be movies or handbags.
What’s required to get started?
To train a matrix factorization model you need a table that includes three input columns: user(s), item(s), and an implicit or explicit feedback variable (e.g., ratings is an example of explicit feedback). With the base input dataset in place, you can then easily run your model in BigQuery after specifying several hyperparameters in your CREATE MODEL SQL statement. Hyperparameters are available to specify the number of embeddings, the feedback type, the amount of L2 regularization applied and so on.
Why use this approach and who is it a good fit for?
As mentioned earlier, Matrix Factorization in BQML is a great way for those new to recommendation systems to get started. Matrix factorization has many benefits:
- Little ML Expertise: Leveraging SQL to build the model lowers the level of ML expertise needed
- Few Input Features: Data inputs are straightforward, requiring a simple interaction matrix
- Additional Insight: Collaborative filtering is adept at discovering new interests or products for users
While Matrix Factorization is a great tool for deriving recommendations it does come with additional considerations and potential drawbacks depending upon the use case.
- Not Amenable to Large Feature Sets: The input table can only contain two feature columns (e.g., user(s), item(s)). If there is a need to include additional features such as contextual signals, Matrix factorization may not be the right method for you.
- New Items: If an item is not available in the training data, the system can’t create an embedding for it and will have difficulty recommending similar items. While there are some workarounds available to address this cold-start issue, if your item catalog often includes new items, Matrix factorization may not be a good fit.
- Input Data Limitations: While the input matrix is expected to be sparse, training examples without feedback can cause problems. Filtering for items and users that have at least a handful of feedback (e.g., ratings) examples can improve the model. More information on limitations can be found here.
In summary, for users with a simplified dataset looking to iterate quickly and develop a baseline recommendation system, Matrix Factorization is a great approach to begin your personalization AI journey.
What is Recommendations AI and how does it work?
Recommendations AI is a fully managed service which helps organizations deploy scalable recommendation systems that use state-of-the-art deep learning techniques, including cutting-edge architectures such as two-tower encoders, to serve personalized and contextually relevant recommendations throughout the customer journey.
Deep learning models are able to improve the context and relevance of recommendations in part because they can easily address the previously mentioned limitations of Matrix Factorization. They incorporate a wide set of user and item features, and by definition they emphasize learning successive layers of increasingly meaningful representations from these features. This flexibility and expressivity allows them to capture complex relationships like short-lived fashion trends and niche user behaviors. However, this increased relevance comes at a cost, as deep learning recommenders can be difficult to train and expensive to serve at scale.
Recommendations AI helps organizations take advantage of serving these deep learning models and handles the MLOps required to serve these models globally with low latency. Models are automatically retrained daily and tuned quarterly to capture changes in customer behavior, product assortment, pricing, and promotions. Newly trained models follow a resilient CI/CD routine which validates they are fit to serve and promotes them to production without service interruption. The models achieve low serving latency by using a scalable approximate nearest neighbors (ANN) service for efficient item retrieval at inference time. And, to maintain consistency between online and offline tasks, a scalable feature store is used, preventing common production challenges such as data leakage and training-serving skew.
What’s required to get started?
To get started with Recommendations AI we first need to ingest product and user data into the API:
- Import product catalog: For large product catalog updates, ingest catalog items in bulk using the catalogItems.import method. Frequent catalog updates can be schedule with Google Merchant Center or BigQuery
- Record user events: User events track actions such as clicking on a product, adding items to cart, or even purchasing an item. These events need to be ingested in real time to reflect the latest user behavior and then joined to items imported in the product catalog
- Import historical user events: The models need sufficient training data before they can provide accurate predictions. The recommended user event data requirements are different across model types (learn more here)
Once the data requirements are met, we are able to create one or multiple models to serve recommendations:
- Determine your recommendation types and placements: The location of the recommendation panel and the objective for that panel impact model training and tuning. Review the available recommendations types, optimization objectives, and other model tuning options to determine the best options for your business objectives.
- Create model(s): Initial model training and tuning can take 2-5 days depending on the number of user events and size of the product catalog
- Create serving configurations and preview recommendations: After the model is activated, create serving configurations and preview the recommendations to ensure your setup is functioning as expected before serving to production traffic
Once models are ready to serve, consider setting up A/B experiments to understand how newly trained models impact your customer experience before serving them to 100% of your traffic. In the Recommendations AI console, see the Monitoring & Analytics page for summary and placement-specific metrics (e.g., recommender-engaged revenue, click-through-rate, conversion rate, and more).
Why use this approach and who is it a good fit for?
Recommendations AI is a great way to engage customers and grow your online presence through personalization. It’s used by teams who lack technical experience with production recommendation systems, as well as customers who have this technical depth but want to allocate their team’s effort towards other priorities and challenges. No matter your team’s technical experience or bandwidth, you can expect several benefits with Recommendations AI:
- Fully managed service: no need to preprocess data, train or hypertune machine learning models, load balance or manually provision you infrastructure – this is all taken care of for you. The recommendation API also provides a user-friendly console to monitor performance over time.
- State-of-the-art AI: take advantage of the same modeling techniques used to serve recommendations across Google Ads, Google Search, and YouTube. These models excel in scenarios with long-tail products and cold-starts users and items
- Deliver at any touchpoint: serve high-quality recommendations to both first-time users and loyal customers anywhere in their journey via web, mobile, email, and more
- Deliver globally: serve recommendations in any language anywhere in the world at low-latency with a fully automated global serving infrastructure
- Your data, your models: Your data and models are yours. They’ll never be used for any other Google product nor shown to any other Google customer
For users looking to leverage state of the art AI to fuel their recommendation systems but need an existing solution to get up and running more quickly, Recommendations AI is the right solution for you.
What are Two Tower encoders and how do they work?
As a reminder, in recommendation system design, our objective is to surface the most relevant set of items for a given user or set of users. The items are usually referred to as the candidate(s) where we might include information about the items such as the title or description of the item, other metadata about the item like language, number of views, or even clicks on the item over time. User(s) are often represented in the form of a query to a recommendation system where we might provide details about the user such as the location of the user, preferred languages, and what they have searched for in the past.
Let’s start with a common example. Imagine that you are creating a movie recommendation system. The input candidates for such a system would be thousands of movies and the query set can consist of millions of viewers. The goal of the retrieval stage is to select a smaller subset of movies(candidates) for each user and then score and rank order them before presenting the final recommended list to the query/user.
The retrieval stage is able to refine our list of candidates by encoding both the candidate and the query data so they share the same embedding space. A good embedding space will place candidates which are similar to one another closer together and dissimilar items/queries farther apart in the embedding space.
Once we have a database of query and candidate embeddings we can then use an approximate nearest neighbor search method to then generate a list of final “like” candidates, i.e. find a certain number of nearest neighbors for a given query/user and surface final recommendations.
What’s required to get started?
At the most basic level, in order to train a two-tower model you need the following inputs:
- Training Data: Training data is created by combining your query/user data with data about the candidates/items. The data must include matched pairs, cases where both user and item information is available. Data in the training set can include many formats from text, numeric data, or even images.
- Input Schema: The input schema describes the schema of the combined training data along with any specific feature configurations.
Several services within Vertex AI have come available that complement the existing Two-Tower built-in algorithm and can be leveraged in your execution:
- Nearest Neighbor (ANN) Service: Vertex AI Matching Engine and ScANN provide a high-scale and low-latency Approximate Nearest Neighbor (ANN) service so you can more easily identify similar embeddings.
- Hyperparameter Tuning Service: A hyperparameter tuning service such as Vizier can help you identify the optimal hyperparameters such as the number of hidden layers, the size of the hidden layers, and the learning rate in fewer trials.
- Hardware Accelerators: Specialized hardware, such as GPUs or TPUs, can be valuable in your recommendation system to help accelerate experiments and improve the speed of training cycles.
Why use this approach and who is it a good fit for?
The Two-Tower built-in algorithm can be considered the “custom sports car” of recommendation systems and comes with several benefits:
- Greater Control: While Recommendations AI uses the two-tower architecture as one of the available architectures it doesn’t provide granular control or visibility into model training, example generation, and model validation details. In comparison, the Two-Tower built in algorithm provides a more customizable approach as you are training a model directly in a notebook environment.
- More Feature Options: The Two Tower approach can handle additional contextual signals ranging from text to images.
- Cold Start Cases: Leveraging a rich set of features not only enhances performance but also allows the candidate generation to work for new users or new candidates.
While the Two-Tower built in algorithm is an excellent and best-in class solution for deriving recommendations, it does come with additional considerations and potential drawbacks depending upon the use case.
- Technical ML Expertise Required: Two tower encoders are not a “plug and play” solution like the other approaches mentioned above. In order to effectively leverage this approach, appropriate coding and ML expertise is required.
- Speed to Insight: Building out a custom solution via two-tower encoders may require additional time as the solution is not pre-built for the user.
For users looking for greater control, increased flexibility, and have the technical chops to easily work within a managed notebook environment – the two-tower built in algorithm is the right solution for them.
What’s next?
In this article, we explored three common methods for building recommendation systems on Google Cloud Platform. As you can see thus far, there are alot of considerations to take into account before choosing a final approach. In an effort to help you align more quickly we have distilled the decision criteria down to a few simple steps (see below for more details).
In the next installments of this series, we will dive more deeply into each method, explore how hardware accelerators can play a key role in recommendation system design, and discuss how recommendation systems may be leveraged in key verticals. Stay tuned for future posts in our recommendation systems series. Thank you for reading! Have a question or want to chat? Find authors here – R.E. [Twitter | LinkedIn], Jordan [LinkedIn], and Vaibhav [LinkedIn].
Acknowledgements
Special thanks to Pallav Mehta, Henry Tappen,Abhinav Khushraj, and Nicholas Edelman for helping to review this post.
References
Is it Cheaper to Run Oracle Workloads on Google Cloud Than On-Prem? Yes, By 78%
DOWNLOAD RESEARCH REPORTS4239
Of your peers have already downloaded this article
3:30 Minutes
The most insightful time you'll spend today!
On-premises relational database management systems (RDBMS) are the traditional heart of many core business operations, but cloud-based relational databases provide the agility and scalability of the cloud while eliminating many of the onerous tasks associated with maintaining your own infrastructure.
To find out how much more value cloud-based relational databases provide, research firm ESG compared an on-premises RDBMS strategy against hosting the same data on one of Google Cloud’s database offerings, called Cloud Spanner.
Its analysis showed that hosting the same data using Google Cloud Spanner is 78% less expensive than using on-premises servers–and up to 37% less expensive than using other cloud databases.
Here’s a quick snapshot of where those benefits stem from:
To read the full report and discover how Google Cloud Spanner ranked against other cloud-based database offerings, download ESG’s report, Analyzing the Economic Benefits of Google Cloud Spanner Relational Database Service.
Unlock Powerful Insights: Monitor Cloud Expenses with BigQuery’s Data Export
1316
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
TL;DR – Regular file export for Google Cloud billing is going away on May 15th! Try out this interactive tutorial to set up exporting and explore your billing data with BigQuery
You may have seen an email from Google Cloud reminding you to set up billing data export to BigQuery, and letting you know that file export options are going away. More specifically:
Starting May 15, 2023, your daily usage and cost estimates won’t be automatically exported to a Cloud Storage bucket. Your existing Cloud Storage buckets will continue to exist, but no new files will be added on an ongoing basis.
In fact, I even received one of these notifications since I had an old billing account (or two) that still had file export enabled. Even though the file export is going away, exporting your billing data to BigQuery gives you more detail and more capabilities to analyze, so it’s a big upgrade.
Start from the start
While Google Cloud does offer a free trial and free tier of services, it shouldn’t come as a surprise that most cloud services cost money to use. With the unique properties of the cloud, it’s much easier to bill for exact usage of various services, like pay-per-second for virtual machines. However, this can also become an overwhelming amount of information when a large number of services are being used across multiple teams. In order to make sure you understand your billing data, Google Cloud offers that data through direct reporting in the Cloud console, file exports, and exporting to BigQuery.
Most importantly, the file export for Cloud Billing data is going away on May 15th (the file exports for specific reports, like the cost table, aren’t going away). This was an option for billing accounts to generate daily CSV or JSON files into a Cloud Storage bucket so you could download the files and analyze them as needed. While these exports were definitely helpful for looking at your billing data, over time they couldn’t keep up with the complexity and volume of data that comes out of a billing account.
The most important thing to note here is simple: this export option is being deprecated entirely, so if you’re currently using it (and even if you’re not), you should make sure exporting to BigQuery is enabled.
Large data, large queries
Exporting from your billing account into BigQuery has been an option for the past few years, and now more than ever it’s the central option for managing your billing data. Of course, BigQuery is a natural home for large datasets with the scale and granularity of your billing data. In addition to that, exporting to BigQuery also happens more frequently than a daily file export, so you can do cost anomaly detection much faster.
As teams and organizations become larger, it can be increasingly difficult to figure out which groups are responsible for different costs. If your organization is using a chargeback model (where one group pays for all resources and then breaks down cost details for individual teams/groups to be responsible for their usage), then using BigQuery is the best way for you to accurately collect and analyze costs. On top of being able to query directly against your data, BigQuery also makes it much easier to integrate with other tools, like Looker or Looker Studio for visualization.
The actual data being exported to BigQuery are hourly records of every service your billing account is responsible for, along with detailed information like the SKU, what type of unit is being measured, and how many of that unit you’re paying for.
I suspect most folks using the file export (and probably most folks overall) will be fine with the standard export for BigQuery, and can use the detailed cost and pricing options if they need even more details. You can find some more details on all the exports in this blog post.
With the file export going away, it’s also important to mention: exporting your Cloud Billing data to BigQuery starts when you enable it, and doesn’t include older data. So make sure you set yourself up for future success by enabling billing export as soon as possible. Try using our built-in walkthrough to help you enable the export and to get started analyzing the data! You can also read more in the documentation.
Updating Twitter’s Ad Engagement Analytics Platform for the Modern Age
3110
Of your peers have already read this article.
3:30 Minutes
The most insightful time you'll spend today!
As part of the daily business operations on its advertising platform, Twitter serves billions of ad engagement events, each of which potentially affects hundreds of downstream aggregate metrics. To enable its advertisers to measure user engagement and track ad campaign efficiency, Twitter offers a variety of analytics tools, APIs, and dashboards that can aggregate millions of metrics per second in near-real time.
In this post, you’ll get details on how the Twitter Revenue Data Platform engineering team, led by Steve Niemitz, migrated their on-prem architecture to Google Cloud to boost the reliability and accuracy of Twitter’s ad analytics platform.
Deciding to migrate
Over the past decade, Twitter has developed powerful data transformation pipelines to handle the load of its ever-growing user base worldwide. The first deployments for those pipelines were initially all running in Twitter’s own data centers. The input data streamed from various sources into Hadoop Distributed File System (HDFS) as LZO-compressed Thrift files in an Elephant Bird container format. The data was then processed and aggregated in batches by Scalding data transformation pipelines. Then, aggregation results were output into Manhattan, Twitter’s homegrown distributed key-value store, for serving. Additionally, a streaming system using Twitter’s homegrown systems Eventbus (a messaging tool built on top of DistributedLog), Heron (a stream processing engine), and Nighthawk (a sharded Redis deployment) powered the real-time analytics that Twitter had to provide, filling the gap between the current time and the last batch run.
While this system consistently sustained massive scale, its original design and implementation was starting to reach some limits. In particular, some parts of the system that had grown organically over the years were difficult to configure and extend with new features. Some intricate, long-running jobs were also unreliable, leading to sporadic failures. The legacy end-user serving system was very expensive to run and couldn’t support large queries.
To accommodate for the projected growth in user engagement over the next few years and streamline the development of new features, the Twitter Revenue Data Platform engineering team decided to rethink the architecture and deploy a more flexible and scalable system in Google Cloud.
Platform modernization: First iteration
In the middle of 2017, Steve and his team tackled the first redesign iteration of its advertising data platform modernization, leading to Twitter’s collaboration with Google Cloud.
At first, the team left the data aggregation legacy Scalding pipelines unchanged and continued to run them in Twitter’s data centers. But the batch layer’s output was switched from Manhattan to two separate storage locations in Google Cloud:
- BigQuery—Google’s serverless and highly scalable data warehouse, to support ad-hoc and batch queries.
- Cloud Bigtable—Google’s low-latency, fully managed NoSQL database, to serve as a back end for online dashboards and consumer APIs.
The output aggregations from the Scalding pipelines were first transcoded from Hadoop sequence files to Avro on-prem, staged in four-hour batches to Cloud Storage, and then loaded into BigQuery. A simple pipeline deployed on Dataflow, Google Cloud’s fully managed streaming and batch analytics service, then read the data from BigQuery and applied some light transformations. Finally, the Dataflow pipeline wrote the results into Bigtable.
The team built a new query service to fetch aggregated values from Bigtable and process end-user queries. They deployed this query service in a Google Kubernetes Engine (GKE) cluster in the same region as the Bigtable instance to optimize for data access latency.
Here’s a look at the architecture:
This first iteration already brought many important benefits:
- It de-risked the overall migration effort, letting Twitter avoid migrating both the aggregation business logic and storage at the same time.
- The end-user serving system’s performance improved substantially. Thanks to Bigtable’s linear scalability and extremely low latency for data access, the serving system’s P99 latencies decreased from 2+ seconds to 300ms.
- Reliability increased significantly. The team now rarely, if ever, gets paged for the serving system anymore.
Platform modernization: second iteration
With the new serving system in place, in 2019 the Twitter team began to redesign the rest of the data analytics pipeline using Google Cloud technologies. The redesign sought to solve several existing pain points:
- Because the batch and streaming layers ran on different systems, much of the logic was duplicated between systems.
- While the serving system had been moved into the cloud, the existing pain points of the Hadoop aggregation process still existed.
- The real-time layer was expensive to run and required significant operational attention.
With these pain points in mind, the team began evaluating technologies that could help solve them. They considered several open-source stream processing frameworks initially: Apache Flink, Apache Kafka Streams, and Apache Beam. After evaluating all possible options, the team chose Apache Beam for a few key reasons:
- Beam’s built-in support for exactly-once operations at extremely large scale across multiple clusters.
- Deep integration with other Google Cloud products, such as Bigtable, BigQuery, and Pub/Sub, Google Cloud’s fully managed, real-time messaging service.
- Beam’s programming model, which unifies batch and streaming and lets a single job operate on either batch inputs (Cloud Storage), or streaming inputs (Pub/Sub).
- The ability to deploy Beam pipelines on Dataflow’s fully managed service.
The combination of Dataflow’s fully managed approach and Beam’s comprehensive feature set let Twitter simplify the structure of its data transformation pipeline, as well as increase overall data processing capacity and reliability.
Here’s what the architecture looks like after the second iteration:
In this second iteration, the Twitter team re-implemented the batch layer as follows: Data is first staged from on-prem HDFS to Cloud Storage. A batch Dataflow job then regularly loads the data from Cloud Storage, processes the aggregations, and dual-writes the results to BigQuery for ad-hoc analysis and Bigtable for the serving system.
The Twitter team also deployed an entirely new streaming layer in Google Cloud. For data ingestion, an on-prem service now pushes two different streams of Avro-formatted messages to Pub/Sub. Each message contains a bundle of multiple raw events and affects between 100 and 1,000 aggregations. This leads to more than 3 million aggregations per second performed by four Dataflow jobs (J0-3 in the diagram above). All Dataflow jobs share the same topology, although each job consumes messages from different streams or topics.
One stream, which contains critical data, enters the system at a rate of 200,000 messages per second and is partitioned in two separate Pub/Sub topics. A Dataflow job (J3 in the diagram) consumes those two streams, performs 400,000 aggregations per second, and outputs the results to a table in Bigtable.
The other stream, which contains less critical but higher volume data, enters the system at a rate of around 80,000 messages per second and is partitioned into six separate topics. Three Dataflow jobs (J0, J1, and J2) share the processing of this larger stream, with each of them handling two of the available six topics in parallel, then also outputting the results to a table in Bigtable. In total, those three jobs process over 2 million aggregations/second.
Partitioning the high-volume stream into multiple topics offers a number of advantages:
- The partitioning is organized by applying a hash function on the aggregation key and then dividing the function’s result by the number of available partitions (in this case, six). This guarantees that any per-key grouping operation in downstream pipelines is scoped to a single partition, which is required for consistent aggregation results.
- When deploying updates to the Dataflow jobs, admins can drain and relaunch each job individually in sequence, allowing the remaining pipelines to continue uninterrupted and minimizing impact on the end users.
- The three jobs can each handle two topics without issue currently, and there is still room to scale horizontally up to six jobs if needed. The number of topics (six) is arbitrary, but is a good balance at the moment based on current needs and potential spikes in traffic.
To assist with job configuration, Twitter initially considered using Dataflow’s template system, a powerful feature that enables the encapsulation of Dataflow pipelines into repeatable templates that can be configured at runtime. However, since Twitter needed to deploy jobs with topologies that might change over time, the team decided instead to implement a custom declarative system where developers can specify different parameters for their jobs in a pystachio DSL: tuning parameters, data sources to operate on, sink tables for aggregation outputs, and the jobs’ source code location. A new major version of Dataflow templates, called Flex Templates, will remove some of the previous limitations with the template architecture and allow any Dataflow job to be templatized.
For job orchestration, the Twitter team built a custom command line tool that processes the configuration files to call the Dataflow API and submit jobs. The tool also allows developers to submit a job update by automatically performing a multi-step process, like this:
- Drain the old job:
- Call the Dataflow API to identify which data sources are used in the job (for example, a Pub/Sub topic reader).
- Initiate a drain request.
- Poll the Dataflow API for the watermark of the identified sources until the maximum watermark is hit, which indicates that the draining operation is complete.
- Launch the new job with the updated code.
This simple, flexible, and powerful system allows developers to focus on their data transformation code without having to be concerned about job orchestration or the underlying infrastructure details.
Looking ahead
Six months after fully transitioning its ad analytics data platform to Google Cloud, Twitter has already seen huge benefits. Twitter’s developers have gained in agility as they can more easily configure existing data pipelines and build new features much faster. The real-time data pipeline has also greatly improved its reliability and accuracy, thanks to Beam’s exactly-once semantics and the increased processing speed and ingestion capacity enabled by Pub/Sub, Dataflow, and Bigtable.
Twitter engineers have enjoyed working with Dataflow and Beam for several years now, since version 2.2, and plan to continue expanding their usage. Most importantly, they’ll soon merge the batch and streaming layers into a single, authoritative streaming layer.
Throughout this project, the Twitter team collaborated very closely with Google engineers to exchange feedback and discuss product enhancements. We look forward to continuing this joint technical effort on several ongoing large-scale cloud migration projects at Twitter. Stay tuned for more updates!
The Right Datawarehouse Helps Fight Climate Change, While Improving Customer Experience
7084
Of your peers have already read this article.
4:30 Minutes
The most insightful time you'll spend today!
When you think about climate change, you might not consider a daily commute to work or a drive around town as big contributing factors. And yet, transport is the fastest growing source of CO2 emissions from fossil fuel, which in turn is the largest contributor to climate change. This is the key insight behind the mission of Spanish carbon-neutral multinational Acciona. The group, which provides sustainable solutions for infrastructure and renewable energy projects across 65 countries, launched an electric scooter sharing service called Acciona Mobility in 2018.
Aiming to contribute to the decarbonization of the transport sector while helping relieve traffic congestion within cities, Acciona Mobility enables users to rent electric scooters powered 100% by energy from renewable sources. Users can find, reserve, and rent the scooters at the tap of their mobile screens via the Acciona Mobility application and pay on the basis of minutes spent riding. The app, which is available free to download online, is also the gateway through which new users can register for the service, which welcomes anyone with a valid driver’s license for motorbikes. After having their right to drive, identity, and card details validated, citizens can start enjoying the service, available 24 hours a day, every day of the year.
“We needed to deploy a reliable infrastructure for a new service before understanding exactly what its scale and demand would be. Our strategy was to adopt a serverless architecture that can grow with us to sustain our long-term vision. This kind of thinking led us directly to Google Cloud.”
—Jose Luis Rosell, CIO Services Division, Acciona
“Acciona Mobility is exciting because it stems from our vision to make people’s lives easier in a sustainable way,” says Jose Luis Rosell, CIO of Acciona’s Services Division. “40% of the pollution near cities comes from private transportation. We want to help solve that problem with zero-carbon, electric, multimodal transportation that’s also convenient for people to use.”
However, Jose Luis says that building the appropriate IT infrastructure to bring the new service to life required external support. “We needed to deploy a reliable infrastructure for a new service before understanding exactly what its scale and demand would be. Our strategy was to adopt a serverless architecture that can grow with us to sustain our long-term vision,” he explains. “This kind of thinking led us directly to Google Cloud.”
Developing the transportation platform of the future
Because Acciona was already a Google Cloud customer, Jose Luis and his team reached the decision to develop the new service on Google Cloud with ease, as he explains: “We had a very positive experience working closely with the Google Cloud team during Acciona’s first project on Google Cloud, and that collaborative mentality is very important for us. We feel Google Cloud is a trustworthy partner that is willing to take risks with us and is flexible enough to lead us through business uncertainty with the right technology,” he says. “So we were confident that its technology and people could help us bring the Acciona Mobility service from theory to reality.”
This time, Acciona partnered with cloud consultancy Altostratus as well, to consult on a selection of Google Cloud products that would help develop the application quickly by making use of managed services. Acciona’s strategy was to launch the new services first in Madrid, where the company is headquartered, and then spread the vision across more cities later, if the idea worked out well in practice. Due to the scale of the project and its data needs, BigQuery was chosen as the primary data warehouse to store all the information related to the service, such as the location of scooters and their availability status. With all its data readily available on BigQuery, Acciona is able to run specific queries that help it to gain insights such as which urban areas have the highest volume of scooters being rented. Knowing this, Acciona can reorganize availability to make sure there are always enough scooters in that specific location.
Integrating it with Pub/Sub and Dataflow, Acciona ensures that the data generated by scooters is ingested and processed in real time so that users searching for scooters nearby, using the Acciona Mobility application on their phones, can always have up-to-date information at hand. Using Google Maps Platform APIs such as the Directions API, Acciona’s mobile app translates the geographical coordinates of scooters into an easy-to-read address displayed beside a visual map. Using Cloud SQL, Acciona automates the storage capacity management of the database as the number of scooters and active users grow.
Meanwhile, Google Cloud Armor protects the service against cyber breaches and distributed denial of service attacks to keep it running uninterrupted. In addition, Google Cloud itself helps to reinforce the cyber security measures with secure-by-default managed services, such as data encryption. “The managed services mean more resilience and more uptime, because we don’t need to worry about maintenance or external threats,“ says Jose Luis of the solutions protecting all information generated by the new service, which as CIO is a topic he holds dearly. “I face cybersecurity issues more confidently as a Google Cloud partner,” he adds.
“Today, we have data coming in every 15 seconds from 10,000 electric scooters around Europe. BigQuery and Cloud SQL-managed services help us handle that data very carefully and with precision so that our users don’t experience any lags when searching for scooters or returning them.”
—Jose Luis Rosell, CIO Services Division, Acciona
Expanding an 100% sustainable transport solution throughout Spain
Within two months of the decision to deploy Acciona Mobility on Google Cloud, the service was ready for launch, filling the streets of Madrid with 500 electric scooters that run 100% on renewable energy. By 2020, the number of scooters has grown to 10,000 and the service has scaled to multiple cities in and outside of Spain, including Lisbon, Milan, and Rome. To date, more than 3.5 million intracity rides have used the Acciona Mobility app, reducing CO2 emissions by more than 1,000 tons.
“Today, we have data coming in every 15 seconds from 10,000 electric scooters around Europe. BigQuery and Cloud SQL-managed services help us handle that data very carefully and with precision so that our users don’t experience any lags when searching for scooters or returning them,” says Jose Luis of the fast-growing scale of the project.
“Google Cloud enables us to keep our services always available and to scale quickly to respond to its growing demand. Our ratings have been very positive as a result. More importantly, our vision of transportation as a cleaner, more sustainable commodity is really spreading and resonating with the public.”
—Jose Luis Rosell, CIO Services Division, Acciona
Set to continue expanding, Acciona is using Google Kubernetes Engine to make sure the Acciona Mobility application has the computing capacity it needs to continue scaling to reach new markets and serve more users. On that note, Google operations tools (formerly Stackdriver) are also being deployed to help monitor, troubleshoot, and improve the performance of the application as it scales.
“Google Cloud enables us to keep our services always available and to scale quickly to respond to its growing demand. Our ratings have been very positive as a result. More importantly, our vision of transportation as a cleaner, more sustainable commodity is really spreading and resonating with the public,” he concludes.
Why More DB Admins Love Google Cloud Spanner, Its Best Uses Cases—and What it Costs
3772
Of your peers have already read this article.
2:45 Minutes
The most insightful time you'll spend today!
Cloud Spanner is the first scalable, enterprise-grade, globally-distributed, and strongly consistent database service built for the cloud specifically to combine the benefits of relational database structure with non-relational horizontal scale.
This combination delivers high-performance transactions and strong consistency across rows, regions, and continents with an industry-leading 99.999% availability SLA, no planned downtime, and enterprise-grade security.
Cloud Spanner revolutionizes database administration and management and makes application development more efficient.
Find out How Streak built a graph database on Cloud Spanner to wrangle billions of emails
What Are Cloud Spanner’s Strengths?
What is Cloud Spanner Useful For?
What Does it Cost?
Pricing for Cloud Spanner is simple and predictable. You are only charged for the number of nodes in your instance, the amount of storage that your tables and secondary indexes use (not pre-provisioned), and the amount of network bandwidth used. Note that there is no additional charge for replication. For more detailed pricing information, please view the pricing guide.
More Relevant Stories for Your Company
Speed Up Data-driven Innovation in Life Sciences with Google Cloud
The last few years have underscored the importance of speed in bringing new drugs and medical devices to market, while ensuring safety and efficacy. Over this time, healthcare and life sciences organizations have transformed the way they research, develop, and deliver patient care by embracing agility and innovation. Now, the
11 Reasons Why Developers and DB Admins in APAC Are Moving to Google Cloud SQL From AWS RDS
We wanted to find out which of the most popular database services on two of the biggest cloud provider (AWS and Google), developers preferred to use. More specifically, we wanted views from enterprises in the region (it's hard to impossible to find data that's specific to India.) To simplify our
Architecting Data Pipelines Directly Improves Customer Experience at Universe.com
At Universe, we serve customers day and night and are always working to make sure they have a great experience, whether online or at one of our live events. Our technology has to make that possible, and our legacy systems weren’t cutting it anymore. What we needed was a consistent,
Turning the Tide: How PrestaShop Regained Trust in Data
Since 2007, PrestaShop has helped companies unlock the power of e-commerce through its open-source platform. Over 300,000 merchants worldwide use the PrestaShop platform to grow their business and serve online shoppers. "Our open-source strategy to ecommerce enablement sets us apart," says Rémi Paulin, Ph.D., Data Architect at PrestaShop. "Customization is
