There are just under 8 billion people on Earth, depending on the source. Here at Recruit, our work is to develop and maintain an email marketing system that sends personalized emails to tens of millions of customers of hundreds of web services, all with the goal of providing the best, most relevant customer experience.

When Recruit was founded in 1960, the company was focused on helping match graduates to jobs. Over the years, we’ve expanded to help providers that deal with almost every personal moment and event a person encounters in their life. From travel plans to real estate, restaurant choices to haircuts, we offer software and services to help providers deliver on virtually everything and connect to their end-customers.

Recruit depends on email as a key marketing vehicle to end-customers and to provide a communications channel to clients and advertisers across its services. To maximize the impact of these emails, we customize each email we send. To help power this business objective, we developed a proprietary system named “Soup” that we host on Google Cloud Platform (GCP). Making use of Google Cloud Spanner, Soup is the connective tissue that manages the complex customization data needed for this system.

Of course, getting from idea to functioning product is easier said than done. We have massive datasets so requirements like high availability and serving data in real-time are particularly tricky. Add in a complex existing on-premises environment, some of which we had to maintain in our journey to the cloud creating a hybrid environment, and the project became even more challenging.

A Soup Primer

First, why the name “Soup”? The name of the app is actually “dashi-wake” in Japanese, from “dashi,” a type of soup. In theory, Soup is a fairly simple application: its API returns recommendation results based on the data we compute about a user via the user’s user ID. Soup ingests pre-computed recommendations and then serves those recommendations to the email generation engine and tracks metrics. While Soup doesn’t actually send the customer emails, it manages the entire volume of personalization and customization data for tens of millions of users. It also manages the computed metrics associated with these email sends such as opens, clicks, and other metadata.

Soup leverages other GCP services such as App Engine Flex (Node.js), BigQuery, Data Studio, and Stackdriver in addition to Cloud Spanner.

Soup Requirements

High availability

If the system is unavailable when a user decides to open an email they see a white screen with no content at all. Not only is that lost revenue for that particular email, it makes customers less likely to open future emails from us.

Low latency

Given a user ID, the system needs to search all its prediction data and generate the appropriate content—an HTML file, an image, multiple images, or other content—and deliver it, all very quickly.

Real-time log ingestion and fast JOINs

In today’s marketing environment, tracking user activity and being able to make dynamic recommendations based on it is a must-have. We live in an increasingly real-time world. In the past, it might have been OK to take a week or longer to adapt content based on customer behavior. Now? A delay of even a day can make the difference between a conversion and a lost opportunity.

The Problem

Pushing out billions of personalized emails to tens of millions of customers comes with some unique challenges. Our previous on-premises system was based on Apache HBase, the open-source NoSQL database, and Hive data warehouse software. This setup presented three major obstacles:

Cluster sizing

Email marketing is a bursty workload. You typically send a large batch of emails, which requires a lot of compute, and then there’s a quiet period. For our email workloads, we pre-compute a large set of recommendations and then serve those recommendations dynamically upon email open. On-premises, there wasn’t much flexibility and we had to resize clusters manually. We were plagued by errors whenever loads of email opens and the resulting requests to the system outpaced the traffic we could handle, because the cluster size of our HBase/Hive system couldn’t keep up.

Performance

The next issue was optimizing the schema model for performance. Soup has a couple of main functions: services write customer tracking data to it, and downstream “customers” read that data from it to create the personalized emails. On the write side, after the data is written to Soup, the writes need to be aggregated. We initially did this on-premises, which was quite difficult and time consuming because Hbase’s doesn’t offer aggregation queries, and because it was hard to scale in response to traffic bursts.

Transfer delays

Finally, every time we needed to generate a recommendation model for a personalized email blast, we needed to transfer the necessary data from HBase to Hive to create the model, then back to HBase. These complex data transfers were taking two-to-three days. Needless to say, this didn’t allow for the type of agility that we need to provide the best service to our customers.

Cloud Spanner allows us to store all our data in one place, and simply join the data tables and do aggregates; there’s no need for a time-intensive data transfer. Using this model, we believe we can cut the recommendation generation time from days to under a minute, bringing real-time back into the equation.

Why Cloud Spanner?

Compared to the previous application running on-premises, Cloud Spanner offers lower cost, lower operations requirements and higher availability. Most critically, we wanted to calculate metrics (KPIs) in real time without data transfer. Cloud Spanner allows us to do this by pumping SQL queries into a custom dashboard that monitors KPIs in real time.

Soup now runs on GCP, although the recommendations themselves are still generated in an on-premise Hadoop cluster. The computed recommendations are stored in Cloud Spanner for the reasons mentioned above. After moving to GCP and architecting for the cloud, we see an error rate of .005% per second vs. a previous rate of 4% per second, an improvement of 1/800. This means that for an email blast sent to all users in Japan, one user won’t be able to see one image in one email. Since these emails often contain 10 images or more, this error rate is acceptable.

Cloud Spanner also solved our scaling problem. In the future, Soup will have to support one million concurrent users in different geographical areas. Likewise, Soup has to perform 5,000 queries per second (QPS) at peak times on the read side, and will expand this requirement to 20,000 to 30,000 QPS in the near future. Cloud Spanner can handle all the different, complex transactions Soup has to run, while scaling horizontally with ease.

Takeaways

In migrating our database to Cloud Spanner, we learned many things that are worth taking note of, whether you have 10 or 10 million users.

Be prepared to scale

We took scaling into account from Day One, sketching out specific requirements for speed, high availability, and other metrics. Only by having these requirements specifically laid out were we able to choose—and build—a solution that could meet them. We knew we needed elastic scale.

With Cloud Spanner, we didn’t have to make any of the common trade-offs between the relational database structure we wanted, and the scalability and availability needed to keep up with the business requirements. Likewise, with a growing company, you don’t want to place any artificial limits on growth, and Cloud Spanner’s ability to scale to “arbitrarily large” database sizes eliminates this cap, as well as the need to rewrite or migrate in the future as our data needs grow.

Be realistic about downtime

For us, any downtime can result in literally thousands of lost opportunities. That meant that we had to demand virtually zero downtime from any solution, to avoid serving up errors to our users. This was an important realization. Google Cloud provides an SLA guarantee for Cloud Spanner. This solution is more available and resistant to outages than anything we would build on our own.

Don’t waste time on management overhead

When you’re worrying about millions of users and billions of emails, the last thing you have time to do is all the maintenance and administrative tasks required to keep a database system healthy and running. Of course, this is true for the smallest installations, as well. Nobody has a lot of extra time to do things that should be taken care of automatically.

Don’t be afraid of hybrid

We found that a hybrid architecture that leverages the cloud for fast data access but still using our existing on-premises investments for batch processing to be effective. In the future, we may move the entire workload to the cloud but data has gravity, and we currently have lots of data stored on-premises.

Aim for real-time

At this time, we can only move data in and out of Cloud Spanner in small volumes. This prevents us from making real-time changes to recommendations. Once Cloud Spanner supports batch and streaming connections, we’ll be able to enable an implementation to provide more real-time recommendations to deliver even more relevant results and outcomes.

Overall, we’re extremely happy with Cloud Spanner and GCP. Google Cloud has been a great partner in our move to the cloud, and the unique services provided enable us to offer the best service to our customers and stay competitive.

We are thrilled that Gartner has positioned Google as a Leader for the second year in a row in the 2021 Gartner® Magic Quadrant™ for Cloud Database Management Systems (DBMS).

We believe the report evaluated Google’s unified capabilities across both transactional and analytical use cases, and showcases innovation progress in areas like data management consistency, high speed processing and ingestion, security, elasticity, advanced analytics, and more. 

With the recent announcement of Dataplex, organizations can centrally manage, monitor, and govern their data across data lakes, data warehouses, and data marts with consistent controls.  Solutions like BigQuery ML provide a “built-in” approach for advanced analytics capabilities and Analytics Hub offer the infrastructure customers need to share data analytics solutions securely and at scale in ways never before achieved. For example, over a seven-day period in April, more than 3,000 different organizations shared over 200 petabytes of data using BigQuery. 

Research shows 90% of organizations have a multicloud strategy, which is why we have invested in a cross-cloud data analytics solution for Google Cloud, AWS, and Azure with BigQuery Omni. Additionally, our progress with Anthos and our Distributed Cloud this past year further advance our ability to support multi and hybrid cloud scenarios. To gain a competitive advantage using data, organizations need a data platform that transcends transactional and analytical workloads and can be run with the highest level of reliability, availability and security. Cloud Spanner, our globally distributed relational database has redefined the scale, global consistency, and availability of Online Transaction Processing (OLTP) systems. Spanner processes over 1 billion requests per second at peak, and has been battle-tested with some of the most-demanding applications, including Google services such as Search, YouTube, Gmail, Maps, and Payments. And what’s unique about our core services Spanner and BigQuery, is that they leverage common infrastructure such as our highly durable distributed file system (Colossus), our large-scale cluster management system (Borg), and Jupiter, our high-performance networking infrastructure, enabling features such as federation between Spanner and BigQuery.

We remain focused on the integration of Google Trends, Maps, Search, Ads, and have increased industry domain expertise in areas such as retail, financial services, healthcare, and gaming. We’re continuing to develop industry white papers such as this one – How to develop Global Multiplayer Games using Cloud Spanner – and we’re proud of the work the team has done to create and share industry and horizontal architecture patterns built from industry leaders to serve as solution accelerators for customer use cases. 

Innovation momentum continues with a unified and open data cloud

We continue to innovate across our data cloud portfolio, especially with the innovations we announced at Google Cloud NEXT’21. BigQuery Omni is now available for AWS and Azure, supporting cross-cloud analytics for customers. We’ve added additional capabilities for enterprise data management and governance with Dataplex, which recently went GA. We’ve made migrations to Cloud SQL easier and faster with the Database Migration Service. More than 85% of all migrations are underway in under an hour, with the majority of customers migrating their databases from other clouds. We are embracing openness with Spanner by adding a PostgreSQL interface, allowing enterprises to take advantage of Spanner’s unmatched global scale, 99.999% availability, and strong consistency using skills and tools from the popular PostgreSQL ecosystem. We are also automating data processing with Spark on Google Cloud, which enables developers to spend less time on infrastructure management and more time on data science, modeling, and delivering business value. Finally, we announced Google Earth Engine on Google Cloud, allowing customers to integrate Earth Engine with BigQuery, Google Cloud’s ML technologies, and the Google Maps Platform. 

With these innovations, enterprises like PayPal, Deutsche Bank, and Equifax use Google Cloud to solve for their end-to-end data lifecycle use cases. Organizations like Telefonica use Google Cloud to deliver new customer experiences. Telefónica has transformed every aspect of how they store, share, and analyze data while doubling processing power and lowering costs.

We continue to support an open ecosystem of data partners including Informatica, Tableau, MongoDB, Neo4j, C3.ai, and Databricks, giving customers the flexibility of choice to build their data clouds without being locked into a single approach. 

We are honored to be a Leader in the 2021 Gartner Magic Quadrant for Cloud Database Management Systems (DBMS), and look forward to continuing to innovate and partner with you on your digital transformation journey. 

Download the complimentary 2021 Gartner Magic Quadrant for Cloud Database Management Systems report.

Learn more about how organizations are building their data clouds with Google Cloud solutions. 

^{Gartner, Magic Quadrant for Cloud Database Management Systems, Henry Cook, Merv Adrian, Rick Greenwald, Adam Ronthal, Philip Russom, 14 December 2021.}

^{Gartner and Magic Quadrant are registered trademarks of Gartner, Inc. and/or its affiliates in the U.S. and internationally and are used herein with permission. All rights reserved.}

^{Gartner does not endorse any vendor, product or service depicted in its research publications and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s Research & Advisory organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.}

^{This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document. The Gartner document is available upon request from Google Cloud.}

Hundreds of millions of residents of South East Asia have only limited access to banking and finance services. However, help is at hand. One of South East Asia’s largest fintech businesses, Ascend Money is using digital technologies to realize its mission of enabling as many people as possible to access innovative financial services and live better lives.

According to Ascend Money, 60 percent of the region’s 620 million residents do not have access to a full suite of financial services. Even a relatively small proportion of this market represents a big opportunity for the business.

Ascend Money’s offerings include the TrueMoney regional payment platform for underserved and digital consumers. The TrueMoney platform supports more than 40 million consumers across six countries: Cambodia, Indonesia, Myanmar, the Philippines, Thailand, and Vietnam.

“We found, based on value, ease of use, effectiveness, and the skills and adaptability of our own team members, BigQuery and other Google Cloud Platform services were the best fit for our business.”

– Abraham Jarrett, Head of Engineering, Ascend Money

TrueMoney also provides an e-wallet app that provides easy ways to top up mobile phones, undertake online shopping, and pay for products and services; a network of 65,000 branded shops; and international remittances, initially between Thailand and Myanmar. In addition, Ascend Money offers financial products to small to medium businesses.

Ascend Money is part of the South East Asian online business Ascend Group, which also operates e-commerce, e-procurement, data centers, cloud services, fulfilment, and digital marketing services.

Cost is a key criteria

Ascend Money initially started operations using physical infrastructure and on-premises workforce productivity applications. However, as the business grew its customer base and expanded into new markets, its technology leaders began to explore options to improve value for money; reduce the maintenance load on in-house team members; improve its data management and analysis; and collaborate more effectively.

“Operating our own on-premises infrastructure entails a higher cost of ownership and maintenance, as well as requiring us to scale up our hardware as needed,” says Abraham Jarrett, Head of Engineering, Ascend Money. “We conducted an evaluation and found that moving to the cloud would deliver a range of benefits.”

G Suite and Google Cloud Platform the best fit

The business evaluated solutions available in the market and opted to move to G Suite and Google Cloud Platform.

“Moving to Google Cloud Platform was an optimization exercise, with lowering our costs our primary goal, and achieving better performance an added benefit,” says Jarrett. Google Cloud Platform monitoring, diagnostic, and analytics tools have enabled Ascend Money to reduce its infrastructure spending, becoming more efficient and cost-effective in the process.

“Managing software is a big cost for us. Through G Suite, we are significantly reducing our software deployment, licensing, and repair costs.”

– Abraham Jarrett, Head of Engineering, Ascend Money

The Ascend Money team saw Google Kubernetes Engine as enabling a seamless way of transitioning from an on-premises data center to a cloud service. “We considered industry trends such as what people were adopting and who we could hire when making our decision,” says Jarrett. “Engineering teams are focusing on using Kubernetes open source container management at scale and as a way of doing business, which was attractive to us.

“Google Kubernetes Engine presented the easiest way to manage and orchestrate our containers, and drive us away from our existing infrastructure.”

BigQuery best for cost and ease of use

Ascend Money also reviewed data infrastructure solutions, with its business intelligence and data platform teams considering a range of options. The teams quickly ruled out an on-premises solution due to the capital investment required and opted for a cloud service. Following a rigorous evaluation of Google Cloud Platform against the services offered by another cloud provider, Ascend Money opted to run on Google Cloud.

“We found, based on value, ease of use, effectiveness, and the skills and adaptability of our own team members, BigQuery and other Google Cloud Platform services were the best fit for our business,” says Jarrett. The business is now running an architecture comprising a BigQuery analytics data warehouse; Cloud Storage to store raw and archive data; Cloud Dataflow to process stream and batch data, and Cloud Pub/Sub for event ingestion and delivery.

“We’re expanding our utilization of BigQuery and other services on a daily basis,” says Jarrett.

Ascend Money is also using a range of Google Cloud Platform services for the infrastructure outside its data platform, including Stackdriver logging and monitoring; Cloud KMS to manage cryptographic keys for its cloud services, Cloud Build to undertake continuous integration, delivery, and deployment; Cloud DNS to provide domain name system services; and Cloud Functions to enable its developers to run and scale code in the cloud.

With Google Cloud Platform, Ascend Money is now processing about 12GB of batch data per day and streaming data from about 200 data marts per day in Thailand alone.

Cost effective scalability and faster to market

Google Kubernetes Engine has enabled the business to scale and deliver to market faster. “We can build on the platform, take the application and container and deploy them to scale out,” says Jarrett. “The Google Kubernetes Engine orchestration mechanism that provides containerized, elastic scalability is extremely important in allowing us to manage costs and expend our effort efficiently. Through Kubernetes, we can write once and deploy everywhere.”

With Google Cloud Platform, the business has saved 3,000,000 THB (about US$90,000) in licensing costs and, through automation tools, reduced the time to complete infrastructure activities by 50 percent.

“We don’t have as many meetings since we deployed G Suite and we can work effectively in a distributed fashion.”

– Abraham Jarrett, Head of Engineering, Ascend Money

Ascend Money’s entire workforce is now using G Suite to collaborate and operate productively. The business is achieving a range of benefits including being able to get new team members up and running more quickly and reduced cost. “Managing software is a big cost for us,” says Jarrett. “Through G Suite, we are significantly reducing our software deployment, licensing, and repair costs.” Meanwhile, the shorter time to onboard team members to G Suite is paying off with improved productivity and an accelerated ability to collaborate.

Ascend Money team members primarily use Sheets, Slides, and Docs to create internal materials, including product requirement documents in Docs and workforce and project planning spreadsheets in Sheets.

The organization also uses Hangouts Meet to collaborate when working from different locations. “We don’t have as many meetings since we deployed G Suite and we can work effectively in a distributed fashion,” says Jarrett. “People can work from home and we have five regions plus Thailand where they can collaborate live on a document, such as a slide deck for board meetings or meetings with our chief executive officer, head of engineering, or other senior managers.”

20 hours per week saved

Jarrett describes the time savings of using web browser-based productivity software and G Suite as saving them an average of 20 hours per week. This has improved productivity and the quality of life for Ascend Money’s hard-working team, increasing overall staff satisfaction.

“The ability to work remotely is a big win for us,” Jarrett says. “Our team members can keep one computer at work and one at home and access the same information, they can work in a plane while it sits on the tarmac, or update a Google Doc or a Google Sheet in real time on their phone. They can work anytime, from any location, as long as they have an internet connection. They have less dead time, meaning more uptime.”

With Google Cloud Platform and G Suite, Ascend Money is ideally positioned to support growing demand for its fintech services in South East Asia, particularly among people with limited access to banking. “We have the agility and dynamism to support rising demand and look forward to continuing to work with Google to realize our business ambitions,” says Jarrett.