With access to more data than ever before, companies have never been better positioned to adopt precision marketing methods and target the right customers at the right time. Emarsys, a digital marketing platform, enables its clients to collect, analyze, and act on a wide variety of data. From websites to mobile apps to emails, Emarsys’ customers can handle data from all its digital channels on a single, easy-to-use platform. Emarsys also makes sure that customers receive the highest quality data possible, making for smarter decisions and better business practices.

“We were close to the limits of our internal data warehouse, scalability-wise. We didn’t want to get to the point where we’d have to delete data or cancel projects. In Google Cloud we saw a platform that could scale with our ambitions and be optimized for AI and real-time solutions.”

—Levente Otti, Head of Data, Emarsys

Since launching as an email solutions provider in 2000, Emarsys has grown into the world’s largest independent digital marketing platform, with more than 2,500 clients worldwide and reaching more than 1.4 billion people. By 2016, the company felt that its existing data warehouse platform was close to its limits, affecting not just day-to-day operations but also important strategic goals.

“We were close to the limits of our internal data warehouse, scalability-wise. We didn’t want to get to the point where we’d have to delete data or cancel projects,” says Levente Otti, Head of Data at Emarsys. “In Google Cloud, we saw a platform that could scale with our ambitions and be optimized for AI and real-time solutions.”

Minimal maintenance, unlimited scale with Google Cloud

Digital marketing is a highly competitive environment. Emarsys works alongside big players with a huge market share on the one hand and smaller, specialist companies on the other. It has thrived by successfully combining the all-inclusive offerings of the former with the agility of the latter, constantly looking for ways to innovate and improve. In recent years, the company had started to feel that the ability to handle large quantities of data was no longer enough. The next challenge was speed. “We truly believe that in the future, everything will be done in real time, including data processing, analytics, and AI predictive models,” Levente says.

At the start of 2016, Emarsys’ existing data warehouse was a software-as-a-service solution running on-premises, which required hardware and software maintenance in order to keep up with the company’s growing appetite for data-heavy use cases such as prediction and analytics. The existing platform had proven its worth processing large amounts of data in batches, but its real-time capabilities were limited. Moreover, Emarsys had begun to experiment with AI technology, but found that its data warehouse couldn’t scale to accommodate some of the more resource-intensive processes, such as training the predictive models. The company decided that it needed a new, cloud-based data platform.

After evaluating some of the leading cloud providers, Emarsys chose Google Cloud for its mature AI capabilities and its ease of use. “With the other solutions, we still had to rent virtual machines and hardware and be responsible for maintenance. At the time, Google Cloud was the only provider that could take that management overhead away from us, while keeping customers accounted for on every query level,” Levente says.

To implement its new data platform, Emarsys teamed up with Google Cloud Partner Aliz. Over a series of meetings, workshops, and architecture reviews, Aliz helped Emarsys navigate the Google Cloud ecosystem to find the right products for the solution it was looking for. “Aliz really helped us set off in the right direction,” explains Levente.

With Google BigQuery, we can run queries which process terabytes of data, in seconds. We can also develop our own user-defined functions, incorporating Bayesian statistics into our predictive algorithms. That means we can take into account historical data, resulting in much more accurate predictions in a scalable way within seconds.”

—Levente Otti, Head of Data, Emarsys

Emarsys’ new data platform would actually be two: one platform for batch processing data and one for real-time analysis and interactions. Firstly, a proprietary publishing component gathered all the data points from Emarsys’ various channels including the website, mobile, emails, and custom events. With Cloud Pub/Sub and Cloud Dataflow, Emarsys transported and processed the data into BigQuery, which allows for further work and reviews that take into account errors or delayed events. After this, the data was exported to the main batch processing platform, which ran on BigQuery. For the real-time analytics, Emarsys used Cloud Bigtable to access data and Cloud Dataflow to pipeline it into the real-time platform, which could communicate with AI components or interaction components via an API to deliver real-time interactions with customers.

On top of the overall data infrastructure, Emarsys built a new AI platform with Google Cloud components. Training the predictive models had been an issue in the past due to the large number of resources required, so Emarsys chose to use Google Kubernetes Engine clusters, which can scale up and down on demand, without the need for hardware configuration or management. The trained models were held securely in Cloud Storage. From here, they were integrated with BigQuery for power and flexibility, allowing Emarsys to improve not just the speed of its AI predictions but also the quality.

“With Google BigQuery, we can run queries which process terabytes of data, in seconds,” shares Levente. “We can also develop our own user-defined functions incorporating Bayesian statistics into our predictive algorithms. That means we can take into account historical data, resulting in much more accurate predictions in a scalable way within seconds.”

Real-time insight, long-term satisfaction

Google Cloud enabled Emarsys to build a scalable data and AI platform that delivers powerful, actionable insights in real time. According to Levente, the company wanted to spend less time managing overload and more time considering how it should handle data. An immediate result of the new platform has been that data is now available in a scalable way, without hardware additions and management.

“With Google Cloud, we’ve been able to build a truly real-time data platform. The norm used to be daily batch processing of data. Now, if an event happens, marketing actions can be executed within seconds, and customers can react immediately. That makes us very competitive in our market.”

—Levente Otti, Head of Data, Emarsys

The clear and innovative pricing schemes of Google Cloud have also brought a new level of accountability to Emarsys’ costs in a way that wasn’t possible with its on-premises infrastructure. “Now that we only pay for what we use, we can assign costs to specific customers or queries, which has a huge impact on our pricing and product development strategies,” says Levente.

Thanks to the power of BigQuery and the scale at which it can handle data, Emarsys can now apply its analytics and AI tools to their full potential. “It’s very important to enable our clients to create the best possible experience for customers,” says Levente. At the same time, the company has cut its AI platform costs by 70% with Kubernetes while increasing scalability compared to the previous solution. The whole data platform was built to be scalable, and its first big test came during the retail peak of Black Friday, when it comfortably handled 250,000 events per second. “Perhaps the biggest impact on the business came with the real-time nature of the new platform,” says Levente.

“With Google Cloud, we’ve been able to build a truly real-time data platform,” he explains. “The norm used to be daily batch processing of data. Now, if an event happens, marketing actions can be executed within seconds, and customers can react immediately. That makes us very competitive in our market.”

Since implementing the new platform, Emarsys has continued to innovate with it and is about to release a new Real-Time Decision Framework, which will provide customers with even more real-time products and tools. The company continues to work with Aliz and Google Cloud, exploring other products such as Google BigQuery ML and TensorFlow to improve its AI processes. “We had a problem that we wanted to tackle now, and for us, Google Cloud was the best way of doing that,” says Levente. “But it was also about looking ahead. We felt that Google Cloud offered us the best way of future-proofing our platform.”

2020 saw an unprecedented change in consumer behaviour around the world, with shoppers finding new ways of discovering, evaluating and buying products. This has created fresh expectations for both brands and retailers to drive consumer closeness, embrace the digital moment and transform their operations to be more agile and sustainable. These themes have been top of mind in my conversations with our CPG customers, and set the tone at Google Cloud Summit for Retail and Consumer Goods, which concluded this week. I couldn’t be more proud of our team and thankful for our customers that supported us by participating in our sessions and attending the event.

I joined Google Cloud in January 2021 after a long career in the CPG industry, and as I shared in my CPG keynote at the summit, I am thrilled to be helping bring the power of Google to drive industry innovation in CPG. At Google we call this ‘new normal’ the transformation cloud era, where we’re working with customers who want to not just save money on storage or compute, but to use cloud and digital technologies to drive agility across their business. I also call it the era of ‘consumer switching’, because Covid-19 has accelerated the likelihood of consumers to switch brands or the way they shop. Some of these changes are still happening; over 40% shoppers in a recent Google survey reported that in March 2021 they changed brands or shopped online for something they were previously buying in store.

At Google, we have an amazing team of strategists who have been researching, observing, and analyzing the many facets of consumer behavior over the last few years. In the opening keynote at the Summit Capturing the Hearts and Minds of Today’s Consumers, Google’s Human Truths Team kicked off the Retail & Consumer Goods summit by sharing some of their consumer insights, including what behavior patterns they think will “stick” as we move into a post-pandemic world.  I think these insights are especially relevant for brands, as they speak to some of our latest findings on the CPG shopper’s mindset.

Accenture estimates that there could be a 3 trillion dollar shift in value between companies as a result of consumers shifting brands and behaviours. While it is not known who the winners of the shift will be, one thing is certain – those who will be able to leverage data and analytics fastest will benefit the most from these times of rapid change. I shared some of the implications for the CPG industry in my keynote How to grow brands in times of rapid change along with the three key areas in which Google cloud is helping CPG companies drive brand success: 

1. Unlocking consumer growth with data powered insights 
2. Transforming go-to-market in the omnichannel ecosystem
3. Driving connected, efficient, and sustainable operations

Let’s take a quick look at each of them:

Unlocking consumer growth with data-powered insights

The digital marketing ecosystem is transforming for a privacy centric world, and brands are seeing a direct impact in marketing effectiveness. As the CPG industry becomes more consumer-centric and shifts more toward direct-to-consumer (D2C) business models, acquiring and activating consented first-party consumer data presents a clear opportunity to capitalize on new consumer demands. 

As a result, CPGs are turning to consumer data platforms (CDPs) to help them unify, manage, enrich, and secure all of their disparate data from different marketing tools, website analytics, email campaigns, loyalty programs, and more. Google Cloud and our ecosystem of partners can help CPGs build a privacy-centric CDP which brings together all their customer and marketing data into a modern data warehouse, with built-in predictive data visualization tools and models. With a CDP built on Google Cloud, you can integrate data from Google Marketing Platform to drive predictive marketing and media effectiveness. And with pre-built connectors you can also easily integrate non-Google media and data from other enterprise platforms like SAP and Oracle to leverage consumer data for more integrated decision making. Democratize access to data across the organization with our Business Intelligence tool Looker to enable faster decisions in real-time from marketing to supply chain to product innovation.

At the Retail & CPG Summit we shared how retailers and brands can drive consumer closeness in a privacy-centric world  featuring Procter & Gamble’s experience building and activating consumer data in a privacy-safe way to serve consumers better and maximize marketing effectiveness and drive growth across their business. And in a demo, Constellation Brands shared how they’re leveraging real-time data from several commercial sources using Looker to unpack insights and develop action plans.

Transforming go-to-market in the omnichannel ecosystem

Amidst COVID-19 restrictions and rolling lockdowns, ecommerce activity has surged past a point of no return. Direct to consumer (D2C)  or digitally native brands were able to minimize consumers switching during the pandemic  by offering a great online experience. While in-person shopping remains important, consumers are expecting more digital and omnichannel experiences and many will continue to explore new brands and purchase online. CPGs that want to maintain their market leadership can no longer afford to ignore the key role omnichannel capabilities will play in capturing the attention of both retailers and consumers.  Even if D2C sales are not a big portion of your business, you will benefit from first-party data that can help you drive insights and product innovation. 

CPG brands want to transform their older, clunky ordering processes into modern digital shopping experiences. Re-platforming legacy solutions on Google Cloud not only accelerates application innovation, but also makes it easier to quickly launch new features and products. At Google Cloud we have transformed ecommerce for large D2C brands and traditional retailers, so we know what a best in class D2C experience looks like. And we can bring it to your brands. We already help some of the biggest brands in retail modernize ecommerce and enhance product discovery with solutions like Visual Search and Recommendations AI. All of these can help you build a best in class omni channel presence for your brands.

We had several sessions on improving your omni channel experience, including 

Why search abandonment is the metric that matters featuring Macy’s and Conversational Commerce with Google with Albertsons, where we shared how Google’s conversational experiences can help consumers message businesses from wherever they are, and whenever they need them.

Driving connected, efficient, and sustainable operations

The superpowers of  AI/ML are not just for marketing.  Did you know that research from MIT and Google Cloud has found companies that use AI/ML can drive 2x more data-driven decisions, 5x faster decision making, and 3x faster execution? By connecting your operations in real time with demand signals like search, trends, weather, mobility and supercharging this data with AI and ML, you can make smarter and quicker business decisions. For example, you can use search trends to drive demand forecasting and ramp up manufacturing for popular products.

Google Cloud can help you modernize legacy business applications by migrating them to the cloud and using AI/ML and smart analytics to drive business outcomes. Take SAP for example – a Forrester study found that modernizing SAP with Google not only resulted in 56% more efficient IT teams – it also generated 160% 3-year ROI. 

SAP data on Google Cloud breaks down silos across SAP, marketing, manufacturing systems, and external data sources for next-level intelligent operations. For example, with SAP and Google Cloud, you can combine product, media, CRM, digital commerce and site data from SAP and non-SAP sources to uncover stronger consumer insights and fuel product discovery along the path to purchase. You can merge SAP product and sales data with consumer,  market data and Google geo trends to drive targeted promotion outcomes, maximizing the ROI of promotional dollars across retail channels. You can also integrate supply chain and manufacturing data from SAP systems with consumer, marketing and Google geo market data to improve demand forecasting and optimize supply chain logistics. The possibilities are endless. This is why we describe SAP modernization as The Gift That Keeps On Giving. Check out the session on SAP from the Summit and hear from Rodan + Fields on their experience of modernizing SAP on Google Cloud. 

Another solution that excites me is Vertex AI, which transforms the demand forecasting process. Traditional demand forecasting accuracy is a challenge for most CPGs. Current forecasting methods do not take into account granular factors that impact demand, like local weather, demographics, or unforeseen events. With our recently launched Vertex Forecast, Google is making it much easier to start using cutting-edge machine learning models for demand forecasting. In our session Demand Forecasting: Time for Intelligence, Not Intuition featuring American Eagle Outfitters, we share how you can adopt a data science approach to demand forecasting that’s customized to your unique needs.

CPG organizations come to Google for help solving their toughest problems, whether it be driving new consumer growth, unlocking new routes to market, or building connected, sustainable operations. And we bring the best of Google: innovation, culture, infrastructure, AI/ML, and a deep understanding of consumer behaviour to help them build best-in-class brands. 

I’d like to end with a topic that’s very close to my heart. This is around Solving for Sustainability in Retail and Consumer Goods. Our research shows that 62% of shoppers cared about at least one sustainability aspect when purchasing online in 2020. In addition, the events of the past year have triggered consumers to re-evaluate their relationships with brands and prioritize those that are more sustainable in the context of the pandemic. Watch this session to learn more about how retailers and consumer goods companies can leverage technology, data, and machine learning to help make sustainability a core part of the recovery.

All our session content is available on demand. Ready to learn more about how we’re helping CPG brands and manufacturers drive results? Learn more about Google Cloud’s consumer packaged good solutions and reach out to your Google Cloud sales executive to set up a deeper conversation on how we can help you grow your brands today and in the future.

Visualizing trends in your logs is critical when troubleshooting an issue with your application. Using the histogram in Logs Explorer, you can quickly visualize log volumes over time to help spot anomalies, detect when errors started and see a breakdown of log volumes. But static visualizations are not as helpful as having more options for customization during your investigations. 

That’s why we’re excited to announce that we recently added three new query controls along with separate colors for log severity to the histogram. These new features make it even easier to refine and analyze your logs by time range. The new histogram controls help find logs before or after the current period, jump to a specific time range represented in a histogram bar and zoom in/out of the current time window in the histogram.

Histogram colors

The histogram now makes it easier to view the breakdown of logs by severity with the introduction of color coding. For example, the severity colors make it easy to spot an increasing number of errors even when the volume of requests is relatively constant. Looking at the histogram below, the red vs blue shading makes it clear that there has been an increase in overall log volume and provides a visual breakdown of errors within that log volume.

Pan left/right to scroll through time

Sometimes in your troubleshooting journey, you may want to look at the logs directly before or after the current set of logs. Perhaps there was an unexpected spike in errors at the beginning of the time range and you need to see the logs in the time period directly preceding the current time range. Pressing the left arrow on the left side of the histogram shifts the time range earlier while the arrow on the right side of the histogram shifts the time range ahead. Either arrow will refine the time range in the query and rerun the query to return the logs in the new time range.

Zooming in or out 

Zooming in or out from a given time range may be useful to visualize fine-grained details or a broader trend Clicking the zoom in or out icons in the upper right corner of the histogram refines the time range in the query and then reruns the query, returning the logs in the newly defined time range.

Scrolling to time 

If you see a large spike in logs volume in the histogram, it’s useful to quickly review the logs generated during that spike. Clicking on the histogram bar that contains the spike now scrolls you to the logs generated during that time period.

Where to find the histogram 

The histogram is a panel in Logs Explorer that can be displayed or hidden using the controls in the Page Layout menu. When you no longer want to display the histogram, click the “X” button in the upper right corner to quickly close it. To open it again, use the same Page Layout menu to enable the histogram display.

Get started with the histogram

These improvements move the histogram from a utility for visualization to an integral part of the troubleshooting journey. We are continuously working to launch new features that make Cloud Logging the best place to troubleshoot your Google Cloud logs. If you are not already a Cloud Logging user, review this getting started documentation or watch a quick video on troubleshooting services on Google Kubernetes Engine (GKE) to learn more. If you have specific questions or feedback, please join the discussion on our Google Cloud Community, Cloud Operations page.

If you’re managing a crucial application that has to be fully fault-tolerant, you need your system to be able to handle every fault, no matter the type and scope of failure, with minimal downtime and data loss. Protecting against these faults means juggling numerous variables that can impact performance as well as recovery time and cost.

Today’s managed database services take over the operational complexity that used to exist for database administrators. Growing your organization’s tolerance required adding machines, compute, and storage, plus the operational costs of IT management: performing backups, writing scripts, creating dashboards, and carrying out testing to make sure your platform is ready when problems arise–all in a secure way.

At Google, our Cloud SQL managed database service offers three fault tolerance mechanisms —backup, high availability, and replication—and there are three major factors to consider for each of them:

• RTO (recovery time objective): When a failure happens, how much time can be lost before significant harm occurs?
• RPO (recovery point objective): When a failure happens, how much data can be lost before significant harm occurs?
• Cost: How cost-effective is this solution?  

We’ve heard from customers like Major League Baseball, HSBC, and Equifax that they have strict data-protection needs and require highly fault-tolerant multi-region applications—and they’ve all chosen Cloud SQL to meet those needs. 

Let’s take a closer look at how the decision-making process plays out for each recovery solution.  

High availability (HA)

If your application is business critical, you require minimum RTO and zero RPO— a high availability configuration ensures that you and your customers are protected. If the primary instance fails, there’s another standby instance ready to take over with no data loss. There’s an additional cost here, but doing this manually brings a great operational cost, since you have to detect and verify the fault, do the failover, and make sure it’s correct—you can’t have two primary instances or you risk data corruption—then finally connect the application to the new database.

Cloud SQL removes all that complexity. Choose high availability for a given instance and we’ll replicate the data across multiple zones, synchronously, to each zone’s persistent disk. If an HA instance has a failure, you don’t have to think about when to fail over because Cloud SQL detects the failure and automatically initiates failover, for a full recovery and no data loss within minutes. Cloud SQL also moves the IP address during failover so your application can easily reconnect. MLB, for example, uses Cloud SQL high availability to serve prediction data to live games with minimal downtime. Dev/test instances don’t need those same guarantees, but can use local backups to recover from any potential failure.

Cross-region replica

If a whole Google Cloud region goes down you still need your business to continue to run. That’s where cross-region replication comes in, a hot standby replica in another Google Cloud region provides RTO of minutes and RPO typically less than a minute . If you create a read replica in a region separate from your primary instance and you get hit with a regional outage, your application and database can start serving customers from another region within minutes. But this solution can be complex and enabling it yourself can be difficult and time-consuming. Securing cross-geography traffic demands end-to-end encryption and can bring connectivity issues too. 

This is where the fully managed Cloud SQL solution shines. We offer MySQL, PostgreSQL and SQL Server database engines as a cross-region replication solution that’s easily configured and bolstered by Google’s interconnected global network. Just say, “I’m in U.S. East, I want to create a replica in U.S. West,” and it’s done, reliably and securely.

Backup

When you suffer data loss because of an operations error (for example, a bug in a script dropped your tables) or human error (for example, someone dropped the wrong table by accident), backups help you restore lost data to your Cloud SQL instance. Our low cost backup mechanism features point-in-time, granular recovery, meaning that if you accidentally delete data or something else goes wrong, you can ask for recovery of, for example, the state of that database down to the millisecond, such as Monday at 12:53pm. Your valuable data is replicated multiple times in multiple geographic locations automatically. This enables the automatic handling of failover in cases of major failure. You can always rest assured that your database is available and data is secure, even in the times of major failure crises. 

Cloud SQL provides automated and on-demand backups. With automated backups, Google manages the backups so that you can easily restore them when required. Also, the scheduled backing is automatically taken by default. With on-demand backup, you can create a backup at any time. This could be useful if you are about to perform a risky operation on your database, as Cloud SQL lets you select a custom location for your backup data. When the backup is stored in multiple regions, and there’s an outage in the region that contains the source instance, you can restore a backup to a new or existing instance in a different region. This is also useful if your organization needs to comply with data residency regulations that require you to keep your backups within a specific geographic boundary.

Putting it all together

For critical workloads, MLB configures their Cloud SQL instances with backups, high availability, and cross-region replication. Doing so ensures they can recover from many failure types.

• To recover from human error (“Oops, I didn’t mean to delete that”), MLB uses backups and point-in-time recovery to recovery to a millisecond or specific database transaction
• To automatically recover from primary instance failures and zonal outages, MLB uses Cloud SQL’s high availability configuration
• To protect against regional outages, MLB uses cross-region replication

Creating a robust configuration, like MLB did, takes just a few minutes. Get started in our Console or review documentation.

Today we’re announcing the HarbourBridge Schema Assistant, which provides a guided schema-design workflow for migrating from MySQL or PostgreSQL to Spanner. HarbourBridge imports dump files (from mysqldump or pg_dump) or directly connects to your source database, and converts the source database schema to an equivalent Spanner schema. The new Schema Assistant capability displays the source schema and Spanner schema side-by-side, highlights errors and walks you through a series of steps to validate and optimize your Spanner schema. It also produces a browsable assessment report with an overall migration-fitness score for Spanner, a table-by-table detailed analysis of type mappings and a list of features used in the source database that aren’t supported by Spanner. It supports editing of table and column names, column types, primary keys and constraints, as well as dropping of tables, columns, foreign keys and secondary indexes.

The new Schema Assistant complements HarbourBridge’s existing data and schema migration capabilities and is a critical step towards our goal of building a complete open-source migration toolkit. HarbourBridge continues to support command-line schema and data migration and turn-key Spanner evaluation.

Complementing the bulk data migration capabilities of HarbourBridge, we are also announcing the ability to migrate change events from MySQL to Cloud Spanner.

Supported Features in Schema Assistant

1. Global type mapping. Users can customize the global mapping for how types should be mapped to Spanner consistently across the schema. For example, mapping large integers in source schema to Spanner’s NUMERIC.
2. Local type mapping. Users can override the custom type mapping for a given table/column.
3. Session management. A session keeps track of all the changes made to the schema mapping.
4. Customization of secondary indexes. Users can add, edit and delete secondary indexes to optimize their Spanner performance.
5. Customization of foreign keys and interleaved tables. Table interleaving is an important design consideration when migrating to Cloud Spanner as explained in more detail in this blog post.

Features in the pipeline

We are already working to further expand the supported set of schema editing features and welcome your feedback. We are particularly excited to expand the Schema Assistant’s design recommendations for optimizing Spanner schemas e.g. in-depth recommendations for primary key design.

HarbourBridge is open source and we gladly accept contributions from the wider community.