Architecting Data Pipelines Directly Improves Customer Experience at Universe.com

6092
Of your peers have already read this article.
3:30 Minutes
The most insightful time you'll spend today!
At Universe, we serve customers day and night and are always working to make sure they have a great experience, whether online or at one of our live events.
Our technology has to make that possible, and our legacy systems weren’t cutting it anymore. What we needed was a consistent, reliable infrastructure that would help our internal teams provide a fast and innovative ticket-buying experience to customers.
With our data well-managed, we could free up time for our developers to bring new web features to customers, like tailored add-ons at checkout.
Our team of about 20 software engineers needed more flexibility and agility in our infrastructure; we were using various data processing tools, and it wasn’t easy to share data across teams so that everyone saw the same information.
We also needed to incorporate streaming data into the data warehouse to ensure the consistency and integrity of data that’s read in a particular window of time from multiple sources. Our developer teams needed to be able to ship new features faster, and the data back ends were getting in the way.
In addition, when GDPR regulations went into effect, we needed to make sure all our data was anonymized, and we couldn’t do that with our legacy tools.
Finding the right data tools for the job
To make sure our customers were getting a top-notch online experience, we had to make the right technology choices.
Our first step was to centralize multiple data sources and create a single data warehouse that could serve as the foundation for all of our reporting requirements, both internal and external.
The new technology infrastructure we built had to let us move and analyze data easily, so our teams could focus on using that data and insights to better serve our customers.
Previously, we had lots of siloed systems and applications running in AWS. We did a trial using Redshift, but we needed more flexibility than it offered in how we loaded historical data into our cloud data warehouse. Though we were using MongoDB Atlas for our transactional database, it was important to continue using SQL for querying data.
The trial task that really sold us on BigQuery was when we wanted to alter a small table that had about 20 million rows, used for internal reporting.
We needed to add a value, but our PostgreSQL system wouldn’t allow it. Using Apache Beam, we set up a simple pipeline that moved data from the original source into BigQuery to see if we could add the column there.
BigQuery ingested the data and let us add the new value in seconds. That was a significant moment that led us to start looking at how we could build end-to-end solutions on Google Cloud. BigQuery gave us multiple options to load our historical data in batches and build powerful pipelines.
We also explored Google Cloud’s migration tools and data pipeline options. Once we saw how Cloud Dataflow worked, with its Apache Beam back-end, we never looked back. Google Cloud provided us with the data tools we needed to build our data infrastructure.
Cloud for data, and for users
Introducing new technologies isn’t always simple—companies sometimes avoid it altogether because it’s so hard. But our Google Cloud onboarding process has been easy.
It took us less than two months to fully deploy our BigQuery data warehouse using the Cloud Dataflow-Apache Beam combination.
Moving to Google Cloud brought us a lot of technology advantages, but it’s also been hugely helpful for our internal users and customers. The data analytics capabilities that we’re now able to offer users has really impressed our internal teams, including developers and DevOps, even those who haven’t used this type of technology before.
Some internal clients are already entirely self-service. We’ve hosted frequent demos, and also hosted some “hack days,” where we share knowledge with our internal teams to show them what’s possible.
We quickly found that BigQuery helped us solve scale and speed problems.
One of our main pain points had been adding upsell opportunities for customers during the checkout process. The legacy technology hadn’t allowed us to quickly reflect those changes in the data warehouse. With BigQuery, we’re able to do that, and devote fewer resources to making it happen. We’ve also eliminated the time we were spending tuning memory and availability, since BigQuery handles that. Database administration and tuning required specialized knowledge and experience and took up time. With BigQuery, we don’t have to worry about configuring that hardware and software. It just works.
We’ve also eliminated the time we were spending tuning memory and availability, since BigQuery handles that. Database administration and tuning required specialized knowledge and experience and took up time. With BigQuery, we don’t have to worry about configuring that hardware and software. It just works.
Two features in particular that we implemented using BigQuery have helped us improve the performance of our core transactional database. First, using Cloud Dataflow to convert raw MongoDB logs to structured rows under a BigQuery table, which we can then query using SQL to identify slow or underperforming queries. Second, we can now query multiple logging tables using wildcards, since we load Fastly logs to BigQuery.
Along with MongoDB Atlas as our main transactional database, much of our infrastructure now runs as Google Cloud microservices using Google Kubernetes Engine (GKE), including the home page and our payment system. Kubernetes cron jobs power background scheduled jobs, and we also use Cloud Pub/Sub. Cloud Storage handles any data storage if any space constraints emerge.
Our overall performance has increased by about 10x with BigQuery. Both our customers and internal clients, like our sales and finance teams, benefit from the new low-latency reporting. Reports that used to be weekly or monthly are now available in near-real time. It’s not only faster to read records, but faster to move the data, too.
We have Cloud Dataflow pipelines that write to multiple places, and the speed of moving and processing data is incredibly helpful. We stream in financial data using Cloud Dataflow in streaming mode, and plan to have different streaming pipelines as we grow. We have several batch pipelines that run every day. We can move terabytes of data without performance issues, and process more than 100,000 rows of data per second from the underlying database. It used to take us a month to move that volume of data into our data warehouse. With BigQuery, it takes two days.
We’re also enjoying how easy and productive these tools are. They make our life as software engineers easier, so we can focus on the problem at hand, not fighting with our tools.
What’s next for Universe
Our team will continue to push even more into Google Cloud’s data platform. We have plans to explore Cloud Datastore next. We’re also moving our databases to PostgreSQL on GCP, using Cloud Dataflow and Beam. BigQuery’s machine learning tools may also come into play as Universe’s cloud journey evolves, so we can start doing predictive analytics based on our data. We’re looking forward to gaining even more speed and agility to meet our business goals and customer needs.
An Expert’s Opinion on What Early-stage Startups Must Know

6447
Of your peers have already read this article.
3:00 Minutes
The most insightful time you'll spend today!
As lead for analytics and AI solutions at Google Cloud, my team works with startups building on Google Cloud. This puts us in the fortunate position to learn from founders and engineers about how early-stage startups’ investments can either constrain them or position them for success, even at the seed level. In this post, I want to share a few of the best practices to keep in mind as you’re building.
Understand your value proposition before diving into a technology stack
If you’re launching a startup in the cloud, you’re no doubt thinking about a technology stack, but it’s important to step back a bit and think carefully about the major value proposition that your startup offers to your customers. That value proposition is going to fundamentally drive the kind of technology that you should pick.
For example, does your system need processing in real time, or can it be done in a batch mode? Can you rely on once-a-day insights or do the insights have to come in as events happen?
Additionally, what kind of latency will your customers face? That latency makes your value proposition either usable or unusable. Early on in Google’s development, leaders realized that no one was going to wait more than a few hundred milliseconds for a web page to show them their results, and that realization drove the technology decisions that have allowed Google to scale from being a startup in a garage to being a trillion dollar company. Your startup needs to define its value to customers with this level of specificity before it can build a technology stack suited to its needs.
Focus on customer interactions
A few companies have gracefully pulled off big IT pivots that reshaped their value proposition. Netflix, for example, moved from mostly sending DVDs through the mail to becoming a streaming service and major content producer. That’s a huge shift in the user experience and the technology stack necessary to support it, even if the underlying value proposition (i.e., get content to customers) was broadly the same. But it’s also an outlier. If you’re planning for potential changes of this magnitude, rather than focused on getting your value proposition to users, you probably need to sharpen what that value proposition is.
Specifically, you need a clear vision of how customers will access and interact with your business. Typically, they’ll do so over a website or a mobile app, but there are still so many variables.
Are customers going to transmit documents? If so, in what format? Is handwriting supported or is input limited to typing? Can they use images for optical character recognition? Will it mostly be forms? Will the data be structured or unstructured? If all that sounds a little overwhelming, don’t worry, it’ll seem simpler by the end of this article—but also be aware: we’re just getting warmed up.
Imagine that most of your customers will access your business via voice, so you know you’ll want to prioritize conversational workflows. That’s a start—but dig deeper. Even if we suppose you’re usingDialogflow, a Google Cloud conversational AI platform that lets you build and deploy virtual agents, we’re still not really seeing the value proposition. How will all this work, from the beginning of a typical full customer interaction to the resolution? How many interactions will have to be facilitated over low-bandwidth connections, for example? When it comes to user interactions, make sure you can see an end-to-end use case.
Another example: you’re building a retail website, and one of your end-to-end use cases involves the customer asking if a certain amount of a given product is in stock, whether it’s one unit of the product, ten or hundreds. If the product is not sufficiently stocked, you want your app to offer similar items that are. Will your technology stack support this end-to-end use case?
These considerations are not an argument for premature optimization. There’s value in moving fast, getting minimum viable products to users, and then iterating. But in the early stages, you only get one chance to start on the right foot—and how you navigate that chance will influence a lot of dollars and effort down the road. You need to make sure you have business use cases, not just an idea, before you can start designing a technology stack.
Here’s how to get in the right frame of mind. Pick three use cases: two that are “bread and butter” and one that is technologically complex. Make sure your proposed technology stack can support all three, end to end.
Default toward higher levels of abstraction
Now that we’re in the right frame of mind, we’re ready to think about the technology stack more directly.
As a startup, you’ll need to conserve resources, and to do that, you’ll want to build at the highest level of abstraction possible for your value proposition. For example, you probably don’t want your people setting up clusters. You don’t want them configuring things if they can use a fully managed service. You want them focused on building your prototype, not managing infrastructure.

This focus has definitely informed how we create products at Google Cloud, as our canonical data stack—Pub/Sub, Dataflow, BigQuery, and Vertex AI—consists of auto-scaling and serverless products.
But management of infrastructure is not the only place where you should err toward a less-is-more philosophy.
When it comes to architecture, choose no-code over low-code and low-code over writing custom code. For example, rather than writing ETL pipelines to transform the data you need before you land it into BigQuery, you could use pre-built connectors to directly land the raw data into BigQuery. That’s no code right there. Then, transform the data into the form you need using SQL views directly in the data warehouse. This is called ELT, and it is low code. You will be a lot more agile if you choose an ELT approach over an ETL approach.
Another place is when you choose your ML modeling framework. Don’t start with custom TensorFlow models. Start with AutoML. That’s no-code. You can invoke AutoML directly from BigQuery, avoiding the need to build complex data and ML pipelines. If necessary, move on to pre-built models from TensorFlow Hub, HuggingFace, etc. That’s low-code. Build your own custom ML models only as a last resort.

Focus on getting your vision to market, not chasing technology hype
The goal is to pick the right technology stack for bringing your vision to market, generating value for customers, conserving resources, and maintaining flexibility for growth. Early IT investments should usually gravitate toward things that preserve flexibility, such as managed services built on standard protocols or open APIs, but they needn’t always rush to the flashiest technologies. The answer isn’t always ML, for example. The answer might be heuristics to start, with a path to ML once you have collected enough data. You want to make sure that your intelligence layer has enough abstraction so you can mark it up with simple rules at first, but then replace it with a more robust system as you go along.
Launch and iterate fast with these principles
The preceding discussion is a reminder that your most expensive resource is your people—and that you really want them to be focused on building your prototype, minimum viable product or production app You want to launch fast and iterate fast, and the only way you can do that is by focusing on the things that differentiate you.
But regardless of the technologies you use, the bottom line is the same: follow these four principles.
- Figure out your major value proposition and design your tech stack around it.
- Be very careful about user interactions. User experience is super important; you need to make sure you deliver the kind of experience that your customers have grown to expect.
- When you’re building, pick the highest possible level of abstraction possible—the most fully managed tools and no-code/low-code frameworks that give you the functionality that you need.
- Instead of choosing new or flashy technologies, consider if you can build a “good enough” minimum viable product quickly and come back to a better implementation later.
To learn more about why startups are choosing Google Cloud, click here.
2842
Of your peers have already watched this video.
31:00 Minutes
The most insightful time you'll spend today!
Tools to Simplify Streaming Analytics
Streaming analytics is transforming how companies interact with their customers and operate with agility.
Dataflow supports a number of features to guide you through your journey to adopt streaming, including notebooks for writing your first pipeline, SQL to accelerate your streaming deployments, Flex Templates to scale streaming in your organization, and enhanced monitoring for operating your pipelines at scale.
In this video Mehran Nazir, Product Manager, Google Cloud, introduces Beam and Dataflow, and other tools, and how to start, how to scale and how to optimize. See how these capabilities can set you up for streaming success.

6697
Of your peers have already downloaded this article
4:30 Minutes
The most insightful time you'll spend today!
Unlocking the Power of Data and Creativity with
the Cloud: The WPP Story
The Real Drivers of Efficiency, Growth, and Customer Experience

5132
Of your peers have already read this article.
4:30 Minutes
The most insightful time you'll spend today!
The increasing adoption of technologies like connected devices, augmented reality, and machine learning has changed the way we shop, and retailers are evolving how they do business to meet the needs of their customers.
Retailers say it’s no longer enough to keep pace with shoppers’ growing expectations—they must get ahead of them. That’s why more and more are turning to the cloud. They’re using it to eliminate data silos and take advantage of cloud-based analytics. They’re tapping into machine learning to improve all aspects of the value chain. And they’re making use of reliable and secure cloud infrastructure to scale their businesses.
Although every retail customer is different, many of them share similar objectives. Here are three major ways retailers take advantage of the cloud.
Storing and Analyzing Data in the Cloud
Data presents both a challenge and an opportunity for retailers. Which is why Ulta Beauty, the largest beauty retailer in the US, is moving to Google Cloud Platform (GCP). Now, with the help of BigQuery, Ulta Beauty will be able to more efficiently predict and analyze outcomes and develop more meaningful data insights that can be leveraged to deliver a more personalized, relevant guest journey.
They are not alone. DSW has also chosen to use GCP to help relaunch their DSW VIP loyalty program for the first time in over 10 years. With more than 90% of transactions running through their loyalty program, DSW needed a flexible and scalable solution to deliver a real-time loyalty program for 26 million active members. They’ve already seen a 9% uptick in new customers and have improved their already strong retention rate.
Improving Customer Experiences with AI and Machine Learning
Once retailers are able to access these insights, they are turning to AI to help personalize the overall shopping experience. At first, retail companies leveraged AI tools such as machine learning for product recommendations.
Now, more retailers use AI to forecast trends, predict inventory needs and prevent
Just look at METRO AG, one of the largest B2B wholesalers globally. They’re using AI and machine learning to better serve their customers. For example, many of their customers are restaurant owners. With Google Cloud AI capabilities, they can create tools that identify when a restaurant is out of a particular ingredient and automatically order more.
Ocado is another great example. The world’s largest online-only grocery retailer drove a 3.5% increase in contact center efficiency by using Google Cloud machine learning technology to respond to customer emails four times faster.
To help businesses further accelerate their AI solutions, Google has developed the Advanced Solutions Lab (ASL), which gives businesses the opportunity to work side-by-side with Google’s AI and ML experts to solve high impact challenges.
Fast Retailing, the Japanese retailer behind Uniqlo, is working with Google Cloud and ASL to help them better analyze customer data to forecast demand and deeply understand what their customers want.
Carrefour, one of the world’s leading retailers, also announced last year that its engineers will be working side-by-side with our AI experts to co-create new consumer experiences. This is in addition to deploying G Suite to their employees to support the company’s digital transformation.
Scaling Infrastructure to Meet Demand
Of course, none of this innovation is possible without a reliable infrastructure that can scale instantly to meet surges in traffic.
And many have found the reliability and security they need with the cloud. That’s why global cosmetics brand Lush chose Google Cloud. They migrated their e-commerce platform to GCP to handle increased traffic without compromising stability.
This move that ultimately reduced infrastructure hosting costs by 40 percent.
L.L.Bean also modernized its IT infrastructure by moving capabilities from its on-premises systems to GCP, improving customer satisfaction and IT efficiency across multiple sales channels.
3130
Of your peers have already watched this video.
38:30 Minutes
The most insightful time you'll spend today!
Building Data Lakes on Google Cloud
A data lake is a centralized and a secure data platform for ingesting, storing, processing, and analyzing all your data–structured semi-structured and unstructured data–at scale.
“When we work with our customers we find out that the traditional approach for building data lakes, where you bring data from different sources and land them into a single central store has not worked. The traditional approach ended up creating swamps instead of lakes,” says Nitin Motgi, Product Manager, Google Cloud.
In this video, Motgi will talk about how to build data lakes correctly.
Google Cloud provides all the capabilities enterprises need to create and manage data lakes. Enterprises can use Google Cloud to aggregate their data and efficiently analyze it using cloud-native or open source tools irrespective of where the data is managed.
Also, learn how Google is making investments to simplify data lake deployment and management.
More Relevant Stories for Your Company

BURGER KING Germany: Serving Up Marketing Insights and Supply Chain Visibility Easily
Do hamburgers really come from Hamburg? This may still be a matter of debate, but the popularity of American-style burger joints not just in Hamburg but all over Germany, is clear. Germany’s top two fast food companies are both burger chains. One of them is BURGER KING®, a global brand that welcomes more

What’s BigQuery and Why Should IT Managers Care?
Today it’s impossible for anyone working in IT teams to have a day go by without a discussion about data, analytics, and big data. Often these conversations veer towards BigQuery. Many IT managers are unsure what BigQuery is and what it does. If you are one of them, you aren’t

How Google Cloud’s Scalable Data Storage and High Compute Resources Fuel Investment Research
Investment management is a heavily data-driven industry—portfolio managers and investment researchers require a large number of data sources to guide them in shaping their investment strategies. New cloud capabilities and technologies enable investment managers to process data faster than ever before and iterate on ideas quickly to fuel innovation in

Accelerating Insights with the Health Analytics Platform
The healthcare industry is beset with a data challenge, specifically clinical data, and challenges around the proliferation of EHR instances and ancillary clinical systems. In this video, Vivian Neilley, Solution Architect, Google Cloud and Marianne Slight, Product Manager, Google Cloud discuss how some of these challenges can be overcome with






