At Universe, we serve customers day and night and are always working to make sure they have a great experience, whether online or at one of our live events.

Our technology has to make that possible, and our legacy systems weren’t cutting it anymore. What we needed was a consistent, reliable infrastructure that would help our internal teams provide a fast and innovative ticket-buying experience to customers.

With our data well-managed, we could free up time for our developers to bring new web features to customers, like tailored add-ons at checkout.

Our team of about 20 software engineers needed more flexibility and agility in our infrastructure; we were using various data processing tools, and it wasn’t easy to share data across teams so that everyone saw the same information.

We also needed to incorporate streaming data into the data warehouse to ensure the consistency and integrity of data that’s read in a particular window of time from multiple sources. Our developer teams needed to be able to ship new features faster, and the data back ends were getting in the way.   

In addition, when GDPR regulations went into effect, we needed to make sure all our data was anonymized, and we couldn’t do that with our legacy tools.

Finding the right data tools for the job
To make sure our customers were getting a top-notch online experience, we had to make the right technology choices.

Our first step was to centralize multiple data sources and create a single data warehouse that could serve as the foundation for all of our reporting requirements, both internal and external.

The new technology infrastructure we built had to let us move and analyze data easily, so our teams could focus on using that data and insights to better serve our customers.

Previously, we had lots of siloed systems and applications running in AWS. We did a trial using Redshift, but we needed more flexibility than it offered in how we loaded historical data into our cloud data warehouse. Though we were using MongoDB Atlas for our transactional database, it was important to continue using SQL for querying data.  

The trial task that really sold us on BigQuery was when we wanted to alter a small table that had about 20 million rows, used for internal reporting.

We needed to add a value, but our PostgreSQL system wouldn’t allow it. Using Apache Beam, we set up a simple pipeline that moved data from the original source into BigQuery to see if we could add the column there.

BigQuery ingested the data and let us add the new value in seconds. That was a significant moment that led us to start looking at how we could build end-to-end solutions on Google Cloud. BigQuery gave us multiple options to load our historical data in batches and build powerful pipelines. 

We also explored Google Cloud’s migration tools and data pipeline options. Once we saw how Cloud Dataflow worked, with its Apache Beam back-end, we never looked back. Google Cloud provided us with the data tools we needed to build our data infrastructure. 

Cloud for data, and for users
Introducing new technologies isn’t always simple—companies sometimes avoid it altogether because it’s so hard. But our Google Cloud onboarding process has been easy. 

It took us less than two months to fully deploy our BigQuery data warehouse using the Cloud Dataflow-Apache Beam combination.

Moving to Google Cloud brought us a lot of technology advantages, but it’s also been hugely helpful for our internal users and customers. The data analytics capabilities that we’re now able to offer users has really impressed our internal teams, including developers and DevOps, even those who haven’t used this type of technology before.

Some internal clients are already entirely self-service. We’ve hosted frequent demos, and also hosted some “hack days,” where we share knowledge with our internal teams to show them what’s possible. 

We quickly found that BigQuery helped us solve scale and speed problems.

One of our main pain points had been adding upsell opportunities for customers during the checkout process. The legacy technology hadn’t allowed us to quickly reflect those changes in the data warehouse. With BigQuery, we’re able to do that, and devote fewer resources to making it happen. We’ve also eliminated the time we were spending tuning memory and availability, since BigQuery handles that. Database administration and tuning required specialized knowledge and experience and took up time. With BigQuery, we don’t have to worry about configuring that hardware and software. It just works.

We’ve also eliminated the time we were spending tuning memory and availability, since BigQuery handles that. Database administration and tuning required specialized knowledge and experience and took up time. With BigQuery, we don’t have to worry about configuring that hardware and software. It just works.

Two features in particular that we implemented using BigQuery have helped us improve the performance of our core transactional database. First, using Cloud Dataflow to convert raw MongoDB logs to structured rows under a BigQuery table, which we can then query using SQL to identify slow or underperforming queries. Second, we can now query multiple logging tables using wildcards, since we load Fastly logs to BigQuery. 

Along with MongoDB Atlas as our main transactional database, much of our infrastructure now runs as Google Cloud microservices using Google Kubernetes Engine (GKE), including the home page and our payment system. Kubernetes cron jobs power background scheduled jobs, and we also use Cloud Pub/Sub. Cloud Storage handles any data storage if any space constraints emerge. 

Our overall performance has increased by about 10x with BigQuery. Both our customers and internal clients, like our sales and finance teams, benefit from the new low-latency reporting. Reports that used to be weekly or monthly are now available in near-real time. It’s not only faster to read records, but faster to move the data, too.

We have Cloud Dataflow pipelines that write to multiple places, and the speed of moving and processing data is incredibly helpful. We stream in financial data using Cloud Dataflow in streaming mode, and plan to have different streaming pipelines as we grow. We have several batch pipelines that run every day. We can move terabytes of data without performance issues, and process more than 100,000 rows of data per second from the underlying database. It used to take us a month to move that volume of data into our data warehouse. With BigQuery, it takes two days.

We’re also enjoying how easy and productive these tools are. They make our life as software engineers easier, so we can focus on the problem at hand, not fighting with our tools. 

What’s next for Universe
Our team will continue to push even more into Google Cloud’s data platform. We have plans to explore Cloud Datastore next. We’re also moving our databases to PostgreSQL on GCP, using Cloud Dataflow and Beam. BigQuery’s machine learning tools may also come into play as Universe’s cloud journey evolves, so we can start doing predictive analytics based on our data. We’re looking forward to gaining even more speed and agility to meet our business goals and customer needs.    

I’m so excited to be part of Google Cloud. Data has been a longstanding part of my career and it is at the heart of business transformation. Many companies have mastered the ability to collect data and have mechanisms in place to draw on some of it to solve business problems.  But most data collected piles up and is never put to a useful purpose. Accessing it and mining it for helpful insights is practically impossible at many companies. It’s always stuck in hard to reach places, fragmented across departments and unavailable when you need it the most.  

Our mission at Google Cloud is to accelerate your ability to digitally transform your business with data. Solving data challenges is in our DNA, and over the last two decades we’ve been in a unique position to help our customers get the most out of data to drive real business value.  

Google products are used and loved by billions of people across the globe. These products bring together the complex web of disconnected, disparate, and rapidly changing data that makes up the internet. When you get an answer in milliseconds from google.com via a simple search bar, you know we have this down to a science. Google Cloud brings this expertise in data and software together for businesses of all sizes so that you can gain advantage from your data. We call this the data cloud. 

Enter the data cloud

A data cloud offers a comprehensive and proven approach to cloud and embraces the full data lifecycle, from the systems that run your business, where data is born, to analytics that support decision making, to AI and machine learning (ML) that predict and automate the future. A data cloud allows you a way to securely unify data across your entire organization, so you can break down silos, increase agility, innovate faster, get value from your data, and support business transformation. This is the heart of the data cloud.

Why a data cloud is essential

Building a data cloud using Google Cloud’s technologies helps organizations accelerate business transformation by giving everyone access to the right information at the right time so that they can act more intelligently based on it.

Since I’ve joined Google, I’ve been not only inspired by the work that the team has done to build products with a user-first mindset, but our customers have been an inspiration to each of us in what’s possible. 

• The Home Depot built a data cloud using Google Cloud technologies to help keep 50,000+ items stocked at over 2,000 locations. They’re making their 400,000+ associates smarter by giving them visibility into the things each customer needs, like item location within a local store. By leveraging BigQuery, their query performance dropped from hours and days to seconds and minutes. The Home Depot also uses Cloud SQL, Spanner, and Bigtable for their operational use cases and AI to help locate goods using their mobile apps for in-store navigation. 
• Major League Baseball (MLB) is reimagining the fan experience with their data cloud. To build engagement with today’s fans, drive engagement with future generations, and lay the groundwork for future innovation, MLB consolidated its infrastructure and migrated to Google Cloud’s Anthos, Google Kubernetes Engine, Cloud SQL, and BigQuery. MLB tracks every moment of every game for an audience on seven continents with Cloud SQL,  this valuable data to drive deeper engagement with fans.
• Vodafone is using their data cloud to offer their customers new, personalized products and services across multiple markets. By identifying more than 700 use cases to deliver new products and services, Vodafone can support fact-based decision-making, reduce costs, remove duplication of data sources, and simplify operations.  With Google Cloud,  Vodafone’s operating companies in multiple countries can access improved data analytics, intelligence, and machine-learning capabilities. 

Here are four reasons why customers trust  Google Cloud to build their data cloud strategy: 

First, Google  delivers insights at planet scale  

Customers often gravitate to Google Cloud for our specific data tools that were built for Google’s internal data needs and are unmatched for speed, scale, security, and capability for any size organization. BigQuery is the leading solution for analytics and allows you to run analytics at scale with a 99.99% SLA and up to 34% lower TCO than cloud data warehouse alternatives. Spanner provides unlimited scale, global consistency across regions, and high availability up to 99.999%, at a TCO that is 78% lower compared to on-prem databases and 37% lower than other cloud options. Firestore continues to see rapid adoption with over 2M databases created to power mobile, web, and IoT applications across customer environments. And finally Looker, an API for all your data, offers a single shared place for people and apps to interact with it, no matter the cloud environment. 

Second, Google’s AI helps your business be more intelligent

Google was built on pioneering AI research and the principle of making the world’s information useful to people and businesses everywhere. AI powers some of Google’s most popular products, such as Search, Maps, Ads, and YouTube. We have leveraged this expertise to deliver a new, unified AI platform that gives every data scientist, data analyst, and ML engineer access to the same AI toolkit Google uses. Automated machine learning, accelerated experimentation and custom training, and more deployed models than any other platform enable your entire data team to drive business outcomes at any scale. 

Third, Google is the open data platform 

Google Cloud’s open platform gives customers maximum flexibility for managing transactional, analytical, and AI-based applications. Customers can choose from a wide range of transactional, processing, and analytics engines, open source tools, open APIs, and ML services to eliminate lock-in. This includes choice of deployment across multi-cloud and hybrid environments and easy interoperability with existing partner solutions and investments. With BigQuery Omni, organizations can choose to deploy their data warehousing solution to work natively with AWS or with Azure (coming soon). Looker supports 50+ distinct SQL dialects across multiple clouds and our database services like Cloud SQL, one of the fastest growing services at Google Cloud, offers familiar open source MySQL and PostgreSQL standard connection drivers, so you can work with your preferred tools and stay up-to-date with the latest community enhancements. In addition, Google offers an unrivaled developer community across the fields of AI, machine learning, mobile, application development, microservices, and access to third party solutions and open source systems. 

Fourth, Google offers a trusted platform for your data needs

Customers can take advantage of the same secure-by-design infrastructure, built-in data protection, and global network that Google uses to ensure compliance, redundancy, and reliability. All of Google’s data is encrypted in transit and at rest, by default. Google offers industry-leading reliability across regions so you’re always up and running. Spanner offers a 99.999% SLA and BigQuery offers a 99.99% SLA. For BI and embedded analytics, Looker supports data governance via a semantic layer that organizes your data and stores your business logic centrally, delivering consistent and trusted KPIs. And finally, our multi-layered security approach across hardware, services, user identity, storage, internet communication and operations provides peace of mind that your data is protected.

Learn more at Data Cloud Summit

We are committed to helping you build a data cloud that gives you deep insights into your business and process automation. Join me as I welcome Anders Gustafsson, CEO of Zebra and 

Gil Perez, CIO of Deutsche Bank at the Data Cloud Summit on May 26, 2021 to learn and share new ways to use data for good. I can’t wait to hear what you accomplish.

Cloud-native features are helping public sector teams innovate faster than ever. Ideas discussed in a morning meeting can be a working proof of concept later that day. Managed services can remove administrative burden and reduce the steps needed to design and provision cloud infrastructure. Security can be built-in from the beginning with Identity and Access Management (IAM), Virtual Private Cloud Service Controls (VPC-SC), and Data Loss Prevention (DLP). Short-lived services and infrastructure-as-code allow rapid and cost-effective prototyping. These technologies can be used to architect a solution that follows the principle of least privilege and helps you secure your data.

So the question becomes: given the complex problems agencies face, where do you start? Google Public Sector now offers “Getting Started” and “Scaling” service offerings for CCAI, DocAI, and BigQuery to help you jumpstart your AI journey, based on where you are.

Complex problems, simpler AI-based solutions

Solving more challenging problems with cloud-native technology doesn’t have to be overwhelming. You can approach them the same way you might solve a puzzle: start with one piece that follows another until the larger picture takes shape. Though you can simplify the steps, solving these problems still requires powerful tools. Google Cloud’s AI/ML capabilities may be the answer for your team.

Google Public Sector is making it easier to get started with advanced technologies, beginning with artificial intelligence and machine learning (AI/ML) workloads for government organizations, and it’s something you can do now, one piece at a time.

Automating your FAQs with CCAI

Does your agency require a team to answer commonly asked questions? What if you could train an agent to answer questions immediately and operate 24/7? Contact Center AI (CCAI) can do this and more. Already using CCAI and need the agent to level up to address complex interactive dialogs? Getting Started with CCAI and Scaling with CCAI are new Google service offerings specifically designed to help public sector organizations tackle situations like these.

Automate data entry with DocAI

How many hours does your team spend manually reviewing or entering data from standardized forms? What if you could automatically pull data right from the page? Google Document AI (DocAI) specializes in exactly this—even if the form has handwritten text. Getting Started with DocAI and Scaling with DocAI are new service offerings that help you remove this burden from your team. DocAI automates data entry and makes that data available to other teams while prioritizing both security and ease of use.

Making data and insights accessible with BigQuery

Then there’s all your existing data. You may have years of it stored in many places, and you may not have a way to make use of it when you need it. BigQuery is Google’s enterprise data warehouse. It was designed for data analytics—looking back at historical data to make conclusions about it. But BigQuery’s analytics don’t stop there. It can also look forward in time to make predictions, often using the same datasets. Getting Started with BigQuery and Scaling with BigQuery are new service offerings that help you take your first steps toward AI/ML capabilities by starting with a single table or pipeline that can help make sense of all your data.

The best help is the kind that meets you where you are and gets you where you want to be. Google Public Sector’s new service offerings do just that: help you work through complex problems by meeting you wherever your starting line is, whether you’re ready to start or ready to scale. Let us know if you would like us to contact you about the services mentioned in this article. Let’s solve your highest impact problems together, one puzzle piece at a time.

Education data can inform strategy, enhance both teaching and learning, and help us better understand customer needs. But that data is only as useful as an organization’s ability to access, analyze, and act upon it. Education Technology (EdTech) companies with a strong data warehouse management policy can help to make the most of educational data at all levels.

Creating a data team

When an EdTech doesn’t have a dedicated data team, databases will likely be poorly maintained and data can be hard to find, it can lack critical security, or fail to meet a customer’s specific requirements. A dedicated data team can prevent this and better support an organization’s data needs.  There’s a correlation between the maturity of a data team and the success of an organization’s data utilization. When considering the three primary components of team, outlook, and results, we can better understand an organization’s maturity. By using this matrix from Looker’s Analytical Maturity Playbook, you can assess the maturity of your organization’s data team. Most would agree, the goal is for your operation to move from data chaos to data-driven. But, how can you do that? Think about your data this way: the intelligence you gain from your data is not only an asset…it’s a product. When your data team views your business intelligence as a product—one that’s just as important as the services you offer—they can set the stage for creating a successful data-driven company. Mature data organizations:

• Report functionality regularly
• Constantly assess potential improvements to existing systems
• Manage content rollout to internal employees

Redivis: Making data accessible, connected & secure

Redivis is a platform that connects academic researchers with data and the tools to understand education intelligence. Their customers, who are researchers, expressed how difficult it is to find new datasets for their studies. Once they found a dataset, they often had to access and work with it before they could tell if the information would be useful for their studies. Redivis Data administrators were also concerned about data security. Redivis built their platform on top of Google Cloud’s security infrastructure, making it a highly scalable data processing environment. 

Redivis developed a transparent, tiered access system for datasets that allows researchers to request separate access to a dataset’s documentation to determine whether they can use the dataset. Detailed audit logs (supported by Google Cloud Logging) and robust application-level security controls tighten data access. To offer more insights for their researchers, they securely connected their private datasets with public datasets hosted in BigQuery allowing researchers to run queries on billions of records in seconds, not hours.

With a data team in place, it’s time to visualize where to store and organize your data. Data is an asset. You want to invest in collecting, operationalizing, and storing it properly. “Inadequate data aggregation, harmonization, and processing impedes teams from making the right data-driven decisions and pivoting when needed,” according to Jesus Trujillo Gomez, a strategic business executive at Google Cloud. You can determine how mature your warehouse is by using this matrix to help you identify where your organization falls on the spectrum (least mature on the left, most mature on the right). A company’s data warehouse can be assessed in four areas: Technology, Pipeline, Quality, and Performance. Here are some suggested steps to bring your EdTech company one step closer to a data powerhouse. 

Classcraft: Using a data-driven approach to make learning fun

Launched in 2014, learning company Classcraft is reimagining the classroom, creating better learning environments with gamification. To support their mission, their team has created a complete data platform that uses data analytics to find new ways to engage students in the classroom. Classcraft became a Google for Education Build Partner to keep up with rising demand. They expanded their footprint to reach 6 million students and educators in 165 countries. 

They also migrated to Google Cloud, building on several integrations with Google Workspace for Education, Google Classroom, and Chrome for Education, allowing them to support customers ranging from individual teachers to entire school districts.  Classcraft migrated its database and analytics infrastructure to BigQuery and Cloud Storage. “We capture about 10 million new data points every month related to school culture and student performance,” says Chief Technology Officer and VP of Product Stéphane Guillemette. “With BigQuery and other Google Cloud tools, we can quickly deliver valuable insights to educators.”