Verizon Media Shows the Solution Architecture it Uses for a 100+ PB Analytics Platform - Build What's Next

2994

Of your peers have already watched this video.

39:00 Minutes

The most insightful time you'll spend today!

Case Study

Verizon Media Shows the Solution Architecture it Uses for a 100+ PB Analytics Platform

Verizon Media owns and operates more than a dozen brands including Yahoo Mail, Yahoo News, AOL, Huffington Post, TechCrunch, and Engadget among others.

These web properties are visited by millions of users on a daily basis.

In this session, Shakil Memon, Customer Engineer, Google Cloud and Nikhil Mishra, Sr Director, Engineering, Verizon Media, present how Verizon Media is generating actionable insights from the wealth of data that they have.

They will discuss:

  • Challenge with large scale and large volumes of data
  • Basic principles of a data warehousing and data analytics project
  • Showcase a reference architecture
  • A complete end-to-end solution architecture for building a 100+ PB internet-scale analytics platform on Google Cloud, including how Looker fits in the end-to-end solution and how actionable insights are generated from data.

They will also provide unique perspectives, behind-the-scenes thinking, and some insight on how the architecture has evolved in its current form over the years.

3004

Of your peers have already watched this video.

34:30 Minutes

The most insightful time you'll spend today!

Explainer

Reinventing geospatial analytics as only Google can do: Intelligent platform

Geospatial analytics is undergoing a fundamental change with the application of cloud computing products. Every day, data analysts need to understand the geospatial impact on their data and GIS professionals are finding themselves constrained by legacy technology that can’t keep pace with increasing data volume.

Google Cloud’s geospatial analytics platform brings the full suite of Google’s Geo offerings to both sets of users to unblock workloads and free them from legacy capacity constraints.

Listen to Joe Bettridge, Consulting Lead, TELUS Insights, TELUS and Chad Jennings, Product Manager and GIS Lead, Google Cloud, and learn about Google Cloud’s uniquely powerful, full–platform solution for geospatial analytics and how organizations are using it to reduce query times from hours, days, or months to minutes. These workloads can span both structured and imagery data.

In this session, they will cover:

  • An overview of BigQuery GIS
  • Some use cases
  • Examples of algorithms TELUS is using
Case Study

Vodafone Leverages Google Cloud to Aid COVID-19 Frontline with Anonymized Insights on Population Mobility

11222

Of your peers have already read this article.

1:30 Minutes

The most insightful time you'll spend today!

Vodafone and Google Cloud work together to retrieve anonymous, network-based insights from Google Cloud Storage and validate that data on Dataflow to power research on populations' mobility patterns across the EU for navigating COVID-19 challenges.

Editor’s note: When Europe’s largest mobile communications company, Vodafone, was asked by the European Commission to help understand population movement across the European Union and the UK to help fight COVID-19, it was able to provide anonymized mobile network-based insights to answer the call. Here’s how Vodafone, with the support of Google Cloud, rapidly mobilized the COVID-19 frontline, while respecting its customers’ privacy.

With the emergence of COVID-19 in early 2020, the European Commission—the executive branch of the European Union (EU)—knew that technology would be instrumental in its fight to control the pandemic. With various lockdowns imposed across its member states, the Commission was keen to predict and prevent the spread of COVID-19 and to manage the related social, political and financial impacts. 

Mobile network data helps track COVID-19 across the EU

Mobile networks produce location data, which can be turned into useful anonymous insights to understand population movement within a geographic area. The European Commission, working with mobile industry association GSMA (Groupe Speciale Mobile Association), asked Europe’s major mobile phone operators for help in producing insights to support the fight against COVID-19. As the largest mobile network operator within the EU, Vodafone saw this as a critical opportunity to participate. 

Vodafone had previous experience of using mobile network data to support pandemic research. For example, in 2019, Vodafone provided mobility pattern analysis to help track the spread of Malaria in Mozambique. And, during the early stages of the COVID-19 pandemic (prior to working with the European Commission), Vodafone assisted the Italian and Spanish governments in understanding their citizens’ mobility patterns. Vodafone had also previously offered anonymized and aggregated population mobility insights to support public transport and tourism authorities and retail organizations in a number of countries. Consequently, Vodafone was perfectly placed to play a greater role in supporting the European Commission’s response to the pandemic. 

When asked to assist the European Commission, Vodafone first considered how it could safely share its data with the governing body without providing details on the individual movements of its customers. It realized it could achieve this through an elaborate set of anonymization and aggregation techniques. Insights are aggregated from a minimum of 50 users and Vodafone only shared these anonymous insights and never the actual raw data with the Commission. As specified by the EU, these insights are then presented onto a large geographical region, typically a city or a county with thousands of people living in that area.

These insights illustrate how people move, helping to determine how lockdowns and self-isolation measures were impacting behaviors.

Using Google Cloud to collate and store population mobility data

In April 2020, Vodafone began migrating its operations, including its mobile data, to Google Cloud on servers in Europe and the UK with elaborate security safeguards, including encryption, building on a previous partnership. 

With the data residing in EU and UK data centers and not the United States, Vodafone could then retrieve anonymous insights from Google Cloud Storage instantaneously. Before supplying any information to the European Commission, however, Vodafone used Dataflow to validate the data and run a series of tests to ensure the database had accurate data, before ingesting and archiving the relevant metrics. For instant access, the data was then made available to the European Commission using a Redis database on Google Kubernetes Engine.

To ensure aggregate Vodafone customer data was always safe, secure, and anonymous, all entry points to the front-end were protected behind Google Cloud Armor, where only specific IP addresses were allowed. Using these tools, seamless data pipelines fed in predefined key performance indicators from each specified European market. While data quality measures ensured the definitions for metrics across markets were consistent and could be accurately compared.

The architecture (pictured below) shows how Vodafone integrated and anonymized its data on Google Cloud.

Vodafone.jpg

Live interactive dashboard shows population mobility in real-time

With its data integrated on Google Cloud, Vodafone created a live, interactive dashboard to track mobility patterns and share relevant information with the European Commission in real-time. 

The European Commission Joint Research Center (JRC) was able to gather valuable information from these insights, which enabled them to see where population mobility was aiding the spread of the disease, when cross-referenced with health data. It could also assess the implications of lockdowns on different populations and forecast cross-country spreading.

Mobile data aids disease modeling for multiple stakeholders

The Vodafone data became instrumental in modeling the likely course of the disease too. For example, the University of Southampton in the UK used it to predict the outcome of different coordinated COVID-19 exit strategies across Europe. This research was published in Science Magazine in September 2020. 

The Vodafone data dashboard continues to be used by individual governments, NGOs and organizations to further investigate the impacts of the pandemic and to measure the effectiveness of response strategies alongside the rollout of vaccination programs. The project also helped Vodafone win a DataIQ award for most effective stakeholder engagement

Using the learnings from this project, Vodafone has been able to adapt its own B2B solution, called Vodafone Analytics, by adaptIng and migrating the code to work in Google Cloud Platform. This solution has been rolled out across Germany, Greece, Portugal and South Africa, and new countries are being onboarded every day. Vodafone Analytics already has more than 100 customers leveraging it for a variety of use cases—Italian fashion retailer OVS, uses it for its smart retail operation, while global real estate company, JLL, uses it to understand the footfall passing through its properties. 

Working together, Vodafone and Google Cloud continue to help a range of organizations, governments, and NGOs navigate through the ongoing pandemic,  optimize their operations, and help the greater good, without infringing individuals’ fundamental rights to privacy.

To learn more about Google Cloud and Vodafone, watch our full interview here.

Case Study

IT Team Figures Out Easiest Way to Build Data Pipelines and Create ML Models

14133

Of your peers have already read this article.

5:15 Minutes

The most insightful time you'll spend today!

To give brands greater agility to rapidly create high-impact customer experiences and increase its own competitive edge, Brandfolder moved to Google Cloud Platform, using AI-powered solutions and fully managed cloud services to enable an efficient and focused development team to improve customer experiences.

Building a strong brand in today’s hyper-competitive business environment takes vision. It also requires a flexible, easily managed approach to digital asset management (DAM), so marketing professionals and other stakeholders can easily share, store, track, and manipulate assets to build the brand.

Many of today’s leading companies, including JetBlue, Slack, TripAdvisor, Lyft, and HealthONE, rely on Brandfolder to deliver consistent, organized, and efficient brand experiences. Brandfolder provides an easy-to-use platform that can scale across an entire company with little end-user training, empowering customers to distribute digital assets wherever they are needed. Customers also gain much greater insight into how those assets are used, and how to use them more effectively in marketing campaigns and brand messaging.

“Google Cloud made it easy to build an ML platform to quickly iterate through different brand intelligence use cases and release data-driven product features into the Brandfolder platform.”

Ajay Rajasekharan, Head of Data Science, Brandfolder

Brandfolder is constantly advancing its development efforts to introduce new data-driven features without complicating the user experience. Big data, artificial intelligence (AI), and machine learning (ML) are key to meeting customers’ unique business needs, and essential for Brandfolder to compete in the fast-moving DAM industry. To enhance these capabilities, Brandfolder sought a public cloud provider that could help it scale its data pipeline cost effectively while providing access to advanced AI technologies.

After graduating from the Techstars startup accelerator program in 2013, Brandfolder tried two other cloud providers before standardizing on Google Cloud Platform (GCP).

“We saw a difference with Google Cloud from the very beginning because the interactions felt like a strategic relationship,” says Jim Hanifen, Head of Product at Brandfolder. “Google gave us startup credits and a lot of face-to-face support, which we hadn’t experienced with other cloud providers. We decided to move our entire infrastructure to Google Cloud Platform.”

Building an ML platform for brand intelligence

After performing an initial lift-and-shift migration of virtual machines (VMs) onto Compute Engine, Brandfolder built an ML platform using GCP managed services to seamlessly deliver its data products. The platform leverages Cloud SQLCloud Storage as the data lake, Cloud Dataproc for cloud-native Apache Spark computing clusters, Cloud Composer as the batch job scheduler, Cloud Pub/Sub as the backbone data pipeline, Container Registry to store Docker images, and Google Kubernetes Engine (GKE) as the application orchestrator. Cloud Dataflow brings data into the data lake and into BigQuery for analysis.

“Google Cloud made it easy to build an ML platform to quickly iterate through different brand intelligence use cases and release data-driven product features into the Brandfolder platform,” says Ajay Rajasekharan, Head of Data Science at Brandfolder, who describes the architecture in a detailed blog. “We simply ingest raw application and event data on one end and output an ML service on the other.”

“Moving to Google Cloud Platform allows us to complete more sophisticated data analysis and ML models much faster, and at a much lower cost. We can create brand-specific ML models 12x faster and get them into production quickly to address our customers’ unique business needs.”

Brett Nekolny, Head of Engineering, Brandfolder

For many general use cases, Brandfolder does not need to build custom ML models, and instead relies on pre-trained API models from GCP. For example, it uses Vision API and Video Intelligence API to auto-tag creative assets on import to enable fast, intuitive searches across images and videos. When more product- and brand-specific modeling is required to address unique customer use cases, Brandfolder builds and trains custom ML models using its GCP pipeline or Cloud AutoML, a suite of products built on Google transfer learning and neural architecture search technology. For example, if a Brandfolder customer makes different types of grills, Brandfolder can use AutoML Vision to train a model to recognize the different grills.

“Moving to Google Cloud Platform allows us to complete more sophisticated data analysis and ML models much faster, and at a much lower cost,” explains Brett Nekolny, Head of Engineering at Brandfolder. “We can create brand-specific ML models 12x faster and get them into production quickly to address our customers’ unique business needs.”

Industry-leading security and performance

Google Cloud’s security model helps Brandfolder give existing and prospective customers peace of mind that their data will be protected. Cloud Identity & Access Management (Cloud IAM) provides enterprise-grade access control, while Cloud Identity-Aware Proxy (Cloud IAP) enables remote users to work more securely without the hassles of a VPN client. GCP also isolates cloud resources into projects, making it easy to assign permissions and keep data and VMs organized and segregated.

“With Google Cloud, everything begins and ends with security, which makes things very easy for us,” says Jim. “If we’re under a security review, we can submit a Google security white paper. If a potential customer has security concerns, we tell them we are hosted on GCP, and those concerns go away.”

To give customers even better application performance for accessing their brand assets, Brandfolder uses Cloud Memorystore, an in-memory data store service for Redis, to cache data and provide sub-millisecond data access for production applications.

“It was much easier for us to use Cloud Memorystore versus running Redis on our compute instances,” says Brett. “The high availability, replication across zones, and automatic failover with no data loss are big for us.”

Global private network interconnects between Google Cloud and the Fastly content delivery network (CDN) dramatically reduce latency, allowing Brandfolder’s customers to deliver and update even very large creative assets quickly around the world.

“What’s beautiful about the relationship between Google and Fastly is that if one of our customers uploads a new version of an asset, we can propagate that out to Fastly, and the new version will automatically show up in all the places where it’s referenced,” says Brett.

“The ability to quickly solve problems with AI has a substantial impact on our revenue, and that’s more apparent every quarter. Few of our competitors are doing product- or brand-specific modeling because it takes a lot of time and resources. We overcame those hurdles with Google Cloud.”

Jim Hanifen, Head of Product, Brandfolder

Improving employee and customer productivity

Brandfolder also uses Google solutions for real-time collaboration and productivity, using G Suite to connect employees with intuitive, cloud-based apps. Teams use GmailCalendarDocsDriveSheetsSlides, and Hangouts Meet every day to move the business forward. Many of Brandfolder’s customers are also G Suite users, and Brandfolder offers a plug-in that allows them to view their creative assets inside of Docs and pull images in as needed. Customers can also log into Brandfolder with their G Suite credentials, making the solution even easier to use.

“We’ve been using G Suite since the beginning, and it’s helped us collaborate efficiently to build a successful, growing company,” says Jim. “Our teams expect to have that kind of close collaboration, and everyone here enjoys the G Suite experience.”

Driving 99 percent annual business growth

With automated tagging and other innovative AI-based features, Brandfolder is helping customers locate and distribute assets faster. As a result, Brandfolder is building customer loyalty and increasing sales, growing its business by 99 percent year-over-year. Since moving to GCP, Brandfolder has been able to scale its analytics and data pipeline 50x without a corresponding increase in costs and has not had to expand its development team.

“The ability to quickly solve problems with AI has a substantial impact on our revenue, and that’s more apparent every quarter,” says Jim. “Few of our competitors are doing product- or brand-specific modeling because it takes a lot of time and resources. We overcame those hurdles with Google Cloud.”

3138

Of your peers have already watched this video.

36:30 Minutes

The most insightful time you'll spend today!

Case Study

Don’t sweat the big stuff. Make it Google’s problem

Need to interpolate new time series data values over 5 billion rows? Don’t reach for python. Make that Google’s problem and do it in BigQuery.

Need to aggregate petabytes of geospatial data across arbitrary polygons and put it on a map for analysis? Make that Google’s problem and use BQ-GIS.

Great – your map was awesome and now we need it every hour. Roll your own Airflow server? Nope. Make that Google’s problem. You see the pattern.

Geotab, which operates in in the telematics space, applies instrumenting to vehicles to learn how to optimize fleet maintenance, routes and costs, and even find way to optimize city infrastructure.

Geotab has been a long-time customer of Google Cloud products. They leverage the entire GCP suite to empower data scientists to efficiently ingest, process, and analyze petabytes of IoT data.

One of the keys to their pace of innovation and growth has been to focus on their key competencies and partner with others to fill in the gaps.

For Geotab’s data scientists this means focusing on the data and model-building and whenever possible making the processing, storage, and orchestration, well Google’s problem.

Whitepaper

Google Cloud named a leader in the Forrester Wave: Streaming Analytics

DOWNLOAD WHITEPAPER

4194

Of your peers have already downloaded this article

15:30 Minutes

The most insightful time you'll spend today!

There’s a lot of pressure on companies today to become data-driven, but they need high-performing technology to make that a reality.

As part of Google’s larger cloud data analytics platform, Cloud Pub/Sub and Cloud Dataflow are designed for ease of use, scalability, and performance.

Google Cloud unifies streaming analytics and batch processing the way it should be. No compromises.

Forrester

Forrester has named Google Cloud as a leader in The Forrester Wave™: Streaming Analytics, Q3 2019. This findings reflect Google Cloud’s market momentum, and what Google hears from enterprise customers who are using Cloud Pub/Sub and Cloud Dataflow in production for streaming analytics in conjunction with our broader platform.  

According to Forrester, many leading enterprises realize that real-time analytics—the analytics of what’s happening with data in the present—is an incredible competitive advantage. With it, they can act right away to serve customers, fix operational problems, power internet of things (IoT) apps, and respond decisively to competitors.

The report evaluates the top 11 vendors against 26 rigorous criteria for streaming analytics to help enterprise IT teams understand their options and make informed choices for their organizations. Google scored 5 out of 5 in Forrester’s report evaluation criteria of scalability, availability, aggregates, management, security, extensibility, ability to execute, solution roadmap, partners, community, and customer adoption. 

We are very excited about the productivity benefits offered by Cloud Dataflow and Cloud Pub/Sub. It took half a day to rewrite something that had previously taken over six months to build using Apache Spark.
Paul Clarke, Director of Technology, Ocado

While stream analytics is a leading business priority, real-life use cases frequently require batch data as an input as well. Google Cloud Platform (GCP) customers can simplify their pipeline development by reusing code across both batch and stream processing using Apache Beam, which also provides pipeline portability to OSS projects. Further, Google Cloud’s data analytics portfolio autoscales across ingestion, processing, and analysis, which eliminates the need for provisioning and makes handling varying volumes of streaming data automatic. We hear from users that they’re able to ingest and process data much more quickly than in the past, helping to get new business insights faster and allowing more users to do self-serve analytics.

More Relevant Stories for Your Company

Blog

DocAI Lowers Customer’s Document Processing Cost by 60 Percent. Learn How

Some of the most important data at your company isn’t living in databases, but in documents, and most business processes begin, involve or end with a document.  Yet most companies are still manually entering data and reliant on guesswork to make sense of it all as the volume and variety

Case Study

This Diagnostic Company is Revolutionising Healthcare Delivery with AI

Dr. Elliot Smith cannot be accused of lacking ambition. A high achiever with a Ph.D. in Electrical Engineering and a specialist in magnetic resonance imaging (MRI) systems, Smith aims to deliver top quality healthcare to anyone in the world -- regardless of their location or wealth. Dr. Smith has already

How-to

How Modern is Your Data Warehouse? Find Out With This Test

As more and more businesses turn to advanced data analytics to help them make smarter decisions, run real-time analytics, and improve business operations, an increasing number are modernizing their data warehouses to make it all possible. For many businesses, knowing how to modernize means understanding where their data warehouse sits

Case Study

Customer Voices: How Firms from Across Industries Leverage Google Cloud

From powering everyday operations and accelerating application innovation, to providing tools for specific business needs and executing on big ideas, to advancing the security of technology solutions, companies from across industries have leveraged Google Cloud for business benefits. Companies from across industries have turned to Google Cloud for transforming their

SHOW MORE STORIES