Google Migration and BigQuery Brings PedidosYa Closer towards its Goal of Becoming Data-driven - Build What's Next
Case Study

Google Migration and BigQuery Brings PedidosYa Closer towards its Goal of Becoming Data-driven

7239

Of your peers have already read this article.

2:00 Minutes

The most insightful time you'll spend today!

PedidosYa, a Latin American leader in the online food ordering space with over 20 million app downloads was looking to democratize data by gaining a secured access and building a comprehensive information ecosystem. PedidosYa was also challenged with its legacy data warehouse that couldn't keep with the brand's increasing analytics demands and was time-consuming and costly. To modernize their data warehouse and transform the analytics environment, PedidosYa chose Google Cloud for its serverless, managed, and integrated data platform, coupled with its seamless integration across open-source solutions. Also, Google Cloud's advanced cost and workload management coupled with its transparent log analytic gave the brand full visibility into any query performance issues to make improvements as they wanted. BigQuery helped PedidosYa achieve cost reduction per query by 5x and leverage its AL/ML-based stack for productivity benefits. Read on to learn more about PedidosYa's BigQuery and Google Cloud migration journey as a step closer towards becoming a data-driven organization.

Editor’s note: PedidosYa is the market leader for online food ordering in Latin America, serving 15 markets and over 400 cities. It’s also one of the largest brands within the German multinational company Delivery Hero SE. With over 20 million app downloads, PedidosYa provides the best online delivery experience through 71,000+ online partners, including restaurants, shops, drugstores, and specialized markets. 

Having constant access to fresh customer data is a key requirement for PedidosYa to improve and innovate our customer’s experience. Our internal stakeholders also require faster insights to drive agile business decisions. Back in early 2020, PedidosYa’s leadership tasked the data team to make the impossible possible. Our team’s mission was to democratize data by providing universal and secure access while creating a comprehensive information ecosystem across PedidosYa. We also had to achieve this goal while keeping costs under control— even during the migration stage and removing operational bottlenecks. 


Challenges with legacy cloud infrastructure

PedidosYa first built its data platform on top of AWS. Our data warehouse ran on Redshift, and our data lake was in S3. We used Presto and Hue as the user interfaces for our data analysts. However, maintaining this infrastructure was a daunting task. Our legacy platform couldn’t keep up with the increasing analytics demands. For example, the data stored on S3 complemented by Presto/Hue required high operational overhead. This was because Presto and our IAM (identity access management) didn’t integrate well in our legacy ecosystem. Managing individual users and mapping IAM roles with groups and Kerberos was operationally time-consuming and costly. Further, sharding access on the S3 files was far too complicated to enable seamless ACLs (access control lists).  

There were also challenges with workload management. Our data warehouse had batch data loaded overnight. If one analyst scheduled a query to run during the overnight ETL (extract, transform, load) workload, it would disrupt the current ETL task. This could stop the entire data pipeline. We’d have to wait until data engineers intervened with a manual fix.

It was also difficult to understand whether a query error was due to performance issues or platform resource exhaustion. This lack of clarity affected our data analysts’ ability to autonomously improve querying efficiency. Data team members needed to manually inspect personal queries looking for performance issues. Also,  the current architecture was prone to a ‘tragedy of the commons’ situation; it was seen as an unlimited and free resource. As a result, it was impossible to disentangle the infrastructure from different stakeholder teams, as all had very different needs. 

The decision to modernize our data warehouse

Given the growing challenges from our legacy platform, our tech team decided to transform our analytics environment with a modern data warehouse. They required the following key criteria from their next data platform: 

  • Scalability – The ability to grow with elastic infrastructure.
  • Cost control – Cost management and transparency. These factors promote efficiency and ownership—both key aspects of data democratization.
  • Metadata management – Intuitive data platform focusing on users’ previous SQL knowledge. Plus, being able to enrich the informational ecosystem with metadata,  to diminish data gatekeepers.
  • Ease of management – The team needed to reduce operational costs with a serverless solution. Data engineers wanted to focus on their key roles rather than acting as database administrators and infrastructure engineers. The team also wanted much higher availability, and to reduce the impact of maintenance windows and vacuum/analysis.
  • Data governance and access rights – With a growing employee base with varying data access requirements, the team needed a simple yet comprehensive solution to understand and track user access to data.

Migrating to Google Cloud

After exploring other alternatives, we concluded Google Cloud had an answer to each of our decision drivers. Google Cloud’s serverless, managed, and integrated data platform, coupled with its seamless integration across open-source solutions, was the perfect answer for our organization. In particular, the natural integration with Airflow as a job orchestrator and Kubernetes for flexible on-demand infrastructure was key.  

We  used Dataflow together with Pub/Sub and Cloud Functions for our data ingestion requirements, which has made our deployment process with Terraform seamless. Because we set up everything in our environment programmatically, operation time has diminished. Google Cloud reduced the deployment process from about 16 hours in our legacy platform to 4 hours.  This is partly due to the friendliness of automating the deployment (such as schema check, load test, table creation, build.) process with Terraform, Cloud Functions, Pub/Sub, Dataflow, and BigQuery on GCP. Input messages processed with Dataflow allow us to abstract and plan the schema changes according to the needs of the functional team. For example, schema changes raise an alarm, and then we can modify the raw layer table schema. By doing this, we ensure that backend modifications that we don’t control do not affect upper layers.

A key reason why we picked Google Cloud was because of its advanced cost and workload management coupled with its transparent log analytics. This information gives us a complete view into any query performance issues to make improvements on the fly. Further, we achieved a significant amount of cost savings by consolidating multiple tools to BigQuery.With BigQuery, we’ve been able to reduce our total cost per query by 5x.

This was due to a number of reasons:

  • Automating pipeline deployment made it much simpler to maintain the data processing processes. 
  • Analysts are conscious about what queries they’re running, resulting in running better, more optimized queries. 
  • Analysts use a Data Studio dashboard to see their queries and all the associated costs. As a result, there’s a lot more transparency for each persona.

 With these changes, we can easily manage and assign costs associated with each workload with their own cost centers using specific Google Cloud projects.

Change management is always challenging. However, BigQuery is intuitive and doesn’t have a steep learning curve from Hue/Hive on SQL basics. BigQuery also allowed the team to expand its capabilities and enabled them to properly work with nested structures, avoiding unnecessary joins and improving query efficiency. Additionally, we now use Data Catalog as our unique point of truth for metadata management. This allows our team to break the data access barriers and enable federation of data across the organization. By using Airflow to orchestrate everything, we keep track of every data stream. With this information, each end user can see their regularly used data entities’ status via the dashboard. This also adds transparency to our everyday data processes.

Finally, with Google Cloud’s IAM rules applied across the different products, data sharing and access is close to a noOps experience. We have programmatically implemented access according to roles and level access within the company. This allows certain pre-validated roles to view more sensitive information. These solutions help drive a more automated data governance experience. 

Up next: Google Cloud AI/ML

The new stack based on BigQuery has created significant productivity gains. Freed from the burden of operational management, PedidosYa’s data team can now focus on adding value through data tools and products.  

  • Our data engineers are better equipped to integrate constantly changing transactional and operational data.
  • The dataOps team can automate the infrastructure and provide autonomy to the end user.
  • Our data quality team can focus on bringing added value to data stakeholders. 
  • Data scientists and data analytics can spend more time analyzing data and less time asking data gatekeepers for data access.

PedidosYa can now democratize data access with a well-governed architecture. We are still at the beginning of our journey, but we are closer to achieving our vision of building a data-driven organization. Up next: expanding our artificial intelligence and machine learning capabilities.

Tune in to Google Cloud’s Applied ML Summit on June 10th, 2021, or listen on-demand later, to learn how to apply groundbreaking machine learning technology in your projects.

Blog

Navigating the Next Wave of B2B Digital Commerce: Trends and Insights for 2023

1459

Of your peers have already read this article.

1:30 Minutes

The most insightful time you'll spend today!

B2B eCommerce is expanding with consumer-like experiences, omnichannel sales, and automation. As demand grows, Commercetools predicts continued digital transformation in B2B commerce in 2023, driven by new technologies and evolving customer needs.

Editor’s note: Google Cloud partner commercetools shares how modern technologies like composable commerce, cloud-native infrastructure and artificial intelligence/machine learning (AI/ML) will lead the way in business-to-business (B2B) digital commerce this year.


Digital commerce in B2B has been predicted as the next big thing for years; yet, at the start of COVID-19, 60% of B2B companies had zero or limited eCommerce capabilities. The pandemic accelerated digitization and eCommerce has finally taken off: As of February 2022, 65% of B2B companies offered eCommerce capabilities.

The behavior of B2B buyers is also changing: Consumer-like expectations are at the heart of successful B2B commerce, and this is how manufacturers, distributors and wholesalers will shape their customer experiences. Today, 73% of B2B buyers want a personalized business-to-consumer or B2C-like experience. 83% prefer ordering or paying through digital commerce and 72% are eager to purchase across channels.

With digital commerce dictating how B2Bs will grow in 2023 and beyond, what trends will spur digital transformations across this business model? Here’s what the team at commercetools expects to unfold in B2B eCommerce this year.

#1 B2B firms are switching to cloud-native, composable commerce

B2B players still plagued with manual processes and siloed backend systems will move away from monolithic platforms and choose composable commerce. In a nutshell, composability enables businesses to select best-of-breed components, such as search, cart or checkout, and “compose” them into a custom application.


B2B firms will modernize their commerce backend, interoperating siloed systems like Configure Price Quote solutions (CPQs) for sales and enterprise resource planning solutions (ERPs) for order entry with an API-first and composable commerce stack. They will also pivot from on-premise deployments to cloud-native architectures as the baseline for auto-scaling capabilities instead of pre-provisioning online capacity during traffic peaks. That way, B2Bs can customize customer-centric experiences to boost revenue while reducing the complexity and cost of in-house IT infrastructure, as well as gaining operational efficiencies

B2Bs will maximize the cross-section of composable commerce and cloud-native infrastructure by leveraging a commerce backend like commercetools Composable Commerce hosted on Google Cloud. This combined solution provides commercetools’ ready-to-use components built as microservices and exposed as APIs, such as product information management (PIM) and unified cart, integrated through the Google Cloud Marketplace.

#2 Strong focus on data quality and personalization

Focusing on data quality continues to be a big trend in 2023. B2B buyers expect product, pricing, inventory and shipping data points to be accurate across every touchpoint so they can make better purchasing decisions, such as when to order products and calculate quantities.

With so many data points to capture throughout the customer journey — product, inventory, pricing and customer data — we’ll see more B2B companies reorganizing their vast information pools to elevate customer experiences. They will pivot to modular and API-first solutions, plus flexible data models, so they can break data silos from legacy monolithic platforms and access such data when needed.

We also expect to see more customer analytics to unlock data on buyer behavior. By understanding what customers see, click and add to their shopping lists, B2B businesses get valuable insights into how buyers behave, using this data in the shopping journey according to product interests. That way, it’s possible to offer personalized experiences across touchpoints without hassle.

“It is important for B2B companies to look at their data as if it is one of their products; invest in its upkeep and integrity while finding ways to continuously improve it. Using advanced analytics powered by AI and ML to identify patterns from large amounts of data, B2B companies can activate insights into customer decision journeys to maintain loyalty, personalize experiences to improve satisfaction and boost revenue, while also finding ways to optimize costs. For example, with analytics, enterprises can streamline spend to focus on the highest-performing channels and reduce waste.”Carrie Tharp, Google Cloud VP of Retail and Consumer

With data-driven tools coming into play like Google Cloud’s Discovery AI, Recommendations AI and Vision Product Search connected with composable commerce, B2B players can boost customer analytics to personalize experiences, improve customer satisfaction and reduce churn.

#3 The B2B customer experience will be redesigned

B2B players are taking a page out of the B2C playbook to elevate experiences throughout the customer journey. While intense work needs to happen in the backend commerce engine, B2B players will also redesign their digital frontends. That means boosting website performance, while mobile responsiveness and personalization will be at the forefront of these advanced digital initiatives.

More than ever, B2B companies are looking for digital storefronts delivered as progressive web applications (PWAs) for optimized performance and responsiveness across devices, as well as fast-loading and responsive experiences to boost your digital presence, SEO rankings and conversion rate. B2Bs can further streamline frontend development with solutions natively connecting to Google Cloud Marketplace, which supports a variety of storefront providers, including commercetools Frontend.

Leveraging Google Cloud’s unique capabilities, such as PWA web app development, Google Cloud Discovery AI solutions that include Retail Search and Vision API Product Search, among many others, B2B companies are well positioned to boost digital commerce in the years to come.

What’s next in 2023?

2022 was already a turbulent year; for better or worse, 2023 is expected to have a similar fate. For B2Bs, even the ones with tight budgets, investing in digital commerce can help future-proof businesses for whatever’s happening this year. To dive deeper into all predictions and insights by commercetools in collaboration with Google Cloud, read the guide Pivotal Trends and Predictions in B2B Digital Commerce in 2023.

How-to

Ease Your Migration and Modernization Journey with Microsoft and Windows on Google Cloud Demo Center

8286

Of your peers have already read this article.

2:00 Minutes

The most insightful time you'll spend today!

If you are looking to migrate and modernize Microsoft and Windows, check Google Cloud Demo Center that includes several simulations to walk you through many scenarios without the need for deployment, configuration and commitment. Discover more.

If you’re looking to migrate and modernize your Microsoft and Windows workloads, Google Cloud is your premiere destination. No matter what migration strategy you’ve selected or what value you’re looking to achieve, with Google Cloud you’re able to:

  • simplify your migration and modernization journey
  • reduce your on-prem footprint and increase agility
  • optimize license usage to reduce costs
  • modernize to reduce single-vendor dependencies
  • rely on enterprise-class support backed by Microsoft

Whether you’re looking to migrate applications running on Windows virtual machines, adopt Windows containers in Google Kubernetes Engine (GKE), convert SQL databases to Cloud SQL, or something else, Google Cloud offers you the first-class experience you need. 

But we don’t want you to take our word for it. Try it out yourself with our new online Microsoft and Windows on Google Cloud Demo Center without any commitment or friction.

simulated demos.jpg
Click to enlarge

The demo center uses hands-on guided simulations to walk you through several scenarios for solving business critical challenges with Google Cloud’s Microsoft and Windows solutions. Because these are all simulated, you’ll see how it works without any deployment, configuration, or commitment. It’s a seamless way for you to see exactly how Google Cloud can help you.

Run dedicated hardware and optimize with sole tenant

Sometimes you might want to run your workloads on dedicated hardware (with oversubscription options) for compliance, licensing, and management. Google Cloud provides sole-tenant nodes that allow you to easily deploy your virtual machines onto dedicated machines to avoid “noisy neighbor” issues, address regulatory or licensing constraints, and optimize inter-VM communications.

Plus, the CPU Overcommit option allows oversubscribing sole-tenant node resources by up to 2x, therefore helping save on per-physical core licensing for many licensed workloads like SQL Server.

Learn how to set up a sole tenant group and node.

Optimize license costs with premium images & custom VMs

One of the easiest ways to optimize your cloud experience with virtual machines (VM) is to pick the right VM image. Google Cloud provides premium license-included VM images that are thoroughly tested and optimized, including SQL Server options with pay-as-you-go licensing. These are great for workloads that don’t need to run all the time or when you do not have spare licenses for bring your own license (BYOL). 

Explore some Windows & SQL images and learn how CPU/Memory options can help optimize deployment and save on licensing.

Modernize your databases with Managed SQL Server

Sometimes you need to manage your SQL Server instance to achieve certain business or operational goals. But more often, managing SQL Server deployments can be undifferentiated: backups, high-availability, updates, and patching are just some of the many things you have to take care of when going the do-it-yourself route. One way to modernize your database tier is to migrate to a managed service like Cloud SQL, which is a fully managed Relational Database service for SQL. 

Explore the process of creating an instance in just a few clicks!

Extract apps from VMs and move to containers in GKE with Migrate for Anthos

Many Windows workloads running on virtual machines such as Internet Information Services (IIS) are ideal candidates for migrating to containers without major changes like rewriting or rearchitecting. However, doing this migration manually can be tedious, which is why Migrate for Anthos can help easily re-platform a .NET app running on IIS into a container-based app. 

Simulate intelligently extracting, migrating, and modernizing applications to run natively on containers in GKE and Anthos clusters.

Move .NET applications to GKE on Windows without code changes

When you’re looking to go fully cloud native, you can leverage Windows containers in GKE without rewriting your .NET applications. Simply create clusters with Windows nodes and deploy containerized Windows workloads in a few clicks, even alongside Linux containers. These deployments reduce operational overhead with features such as auto-upgrade, auto-repair, and release channels. 

Learn  how easy it is to build a GKE cluster with a Windows node and deploy an app. 

Now that you’ve gotten a feel for what’s available, go check out the Demo Center. You can also visit us at Windows and Microsoft on Google Cloud to learn more.

E-book

Security in the Cloud: Google’s Answer

DOWNLOAD E-BOOK

5568

Of your peers have already downloaded this article

1:30 Minutes

The most insightful time you'll spend today!

Protecting a global network against persistent and constantly evolving cyber threats is one of the most important challenges faced by Google Cloud. So, how does Google’s global network protects seven different global businesses, each with over 1 billion customers, including popular Google services such as Google Search, YouTube, Maps, and Gmail?

The answer is a multi-step process, which is constantly changing and evolving to stay one step ahead of the malicious hackers and attackers. For instance, it’s network communications protocols—the rules that enable communications between systems—change multiple times per second to make malicious intrusions much harder.

Data in Google Cloud is encrypted both in transit and at rest. Google’s network capacity far exceeds any traffic load it hosts to thwart and DDoS attack. In addition, it has numerous other products, tools, and processes at work to provide defense in depth.

Download this e-book to get a detailed overview of Google Cloud’s approach to security and privacy.

Whitepaper

The Cloud-First Imperative To Accelerate Digital Transformation

DOWNLOAD WHITEPAPER

3548

Of your peers have already downloaded this article

1:30 Minutes

The most insightful time you'll spend today!

In a study of 360 business and technology decision makers and influencers from Indian enterprises, conducted by Forrester Consulting, it is revealed that public cloud adoption in India is on the rise with 2 in 3 organizations planning to increase their cloud spending by 5% or more in the next 12 months.

According to the study, public cloud platforms help organizations to:

  • Enable digital customer experience
  • Improve digital operational excellence
  • Power digital ecosystem expansion
  • Accelerate digital innovation

Download this infographic to understand why Indian companies are adopting a cloud-first approach to accelerate digital transformation.

Trend Analysis

Digital Maturity in Higher Ed Tied to Improvements in Students’ Journey: Study

5129

Of your peers have already read this article.

3:00 Minutes

The most insightful time you'll spend today!

BCG and Google's 2021 study on digital maturity in higher education reveals 'going all-in' on digital helps universities become more agile and efficient in delivering education. It meets students' preferences and fosters future disruptions.

Why Higher Ed Needs to Go All-in on Digital

In the wake of the COVID-19 pandemic, the majority of students within the 18-24-year-old demographic now expect hybrid learning environments–even once we are beyond the pandemic. And a vast number of adult learners are seeking options that accommodate their work and family lives now that it’s clear that effective learning can indeed occur virtually. Implementing cloud technologies and achieving digital maturity within higher education will enable institutions to be innovative and responsive to evolving student preferences, while being prepared for future disruptions.

Exhibit 1: BCG

The state of digital maturity

In February and March 2021, Boston Consulting Group (BCG), in partnership with Google, surveyed U.S. higher education leaders on their views of the state of digital maturity in the higher education sector. This survey found that institutional and technology leaders strongly agreed that moving legacy IT systems to the cloud, centralizing and integrating data, and increasing the use of advanced analytics is necessary to make a successful digital transformation, and ultimately achieve digital maturity.

But what is digital maturity? Digital maturity—a measure of an organization’s ability to create value through digital delivery—focuses on three areas of technological advancement that drive large-scale innovation:

  1. Using cloud infrastructure
  2. Expanding access to data
  3. Using that data to improve processes through advanced analytics, such as Artificial Intelligence and Machine Learning (AI/ML)

Although university leaders agree on prioritizing digital maturity, more than 55% said they considered their schools to be “digital performers” or “digital leaders.” However, only 25% of tech leaders at these universities stated that their schools regularly use data analytics. As with corporations and governments, higher education institutions face barriers to technological innovation, such as:

  • Competing priorities to meet step-change goals and decentralized decision making
  • Budget constraints
  • Cultural resistance to change
  • Tech staff skillset gaps

Still, leaders understand that the way to overcome institutional inertia is with a strong, goal-oriented vision of what is best for the institution overall. Although only a handful of schools have reached digital maturity as we define it, others can learn a great deal from their examples. Here are the top takeaways from higher education leaders who successfully transformed their institutions:

Digital solutions can improve the student journey in many ways

Exhibit 2: BCG

As digital capabilities hold the key to dealing effectively with declining enrollment and rising costs, higher ed leaders identified four goals that are critical to improving performance:

  1. Improve the student journey
  2. Increase operational efficiency
  3. Scale computing power in advanced research
  4. Innovate education delivery

The research found that technology investments can help enhance the student journey in the recruiting and retention of students, improving digital education delivery, government funding, and donations from alumni. Digital maturity can make institutions more agile and efficient in delivering education that aligns with the changing societal norms, evolving student preferences, and future disruptions. Survey participants shared that they plan to increase the use of the cloud by more than 50% over the next three years. By shifting legacy IT systems to the cloud, institutions can increase scalability, lower the cost of ownership, and improve operational agility, while offering a more secure, long-term data storage solution.

Cloud-native software-as-a-service (SaaS) solutions provide an excellent platform for centralizing data. However, institutions that attempt to “lift and shift” their legacy systems to the cloud may encounter challenges to achieving measurable improvements in data integration and cost reduction. Higher ed leaders must realize that centralizing data and transitioning to the cloud do not happen simultaneously.

Leaders who are able to articulate a strong vision and commitment will experience a more successful technology transformation. By linking their vision to specific needs, such as more effective recruiting, leaders will find their technology investments will have a more substantial return. University presidents should base their decisions about which systems to move, when, and how on desired performance outcomes.

Big visions become a reality with small steps. Small pilot projects are an excellent way to start the journey toward digital maturity. Small steps toward a significant transformation can reduce resistance to change, build positive momentum, and produce better student outcomes. Read the full report here. If you’d like to talk to a Google Cloud expert, get in touch

More Relevant Stories for Your Company

Explainer

Delivering 10X Improvement to Risk and Regulatory Reporting Through Cloud and AI

Enterprise agility and the ability to innovate, adapt and respond quickly to the ever-changing risk and regulatory landscape is no longer a choice, but the cornerstone of successful digital transformation and commercial growth. Traditional access to and ways of managing data invariably create challenges in dealing with multiple data repositories,

Trend Analysis

2022 is a Big Year for the Gaming Industry!

Editor’s note: This post was originally published in TechPulse Belgium, where Jack Buser, Google Cloud’s Director of Game Industry Solutions, shared his trends for the industry this year. The year 2022 will hold many surprises (with a few already dropping!), but there's one near certainty: By this time next year there will

Whitepaper

The Pathway to Innovation: Migrating SAP to the Cloud

While innovation remains a top priority across all industries, many companies are not achieving the results they want from their innovation efforts. As companies migrate their SAP enterprise resource planning (ERP) systems to the cloud, technology leaders see this as an opportunity to reimagine business processes and propel their innovation

Blog

Google Introduces Cloud Career Jump Start Certification for the U.S. Underrepresented Communities

It’s no question that the cloud computing industry is booming and cloud experts are in high demand.  In 2020, 67 percent of organizations surveyed in the IDG Cloud Computing Report added new cloud roles and functions (source).  So there are plenty of cloud jobs out there, but if you don’t

SHOW MORE STORIES