5044
Of your peers have already watched this video.
21:00 Minutes
The most insightful time you'll spend today!
Uber’s Story of Scaling Their App with Millions of Concurrent Requests
Uber has millions of concurrent customers who use the platform to book rides and place food delivery orders, generating billions of database transactions per day. In just a click of a button, Uber captures users’ intent which bases their fulfillment model to meet the customers’ demand, and match it to active providers that address their demand with supply.
Uber’s fulfillment platform capability is the lifeline powering every line of business and also allows rapid scaling of new verticals. Hundreds of microservices at Uber depend on this fulfillment platform as a source of truth for all their active booking, delivery or order session. The events generated by this platform are used by hundreds of offline data sets to power business decisions. Additionally over hundreds of developers at Uber extend the platform with APIs, events and codes to build 120+ unique fulfillment flows!
Watch the video to learn about Uber’s decision to move from on-prem to Google Cloud’s Cloud Spanner while still taking live orders at scale and meeting customers’ expectations.
6806
Of your peers have already watched this video.
2:30 Minutes
The most insightful time you'll spend today!
What is BigQuery?
BigQuery is Google Cloud’s enterprise data warehouse designed to help you ingest, store, analyze, and visualize big data with ease.
Organizations rely on data warehouses to aggregate data from disparate sources, process it, and make it readily available for data analysis that supports their strategic decision-making.
You can ingest data into BigQuery either through batch uploading or streaming data directly to deliver real-time insights.
As a fully-managed data warehouse, Google takes care of the infrastructure so you can focus on analyzing your data up to petabyte scale.
BigQuery supports the same Structured Query Language, or SQL, for analyzing your data, which you may be familiar with if you’ve worked with ANSI-compliant relational databases in the past.
If you’re looking to create machine learning models using your enterprise data, you can do so with BigQuery ML.
With only a few lines of SQL, you can train and execute models on your BigQuery data without needing to move it around.
When it comes time to visualize your data, BigQuery integrates with Looker, as well as several other business intelligence tools across our partner ecosystem.
Now, how do you use BigQuery?
Luckily, it’s straightforward to get up and running with BigQuery.
After creating a GCP project, you can immediately start querying public data sets, which Google Cloud hosts and makes available to all BigQuery users, or you can load your own data into BigQuery to analyze.
Interacting with BigQuery to load data, run queries, or even create ML models can be done in three different ways.
First is by using the UI and the Cloud Console. Second is by using the BigQuery command line tool. And third is by making calls to the BigQuery API, using client libraries available in several languages.
BigQuery is integrated with Google Cloud’s Identity and Access Management Service so you can securely share your data and analytical insights across the organization.
What does it cost to use BigQuery?
With BigQuery, you pay for storing and querying data and streaming inserts.
Loading and exporting data are free of charge.
Storage costs are based on the amount of data stored and have two rates based on how often the data is changing.
Query costs can be either on demand, meaning you are charged per query by the amount of data processed, or flat rate for customers who want to purchase dedicated resources.
3502
Of your peers have already downloaded this article
19:30 Minutes
The most insightful time you'll spend today!
While data has influenced major business decisions throughout the ages, the term “data warehouse” isn’t even half a century old. But it’s an important concept; by centralizing data from many disparate sources, analysts can make more informed decisions. Traditionally, this has meant collecting data in on-premises infrastructure. But as data volumes have grown, these legacy systems have not been able to keep up with changing demands: They are difficult to scale, offer no support for machine learning and artificial intelligence initiatives, and are simply too expensive to maintain.
With increased advances in technology, the introduction of more cost-effective compute and storage options have changed the data warehouse landscape. Now more than ever, businesses need an agile and efficient analytics infrastructure that allows them to derive insights at a fraction of the cost of legacy systems. Modern data warehouses can give organizations the competitive edge, offering customers fast and easy access to business intelligence while seamlessly scaling to meet increased demands.
In this ebook, based on the keynote presentation featuring Forrester Analyst Michele Goetz at the 2019 Future of Your Data Warehouse digital conference, we’ll look at the evolution of data warehousing and how a cloud infrastructure can offer immense benefits for the future.
3691
Of your peers have already watched this video.
61:30 Minutes
The most insightful time you'll spend today!
How to Move From Redshift to BigQuery Easily
Enterprise data warehouses are getting more expensive to maintain. Traditional data warehouses are hard to scale and often involve lots of data silos. Business teams need data insights quickly, but technology teams have to grapple with managing and providing that data using old tools that aren’t keeping up with demand. Increasingly, enterprises are migrating their data warehouses to the cloud to take advantage of the speed, scalability, and access to advanced analytics it offers.
With this in mind, we introduced the BigQuery Data Transfer Service to automate data movement to BigQuery, so you can lay the foundation for a cloud data warehouse without writing a single line of code. Earlier this year, we added the capability to move data and schema from Teradata and S3 to BigQuery via the BigQuery Data Transfer Service. To help you take advantage of the scalability of BigQuery, we’ve now added a service to transfer data from Amazon Redshift, in beta, to that list.
Data and schema migration from Redshift to BigQuery is provided by a combination of the BigQuery Data Transfer Service and a special migration agent running on Google Kubernetes Engine (GKE), and can be performed via UI, CLI or API. In the UI, Redshift to BigQuery migration can be initiated from BigQuery Data Transfer Service by choosing Redshift as a source.
The migration process has three steps:
- UNLOAD from Redshift to S3—The GKE agent initiates an UNLOAD operation from Redshift to S3. The agent extracts Redshift data as a compressed file, which helps customers minimize the egress costs.
- Transfer from S3 to Cloud Storage—The agent then moves data from Amazon S3 to a Cloud Storage bucket using Cloud Storage Transfer Service.
- Load from Cloud Storage to BigQuery—Cloud Storage data is loaded into BigQuery (up to 10 million files).
Watch the video to learn more!
Using Google Cloud’s Backup and DR Service with Logging and Monitoring Tools
1104
Of your peers have already read this article.
2:30 Minutes
The most insightful time you'll spend today!
Backup and DR data is a valuable business asset, and ensuring that it’s safe and accessible is essential. In particular, you want to be able to monitor your backups to ensure that the data is indeed protected and that you can quickly recover it in the event of a disaster, user error, or failed upgrade.
Users of Google Cloud’s Backup and DR service often ask:
- What are the best practices for monitoring my backup and recovery jobs?
- How can I proactively identify issues and troubleshoot?
- How can I create alerts and be notified about important events?
The good news is that Google Cloud Backup and DR now integrates with Cloud Logging and Cloud Monitoring tools. Now, you can monitor backup events, jobs, appliance health, resource consumption and user actions in the same way you monitor other Google Cloud workloads and services.
With this integration, you can now:
- Monitor events related to backup and restore – With detailed Google Cloud Backup and DR service event logs in Cloud Logging, you can now use log based alerts to create a customisable notification, in near-real time. This lets you monitor important events such as backup job failures, jobs that did not run, local backup storage saturation, and network failures.
- Configure fine grained alerting – You can write custom event queries on a wide variety of dimensions such as event severity, application type, application name, job type, job name, error message and many more. This gives a lot of flexibility to define alerts for specific events or conditions within a system, thus reducing noise and helping you identify and fix issues easily.
- Get notified on your preferred notification channel – Google Cloud offers seven pre-built notification channels that let you receive customisable notifications directly over email, SMS, Slack or the Google Cloud mobile app. Alternatively, you can integrate with your own monitoring and event management tools using webhooks or Pub/Sub.
- Derive useful insights to troubleshoot issues – With Log Analytics, you can use BigQuery to query your data using SQL queries and generate operational insights, which can help you reduce time spent troubleshooting. For example, you can analyze the key reasons that backup and restore jobs fail for a given application or application type.
The integration also lets you proactively discover deeper issues related to backup and recovery, such as capacity-related issues or certain job failures that happen periodically, by monitoring recurring events in the logs over time. With log-based metrics, you can:
- Create trend charts to track important metrics such as the number of backup or restore jobs that failed during a day/week/month
- Receive a notification when the number of occurrences crosses a threshold, for example receive a notification if a snapshot pool saturates more than five times a week
- Monitor trends in data, such as latency values in logs, and receive a notification if the values change in an unacceptable way.
Getting started
The time to learn that your backup failed should never be when you go to restore. By integrating our Backup and DR service with Cloud Monitoring and Cloud Logging, you can get valuable assurances about the health of your backup and business continuity processes with the same tools that you use to manage other Google Cloud workloads. To get started, check out the Backup and DR event logs page. You can also watch a video that shows you how to set up and use the service and configure some custom alerts.
How Google Cloud & NGIS’ Partnership Powers Sustainability & Responsible Sourcing for Consumer Brands
6201
Of your peers have already read this article.
2:00 Minutes
The most insightful time you'll spend today!
In the competitive world of consumer goods, sustainability matters more than ever. In recent Google survey, 82% of consumers said sustainability is more top of mind now than it was before COVID-191 and 78% said that big businesses have a role to play in helping to fight climate change.2 As well as delivering on customers’ heightened expectations, sustainable business practices can help organizations reduce waste and lower operational costs and even help attract top talent.
Not only is sustainability good for business performance and the bottom line, it’s great for the planet, too. And it’s something we care deeply about at Google. Since our earliest days, we have focused on developing services that significantly improve the lives of billions of people while operating our business in an environmentally sustainable way. In 2007, we were the first major company to become carbon neutral, and in 2017 we were the first to achieve 100% renewable energy. Today, we proudly operate the cleanest cloud in the industry. We’ve also been powering humanitarian, scientific, and environmental initiatives studying geospatial information and making it available for analysis via Google Earth Engine.
Powering the future of responsible sourcing
How does all this relate to CPG supply chains? The world relies on raw materials like palm oil, soy, and cocoa to produce the items we consume every day—such as coffee, chocolate, frozen foods, shampoo, toothpaste, cosmetics, and even household cleaning products. Yet as demand for these materials continues to grow, forests are under threat. Change is needed.
As sustainability moves further into the spotlight, in-demand crops face increased supplier scrutiny. Environmentally conscious companies want to know what percentage of their raw materials is sourced from deforestation-free suppliers—and how they can improve that number. Until recently, many CPG brands have found it hard to get real-time, reliable visibility into operations at a local supplier level, globally.
Google Cloud, in partnership with NGIS, is helping brands gain a deeper understanding of raw material sourcing practices across supplier networks, so they can improve supplier performance and compliance in the fight against deforestation. The TraceMark solution, developed by NGIS, uses Google Earth Engine and BigQuery to analyze and visualize how suppliers behave over time.
Google Earth Engine is the world’s largest archive of open Earth data. It combines a multi-petabyte catalog of satellite imagery and geospatial datasets with planetary-scale analysis capabilities to help people detect changes, map trends, and quantify differences on the Earth’s surface.With Earth Engine, organizations can analyze the potential social, and environmental impacts of the decision they make.
Together, TraceMark, Google Earth Engine and BigQuery help companies visualize, monitor, and measure the impact of suppliers’ farming practices. The process starts with Earth Engine, which aggregates and harmonizes planetary data into images that it has been collecting for decades. Then, boundary maps for different suppliers are created and NGIS applies climate data science and machine learning to turn pixels into insights. Finally, BigQuery and Vertex AI -Google Cloud’s unified artificial intelligence platform are used to produce supplier scoring, which is integrated into downstream ERP and procurement systems to support more informed decision-making.
These efforts support the responsible sourcing of raw materials by making supply chains more agile, transparent and traceable.
“NGIS is thrilled to work with Google Cloud and its partners to enable business accountability for sustainable practices at all levels of their supply chain,” said Nathan Eaton, Executive Director, NGIS. “Google Earth Engine provides unique geospatial capability that [gives] leaders visibility and control over their environmental footprint and that of their suppliers in a way that was not previously possible.”
Unilever commits to working towards a deforestation-free supply chain
Unilever is a great example of a company committed to using technology in the quest to become more sustainable. Since 2020, Unilever has partnered with Google Cloud to use data for eco-friendly decision-making, particularly when it comes to sustainable commodity sourcing.
By combining the power of cloud computing with satellite imagery and AI, we’re helping Unilever build a more holistic view of the forests, water cycles, and biodiversity that intersect its supply chain. By gaining a complete picture of these ecosystems, Unilever can detect deforestation while simultaneously prioritizing critical areas of forest and habitats in need of protection.
The cleanest cloud for your sustainable transformation
We are excited to bring Google Earth Engine and NGIS to our CPG customers to help them meet their sustainability goals. Looking ahead, we’re committed to furthering our own ambitious sustainability goals and empowering CPG brands with the technology to do more for our environment and our shared future.
Contact your cloud seller to learn more about Tracemark sustainable sourcing solutions and how Google Cloud can help you advance your sustainability initiatives.
Explore TraceMark on the Google Cloud Marketplace
To know more about how we are helping CPGs transform digitally read this ebook
1. Google/C Space, BR, FR, DE, IN, MX, U.K., U.S., qualitative survey activity, n = 528, Nov. 24–Nov. 26, 2020.
2. Google/Ipsos, Google Sustainability, BR, FR, DE, IN, JP, U.K., U.S., n=16,959 online population 18–70, July 2021.
More Relevant Stories for Your Company
Google Launches Open Saves To Power Gaming Platforms Scale to User Demands
Many of today’s games are rich, immersive worlds that engage the audience in ways that make a gamer a part of a continuing storyline. To create these persistent experiences, numerous storage technologies are required to ensure game data can scale to the standards of gamers’ demands. Not only do game
Case Study: When Database Choice Powers New Revenue-driving Product Features
Editor’s note: Streak makes a CRM add-on for Gmail, and recently adopted Cloud Spanner to take advantage of its scalability and SQL capabilities to implement a graph data model. Read on to learn about their decision, what they love about the system, and the ways in which it still needs
Creating and Using Storage Buckets for Your Data Needs
So, you want to store objects on the cloud? But you're really new to Google Cloud or Cloud Storage and would like someone to walk you through the process step by step? Today is your lucky day! I will help you understand the big steps involved in setting up and
Manhattan Associates’ Seamless Migration to Google Cloud SQL
Editor’s note: Manhattan Associates provides transformative, modern supply chain and omnichannel commerce solutions. It enhanced the scalability, availability, and reliability of its software-as-a-service through a seamless migration to Google Cloud SQL for MySQL. Geopolitical shifts and global pandemics have made the global supply chain increasingly unpredictable and complex. At Manhattan
