6792
Of your peers have already watched this video.
2:30 Minutes
The most insightful time you'll spend today!
What is BigQuery?
BigQuery is Google Cloud’s enterprise data warehouse designed to help you ingest, store, analyze, and visualize big data with ease.
Organizations rely on data warehouses to aggregate data from disparate sources, process it, and make it readily available for data analysis that supports their strategic decision-making.
You can ingest data into BigQuery either through batch uploading or streaming data directly to deliver real-time insights.
As a fully-managed data warehouse, Google takes care of the infrastructure so you can focus on analyzing your data up to petabyte scale.
BigQuery supports the same Structured Query Language, or SQL, for analyzing your data, which you may be familiar with if you’ve worked with ANSI-compliant relational databases in the past.
If you’re looking to create machine learning models using your enterprise data, you can do so with BigQuery ML.
With only a few lines of SQL, you can train and execute models on your BigQuery data without needing to move it around.
When it comes time to visualize your data, BigQuery integrates with Looker, as well as several other business intelligence tools across our partner ecosystem.
Now, how do you use BigQuery?
Luckily, it’s straightforward to get up and running with BigQuery.
After creating a GCP project, you can immediately start querying public data sets, which Google Cloud hosts and makes available to all BigQuery users, or you can load your own data into BigQuery to analyze.
Interacting with BigQuery to load data, run queries, or even create ML models can be done in three different ways.
First is by using the UI and the Cloud Console. Second is by using the BigQuery command line tool. And third is by making calls to the BigQuery API, using client libraries available in several languages.
BigQuery is integrated with Google Cloud’s Identity and Access Management Service so you can securely share your data and analytical insights across the organization.
What does it cost to use BigQuery?
With BigQuery, you pay for storing and querying data and streaming inserts.
Loading and exporting data are free of charge.
Storage costs are based on the amount of data stored and have two rates based on how often the data is changing.
Query costs can be either on demand, meaning you are charged per query by the amount of data processed, or flat rate for customers who want to purchase dedicated resources.
Beyond Traditional Learning: AI-based Online Learning Platform and Google Cloud Solutions Push Learners to Get Ahead

10318
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
The combination of a vital need for IT experts among businesses and a digital skills gap is making lifelong learning increasingly critical. Beyond professional development, learning new skills offers additional rewards from building peer connections to boosting your creativity. That’s why in 2020 Krishna Deepak Nallamilli and I launched KIMO.ai to reimagine how people approach learning, especially in developing markets. Our team is building the artificial intelligence needed to generate individual learning paths through a wide range of quality digital learning content.
Google Cloud and its Startup Program have been instrumental in connecting our team with the tools, people, processes, and best practices to grow our business.
Existing learning platforms lack engagement
Outside of traditional education settings, massive open online learning courses (MOOCs)—often modeled after university courses—can provide a flexible and affordable way to upskill or reskill. But the vast majority of people who participate in MOOC programs fail to complete courses. Based on our research, the challenge with existing learning platforms is a lack of engagement, primarily caused by limited direction on which skills to learn, whether AI, fintech, blockchain, or other in-demand disciplines.
We’ve also received feedback that many corporate learning management systems–developed as online training systems to upskill employees–tend to be poorly designed and time-consuming to use.
Overall, a significant challenge with most existing learning platforms is that they’re generic. For example, suppose you’re interested in learning about AI. In that case, you need AI-related coursework that applies to your industry and the job you want because AI in medicine is vastly different from AI in financial services. Today’s online learning options typically take a one-size-fits-all approach and fail to capture the nuances of what learners really need to get ahead.
Building a future-proof learning platform
The commitment to highly personalized, accessible learning inspired KIMO.ai, a platform that we believe is the future of education. Depending on your goals, current skills, location, and other factors, our AI-based platform will identify which coursework (and where to find those classes) to build the skills you need. The more personalized, relevant learning recommendations even take into account people’s preferences for podcasts, MOOCs, books, articles, videos, courses, publications, and more.
In a mix of cooperation and competition we call “coopetition,” KIMO.ai will regularly recommend courses from other established online learning systems if, based on our automated assessment, it’s the best option for a learner. There’s also the option to access free content only.
Google cultural alignment fosters trust
Our platform started with one developer exploring NLP models and Google APIs. As we’ve grown our team and launched our beta to 110,000 users in developing markets, we discovered there is a lot of interest in our platform, and we believe we can make a significant impact. In feedback forums, we also learned that we need to focus our efforts on the mobile experience to improve engagement since 99% of the beta testers use mobile devices.
Beyond our team’s high level of trust in Google Cloud solutions, our team also appreciates the cultural alignment with Google. We value Google’s developer-centric approach and rely on tools like Dataflow for batch data processing and Cloud TPU to reliably run machine learning models with AI services on Google Cloud. We also build all of our deployments on Google Kubernetes Engine (GKE), which makes it easy to manage all our containerized workloads
On the front end, Google App Engine makes it easy to deploy apps and experiment, and it integrates seamlessly with Firebase for authentication and more. BigQuery is our serverless data warehouse that efficiently scales to support the millions of articles, videos, and other learning resources we need to analyze to provide the targeted coursework recommendations our learners require.
As we grow our business having a network of trusted advisors is also extremely valuable. By working closely with DoIT International, the 2020 Google Cloud Global Reseller Partner of the Year, our team has access to their cloud, Kubernetes, and machine learning expertise. DoIT has already helped us quickly resolve IT issues and create analytics dashboards that give us insights to continually enhance our services.
Building for a growing industry
The dynamic edtech market is growing rapidly and estimated to become an $11B industry by 2025. We’re proud to be part of the next wave of personalized education that has the potential to empower people in developing markets and beyond to grow their skills with coursework tailored to their exact needs and how they like to learn. This year, we will deliver our platform to at least 400,000 more people. We’re excited to see how they use it and where it takes them.
If you want to learn more about how Google Cloud can help your startup, visit our Startup Program application page here to get more information about our program, and sign up for our communications to get a look at our community activities, digital events, special offers, and more.
3887
Of your peers have already watched this video.
2:30 Minutes
The most insightful time you'll spend today!
How Can an IT Manager Get into the Machine Learning Game?
It’s a rare technology forecast that doesn’t include machine learning and AI. But for IT managers these technologies can seem like a world away. This can be worrying for many IT managers who feel like they are being left out of the future.
It needn’t be this way.
One of the largest challenges data scientists and analysts face when undertaking a big data or a machine learning initiative is cleaning and preparing data. Some estimate that data scientists spend up to 80% of their time wrangling with data so that it can be used.
This space is primed for IT managers. With tools like Google Dataprep, IT managers can get into the machine learning and big data space and simultaneously fill a deeply-felt business need.
With tools like Google Dataprep, IT managers can help data teams clean and prepare structured and unstructured data and have it ready for analysis. It’s a browser-based service, with user-friendly drag-and-drop interface. It automatically detects mis-matched data, and predicts transformation you might need.
See how easy it is to use Google Dataprep. Watch this short video.
Countries can Tackle Food Wastage Crisis Using Google’s Data Analytics

4972
Of your peers have already read this article.
4:00 Minutes
The most insightful time you'll spend today!
With over ⅓ of the food in the USA ending up as waste according to the USDA, it is a compelling challenge to address this travesty. What will happen to hunger, food prices, trash reduction, water consumption, and overall sustainability when we stop squandering this abundance?

Beginning with the departure from the farm to the back of the store, the freshness clock continues to run. Grocers work very hard to purchase high quality produce items for their customers and the journey to the shelf can take a toll in both quality and remaining shelf life. Suppliers focus on delivering their items through the arduous supply chain journey to the store with speed and gentle handling. The baton is then passed to the store to unload and present the items to customers with care to sell through each lot significantly before the expiration or sell by date. This is to ensure that the time spent in the customer’s home is ample to ensure a great eating experience as well. Food waste is a farm to fork problem with opportunity at every step of the chain, but today we will focus on the segment that the grocery industry oversees.
With the complexities of weather, geopolitical issues, distribution, sales variability, pricing, promotions, and inventory management, it seems daunting to impact waste. Fortunately, data analytics and machine learning in the cloud is a powerful weapon in the fight against food waste. Data Scientists harness knowledge to draw meaning from data turning that data into decision driving information.
One key Google has been working on to accelerate value is to break down data silos and leverage machine learning to realize better outcomes, using our Google Data Cloud platform. This enables better planning through demand forecasting, Inventory management, assortment planning, and dynamic pricing and promotions.
That sounds great but how does it work?
Let’s walk through a day in the life journey to see how the integrated Google Data Cloud platform can change the game for good. Our friendly fictitious grocer FastFreshFood is committed to selling high quality perishable items to their local market. Their goal is to minimize food waste and maximize revenue by selling as much perishable fresh food as possible before the sell by date. Our fictitious grocer in partnership with Google Cloud could build a solution that will take a significant bite out of their food waste volume and better satisfy customers.
- Sales through the register and online are processed in real time with Datastream, Dataflow to keep an accurate perpetual inventory by minute of every single item.
- A Demand forecasting model using machine learning algorithms in BigQuery then identifies needs for back room replenishment, so Direct Store Delivery and daily store Distribution Centers manage ordering more efficiently to ensure just the right amount of each product each day.
- Realtime reporting dashboards in Looker with alerting capabilities enable the system to operate with strong associate support and understanding. The reporting suite shows inventory levels into the future, daily orders, and at risk items.
The pricing algorithm could also alert store leadership concerning any items that will not sell through and suggest real time in store specials resulting in zero waste at shelf and maximized revenue.

This approach is not just for perishable categories and is a pattern that works well for in-store produced items and center store items. The key point is that by bringing ML/AI to difficult business problems grocers are reinventing what is possible for both their profitability and sustainability.
The technical implementation of this design pattern in Google Cloud leverages Datastream, Dataflow, BigQuery and Looker products, it is detailed in a technical tutorial accompanying this blog post.
In partnership with Google Cloud, retailers can solve complex problems with innovative solutions to achieve higher quality, lower cost, and provide great customer experiences. To learn more from this and other use cases, please visit our Design Patterns website.
Curious to learn more?
We’re excited to share what we know about tackling food waste at Google, a topic we’ve been working on in the last decade as we’ve embarked on reducing our own food waste in our operations in over 50 countries in the world. The Google Food for Good team works exclusively on Google Cloud Platform with our partners on this topic. Two additional articles below.
Silos are for food, not for data – tackling food waste with technology
This business Cloud blog directly addresses information silos that currently exist across many nodes in the food system and how to break down cultural and organizational barriers to sharing.
“Unsiloing” data to work toward solving food waste and food insecurity
This follow-on technical Cloud blog articulates the path to setting up data pipelines, translating between data sets (not everyone calls a tomato a tomato!) and making sense of emergent insights.
5196
Of your peers have already watched this video.
5:00 Minutes
The most insightful time you'll spend today!
How to Build and Maintain Spark and Hadoop Clusters Quickly, Easily and Inexpensively
How often have you struggled to create Spark and Hadoop clusters only to have them become unstable?
Building and maintaining clusters that are the building blocks of big data analytics should not be a time-consuming, repetitive, and expensive process. But with the current set of tools, it often is.
Google Cloud Dataproc solves this. It is a fast, easy-to-use, fully-managed cloud service for running Spark and Hadoop clusters in a simple, more cost-efficient way.
Operations that used to take hours or days take now seconds or minutes instead, and you pay only for the resources you use (with per-second billing).
Watch this five-minute demo to find out how—with just a few clicks—you can set up clusters that build and heal themselves. You’ll find out how to creating a large Dataproc cluster with preemptible VMs, which help run large clusters at a low total cost.
You Can Quantify and Maximize Value of Your Org’s AI/ML and Analytics Teams!

4414
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
Investing in Artificial Intelligence (AI) can bring a competitive advantage to your organization. If you’re in charge of an AI or Data Science team, you’ll want to measure and maximize the value that you’re providing. Here is some advice from our years of experience in the field.
A checklist to embark on a project:
As you embark on projects we’ve found it’s good to have the following areas covered:
- Have a customer. It’s important to have a customer for your work, and that they agree with what you’re trying to achieve. Be sure to know what value you’re delivering to them.
- Have a business case. This will rely on estimates and assumptions, and may take no more than a few minute’s work. You should revise this, but always know what justifies your team’s effort, and what you (and your customer) expect to get in return.
- Know what process you will change or create. You’ll want to put your work in production, so you have to be clear about what business operations are changing or created around your work and who needs to be involved to make it happen
- Have a measurement plan. You’ll want to show that ongoing work is impacting some relevant business indicator. Measure and show incremental value. The goal of these measurements is to establish what has changed because of your project that would otherwise not have changed. Be sure to account for other factors like seasonality or other business changes that may affect your measurements.
- Use all the above to get your organization’s support for your team and your work.
What measures to use?
As you start the work, what measures and indicators can you use to show that your team’s work is useful for your organization?
How many decisions you make. A major function of ML is to automate and optimize decisions: which product to recommend, which route to follow, etc. Use logs to track how many decisions your systems are making.
Changes to revenue or costs. Better and quicker decisions often lead to increased revenue or savings. If possible, measure it directly, otherwise estimate it (for example fuel costs saved from less distance traveled, or increased purchases from personalized offers).
As an example, the Illinois Department of Employment Security is using Contact Center AI to rapidly deploy virtual agents to help more than 1 million citizens file unemployment claims. To measure success the team tracked the two outcomes: (1) the number of web inquiries and voice calls they were able to handle, and (2) the overall cost of the call center after the implementation. Post implementation, they were able to observe more than 140,000 phone and web inquiries per day and over 40,000 after-hours calls per night. They also anticipate an estimated annual cost savings of $100M based on an initial analysis of IDES’s virtual agent data (see more in the link to case study).
Implementation costs. The other side of increased revenue or savings, is to put your achievements in the context of how much they cost. Show the technology costs that your team incurs and, ideally, how you can deliver more value, more efficiently.
How much time was saved. If the team built a routing system then it saved travel time, if it built an email classifier then it saved reading time, etc. Quantify how many hours were given back to the organization thanks to the efficiency of your system.
In the medical field, quicker diagnostics matter. Johns Hopkins University’s Brain Injury Outcomes (BIOS) Division has focused on studying brain hemorrhage aiming to improve medical outcomes. The team identified the time to insights as a key metric in measuring business success. They experimented with a range of cloud computing solutions like Dataflow, Cloud Healthcare API, Compute Engine, and AI Platform for distributed training to accelerate iterations. As a result, in their recent work they were able to accelerate insights from scans from approximately 500 patients from 2,500 hours to 90 minutes.
How many applications your team supports. Some of your organization’s operations don’t use ML (say reconciling financial ledgers) but others do. Know how many parts of your organization benefit from the optimization and automation your team builds.
User experience. You may be able to measure your customer’s experience: fewer complaints, better reviews, reduced latency, more interactions, etc. This is valid both for internal and external stakeholders. At Google we measure usage and regularly ask for feedback on any internal system or process.
One of our customers, The City of Memphis, is using VisionAI and ML to tackle a common but very challenging issue: identifying and addressing potholes. The implementation team identified the percentage increase of potholes identified as one of the key metrics along with accuracy and cost savings. The solution captures video footage from it’s public vehicles and leverages Google Cloud capabilities like Compute Engine, AI Platform, and BigQuery to automate the review of videos. The project increased pothole detection by 75% with over 90% accuracy. By measuring and demonstrating these outcomes, the team proved the viability of a cost-effective, cloud-based machine learning model and is looking into new applications of AI and ML that will further improve city services and help it build a better future for its 652,000 residents.
Acknowledgements
Filipe and Payam would like to thank our colleague and co-author Mona Mona (AI/ML Customer Engineer, Healthcare and lifesciences) who contributed equally to the writing.
More Relevant Stories for Your Company

Delivering analysis-ready financial data at scale
Refinitiv, is a leading provider of financial market data. It serves over 40,000 institutions and operates in 190 countries. “We're constantly looking at solving some of the most difficult problems our users face when accessing new data sets as well as onboarding new sources that will be relevant for workflows

“It’s an Astonishing Difference”: What Data Operation Execs say About Google Cloud’s Data Warehouse
The popularity of meal kit delivery services has surged in recent years as consumer attitudes toward home cooking and grocery shopping have shifted. As a pioneer in the category, Blue Apron helps its customers create incredible home cooking experiences by sending culinary-driven recipes with high-quality ingredients and step-by-step instructions straight to customers’

Vodafone Leverages Google Cloud to Aid COVID-19 Frontline with Anonymized Insights on Population Mobility
Editor’s note: When Europe’s largest mobile communications company, Vodafone, was asked by the European Commission to help understand population movement across the European Union and the UK to help fight COVID-19, it was able to provide anonymized mobile network-based insights to answer the call. Here’s how Vodafone, with the support

Vector Search: The Tech Powering Billions of Search Results for Google Users
Recently, Google Cloud partner Groovenauts, Inc. published a live demo of MatchIt Fast. As the demo shows, you can find images and text similar to a selected sample from a collection of millions in a matter of milliseconds: Image similarity search with MatchIt Fast Give it a try — and either select a preset






