Make Your Data Useful with Google Cloud Products and Services
3237
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
While you likely know that data science is the practice of making data useful, you may not have a clear landscape around the tools that can aid each stage of the data science workflow as you use machine learning to tackle your challenges.
Read on to discover the six broad areas that are critical to the process of making data useful, and some corresponding Google Cloud products and services for those areas.
https://youtube.com/watch?v=EQvLUMjz-g4%3Fenablejsapi%3D1%26
Data engineering
Perhaps the greatest missed opportunities in data science stem from data that exists somewhere, but hasn’t been made accessible for use in further analysis. Laying the critical foundation for downstream systems, data engineering involves the transporting, shaping, and enriching of data for the purposes of making it available and accessible.
Data ingestion and data preprocessing on Google Cloud
Here we consider data ingestion as moving data from one place to another, and data preparation the process of transformation, augmentation, or enrichment prior to consumption. Global scalability, high throughput, real-time access, and robustness are common challenges in this stage. For scalable, real-time, and batch data processing, look into building data ingestion and preprocessing pipelines with Dataflow, a managed Apache Beam service. There’s a reason why Dataflow is called the backbone of analytics on Google Cloud.
If you’re looking for a scalable messaging system to help you ingest data, consider Cloud Pub/Sub, a global, horizontally scalable messaging infrastructure. Cloud Pub/Sub was built using the same infrastructure component that enabled Google products, including Ads, Search, and Gmail, to handle hundreds of millions of events per second.
If you want an easy way to automate data movement to BigQuery, a serverless data warehouse on Google Cloud, look into the BigQuery Data Transfer Service. For transferring data to Cloud Storage, take a look at the Storage Transfer Service. Or, for a no-code data ingestion and transformation tool, check out Data Fusion, which has over 150 preconfigured connectors and transformations. In addition to Dataflow and Data Fusion for data preparation, Spark users may want to look at related products and features for Spark on Google Cloud.
Data storage and data cataloging on Google Cloud
For structured data, consider a data warehouse like BigQuery, or any of the Cloud Databases (relational ones like Cloud SQL and NoSQL ones like Cloud BigTable and Cloud Firestore). For unstructured data, you can always use Cloud Storage. You may also want to consider a data lake. For data discovery, cataloging, and metadata management, consider Data Catalog. For a unified solution, take a look at Dataplex, which integrates a unified data management solution with an integrated analytics experience.
Learn more about data engineering on Google Cloud
- Explore the data engineering learning path
- Discover reference patterns
- Get certified by Google Cloud as a Professional Data Engineer
Data Analysis
From descriptive statistics to visualizations, data analysis is where the value of data starts to appear.
Data exploration, data preprocessing, and data insights
Data exploration, a highly iterative process, involves slicing and dicing data via data preprocessing before data insights can start to manifest through visualizations or simply via simple group-by, order-by operations. One hallmark of this phase is that the data scientist may not yet know which questions to ask about the data. In this somewhat ephemeral phase, a data analyst or scientist has likely uncovered some aha-moments, but hasn’t shared them yet. Once insights are shared, the flow enters the Insights Activation stage, where those insights become used to guide business decisions, influence consumer choices, or become embedded in other applications or services.
On Google Cloud, there are many ways to explore, preprocess, and uncover insights in your data. If you are looking for a notebook-based end-to-end data science environment, check out Vertex AI Workbench, which enables you to access, analyze, and visualize your entire data estate: from structured data at the petabyte-scale in SQL with BigQuery, to processing data with Spark on Google Cloud and its serverless, auto-scaling, and GPU acceleration capabilities. As a unified data science environment, Vertex AI Workbench also makes it easy to do machine learning with TensorFlow, PyTorch, and Spark, with built-in MLOps capabilities.
Finally, if your focus is on analyzing structured data from data warehouses and insight activation for business intelligence, you may want to also consider using Looker, with its rich interactive analytics, visualizations, dashboarding tools, and Looker Blocks to help you accelerate your time-to-insight.
Learn more about data analysis on Google Cloud
- Learn about Vertex AI Workbench for a Jupyter-based fully managed notebook environment
- Learn about how you can use BigQuery for petabyte-scale data analysis
- Learn about Spark on Google Cloud
- Discover the data analyst learning path
- Explore reference patterns for common analytics use cases
Model development
From linear regression to XGBoost, from TensorFlow to PyTorch, the model development stage is where machine learning starts to provide new ways of unlocking value from your data. Experimentation is a strong theme here, with data scientists looking to accelerate iteration speed between models without worrying about infrastructure overhead or context-switching between tools for data analysis and tools for productionizing models with MLOps.
To solve these challenges, once again, as a Jupyter-based fully managed, scalable, and enterprise-ready environment, Vertex AI Workbench makes it easy as the one-stop-shop for data science, combining analytics and machine learning, including Vertex AI services. Apache Spark, XGBoost, TensorFlow, and PyTorch are just some of the frameworks supported on Vertex AI Workbench. Vertex AI Workbench makes managing the underlying compute infrastructure needed for model training easy with the ability to scale vertically and horizontally, and with idle timeouts and auto shutdown capabilities to reduce unnecessary costs. Notebooks themselves can be used for distributed training and hyperparameter optimization, and they include Git integration for version control. Due to the significant reduction in context switching required, data scientists can build and train models 5x faster using Vertex AI Workbench than when using traditional notebooks.
With Vertex AI, custom models can be trained and deployed using containers. You can take advantage of pre-built containers or custom containers to train and deploy your models.
For low-code model development, data analysts and data scientists can use SQL with BigQuery ML to train and deploy models (including XGBoost, deep neural networks, and PCA), directly using BigQuery’s built-in serverless, autoscaling capabilities. Behind-the-scenes, BigQuery ML leverages Vertex AI to enable automated hyperparameter tuning, and explainable AI. For no-code model development, Vertex AI Training provides a point-and-click interface to train powerful models using AutoML, which comes in multiple flavors: AutoML Tables, AutoML Image, AutoML Text, AutoML Video, and AutoML Translation.
Learn more about model development on Google Cloud
- Learn about Vertex AI Workbench for a Jupyter-based fully managed notebook environment
- Learn more about Vertex AI
ML engineering
Once a satisfactory model is developed, the next step is to incorporate all the activities of a well-engineered application lifecycle, including testing, deployment, and monitoring. And all of those activities should be as automated and robust as possible.
Managed datasets and Feature Store on Vertex AI provide shared repositories for datasets and engineered features, respectively, which provide a single source of truth for data and promote reuse and collaboration within and across teams. Vertex AI’s model serving capability enables deployment of models with multiple versions, automatic capacity scaling, and user-specified load balancing. Finally, Vertex AI Model Monitoring provides the ability to monitor prediction requests flowing into a deployed model and automatically alert model owners whenever the production traffic deviates beyond user-defined thresholds and previous historical prediction requests.
MLOps is the industry term for modern, well engineered ML services, with scalability, monitoring, reliability, automated CI/CD, and many other characteristics and functions that are now taken for granted in the application domain. The ML engineering features provided by Vertex AI are informed by Google’s extensive experience deploying and operating internal ML services. Our goal with Vertex AI is to provide everyone with easy access to essential MLOps services and best practices.
Learn more about ML engineering and MLOps on Google Cloud
- Follow the guides, tutorials and documentation for Vertex AI
- Watch this video to learn more about Vertex AI
- Discover the data scientist/machine learning engineer learning path
- Get certified as a Professional ML Engineer
Insights activation
The insights activation stage is where your data has now become useful to other teams and processes. You can use Looker and Data Studio to enable use cases in which data is used to influence business decisions with charts, reports, and alerts.
Data can also influence customer decisions and as a result increase usage or decrease churn, for example. Finally, the data can also be used by other services to drive insights; these services can run outside Google Cloud, inside Google Cloud on Cloud Run or Cloud Functions, and/or using Apigee API Management as an interface.
Learn more about insights activation on Google Cloud
- Watch this video to learn about building interactive ML apps using Looker and Vertex AI
- Learn about Looker, and Looker solutions for eCommerce, Digital Media and more
- Discover a gallery of interactive dashboards created with Data Studio
- Watch this video to understand the difference between Cloud Run and Cloud Functions
Orchestration
All of the capabilities discussed above provide the key building blocks to a modern data science solution, but a practical application of those capabilities requires orchestration to automatically manage the flow of data from one service to another. This is where a combination of data pipelines, ML pipelines, and MLOps comes into play. Effective orchestration reduces the amount of time that it takes to reliably go from data ingestion to deploying your model in production, in a way that lets you monitor and understand your ML system.
For data pipeline orchestration, Cloud Composer and Cloud Scheduler are both used to kick off and maintain the pipeline.
For ML pipeline orchestration, Vertex AI Pipelines is a managed machine learning service that enables you to increase the pace at which you experiment with and develop machine learning models and the pace at which you transition those models to production. Vertex Pipelines is serverless, which means that you don’t need to deal with managing an underlying GKE cluster or infrastructure. It scales up when you need it to, and you pay only for what you use. In short, it lets you just focus on building your data science pipelines.
Learn more about orchestration on Google Cloud
- Read more about Cloud Composer for Airflow-based pipelines
- Try some example notebooks on Github with Vertex AI Pipelines
- Learn different ways to trigger Vertex AI Pipeline runs
- Read the whitepaper on Practitioners Guide to MLOps: A framework for continuous delivery and automation of machine learning
Summary
Google Cloud offers a complete suite of data management, analytics, and machine learning tools to generate insights from data. Want to learn more? Check out the following resources:
Special thanks to the following contributors to this blogpost: Alok Pattani, Brad Miro, Saeed Aghabozorgi, Diptiman Raichaudhuri, Reza Rokni.
4900
Of your peers have already watched this video.
11:30 Minutes
The most insightful time you'll spend today!
Driving Business Transformation in Manufacturing, Industrial, and Transportation Using Google Cloud and AI/ML
Google Cloud partners closely with manufacturing, industrial, and transportation organizations to drive business transformation.
In this video, Mandeep Waraich, Head of Product – Industrial AI, Google Cloud, shares customer stories as well as Google Cloud’s differentiated AI products and solutions.
Waraich covers the current state of automation and industrial efficiency and how artificial intelligence is revealing an entirely new universe of possibilities.
He also speaks about Google Cloud’s approach to bringing these AI technologies to the market, and Google Cloud’s “deploy anywhere” methodology that helps achieve the impact of AI at a global enterprise scale.
3314
Of your peers have already watched this video.
19:00 Minutes
The most insightful time you'll spend today!
Breakthrough Insights Across the Life Sciences Value Chain
New science and patient centricity require increasing reliance on data and deeper insights.
Accenture’s life sciences customers have been demanding answers to this need, and want a solution with an open platform.
Enter INTIENT, Accenture’s life sciences platform that is powered by Google Cloud.
INTIENT generates insights throughout the life sciences value chain from early research through pivotal clinical trials all the way through to delivering care to patients.
In this video, Kevin Julian, Senior Managing Director, Life Sciences, Accenture and Michael Stapleton, Senior Industry Executive, Accenture, discuss enables ground-breaking possibilities through applied intelligence, including the development of personalized therapeutics tailored to a patient’s genetic makeup, the utilization of real world data/evidence in clinical trials, better identification of patients for trials, development of digital therapeutics, and improved patient outcomes — in other words, better regimen adherence.
6087
Of your peers have already watched this video.
48:30 Minutes
The most insightful time you'll spend today!
Can Your Data Warehouse Handle a 100-Trillion Row Query?
Today’s enterprise demands from data go far beyond the capabilities of traditional data warehousing and for many leaders, the need to digitally transform their businesses is a key driver for data analytics spending.
Businesses want to make real-time decisions from fresh information as well as make future predictions from their data in order to remain competitive.
In this video, Jordan Tigani, Director of Product Management, Google BigQuery reveals the power of Google Cloud’s modern data warehouse, BigQuery, that helps businesses make informed decisions quickly.
In addition, he talks about how big Google BigQuery can get. He shares examples of how one customer ran a query against a giant table of 100 trillion rows. “I think it was something like 19 petabytes of data scanned. It took about took about 20 minutes. It used 39,000 slots, which is about 20,000 cores,” says Tigani.
He also shares examples of how businesses, such as online retailer, Zulily generate real business benefits from being able to query large datasets faster, and more easily than ever–without having to invest time managing infrastructure.
Finally, Amir Aryanpour, Technical Architect, Channel 4, talks abouut how connecting connecting Google BigQuery to other solutions with the Google Cloud Platform, including storage, data visualisation, and a sentiment analysis engine, among others, helped the company.
Smooth AI Adoption for Companies involves Three Principled Phases!
4089
Of your peers have already read this article.
4:00 Minutes
The most insightful time you'll spend today!
Business leaders see much in Artificial Intelligence, including new ways to save money, serve customers, and figure out what to build. What’s often tougher is deciding how to engage. That’s understandable, since getting into AI often raises questions about costs, data integrity, project length, and similar issues of planning and execution.
For CIOs concerned with the deployment and use of IT at their companies, these are critical issues. Here is one way to think about this complexity, by breaking it down into three different areas; Early Automation, Learning and working, and System Views.
First, a bit of good news and myth dispelling for readers who think AI is some far off promise, still in the labs and not safe for work. The reality is that AI is very much here, and almost everyone in your workplace is using it every day. People work with AI when they use Google Search, Photos, the autocomplete feature in Docs, live translation in Pixel phones, and many other areas. It is in the products of many other companies too.
“IDC reports that the market for AI software, hardware and services is expected to break the $500 billion mark in 2023.”
Additionally, it’s increasingly clear that AI can be adapted as a virtuous process, not just a one-off purchase (though there is much to recommend that.) During a recent Alphabet earnings call, CEO Sundar Pichai noted that “investments in AI will be key” to its near-term strategy, with new techniques that make it faster and easier to train and build AI for a number of uses. Additionally, the company is offering AI-driven “insights, new tools, and automation” to its advertising clients. The striking thing in this was the way that developing AI in one area could lead to growth in many others.
So, how does an IT leader foster a growth process like this for their stakeholders? By leading people through the well-established stages of Awareness, Learning, and Extension. Here’s what I mean.
Awareness: Early Automation
Consumer-facing AI is particularly strong in communications functions like voice recognition, translation, and writing tips. It’s similar in business uses: One of the most effective early instances of AI in the workplace has been Contact Center AI (CCAI), which manages basic customer communications, automatically answering common questions and prioritizing calls that require human assistance. It is doing what automation has always done best, automating the rote stuff and leaving the higher-value imaginative activity to people. It has been used by governments, retailers, telecommunications companies, and others, in a wide variety of use cases.
These and similar language-centric products, like DocAI for extracting information from things like invoices, receipts, or AI that extracts information from business contracts, have a number of benefits. For one, the investment is relatively easy to control, unlike with a research project leading to a formal launch. The payoff is also clearer. In the case of contact centers, in particular, the automation relieves stress, wins loyalty and slows disaffection in a high-turnover area. In both cases, successful results build allies in the business, who can testify to the earlier benefits when it’s time to take on something more complex.
Perhaps best of all, creating interest in basic AI services for business, right now, means people become engaged in learning more, since they see the early benefits and wonder what else might be done.
Learning: The Human Factor
The Natural Language Processing (NLP) that goes into these ready made, “out of the box” AI products can, not surprisingly, be used on much more sophisticated levels. Twitter, for example, processes 400 billion different events in real time, and its staff queries this trove using advanced NLP, answering questions and improving customer experiences.
There is clearly an enormous gap between Call Center AI and processing Twitter’s 400 billion events per day, but it’s not noticed enough how quickly that gap is closing. Look at how many products, partners, and training resources have emerged in the past few years. The gap makes sense, insofar as both the means of AI (like large data sets, good algorithms, and sufficient computing) and the value of AI, are new.
Increasingly, as AI is incorporated into standard enterprise tools like spreadsheets and analytic tools, easier to use AI becomes a skill within reach for many (even as the advanced end becomes more complex, meaning this ease of use process will continue for some time.)
It’s so new that AI skills demand isn’t met by conventional education means, creating lots of good opportunities for both nonstandard skills training, and in-house learning in the workforce. Companies offering AI skills training could well gain a competitive advantage and retain staff better.
Extension: Building System Views
When a new technology lands and gains in popularity, people seek to find new uses for it, or build connections among its different uses. Networked computing is one example, but think also of the way cars were soon followed by trucks and fire engines, or the way the data services on wireless phones soon morphed into the App Economy. If something is useful, people look for ways to grow it.
How will AI grow? My colleague Dominik Wee recently wrote about ways that AI will soon change supply chains, change product design, and improve sustainability. Most interestingly, he talked about how customers in manufacturing were realizing savings and gaining insights when once separate quality control data was combined with system wide views of the quality process.
There are several reasons to think AI will promote many such system views. For one thing, successful AI promotes the collection of data from more places, at greater frequency, since that leads to insight (and the cost of data collection is dropping.) Additionally, AI is good at spotting patterns and interactions that are not currently known. As well, AI is used in prediction and scenario planning, which leads to better understanding of how large-scale systems interact.
This comes at a time when we have more ways of seeing the world, from satellites, sensors, social media, and much more. We have more awareness of interactions, and a demand to understand them, in everything from the supply chain crisis, to human rights and sourcing regulations, or in the business realities of partnering, and serving customers in all sorts of ways, online and in the physical world.
Whether by coincidence or design, the Age of AI is also an age when organizations see themselves more accurately with a rich web of connections, with their choices and actions having more resonance than ever. That awareness is both a competitive tool, and a call to greater responsibility, potentially affording more customer loyalty and a more satisfied workforce for those who get it right.
That transformation won’t happen everywhere overnight, but it seems to be happening at all sorts of companies. And the trend for AI to assist, to be studied and grown, and to provide a richer understanding of the world, is happening every time someone touches this technology, at whatever level they need.
3306
Of your peers have already watched this video.
16:30 Minutes
The most insightful time you'll spend today!
Engage and Translate: Text and Audio Chat in 100+ Languages
As users worldwide connect to the internet and work remotely, it’s more important than ever to make interactive applications chat in many languages.
But when users speak 100s of languages, this quickly can get challenging. Where to start? If you are new to translation, Sarah Weldon, the Product Manager for Cloud Translation, and Dale Markowitz, an Applied AI Engineer and Developer Advocate touch on a couple of simple starters, then provide examples of how you could advance your translations once you have gone further along the learning curve.
They also share how the Translation API can quickly globalize an app, with no multilingual expertise required. Increasing language coverage can drastically increase engagement, even for internal applications. For example, when Mercy Corps integrated Translation API in their internal hub, traffic volume increased 70%. Learn about integrating the Google Cloud Translation API Advanced with a chatbot client, using the machine translation glossary feature to control a set of terms for more relevant translation.
More Relevant Stories for Your Company
How Apna is using data and AI to drive the gig economy in India
In this episode, Theo speaks to Ronak Shah, Head of Data at apna.co. As one of the largest gig economies in the world, India is seeing rising numbers among younger people joining the ecosystem. Driven by digital adoption and new ways of working, Apna saw an opportunity to help bridge
Combining IoT and Analytics to Warn Manufacturers of Line Break Downs and Increase Profitability
Oden Technologies is using the Internet of Things (IoT) to improve the factories of today. The giant network of “things” (including people) connected to each other via the Internet has the potential to reduce waste, increase efficiency, and improve safety across all walks of life. Oden is leading IoT innovation in
Baking Gets Sweeter: Build ML Models that Help Predict the Best Recipe!
Baking recipes and ML models have one thing in common—they follow a pattern. Machine Learning is all about finding pattern in data sets, you can predict what you are baking based on the core ingredients and their respective amounts! Bread, cake or cookies, watch the video to make you make
Maximize Efficiency in Document Extraction Models with Document AI Workbench GA Release
Each day, more documents are created and used across companies to make decisions. However, the value in these documents is primarily expressed as unstructured data, which makes the value difficult and manually intensive to extract and use for business processes. As the number and variety of documents used by businesses
