One of the most vibrant elements of culture is the use of native languages and the time-honored tradition of storytelling. Anthropologists and linguists have been vocal on the role that language plays in the preservation of culture and how it contributes to the appreciation of heritage. 

Unfortunately, of the more than 7,000 languages that are spoken around the globe, nearly 3,000  are at risk of disappearing. In fact, it’s estimated that on average a language becomes extinct every fourteen days. Google Arts & Culture realized that with some creative technology and partnering with language organisations, we could help create an interactive and educational tool to help promote them.

Enter Woolaroo, an open-source photo-translation platform powered by machine learning and image recognition. The application was built on Google Cloud to encourage users to explore endangered languages around the world. Users are able to take a picture of an object in real-time, and the application returns the word in its native language, along with its pronunciation. 

Woolaroo was created with the philosophy that learning languages is greatly enhanced through engagement and context. By seeing an object in its environment, it’s easier to retain the information and then use it more naturally in conversation. 

With the help of Googlers, Woolaroo was launched in 10 languages, including Calabrian Greek, Louisiana Creole, Maori and Yiddish. During the conception stage of the app, teams from Partner Innovation and Google Arts & Culture put out an open call to the rest of Google to see what lesser-known languages our employees spoke. They then worked with the individuals that responded to develop dictionaries that were reviewed by partner institutions to ensure translations were correct and consistent. 

Woolaroo uses Google Cloud Vision API, which derives insights from images using AutoML or pre-trained models to quickly classify images into millions of predefined categories. This makes AI accessible and useful to more people as AutoML automates the training of these machine learning models.

Our team at Google Arts & Culture creates immersive experiences for people to learn about art, history, culture and more. We are committed to supporting the preservation of heritage and cultural landmarks – including spoken language – through the use of modern technology. The magic of Woolaroo is that it is open source, which means any person or organisation can use it to build something for their own endangered language. To learn about the efforts Google Arts & Culture is involved in, download the Google Arts & Culture app or visit our blog.

First, What is Google Cloud Talent Solution?

Cloud Talent Solution is a service that brings machine learning to the job search experience, returning high quality results to job seekers far beyond the limitations of typical keyword-based methods. Once integrated with your job content, Cloud Talent Solution automatically detects and infers various kinds of data, such as related titles, seniority, and industry.

Show Me How it Works

Try it online now.

Show Me an Example of Who’s Using It

There’s a number of enterprises leveraging this service. Here are a few easy-to-watch examples

Watch how FedEx Ground Employs Google Cloud Talent Solution

Read how Johnson & Johnson is Reimagining Recruiting with Jibe and Google

How Much Does it Cost?

Ok, Let’s See How it Works

There’s a lot we can learn from combining technology with science to help support the development of amazing discoveries. By using an AI system to predict protein shapes, we have the potential to accelerate research in every field of biology. Inside every cell in your body, billions of tiny molecular machines are hard at work. They are what allow your eyes to detect light, your neurons to fire, and the ‘instructions’ in your DNA to be read.

These intricate machines are known as proteins.

The protein folding puzzle

Protein folding is something that occurs naturally so that proteins become biologically functional, but it’s a complex process that sometimes fails. For decades, scientists have been trying to find a method to reliably predict a protein’s structure from its sequence of amino acids so we can better understand how proteins work.

The challenge? There are over 200 million known distinct proteins. Each one has a unique 3D shape that determines how it works and what it does. Because there are so many sequences and determining their 3-D structure experimentally is so time-consuming and expensive, scientists only know the exact structure of a tiny fraction of the proteins. And these experimental methods still fall far short of reliable statistical accuracy.

Deepmind’s gigantic leap

In 2020, Alphabet’s artificial intelligence research arm, DeepMind, made a massive breakthrough in predicting protein structures using a deep learning model called AlphaFold.

AlphaFold is trained on publicly available data consisting of about 170,000 protein structures, and is the first computational method that can regularly predict the 3D shape of a protein, at scale with a high degree of accuracy.

AlphaFold has already sent waves throughout the scientific community and has demonstrated the potential for AI to aid fundamental scientific discovery. Recently, Deepmind has made AlphaFold predictions available and open source to anyone. To date, more than 500,000 researchers from 190 countries have accessed the AlphaFold protein structure database to get closer to finding life-saving cures for diseases like Leishmaniasis and Chagas.

And now Deepmind has expanded the set of available predictions by more than 200 times (from nearly 1 million to nearly 214 million) to cover almost all cataloged proteins found in nature.

Open source predictions available on Google Cloud

Together, Google Cloud and Deepmind have released this dataset of predicted protein structures for plants, bacteria, animals, and other organisms as part of the Google Cloud Public Dataset program to enable bulk downloads at no cost. That means you can also create custom queries of the dataset using BigQuery!

Running AlphaFold on Google Cloud Vertex AI

Let’s say you want to run AlphaFold on your own in order to get protein structure predictions against your own set of data. There are a few challenges to keep in mind:

• You need to set up feature engineering against genetic sequence databases
• Preprocess data
• And run those inputs against pre-trained models

All of this requires allocating CPUs or GPUs, hosting a notebook environment, and scaling up for larger experiments. It’s hard to build and configure an on-premise system or cloud server to use AlphaFold whether you just want to try it out or run it at scale as a large organization.

That’s why we’re excited to share a deep integration between Google Cloud and Deepmind. On top of the Public Datasets program we have created end-to-end code samples for AlphaFold on Vertex AI, a managed end-to-end ML platform, to help address these challenges and speed up deployment. With AlphaFold on Vertex AI, you can manage a data science or machine learning workflow in a single development environment. You get access to pre-configured compute, storage, and end-to-end production notebooks. We have removed the heavy lifting needed to set up new ML environments, automate orchestration, and manage large clusters.

The AlphaFold inference workflow can be simplified with Vertex AI: from data preparation to feature engineering and deployment. Unlike the manual set up, the orchestrator makes it possible to parallelize steps, get predictions faster, and with better tracking.

Try it out first using Vertex AI Workbench

For those of you who want to try out a simplified version of AlphaFold, we have a Colab notebook that uses no templates (homologous structures) and a selected portion of the BFD database. You can deploy right on Vertex AI Workbench, which lets you specify a custom container image that we’ve already created for you. You’ll be able to:

1. Configure access to genetic databases
2. Configure GPU acceleration
3. Search against genetic databases
4. Use the pre-processed results as inputs to the AlphaFold model locally

In a little over an hour you can harness the power of AlphaFold to generate 3-D protein structures from amino acid sequences.

Run hundreds of experiments reliably using Vertex AI Pipelines

For organizations that want to run a full blown version of AlphaFold for many protein folding experiments a week, you’ll want an ML pipeline orchestrator. The AlphaFold Batch Inference solution is a set of code samples that uses Vertex AI Pipelines to support hundreds of concurrent inference pipelines with higher throughput to help you run experiments at scale. The solution uses Vertex AI Pipelines as an orchestrator and runtime, Vertex ML Metadata for metadata and artifacts, and Cloud Filestore to manage databases.

Because it’s built on Vertex AI Pipelines, you can automate, monitor, and experiment with interdependent parts of an ML workflow. The minimized inference elapsed times mean what normally would take you days, can now take you hours.

The solution includes two example pipelines:

1. The universal pipeline solution mirrors the exact logic in DeepMind’s open source inference script but decoupled into discrete tasks so you can run the same experiments faster, more efficiently, and with better tracking.

2. The customized pipeline solution shows you how to further optimize the inference workflow by parallelizing feature engineering steps so you can plug in your own database sources.

You get example components, pipelines, and notebooks to start, analyze, and recompile pipelines on different GPUs.

The AlphaFold Vertex AI Workbench solution is great for experimental use, while the AlphaFold Batch Inference solution on Vertex AI Pipelines is great for doing protein folding at scale with a strong process for reproducibility and tracking.

Now go forth and save the world!

Okay maybe that’s a bit hyperbolic, but this is inspiring stuff! What started as a 50 year challenge, to the discovery of AlphaFold, to being able to run it on Google Cloud, researchers, developers, and science enthusiasts now have access to one of the most pivotal advancements in the medical world. Even a non-specialist can easily use a Vertex AI notebook to exercise a simplified version of AlphaFold. The next answers to the mysteries of life and discovery of disease treatments have never felt more attainable. With these no-cost solutions to run AlphaFold on Vertex AI and the Public Dataset, you can help propel us in this worldwide endeavor.

Learn more about healthcare and life sciences solutions on Google Cloud here.

If you have feedback or want to share your experience with me, reach out to me at @stephr_wong.

While you likely know that data science is the practice of making data useful, you may not have a clear landscape around the tools that can aid each stage of the data science workflow as you use machine learning to tackle your challenges.

Read on to discover the six broad areas that are critical to the process of making data useful, and some corresponding Google Cloud products and services for those areas.

https://youtube.com/watch?v=EQvLUMjz-g4%3Fenablejsapi%3D1%26

Data engineering 

Perhaps the greatest missed opportunities in data science stem from  data that exists somewhere, but hasn’t been made accessible for use in further analysis. Laying the critical foundation for downstream systems, data engineering involves the transporting, shaping, and enriching of data for the purposes of making it available and accessible.

Data ingestion and data preprocessing on Google Cloud

Here we consider data ingestion as moving data from one place to another, and data preparation the process of transformation, augmentation, or enrichment prior to consumption. Global scalability, high throughput, real-time access, and robustness are common challenges in this stage. For scalable, real-time, and batch data processing, look into building data ingestion and preprocessing pipelines with Dataflow, a managed Apache Beam service. There’s a reason why Dataflow is called the backbone of analytics on Google Cloud. 
If you’re looking for a scalable messaging system to help you ingest data, consider Cloud Pub/Sub, a global, horizontally scalable messaging infrastructure. Cloud Pub/Sub was built using the same infrastructure component that enabled Google products, including Ads, Search, and Gmail, to handle hundreds of millions of events per second. 
If you want an easy way to automate data movement to BigQuery, a serverless data warehouse on Google Cloud, look into the BigQuery Data Transfer Service. For transferring data to Cloud Storage, take a look at the Storage Transfer Service. Or, for a no-code data ingestion and transformation tool, check out Data Fusion, which has over 150 preconfigured connectors and transformations. In addition to Dataflow and Data Fusion for data preparation, Spark users may want to look at related products and features for Spark on Google Cloud.

Data storage and data cataloging on Google Cloud

For structured data, consider a data warehouse like BigQuery, or any of the Cloud Databases (relational ones like Cloud SQL and NoSQL ones like Cloud BigTable and Cloud Firestore). For unstructured data, you can always use Cloud Storage. You may also want to consider a data lake. For data discovery, cataloging, and metadata management, consider Data Catalog. For a unified solution, take a look at Dataplex, which integrates a unified data management solution with an integrated analytics experience.

Learn more about data engineering on Google Cloud

• Explore the data engineering learning path
• Discover reference patterns
• Get certified by Google Cloud as a Professional Data Engineer

Data Analysis 

From descriptive statistics to visualizations, data analysis is where the value of data starts to appear.

Data exploration, data preprocessing, and data insights

Data exploration, a highly iterative process, involves slicing and dicing data via data preprocessing before data insights can start to manifest through visualizations or simply via simple group-by, order-by operations. One hallmark of this phase is that the data scientist may not yet know which questions to ask about the data. In this somewhat ephemeral phase, a data analyst or scientist has likely uncovered some aha-moments, but hasn’t shared them yet. Once insights are shared, the flow enters the Insights Activation stage, where those insights become used to guide business decisions, influence consumer choices, or become embedded in other applications or services. 

On Google Cloud, there are many ways to explore, preprocess, and uncover insights in your data. If you are looking for a notebook-based end-to-end data science environment, check out Vertex AI Workbench, which enables you to access, analyze, and visualize your entire data estate: from structured data at the petabyte-scale in SQL with BigQuery, to processing data with Spark on Google Cloud and its serverless, auto-scaling, and GPU acceleration capabilities. As a unified data science environment, Vertex AI Workbench also makes it easy to do machine learning with TensorFlow, PyTorch, and Spark, with built-in MLOps capabilities.

Finally, if your focus is on analyzing structured data from data warehouses and insight activation for business intelligence, you may want to also consider using Looker, with its rich interactive analytics, visualizations, dashboarding tools, and Looker Blocks to help you accelerate your time-to-insight.

Learn more about data analysis on Google Cloud

• Learn about Vertex AI Workbench for a Jupyter-based fully managed notebook environment
• Learn about how you can use BigQuery for petabyte-scale data analysis
• Learn about Spark on Google Cloud
• Discover the data analyst learning path
• Explore reference patterns for common analytics use cases

Model development

From linear regression to XGBoost, from TensorFlow to PyTorch, the model development stage is where machine learning starts to provide new ways of unlocking value from your data. Experimentation is a strong theme here, with data scientists looking to accelerate iteration speed between models without worrying about infrastructure overhead or context-switching between tools for data analysis and tools for productionizing models with MLOps. 

To solve these challenges, once again, as a Jupyter-based fully managed, scalable, and enterprise-ready environment, Vertex AI Workbench makes it easy as the one-stop-shop for data science, combining analytics and machine learning, including Vertex AI services. Apache Spark, XGBoost, TensorFlow, and PyTorch are just some of the frameworks supported on Vertex AI Workbench. Vertex AI Workbench makes managing the underlying compute infrastructure needed for model training easy with the ability to scale vertically and horizontally, and with idle timeouts and auto shutdown capabilities to reduce unnecessary costs. Notebooks themselves can be used for distributed training and hyperparameter optimization, and they include Git integration for version control. Due to the significant reduction in context switching required, data scientists can build and train models 5x faster using Vertex AI Workbench than when using traditional notebooks. 

With Vertex AI, custom models can be trained and deployed using containers. You can take advantage of pre-built containers or custom containers to train and deploy your models.

For low-code model development, data analysts and data scientists can use SQL with BigQuery ML to train and deploy models (including XGBoost, deep neural networks, and PCA),  directly using BigQuery’s built-in serverless, autoscaling capabilities. Behind-the-scenes, BigQuery ML leverages Vertex AI to enable automated hyperparameter tuning, and explainable AI. For no-code model development, Vertex AI Training provides a point-and-click interface to train powerful models using AutoML, which comes in multiple flavors: AutoML Tables, AutoML Image, AutoML Text, AutoML Video, and AutoML Translation.

Learn more about model development on Google Cloud

• Learn about Vertex AI Workbench for a Jupyter-based fully managed notebook environment
• Learn more about Vertex AI

ML engineering 

Once a satisfactory model is developed, the next step is to incorporate all the activities of a well-engineered application lifecycle, including testing, deployment, and monitoring. And all of those activities should be as automated and robust as possible.

Managed datasets and Feature Store on Vertex AI provide shared repositories for datasets and engineered features, respectively, which provide a single source of truth for data and promote reuse and collaboration within and across teams. Vertex AI’s model serving capability enables deployment of models with multiple versions, automatic capacity scaling, and user-specified load balancing. Finally, Vertex AI Model Monitoring provides the ability to monitor prediction requests flowing into a deployed model and automatically alert model owners whenever the production traffic deviates beyond user-defined thresholds and previous historical prediction requests.

MLOps is the industry term for modern, well engineered ML services, with scalability, monitoring, reliability, automated CI/CD, and many other characteristics and functions that are now taken for granted in the application domain. The ML engineering features provided by Vertex AI are informed by Google’s extensive experience deploying and operating internal ML services. Our goal with Vertex AI is to provide everyone with easy access to essential MLOps services and best practices.

Learn more about ML engineering and MLOps on Google Cloud

• Follow the guides, tutorials and documentation for Vertex AI
• Watch this video to learn more about Vertex AI
• Discover the data scientist/machine learning engineer learning path
• Get certified as a Professional ML Engineer

Insights activation 

The insights activation stage is where your data has now become useful to other teams and processes. You can use Looker and Data Studio to enable use cases in which data is used to influence business decisions with charts, reports, and alerts.

Data can also influence customer decisions and as a result increase usage or decrease churn, for example. Finally, the data can also be used by other services to drive insights; these services can run outside Google Cloud, inside Google Cloud on Cloud Run or Cloud Functions, and/or using Apigee API Management as an interface.  

Learn more about insights activation on Google Cloud

• Watch this video to learn about building interactive ML apps using Looker and Vertex AI
• Learn about Looker, and Looker solutions for eCommerce, Digital Media and more
• Discover a gallery of interactive dashboards created with Data Studio 
• Watch this video to understand the difference between Cloud Run and Cloud Functions

Orchestration 

All of the capabilities discussed above provide the key building blocks to a modern data science solution, but a practical application of those capabilities requires orchestration to automatically manage the flow of data from one service to another. This is where a combination of data pipelines, ML pipelines, and MLOps comes into play.  Effective orchestration reduces the amount of time that it takes to reliably go from data ingestion to deploying your model in production, in a way that lets you monitor and understand your ML system.

For data pipeline orchestration, Cloud Composer and Cloud Scheduler are both used to kick off and maintain the pipeline. 

For ML pipeline orchestration, Vertex AI Pipelines is a managed machine learning service that enables you to increase the pace at which you experiment with and develop machine learning models and the pace at which you transition those models to production. Vertex Pipelines is serverless, which means that you don’t need to deal with managing an underlying GKE cluster or infrastructure. It scales up when you need it to, and you pay only for what you use. In short, it lets you just focus on building your data science pipelines. 

Learn more about orchestration on Google Cloud

• Read more about Cloud Composer for Airflow-based pipelines
• Try some example notebooks on Github with Vertex AI Pipelines
• Learn different ways to trigger Vertex AI Pipeline runs
• Read the whitepaper on Practitioners Guide to MLOps: A framework for continuous delivery and automation of machine learning

Summary

Google Cloud offers a complete suite of data management, analytics, and machine learning tools to generate insights from data. Want to learn more? Check out the following resources:

• Building the data science driven organization from the first principles

Special thanks to the following contributors to this blogpost: Alok Pattani, Brad Miro, Saeed Aghabozorgi, Diptiman Raichaudhuri, Reza Rokni.