Establishing a mature MLOps practice to build and operationalize ML systems can take years to get right.  We recently published our MLOps framework to help organizations come up to speed faster in this important domain.  

As you start your MLOps journey, you might not need to implement all of these processes and capabilities. Some will have a higher priority than others, depending on the type of workload and business value that they create for you, balanced against the cost of building or buying processes or capabilities. 

To help ML practitioners translate the framework into actionable steps, this blog post highlights some of the factors that influence where to begin, based on our experience in working with customers. 

The following table shows the recommended capabilities (indicated by check marks) based on the characteristics of your use case, but remember that each use case is unique and might have exceptions. (For definitions of the capabilities, see the MLOps framework.)

Your use case might have multiple characteristics. For example, consider a recommender system that’s retrained frequently and that serves batch predictions. In that case, you need the data processing, model training, model evaluation, ML pipelines, model registry, and metadata and artifact tracking capabilities for frequent retraining. You also need a model serving capability for batch serving.

In the following sections, we provide details about each of the characteristics and the capabilities that we recommend for them.

Pilot

Example: A research project for experimenting with a new natural language model for sentiment analysis.

For testing a proof of concept, your focus is typically on data preparation, feature engineering, model prototyping, and validation. You perform these tasks using the experimentation and data processing capabilities. Data scientists want to set up experiments quickly and easily and track and compare them. Therefore, you need the ML metadata and artifact tracking capability in order to debug, to provide traceability and lineage, to share and track experimentation configurations, and to manage ML artifacts. For large-scale pilots, you might also require dedicated model training and evaluation capabilities.

Mission-critical

Example: An equities trading model where model performance degradation in production can put millions of dollars at stake.

In a mission-critical use case, failure with the training process or production model has a significant negative impact on the business (a legal, ethical, reputational, or financial risk). The model evaluation capability is important to identify bias and fairness, as well as to provide explainability of the model. Additionally, monitoring is essential to assess the quality of the model during training and to assess how it performs in production. Online experimentation lets you test newly trained models against the one in production using a controlled environment before you replace the deployed model. Such use cases also need a robust model governance process to store, evaluate, check, release, and report on models and to protect against risks. You can enable model governance by using the model registry and metadata and artifact tracking capabilities. Additionally, datasets and feature repositories provide you with high-quality data assets that are consistent and versioned.

Reusable and collaborative

Example: Customer Analytic Record (CAR) features that are used across various propensity modeling use cases.

Reusable and collaborative assets allow your organization to share, discover, and reuse AI data, source code, and artifacts. A feature store helps you standardize the processes of registering, storing, and accessing features for training and serving ML models. Once features are curated and stored, they can be discovered and reused by multiple data science teams. Having a feature store helps you avoid reengineering features that already exist, and saves time on experimentation. You can also use tools to unify data annotation and categorization. Finally, by using ML metadata and artifacts tracking, you help provide consistency, testability, security and repeatability of the ML workflows. 

Ad hoc retraining

Example: An object detection model to detect various car parts, which needs to be retrained only when new parts are introduced. 

In ad hoc retraining, models are fairly static and you do not retrain them except when the model performance degrades. In these cases, you need data processing, model training, and model evaluation capabilities to train the models. Additionally, because your models are not updated for long periods, you need model monitoring. Model monitoring detects data skews, including schema anomalies, as well as data and concept drifts and shifts. Monitoring also lets you continuously evaluate your model performance, and it alerts you when performance decreases or when data issues are detected. 

Frequent retraining

Example: A fraud detection model that’s trained daily in order to capture recent fraud patterns.

Use cases for frequent retraining are ones where model performance relies on changes in the training data. The retraining might be based on time intervals (for example, daily or weekly), or it could be triggered based on events like when new training data becomes available. For this scenario, you need ML pipelines to connect multiple steps like data extraction, preprocessing, and model training. You also need the model evaluation capability to ensure that the accuracy of the newly trained model meets your business requirements. As the number of models you train grows, both a model registry and metadata and artifact tracking help you keep track of the training jobs and model versions.

Frequent implementation updates

Example: A promotion model with frequent changes to the architecture to maximize conversion rate.

Frequent implementation updates involve changes to the training process itself. That might mean switching to a different ML framework, such as changing the model architecture (for example, LSTM to Attention) or adding a data transformation step in your training pipeline. Such changes in the foundation of your ML workflow require controls to ensure that the new code is functional and that the new model matches or outperforms the previous one. Additionally, the CI/CD process accelerates the time from ML experimentation to production, as well as reducing the possibility for human error. Because the changes are significant, online experimentation is necessary to ensure that the new release is performing as expected. You also need other capabilities such as experimentation, model evaluation, model registry, and metadata and artifact tracking to help you operationalize and track your implementation updates. 

Batch serving

Example: A model that serves weekly recommendations to a user who has just signed up for a video-streaming service.

For batch predictions, there is no need to score in real time. You precompute the scores and you store them for later consumption, so latency is less of a concern than in online serving. However, because you process a large amount of data at a time, throughput is important. Often batch serving is a step in a larger ETL workflow that extracts, pre-processes, scores, and stores data. Therefore, you need the data processing capability and ML pipelines for orchestration. In addition, a model registry can provide your batch serving process with the latest validated model to use for scoring.

Online serving

Example: A RESTful microservice that uses a model to translate text between multiple languages. 

Online inference requires tooling and systems in order to meet latency requirements. The system often needs to retrieve features, to perform inference, and then to return the results according to your serving configurations. A feature repository lets you retrieve features in near real time, and model serving allows you to easily deploy models as an endpoint. Additionally, online experiments help you test new models with a small sample of the serving traffic before you roll the model out to production (for example, by performing A/B testing).

Get started with MLOps using Vertex AI

We recently announced Vertex AI, our unified machine learning platform that helps you implement MLOps to efficiently build and manage ML projects throughout the development lifecycle. You can get started using the following resources: 

• MLOps: Continuous delivery and automation pipelines in machine learning
• Getting started with Vertex AI
• Best practices for implementing machine learning on Google Cloud

^{Acknowledgements: I’d like to thank all the subject matter experts who contributed,  including Alessio Bagnaresi, Alexander Del Toro, Alexander Shires, Erin Kiernan, Erwin Huizenga, Hamsa Buvaraghan, Jo Maitland, Ivan Nardini, Michael Menzel, Nate Keating, Nathan Faggian, Nitin Aggarwal, Olivia Burgess, Satish Iyer, Tuba Islam, and Turan Bulmus. A special thanks to the team that helped create this, Donna Schut, Khalid Salama, and Lara Suzuki, and Mike Pope for his ongoing support.}

Editor’s note: The post is part of a series highlighting our awesome partners, and their solutions, that are Built with BigQuery

In the field of producing engaging video content such as ads, many marketers ignore the power of data to improve their creative efforts to meet the consumers’ need for personalized messages. The demand for creative tech to efficiently personalize is real as marketers need personalized video Ads to reach their audience with the right message at the right time. Data, Insights and Technology are the main ingredients to deliver this value while ensuring security and privacy requirements are met. The Connected-Stories team partnered with Google Cloud to build a platform for Ad personalization. Google Data Cloud and BigQuery are at the forefront to assimilate data, leverage ML models, create personalized ads, and capitalize on real-time intelligence as the core features of the Connected-Stories NEXT platform.

Connected-Stories NEXT is an end-to-end creative management platform to develop, serve, and optimize interactive video and display ads that scale across any channel. The platform ingests first-party data to create custom ML models, measure numerous third-party data points to help brands develop unique customer journeys and create videos that their data signals can drive. An intelligent feedback loop passes real-time data back, enabling brands to make data-driven and actionable video ads that take the brand’s campaigns to the next level.

The core use case of the NEXT platform revolves around collecting user’s interaction data and optimizing for precision and speed to create an actionable Ad experience that is personalized for each user. The platform processes complex data points to create interactive data visualizations that allow for accurate analysis. The platform uses Vertex AI to access managed tools, workflows, and infrastructure to build, deploy, and scale ML models that have improved the accuracy to identify segments for further analysis.

The platform ingests 200M data events with peaks and valleys of activity. These events are processed to generate dashboards that enable users to visualize metrics based on filters in real-time. These dashboards have high performance requirements in terms of a responsive user interface under constantly changing data dimensions.

Google Cloud’s serverless stack coupled with limitless data cloud infrastructure has been the core to the NEXT platform’s data-driven innovation. The growing volume of data ingested, streamed and processed were scaled uniformly across the compute, storage and analytical layers of solution. A lean development team at Connected-Stories were able to focus all-in on the solution, while the serverless stack scaled, lowered attack service in terms of security and optimized the cost footprint through pay-as-you-go features.

BigQuery has been the backbone to support the vast amounts of data spreading over multiple geos resulting in workloads running at petabyte scale. BigQuery’s fully managed serverless architecture, real-time streaming, built-in machine learning and rich business intelligence capabilities distinguishes itself from a cloud data warehouse. It is the foundation needed to approach data and serve users in an unlimited number of ways. For an application with zero tolerance for failure, given its fully managed nature, BigQuery handles replication, recovery, data distributed optimization and management.

The platform’s requirements include the need for low maintenance, constantly ingesting and refreshing data and smart-tuning of aggregated data. These capabilities can be implemented by BigQuery’s materialized views feature. Materialized views are useful for precomputed views that regularly cache query results for better performance. These views possess the innate feature to read only the delta change from base tables and calculate the up-to-date aggregations. Materialized views impart faster outputs and consume fewer resources while reducing the cost footprint.

Some key considerations in using Google cloud and focusing on the Serverless stack include: quick onboarding to development, prototyping in short sprints and ease of preparing data in a rapidly changing environment. Typical considerations around low code / no code include data transformation, aggregation and reduced deployment time. These considerations are fulfilled through using serverless capabilities within Google Cloud such as PubSub, Cloud Storage, Cloud Run, Cloud Composer, Dataflow and BigQuery as described in the Architecture diagram below. The use of each of these components and services are described below.

1. Input/Ingest: At a high-level, microservices hosted in Cloud Run collect and aggregate incoming Ads events.
2. Enrichment: The output of this stage is a Pub-Sub message enriched with more attributes based on a pre-configured campaign.
3. Store: a Cloud Dataflow streaming job to create text files in Cloud Storage buckets.
4. Trigger: Cloud Composer triggers the spark jobs based on text files to process and group them to produce desired output as one record per impression, a logical group of events.
5. Deploy: Cloud Build is then used to automate all deployments.

Thus far, all Google cloud managed services work together to ingest, store and trigger the orchestration, all of which are scalable based on configurations including autoscaling capabilities.

1. Visualization: A visualization tool reads data from BigQuery to compute pre-aggregations required for each dashboard.
2. Data Model Evolution considerations: Though the solution served the purpose of creating pre-aggregations, as the data model evolved by adding a column or creating a new table, it led to recreating pre-aggregations and querying the data again. Alternatively, creating aggregate tables as an extra output of current ETLs seemed like a viable option. However, this would increase the cost and complexity of jobs. A similar situation to reprocess or update aggregated tables would occur as data is updated.

Precomputed views of data that is periodically cached are critical to reach the audience with the right message at the right time.

1. Performance: In order to increase the performance of the platform, we need to have regularly precomputed views of the data, cached .
2. Materialized Views: Consumers of these views needed faster response times, to consume fewer resources and output only the changes in comparison to a base table. BigQuery Materialized views were used to solve this very requirement. Materialized views have been highly leveraged to optimize the design resulting in lesser maintenance and access to fresh data with high performance with a relatively low technical investment in creating and maintaining SQL code.
3. Dashboards: Application dashboards pointing to the Materialized views are highly performant and provide a view into fresh data.
4. Custom Reports with Vertex AI Notebooks: Vertex AI notebooks directly read data from BigQuery to produce custom reports for a subset of customers. Vertex AI has been hugely beneficial to data analysts, where an environment with pre-installed libraries simplifies the readiness to use. Vertex AI Workbench notebooks are used to share these reports within the team allowing them to work always on the cloud without having the need to download data at any time. Besides, it increases the velocity to develop and test ML models faster.

The NEXT platform has yielded benefits such as customers having the ability to create unique consumer journeys powered by AI / ML personalization triggers, using first-party data and business intelligence tools to capitalize on real-time creative intelligence, which is a dashboard to measure campaign performance for cross-functional teams to analyze the impact of Ad content experience at a granular level. All of these while ensuring controlled access to data to enrich data without moving across clouds. The NEXT platform can keep up with increased demands for agility, scalability and reliability through the underlying usage of Google Cloud.

Partnering with Google, in the context of the Google Built with BigQuery program has surfaced the differentiated value in areas of creating interactive personalized Ads by using real-time data. In addition, by sharing this data across organizations as assets, ML models have fueled higher levels of innovation. Connected-Stories plan to deepen the penetration into the entire spectrum of services offered in the AI/ML area to enhance core functionality and provide newer capabilities to the platform.

Click here to learn more about Connected-Stories NEXT Platform capabilities.

The Built with BigQuery Advantage for ISVs

Through Built with BigQuery, launched in April ‘22 as part of Google Data Cloud Summit, Google is helping tech companies like Connected-Stories co-innovate in building applications that leverage Google’s data cloud with simplified access to technology, helpful and dedicated engineering support, and joint go-to-market programs. Participating companies can:

• Get started fast with a Google-funded, pre-configured sandbox.
• Accelerate product design and architecture through access to designated technical experts from the ISV Center of Excellence who can share insights from key use cases, architectural patterns, and best practices encountered in the field.
• Amplify success with joint marketing programs to drive awareness, generate demand, and increase adoption.

The Google Data Cloud spectrum of products and specifically BigQuery give ISVs the advantage of a powerful, highly scalable data warehouse that’s integrated with Google Cloud’s open, secure, sustainable platform. And with a huge and expanding partner ecosystem and support for multi-cloud, open source tools and APIs, Google provides technology companies the portability and extensibility they need to avoid data lock-in and exercise choice.

We thank the Google Cloud and Connected-Stories team members who co-authored the blog: Connected-Stories: Luna Catini, Marketing Director, Google: Sujit Khasnis, Cloud Partner Engineering

Google Cloud introduced Document AI to automate document processing and to streamline workflows with state-of-the-art machine learning models. With the deep neural networks, the models generalize the learning from seeing hundreds of thousands variations of the documents. But when information is missing or ambiguous on a document – like a missing address or entity name – a human may need to search for it…often on Google.

With Document AI, we are bringing the power of this “Google search” to help customers understand their documents. This means that the same Google knowledge graph technology that helps you find the name, address or phone number of your favorite restaurant can now enrich your document extraction with the right name, fully qualified address, and updated phone number.

Here is a sample payslip…

Imagine a bank employee entering this to capture a customer’s income to qualify them for a loan. When extracting information from this payslip, what employer name should she key in? She might take the time to go into Google and find the right correct legal entity name; or she might just guess and move on, potentially creating data reconciliation headaches down the line.

With Document AI, there is a better way. Our native integration with the knowledge graph means that we can deliver both the specific text from the payslip as well as Google’s best understanding of the actual name of the company that operates at this address. This is an important step to translate from “what has been said” on a document to “what does it mean”.  By normalizing the value as you process millions of documents, you are improving accuracy and consistency at the beginning of the data processing workflow, making downstream integration, data analytics and business intelligence tasks at ease.

How EKG Enrichment Works

In a nutshell, Knowledge Graph is a knowledge base that uses a graph data model to integrate interlinked entities, including objects, events, processes or abstracted concepts. Google announced its Knowledge Graph in 2012 as a way to organize information from the Web and to enhance Search results. Different from Google Knowledge Graph, Cloud EKG (Enterprise Knowledge Graph) focuses on entities that are more relevant to enterprise customers, such as organization, product, people, locations, etc.

In EKG, every node is called an entity. Each entity in the graph represents an object, such as an organization. EKG aggregates all the information about a thing into a single entity, thus each entity represents a distinct and identifiable real world concept. The uniqueness of these entities in the graph is one of the reasons that make EKG useful. The edges between nodes are called relationships. When representing attributes, the relationships can be considered as properties, such as the name of a company, the price of a product, etc. When representing relationships, they connect entities in the graph, such as the CEO of a company, the seller of a product.

Entity linking, as its name suggests, is the task of assigning a unique identity to entities mentioned in text, images or videos. In the context of EKG, it connects the text mentions to entities in the graph. Under the hood, the recognition takes both the mention and its context information into consideration for linking to the best matching candidates in the graph.

Entity Enrichment, is essentially linking entities in EKG to documents, and using the attributes of linked entities to enrich the extracted information.  The entity linking on documents is based on the understanding of both documents and the entities in the graph. First the document parser annotates entity mentions, which provides semantic meanings to the content of the document. Then Entity Linking selects a list of entities from EKG based on the types of these mentions, and matches to the best entity by comparing the attributes found in the document with the the relationships of the selected entities.

How to leverage the enrichment result

Knowledge graph enrichment is a built-in feature for Lending DocAI , Procurement DocAI and Contract DocAI today, and we are actively working on expanding it to cover more document types and parsers on the platform. To use the knowledge graph enriched values, look out for the entities fields under normalizedValue, returned by the API.

  {
      entities: [
         {
          "textAnchor": {
             "content": "Google Singapore"
          },
          ….
           "normalizedValue": {
              "text": "Google Asia Pacific, Singapore"
          }
        }
      ]
    }

To learn more, check out the Document AI webpage, and EKG Enrichment page to see the list of supported parsers and fields.

At Google, we believe anyone should be able to quickly and easily turn their artificial intelligence (AI) idea into reality. Open source software (OSS) has become increasingly important to this goal, heavily influencing the pace of innovation in AI and machine learning (ML) ecosystems. Over the last two decades, ML has transformed Google services including Search, YouTube, Assistant, and Maps, and the basis for this transformation has always been our “open first” approach through investments in projects and ecosystems like TensorFlow, Jax, and PyTorch.

These OSS efforts are important because many AI technologies rely on closed or exclusive approaches. This wall-garden approach creates high barriers to entry for developers; limits efforts to make AI explainable, ethical, equitable; and stunts innovation. We’re committed to open ecosystems, as we firmly believe no one company should own AI/ML innovation. In this blog post, we’ll explore some of Google’s most significant OSS AI and ML contributions from recent years, as well as how our commitment to open technologies can help organizations innovate faster and more flexibly.

Openness is the way to operate as an ecosystem, not a single project

Google’s OSS initiatives extend and enable AI initiatives according to three pillars:

• Access — OSS allows developers, researchers and organizations of all sizes to leverage the latest ML technology. It is a key part of democratizing innovation in ML, fostering software diversity and choice for customers, and lowering operating cost while accelerating scale for everyone.
• Transparency — Open source datasets, ML algorithms, training models, frameworks, and compilers ensure due diligence and validation by the larger community. This is paramount when it comes to ML as it bolsters reproducibility, interpretability, ensures equity, and boosts security.
• Innovation — With more access and transparency, more innovation comes naturally. Our customers and partners take advantage of open source ML toolsets and frameworks to create more innovation in the field by contributing their own OSS.

Google’s ongoing commitment to open source AI

Google’s commitment to open standards spans over two decades of OSS contributions like TensorFlow, JAX, TFX, MLIR, KubeFlow, and Kubernetes, as well as sponsorship for critical OSS data science initiatives like Project Jupyter and NumFOCUS. Initiatives like these have helped Google become the leading Cloud Native Computing Foundation (CNCF) contributor—and by building on these efforts, Google Cloud seeks to be the best platform for the OSS AI community and ecosystem.

The perils of closed technologies can emerge at many points across ML pipelines, which is why Google’s OSS strategy encompasses the entire “idea-to-production” lifecycle, from acquiring data, to training models, to managing infrastructure, to facilitating experimentation and model refinement:

Data acquisition: starting the journey from idea to production-ready ML model

The journey from an idea to a production ML model starts with data. TensorFlow Datasets not only help users acquire ready-to-use, customizable, and highly-optimized datasets (including image, audio, and text), but also provides a set of helpful APIs that make it easy for users to organize their own datasets, regardless of whether they build with TensorFlow, Jax, or other ML frameworks.

Model development and training: shortening the path from data to useful ML

OSS libraries help developers and researchers design, implement, train, test, and debug ML algorithms. Our contributors on this front include:

• The TensorFlow core framework, which offers APIs to help data scientists and developers build and train production-grade ML models on distributed and accelerated infrastructure powered by GPUs or TPUs;
• Google’s founding membership of the PyTorch Foundation, which positions us to increase adoption of ML by building an ecosystem of open-source projects with PyTorch;
• Keras, a simple and powerful ML framework, well integrated with TensorFLow, that makes it easy for developers to quickly build and train ML models, or to leverage pre-trained AI applications;
• Model Garden, which provides implementations of many state-of-the-art computer vision and natural language processing models, maintained by Google and accessible to all, alongside APIs to accelerate training and experiments;
• Jax, a lean, intuitive, and composable system that brings together automatic differentiation (Autograd) and the Accelerated Linear Algebra (XLA) optimizing compiler to offer high-performance ML for fast research and production;
• TensorFlow Hub, a repository of trained ML models ready for fine-tuning and deployment; and,
• MediaPipe open source cross-platform, which lets users leverage customizable ML solutions for live and streaming media, including text and video.

ML infrastructure management: scaling valuable models with powerful backends

Accessing and managing infrastructure for ML, especially at scale, can be a blocker for many organizations, which is why Google has invested in initiatives including:

• The TFX (or TensorFlow Extended) platform, which offers software frameworks and tooling for full MLOps deployments, helping developers with data automation, model tracking, performance monitoring, and model retraining;
• Kubeflow, which makes deployments of ML workflows on Kubernetes simple, portable and scalable; and,
• TRC (TPU Research Cloud), which gives access to a cluster of more than 1,000 Cloud TPU devices at no charge to selected researchers who publish peer-reviewed papers and/or open source code.

Experimentation and model optimization: encouraging discovery and iteration

Data, tools for model training, and infrastructure can achieve only so much without strong processes for experimentation and optimization—which is why we’ve contribution to projects like xManager, which enables anyone to run and keep track of ML experiments locally or on Vertex AI and Tensorboard, which simplifies tracking and visualizing of model performance metrics.

These areas of focus will help not only our customers but the open-source AI community as a whole, and we’re excited to share more OSS news in coming days and months. To start exploring why many organizations choose Google Cloud for their open-source AI needs, visit our “open cloud” page and be sure to register for Google Cloud Next ‘22 for all our latest news.

Thanks to all the contributors to this blog post: Matt Vasey, George Elissaios, Warren Barkley, Manvinder Singh, James Rubin, Abhishek Ratna, Thea Lamkin, Amin Vahdat, Andrew Moore, Max Sapozhnikov, Gandhi, Vikram Kasivajhula