Practical machine learning (ML) is a trial and error process. ML practitioners compare different performance metrics by running ML experiments till you find the best model with a given set of parameters. Because of the experimental nature of ML, there are many reasons for tracking ML experiments and making them reproducible including debugging and compliance.

But tracking experiments is challenging: you need to organize experiments so that other team members can quickly understand, reproduce and compare them. That adds overhead that you don’t need.

We are happy to announce Vertex AI Experiments autologging, a solution which provides automated experiment tracking for your models, which streamlines your ML experimentation

With Vertex AI Experiments autologging, you can now log parameters, performance metrics and lineage artifacts by adding one line of code to your training script without needing to explicitly call any other logging methods.

How to use Vertex AI autologging

As a data scientist or ML practitioner, you conduct your experiment in a notebook environment such as Colab or Vertex AI Workbench. To enable Vertex AI Experiments autologging, you call aiplatform.autolog() in your Vertex AI Experiment session. After that call, any parameters, metrics and artifacts associated with model training are automatically logged and then accessible within the Vertex AI Experiment console. 

Here’s  how to enable autologging in your training session with a Scikit-learn model.

# Enable autologging
aiplatform.autolog()

# Build training pipeline
ml_pipeline = Pipeline(...)

# Train model
ml_pipeline.fit(x_train, y_train)

This video shows parameters and training/post-training metrics in the Vertex AI Experiment console.

Vertex AI SDK autologging uses MLFlow’s autologging in its implementation and it supports several frameworks including XGBoost, Keras and Pytorch Lighting. See documentation for all supported frameworks. 

Vertex AI Experiments autologging automatically logs model time series metrics when you train models along multiple epochs. That’s because of the integration between Vertex AI Experiments autologging and Vertex AI Tensorboard. 

Furthermore, you can adapt Vertex AI Experiments autologging to your needs. For example, let’s say your team has a specific experiment naming convention. By default, Vertex AI Experiments autologging automatically creates Experiment Runs for you without requiring you to call `aiplatform.start_run()` or `aiplatform.end_run()`. If you’d like to specify your own Experiment Run names for autologging, you can manually initialize a specific run within the experiment using aiplatform.start_run() and aiplatform.end_run() after autologging has been enabled. 

What’s next

You can access Vertex AI Experiments autologging with the latest version of Vertex AI SDK for Python. To learn more, check out these resources :

• Documentation: Autolog data to an experiment run
• Github:  Get started with Vertex AI Experiments autologging

While I’m thinking about the next blog post, let me know if there is Vertex AI content you’d like to see on Linkedin or Twitter.

At enterprises across industries, documents are at the center of core business processes. Documents store a treasure trove of valuable information whether it’s a company’s invoices, HR documents, tax forms and much more. However, the unstructured nature of documents make them difficult to work with as a data source. We call this “dark data” or unstructured data that businesses collect, process and store but do not utilize for purposes such as analytics, monetization, etc. These documents in pdf or image formats, often trigger complex processes that have historically relied on fragmented technology and manual steps. With compute solutions on Google Cloud and Document AI, you can create seamless integrations and easy to use applications for your users. Document AI is a platform and a family of solutions that help businesses to transform documents into structured data backed by machine learning. In this blog post we’ll walk you through how to use Serverless technology to process documents with Cloud Functions, and with workflows of business processes orchestrating microservices, API calls, and functions, thanks to Workflows.

At Cloud Next 2021, we presented how to build easy AI-powered applications with Google Cloud. We introduced a sample application for handling incoming expense reports, analyzing expense receipts with Procurement Document AI, a DocAI solution for automating procurement data capture from forms including invoices, utility statements and more. Then organizing the logic of a report approval process with Workflows, and used Cloud Functions as glue to invoke the workflow, and do analysis of the parsed document.

We also open sourced the code on this Github repository, if you’re interested in learning more about this application.

In the above diagram, there are two user journeys: the employee submitting an expense report where multiple receipts are processed at once, and the manager validating or rejecting the expense report. 

First, the employee goes to the website, powered by Vue.js for the frontend progressive JavaScript framework and Shoelace for the library of web components. The website is hosted via Firebase Hosting. The frontend invokes an HTTP function that triggers the execution of our business workflow, defined using the Workflows YAML syntax. 

Workflows is able to handle long-running operations without any additional code required, in our case we are asynchronously processing a set receipt files. Here, the Document AI connector directly calls the batch processing endpoint for service. This API returns a long-running operation: if you poll the API, the operation state will be “RUNNING” until it has reached a “SUCCEEDED” or “FAILED” state. You would have to wait for its completion. However, Workflows’ connectors handle such long-running operations, without you having to poll the API multiple times till the state changes. Here’s how we call the batch processing operation of the Document AI connector:

  - invoke_document_ai:
    call: googleapis.documentai.v1.projects.locations.processors.batchProcess
    args:
        name: ${"projects/" + project + "/locations/eu/processors/" + processorId}
        location: "eu"
        body:
            inputDocuments:
                gcsPrefix:
                    gcsUriPrefix: ${bucket_input + report_id}
            documentOutputConfig:
                gcsOutputConfig: 
                    gcsUri: ${bucket_output + report_id}
            skipHumanReview: true
    result: document_ai_response

Machine learning uses state of the art Vision and Natural Language Processing models to intelligently extract schematized data from documents with Document AI. As a developer, you don’t have to figure out how to fine tune or reframe the receipt pictures, or how to find the relevant field and information in the receipt. It’s Document AI’s job to help you here: it will return a JSON document whose fields are: line_item, currency, supplier_name, total_amount, etc. Document AI is capable of understanding standardized papers and forms, including invoices, lending documents, pay slips, driver licenses, and more.

A cloud function retrieves all the relevant fields of the receipts, and makes its own tallies, before submitting the expense report for approval to the manager. Another useful feature of Workflows is put to good use: Callbacks, that we introduced last year. In the workflow definition we create a callback endpoint, and the workflow execution will wait for the callback to be called to continue its flow, thanks to those two instructions:

  - create_callback:
    call: events.create_callback_endpoint
    args:
        http_callback_method: "POST"
    result: callback_details
...
- await_callback:
    try:
        call: events.await_callback
        args:
            callback: ${callback_details}
            timeout: 3600
        result: callback_request
    except:
        as: e
        steps:
            - update_status_to_error:
              ...

In this example application, we combined the intelligent capabilities of Document AI to transform complex image documents into usable structured data, with Cloud Functions for data transformation, process triggering, and callback handling logic, and Workflows enabled us to orchestrate the underlying business process and its service call logic.

Going further 

If you’re looking to make sense of your documents, turning dark data into structured information, be sure to check out what Document AI offers. You can also get your hands on a codelab to get started quickly, in which you’ll get a chance at processing handwritten forms. If you want to explore Workflows, quickstarts are available to guide you through your first steps, and likewise, another codelab explores the basics of Workflows. As mentioned earlier, for a concrete example, the source code of our smart expense application is available on Github. Don’t hesitate to reach out to us at @glaforge and @asrivas_dev to discuss smart scalable apps with us.

In recent years, machine learning (ML) has gained tremendous popularity as a powerful tool for solving complex problems across various domains. However, building and deploying ML models at scale can be challenging, as it involves a range of tasks such as data preparation, feature engineering, model training, deployment, monitoring, maintenance and so on. 

According to “The Art of AI maturity” report published by Accenture “87% of data science projects never make it into production.” This is where MLOps comes in – it can help to address the core challenges by providing a framework for managing the entire ML lifecycle, from data collection and preparation to model development, testing, and deployment. It also reduces the time from ML model development to production and increases the success rate of ML projects.

In Google Cloud, we understand how important MLOps is to successfully productionize ML models. So we collaborated with Equinix, the world’s digital infrastructure company™ and  a leader in global colocation data center market share, with 248 data centers in 27 countries on five continents. We helped them by providing the advisory services on the MLOps best practices and architecture. 

Let’s take a sneak peek at the MLOps requirements at Equinix and the final architecture that was proposed.

What does MLOps mean for Equinix?

After multiple discovery sessions with the Equinix Team, we identified the core requirements and pain points to address in their new MLOps architecture:

• Reusability: Components such as features and pipeline components should be reused across projects to reduce costs and improve efficiency.
• Foundations: The foundations of the infrastructure, such as environments, folder structure, and project hierarchy, should be well-designed to support scalability and reliability.
• Early identification of problems: Problems should be identified early by including data validation, notifications, and retry mechanisms.
• Cost optimization: Costs should be optimized by paying only for what is used.
• Enterprise CI/CD requirements: Enterprise CI/CD requirements should be met by integrating with GitHub and GitActions.
• Scaling: The infrastructure must scale to support future growth.
• Security: Enterprise security requirements should be met in terms of IAM roles, network, etc.

MLOps Architecture

Based on the above requirements from Equinix, key design considerations were made for example – using Vertex AI Feature Store instead of Big Query for online feature serving, using DataFlow for pre-processing vs using the existing python based pre-processing and so on. After carefully assessing all the alternatives, the below reference architecture for MLOps in GCP was proposed:

Reference architecture for MLOPs using GCP is illustrated in Figure 1. This architecture includes the following pipeline stages:

• Vertex AI Workbench enables data scientists to quickly explore new ideas and develop/experiment new models. The source code is saved in GitHub repository
• Github Actions with self-hosted runners are integrated with Github as the source code repository. This enables continuous integration of code, quality and security scans. The artifacts generated during the process are saved in Artifact Registry and Google Cloud Storage. These artifacts are deployed to implement the pipeline.
• Unit tests and integration tests can be performed during the continuous integration in Github Actions with self-hosted runners. End-to-end tests are performed on demand in the continuous integration pipeline. 
• Metadata about the artifacts is generated and saved in Vertex ML Metadata.
• Automated triggers can be enabled to run pipelines. For example, one trigger is the availability of new training data. These triggers can run the model training pipeline and the new trained model can be pushed to Vertex AI Model Registry.
  • To train a new ML model with new data, the deployed Vertex AI Pipeline is executed. 
  • To train a new ML model with new implementation, a new pipeline will be deployed through CI/CD pipeline.

“The proposed architecture design covers the requirements and scenarios that we were looking for. As our AI and ML portfolio is growing in scale and complexity, it’s important to follow a clear and up-to-date architecture if we want to keep increasing the value delivered by our solutions. As part of the process, the team also acquired the skills required to fully implement it” according to Bernardo Fernandes, Data Science Senior Manager at Equinix.

Below is the snapshot of the features before and after MLOps implementation at Equinix:

As businesses increasingly rely on machine learning to gain a competitive edge, MLOps has become a critical component of their strategy. By adopting MLOps practices, organizations can achieve faster time-to-market, better performance, and higher ROI for their machine learning initiatives.

Fast track end-to-end deployment with Google Cloud AI Services (AIS)

The partnership between Google Cloud and Equinix is just one of the latest examples of how we’re providing AI-powered solutions to solve complex problems to help organizations drive the desired outcomes. To learn more about Google Cloud’s AI services, visit our AI & ML Products page.

^{We’d like to give special thanks to Nitin Aggarwal, Vijay Surampudi, Parag Mhatre and Anantha Narayanan Krishnamurthy for their support and guidance throughout the project. We are also grateful to the super awesome collaboration with the Equinix Team (Ravi Pasula and Brendan Coffey, Bernardo Fernandes, Łukasz Murawski, Jakub Michałowski, Sonia Przygocka-Groszyk, Marek Opechowski, Daria Bondara, Nila Velu, Vijay Narayanan, Dharmendra Kumar, Shailesh Sukare, Arunraj Kumar Raje, Seng Cheong Lee).}

The digital era turned the traditional formula for media and entertainment success on its head, ushering in new technologies that have changed how content is produced, distributed, experienced, and monetized. Audiences have more choice, flexibility, and power over what they consume, and today’s media companies have to embrace ongoing transformation or risk falling behind – or becoming irrelevant. 

A new wave of transformation is arriving with generative AI, a type of artificial intelligence that can interact with users in natural language and create novel data, ranging from story outlines, reports, and other text outputs to multimodal content like images, videos, and audio. Media and entertainment are inherently about content creation and creativity—so what does this new technology mean for the industry? 

At Google Cloud, we see tremendous opportunity for creative industries, from more efficient creation methods to improved user experiences. Let’s explore.

AI for media with Google Cloud

Google Cloud has a long history with large language models (LLMs) and other generative AI technologies—from their influence over the years on products like Document AI, to recent announcements like Generative AI support in Vertex AI, which lets businesses access and tune generative AI foundation models, and Generative AI App Builder, which lets developers build chatbots and other generative apps in minutes.  

We’ve helped our global media and entertainment customers with AI for personalization, search and recommendations, predictive analytics, and much more — and with generative AI now on the rise, we have some ideas to help media leaders, technologists, and creators think about and prepare to utilize powerful AI in their work. 

Three lenses on innovation in media

The media and entertainment industry is increasingly diverse and complex, with companies spanning over-the-top (OTT) subscription streaming services, 24-hour linear channels, live broadcasts of sporting events, digital journalism, traditional publishing, short-form user-generated social video, and more. More and more, the boundaries between these segments of the media industry are blurring — but common to them all is the focus on providing compelling content in an engaging audience experience that can be directly or indirectly monetized.

With this in mind, we suggest media and entertainment companies look at the application of innovative technologies like generative AI through the following three lenses:

1. Improving content creation, production, and management
2. Enhancing and personalizing audience experiences
3. Improving monetization

Improving content creation, production, and management

Generative AI democratizes many aspects of content creation, opening new ways to create written material, illustrations, sound effects, special effects, and more. Its recent maturation has been so rapid, some in the media industry have expressed concern that generative AI implies the end of creative professions. We think the opposite is more likely: just as photography, audio recordings, and computer generated images have enabled new modes of creativity, rather than making old ones obsolete, generative AI has the potential to both enable new forms of expression and enhance familiar ones. 

For example, journalists could use generative AI to speed up research by helping them synthesize and analyze large volumes of information, or to help them create initial drafts or summaries of editorial content. Film and television producers could leverage the technology to accelerate the post-production editing process, with new AI-enabled interfaces for rapidly adjusting or enhancing scene details such as lighting and color. Broadcasters could use generative AI to make vast libraries of video footage searchable and accessible for use in telling more compelling stories. The potential use cases go on and on.

Far from undermining incredible creative professions, generative AI is poised to free writers, artists, editors, and many others from the tedious and mundane aspects of their work, empowering them to focus more of their time on creativity.

Enhancing and personalizing audience experiences

Every media organization in the world today faces the reality that for most consumers, switching costs are extremely low. This puts incredible pressure on these companies to invest in delivering low-friction and compelling audience experiences that help mitigate subscribers from churning and viewers from abandoning content experiences for competitive platforms. 

Generative AI can help media companies engage and retain viewers, such as by enabling more powerful search and recommendations on their digital content platforms. With its increasingly multimodal capabilities extending from natural language to both audio and video content, generative AI is well-positioned to power more personalized audience experiences. 

Consumers often complain about “the paradox of choice” or their inability to find something interesting to watch on streaming platforms that have incredibly vast libraries of content available on demand. Imagine a not-too-distant future wherein a consumer can simply ask the content platform they’re using to help them find a specific show to watch based on mood, specific types of scenes, combinations of actors, award nominations, or practically anything they can think to ask. And that’s just the tip of the iceberg — imagine generative AI’s potential to curate, assemble, and even create personalized content for a viewer to consume!

Improving monetization

As consumers’ content consumption further expands from traditional theatrical and linear television programming to include digital offerings across an array of platforms, devices, and content types, media companies face the challenge of maintaining and improving monetization. The conventional economics and approaches to advertising and subscription models are proving, in many cases, not to deliver sufficient ROI. 

Generative AI has the potential to help media companies improve their monetization of audience experiences. As mentioned previously, enhanced personalization can play a role in mitigating churn, which in turn can help sustain and grow subscription and advertising revenues. Going beyond this, generative AI can be leveraged to drive even greater advertising revenues via more targeted, contextual, and personalized advertisements. Imagine both display and video advertisements that are generated on the fly to personalize product specifics, messaging, style, colors, and innumerable other characteristics to drive greater engagement and higher click-through rates (CTR), and thus higher advertising CPMs (cost per thousand impressions).

Coming up next

Generative AI presents a significant opportunity for media companies to fundamentally transform content creation, engagement, and monetization. Compelling services are already on the market — but there is far more to come. 

Google Cloud continues to build on its deep experience and expertise with AI, and we are committed to working with the industry to develop compelling, accessible, trusted, and responsible AI solutions that will drive meaningful business outcomes. We are excited to create the future together with our global media customers and partners across the ecosystem. To learn more about this disruptive topic, read “Debunking five generative AI misconceptions” from Google Cloud vice president of AI & Business Solutions Phil Moyer, or explore our Trusted Tester Program for generative AI.

The disruptions of 2020 elevated the importance of having the right data and insights to pivot quickly when necessary. Here’s a look at how businesses can use embedded intelligence to prepare for uncertainty and meet ever-changing customer expectations.

At work and in life, some unpredictability is always a part of the package. Ten years ago, your business might have experienced sudden product demand or a system outage that slowed down deliveries. Servers or data platforms might have run out of capacity earlier than expected.

In 2020, though, the concept of unpredictability in business reached new heights. These disruptions have elevated the importance of embedded intelligence—that is, having machine learning built into the tools people use every day, so that when a pivot is necessary, everyone has the data and insights they need at their fingertips. In fact, 2020 was so disruptive, with so many changes in customer behavior and so many ripple effects, a lot of historical data and forecasting assumptions may not be helpful in 2021. This only increases the onus on businesses to make the freshest data actionable for more of the workforce. 

A decade or so ago, the idea of embedded intelligence might have seemed like science fiction. You might remember hearing that analytics would be able to make predictions, and that technology would be able to take on the complex work of predicting retail demand or helping to create a responsive supply chain. But insufficient hardware, older architecture models, slow queries, and untrustworthy data often got in the way. 

Now, that concept has become reality as enterprise decision-making has moved from legacy tools to cloud-powered data intelligence services. Today, it’s possible to perform complex analytics tasks and obtain valuable, trusted outputs much faster than ever before. That speed and scale has allowed businesses to tackle entirely new projects and release new features and products very quickly. In addition, APIs have become a lot more intelligent, making it easy to connect siloed solutions. No matter the industry, businesses can access the technology to get to the bottom of what customers need.

Related: Top 5 trends for API-powered digital transformation in 2021

Meeting ever-changing customer expectations with embedded ML

Bringing embedded analytics to real-world uses continues to evolve, with a number of inspiring examples surfacing in the past year. As a result of the pandemic and shifting public health guidelines, many businesses didn’t know month by month if they’d be interacting with customers primarily through in-person or digital channels. And even if both channels were available, it wasn’t obvious how changing customer behaviors would net out.

At patient engagement platform Force Therapeutics, for example, daily activity on their virtual care platform went up by over 140% during the pandemic. With such a large influx of incoming data, it would have been difficult—if not impossible—for a team of humans to gather, organize, and draw insights from all of that information, especially in a timely enough manner to be of use to healthcare providers. 

To deliver the necessary care when and how it was needed, Force Therapeutics required a machine learning solution that could identify patient needs based on a wide range of data. Using an embedded analytics platform, they created an application that allowed them to monitor the progress of post-op patients, answer questions, or triage concerns remotely. The platform also enabled providers to check for spikes and anomalies, in order to identify patients who needed to come in due to a critical issue.

Amidst all of the disruption, it became clear that teamwork is essential, and that effective teamwork relies on having the right data-driven tools to get the job done.

Likewise, home delivery became a bigger part of consumers’ routines. This increased pressure on companies to adapt quickly to changes that might prevent packages from arriving on time, such as worsening weather conditions or upstream supply chain disruptions. Amidst all of the disruption, it became clear that teamwork is essential, and that effective teamwork relies on having the right data-driven tools to get the job done.

One example of this can be seen in Google Cloud customers who are using public data to accelerate their journey from data to actionable insights. Some retailers are utilizing the Google Cloud Public Datasets Program to leverage NOAA’s Global Surface Summary of the Day (GSOD) and Severe Weather Data Inventory datasets in order to better understand disruptive weather events, reroute their supply chains to prevent disruptions, and predict their in-store inventory needs to support communities as they recover from natural disasters. 

Implementing ML without the complexity

The idea of embedded ML has been hyped for years, but for many use cases, the status quo tools have not caught up to the enthusiasm. Many business intelligence tools rooted in older database architectures require intense engineering work to deliver insights, queries are often slow, and the output is not always consistent or accurate. Part of the challenge is that building ML pipelines is difficult. Data in a database or data warehouse typically needs to move to an intelligence platform so models can be trained, and the models then need to be deployed and integrated into business workflows.

But modern data warehouses such as BigQuery let users train models in the warehouse itself, without having to move the data—and once the models are created, they can be applied and integrated into business processes using simple SQL. When it comes to embedding ML into enterprise processes, these modern approaches significantly lower the barrier for entry. Plus, tools like Looker, Google Cloud’s platform for modern BI and data applications, were created specifically for modern data needs, with the assumption that data needs would constantly evolve and that iterations should be made quickly without eating up inordinate engineering resources.

Related: Google (Looker) recognized in the Gartner 2021 Magic Quadrant for Analytics and Business Intelligence Platforms.

For Commonwealth Care Alliance (CCA), Looker was originally implemented to alleviate their pain points around data bottlenecks and data chaos. But when the pandemic hit, the nonprofit, community-based healthcare organization pivoted to make use of Looker’s tools to better serve patients. CCA used BigQuery and Looker to combine numerous datasources, create a predictive model that assesses risk, and distribute that model to its clinicians. This has given response teams the insights to determine who is too high risk to come in for care so they can reach out with home care solutions.

This kind of agility is not a one-time antidote to a one-time disruption, but rather the norm to which organizations must aspire if they want to remain competitive and protect themselves against future disruptions.

This same functionality is also helping businesses like SoundCommerce. Retailers like Constellation Brands, Eddie Bauer, and FTD/ProFlowers use SoundCommerce’s out-of-the-box data platform, which is powered by BigQuery and Looker, to collect retail data from any source and build a model around the metrics and relationships that are most crucial to retail. This has saved brands hundreds of manual reporting hours each month, and reduced platform licensing costs by almost 75%. Just as importantly, during the uncertain times of 2020, brands that used SoundCommerce were able to align real-time and predictive business decisions across marketing and operations with critical retail KPIs like contribution margin and customer lifetime value (CLV). 

As 2020 showed us, we can never predict the future—but we can prepare for unpredictability by having the agility to always improve, and by positioning ourselves to make quick, intelligent pivots when the time comes. Last year was in many ways a rubicon: This kind of agility is not a one-time antidote to a one-time disruption, but rather the norm to which organizations must aspire if they want to remain competitive and protect themselves against future disruptions.

Looking for an ‘easy button’ to speed up your BI workloads running on BigQuery? Check out our latest announcement about BI Engine on the Google Cloud Blog.

Debanjan Saha

Debanjan Saha is GM of Data Analytics at Google Cloud, where he leads the strategy and execution of analytics services in GCP. Prior to joining Google, Debanjan was VP of Amazon Aurora and RDS at Amazon Web Services. Earlier in his career Debanjan held multiple executive and technical leadership positions at IBM and Tellium, an optical networking pioneer that he helped grow from an early stage start-up to a public company. 

Debanjan is a Fellow of the IEEE and a Distinguished Scientist of the ACM. He has co-authored a book, 50 patent applications, and numerous technical articles including award winning papers and Internet standards. He received MS and PhD degrees from the University of Maryland, and a B.Tech from IIT, all in Computer Science. In 2019, Business Insider named him as one of the top 10 technology executives transforming business.