The digital era turned the traditional formula for media and entertainment success on its head, ushering in new technologies that have changed how content is produced, distributed, experienced, and monetized. Audiences have more choice, flexibility, and power over what they consume, and today’s media companies have to embrace ongoing transformation or risk falling behind – or becoming irrelevant. 

A new wave of transformation is arriving with generative AI, a type of artificial intelligence that can interact with users in natural language and create novel data, ranging from story outlines, reports, and other text outputs to multimodal content like images, videos, and audio. Media and entertainment are inherently about content creation and creativity—so what does this new technology mean for the industry? 

At Google Cloud, we see tremendous opportunity for creative industries, from more efficient creation methods to improved user experiences. Let’s explore.

AI for media with Google Cloud

Google Cloud has a long history with large language models (LLMs) and other generative AI technologies—from their influence over the years on products like Document AI, to recent announcements like Generative AI support in Vertex AI, which lets businesses access and tune generative AI foundation models, and Generative AI App Builder, which lets developers build chatbots and other generative apps in minutes.  

We’ve helped our global media and entertainment customers with AI for personalization, search and recommendations, predictive analytics, and much more — and with generative AI now on the rise, we have some ideas to help media leaders, technologists, and creators think about and prepare to utilize powerful AI in their work. 

Three lenses on innovation in media

The media and entertainment industry is increasingly diverse and complex, with companies spanning over-the-top (OTT) subscription streaming services, 24-hour linear channels, live broadcasts of sporting events, digital journalism, traditional publishing, short-form user-generated social video, and more. More and more, the boundaries between these segments of the media industry are blurring — but common to them all is the focus on providing compelling content in an engaging audience experience that can be directly or indirectly monetized.

With this in mind, we suggest media and entertainment companies look at the application of innovative technologies like generative AI through the following three lenses:

1. Improving content creation, production, and management
2. Enhancing and personalizing audience experiences
3. Improving monetization

Improving content creation, production, and management

Generative AI democratizes many aspects of content creation, opening new ways to create written material, illustrations, sound effects, special effects, and more. Its recent maturation has been so rapid, some in the media industry have expressed concern that generative AI implies the end of creative professions. We think the opposite is more likely: just as photography, audio recordings, and computer generated images have enabled new modes of creativity, rather than making old ones obsolete, generative AI has the potential to both enable new forms of expression and enhance familiar ones. 

For example, journalists could use generative AI to speed up research by helping them synthesize and analyze large volumes of information, or to help them create initial drafts or summaries of editorial content. Film and television producers could leverage the technology to accelerate the post-production editing process, with new AI-enabled interfaces for rapidly adjusting or enhancing scene details such as lighting and color. Broadcasters could use generative AI to make vast libraries of video footage searchable and accessible for use in telling more compelling stories. The potential use cases go on and on.

Far from undermining incredible creative professions, generative AI is poised to free writers, artists, editors, and many others from the tedious and mundane aspects of their work, empowering them to focus more of their time on creativity.

Enhancing and personalizing audience experiences

Every media organization in the world today faces the reality that for most consumers, switching costs are extremely low. This puts incredible pressure on these companies to invest in delivering low-friction and compelling audience experiences that help mitigate subscribers from churning and viewers from abandoning content experiences for competitive platforms. 

Generative AI can help media companies engage and retain viewers, such as by enabling more powerful search and recommendations on their digital content platforms. With its increasingly multimodal capabilities extending from natural language to both audio and video content, generative AI is well-positioned to power more personalized audience experiences. 

Consumers often complain about “the paradox of choice” or their inability to find something interesting to watch on streaming platforms that have incredibly vast libraries of content available on demand. Imagine a not-too-distant future wherein a consumer can simply ask the content platform they’re using to help them find a specific show to watch based on mood, specific types of scenes, combinations of actors, award nominations, or practically anything they can think to ask. And that’s just the tip of the iceberg — imagine generative AI’s potential to curate, assemble, and even create personalized content for a viewer to consume!

Improving monetization

As consumers’ content consumption further expands from traditional theatrical and linear television programming to include digital offerings across an array of platforms, devices, and content types, media companies face the challenge of maintaining and improving monetization. The conventional economics and approaches to advertising and subscription models are proving, in many cases, not to deliver sufficient ROI. 

Generative AI has the potential to help media companies improve their monetization of audience experiences. As mentioned previously, enhanced personalization can play a role in mitigating churn, which in turn can help sustain and grow subscription and advertising revenues. Going beyond this, generative AI can be leveraged to drive even greater advertising revenues via more targeted, contextual, and personalized advertisements. Imagine both display and video advertisements that are generated on the fly to personalize product specifics, messaging, style, colors, and innumerable other characteristics to drive greater engagement and higher click-through rates (CTR), and thus higher advertising CPMs (cost per thousand impressions).

Coming up next

Generative AI presents a significant opportunity for media companies to fundamentally transform content creation, engagement, and monetization. Compelling services are already on the market — but there is far more to come. 

Google Cloud continues to build on its deep experience and expertise with AI, and we are committed to working with the industry to develop compelling, accessible, trusted, and responsible AI solutions that will drive meaningful business outcomes. We are excited to create the future together with our global media customers and partners across the ecosystem. To learn more about this disruptive topic, read “Debunking five generative AI misconceptions” from Google Cloud vice president of AI & Business Solutions Phil Moyer, or explore our Trusted Tester Program for generative AI.

Let’s face it: in the globalized world, which is now more than ever a digital demand world, you need to scale and reach your customers right where they’re at. Translation is a critical piece of that, whether you’re translating a website in multiple languages or releasing a document, a piece of software, or training materials.

Manual translation does not scale, which is why machine translation, powered by machine learning (ML), is becoming more important to our customers.  Machine translation has historically been challenging because of the sheer volume and breadth of content that can add value when translated into multiple languages. Companies acquire and share content in many languages and formats, and scaling translation to meet needs is a tall order due to multiple document formats, integrations with optical character recognition (OCR), and the need to correct for domain-specific terminology.

Our goal is to simplify translation services, while enabling flexibility and control for our customers’ unique needs across industries. Read on to learn more about recent features and updates. 

Formatting matters: Document Translation is now GA 

In many cases, the layout of a document dictates how it should be interpreted—e.g., readers navigate text and discern meaning based on formatting, like bold or italicized text, or markups for headers, paragraphs, and columns. Previously, to automate translation of documents, text needed to be separated from these layout attributes, meaning the document’s structure was either lost or needed to be recreated later in the developer pipeline, after the text had been translated. This required translation teams to do a lot of extra work and maintain a lot of additional code. But now, those steps are unnecessary. Formatting can be retained throughout the translation process, handled directly by the Translation API Advanced. 

This feature lets customers translate documents in 100+ languages and supports document types such as Docx, PPTx, XLSx, and PDF while preserving document formatting.

And if your needs go beyond Document Translation, we can help you translate audio as well. For real-time streaming translation, check out the Media Translation API, and for offline transcription translation, combine the Translation API with the Video Intelligence API.   

Real-Time translation when you need it, Batch when you don’t

One of the biggest differentiators for Translation API Advanced’s document translation capabilities is the ability to do real-time, synchronous processing for a single file. 

For example, if you are translating a business document such as HR documentation, online translation provides flexibility for smaller files and provides faster results. You can easily integrate with our APIs via REST or gRPC with mobile or browser applications, with instant access to 100+ language pairs so that content can be understandable in any supported language. 

Meanwhile, batch translation allows customers to translate multiple files into multiple languages in a single request. For each request, customers can send up to 100 files with a total content size of up to 1 GB or 100 million Unicode codepoints, whichever limit is hit first.

State of the Art (SOTA) accuracy, with flexibility for customization

In order to achieve the highest level of accuracy for your translation, we now support multiple options:

• Use Google’s SOTA translation models: Each year, Google heavily invests to improve the quality of our translations across Apps, Cloud APIs, and Chrome, as well to enable multilanguage answers in Search. A popular metric for automatic quality evaluation of Machine translation systems is the BLEU score, which is based on the similarity between machine translation and the reference translations that were generated by people. While we push out incremental improvements for individual models on a monthly cadence, there are also times where we make significant leaps. In the releases since 2019, we have improved our average BLEU score by 5pts on average across 100+ languages and 7pts on low resource languages.
• Leverage glossaries for specific terms and phrases: Glossary is our terminology control feature. It allows you to import source content to define preferred translations, such as product names or department names. Then, when calling the glossary in the API request, your preferred translations will be enforced. This will work for words as well as phrase translation.
• Pick a pre-trained model with model selection: If you create custom models for machine translation, we don’t think you should have multiple client libraries and multiple APIs to maintain in order for you to use the best model for your needs. Translation API Advanced now supports Model Selection. Pick your pretrained model or pick your custom ML model built on AutoML for any language pair you’ve created and use the same API and the same client library. 
• Build custom translation models with AutoML: AutoML Translation is a suite of ML products that enable you to build high quality models for your own use case or data, with limited-to-no ML expertise or coding required. Bring your past human-validated translations to improve translation specificity for your domain.

Keep localization local with Regional Endpoints

If you are a customer operating in the EU, we recently launched an endpoint specifically for EU regionalization. This is a configurable endpoint for customers to store and perform machine translation processing of customer data only in the EU multi region. For now, this only supports our pretrained translation models and glossary, but batch translations will be coming soon.

How Eli Lilly uses Cloud Translation to translate content globally

Historically, translations at Eli Lilly have been complicated: numerous translation vendors have been needed for different languages and organizations, all with their own processes and expectations. On top of that, translations have been costly and slow. 

To solve this, Eli Lilly took a codified approach to enable users and systems to spend less time and resources to safely generate quality translations. 

Learn more, and even catch a demo, from Thomas Griffin, Translation Tech Lead & Global Regulatory Architect for Eli Lilly.

Learn more 

• To get started using Cloud Translation – Advanced, complete the setup and then try the Translate text (Advanced edition) quickstart.
• Document Translation is priced per page. For more information, see pricing.

Unstructured data such as images, speech and textual data can be notoriously difficult to manage, and even harder to analyze. The analysis of unstructured data includes use cases such as extracting text from images using OCR, sentiment analysis on customer reviews and simplifying translation for analytics. All of this data needs to be stored, managed and made available for machine learning.

The new BigQuery ML inference engine empowers practitioners to run inferences on unstructured data using pre-trained AI models. The results of these inferences can be analyzed to extract insights and improve decision making. This can all be done in BigQuery, using just a few lines of SQL.

In this blog, we’ll explore how the new BigQuery ML inference engine can be used to run inferences against unstructured data in BigQuery. We’ll demonstrate how to detect and translate text from movie poster images, and run sentiment analysis against movie reviews.

BigQuery ML’s new inference engine

Google Cloud is home to a suite of pre-trained AI models and APIs. The BigQuery ML inference engine can call these APIs and manage the responses on your behalf. All you have to do is define the model you want to use and run inferences against your data. All of this is done in BigQuery using SQL. The inference results are returned in JSON format and stored in BigQuery for analysis.

Why run your inferences in BigQuery?

Traditionally, working with AI models to run inferences required expertise in programming languages like Python. The ability to run inferences in BigQuery using just SQL can make generating insights from your data using AI simple and accessible. BigQuery is also serverless, so you can focus on analyzing your data without worrying about scalability and infrastructure.

The inference results are stored in BigQuery, which allows you to analyze your unstructured data immediately, without the need to move or copy your data. A key advantage here is that this analysis can also be joined with structured data stored in BigQuery, giving you the opportunity to deepen your insights. This can simplify data management and minimize the amount of data movement and duplication required.

Which models are supported?

For now, the BigQuery ML inference engine can be used with these pre-trained Vertex AI models:

• Vision AI API: This model can be used to extract features from images managed by BigQuery Object Tables and stored on Cloud Storage. For example, Vision AI can detect and classify objects, or read handwritten text.
• Translation AI API: This model can be used to translate text in BigQuery tables into over one hundred languages.
• Natural Language Processing API: This model can be used to derive meaning from textual data stored in BigQuery tables. For example, features like sentiment analysis can be used to determine whether the emotional tone of text is positive or negative.

So, how does this work in practice? Let’s look at an example using images of movie posters

1. We will define our pre-trained models for Vision AI, Translation AI and NLP AI in BigQuery ML.
2. We’ll then use Vision AI to detect the text from some classic movie posters images. 
3. Next, we’ll use Translation AI to detect any foreign posters and translate them to a language of our choosing – English in this case. 
4. Finally, we’ll combine our unstructured data with structured data in BigQuery.
   We’ll use the extracted movie titles from our movie posters to look up the viewer reviews from the BigQuery IMDB public dataset. We can then run sentiment analysis against these reviews using NLP AI.

Note: The BigQuery ML inference engine is currently in Preview. You will need to complete this enrollment form to have your project allowlisted for use with the BQML Inference Engine.

We’ll give examples of the BigQuery SQL needed to define your models and run your inferences. You’ll want to check out our notebook for a detailed guide on how to get this up and running in your Google Cloud project.

1. Define your AI Models in BigQuery

You will need to enable the APIs listed below, and also create a Cloud resource connection to enable BigQuery to interact with these services.

You can then run the CREATE MODEL query for each AI service to create your pretrained models, replacing the model_name as required.

CREATE OR REPLACE MODEL 
`{PROJECT_ID}.{DATASET_ID}.{VISION_MODEL_NAME}`
REMOTE WITH 
CONNECTION `{PROJECT_ID}.{REGION}.{CONN_NAME}`
OPTIONS ( remote_service_type = '<model_name>' );

2. Use the Vision AI API to detect text in images stored in Cloud Storage

You will need to create an object table for your images in Cloud Storage. This read-only object table provides metadata for images stored in Cloud Storage:

CREATE OR REPLACE EXTERNAL TABLE
`{PROJECT_ID}.{DATASET_ID}.{OBJECT_TABLE_NAME}`
WITH
CONNECTION `{REGION}.{CONN_NAME}`
OPTIONS (object_metadata = 'SIMPLE', uris = ['{BUCKET_LOCATION}/*']);

To detect the text from our posters, you can then use ML.ANNOTATE_IMAGE and specify the text_detection feature.

SELECT
       ml_annotate_image_result.full_text_annotation.text AS text_content,
       *
FROM
       ML.ANNOTATE_IMAGE(
         MODEL `{PROJECT_ID}.{DATASET_ID}.{VISION_MODEL_NAME}`,
         TABLE `{DATASET_ID}.{OBJECT_TABLE_NAME}`,
         STRUCT(['TEXT_DETECTION'] AS vision_features));

A JSON response will be returned to BigQuery that includes the text content and language code of the text. You can parse the JSON to a scalar result using the dot annotation highlighted above.

3. Use the Translation AI API to translate foreign movie titles 

ML.TRANSLATE can now be used to translate the foreign titles we’ve extracted from our images into English. You just need to specify the target language and the table of the movie posters for translation:

SELECT
text_content,
STRING(ml_translate_result.translations[0].detected_language_code)
as original_language,
STRING(ml_translate_result.translations[0].translated_text)
as translated_title
FROM
  ML.TRANSLATE(
MODEL `{PROJECT_ID}.{DATASET_ID}.{TRANSLATE_MODEL_NAME}`,
TABLE `{DATASET_ID}.image_results`,
STRUCT('TRANSLATE_TEXT' as translate_mode, "en" as target_language_code));

Note: The table column with the text you want to translate must be named text_content:

The table of results will include json that can be parsed to extract both the original language and the translated text. In this case, the model has detected that title text is in French and has translated it to English:

4. Finally, use natural language processing (NLP) to run sentiment analysis against movie reviews

You can easily join inference results from your unstructured data with other BigQuery datasets to bolster your analysis. For example, we can now join the movie titles we extracted from our posters with thousands of movie reviews stored in BigQuery’s IMDB public dataset `bigquery-public-data.imdb.reviews`.

You can use ML.UNDERSTAND_TEXT with the analyze_sentiment feature to run sentiment analysis against some of these reviews to determine whether they are positive or negative:

SELECT
   primary_title, start_year, text_content AS review,
   FLOAT64(ml_understand_text_result.document_sentiment.score) AS score,
   FLOAT64(ml_understand_text_result.document_sentiment.magnitude) AS magnitude,
FROM
   ML.UNDERSTAND_TEXT(
     MODEL `{PROJECT_ID}.{DATASET_ID}.{NLP_MODEL_NAME}`,
           (
           SELECT
             primary_title, start_year, review AS text_content
           FROM
             `bigquery-public-data.imdb.title_basics` titles
           JOIN
             `bigquery-public-data.imdb.reviews` reviews
           ON
             reviews.movie_id = titles.tconst
           WHERE
             UPPER(titles.primary_title) = 'THE LOST WORLD' AND
             start_year = 1925
           ),
      STRUCT("analyze_sentiment" AS nlu_option)) ;

Note: The table column with the text you want to analyze must be named text_content:

The JSON response will include a score and magnitude. The score indicates the overall emotion of the text while the magnitude indicates how much emotional content is present:

So, how did the Lost World compare with other movies that year?

To wrap up, we’ll compare the average review score of the 1925 Lost World movie to other movies released that year to see which was more popular. This can be done using familiar SQL analysis:

SELECT
 primary_title, start_year,
 AVG(FLOAT64(ml_understand_text_result.document_sentiment.score))AS av_score,
 AVG(FLOAT64(ml_understand_text_result.document_sentiment.magnitude)) AS av_magnitude
FROM
 ML.UNDERSTAND_TEXT(
   MODEL `{PROJECT_ID}.{DATASET_ID}.{NLP_MODEL_NAME}`,
         (
         SELECT
           primary_title, start_year, movie_id, review AS text_content
         FROM
           `bigquery-public-data.imdb.title_basics` titles
         JOIN
           `bigquery-public-data.imdb.reviews` reviews
         ON
           reviews.movie_id = titles.tconst
         WHERE
           start_year = 1925
         ),
   STRUCT("analyze_sentiment" AS nlu_option))
GROUP BY
   primary_title, start_year
ORDER BY
   av_score DESC;

It looks like The Lost World narrowly missed out on the top spot to Sally of the Sawdust!

Want to learn more?

Check out our notebook for a step by step guide on using the BQML inference engine for unstructured data in Google Cloud. You can also check out our Cloud AI service table-valued functions overview page for more details. Curious about pricing? The BQML Pricing page gives a breakdown of how costs are applied across these services.