At Google, we believe anyone should be able to quickly and easily turn their artificial intelligence (AI) idea into reality. Open source software (OSS) has become increasingly important to this goal, heavily influencing the pace of innovation in AI and machine learning (ML) ecosystems. Over the last two decades, ML has transformed Google services including Search, YouTube, Assistant, and Maps, and the basis for this transformation has always been our “open first” approach through investments in projects and ecosystems like TensorFlow, Jax, and PyTorch.

These OSS efforts are important because many AI technologies rely on closed or exclusive approaches. This wall-garden approach creates high barriers to entry for developers; limits efforts to make AI explainable, ethical, equitable; and stunts innovation. We’re committed to open ecosystems, as we firmly believe no one company should own AI/ML innovation. In this blog post, we’ll explore some of Google’s most significant OSS AI and ML contributions from recent years, as well as how our commitment to open technologies can help organizations innovate faster and more flexibly.

Openness is the way to operate as an ecosystem, not a single project

Google’s OSS initiatives extend and enable AI initiatives according to three pillars:

• Access — OSS allows developers, researchers and organizations of all sizes to leverage the latest ML technology. It is a key part of democratizing innovation in ML, fostering software diversity and choice for customers, and lowering operating cost while accelerating scale for everyone.
• Transparency — Open source datasets, ML algorithms, training models, frameworks, and compilers ensure due diligence and validation by the larger community. This is paramount when it comes to ML as it bolsters reproducibility, interpretability, ensures equity, and boosts security.
• Innovation — With more access and transparency, more innovation comes naturally. Our customers and partners take advantage of open source ML toolsets and frameworks to create more innovation in the field by contributing their own OSS.

Google’s ongoing commitment to open source AI

Google’s commitment to open standards spans over two decades of OSS contributions like TensorFlow, JAX, TFX, MLIR, KubeFlow, and Kubernetes, as well as sponsorship for critical OSS data science initiatives like Project Jupyter and NumFOCUS. Initiatives like these have helped Google become the leading Cloud Native Computing Foundation (CNCF) contributor—and by building on these efforts, Google Cloud seeks to be the best platform for the OSS AI community and ecosystem.

The perils of closed technologies can emerge at many points across ML pipelines, which is why Google’s OSS strategy encompasses the entire “idea-to-production” lifecycle, from acquiring data, to training models, to managing infrastructure, to facilitating experimentation and model refinement:

Data acquisition: starting the journey from idea to production-ready ML model

The journey from an idea to a production ML model starts with data. TensorFlow Datasets not only help users acquire ready-to-use, customizable, and highly-optimized datasets (including image, audio, and text), but also provides a set of helpful APIs that make it easy for users to organize their own datasets, regardless of whether they build with TensorFlow, Jax, or other ML frameworks.

Model development and training: shortening the path from data to useful ML

OSS libraries help developers and researchers design, implement, train, test, and debug ML algorithms. Our contributors on this front include:

• The TensorFlow core framework, which offers APIs to help data scientists and developers build and train production-grade ML models on distributed and accelerated infrastructure powered by GPUs or TPUs;
• Google’s founding membership of the PyTorch Foundation, which positions us to increase adoption of ML by building an ecosystem of open-source projects with PyTorch;
• Keras, a simple and powerful ML framework, well integrated with TensorFLow, that makes it easy for developers to quickly build and train ML models, or to leverage pre-trained AI applications;
• Model Garden, which provides implementations of many state-of-the-art computer vision and natural language processing models, maintained by Google and accessible to all, alongside APIs to accelerate training and experiments;
• Jax, a lean, intuitive, and composable system that brings together automatic differentiation (Autograd) and the Accelerated Linear Algebra (XLA) optimizing compiler to offer high-performance ML for fast research and production;
• TensorFlow Hub, a repository of trained ML models ready for fine-tuning and deployment; and,
• MediaPipe open source cross-platform, which lets users leverage customizable ML solutions for live and streaming media, including text and video.

ML infrastructure management: scaling valuable models with powerful backends

Accessing and managing infrastructure for ML, especially at scale, can be a blocker for many organizations, which is why Google has invested in initiatives including:

• The TFX (or TensorFlow Extended) platform, which offers software frameworks and tooling for full MLOps deployments, helping developers with data automation, model tracking, performance monitoring, and model retraining;
• Kubeflow, which makes deployments of ML workflows on Kubernetes simple, portable and scalable; and,
• TRC (TPU Research Cloud), which gives access to a cluster of more than 1,000 Cloud TPU devices at no charge to selected researchers who publish peer-reviewed papers and/or open source code.

Experimentation and model optimization: encouraging discovery and iteration

Data, tools for model training, and infrastructure can achieve only so much without strong processes for experimentation and optimization—which is why we’ve contribution to projects like xManager, which enables anyone to run and keep track of ML experiments locally or on Vertex AI and Tensorboard, which simplifies tracking and visualizing of model performance metrics.

These areas of focus will help not only our customers but the open-source AI community as a whole, and we’re excited to share more OSS news in coming days and months. To start exploring why many organizations choose Google Cloud for their open-source AI needs, visit our “open cloud” page and be sure to register for Google Cloud Next ‘22 for all our latest news.

Thanks to all the contributors to this blog post: Matt Vasey, George Elissaios, Warren Barkley, Manvinder Singh, James Rubin, Abhishek Ratna, Thea Lamkin, Amin Vahdat, Andrew Moore, Max Sapozhnikov, Gandhi, Vikram Kasivajhula

Editor’s note: Earth Day reminds us that we all can contribute to creating a cleaner, healthier, and more sustainable future. Google Cloud is excited to celebrate innovative startup companies developing new technology and driving sustainable change. On this year’s Earth Day we’re highlighting Enexor BioEnergy and their initiative to produce clean, sustainable energy from plastic and agro-waste with the help of Google Cloud solutions.

Picture a world without electricity where billions of people burn dangerous fuels and trash from landfills to cook meals and heat their homes. Children constantly cough, choking on toxic smoke that lingers in houses and sickens entire families. Because there are no pumps or purification plants, contaminated water is hauled in buckets from dirty rivers and polluted wells. Without power or lights, hospitals and emergency medical clinics can only provide basic services during the day.

Although it is 2022, this harsh life is a reality for billions of people living in developing countries around the world.

Producing clean energy from plastics and organic-waste

This Earth Day, let’s imagine what life would be like in these countries if people had access to safe and inexpensive bioenergy. In this world, indoor air is fresher. Clean water runs from faucets, while homes, businesses, and schools have electricity. Doctors and nurses have the power to treat patients 24 hours a day and save lives with advanced medical equipment. Local economies boom and entrepreneurs thrive.

This is the sustainable future Enexor BioEnergy and its partners are building—one village at a time. With the Enexor Bio-CHP™ system, we produce clean and sustainable energy from discarded plastics, organic, and biomass-waste such as rice and corn husks, seaweed, coconuts, and other biomatter while offsetting significant Carbon emissions. The Bio-CHP can be commissioned and easily installed in just a single day, with multiple systems installed side-by-side when more power is needed. By being offered under its novel Energy-as-a-Service business model, Enexor ensures that the Bio-CHP can generate immediate environmental positive impact and customer adoption.

Each Bio-CHP unit generates enough energy to power over 100 homes—and provides much-needed renewable electricity and thermal power for schools, businesses, manufacturing facilities, water pumps, Wi-Fi systems, and telecommunications towers. The Bio-CHP creates clean energy by safely oxidizing plastics and biomatter in a secure container using a high-temperature system to power a micro-turbine. This produces bioenergy, significantly reducing harmful greenhouse gas emissions.

By solving a community’s waste issues while providing more affordable renewable energy, the Bio-CHP also creates new economic opportunities and improved outcomes where it is installed. This includes using its inexpensive power to expand local services and offerings, and empowering local community clean up efforts by incentivizing waste collection via blockchain-enabled digital currencies and services such as PayGo. Additionally, the Bio-CHP is an ideal solution for businesses who desire to maximize their own sustainability efforts, gain greater energy resiliency, and save money on their current energy and waste costs.

Engaging in Google for Startups Accelerator: Climate Change

When developing the Bio-CHP, we realized we needed a reliable and experienced partner to help us incorporate the best in class machine learning and artificial intelligence into our technology. That’s why we joined the Google for Startups Accelerator and participated in the Google for Startups Accelerator: Climate Change.

The accelerator introduced us to other companies working to create a sustainable future and opened new opportunities for collaboration and partnerships. The accelerator also continues to give our small team access to the best of Google Cloud’s people, programs, and solutions. We especially want to highlight Google Cloud’s dedicated startup experts and the incredible technical support they provide, as well as the Google Cloud research credits we used to explore new solutions.

As an example, Enexor is developing a predictive maintenance tool that automatically adjusts Bio-CHP operations to match fuel composition. Another tool in development proactively identifies and flags potential Bio-CHP system and component failures—before they occur.

Google Cloud’s dedicated startup experts work closely with Enexor to develop predictive models for these tools and Enexor also used the Google Cloud research credits that were provided to build these advanced models on TensorFlow, Vertex AI, Cloud CDN and AutoML. Since each system is remotely monitored from Enexor’s global headquarters in Tennessee, predictive maintenance tools also increase the safety, reliability, and efficiency of Bio-CHP in off-grid locations.

Generating 13 billion kWh of clean power by 2040

In the future, Enexor plans to explore additional Google Cloud solutions such as BigQuery to crunch larger datasets, Google Data Studio to build dashboards, and perhaps even Google Cloud Tensor Processing Units (TPUs) to more efficiently process machine learning (ML) workloads.

These solutions will help Enexor to achieve three primary goals by 2040: to generate 13 billion kWh of clean power, reduce 120 million tons of CO₂, and provide one million people with access to clean and sustainable energy. With each Bio-CHP system, Enexor annually reduces up to 2,000 metric tons of CO₂ equivalent emissions by decreasing methane emissions released from landfills, offsetting fossil fuel-based power generation with carbon credits and minimizing waste disposal transportation emissions.

Enexor is now preparing to deploy the Bio-CHP in Accra, Ghana where it will convert organic and plastic manufacturing waste into sustainable energy. Enexor is also looking forward to working with the local Accra community and NGOs to collect discarded plastics and organic waste diverting it from ending in the rivers and oceans and instead using them as renewable fuel sources. We can’t wait to see what we accomplish next as we celebrate Earth Day 2022 and think about the sustainable future Enexor is helping to empower.

If you want to learn more about how Google Cloud can help your startup, visit our page here to get more information about our program, and sign up for our communications to get a look at our community activities, digital events, special offers, and more.

In fast-moving organizations, it’s not uncommon for cloud resources, including entire projects, to occasionally be forgotten about. Not only such unattended resources can be difficult to identify, but they also tend to create a lot of headaches for product teams down the road, including unnecessary waste and security risks. 

To help you prune your idle cloud resources, we’re excited to introduce Unattended Project Recommender. It’s a new feature of Active Assist that provides you with a one-stop shop for discovering, reclaiming, and shutting down unattended projects. With actionable and automatic recommendations, you no longer have to worry about wasting money or mitigating security risks presented by your idle resources. Unattended Project Recommender uses machine learning to identify, with a high degree of confidence, projects that are likely abandoned based on API and networking activity, billing, usage of cloud services, and other signals. This feature is available via the Recommender API today, making it easy for you to integrate with your company’s existing workflow management and communication tools, or export results to a BigQuery table for custom analysis.

Thousands of projects can be unattended in large organizations, presenting major security risks

Your cloud projects can go abandoned or unattended for a number of reasons — ranging from a test environment that’s no longer needed, to project cancellation, to project owner switching jobs, and more. Not only can such projects contribute to your cloud bill (waste) but they may contain security issues such as open firewalls or privileged service account keys that attackers can exploit to get a hold of your cloud resources for cryptocurrency mining or, worse, compromise your company’s sensitive data. These security risks tend to grow over time because the latest best practices and patches are usually not applied to unattended projects. 

We experience this issue here at Google, too… In fact, it has been on Google’s internal security team’s radar for some time now, so we joined forces and looked into this problem together, starting with our very own “google.com” organization cloud projects. We quickly found some projects that were unattended, but remediating this issue was easier said than done due to challenges in several areas:

• Detection: With lots of signals available to you via sources like Cloud Monitoring, what are the right ones you should look at (e.g. API, networking, user activity)? How can you tell the difference between an unattended project and a project that has a low level of activity by design (e.g. a “shell” project that holds an auth token)?
• Remediation: Once you have identified a project that seems abandoned, how do you go about ensuring that it’s indeed an unattended project? How do you reduce the risk of deleting something that might be essential to a production workload, causing irreversible data loss? How do you solve this at the scale of your entire organization, beyond a one-time cleanup? 

Over the course of 2021 we built and tested a Google-internal prototype first, cleaning up many of our internal unattended projects, and then worked with a number of Google Cloud customers to build and tune this feature based on real-life data (thank you to all of our early adopters for working with us and your generous feedback that helped us shape this feature!) It was not uncommon for us to come across organizations with thousands of unattended projects, and we’re very excited to bring Unattended Project Recommender to all customers, in public preview.

Discovering and acting on unattended project recommendations

Unattended Project Recommender analyzes usage activity across all projects under your organization, including the following data:

• API activity (e.g. service accounts with authentication activity, API calls consumed)
• Networking activity (ingress and egress)
• Billing activity (e.g. services with billable usage)
• User activity (e.g. active project owners)
• Cloud services usage (e.g. number of active VMs, BigQuery jobs, storage requests)

Based on these signals, it can generate recommendations to clean up projects that have low usage activity (where “low usage” is defined using a machine learning model that ranks projects in your organization by level of usage), or recommendations to reclaim projects that have high usage activity but no active project owners. Here’s what an example post-processed summary list of recommendations can look like for the “foobar” organization that has 3 projects:

  Project ID: demo-project-307815
Recommendation: CLEANUP_PROJECT

Project ID: new-project
Recommendation: N/A

Project ID: bobs-playground-project
Recommendation: RECLAIM_PROJECT

In addition to the recommendations, you can also examine the underlying project activity insights that the recommendations are based upon. The insights provide additional information that can be useful for integration with your organization’s existing workflows and automation (e.g. send an auto-generated email or chat message to project owners based on the list provided by the owners field). Here’s an example insight payload:

  content:
  activeAppengineInstanceDailyCount: 0
  activeCloudsqlInstanceDailyCount: 0
  activeGceInstanceDailyCount: 3
  activeServiceAccountDailyCount: 1
  apiClientDailyCount: 18922           // Daily average API calls produced
  bigqueryInflightJobDailyCount: 0
  bigqueryInflightQueryDailyCount: 0
  bigqueryStorageDailyBytes: 0
  bigqueryTableDailyCount: 0
  consumedApiDailyCount: 0             // Daily average API calls consumed
  datastoreApiDailyCount: 0
  gcsObjectDailyCount: 11
  gcsRequestDailyCount: 0
  gcsStorageDailyBytes: 2663548
  hasActiveOauthTokens: false          // OAuth tokens used in the last 180 days
  hasBillingAccount: true
  numActiveUserOwners: 1
  owners:                              // List of project owners
  - activeOnProject: false
    member: user:user1@example.com
  - activeOnProject: true
    member: user:user2@example.com
  serviceWithBillableUsage:
  – Cloud Storage
  - Compute Engine
  vpcEgressDailyBytes: 264456938       // Daily average VPC egress bytes
  vpcIngressDailyBytes: 392435047      // Daily average VPC ingress bytes
  usagePercentile: 20                  // Level of usage relative to other projects

GCP projects are used in many different ways and for many different purposes. In case you get a recommendation to delete a project that’s being used in a way that’s out of the scope for this feature, you can dismiss the recommendation and it will stop showing up for the given project. 

Restoring deleted projects

When you choose to shut down a project using the projects.delete() method, it gets marked for deletion. After a project is marked for deletion, it becomes unusable, all resources within that project are shut down, and a 30-day wait period for the project and all of its data to get fully deleted begins.

In case a useful project is accidentally shut down, you have the option to restore the project within that 30-day wait period. Since restoring allows you to recover most but not necessarily all of your project data and resources, we recommend carefully examining the utilization insights associated with a project and considering any additional utilization signals that may not be captured by the Unattended Project Recommender before taking the cleanup action.

Early customer success stories

A number of enterprise customers are already using Unattended Project Recommender to keep their organizations clean of unattended projects and resources.

Decathlon, a French sporting goods retailer, is excited for the insight Unattended Project Recommender will bring to their environment, and are already deploying it as a part of their latest cloud security initiatives.

“After a thorough test of this feature and the validation of our CISO, we ended up deleting our first 775 projects, and no one complained! A great help to improve our security. The next step for us will be to operationalize it at scale, and implement a company wide policy for unattended resource management.” —Adeline Villette, Cloud Security Officer

For Veolia, one of the world’s largest water, waste and energy management companies, not only does this feature reduce security risks and waste, but also helps drive cultural shift and alignment with its ecological transformation strategy.

“This feature allows us to reduce our costs and security debt on assets that are no longer in use, and is also fully in line with Veolia’s philosophy of limiting its carbon footprint. After having tested Unattended Project Recommender on more than 3,000 projects throughout our organization, we are looking to bring it as proactive alerts to our project owners at scale.”—Thomas Meriadec, Product Manager

Box, a secure cloud content management provider, views it as a foundation for building a repeatable process to remediate unused resources.

“Unattended Project Recommender is a great fit for us. It gives us a unified view of project usage across our entire organization and enables us to address security risks of legacy projects in a systematic and organized manner, ensuring an even safer environment.” —Matt Bowes, Staff Security Engineer

Getting started with the Unattended Project Recommender

To help you get started, we’ve prepared a Cloud Shell tutorial (source code) that you can use to find unattended project recommendations within your own Projects/Folders/Organization. Click this button to clone the tutorial from GitHub and run in your Cloud Shell environment:

As you can see, listing recommendations for your projects only takes a few clicks with the tutorial (special thanks to Lanre Ogunmola, Security & Compliance Specialist, for making this look so easy)! For additional detail on using the gcloud CLI or API to discover unattended project recommendations, please refer to the documentation page.

You can also automatically export all recommendations from your Organization to BigQuery and then investigate the recommendations with DataStudio or Looker, or use Connected Sheets that let you use Google Workspace Sheets to interact with the data stored in BigQuery without having to write SQL queries.

As with any other Recommender, you can choose to opt out of data processing at any time by disabling the appropriate data groups in the Transparency & control tab under Privacy & Security settings.

We hope that you can leverage Unattended Project Recommender to improve your cloud security posture and reduce cost, and can’t wait to hear your feedback and thoughts about this feature! Please feel free to reach us at active-assist-feedback@google.com and we also invite you to sign up for our Active Assist Trusted Tester Group if you would like to get early access to the newest features as they are developed.

After a seamless migration to Google Cloud Platform with CloudCover and Google Cloud Professional Services, Hike has reduced its costs by 20% and processed analytics queries 20 times faster than with its previous cloud provider. The business is also using AI and machine learning to enhance the experience provided by a new sticker-based messaging app, Hike Sticker Chat.

India is a market of opportunity for businesses that provide messaging apps to consumers. With more than 1.3 billion people, the country is the second most populous in the world. However, global messaging app providers face a robust market challenge from Hike, a home-grown internet and technology startup. Launched in 2012, Hike provides innovative products such as Hike Messenger and more recently the AI- and machine-learning-enabled Hike Sticker Chat, a service that enables young people in the country to express themselves through digital stickers.

The business says it understands the people of India and communication like no one else, while its mission is to reduce individuals’ dependency on the keyboard. To do this, Hike is building one of the largest repositories of AI and machine-learning-enabled stickers for Hike Sticker Chat. This messaging platform is, according to Hike, the only product of its type that enables conversations through stickers covering more than 40 languages and local dialects.

Google Cloud Results

• Processes analytics queries 20X faster than previously
• Doubles compute throughput
• Uses Google Cloud Machine Learning Engine managed, distributed capabilities to train complex models on TensorFlow that provide delightful local sticker recommendations through Hike Sticker Chat

Founded by Kavin Bharti Mittal, the Delhi-based venture is backed by SoftBank, Tencent, Tiger Global, Foxconn, and Bharti. To date, Hike has raised $261 million in funding. In August 2016, Hike raised its Series D round of funding, led by Tencent and Foxconn, at a valuation of $1.4 billion. The business is one of the fastest Indian startups to achieve Unicorn status, doing so in less than four years.

Hike started operations on a multinational cloud service. However, as user numbers and usage grew, the business began exploring options to improve performance and stability, reduce costs, and cut administration loads. In particular, Hike wanted to reduce latency between cloud data centers.

Focus on product development

“We aimed to move away from a technology stack with single points of failure to a horizontally scaled, highly reliable, distributed systems and managed services environment that enabled us to focus on product development rather than operations,” says Aditya Gupta, Director, Engineering, Hike.

Hike then began exploring the opportunities presented by Google Cloud Platform. The business held a number of executive-level meetings with Google to understand the capabilities, roadmap, and track record of the cloud service. It then decided to proceed with a proof of concept with Google Cloud Premier Partner CloudCover.

The proof of concept revealed that when Cloud Load Balancing was operating, latency between the Google Cloud data center in Taiwan and Delhi, India, was less than the latency between the incumbent cloud provider’s data center and Delhi. Further, compute throughput was up to two times greater on Compute Engine than on the equivalent service, while Hike could complete more then 1 million connections on Compute Engine – up from 500,000 connections on the incumbent service.

Migrate to GCP

The success of the exercise prompted Hike to migrate its messaging app to Google Cloud Platform. “We chose Google Cloud Platform because of its very broad set of services and features,” explains Gupta. “In addition, Google’s innovation mindset and the richness of the partnership would allow us to be onboarded quickly to machine learning services such as Cloud Machine Learning Engine.”

The business called on Google Cloud Professional Services (Technical Account Management) to help ensure a seamless lift-and-shift migration over two months. Google Cloud Professional Services initially undertook a technical infrastructure kickoff to establish a foundation for architecture requirements such as identity and access management and security.

Google Cloud Professional Services team delivers smooth migration

Google Cloud Professional Services worked closely with Hike to map out and deliver the Google Cloud Platform architecture that would deliver the greatest value to the business. The Professional Services team also worked with Hike to resolve product and support queries quickly; provided project background for product and support teams; and organized project meetings and early adopter program access.

In addition, Professional Services team members worked on site at least once a week, coordinated external support during critical migration periods, and coordinated teams in five countries for a single, 17-hour migration marathon. Over 60 days, the business migrated 7,000 processor cores, running virtual machine instances used for messaging infrastructure and analytics, to Google Cloud Platform.

Throughout the exercise, Google Cloud Professional Services worked with CloudCover to educate the customers’ technology teams to achieve proficiency with Google Cloud Platform. The teams soon built up skills and knowledge of best practices and began applying them to the Google Cloud Platform environment.

The Hike Google Cloud Platform architecture comprises virtual machine instances running in Compute Engine; Cloud Storage for unified object storage; networking; a BigQuery analytics data warehouse; Cloud Dataflow to transform and enrich data; Cloud Load Balancing to distribute workloads to maximize efficiency; and Cloud Dataproc to run Hadoop clusters.

Hike is also stepping up its AI & machine learning capabilities. It uses Google Cloud Machine Learning Engine managed, distributed computing capabilities to train complex models on TensorFlow. This powers key use cases such as delightful local sticker recommendations on Hike Sticker Chat. Hike is also investing heavily on AI and machine learning research.

Hike has achieved a range of benefits from its Google Cloud Platform deployment. As well as reduced latency, improved compute throughput, and increased connection handling, Google Cloud Platform managed services have enabled the business to reduce the time and effort required to administer core infrastructure, with the saved resources allocated to improving its messaging product.

“Managed services are beginning to reduce our operational overheads,” says Gupta. “For example, managed instance groups and Cloud Load Balancing are reducing our instance count and costs, thereby reducing involvement from DevOps and developer teams.”

Google Cloud Platform 20% cheaper

Gupta and his team have calculated that running for three years on Google Cloud Platform will cost, including the cost of migration, 20 percent less than on its previous platform. BigQuery is processing queries 20 times faster than a similar service offered by the previous provider, while storing 125TB of data and streaming 1.5TB of data daily. Furthermore, Hike’s analytics pipeline costs 80 percent less than in its previous environment.

“Google Cloud Platform has played an important role in enabling us to continue to innovate and realize our mission of reducing dependency on the keyboard,” says Gupta.

Generative AI continues to develop at a blistering pace, making it more important than ever that organizations have access to enterprise-ready capabilities to help them leverage this disruptive technology. 

Harnessing the power of decades of Google’s research, innovation, and investment in AI, Google Cloud continues to make generative AI available with baked-in security, data governance, and scalability across the board. 

To this end, last month, we announced the general availability of Generative AI support on Vertex AI, giving our customers the ability to access powerful foundation models from Google Research and tools for customizing and applying them. 

Today we are announcing the general availability (GA) of four important foundation models for Vertex AI. These include Imagen, PaLM 2 for Chat, Codey, and Chirp. For each of these models, organizations can access APIs on Model Garden and do prompt design and tuning on Generative AI Studio.

• Imagen includes four key features:
  • Image generation for creating studio-grade images at scale
  • Image editing to edit generated or existing images via text prompts 
  • Image captioning for creating captions of images at scale
  • Visual Question & Answering (VQA) for interacting with, analyzing, and explaining images
• PaLM 2 for Chat follows the general availability of PaLM 2 for Text in June 
• Codey supports code generation, completion, and code chat
• Chirp supports multilingual Speech AI 

We’re also announcing Multimodal Embeddings API in preview, which lets customers combine the power of Vertex AI’s generative AI models with their proprietary data, to generate embeddings, or interchangeable vector representations, of their text and image data. These capabilities can enable data science teams to deliver a variety of downstream tasks such as image classification, content recommendations, and visual search. 

In this blog post, we’ll explore what your organization can do with these powerful models and how Vertex AI provides the enterprise-ready capabilities you can use to get up and running with generative AI. 

Helping to drive enterprise value from Generative AI models

Powerful models are the foundation of generative AI, but the software, tools, and infrastructure that surround these models are equally important for enterprise adoption. Organizations face challenges not only accessing these models, but also integrating AI while maintaining protection over intellectual property, adhering to regulations around data security and privacy, and ensuring models and applications are safe to use. Many organizations also want to use generative AI without incurring large costs or managing huge clusters.

We help address these challenges head-on with Vertex AI’s platform capabilities for scalable application integration, purpose-built AI infrastructure, secure and private data customization, and responsible use of this technology. 

Let’s see how each of these pillars can help your organization. 

Access models to build production-ready generative applications 
Vertex AI can make it easy to access foundation models, as today’s model announcements attest. While models are an inextricable part of generative AI, the software that helps enterprises use this technology is equally important—which is why Vertex AI also offers a range of tools for tuning, deploying, monitoring, and maintaining models, so you can build differentiated applications using your own data. 

Turning to today’s announcements, in May we announced Imagen, our foundation model for image generation. Now, we are excited to announce Imagen is generally available with an allowlist (i.e., approved access via your sales representative), letting onboarded customers start using image generation and editing capabilities. Visual Q&A and Captioning for production workloads are also generally available for all customers. Visual Q&A provides new ways to engage with image-based data like retail products or image libraries. This new capability can give you answers to questions about an image, helping you analyze large amounts of data quickly, and it can even help the visually impaired understand images or graphs that they wouldn’t be able to otherwise. Captioning, meanwhile, can make it easy to generate relevant descriptions for your images. Captions can help with indexing and searching, as well as assigning image descriptions to product listings on eCommerce websites. 

“Imagen is beginning to power key capabilities within Omni, Omnicom’s open operating system, that will enable 17,000+ trained and certified users to create audience-driven customized images in minutes. Imagen has been instrumental in offering a scalable platform for image generation and customization. Integrating it into our platform allows us to expand the scope of audience-powered creative inspiration, at a scale that wasn’t previously possible,” said Art Schram, Annalect Chief Product Officer at Omnicom. “We’re starting to adopt the latest features like styles and fine tuning, and engineering data-driven prompts. We look forward to continuing to provide our users relevant visual inspiration in a responsible way.”

“The latest improvements in Imagen’s product preservation capabilities are a perfect match for Typeface’s focus on personalized AI for brands,” explained Vishal Sood, Head of Product at Typeface. “By combining Google Vertex AI’s Imagen with Typeface’s brand-personalized AI, we are able to help enterprises to create 10x personalized content in a fraction of time.”

Google Shopping recently built an application called Product Studio using Imagen on Vertex AI. Product Studio can enable merchants to create rich product images quickly and easily, at a fraction of the time it takes to do professional product photo shoots. “We’re excited about the feedback we’re getting from merchants in our early pilots, who say that Product Studio, which leverages Imagen on Vertex AI, helps them generate and publish lifestyle product photos directly to their product catalogs,” says Jeff Harrell, Google’s Senior Director of Product Management for Merchant Shopping. 

Announced in May, PaLM 2 is a family of models that power dozens of Google products, including Bard and Duet AI in Google Cloud. With the PaLM 2 for Chat model, now generally available, you can leverage Google’s PaLM’s variety of abilities for multi-turn chat applications, such as shopping assistants, customer support agents, and more. 

ThoughtSpot, provider of a widely-adopted business intelligence platform, is using PaLM 2 to build a new feature in ThoughtSpot for Google Sheets called “AI Explain,” which can instantly generate explanations of charts, visuals, and anomalies, and will launch new conversational AI and ML-enabled predictive forecasting capabilities into its analytics platform.

With Codey, your organization’s developers can accelerate a wide variety of coding tasks, helping to empower them to work efficiently and close skills gaps. The model enables not only code completion and code generation capabilities, but also chat to help with debugging, documentation, learning new concepts, and more. Since launching in preview in May, we’ve added additional programming languages including Go, Google Standard SQL, Java, Javascript, Python, and Typescript. We’ve also improved the quality of code responses and increased serving capacity, enabling your developers with the right tools to enter the era of generative engineering.  

“Security and privacy are key to incorporating AI into the software development lifecycle,” said David DeSanto, Chief Product Officer at GitLab. “GitLab leverages Vertex AI to deliver new, AI-powered features with a privacy-first approach, including the ability to run our own models and leverage Codey foundation models built on top of PaLM 2. The GitLab DevSecOps platform empowers organizations to harness the benefits of AI for faster software delivery, while ensuring their data, intellectual property, and source code are protected.”

Originally released in May in preview, Chirp is a version of our 2 billion-parameter speech model, which was trained on millions of hours of audio and supports over 100 languages. Chirp achieves 98% accuracy on English and relative improvement of up to 300% in languages with less than 10 million speakers. Whether the use case involves customer support, transcriptions, or voice control, Chirp can help your organization communicate with customers and constituents inclusively, by engaging audiences in their native languages. 

Last but not least, our Multimodal Embeddings API, now in preview, can unlock an array of new applications, such as image and text-based recommendations, by enabling the processing of text and images interchangeably. This capability complements our Text Embeddings API, which became generally available in June, and remains a recommended choice for those with fully text-based use cases. Multimodal Embeddings API makes it possible to categorize images and text together and can be crucial for use cases like retail recommendation systems that can provide relevant outputs from both images of products and text descriptions.

Match generative AI with infrastructure 
Beyond access to models and tools for building generative AI apps, you need infrastructure to make sure your apps can scale and reliably perform — ideally without running into daunting compute costs or management overhead that distracts your technical talent from building innovative products. Google Cloud offers the choice and power to run smaller models running finite tasks at the lowest latency levels, as well as to run large models capable of cutting-edge experiments. 

As our large language model customers are looking to scale up their projects and applications using our models, they often need assurances that their requests will be serviced with acceptable performance. This is especially critical for delivering real-time applications where customer service is paramount. Starting in August, Vertex AI will support provisioned, dedicated generative AI capacity that can deliver guaranteed throughput. This feature can be especially beneficial to customers who have a high volume of sustained workloads.  

Leverage generative AI while protecting data and privacy 
One capability enabled by Google Cloud is the ability to customize models using your own data. Vertex AI can help customers keep their data protected, secure, and private. When a company tunes a foundation model in Vertex AI, private data, model outputs, and prompts can be kept private, and they are never used in the foundation model training corpus. We recently published a whitepaper, “Adaptation of Large Foundation Models,” which outlines how we help protect customer data. 

Auditability and compliance are essential to helping ensure the security and privacy of customer data. We also engage in comprehensive GDPR privacy efforts, including our transparency commitments for customer data usage and the support for our customer’s Data Protection Impact Assessments (DPIAs). Now, we’re excited to support HIPAA compliance for many of our generally available models on Vertex AI, so that healthcare and life science customers with whom we have a Business Associate Agreement can run workloads with Protected Health Information (PHI) data on Google Cloud. 

Innovate responsibly 
Our AI Principles put beneficial use, user safety, and avoidance of harms above business outcomes and are embedded in how we develop our AI products. We’ve conducted extensive reviews on our generative AI products to identify potential risks and have developed guardrails to mitigate these impacts. For example, to address concerns around safety, we’ve implemented safety filters for bias, toxicity, and other harmful content. We also equip our customers with the tools they need to help reduce risk within their applications and provide recommendations to help navigate responsible AI. 

Bring the power of generative AI to your organization

With both a wide selection of foundation models and extensive, enterprise-grade platform capabilities, Vertex AI continues to unlock ways for your business or organization to access foundation models, tune them on your proprietary data, and leverage them for differentiated apps and digital experiences. To take the next step, visit our product page or reach out to our sales representatives to gain access to our latest capabilities.