Gartner® named Google as a Leader in the 2022 Magic Quadrant™ for Cloud AI Developer Services report. This evaluation covered Google’s language, vision and structured data products including AutoML, all of which we deliver through Google Cloud. We believe this recognition is a reflection of the confidence and satisfaction that customers have in our language, vision, and AutoML products for developers. Google remains a Leader for the third year in a row, based upon the completeness of our vision and our ability to execute.

Developers benefit in many ways by using Cloud AI services and solutions. Customers recognize the advantages of Google’s AI and ML services for developers, such as Vertex AI, BigQuery ML, AutoML and AI APIs. In addition, customers benefit from the pace of progress in the field of Responsible AI and actionable ethics processes applied to all customer and partner solutions leveraging Google Cloud technology, as well as our core architecture including the Vertex AI platform, vision, conversational AI, language and structured data, and optimization services and key vertical industry solutions.

We believe that our ‘Leader’ placement validates this vision for AI developer tools. Let’s take a closer look at some of the report findings.

ML tools purpose-built for developers

Google’s machine learning tools have been built by developers, for developers, based on the groundbreaking research generated from Google Research and DeepMind. This developer empathy drives product development, which supports the developer community to achieve deep value from Google’s AI and ML services. An example of this is the unification of all of the tools needed for building, deploying and managing ML models into one ML platform, Vertex AI, resulting in accelerated time to production. They also cite BigQuery ML, AutoML for language, vision video and tabular data) and prebuilt ML APIs (such as speech and translation) as having high utility for developers at all levels of ML expertise to build custom AI and quickly infuse AI into their applications.

Leading organizations like OTOY, Allen Institute for AI and DeepMind (an Alphabet subsidiary) choose Google for ML, and enterprises like Twitter, Wayfair and The Home Depot shared more about their partnership with Google in their recent sessions at Google Next 2021.

Responsible AI principles and practices

Responsible AI is a critical component of successful AI. A 2020 study commissioned by Google Cloud and the Economic Intelligence Unit highlighted that ethical AI does not only prevent organizations from making egregious mistakes, but that the value of responsible AI practices for competitive edge, as well as talent acquisition and retention are notable. At Google, we not only apply our ethics review process to first party platforms and solutions, to ensure that our services design-in responsible AI from the outset, we also consult with customers and partners based on AI principles to deliver accountability and avoid unfair biases. In addition, our best-in-class tools provide developers with the functionality they need to evaluate fairness and biases in datasets and models. Our Explainable AI tools such as model cards provide model transparency in a structured, accessible way, and the What-If Tool is essential for developers and data scientists to evaluate, debug and improve their ML models.

Clear and understandable product architecture

Google Cloud’s investment in our ML product portfolio has led to a comprehensive, integrated and open offering that spans breadth (across vision, conversational AI, language and structured data, and optimization services) and depth (core AI services, with features such as Vertex AI Pipelines and Vertex Explainable AI built on top). Industry-specific solutions tailored by Google for retail, financial services, manufacturing, media and healthcare customers, such as Recommendations AI, Visual Inspection AI, Media Translation, Healthcare Data Engine, add another layer leveraging this foundational platform to help organizations and users adopt machine learning solutions more easily.

At Google Cloud, we refuse to make developers jump through hoops to derive value out of our technology; instead, we bring the value directly to them by ensuring that all of our AI and ML products and solutions work seamlessly together. To download the full report, click here. Get started on Vertex AI and talk with our sales team.

Disclaimer:

Gartner, Magic Quadrant for Cloud AI Developer Services, Van Baker, Arun Batchu, Erick Brethenoux, Svetlana Sicular, Mike Fang, May 23, 2022.

Gartner and Magic Quadrant are registered trademarks of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved.

Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

Today, more and more applications are being deployed in containers on Kubernetes—so much so that we’ve heard Kubernetes called the Linux of the cloud.

Despite all that growth on the application layer, the data layer hasn’t gotten as much traction with containerization. That’s not surprising, since containerized workloads inherently have to be resilient to restarts, scale-out, virtualization, and other constraints. So handling things like state (the database), availability to other layers of the application, and redundancy for a database can have very specific requirements. That makes it challenging to run a database in a distributed environment. 

However, the data layer is getting more attention, since many developers want to treat data infrastructure the same as application stacks.

Operators want to use the same tools for databases and applications, and get the same benefits as the application layer in the data layer: rapid spin-up and repeatability across environments. In this blog, we’ll explore when and what types of databases can be effectively run on Kubernetes.

Before we dive into the considerations for running a database on Kubernetes, let’s briefly review our options for running databases on Google Cloud Platform (GCP) and what they’re best used for.

• Fully managed databases. This includes Cloud Spanner, Cloud Bigtable and Cloud SQL, among others. This is the low-ops choice, since Google Cloud handles many of the maintenance tasks, like backups, patching and scaling. As a developer or operator, you don’t need to mess with them. You just create a database, build your app, and let Google Cloud scale it for you. This also means you might not have access to the exact version of a database, extension, or the exact flavor of database that you want.
• Do-it-yourself on a VM. This might best be described as the full-ops option, where you take full responsibility for building your database, scaling it, managing reliability, setting up backups, and more. All of that can be a lot of work, but you have all the features and database flavors at your disposal.
• Run it on Kubernetes. Running a database on Kubernetes is closer to the full-ops option, but you do get some benefits in terms of the automation Kubernetes provides to keep the database application running. That said, it is important to remember that pods (the database application containers) are transient, so the likelihood of database application restarts or failovers is higher. Also, some of the more database-specific administrative tasks—backups, scaling, tuning, etc.—are different due to the added abstractions that come with containerization.

Tips for running your database on Kubernetes

When choosing to go down the Kubernetes route, think about what database you will be running, and how well it will work given the trade-offs previously discussed.

Since pods are mortal, the likelihood of failover events is higher than a traditionally hosted or fully managed database. It will be easier to run a database on Kubernetes if it includes concepts like sharding, failover elections and replication built into its DNA (for example, ElasticSearch, Cassandra, or MongoDB). Some open source projects provide custom resources and operators to help with managing the database.

Next, consider the function that database is performing in the context of your application and business. Databases that are storing more transient and caching layers are better fits for Kubernetes. Data layers of that type typically have more resilience built into the applications, making for a better overall experience.  

Finally, be sure you understand the replication modes available in the database. Asynchronous modes of replication leave room for data loss, because transactions might be committed to the primary database but not to the secondary database(s). So, be sure to understand whether you might incur data loss, and how much of that is acceptable in the context of your application.

After evaluating all of those considerations, you’ll end up with a decision tree looking something like this:

How to deploy a database on Kubernetes

Now, let’s dive into more details on how to deploy a database on Kubernetes using StatefulSets.

With a StatefulSet, your data can be stored on persistent volumes, decoupling the database application from the persistent storage, so when a pod (such as the database application) is recreated, all the data is still there.

Additionally, when a pod is recreated in a StatefulSet, it keeps the same name, so you have a consistent endpoint to connect to. Persistent data and consistent naming are two of the largest benefits of StatefulSets. You can check out the Kubernetes documentation for more details.

If you need to run a database that doesn’t perfectly fit the model of a Kubernetes-friendly database (such as MySQL or PostgreSQL), consider using Kubernetes Operators or projects that wrap those database with additional features. Operators will help you spin up those databases and perform database maintenance tasks like backups and replication. For MySQL in particular, take a look at the Oracle MySQL Operator and Crunchy Data for PostgreSQL. 

Operators use custom resources and controllers to expose application-specific operations through the Kubernetes API. For example, to perform a backup using Crunchy Data, simply execute pgo backup [cluster_name]. To add a Postgres replica, use pgo scale cluster [cluster_name].

There are some other projects out there that you might explore, such as Patroni for PostgreSQL. These projects use Operators, but go one step further. They’ve built many tools around their respective databases to aid their operation inside of Kubernetes. They may include additional features like sharding, leader election, and failover functionality needed to successfully deploy MySQL or PostgreSQL in Kubernetes.

While running a database in Kubernetes is gaining traction, it is still far from an exact science. There is a lot of work being done in this area, so keep an eye out as technologies and tools evolve toward making running databases in Kubernetes much more the norm. 

When you’re ready to get started, check out GCP Marketplace for easy-to-deploy SaaS, VM, and containerized database solutions and operators that can be deployed to GCP or Kubernetes clusters anywhere.

Google Cloud Functions provides a simple and intuitive developer experience to execute code from Google Cloud, Firebase, Google Assistant, or any web, mobile, or backend application. Oftentimes that code needs secrets—like API keys, passwords, or certificates—to authenticate to or invoke upstream APIs and services.

While Google Secret Manager is a fully-managed, secure, and convenient storage system for such secrets, developers have historically leveraged environment variables or the filesystem for managing secrets in Cloud Functions. This was largely because integrating with Secret Manager required developers to write custom code… until now. We listened to customer feedback and today we are announcing Cloud Functions has a native integration with Secret Manager!

This native integration has many key benefits including:

• Zero required code changes. Cloud functions that already consume secrets via environment variables or files bundled with the source upload simply require an additional flag during deployment. The Cloud Functions service resolves and injects the secrets at runtime and the plaintext values are only visible inside the process.
• Easy environment separation. It’s easy to use the same codebase across multiple environments, e.g., dev, staging, and prod, because the secrets are decoupled from the code and are resolved at runtime.
• Supports the 12-factor app pattern. Because secrets can be injected into environment variables at runtime, the native integration supports the 12-factor pattern while providing stronger security guarantees.
• Centralized secret storage, access, and auditing. Leveraging Secret Manager as the centralized secrets management solution enables easy management of access controls, auditing, and access logs.

Cloud Functions’ native integration with Secret Manager is available in preview to all Google Cloud customers today. Let’s take a deeper dive into this new integration.

Example

Suppose the following cloud function invoked via HTTP uses a secret token to invoke an upstream API:

  const https = require('https');
const token = process.env.TOKEN;

exports.secretDemo = (req, res) => {
  https.get(`https://upstream-api.example.com?token=${token}`, (innerRes) => {
    innerRes.on('end', () => { res.send('OK') });
  }).on('error', (err) => {
    res.send(`Error: ${err}`);
  });
}

Without Cloud Functions’ native integration with Secret Manager, this function is deployed via:

  $ gcloud functions deploy "secretDemo" \
    --runtime "nodejs14" \
    --trigger-http \
    --allow-unauthenticated \
    --set-env-vars "TOKEN=abcd1234"

This approach has a number of drawbacks, the biggest of which being that anyone with viewer permissions on the Google Cloud project can see the environment variables set on a cloud function.

To improve the security of this code, migrate the secret to Secret Manager in the same project:

  $ gcloud secrets create "token" \
    --replication-policy "automatic" \
    --data-file - <<< "abcd1234"

Finally, without changing any code in the function re-deploy with slightly different flags:

  $ gcloud beta functions deploy "secretDemo" \
    --runtime "nodejs14" \
    --trigger-http \
    --allow-unauthenticated \
    --set-secrets "TOKEN=my_token:latest"

It’s truly that easy to migrate from hard-coded secrets to using secure secret storage with Secret Manager! To add even more layers of security, consider running each cloud function with a dedicated service account and practice the principle of least privilege. Learn more in the Secret Manager Best Practices guide.

Making security easy

Security and proper secrets management are core pillars of modern software development, and we’re excited to provide customers a way to improve the security of their cloud functions. To learn more about the new native integration, check out the Cloud Functions documentation. You can also learn more about Cloud Functions or learn more about Secret Manager.

In our conversations with technology leaders about data-driven transformation using Google Data Cloud –  industry’s leading unified data and AI solution – , one important topic is incorporating continuous intelligence to move from answering questions such as “What has happened? to questions like “What is happening?” and “What might happen?”. The core to this evolution is the need for an underlying data processing that not only provides powerful real-time capabilities for events happening close to origination, but also brings together existing data sources under one unified data platform to enable organizations to draw insights and take actions holistically. Dataflow, Google’s cloud-native data processing and streaming analytics platform, is a key component of any modern data and AI architecture and data transformation journey, along with BigQuery, Google’s internet-scale warehouse with built-in streaming, BI engine and ML; Pub/Sub, a global no-ops event delivery service; and Looker, a modern BI and embedded analytics platform. One of the key evaluation factors is potential economic value of Dataflow to their organization, particularly in the context of engaging other stakeholders is key for many of the leaders that we engage with. So we commissioned Forrester Consulting to conduct a comprehensive study on the impact that Dataflow had on their organization by interviewing actual customers . 

Today we’re excited to share our commissioned study conducted by Forrester Consulting, the Total Economic Impact™ of Google Cloud Dataflow, which allows data leaders to understand and quantify the benefits of Dataflow, and use cases it enables. Forrester conducted interviews with Dataflow customers to evaluate the benefits, costs, and risks of investing in Dataflow across an organization. Based on their interviews, Forrester identified major financial benefits across four different areas: business growth, infrastructure cost savings, data engineer productivity, and administration efficiency. In fact, Forrester found that customers adopting Dataflow can achieve a 55% boost in developer productivity and a 50% reduction in infrastructure costs. In fact, Forrester projects that customers adopting Dataflow can achieve a range of up to 171% Return on Investment (ROI) and a less than six months payback period. Customers can now use figures in the report to compute their own Return on Investment (ROI) and payback period.

“Dataflow is integral to accelerating time-to-market, decreasing time-to-production, reducing time to figure out how to use data for use cases, focusing time on value-add tasks, streamlining ingestion, and reducing total cost of ownership.” – Lead technical architect, CPG

Let’s take a deeper look at the ways that Forrester found that Dataflow can help you achieve your goals and unlock your business potential. 

Benefit #1: Increase data engineer productivity by 55%

Developers can choose among a variety of programming languages to define and execute data workflows. Dataflow also seamlessly integrates with other Google Cloud Platform and open source technologies to maximize value and applicability to a wide variety of use cases. Dataflow streamlined workflows with code reusability,dynamic templates, and the simplicity of a managed service. Engineers trusted pipelines to run correctly and adhere to governance. Data engineers avoided laborious issue-monitoring and remediation tasks that were common in the legacy environments such as poor performance, lack of availability, and failed jobs. Teams valued the language flexibility and open source base.

“Dataflow provided us with ETL replacement that opened limitless potential use cases and enabled us to do smarter data enhancement while data remains in motion.” — Director of data projects, financial services

Benefit #2: Reduce infrastructure costs by up-to 50% for batch and streaming workloads 

Dataflow’s serverless autoscaling and discrete control of job needs, scheduling, and regions eliminated overhead and optimized technology spending. Consolidating global data processing solutions to Dataflow further eliminated excess costs while ensuring performance, resilience, and governance across environments. Dataflow’s unified streaming and batch data platform gives organizations the flexibility to define either workload in the same programming model, run it on the same infrastructure, and manage it from a single operational management tool. 

“Our costs with our cloud data platform using Dataflow are just a fraction of the costs we faced before. Now we only pay for cloud infrastructure consumption because the open source base helps us avoid licensing costs. We spend about $120,000 per year with Dataflow, but we’d be spending millions with our old technologies.” – Lead technical architect, CPG

Benefit #3: Increase top-line revenue by improving customer experience and retention with payback time of < 6 months

Streaming analytics is an essential capability in today’s digital world to gain real-time actionable insights. Likewise, organizations must also have flexible, high- performance batch environments to analyze historical data for building machine learning models, business intelligence, and advanced analytics. Dataflow enabled real-time streaming use cases, improved data enrichment, encouraged data exploration,improved performance and resiliency, reduced errors, increased trust, and eliminated barriers to scale. As a result, organizations provided customers with more accurate, relevant, and in-the-moment data-backed services and insights — boosting customer experience, creating new revenue streams, and improving acquisition, retention, and enrichment.

“It’s already been proven that we are getting more business [with Dataflow] because we can turn around results faster for customers.” – VP of technology, financial services technology

“When we provide data to our customers and partners with Dataflow, we are much more confident in those numbers and can provide accurate data within a minute. Our customers and partners have taken note and commented on this. It’s reduced complaints and prevented churn.” – Senior software engineer, media

Other benefits 

Eliminated administrative overhead and toil

As a cloud-native managed service, all administration tasks such as provisioning, scaling, and updates are automatically handled by Google Cloud. Teams no longer need to manage servers and related software for legacy data processing solutions. Admins also streamlined processes for setting up data sources, adding pipelines, and enforcing governance.

Saved business operations costs for support teams and data end users

Dataflow improved the speed, quality, reliability, and ease of access to data for insights for general business users, saving time and empowering users to drive better data-backed outcomes. It also reduced support inquiry volume while automating manual job creation.

What’s next?

Download the Forrester Total Economic Impact study today to dive deep into the economic impact Dataflow can deliver your organization. We would love to partner with you to explore the potential Dataflow can unlock in your teams. Please reach out to our sales team to start a conversation about your data transformation with Google Cloud.