May was a very busy month for data analytics product innovation. If you didn’t have the chance to attend our inaugural Data Cloud Summit, video replays of all our sessions are now available so feel free to watch them at your own pace. 

In this blog, I’d like to share some background behind the innovations we released in May, why we built them the way we did, and the type of value they can bring your company and your team.

But first, a huge thank you!

This week, we had the honor to announce that Google has been named a Leader in The Forrester Wave™: Streaming Analytics, Q2 2021 report. Forrester gave Dataflow a score of 5 out of 5 across 12 different criteria, stating: “Google Cloud Dataflow has strengths in data sequencing, advanced analytics, performance, and high-availability”. 

Google has more than a decade of experience in building real-time and internet-scale systems for its own needs, and we are excited to see that our ability to provide customers with a reliable, scalable, and performant platform is bearing fruit. 

This announcement comes on the back of the release of The Forrester Wave™: Cloud Data Warehouse, Q1 2021 report, which also named Google Cloud as a Leader.

We couldn’t be more excited about the recognition and appreciate all your feedback and trust in the work that we do to support your goal in accelerating data-powered innovation.

Innovation galore

Your feedback and your passion is the fuel that drives our ambition to deliver more and better services to you. That’s why, this year, we didn’t want to wait until Google Cloud Next to share some great products we have been working on. On May 26, our team announced a slew of new products, services and programs. Watch a quick summary below:

https://youtube.com/watch?v=DG1mOPMXJvw%3Fenablejsapi%3D1%26

Meeting you where you are

An important design principle behind all of our services is “meeting you where you are”. This means we aim to provide you with the tools and software you need to innovate on your own terms. Here are three new services that will help you do just that:

Datastream

Datastream, our new serverless change data capture (CDC) and replication service, allows your company to synchronize data across heterogeneous databases, storage systems, and applications reliably and with minimal latency to support real-time analytics, database replication, and event-driven architectures. Datastream delivers change streams from Oracle and MySQL databases into Google Cloud services such as BigQuery, Cloud SQL, Cloud Storage, and Cloud Spanner, saving time and resources while ensuring your data is accurate and up-to-date. 

• Under the hood, Datastream reads CDC events (inserts, updates, and deletes) from source databases, and writes those events with minimal latency to a data destination. It leverages the fact that each database source has its own CDC log—binlog for MySQL and LogMiner for Oracle—which it uses for its own internal replication and consistency purposes. 
• Datastream integrates with purpose-built and extensible Dataflow templates to pull the change streams written to Cloud Storage, and create up-to-date replicated tables in BigQuery for analytics. It also leverages Dataflow templates to replicate and synchronize databases into Cloud SQL or Cloud Spanner for database migrations and hybrid cloud configurations. 
• Datastream also powers a Google-native Oracle connector in Cloud Data Fusion’s new replication feature for easy ETL/ELT pipelining. By delivering change streams directly into Cloud Storage, customers can leverage Datastream to implement modern, event-driven architectures.

Looker and BigQuery Omni on Microsoft Azure

Research on multi cloud adoption is unequivocal — 92% of businesses in 2021 report having a multi cloud strategy. We want to continue supporting your choice by providing the flexibility you need to see your strategy through. 

• This past month, we introduced Looker, hosted on Microsoft Azure. For the first time, you can now choose Azure, Google Cloud, or AWS for your Looker instance. You can also self-host your Looker instance on-premises.
• We also introduced BigQuery Omni for Azure, which along with last year’s introduction of BigQuery Omni for AWS, will help you access and securely analyze data across Google Cloud, AWS, and Azure. 

The cost of moving data between cloud providers isn’t sustainable for many, and it’s still difficult to seamlessly work across clouds. BigQuery Omni represents a new way of analyzing data stored in multiple public clouds, which is made possible by BigQuery’s separation of compute and storage. By decoupling these two, BigQuery provides scalable storage that can reside in Google Cloud or other public clouds, and stateless resilient compute that executes standard SQL queries. 

• Unlike competitors, BigQuery Omni doesn’t require you to move or copy your data from one public cloud to another, where you might incur egress costs. You also benefit from the same BigQuery interface on Google Cloud, enabling you to query data stored in Google Cloud, AWS, and Azure without any cross-cloud movement or copies of data. 
• BigQuery Omni’s query engine runs the necessary compute on clusters in the same region where your data resides. For example, you can query Google Analytics 360 Ads data stored in Google Cloud and query logs data from your ecommerce platform and applications that are stored in AWS S3 and/or Microsoft Azure. 

Then, using Looker, you can build a dashboard that allows you to visualize your audience behavior and purchases alongside your advertising spend. 

Dataplex

We understand that most organizations still struggle to make high-quality data easily discoverable and accessible for analytics, across multiple silos, to a growing number of people and tools within their organization. 

They are often forced to make tradeoffs. For instance, moving and duplicating data across silos to enable diverse analytics use cases or leaving their data distributed but limiting the agility of decisions. 

• Dataplex provides an intelligent data fabric that enables you to centrally manage, monitor, and govern your data across data lakes, data warehouses, and data marts, while also ensuring data is securely accessible to a variety of analytics and data science tools. 
• One of the core tenets of Dataplex is letting you organize and manage your data in a way that makes sense for your business, without data movement or duplication. For that, we provide logical constructs like lakes, data zones, and assets. These constructs enable you to abstract away the underlying storage systems and become the foundation for setting policies around data access, security, lifecycle management, and so on. 
• For example, you can create a lake per department within your organization (e.g. Retail, Sales, Finance, etc.) and create data zones that map to data readiness and usage (e.g. landing, raw, curated_data_analytics, curated_data_science, etc.). 

Once you have your lakes and zones setup, you can attach data to these zones as assets. You can add data from different types of storage (e.g. GCS Bucket and BigQuery dataset) under the same zone. You can also attach data across multiple projects under the same zone. You can ingest data into your lakes and zones using the tools of your choice, including services such as Dataflow, Data Fusion, Dataproc, Pub/Sub, or choose from one of our partner products. Dataplex comes with built-in 1-click templates for common data management tasks. 
To find out more about Dataplex, head to cloud.google.com/dataplex or watch the video below:

https://youtube.com/watch?v=bbFeAt7cw1g%3Fenablejsapi%3D1%26

Helping you innovate everyday

Sharing data is hard. Traditional data sharing techniques use batch data pipelines that are expensive to run, create late arriving data, and can break with any changes to the source data. These techniques also create multiple copies of data, which brings unnecessary costs and can bypass data governance processes. They also fail to offer features for data monetization, such as managing subscriptions and entitlements. Altogether, these challenges mean that organizations are unable to realize the full potential of transforming their business with shared data.

Analytics Hub

To address these limitations, we are introducing Analytics Hub, a new fully managed service that helps organizations unlock the value of data sharing, leading to new insights and increased business value. 

This new service is built on the tremendous experience and feedback we have received over the years. For example, BigQuery has had cross-organizational, in-place data sharing capabilities since its inception in 2010—and the functionality is very popular. Over a 7-day period in April, we had over 3,000 different organizations sharing over 200 petabytes of data. These numbers don’t include data sharing between departments within the same organization.

Analytics Hub takes sharing to the next level, making it easy for you to publish, discover, and subscribe to valuable datasets that you can combine with your own data to derive unique insights. 

This includes: 

• Shared datasets: As a data publisher, you create shared datasets that contain the views of data that you want to deliver to your subscribers. Data subscribers can search through the datasets that are available across all exchanges for which they have access and subscribe to relevant datasets. In addition, the publisher can track subscribers, disable subscriptions, and see aggregated usage information for the shared data.
• Curated, self-service data exchanges: Exchanges are collections used to organize and secure shared datasets. By default, exchanges are completely private, but granular roles and permissions make it easy to deliver data to the right audience—whether internal or public. 

This is just the beginning for Analytics Hub. Please sign up for the preview, which is scheduled to be available in the third quarter of 2021.

Dataflow Prime

At Google Cloud, we have the great privilege of working with some of the most innovative organizations in the world. And this work provides us with a unique perspective into the future of big data processing. Dataflow Prime is a new platform based on a serverless, no-ops, and auto-tuning architecture that brings unparalleled resource utilization and radical operational simplicity to big data processing. This new service introduces a large number of exciting capabilities but I’d like to highlight three key aspects of the product:

• Vertical Autoscaling: Dataflow Prime dynamically adjusts the compute capacity allocated to each worker based on utilization, detecting when jobs are limited by worker resources and automatically adding more resources. Vertical Autoscaling works hand in hand with Horizontal Autoscaling to seamlessly scale workers to best fit the needs of the pipeline. As a result, it no longer takes hours or days to determine the perfect worker configuration to maximize utilization. 
• Right Fitting: Each stage of a pipeline typically has a different resource requirement than the others. Until now, either all workers in the pipeline would have had the higher memory and GPU, or none of them would. Pipelines either had to waste resources or suffer slower workloads. Right Fitting solves this problem by creating stage-specific pools of resources, optimized for each stage. 
• Smart Recommendations: Smart Recommendations automatically detects problems in your pipeline and shows potential fixes. For example, if your pipeline is running into permissions issues, a Smart Recommendation will detect which IAM permissions you need to enable to unblock your job. If you are using an inefficient coder in your job, Smart Recommendations will surface more performant coder implementations that can help you save on costs.

What’s next

We’re excited to hear your thoughts and feedback about all these exciting new services. I would also highly recommend that you connect with members of the community to learn more about their story and journey. A good example to start with is the Data To Value customer panel we produced at our inaugural Data Cloud Summit with the Chief Data Officers of Keybank and Rackspace. You can watch it for free below:

https://youtube.com/watch?v=ITI2Q3MkxuA%3Fenablejsapi%3D1%26

IT leaders today are being asked to simultaneously support their company’s infrastructure, find opportunities for growth, and meet their goals with fewer resources and smaller budgets than before. Recently we highlighted three customers who are leveraging Google Cloud VMware Engine to achieve these goals while lowering their TCO and transforming their organization. 

It’s because of these successful customer outcomes that we have been awarded the 2023 VMware Cloud Innovation and SaaS Transformation partner achievement award for delivering solutions that accelerate customers’ digital transformation journey. We’re honored to receive this award and continue to stay focused on delivering tremendous value to our customers.

In the past few months, we’ve also made several updates to Google Cloud VMware Engine. Today’s post provides a recap of the latest milestones that make it easier for you to migrate and run your vSphere workloads in a cloud-first, enterprise-class VMware environment in Google Cloud. 

Back in September 2022, we announced a number of updates including the preview of API/CLI support (which is now available). In February 2023, we also talked about how to use NetApp CVS as datastores for VMware Engine.

Key updates this time around include:

Availability of VMware Engine in Delhi, Santiago and Milan regions: This brings the availability of VMware Engine to 17 regions worldwide, each supporting 4 9’s of uptime SLA for clusters 5 or more, serving the needs of our regional and multi-national customers. In addition, we have also added a second zone in the London region.

Filestore datastore support for VMware Engine: Generally Available in all VMware Engine regions, you can use Filestore High Scale and Enterprise tier instances as external NFS datastores for VMware Engine nodes. Filestore is VMware certified as an NFS datastore with VMware Engine. You can size compute and storage capacity independently to meet your workload requirements for your storage-intensive VMs. You can also leverage vSAN for low-latency VM requirements and scale Filestore from TBs to PBs for the capacity hungry VMs. If interested in this feature, please contact your Google account team.

Stretched private clouds: These private clouds stretch across two data zones and a witness zone all within the same Google Cloud region. Stretched private clouds use vSphere and vSAN stretched clusters to provide compute and storage high availability against zone-level failures. This capability is now available in Frankfurt and Sydney regions. Learn more here.

Zerto solution version 9.5u1 support: This recovery solution allows critical infrastructure and application virtual machines (VMs) to be replicated continuously from your on-premises vCenter to your private cloud. Learn more about setting up Zerto here.

Google Cloud Backup and Disaster Recovery (GCBDR): GCBDR is available to protect applications running in VMware Engine, and can be managed within the Google Cloud Console. We recently launched GCBDR under Google Cloud Platform Terms of Service simplifying customers’ purchasing and support experience. 

vTPM support: Google Cloud VMware Engine private clouds now support the addition of a Trusted Platform Module (TPM) 2.0 virtual cryptoprocessor to a virtual machine. You can add vTPMs to VMs by following VMware instructions or upgrading your existing VMs to include a vTPM. You can read more about this in the VMware blog.

This brings us to the end of our updates this time. For the latest updates to the service, please bookmark our release notes.

At Google Cloud, our customers are at the forefront of digital transformation—launching entirely new businesses and products built in the cloud, redefining entire industries with data and artificial intelligence, delivering innovative new consumer experiences, or committing to sustainable new ways of doing business.

We’re committed to our customers’ success, and over the past two years we’ve invested significantly in providing ongoing and integrated support through our customer care portfolio, including launching new Premium and Mission Critical Support offerings that allow us to monitor, prevent, and mitigate impacts quickly, while delivering the fastest response times in the industry.

Today, I’m proud to unveil several new resources and offerings for our customers, including a new Custom AI Solutions practice, new Global Delivery Centers, expanded Executive Briefing Centers, and new leaders to help continue to drive our organization forward.

Helping businesses innovate with a new Custom AI Solutions practice

We are investing in services and offerings to help our customers innovate and drive innovative change to business processes, culture, and customer experiences with Google Cloud products and services. Artificial intelligence (AI) and machine learning (ML) technologies are foundational for many such digital transformations, and to help customers create real business value with these technologies, we’re excited to launch a Custom AI offering to address our customers’ most critical innovation needs. 

AI and ML technologies are foundational to digital transformations, yet they are not “one size fits all.” Each customers’ problems and opportunities are unique; a rideshare company will use AI differently than a brick and mortar retailer, or a healthcare company, or an insurance firm. To help organizations deploy AI and ML more effectively, we’re launching a new Custom AI Solutions practice, offering customers custom-built AI and ML solutions built into Vertex AI; access to Google’s engineering expertise; and predictable, subscription-based pricing. 

Our teams are already partnering closely with early customers to build and deploy custom AI solutions. For instance, USAA, the large North American insurer, is using Google Cloud ML to process near-real-time damage estimates based on digital images to create a streamlined operations experience.

Learn more about our Custom AI Solutions practice offering here, and how we work together with our customers here. To get in touch, please contact our sales team.

Launching new Global Delivery Centers

Our drive to digitally transform our customers’ businesses is often manifested through our Global Delivery Centers, which expand our professional consulting and emerging practices capabilities available to customers and partners around the world.

The teams at our Global Delivery Centers help customers get up-and-running on Google Cloud quickly and cost effectively, and consult with customers to rapidly build capacity in areas like data analytics, hybrid and multi-cloud, artificial intelligence, and machine learning. More importantly, they help customers successfully execute projects in support of their most mission-critical business objectives. Critically, these centers also help our global partner ecosystem quickly ramp their Google Cloud practices, and get fast, expert consultation for their customers too.

In 2022, we aim to triple the size of our Global Delivery Center teams in Argentina, Poland, and India. In addition, we’ll invest in building deep Google Cloud talent in Mexico and Portugal, furthering our commitment to the industry’s best expert consultation for customers and partners around the world.

Expanding our Executive Briefing Centers footprint

In addition to our Global Delivery Centers, we’re also pleased to expand our resources and facilities that enable digital and face-to-face meetings and working sessions with our customers. This year, we will launch four new Executive Briefing Centers, located on Google Cloud campuses in London, Paris, Singapore, and Munich. 

These centers provide an opportunity to listen to our customers, share the best of Google Cloud’s solutions, and inspire digital transformation, in conversations facilitated by Google Cloud leadership, engineers, and industry experts. By bringing this experience to our customers in-region we can foster deeper partnerships and develop cloud solutions that meet their requirements for security, privacy, and digital sovereignty without compromising on functionality or innovation. 

Adding new leadership to enable customer success

Finally, I’m excited to welcome two new leaders to Google Cloud on the Customer Experience team, who will help scale our Delivery Centers and deliver exceptional experiences for our customers.

Heading our Global Delivery Center experience is Sunil Rao. Sunil comes to Google Cloud from Accenture, where he spent 18-plus years working with large technology customers across many industry verticals and managed large global teams in Accenture Advanced Technology Center. Sunil will also lead our Technical Onboarding Center, which helps businesses around the world get up to speed with technologies that are critical to understanding customer and business process contexts, and delivering great experiences.

Additionally, Lee Moore is joining Google Cloud to lead Customer Experience in North America. Lee spent nearly 30 years at Accenture in various leadership positions, including services integration for complex problem-solving, product development across a number of industry verticals, and building long-term customer relationships.

You’ll be hearing much more from us in the coming months, as we build out even more powerful and effective cloud-based services and offerings, work with customers to deliver new analytics- and AI-based tools and services, and work with our growing list of partners to help ensure customer success, across the globe.