Dataplex: Google Cloud’s Intelligent Data Fabric to Manage Data and Analytics at Scale
4860
Of your peers have already read this article.
2:30 Minutes
The most insightful time you'll spend today!
Enterprises are struggling to make high quality data easily discoverable and accessible for analytics, across multiple silos, to a growing number of people and tools within their organization. They are often forced to make tradeoffs—to move and duplicate data across silos to enable diverse analytics use cases or leave their data distributed but limit the agility of decisions.
Today we are excited to announce Dataplex, an intelligent data fabric that provides a way to centrally manage, monitor, and govern your data across data lakes, data warehouses and data marts, and make this data securely accessible to a variety of analytics and data science tools.
Dataplex provides an integrated analytics experience, bringing together the best of Google Cloud and open source tools, so you can rapidly curate, secure, integrate, and analyze data at scale. With built-in data intelligence using Google Artificial Intelligence (AI) and machine learning (ML) capabilities and a flexible consumption model, you can now spend less time wrestling with infrastructure and more time focused on driving business outcomes.
Dataplex enables you to:
- Achieve freedom of choice to store data wherever you want for the right price/performance and choose the best analytics tools for the job, including Google Cloud and open source analytics technologies such as Apache Spark and Presto.
- Enforce consistent controls across your data to ensure unified security and governance
- Take advantage of the built-in data intelligence using Google’s best in class AI/ML capabilities to automate much of the manual toil around data management and get access to higher quality data.
Early customers like Equifax, Loblaw, and ANZ are excited about using Dataplex to address data management complexity.
“Dataplex will greatly simplify the existing analytics workflows within Equifax with its unified data fabric and single interface for policy management and governance across all our analytics data. Its built-in data discovery and data quality features will ensure that our data scientists and analysts always have access to high quality data that they can trust. Dataplex aligns well with our enterprise data strategy and we are excited to partner with Google Cloud on this.”
—Kumar Menon, SVP, Data Fabric & Decision Science Technology, Equifax.
“Loblaw is Canada’s food and pharmacy leader, and we are excited to be an early adopter of Dataplex. We could significantly benefit from Dataplex as it provides a single pane of glass for end-to-end data management and governance. We are particularly interested in improving platform resilience and data quality by detecting anomalies as early as possible in the data pipeline with the help of Dataplex.”
—Elton Martins, Senior Director of Data Insights & Analytics, Loblaw
“We are undergoing a major data transformation at ANZ, bringing together our various data assets and building a cohesive data ecosystem for customer benefit. Dataplex’s vision and capabilities align well with our current data strategy to build a unified data fabric for all our analytics and AI/ML use cases. We are excited to partner with GCP on Dataplex and test the product in private preview.”
—Ashish Shekhar, Head of Technology – Enterprise Analytics & Applied AI, ANZ
Dataplex is built for distributed data. We are starting with data stored in Google Cloud Storage and BigQuery, with support for other data sources coming soon. It provides a workflow-driven experience helping you build an open data platform and make data easily accessible to your end users while ensuring your policies and best practices are consistently enforced.
Organizing and curating your data
One of the core tenets of Dataplex is letting you organize and manage your data in a way that makes sense for your business, without data movement or duplication. For that, we are providing logical constructs like lakes, data zones and assets. Those constructs enable you to abstract away the underlying storage systems and become the foundation for setting policies around data access, security, lifecycle management, and so on.
For example, you can create a lake per department within your organization (Retail, Sales, Finance, etc.) and create data zones that map to data readiness and usage (landing, raw, curated_data_analytics, curated_data_science, etc.).
Once you have your lakes and zones setup, you can now attach data to these zones as assets. You can add data from different types of storage (e.g. GCS Bucket and BigQuery dataset) under the same zone. You can also attach data across multiple projects under the same zone.
You can ingest data into your lakes and zones using the tools of your choice including services such as Dataflow, Data Fusion, Dataproc, Pub/Sub or choose from one of our partner products. Dataplex provides built-in 1-click templates for common data management tasks.
Securing your data
Dataplex enables you to define and enforce consistent policies across your data, irrespective of where it physically resides. Data owners can easily set up policies for specific data domains based on business needs, without thinking about where the data is stored while data stewards get global visibility into governance policies and permissions across their data.
You can apply security and governance policies for your entire lake, a specific zone, or an asset. Dataplex maps the policies to the underlying storage and pushes down permissions to the storage layer to provide end-to-end secure data access. Additionally, you can secure not just data but also related artifacts like notebooks, scripts, and models using the same set of access policies.
Making high quality data available for analytics and data science
One of the biggest differentiators for Dataplex is our data intelligence capabilities using Google’s best in class AI/ML technologies. As you bring the data under management, Dataplex will automatically harvest the metadata for both structured and unstructured data, with built-in data quality checks. All of the metadata is automatically registered in a unified metastore, and made available for search and discovery. It is also published to Bigquery, Dataproc Metastore, and Data Catalog – ensuring that you have the same consistent data context and access across your tools.
For example, when you write parquet files to a Google Cloud Storage bucket – Dataplex will automatically extract metadata of these files, detect a tabular schema, including hive-style partitions, run data quality checks, and make this data queryable in BigQuery as an external table and from any open source or partner application – with the same consistent security and access policies you defined at the logical data layer.
Your data scientists and analysts now have secured access to this data that meets your quality bar and governance rules via the tools of their choice, without needing any additional processing.
One-click access to collaborative analytics
Dataplex provides fully managed, one-click analytics environments enabling you to use the power of Apache Spark and BigQuery with support for other engines coming in the future.
As data administrators, you now have the flexibility to pre-configure these environments with the right cost and financial governance measures without taking on the overhead of managing and maintaining the infrastructure required to power these environments. You can easily configure different environments for different types of workloads and share it with multiple users using their IAM credentials. Dataplex manages the provisioning, monitoring, scaling, and shutdown of these environments.
As data scientists, analysts, and engineers, you now have a turn-key experience to run your analysis using notebooks and a SQL workbench. You can search for notebooks and scripts alongside data, save and share your work with other users, and schedule your notebooks or scripts for recurring workloads – all using the same integrated experience within Dataplex.
Building an open platform with industry leaders
We are partnering with industry leaders such as Accenture, Collibra, Confluent, Informatica, HCL, Starburst, NVIDIA, Trifacta, and others to build an open platform to power analytics at scale. Our partners are excited about the capabilities that Dataplex will provide:
“Collibra is excited to partner with Dataplex to provide data governance and data quality for consistent controls across distributed data. Pairing Collibra’s multi-cloud and hybrid solution with Dataplex allows enterprises to securely open up access to more, higher quality data for users and analytics using a single unified view.”
—Jim Cushman, Chief Product Officer, Collibra
“Dataplex builds on Google Cloud’s commitment to open source by integrating with Apache Kafka®, a leading open source platform for event streaming. We at Confluent, the platform for data in motion that completes Apache Kafka® to be enterprise-ready, are excited to partner with Dataplex to enable customers to bring together distributed, real-time data and build a unified data fabric for end to end analytics.”
—Paul Mac Farland, VP, Head of Customer Solutions and Innovation, Confluent
“We are excited to partner with Google Cloud’s Dataplex team as we look to provide our joint customers an integrated and open data fabric for analytics at scale. Extending Dataplex’s data management and data quality capabilities with Starburst Enterprise will accelerate time to value for enterprises looking to connect distributed data without having to move data.”
—Justin Borgman, CEO and Co-founder, Starburst
Next Steps
Dataplex is now available in preview for a select number of customers. For more information, visit our website or watch the recording. If you would like to sign up, please click here.
How Google Cloud and SAP Address Global Supply Chain Initiatives
3514
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
With SAP Sapphire kicking off today in Orlando, we’re looking forward to seeing our customers and discussing how they can make core processes more efficient and improve how they serve their customers.
One thing is certain to be top of mind – the global supply chain challenges facing the world today. It’s affecting every business across every industry, from common household items that once filled store shelves and are now on backorder, to essential goods and services like food and medical treatments, which are at risk. Even cloud-native companies are making changes to ensure they have the insights, equipment, and other assets they need to continue serving customers.
We are proud to work with SAP on many initiatives that are driving results for our customers and helping them run more intelligent and sustainable companies. I’d like to highlight three of these important initiatives and how they are helping address global supply chain challenges.
Enabling more efficient migrations of critical workloads
We know a key barrier to entry in the cloud is the ability to easily migrate from on-premises environments. Our cloud provides a safe path to help companies including Johnson Controls, PayPal, and Kaeser Compressor to digitize and solve large, complex business problems, reduce costs, scale without cycles of investment, and gain access to key services and capabilities that can unlock value and enable growth.
Singapore-based shipping company Ocean Network Express (ONE) has become more agile by running their mission-critical SAP workloads on Google Cloud and using our data analytics to improve operational efficiency and make faster decisions. They have gone from an on-premises data warehouse solution that would take a full day loading data from SAP S/4HANA, to using our BigQuery solution that delivers business insights in minutes.
Since The Home Depot moved its critical SAP workloads to Google Cloud, the company has been able to shorten the time it takes to prepare a supply chain use case from 8 hours to 5 minutes by using BigQuery to analyze large volumes of internal and external data. This helps improve forecast accuracy and more effectively replenish inventory by being able to create a new plan when circumstances change unexpectedly with demand or a supplier.
Accelerating cloud benefits through RISE and LiveMigration
At Google Cloud, we have dedicated programs to help migrate SAP and other mission-critical workloads to our cloud with our Cloud Acceleration Program for SAP.
For SAP customers moving to Google Cloud, we provide LiveMigration to provide superior uptime and business continuity. LiveMigration eliminates downtime required for planned infrastructure maintenance. This means that your SAP system continues running even when Google Cloud is performing planned infrastructure maintenance upgrades thus ensuring superior business continuity for your mission critical workloads.
We are also proud to be a strategic partner with the RISE with SAP program, which helps accelerate cloud migration for SAP’s global customer base while minimizing risks along the migration journey. This program provides solutions and expertise from SAP and technology ecosystem partners to help companies transform through process consulting, workload migration services, cloud infrastructure, and ongoing training and support. To secure your mission critical workloads, SAP and Google Cloud can provide a 99.9% uptime SLA as part of the RISE with SAP program.
Many large manufacturers have taken advantage of RISE with SAP to forge a secure, proven path to our cloud, including Energizer Holdings Inc., a leading manufacturer and distributor of primary batteries, portable lights, and auto care products. Energizer has turned to RISE with SAP on Google Cloud to power its move to SAP S/4HANA. The company wants to automate essential business processes, improve customer service, and boost innovation. It had been using a private cloud solution but needed to gain flexibility while better containing costs.
“SAP S/4HANA for central finance will help us automate essential business processes, improve customer service, and fuel innovation that grows our company’s leadership position globally. We selected RISE with SAP to begin our journey to SAP S/4HANA and maintain the freedom and flexibility to move at our own pace,” said Energizer Chief Information Officer Dan McCarthy.
Another example is global automotive distributor Inchcape, which moved its mission-critical sales, marketing, finance, and operations systems and data to Google Cloud. With its diverse data sets now in a single, secure cloud platform, Inchcape is applying Google Cloud AI and ML capabilities to manage and analyze its data, automate operations, and ultimately transform the car ownership experience for millions.
“Google Cloud’s close relationship with SAP and its strong technical expertise in this space were a big pull for us,” said Mark Dearnley, Chief Digital Officer at Inchcape. “Ultimately, we wanted a headache-free RISE with SAP implementation and to unlock value for auto makers and consumers in all our regions, while continuing to have the choice and flexibility to modernize our 150-year old business in a way that works for us.”
A new intelligence layer for all SAP Google Cloud customers
When moving mission-critical workloads to the cloud, companies not only need to migrate safely, they also need to quickly realize value, which we enable with Google Cloud Cortex Framework — a layer of intelligence that integrates with SAP Business Technology Platform (SAP BTP). Google Cloud Cortex Framework provides reference architectures, deployment accelerators, and integration services for analytics scenarios.
Like many large e-commerce companies, Mercado Libre experienced skyrocketing transactions that more than doubled in 2020 as people sheltered at home during the pandemic, and they are anticipating more growth. The Google Cloud Cortex Framework is enabling Mercado Libre to respond, run more efficiently, and make faster, data-driven decisions.
Continued partnership to support organizations around the world
Our longstanding partnership with SAP continues to yield exciting innovations for our customers, and we’re honored to work with them to help customers address the ongoing impact of global supply chain challenges. We’re looking forward to sharing new insights and innovations at SAP Sapphire this week, and to listening and learning from you about your plans and challenges, and how we can best support your transformation to the cloud.
Explore The New Era of Flexibility: Streamlined AWS-to-Google Cloud Migration
1347
Of your peers have already read this article.
3:30 Minutes
The most insightful time you'll spend today!
As an IT leader, you’re asked to do it all: innovate and optimize your tech stack for business outcomes — all while being secure and compliant. It takes heroic efforts to achieve innovation and progress while also tightening budgets and teams. This is why many of you are considering migrating applications to Google Cloud, for benefits like scalability and flexibility, security and compliance, disaster recovery and business continuity, and cutting-edge technologies at lower costs.
To help you do this, we suggest using Google Cloud’s Migrate to Virtual Machines – part of Migration Center. This managed cloud service lets you lift and shift workloads at scale to Google Cloud Compute Engine with minimal changes and risk.
And recently, we rolled out our latest release which introduces support for workload migration from AWS to Google Compute Engine. With this addition, you can now migrate both your on-prem and AWS workloads at scale. This means centralized management for your end to end workload migration journey from both sources via Cloud Console or APIs.
Simple and easy migrations from AWS and VMware sources to Google Cloud
Migration of AWS EC2 instances directly to Google Compute Engine using Migrate to Virtual Machines follows a well established and easy journey, which means a minimal learning curve for users who are already migrating workloads from VMware. Workload migration is agent-less, which means you do not need to access or alter workloads as a prerequisite for migration, allowing you to execute zero-touch migrations. Migrate instance data with no interruptions to the running workload at the source for a fast cutover to Google Cloud. In addition, our end-to-end cloud console interface surfaces your AWS EC2 inventory, migrations, and groups so you can execute migrations without ever leaving the cloud console interface.
Large-scale migrations
Completing a large-scale migration project in a timely manner calls for careful planning and streamlined migration sprints. The Migrate to Virtual Machines’ Groups construct enables you to group source VMs together in the planning phase. When it’s time to execute the planned migration, VM Groups let you execute migration operations on a group level, or on a subset of the group, streamlining the process at scale.
Minimal downtime and risk
Application uptime is key to keeping your business running. Every migration with this latest release of the service periodically replicates data from the source workload to the destination without manual steps or interruptions to the running workload, minimizing workload downtime and enabling fast cutover to Google Cloud. You can also launch non-disruptive migration tests — referred to as test-clones — to help you validate that these workloads will work properly in the cloud before cutting over. This helps avoid issues that might have otherwise been costly or disruptive to your business.
How the service works
Migrations simply work, at scale, in a managed service fashion. With Migrate to Virtual Machines, there’s no requirement to provision or manage migration-specific resources in the cloud. The service uses replication-based migration technology to lift and shift workloads from source environments to Google Cloud. The Migrate Connection replicates source VM disk snapshots in the background with no interruption to the source workload. Replicated data is encrypted in transit and at rest, and when you instantiate a migrating VM using a test-clone or cut-over, the service seamlessly adapts your source VM operating system to boot and run natively in the cloud — including configuring network settings and deploying Google Cloud guest packages.
The migration journey of an EC2 instance — or VMware VM — to Google Cloud is comprised of the following steps:
1. Onboarding a source VM for migration: Onboard one or more VMs for migration from the source environment fleet.
2. Configure landing zone target: You can migrate an instance to any Google Cloud project in your environment and update landing zone details at any time before executing a test-clone or cutover.
3. Initiate VM data replication of source workload: Migrate to Virtual Machines periodically replicates instance disks to the cloud with no interruption to the source instance. You can control replication frequently and pause or resume at any point in time.
4. Test migrating instance: Test-clone creates a copy of your source instance in the defined landing zone to validate the migrating instance in the cloud before executing a cut-over. You can repeat the test-clone multiple times to multiple landing zones for thorough validation
5. Cutover migrating instance: Cutover operation shuts down your source instance and then performs the short final sync to Google Cloud. Migrated VM is instantiated in the target landing zone.
Getting started with Migrate to Virtual Machines
It’s quick and easy to start migrating your AWS EC2 instances and on-premises VMs today:
- Enable the vmmigration API in a Google Cloud project
- Create an AWS source in your environment
- Onboard and initiate replication of instance data from source
- Set migrating instance target details.
- Perform non-disruptive tests of your migrating instance using test-clone
- Cutover your instance to the cloud with minimal down time
You can also visit our website to learn more about Migrate to Virtual Machines. If you know you have to migrate in 2023 but aren’t sure how to get started, you can sign up for a free discovery and assessment of your current IT landscape so we can help craft the ideal migration plan for you and your business.
3335
Of your peers have already downloaded this article
10:00 Minutes
The most insightful time you'll spend today!
Any organization that has made a significant investments in Paas or IaaS capabilities requires a FinOps (financial operations) strategy. It involves linking their cloud migration business cases with value metrics, creating detailed cost visibility dashboards and having an automated expense control to ensure value realization from the cloud transformation journey. To fast-track your FinOps journey on Google Cloud, experts offer a detailed intro on the FinOps concept, the five Cloud FinOps pillars and important metrics for business value realization in this whitepaper.
To keep up with the digital transformations and demands of an ever-evolving virtual workforce, Google Cloud has distilled insights from several organizations that began their cloud migration journey and the FinOps Foundation community. Download the ‘Maximize Business Value with Cloud FinOps’ document to build a solid foundation for your organization’s cloud FinOps.
3542
Of your peers have already watched this video.
1:30 Minutes
The most insightful time you'll spend today!
See How Rémy Cointreau Drives Customer Centricity with SAP on Google Cloud
Rémy Cointreau is a French, family-owned business group whose origins date back to 1724. Rémy Cointreau is working to be a more customer centric organization. In order to fulfill this goal and to modernize, they determined they needed to get away from infrastructure management and decided to move their SAP landscape to Google Cloud.
Additionally Rémy Cointreau wanted to become more data centric. Learn how operations that used to take five weeks now take five minutes. Learn how Rémy Cointreau is leveraging live data analysis and is preparing for the future with SAP on Google Cloud.
4044
Of your peers have already watched this video.
15:00 Minutes
The most insightful time you'll spend today!
Modernize your Windows Workloads by Migrating them to Google Cloud
Google has plenty to offer when it comes to migrating and modernizing traditional enterprise Windows workloads to the cloud. Explore different approaches for re-hosting, modernizing, and transforming Windows applications, and the benefits of moving to Google Cloud.
Learn from demos on some of the cutting-edge technologies that can offload some of the operational IT burden into optimized managed services.
More Relevant Stories for Your Company
Transitioning from Amazon Glacier to Google Cloud
This comprehensive guide provides a detailed process for migrating archival workloads from Amazon Glacier to Google Cloud Storage Nearline in the Indian context. The process involves carefully planning data retrieval and staging strategies to ensure an efficient and cost-effective migration. The whitepaper includes: Different storage methods on Amazon Glacier and
Google Leads the Pack in Cloud Data Warehouse: Forrester Research
Cloud data warehouse solutions are changing the way firms build and support data platforms for insights. From provisioning a cloud data warehouse in minutes without requiring any technical expertise to allowing business analysts and other nontechnical users to access, store, and process large amounts of data for insights, they allow
Three German Retail Firms Choose Google Cloud to Migrate SAP Workloads for Business Transformation
The retail industry is rapidly evolving, with customers demanding exceptional digital experiences, and retailers adjusting their businesses to be more efficient. With many retailers relying on SAP for critical business functions like digital transactions, finance, supply chains and more, this shift has led them to look for ways to modernize
Trading and Investment Companies will Increase Consumption of Cloud Services: Study Confirms
While some traditional financial services companies have more slowly transitioned to the cloud, capital markets firms have embraced cloud computing across their entire value chains — front-, middle-, and back-office. We wanted to understand the dynamics behind this rapid adoption, the most common use cases, and the types of technology
