Enterprises are struggling to make high quality data easily discoverable and accessible for analytics, across multiple silos, to a growing number of people and tools within their organization. They are often forced to make tradeoffs—to move and duplicate data across silos to enable diverse analytics use cases or leave their data distributed but limit the agility of decisions.  

Today we are excited to announce Dataplex, an intelligent data fabric that provides a way to centrally manage, monitor, and govern your data across data lakes, data warehouses and data marts, and make this data securely accessible to a variety of analytics and data science tools.  

Dataplex provides an integrated analytics experience, bringing together the best of Google Cloud and open source tools, so you can rapidly curate, secure, integrate, and analyze data at scale. With built-in data intelligence using Google Artificial Intelligence (AI) and machine learning (ML) capabilities and a flexible consumption model, you can now spend less time wrestling with infrastructure and more time focused on driving business outcomes.

Dataplex enables you to:

• Achieve freedom of choice to store data wherever you want for the right price/performance and choose the best analytics tools for the job, including Google Cloud and open source analytics technologies such as Apache Spark and Presto.
• Enforce consistent controls across your data to ensure unified security and governance
• Take advantage of the built-in data intelligence using Google’s best in class AI/ML capabilities to automate much of the manual toil around data management and get access to higher quality data.

Early customers like Equifax, Loblaw, and ANZ are excited about using Dataplex to address data management complexity.

“Dataplex will greatly simplify the existing analytics workflows within Equifax with its unified data fabric and single interface for policy management and governance across all our analytics data. Its built-in data discovery and data quality features will ensure that our data scientists and analysts always have access to high quality data that they can trust. Dataplex aligns well with our enterprise data strategy and we are excited to partner with Google Cloud on this.”
—Kumar Menon, SVP, Data Fabric & Decision Science Technology, Equifax.

“Loblaw is Canada’s food and pharmacy leader, and we are excited to be an early adopter of Dataplex. We could significantly benefit from Dataplex as it provides a single pane of glass for end-to-end data management and governance. We are particularly interested in improving platform resilience and data quality by detecting anomalies as early as possible in the data pipeline with the help of Dataplex.”
—Elton Martins, Senior Director of Data Insights & Analytics, Loblaw

“We are undergoing a major data transformation at ANZ, bringing together our various data assets and building a cohesive data ecosystem for customer benefit. Dataplex’s vision and capabilities align well with our current data strategy to build a unified data fabric for all our analytics and AI/ML use cases. We are excited to partner with GCP on Dataplex and test the product in private preview.”
—Ashish Shekhar, Head of Technology – Enterprise Analytics & Applied AI, ANZ 

Dataplex is built for distributed data. We are starting with data stored in Google Cloud Storage and BigQuery, with support for other data sources coming soon. It provides a workflow-driven experience helping you build an open data platform and make data easily accessible to your end users while ensuring your policies and best practices are consistently enforced.

Organizing and curating your data

One of the core tenets of Dataplex is letting you organize and manage your data in a way that makes sense for your business, without data movement or duplication. For that, we are providing logical constructs like lakes, data zones and assets. Those constructs enable you to abstract away the underlying storage systems and become the foundation for setting policies around data access, security, lifecycle management, and so on. 

For example, you can create a lake per department within your organization (Retail, Sales, Finance, etc.) and create data zones that map to data readiness and usage (landing, raw, curated_data_analytics, curated_data_science, etc.). 

Once you have your lakes and zones setup, you can now attach data to these zones as assets. You can add data from different types of storage (e.g. GCS Bucket and BigQuery dataset) under the same zone. You can also attach data across multiple projects under the same zone.

You can ingest data into your lakes and zones using the tools of your choice including services such as Dataflow, Data Fusion, Dataproc, Pub/Sub or choose from one of our partner products. Dataplex provides built-in 1-click templates for common data management tasks. 

Securing your data 

Dataplex enables you to define and enforce consistent policies across your data, irrespective of where it physically resides. Data owners can easily set up policies for specific data domains based on business needs, without thinking about where the data is stored while data stewards get global visibility into governance policies and permissions across their data. 

You can apply security and governance policies for your entire lake, a specific zone, or an asset. Dataplex maps the policies to the underlying storage and pushes down permissions to the storage layer to provide end-to-end secure data access. Additionally, you can secure not just data but also related artifacts like notebooks, scripts, and models using the same set of access policies.

Making high quality data available for analytics and data science

One of the biggest differentiators for Dataplex is our data intelligence capabilities using Google’s best in class AI/ML technologies.  As you bring the data under management, Dataplex will automatically harvest the metadata for both structured and unstructured data, with built-in data quality checks. All of the metadata is automatically registered in a unified metastore, and made available for search and discovery. It is also published to Bigquery, Dataproc Metastore, and Data Catalog – ensuring that you have the same consistent data context and access across your tools.

For example, when you write parquet files to a Google Cloud Storage bucket – Dataplex will automatically extract metadata of these files, detect a tabular schema, including hive-style partitions, run data quality checks, and make this data queryable in BigQuery as an external table and from any open source or partner application – with the same consistent security and access policies you defined at the logical data layer. 

Your data scientists and analysts now have secured access to this data that meets your quality bar and governance rules via the tools of their choice, without needing any additional processing.

One-click access to collaborative analytics

Dataplex provides fully managed, one-click analytics environments enabling you to use the power of Apache Spark and BigQuery with support for other engines coming in the future. 

As data administrators, you now have the flexibility to pre-configure these environments with the right cost and financial governance measures without taking on the overhead of managing and maintaining the infrastructure required to power these environments. You can easily configure different environments for different types of workloads and share it with multiple users using their IAM credentials. Dataplex manages the provisioning, monitoring, scaling, and shutdown of these environments. 

As data scientists, analysts, and engineers, you now have a turn-key experience to run your analysis using notebooks and a SQL workbench. You can search for notebooks and scripts alongside data, save and share your work with other users, and schedule your notebooks or scripts for recurring workloads – all using the same integrated experience within Dataplex.

Building an open platform with industry leaders

We are partnering with industry leaders such as Accenture, Collibra, Confluent, Informatica, HCL, Starburst, NVIDIA, Trifacta, and others to build an open platform to power analytics at scale. Our partners are excited about the capabilities that Dataplex will provide:

“Collibra is excited to partner with Dataplex to provide data governance and data quality for consistent controls across distributed data. Pairing Collibra’s multi-cloud and hybrid solution with Dataplex allows enterprises to securely open up access to more, higher quality data for users and analytics using a single unified view.”
—Jim Cushman, Chief Product Officer, Collibra

“Dataplex builds on Google Cloud’s commitment to open source by integrating with Apache Kafka®, a leading open source platform for event streaming. We at Confluent, the platform for data in motion that completes Apache Kafka® to be enterprise-ready, are excited to partner with Dataplex to enable customers to bring together distributed, real-time data and build a unified data fabric for end to end analytics.”
—Paul Mac Farland, VP, Head of Customer Solutions and Innovation, Confluent

“We are excited to partner with Google Cloud’s Dataplex team as we look to provide our joint customers an integrated and open data fabric for analytics at scale. Extending Dataplex’s data management and data quality capabilities with Starburst Enterprise will accelerate time to value for enterprises looking to connect distributed data without having to move data.”
—Justin Borgman, CEO and Co-founder, Starburst

Next Steps

Dataplex is now available in preview for a select number of customers. For more information, visit our website or watch the recording. If you would like to sign up, please click here.

Organizations that collect geospatial data can use that information to understand their operations, help make better business decisions, and power innovation. Traditionally, organizations have required deep GIS expertise and tooling in order to deliver geospatial insights. In this post, we outline some ways that geospatial data can be used in various business applications. 

Assessing environmental risk 

Governments and businesses involved in insurance underwriting, property management, agriculture technology, and related areas are increasingly concerned with risks posed by environmental conditions. Historical models that predict natural disasters like pollution, flooding, and wildfires are becoming less accurate as real-world conditions change. Therefore, organizations are incorporating real-time and historical data into a geospatial analytics platform and using predictive modeling to more effectively plan for risk and to forecast weather.

Selecting sites and planning expansion

Businesses that have storefronts, such as retailers and restaurants, can find the best locations for their stores by using geospatial data like population density to simulate new locations and to predict financial outcomes. Telecom providers can use geospatial data in a similar way to determine the optimal locations for cell towers. A site selection solution can combine proprietary site metrics with publicly-available data like traffic patterns and geographic mobility to help organizations make better decisions about site selection, site rationalization, and expansion strategy.

Planning logistics and transport

For freight companies, courier services, ride-hailing services, and other companies that manage fleets, it’s critical to incorporate geospatial context into business decision-making. Fleet management operations include optimizing last-mile logistics, analyzing telematics data from vehicles for self-driving cars, managing precision railroading, and improving mobility planning. Managing all of these operations relies extensively on geospatial context. Organizations can create a digital twin of their supply chain that includes geospatial data to mitigate supply chain risk, design for sustainability, and minimize their carbon footprint. 

Understanding and improving soil health and yield

AgTech companies and other organizations that practice precision agriculture can use a scalable analytics platform to analyze millions of acres of land. These insights help organizations understand soil characteristics and help them analyze the interactions among variables that affect crop production. Companies can load topography data, climate data, soil biomass data, and other contextual data from public data sources. They can then combine this information with data about local conditions to make better planting and land-management decisions. Mapping this information using geospatial analytics not only lets organizations actively monitor crop health and manage crops, but it can help farmers determine the most suitable land for a given crop and to assess risk from weather conditions.

Managing sustainable development

Geospatial data can help organizations map economic, environmental, and social conditions to better understand the geographies in which they conduct business. By taking into account environmental and socio-economic phenomena like poverty, pollution, and vulnerable populations, organizations can determine focus areas for protecting and preserving the environment, such as reducing deforestation and soil erosion. Similarly, geospatial data can help organizations design data-driven health and safety interventions. Geospatial analytics can also help an organization meet its commitments to sustainability standards through sustainable and ethical sourcing. Using geospatial analytics, organizations can track, monitor, and optimize the end-to-end supply chain from the source of raw materials to the destination of the final product.

What’s next

Google Cloud provides a full suite of geospatial analytics and machine learning capabilities that can help you make more accurate and sustainable business decisions without the complexity and expense of managing traditional GIS infrastructure. Get started today by learning how you can use Google Cloud features to get insights from your geospatial data, see Geospatial analytics architecture.

Acknowledgements: We’d like to thank Chad Jennings, Lak Lakshmanan, Kannappan Sirchabesan, Mike Pope, and Michael Hao for their contributions to this blog post and the Geospatial Analytics architecture.

If you’ve opened an account at a bank, applied for a government benefit, or provided a proof of age document on an ecommerce website, chances are you’ve had to share a physical or digital copy of a Driver’s License or a passport as proof of your identity. For businesses or public sector organizations that need this information to provide services, processing images of identity documents has long been a time- and resource-intensive process that requires extensive human intervention. Solutions exist to help digitally capture the data, but they require extensive human intervention that impacts the speed and cost of processing and ultimately the time to service customers.

The Google Cloud Document AI family of solutions has been designed to help solve some of the hardest problems for data capture at scale by extracting structured data from unstructured documents to help reduce processing costs and improve business speed and efficiency. Today, we’re announcing the general availability of identity parsers that bring the power of Document AI to customer verification, KYC, and other identity-based workflows.

With Document AI for Identity, businesses can leverage automation to extract information from identity documents with a high degree of accuracy, without having to bear the cost and turnaround time of manual tasks by a service provider. Document AI for Identity leverages artificial Intelligence to provide a set of pre-trained models that can parse identity and supports US driver’s licenses (generally available), US passports (generally available), French driver’s licenses (preview) and French National ID cards (preview), with more documents to be added from around the world over the coming months.

When our customers process high-volume workloads or complex workflows, they need a high degree of accuracy, since getting the first step wrong can derail the entire workflow. The introduction of special parsers for Identity processing can help solve one of the most commonly required document processing needs that our financial services and public sector customers face.

Along with the identity parsers, Google Cloud is also offering its “Human in the Loop” service, in which verification for a subset of identity documents can be automatically assigned to a pool of humans (internal or external) for manual review, based on confidence scores.

While there are multiple industries and applications that could benefit from Document AI for Identity, we’ve seen two main kinds of applications being adopted during the solution’s preview. One is around processing ID cards uploaded as unstructured images at scale, so that enterprises can have IDs on file. The second use case is to perform advanced checks on identity documents to validate their authenticity and / or to detect fraud. Google Cloud’s fraud detector API (which is currently in preview) can complement Document AI for Identity and apply an extra layer of normalization to help validate the identity as a government-issued ID by checking for suspicious words, image manipulation, and other common issues with forged identity documents. With new versions of driver’s licenses being frequently released, Document AI for identity uses specialized models and constantly-updated training data to help make sure the parsers can offer a high degree of accuracy. For all use cases, Document AI does not retain any customer data after completing the processing request (successfully or with an error).

Check out this demo and visit the Document AI for Identity landing page for more information on how Document AI can help solve your identity processing needs, and ask your Google Cloud account team to help you integrate Identity Document AI into your workflows.

For practitioners who’re interested in trying out Identity DocAI, check out our companion practitioner blog for step by step instructions on how to get started.

When it comes to the adoption of artificial intelligence (AI), we have reached a tipping point. Technologies that were once accessible to only a few are now broadly available. This has led to an explosion in AI investment. However, according to research firm McKinsey, for AI to make a sizable contribution to a company’s bottom line, they “must scale the technology across the organization, infusing it in core business processes” — and based on conversations with our customers, we couldn’t agree more.

While investments in pure data science continue to be essential for many, widespread adoption of AI increasingly involves a category of applications and services that we call AI Agents. These are technologies that let customers apply the best of AI to common business challenges, with limited technical expertise required by employees, and include Google Cloud products like Document AI and Contact Center AI. Today, at Google Cloud Next ‘22, we’re announcing new features to our existing AI Agents and a brand new one with Translation Hub.

“AI is becoming a key investment for many companies’ long term success. However, most companies are still in the experimental phases with AI and haven’t fully put the technology into production because of long deployment timelines, IT staffing needs, and more,” said Ritu Jyoti, group vice president, worldwide AI and automation research practice global AI research lead, at IDC. “Organizations need AI products that can be immediately applied to automate processes and solve business problems. Google Cloud is answering this problem by providing fully managed, scalable AI Agents that can be deployed fast and deliver immediate results.”

Translation Hub: An enterprise-scale translation AI Agent

At I/O this year, we announced the addition of 24 new languages to Google Translate to allow consumers in more locations, especially those whose languages aren’t represented in most technology, to help reduce communication barriers through the power of translation. Businesses strive for the same goals, but unfortunately it is often out of reach due to the high costs that come with scaling translation.

That’s why today, we are announcing Translation Hub, our AI Agent that provides customers with self-service document translation. With 135 languages, Translation Hub can create impactful, inclusive, and cost-effective global communications in a few clicks.

With Translation Hub, now researchers are able to share their findings instantly across the world, goods and services providers can reach underserved markets, and public sector administrators can reach more members of their communities in a language they understand — all of which ultimately help make for a more connected, inclusive world.

Translation Hub brings together Google Cloud AI technology, like Neural Machine Translation and AutoML, to help make it easy to ingest and translate content from the most common enterprise document types, including Google Docs and Slides, PDFs, and Microsoft Word. It not only preserves layouts and formatting, but also provides granular management controls such as support for post-editing human-in-the-loop feedback and document review.

“In just three months of using Translation Hub and AutoML translation models, we saw our translated page count go up by 700% and translation cost reduced by 90%,” said Murali Nathan, digital innovation and employee experience lead, at materials science company Avery Dennison. “Beyond numbers, Google’s enterprise translation technology is driving a feeling of inclusion among our employees. Every Avery Dennison employee has access to on-demand, general purpose, and company-specific translations. English language fluency is no longer a barrier, and our employees are beginning to broadly express themselves right in their native language.”

Document AI: A document processing AI Agent to automate workflows

Every organization needs to process documents, understand their content, and make them available to the appropriate people. Whether it’s during procurement cycles involving invoices and receipts, contract processes to close deals, or for general increases in efficiency, Document AI simplifies and automates various document processing. With two new features launching today, Document AI can allow employees to focus on higher impact tasks and better serve their own customers.

For example, payments provider Libeo used Document AI to uptrain an invoice parser with 1,600 documents and increase its testing accuracy from 75.6% to 83.9%. “Thanks to uptraining, the Document AI results now beat the results of a competitor and will help Libeo save ~20% on the overall cost for model training over the long run,” said Libeo chief technology officer, Pierre-Antoine Glandier.

Today, we’re announcing these new features to our existing Document AI Agent:

• Document AI Workbench can remove the barriers around building custom document parsers, helping organizations extract fields of interest that are specific to their business needs. Relative to more traditional development approaches, it requires less training data and offers a simple interface for both labeling data and one-click model training.
• Document AI Warehouse can eliminate the challenges that many enterprises face when tagging and extracting data in documents by bringing Google’s Search technologies to Document AI. This feature can make it simpler and easier to search for and manage documents like workflow controls to accommodate invoice processing, contracts, approvals, and custom workflows.

Contact Center AI: A contact center AI Agent to improve customer experiences

Scaling call center support can be expensive and difficult, especially when implementing AI technologies to support representatives. Contact Center AI is an AI Agent for virtually all contact center needs, from intelligently routing customers, to facilitating handoffs between virtual and human customer support representatives, to analyzing call center transcripts for trends and much more.

Just days ago, we announced that Contact Center AI Platform is now generally available to provide additional deployment choice and flexibility. With this addition to Contact Center AI, we are furthering our commitment to providing an AI Agent that can assist organizations to quickly scale their contact centers to improve customer experiences and create value via data-driven decisions.

Dean Kontul, division chief information officer at KeyBank, had this to say about powering their contact center with Contact Center AI from Google Cloud: “With Google Cloud and Contact Center AI, we will quickly move our contact center to the Cloud, supporting both our customers and agents with industry-leading customer experience innovations, all while streamlining operations through more efficient customer care operations.”

Start delivering business results with AI Agents, today!

If you’re ready to get started with Translation Hub, this Next ‘22 session has the details, including a deeper dive into Avery Dennison’s use of the AI Agent.

To learn more about our Document AI announcements, check out our session with Commerzbank, “Improve document efficiency with AI,” as well as “Transform digital experiences with Google AI powered search and recommendations.”

And, to explore Contact Center AI Platform, watch “Delight customers in every interaction with Contact Center AI,” featuring more insight into KeyBank’s use case.