Setting up Data Pipelines Easily for Streaming and Non-Streaming Data - Build What's Next

3056

Of your peers have already listened to this podcast

25:30 Minutes

The most insightful time you'll spend today!

Podcast

Setting up Data Pipelines Easily for Streaming and Non-Streaming Data

We live in a world of incessant data. Our phones, factory equipment, smart watches, health devices, and other IoT devices are constantly streaming data. Some of the most popular uses of machine learning and AI are also based on streaming data. Think real-time credit card fraud detection, or software that helps customer care representatives figure out a customer’s state of mind, just from their voice.

Yet, many machine learning, AI, and big data initiatives tend to start with batch processing data. This can limit the business gains that machine learning can accrue.

Tools such as Google’s Cloud Dataflow can help.

Cloud Dataflow is a fully-managed service that’s integrated with Google Cloud Platform for executing batch and streaming data processing pipelines. It’s kind of a successor to systems like MapReduce. But it also brings in a lot of the more recent streaming technologies. It can do both traditional batch ETL-style processing as well as real-time applications.

In this podcast Frances Perry tells you what you can achieve with Cloud Dataflow, using example and anecdotes. Frances Perry is a software engineer at Google who spend many years working on Google’s internal data processing stack before joining the Cloud Dataflow team to make this technology available to external cloud customers.

Listen in now!

4424

Of your peers have already watched this video.

31:30 Minutes

The most insightful time you'll spend today!

Explainer

Smart analytics: Deep dive on roadmap

Data across organizations is growing and that organizations need a very strong analytics platform to leverage this data create insights and make real-time decisions on top of this data.

That’s driving the advent of three large trends. First is the convergence of data lakes and data warehouses, that’s enable organizations to maximize the value of their data.

Second, is the growing phenomena of real-time decision-making which is forcing enterprises to think of how they can support the needs of batch processing and streaming data.

Finally, there is the rise of artificial intelligence and machine learning, which allows enterprises to leverage their data and create competitive differentiation.

With this background, Sudhir Hasbe, Director of Product Management, Data Analytics, Google Cloud, walks us through Google Cloud’s smart analytics offerings—and what’s new.

He takes us on a tour through the technical value of Google Cloud’s smart analytics platform end-to-end. He provides a comprehensive overview and demos what’s new and what’s next in Google Cloud’s smart analytics portfolio across products like BigQuery, Dataflow, Dataproc, Data Fusion, PubSub, Data Catalog, Dataprep, and Looker.

2992

Of your peers have already watched this video.

39:00 Minutes

The most insightful time you'll spend today!

Case Study

Verizon Media Shows the Solution Architecture it Uses for a 100+ PB Analytics Platform

Verizon Media owns and operates more than a dozen brands including Yahoo Mail, Yahoo News, AOL, Huffington Post, TechCrunch, and Engadget among others.

These web properties are visited by millions of users on a daily basis.

In this session, Shakil Memon, Customer Engineer, Google Cloud and Nikhil Mishra, Sr Director, Engineering, Verizon Media, present how Verizon Media is generating actionable insights from the wealth of data that they have.

They will discuss:

  • Challenge with large scale and large volumes of data
  • Basic principles of a data warehousing and data analytics project
  • Showcase a reference architecture
  • A complete end-to-end solution architecture for building a 100+ PB internet-scale analytics platform on Google Cloud, including how Looker fits in the end-to-end solution and how actionable insights are generated from data.

They will also provide unique perspectives, behind-the-scenes thinking, and some insight on how the architecture has evolved in its current form over the years.

5369

Of your peers have already watched this video.

44:30 Minutes

The most insightful time you'll spend today!

Case Study

How Go-Jek, Indonesia’s First Billion-dollar Startup, Improved the Productivity of Data Scientists

Go-Jek, Indonesia’s first billion-dollar startup, has seen an incredible amount of growth in both users and data over the past two years. Many of the ride-hailing company’s services are backed by machine learning models hosted on Google Cloud Platform.

Models range from driver allocation, to dynamic surge pricing, to food recommendation, and process millions of bookings every day, leading to substantial increases in revenue and customer retention.

But senior executives at Go-Jek realized something: One of their most important and expensive resources, data scientists, were spending far too much time cleaning data. That wasn’t part of their remit and resulted in a waste of time and money.

As a COO, this a major concern for any company undertaking a machine learning initiative. Data scientists are hard to come by and their salaries have been on the rise for the last few years. Yet according to some reports data scientists spend upto 80% of their time just preparing data—not creating models.

Watch how operational teams at Go-Jek combined the right Google tools and processes to improve the productivity of their data scientists.

Blog

How Google Cloud & NGIS’ Partnership Powers Sustainability & Responsible Sourcing for Consumer Brands

6211

Of your peers have already read this article.

2:00 Minutes

The most insightful time you'll spend today!

TraceMark, a sourcing platform of NGIS built on Google Cloud helps CPG brands with responsible raw material sourcing by making their supply chain more agile, transparent and traceable. Read the to learn how this gears CPGs towards sustainability!

In the competitive world of consumer goods, sustainability matters more than ever. In recent Google survey, 82% of consumers said sustainability is more top of mind now than it was before COVID-191 and 78% said that big businesses have a role to play in helping to fight climate change.2 As well as delivering on customers’ heightened expectations, sustainable business practices can help organizations reduce waste and lower operational costs and even help attract top talent.

Not only is sustainability good for business performance and the bottom line, it’s great for the planet, too. And it’s something we care deeply about at Google. Since our earliest days, we have focused on developing services that significantly improve the lives of billions of people while operating our business in an environmentally sustainable way. In 2007, we were the first major company to become carbon neutral, and in 2017 we were the first to achieve 100% renewable energy. Today, we proudly operate the cleanest cloud in the industry. We’ve also been powering humanitarian, scientific, and environmental initiatives studying geospatial information and making it available for analysis via Google Earth Engine

Powering the future of responsible sourcing 

How does all this relate to CPG supply chains? The world relies on raw materials like palm oil, soy, and cocoa to produce the items we consume every day—such as coffee, chocolate, frozen foods, shampoo, toothpaste, cosmetics, and even household cleaning products. Yet as demand for these materials continues to grow, forests are under threat. Change is needed.

As sustainability moves further into the spotlight, in-demand crops face increased supplier scrutiny. Environmentally conscious companies want to know what percentage of their raw materials is sourced from deforestation-free suppliers—and how they can improve that number. Until recently, many CPG brands have found it hard to get real-time, reliable visibility into operations at a local supplier level, globally. 

Google Cloud, in partnership with NGIS, is helping brands gain a deeper understanding of raw material sourcing practices across supplier networks, so they can improve supplier performance and compliance in the fight against deforestation. The TraceMark solution, developed by NGIS, uses Google Earth Engine and BigQuery to analyze and visualize how suppliers behave over time. 

Google Earth Engine is the world’s largest archive of open Earth data. It combines a multi-petabyte catalog of satellite imagery and geospatial datasets with planetary-scale analysis capabilities to help people detect changes, map trends, and quantify differences on the Earth’s surface.With Earth Engine, organizations can analyze the potential social, and environmental impacts of the decision they make. 

Together, TraceMark, Google Earth Engine and BigQuery help companies visualize, monitor, and measure the impact of suppliers’ farming practices. The process starts with Earth Engine, which aggregates and harmonizes planetary data into images that it has been collecting for decades. Then, boundary maps for different suppliers are created and NGIS applies climate data science and machine learning to turn pixels into insights. Finally, BigQuery and Vertex AI -Google Cloud’s unified artificial intelligence platform are used to produce supplier scoring, which is integrated into downstream ERP and procurement systems to support more informed decision-making.

These efforts support the responsible sourcing of raw materials by making supply chains more agile, transparent and traceable.

cpg.jpg

“NGIS is thrilled to work with Google Cloud and its partners to enable business accountability for sustainable practices at all levels of their supply chain,” said Nathan Eaton, Executive Director, NGIS. “Google Earth Engine provides unique geospatial capability that [gives] leaders visibility and control over their environmental footprint and that of their suppliers in a way that was not previously possible.” 

Unilever commits to working towards a deforestation-free supply chain 

Unilever is a great example of a company committed to using technology in the quest to become more sustainable. Since 2020, Unilever has partnered with Google Cloud to use data for eco-friendly decision-making, particularly when it comes to sustainable commodity sourcing. 

By combining the power of cloud computing with satellite imagery and AI, we’re helping Unilever build a more holistic view of the forests, water cycles, and biodiversity that intersect its supply chain. By gaining a complete picture of these ecosystems, Unilever can detect deforestation while simultaneously prioritizing critical areas of forest and habitats in need of protection.

The cleanest cloud for your sustainable transformation

We are excited to bring Google Earth Engine and NGIS to our CPG customers to help them meet their sustainability goals. Looking ahead, we’re committed to furthering our own ambitious sustainability goals and empowering CPG brands with the technology to do more for our environment and our shared future. 

Contact your cloud seller to learn more about Tracemark sustainable sourcing solutions and how Google Cloud can help you advance your sustainability initiatives.

Explore TraceMark on the Google Cloud Marketplace

To know more about how we are helping CPGs transform digitally read this ebook


1. Google/C Space, BR, FR, DE, IN, MX, U.K., U.S., qualitative survey activity, n = 528, Nov. 24–Nov. 26, 2020.
2. Google/Ipsos, Google Sustainability, BR, FR, DE, IN, JP, U.K., U.S., n=16,959 online population 18–70, July 2021.

Blog

DocAI Lowers Customer’s Document Processing Cost by 60 Percent. Learn How

5625

Of your peers have already read this article.

2:00 Minutes

The most insightful time you'll spend today!

Document processing in most organizations require human intervention for review, data extraction and unlocking valuable insights. With Google Cloud Doc AI platform, document-intensive industries and businesses can end their guess work through automation, document processing accuracy, ML-based predictions and better UX at 60 percent lesser document processing cost.

Some of the most important data at your company isn’t living in databases, but in documents, and most business processes begin, involve or end with a document. 

Yet most companies are still manually entering data and reliant on guesswork to make sense of it all as the volume and variety of data explodes. Organizations are also leaving heaps of value on the table in the form of new and better customer experiences that can be unlocked with artificial intelligence (AI) applied to documents. 

The latest releases of Document (Doc) AI platformLending DocAI and Procurement DocAI, built on decades of AI innovation at Google, bring powerful and useful solutions to these challenges. Under the hood are Google’s industry-leading technologies:

  • Computer vision (including OCR) and Natural Language Processing (NLP) that creates pre-trained models for high-value, high-volume documents. 
  • Google Knowledge Graph to validate and enhance the fields in your documents.
  • Training and creation of your own custom document models. 
  • Human interaction with AI to ensure accuracy where needed.

Google Cloud DocAI platform, Lending DocAI and Procurement DocAI are now generally available. Thousands of customers have tried these products in the preview phase—and DocAI has already processed tens of billions of pages of documents across lending, insurance, government and other industries.

“The vast majority of enterprise content still resides in unstructured sources like documents. Google Cloud’s Document AI brings a fresh new perspective to the problem informed by the company’s decades of experience making sense of the largest unstructured corpus in the world—the world wide web.” Ritu Jyoti, VP of AI Research, IDC

Cut document processing costs by up to 60%

Lending DocAI helps banks, mortgage brokers and other lending institutions fast track the loan application process from weeks to days, dramatically reducing the cost of issuing a loan. And Procurement DocAI enables companies to automate procurement data capture at scale, lowering processing costs by up to 60%.

These solutions are built on DocAI platform, a unified console for document processing that lets you quickly access all parsers and tools. From the platform, you can automate and validate documents to streamline workflows, reduce guesswork, and keep data accurate and compliant. 

Get more value from AI with DocAI’s industry-specific solutions

According to Accenture’s AI: Built to Scale report: “Companies that scale successfully see 3x the return on their AI investments compared to those who have not fully rolled out AI capabilities.” 

Core to our strategy at Google Cloud is the creation of industry-specific solutions that help companies get maximum value out of their investments in AI. We announced Lending DocAI, our first solution designed specifically for the financial services industry, at the Mortgage Bankers Association convention last year. It processes borrowers’ income and asset documents using a set of specialized machine learning (ML) models, and automates routine document reviews so that mortgage providers can focus on more important work. 

Lending DocAI is now generally available and includes more specialized parsers for critical loan documents including paystubs, bank statements, and more. Our goal is to provide the right tools to help borrowers and lenders have a better experience and close home loans faster. For more, watch this video.

Procurement DocAI is also now generally available. This solution helps companies accelerate document processing for invoices, receipts, and other valuable documents in the procurement cycle. 

Automating data capture is helping our customers increase accuracy and also lower their procure-to-pay processing costs. We are continually expanding the types of documents Procurement DocAI can process—the latest is a utility parser for electric, water and other bills. In addition, Procurement DocAI leverages Google Knowledge Graph to validate and enrich parsed information to make the data even more useful. Check out this overview video for more details. 

One company that lives and breathes AI-enabled document management is AODocs. It uses Procurement DocAI to simplify invoice processing for enterprise customers and launched a new Gmail add-on, Invoice to Sheet, for SMB customers who just want to track their invoices in Google Sheets.

“Google Cloud’s Procurement DocAI service allows our document management platform to better automate the processing of invoices; AODocs customers who have tested our new account payables workflow estimate that the productivity of their A/P team has more than doubled, thanks to the reduction of manual data input brought by the Procurement DocAI.”Stéphan Donzé, Founder and CEO, AODocs

The new specialized parsers for Lending and Procurement DocAI can be used alongside our existing AutoML Text & Document Classification and AutoML Document Extraction services. These technologies provide a state-of-the-art toolset for creating new document models and have been widely deployed by customers in financial services and other industries. 

Partner to accelerate your AI deployment and results

Having the right partner to ease the complexity of rolling out your AI-strategy in mortgage document processing is critical to transforming your customers’ experience. We’re excited to announce a partnership with Mr. Cooper, a leader in mortgage servicing, to provide customers with more automation and workflow tools throughout their entire mortgage life cycle. As part of this agreement, both companies will collaborate on digitizing Mr. Cooper’s core mortgage platform, creating a more personal customer experience utilizing AI, and driving a broader culture of innovation to imagine and develop services and solutions that will transform the mortgage experience for American homeowners.

“Over the last few years, we have made substantial investments in our servicing technology and core mortgage platform that have revolutionized the customer experience, while providing dramatic efficiencies in operating cost. Our partnership with Google Cloud AI will build on those advances and help make these technologies available for the mortgage industry.” —Jay Bray, Chairman and CEO, Mr. Cooper Group

This builds upon the robust partner ecosystem we’re creating to help customers revolutionize the home loan experience, which includes last year’s partnership announcement with Roostify.

Integrate human review into ML predictions 

Next up is the general availability of Human-in-the-Loop AI, a new DocAI feature that will help companies achieve higher document processing accuracy with the assurance of human review. Adding human review can increase accuracy and help businesses interpret predictions using purpose-built tools to enable those reviews. 

Processing documents quickly and cost-effectively is important. But it’s often necessary to have a high level of assurance on data accuracy for compliance. CIOs and IT decision-makers need highly accurate ML predictions to fulfill compliance requirements, improve employee experience (e.g. less rework), and raise customer satisfaction (e.g. fewer data errors). Including human participation in ML processes allows AI and humans to work together for the best possible results.

gcp human-in-the-loop ai.gif

Human-in-the-Loop AI provides the workflow to manage human review tasks and produces a percentage confidence score of how “sure” it is that the AI ingested the document correctly. Document AI extracts data from documents with ML, and when paired with Human-in-the-Loop AI, human reviewers are able to verify the data captured. This system is customizable, providing the flexibility to set different thresholds and assign individual groups of reviewers to various stages of the workflow. With Human-in-the-Loop AI, developers can choose trusted reviewers to assign to the task; these reviewers can be from within their own or partner organizations.

More Document AI resources

To learn more, check out the Document AI webpage and watch a demo of how to process sample forms in AI Platform notebooks to inspect data extraction and confidence scores. For more on how customers and partners like Workday, AODocs, and Mr. Cooper are using Document AI, listen to our fireside chat. And stay tuned for the exciting evolution of these technologies in future releases of DocAI.

More Relevant Stories for Your Company

Explainer

New Capabilities in Google Cloud Dataprep: A Must-Read for Data Teams

In this video, Bertrand Cariou, Sr. Director Product Marketing, Trifacta and Sean Ma, Sr. Director, Product Management, Trifacta take a deep dive into new capabilities for Cloud Dataprep. They walk us through several new features to enable a wider range of use cases with Cloud Dataprep; and, major enhancements to

Blog

Google’s Intelligent Products Essentials Assist Manufacturers in Product Development Journey

Expectations for both consumer and commercial products have changed. Consumers want products that evolve with their needs, adapt to their preferences, and stay up-to-date over time. Manufacturers, in turn, need to create products that provide engaging customer experiences not only to better compete in the marketplace, but also to provide

Blog

Google and AI Researchers Work towards Building Data-centric AI

AI researchers and engineers need better data to enable better AI solutions. The quality of an AI solution is determined by both the learning algorithm (such as a deep-neural network model) and the datasets used to train and evaluate that algorithm. Historically, AI research has focused much more on algorithms

Blog

Google Cloud’s Data Analytics May Recap

May was a very busy month for data analytics product innovation. If you didn’t have the chance to attend our inaugural Data Cloud Summit, video replays of all our sessions are now available so feel free to watch them at your own pace.  In this blog, I’d like to share some background behind

SHOW MORE STORIES