Countries can Tackle Food Wastage Crisis Using Google’s Data Analytics

4994
Of your peers have already read this article.
4:00 Minutes
The most insightful time you'll spend today!
With over ⅓ of the food in the USA ending up as waste according to the USDA, it is a compelling challenge to address this travesty. What will happen to hunger, food prices, trash reduction, water consumption, and overall sustainability when we stop squandering this abundance?

Beginning with the departure from the farm to the back of the store, the freshness clock continues to run. Grocers work very hard to purchase high quality produce items for their customers and the journey to the shelf can take a toll in both quality and remaining shelf life. Suppliers focus on delivering their items through the arduous supply chain journey to the store with speed and gentle handling. The baton is then passed to the store to unload and present the items to customers with care to sell through each lot significantly before the expiration or sell by date. This is to ensure that the time spent in the customer’s home is ample to ensure a great eating experience as well. Food waste is a farm to fork problem with opportunity at every step of the chain, but today we will focus on the segment that the grocery industry oversees.
With the complexities of weather, geopolitical issues, distribution, sales variability, pricing, promotions, and inventory management, it seems daunting to impact waste. Fortunately, data analytics and machine learning in the cloud is a powerful weapon in the fight against food waste. Data Scientists harness knowledge to draw meaning from data turning that data into decision driving information.
One key Google has been working on to accelerate value is to break down data silos and leverage machine learning to realize better outcomes, using our Google Data Cloud platform. This enables better planning through demand forecasting, Inventory management, assortment planning, and dynamic pricing and promotions.
That sounds great but how does it work?
Let’s walk through a day in the life journey to see how the integrated Google Data Cloud platform can change the game for good. Our friendly fictitious grocer FastFreshFood is committed to selling high quality perishable items to their local market. Their goal is to minimize food waste and maximize revenue by selling as much perishable fresh food as possible before the sell by date. Our fictitious grocer in partnership with Google Cloud could build a solution that will take a significant bite out of their food waste volume and better satisfy customers.
- Sales through the register and online are processed in real time with Datastream, Dataflow to keep an accurate perpetual inventory by minute of every single item.
- A Demand forecasting model using machine learning algorithms in BigQuery then identifies needs for back room replenishment, so Direct Store Delivery and daily store Distribution Centers manage ordering more efficiently to ensure just the right amount of each product each day.
- Realtime reporting dashboards in Looker with alerting capabilities enable the system to operate with strong associate support and understanding. The reporting suite shows inventory levels into the future, daily orders, and at risk items.
The pricing algorithm could also alert store leadership concerning any items that will not sell through and suggest real time in store specials resulting in zero waste at shelf and maximized revenue.

This approach is not just for perishable categories and is a pattern that works well for in-store produced items and center store items. The key point is that by bringing ML/AI to difficult business problems grocers are reinventing what is possible for both their profitability and sustainability.
The technical implementation of this design pattern in Google Cloud leverages Datastream, Dataflow, BigQuery and Looker products, it is detailed in a technical tutorial accompanying this blog post.
In partnership with Google Cloud, retailers can solve complex problems with innovative solutions to achieve higher quality, lower cost, and provide great customer experiences. To learn more from this and other use cases, please visit our Design Patterns website.
Curious to learn more?
We’re excited to share what we know about tackling food waste at Google, a topic we’ve been working on in the last decade as we’ve embarked on reducing our own food waste in our operations in over 50 countries in the world. The Google Food for Good team works exclusively on Google Cloud Platform with our partners on this topic. Two additional articles below.
Silos are for food, not for data – tackling food waste with technology
This business Cloud blog directly addresses information silos that currently exist across many nodes in the food system and how to break down cultural and organizational barriers to sharing.
“Unsiloing” data to work toward solving food waste and food insecurity
This follow-on technical Cloud blog articulates the path to setting up data pipelines, translating between data sets (not everyone calls a tomato a tomato!) and making sense of emergent insights.
Transform ‘Dark Data’ from Documents with Document AI, Cloud Functions and Workflows

8224
Of your peers have already read this article.
2:00 Minutes
The most insightful time you'll spend today!
At enterprises across industries, documents are at the center of core business processes. Documents store a treasure trove of valuable information whether it’s a company’s invoices, HR documents, tax forms and much more. However, the unstructured nature of documents make them difficult to work with as a data source. We call this “dark data” or unstructured data that businesses collect, process and store but do not utilize for purposes such as analytics, monetization, etc. These documents in pdf or image formats, often trigger complex processes that have historically relied on fragmented technology and manual steps. With compute solutions on Google Cloud and Document AI, you can create seamless integrations and easy to use applications for your users. Document AI is a platform and a family of solutions that help businesses to transform documents into structured data backed by machine learning. In this blog post we’ll walk you through how to use Serverless technology to process documents with Cloud Functions, and with workflows of business processes orchestrating microservices, API calls, and functions, thanks to Workflows.
At Cloud Next 2021, we presented how to build easy AI-powered applications with Google Cloud. We introduced a sample application for handling incoming expense reports, analyzing expense receipts with Procurement Document AI, a DocAI solution for automating procurement data capture from forms including invoices, utility statements and more. Then organizing the logic of a report approval process with Workflows, and used Cloud Functions as glue to invoke the workflow, and do analysis of the parsed document.

We also open sourced the code on this Github repository, if you’re interested in learning more about this application.

In the above diagram, there are two user journeys: the employee submitting an expense report where multiple receipts are processed at once, and the manager validating or rejecting the expense report.
First, the employee goes to the website, powered by Vue.js for the frontend progressive JavaScript framework and Shoelace for the library of web components. The website is hosted via Firebase Hosting. The frontend invokes an HTTP function that triggers the execution of our business workflow, defined using the Workflows YAML syntax.
Workflows is able to handle long-running operations without any additional code required, in our case we are asynchronously processing a set receipt files. Here, the Document AI connector directly calls the batch processing endpoint for service. This API returns a long-running operation: if you poll the API, the operation state will be “RUNNING” until it has reached a “SUCCEEDED” or “FAILED” state. You would have to wait for its completion. However, Workflows’ connectors handle such long-running operations, without you having to poll the API multiple times till the state changes. Here’s how we call the batch processing operation of the Document AI connector:
- invoke_document_ai:call: googleapis.documentai.v1.projects.locations.processors.batchProcessargs:name: ${"projects/" + project + "/locations/eu/processors/" + processorId}location: "eu"body:inputDocuments:gcsPrefix:gcsUriPrefix: ${bucket_input + report_id}documentOutputConfig:gcsOutputConfig:gcsUri: ${bucket_output + report_id}skipHumanReview: trueresult: document_ai_response
Machine learning uses state of the art Vision and Natural Language Processing models to intelligently extract schematized data from documents with Document AI. As a developer, you don’t have to figure out how to fine tune or reframe the receipt pictures, or how to find the relevant field and information in the receipt. It’s Document AI’s job to help you here: it will return a JSON document whose fields are: line_item, currency, supplier_name, total_amount, etc. Document AI is capable of understanding standardized papers and forms, including invoices, lending documents, pay slips, driver licenses, and more.
A cloud function retrieves all the relevant fields of the receipts, and makes its own tallies, before submitting the expense report for approval to the manager. Another useful feature of Workflows is put to good use: Callbacks, that we introduced last year. In the workflow definition we create a callback endpoint, and the workflow execution will wait for the callback to be called to continue its flow, thanks to those two instructions:
- create_callback:call: events.create_callback_endpointargs:http_callback_method: "POST"result: callback_details...- await_callback:try:call: events.await_callbackargs:callback: ${callback_details}timeout: 3600result: callback_requestexcept:as: esteps:- update_status_to_error:...
In this example application, we combined the intelligent capabilities of Document AI to transform complex image documents into usable structured data, with Cloud Functions for data transformation, process triggering, and callback handling logic, and Workflows enabled us to orchestrate the underlying business process and its service call logic.
Going further
If you’re looking to make sense of your documents, turning dark data into structured information, be sure to check out what Document AI offers. You can also get your hands on a codelab to get started quickly, in which you’ll get a chance at processing handwritten forms. If you want to explore Workflows, quickstarts are available to guide you through your first steps, and likewise, another codelab explores the basics of Workflows. As mentioned earlier, for a concrete example, the source code of our smart expense application is available on Github. Don’t hesitate to reach out to us at @glaforge and @asrivas_dev to discuss smart scalable apps with us.
Google Cloud Celebrates Journey of 3 Inspiring Founders for the Asian Pacific American Heritage Month

8168
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
May is Asian Pacific American Heritage Month —a time for us to come together to celebrate and remember the important people and history of Asian and Pacific Island heritage. This feature highlights three AAPI founders from the Google For Startups community.
Read on to learn how these three founders built their businesses, leverage Google tech, and suggestion they have for aspiring entrepreneurs.

CultivatePeople
Founder: Lola Han
Description: CultivatePeople’s compensation software, Kamsa, provides global market pay rate data and helps companies make data-driven salary decisions so they can attract and retain their most valuable asset: employees.
Why GCP: “CultivatePeople began using GCP when integrating SSO/SAML authentication after our SaaS product became a high priority. We were able to exceed our clients’ expectations and increase their level of trust in our platform’s capabilities. We’re currently investigating additional machine learning products, such as Cloud AutoML, that will allow us to quickly deliver exciting features.”
Note from the Founder: “I grew up with immigrant parents who value stability and are risk-averse. My parents discouraged me from starting my own company because they didn’t want to see me struggle financially or see my health suffer (due to stress). I felt strongly about what CultivatePeople could do, so I started the company as a sole founder in 2017 and watched it double in size year over year since. While the ones I love most may not have cheered me on initially, it was important for me to hang on to the encouraging words of former bosses, executives, and founders to keep me focused on my mission.
My advice for other AAPI founders is to be a “silent assassin” and believe in the mission and values of your organization. Always remember to stop along the way and:
1) Enjoy the journey by celebrating wins and giving yourself credit;
2) Follow your intuition—it’s (almost) always right;
3) Recognize and invest in your people regularly (ie. give increases more than once a year, if warranted);
4) Give regular words of affirmation to employees on even small achievements.”

Swit
Founder: Josh Lee and Max Lim
Description: Swit is a team collaboration platform that seamlessly combines team chat with task management by allowing teams to turn their conversations into trackable tasks and share tasks to chat with simple drag-and-drop functionality, ensuring everyone is on the same page and projects get done faster.
Why GCP: “Swit is a cross-category hybrid work tool for chat and tasks. This functionality requires more complicated and heavier architecture for performance. So, configuring and managing virtual machines was really challenging to scale up our systems, while handling occasional unexpected traffic surges and frequent updates. Eventually we divided our monolithic architecture into 35 microservices when we launched our official product. The migration to GKE took around one month, and it turned out to be well worth the effort—our systems became able to offer high scalability and enough resilience to keep its uptime no matter what happened. Now we’re operating 84 workloads and 252 microservices with high stability with remarkably low downtime – less than 0.00001%/year.”
Note from the Founder: “As an AAPI founder based in Silicon Valley, I feel proud of the work ethics and diligence fellow Korean American entrepreneurs and professionals have long demonstrated here. Especially with K-pop breaking into the mainstream, I feel even more proud of our culture that strives toward an absolute perfection molded through years of training and dedication. The mission-driven culture of Silicon Valley coupled with Google’s edging technology and creativity really helped us build a product that not only encompasses verticals but also transcends cultures. Swit is growing at an unprecedented rate, and we hope to join the long list of successful AAPI entrepreneurs here. Swit’s close network with the AAPI community wouldn’t have been possible without Google support. We are grateful for this collaborative environment, and we hope to become the next-generation ambassador for collaboration after Google.”
Check out more from Swit in their founder story.

WISY
Founder: Min Chen
Description: Wisy develops technology to bring digital efficiency into the physical world, supporting consumer products businesses and making them thrive in the new economy. All of us have a bad experience when we can’t find the product we want to buy. That is a $1.9T problem in the consumer-packaged goods industry that Wisy is solving with AI and analytics to help manufacturers and retailers sell more by reducing out-of-stocks and waste at a global scale.
Why GCP: “GCP has an intuitive, easy to use interface, was lower cost, and offered preemptible instances with flexible compute options. Some of the reasons why Wisy decided to use GCP include instance and payment configurability, privacy and traffic security, cost-efficiency, and Machine Learning.
Wisy has been able to advance quickly with product development, as well as collaborate better and iterate faster in the creation of our AI models, while reducing costs by 40%. At Wisy, we are solving a problem that affects everyone who shops at a store.“
Note from the Founder: “Two years ago, I moved to San Francisco to expand my second startup, Wisy. This is when I learned that my name ‘Min’ stands for ‘minority.’ I was born in China, raised in a Black community in Panama, received scholarships to attend both Carnegie Mellon and UC Berkeley. I worked for 20 years in several countries, but I have never felt so discriminated against due to my race, ethnicity, gender and age than during my time in Silicon Valley. However, this is also the place I learned that my diverse life experience is my competitive advantage. My background enables me to recruit and relate to people in different countries, create scalable and flexible products for multinational customers, and run global operations efficiently.
My recommendation to AAPI founders is to find strength in their multicultural background. Don’t hide what makes you unique, do not limit yourselves, and do not let others limit you. You will lose your edge when trading authenticity for validation. Be proud and own your story.”
If you want to learn more about how Google Cloud can help your startup, visit our Startup Program application page here and sign up for our monthly startup newsletter to get a peek at our community activities, digital events, special offers, and more.To learn more about how you can help #StopAsianHate during Asian Pacific American Heritage Month and beyond, visit their website here.
How to Enhance Incremental Pipeline Performance while Ingesting Data into BigQuery

6341
Of your peers have already read this article.
3:00 Minutes
The most insightful time you'll spend today!
When you build a data warehouse, the important question is how to ingest data from the source system to the data warehouse. If the table is small you can fully reload a table on a regular basis, however, if the table is large a common technique is to perform incremental table updates. This post demonstrates how you can enhance incremental pipeline performance when you ingest data into BigQuery.
Setting up a standard incremental data ingestion pipeline
We will use the below example to illustrate a common ingestion pipeline that incrementally updates a data warehouse table. Let’s say that you ingest data into BigQuery from a large and frequently updated table in the source system, and you have Staging and Reporting areas (datasets) in BigQuery.

The Reporting area in BigQuery stores the most recent, full data that has been ingested from the source system tables. Usually you create the base table as a full snapshot of the source system table. In our running example, we use BigQuery public data as the source system and create reporting.base_table as shown below. In our example each row is identified by a unique key which consists of two columns: block_hash and log_index.
CREATE TABLE reporting.base_table --156 GB processedPARTITION BY TIMESTAMP_TRUNC(block_timestamp, DAY) ASSELECT log_index, data, topics, block_timestamp, block_hashFROM bigquery-public-data.crypto_ethereum.logsWHERE block_timestamp BETWEEN TIMESTAMP '2020-01-01' AND TIMESTAMP '2020-11-30';
In data warehouses it is common to partition a large base table by a datetime column that has a business meaning. For example, it may be a transaction timestamp, or datetime when some business event happened, etc. The idea is that data analysts who use the data warehouse usually need to analyze only some range of dates and rarely need the full data. In our example, we partition the base table by block_timestamp which comes from the source system.
After ingesting the initial snapshot you need to capture changes that happen in the source system table and update the reporting base table accordingly. This is when the Staging area comes into the picture. The staging table will contain captured data changes that you will merge into the base table. Let’s say that in our source system on a regular basis we have a set of new rows and also some updated records. In our example we mock the staging data as follows: first, we create new data, than we mock the updated records:
CREATE TABLE staging.load_delta AS --5 GB processedSELECT log_index, data, topics, block_timestamp, block_hashFROM bigquery-public-data.crypto_ethereum.logsWHERE block_timestamp BETWEEN TIMESTAMP '2020-12-01' AND TIMESTAMP '2020-12-07';INSERT INTO staging.load_delta --2 GB processedSELECT log_index, CONCAT(data, RAND()), topics, block_timestamp, block_hashFROM bigquery-public-data.crypto_ethereum.logs TABLESAMPLE SYSTEM (5 PERCENT)WHERE block_timestamp BETWEEN TIMESTAMP '2020-10-01' AND TIMESTAMP '2020-11-30';
Next, the pipeline merges the staging data into the base table. It joins two tables by unique key and than updates the changed value or inserts a new row
MERGE INTO reporting.base_table T --161 GB processedUSING staging.load_delta SON T.block_hash = S.block_hashAND T.log_index = S.log_indexWHEN MATCHED THEN UPDATE SETT.data = S.dataWHEN NOT MATCHED THEN INSERT (log_index, data, topics, block_timestamp, block_hash)VALUES (log_index, data, topics, block_timestamp, block_hash);
It is often the case that the staging table contains keys from various partitions but the number of those partitions are relatively small. It holds, for instance, because in the source system the recently added data may get changed due to some initial errors or ongoing processes but older records are rarely updated. However, when the above MERGE gets executed, BigQuery scans all partitions in the base table and processes 161 GB of data. You might add additional join condition on block_timestamp:
MERGE INTO reporting.base_table T --161 GB processedUSING staging.load_delta SON T.block_hash = S.block_hashAND T.log_index = S.log_indexAND T.block_timestamp = S.block_timestampWHEN MATCHED THEN UPDATE SETT.data = S.dataWHEN NOT MATCHED THEN INSERT (log_index, data, topics, block_timestamp, block_hash)VALUES (log_index, data, topics, block_timestamp, block_hash);
But BigQuery would still scan all partitions in the base table because condition T.block_timestamp = S.block_timestamp is a dynamic predicate and BigQuery doesn’t automatically push such predicates down from one table to another in MERGE.
Can you improve the MERGE efficiency by making it scan less data? The answer is Yes.
As described in the MERGE documentation, pruning conditions may be located in a subquery filter, a merge_condition filter, or a search_condition filter. In this post we show how you can leverage the first two. The main idea is to turn a dynamic predicate into a static predicate.
Steps to enhance your ingestion pipeline
The initial step is to compute the range of partitions that will be updated during the MERGE and store it in a variable. As was mentioned above, in data ingestion pipelines, staging tables are usually small so the cost of the computation is relatively low.
DECLARE src_range STRUCT<date_min TIMESTAMP, date_max TIMESTAMP> --115 MB processedDEFAULT(SELECT STRUCT(MIN(block_timestamp) AS date_min,MAX(block_timestamp) AS date_max) FROM staging.load_delta);
Based on your existing ETL/ELT pipeline, you can add the above code as-is to your pipeline or you can compute date_min, data_max as part of some already existing transformation step. Alternatively, date_min, data_max can be computed on the Source System side while capturing the next ingestion data batch.
After computing date_min, date_max you pass those values to the MERGE statement as static predicates. There are several ways to enhance the MERGE and prune partitions in the base table based on precomputed date_min, data_max.
If your initial MERGE statement uses a subquery, you can incorporate a new filter into it:
BEGINDECLARE src_range STRUCT<date_min TIMESTAMP, date_max TIMESTAMP> --115 MB processedDEFAULT(SELECT STRUCT(MIN(block_timestamp) AS date_min,MAX(block_timestamp) AS date_max) FROM staging.load_delta);MERGE INTO reporting.base_table T --41 GB processedUSING (SELECT *FROM staging.load_deltaWHERE block_timestamp BETWEEN src_range.date_min AND src_range.date_max) SON T.block_hash = S.block_hashAND T.log_index = S.log_indexAND T.block_timestamp = S.block_timestampWHEN MATCHED THEN UPDATE SETT.data = S.dataWHEN NOT MATCHED THEN INSERT (log_index, data, topics, block_timestamp, block_hash)VALUES (log_index, data, topics, block_timestamp, block_hash);END;
Note that you add the static filter to the staging table and keep T.block_timestamp = S.block_timestamp to convey to BigQuery that it can push that filter to the base table. This MERGE processes 41 GB of data in contrast to the initial 161 GB. You can see in the query plan that BigQuery pushes the partition filter from the staging table to the base table:

This type of optimization, when a pruning condition is pushed from a subquery to a large partitioned or clustered table, is not unique for MERGE. It also works for other types of queries. For instance:
SELECT * -- 41 GB processedFROM reporting.base_table TINNER JOIN staging.load_delta SON T.block_hash = S.block_hashAND T.log_index = S.log_indexAND T.block_timestamp = S.block_timestampWHERE S.block_timestamp BETWEEN TIMESTAMP '2020-10-05' AND TIMESTAMP '2020-12-07'
And you can check the query plan to verify that BigQuery pushed down the partition filter from one table to another.
Moreover, for SELECT statements, BigQuery can automatically infer a filter predicate on a join column and push it down from one table to another if your query meets the following criteria:
- The target table must be clustered or partitioned.
- The result size of the other table, i.e. after applying all filters, must qualify for broadcast join. Namly, the result set must be relatively small, less than ~100MB.
In our running example, reporting.base_table is partitioned by block_timestamp. If you define a selective filter on staging.load_delta and join two tables, you can see an inferred filter on the join key pushed to the target table
SELECT *FROM reporting.base_table TINNER JOIN staging.load_delta SON T.block_timestamp = S.block_timestampWHERE S.block_hash = '0x0c1caa16b34d94843aabfebc0d5a961db358135988f7498a6fdc450ad55f0870'

There is no requirement to join tables by partitioning or clustering key to kick off this type of optimization. However, in this case the pruning effect on the target table would be less significant.
But let us get back to the pipeline optimizations. Another way to enhance MERGE is to modify the merge_condition filter by adding static predicate on the base table:
BEGINDECLARE src_range STRUCT<date_min TIMESTAMP, date_max TIMESTAMP> --115 MB processedDEFAULT(SELECT STRUCT(MIN(block_timestamp) AS date_min,MAX(block_timestamp) AS date_max) FROM staging.load_delta);MERGE INTO reporting.base_table T --41 GB processedUSING staging.load_delta SON T.block_hash = S.block_hashAND T.log_index = S.log_indexAND T.block_timestamp BETWEEN src_range.date_min AND src_range.date_maxWHEN MATCHED THEN UPDATE SETT.data = S.dataWHEN NOT MATCHED THEN INSERT (log_index, data, topics, block_timestamp, block_hash)VALUES (log_index, data, topics, block_timestamp, block_hash);END;
To summarize, here are the steps that you can perform to enhance incremental ingestion pipelines in BigQuery. First you compute the range of updated partitions based on the small staging table. Next, you tweak the MERGE statement a bit to let BigQuery know to prune data in the base table.
All the enhanced MERGE statements scanned 41 GB of data, and setting up the src_range variable took 115 MB. Compare it with the initial 161 GB scan. Moreover, given that computing src_range may be incorporated into some existing transformation in your ETL/ELT, it results in a good performance improvement which you can leverage in your pipelines.
In this post we described how to enhance data ingestion pipelines by turning dynamic filter predicates into static predicates and letting BiQuery prune data for us. You can find more tips on BigQuery DML tuning here.
Special thanks to Daniel De Leo, who helped with examples and provided valuable feedback on this content.

7540
Of your peers have already downloaded this article
15:52 Minutes
The most insightful time you'll spend today!
Breakthroughs in artificial intelligence (AI) have captured the imaginations of business and technical leaders alike. The AI techniques underlying these breakthroughs are finding diverse application across every industry. Early adopters are seeing results, particularly encouraging is that AI is starting to transform processes in established industries, from retail to financial services to manufacturing.
However, an organization’s effectiveness in applying these breakthroughs is anchored in the basics: A disciplined foundation in capturing, preparing and analyzing data.
Data scientists spend up to 80% of their time on the “data wrangling,” “data munging” and “data janitor” work required well before the predictive capabilities promised by AI can be realized.
Capturing, preparing and analyzing data creates the foundation for successful AI initiatives. To help business and IT leaders create this virtuous cycle, Google Cloud has prepared a CIO’s guide to data analytics and machine learning that outlines key enabling technologies at each step. Crucially, the guide illustrates how managed cloud services greatly simplify the journey—regardless of an organization’s maturity in handling big data.
This is important because, for many companies, the more fundamental levels of data management present a larger challenge than new capabilities like AI. “Management teams often assume they can leapfrog best practices for basic data analytics by going directly to adopting artificial intelligence and other advanced technologies,” noted Oliver Wyman consultants Nick Harrison and Deborah O’Neill in a recent Harvard Business Review article (aptly titled If Your Company Isn’t Good at Analytics, It’s Not Ready for AI). “Like it or not, you can’t afford to skip the basics.
Building on new research and Google Cloud’s own contributions to big data since the beginning, this guide walks readers through each step in the data management cycle, illustrating what’s possible alongside examples.
Specifically, the CIO’s guide to data analytics and machine learning is designed to help business and IT leaders address some of the essential questions companies face in modernizing data strategy:
- For my most important business processes, how can I capture raw data to ensure a proper foundation for future business questions? How can I do this cost-effectively?
- What about unstructured data outside of my operational/transactional databases: raw files, documents, images, system logs, chat and support transcripts, social media?
- How can I tap the same base of raw data I’ve collected to quickly get answers as new business questions arise?
- Rather than processing historical data in batch, what about processes where I need a real-time view of the business? How can I easily handle data streaming in real time?
- How can I unify the scattered silos of data across my organization to provide a current, end-to-end view? What about data stored off-premises in the multiple cloud and SaaS providers I work with?
- How can I disseminate this capability across my organization—especially to business users, not just developers and data scientists?
Because managed cloud services deal with an organization’s sensitive data, security is a top consideration at each step of the data management cycle. From data ingestion into the cloud, followed by storage, preparation and ongoing analysis as additional data flows in, techniques like data encryption and the ability to connect your network directly to the Google Cloud must reflect data security best practices that keep data assets safe as they yield insights.
Wherever your company is on its path to data maturity, Google Cloud is here to help. We welcome the opportunity to learn more about your challenges and how we can help you unlock the transformational potential of data.
Leverage the Power of Looker to Extract Data Value at Web Scale

4458
Of your peers have already read this article.
3:00 Minutes
The most insightful time you'll spend today!
Data science can be applied to business problems to improve practices and help to strengthen customer satisfaction. In this blog, we address how the addition of Looker extends the value of your Google Cloud investments to help you understand your customer journey, unlock value from first-party data, and optimize existing cloud infrastructure.
Data is a powerful tool
While everyone doesn’t need to understand the nuts and bolts of data technologies, most people do care about the value data can create for them — how it can help them do their jobs better. Within the data space, we are seeing a trend of ongoing failure to make data accessible to “humans” – the industry still hasn’t figured out how to put data into the hands of people when and where they need it, how they need it.
What if everyone in your organization could analyze data at scale, and make more-informed, fact-based decisions? Data and insights derived from data are valuable but only if your users see it. We think Looker helps solve that.
Solutions tell a larger story of how it all fits together
Across all verticals and industries, businesses benefit from knowing and understanding their customers better. Many business goals are to increase revenue by improving product recommendations and pricing optimization, improving the user experience through targeted marketing and personalization, and reducing churn while improving retention rates. To help reach these goals, key strategies should focus on understanding customer needs, their motivations, likes and dislikes, and using all available data – in other words, put yourself in the shoes of your customer.
Our goal with Looker solutions is to offer the right level of out-of-box support that allows customers to get to value quickly, while maintaining the necessary flexibility. We aim to offer a library of data-driven solutions that accelerate data projects. Many solutions include Looker Blocks (pre-built pieces of code that accelerate data exploration environments) and Actions (custom integrations) that get customers up and running quickly and lets you build business-friendly access points for Google Cloud functionality like BQML, App Engine and Cloud Functions.
Below, you’ll find a sampling of the newest Looker solutions.
Listening to customers by looking at the data
Looker’s solution for Contact Center AI (CCAI), helps businesses gain a deeper understanding and appreciation of their customers’ full journey by unlocking insights from all their company’s first-party data. Call centers can converse naturally with customers and deliver outstanding experiences by leveraging artificial intelligence. CCAI‘s newest product —CCAI Insights — reviews conversations support agents are having, finding and annotating the data with the important information, and identifying the calls that need review. We’ve partnered with the product teams at CCAI to build Looker’s Block for CCAI Insights, which sets you on the path to integrating the advanced insights into your first-party data in Looker, overlaying business data with the customer experience.

Businesses can better understand contact center experiences and take immediate action when necessary to make sure the most valuable customers receive the best service.
Realizing full business value of First-Party data
Looker for Google Marketing Platform (GMP) provides marketers the power to unlock the value of their company’s first-party data to more effectively target audiences. The Looker Blocks and Actions for GMP offer interactive data exploration, slices of data with built-in ML predictions and activation paths back to the GMP. This strategic solution continues to evolve with the Looker Action for Google Ads (Customer Match), the Looker Action for Google Analytics (Data Import) and the Looker Block for Google Analytics 4 (GA4).
- The Looker Action for Customer Match allows marketers to send segments and audiences based on first-party data directly into Google Ads. Reach users cross-device and across the web’s most powerful channels such as Display, Video, YouTube, and Gmail. The entire process is performed within a single screen in Looker, and is able to be completed in a few minutes by a non-technical user.
- The Looker Action for Data Import can be used to enhance user segmentation and remarketing audiences in Google Analytics by taking advantage of user information accessible in Looker, such as in CRM systems or transactional data warehouses.
- The Looker Block for Google Analytics 4 (GA4) expands the solution’s support with out-of-the-box dashboards and pre-baked BigQuery ML models for the newest version of Google Analytics.The Looker Block offers up reports with flexible configuration capabilities to unlock custom insights beyond the standard GA reporting. Customize audience segments, define custom goals to track and share these reports with teams who do not have access to the GA console.
From clinical notes to patient insights at scale
Taking a look at the Healthcare vertical, the Looker Healthcare NLP API Block serves as a critical bridge between existing care systems and applications hosted on Google Cloud providing a managed solution for storing and accessing healthcare data in Google Cloud. The Healthcare NLP API uses natural language models to extract healthcare information from medical text, rapidly unlocking insights from unstructured medical text and providing medical providers with simplified access to intelligent insights. Healthcare providers, payers, and pharma companies can quickly understand the context and relationships of medical concepts within the text, such as medications, procedures, conditions, clinical history, and begin to link this to other clinical data sources for downstream AI/ML.
Specifically, the natural language processing (NLP) Patient View (pictured below) allows you to review a single selected patient of interest, surfacing their clinical notes history over time. It informs clinical diagnosis with family history insights, which is not currently captured in claims, and captures additional procedure coding for revenue cycle purposes.

The dashboard below shows the NLP Term View which allows users to focus on chosen medical terms across the entire patient population in the dataset so they can start to view trends and patterns across groups of patients.

This data can be used to:
- Enhance patient matching for clinical trials
- Identify re-purposed medications
- Drive advancements for research in cancer and rare diseases
- Identify how social determinants of health impact access to care
Managing Costs Across Clouds
Effective cloud cost management is important for reasons beyond cost control — it provides you the ability to reduce waste and predictably forecast both costs and resource needs. Looker’s solution for Cloud Cost Management offers quick access to necessary reporting and clear insights into cloud expenditures and utilization.
This solution brings together billing data from different cloud providers in a phased approach: get up and running quickly with Blocks optimized for where the data is today (Google Cloud, AWS or Azure) as you work towards more sophisticated analysis for cross-platform planning and even cloud spend optimization with the mapping of tags, labels and cost centers across clouds.

The Looker Cloud Cost Management solution provides operational teams struggling to monitor, understand, and manage the costs and needs associated with their cloud technology with a comprehensive view into what, where and why they are spending money.
Making better decisions with Looker-powered data
Leading companies are discovering ways to get value from all of that data beyond displaying it in a report or a dashboard. They want to enable everyone to make better decisions but that’s only going to happen if everyone can ask questions of the data, and get reliable, correct answers without using outdated or incomplete data and without waiting for it. People and systems need to have data available to them in the way that makes the most sense for them at that moment.
It’s clear that successful data-driven organizations will lead their respective segments not because they use data to create reports, but because they use it to power data experiences tailored to every part of the business, including employees, customers, operational workflows, products and services.
As people’s way of experiencing data has evolved, more than ever before, dashboards alone are not enough. You can use data as fuel for data-driven business workflows, and to power digital experiences that improve customer engagement, conversions, and advocacy.
From Nov. 9 – 11th, Looker is hosting its annual conference JOIN, where we’ll be showing new features, including how we help to:
- Build data experiences at the speed of business
- Accelerate the business value with packaged experiences
- Unleash more insights for more people in the right way – Deliver data experiences at scale
There is no cost to attend JOIN. Register here and learn how Looker helps organizations build and deliver custom data-driven experiences that goes beyond just reports and dashboards, scales and grows with your business, allows developers to build innovative data products faster, and ensures data reaches everyone.
More Relevant Stories for Your Company

Cart.com to Transform e-Commerce for Brands Globally
The ecommerce playing field has been hard to navigate for most retailers, and Cart.com is on a mission to change that. Traditionally, retailers needing to run their online store, order fulfillment, customer service, marketing, and other essential activities have had to cobble together systems to get the capabilities they need

How One Company Uses AI and Data Analysis to Boost Revenue
AI, deep learning, and image recognition is transforming the shopping experience. These technologies enable consumers to use product images or screenshots rather than text to search for similar products. This improves the customer experience and enables retailers with online and offline outlets to provide a genuine omnichannel experience. The lack of

Indian Retailer Figures Optimizes Hyperlocal Delivery to Increase Customer Experience
Anyone who follows the Indian e-commerce scene knows that one of the largest challenges these companies face is hyperlocal delivery. That was a problem facing Wellness Forever, a retail chain of pharmacies with 150-plus stores across India. “Exactly a year ago, we started our journey of hyperlocal deliveries. This optimization

Looking for a Cloud Data Warehouse? Find out Why Forrester Thinks Google BigQuery is a Leader
We are thrilled to announce that Google has been named a Leader in The Forrester Wave™: Cloud Data Warehouse, Q1 2021 report. For more than a decade, BigQuery, our petabyte-scale cloud data warehouse, has been in a class of its own. We're excited to share this recognition and we want to thank






