We are increasingly seeing one question arise in virtually every customer conversation: How can the organization save costs and drive new revenue streams? 

Everyone would love a crystal ball, but what you may not realize is that you already have one. It’s in your data. By leveraging Data Cloud and AI solutions, you can put your data to work to achieve your financial objectives. Combining your data and AI reveals opportunities for your business to reduce expenses and increase profitability, which is especially valuable in an uncertain economy. 

Google Cloud customers globally are succeeding in this effort, across industries and geographies. They are improving ROI by saving money and creating new revenue streams. We have distilled the strategies and actions they are implementing—along with customer examples and tips—in our eBook, “Make Data Work for You.” In it, you’ll find ways you can pare costs, increase profitability, and monetize your data.  

Find money in your data 

Our Google Cloud teams have identified eight strategies that successful organizations are pursuing to trim expenses and uncover new sources of revenue through intelligent use of data and AI. These use cases range from scaling small efficiencies in logistics to accelerating document-based workflows, monetizing data, and optimizing marketing spend.

The results are impressive. They include massive cost savings and additional revenue. On-time deliveries have increased sharply at one company, and procure-to-pay processing costs have fallen by more than half at another. Other organizations have reaped big gains in ecommerce upselling and customer satisfaction.

We’ve found that businesses across every industry and around the globe are able to take action on at least one of these eight strategies. Contrary to common misperceptions, implementation does not require massive technology changes, crippling disruption to your business, or burdensome new investments. 

What success looks like 

If you worry your business is not ready or you need to gain buy-in from leadership, the success stories of the 15 companies in this report are helpful examples. Learning how organizations big and small, in different industries and parts of the world, have implemented these data and AI strategies makes the opportunities more tangible.

Carrefour 
Among the world’s largest retailers, Carrefour operates supermarkets, ecommerce, and other store formats in more than 30 countries. To retain leadership in its markets, the company wanted to strengthen its omnichannel experience.

Carrefour moved to Google Data Cloud and developed a platform that gives its data scientists secure, structured access to a massive volume of data in minutes. This paved the way for smarter models of customer behavior and enabled a personalized recommendation engine for ecommerce services. 

The company saw a 60% increase in ecommerce revenue during the pandemic, which it partly attributes to this personalization. 

ATB Financial 
ATB Financial, a bank in the Canadian province of Alberta, uses its data and AI to provide real-time personalized customer service, generating more than 20,000 AI-assisted conversations monthly. Machine learning models enable agents to offer clients real-time tailored advice and product suggestions. 

Moreover, marketing campaigns and month-end processes that used to take five to eight hours now run in seconds, saving over CA$2.24 million a year. 

Bank BRI
Bank BRI, which is owned by the Indonesian government, has 75.5 million clients. Through its use of digital technologies, the institution amasses a lot of valuable data about this large customer base. 

Using Google Cloud, the bank packages this data through more than 50 monetized open APIs for more than 70 ecosystem partners who use it for credit scoring, risk management, and other applications. Fintechs, insurance companies, and financial institutions don’t have the talent or the financial resources to do quality credit scoring and fraud detection on their own, so they are turning to Bank BRI. 

Early in the effort, the project generated an additional $50 million in revenue, showing how data can drive new sources of income. 

How to get going now

“Make Data Work for You” will help you launch your financial resiliency initiatives by outlining the steps to get going. The process lays the groundwork for realizing your own cost savings and new revenue streams by leveraging data and AI.

Among these steps include building frameworks to operate cost efficiently, make informed decisions related to spending and optimize your data and AI budgets.

Operate: Billing that’s specific to your use-case
Control your costs by choosing data and analytics vendors who offer industry-leading data storage solutions and flexible pricing options. For example, multiple pricing options such as flat rate and pay-as-you-go allow you to optimize your spend for best price-performance.

Inform: make informed decisions based on usage
Use your cloud vendor’s dashboards or build a billing data report to gain insights on your spending over time. Make use of cost recommendations and other forecasting tools to predict what your future expenses are going to be.

Optimize: Never pay more than you use 
While planning data analytics capacity, organizations often overprovision and overpay than what they actually use. Consider migrating your workloads that have unpredictable demand to a data warehousing solution that offers granular level autoscaling features so that you never have to pay for more than what you use.

There are other key moves that will set your initiative up for success including how to shorten time to value in building AI models and measuring impact. You can find details in the report.

A brighter future

The teams at Google Cloud helped the companies in “Make Data Work for You,” along with many more organizations, use their data and AI to achieve meaningful results. Download the full report to see how you can too.

Data security is an ongoing concern for anyone managing a data warehouse. Organizations need to control access to data, down to the granular level, for secure access to data both internally and externally. With the complexity of data platforms increasing day by day, it’s become even more critical to identify and monitor access to sensitive data. In many cases, sensitive data is co-mingled with non-sensitive data, and access restrictions to sensitive data need to be enabled based on factors like data location or presence of financial information. There may also be nuances where data is sensitive for some groups of users, while for others, it is not. 

Today, we’re pleased to announce the general availability of BigQuery row-level security, which gives customers a way to control access to subsets of data in the same table for different groups of users. Row-level security (RLS) extends the principle of least privilege access and enables fine-grained access control policies in BigQuery tables. BigQuery currently supports access controls at the project-, dataset-, table- and column-level. Adding RLS to the portfolio of access controls now enables customers to filter and define access to specific rows in a table based on qualifying user conditions—providing much needed peace of mind for data professionals. 

“Our digital transformation and migration of data to the cloud magnifies the business value we can extract from our information assets. However, granular data access control is essential to comply with international regulatory and contractual requirements. BigQuery row-level security helps us comply with data residency and export restrictions,” says Jarrett Garcia, Iron Mountain’s Enterprise Data Platform Senior Director. “It enables us to manage fine-grained access controls without replicating data. What used to take months for approval and access provisioning can now be done more efficiently and effectively. We are looking forward to implementing additional data security capabilities on the BigQuery roadmap to address other critical business use cases.”

How BigQuery row-level security works

Row-level security in BigQuery enables different user personas access to subsets of data in the same table. Customers who are currently using authorized views to enable these use cases can leverage RLS for ease of management. To express the concept of RLS, we have introduced a new entity in BigQuery called row access policy. Row access policies map a group of user principals to the rows that they can see, defined by a SQL filter predicate. 

Secure logic rules created by data owners and administrators determines which user can see which rows through the creation of a row-level access policy. The row-level access policies created on a target table by administrators or data owners are applied when a query is run on the table. One table can have multiple policies applied to it.

Below is an example, where row-level access policies have been created to filter data based on users’ “region”.

In the illustrated scenario above, row-level access policies have been created to verify a querying user’s region and to give them access only to the subset of data relevant to that region. Access policies are granted to a grantee list which support all types of IAM principles such as individual users, groups, domains or service accounts. In this example, when a user queries the table, row-level access policies are evaluated to assess which, if any, policies are applicable to that user. The group ‘sales-apac’ is granted access to view a subset of rows where region = ‘APAC’ whereas the group ‘sales-us’ is granted access to view a subset of rows where the region = ’US’. Likewise, users in both groups will see rows in both regions, and users in neither group will not see any rows.

Row-level access policies can also be created using the SESSION_USER() function to restrict access only to rows that belong to the user running the query. If none of the row access policies are applicable to the querying user, the user will have no access to the data in the table.

When a user queries a table with a row-level access policy, BigQuery displays a banner notice indicating that their results may be filtered by a row-level access policy. This notice displays even if the user is a member of the `grantee_list`.

When to put BigQuery row-level security to work

Row-level access policies are useful when you have a need to limit access to data based on filter conditions. The row-access policies’ filter predicate supports arbitrary SQL, and is conceptually similar to the WHERE clause of a SQL query. Filter predicates support the SESSION_USER() function to restrict access only to rows that belong to the user running the query. If none of the row access policies are applicable to the querying user, the user will have no access to the data in the table. Currently, the column used for filtering must be in the table, but we anticipate adding support for subqueries in the filter expression, opening up access to use cases where data is filtered based on lookup tables and calculated values. Row-level access policies can be created, updated and dropped using DDL statements. You will be able to see the list of row-level access policies applied to a table using the BigQuery schema pane in the Cloud Console,  which simplifies the management of policies per table, or by using the bq command-line tool.

Row-level security is compatible with other BigQuery security features, and can be used along with column-level security for further granularity.  Since row-level access policies are applied on the source tables, any actions performed on the table will inherit the table’s associated access policies, to ensure access to secure data is protected. Row-level access policies are applicable to every method used to access BigQuery data (API, Views, etc). 

Try it out

We’re always working to enhance BigQuery’s (and Google Cloud’s) data governance capabilities, to provide more controls around managing your data. With row-level security, we are adding deeper protections for your data. You can learn more about BigQuery row-level security in our documentation and best practices.

Japan is home to millions of vending machines installed on streets and in buildings, sports stadiums and other facilities. Vending machine owners and operators, including beverage manufacturers, stock these machines with different product combinations depending on location and demand. For example, they primarily display coffee and energy drinks in machines placed in offices and sports drinks and mineral water in machines at sports facilities. The combinations also vary by season: for example, owners and operators may display cold beverages in summer and hot beverages in winter. 

Traditionally, vending machine operators have relied on the intuition and experience of sales managers to determine the optimum product mix for each vending machine. However, in recent years, manufacturers such as Coca-Cola Bottlers Japan (CCBJ) have turned to data to analyze and make strategic decisions about when and where to locate products in machines.

CCBJ is the number one Coca-Cola bottler in Asia and vending machines comprise the bulk of its business. The organization operates about 700,000 machines across Tokyo, Osaka, Kyoto, and 35 prefectures. Minori Matsuda, Google Developer Expert and also Data Science Manager at CCBJ, says “The billions of data records collected from 700,000 physical devices are a great asset and a treasure trove we can take advantage of.”

Minori points out that when considering the mix of products in vending machines in sporting facilities, the managers naturally assume sports drinks would generally sell well. However, analysis of purchase data – including hot drinks and hot drinks plus sports drinks – found many parents purchased sweet drinks such as milk tea when they attended games or sessions involving their children.  “Analyzing data gives us new discoveries and, by using catchy storytelling techniques from exploratory data analysis, we are instilling a data culture within our company,” he says. “It’s worth creating by looking at facts rather than making assumptions!”

Minori believes that to analyze the vast amount of data collected from more than 700,000 vending machines, the business needs a powerful analytical platform. However, until recently, CCBJ had to extract data for analysis from its core systems, load this data into a warehouse it created and perform the required analyses.  The billions of records of data generated across the fleet – including transaction data – exposed some challenges for traditional analysis platforms. They could not efficiently process data at a considerable scale: it could take a day to return results and required extensive maintenance due to the size.

CCBJ considered building a machine learning (ML) platform as a layer on top of existing systems in August 2020 and opted for Google Cloud the following month.  “I feel that Google Cloud has an edge in all products and is very well thought out,“ says Minori, noting the scalability and cost of the platform allow the business to take a ‘trial and error’ approach to achieve the best outcomes from ML. Google Cloud also delivered the required visibility and flexibility to help the business deliver change every day against key performance indicators. 

MLOps platform streamlines ML pipeline development

CCBJ built its analysis platform using Vertex AI (formerly AI Platform) centered on a BigQuery analytics data warehouse, and partly using AutoML for tabular data. “We have created a prediction model of where to place vending machines, what products are lined up in the machines and at what price, how much they will sell, and implemented a mechanism that can be analyzed on a map,” says Minori, adding that building the platform with Google Cloud was not difficult. “We were able to realize it in a short period of time with a sense of speed, from platform examination to introduction, prediction model training, on-site proof of concept to rollout.”

The new data analytics platform of CCBJ consists of the following parts:

Data Sources

• The data collected from the vending machines are all stored on BigQuery.

Data Discovery and Feature Engineering

• Minori and other data scientists at CCBJ are using Vertex Notebooks, where they access the data on BigQuery by executing SQL queries directly from the Notebooks. This environment is used for the data discovery process and feature engineering. 

ML Training

• For ML training, CCBJ uses AutoML for Tabular data, Custom model training on Vertex AI, and BigQuery ML. AutoML gives model performance with AUC curves and also feature importance graphs.

ML Prediction and Serving

• For ML prediction, CCBJ uses Online Prediction for AutoML models and Online Prediction for custom models for real-time prediction when the salesperson finds the interesting point
• Batch Prediction is used for generating a large prediction map that covers the whole country
• The prediction results are distributed to sales managers’ tablets

CCBJ started constructing the platform in September 2020, and completed it within a month. The business has conducted proofs of concept at its base in Kyoto since February 2021, and since April, has rolled out the platform to sales managers in 35 prefectures in one metropolitan area. “Data analysis is built into the day-to-day routines of sales managers with 100% utilization,” says Minori. “They can utilize the prediction results on tablets that were able to achieve pretty high accuracy from the start.”

The hardest part was the education of sales managers in the field; having them understand the reasoning behind the ML prediction results for particular outcomes, so they could be convinced to make use of the results. “For example, regarding a new installation location predicted by the model, it seemed that there was no effective information for installation from the map information, but when I actually went there, there was a motorcycle shop and it was a place where young people who like motorcycles gathered,” says Minori. “Or there is a small meeting place where the elderly in the neighborhood are active. 

“In many cases, new discoveries that cannot be understood from map information alone can be derived from the data.”

Minori also points to a phenomenon whereby humans pursued and confirmed factors inferred by the model – meaning that once they experienced analysis and it worked effectively, they asked why the same type of analysis or prediction could not be undertaken next time. The resulting cycle of more inquiries generated, more information gathered and more data captured for analysis meant the accuracy of results was improved.

Minori describes Vertex AI as having a number of strengths in helping CCBJ build a ML data analysis platform. “One of the major merits of Vertex AI was that we were able to realize MLOps that streamlines the entire development life cycle from construction of the ML pipeline to its execution,” he says.

With near real-time data analysis through Google Cloud, CCBJ teams can spend time developing strategies rather than waiting for data requested from the IT systems department. Exploratory data analysis is also considerably easier as repeated trial and error has greatly improved the accuracy of analyses. Before we used Machine Learning, most machine placement processes were done by human senses, by looking at a map to find the suggestion points. By using Machine Learning to generate a massive number of placement point suggestions, the efficiency of routing of salespeople has been dramatically improved. 

In the future, CCBJ aims to automate the continuous training pipeline with Vertex AI. “CCBJ is a tech company that operates in the food industry,” says Minori. With the organization operating a vending machine network of 700,000 units, it would like to create new businesses based on utilization and analyzing data. Some of these businesses may be based on Sustainable Development Goals (SDGs) initiatives such as the utilization of recycled PET bottles, measures to prevent food loss and ways of using vending machines to contribute to local communities, which we have been working on for some time. It would be interesting if we could collaborate with Google Cloud on these in the future.”

Editor’s note: Streak makes a CRM add-on for Gmail, and recently adopted Cloud Spanner to take advantage of its scalability and SQL capabilities to implement a graph data model. Read on to learn about their decision, what they love about the system, and the ways in which it still needs work.]  

Streak is a customer relationship management (CRM) tool built directly into Gmail. It is used for sales, marketing, hiring, and just about anything else you can think of.

We built it because out of the box, email is actually a really crummy team sharing system. By adding a layer of organization on top of email, Streak lets you add email threads directly into its spreadsheet view, making it useful as a workflow tool with capabilities including task creation, email template management, and easy data entry.

Streak has been integrated with G Suite (originally Google Apps For Your Domain) since its inception so when choosing a cloud, it made sense to colocate our server stack with Google Cloud.

Likewise, Streak was on Google App Engine from the start, and we slowly added other GCP services as the offerings improved or as our use cases became more complex.

In addition to App Engine, we use Google Kubernetes Engine to run a bunch of our compute workload, including both application servers and our offline processes like indexers and task queue consumers.

We use Cloud Dataflow for both streaming event processing and logs ETL, and BigQuery for all of our analytics queries. We use Cloud Pub/Sub for interacting with the Gmail watch API, as well as Stackdriver (logging, tracing, monitoring, errors) and OpenCensus to dig into any operational issues as they arise.

Then, on the database front, we recently started using Cloud Spanner, Google Cloud’s scalable relational database service. Before that, we stored most of our business data in Cloud Datastore, Google Cloud’s NoSQL document database.

Partially, that was historical, since Cloud Datastore was GCP’s only managed database when we wrote the Streak backend. And we’ve been very happy with how easy Cloud Datastore is to maintain. Between Google App Engine and Cloud Datastore, we’ve never had to have an explicit infrastructure on-call rotation.

But as more users rely on Streak to collaborate with larger and larger teams, we were feeling the pain of not having a fully relational database. We found ourselves having to manually join data in our application, which increased application latency and increased the time developers spent coding workarounds and debugging that complexity.

We found we needed two things out of our database: a scalable relational store and a graph store that could power next-gen Streak features. At the same time, we wanted a single database that could handle both use cases and wouldn’t increase our operational burden. This meant finding a managed service to give us more query flexibility, so we decided to give Cloud Spanner a try.

Of course, we didn’t want to migrate our existing stack to a new data platform without first testing it out (never a smart strategy). But since most of our existing data model required transactional updates with other entities, pulling out a single entity to test was challenging. We did have a feature in our pipeline that necessitated a graph data store and that was removed from our other data: our email metadata indexing system.

How your client software handles email metadata indexing can make or break the useability of a system.  Think about how many times somebody forgets to reply-all or that you receive a forwarded thread with thirty emails in reverse-chronological order. Within our own inboxes, we rely on Gmail’s UI to nicely organize email threads, but that organization breaks down when working with a team or across organizational boundaries.

We decided to fix that in the Streak product by organizing metadata (i.e., headers but not message content) from users’ email by using Cloud Spanner as a graph database. Using a graph database lets us answer questions like “What are all the emails on this thread in the inboxes of everybody on my team?” and “Who on my team has previously talked with the organization that this prospect works at?”

In our model, the nodes of the graph are either an email message, a person (email address) or a company (a domain). Then we have four different types of “edges”— properties by which nodes in a graph connect to one another:

1. Message to message (thread): messages that are on the same thread have an edge between them. The reason we do this is because we want to show users a list of threads to answer their questions, not messages, so we need to be able to get the spanning set of messages.
2. Message to message (same RFC id): A core value proposition of Streak is being able to see the “unified” version of a thread that shows each person on a team’s version of the email thread. To make sure we are getting each user’s version of a thread when we issue a query, there needs to be an edge between a message in the queryer’s inbox and the same message in their team’s inbox. In case you’re curious, Streak uses the RFC message id to determine that two messages across inboxes are actually the same.
3. Email address to message: a message has an edge to an email address if it was either the from, to, cc, or bcc on the message. This edge is crucial for queries that start with: “Show me all threads between this person and our team.”

Domain to message: a message has an edge to a domain if the domain is present in any of the from, to, cc, or bcc addresses on the message. This edge is similarly used for queries that start with “Show me all threads between this company and our team.”

Using Cloud Spanner’s distributed SQL capabilities and scalability to build a graph database also let us answer the important follow-up question: “Which threads have I been granted permission to view?” And while a lot of these questions could be answered per-user by a traditional relational database, scale limitations have to be taken into consideration, especially as we plan for 10x or more data volume growth as both our user base grows and as their inboxes accumulate more emails. A graph database model is simply a better fit for Streak’s collaboration model with many-to-many mappings between users and teams, and will allow us to query the data in any number of configurations, without worrying about scale limitations or having to manually shard a relational database. Cloud Spanner gives us queryability and scalability.

Taking the Cloud Spanner plunge

With so many advantages to it, we went ahead and began building out our metadata system with Cloud Spanner as a back-end.

Adopting Cloud Spanner has been great. Here are some of the high points:

1. The fast distributed queries and transactions are absolutely real. We have global indexes across our entire dataset and we haven’t had to spend very much time at all thinking about co-locating data. In particular, we only use interleaved tables for values that would be repeated fields in Cloud Datastore, and that hasn’t been a problem for us yet.
2. We haven’t had any reliability problems whatsoever, despite averaging 20K writes/sec in steady state.
3. Once we optimized our queries on realistic data, Cloud Spanner scaled up in a surprisingly predictable way. You need to run queries after you’ve populated data, do the explain to figure out how the query planner is executing the query, and add indexes/modify queries to make sure they’re performant.
4. Compared to the hoops some traditional relational databases make you jump through, Cloud Spanner’s online schema changes and index builds are magical. There is no downtime for these operations.

Overall, the experience has been encouraging, and we’re planning to move 20 TB of existing data in Cloud Datastore to Cloud Spanner as well. We built out an ORM library for Java on top of Cloud Spanner called Ratchet and are testing a framework for dual-writing entities to both Cloud Datastore and Cloud Spanner to support the rest of the migration. We now store about 40 TB of email metadata in Cloud Spanner, which makes us a large user of Cloud Spanner.

In short, if you’re starting to outgrow your NoSQL database, and want to move to a managed SQL database, give Cloud Spanner a try. You definitely want to model out your costs and try out a proof of concept, both to see how it works on your workload and to get familiar with the quirks of the system. But you don’t need to spend much time worrying about the reliability of the product: it’s there.