Over the past two decades, consumers have become accustomed to receiving personalized recommendations in all facets of their online life. Whether that be recommended products while shopping on Amazon, a curated list of apps in the Google Play store, or relevant videos to watch next on YouTube. In fact, in a Verge article “​​How YouTube perfected the feed: Google Brain gave YouTube new life,” the Google Brain team reveals how their recommendation engine has impacted the platform with “more than 70 percent of the time people spend watching videos on the site being driven by YouTube’s algorithmic recommendations” thereby increasing time spent on the platform by 20X in three years. 

It’s become clear that personalized recommendations are no longer a differentiator for an organization but rather something consumers have come to expect in their day-to-day experiences online. So what should you do if you are behind the curve and want to get started or simply want to improve upon what you already have? While there are all sorts of techniques, from content-based systems to deep learning methods, our goal in this recommender-focused blog series is to demystify three available approaches to building recommendation systems on Google Cloud: Matrix Factorization in BigQuery Machine Learning (BQML),  Recommendations AI, and deep retrieval techniques available via the Two-Tower built-in algorithm.

One of these approaches can be used to meet you where you are in your personalization journey, no matter if you are just starting or if you are well into it. This first blog post will introduce our three approaches and when to use them. 

What is Matrix Factorization and how does it work?

Collaborative filtering is a foundational model for building a recommendation system as the input dataset is simple and the embeddings are learned for you. How does Matrix factorization fit into the mix you might be wondering?  Matrix factorization is simply the model that applies collaborative filtering. BQML enables users to create and execute a matrix factorization model by using standard SQL directly in the data warehouse. 

Collaborative filtering begins by creating an interaction matrix. The interaction matrix represents users as a row and items as columns in your dataset. This interaction matrix often is sparse in nature as not all users will have interacted with many items in your catalog. This is where embeddings come into play. Generating embeddings for users and items not only allows you to collapse many sparse features into a lower dimensional space but they also allow you to derive a similarity measure so that similar users/items fall nearby in the embedding space. These similarity measures are key as collaborative filtering uses similarities between users and items to make the end recommendations. The underlying assumption being that similar users will like similar items whether that be movies or handbags. 

What’s required to get started?

To train a matrix factorization model you need a table that includes three input columns: user(s), item(s), and an implicit or explicit feedback variable (e.g., ratings is an example of explicit feedback). With the base input dataset in place, you can then easily run your model in BigQuery after specifying several hyperparameters in your CREATE MODEL SQL statement. Hyperparameters are available to specify the number of embeddings, the feedback type, the amount of L2 regularization applied and so on.    

Why use this approach and who is it a good fit for?  

As mentioned earlier, Matrix Factorization in BQML is a great way for those new to recommendation systems to get started. Matrix factorization has many benefits: 

• Little ML Expertise: Leveraging SQL to build the model lowers the level of ML expertise needed
• Few Input Features: Data inputs are straightforward, requiring a simple interaction matrix
• Additional Insight: Collaborative filtering is adept at discovering new interests or products for users 

While Matrix Factorization is a great tool for deriving recommendations it does come with additional considerations and potential drawbacks depending upon the use case. 

• Not Amenable to Large Feature Sets: The input table can only contain two feature columns (e.g., user(s), item(s)). If there is a need to include additional features such as contextual signals, Matrix factorization may not be the right method for you.  
• New Items: If an item is not available in the training data, the system can’t create an embedding for it and will have difficulty recommending similar items. While there are some workarounds available to address this cold-start issue, if your item catalog often includes new items, Matrix factorization may not be a good fit.  
• Input Data Limitations: While the input matrix is expected to be sparse, training examples without feedback can cause problems. Filtering for items and users that have at least a handful of feedback (e.g., ratings) examples can improve the model. More information on limitations can be found here. 

In summary, for users with a simplified dataset looking to iterate quickly and develop a baseline recommendation system, Matrix Factorization is a great approach to begin your personalization AI journey. 

What is Recommendations AI and how does it work?

Recommendations AI is a fully managed service which helps organizations deploy scalable recommendation systems that use state-of-the-art deep learning techniques, including cutting-edge architectures such as two-tower encoders, to serve personalized and contextually relevant recommendations throughout the customer journey.

Deep learning models are able to improve the context and relevance of recommendations in part because they can easily address the previously mentioned limitations of Matrix Factorization. They incorporate a wide set of user and item features, and by definition they emphasize learning successive layers of increasingly meaningful representations from these features. This flexibility and expressivity allows them to capture complex relationships like short-lived fashion trends and niche user behaviors. However, this increased relevance comes at a cost, as deep learning recommenders can be difficult to train and expensive to serve at scale. 

Recommendations AI helps organizations take advantage of serving these deep learning models and handles the MLOps required to serve these models globally with low latency. Models are automatically retrained daily and tuned quarterly to capture changes in customer behavior, product assortment, pricing, and promotions. Newly trained models follow a resilient CI/CD routine which validates they are fit to serve and promotes them to production without service interruption. The models achieve low serving latency by using a scalable approximate nearest neighbors (ANN) service for efficient item retrieval at inference time. And, to maintain consistency between online and offline tasks, a scalable feature store is used, preventing common production challenges such as data leakage and training-serving skew.  

What’s required to get started?

To get started with Recommendations AI we first need to ingest product and user data into the API:

• Import product catalog: For large product catalog updates, ingest catalog items in bulk using the catalogItems.import method. Frequent catalog updates can be schedule with Google Merchant Center or BigQuery
• Record user events: User events track actions such as clicking on a product, adding items to cart, or even purchasing an item. These events need to be ingested in real time to reflect the latest user behavior and then joined to items imported in the product catalog 
• Import historical user events: The models need sufficient training data before they can provide accurate predictions. The recommended user event data requirements are different across model types (learn more here)

Once the data requirements are met, we are able to create one or multiple models to serve recommendations: 

• Determine your recommendation types and placements:  The location of the recommendation panel and the objective for that panel impact model training and tuning. Review the available recommendations types, optimization objectives, and other model tuning options to determine the best options for your business objectives.
• Create model(s): Initial model training and tuning can take 2-5 days depending on the number of user events and size of the product catalog 
• Create serving configurations and preview recommendations: After the model is activated, create serving configurations and preview the recommendations to ensure your setup is functioning as expected before serving to production traffic

Once models are ready to serve, consider setting up A/B experiments to understand how newly trained models impact your customer experience before serving them to 100% of your traffic. In the Recommendations AI console, see the Monitoring & Analytics  page for summary and placement-specific metrics (e.g., recommender-engaged revenue, click-through-rate, conversion rate, and more).

Why use this approach and who is it a good fit for?  

Recommendations AI is a great way to engage customers and grow your online presence through personalization. It’s used by teams who lack technical experience with production recommendation systems, as well as customers who have this technical depth but want to allocate their team’s effort towards other priorities and challenges. No matter your team’s technical experience or bandwidth, you can expect several benefits with Recommendations AI: 

• Fully managed service: no need to preprocess data, train or hypertune machine learning models, load balance or manually provision you infrastructure – this is all taken care of for you. The recommendation API also provides a user-friendly console to monitor performance over time. 
• State-of-the-art AI: take advantage of the same modeling techniques used to serve recommendations across Google Ads, Google Search, and YouTube. These models excel in scenarios with long-tail products and cold-starts users and items
• Deliver at any touchpoint: serve high-quality recommendations to both first-time users and loyal customers anywhere in their journey via web, mobile, email, and more
• Deliver globally: serve recommendations in any language anywhere in the world at low-latency with a fully automated global serving infrastructure
• Your data, your models: Your data and models are yours. They’ll never be used for any other Google product nor shown to any other Google customer

For users looking to leverage state of the art AI to fuel their recommendation systems but need an existing solution to get up and running more quickly, Recommendations AI is the right solution for you. 

What are Two Tower encoders and how do they work?

As a reminder, in recommendation system design, our objective is to surface the most relevant set of items for a given user or set of users. The items are usually referred to as the candidate(s) where we might include information about the items such as the title or description of the item, other metadata about the item like language, number of views, or even clicks on the item over time. User(s) are often represented in the form of a query to a recommendation system where we might provide details about the user such as the location of the user, preferred languages, and what they have searched for in the past.   

Let’s start with a common example. Imagine that you are creating a movie recommendation system. The input candidates for such a system would be thousands of movies and the query set can consist of millions of viewers. The goal of the retrieval stage is to select a smaller subset of movies(candidates) for each user and then score and rank order them before presenting the final recommended list to the query/user.

The retrieval stage is able to refine our list of candidates by encoding both the candidate and the query data so they share the same embedding space. A good embedding space will place candidates which are similar to one another closer together and dissimilar items/queries farther apart in the embedding space.

Once we have a database of query and candidate embeddings we can then use an approximate nearest neighbor search method to then generate a list of final “like” candidates, i.e. find a certain number of nearest neighbors for a given query/user and surface final recommendations.

What’s required to get started?

At the most basic level, in order to train a two-tower model you need the following inputs:

• Training Data: Training data is created by combining your query/user data with data about the candidates/items. The data must include matched pairs, cases where both user and item information is available. Data in the training set can include many formats from text, numeric data, or even images. 
• Input Schema: The input schema describes the schema of the combined training data along with any specific feature configurations.

Several services within Vertex AI have come available that complement the existing Two-Tower built-in algorithm and can be leveraged in your execution: 

• Nearest Neighbor (ANN) Service: Vertex AI Matching Engine and  ScANN provide a high-scale and low-latency Approximate Nearest Neighbor (ANN) service so you can more easily identify similar embeddings.
• Hyperparameter Tuning Service: A hyperparameter tuning service such as Vizier can help you identify the optimal hyperparameters such as the number of hidden layers, the size of the hidden layers, and the learning rate in fewer trials. 
• Hardware Accelerators: Specialized hardware, such as GPUs or TPUs, can be valuable in your recommendation system to help accelerate experiments and improve the speed of training cycles. 

Why use this approach and who is it a good fit for?  

The Two-Tower built-in algorithm can be considered the “custom sports car” of recommendation systems and comes with several benefits: 

• Greater Control: While Recommendations AI uses the two-tower architecture as one of the available architectures it doesn’t provide granular control or visibility into model training, example generation, and model validation details. In comparison, the Two-Tower built in algorithm provides a more customizable approach as you are training a model directly in a notebook environment. 

• More Feature Options: The Two Tower approach can handle additional contextual signals ranging from text to images. 
• Cold Start Cases: Leveraging a rich set of features not only enhances performance but also allows the candidate generation to work for new users or new candidates.

While the Two-Tower built in algorithm is an excellent and best-in class solution for deriving   recommendations, it does come with additional considerations and potential drawbacks depending upon the use case.

• Technical ML Expertise Required: Two tower encoders are not a “plug and play” solution like the other approaches mentioned above. In order to effectively leverage this approach, appropriate coding and ML expertise is required. 
• Speed to Insight: Building out a custom solution via two-tower encoders may require additional time as the solution is not pre-built for the user.   

For users looking for greater control, increased flexibility, and have the technical chops to easily work within a managed notebook environment – the two-tower built in algorithm is the right solution for them. 

What’s next?

In this article, we explored three common methods for building recommendation systems on Google Cloud Platform. As you can see thus far, there are alot of considerations to take into account before choosing a final approach. In an effort to help you align more quickly we have distilled the decision criteria down to a few simple steps (see below for more details).

In the next installments of this series, we will dive more deeply into each method, explore how hardware accelerators can play a key role in recommendation system design, and discuss how recommendation systems may be leveraged in key verticals. Stay tuned for future posts in our recommendation systems series. Thank you for reading! Have a question or want to chat? Find authors here – R.E. [Twitter | LinkedIn], Jordan [LinkedIn], and Vaibhav [LinkedIn].

Acknowledgements

Special thanks to Pallav Mehta, Henry Tappen,Abhinav Khushraj, and Nicholas Edelman for helping to review this post. 

References

• Announcing ScaNN: Efficient Vector Similarity Search
• Factorization Machines for Item Recommendation with Implicit Feedback Data

Google Workspace recently commissioned Economist Impact to complete a global survey (October 2021)* on the state of hybrid work, including its challenges and opportunities. We already knew that the pandemic had fundamentally changed the world of work, but the survey emphasizes the scale, reach, and longevity of those changes.. 

Over 75% of respondents believe that hybrid/flexible work will be a standard practice within their organizations in the coming three years. Given that 70% of respondents said they never worked remotely before the pandemic, it’s clear that hybrid has become the dominant model for work and that it’s here to stay. 

But it’s also clear, as we’ll see below, that hybrid has some serious gaps that need to be addressed if it’s going to be sustainable and successful in the long term.

Defining hybrid work

As part of our work on the hybrid work survey and a broader project with The Economist Group, we interviewed a group of experts across research, consulting, business, and the advocacy world to arrive at a definition of hybrid work that captures all ways work is changing.

As Brian Kropp, one of the interviewees and vice president at Gartner, puts it: “Hybrid work is not just about different locations, but also different timings and different schedules.”

Harriet Molyneaux, managing director at HSM Advisory, a future-of-work research and advisory group, adds to this notion by describing the time-location work spectrum: “At one end of the spectrum is everyone in the office, nine till five, so both restricted time and location. And then at the other end of the spectrum is anywhere around the world at any time. So no restricted time or location. A hybrid is anything that sits in the middle of that.”

I agree with Brian and Harriet’s views. Flexibility in both location and hours is a core part of our working definition of hybrid work: a spectrum of flexible work arrangements in which an employee’s work location and/or hours are not strictly standardized.

Given this definition and framework, how is hybrid work faring and what lies ahead? 

Individual wellbeing is coming at the cost of organizational connection

Early on in the pandemic, productivity remained steady or even increased for many organizations. But it came at a cost, with levels of burnout spiking as employees juggled caregiving, homeschooling, and other demands in their personal lives with work responsibilities.

Based on the survey data, wellbeing has made an upward shift, no doubt aided by things like students returning to schools in many regions and fewer demands being made of working parents. The majority of respondents said that hybrid work, based on their own experiences, can have a positive impact on the physical, mental, financial, and social wellbeing of employees.

But it appears that individual wellbeing is coming at the cost of organizational connection. The majority of respondents said they feel disconnected from their organization and co-workers (57%), that limited networking opportunities negatively impact career growth (62%), and that limited social interactions with co-workers has had a negative impact on their mental health (54%).

To be sustainable, hybrid work models must address this sense of disconnection in real and tangible ways. And it won’t happen just by increasing the number of virtual meetings. Although 72% of people say that virtual meetings improve inclusion and participation, 68% also say there are too many virtual meetings to begin with. We need new ways for people to connect spontaneously.

Hybrid workers are often using tools built for a bygone desktop era

When asked about the most important conditions needed to achieve the long-term success of hybrid work models, the number one choice globally was “new technologies that allow for time and location flexibility.” 

Naturally, it was gratifying to see this, since Google Workspace is built on a cloud-based platform that empowers collaboration from anywhere, on any device, but the survey answer also highlights how much organizations have had to scramble to meet the demands of the pandemic. Some had to take legacy, office-centric systems and make them work for a distributed workforce overnight, often leveraging less-than-ideal technologies like VPNs that introduce digital friction.

As a result, top technology concerns of respondents include:

• unreliable internet access (if only all our WiFi connections were hybrid-ready!)
• reliance on slow or outdated tools
• accessing and maintaining files in multiple places
• relying on too many applications in order to get work done

Hybrid work, it seems to me, doesn’t need more applications and collaboration surfaces. It needs deeper, more meaningful connections in the tools and surfaces we already have. 

As organizations assess their tech stacks, they should consider whether they can shape the behaviors they want from hybrid employees (e.g., real-time collaboration, ease of information sharing) with the tools they have in place. Or are the limits of the technology determining hybrid employee behavior?

The management and culture gap

In the same way that tools have often been built for a shared physical workspace, organizational culture seems to lag behind the hybrid moment. The majority of respondents said that a lack of face-to-face supervision creates a sense of distrust among managers and employees, and that they feel stressed by increased monitoring associated with flexible work. And more than 62% said that limited networking opportunities with senior employees and co-workers has a negative impact on career growth.

Strikingly, more than 70% of respondents indicated that the culture of trust between managers and employees needed improvement. Training and management best practices (60% of people want more of both) might help fill some of the gaps, but a manager can’t build trust with their hybrid teams by going through a training program. 

The role of manager must itself evolve to meet the demands of a hybrid model. How can we empower managers to be the bridge between the “office of one” and the “office of many”? And as I discussed previously on Forbes, driving towards impact rather than output is a more sustainable metric for team engagement and productivity. It also dispenses with monitoring as a core part of a manager’s role, freeing them up to be a coach-and-connector.

How Google Workspace can help bridge the hybrid work gaps

No one has all the answers for how to make hybrid work successful, and I suspect we will see iterations of the model into 2022 and beyond as organizations—including Google—experiment with the right mix of location, culture, processes, and tools. 

On the technology front, Google Workspace is uniquely positioned to bridge many of the emerging hybrid work gaps. Over the last year, we’ve delivered a set of innovations that are designed to deepen collaboration experiences while strengthening social connections within and across teams. 

For example, we launched smart canvas to bring new collaboration capabilities to the places where people are already working together, helping to keep them connected, rather than having them switch tabs or open new apps. And we delivered Spaces as a central place for collaboration, where teams can share ideas, work on documents together, and manage tasks from a single place. Because all their work is preserved for future reference, team members can easily jump in and contribute at a time that works best for them, seeing a full history of the conversations, context, and content along the way. Spaces helps people maintain individual wellbeing while preserving the health of a group project.

In Google Meet, we’ve introduced real-time captions in multiple languages (in preview now), the ability for people to customize their video tiles, including being able to turn off their own to help with meeting fatigue, and we’ve implemented automatic light adjustments and noise cancellation. These changes help ensure that everyone can be seen and heard.

Meanwhile, features like hand-raise, Q&A, and polls have helped make meetings more inclusive and companion mode (in preview) can bring these to life in hybrid settings, to help ensure that there’s one cohesive conversation between the people in the office and their colleagues working somewhere else. We view companion mode as a bridge between the office and “somewhere else.”

On the personal wellbeing front, we launched Focus Time, which lets people block out their calendars for uninterrupted focus work, and Time Insights in Google Calendar, where employees can look at how they’re spending their time and adjust as needed.

As hybrid work continues to evolve, Google Workspace is committed to supporting wellbeing and seamless collaboration with tools that can remove digital friction and help people maintain connections—to each other and the organizations they work for.

^{*Survey details: The survey, completed in October 2021, polled a total of 1,244 employees and managers in four regions (North America, Europe, APAC, and Latin America), from more than 15 industries, in every age group, and from both small and large organizations. The focus was on knowledge workers, though it’s important to note that some of those people have been working on the frontlines; 20% of respondents indicated they haven’t worked remotely at all during the pandemic. This includes people in hospitality, retail, transportation and logistics, and healthcare.}

Access to the cloud has advanced the distribution and consumption of financial information on a global scale. In parallel, the global financial data landscape has been transformed by an influx of alternative data sources, including social media, meteorological data, satellite imagery, and other data. Exchanges and market data providers now find they need to include these new datasets to enrich their products and compete, which has meant they now must consider cloud-based models to keep up with the demands of their customers who expect easy, quick, flexible and cost-efficient ways to consume market data.

To address these needs, today we’re announcing the general availability of Datashare for financial services, a new Google Cloud solution that brings together the entire capital markets ecosystem—data publishers, and data consumers—to exchange market data securely and easily.

Datashare helps organize third-party financial information, making it accessible and useful to market data publishers and data consumers. We open-sourced the entire Datashare solution so market data publishers can now onboard their licensed datasets to Google Cloud securely, quickly and easily, while data consumers can consume that data as a service in tools of their preference, such as BigQuery.

Three ways to distribute and consume your data

Batch data delivery

Datashare provides a batch data delivery mechanism for data publishers to deliver their reference data, historical tick data, alternative market data sources and more via BigQuery, reducing the administrative burden on data consumers to extract insights from data. 

Real-time data streaming delivery

By using this event-based data delivery channel for rapidly changing instrument prices, tick data, orders, news and others via Pub/Sub, data consumers can reliably process individual messages or rewind to a point in time to replay a prior market scenario and test model changes.

Monetizing licensed datasets

Market data publishers can onboard their licensed datasets to Google Cloud and make them available via a one-stop-shop on Google Cloud Marketplace, enabling a new sales channel to expand market reach.

Reference architecture

Check out the diagram below to see how you can share your batch and real-time data directly to your Google Cloud customers with BigQuery and Pub/Sub.

As you can see in the above reference architecture, both publishers and consumers can derive several benefits from the solution:

Benefits for data publishers

• You no longer have to maintain your own delivery and licensing infrastructure.
• You can easily package and deliver granular data products and experiments with SQL.
• You can have a solution that scales with your business as data volumes and number of customers grow.

Benefits for data consumers

• Your data is ready for analysis and machine learning (ML)— you no longer have to maintain extract, transform, and load (ETL) pipelines to load files and transform data.
• You can avoid the expense and burden of maintaining multiple copies of large data files.
• You can be more targeted with consumption of data using BigQuery queries, improving performance, and compliance, and reducing cost.

Accessing the datasets

Google Cloud has been working with multiple industry firms on innovating in the market data space. By using Datashare for publishing, data publishers can make their entire datasets available on Google Cloud. Early adopters of Datashare include firms such as OneTick and Accern. OneTick’s datasets include reference and historical futures data (that can be accessed in our console with your login). Accern’s datasets include alternative data such as market sentiment and credit analysis data (that can be accessed in our console with your login).

To make it more helpful, we partnered with Accern to create a hypothetical scenario to describe the data acquisition and analytics process step-by-step.

Accern use case 

As a sustainability analyst, you require an economic, social and governance (ESG) dataset to determine which sector is the most widely covered ESG sector by analysts, and to also identify the sector with the lowest ESG sentiment score. Now, you can discover and acquire an ESG dataset in Google Cloud.

Step 1. Navigate to the Financial Services solutions page in the Google Cloud console:

Step 2. Click a dataset, for example Accern AI-Generated ESG Insights, then review the overview details, plans and pricing, documentation and support information. To view the available pricing tiers, click ‘View All Plans’. Once you’ve decided on a tier that you would like to subscribe to, click ‘Select’, choose a billing account and review and accept the terms of service to complete the subscription. Once the steps are complete, click ‘Subscribe’ at the bottom. An overlay window will appear, click ‘Register with Accern’ to activate and complete the subscription.

Step 3. Once activation is complete, you’ll be directed to the Datashare ‘My Products’ screen. Voila! You are now subscribed to Accern’s ESG Scores dataset and can access it in your Google Cloud instance using BigQuery. To access the data, click the hour glass icon on the corresponding ‘My Products’ record that you just purchased. An overlay will present you with the details on the dataset and/or table. Click the ‘Navigate to Table’ button to navigate through to the BigQuery console.

Step 4. Now that you have access and are in the BigQuery console, it’s time to generate data insights.

^{For this example, we’ve eliminated the company identifying information that is included as part of the subscription and aggregated company ESG in a view where each row represents a day, an industry sector, a specific identified ‘ESG Issues’ (event_group and event) and the respective ‘ESG Sentiment’ per issue.}

For example, row 1 indicates that within the ‘Healthcare’ sector, there was a ‘Social – Civil Society’ issue identified and it had a negative ESG sentiment score of -15.35.

Step 5. Generate a report by exporting it to Data Studio to build visualizations and conduct additional analysis on the ESG data.

Select ‘Export’ and ‘Explore with Data Studio’.

Step 6. Build a simple/basic report.

Now that the ESG data appears in Data Studio, you can start by building a simple chart to help you understand which industry sectors have the highest volume of discussions around ESG and the overall ESG Sentiment per industry sector.

To build the chart:

• Select the chart type ‘Table’.

• Include Entity_Sector as your dimension to aggregate results by ‘Industry Sector.’
• Include Signal_ID as a measure to count the number of ESG passages identified per ‘Industry Sector.’
• Include AVG(Event_Sentiment) as a measure to display the overall ESG Sentiment per ‘Industry Sector’ across ESG Issues.

You can see sectors that are  most discussed when it comes to ESG related topics and their corresponding ‘ESG Sentiment’ scores.

Step 7. Build your final report in Data Studio.

As a next step you can further drill into the data to understand ESG data specific to each ‘Industry Sector’ and identify positive and negative ESG practices. 

Accern has built a more complex sample dashboard and made it available publicly here. You can interact with this report and play around with the data. The dashboard can help to identify material ESG insights for each sector to inform your investment and risk processes. If you have additional questions, you can reach out to Accern directly.

Discovering, accessing and analyzing licensed datasets is quick and easy. Stay tuned for more updates on new licensed datasets.

Publishing your data via Datashare

If you are a publisher of market data, alternative, or exotic data, you can use Datashare to get it published on Google Cloud Marketplace.

Start by joining the Partner Advantage program by registering for the Partner Advantage Portal and applying for the Partner Advantage Build Model engagement. Visit our getting started guide for information to get started on publishing licensed datasets in the Marketplace. Stay tuned for a future blog post about using Datashare to publish datasets in the Marketplace.

More solutions for capital markets

Check out other Google Cloud solutions for capital markets.

For the past few years, we’ve applied core Google AI research and engineering to projects with positive societal impact, including forecasting floods, protecting whales and predicting famine. Artificial intelligence has incredible potential to address big social, humanitarian and environmental problems, but in order to achieve this potential, it needs to be accessible to organizations already making strides in these areas. So, the Google AI Impact Challenge, which kicked off in October 2018, was our open call to organizations around the world to submit their ideas for how they could use AI to help address societal challenges.

“Accelerating social good with artificial intelligence” sheds light on the range of organizations using AI to address big problems. It also identifies several trends around the opportunities and challenges related to using AI for social good. Here are some of the things  we learned—check out the report for more details.

AI is globally relevant 

We received 2,602 applications from six continents and 119 countries, with projects addressing a wide range of issue areas, from education to the environment. Some of the applicants had experience with AI, but 55 percent of not-for-profit organizations and 40 percent of for-profit social enterprises reported no prior experience with AI. 

Similar projects can benefit from shared resources

When we reviewed all the applications, we saw that many people are trying to tackle the same problems and are even using the same approaches to do so. For example, we received more than 30 applications proposing to use AI to identify and manage agricultural pests. The report includes a list of common project submissions, which will hopefully encourage people to collaborate and share resources with others working to solve similar problems.  

You don’t need to be an expert to use AI for social good

AI is becoming more accessible as new machine learning libraries and other open-source tools, such as Tensorflow and ML Kit, reduce the technical expertise required to implement AI. Organizations no longer need someone with a deep background in AI, and they don’t have to start from scratch. More than 70 percent of submissions, across all sectors and organization types, used existing AI frameworks to tackle their proposed challenge. 

Successful projects combine technical ability with sector expertise 

Few organizations had both the social sector and AI technical expertise to successfully design and implement their projects from start to finish. The most comprehensive applications established partnerships between nonprofits with deep sector expertise, and academic institutions or technology companies with technical experience.

ML isn’t the only answer 

Some problems can be addressed by using alternative methods to AI—and result in faster, simpler and cheaper execution. For example, several organizations proposed using machine learning to match underserved populations to legal knowledge and tools. While AI could be helpful, similar results could be achieved through a well-designed website. While we’ve seen the impact AI can have in solving big problems, you shouldn’t rule out more simple approaches as well. 

Global momentum around AI for social good is growing—and many organizations are already using AI to address a wide array of societal challenges. As more social sector organizations recognize AI’s potential, we all have a role to play in supporting their work for a better world.