Enabling Real-time AI with Streaming Ingestion in Vertex AI
2520
Of your peers have already read this article.
2:30 Minutes
The most insightful time you'll spend today!
Many machine learning (ML) use cases, like fraud detection, ad targeting, and recommendation engines, require near real-time predictions. The performance of these predictions is heavily dependent on access to the most up-to-date data, with delays of even a few seconds making all the difference. But it’s difficult to set up the infrastructure needed to support high-throughput updates and low-latency retrieval of data.
Starting this month, Vertex AI Matching Engine and Feature Store will support real-time Streaming Ingestion as Preview features. With Streaming Ingestion for Matching Engine, a fully managed vector database for vector similarity search, items in an index are updated continuously and reflected in similarity search results immediately. With Streaming Ingestion for Feature Store, you can retrieve the latest feature values with low latency for highly accurate predictions, and extract real-time datasets for training.
For example, Digits is taking advantage of Vertex AI Matching Engine Streaming Ingestion to help power their product, Boost, a tool that saves accountants time by automating manual quality control work.“Vertex AI Matching Engine Streaming Ingestion has been key to Digits Boost being able to deliver features and analysis in real-time. Before Matching Engine, transactions were classified on a 24 hour batch schedule, but now with Matching Engine Streaming Ingestion, we can perform near real time incremental indexing – activities like inserting, updating or deleting embeddings on an existing index, which helped us speed up the process. Now feedback to customers is immediate, and we can handle more transactions, more quickly,” said Hannes Hapke, Machine Learning Engineer at Digits.
This blog post covers how these new features can improve predictions and enable near real-time use cases, such as recommendations, content personalization, and cybersecurity monitoring.
Streaming Ingestion enables real-time AI
As organizations recognize the potential business impact of better predictions based on up-to-date data, more real-time AI use cases are being implemented. Here are some examples:
- Real-time recommendations and a real-time marketplace: By adding Streaming Ingestion to their existing Matching Engine-based product recommendations, Mercari is creating a real-time marketplace where users can browse products based on their specific interests, and where results are updated instantly when sellers add new products. Once it’s fully implemented, the experience will be like visiting an early-morning farmer’s market, with fresh food being brought in as you shop. By combining Streaming Ingestion with Matching Engine’s filtering capability, Mercari can specify whether or not an item should be included in the search results, based on tags such as “online/offline” or “instock/nostock.”
Mercari Shops: Streaming Ingestion enables real-time shopping experiment
- Large-scale personalized content streaming: For any stream of content representable with feature vectors (including text, images, or documents), you can design pub-sub channels to pick up valuable content for each subscriber’s specific interests. Because Matching Engine is scalable (i.e., it can process millions of queries each second), you can support millions of online subscribers for content streaming, serving a wide variety of topics that are changing dynamically. With Matching Engine’s filtering capability, you also have real-time control over what content should be included, by assigning tags such as “explicit” or “spam” to each object. You can use Feature Store as a central repository for storing and serving the feature vectors of the contents in near real time.
- Monitoring: Content streaming can also be used for monitoring events or signals from IT infrastructure, IoT devices, manufacturing production lines, and security systems, among other commercial use cases. For example, you can extract signals from millions of sensors and devices and represent them as feature vectors. Matching Engine can be used to continuously update a list of “the top 100 devices with possible defective signals,” or “top 100 sensor events with outliers,” all in near real time.
- Threat/spam detection: If you are monitoring signals from security threat signatures or spam activity patterns, you can use Matching Engine to instantly identify possible attacks from millions of monitoring points. In contrast, security threat identification based on batch processing often involves potentially significant lag, leaving the company vulnerable. With real-time data, your models are better able to catch threats or spams as they happen in your enterprise network, web services, online games, etc.
Implementing streaming use cases
Let’s take a closer look at how you can implement some of these use cases.
Real-time recommendations for retail
Mercari built a feature extraction pipeline with Streaming Ingestion.
The feature extraction pipeline is defined with Vertex AI Pipelines, and is periodically invoked by Cloud Scheduler and Cloud Functions to initiate the following process:
- Get item data: The pipeline issues a query to fetch the updated item data from BigQuery.
- Extract feature vector: The pipeline runs predictions on the data with the word2vec model to extract feature vectors.
- Update index: The pipeline calls Matching Engine APIs to add the feature vectors to the vector index. The vectors are also saved to Cloud Bigtable (and can be replaced with Feature Store in the future).
“We have been evaluating the Matching Engine Streaming Ingestion and couldn’t believe the super short latency of the index update for the first time. We would like to introduce the functionality to our production service as soon as it becomes GA, ” said Nogami Wakana, Software Engineer at Souzoh (a Mercari group company).
This architecture design can be also applied to any retail businesses that need real-time updates for product recommendations.
Ad targeting
Ad recommender systems benefit significantly from real-time features and item matching with the most up-to-date information. Let’s see how Vertex AI can help build a real-time ad targeting system.
The first step is generating a set of candidates from the ad corpus. This is challenging because you must generate relevant candidates in milliseconds and ensure they are up to date. Here you can use Vertex AI Matching Engine to perform low-latency vector similarity matching, generate suitable candidates, and use Streaming Ingestion to ensure that your index is up-to-date with the latest ads.
Next is reranking the candidate selection using a machine learning model to ensure that you have a relevant order of ad candidates. For the model to use the latest data, you can use Feature Store Streaming Ingestion to import the latest features and use online serving to serve feature values at low latency to improve accuracy.
After reranking the ads candidates, you can apply final optimizations, such as applying the latest business logic. You can implement the optimization step using a Cloud Function or Cloud Run.
What’s Next?
Interested? The documents for Streaming Ingestion are available and you can try it out now. Using the new feature is easy: For example, when you create an index on Matching Engine with the REST API, you can specify the indexUpdateMethod attribute as STREAM_UPDATE.
{
displayName: "'${DISPLAY_NAME}'",
description: "'${DISPLAY_NAME}'",
metadata: {
contentsDeltaUri: "'${INPUT_GCS_DIR}'",
config: {
dimensions: "'${DIMENSIONS}'",
approximateNeighborsCount: 150,
distanceMeasureType: "DOT_PRODUCT_DISTANCE",
algorithmConfig: {treeAhConfig: {leafNodeEmbeddingCount: 10000, leafNodesToSearchPercent: 20}}
},
},
indexUpdateMethod: "STREAM_UPDATE"
}After deploying the index, you can update or rebuild the index (feature vectors) with the following format. If the data point ID exists in the index, the data point is updated, otherwise, a new data point is inserted.
{
datapoints: [
{datapoint_id: "'${DATAPOINT_ID_1}'", feature_vector: [...]},
{datapoint_id: "'${DATAPOINT_ID_2}'", feature_vector: [...]}
]
}It can handle the data point insertion/update at high throughput with low latency. The new data point values will be applied in any new queries within a few seconds or milliseconds (the latency varies depending on the various conditions).
The Streaming Ingestion is a powerful functionality and very easy to use. No need to build and operate your own streaming data pipeline for real-time indexing and storage. Yet, it adds significant value to your business with its real-time responsiveness.
To learn more, take a look at the following blog posts for learning Matching Engine and Feature Store concepts and use cases:
5371
Of your peers have already watched this video.
44:30 Minutes
The most insightful time you'll spend today!
How Go-Jek, Indonesia’s First Billion-dollar Startup, Improved the Productivity of Data Scientists
Go-Jek, Indonesia’s first billion-dollar startup, has seen an incredible amount of growth in both users and data over the past two years. Many of the ride-hailing company’s services are backed by machine learning models hosted on Google Cloud Platform.
Models range from driver
But senior executives at Go-Jek realized something: One of their most important and expensive resources, data scientists, were spending far too much time cleaning data. That wasn’t part of their remit and resulted in a waste of time and money.
As a COO, this a major concern for any company undertaking a machine learning initiative. Data scientists are hard to come by and their salaries have been on the rise for the last few years. Yet according to some reports data scientists spend upto 80% of their time just preparing data—not creating models.
Watch how operational teams at Go-Jek combined the right Google tools and processes to improve the productivity of their data scientists.
AgroStar: Small farms in India getting big help from the cloud
13271
Of your peers have already read this article.
3:30 Minutes
The most insightful time you'll spend today!
AgroStar has launched a cloud-based mobile app that is helping to boost crop yields and encourage best practices for small farmers in India. Launched as an on-premises ecommerce platform selling farm tools in 2008, the firm turned to Google Cloud Platform (GCP) to expand its offering. It now uses cloud-based analytics and is deploying ML models to provide timely advice in five languages on everything from seed optimization, crop rotation, and soil nutrition to pest control.
A 2018 survey underscored the demand for agricultural planning for Indian farmers. While farming remains a dominant sector in India, employing half of its labor force, 70 percent of small farmers – those cultivating fewer than three acres – said their crops are damaged by unforeseen weather and pests. An even higher number – 74 percent – say they lack access to farming-related information.
Widening that gap is the relative lack of access to new, higher yield seeds and improved soil analyses for small farmers, who must otherwise rely on traditional methods. “It could take a few years for innovative information to trickle down from universities to small, grassroots farmers,” says Pritesh Gudge, AgroStar Software Engineer. “Today, just by clicking through our Android application, farmers learn about new, effective farming practices and receive advice customized to their crop and soil.”
Connecting a million farmers in the cloud
Operating in the Indian states of Gujarat, Maharashtra, Rajasthan, Orissa, Bihar, and Karnataka, AgroStar is closing the knowledge gap with a full-service, cloud-based SaaS solution – the only one of its kind in India. It combines agronomy, data science, and analytics to help farmers by providing a variety of resources.
AgroStar has reached over a million farmers through its Android app, the AgroStar Agri-Doctor. The mobile client is available as a web-based or full-featured native app. Both provide access to the firm’s knowledge base hosted on GCP, a Q&A forum that connects farmers to each other to help understand and better solve problems and to learn about innovative practices and products. Farmers can also click through to follow local and national market trends that help forecast crop prices.
In addition to the self-service knowledge base, AgroStar provides access to agronomy experts who use cloud-based analytics tools and historical data to provide season-and locale-specific advice to each farmer. “We are now tracking thousands of calls in 5 languages each day,” says Pritesh.
The AgroStar app also provides links to purchase and then track the delivery of farm tools and supplies such as cultivators and fertilizers. An in-house platform manages fulfillment centers and a doorstep delivery network simplifies the supply chain while giving farmers what they need, when they need it. By procuring directly from the manufacturers and primary distributors of farm supplies, Agrostar is achieving cost savings, which it passes on to farmers.
Build fast, pivot faster
From the start, the human and environmental variables of farming in India, not to mention the volume of AgroStar’s few hundred thousand monthly active users, made a highly scalable cloud-based solution inevitable. Farmers rely on the firm’s Agri-Doctor app to provide advice in multiple languages on topics that range widely throughout three growing seasons, each with distinct crop nutrition and rotation cycles and farm implementation requirements.
“For farmers, the focus keeps changing every month, and every season,” says Pritesh. “To serve our growing community, we needed a platform that could process images at high volume, fulfill tools and seed orders across thousands of miles, and respond to multilingual queries. We quickly moved away from spreadsheets and server-based solutions – we needed to build fast and pivot faster.”
Ending late-night deployments
The firm’s first cloud experience was with an AWS solution. At the time, AWS was the only cloud provider in India, but AgroStar wanted to find a solution that was easier to use and offered better integration with Android devices. “Deployment and processing costs were very high, and the developer tools and documentation were not as intuitive as we needed,” says Pritesh.
When GCP service arrived in India in October 2017, AgroStar embarked on a platform re-implementation that made possible dramatic changes in the way it developed and deployed its solution. Using Google Kubernetes Engine (GKE) for crop advice management and Compute Engine for its production application services, the firm built the backend for the Agri-Doctor discussion forum in only three weeks. The platform’s microservice architecture is implemented in Python and Golang and deployed on GCP.
AgroStar began to realize significant efficiencies in its build, deploy, and test cycles. “We previously needed to work overnight to deploy to production,” says Pritesh. “Now using Google for Kubernetes containers and a rolling update strategy, we can deploy during the day without any problems or interruptions to service.”
The move to GCP streamlined AgroStar’s stack. “We were running 12 independent instances on AWS,” says Pritesh. “With Google Kubernetes Engine, we are deployed on a single cluster at a cost savings of $1,300 per month and growing.”
Improving customer response times by 85 percent
With a managed deployment capability, AgroStar can devote more time and resources to executing on its platform and Agri-Doctor app development plan. A strategic goal was managing customer response times as the firm grew its base. GCP has helped the firm meet that goal, achieving an 85 percent improvement in customer response times even as traffic grew significantly.
“With our on-premises solution, we could handle around 100 customers daily, which took 30 to 50 minutes for each customer,” says Pritesh. “We now handle thousands of customers daily, taking only 4 to 5 minutes for each one.”
AgroStar used Firebase to implement its Agri-Doctor app. A real-time cloud database, Firebase provides an API that enables the Agri-Doctor advice forum to be synchronized across all its far-flung mobile clients, effectively sharing knowledge base updates with one million users in near real time.
Using cloud tools to manage and monitor
Cloud Pub/Sub, Kafka, and Cloud Dataflow manage data ingestion and queueing of event and transaction data to the analytics layer. BigQuery fetches and persists data to Cloud Storage. Cloud SQL and dashboards powered by Tableau deliver farmer crop and soil profiles within minutes.
Cloud IAM helps AgroStar control access to all its cloud resources. And Stackdriver, the integrated logging aggregation capability for GCP, helps monitor and speed debugging on every tier of the AgroStar solution.
Machine learning to enhance yields
AgroStar is developing a variety of ML components to improve responsiveness and extend its platform offerings.
To speed up the diagnosis of and treatment for crop blight, AgroStar is building a deep learning pipeline using TensorFlow. The pipeline relies on GoogLeNet models that use multi-layered convolutional visual pattern recognition. It will assess uploaded images to support a disease-detection capability on the mobile app. Based on the commercially successful AI algorithms that automated postal code processing, GoogLeNet offers improved performance and computational efficiencies by using a creative layering technique that distinguishes them from older, sequential recognition engines.
To improve its customer search experience, AgroStar is developing an ML pipeline that shrinks fetch times by suggesting tags mapped to stored data. Processed using TPUs, Cloud Natural Language and Video AI, the tags provide a metadata layer that supports queries in any of the ten natural languages that AgroStar farmers can use.
The AgroStar search pipeline consists of Long Short-Term Memory (LSTM) models of Recurrent Neural Networks. Recurrent networks exhibit “memory” through iterative processing and are distinguished from feedforward networks by a feedback loop connected to their past decisions, ingesting their own outputs moment after moment as input.
Implementing a recommendation engine
The firm is also adapting the Random Forests TensorFlow AI model to develop a crop and product recommendation engine. The model is trained by consuming numerical (rainfall, humidity, water availability per acre) and categorical (soil type, water sources) parameters to suggest appropriate products by season, region, and locale.
To simplify the product suggestion experience, AgroStar developers are testing Cloud Dialogflow, the Google Cloud conversational interface, to build a chatbot capability into its mobile app. The bot will track a farmer’s crop schedules and answer simple questions by linking to the recommendation engine.
AgroStar is also extending its analytics platform with AI-powered sales planning and forecasting. Using linear regression models implemented in TensorFlow and powered by Cloud ML Engine, the capability will enhance supply chain logistics as the company scales its operations across India.
To provide a credit on-demand offering for a range of seed-to-harvest cycle products, AgroStar is attempting to use Vision API to create an AI model that will convert uploaded photos of customer application records into standard data formats. The firm’s credit policy features a grace period in which farmers begin paying back loans after harvested crops go to market.
A versatile and friendly development ecosystem
AgroStar credits the convivial tools and documentation that GCP offers and its incremental, pay-as-you-go pricing model for both the firm’s success and its ability to manage growth.
“What Google Cloud offers is extremely good documentation and extremely simple-to-use tools and interfaces across all services,” says Pritesh. “It helped us initially deploy our platform and at every scale that we have required since then, and its cost effectiveness enabled us to staff up to meet new feature milestones.”
3140
Of your peers have already watched this video.
17:00 Minutes
The most insightful time you'll spend today!
How FLYR and Google Cloud Help Airlines Forecast Demand and Set Prices
FLYR Labs is an international team of industry experts and specialists in revenue management that works to bring in intelligence to the airlines companies. FLYR uses machine learning and AI to help predict demand and optimize price so that every airline is operating its complete capacity. Watch the video from Architecting with Google Cloud to deep-dive into a use case with FLYR involving the use of historic data, competitors data and future information to build model for outputting demand, set prices and optimize revenue. You can can even have a quick view of the FLYR ML platform!
AirAsia Turns to Google Cloud to refine Pricing, Increase Revenue, and Improve Customer Experience
DOWNLOAD CASE STUDY7139
Of your peers have already downloaded this article
1:30 Minutes
The most insightful time you'll spend today!
AirAsia needed a platform incorporating products that could capture, process, analyze, and report on data, while delivering value for money and meeting its speed and availability requirements. The airline also wanted to minimise infrastructure management and system administration demands on its technology team members.
The airline conducted a proof of concept and found Google Cloud Platform—including the Google BigQuery analytics data warehouse—was the best fit.
AirAsia was impressed by the ease and flexibility with which it could extract, transform, and load customer data from its systems, websites, and mobile applications into Google BigQuery for analysis. Reporting and dashboards were quickly and effectively delivered through Google Data Studio.
“With a minimal number of people involved, we can very quickly transform an idea or thought process into a deliverable. Prior to Google Cloud Platform, bringing those ideas to fruition would have been impossible,” says Nikunj Shanti, Chief Data and Digital Officer, AirAsia.
ML Workflow Made Simple: How to Automate ML Experiment Tracking with Vertex AI Experiments Autologging
1274
Of your peers have already read this article.
3:30 Minutes
The most insightful time you'll spend today!
Practical machine learning (ML) is a trial and error process. ML practitioners compare different performance metrics by running ML experiments till you find the best model with a given set of parameters. Because of the experimental nature of ML, there are many reasons for tracking ML experiments and making them reproducible including debugging and compliance.
But tracking experiments is challenging: you need to organize experiments so that other team members can quickly understand, reproduce and compare them. That adds overhead that you don’t need.
We are happy to announce Vertex AI Experiments autologging, a solution which provides automated experiment tracking for your models, which streamlines your ML experimentation
With Vertex AI Experiments autologging, you can now log parameters, performance metrics and lineage artifacts by adding one line of code to your training script without needing to explicitly call any other logging methods.
How to use Vertex AI autologging
As a data scientist or ML practitioner, you conduct your experiment in a notebook environment such as Colab or Vertex AI Workbench. To enable Vertex AI Experiments autologging, you call aiplatform.autolog() in your Vertex AI Experiment session. After that call, any parameters, metrics and artifacts associated with model training are automatically logged and then accessible within the Vertex AI Experiment console.
Here’s how to enable autologging in your training session with a Scikit-learn model.
# Enable autologging
aiplatform.autolog()
# Build training pipeline
ml_pipeline = Pipeline(...)
# Train model
ml_pipeline.fit(x_train, y_train)This video shows parameters and training/post-training metrics in the Vertex AI Experiment console.
Vertex AI SDK autologging uses MLFlow’s autologging in its implementation and it supports several frameworks including XGBoost, Keras and Pytorch Lighting. See documentation for all supported frameworks.
Vertex AI Experiments autologging automatically logs model time series metrics when you train models along multiple epochs. That’s because of the integration between Vertex AI Experiments autologging and Vertex AI Tensorboard.
Furthermore, you can adapt Vertex AI Experiments autologging to your needs. For example, let’s say your team has a specific experiment naming convention. By default, Vertex AI Experiments autologging automatically creates Experiment Runs for you without requiring you to call `aiplatform.start_run()` or `aiplatform.end_run()`. If you’d like to specify your own Experiment Run names for autologging, you can manually initialize a specific run within the experiment using aiplatform.start_run() and aiplatform.end_run() after autologging has been enabled.
What’s next
You can access Vertex AI Experiments autologging with the latest version of Vertex AI SDK for Python. To learn more, check out these resources :
- Documentation: Autolog data to an experiment run
- Github: Get started with Vertex AI Experiments autologging
While I’m thinking about the next blog post, let me know if there is Vertex AI content you’d like to see on Linkedin or Twitter.
More Relevant Stories for Your Company
Vodafone Leverages Google Cloud to Aid COVID-19 Frontline with Anonymized Insights on Population Mobility
Editor’s note: When Europe’s largest mobile communications company, Vodafone, was asked by the European Commission to help understand population movement across the European Union and the UK to help fight COVID-19, it was able to provide anonymized mobile network-based insights to answer the call. Here’s how Vodafone, with the support
Revolutionizing Generative AI Applications with Google’s Vertex AI
At Google Cloud, we’re committed to making generative AI useful for everyone. Doing so requires more than making powerful foundation models available to businesses, governments, and developers. Models also need to be backed by platforms that make adoption faster and safer, with onramps to meet organizations wherever they are, regardless of their
Google Cloud Partnership Fuels ListenField’s Agriculture Revolution
When I was growing up in Thailand, I witnessed the challenges facing farmers including rising food demand, shortage of labor, and uneven crop yields caused by climate change. As a result, many smallholder farmers found themselves trapped in a vicious circle, unable to reduce food insecurity due to low yields,
Driving Business Transformation in Financial Services Using Google Cloud and AI/ML
These are challenging times with financial services institutions navigating the pandemic. Interest rates are an all-time low, and customer expectations are changing with the shift to digital. For example, digital banking has increased from 63 in 2019 to 72 percent today. Then regulatory requirements are evolving and compliance costs are
