One of the biggest challenges when serving machine learning models is delivering predictions in near real-time. Whether you’re a retailer generating recommendations for users shopping on your site, or a food service company estimating delivery time, being able to serve results with low latency is crucial. That’s why we’re excited to announce Private Endpoints on Vertex AI, a new feature in Vertex Predictions. Through VPC Peering, you can set up a private connection to talk to your endpoint without your data ever traversing the public internet, resulting in increased security and lower latency for online predictions. 

Configuring VPC Network Peering

Before you make use of a Private Endpoint, you’ll first need to create connections between your VPC (Virtual Private Cloud) network and Vertex AI.  A VPC network is a global resource that consists of regional virtual subnetworks, known as subnets, in data centers, all connected by a global network. You can think of a VPC network the same way you’d think of a physical network, except that it’s virtualized within GCP. If you’re new to cloud networking and would like to learn more, check out this introductory video on VPCs.

With VPC Network Peering, you can connect internal IP addresses across two VPC networks, regardless of whether they belong to the same project or the same organization. As a result, all traffic stays within Google’s network.

Deploying Models with Vertex Predictions

Vertex Predictions is a serverless way to serve machine learning models. You can host your model in the cloud and make predictions through a REST API. If your use case requires online predictions, you’ll need to deploy your model to an endpoint. Deploying a model to an endpoint associates physical resources with the model so it can serve predictions with low latency. 

When deploying a model to an endpoint, you can specify details such as the machine type, and parameters for autoscaling. Additionally, you now have the option to create a Private Endpoint. Because your data never traverses the public internet, Private Endpoints offer security benefits in addition to reducing the time your system takes to serve the prediction when it receives the request. The overhead introduced by Private Endpoints is minimal, achieving performance nearly identical to DIY serving on GKE or GCE.  There is also no payload size limit for models deployed on the private endpoint.

Creating a Private Endpoint on Vertex AI is simple.

In the Models section of the Cloud console, select the model resource you want to deploy.

Next, select DEPLOY TO ENDPOINT

In the window on the right hand side of the console, navigate to the Access section and select Private. You’ll need to add the full name of the VPC network for which your deployment should be peered.

Note that many other managed services on GCP support VPC peering, such as Vertex Training, Cloud SQL, and Firestore. Endpoints is the latest to join that list.

What’s Next?

Now you know the basics of VPC Peering and how to use Private Endpoints on Vertex AI. If you want to learn more about configuring VPCs, check out this overview guide. And if you’re interested to learn more about how to use Vertex AI to support your ML workflow, check out this introductory video. Now it’s time for you to deploy your own ML model to a Private Endpoint for super speedy predictions!

The magnitude and direction of wind significantly impacts airport operations, and Lufthansa Group Airlines are no exception. A particularly troublesome kind is called BISE: it is a cold, dry wind that blows from the northeast to southwest in Switzerland, through the Swiss Plateau. Its effects on flight schedules can be severe, such as forcing planes to change runways, which can create a chain reaction of flight delays and possible cancellations. In Zurich Airport, in particular, BISE can potentially reduce capacity by up to 30%, leading to further flight delays and cancellations, and to millions in lost revenue for Lufthansa (as well as dissatisfaction among their passengers).

Being able to predict this kind of wind well in advance lets the Network Operations Control team schedule flight operations optimally across runways and timeslots, to minimize disruptions to the schedule. However, predicting speed and magnitude can be incredibly difficult to model and thus to predict— which is why Lufthansa reached out to Google Cloud.

Machine learning (ML) can help airports and airlines to better anticipate and manage these types of disruptive weather events. In this blog post, we’ll explore an experiment Lufthansa did together with Google Cloud and its Vertex AI Forecast service, accurately predicting BISE hours in advance, with more than 40% relative improvement in accuracy over internal heuristics, all within days instead of the months it often takes to do ML projects of this magnitude and performance.

“Being impressed with Google’s technology and prowess in the field of AI and machine learning, we were certain that my working together with their expert, to combine our technology with their domain expertise, we would achieve the best results possible,“ said Christian Most, Senior Director, Digital Operations Optimization at Lufthansa Group.

Collecting and preparing the dataset

The goal of Lufthansa and Google Cloud’s project was to forecast the BISE wind for Zurich’s Kloten Airport using deep learning-based ML approaches, then to see if the prediction surpasses internal heuristics-driven solutions and gauge the ease of use and practicality of the deep learning approach in production.

Since deep learning-based techniques require large datasets, the project relies on Meteoswiss simulation data, a dataset consisting of multiple meteorological sensor measurements collected from several weather stations across Switzerland over the past five years. By using this dataset, we obtained data on factors like wind direction, speed, pressure, temperature, humidity and more, at a 10 min resolution, along with some information about the location of the weather stations, such as altitude. These factors, which we hypothesized to be predictive of the BISE, ended up carrying valuable signals, as we would see later.

This collected data was next subjected to an extensive cleaning and feature engineering process using Vertex AI Workbench, in order to prepare the final dataset for training. The cleaning phase included steps to drop the features, or rows, that contained too many missing values, or failed statistical tests for entropy, etc. Since the direction of wind is a circular feature (between 0 and 360 degrees), this column/feature was replaced with two features: the corresponding sine and cosine embedding. The dataset was then flattened such that the columns contained all the relevant features and sensor measurements from all the weather stations at a particular 10-minute interval.

Since the target variable — i.e,. BISE — was not directly available, we engineered a proxy target variable for BISE called “tailwind speed around runway,” which above a certain threshold indicates the presence of BISE along the runway.

Forecasting wind in the Cloud

Once the dataset was ready, Lufthansa and Google Cloud evaluated several options before deciding to experiment and tune Vertex AI Forecast, Google’s AutoML-powered forecasting service, in order to achieve optimum results. Vertex Forecast is capable of the required feature engineering, neural architecture search, and hyper parameter tuning, and it is managed by Google Cloud to score in the top 2.5% in the M5 Forecasting Competition on Kaggle, in a completely automated fashion. These qualities made it an excellent choice for Lufthansa, to reduce the manual overhead of creating, deploying, and maintaining top performing deep learning models.

The raw data files were loaded from cloud storage, preprocessed on Vertex AI Workbench. Then, a training pipeline was initiated on Vertex AI Pipelines, which performed the following steps in sequence:

The .csv data file was loaded from Cloud Storage into a Vertex AI managed dataset.

A Vertex AI forecasting training job was initiated with the dataset, and it was also registered as a model in the Vertex AI Model Registry.

Upon completion, the model was evaluated on the test set, and the model’s predictions and the input features and ground truth of the test set, were stored in a user-defined table in BigQuery. Several test metrics were also available on the service and model dashboards.

One of the biggest challenges was the severe imbalance in the dataset, as measurements with BISE were very far and few in between. In order to account for this, instances where BISE occurred, as well the occurrences temporally close to them, were upweighted using weights calculated with methods including Inverse of Square Root of Number of Samples (ISNS), Effective Number of Samples (ENS), and Gaussian reweighting. The formulas for the methods are given below. These weights were supplied as separate columns in the dataset, and were iteratively used thereafter by the service as the “weight” column.

ISNS

ENS

Weighted gaussian

Results and next steps

In the above figures, the x-axis represents the forecast horizon and the Y-axis shows the respective metrics (Recall/F1-score). As shown after multiple experiments, we can see Vertex AI Forecast achieved higher recall and precision t (red bar), outperforming Lufthansa’s internal baseline heuristics, with the performance gap widening steadily as the forecast horizon extends further into the future. At the two-hour mark, our custom-configured Vertex AI Forecast model improved by 40% relative to the internal heuristics and 1700% compared to the random guess baseline. As we saw with other experiments, at a six-hour forecast horizon, the performance gap widens even more, with Vertex AI Forecast in the lead. Since forecasting BISE a few hours in advance is very beneficial to prevent flight delays for Lufthansa, this was a great solution for them.

“We are very excited to be able to not only do accurate long term forecasts for the BISE, but also that Vertex AI Forecasting makes training and deploying such models much easier and faster, allowing us to innovate rapidly to serve our customers and stakeholders in the best possible manner,” said Swiss Oliver Rueegg, Product Owner, Swiss International Airlines.

Lufthansa plans to explore productionizing this solution by integrating it into their Operations Decision Support Suite, which is used by the network controllers in the Operations Control Center in Kloten, as well as to work closely with Google’s specialists to integrate both Vertex AI Forecast and other of Google’s AI/ML offerings for their use cases.

Editor’s note: Today’s guest post comes from Indian social media platform ShareChat. Here’s the story of how they improved performance, app development, and analytics for serving regional content to millions of users using Google Cloud. 

How do you create a social network when your country has 22 major official languages and countless active regional dialects? At ShareChat, we serve more than 160 million monthly active users who share and view videos, images, GIFs, songs, and more in 15 different Indian languages. We also launched a short video platform in 2020, Moj, which already supports over 80 million monthly active users. 

Connecting with people in the language they understand

As mobile data and smartphones have become more affordable in India, we noticed a large new segment of people, many in rural areas, being welcomed onto the internet. However, many of them didn’t speak English, and when it comes to accessing content and information—language plays a significant role. Instead of joining other social media sites where English reigned supreme, new internet users chose to join language or dialect-specific Whatsapp groups where they felt more comfortable instead.

So, we set out to build a platform where people can share their opinions, document their lives, and make new friends, all in their native language. ShareChat simplifies content and people discovery by using a personalized content newsfeed to deliver language-specific content to the right audience.

Given the high-intensity data and high volume of content and traffic, we rely heavily on IT infrastructure. On top of that, a large number of our users rely on 2G networks to post, like, view, or follow each other. Our platform needs to deliver great experiences to people who are spread out across the country and different networks without any reduction in performance.

The right cloud partner to support future growth

ShareChat was born in the cloud—we already knew how to scale systems to serve a large customer base with our existing cloud provider. But like many companies, we struggled with over-provisioning compute and storage to accommodate unpredictable traffic and avoid running out of storage. With demand rising for local language content and an increase in online interactions in response to the COVID-19 crisis, we realized that we would need a more efficient way to scale dynamically and allocate resources as needed.

Google Cloud was a natural choice for us. We wanted to partner with a technology-first company that would make it easy (and cost-effective) to manage a strong technology portfolio that would allow us to build whatever we wanted. Google is at the forefront of technology innovation and provided everything we needed to build, run, and manage our applications (including creating an efficient DevOps pipeline to fix and release new features quickly). 

We had a few issues in mind at the start of discussions with the Google Cloud team, but over time, as we got information and support from them, we realized that these were the partners we wanted in our corner when it came time to tackle our most challenging problems. In the end, we decided to take our entire infrastructure to Google Cloud.

To support millions of users, we deploy and scale using Google Kubernetes Engine. While we analyze our data using a combination of managed data cloud services, such as Pub/Sub for data pipelines, BigQuery for analytics, Cloud Spanner for real-time app serving workloads, and Cloud Bigtable for less-indexed databases. We also rely on Cloud CDN to help us distribute high-quality and reliable content delivery at low latency to our users. 

We now use just half the total core consumption of our legacy environment to run ShareChat’s existing workloads.

Google Cloud delivers better outcomes at every level 

By moving to Google Cloud, we saw major benefits in several key areas: 

Zero-downtime migration for users

At the time of migration, we had over 70 terabytes of data, consisting of 220 tables—some of which were up to 14 terabytes with nearly 50 billion rows. Due to our data’s interdependencies, moving services over one at a time wasn’t an option for us. 

Even though we were migrating such large volumes of data, we didn’t want to impact any of our customers. Latency spikes for out-of-sync data might affect message delivery. For instance, if a message or notification was delayed, we didn’t want to risk a bad user experience causing someone to abandon ShareChat. 

To prepare for the move, we ran a proof-of-concept cluster for over four months to test database performance in a real-world scenario for handling more than a million queries per second. Using an open-source API gateway, we replicated our legacy data environment into Google Cloud for performance testing and capacity analysis. As soon as we were confident Google Cloud could handle the same traffic as our previous cloud environment, we were ready to execute.

Using wrappers, we were able to migrate without having to change anything in our existing application code. The entire migration of 60 million users to Google Cloud took five hours—without any data loss or downtime. Today, ShareChat has grown to 160 million users, and Google Cloud continues to give us the support we need.

Scaling globally to meet unexpected demand

We rely on real-time data to drive everything on ShareChat by tracking everything that goes on in our app—from messages and new groups to content people like or who they follow. Our users create more than a million posts per day, so it’s critical that our systems can process massive amounts of data efficiently. 

We chose to migrate to Spanner for its global consistency and secondary index. Unlike our legacy NoSQL database, we could scale without having to rethink existing tables or schema definitions and keep our data systems in sync across multiple locations. It’s also cost-effective for us—moving over 120 tables with 17 indexes into Cloud Spanner reduced our costs by 30%.

Spanner also replicates data seamlessly in multiple locations in real time, enabling us to retrieve documents if one region fails. For instance, when our traffic unexpectedly grew by 500% over just a few days, we were able to scale horizontally with zero lines of code change. We were also launching our Moj video app simultaneously, and we were able to move it to another region without a single issue. 

Simplifying development and deployment

On average, we experience about 80,000 requests per second (RPS) –nearly 7 billion RPS per day. That means daily push notifications sent out to the entire user base about daily trending topics can often result in a spike of 130,000 RPS in just a few seconds. 

Instead of over-provisioning, Google Kubernetes Engine (GKE) enables us to pre-scale for traffic spikes around scheduled events, such as holidays like Diwali, when millions of Indians send each other greetings. 

Migrating to GKE has also enabled us to adopt more agile ways of work, such as automating deployment and saving time with writing scripts. Even though we were already using container-based solutions, they lacked transparency and coverage across the entire deployment funnel. 

Kubernetes features, such as sidecar proxy, allows us to attach peripheral tasks like logging into the application without requiring us to make code changes. Kubernetes upgrades are managed by default, so we don’t have to worry about maintenance and stay focused on more valuable work. Clusters and nodes automatically upgrade to run the latest version, minimizing security risks and ensuring we always have access to the latest features.

Low latency and real-time ML predictions

Even though many of our users may be accessing ShareChat outside of metropolitan areas, it doesn’t mean they’re more patient if the app loads slowly or their messages are delayed. We strive to deliver a high-performance experience, regardless of where our users are. 

We use Cloud CDN to cache data in five Google Cloud Point of Presence (PoP) locations at the edge in India, allowing us to bring content as close as possible to people and speeding up load time. Since moving to Cloud CDN, our cache hit ratio has improved from 90% to 98.5%—meaning our cache can handle 98.5% of content requests. 

As we expand globally, we’d like to use machine learning to reach new people with content in different languages. We want to build new algorithms to process real-time datasets in regional languages and accurately predict what people want to see. Google Cloud gives us an infrastructure optimized to handle compute-intensive workloads that will be useful to us both now—and in the future.  

The confidence to build the best platform

Our current system now performs better than before we migrated, but we are continuously building new features on top of it. Google’s data cloud has provided us with an elegant ecosystem of services that allows us to build whatever we want, more easily and faster than ever before. 

Perhaps the biggest advantage of partnering with Google Cloud has been the connection we have with the engineers at Google. If we’re working to solve a specific problem statement and find a specific solution in a library or a piece of code, we have the ability to immediately connect with the team responsible for it. 

As a result, we have experienced a massive boost in our confidence. We know that we can build a really good system because we not only have a good process in place to solve problems—we have the right support behind us.