The rapidly evolving landscape of distributed computing demands efficient and scalable frameworks. Ray.io is an open-source framework to easily scale up Python applications across multiple nodes in a cluster. Ray provides a simple API for building distributed, parallelized applications, especially for deep learning applications.

Google Kubernetes Engine (GKE) is a managed container orchestration service that makes it easy to deploy and manage containerized applications. GKE provides a scalable and flexible platform that abstracts away the underlying infrastructure.

KubeRay enables Ray to be deployed on Kubernetes. You get the wonderful Pythonic unified experience delivered by Ray, and the enterprise reliability and scale of GKE managed Kubernetes. Together, they offer scalability, fault tolerance, and ease of use for building, deploying, and managing distributed applications.

In this blog post, we share a solution template to get you started easily with Ray on GKE. We discuss the components of the solution and showcase an inference example using Ray Serve for Stable Diffusion.

Overview of the Solution

In this solution template we use KubeRay, an OSS solution for managing Ray clusters on Kubernetes, as the operator for provisioning our workloads. Follow the step-by-step instructions in the README file to get started. The solution contains two groups of resources: platform-level and user-level.

Platform-level resources are expected to be deployed once for each development environment by the system administrator. These include the common infrastructure and GCP service integrations that are shared by all users.

• GKE cluster and node pool. Configurations can be changed in the main.tf file. This module deploys a GKE cluster with a GPU node pool, including required Nvidia drivers for GPUs. You can replace these with other machine types.
• Kubernetes system namespace and service accounts, along with the necessary IAM policy bindings. This allows the platform administrator to provide fine-grained user access control and quota policies for Ray cluster resources.
• KubeRay operator. The operator is responsible for watching for changes in KubeRay resources and reconciling the state of the KubeRay clusters.  
• Logging. The `logging_config` section enables logs from system components and workloads to write logs to Cloud logging.
• Monitoring. The `monitoring_config` section enables Managed Prometheus integration. This allows the deployment to automatically scrape system-level metrics and writes them to the managed metrics service.
• Workload identity. This enables your workloads to authenticate with other GCP services using Google IAM service accounts.

User-level resources are expected to be deployed once by each user in the development environment.

• KubeRay cluster. This is the actual Ray cluster that we will be used for your workloads. It is configured to use a Workload Identity pool and a IAM-binded service account that provides fine-grained access to GCP services. You can customize the Ray cluster settings by editing the kuberay-values.yaml file.

• Logging. The solution adds a side car container deployed alongside each KubeRay worker node. This uses fluentbit to forward Ray logs from the head node to Cloud logging. You can edit the fluentbit-config file to change how the logging container filters and flushes logs.
• Monitoring. This module provides a PodMonitoring resource that scrapes metrics from the user’s Ray cluster and uploads data points to Google Managed Prometheus. An optional installation for Grafana dashboard is included and can be accessed through a web browser.
• JupyterHub server. This module installs a JupyterHub notebook server in the user namespace, enabling users to interact directly with their Ray clusters.

Run a Workload on Your Ray Cluster

Let’s try running the provided example with Ray Serve to deploy Stable Diffusion. This example was originally taken from the Ray Serve documentations here. To open the example in Jupyter notebook, go to the external IP for proxy-public in your browser (instructions to get the IP). And then click on File -> Open from URL, and input the raw URL of the notebook to open it.

Since the notebook runs in the same Kubernetes cluster as the Ray cluster, it is able to talk directly to the latter using its cluster-internal service endpoint – thus there is no need to expose the Ray cluster to public internet traffic. For production workloads, you should secure your endpoints with GCP account credentials. Google Cloud Identity Aware Proxy (IAP) can be used to enable fine-grained access control to user resources, such as our Ray cluster, to protect your GCP resources from unnecessary exposure. A full tutorial on how to enable IAP on your GKE cluster can be found here.

The notebook contains code for deploying a pre-trained model to a live endpoint. The last cell makes a call to the created service endpoint:prompt = “a cute cat is dancing on the grass.”input = “%20″.join(prompt.split(” “))resp = requests.get(f”http://example-cluster-kuberay-head-svc:8000/imagine?prompt={input}”)with open(“output.png”, ‘wb’) as f:   f.write(resp.content)

Executing the notebook will generate a file with a unique picture of a cute cat. Here is an example we got:

Congratulations! You have now deployed a large model for image generation on GKE.

Logging and Monitoring

As mentioned earlier, this solution enables logging and monitoring automatically. Let’s find those logs.

In your Cloud Console, open up Logging -> Log Explorer. In the query text box, enter the following:resource.type=”k8s_container”resource.labels.cluster_name=%CLUSTER_NAME%resource.labels.pod_name=%RAY_HEAD_POD_NAME%resource.labels.container_name=”fluentbit”

You should see the Ray logs from your cluster forwarded here.

To see your monitoring metrics, go to Metrics Explorer in the Cloud Console. Under the menu for “Target”, select “Prometheus Target” and then “Ray”. Select the metric that you want to see, for instance `prometheus/ray_component_cpu_percentage/gauge`:

The deployment also comes with a Grafana deployment. Follow this guide to open it up and view your Ray cluster’s metrics.

Conclusion

The combination of Ray and GKE offers a simple and powerful solution for building, deploying, and managing distributed applications. Ray’s simplicity makes it an attractive choice for data and model developers while GKE’s scalability and reliability is the defacto choice for enterprise platforms. The solution template presented in this blog post offers a convenient way to get started quickly with KubeRay, the recommended approach to deploy Ray on GKE.

If you have any questions for building Ray on Kubernetes and GKE, you can contact us directly at ray-on-gke@google.com or comment in GitHub. Learn more about building AI Platforms with GKE by visiting our User Guide.

Editor’s note: Today’s guest post comes from Indian social media platform ShareChat. Here’s the story of how they improved performance, app development, and analytics for serving regional content to millions of users using Google Cloud. 

How do you create a social network when your country has 22 major official languages and countless active regional dialects? At ShareChat, we serve more than 160 million monthly active users who share and view videos, images, GIFs, songs, and more in 15 different Indian languages. We also launched a short video platform in 2020, Moj, which already supports over 80 million monthly active users. 

Connecting with people in the language they understand

As mobile data and smartphones have become more affordable in India, we noticed a large new segment of people, many in rural areas, being welcomed onto the internet. However, many of them didn’t speak English, and when it comes to accessing content and information—language plays a significant role. Instead of joining other social media sites where English reigned supreme, new internet users chose to join language or dialect-specific Whatsapp groups where they felt more comfortable instead.

So, we set out to build a platform where people can share their opinions, document their lives, and make new friends, all in their native language. ShareChat simplifies content and people discovery by using a personalized content newsfeed to deliver language-specific content to the right audience.

Given the high-intensity data and high volume of content and traffic, we rely heavily on IT infrastructure. On top of that, a large number of our users rely on 2G networks to post, like, view, or follow each other. Our platform needs to deliver great experiences to people who are spread out across the country and different networks without any reduction in performance.

The right cloud partner to support future growth

ShareChat was born in the cloud—we already knew how to scale systems to serve a large customer base with our existing cloud provider. But like many companies, we struggled with over-provisioning compute and storage to accommodate unpredictable traffic and avoid running out of storage. With demand rising for local language content and an increase in online interactions in response to the COVID-19 crisis, we realized that we would need a more efficient way to scale dynamically and allocate resources as needed.

Google Cloud was a natural choice for us. We wanted to partner with a technology-first company that would make it easy (and cost-effective) to manage a strong technology portfolio that would allow us to build whatever we wanted. Google is at the forefront of technology innovation and provided everything we needed to build, run, and manage our applications (including creating an efficient DevOps pipeline to fix and release new features quickly). 

We had a few issues in mind at the start of discussions with the Google Cloud team, but over time, as we got information and support from them, we realized that these were the partners we wanted in our corner when it came time to tackle our most challenging problems. In the end, we decided to take our entire infrastructure to Google Cloud.

To support millions of users, we deploy and scale using Google Kubernetes Engine. While we analyze our data using a combination of managed data cloud services, such as Pub/Sub for data pipelines, BigQuery for analytics, Cloud Spanner for real-time app serving workloads, and Cloud Bigtable for less-indexed databases. We also rely on Cloud CDN to help us distribute high-quality and reliable content delivery at low latency to our users. 

We now use just half the total core consumption of our legacy environment to run ShareChat’s existing workloads.

Google Cloud delivers better outcomes at every level 

By moving to Google Cloud, we saw major benefits in several key areas: 

Zero-downtime migration for users

At the time of migration, we had over 70 terabytes of data, consisting of 220 tables—some of which were up to 14 terabytes with nearly 50 billion rows. Due to our data’s interdependencies, moving services over one at a time wasn’t an option for us. 

Even though we were migrating such large volumes of data, we didn’t want to impact any of our customers. Latency spikes for out-of-sync data might affect message delivery. For instance, if a message or notification was delayed, we didn’t want to risk a bad user experience causing someone to abandon ShareChat. 

To prepare for the move, we ran a proof-of-concept cluster for over four months to test database performance in a real-world scenario for handling more than a million queries per second. Using an open-source API gateway, we replicated our legacy data environment into Google Cloud for performance testing and capacity analysis. As soon as we were confident Google Cloud could handle the same traffic as our previous cloud environment, we were ready to execute.

Using wrappers, we were able to migrate without having to change anything in our existing application code. The entire migration of 60 million users to Google Cloud took five hours—without any data loss or downtime. Today, ShareChat has grown to 160 million users, and Google Cloud continues to give us the support we need.

Scaling globally to meet unexpected demand

We rely on real-time data to drive everything on ShareChat by tracking everything that goes on in our app—from messages and new groups to content people like or who they follow. Our users create more than a million posts per day, so it’s critical that our systems can process massive amounts of data efficiently. 

We chose to migrate to Spanner for its global consistency and secondary index. Unlike our legacy NoSQL database, we could scale without having to rethink existing tables or schema definitions and keep our data systems in sync across multiple locations. It’s also cost-effective for us—moving over 120 tables with 17 indexes into Cloud Spanner reduced our costs by 30%.

Spanner also replicates data seamlessly in multiple locations in real time, enabling us to retrieve documents if one region fails. For instance, when our traffic unexpectedly grew by 500% over just a few days, we were able to scale horizontally with zero lines of code change. We were also launching our Moj video app simultaneously, and we were able to move it to another region without a single issue. 

Simplifying development and deployment

On average, we experience about 80,000 requests per second (RPS) –nearly 7 billion RPS per day. That means daily push notifications sent out to the entire user base about daily trending topics can often result in a spike of 130,000 RPS in just a few seconds. 

Instead of over-provisioning, Google Kubernetes Engine (GKE) enables us to pre-scale for traffic spikes around scheduled events, such as holidays like Diwali, when millions of Indians send each other greetings. 

Migrating to GKE has also enabled us to adopt more agile ways of work, such as automating deployment and saving time with writing scripts. Even though we were already using container-based solutions, they lacked transparency and coverage across the entire deployment funnel. 

Kubernetes features, such as sidecar proxy, allows us to attach peripheral tasks like logging into the application without requiring us to make code changes. Kubernetes upgrades are managed by default, so we don’t have to worry about maintenance and stay focused on more valuable work. Clusters and nodes automatically upgrade to run the latest version, minimizing security risks and ensuring we always have access to the latest features.

Low latency and real-time ML predictions

Even though many of our users may be accessing ShareChat outside of metropolitan areas, it doesn’t mean they’re more patient if the app loads slowly or their messages are delayed. We strive to deliver a high-performance experience, regardless of where our users are. 

We use Cloud CDN to cache data in five Google Cloud Point of Presence (PoP) locations at the edge in India, allowing us to bring content as close as possible to people and speeding up load time. Since moving to Cloud CDN, our cache hit ratio has improved from 90% to 98.5%—meaning our cache can handle 98.5% of content requests. 

As we expand globally, we’d like to use machine learning to reach new people with content in different languages. We want to build new algorithms to process real-time datasets in regional languages and accurately predict what people want to see. Google Cloud gives us an infrastructure optimized to handle compute-intensive workloads that will be useful to us both now—and in the future.  

The confidence to build the best platform

Our current system now performs better than before we migrated, but we are continuously building new features on top of it. Google’s data cloud has provided us with an elegant ecosystem of services that allows us to build whatever we want, more easily and faster than ever before. 

Perhaps the biggest advantage of partnering with Google Cloud has been the connection we have with the engineers at Google. If we’re working to solve a specific problem statement and find a specific solution in a library or a piece of code, we have the ability to immediately connect with the team responsible for it. 

As a result, we have experienced a massive boost in our confidence. We know that we can build a really good system because we not only have a good process in place to solve problems—we have the right support behind us.

Enterprise application teams need to manage more than just containers in their software supply chain. That’s why we created Artifact Registry, a fully-managed service with support for both container images and non-container artifacts.

Artifact Registry improves and extends upon the existing capabilities of Container Registry, such as customer-managed encryption keys, VPC-SC support, Pub/Sub notifications, and more, providing a foundation for major upgrades in security, scalability and control. While Container Registry is still available and will continue to be supported as a Google Enterprise API, going forward new features will only be available in Artifact Registry, and Container Registry will only receive critical security fixes.

Below, we’ll highlight the key improvements Artifact Registry provides over Container Registry, as well as the steps to start using it today.

A unified control plane for container, OS and language repositories

Artifact Registry includes more than just container images: as a developer, you can store multiple artifact formats, including OS packages for Debian and RPM, as well as language packages for popular languages like Python, Java, and Node. In addition, you can manage them all from a single, unified interface. 

A more granular permission model with Cloud IAM

Artifact Registry comes with fine-grained access control via Cloud IAM. Unlike Container Registry, this allows you to control access on a per-repository basis, rather than all images stored in a project. This enables you to scope permissions as granularly as possible, for example to specific regions or environments as necessary.

Repositories in the region of your choice

Artifact Registry supports the creation of regional repositories, which allows you to put your artifacts and data directly in the location that they’ll be used, allowing for higher availability and speed. In Container Registry, you’re limited to “multi-regions”: for example, the closest multi-region for Australia is Asia. However, with Artifact Registry’s regional support, you can create a repository directly in the Sydney data center.

A pricing model that respects your region

While Artifact Registry’s pricing is still based on a combination of network egress and storage usage, support for regional repositories means that you can choose in what region to host your container repositories. Although per unit storage costs are higher for Artifact Registry, optimizing the locations of your repositories to be hosted in the same region where they are used can result in cost savings, because any network traffic within the same region is not considered egress and is thus free.

Part of a secure supply chain

Artifact Registry was designed from the ground up to integrate into our suite of secure supply chain products. This means that it can optionally use Container Analysis to scan your container images for vulnerabilities as they’re uploaded to Artifact Registry, and works directly with Binary Authorization to secure your deployments.

We’re here to help you migrate

If you already use Container Registry, you can take advantage of all the current and upcoming features of container image storage with Artifact Registry by migrating to it. To help, we’ve prepared the following guides:

• Transitioning from Container Registry provides an overview of how to use Artifact Registry instead of Container Registry in a backwards-compatible way
• Copying images from Container Registry guide you to move container images from an existing repository to an Artifact Registry repository

If you’re currently hosting your container images with a third party, you can begin using Artifact Registry directly, by following the instructions in our guide, Migrating containers from a third-party registry, which shows you how to avoid rate limits on image pulls or third-party outages which can disrupt your builds and deployments.

And if you’re just getting started storing container images, you can begin using Artifact Registry as your image repository right away. To learn how, check out Artifact Registry quickstart for Docker, a guide to using Artifact Registry as a single location for managing private packages and Docker container images.

Join our community 

Our Artifact Registry communities are also great resources to help answer your questions and for guidance on best practices: 

• Ask questions on Stack Overflow using the google-artifact-registry tag
• Visit the Google Cloud Slack community and ask a question in the #artifact-registry channel. If you haven’t already joined the Slack community, use this form to sign up.