Deploying Ray on GKE: Distributed Computing Made Easy
958
Of your peers have already read this article.
3:30 Minutes
The most insightful time you'll spend today!
The rapidly evolving landscape of distributed computing demands efficient and scalable frameworks. Ray.io is an open-source framework to easily scale up Python applications across multiple nodes in a cluster. Ray provides a simple API for building distributed, parallelized applications, especially for deep learning applications.
Google Kubernetes Engine (GKE) is a managed container orchestration service that makes it easy to deploy and manage containerized applications. GKE provides a scalable and flexible platform that abstracts away the underlying infrastructure.
KubeRay enables Ray to be deployed on Kubernetes. You get the wonderful Pythonic unified experience delivered by Ray, and the enterprise reliability and scale of GKE managed Kubernetes. Together, they offer scalability, fault tolerance, and ease of use for building, deploying, and managing distributed applications.
In this blog post, we share a solution template to get you started easily with Ray on GKE. We discuss the components of the solution and showcase an inference example using Ray Serve for Stable Diffusion.
Overview of the Solution
In this solution template we use KubeRay, an OSS solution for managing Ray clusters on Kubernetes, as the operator for provisioning our workloads. Follow the step-by-step instructions in the README file to get started. The solution contains two groups of resources: platform-level and user-level.
Platform-level resources are expected to be deployed once for each development environment by the system administrator. These include the common infrastructure and GCP service integrations that are shared by all users.
- GKE cluster and node pool. Configurations can be changed in the main.tf file. This module deploys a GKE cluster with a GPU node pool, including required Nvidia drivers for GPUs. You can replace these with other machine types.
- Kubernetes system namespace and service accounts, along with the necessary IAM policy bindings. This allows the platform administrator to provide fine-grained user access control and quota policies for Ray cluster resources.
- KubeRay operator. The operator is responsible for watching for changes in KubeRay resources and reconciling the state of the KubeRay clusters.
- Logging. The `logging_config` section enables logs from system components and workloads to write logs to Cloud logging.
- Monitoring. The `monitoring_config` section enables Managed Prometheus integration. This allows the deployment to automatically scrape system-level metrics and writes them to the managed metrics service.
- Workload identity. This enables your workloads to authenticate with other GCP services using Google IAM service accounts.
User-level resources are expected to be deployed once by each user in the development environment.
- KubeRay cluster. This is the actual Ray cluster that we will be used for your workloads. It is configured to use a Workload Identity pool and a IAM-binded service account that provides fine-grained access to GCP services. You can customize the Ray cluster settings by editing the kuberay-values.yaml file.
- Logging. The solution adds a side car container deployed alongside each KubeRay worker node. This uses fluentbit to forward Ray logs from the head node to Cloud logging. You can edit the fluentbit-config file to change how the logging container filters and flushes logs.
- Monitoring. This module provides a PodMonitoring resource that scrapes metrics from the user’s Ray cluster and uploads data points to Google Managed Prometheus. An optional installation for Grafana dashboard is included and can be accessed through a web browser.
- JupyterHub server. This module installs a JupyterHub notebook server in the user namespace, enabling users to interact directly with their Ray clusters.
Run a Workload on Your Ray Cluster
Let’s try running the provided example with Ray Serve to deploy Stable Diffusion. This example was originally taken from the Ray Serve documentations here. To open the example in Jupyter notebook, go to the external IP for proxy-public in your browser (instructions to get the IP). And then click on File -> Open from URL, and input the raw URL of the notebook to open it.
Since the notebook runs in the same Kubernetes cluster as the Ray cluster, it is able to talk directly to the latter using its cluster-internal service endpoint – thus there is no need to expose the Ray cluster to public internet traffic. For production workloads, you should secure your endpoints with GCP account credentials. Google Cloud Identity Aware Proxy (IAP) can be used to enable fine-grained access control to user resources, such as our Ray cluster, to protect your GCP resources from unnecessary exposure. A full tutorial on how to enable IAP on your GKE cluster can be found here.
The notebook contains code for deploying a pre-trained model to a live endpoint. The last cell makes a call to the created service endpoint:prompt = “a cute cat is dancing on the grass.”input = “%20″.join(prompt.split(” “))resp = requests.get(f”http://example-cluster-kuberay-head-svc:8000/imagine?prompt={input}”)with open(“output.png”, ‘wb’) as f: f.write(resp.content)
Executing the notebook will generate a file with a unique picture of a cute cat. Here is an example we got:
Congratulations! You have now deployed a large model for image generation on GKE.
Logging and Monitoring
As mentioned earlier, this solution enables logging and monitoring automatically. Let’s find those logs.
In your Cloud Console, open up Logging -> Log Explorer. In the query text box, enter the following:resource.type=”k8s_container”resource.labels.cluster_name=%CLUSTER_NAME%resource.labels.pod_name=%RAY_HEAD_POD_NAME%resource.labels.container_name=”fluentbit”
You should see the Ray logs from your cluster forwarded here.
To see your monitoring metrics, go to Metrics Explorer in the Cloud Console. Under the menu for “Target”, select “Prometheus Target” and then “Ray”. Select the metric that you want to see, for instance `prometheus/ray_component_cpu_percentage/gauge`:
The deployment also comes with a Grafana deployment. Follow this guide to open it up and view your Ray cluster’s metrics.
Conclusion
The combination of Ray and GKE offers a simple and powerful solution for building, deploying, and managing distributed applications. Ray’s simplicity makes it an attractive choice for data and model developers while GKE’s scalability and reliability is the defacto choice for enterprise platforms. The solution template presented in this blog post offers a convenient way to get started quickly with KubeRay, the recommended approach to deploy Ray on GKE.
If you have any questions for building Ray on Kubernetes and GKE, you can contact us directly at ray-on-gke@google.com or comment in GitHub. Learn more about building AI Platforms with GKE by visiting our User Guide.
4416
Of your peers have already downloaded this article
7:00 Minutes
The most insightful time you'll spend today!
Aspiration is easy, but execution is hard. Few concepts illustrate this as well as digital transformations.
Most businesses today are at some phase of their digital transformation journey. While some are evaluating strategies and solutions, many companies have already embarked on full-scale project implementations. However, one thing is clear. While most business and IT leaders aspire to revamp their businesses, executing a well-defined digital transformation strategy is extremely complicated. The hurdles are both cultural and technological.
This e-Book breaks down the process of creating a digital transformation strategy into ten core dimensions: platform, APIs, outside-in, ecosystem, leadership, funding, metrics, software development lifecycle, talent, and self-service. Read on to understand how companies like Walgreens, Ticketmaster, Magazine Luiza, AccuWeather, and Pitney Bowes, are successfully charting their digital transformation journey with the help of Apigee Compass and evolving their businesses.
Beyond Traditional Learning: AI-based Online Learning Platform and Google Cloud Solutions Push Learners to Get Ahead
10312
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
The combination of a vital need for IT experts among businesses and a digital skills gap is making lifelong learning increasingly critical. Beyond professional development, learning new skills offers additional rewards from building peer connections to boosting your creativity. That’s why in 2020 Krishna Deepak Nallamilli and I launched KIMO.ai to reimagine how people approach learning, especially in developing markets. Our team is building the artificial intelligence needed to generate individual learning paths through a wide range of quality digital learning content.
Google Cloud and its Startup Program have been instrumental in connecting our team with the tools, people, processes, and best practices to grow our business.
Existing learning platforms lack engagement
Outside of traditional education settings, massive open online learning courses (MOOCs)—often modeled after university courses—can provide a flexible and affordable way to upskill or reskill. But the vast majority of people who participate in MOOC programs fail to complete courses. Based on our research, the challenge with existing learning platforms is a lack of engagement, primarily caused by limited direction on which skills to learn, whether AI, fintech, blockchain, or other in-demand disciplines.
We’ve also received feedback that many corporate learning management systems–developed as online training systems to upskill employees–tend to be poorly designed and time-consuming to use.
Overall, a significant challenge with most existing learning platforms is that they’re generic. For example, suppose you’re interested in learning about AI. In that case, you need AI-related coursework that applies to your industry and the job you want because AI in medicine is vastly different from AI in financial services. Today’s online learning options typically take a one-size-fits-all approach and fail to capture the nuances of what learners really need to get ahead.
Building a future-proof learning platform
The commitment to highly personalized, accessible learning inspired KIMO.ai, a platform that we believe is the future of education. Depending on your goals, current skills, location, and other factors, our AI-based platform will identify which coursework (and where to find those classes) to build the skills you need. The more personalized, relevant learning recommendations even take into account people’s preferences for podcasts, MOOCs, books, articles, videos, courses, publications, and more.
In a mix of cooperation and competition we call “coopetition,” KIMO.ai will regularly recommend courses from other established online learning systems if, based on our automated assessment, it’s the best option for a learner. There’s also the option to access free content only.
Google cultural alignment fosters trust
Our platform started with one developer exploring NLP models and Google APIs. As we’ve grown our team and launched our beta to 110,000 users in developing markets, we discovered there is a lot of interest in our platform, and we believe we can make a significant impact. In feedback forums, we also learned that we need to focus our efforts on the mobile experience to improve engagement since 99% of the beta testers use mobile devices.
Beyond our team’s high level of trust in Google Cloud solutions, our team also appreciates the cultural alignment with Google. We value Google’s developer-centric approach and rely on tools like Dataflow for batch data processing and Cloud TPU to reliably run machine learning models with AI services on Google Cloud. We also build all of our deployments on Google Kubernetes Engine (GKE), which makes it easy to manage all our containerized workloads
On the front end, Google App Engine makes it easy to deploy apps and experiment, and it integrates seamlessly with Firebase for authentication and more. BigQuery is our serverless data warehouse that efficiently scales to support the millions of articles, videos, and other learning resources we need to analyze to provide the targeted coursework recommendations our learners require.
As we grow our business having a network of trusted advisors is also extremely valuable. By working closely with DoIT International, the 2020 Google Cloud Global Reseller Partner of the Year, our team has access to their cloud, Kubernetes, and machine learning expertise. DoIT has already helped us quickly resolve IT issues and create analytics dashboards that give us insights to continually enhance our services.
Building for a growing industry
The dynamic edtech market is growing rapidly and estimated to become an $11B industry by 2025. We’re proud to be part of the next wave of personalized education that has the potential to empower people in developing markets and beyond to grow their skills with coursework tailored to their exact needs and how they like to learn. This year, we will deliver our platform to at least 400,000 more people. We’re excited to see how they use it and where it takes them.
If you want to learn more about how Google Cloud can help your startup, visit our Startup Program application page here to get more information about our program, and sign up for our communications to get a look at our community activities, digital events, special offers, and more.
5677
Of your peers have already downloaded this article
5:30 Minutes
The most insightful time you'll spend today!
Modernizing apps on the cloud isn’t an “all or nothing” decision. Businesses want the option to modernize on-premises or choose multi-cloud solutions that meet their needs. That’s why we created a new solution for running apps anywhere – simply, flexibly, and securely. Embracing open standards, Anthos lets you run your applications, unmodified, on existing on-prem hardware investments or in the public cloud. So that you write once and deploy anywhere.
Download this report and find out how to:
- Decouple infrastructure and applications with containers and Kubernetes
- Decouple cloud teams from one another so they can work independently
- Meet the challenges of microservice management using service mesh
- Implement a zero-trust security model to enforce more granular controls while maintaining a consistent user experience
Project to Platform: Financing Digital Transformation
4202
Of your peers have already read this article.
3.45 Minutes
The most insightful time you'll spend today!
Financing digital transformation poses a significant problem for many enterprises we work with. It’s not because the C-suite doesn’t support it: the need to become more agile, digital, and data-driven is nearly ubiquitous in executive public pronouncements. But most large organizations look at IT investment in a way that’s ineffective for a successful digital transformation effort.
A varied combination of factors affect an organization’s success in the digital ecosystem, and a lack of alignment between IT and business leaders can hamper even the most digitally sophisticated organizations. Empirical research by the Apigee Institute shows clear evidence that a solid digital transformation effort, aligned across business and marketing, whose progress is carefully tied to core enterprise metrics, must be supported by an investment strategy that uses a “real options” portfolio approach.
Let’s explore ways to think about funding digital transformation.
Planning for the Unimaginable
Common IT investment strategies are focused on straight-line net present value (NPV) business case calculations, which emphasize linear solutions and impacts. These methods ignore the value of enabling a range of options and potential future use cases, which an API program, with its reusable, highly consumable digital assets, makes available for the enterprise. In addition, these methods often ignore the value of the new categories of business data generated by such programs, as well as business capabilities that were impossible to imagine before the capabilities came online.
One media company we’re working with was astounded by a project that now makes key sales provisioning available to large distribution partners within two months, when previous estimates predicted two years and many millions of dollars, each.
More importantly, that functionality will now be reused across all of their numerous current partners and sales channels, and is enabling exciting new partnerships that were too costly before. This functionality was an unexpected by-product of their original API effort—it wasn’t even the main focus of the initial project.
There was no way that the impact of this capability or its implications on new business could have been predicted at the outset, and no way to formally model the outsized impact this functionality will have on the enterprise. Any attempts to do so would have been deemed ludicrous.
Breaking from the Status Quo
So how do you fund an API-driven digital transformation, which by nature emphasizes speed and optionality to unlock new ways of doing business, instead of the business-as-usual single-use, point-to-point integrations that perpetuate status quo thinking? For a world where the latter is the dominant paradigm and you don’t have a clear C-level mandate, here are three suggestions.
1. Enable your web/e-commerce teams. Many such teams find their responsibilities expanded to include mobile, kiosk, and many other digital initiatives, while being trapped in the traditional web server, do-it-yourself paradigm. These teams are actually quite open to technology, and often harbour weekend hackers who love to explore the new capabilities that are expanding the digital ecosystem at light speed. Tapping this passion by funding new approaches and quickly realizing immediate projects can create first the technical, and then the business proof points that resonate with a strategy team. These capabilities must be nurtured to move out of the lab into production, where real results are created.
2. Focus on partnerships. Opening and accelerating your relationships with partners via APIs (read more about partner API initiatives here) is high-impact and relatively easy to model. This business area can prove out an API platform, and in our experience creates clear business value that can support future justifications for other business units. What’s best about these wins is that sales operations groups are typically very tactical and demanding, so when they see the value, these groups will send a message to the rest of the organization that APIs have business value. Further, your enterprise’s core partnerships have the focus of executive attention. Adding value here will elevate the stature of the API program in the C-suite.
3. Work on your CFO/financial organization. Using your successes in e-commerce and partnerships, you can propose a more robust investment using a real options approach that leads to treatment of APIs as products. This path is predicated on what research has proven to work: the adoption of a digital KPI target, coupled with a portfolio of programs whose net result is to achieve the target KPI.
Building Toward the Big Vision
But what about the visions of ground-breaking business models, ubiquitous connectivity, and enablement of an ever-shifting set of opportunities on big platforms and with unlimited partnership possibilities? Yes, these are still the endpoints to work toward, and fundamentally comprise the strategic reasons to embark on a digital transformation. If your enterprise has the C-Level leadership (in the form of a CDO or CDO-like figure), supported by a well-funded, sophisticated investment approach, by all means, build that new world now.
However, should that vision be shared only within your business unit, a forward-thinking IT group, or a dedicated innovation team, the most important thing to do is to build your team’s capability—and your program’s credibility—while delivering value along the way. The approach we suggest here has been proven in many of the organizations we work with; once your organization begins working faster and at lower costs by orders of magnitude, you’ll have created enough leverage to change the way your organization invests in its API program.
Cloud IoT Core Helps Businesses Leverage their IoT Data to Build a Competitive Edge
7082
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
The ability to gain real-time insights from IoT data can redefine competitiveness for businesses. Intelligence allows connected devices and assets to interact efficiently with applications and with human beings in an intuitive and non-disruptive way. After your IoT project is up and running, many devices will be producing lots of data. You need an efficient, scalable, affordable way to both manage those devices and handle all that information.
IoT Core is a fully managed service for managing IoT devices. It supports registration, authentication, and authorization inside the Google Cloud resource hierarchy as well as device metadata stored in the cloud, and the ability to send device configuration from other GCP or third-party services to devices.
Main components
The main components of Cloud IoT Core are the device manager and the protocol bridges:
- The device manager registers devices with the service, so you can then monitor and configure them. It provides:
- Device identity management
- Support for configuring, updating, and controlling individual devices
- Role-level access control
- Console and APIs for device deployment and monitoring
- Two protocol bridges (MQTT and HTTP) can be used by devices to connect to Google Cloud Platform for:
- Bi-directional messaging
- Automatic load balancing
- Global data access with Pub/Sub
How does Cloud IoT Core work?
Device telemetry data is forwarded to a Cloud Pub/Sub topic, which can then be used to trigger Cloud Functions as well as other third-party apps to consume the data. You can also perform streaming analysis with Dataflow or custom analysis with your own subscribers.
Cloud IoT Core supports direct device connections as well as gateway-based architectures. In both cases the real time state of the device and the operational data is ingested into Cloud IoT Core and the key and certificates at the edge are also managed by Cloud IoT Core. From Pub/Sub the raw input is fed into Dataflow for transformation, and the cleaned output is populated in Cloud Bigtable for real-time monitoring or BigQuery for warehousing and machine learning. From BigQuery the data can be used for visualization in Looker or Data Studio and it can be used in Vertex AI for creating machine learning models. The models created can be deployed at the edge using Edge Manager (in experimental phase). Device configuration updates or device commands can be triggered by Cloud Functions or Dataflow to Cloud IoT Core, which then updates the device.
Design principles of Cloud IoT Core
As a managed service to securely connect, manage, and ingest data from global device fleets, Cloud IoT COre is designed to be:
- Flexible, providing easy provisioning of device identities and enabling devices to access most of Google Cloud
- IThe industry leader in IoT scalability and performance
- Interoperable, with supports for the most common industry-standard IoT protocols
Use cases
IoT use cases range across numerous industries. Some typical examples include:
- Asset tracking, visual inspection, and quality control in retail, automotive, industrial, supply chain and logistics
- Remote monitoring and predictive maintenance in oil & gas, utilities, manufacturing, and transportation
- Connected homes and consumer technologies.
- Vision intelligence in retail, security, manufacturing, and industrial sectors
- Smart living in commercial, residential, and smart spaces
- Smart factories with predictive maintenance and real-time plant floor analytics
For a more in-depth look into Cloud IoT Core check out the documentation.
https://youtube.com/watch?v=76v16P-Wqe4%3Fenablejsapi%3D1%26
For more #GCPSketchnote, follow the GitHub repo. For similar cloud content follow me on Twitter @pvergadia and keep an eye out on thecloudgirl.dev.
More Relevant Stories for Your Company
The Future of Workloads: Google Cloud’s Purpose-Built Infrastructure Evolution
For far too long, cloud infrastructure has focused on raw speeds and feeds of building blocks such as VMs, containers, networks, and storage. Today, Moore’s law is slowing, and the burden of picking the right combination of infrastructure components increasingly falls on IT. At Google Cloud we are committed to
Harnessing the Power of AI with Google Cloud: What Every IT Pro Needs to Know
As an IT architect or IT professional, you are essential to the success of your organization, responsible for designing, building, and maintaining the IT systems that your businesses and governments rely on. As if that wasn’t enough, you’ve also got to constantly learn and adapt to keep up with the
Transforming Media Industry: Three Strategies for Media Leaders to Leverage Generative AI
The digital era turned the traditional formula for media and entertainment success on its head, ushering in new technologies that have changed how content is produced, distributed, experienced, and monetized. Audiences have more choice, flexibility, and power over what they consume, and today’s media companies have to embrace ongoing transformation
Google Products Helps HMH’s Healthcare Staff Work from Anywhere Efficiently and Securely!
Hackensack Meridian Health (HMH) executive Mark Eimer explains how an ambitiously-timed rollout of a comprehensive suite of Google products helped the entire organization—from doctors to IT staff—achieve better security, cultivate a more equitable work environment, and ultimately, improve patient outcomes. How does a recently merged, 17-hospital healthcare system fast-track a platform migration
