Traditionally, dedicated game servers for real time multiplayer games have used bespoke UDP protocols for communication and synchronization of gameplay among the players within a game. This communication is most often bundled into monolithic game servers and clients, pairing the technical functionality of communication protocols, such as custom network physics synchronisation, security, access control, telemetry and metrics, with the extremely high computational requirements of physics simulations, AI computation and more.

Developed in collaboration with Embark Studios, Quilkin is a UDP proxy, tailor-made for high performance real-time multiplayer games. Its aim is twofold:

1. Pull common functionality, such as security, access control, telemetry and metrics out of monolithic dedicated game servers and clients.
2. Provide this common functionality in a composable and configurable way, such that it can be reused across a wide set of multiplayer games.

This reusable foundation then allows game developers to spend more of their time focusing on building the game-specific aspects of building their multiplayer communication protocols, rather than these common aspects.

Challenges with multiplayer Game Server communication

In fast-paced, multiplayer games, the full simulation of a session of gameplay generally occurs within the memory of a monolithic dedicated game server, whose responsibility covers everything from network physics and AI simulation to communications from client back to server and more.

Since the entire state of the game is memory resident, each client connects directly to the dedicated game server the player is playing on, which presents several challenges:

• Each dedicated game server is a single point of failure. If it goes down, then the whole game session (or sometimes multiple sessions) fails. This makes it a target for malicious actors.
• The IP and port of connection to the game server is public, and exposed to the game client, making it easy to discover and target.
• Multiple aspects of game server simulation and network communication are tightly coupled in the same process, making reuse and modularity more difficult, and expanding risk of performance issues.

If we look at both web and mobile technologies over the past several years, some of these challenges start to look very familiar. Thankfully, one of the solutions to help drive dedicated server workloads to more redundant and distributed orchestration is the utilisation of traffic proxies!

By using a proxy for multiplayer UDP traffic, in front of our dedicated games servers within a low latency network such as what is available on Google Cloud, we can address these key challenges as follows:

• Greater reliability. Proxies provide redundant points of communication entry. UDP packets can be sent to any number of proxies and routed to the dedicated game server. While a dedicated game server will still generally be a single point of failure, proxies improve redundancy and potential failover at the communication layer.
• Greater security. The IP and port of the dedicated game server is no longer public. Game clients may only have visibility into a subset of the proxy pool, limiting a potential attack surface.
• Greater scalability. We start to break apart the single process, as we can move aspects of the communication protocol, metrics, communication security and access control into the proxy. This removes the non-game specific computation out of your game server’s processing loop.

As a result, the entire system is now more resilient as proxies can be scaled independently, not only for performance reasons but also to distribute load in case of malicious actors.

Introducing Quilkin: The UDP proxy for Game Servers

Embark Studios and Google Cloud came together and built Quilkin, to provide a standard, open source solution.  Based out of Stockholm, Embark Studios is a (relatively) new studio made up of seasoned industry veterans.  They were the perfect collaboration partner to create Quilkin with, given their team’s experience with large scale real time multiplayer games.  

Quilkin is an open-source, non-transparent UDP proxy specifically designed for use with large scale multiplayer dedicated game server deployments, to ensure security, access control, telemetry data, metrics and more.

Quilkin is designed to be used behind game clients as well as in front of dedicated game servers, and offers the following major benefits:

• Obfuscation. Non-transparent proxying of UDP data, making the internal state of your game architecture less visible to bad actors.
• Out of the box metrics. For UDP packet traffic and communication.
• Visibility. A composable set of processing filters that can be applied for routing, access control, rate limiting, and more.
• Flexibility. Ability to to be utilised as a standalone binary, with no client/server changes required or as a Rust library, depending on how deep an integration you wish for your system and/or custom processing Filters you wish to build.
• Compatibility. Can be integrated with existing C/C++ code bases via Rust FFI, if required.
• Onboarding. Multiple integration patterns, allowing you to choose the level of integration that makes sense for your architecture and existing platform.

Until now, these sorts of capabilities are only available to large game studios with resources to build their own proprietary technology. 

We think leveling the playing field for everyone in the games industry is an important and worthy endeavor. That’s why we collaborated with Google Cloud and initiated this project together.

At Embark, we believe open source is the future of the games industry and that open, cross-company collaboration is the way forward, so that all studios, regardless of size, are able to achieve the same level of technical capabilities. —Luna Duclos, Tech Lead, Embark Studios

Google Cloud is excited to announce Quilkin as the latest entry in our portfolio of open-source solutions for gaming.  Quilkin complements our existing OSS solutions including Agones for game servers, Open Match for matchmaking, and Open Saves for persistence.  These are designed to work together as an open and integrated ecosystem for gaming.  We’re proud to include Embark Studios as our latest open source collaborator for gaming along with Ubisoft, Unity, and 2K Games.  Google Cloud will continue to work closely with our partners in industry and the community to offer planet-scale solutions to power the world’s largest games. —Rob Martin, Chief Architect, Google Cloud for Games

Getting started with Quilkin

While Quilkin can support more advanced deployment scenarios like above, the easiest way to get started with Quilkin is to deploy it as a sidecar to your existing dedicated game server. This may initially limit some of the benefits, but it’s an easy path to getting metrics and telemetry data about your UDP communication, with a very low barrier to entry and the ability to expand over time.

While Quilkin is released as both binaries and container images, and is not tied to any specific hosting platform, we’ll use Agones and Google Cloud Game Servers as our game server hosting platform for this example.

First we will create a ConfigMap to store the yaml for a static configuration for Quilkin that will accept connections on port 26001 and route then to the Xonotic (an open source, multiplayer FPS game) dedicated game server on port 26000:

  apiVersion: v1
kind: ConfigMap
metadata:
  name: quilkin-config
data:
  quilkin.yaml: |  # quilkin configuration
    version: v1alpha1
    proxy:
      port: 26001
    static:
      endpoints:
        - address: 127.0.0.1:26000

Second, we’ll take the example container that Agones provides for the Xonotic dedicated game server, and  run Quilkin alongside each dedicated game server as a sidecar, in an Agones Fleet of game servers like so:

  apiVersion: "agones.dev/v1"
kind: Fleet
metadata:
  name: xonotic-sidecar
spec:
  replicas: 2
  template:
    spec:
      container: xonotic
      ports:
        - name: default
          containerPort: 26001
          container: quilkin
      health:
        initialDelaySeconds: 30
        periodSeconds: 60
      template:
        spec:
          containers:
            - name: xonotic
              image: gcr.io/agones-images/xonotic-example:0.8
            - name: quilkin  # quilkin sidecar
              image: us-docker.pkg.dev/quilkin/release/quilkin:0.1.0
              volumeMounts:
                - name: config
                  mountPath: "/etc/quilkin"
              livenessProbe:
                httpGet:
                  path: /live
                  port: 9091
                initialDelaySeconds: 3
                periodSeconds: 2
          volumes:
            - name: config
              configMap:
                name: quilkin-config

Once applied, when we query the cluster for the running GameServers, everything looks the same as it would without Quilkin! Nothing else in our system needs to be aware that the traffic is being intercepted, and we can freely take advantage of the functionality of Quilkin without adjusting either client or server code.

  $ kubectl get gameservers
NAME                          STATE   ADDRESS         PORT   NODE           AGE
xonotic-sidecar-gdpgn-2pfkc   Ready   34.95.106.201   7929   gke-0f7d8adc   25m
xonotic-sidecar-gdpgn-c8bds   Ready   34.95.106.201   7028   gke-0f7d8adc   25m

If this has piqued your interest, make sure to have a look at the walkthrough, where we step through this same scenario and then extend it to compress UDP packets from the game client to server, without having to change either programs.

This just scratches the surface, however: there’s even more to Quilkin, including an xDS compliant admin API, a variety of existing Filters to manipulate and route UDP packets and more.

What’s next for Quilkin

Quilkin is still in its early stages, with this 0.1.0 alpha release, but we’re very happy with the foundation that has been laid.

There are a variety of features in the roadmap, from enhanced metrics and telemetry, new filters and filter types, and more.

If you would like to try out this release, you can grab the binaries or container images from our releases page, step through our quickstarts and review different integration options with your dedicated game servers.

To get involved with the project, please:

• Check out our Github repository
• Join our Discord community
• Join the quilkin-discuss mailing list
• Follow us on Twitter

Embark Studios has also released their own announcement blog post, going deeper into the plans they have for their own production game backend infrastructure, and where Quilkin fits in.

^{Thanks to everyone who has been involved with this project across Google Cloud and Embark Studios, and we look forward to the future for Quilkin!}

At Precognitive, we were able to start with a blank technology slate to support our fraud detection software products. When we started building the initial version of our platform in 2017, we had some decisions to make: What coding language to use? What cloud infrastructure provider to choose? What database to use? The majority of the decisions were straightforward, but we struggled to decide upon a database. We had plenty of collective experience with relational databases, but not with a wide-column database like Cloud Bigtable—which we knew we’d need to scale our behavior and device workloads. At launch, our products were supported by a self-managed database, but we quickly migrated to Cloud Bigtable, and we love it.  

To efficiently support our bursty, real-time fraud detection workloads, we needed a cloud database that could satisfy the following key requirements:

• Stability to keep up with increased adoption of our products
• Intelligent scaling that avoids bottlenecks
• Native integrations with BigQuery and Cloud Dataproc
• Managed services that free up our engineers’ time to work on our products

Adding Cloud Bigtable as our performance database

As we scaled our services and added customers, our data collection services for our Device Intelligence and Behavioral Analytics products were seeing thousands of events per second. Cloud Bigtable provided a stable managed database that could handle the volume we were receiving during peak hours. We weren’t always able to handle this scale, as an early version of our product utilized a self-managed database.

Every month, two or three engineers spent hours managing the database instances. Whenever the instances crashed, it would cost at least one engineer a day or two of productivity attempting to restore the instances and recovering any data from our backup database. Managing this database internally was taking precious time away from product development.

We circled back to Cloud Bigtable. After two weeks of R&D, we decided to switch the Device Intelligence and Behavioral Analytics services to Cloud Bigtable.

Cloud Bigtable solved our scaling issues. Cloud Bigtable had been attractive to us from the start because it was fully managed, and offered regional replication and other features we were lacking in our own managed instances. Cloud Bigtable provides horizontal scaling and automatically rebalances row keys (equivalent to a shard key) over time to prevent “hot” nodes. In addition, Cloud Bigtable provides a connector to BigQuery and Cloud Dataproc that allows us to analyze the terabytes of data we are processing and use that data for unsupervised machine learning.

The perks of using Cloud Bigtable

After the migration to Cloud Bigtable, we noticed a number of additional benefits: improved I/O performance, a significant cost reduction, and a sizable decrease in hours spent on database maintenance.

We measured some of our typical metrics before and after implementing Cloud Bigtable. Our request latency dropped by about 30 ms on average (to sub-10 ms) for API requests. Prior to the change, we were seeing latencies of 40+ ms on average. This latency drop on our Behavioral Analytics and Device Intelligence products allowed us to trim about an additional 10 to 15 ms off our average response time across all dependent services.

Before we moved to Cloud Bigtable, we had to scale our database instances every time a new customer was onboarded. We were over-scaling in an attempt to avoid constantly resizing our database servers. By sunsetting our self-managed database and switching to Cloud Bigtable, we cut database infrastructure costs by approximately 35% and can now scale as needed, with a couple of clicks, during onboarding.

We have spent zero hours managing a Cloud Bigtable database since launch, and we put the time we are saving every month toward product development.

Moving forward with Cloud Bigtable

As an engineering team, we love working with Cloud Bigtable. We are not only seeing improved developer experience and reduced latency, which keeps the engineers happy, but also reduced costs, which keeps the business happy. We’re able to build more product, too, with the time we’ve saved by switching to Cloud Bigtable. Stay tuned to our engineering blog for more on the lessons we’ve learned and our contributions to the wider Cloud Bigtable community.

Editor’s note: In honor of Mother’s Day, we look at how 1-800-FLOWERS.COM, Inc. migrated to Google Cloud as part of its digital transformation to quickly deploy seamless and convenient customer experiences across multiple brands on Mother’s Day and every day. 

As a leading provider of gifts designed to help customers express, connect, and celebrate,  1-800-FLOWERS.COM, Inc. has embraced cloud technologies to grow and transform its business through constant innovation. As part of our digital transformation, we recently completed the migration of our ecommerce platform and other services to Google Cloud. We’ve transitioned from a monolithic to a microservices platform, moved many workloads from our on-premises data centers to our Google Cloud environment, and scaled both horizontally and vertically. 

Since our migration, we’ve developed efficient processes to launch new brands, improved the customer experience across all brands, and seen significantly increased site traffic. 

Nurturing a more delightful customer journey

Customer delight is at the core of everything we do. Whether it be with a flower bouquet, a sweet treat, or a personalized keepsake, our mission is to deliver smiles. With the rise of the COVID-19 pandemic, we’ve all been challenged to find unique and safe ways to continue honoring the special connections in our lives and celebrating occasions with loved ones. Our customers have adapted by doing things such as sending gifts to isolated loved ones, sharing the same meal together virtually, or using video to engage with others through group activities like flower arranging and building charcuterie boards. 

The customer experience is a top priority for us, and we constantly look for innovative ways to enhance the customer journey across our ecommerce platform of more than a dozen brands. As a result, we’ve continued to see a rise in demand as customers enjoy the ease and convenience of our site and discover our full family of brands. 

Migrating to a cloud-first mindset

As we’ve continued to innovate and iterate on the customer experience, we knew we wanted to evolve our platform. We wanted to shift to a microservices platform, which would allow our team the opportunity to release updates to our site more often and set up the right continuous integration/continuous deployment (CI/CD) practices. 

Working with Google Cloud, we were able to move our ecommerce platform to the cloud and standardize our site and brand deployment by building one release that could then be repeated across all of our brands. We built everything in a modular fashion, including microservices and      code libraries, so that sites could be easily constructed and replicated for each brand. And because of this, we were able to launch Shari’s Berries extremely quickly after we acquired the brand in 2019.   

Moving our platform to a completely homegrown solution of microservices was a daunting task. But our team handled it beautifully through load-testing, stress-testing, and building new monitoring tools. And with Google Cloud supporting us all along the way, managing the migration process was simple and easy from start to finish. 

Arranging a better bouquet of services

Currently, we’ve migrated every customer-facing touchpoint for all of our brands to Google Cloud—whether it’s on the web or mobile, our AI bots, or our chat interfaces. 

• We run on Google Kubernetes Engine and Istio.
• We have nearly 200 microservices built to help power our entire ecommerce stack across several cloud services running on Google Cloud. 
• We’re utilizing BigQuery for our offline intelligence.

Results are coming up roses

Our new stack on Google Cloud has benefits for both us and our customers. We moved from a session-based to a token-based system, which provides enhanced security as well as a consistent, convenient experience across all our brands. Using service workers and a single-page app, we are able to download all the relevant site content to the browser in under two seconds to create an instant-click experience for each and every customer. We also use Google Analytics to measure our user interactions and provide personalized results to each customer. Our hope is that with this new system, we can learn from customer behavior to offer gift givers a more personalized shopping experience during each visit. 

The benefits of our new tech stack have not only helped us enhance the solutions we offer to customers today, they’ve also enabled us to offer new ones at lightning speed. With our legacy system, we used to release new code once a week or once a month. Now, even during our peak periods, we’re able to release 10 to 15 times a day and can deploy and pivot quickly to create new microservices and microsites on the fly—often without having to touch any code. 

Efficiencies abound     

The benefits of moving our platform to Google Cloud have extended to our internal teams as well. Before the migration, we had only two environments for developing and testing, which made it time-consuming to test updates before they went into production. Now with Google Cloud, we have several different journey teams—which are made up of developers, product owners, and technical owners—all working in several different environments, solving problems, and creating new solutions together. 

Everyone is now empowered to be self-sufficient, developing and releasing microservices on their own when they’re ready. This has given our developers more time to take part in continued development and learning opportunities. For example, we offer lunch-and-learn sessions as well as other resources for everyone to take advantage of so they can continue to learn and refine their skills.

Planting the seeds for future growth

As we look to the future and think about how we help our customers express, connect, and celebrate, we’ll continue to collaborate across teams to deliver solutions that spread smiles. Specifically, we’re exploring additional use of AI to help us better serve our customers across all our brands. 

We’ve enjoyed the ongoing support we’ve received from the Google Cloud team as they help us build new solutions and design a road map for the future. Their support has helped the 1-800-FLOWERS.COM, Inc. team to realize the power of the cloud and bring the very best experience to our customers.

Learn more about 1-800-FLOWERS.COM, Inc., or check out our recent blog about cloud migration for the real world.

Organizations across industries are looking to AI to turn troves of data into intelligence, powered by the latest advances in generative AI. Yet for many organizations, there is a barrier to adopting the latest models because they can be costly to train or serve. A new class of cloud GPUs is needed to lower the cost of entry for businesses that want to tap the power of AI. 

Today, we’re introducing G2, the newest addition to the Compute Engine GPU family in Google Cloud. G2 is the industry’s first cloud VM powered by the newly announced NVIDIA L4 Tensor Core GPU, and is purpose-built for large inference AI workloads like generative AI. G2 delivers cutting-edge performance-per-dollar for AI inference workloads that run on GPUs in the cloud. By switching from NVIDIA A10G GPUs to G2 instances with L4 GPUs, organizations can lower their production infrastructure costs up to 40%. We also found that customers switching from NVIDIA T4 GPUs to L4 GPUs can achieve 2x-4x better performance. As a universal GPU offering, G2 instances also help accelerate other workloads, offering significant performance improvements on HPC, graphics, and video transcoding. Currently in private preview, G2 VMs are both powerful and flexible, and scale easily from one up to eight GPUs. 

Currently organizations require end-to-end enterprise ready infrastructure that will future proof their AI and HPC initiatives for a new era. G2s will be ready to be deployed on Vertex AI, GKE, and GCE, giving customers the freedom to architect their own custom software stack to meet their performance requirements and budget. With optimized Vertex AI support for G2 VMs, AI users can tap the latest generative AI models and technologies. With an easy to use UI and automated workflows, customers can access, tune and serve modern models for video, text, images, and audio without the toil of manual optimizations. The combination of these services with the power of G2 will help customers harness the power of complex machine models for their business.

NVIDIA L4 GPUs with Ada Lovelace Architecture

G2 machine families enable machine learning customers to run their production infrastructure in the cloud for a variety of applications such as language models, image classification, object detection, automated speech recognition, and language translation. Built on the Ada Lovelace architecture with fourth-generation Tensor Cores, the NVIDIA L4 GPU provides up to 30 TFLOPS of performance for FP32, and 242 TFLOPs for FP16. Newly added FP8 support, on top of existing INT8, BFLOAT16 and TF32 capabilities, makes the L4 ideal for ML inference. 

With the latest third-generation RT Cores and DLSS 3.0 technology, G2 instances are also great for graphics-intensive workloads such as rendering and remote workstations when paired with NVIDIA RTX Virtual Workstation. NVIDIA L4 provides 3x video encoding and decoding performance, and adds new AV1 hardware-encoding capabilities. For example, G2 can enable gaming customers running game engines such as Unreal and Unity with modern graphics cards to run real-time applications. Likewise, media and entertainment customers that need GPU-enabled virtual workstations can use the L4 to create photo-realistic, high-resolution 3D content for movies, games, and AR/VR experiences using applications such as Autodesk Maya or 3D Studio Max.

What customers are saying

A handful of early customers have been testing G2 and have seen great results in real-world applications. Here are what some of them have to say about the benefits that G2 with NVIDIA L4 GPUs bring:

AppLovin

AppLovin enables developers and marketers to grow with market leading technologies. Businesses rely on AppLovin to solve their mission-critical functions with a powerful, full stack solution including user acquisition, retention, monetization and measurement.

“AppLovin serves billions of AI powered recommendations per day, so scalability and value are essential to our business,” said Omer Hasan, Vice President, Operations at AppLovin. “With Google Cloud’s G2 we’re seeing that NVIDIA L4 GPUs offer a significant increase in the scalability of our business, giving us the power to grow faster than ever before.”

WOMBO

WOMBO aims to unleash everyone’s creativity through the magic of AI, transforming the way content is created, consumed, and distributed. 

“WOMBO relies upon the latest AI technology for people to create immersive digital artwork from users’ prompts, letting them create high-quality, realistic art in any style with just an idea,” said Ben-Zion Benkhin, Co-Founder and CEO of WOMBO. “Google Cloud’s G2 instances powered by NVIDIA’s L4 GPUs will enable us to offer a better, more efficient image-generation experience for users seeking to create and share unique artwork.”

Descript

Descript’s AI-powered features and intuitive interface fuel YouTube and TikTok channels, top podcasts, and businesses using video for marketing, sales, and internal training and collaboration. Descript aims to make video a staple of every communicator’s toolkit, alongside docs and slides. 

“G2 with L4’s AI Video capabilities allow us to deploy new features augmented by natural-language processing and generative AI to create studio-quality media with excellent performance and energy efficiency” said Kundan Kumar, Head of Artificial Intelligence at Descript.

Workspot

Workspot believes that the software-as-a-service (SaaS) model is the most secure, accessible and cost-effective way to deliver an enterprise desktop and should be central to accelerating the digital transformation of the modern enterprise.

“The Workspot team looks forward to continuing to evolve our partnership with Google Cloud and NVIDIA. Our customers have been seeing incredible performance leveraging NVIDIA’s T4 GPUs. The new G2 instances with L4 GPUS through Workspot’s remote Cloud PC workstations provide 2x and higher frame rates at 1280×711 and higher resolutions” said Jimmy Chang, Chief Product Officer at Workspot.

Pricing and availability

G2 instances are currently in private preview in the following regions: us-central1, asia-southeast1 and europe-west4. Submit your request here to join the private preview, or to receive a notification as when the public preview begins. Support will be coming to Google Kubernetes Engine (GKE), Vertex AI, and other Google Cloud services as well. We’ll share G2 public availability and pricing information later in the year.

A “one size fits all” approach to education no longer works in today’s classrooms. Using cloud-based technologies, schools and educators can take a more personalized approach to education–one that suits each student’s unique learning style, abilities, and needs. 

Taking the lead toward more equitable educational opportunities worldwide, education technology pioneer, LearningMate, is partnering with Google Cloud to offer intelligent, personalized learning via Google Cloud’s Student Success Services.

In a student survey by EDUCAUSE, a nonprofit association whose mission is to advance higher education through the use of information technology, nearly all respondents asked for more digital learning and study options. Given a list of educational material types, such as study guides and recorded lessons, 93 percent said they would like to have online access to at least two options and more than half (56 percent) chose seven or more. 

The trend towards student choice challenges educators to reconsider how they teach and support learners. Students expect the same qualities in their lessons as they encounter in their other, non-school related digital experiences: personalization; user-friendliness; and engagement. Educators who are used to a more top-down education model can struggle to meet these new expectations. 

Google Cloud created Student Success Services to help education institutions meet learners where they are—in the digital age. This suite of tools and services uses Google’s advanced artificial intelligence (AI) and analytic tools to:

• Engage with students via an AI-powered learning platform and interactive tutor
• Help educators and learners collaborate more effectively
• Provide current, actionable data on student progress 

To bring our Student Success Services to more schools and organizations worldwide, we’re partnering with LearningMate — a global leader in digital learning infrastructure. LearningMate is a key go-to-market partner, helping schools design, launch, and maintain their own learning infrastructure. To start, LearningMate is adding Google’s AI-powered learning platform to its Frost platform, a popular content management for education. 

As the education model continues to change and digital learning becomes the norm, disadvantaged students risk getting left behind. For these learners, the “digital divide” is very real, and stands to hinder them further–not only in school, but in society and the workplace in later life. 

“The gap will only widen between those with digital advantages and those who struggle to gain access to devices and network necessities,” EDUCAUSE states. To meet all these challenges, schools need a diverse mix of tools, methods, and partnerships.

To help close the digital divide among students, digital tools need to be able to scale up or down so institutions and districts of any size and budget can use them. Google Cloud designed Student Success Services with this ability in mind. By offering these services, LearningMate aims to ensure that even schools with smaller teams and fewer resources can offer individualized learning experiences to their students. 

With LearningMate and Google’s nearly 40 years of combined experience in education, this partnership aims to help educators better understand their students’ engagement, performance, and preferences. To learn more about Student Success Services and our collaboration with LearningMate, watch this session from our Government and Education Summit.