Editor’s note: Today’s post highlights five takeaways from our session at the first ever Google Cloud Public Sector Summit. To watch the full session, check out All the Right Moves: Prioritizing Investments in Technology.

Now more than ever, government agencies need to invest in digital services to fulfill their missions and better serve communities. Yet modernization isn’t a one-and-done approach; it’s a sustained effort, with multi-year implications, and requires careful consideration of how to integrate existing investments to optimize costs. Digital transformation requires coordination between different programs and agencies, including all of their many competing considerations. In short, maximizing technology investments requires careful planning, strategic thinking, and industry partners that can provide flexibility and security and meet agencies where they are. 

I sat down with Suzette Kent, former U.S. federal chief information officer, and Dominic Sale, assistant commissioner for Technology Transformation Services at the General Services Administration (GSA) for a conversation to unpack this important topic and discuss industry best practices. 

The panelists had five tips for government employees who are making technology purchasing decisions for their agency.

1.   Put the agency’s mission first.Avoid getting distracted by exciting new trends and focus on long-term goals that can impact which procurement strategies or funds could be used. The discussion started with how government agencies could cut through the noise about technology and prioritize which technology is best for their needs. Sale and Kent agreed that an agency should focus on its core mission outcomes and let its technology needs flow from that. Sale also emphasized the importance of having technology design respond to humans’ needs, which has historically been a challenge for government agencies.

For agencies to stay focused on their mission, final decisions about technology need to be made by the program manager who best understands each program’s mission. The government CIO’s role is to be the enabler for the technology and leverage it at the enterprise level, particularly when it comes to sharing infrastructure.  The takeaway: enable mission programs by empowering your teams and providing access to authorized, compliant, innovative data platforms that programs can move confidently and quickly with.

2.   Invest in interoperability. Agency employees often struggle to balance the need for a positive return on investment (ROI) with requirements for meeting mission objectives. While the panelists agreed that the total cost of ownership was important, they also emphasized taking an expanded view of ROI, including future-proofing and investing in functionality that may not realize its return for many years based on the initiative. Saving money isn’t particularly valuable if the solution doesn’t meet an agency’s needs. When choosing a technology partner, government employees should understand its long-term vision to ensure that the partner fits agency priorities. Partners’ technologies should also integrate seamlessly with existing systems so agencies don’t duplicate investment costs.

For example, Google Anthos extends Google Cloud services and engineering practices into an organization’s existing environment, establishing operational consistency across apps and modernization. With Anthos, agencies can simply and securely build and deploy applications anywhere, integrating cloud services across platforms. This allows them to enjoy a consistent DevOps experience for hybrid and multi-cloud environments and enables new innovation. Most importantly, this enables an enterprise data platform, one of the largest catalysts for mission transformation and applied AI.

3.   Take advantage of artificial intelligence (AI) benefits. Over the course of the pandemic, the rapid application of AI has improved government productivity, efficiency, and the ability to deliver critical new services to the public at scale. This has further cemented AI’s role as an essential government technology for the present and future. In fact, Nextgov reports that “46% of government IT specialists plan to use AI and machine learning (ML) for embedded systems in the near future.”

As we’ve seen over the course of the pandemic, government programs can start small with AI  pilots before moving into broad deployment. This can help agencies understand AI’s potential before moving to full production. People always supervise AI technologies, and the possibilities are endless. Google Cloud’s Contact Center AI (CCAI) has helped government agencies improve the customer experience, by allowing citizens to schedule vaccine appointments via a platform of their choice with up to 28 languages and dialects, and manage vaccine deployment. The U.S. Navy spends billions annually to fight rust and corrosion on its ships. Inspections of ships, aircraft and vehicles are a time-consuming and critical part of keeping the U.S. Navy at top performance so Google Cloud and Simple Technology Solutions (STS) rapidly built an AI-based corrosion-detection and analysis system. The system detected and analyzed corrosion on vessels with 90% accuracy and will eventually be used to automate inspections of vessels, aircraft, and vehicles—saving billions of dollars. Document AI helps a variety of government agencies scale their document processing, reducing the time it typically takes to process enormous amounts of data and related citizen claims.

Successful adoption of AI also depends on the quality of the data. Ultimately, agencies need high-quality enterprise data pipes so that employees and the community trust the system and public sector agencies. Sale described a GSA project that used AI bots to read legal contracts and look for particular phrases that would indicate a specific use case. Previously, an employee would have had to read through the contracts and search for the information. In this way, AI is saving the government both money and time.

4.   Creating better experiences for the public. Sale observed that, “trust is the government’s currency and profit motive.” And trust comes when the public can be served with the same modern tools and technology they’re used to – in real-time and with transparency in mind. For example, agencies can provide transparency in public-facing dashboards for programs and supply services that deliver information in real-time through solutions like CCAI.

Trust also requires that constituents feel that government agencies will keep their data safe and secure. The need for a globally secure infrastructure with systems that are up-to-date and designed with security at every level, underpinned by zero-trust enterprise-wide remains paramount – particularly after the series of recent cyberattacks targeting government IT infrastructure.

5.   Finding the right technology partner. Government leaders need technology partners who  provide a flexible and interoperable platform to integrate existing investments and maximize technical value. Historically, public sector agencies have largely been forced to adopt private clouds, which has reduced their access to richer features, and hindered their ability to adopt a full range of security and product capabilities. The right partner won’t require government leaders to compromise on functionality or service availability to achieve compliance. The right partner can harness the power of emerging technology to make it Government-ready and the true promise of cloud– the access and integration of open data– to make missions more powerful and impactful for the constituencies they serve.

Environmental sustainability is a key priority for TELUS, a world-leading communications technology company. It continues to rank in the top 100 most sustainably managed companies in the world, and seeks to make a healthier planet for all by leveraging its global-leading technology, compassion to drive social change and reduce our collective carbon footprint through innovative technologies and sustainable business practices.

TELUS surpassed its sustainability objectives in 2019 and is now on a journey to procure all of its electricity from renewable or low-emitting sources by 2025. Next, it aims to achieve net carbon neutrality for its operations by 2030. TELUS has also been named to the Dow Jones Sustainability Index for 21 consecutive years, a feat unmatched by any other North American telecom or cable company. In 2021, it became the first company in Canada to release a Sustainability-linked bond (SLB) framework and complete an SLB offering, formally linking TELUS financing to its environmental performance.

“We’ve spent the last decade becoming a global leader in sustainability, helping make the planet healthier by ensuring that our operations are as environmentally responsible as possible,” said Geoff Pegg, Head of Sustainability and Environment at TELUS.

In part, TELUS’ strategy is focused on three key areas:

1. Seek the best renewable energy options available
2. Focus on migrating workloads to the cloud
3. Embrace a multiplier effect through the use of sustainable partners

Renewable energy impact

Part of this environmental responsibility involves investing heavily in renewable energy sources through power purchase agreements (PPAs) that help renewable energy providers like wind farms and solar companies develop their infrastructure. TELUS executed PPAs with four Alberta-based solar and wind facilities to provide 100 per cent of its electricity load demand in a province where one-third of the grid is powered by coal.

As a technology company, electricity represents a large portion of TELUS’ energy needs: 80 percent of the operational carbon footprint comes from the power requirements for TELUS’ network and administrative buildings, Pegg explains. While TELUS is using renewable energy sources and low-emitting energy grids to power its buildings and network, there’s also the often-forgotten part of the carbon emissions equation: the energy it takes to power data centers. As the International Energy Agency recently reported, data centers represent 1 percent of the global electricity demand and that figure is expected to keep rising as the world increases usage of data-heavy technologies.

“It’s probably no surprise that everyone, whether you’re a business or a consumer, is concerned about reducing carbon emissions,” said Chris Talbott, the Google Cloud Sustainability Lead. “A lot of us think about the carbon emissions associated with our cars or with the electricity that powers our homes, but oftentimes we forget about the carbon emissions that come from the digital services that we use or the networks required to deliver that data.”

As a leader in sustainability, how can TELUS meet the energy demands of its customers while also protecting the environment? One way is through the company’s previously announced collaboration with Google Cloud. The two companies are working together to build a more sustainable world through technology and reduce TELUS’ carbon footprint, create value along the entire supply chain, and optimize industry solutions for social impact through data analytics and machine learning.

Taking a cloud first approach — reducing carbon emissions with green cloud computing

Google became carbon neutral in 2007 and has achieved 100 per cent renewable energy matching every year since 2017. Google has invested in renewable energy to match the electricity we use across our entire operations, including Google Cloud, meaning every workload that TELUS runs on Google Cloud has been matched with renewable energy purchases.

“The operational carbon footprint of running anything on Google Cloud is zero,” Talbott said. Also, by working with Google, TELUS gets the benefit of economies of scale using less electricity. Not only is TELUS leveraging Google data centers, it’s also relying on the digital collaboration made possible by Google Workspace to reduce the amount of travel required by employees attending meetings in different offices. Collaboration tools like Google Meet can reduce the carbon footprint of in-person conferences by 94 percent.

Google compensates for the environmental footprint of any electricity used in the data center and out to the edge network. “You can feel pretty good about using Google Meet because it’s carbon-neutral,” Talbott said.

Multiplier through sustainable partnerships — green cloud computing radiates out

By supporting TELUS in its environmental sustainability efforts, Google Cloud is also enabling TELUS to do the same for its various partnerships. For example, powered by Google Cloud’s infrastructure and data analytics capabilities, TELUS is partnering with Picacity (formerly NXN Digital) and Google Cloud to deliver an ecosystem of integrated smart technologies that enable cities to improve the lives of their residents.

From dynamic traffic signaling that reduces congestion and emissions, to data analytics that create smarter, more efficient city planning, the partnership is transforming the way municipalities operate in our increasingly digital world.The partnership is built on four foundational pillars of infrastructure and environmental sustainability, intelligent transportation, public safety and security, and health. In the case of intelligent transportation, this means sensors, cameras, and other devices are built into or near roads, sidewalks, and bike paths to provide data for innovative software to improve traffic flow in real time. The data can then foster informed decisions about infrastructure, city planning, fleet optimization, and public safety.

All of these environmental measures may seem small when compared with the enormity of the problem that is climate change, but as Talbott said, “Change begins with the small decisions we make every day such as paying attention to the practices of companies that we’ve come to rely on daily in the modern world. They may seem small and in the margins, but at scale, this is how we can make a real impact.”

Traditionally, dedicated game servers for real time multiplayer games have used bespoke UDP protocols for communication and synchronization of gameplay among the players within a game. This communication is most often bundled into monolithic game servers and clients, pairing the technical functionality of communication protocols, such as custom network physics synchronisation, security, access control, telemetry and metrics, with the extremely high computational requirements of physics simulations, AI computation and more.

Developed in collaboration with Embark Studios, Quilkin is a UDP proxy, tailor-made for high performance real-time multiplayer games. Its aim is twofold:

1. Pull common functionality, such as security, access control, telemetry and metrics out of monolithic dedicated game servers and clients.
2. Provide this common functionality in a composable and configurable way, such that it can be reused across a wide set of multiplayer games.

This reusable foundation then allows game developers to spend more of their time focusing on building the game-specific aspects of building their multiplayer communication protocols, rather than these common aspects.

Challenges with multiplayer Game Server communication

In fast-paced, multiplayer games, the full simulation of a session of gameplay generally occurs within the memory of a monolithic dedicated game server, whose responsibility covers everything from network physics and AI simulation to communications from client back to server and more.

Since the entire state of the game is memory resident, each client connects directly to the dedicated game server the player is playing on, which presents several challenges:

• Each dedicated game server is a single point of failure. If it goes down, then the whole game session (or sometimes multiple sessions) fails. This makes it a target for malicious actors.
• The IP and port of connection to the game server is public, and exposed to the game client, making it easy to discover and target.
• Multiple aspects of game server simulation and network communication are tightly coupled in the same process, making reuse and modularity more difficult, and expanding risk of performance issues.

If we look at both web and mobile technologies over the past several years, some of these challenges start to look very familiar. Thankfully, one of the solutions to help drive dedicated server workloads to more redundant and distributed orchestration is the utilisation of traffic proxies!

By using a proxy for multiplayer UDP traffic, in front of our dedicated games servers within a low latency network such as what is available on Google Cloud, we can address these key challenges as follows:

• Greater reliability. Proxies provide redundant points of communication entry. UDP packets can be sent to any number of proxies and routed to the dedicated game server. While a dedicated game server will still generally be a single point of failure, proxies improve redundancy and potential failover at the communication layer.
• Greater security. The IP and port of the dedicated game server is no longer public. Game clients may only have visibility into a subset of the proxy pool, limiting a potential attack surface.
• Greater scalability. We start to break apart the single process, as we can move aspects of the communication protocol, metrics, communication security and access control into the proxy. This removes the non-game specific computation out of your game server’s processing loop.

As a result, the entire system is now more resilient as proxies can be scaled independently, not only for performance reasons but also to distribute load in case of malicious actors.

Introducing Quilkin: The UDP proxy for Game Servers

Embark Studios and Google Cloud came together and built Quilkin, to provide a standard, open source solution.  Based out of Stockholm, Embark Studios is a (relatively) new studio made up of seasoned industry veterans.  They were the perfect collaboration partner to create Quilkin with, given their team’s experience with large scale real time multiplayer games.  

Quilkin is an open-source, non-transparent UDP proxy specifically designed for use with large scale multiplayer dedicated game server deployments, to ensure security, access control, telemetry data, metrics and more.

Quilkin is designed to be used behind game clients as well as in front of dedicated game servers, and offers the following major benefits:

• Obfuscation. Non-transparent proxying of UDP data, making the internal state of your game architecture less visible to bad actors.
• Out of the box metrics. For UDP packet traffic and communication.
• Visibility. A composable set of processing filters that can be applied for routing, access control, rate limiting, and more.
• Flexibility. Ability to to be utilised as a standalone binary, with no client/server changes required or as a Rust library, depending on how deep an integration you wish for your system and/or custom processing Filters you wish to build.
• Compatibility. Can be integrated with existing C/C++ code bases via Rust FFI, if required.
• Onboarding. Multiple integration patterns, allowing you to choose the level of integration that makes sense for your architecture and existing platform.

Until now, these sorts of capabilities are only available to large game studios with resources to build their own proprietary technology. 

We think leveling the playing field for everyone in the games industry is an important and worthy endeavor. That’s why we collaborated with Google Cloud and initiated this project together.

At Embark, we believe open source is the future of the games industry and that open, cross-company collaboration is the way forward, so that all studios, regardless of size, are able to achieve the same level of technical capabilities. —Luna Duclos, Tech Lead, Embark Studios

Google Cloud is excited to announce Quilkin as the latest entry in our portfolio of open-source solutions for gaming.  Quilkin complements our existing OSS solutions including Agones for game servers, Open Match for matchmaking, and Open Saves for persistence.  These are designed to work together as an open and integrated ecosystem for gaming.  We’re proud to include Embark Studios as our latest open source collaborator for gaming along with Ubisoft, Unity, and 2K Games.  Google Cloud will continue to work closely with our partners in industry and the community to offer planet-scale solutions to power the world’s largest games. —Rob Martin, Chief Architect, Google Cloud for Games

Getting started with Quilkin

While Quilkin can support more advanced deployment scenarios like above, the easiest way to get started with Quilkin is to deploy it as a sidecar to your existing dedicated game server. This may initially limit some of the benefits, but it’s an easy path to getting metrics and telemetry data about your UDP communication, with a very low barrier to entry and the ability to expand over time.

While Quilkin is released as both binaries and container images, and is not tied to any specific hosting platform, we’ll use Agones and Google Cloud Game Servers as our game server hosting platform for this example.

First we will create a ConfigMap to store the yaml for a static configuration for Quilkin that will accept connections on port 26001 and route then to the Xonotic (an open source, multiplayer FPS game) dedicated game server on port 26000:

  apiVersion: v1
kind: ConfigMap
metadata:
  name: quilkin-config
data:
  quilkin.yaml: |  # quilkin configuration
    version: v1alpha1
    proxy:
      port: 26001
    static:
      endpoints:
        - address: 127.0.0.1:26000

Second, we’ll take the example container that Agones provides for the Xonotic dedicated game server, and  run Quilkin alongside each dedicated game server as a sidecar, in an Agones Fleet of game servers like so:

  apiVersion: "agones.dev/v1"
kind: Fleet
metadata:
  name: xonotic-sidecar
spec:
  replicas: 2
  template:
    spec:
      container: xonotic
      ports:
        - name: default
          containerPort: 26001
          container: quilkin
      health:
        initialDelaySeconds: 30
        periodSeconds: 60
      template:
        spec:
          containers:
            - name: xonotic
              image: gcr.io/agones-images/xonotic-example:0.8
            - name: quilkin  # quilkin sidecar
              image: us-docker.pkg.dev/quilkin/release/quilkin:0.1.0
              volumeMounts:
                - name: config
                  mountPath: "/etc/quilkin"
              livenessProbe:
                httpGet:
                  path: /live
                  port: 9091
                initialDelaySeconds: 3
                periodSeconds: 2
          volumes:
            - name: config
              configMap:
                name: quilkin-config

Once applied, when we query the cluster for the running GameServers, everything looks the same as it would without Quilkin! Nothing else in our system needs to be aware that the traffic is being intercepted, and we can freely take advantage of the functionality of Quilkin without adjusting either client or server code.

  $ kubectl get gameservers
NAME                          STATE   ADDRESS         PORT   NODE           AGE
xonotic-sidecar-gdpgn-2pfkc   Ready   34.95.106.201   7929   gke-0f7d8adc   25m
xonotic-sidecar-gdpgn-c8bds   Ready   34.95.106.201   7028   gke-0f7d8adc   25m

If this has piqued your interest, make sure to have a look at the walkthrough, where we step through this same scenario and then extend it to compress UDP packets from the game client to server, without having to change either programs.

This just scratches the surface, however: there’s even more to Quilkin, including an xDS compliant admin API, a variety of existing Filters to manipulate and route UDP packets and more.

What’s next for Quilkin

Quilkin is still in its early stages, with this 0.1.0 alpha release, but we’re very happy with the foundation that has been laid.

There are a variety of features in the roadmap, from enhanced metrics and telemetry, new filters and filter types, and more.

If you would like to try out this release, you can grab the binaries or container images from our releases page, step through our quickstarts and review different integration options with your dedicated game servers.

To get involved with the project, please:

• Check out our Github repository
• Join our Discord community
• Join the quilkin-discuss mailing list
• Follow us on Twitter

Embark Studios has also released their own announcement blog post, going deeper into the plans they have for their own production game backend infrastructure, and where Quilkin fits in.

^{Thanks to everyone who has been involved with this project across Google Cloud and Embark Studios, and we look forward to the future for Quilkin!}