Start your 2022 New Year’s resolutions by learning at no cost how to use Google Cloud with the following training opportunities:

30 day access to Google Cloud Skills Boost 

Register by January 31, 2022 and claim 30 days free access to Google Cloud Skills Boost to complete the Getting Started with Google Cloud learning path. 

Google Cloud Skills Boost is the definitive destination for skills development where you can personalize learning paths, track progress, and validate your newly-earned expertise with skill badges. 

The Getting Started with Google Cloud learning path will give you the opportunity to earn three skill badges after you complete hands-on labs and courses designed for aspiring cloud engineers and architects. It covers the fundamentals of Google Cloud including core infrastructure, big data and ML, writing gcloud commands, using Cloud Shell, deploying virtual machines, and running containerized applications on GKE.

Cloud OnBoard: half day training on getting started with Google Cloud fundamentals

Attend the Getting Started Cloud OnBoard on January 20 for a comprehensive Google Cloud orientation. Google Cloud experts will show you how to execute your compute, available storage options, how to secure your data, and available Google Cloud managed services. 

Cloud Study Jam: expert-guided hands-on lab

Google Cloud experts will walk you through a hands-on lab included in Google Cloud Skill Boost’s Getting Started with Google Cloud learning path when you join our Cloud Study Jam on January 27. Google Cloud experts will also answer questions live via chat during this event.

Traditionally, dedicated game servers for real time multiplayer games have used bespoke UDP protocols for communication and synchronization of gameplay among the players within a game. This communication is most often bundled into monolithic game servers and clients, pairing the technical functionality of communication protocols, such as custom network physics synchronisation, security, access control, telemetry and metrics, with the extremely high computational requirements of physics simulations, AI computation and more.

Developed in collaboration with Embark Studios, Quilkin is a UDP proxy, tailor-made for high performance real-time multiplayer games. Its aim is twofold:

1. Pull common functionality, such as security, access control, telemetry and metrics out of monolithic dedicated game servers and clients.
2. Provide this common functionality in a composable and configurable way, such that it can be reused across a wide set of multiplayer games.

This reusable foundation then allows game developers to spend more of their time focusing on building the game-specific aspects of building their multiplayer communication protocols, rather than these common aspects.

Challenges with multiplayer Game Server communication

In fast-paced, multiplayer games, the full simulation of a session of gameplay generally occurs within the memory of a monolithic dedicated game server, whose responsibility covers everything from network physics and AI simulation to communications from client back to server and more.

Since the entire state of the game is memory resident, each client connects directly to the dedicated game server the player is playing on, which presents several challenges:

• Each dedicated game server is a single point of failure. If it goes down, then the whole game session (or sometimes multiple sessions) fails. This makes it a target for malicious actors.
• The IP and port of connection to the game server is public, and exposed to the game client, making it easy to discover and target.
• Multiple aspects of game server simulation and network communication are tightly coupled in the same process, making reuse and modularity more difficult, and expanding risk of performance issues.

If we look at both web and mobile technologies over the past several years, some of these challenges start to look very familiar. Thankfully, one of the solutions to help drive dedicated server workloads to more redundant and distributed orchestration is the utilisation of traffic proxies!

By using a proxy for multiplayer UDP traffic, in front of our dedicated games servers within a low latency network such as what is available on Google Cloud, we can address these key challenges as follows:

• Greater reliability. Proxies provide redundant points of communication entry. UDP packets can be sent to any number of proxies and routed to the dedicated game server. While a dedicated game server will still generally be a single point of failure, proxies improve redundancy and potential failover at the communication layer.
• Greater security. The IP and port of the dedicated game server is no longer public. Game clients may only have visibility into a subset of the proxy pool, limiting a potential attack surface.
• Greater scalability. We start to break apart the single process, as we can move aspects of the communication protocol, metrics, communication security and access control into the proxy. This removes the non-game specific computation out of your game server’s processing loop.

As a result, the entire system is now more resilient as proxies can be scaled independently, not only for performance reasons but also to distribute load in case of malicious actors.

Introducing Quilkin: The UDP proxy for Game Servers

Embark Studios and Google Cloud came together and built Quilkin, to provide a standard, open source solution.  Based out of Stockholm, Embark Studios is a (relatively) new studio made up of seasoned industry veterans.  They were the perfect collaboration partner to create Quilkin with, given their team’s experience with large scale real time multiplayer games.  

Quilkin is an open-source, non-transparent UDP proxy specifically designed for use with large scale multiplayer dedicated game server deployments, to ensure security, access control, telemetry data, metrics and more.

Quilkin is designed to be used behind game clients as well as in front of dedicated game servers, and offers the following major benefits:

• Obfuscation. Non-transparent proxying of UDP data, making the internal state of your game architecture less visible to bad actors.
• Out of the box metrics. For UDP packet traffic and communication.
• Visibility. A composable set of processing filters that can be applied for routing, access control, rate limiting, and more.
• Flexibility. Ability to to be utilised as a standalone binary, with no client/server changes required or as a Rust library, depending on how deep an integration you wish for your system and/or custom processing Filters you wish to build.
• Compatibility. Can be integrated with existing C/C++ code bases via Rust FFI, if required.
• Onboarding. Multiple integration patterns, allowing you to choose the level of integration that makes sense for your architecture and existing platform.

Until now, these sorts of capabilities are only available to large game studios with resources to build their own proprietary technology. 

We think leveling the playing field for everyone in the games industry is an important and worthy endeavor. That’s why we collaborated with Google Cloud and initiated this project together.

At Embark, we believe open source is the future of the games industry and that open, cross-company collaboration is the way forward, so that all studios, regardless of size, are able to achieve the same level of technical capabilities. —Luna Duclos, Tech Lead, Embark Studios

Google Cloud is excited to announce Quilkin as the latest entry in our portfolio of open-source solutions for gaming.  Quilkin complements our existing OSS solutions including Agones for game servers, Open Match for matchmaking, and Open Saves for persistence.  These are designed to work together as an open and integrated ecosystem for gaming.  We’re proud to include Embark Studios as our latest open source collaborator for gaming along with Ubisoft, Unity, and 2K Games.  Google Cloud will continue to work closely with our partners in industry and the community to offer planet-scale solutions to power the world’s largest games. —Rob Martin, Chief Architect, Google Cloud for Games

Getting started with Quilkin

While Quilkin can support more advanced deployment scenarios like above, the easiest way to get started with Quilkin is to deploy it as a sidecar to your existing dedicated game server. This may initially limit some of the benefits, but it’s an easy path to getting metrics and telemetry data about your UDP communication, with a very low barrier to entry and the ability to expand over time.

While Quilkin is released as both binaries and container images, and is not tied to any specific hosting platform, we’ll use Agones and Google Cloud Game Servers as our game server hosting platform for this example.

First we will create a ConfigMap to store the yaml for a static configuration for Quilkin that will accept connections on port 26001 and route then to the Xonotic (an open source, multiplayer FPS game) dedicated game server on port 26000:

  apiVersion: v1
kind: ConfigMap
metadata:
  name: quilkin-config
data:
  quilkin.yaml: |  # quilkin configuration
    version: v1alpha1
    proxy:
      port: 26001
    static:
      endpoints:
        - address: 127.0.0.1:26000

Second, we’ll take the example container that Agones provides for the Xonotic dedicated game server, and  run Quilkin alongside each dedicated game server as a sidecar, in an Agones Fleet of game servers like so:

  apiVersion: "agones.dev/v1"
kind: Fleet
metadata:
  name: xonotic-sidecar
spec:
  replicas: 2
  template:
    spec:
      container: xonotic
      ports:
        - name: default
          containerPort: 26001
          container: quilkin
      health:
        initialDelaySeconds: 30
        periodSeconds: 60
      template:
        spec:
          containers:
            - name: xonotic
              image: gcr.io/agones-images/xonotic-example:0.8
            - name: quilkin  # quilkin sidecar
              image: us-docker.pkg.dev/quilkin/release/quilkin:0.1.0
              volumeMounts:
                - name: config
                  mountPath: "/etc/quilkin"
              livenessProbe:
                httpGet:
                  path: /live
                  port: 9091
                initialDelaySeconds: 3
                periodSeconds: 2
          volumes:
            - name: config
              configMap:
                name: quilkin-config

Once applied, when we query the cluster for the running GameServers, everything looks the same as it would without Quilkin! Nothing else in our system needs to be aware that the traffic is being intercepted, and we can freely take advantage of the functionality of Quilkin without adjusting either client or server code.

  $ kubectl get gameservers
NAME                          STATE   ADDRESS         PORT   NODE           AGE
xonotic-sidecar-gdpgn-2pfkc   Ready   34.95.106.201   7929   gke-0f7d8adc   25m
xonotic-sidecar-gdpgn-c8bds   Ready   34.95.106.201   7028   gke-0f7d8adc   25m

If this has piqued your interest, make sure to have a look at the walkthrough, where we step through this same scenario and then extend it to compress UDP packets from the game client to server, without having to change either programs.

This just scratches the surface, however: there’s even more to Quilkin, including an xDS compliant admin API, a variety of existing Filters to manipulate and route UDP packets and more.

What’s next for Quilkin

Quilkin is still in its early stages, with this 0.1.0 alpha release, but we’re very happy with the foundation that has been laid.

There are a variety of features in the roadmap, from enhanced metrics and telemetry, new filters and filter types, and more.

If you would like to try out this release, you can grab the binaries or container images from our releases page, step through our quickstarts and review different integration options with your dedicated game servers.

To get involved with the project, please:

• Check out our Github repository
• Join our Discord community
• Join the quilkin-discuss mailing list
• Follow us on Twitter

Embark Studios has also released their own announcement blog post, going deeper into the plans they have for their own production game backend infrastructure, and where Quilkin fits in.

^{Thanks to everyone who has been involved with this project across Google Cloud and Embark Studios, and we look forward to the future for Quilkin!}

With nearly 2,300 stores, The Home Depot is the world’s largest home-improvement chain — a brand that professional contractors and DIYers alike have come to depend on. The home improvement industry continues to experience unprecedented demand and dramatic increases in online ordering accompanied by expanding consumer expectations for things like curbside pickup and same day delivery. The Home Depot’s decision to migrate to cloud-based infrastructure, including the migration of the company’s SAP applications on Google Cloud which began in 2017, has set it up for success in an increasingly digital world, and helped the company adapt to changing market conditions quickly. 

Interconnected retail at scale

Building on a strong customer-first philosophy, The Home Depot aims to create what it calls interconnected retail—allowing customers to shop however, whenever, and wherever they want. “So many companies are focused on omni-channel retail,” explains Sam Moses, Vice President of Corporate Systems. “At The Home Depot, we wanted to take it to the next level. Interconnected retail puts the customer at the center of everything and enables them to shop in store, online, or both. Customers can begin a transaction online and continue in-store, or vice-versa.”

To support this strategy, the company’s SAP environment needed to be more agile. Running  everything on-premises, from central finance to POS systems, meant that The Home Depot’s IT teams experienced redundancy and repetitive, manual processes. Their data warehouse needed an upgrade to process and analyze growing and increasingly diverse data sets. The Home Depot chose to migrate its SAP environment to Google Cloud to support both the velocity and scale needed for the business as well as critical analytics capabilities needed for its bold digital initiatives. “We chose Google Cloud to support our SAP implementation. Our decision had a lot to do with the relationship between Google Cloud and SAP and also for the applications and services that are offered by Google Cloud, like BigQuery, which are helping to enable data and analytics within our organization,” Moses explains.

After migrating its SAP applications—including S/4HANA, its customer activity repository (CAR), general ledger, e-commerce system, enterprise data warehouse and more to Google Cloud, the company now has the speed, scale and flexibility to tackle enormous spikes in the business, all while staying fully available for their customers. Additionally, The Home Depot was able to transform its financial systems and make them more agile to deliver critical information across multiple business functions in real time.

Maximizing data insights to support customer experiences

By migrating to Google Cloud, The Home Depot is leveraging Google Cloud analytics to build   the industry’s most efficient supply chain including more robust demand forecasting, supplier lead times, estimated delivery times and more, all while maintaining better security than before. “We experienced unprecedented change in our customers’ behavior and their buying patterns, which puts a lot of pressure on our supply chain,” explains Moses. “So having the ability to leverage data and analytics gives us insights to know exactly what it is that our customers need.” 

The company’s analysts now use BigQuery ML for machine learning directly against the company’s BigQuery data and use AutoML to determine the best model for predictions. The Home Depot’s engineers have also adapted BigQuery to monitor, analyze, and act on application performance data across all its stores and warehouses in real time—capabilities that were not as seamless in the on-premises environment.

With hundreds of projects on Google Cloud, The Home Depot’s cloud journey is well on track, but the company is always looking to the future. “As our customers’ needs have continued to evolve, and as technology has continued to evolve, our relationship with Google will continue to advance — to be able to innovate together, to be able to find new solutions together, to better serve our customers.”

Learn more about how The Home Depot is renovating its retail operation with SAP on Google Cloud.

In our conversations with technology leaders about data-driven transformation using Google Data Cloud –  industry’s leading unified data and AI solution – , one important topic is incorporating continuous intelligence to move from answering questions such as “What has happened? to questions like “What is happening?” and “What might happen?”. The core to this evolution is the need for an underlying data processing that not only provides powerful real-time capabilities for events happening close to origination, but also brings together existing data sources under one unified data platform to enable organizations to draw insights and take actions holistically. Dataflow, Google’s cloud-native data processing and streaming analytics platform, is a key component of any modern data and AI architecture and data transformation journey, along with BigQuery, Google’s internet-scale warehouse with built-in streaming, BI engine and ML; Pub/Sub, a global no-ops event delivery service; and Looker, a modern BI and embedded analytics platform. One of the key evaluation factors is potential economic value of Dataflow to their organization, particularly in the context of engaging other stakeholders is key for many of the leaders that we engage with. So we commissioned Forrester Consulting to conduct a comprehensive study on the impact that Dataflow had on their organization by interviewing actual customers . 

Today we’re excited to share our commissioned study conducted by Forrester Consulting, the Total Economic Impact™ of Google Cloud Dataflow, which allows data leaders to understand and quantify the benefits of Dataflow, and use cases it enables. Forrester conducted interviews with Dataflow customers to evaluate the benefits, costs, and risks of investing in Dataflow across an organization. Based on their interviews, Forrester identified major financial benefits across four different areas: business growth, infrastructure cost savings, data engineer productivity, and administration efficiency. In fact, Forrester found that customers adopting Dataflow can achieve a 55% boost in developer productivity and a 50% reduction in infrastructure costs. In fact, Forrester projects that customers adopting Dataflow can achieve a range of up to 171% Return on Investment (ROI) and a less than six months payback period. Customers can now use figures in the report to compute their own Return on Investment (ROI) and payback period.

“Dataflow is integral to accelerating time-to-market, decreasing time-to-production, reducing time to figure out how to use data for use cases, focusing time on value-add tasks, streamlining ingestion, and reducing total cost of ownership.” – Lead technical architect, CPG

Let’s take a deeper look at the ways that Forrester found that Dataflow can help you achieve your goals and unlock your business potential. 

Benefit #1: Increase data engineer productivity by 55%

Developers can choose among a variety of programming languages to define and execute data workflows. Dataflow also seamlessly integrates with other Google Cloud Platform and open source technologies to maximize value and applicability to a wide variety of use cases. Dataflow streamlined workflows with code reusability,dynamic templates, and the simplicity of a managed service. Engineers trusted pipelines to run correctly and adhere to governance. Data engineers avoided laborious issue-monitoring and remediation tasks that were common in the legacy environments such as poor performance, lack of availability, and failed jobs. Teams valued the language flexibility and open source base.

“Dataflow provided us with ETL replacement that opened limitless potential use cases and enabled us to do smarter data enhancement while data remains in motion.” — Director of data projects, financial services

Benefit #2: Reduce infrastructure costs by up-to 50% for batch and streaming workloads 

Dataflow’s serverless autoscaling and discrete control of job needs, scheduling, and regions eliminated overhead and optimized technology spending. Consolidating global data processing solutions to Dataflow further eliminated excess costs while ensuring performance, resilience, and governance across environments. Dataflow’s unified streaming and batch data platform gives organizations the flexibility to define either workload in the same programming model, run it on the same infrastructure, and manage it from a single operational management tool. 

“Our costs with our cloud data platform using Dataflow are just a fraction of the costs we faced before. Now we only pay for cloud infrastructure consumption because the open source base helps us avoid licensing costs. We spend about $120,000 per year with Dataflow, but we’d be spending millions with our old technologies.” – Lead technical architect, CPG

Benefit #3: Increase top-line revenue by improving customer experience and retention with payback time of < 6 months

Streaming analytics is an essential capability in today’s digital world to gain real-time actionable insights. Likewise, organizations must also have flexible, high- performance batch environments to analyze historical data for building machine learning models, business intelligence, and advanced analytics. Dataflow enabled real-time streaming use cases, improved data enrichment, encouraged data exploration,improved performance and resiliency, reduced errors, increased trust, and eliminated barriers to scale. As a result, organizations provided customers with more accurate, relevant, and in-the-moment data-backed services and insights — boosting customer experience, creating new revenue streams, and improving acquisition, retention, and enrichment.

“It’s already been proven that we are getting more business [with Dataflow] because we can turn around results faster for customers.” – VP of technology, financial services technology

“When we provide data to our customers and partners with Dataflow, we are much more confident in those numbers and can provide accurate data within a minute. Our customers and partners have taken note and commented on this. It’s reduced complaints and prevented churn.” – Senior software engineer, media

Other benefits 

Eliminated administrative overhead and toil

As a cloud-native managed service, all administration tasks such as provisioning, scaling, and updates are automatically handled by Google Cloud. Teams no longer need to manage servers and related software for legacy data processing solutions. Admins also streamlined processes for setting up data sources, adding pipelines, and enforcing governance.

Saved business operations costs for support teams and data end users

Dataflow improved the speed, quality, reliability, and ease of access to data for insights for general business users, saving time and empowering users to drive better data-backed outcomes. It also reduced support inquiry volume while automating manual job creation.

What’s next?

Download the Forrester Total Economic Impact study today to dive deep into the economic impact Dataflow can deliver your organization. We would love to partner with you to explore the potential Dataflow can unlock in your teams. Please reach out to our sales team to start a conversation about your data transformation with Google Cloud.

Setting up your new cloud infrastructure is scary. Extra scary when you realize that someone (is it gonna be you?) gets to have phenomenal cosmic power over the whole thing. 

Yes, I’m talking about the admin account, and today we’ll dig into why they are important, dangerous and different.

When the team at pistach.io got their nuts-as-a-service business growing fruitfully, they knew they needed to think carefully about admin accounts. These people would have tremendous control over their use of Google Cloud, and they could potentially cause very big problems if any were compromised. Definitely resources that require a protective shell.

Early in pistach.io’s development, an employee named Walter Nutt wanted to play a prank by changing his co-worker’s profile photo from an almond to a peanut (pretty devious, since a peanut is actually a legume). He didn’t have access to his friend’s computer, but he did have access to the company’s Cloud Storage bucket. While searching for the profile photo in question, Wally inadvertently deleted the entire contents of the bucket! 

As he searched for ways to restore its contents, Wally modified access to two other pistach.io buckets.  The company was ground to a halt for a week while teams worked to crack through the permissions issues.

Time to rethink permissions a bit, so this couldn’t spoil their buttery smooth operations in the future.

Following the resource manager guide, the team made a super admin email address that wasn’t tied to a particular individual or Workspace account, and secured it with strong multi-factor authentication. This would be their backup in case an admin account were to be compromised, so they could recover and repair.

The team already uses Google Workspace, so they have an organization set up already. That creation process established initial super administrators, allowing them to create and modify all other resources inside the organization. As they looked toward using Google Cloud, the super administrators could:

1. Give the admin role to people, for Cloud
2. Act as a point of contact for account recovery
3. Modify or delete the organization if needed

Making admin users for the organization allows other people to then flesh out the resources and policies for pistach.io, before they go nuts and give everyone all the permissions. While that would speed things up, it would make it easy for an attacker to crack through the security shell because any account compromise could give ousize access. Yikes!

Instead, the IT leads specified certain people to act as organization admins, and then gave them permissions to:

1. Define Identity and Access Management policies
2. Structure the Resource Hierarchy
3. Delegate control of specific Cloud elements to others on the team

Once those organization admins were set up, they could give management and oversight of Compute, Storage, Networking and other resource types to the relevant leads, making sure each person had just the right amount of permission for the role they needed to perform. The organization admins don’t have permissions themselves to make these resources. They just delegate. 

Now each person can accomplish the job they’re responsible for, but doesn’t have overly permissive access. Delegating like this keeps the entire organization safer, and limits the blast radius if someone does manage to break in.

You can go through these steps yourself with this tutorial.

By default the creation of an organization resource for the domain gives everyone the ability to create projects and billing accounts. Once they set up their Organization Admin at pistach.io they decided to remove some of these wide permissions and, in a nutshell, bring everything down to a much finer control. So people could get permissions for a folder or a project, but not the entire organization!

Remember to take care of your admin roles, as they have the power, and responsibility, to cause serious harm if not used safely. Be safe with your Identity and Access Management. And keep your data yours! 

Next time we join you we’ll take a crack at creating and provisioning an app to run inside the policies and resource management frameworks created today.