Predict User Churn on Gaming Apps with Google Analytics Data using BigQuery ML - Build What's Next

Build What's Next

API Management
Application Modernization with Google Cloud
Hybrid and Multi-cloud Application Platform

10137

Of your peers have already watched this video.

1:00 Minutes

The most insightful time you'll spend today!

How-to

Predict User Churn on Gaming Apps with Google Analytics Data using BigQuery ML

User retention can be a major challenge for mobile game developers. According to the Mobile Gaming Industry Analysis in 2019, most mobile games only see a 25% retention rate for users after the first day. To retain a larger percentage of users after their first use of an app, developers can take steps to motivate and incentivize certain users to return. But to do so, developers need to identify the propensity of any specific user returning after the first 24 hours. 

In this blog post, we will discuss how you can use BigQuery ML to run propensity models on Google Analytics 4 data from your gaming app to determine the likelihood of specific users returning to your app.

You can also use the same end-to-end solution approach in other types of apps using Google Analytics for Firebase as well as apps and websites using Google Analytics 4. To try out the steps in this blogpost or to implement the solution for your own data, you can use this Jupyter Notebook. 

Using this blog post and the accompanying Jupyter Notebook, you’ll learn how to:

  • Explore the BigQuery export dataset for Google Analytics 4
  • Prepare the training data using demographic and behavioural attributes
  • Train propensity models using BigQuery ML
  • Evaluate BigQuery ML models
  • Make predictions using the BigQuery ML models
  • Implement model insights in practical implementations

Google Analytics 4 (GA4) properties unify app and website measurement on a single platform and are now default in Google Analytics. Any business that wants to measure their website, app, or both, can use GA4 for a more complete view of how customers engage with their business. With the launch of Google Analytics 4, BigQuery export of Google Analytics data is now available to all users. If you are already using a Google Analytics 4 property, you can follow this guide to set up exporting your GA data to BigQuery.

Once you have set up the BigQuery export, you can explore the data in BigQuery. Google Analytics 4 uses an event-based measurement model. Each row in the data is an event with additional parameters and properties. The Schema for BigQuery Export can help you to understand the structure of the data.  

In this blogpost, we use the public sample export data from an actual mobile game app called “Flood It!” (Android, iOS) to build a churn prediction model. But you can use data from your own app or website. 

Here’s what the data looks like. Each row in the dataset is a unique event, which can contain nested fields for event parameters.

  SELECT *
FROM `firebase-public-project.analytics_153293282.events_*`
TABLESAMPLE SYSTEM (1 PERCENT)
table

This dataset contains 5.7M events from over 15k users.

  SELECT 
    COUNT(DISTINCT user_pseudo_id) as count_distinct_users,
    COUNT(event_timestamp) as count_events
FROM
  `firebase-public-project.analytics_153293282.events_*
count

Our goal is to use BigQuery ML on the sample app dataset to predict propensity to user churn or not churn based on users’ demographics and activities within the first 24 hours of app installation.

data

In the following sections, we’ll cover how to:

  1. Pre-process the raw event data from GA4
    1. Identify users & the label feature
    2. Process demographic features
    3. Process behavioral features
  2. Train classification model using BigQuery ML
  3. Evaluate the model using BigQueryML
  4. Make predictions using BigQuery ML
  5. Utilize predictions for activation

Pre-process the raw event data

You cannot simply use raw event data to train a machine learning model as it would not be in the right shape and format to use as training data. So in this section, we’ll go through how to pre-process the raw data into an appropriate format to use as training data for classification models.

This is what the training data should look like for our use case at the end of this section:

user id

Notice that in this training data, each row represents a unique user with a distinct user ID (user_pseudo_id). 

Identify users & the label feature

We first filtered the dataset to remove users who were unlikely to return the app anyway. We defined these ‘bounced’ users as ones who spent less than 10 mins with the app. Then we labeled all remaining users:

  • churned: No event data for the user after 24 hours of first engaging with the app.
  • returned: The user has at least one event record after 24 hours of first engaging with the app.

For your use case, you can have a different definition of bounce and churning. Also you can even try to predict something else other than churning, e.g.:

  • whether a user is likely to spend money on in-game currency 
  • likelihood of completing n-number of game levels
  • likelihood of spending n amount of time in-game etc.

In such cases, label each record accordingly so that whatever you are trying to predict can be identified from the label column.

From our dataset, we found that ~41% users (5,557) bounced. However, from the remaining users (8,031),  ~23% (1,883) churned after 24 hours:

  SELECT
    bounced,
    churned, 
    COUNT(churned) as count_users
FROM
    bqmlga4.returningusers
GROUP BY 1,2
ORDER BY bounced
boucned

To create these bounced and churned columns, we used the following snippet of SQL code. 

  ...
#churned = 1 if last_touch within 24 hr of app installation, else 0
IF (user_last_engagement < TIMESTAMP_ADD(user_first_engagement, 
      INTERVAL 24 HOUR),
    1,
    0 ) AS churned,
#bounced = 1 if last_touch within 10 min, else 0
IF (user_last_engagement <= TIMESTAMP_ADD(user_first_engagement, 
      INTERVAL 10 MINUTE),
    1,
    0 ) AS bounced,
...

You can view the Jupyter Notebook for the full query used for materializing the bounced and churned labels. 

Process demographic features

Next, we added features both for demographic data and for behavioral data spanning across multiple columns. Having a combination of both demographic data and behavioral data helps to create a more predictive model. 

We used the following fields for each user as demographic features:

  • geo.country
  • device.operating_system
  • device.language

A user might have multiple unique values in these fields — for example if a user uses the app from two different devices. To simplify, we used the values from the very first user engagement event.

  CREATE OR REPLACE VIEW bqmlga4.user_demographics AS (
  WITH first_values AS (
      SELECT
          user_pseudo_id,
          geo.country as country,
          device.operating_system as operating_system,
          device.language as language,
          ROW_NUMBER() OVER (PARTITION BY user_pseudo_id ORDER BY event_timestamp DESC) AS row_num
      FROM `firebase-public-project.analytics_153293282.events_*`
      WHERE event_name="user_engagement"
      )
  SELECT * EXCEPT (row_num)
  FROM first_values
  WHERE row_num = 1 #first engagement
);

Process behavioral features

There is additional demographic information present in the GA4 export dataset, e.g. app_info, device, event_params, geo etc. You may also send demographic information to Google Analytics through each hit via user_properties. Furthermore, if you have first-party data on your own system, you can join that with the GA4 export data based on user_ids. 

To extract user behavior from the data, we looked into the user’s activities within the first 24 hours of first user engagement. In addition to the events automatically collected by Google Analytics, there are also the recommended events for games that can be explored to analyze user behavior. For our use case, to predict user churn, we counted the number of times the follow events were collected for a user within 24 hours of first user engagement: 

  • user_engagement
  • level_start_quickplay
  • level_end_quickplay
  • level_complete_quickplay
  • level_reset_quickplay
  • post_score
  • spend_virtual_currency
  • ad_reward
  • challenge_a_friend
  • completed_5_levels
  • use_extra_steps

The following query shows how these features were calculated:

  WITH
  events_first24hr AS (
    SELECT
      e.*
    FROM
      `firebase-public-project.analytics_153293282.events_*` e
    JOIN
      bqmlga4.returningusers r
      ON
        e.user_pseudo_id = r.user_pseudo_id
    WHERE
      TIMESTAMP_MICROS(e.event_timestamp) <= r.ts_24hr_after_first_engagement
  )
SELECT
  user_pseudo_id,
  SUM(IF(event_name = 'user_engagement', 1, 0)) AS cnt_user_engagement,
  # ... repeated for all behavior data ... 
  SUM(IF(event_name = 'use_extra_steps', 1, 0)) AS cnt_use_extra_steps,
FROM
  events_first24hr
GROUP BY
  1

View the notebook for the query used to aggregate and extract the behavioral data. You can use different sets of events for your use case. To view the complete list of events, use the following query:

  SELECT
    event_name,
    COUNT(event_name) as event_count
FROM
    `firebase-public-project.analytics_153293282.events_*`
GROUP BY 1
ORDER BY
   event_count DESC

After this we combined the features to ensure our training dataset reflects the intended structure. We had the following columns in our table:

  • User ID:
    • user_pseudo_id
  • Label:
    • churned
  • Demographic features
    • country
    • device_os
    • device_language
  • Behavioral features
    • cnt_user_engagement
    • cnt_level_start_quickplay
    • cnt_level_end_quickplay
    • cnt_level_complete_quickplay
    • cnt_level_reset_quickplay
    • cnt_post_score
    • cnt_spend_virtual_currency
    • cnt_ad_reward
    • cnt_challenge_a_friend
    • cnt_completed_5_levels
    • cnt_use_extra_steps
    • user_first_engagement

At this point, the dataset was ready to train the classification machine learning model in BigQuery ML. Once trained, the model will output a propensity score between churn (churned=1) or return (churned=0) indicating the probability of a user churning based on the training data.

Train classification model 

When using the CREATE MODEL statement, BigQuery ML automatically splits the data between training and test. Thus the model can be evaluated immediately after training (see the documentation for more information).

For the ML model, we can choose among the following classification algorithms where each type has its own pros and cons:

model

Often logistic regression is used as a starting point because it is the fastest to train. The query below shows how we trained the logistic regression classification models in BigQuery ML.

  CREATE OR REPLACE MODEL bqmlga4.churn_logreg
TRANSFORM(
  EXTRACT(MONTH from user_first_engagement) as month,
  EXTRACT(DAYOFYEAR from user_first_engagement) as julianday,
  EXTRACT(DAYOFWEEK from user_first_engagement) as dayofweek,
  EXTRACT(HOUR from user_first_engagement) as hour,
  * EXCEPT(user_first_engagement, user_pseudo_id)
)
OPTIONS(
  MODEL_TYPE="LOGISTIC_REG",
  INPUT_LABEL_COLS=["churned"]
) AS
SELECT
  *
FROM
  bqmlga4.train

We extracted month, julianday, and dayofweek  from datetimes/timestamps as one simple example of additional feature preprocessing before training. Using TRANSFORM() in your CREATE MODEL query allows the model to remember the extracted values. Thus, when making predictions using the model later on, these values won’t have to be extracted again. View the notebook for the example queries to train other types of models (XGBoost, deep neural network, AutoML Tables).

Evaluate model

Once the model finished training, we ran ML.EVALUATE to generate precision, recall, accuracy and f1_score for the model:

  SELECT
  *
FROM
  ML.EVALUATE(MODEL bqmlga4.churn_logreg)
row

The optional THRESHOLD parameter can be used to modify the default classification threshold of 0.5. For more information on these metrics, you can read through the definitions on precision and recall, accuracy, f1-score, log_loss and roc_auc. Comparing the resulting evaluation metrics can help to decide among multiple models.Furthermore, we used a confusion matrix to inspect how well the model predicted the labels, compared to the actual labels. The confusion matrix is created using the default threshold of 0.5, which you may want to adjust to optimize for recall, precision, or a balance (more information here).

  SELECT
  expected_label,
  _0 AS predicted_0,
  _1 AS predicted_1
FROM
  ML.CONFUSION_MATRIX(MODEL bqmlga4.churn_logreg)
expected

This table can be interpreted in the following way:

actual

Make predictions using BigQuery ML

Once the ideal model was available, we ran ML.PREDICT to make predictions. For propensity modeling, the most important output is the probability of a behavior occurring. The following query returns the probability that the user will return after 24 hrs. The higher the probability and closer it is to 1, the more likely the user is predicted to return, and the closer it is to 0, the more likely the user is predicted to churn.

  SELECT
  user_pseudo_id,
  returned,
  predicted_returned,
  predicted_returned_probs[OFFSET(0)].prob as probability_returned
FROM
  ML.PREDICT(MODEL bqmlga4.churn_logreg,
  (SELECT * FROM bqmlga4.train)) #can be replaced with a proper test dataset

Utilize predictions for activation

Once the model predictions are available for your users, you can activate this insight in different ways. In our analysis, we used user_pseudo_id as the user identifier. However, ideally, your app should send back the user_id from your app to Google Analytics. In addition to using first-party data for model predictions, this will also let you join back the predictions from the model into your own data.

  • You can import the model predictions back into Google Analytics as a user attribute. This can be done using the Data Import feature for Google Analytics 4. Based on the prediction values you can Create and edit audiences and also do Audience targeting. For example, an audience can be users with prediction probability between 0.4 and 0.7, to represent users who are predicted to be “on the fence” between churning and returning.
  • For Firebase Apps, you can use the Import segments feature. You can tailor user experience by targeting your identified users through Firebase services such as Remote Config, Cloud Messaging, and In-App Messaging. This will involve importing the segment data from BigQuery into Firebase. After that you can send notifications to the users, configure the app for them, or follow the user journeys across devices.
  • Run targeted marketing campaigns via CRMs like Salesforce, e.g. send out reminder emails.

You can find all of the code used in this blogpost in the Github repository:

https://github.com/GoogleCloudPlatform/analytics-componentized-patterns/tree/master/gaming/propensity-model/bqml

What’s next? 

Continuous model evaluation and re-training

As you collect more data from your users, you may want to regularly evaluate your model on fresh data and re-train the model if you notice that the model quality is decaying.

Continuous evaluation—the process of ensuring a production machine learning model is still performing well on new data—is an essential part in any ML workflow. Performing continuous evaluation can help you catch model drift, a phenomenon that occurs when the data used to train your model no longer reflects the current environment. 

To learn more about how to do continuous model evaluation and re-train models, you can read the blogpost: Continuous model evaluation with BigQuery ML, Stored Procedures, and Cloud Scheduler

More resources

If you’d like to learn more about any of the topics covered in this post, check out these resources:

Or learn more about how you can use BigQuery ML to easily build other machine learning solutions:

Let us know what you thought of this post, and if you have topics you’d like to see covered in the future! You can find us on Twitter at @polonglin and @_mkazi_.Thanks to reviewers: Abhishek Kashyap, Breen Baker, David Sabater Dinter.

How-to

Transforming Media Industry: Three Strategies for Media Leaders to Leverage Generative AI

1524

Of your peers have already read this article.

3:30 Minutes

The most insightful time you'll spend today!

As the media industry continues to evolve, generative AI is emerging as a powerful tool for innovation and growth. Here are five key strategies for media leaders to leverage this technology and stay ahead of the game. Know more...

The digital era turned the traditional formula for media and entertainment success on its head, ushering in new technologies that have changed how content is produced, distributed, experienced, and monetized. Audiences have more choice, flexibility, and power over what they consume, and today’s media companies have to embrace ongoing transformation or risk falling behind – or becoming irrelevant. 

A new wave of transformation is arriving with generative AI, a type of artificial intelligence that can interact with users in natural language and create novel data, ranging from story outlines, reports, and other text outputs to multimodal content like images, videos, and audio. Media and entertainment are inherently about content creation and creativity—so what does this new technology mean for the industry? 

At Google Cloud, we see tremendous opportunity for creative industries, from more efficient creation methods to improved user experiences. Let’s explore.

AI for media with Google Cloud

Google Cloud has a long history with large language models (LLMs) and other generative AI technologies—from their influence over the years on products like Document AI, to recent announcements like Generative AI support in Vertex AI, which lets businesses access and tune generative AI foundation models, and Generative AI App Builder, which lets developers build chatbots and other generative apps in minutes.  

Build, tune and deploy foundation models with Vertex AI

We’ve helped our global media and entertainment customers with AI for personalization, search and recommendations, predictive analytics, and much more — and with generative AI now on the rise, we have some ideas to help media leaders, technologists, and creators think about and prepare to utilize powerful AI in their work. 

Three lenses on innovation in media

The media and entertainment industry is increasingly diverse and complex, with companies spanning over-the-top (OTT) subscription streaming services, 24-hour linear channels, live broadcasts of sporting events, digital journalism, traditional publishing, short-form user-generated social video, and more. More and more, the boundaries between these segments of the media industry are blurring — but common to them all is the focus on providing compelling content in an engaging audience experience that can be directly or indirectly monetized.

With this in mind, we suggest media and entertainment companies look at the application of innovative technologies like generative AI through the following three lenses:

  1. Improving content creation, production, and management
  2. Enhancing and personalizing audience experiences
  3. Improving monetization

Improving content creation, production, and management

Generative AI democratizes many aspects of content creation, opening new ways to create written material, illustrations, sound effects, special effects, and more. Its recent maturation has been so rapid, some in the media industry have expressed concern that generative AI implies the end of creative professions. We think the opposite is more likely: just as photography, audio recordings, and computer generated images have enabled new modes of creativity, rather than making old ones obsolete, generative AI has the potential to both enable new forms of expression and enhance familiar ones. 

For example, journalists could use generative AI to speed up research by helping them synthesize and analyze large volumes of information, or to help them create initial drafts or summaries of editorial content. Film and television producers could leverage the technology to accelerate the post-production editing process, with new AI-enabled interfaces for rapidly adjusting or enhancing scene details such as lighting and color. Broadcasters could use generative AI to make vast libraries of video footage searchable and accessible for use in telling more compelling stories. The potential use cases go on and on.

Far from undermining incredible creative professions, generative AI is poised to free writers, artists, editors, and many others from the tedious and mundane aspects of their work, empowering them to focus more of their time on creativity.

Enhancing and personalizing audience experiences

Every media organization in the world today faces the reality that for most consumers, switching costs are extremely low. This puts incredible pressure on these companies to invest in delivering low-friction and compelling audience experiences that help mitigate subscribers from churning and viewers from abandoning content experiences for competitive platforms. 

Generative AI can help media companies engage and retain viewers, such as by enabling more powerful search and recommendations on their digital content platforms. With its increasingly multimodal capabilities extending from natural language to both audio and video content, generative AI is well-positioned to power more personalized audience experiences. 

Consumers often complain about “the paradox of choice” or their inability to find something interesting to watch on streaming platforms that have incredibly vast libraries of content available on demand. Imagine a not-too-distant future wherein a consumer can simply ask the content platform they’re using to help them find a specific show to watch based on mood, specific types of scenes, combinations of actors, award nominations, or practically anything they can think to ask. And that’s just the tip of the iceberg — imagine generative AI’s potential to curate, assemble, and even create personalized content for a viewer to consume!

Improving monetization

As consumers’ content consumption further expands from traditional theatrical and linear television programming to include digital offerings across an array of platforms, devices, and content types, media companies face the challenge of maintaining and improving monetization. The conventional economics and approaches to advertising and subscription models are proving, in many cases, not to deliver sufficient ROI. 

Generative AI has the potential to help media companies improve their monetization of audience experiences. As mentioned previously, enhanced personalization can play a role in mitigating churn, which in turn can help sustain and grow subscription and advertising revenues. Going beyond this, generative AI can be leveraged to drive even greater advertising revenues via more targeted, contextual, and personalized advertisements. Imagine both display and video advertisements that are generated on the fly to personalize product specifics, messaging, style, colors, and innumerable other characteristics to drive greater engagement and higher click-through rates (CTR), and thus higher advertising CPMs (cost per thousand impressions).

Coming up next

Generative AI presents a significant opportunity for media companies to fundamentally transform content creation, engagement, and monetization. Compelling services are already on the market — but there is far more to come. 

Google Cloud continues to build on its deep experience and expertise with AI, and we are committed to working with the industry to develop compelling, accessible, trusted, and responsible AI solutions that will drive meaningful business outcomes. We are excited to create the future together with our global media customers and partners across the ecosystem. To learn more about this disruptive topic, read “Debunking five generative AI misconceptions” from Google Cloud vice president of AI & Business Solutions Phil Moyer, or explore our Trusted Tester Program for generative AI.

E-book

Driving Digital Success: Three ROI Criteria for Competitive Advantage

DOWNLOAD E-BOOK

3940

Of your peers have already downloaded this article

6.00 Minutes

The most insightful time you'll spend today!

Choosing ROI criteria that drive smart decisions about digital investments is necessary to win in the app economy. But according to an Apigee Institute survey of IT and Marketing executives, there is a wide gap between what corporate leaders believe are the best ROI metrics and what are actually used in most enterprises today. Those who move first to bridge the gap will be able to move further and faster towards digital transformation and market leadership.

This special report drills down on patterns and practices for digital ROI that drive more confident decision making and stronger results based on empirical analysis of 200 large companies. Read the report to understand how you can drive decisions about digital investments to the next level of actionability and strengthen digital capabilities by building stronger organizational alignment. The report explores how top performing digital businesses evaluate and make decisions about digital investments using data analytics, by deploying apps, and operating APIs.

3338

Of your peers have already watched this video.

25:30 Minutes

The most insightful time you'll spend today!

How-to

Building Globally Scalable Services with Istio and ASM

Building distributed applications is hard! Building globally scalable distributed applications is harder. Maintaining and growing these services as your business grows is even harder.

Learn how to create a globally scalable platform for your business on Google Cloud using service meshes. See how to build a platform on Google Cloud from the ground up.

Blog

Streamlining Workflow Executions: Using Cloud Tasks in Google Cloud

892

Of your peers have already read this article.

3:30 Minutes

The most insightful time you'll spend today!

Discover how to effectively use Cloud Tasks in Google Cloud's Workflows for optimized performance during traffic surges. This guide covers creating, deploying, and executing workflows, as well as managing execution rates to stay within quota limits.

Introduction

In my previous post, I talked about how you can use a parent workflow to execute child workflows in parallel for faster overall processing time and easier detection of errors. Another useful pattern is to use a Cloud Tasks queue to create Workflows executions and that’s the topic of this post.

When your application experiences a sudden surge of traffic, it’s natural to want to handle the increased load by creating a high number of concurrent workflow executions. However, Google Cloud’s Workflows enforces quotas to prevent abuse and ensure fair resource allocation. These quotas limit the maximum number of concurrent workflow executions per region, per project, for example, Workflows currently enforces a maximum of 2000 concurrent executions by default. Once this limit is reached, any new executions beyond the quota will fail with an HTTP 429 error.

A Cloud Tasks queue can help. Rather than creating Workflow executions directly, you can add Workflows execution tasks to the Cloud Tasks queue and let Cloud Tasks drain the queue at a rate that you define. This allows for better utilization of your workflow quota and ensures the smooth execution of workflows.

https://storage.googleapis.com/gweb-cloudblog-publish/images/0_workflows_with_queue.max-1300x1300.png

Let’s dive into how to set this up. 

Create a Cloud Tasks queue

We’ll start by creating a Cloud Tasks queue. The Cloud Tasks queue acts as a buffer between the parent workflow and the child workflows, allowing us to regulate the rate of executions.

Create the Cloud Tasks queue (initially with no dispatch rate limits) with the desired name and location:

QUEUE=queue-workflow-child
LOCATION=us-central1

gcloud tasks queues create $QUEUE --location=$LOCATION

Now that we have our queue in place, let’s proceed to set up the child workflow.

Create and deploy a child workflow

The child workflow performs a specific task and returns a result to the parent workflow.

Create workflow-child.yaml to define the child workflow:

main:

  params: [args]

  steps:

    - init:

        assign:

          - iteration: ${args.iteration}

    - wait:

        call: sys.sleep

        args:

            seconds: 10

    - return_message:

        return: ${"Hello world" + iteration}

In this example, the child workflow receives an iteration argument from the parent workflow, simulates work by waiting for 10 seconds, and returns a string as the result.

Deploy the child workflow:

gcloud workflows deploy workflow-child --source=workflow-child.yaml --location=$LOCATION

Create and deploy a parent workflow

Next, create a parent workflow in workflow-parent.yaml.

The workflow assigns some constants first. Note that it’s referring to the child workflow and the queue name between the parent and child workflows:

main:
  steps:
    - init:
        assign:
          - project_id: ${sys.get_env("GOOGLE_CLOUD_PROJECT_ID")}
          - project_number: ${sys.get_env("GOOGLE_CLOUD_PROJECT_NUMBER")}
          - location: ${sys.get_env("GOOGLE_CLOUD_LOCATION")}
          - workflow_child_name: "workflow-child"
          - queue_name: "queue-workflow-child"

In the next step, Workflows creates and adds a high number of tasks (whose body is an HTTP request to execute the child workflow) to the Cloud Tasks queue:

- enqueue_tasks_to_execute_child_workflow:
        for:
          value: iteration
          range: [1, 100]
          steps:
              - iterate:
                  assign:
                    - data:
                        iteration: ${iteration}
                    - exec:
                        # Need to wrap into argument for Workflows args.
                        argument: ${json.encode_to_string(data)}
              - create_task_to_execute_child_workflow:
                  call: googleapis.cloudtasks.v2.projects.locations.queues.tasks.create
                  args:
                      parent: ${"projects/" + project_id + "/locations/" + location + "/queues/" + queue_name}
                      body:
                        task:
                          httpRequest:
                            body: ${base64.encode(json.encode(exec))}
                            url: ${"https://workflowexecutions.googleapis.com/v1/projects/" + project_id + "/locations/" + location + "/workflows/" + workflow_child_name + "/executions"}
                            oauthToken:
                              serviceAccountEmail: ${project_number + "-compute@developer.gserviceaccount.com"}

Note that task creation is a non-blocking call in Workflows. Cloud Tasks takes care of running those tasks to execute child workflows asynchronously.

Deploy the parent workflow:

gcloud workflows deploy workflow-parent --source=workflow-parent.yaml --location=$LOCATION

Execute the parent workflow with no dispatch rate limits

Time to execute the parent workflow:

gcloud workflows run workflow-parent --location=$LOCATION

As the parent workflow is running, you can see parallel executions of the child workflow, all executed roughly around the same:

https://storage.googleapis.com/gweb-cloudblog-publish/images/1_parallel_executions_allsametime.max-1000x1000.png

In this case, 100 executions is a well under the concurrency limit for Workflows. Quota issues may arise if you submit 1000s of executions all at once. This is when Cloud Tasks queue and its rate limits become useful.

Execute the parent workflow with dispatch rate limits

Let’s now apply a rate limit to the Cloud Tasks queue. In this case, 1 dispatch per second:

gcloud tasks queues update $QUEUE --max-dispatches-per-second=1 --location=$LOCATION

Execute the parent workflow again:

gcloud workflows run workflow-parent --location=$LOCATION

This time, you see a more smooth execution rate (1 execution request per second):

https://storage.googleapis.com/gweb-cloudblog-publish/images/2_parallel_executions_buffered.max-900x900.png

Summary

By introducing a Cloud Tasks queue before executing a workflow and playing with different dispatch rates and concurrency settings, you can better utilize your Workflows quota and stay below the limits without triggering unnecessary quota related failures. 

Check out the Buffer HTTP requests with Cloud Tasks codelab, if you want to get more hands-on experience with Cloud Tasks. As always, feel free to contact me on Twitter @meteatamel for any questions or feedback.

Blog

Develop for Compute Engine in your IDE with Cloud Code

2810

Of your peers have already read this article.

2:30 Minutes

The most insightful time you'll spend today!

Experience seamless development with Compute Engine using Cloud Code's integration. Discover how Cloud Code simplifies managing virtual machines, SSH connections, file uploads, and log viewing, directly from your favorite IDE.

When developing services with Compute Engine, our customizable compute service that lets you create and run virtual machines on Google’s infrastructure, you’ll likely find yourself frequently switching between your code editor, terminal, and the Google Cloud Console.

Cloud Code is a set of IDE plugins for popular IDEs like VS Code and IntelliJ that make it easier to develop applications that use Google Cloud services. And now, Cloud Code makes it easy to develop with Compute Engine by incorporating common workflows with your favorite IDE’s user interface.

Specifically, this new integration between Compute Engine and Cloud Code makes it easier to manage your commonly used virtual machines in the IDE, view details about them, connect to them over SSH, upload your application files to them, and view their logs.

Before you begin

Let’s demonstrate how the new integration works in Cloud Code for VS Code. Install Cloud Code for VS Code, and once installed, open its icon on the activity bar on the left and find “Compute Engine”:

Cloud Code for Jetbrains IDEs (such as IntelliJ) could be installed similarly, and you will find Compute Engine in the list of your IDE tool windows.

View your VMs

Cloud Code makes it easy to see all relevant VMs in your GCP project and view details needed to effectively work with the VM from the IDE. To start working with a Compute Engine VM in the IDE, navigate to Cloud Code’s new Compute Engine explorer. From there, you can see all the VMs in your current Cloud project. Clicking on a VM will display details such as machine type, boot image, IP address and more. You can also right click on a VM for a quick link to the Google Cloud Console where you can take additional action.

Connect to your VMs over SSH

Once you’ve found the VM you want to work with, Cloud Code makes it easy to connect to that VM over SSH. Again, find the VM you want to connect to in Cloud Code’s Compute Engine explorer, right click it, and select “Open SSH”. Cloud Code will then establish an SSH connection from your IDEs terminal into the VM. If there’s any difficulty establishing a connection, Cloud Code can run a troubleshooting diagnostic to help resolve the issue.

Many organizations maintain VMs that don’t have a public IP address, making it difficult to establish an SSH connection to them. For those VMs that use Identity-Aware Proxy, Cloud Code can still securely connect to them over SSH, even without a public IP address.

Upload files to your VMs

You might want to try a debug version of your application, run a script, or try a new code in an environment identical to production, in this case on a development VM instance which might not have access to full source code or is not a part of your CI/CD pipeline. Cloud Code provides an easy way to upload your code files into a VM instance.

Find the VM you want to connect to in Cloud Code’s Compute Engine explorer, right click it, and select “Upload File via SCP”. Choose a file from your local system and Cloud Code will upload it to a VM instance using SCP. Once upload completes, Cloud Code offers to open a new SSH connection to access the files and work with them on a remote VM instance. Again, if there’s any difficulty establishing a connection, Cloud Code can run a troubleshooting diagnostic to help resolve the issue.

View your VM logs

As you’re working with your VM, you can right click it and select to view the VM instance logs. From Visual Studio Code this will open a logs viewer in the IDE. From IntelliJ, the logs viewer in the Cloud Console will be opened. If you’ve configured application logs to be collected with Cloud Logging, you can also view those in these logs viewers as well.

Get Started

We invite you to try out Compute Engine with Cloud Code to better streamline your development workflow. To learn more, check out the Compute Engine documentation for Visual Studio Code and JetBrains IDEs. If you’re new to development with IDEs, you can take the first step by installing Visual Studio Code or IntelliJ.

More Relevant Stories for Your Company

Whitepaper

Google is the Top Provider for Continuous Integration Tools, According to Forrester

Google’s continuous integration (CI) and continuous delivery (CD) platform, Cloud Build, emerges as a Leader for Continuous Integration. “Google Cloud Build comes out swinging, going toe to toe with other cloud giants. Google Cloud Build is relatively new when compared to the other public cloud CI offerings, they had a

77% of app development supporting digital business will occur in-house.

E-book

Best Practices for Crafting Interfaces that Developers Love

Web APIs use HTTP, by definition. In the early days of web APIs, people spent a lot of time and effort figuring out how to implement the features of previous-generation distributed technologies like CORBA and DCOM on top of HTTP. This led to technologies like SOAP and WSDL. Experience showed

“So far it’s been good, especially the elasticity around how we can configure our Jenkins workloads on that Kubernetes shared cluster. That is the win we were pushing for.” Micheal Benedict, Product Manager for the Cloud and the Data Infrastructure at Pinterest

Blog

Transforming the Bank of Anthos App with Spring Cloud GCP 4.0

We’re excited to announce that Spring Cloud GCP version 4.0 is now generally available! In this post, we’ll be describing what the new major version has to offer, and demonstrating the process of using the migration guide on one of our reference architectures, Bank of Anthos. What’s new? With this release, Spring Cloud

Case Study

Google Maps Platform Can Elevate FinTech Experience with Less Risks and Higher Security

The financial services industry is changing—an estimated $68 trillion in wealth transferring from baby boomers to millennials.1 This means financial service providers will have to deliver the speed, ease-of-use, technological sophistication, and tailored services that millennials have come to expect. In fact, half of all millennials are willing to switch to

SHOW MORE STORIES

We use cookies to improve your experience. You accept to receive communications from GetIT Comms.   |   Cookie Policy  

Got It