Streamlining Workflow Executions: Using Cloud Tasks in Google Cloud
890
Of your peers have already read this article.
3:30 Minutes
The most insightful time you'll spend today!
Introduction
In my previous post, I talked about how you can use a parent workflow to execute child workflows in parallel for faster overall processing time and easier detection of errors. Another useful pattern is to use a Cloud Tasks queue to create Workflows executions and that’s the topic of this post.
When your application experiences a sudden surge of traffic, it’s natural to want to handle the increased load by creating a high number of concurrent workflow executions. However, Google Cloud’s Workflows enforces quotas to prevent abuse and ensure fair resource allocation. These quotas limit the maximum number of concurrent workflow executions per region, per project, for example, Workflows currently enforces a maximum of 2000 concurrent executions by default. Once this limit is reached, any new executions beyond the quota will fail with an HTTP 429 error.
A Cloud Tasks queue can help. Rather than creating Workflow executions directly, you can add Workflows execution tasks to the Cloud Tasks queue and let Cloud Tasks drain the queue at a rate that you define. This allows for better utilization of your workflow quota and ensures the smooth execution of workflows.
Let’s dive into how to set this up.
Create a Cloud Tasks queue
We’ll start by creating a Cloud Tasks queue. The Cloud Tasks queue acts as a buffer between the parent workflow and the child workflows, allowing us to regulate the rate of executions.
Create the Cloud Tasks queue (initially with no dispatch rate limits) with the desired name and location:
QUEUE=queue-workflow-child LOCATION=us-central1 gcloud tasks queues create $QUEUE --location=$LOCATION
Now that we have our queue in place, let’s proceed to set up the child workflow.
Create and deploy a child workflow
The child workflow performs a specific task and returns a result to the parent workflow.
Create workflow-child.yaml to define the child workflow:
main:
params: [args]
steps:
- init:
assign:
- iteration: ${args.iteration}
- wait:
call: sys.sleep
args:
seconds: 10
- return_message:
return: ${"Hello world" + iteration}In this example, the child workflow receives an iteration argument from the parent workflow, simulates work by waiting for 10 seconds, and returns a string as the result.
Deploy the child workflow:
gcloud workflows deploy workflow-child --source=workflow-child.yaml --location=$LOCATION
Create and deploy a parent workflow
Next, create a parent workflow in workflow-parent.yaml.
The workflow assigns some constants first. Note that it’s referring to the child workflow and the queue name between the parent and child workflows:
main:
steps:
- init:
assign:
- project_id: ${sys.get_env("GOOGLE_CLOUD_PROJECT_ID")}
- project_number: ${sys.get_env("GOOGLE_CLOUD_PROJECT_NUMBER")}
- location: ${sys.get_env("GOOGLE_CLOUD_LOCATION")}
- workflow_child_name: "workflow-child"
- queue_name: "queue-workflow-child"In the next step, Workflows creates and adds a high number of tasks (whose body is an HTTP request to execute the child workflow) to the Cloud Tasks queue:
- enqueue_tasks_to_execute_child_workflow:
for:
value: iteration
range: [1, 100]
steps:
- iterate:
assign:
- data:
iteration: ${iteration}
- exec:
# Need to wrap into argument for Workflows args.
argument: ${json.encode_to_string(data)}
- create_task_to_execute_child_workflow:
call: googleapis.cloudtasks.v2.projects.locations.queues.tasks.create
args:
parent: ${"projects/" + project_id + "/locations/" + location + "/queues/" + queue_name}
body:
task:
httpRequest:
body: ${base64.encode(json.encode(exec))}
url: ${"https://workflowexecutions.googleapis.com/v1/projects/" + project_id + "/locations/" + location + "/workflows/" + workflow_child_name + "/executions"}
oauthToken:
serviceAccountEmail: ${project_number + "-compute@developer.gserviceaccount.com"}Note that task creation is a non-blocking call in Workflows. Cloud Tasks takes care of running those tasks to execute child workflows asynchronously.
Deploy the parent workflow:
gcloud workflows deploy workflow-parent --source=workflow-parent.yaml --location=$LOCATION
Execute the parent workflow with no dispatch rate limits
Time to execute the parent workflow:
gcloud workflows run workflow-parent --location=$LOCATION
As the parent workflow is running, you can see parallel executions of the child workflow, all executed roughly around the same:
In this case, 100 executions is a well under the concurrency limit for Workflows. Quota issues may arise if you submit 1000s of executions all at once. This is when Cloud Tasks queue and its rate limits become useful.
Execute the parent workflow with dispatch rate limits
Let’s now apply a rate limit to the Cloud Tasks queue. In this case, 1 dispatch per second:
gcloud tasks queues update $QUEUE --max-dispatches-per-second=1 --location=$LOCATION
Execute the parent workflow again:
gcloud workflows run workflow-parent --location=$LOCATION
This time, you see a more smooth execution rate (1 execution request per second):
Summary
By introducing a Cloud Tasks queue before executing a workflow and playing with different dispatch rates and concurrency settings, you can better utilize your Workflows quota and stay below the limits without triggering unnecessary quota related failures.
Check out the Buffer HTTP requests with Cloud Tasks codelab, if you want to get more hands-on experience with Cloud Tasks. As always, feel free to contact me on Twitter @meteatamel for any questions or feedback.
Google Cloud’s Suite of DevOps Speeds Up ForgeRock’s Development
5680
Of your peers have already read this article.
3:00 Minutes
The most insightful time you'll spend today!
Editor’s note: Today we hear from ForgeRock, a multinational identity and access management software company with more than 1,100 enterprise customers, including a major public broadcaster. In total, customers use the ForgeRock Identity Platform to authenticate and log in over 45 million users daily, helping them manage identity, governance, and access management across all platforms, including on-premises and multicloud environments.
Operating at that kind of scale isn’t easy. In this blog post, ForgeRock Engineering Director, Warren Strange discusses the three things that help make their developers efficient and productive, and the Google Cloud tools they use along the way.
At ForgeRock, we’ve been an early adopter of Kubernetes, viewing it as a strategic platform. Running on Kubernetes allows us to drive multicloud support across Google Kubernetes Engine (GKE), Amazon (EKS), and Azure (AKS). So no matter which cloud our customers are running on, we are able to seamlessly integrate our products into customers’ environments.
Making it easier for ForgeRock’s developers and operators to build, deploy and manage applications has been crucial in our ability to continually provide high quality solutions for our customers. We’re always looking for tools to improve productivity and keep our developers focused on coding instead of configuration. Google Cloud’s suite of DevOps tools have streamlined three specific practices to help keep our developers productive:
1. Make developers productive within IDEs
Developer productivity is core to the success of any organization, including ForgeRock. Since developers spend most of their time within their IDE of choice, our goal at ForgeRock has been to make it easier for our developers to write Kubernetes applications within the IDEs they know and love. Cloud Code helps us precisely with that: it makes the process of building, deploying, scaling, and managing Kubernetes infrastructure and applications a breeze.
In particular, working with the Kubernetes YAML syntax and schema takes time, and a lot of trial and error to master. Thanks to YAML authoring support within Cloud Code, we can easily avoid the complicated and time consuming task of writing YAML files at ForgeRock. With YAML authoring support, developers save time on every bug. Cloud Code’s inline snippets, completions, and schema validation, a.k.a. “linting,” further streamline working with YAML files.
The benefits of Cloud Code extend to local development as well. Iterating locally on Kubernetes applications often requires multiple manual steps, including building container images, updating Kubernetes manifests, and redeploying applications. Doing these steps over and over again can be a chore. Cloud Code supports Skaffold under the hood, which tracks changes as they come and automatically rebuilds and redeploys—reducing repetitive development tasks.
Finally, developing for Kubernetes usually involves jumping between the IDE, documentation, samples etc. Cloud Code reduces this context switching with Kubernetes code samples. With samples, we can get new developers up and running quickly. They spend less time learning about configuration and management of the application—and spend more time on writing and evolving the code.
2. Drive end-to-end automation
To further improve developer productivity, we’ve focused on end-to-end automation: from writing code within IDEs, to automatically triggering CI/CD pipelines and running the code in production. In particular, Tekton, Cloud Build, Container Registry, and GKE have been critical to Forgerock as we streamline the flow of code, feedback and remediation through the build and deployment processes. The process looks something like this:
We begin by developing Kubernetes manifests and dockerfiles using Cloud Code. Then we use Skaffold to build containers locally, while Cloud Build helps with continuous integration (CI). The Cloud Build GitHub app allows us to automate builds and tests as part of our GitHub workflow. Cloud Build is differentiated from other continuous integration tools since it is fully serverless. It scales up and scales down in response to load, with no need for us to pre-provision servers or pay in advance for additional capacity. We pay for the exact resources we use.
Once the image is built by Cloud Build, it is stored, managed, and secured in Google’s Container Registry. Just like Cloud Build, Container Registry is serverless, so we only pay for what we use. Additionally, since Container Registry comes with automatic vulnerability scanning, every time we upload a new image to Container Registry, we can also scan it for vulnerabilities.
Next, a Tekton pipeline is triggered, which deploys the docker images stored in Container Registry and Kubernetes manifests to a running GKE cluster. Along with Cloud Build, Tekton is a critical part of our CI/CD process at ForgeRock. Most importantly, since Tekton comes with standardized Kubernetes-native primitives, we can create continuous delivery workflows very quickly.
After deployment, Tekton triggers a functional test suite to ensure that the applications we deploy perform as expected. The test results are posted to our team Slack channel so all developers have instant access and insights about each cluster. From there, we are able to provide our customers with their finished product request.
3. Leverage multicloud patterns and practices
The industry has seen a shift towards multicloud. Organizations have adopted multicloud strategies to minimize vendor lock-in, take advantage of best-in-class solutions, improve cost-efficiencies, and increase flexibility through choice.
At ForgeRock, we’re big proponents of multicloud. Part of that comes from the fact that our identity and access management product works across Google Cloud, AWS, and Azure. Developing products using open-source technologies such as Kubernetes has been particularly helpful in driving this interoperability.
Tekton has been another critical project that has allowed us to prevent vendor lock-in. Thanks to Tekton, our continuous delivery pipelines can deploy across any Kubernetes cluster. Most importantly, since Tekton pipelines run on Kubernetes, these pipelines can be decoupled from the runtime. Like Tekton and Kubernetes, both Cloud Build and Container Registry are based on open technologies. Community-contributed builders and official builder images allow us to connect to a variety of tools as a part of the build process. And finally, with support for open technologies like Google Cloud buildpacks within Cloud Build, we can build containers without even knowing Docker.
Making it easier for developers and operators to build, deploy and manage applications is critical for the success of any organization. Driving developer productivity within IDEs, leveraging end-to-end automation, and support for multi-cloud patterns and practices are just some of the ways we are trying to achieve this at ForgeRock. To learn more about ForgeRock, and to deploy the ForgeRock Identity Platform into your Kubernetes cluster, check out our open-source ForgeOps repository on GitHub.
3343
Of your peers have already watched this video.
17:30 Minutes
The most insightful time you'll spend today!
An App Modernization Story with Cloud Run
Back in 2016, an ASP.NET monolith app was deployed to IIS on Windows. While it worked, it was clunky in every sense of the word.
Over the years, the app was freed from Windows (thanks to .NET Core), containerized to run consistently in different environments (thanks to Docker) and decomposed into a set of loosely-coupled, event-driven, microservices (thanks to Cloud Run). The end result is a simpler and portable serverless architecture that’s much cheaper to run and maintain.
Watch the modernization journey, explore the decision points, and deep dive into the final architecture and code.
Google Maps Platform Can Elevate FinTech Experience with Less Risks and Higher Security
4727
Of your peers have already read this article.
3:00 Minutes
The most insightful time you'll spend today!
The financial services industry is changing—an estimated $68 trillion in wealth transferring from baby boomers to millennials.1 This means financial service providers will have to deliver the speed, ease-of-use, technological sophistication, and tailored services that millennials have come to expect. In fact, half of all millennials are willing to switch to a competing institution if it offers a better digital experience.2 This and many other trends are driving unprecedented growth for mobile Fintech experiences in banking, digital payments, financial management and insurance.3
Google Maps Platform financial services solutions
To help you respond to customer’s changing demands, we’re launching financial services solutions that can help you improve your customer experience, security and operations. We’ve outlined the technical guidance and APIs you need to build out three financial services solutions: Enriched Transactions, Quick and Verified Sign-up, and Branch and ATM Locator Plus. We’ve also highlighted two use cases that customers are using our APIs to solve: Contextual Experiences and Fraud Detection.
Clarify financial statements with Enriched Transactions solution
Transaction statements are often hard for customers to understand, using abbreviations like “ACMEHCORP” instead of customer-facing names like “Acme Houseware”. Our Enriched Transactions solution clarifies these transactions and makes them instantly recognizable by adding the merchant name and business category, a photo of the storefront, its location on a map, and full contact info. Making transactions easier to recognize not only boosts consumer confidence, with reported increases in NPS of 15% or higher, but decreases costly support calls by approximately 67%.4
In addition, you can help customers easily visualize a series of transactions by adding the merchant name to the transaction amount and date, and displaying their transactions on a Google map. This enables you to give customers insights about where and how they spend money. See the guide to implement Enriched Transactions today.
Enable faster sign-up with Quick and Verified Sign-up solution
Manually entered addresses can lead to lowered conversions, erroneous customer data, and costly delivery mistakes. Our Quick and Verified Sign-up solution makes sign-up faster, suggesting nearby addresses with just a few thumb taps—cutting sign-up time by up to 64% and increasing conversion rates by up to 15%.5
The solution also provides one additional level of address verification that helps reduce the risk of fraudulent account sign-ups—and companies have decreased fraudulent account setups by approximately 30% through using geospatial data to verify customer identities.4 See the Quick and Verified Sign-up solution guide to get started today.
- Faster sign-ups 1An application form requires an address
- Faster sign-ups 2Autocomplete quickly suggests addresses
- Faster sign-ups 3Select the address with visual confirmation
- Faster sign-ups 4Address verification options are presented
- Faster sign-ups 5Location permission is granted by the user
- Faster sign-ups 6The address is verified
Help customers visit you with Branch and ATM Locator Plus solution
74% of customers now search for specific details prior to their visit, which makes detailed, accurate profiles for each location a must.5 Our Branch and ATM Locator Plus solution enhances your own websites and apps with the same information shown about your branches and ATMs on Google Maps. Include hours of operation, available services, user reviews, photos of the location, driving directions and more.
Financial services companies using geospatial data to provide additional information (e.g. opening hours, available services, etc.) on branch and ATM services have seen a 14% increase in Net Promoter Score (NPS), and a 7% decrease in customer support calls.4 Implement Branch and ATM Locator Plus today using the guide or build it in minutes with Quick Builder.
- ATM locator 1Customers can enable location permissions, or enter their address
- ATM locator 2Quickly enter the address with Autocomplete
- ATM locator 3Nearby location listings, ranked by distance and ETA
- ATM locator 4Map view and directions
Enable offers and rewards with Contextual Experiences
Real-time, geo-targeted offers can power deals, rewards, and cash-back programs—all visualized with rich Google Maps. By combining the insights of purchase histories with customer opt-in to location-based features, companies can implement the Contextual Experiences use case to enable personalized offers and rewards programs that drive engagement with brands while putting money in customers’ pockets at the same time.This is a win for banks and their customers, validated by encouraging metrics like NPS rating boosts of 8% or higher, and an increase of 8% or more time spent in-app.4 Learn how Current uses Google Maps Platform to create innovative customer rewards programs with location intelligence.
Detect suspicious transactions with Fraud Detection
With the Fraud Detection use case, companies can use customer opted-in mobile device location to flag suspicious activity based on geographic distance, such as an ATM withdrawal that is far from the customer’s phone. Our APIs can also help companies recognize suspicious transaction patterns such as a purchase made at a location that is physically distant from a recent transaction.
Financial services companies that use geospatial data to verify customers’ identities have reduced fraudulent transactions by approximately 70%, and false positives in fraud detection by 45%, on average.4 Learn how Starling Bank uses Google Maps Platform to enable real-time notification of transactions and their locations, and enhance data-driven decision-making.
Start elevating customer experiences, reducing risk and increasing efficiency today with our financial services offerings. Visit our financial services solutions page to learn more about how to start implementing these solutions.
For more information on Google Maps Platform, visit our website.
Anthos Makes Hybrid and Multi-Cloud Deployments Easy
3656
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
Most enterprises have applications in disparate locations—in their own data centers, in multiple public clouds, and at the edge. These apps run on different proprietary technology stacks, which reduces developer velocity, wastes computing resources, and hinders scalability. How can you consistently secure and operate existing apps, while developing and deploying new apps across hybrid and multicloud environments? How can you get centralized visibility and management of the resources? Well, that is why Anthos exists!
This post explores why traditional hybrid and multicloud deployments are difficult, and then shows how Anthos makes it easy to manage applications across multiple environments.
Why is traditional hybrid and multicloud difficult?
In hybrid and multicloud environments, you need to manage infrastructure. Let’s say you use containers on the clouds, and you develop apps using services on Google Cloud and AWS. Regardless of environment, you will need policy enforcement across your IT footprint. To manage your apps across the environment, you need monitoring and logging systems. You need to integrate that data into meaningful categories, like business data, operational data, and alerts.
Digging further, you might use operational data and alerts to inform optimizations, implement automations, and set policies or SLOs. You might use business data to do all those things, and to deploy third-party apps. Then, to actually enact the changes you decide to implement, you need to act on different parts of the system. That means digging into each tool for policy enforcement, securing services, orchestrating containers, and managing infrastructure. Don’t forget, all of this work is in addition to what it takes to develop and deploy your own apps.
Now, consider repeating this set of tasks across a hybrid and multicloud landscape. It becomes very complex, very quickly. Your platform admins, SREs, and DevOps teams who are responsible for security and efficiency have to do manual, cluster-by-cluster management, data collection, and information synthesis. With this complexity, it’s hard to stay current, to understand business implications, and to ensure compliance (not to mention the difficulty of onboarding a new hire). Anthos helps solve these challenges!
How does Anthos make hybrid and multicloud easy?
With Anthos, you get a consistent way to manage your infrastructure, with similar infrastructure management, container management, service management, and policy enforcement across your landscape.
As a result, you have observability across all your platforms in one place, including access to business information, alerts, and operations information. With this information you might decide to optimize, automate, and set policies or SLOs.
Digging deeper into Anthos
Environs
You may have different regions that need different policies, and also have different development, staging, or production environments that need different permissions. Some parts of your work may need more security. That’s where environs come in! Environs are a way to create logical sets of underlying Kubernetes clusters, regardless of which platform those clusters live on.
By considering, grouping, and managing sets of clusters as logical environs, you can think about and work with your applications at the right level of detail for what you need to do, be it acquiring business insights over the entire system, updating settings for a dev environment, or troubleshooting data for a specific cluster. Using environs, each part of the functional stack can take declarative direction about configuration, compliance, and more.
Modernize application development
Anthos also helps modernize application development because it uses environs to enforce policies and processes, and abstracts away the cluster and container management from application teams. Anthos enables you to easily abstract away infrastructure from application teams, making it easy for them to incorporate a wide variety of CI/CD solutions on top of environs. It lets you view and manage your applications at the right level of detail, be it business insights for services across the entire system, or troubleshooting data for a specific cluster. Anthos also works with container-based tools like buildpacks to simplify the packaging process. It offers Migrate for Anthos to take those applications out of the VMs and move them to a more modern hosting environment.
What’s in it for platform administrators?
Anthos provides platform administrators a single place to monitor and manage their landscape, with policy control and marketplace access. This reduces person-hours needed for management, enforcement, discovery, and communication. Anthos also provides administrators an out-of-the-box structured view of their entire system, including services, clusters, and more, so they can improve security, use resources more efficiently, and demonstrate measurable success. Administrators also save time and effort by managing declaratively, and they can communicate the success, cost savings, and efficiency of the platforms without needing to manually combine data.
Interested in getting started with Anthos? Check out the free on-demand training here and my YouTube series.
For more resources, you can also read the Anthos ebook at no cost. For more #GCPSketchnote and similar cloud content, follow me on twitter @pvergadia and keep an eye out on thecloudgirl.dev
Customer Voices: How Firms from Across Industries Leverage Google Cloud
DOWNLOAD CASE STUDY14060
Of your peers have already downloaded this article
1:30 Minutes
The most insightful time you'll spend today!
From powering everyday operations and accelerating application innovation, to providing tools for specific business needs and executing on big ideas, to advancing the security of technology solutions, companies from across industries have leveraged Google Cloud for business benefits.
Companies from across industries have turned to Google Cloud for transforming their business, modernizing their infrastructure, and gleaning intelligence from data. For instance:
- Johnson & Johnson achieved a 41% increase in search results from high-quality job applicants, significantly improving the company’s ability to quickly hire top talent.
- Sony Network Communications now processes 10 billion monthly queries faster, which advances data analysis.
- University College Dublin saw significant 6-figure savings by eliminating legacy hardware, software, and maintenance.
And there are many such examples. Read the collection of case studies to find out how companies from across industries and geographies leveraged Google Cloud for measurable business benefits and for solving complex problems.
More Relevant Stories for Your Company
Cloud IoT Core Helps Businesses Leverage their IoT Data to Build a Competitive Edge
The ability to gain real-time insights from IoT data can redefine competitiveness for businesses. Intelligence allows connected devices and assets to interact efficiently with applications and with human beings in an intuitive and non-disruptive way. After your IoT project is up and running, many devices will be producing lots of
Google is the Top Provider for Continuous Integration Tools, According to Forrester
Google’s continuous integration (CI) and continuous delivery (CD) platform, Cloud Build, emerges as a Leader for Continuous Integration. “Google Cloud Build comes out swinging, going toe to toe with other cloud giants. Google Cloud Build is relatively new when compared to the other public cloud CI offerings, they had a
Neo4J & Google Cloud: Graph Data in Cloud to Address Challenges in FinServ Industry
Over the last decade, financial service organizations have been adopting a cloud-first mindset. According to InformationWeek, lower costs and enhanced scalability were the biggest drivers for cloud adoption in financial services, and cloud-native applications allow access to the latest technology and talent, enabling adopters to rebuild transaction processing systems capable of
Announcing Apigee’s Pay-as-you-go pricing to provide flexibility and scale seamlessly
Apigee is Google Cloud’s API management platform that enables organizations to build, operate, manage and monetize their APIs. Customers from industries around the world trust Apigee to build and scale their API programs. While some organizations operate with mature API-first strategies, others might still be working on a modernization strategy.
