Introduction

In my previous post, I talked about how you can use a parent workflow to execute child workflows in parallel for faster overall processing time and easier detection of errors. Another useful pattern is to use a Cloud Tasks queue to create Workflows executions and that’s the topic of this post.

When your application experiences a sudden surge of traffic, it’s natural to want to handle the increased load by creating a high number of concurrent workflow executions. However, Google Cloud’s Workflows enforces quotas to prevent abuse and ensure fair resource allocation. These quotas limit the maximum number of concurrent workflow executions per region, per project, for example, Workflows currently enforces a maximum of 2000 concurrent executions by default. Once this limit is reached, any new executions beyond the quota will fail with an HTTP 429 error.

A Cloud Tasks queue can help. Rather than creating Workflow executions directly, you can add Workflows execution tasks to the Cloud Tasks queue and let Cloud Tasks drain the queue at a rate that you define. This allows for better utilization of your workflow quota and ensures the smooth execution of workflows.

Let’s dive into how to set this up. 

Create a Cloud Tasks queue

We’ll start by creating a Cloud Tasks queue. The Cloud Tasks queue acts as a buffer between the parent workflow and the child workflows, allowing us to regulate the rate of executions.

Create the Cloud Tasks queue (initially with no dispatch rate limits) with the desired name and location:

QUEUE=queue-workflow-child
LOCATION=us-central1

gcloud tasks queues create $QUEUE --location=$LOCATION

Now that we have our queue in place, let’s proceed to set up the child workflow.

Create and deploy a child workflow

The child workflow performs a specific task and returns a result to the parent workflow.

Create workflow-child.yaml to define the child workflow:

main:

  params: [args]

  steps:

    - init:

        assign:

          - iteration: ${args.iteration}

    - wait:

        call: sys.sleep

        args:

            seconds: 10

    - return_message:

        return: ${"Hello world" + iteration}

In this example, the child workflow receives an iteration argument from the parent workflow, simulates work by waiting for 10 seconds, and returns a string as the result.

Deploy the child workflow:

gcloud workflows deploy workflow-child --source=workflow-child.yaml --location=$LOCATION

Create and deploy a parent workflow

Next, create a parent workflow in workflow-parent.yaml.

The workflow assigns some constants first. Note that it’s referring to the child workflow and the queue name between the parent and child workflows:

main:
  steps:
    - init:
        assign:
          - project_id: ${sys.get_env("GOOGLE_CLOUD_PROJECT_ID")}
          - project_number: ${sys.get_env("GOOGLE_CLOUD_PROJECT_NUMBER")}
          - location: ${sys.get_env("GOOGLE_CLOUD_LOCATION")}
          - workflow_child_name: "workflow-child"
          - queue_name: "queue-workflow-child"

In the next step, Workflows creates and adds a high number of tasks (whose body is an HTTP request to execute the child workflow) to the Cloud Tasks queue:

- enqueue_tasks_to_execute_child_workflow:
        for:
          value: iteration
          range: [1, 100]
          steps:
              - iterate:
                  assign:
                    - data:
                        iteration: ${iteration}
                    - exec:
                        # Need to wrap into argument for Workflows args.
                        argument: ${json.encode_to_string(data)}
              - create_task_to_execute_child_workflow:
                  call: googleapis.cloudtasks.v2.projects.locations.queues.tasks.create
                  args:
                      parent: ${"projects/" + project_id + "/locations/" + location + "/queues/" + queue_name}
                      body:
                        task:
                          httpRequest:
                            body: ${base64.encode(json.encode(exec))}
                            url: ${"https://workflowexecutions.googleapis.com/v1/projects/" + project_id + "/locations/" + location + "/workflows/" + workflow_child_name + "/executions"}
                            oauthToken:
                              serviceAccountEmail: ${project_number + "-compute@developer.gserviceaccount.com"}

Note that task creation is a non-blocking call in Workflows. Cloud Tasks takes care of running those tasks to execute child workflows asynchronously.

Deploy the parent workflow:

gcloud workflows deploy workflow-parent --source=workflow-parent.yaml --location=$LOCATION

Execute the parent workflow with no dispatch rate limits

Time to execute the parent workflow:

gcloud workflows run workflow-parent --location=$LOCATION

As the parent workflow is running, you can see parallel executions of the child workflow, all executed roughly around the same:

In this case, 100 executions is a well under the concurrency limit for Workflows. Quota issues may arise if you submit 1000s of executions all at once. This is when Cloud Tasks queue and its rate limits become useful.

Execute the parent workflow with dispatch rate limits

Let’s now apply a rate limit to the Cloud Tasks queue. In this case, 1 dispatch per second:

gcloud tasks queues update $QUEUE --max-dispatches-per-second=1 --location=$LOCATION

Execute the parent workflow again:

gcloud workflows run workflow-parent --location=$LOCATION

This time, you see a more smooth execution rate (1 execution request per second):

Summary

By introducing a Cloud Tasks queue before executing a workflow and playing with different dispatch rates and concurrency settings, you can better utilize your Workflows quota and stay below the limits without triggering unnecessary quota related failures. 

Check out the Buffer HTTP requests with Cloud Tasks codelab, if you want to get more hands-on experience with Cloud Tasks. As always, feel free to contact me on Twitter @meteatamel for any questions or feedback.

In an industry as highly regulated as healthcare, building a single secure and compliant application that tracks patient care and appointments at a clinic requires a great deal of planning from development and security teams. So, imagine what it would be like to build a solution that includes almost everything related to a patient’s healthcare, including insurance and billing. That’s what Highmark Health (Highmark)—a U.S. health and wellness organization that provides millions of customers with health insurance plans, a physician and hospital network, and a diverse portfolio of businesses–decided to do. 

Highmark is developing a solution called Living Health to re-imagine healthcare delivery, and it is using Google Cloud and the Google Cloud Professional Services Organization (PSO) to build and maintain the innovation platform supporting this forward thinking experience. Considering all the personal information that different parties like insurers, specialists, billers and coders, clinics, and hospitals share, Highmark must build security and compliance into every part of the solution. 

In this blog, we look at how Highmark Health and Google are using a technique called “secure-by-design” to address the security, privacy, and compliance aspects of bringing Living Health to life.

Secure-by-design: Preventive care for development

In healthcare, preventing an illness or condition is the ideal outcome. Preventive care often involves early intervention—a course of ideas and actions to ward off illness, permanent injury, and so on. Interestingly, when developing a groundbreaking delivery model like Living Health, it’s a good idea to take the same approach to security, privacy, and compliance. 

That’s why Highmark’s security and technology teams worked with their Google Cloud PSO team to implement secure-by-design for every step of design, development, and operations. Security is built into the entire development process rather than waiting until after implementation to reactively secure the platform or remediate security gaps. 

It’s analogous to choosing the right brakes for a car before it rolls off the assembly line instead of having an inspector shut down production because the car failed its safety tests. The key aspect of secure-by-design is an underlying application architecture created from foundational building blocks that sit on top of a secure cloud infrastructure. Secure-by-design works to ensure that these building blocks are secure and compliant before moving on to development.

The entire approach requires security, development, and cloud teams to work together with other stakeholders. Most importantly, it requires a cloud partner, cloud services, and a cloud infrastructure that can support it. 

Finding the right cloud and services for secure-by-design 

Highmark chose Google Cloud because of its leadership in analytics, infrastructure services, and platform as a service. In addition, Google Cloud has made strategic investments in healthcare interoperability and innovation, which was another key reason Highmark decided to work with Google. As a result, Highmark felt that Google Cloud and the Google Cloud PSO were best suited for delivering on the vision of Living Health—its security and its outcomes. 

“Google takes security more seriously than the other providers we considered, which is very important to an organization like us. Cloud applications and infrastructure for healthcare must be secure and compliant,” explains Highmark Vice President and Chief Information Security Officer, Omar Khawaja. 

Forming a foundation for security and compliance

How does security-by-design with services work? It starts with the creation and securing of the foundational platform, allowing teams to harden and enforce specified security controls. It’s a collaborative process that starts with input from cross-functional teams—not just technology teams—using terms they understand, so that everyone has a stake in the design. 

A strong data governance and protection program classifies and segments workloads based on risk and sensitivity. Teams build multiple layers of defense into the foundational layers to mitigate against key industry risks. Google managed services such as VPC Service Controls help prevent unauthorized access. Automated controls such as those in Data Loss Prevention help teams quickly classify data and identify and respond to potential sources of data risk. Automation capabilities help ensure that security policies are enforced.

After the foundational work is done, it’s time to assess and apply security controls to the different building blocks, which are Google Cloud services such as Google Kubernetes Engine, Google Compute Engine, and Google Cloud Storage. The goal is to make sure that these and similar building blocks, or any combination of them, do not introduce additional risks and to ensure any identified risks are remediated or mitigated. 

Enabling use cases, step by step

After the foundational security is established, the security-by-design program enables the Google Cloud services that developers then use to build use cases that form Living Health. The service enablement approach allows Highmark to address complexity by providing the controls most relevant for each individual service. 

For each service, the teams begin by determining the risks and the controls that can reduce them. The next step is enforcing preventive and detective controls across various tools. After validation, technical teams can be granted an authorization to operate, also called an ATO. An ATO authorizes the service for development in a use case.

For use cases with greater data sensitivity, the Highmark teams validate the recommended security controls with an external trust assessor, who uses the HITRUST Common Security Framework, which maps to certifications and compliance such as HIPAA, NIST, GDPR, and more. A certification process follows that can take anywhere from a few weeks to a few months. In addition to certification, there is ongoing monitoring of the environment for events, behavior, control effectiveness, and control lapses or any deviation from the controls.

The approach simplifies compliance for developers by abstracting compliance requirements away. The process provides developers a set of security requirements written in the language of the cloud, rather than in the language of compliance, providing more prescriptive guidance as they build solutions. Through the secure-by-design program, the Highmark technology and security teams, Google, the business, and the third-party trust assessor all contribute to a secure foundation for any architectural design with enabled Google Cloud services as building blocks. 

Beating the learning curve 

Thanks to the Living Health project, the Highmark technology and security teams are trying new methods. They are exploring new tools for building secure applications in the cloud. They are paying close attention to processes and the use case steps and, when necessary, aligning different teams to execute. Because everyone is working together collaboratively toward a shared goal, teams are delivering more things on time and with predictability, which has reduced volatility and surprises. 

The secrets to success: Bringing everyone to the table early and with humility

Together, Highmark and Google Cloud PSO have created over 24 secure-by-design building blocks by bringing everyone to the table early and relying on thoughtful, honest communication. Input for the architecture design produced for Highmark came from privacy teams, legal teams, security teams, and the teams that are building the applications. And that degree of collaboration ultimately leads to a much better product because everyone has a shared sense of responsibility and ownership of what was built. 

Delivering a highly complex solution like Living Health takes significant, more purposeful communication and execution. It is also important to be honest and humble. The security, technology, and Google teams have learned to admit when something isn’t working and to ask for help or ideas for a solution. The teams are also able to accept that they don’t have all the answers, and that they need to figure out solutions by experimenting. Khawaja puts it simply, “That level of humility has been really important and enabled us to have the successes that we’ve had. And hopefully that’ll be something that we continue to retain in our DNA.”

Google Cloud Results

• Reduced API management costs 34%
• Develops new features in days versus months
• Fixes issues quickly by writing code
• Manages 15B API calls per month without outages

Austin, Texas-based Bazaarvoice switched to Apigee for its API platform because of its competitive pricing and feature set. But another benefit soon became apparent to Bazaarvoice engineers: the ability to write code on the Apigee platform and “own our destiny,” says Kurtis White, Senior Development Manager in Research and Development for Bazaarvoice.

Bazaarvoice specializes in consumer-generated content marketing solutions that connect brands, retailers, and consumers. APIs do the heavy lifting in making those connections. For example, the Conversations API, the heart of the Bazaarvoice platform, lets the company’s retail clients aggregate and leverage users’ product reviews and ratings. With the Conversations API, API keys are passed along in each query string and enable Bazaarvoice customers to access the relevant customer data.

APIs are essential to the Bazaarvoice platform. During its early days the company didn’t use APIs, relying instead on direct communication links to connect its customers to what consumers were saying about them. “That approach just didn’t scale,” says White. “Without reliable APIs, it would be almost impossible for us to be in business today.”

Eight years ago, Bazaarvoice switched to APIs. From about 2013 to 2017, the company relied upon the same API management SaaS solutions provider. But the company was growing. The call volume that its APIs generated was expanding at a rapid pace, too—especially during the busy year-end shopping season. For example, in early October 2016, Bazaarvoice APIs generated nearly 9 billion calls, compared to about 12 billion calls in early December 2016.

Unfortunately, along with the pace of API calls, the costs associated with Bazaarvoice’s existing API solutions provider were also escalating—increasing 74% from early 2016 to early 2017. To keep rising expenses from stifling its own growth, and to continue offering its reliable, high-performing SaaS solution, the company’s IT team decided it was to time to find a different API solutions provider.

In April 2017, Bazaarvoice migrated to the Apigee API platform because of its feature set and, in particular, its competitive pricing. By switching to Apigee, Bazaarvoice reduced its API spend by 34%, White says.

A ticking clock

Migrating from its previous API provider to Apigee involved importing 66,000 API keys—an arduous task. Apigee engineers and business leaders worked directly with their Bazaarvoice counterparts to help ensure a successful transition. “Everyone at Apigee was very responsive to our needs,” White says. “If a problem came up, the Apigee team owned it and worked through it to give us a great experience.”

During the migration, there was a clock ticking in the background. Bazaarvoice wanted to shift 100% of its API calls to Apigee before the contract with its existing API provider came up for renewal. “I wasn’t sure we could make the deadline,” says White. “But we did, and I give a big credit to the Apigee technical team. They provided 24-hour support to our team. And once we went live, our platform was handling 15 billion API calls a month with no outages.”

Following the migration, Apigee support continues to “knock it out of the park,” he added. “Apigee is a great partner to have.”

Controlling destiny

The Bazaarvoice IT team didn’t specifically choose Apigee because of the ability to write code on the platform—but they quickly came to appreciate the feature.

“Our previous API provider was like a black box to us,” says White. “They had written some custom code for us that we depended upon. But we didn’t own that code. And if we needed changes, we had to pay them to do it for us.”

“Apigee allows us to write our own code that runs on the platform and looks at all of the API calls,” White continues. “We can put the code into a virtual control system. We can update it whenever we want to add a feature. Because of how important APIs are to our business, the ability to write code on Apigee lets us control our own destiny. It’s the Apigee number one benefit, in my opinion.” This also accelerates the time it takes Bazaarvoice to bring new features to market.

“Apigee gives us the flexibility to do things we wouldn’t have even tried before,” says White. “We can test new things easily, without getting locked into a service agreement with an external party. Before, it might have taken months to develop new product functionalities. Now it takes days.”

In addition, writing code on the Apigee platform enables the IT team to quickly address any problems that may arise. “If someone notices a problem, we can write the code to fix it right away,” White says.

“Because Apigee lets us iterate and fix problems quickly, our innovation has gone to the next level,” White added. “Before, if you had an idea, you’d wonder if you really wanted to invest $30,000 and three months in it. Probably not. Now, with Apigee, the investment is maybe one day and zero dollars.”

Experimentation pays off for Bazaarvoice and its customers. For example, during the 2017 holiday season, Bazaarvoice was able to give all its customers the capacity to handle twice the number of API calls.

“With our previous API solutions provider, it would have been such a long process to make that happen, it wouldn’t have been worth it,” says White. “But it was really easy to write the code in Apigee. Plus, we made our customers happy and kept them running smoothly during the peak holiday season—with little effort on our part.”

Personalization and monetization

Apigee has played a key role in enabling Bazaarvoice to develop new products, too. In March 2018, Bazaarvoice released a new personalization data product for its customers, built on top of an Apigee-powered API. The new product enables Bazaarvoice retail customers to recommend relevant products to their online shoppers. The products are personalized to the shoppers’ interests, based on their past behavior and what they’re shopping for.

Apigee is also serving as a strategic business advisor, helping Bazaarvoice determine the best way to price its offerings as its API call volumes continue to increase.

“We’re working with Apigee to build an industry-leading model for monetizing our APIs,” says White. “Apigee is the ideal advisor in this regard. We’re handling such large volume, the volume is continuing to grow, and Apigee enables us to power it.”