At enterprises across industries, documents are at the center of core business processes. Documents store a treasure trove of valuable information whether it’s a company’s invoices, HR documents, tax forms and much more. However, the unstructured nature of documents make them difficult to work with as a data source. We call this “dark data” or unstructured data that businesses collect, process and store but do not utilize for purposes such as analytics, monetization, etc. These documents in pdf or image formats, often trigger complex processes that have historically relied on fragmented technology and manual steps. With compute solutions on Google Cloud and Document AI, you can create seamless integrations and easy to use applications for your users. Document AI is a platform and a family of solutions that help businesses to transform documents into structured data backed by machine learning. In this blog post we’ll walk you through how to use Serverless technology to process documents with Cloud Functions, and with workflows of business processes orchestrating microservices, API calls, and functions, thanks to Workflows.

At Cloud Next 2021, we presented how to build easy AI-powered applications with Google Cloud. We introduced a sample application for handling incoming expense reports, analyzing expense receipts with Procurement Document AI, a DocAI solution for automating procurement data capture from forms including invoices, utility statements and more. Then organizing the logic of a report approval process with Workflows, and used Cloud Functions as glue to invoke the workflow, and do analysis of the parsed document.

We also open sourced the code on this Github repository, if you’re interested in learning more about this application.

In the above diagram, there are two user journeys: the employee submitting an expense report where multiple receipts are processed at once, and the manager validating or rejecting the expense report. 

First, the employee goes to the website, powered by Vue.js for the frontend progressive JavaScript framework and Shoelace for the library of web components. The website is hosted via Firebase Hosting. The frontend invokes an HTTP function that triggers the execution of our business workflow, defined using the Workflows YAML syntax. 

Workflows is able to handle long-running operations without any additional code required, in our case we are asynchronously processing a set receipt files. Here, the Document AI connector directly calls the batch processing endpoint for service. This API returns a long-running operation: if you poll the API, the operation state will be “RUNNING” until it has reached a “SUCCEEDED” or “FAILED” state. You would have to wait for its completion. However, Workflows’ connectors handle such long-running operations, without you having to poll the API multiple times till the state changes. Here’s how we call the batch processing operation of the Document AI connector:

  - invoke_document_ai:
    call: googleapis.documentai.v1.projects.locations.processors.batchProcess
    args:
        name: ${"projects/" + project + "/locations/eu/processors/" + processorId}
        location: "eu"
        body:
            inputDocuments:
                gcsPrefix:
                    gcsUriPrefix: ${bucket_input + report_id}
            documentOutputConfig:
                gcsOutputConfig: 
                    gcsUri: ${bucket_output + report_id}
            skipHumanReview: true
    result: document_ai_response

Machine learning uses state of the art Vision and Natural Language Processing models to intelligently extract schematized data from documents with Document AI. As a developer, you don’t have to figure out how to fine tune or reframe the receipt pictures, or how to find the relevant field and information in the receipt. It’s Document AI’s job to help you here: it will return a JSON document whose fields are: line_item, currency, supplier_name, total_amount, etc. Document AI is capable of understanding standardized papers and forms, including invoices, lending documents, pay slips, driver licenses, and more.

A cloud function retrieves all the relevant fields of the receipts, and makes its own tallies, before submitting the expense report for approval to the manager. Another useful feature of Workflows is put to good use: Callbacks, that we introduced last year. In the workflow definition we create a callback endpoint, and the workflow execution will wait for the callback to be called to continue its flow, thanks to those two instructions:

  - create_callback:
    call: events.create_callback_endpoint
    args:
        http_callback_method: "POST"
    result: callback_details
...
- await_callback:
    try:
        call: events.await_callback
        args:
            callback: ${callback_details}
            timeout: 3600
        result: callback_request
    except:
        as: e
        steps:
            - update_status_to_error:
              ...

In this example application, we combined the intelligent capabilities of Document AI to transform complex image documents into usable structured data, with Cloud Functions for data transformation, process triggering, and callback handling logic, and Workflows enabled us to orchestrate the underlying business process and its service call logic.

Going further 

If you’re looking to make sense of your documents, turning dark data into structured information, be sure to check out what Document AI offers. You can also get your hands on a codelab to get started quickly, in which you’ll get a chance at processing handwritten forms. If you want to explore Workflows, quickstarts are available to guide you through your first steps, and likewise, another codelab explores the basics of Workflows. As mentioned earlier, for a concrete example, the source code of our smart expense application is available on Github. Don’t hesitate to reach out to us at @glaforge and @asrivas_dev to discuss smart scalable apps with us.

API gateways are a critical component of a modern architecture. Apigee X is Google Cloud’s API management platform. It also allows users of the legacy Apigee Edge product to leverage Virtual Private Cloud (VPC) products and features, like Cloud Armor and Load Balancers. With these new security and availability advantages come a new set of challenges. The biggest challenge is accessing API backends in multiple VPCs. This introduces  VPC peering transitivity restrictions. Only two VPCs can be peered in sequence, but the Apigee X Runtime already consumes one peering connection. This blog post discusses overcoming transitivity limits, with an eye on future managed services that solve the problem.

VPC Peering Transitivity Limits

The main challenge with Apigee X is that customers have backend servers in multiple distinct VPCs. This means that Apigee X Runtime traffic must traverse multiple VPC peering  connections. However, peering transitivity limits VPC networking to only one sequential VPC hop.

In the past (Apigee Edge / SaaS) customers did not need to worry about securing their backend APIs across multiple VPCs. Apigee X networking is a new feature, compared to the original Apigee SaaS product.

VPC Connectivity Options

VPC peering is the easiest option. The restriction is that connectivity wise, only one peering hop is allowed, which is already consumed by a connection to the Apigee X Runtime. The Apigee X Runtime resides in a Google managed VPC.

Internal Global Load Balancers (ILBs) with Managed Instance Group (MIG) backends is the second option. This peering solution is complex though.

The open internet is the most parsimonious solution. Apigee X facilitates this, but it would violate the security mandates of many Google Cloud customers.

The final option is Cloud VPN. Cloud VPN supports transitivity across multiple VPC network hops. As well the implementation steps are relatively simple. Traffic does not traverse the open internet for VPCs within Google Cloud.

Apigee X Networking Explained

ApigeeX is a Private Service Access (PSA) based service that is deployed and managed in a tenant project. A tenant project is a project that is tied to the Apigee X Runtime, in a one-to-one relationship. The Apigee X deployment natively manages a load balancer in the customer’s VPC for ingress traffic. By default, egress to non-private routes from Apigee X is through Cloud NAT in the tenant project. The default Apigee X deployment is more suitable for non hub-and-spoke network design.

Single VPC Hub Design

Figure 3 demonstrates the API request flow. External API requests come from the internet and first encounter the XLB. The XLB has the Apigee X Runtime as a backend, using Network Endpoint Group to contain the Apigee X Environment Group domain name. From here the Apigee X Runtime sends the API request over the VPC peering connection to apigee-x-project.

Google Cloud VPN connections allow Virtual Private Clouds (VPCs) to reference each other’s internal resources, with the help of Google Cloud Routers. Cloud Routers advertise each VPCs’ IP address ranges. This enables traffic over the Cloud VPN tunnels to reference resources via their private IP addresses. Additionally, Google Cloud DNS peering facilitates a consumer VPC to reference hostnames in a producer VPC.

From the apigee-x-vpc (consumer), the API request references a backend-project-a-vpc (producer) resource hostname, via DNS peering. This would be the backend identified in an Apigee X API proxy. The request itself does not traverse the DNS peering connection. Instead, the API request goes through a Cloud VPN tunnel. In this way, Cloud VPN is used as a substitute for VPC peering.

Shared VPC Hub Design

A more advanced example is a shared VPC hub project, illustrated in Figure 4. The general idea is the same; use Cloud VPN connections instead of peering connections, advertise routes using Cloud Router and use DNS peering to reference any compute resources by their hostnames.

What is interesting is the idea that the apigee-x-project can use a subnet in a shared VPC, independently of the hub-project, which is in the same shared VPC. Shared VPC subnets do not violate the VPC peering transitivity limit (only two peered VPCs in sequence). This architecture pattern can be scaled to connect multiple shared VPCs via VPN tunnels. This is analogous to cascading hub-and-spoke architecture pattern: the inner hub-and-spoke is the shared VPC (with the apigee-x-project) and the outer hub-and-spoke is the VPN connected backend-projects.

Fixing VPC Peering with Private Service Connect

Private Service Connect (PSC) allows private consumption of services across VPC networks. This feature is coming to Apigee X in the first half of 2022. Private Services Access (PSA) is what enables you to reach services’ internal IP addresses. Apigee X is a PSA service, hence its ability to connect to backends by their internal IP addresses. PSC is the missing piece that will natively support Apigee X backends in other VPCs.

What Next?

Try it out for yourself! We built and shared a working Terraform example in our Github repository. Apigee X opens up users to the products and features of Google Cloud VPCs. VPC transitivity limits do require some work to overcome. Private Service Connect (PSC) will solve Apigee X VPC peering challenges later in 2022. A great next step would be to implement a “wheel” architecture pattern. This is an evolution of the hub-and-spoke design, where spoke VPCs peer to each other. Cloud VPN makes this possible and PSC makes a wheel more feasible to implement. The advantage of a wheel is that spokes no longer rely on a hub VPC; for example productizing APIs in a multi-tenant environment. More details are available in the following ebook: The API Product Mindset

Resources

You can find the Terraform code in the Github repository.

Editor’s note: Today we hear from the Lowe’s SRE team. They share about how they have been able to increase the number of releases they can support by adopting Google’s Site Reliability Engineering (SRE) framework and leveraging their partnership with Google Cloud. 

At Lowe’s, we’ve made significant progress in our multiyear technology transformation. To modernize our systems and build new capabilities for our customers and associates, we leverage Google’s SRE framework and Google Cloud, which helps us meet their needs faster and more effectively. With these efforts, we’ve been able to go from one release every two weeks to 20+ releases daily—about 20X more releases per month. 

Our SRE transformation didn’t happen overnight, though. Every step along the way brought some challenges. But looking back, we are excited to see how much we have accomplished for our customers as a result. 

Back in 2018, before adopting SRE practices, we were more reactive than proactive, following an “eyes on glass” approach. On-call structures and incident management efficiency were not at optimal levels with too many repetitive and manual tasks, resulting in operational toil. Production concerns were not surfaced into the product roadmap, which resulted in delays in making fixes.

Bootstrapping SRE at Lowe’s

As we moved from on-prem to Google Cloud, we decided to move from a monolithic- to microservices-based architecture. And to better manage this new architecture, we embarked on an SRE journey. 

Then as COVID-19 hit, we really had to accelerate this journey as customers increasingly moved to online ordering and delivery to meet their Total Home Improvement needs. To do so, we followed four key principles that allowed us to meet changing customer needs quickly and release fast and reliably.

1. Automate away toil 
   As we moved from traditional Ops to an SRE ecosystem, our biggest opportunity was reducing toil, so that engineers can spend time on activities that drive business impact and customer outcomes. We think of toil as work that is manual, repetitive,  tactical, devoid of enduring value—but automatable. So, to tackle toil, we focused on automating away the need for manual intervention. As an example, we made sure engineers were not the first point of contact for any alert. Any triage or resolution that an engineer can perform, a machine can be trained to do the same. We used supervised and unsupervised learning techniques to automate our toil. With a long-term goal of “no toil,” our SREs work on identifying and reducing toil to a manageable level across the organization.
2. Engineer alignment through roadmaps
   Our goal is to maximize the engineering velocity of developer teams while keeping products reliable. We want an engagement model where product, SRE and development teams are closely aligned. A key way we’ve been able to create this alignment is by having our SREs embedded into domain and product teams. Each domain has an SRE, who is  involved at the beginning stages of product development to ensure that the domain stakeholders are in alignment with the SRE initiatives. As such, SREs are able to improve the reliability, performance, scalability and launch velocity of the services throughout all phases of the service lifecycle. 
3. Adopt one-touch releases
   Our path to production used to contain many manual steps and validations, slowing the rate at which we released features. Additionally, we used to bulk all our releases together to deploy at once, which increased the risk of failure and created a longer feedback loop from production. To tackle this with an SRE mindset, we created a one-touch release process in which SREs review the product team’s pull requests. When approved, this triggers a DevSecOps pipeline that deploys the approved changes to production securely. This process created a safe, reliable and sustainable continuous delivery pipeline with quick feedback loops. Striking the right balance between speed, innovation and stability, we were able to increase our releases exponentially for the year, taking less than 30 minutes per release to deliver quality code, including various automated quality checks and processes, all in just one click. 
4. Embrace capacity planning
   To ensure our services have enough spare capacity to handle any surge in traffic patterns, our SREs emphasize capacity planning, making recommended capacity changes in the continuous delivery (CD) pipeline. They constantly monitor performance to make sure the service is robust, stable and available. And when there’s a sudden surge beyond the forecasted volume, SREs change the capacity on demand and document changes for the performance and domain teams. 

Capacity planning is especially important for us during peak holiday times such as Black Friday and Cyber Monday (BFCM). We lay out our SRE stability plan three months in advance and surface into the domain team’s product roadmap. This way development teams are able to allocate sufficient engineering time to reliability. We do performance testing to ensure the environment is able to sustain increased load over long periods of time and also handle sudden surges in traffic. We also do region failover testing at a global scale to validate the automatic failover duration of service level agreements  (SLAs), SRE and domain readiness. Additionally, we conduct Black Friday and Cyber Monday-specific destructive testing to validate customer experience, reliability and more.

Google Cloud’s Black Friday and Cyber Monday white-glove service played a key role in ensuring our success in both BFCM 2019 and BFCM 2020. This service included on-site visits from Google’s Customer Reliability Engineering (CRE) team who reviewed Lowe’s web architecture, capacity planning, operations practices for event risks, and presented workshops on topics such as incident response best practices. 

Looking ahead

There is always room for improvement, and at Lowe’s we aim to continuously improve our SRE practices. One thing that has worked well for us, which we plan to continue, has been our road shows, where senior SRE leads present to other SREs and application domain teams on the latest SRE principles and best practices, and to get input in real-time from them. 

Google’s tools and methodology have played an instrumental role in helping reshape our SRE practices and better serve our customers. We look forward to building on the momentum and partnership as we continue our SRE journey at Lowe’s. 

If you want to learn more about how to adopt SRE best practices on Google Cloud, check out our documentation. If you want to learn more about Google SRE, visit our website. Stay tuned for the next blogs with Lowe’s discussing how they trained their engineering talent to adopt SRE practices and tooling, and how they improved MTTR using SRE principles.