2020 brought many challenges, especially as in-person operations were shut down, and many were left vulnerable to unemployment.

The State of Rhode Island responded to these challenges, by modernizing their workforce development operations and moving completely to a custom online platform called the Virtual Career Center, also nicknamed “the VCC.”

It was developed in partnership with Google Cloud and a nonprofit called “Research Improving People’s Lives” (RIPL), and was fully built by Google Cloud partner Maven Wave, which helps a wide array of organizations, including public sector customers, with many types of cloud initiatives.

In this episode of Architecting with Google Cloud, we interviewed Joel Osman, head of Digital Experience & Custom Applications at MavenWave who shared a lot of insights such as:

“We looked at how we can apply leading edge emerging technologies to help people get back to work and use them as tools in such a way that we can augment the personal one to one interactions that agencies had been using with job seekers, to help them get back to work.”

One of those key benefits is enabling job seekers to find coaches that are specialized in their respective community. During the in-person walk-in model, applicants were paired on a first come first serve basis with any available coach on site. Meanwhile online scheduling has enabled a greater opportunity to match veterans, college graduates, non-English speakers, etc with coaches with prior experience in that specific area.

How the VCC was built

This VCC web app was built on Angular. It has a custom frontend built on top of 2 key Google Cloud products. The first is Workspace, which includes functionality such as video conferencing, documents, slides, chat, file storage, etc. And the other is Google Cloud computing resources. Here’s a view of the architecture:

There are 3 main types of users at this time, and that’s job coaches, job seekers, and employers.

• 👩‍🔧 Job Coaches all have Google IDs in the Google Workspace domain, and therefore authenticate against the Google identity repository.
• 🕵️‍♂️ Job Seekers are authenticated through a Cognito-based process maintained by the nonprofit I mentioned earlier (RIPL), the Rhode Island infrastructure team, and the Department of Information Technology (DoIT). Cognito was an identity repository setup prior to this project for users interacting with the State, and remained as their form of authentication.
• 🧭 Employers participate directly with Google Meet, and, to an extent, Google Calendar; but not the Angular app. There’s also a focus on building a future dashboard to see how the center has helped employers with applicants.

The specific Google Cloud components used are the following:

• Firestore: realtime Database that keeps data in sync across client apps.
• BigQuery: serverless warehouse for data.
• Data Studio: is used to build filterable dashboards over BigQuery
• Cloud Functions: which serve as triggers to keep scheduling and data workflows in sync.
• Kubernetes cluster: runs & autoscales the server-side code in a single-region deployment, with a minimum of four nodes per zone across three zones of the US East region.
• Cloud Armor: protects applications and websites from attacks, and sets NIST-compliant policies.
• Google’s Content Distribution Network (CDN): content is accessed and cached.

To manage the lifecycle of the infrastructure, a Terraform script is used, which is an open source tool, and is structured into 5 folder environments:

• Admin
• Dev
• QA & UAT
• Networks
• Prod
• Shared Services (for CI/CD pipelines between Dev & Prod).

Adoption outcomes

A universal fear we technical practitioners may have is:

“Will our tool be loved and adopted by our intended audiences?”

 Joel mentioned Job Coaches at the time were not used to working from home, and the team was concerned that they would potentially feel overwhelmed with a lot of new technology.

 As a rewarding surprise, when Job Coaches were presented the proof of concept, it was received with positivity.

“95% of Job Coaches rated the VCC as a valuable solution and 87% reported to find it very or extremely effective.”

This alignment was thanks to designing the tool with the users in mind, and performing user research since the beginning of the journey, which helped address their day to day needs.

Additional innovation for the future

After creating a centralized hub and moving operations to a digital format, many more benefits also arise. For example, there can now be an integrated data analytics view which enables meaningful dashboards that can be customized for different audiences such as job applicants, coaches, program stakeholders, or state agencies.

There can be improved job searchability by integrating machine learning, helping with resume building and parsing that take keywords out of a resume and match them to a variety of relevant job clusters, rather than just performing raw keyword searches.

Embedding chat bots can also help reduce the load of call centers in states, as they utilize natural language processing as well to help guide job seekers with prompt answers.

Conclusion

The State of Rhode Island’s Virtual Career Center is an amazing success story. By having worked with an experienced partner to move their operations to a digital format, they were able to respond to their citizen’s needs in a time where in-person operations were not possible. They also unlocked opportunities such as better matching and reporting along that journey.

For any organization whether they are in the public sector, university, private sector, etc; anyone can take advantage of this platform and customize it to their needs as Maven Wave shared that they offer a menu of options, where you can pick and choose functionality based on your requirements, IT resources, and budget.

Over the decades, technology has helped us organize large amounts of physical information in ways that are streamlined, efficient, and easily accessible. Rows upon rows of encyclopedias are no longer needed; simply punch in or speak a query into Google Search and find numerous results at your fingertips. There’s no need to haul around cases of CDs or cassettes either, when you can access hundreds of thousands of songs on music streaming services. 

We wanted to bring that same level of accessibility to one specific type of publication: the printed car manual. Here’s how we shifted the paper manual to become an easy-to-access, voice-activated digital experience for owners of the all-new Sienna. It’s helped this resource become just as modern and useful as Toyotas themselves.

Putting cloud technology in the driver’s seat

Far too often, the printed car manual remains unused, collecting dust in the glove compartment—that is, until it’s needed during a roadside emergency or to solve the meaning of a mysterious light popping up on the dashboard. Even then, thumbing through hundreds of pages during high-stakes moments can be stressful. On top of that, these manuals can be expensive to print and update.

Digital versions, such as a PDF file, are a nice start. But, they’re often little more than reformatted flat documents. In poor visual conditions on the side of a road, the last thing a driver wants to do is squint at a phone or scroll through pages of tiny text. And similar to printed manuals, digital versions don’t facilitate ongoing, real-time conversations between drivers and automakers when it matters the most; they’re often only text and pictures, and at best, schematic drawings.

With that in mind, we set out to elevate and personalize the car manual by creating a voice-activated digital owner’s manual experience, all powered by Google Cloud. A voice-based assistant was the clear choice because, as it felt like the most intuitive, natural option, especially while driving. In fact, 51% of U.S. adults have used a voice assistant while driving, and 95% of all drivers expect to use a voice assistant in the next three years. 

We’re now putting this technology on the road by powering the Toyota Driver’s Companion for the all-new 2021 Toyota Sienna model. Accessible through the existing Toyota app, this companion provides real-time assistance, any time of the day. Drivers can ask questions and the companion will efficiently provide helpful answers in convenient ways, from voice to 3D walkthroughs and explorable environments. 

What a modern car manual should do

The Toyota Driver’s Companion has interactive features to help drivers discover the Sienna’s dashboard, set up the car’s interior and exterior appearance, better understand vehicle maintenance, and explore Dynamic Radar Cruise Control, Lane Departure Alert and other features. 

Here are a few other additional key features to call out within the Toyota Driver’s Companion: 

• An easily accessible virtual voice through the Toyota Driver’s Companion lets app users ask personal questions about their 2021 Sienna such as “what’s the height of my car?” and receive immediate answers either by voice, display or interactive input. 
• The manual automatically connects with the purchased vehicle’s VIN number to create a completely personalized experience, curated specifically for the driver. For example, if an unfamiliar light on the dashboard pops up, the Toyota Driver’s Companion can help identify the light’s meaning.
• Interactive hotspots throughout the vehicle’s interior let drivers explore the cabin virtually. Drivers can discover button functionalities, find specific dials, and learn more about car functions, such as how to slide seats or open doors, to become acclimated with their new vehicle.

To bring this experience to life, we tapped into some of our key Google Cloud solutions:

• APIs powered by Google Cloud artificial intelligence technology make accessing specific vehicle information easy and effortless, by leveraging Google’s natural language processing:
  • Google Cloud DialogFlow API serves as the decision tree that gives intelligence for both finding an answer for a question, i.e., how the Companion responds to the end user’s questions. 
  • One of our Text-to-Speech APIs—called Wavenet—creates the Companion’s realistic voice. 
  • And finally, our Speech-to-Text API “listens” to the user’s voice and finds the correct information to craft responses. That means a driver can ask a question multiple ways, and the Companion will still respond with the right answer. 

Our Firebase mobile app dev platform gleans analytic insights that help improve the overall experience and services for OEMs and drivers alike.https://www.youtube.com/embed/66QxWS-PzIM?enablejsapi=1&

We’re encouraging better consumer experiences by providing faster access to fresh information, in a natural, accessible format—voice. But these new voice-activated experiences aren’t only an opportunity to help out drivers; it’s also about strengthening connections between drivers, their vehicles, and automakers, too. 

Our hope is that through this information exchange, drivers can provide feedback on the most frequently misunderstood features, enabling OEMs to address questions early on. By understanding the most requested features, OEMs can also predict and inform driver questions about features. We’re incredibly excited to help make the driver’s experience more connected and helpful.

Editor’s note: Today we hear from the Lowe’s SRE team. They share about how they have been able to increase the number of releases they can support by adopting Google’s Site Reliability Engineering (SRE) framework and leveraging their partnership with Google Cloud. 

At Lowe’s, we’ve made significant progress in our multiyear technology transformation. To modernize our systems and build new capabilities for our customers and associates, we leverage Google’s SRE framework and Google Cloud, which helps us meet their needs faster and more effectively. With these efforts, we’ve been able to go from one release every two weeks to 20+ releases daily—about 20X more releases per month. 

Our SRE transformation didn’t happen overnight, though. Every step along the way brought some challenges. But looking back, we are excited to see how much we have accomplished for our customers as a result. 

Back in 2018, before adopting SRE practices, we were more reactive than proactive, following an “eyes on glass” approach. On-call structures and incident management efficiency were not at optimal levels with too many repetitive and manual tasks, resulting in operational toil. Production concerns were not surfaced into the product roadmap, which resulted in delays in making fixes.

Bootstrapping SRE at Lowe’s

As we moved from on-prem to Google Cloud, we decided to move from a monolithic- to microservices-based architecture. And to better manage this new architecture, we embarked on an SRE journey. 

Then as COVID-19 hit, we really had to accelerate this journey as customers increasingly moved to online ordering and delivery to meet their Total Home Improvement needs. To do so, we followed four key principles that allowed us to meet changing customer needs quickly and release fast and reliably.

1. Automate away toil 
   As we moved from traditional Ops to an SRE ecosystem, our biggest opportunity was reducing toil, so that engineers can spend time on activities that drive business impact and customer outcomes. We think of toil as work that is manual, repetitive,  tactical, devoid of enduring value—but automatable. So, to tackle toil, we focused on automating away the need for manual intervention. As an example, we made sure engineers were not the first point of contact for any alert. Any triage or resolution that an engineer can perform, a machine can be trained to do the same. We used supervised and unsupervised learning techniques to automate our toil. With a long-term goal of “no toil,” our SREs work on identifying and reducing toil to a manageable level across the organization.
2. Engineer alignment through roadmaps
   Our goal is to maximize the engineering velocity of developer teams while keeping products reliable. We want an engagement model where product, SRE and development teams are closely aligned. A key way we’ve been able to create this alignment is by having our SREs embedded into domain and product teams. Each domain has an SRE, who is  involved at the beginning stages of product development to ensure that the domain stakeholders are in alignment with the SRE initiatives. As such, SREs are able to improve the reliability, performance, scalability and launch velocity of the services throughout all phases of the service lifecycle. 
3. Adopt one-touch releases
   Our path to production used to contain many manual steps and validations, slowing the rate at which we released features. Additionally, we used to bulk all our releases together to deploy at once, which increased the risk of failure and created a longer feedback loop from production. To tackle this with an SRE mindset, we created a one-touch release process in which SREs review the product team’s pull requests. When approved, this triggers a DevSecOps pipeline that deploys the approved changes to production securely. This process created a safe, reliable and sustainable continuous delivery pipeline with quick feedback loops. Striking the right balance between speed, innovation and stability, we were able to increase our releases exponentially for the year, taking less than 30 minutes per release to deliver quality code, including various automated quality checks and processes, all in just one click. 
4. Embrace capacity planning
   To ensure our services have enough spare capacity to handle any surge in traffic patterns, our SREs emphasize capacity planning, making recommended capacity changes in the continuous delivery (CD) pipeline. They constantly monitor performance to make sure the service is robust, stable and available. And when there’s a sudden surge beyond the forecasted volume, SREs change the capacity on demand and document changes for the performance and domain teams. 

Capacity planning is especially important for us during peak holiday times such as Black Friday and Cyber Monday (BFCM). We lay out our SRE stability plan three months in advance and surface into the domain team’s product roadmap. This way development teams are able to allocate sufficient engineering time to reliability. We do performance testing to ensure the environment is able to sustain increased load over long periods of time and also handle sudden surges in traffic. We also do region failover testing at a global scale to validate the automatic failover duration of service level agreements  (SLAs), SRE and domain readiness. Additionally, we conduct Black Friday and Cyber Monday-specific destructive testing to validate customer experience, reliability and more.

Google Cloud’s Black Friday and Cyber Monday white-glove service played a key role in ensuring our success in both BFCM 2019 and BFCM 2020. This service included on-site visits from Google’s Customer Reliability Engineering (CRE) team who reviewed Lowe’s web architecture, capacity planning, operations practices for event risks, and presented workshops on topics such as incident response best practices. 

Looking ahead

There is always room for improvement, and at Lowe’s we aim to continuously improve our SRE practices. One thing that has worked well for us, which we plan to continue, has been our road shows, where senior SRE leads present to other SREs and application domain teams on the latest SRE principles and best practices, and to get input in real-time from them. 

Google’s tools and methodology have played an instrumental role in helping reshape our SRE practices and better serve our customers. We look forward to building on the momentum and partnership as we continue our SRE journey at Lowe’s. 

If you want to learn more about how to adopt SRE best practices on Google Cloud, check out our documentation. If you want to learn more about Google SRE, visit our website. Stay tuned for the next blogs with Lowe’s discussing how they trained their engineering talent to adopt SRE practices and tooling, and how they improved MTTR using SRE principles.

Today’s enterprise applications are often assembled across distributed environments. This includes the integration of services across multi-cloud, multi-SaaS and on-premises environments. While the approach has the advantage of enabling enterprises to choose the best service available to support their applications, it adds the complexity of delivering services across heterogeneous environments. To solve this, Cloud Load Balancing supports an open cloud strategy, which includes:

• Supporting universal traffic management policies across heterogeneous environment by leveraging open source and open standards
• Enabling a global front-end so applications can leverage a common set of policies and security postures
• Providing tools that give your users the highest possible performance and reliability

Universal traffic management with open source and open standards

Kubernetes is a great solution for managing containers across environments. We believe that traffic management policies should also be supported across environments. Cloud Load Balancing creates homogeneous traffic policies across highly distributed heterogeneous environments by supporting standard-based traffic management in a fully managed solution, and allowing open source Envoy Proxy sidecars to be used on-premises or in a multi-cloud environment, using the same traffic management as our fully managed Cloud Load Balancers.

As enterprises start modernizing services and refactor monolithic applications, they require solutions that can provide consistent traffic management across distributed systems at scale. But organizations want to invest their time and resources innovating and building new applications — not on the infrastructure and networking required to deploy and manage these services. Envoy is an open-source high-performance proxy that runs alongside the application to deliver common platform-agnostic networking capabilities, including:

• New load balancing algorithms, (e.g. round robin, ring hash, least conns)
• Weighted traffic splitting
• Fault injection
• Request mirroring
• Outlier detection
• Additional header transformation options
• Request retries
• Additional backend session affinity options
• Cross-origin resource sharing (CORS)

Hybrid Load Balancing across multi-cloud and private clouds

Over the years Google has deployed Load Balancers across 173+ Edge Pop locations, delivering customer applications at massive-scale on Google infrastructure. And now Google Cloud has introduced Hybrid Load Balancing, extending our Load Balancing capabilities beyond Google’s network to on-premises private clouds and multi-cloud solutions. This allows our customers to migrate applications to the cloud iteratively, or build hybrid applications that are assembled from services that are running across heterogeneous environments.

Supporting modern application delivery with HTTP3/QUIC

Cloud Load Balancing is a fully distributed load balancing solution that balances user traffic (HTTP(s), HTTPS/2, HTTPS/3 with gRPC, TCP/SSL, UDP, and QUIC) to multiple backends to avoid congestion, reduce latency, increase security, and reduce costs. It is built on the same frontend-serving infrastructure that powers Google services, supporting millions of queries per second with consistent high performance and low latency.

To serve massive amounts of traffic, Google built the first scaled-out software-defined load balancing, Maglev, which has been serving global traffic since 2008. It has sustained the rapid global growth of Google services, and it also provides network load balancing functions for Google Cloud Platform customers. To accommodate ever-increasing traffic, Maglev is specifically optimized for packet processing performance with Linux Kernel Offload. Maglev is also equipped with consistent hashing and connection tracking features, to minimize the negative impact of unforeseen faults and failures on connection-oriented protocols.

Another key enabler to support this global-scale is that our Cloud Load Balancers are built on top of QUIC(RFC9000), a protocol developed from the original Google QUIC) (gQUIC). HTTP/3 is supported between the External HTTP(S) Load Balancer, Cloud CDN, and end clients. And once enabled, customers typically see dramatic improvements in performance and throughput.

Google Cloud already supports HTTP3 in Cloud Load Balancer. To use HTTP/3 for your applications, you can enable it on your external HTTPS Load Balancers via the Google Cloud Console or the gCloud SDK with a single click.

If your service is sensitive to latency, QUIC will make it faster because it establishes connections with reduced handshakes. When a web client uses TCP and TLS, it requires two to three round trips with a server to establish a secure connection before the browser can send a request. With QUIC, if a client has connected with a given server before, it can start sending data without any round trips, so your web pages will load faster.

QUIC has advantages over legacy TCP as follows.

Summary

Since 2008, Google has been an innovator in software-defined networking, supporting applications running at massive scale. Google Cloud Load Balancers support HTTP3 and QUIC as a next generation web transport protocol, which significantly improves customer traffic latency. Google Load Balancers also have incorporated the Envoy proxy as a foundational technology, providing our customers with advanced traffic management that’s compatible with the open source Envoy ecosystem. This allows our users to have the choice to combine Google’s fully-managed Cloud Load Balancers with open source Envoy Proxies, to enable consistent traffic management capabilities across a multi-cloud distributed environment. And with Hybrid Load Balancing, customers can leverage our 173+ world wide PoPs to seamlessly manage traffic across Google Cloud, on-premises and other cloud providers.

Google Cloud Load Balancers include all these capabilities natively. And when used together, they support globally-scaled applications that run seamlessly across the heterogeneous environments many enterprises deploy today.