Making Streaming Analytics Simpler and More Cost-Effective With Cloud Dataflow
3889
Of your peers have already read this article.
8:30 Minutes
The most insightful time you'll spend today!
Streaming analytics helps businesses to understand their customers in real time and adjust their offerings and actions to better serve customer needs. It’s an important part of modern data analytics, and can open up possibilities for faster decisions.
For streaming analytics projects to be successful, the tools have to be easy to use, familiar, and cost-effective. Since its launch in 2015, Cloud Dataflow, Google’s fully managed streaming analytics service, has been known for its powerful API available in Java and Python. Businesses have almost unlimited customization capabilities for their streaming pipelines when they use these two languages, but these capabilities come at the cost of needing programming skills that are sometimes hard to find.
To advance Google Cloud’s streaming analytics further, Google Cloud is announcing new features available in the public preview for Cloud Dataflow SQL, as well as the general availability of Cloud Dataflow Flexible Resource Scheduling (FlexRS) for a very cost-effective way of batch processing of events. These new features make streaming analytics easier and more accessible to data engineers, particularly those with database experience.
Here’s more detail about each of these new features:
Cloud Dataflow SQL to create streaming (and batch) pipelines using SQL
We know that SQL is the standard for defining data transformations, and that you want more GUI-based tools for creating pipelines. With that in mind, several months ago we launched a public preview of Cloud Dataflow SQL, an easy way to use SQL queries to develop and run Cloud Dataflow jobs from the BigQuery web UI. Today, we are launching several new features in Cloud Dataflow SQL, including Cloud Storage file support and a visual schema editor.
Cloud Dataflow SQL allows you to join Cloud Pub/Sub streams with BigQuery tables and Cloud Storage files. It also provides several additional features:
- Using Streaming SQL extensions for defining time windows and calculating window-based statistics;
- Integration with Cloud Data Catalog for storing the schema of Cloud Pub/Sub topics and Cloud Storage file sets—a key enabler for using SQL with streaming messages;
- A simple-to-use graphical editor available in the BigQuery web UI. If you are familiar with BigQuery’s SQL editor, you can create Cloud Dataflow SQL jobs.
To switch the BigQuery web UI to Cloud Dataflow SQL editing mode, open the BigQuery web UI, go to More>Query settings and select “Cloud Dataflow” as the query engine.
As an example, let’s say you have a Cloud Pub/Sub stream of sales transactions, and you want to build a real-time dashboard for the sales managers of your organization showing them the up-to-date stats of the sales in their regions. You can accomplish this in a few steps: write a SQL statement like the following one, launch a Cloud Dataflow SQL job, direct the output to a BigQuery table, then use one of the many supported dashboarding tools, including Google Sheets, Data Studio, and others, to visualize the results.
SELECTsr.sales_region,TUMBLE_START("INTERVAL 5 SECOND") AS period_start,SUM(tr.payload.amount) as amountFROM pubsub.topic.`dataflow-sql`.transactions AS trINNER JOIN bigquery.table.`dataflow-sql`.dataflow_sql_ds.us_state_salesregions AS srON tr.payload.state = sr.state_codeGROUP BYsr.sales_region,TUMBLE(tr.event_timestamp, "INTERVAL 5 SECOND")
In this example, we are joining the “transactions” Cloud Pub/Sub topic in the “dataflow-sql” project with a metadata table in BigQuery called “us_state_salesregions.” This table contains a mapping between the state codes (present in the “transactions” Cloud Pub/Sub topic) and the sales regions “Region_1”, “Region_2”, .., “Region_N” that are relevant to the sales managers in our example organization. In addition to the join, we’ll do a streaming aggregation of our data, using one of the several windowing functions supported by Cloud Dataflow. In our case, we will use TUMBLE windows, which will divide our stream into fixed five-second time intervals, group all the data in those time windows by the sales_region field, and calculate the sum of sales in the sales region. We also want to preserve the start of each time window, via TUMBLE_START("INTERVAL 5 SECOND"), to plot sales amounts as a time series.
To start a Cloud Dataflow job, click on “Create Cloud Dataflow job” in the BigQuery web UI.
When data starts flowing into the destination table, it will contain three fields: the sales_region, the timestamp of the period start, and the amount of sales.
In the next step, we will create a BigQuery Connected Sheet that shows a column chart of sales in “Region_1” over time. Select the destination BigQuery table in the nav tree. In our case, it’s the dfsqltable_25 table.
Then, select Export>Explore with Sheets to chart your data. Do it from the tab where your BigQuery table is shown, and not from the tab where your original Cloud Dataflow SQL was. In Sheets, create the column chart using the data connection to BigQuery, and choose period_start for the X-axis, amount as your data series, and add a sales_region filter. This is all you have to do to build a chart in Sheets that visualizes streaming data.
Mixing and joining data in Cloud Pub/Sub and BigQuery helps solve many real-time dashboarding cases, but quite a few customers have also asked for support of their Cloud Storage files to join those files with events in Cloud Pub/Sub or with tables in BigQuery. This is now possible in Dataflow SQL, enabled by our integration with Data Catalog’s Cloud Storage file sets.
In the following example, you’ll see an archive of transactions stored in CSV files in the “transactions_archive” Cloud Storage bucket.
Two gcloud commands can define a Cloud Storage file set and entry group in Data Catalog.
#Create a GCS entrygroupgcloud beta data-catalog entry-groups create transactions_archive_eg --location=us-central1#Create the filesetgcloud beta data-catalog entries create transactions_archive_fs --location=us-central1 --entry-group=transactions_archive_eg --gcs-file-patterns=gs://dataflow-sql/inputs/transactions_archive/*.csv --description="Historical Archive of Transactions"
Notice how we defined the file pattern “gs://dataflow-sql/inputs/transactions_archive/*.csv” as part of the file set entry definition. This pattern is what will allow Cloud Dataflow to find the CSV files once we write the SQL statement that references this file set.
We can even specify the schema of this transactions_archive_fs file set using a GUI editor. For that, go to the BigQuery web UI (make sure it is running in Cloud Dataflow mode), select “Add Data” in the left navigation and choose “Cloud Dataflow sources.” Search for your newly added Cloud Storage file set and add it to your active datasets in the BigQuery UI.
This will allow you to edit the schema of your file set after you select it in the nav tree. The “Edit schema” button is right there on the “Schema” tab. The visual schema editor is new and works for both Cloud Storage file sets as well as for Cloud Pub/Sub topics.
Once you’ve registered the file set in Data Catalog and defined a schema for it, you can query it in Cloud Dataflow SQL. In the next example, we’ll join the transactions archive in Cloud Storage with the metadata mapping table in BigQuery.
SELECTsr.sales_region,TUMBLE_START("INTERVAL 5 SECOND") AS period_start,SUM(tr.amount) as amountFROM datacatalog.entry.`dataflow-sql`.`us-central1`.transactions_archive_eg.transactions_archive_fs AS trINNER JOIN bigquery.table.`dataflow-sql`.dataflow_sql_ds.us_state_salesregions AS srON tr.state = sr.state_codeGROUP BYsr.sales_region,TUMBLE(CAST(tr.tr_time_str AS TIMESTAMP), "INTERVAL 5 SECOND")
Notice how similar this SQL statement is to the one that queries Cloud Pub/Sub. The TUMBLE window function even works on Cloud Storage files, although we will define the windows based on a field “tr_time_str” that is inside the files (the Cloud Pub/Sub SQL statement used the tr.event_timestamp attribute of the Cloud Pub/Sub stream). The only other difference is the reference to the Cloud Storage file set. We accomplish this by specifying datacatalog.entry.`dataflow-sql`.`us-central1`.transactions_archive_eg.transactions_archive_fs AS tr
Because both of the job inputs are bounded (batch) sources, Cloud Dataflow SQL will create a batch job (instead of a streaming job created for the first SQL statement using a Cloud Pub/Sub source) which will join your Cloud Storage files with the BigQuery table, and write the results back to BigQuery.
And now you have both a streaming pipeline feeding your real-time dashboard from a Cloud Pub/Sub topic, as well as a batch pipeline capable of onboarding historical data from CSV files. Check out the SQL Pipelines tutorial to start applying your SQL skills for developing streaming (and batch) Cloud Dataflow pipelines.
FlexRS for cost-effective batch processing of events
While real-time streaming processing is an exciting use case that’s growing rapidly, many streaming practitioners know that every stream processing system needs a sprinkle of batch processing (as you saw in the SQL example). When you bootstrap a streaming pipeline, you usually need to onboard historical data, and this data tends to reside in files stored in Cloud Storage. When streaming events need to be reprocessed due to changes in business logic (i.e., time windows get readjusted, or new fields are added), this reprocessing is also better done in batch mode.
The Apache Beam SDK and the Cloud Dataflow managed service are well-known in the industry for their unified API approach to batch and streaming analytics. The same Cloud Dataflow code can run in either mode with just minimal changes (usually replacing the data source from Cloud Pub/Sub to Cloud Storage). Remember our SQL example, where switching the SQL statement from a streaming source to a batch source was no trouble at all? And while it’s easy to go back and forth between streaming and batch processing in Cloud Dataflow, an important factor influencing the processing choice is cost. Customers who gravitate to batch processing have always looked to find the right balance between the speed of execution and costs. In many cases, that may mean you can be flexible with the amount of time it takes to process a dataset if the overall cost of processing is reduced in a significant fashion.
Our new Cloud Dataflow FlexRS feature reduces batch processing costs by up to 40% using advanced resource scheduling techniques and a mix of different virtual machine (VM) types (including the preemptible VM instances) to decrease processing costs while providing the same job completion guarantees as regular Cloud Dataflow jobs. FlexRS uses the Cloud Dataflow Shuffle service, which allows it to handle the preemption of worker VMs better because the Cloud Dataflow service does not have to redistribute unprocessed data to the remaining workers.
Using FlexRS requires no code changes in your pipeline and can be accomplished by simply specifying the following pipeline parameter:
--flexRSGoal=COST_OPTIMIZED
Running Cloud Dataflow jobs with FlexRS requires autoscaling, a regional endpoint in the intended region, and specific machine types, so you should review the recommendations for other pipeline settings. While Cloud Dataflow SQL does not yet support FlexRS, it will in the future.
Simultaneously with launching FlexRS in general availability, we are also extending its availability to five additional regions, covering all regions now where we have regional endpoints and Cloud Dataflow Shuffle:
- us-central1
- us-east1
- us-west1
- europe-west1
- europe-west4
- asia-east1
- asia-northeast1
To learn more about Cloud Dataflow SQL, check out our tutorial and try creating your SQL pipeline using the BigQuery web UI. Visit our documentation site for additional FlexRS usage and pricing information.
Check out other recently launched streaming and batch processing features:
- We deployed Cloud Dataflow Shuffle and Cloud Dataflow Streaming Engine to three additional regions, bringing total availability to seven regions.
- We launched the ability to protect your pipeline state with customer-managed encryption keys.
Python streaming support from Cloud Dataflow is now generally available.
Google Cloud and Univision Partnership to Up the Ante in UX for Spanish-speaking Audience
6876
Of your peers have already read this article.
1:00 Minutes
The most insightful time you'll spend today!
The past year has given everyone lots to think about—about our priorities as people and as businesses. As the world retreated behind closed doors, we saw how shared interests and experiences can bring us together. As the world grappled with a common enemy, we witnessed just how differently individuals, communities and indeed entire countries can experience a situation. And as we faced seemingly unending obstacles to making it through the pandemic, we saw how making smart decisions based on data can drive meaningful solutions—fast.
That’s why we here at Google Cloud are so proud to partner Univision, the country’s leading Spanish-language content and media company. By partnering with Google Cloud, Univision will be able to accelerate growth across its portfolio of properties, deliver an enhanced user experience for Spanish-speaking audiences and provide the enterprise solutions needed to create the Spanish-language media company of the future.
According to Instituto Cervantes, there are over 580 million Spanish language speakers worldwide. Those viewers, like people everywhere, are avid consumers of streaming content. In Q4 of 2020 alone, viewing time for that content increased by 44%1, and in 2020, from 50%2 more sources. With that surge in demand, Univision needed a cloud provider whose infrastructure could reach Hispanic viewers around the world. With two-plus decades spent building out its network and data centers, as well as global content-delivery capabilities, Google Cloud has the infrastructure Univision needs to reach viewers across the Spanish-speaking world.
At the same time, with such a diverse audience for their content, Univision needs to target that content to viewers’ specific preferences. By applying Google Cloud’s artificial intelligence (AI) and machine learning (ML) technology across its content, Univision intends to personalize content based on shows users have previously watched, enhancing their engagement and viewing experience.
And as Univision transforms the user experience, it can use Google Cloud’s data and analytics suite to garner deeper insights into its audience and forge stronger relationships with them on an individual basis. With Looker and BigQuery, Univision employees will have access to real-time data to help them make business decisions about programming.
Univision will also migrate video distribution and production operations to Google Cloud, where we’ll help them streamline media workflows and develop innovative new capabilities. Meanwhile, Google Cloud’s tight business and technical integration with other Google services will help ensure Univision reaches viewers on the device of their choice, wherever they are in the world. For example, in the coming years, Univision will expand its global YouTube partnership and will integrate with entertainment features on Google Search that help people better discover TV shows and movies. The company will also use Google Ad Manager for global ad decisioning and Google’s Dynamic Ad Insertion for PrendeTV and future video-on-demand offerings. Finally, Univision will distribute its content and services on Google Play across Android phones and tablets, as well as Google TV and other Android TV OS devices.
We’re thrilled to partner with Univision to help them reach the Spanish-speaking world with their content. With our cloud portfolio, we can help them reach individual viewers around the world, with personalized content that they can consume however they see fit. Best of all, together, we can help them achieve this vision fast, leveraging established cloud, content delivery, and data analytics technologies. You can learn more about the partnership here.
Making Mothers’ Day Special: How Google Cloud Migration for 1-800-FLOWERS.COM, Inc Impacts CX
6840
Of your peers have already read this article.
2:30 Minutes
The most insightful time you'll spend today!
Editor’s note: In honor of Mother’s Day, we look at how 1-800-FLOWERS.COM, Inc. migrated to Google Cloud as part of its digital transformation to quickly deploy seamless and convenient customer experiences across multiple brands on Mother’s Day and every day.
As a leading provider of gifts designed to help customers express, connect, and celebrate, 1-800-FLOWERS.COM, Inc. has embraced cloud technologies to grow and transform its business through constant innovation. As part of our digital transformation, we recently completed the migration of our ecommerce platform and other services to Google Cloud. We’ve transitioned from a monolithic to a microservices platform, moved many workloads from our on-premises data centers to our Google Cloud environment, and scaled both horizontally and vertically.
Since our migration, we’ve developed efficient processes to launch new brands, improved the customer experience across all brands, and seen significantly increased site traffic.
Nurturing a more delightful customer journey
Customer delight is at the core of everything we do. Whether it be with a flower bouquet, a sweet treat, or a personalized keepsake, our mission is to deliver smiles. With the rise of the COVID-19 pandemic, we’ve all been challenged to find unique and safe ways to continue honoring the special connections in our lives and celebrating occasions with loved ones. Our customers have adapted by doing things such as sending gifts to isolated loved ones, sharing the same meal together virtually, or using video to engage with others through group activities like flower arranging and building charcuterie boards.
The customer experience is a top priority for us, and we constantly look for innovative ways to enhance the customer journey across our ecommerce platform of more than a dozen brands. As a result, we’ve continued to see a rise in demand as customers enjoy the ease and convenience of our site and discover our full family of brands.
Migrating to a cloud-first mindset
As we’ve continued to innovate and iterate on the customer experience, we knew we wanted to evolve our platform. We wanted to shift to a microservices platform, which would allow our team the opportunity to release updates to our site more often and set up the right continuous integration/continuous deployment (CI/CD) practices.
Working with Google Cloud, we were able to move our ecommerce platform to the cloud and standardize our site and brand deployment by building one release that could then be repeated across all of our brands. We built everything in a modular fashion, including microservices and code libraries, so that sites could be easily constructed and replicated for each brand. And because of this, we were able to launch Shari’s Berries extremely quickly after we acquired the brand in 2019.
Moving our platform to a completely homegrown solution of microservices was a daunting task. But our team handled it beautifully through load-testing, stress-testing, and building new monitoring tools. And with Google Cloud supporting us all along the way, managing the migration process was simple and easy from start to finish.
Arranging a better bouquet of services
Currently, we’ve migrated every customer-facing touchpoint for all of our brands to Google Cloud—whether it’s on the web or mobile, our AI bots, or our chat interfaces.
- We run on Google Kubernetes Engine and Istio.
- We have nearly 200 microservices built to help power our entire ecommerce stack across several cloud services running on Google Cloud.
- We’re utilizing BigQuery for our offline intelligence.
Results are coming up roses
Our new stack on Google Cloud has benefits for both us and our customers. We moved from a session-based to a token-based system, which provides enhanced security as well as a consistent, convenient experience across all our brands. Using service workers and a single-page app, we are able to download all the relevant site content to the browser in under two seconds to create an instant-click experience for each and every customer. We also use Google Analytics to measure our user interactions and provide personalized results to each customer. Our hope is that with this new system, we can learn from customer behavior to offer gift givers a more personalized shopping experience during each visit.
The benefits of our new tech stack have not only helped us enhance the solutions we offer to customers today, they’ve also enabled us to offer new ones at lightning speed. With our legacy system, we used to release new code once a week or once a month. Now, even during our peak periods, we’re able to release 10 to 15 times a day and can deploy and pivot quickly to create new microservices and microsites on the fly—often without having to touch any code.
Efficiencies abound
The benefits of moving our platform to Google Cloud have extended to our internal teams as well. Before the migration, we had only two environments for developing and testing, which made it time-consuming to test updates before they went into production. Now with Google Cloud, we have several different journey teams—which are made up of developers, product owners, and technical owners—all working in several different environments, solving problems, and creating new solutions together.
Everyone is now empowered to be self-sufficient, developing and releasing microservices on their own when they’re ready. This has given our developers more time to take part in continued development and learning opportunities. For example, we offer lunch-and-learn sessions as well as other resources for everyone to take advantage of so they can continue to learn and refine their skills.
Planting the seeds for future growth
As we look to the future and think about how we help our customers express, connect, and celebrate, we’ll continue to collaborate across teams to deliver solutions that spread smiles. Specifically, we’re exploring additional use of AI to help us better serve our customers across all our brands.
We’ve enjoyed the ongoing support we’ve received from the Google Cloud team as they help us build new solutions and design a road map for the future. Their support has helped the 1-800-FLOWERS.COM, Inc. team to realize the power of the cloud and bring the very best experience to our customers.
Learn more about 1-800-FLOWERS.COM, Inc., or check out our recent blog about cloud migration for the real world.
2988
Of your peers have already watched this video.
39:00 Minutes
The most insightful time you'll spend today!
Verizon Media Shows the Solution Architecture it Uses for a 100+ PB Analytics Platform
Verizon Media owns and operates more than a dozen brands including Yahoo Mail, Yahoo News, AOL, Huffington Post, TechCrunch, and Engadget among others.
These web properties are visited by millions of users on a daily basis.
In this session, Shakil Memon, Customer Engineer, Google Cloud and Nikhil Mishra, Sr Director, Engineering, Verizon Media, present how Verizon Media is generating actionable insights from the wealth of data that they have.
They will discuss:
- Challenge with large scale and large volumes of data
- Basic principles of a data warehousing and data analytics project
- Showcase a reference architecture
- A complete end-to-end solution architecture for building a 100+ PB internet-scale analytics platform on Google Cloud, including how Looker fits in the end-to-end solution and how actionable insights are generated from data.
They will also provide unique perspectives, behind-the-scenes thinking, and some insight on how the architecture has evolved in its current form over the years.
How Google Cloud’s Scalable Data Storage and High Compute Resources Fuel Investment Research
6301
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
Investment management is a heavily data-driven industry—portfolio managers and investment researchers require a large number of data sources to guide them in shaping their investment strategies.
New cloud capabilities and technologies enable investment managers to process data faster than ever before and iterate on ideas quickly to fuel innovation in the signal generation process and gain a competitive edge.
Using the cloud for investment research workflows makes it easier to onboard data from data providers, spin up large compute workloads in the midst of market volatility or during heavy research cycles, and manage complex machine learning or natural language workflows to gain market insights.
We hear from industry leaders that they’re exploring new ways to run investment research. “Differentiated investment strategies require new types of information sources, and new ways to process that information,” David Easthope, senior analyst, Market Structure and Technology, Greenwich Associates. “And that, of course, relies heavily on having access to reliable and scalable storage, computational, and AI / ML resources. More specifically, quantitative strategies can benefit from the computational platforms and embedded AI/ML capabilities the cloud can offer.”
Google Cloud gives investment managers essential components to work and operate faster as they bring their investment research workflows to the cloud. Here are the key highlights:
1. Simplify, speed up your data acquisition, discovery, and analytics
The foundation of any investment strategy starts with data—acquiring it, detecting patterns, and analyzing it for insights. Enabling data providers to easily share large datasets such as tick history within a high-performance analytics engine can greatly reduce the data engineering overhead when possible.
Once data is onboarded, you can tag business and technical metadata related to your datasets and provide portfolio managers the ability to discover these datasets via a search interface.
We further review analytics options for various scenarios, including aggregating massive datasets, creating dashboards, and incorporating streaming analytics workloads.
2. Take advantage of burst compute workloads
Data engineers and researchers require ready access to burst compute capabilities to perform backtesting, portfolio simulations and run risk calculations. Cloud works well for these workloads due to its elasticity, consumption-based models, and hardware evolution.
Many investment managers are shifting to a container-based strategy along with a Kubernetes-based scheduler for greater consistency, scaling and efficiency in environments with a large number of researchers. Cloud managed services and a rich suite of CI/CD tools can make this vision a reality while improving security and developer productivity.
3. Tackle machine learning (ML) and model deployment with the help from cloud
Quantitative researchers scour vast amounts of market and alternative data sources searching for signals and correlations, while ML engineers have the challenge of taking these signals and moving them to production.
Google Cloud empowers users to create and operationalize their models without wasting valuable time with a comprehensive set of MLOps tools.
In this paper, we explore multiple solutions for ML and model deployment. Those capabilities reduce the amount of time operationalizing ML models, so quants and data scientists have more time to devote to differentiating activities.
4. Get the data you need in less time with Natural Language and Document AI
Thousands of financial filings, news articles, and sell-side research reports are generated every day, and it’s difficult for humans alone to process this volume of information. These documents are often generated in many languages and the ability to do entity recognition, sentiment or syntactical analysis in those languages, or perhaps translate them into the language of the portfolio manager is of critical importance. Google Cloud provides these capabilities through pre-trained models, or allows you to train high-quality models with your own datasets.
Getting started
There are plenty of emerging technologies, tools, and approaches available to help investment managers today. At Google Cloud, we can help you access, organize, and utilize these essential components to make your research faster, reliable, and more valuable.
To learn more about these four keys to better investment research, check out our whitepaper for more.
5368
Of your peers have already downloaded this article
8:15 Minutes
The most insightful time you'll spend today!
To design and implement a successful machine learning (ML) project, you often need to collaborate with multiple teams, including those in business, sales, research, and engineering. Understanding the essentials of gathering and preparing your data is crucial to align teams, and getting a project off the ground.
Building an ML model that achieves your business’ goal, requires the kind of data that a model can learn from. If the goal is to predict a target output based on some input, you’ll need some data consisting of past input-output examples. For example, to detect (that is, to predict) a fraudulent credit card transaction (the target of the prediction), each example must contain information on a past credit card transaction and whether that transaction was fraudulent or not.
So what are the best practices in preparing and curating data for a machine learning initiative? The answer to this question lies within the guide. You’ll learn:
- To evaluate whether your data is suitable for ML
- What types of data you need–and which ones to eliminate
- How to prevent data leakage
- Tricks to speed up an ML project such as Google’s human labeling service, and Cloud Dataprep
- The importance of the granularity of data
More Relevant Stories for Your Company
Google Cloud’s 2021 Data Analytics Launches
Google Cloud announced closed to 15 services and programs spanning database, analytics, business intelligence and AI to help businesses gain value out of their data. Here's a rundown of announcements throughout 2021 on analytics solutions and services such as Dataplex, an intelligent data fabric solution, Datastream, a serverless change data
What Swiggy and You Can Learn From This Company’s Use of ML to Engage Customers
The app economy has enabled a huge range of unique business models to flourish. One such model is online food ordering and delivery services, in which apps leverage geo-location data to aggregate local food choices and offer personalized options to consumers. A leading company in this space is Just Eat.
Google Cloud Accelerates Financial Organizations’ Journey towards Digital Transformation
When I reflect back on the past year and the pandemic, I’m struck by how the reliance on remote work and operations has changed the fundamentals of business forever. For the financial services industry, this rings particularly true. Many conversations I’m having right now with organizations revolve around embracing a transformation
How Eventrac and Workflows Integration Helps Implement Hybrid Architecture in Google Cloud
I previously talked about Eventarc for choreographed (event-driven) Cloud Run services and introduced Workflows for orchestrated services. Eventarc and Workflows are very useful in strictly choreographed or orchestrated architectures. However, you sometimes need a hybrid architecture that combines choreography and orchestration. For example, imagine a use case where a message to a Pub/Sub topic
