How Eventrac and Workflows Integration Helps Implement Hybrid Architecture in Google Cloud
6555
Of your peers have already read this article.
3:00 Minutes
The most insightful time you'll spend today!
I previously talked about Eventarc for choreographed (event-driven) Cloud Run services and introduced Workflows for orchestrated services.
Eventarc and Workflows are very useful in strictly choreographed or orchestrated architectures. However, you sometimes need a hybrid architecture that combines choreography and orchestration.
For example, imagine a use case where a message to a Pub/Sub topic triggers an automated infrastructure workflow or where a file upload to a Cloud Storage bucket triggers an image processing workflow. In these use cases, the trigger is an event but the actual work is done as an orchestrated workflow.
How do you implement these hybrid architectures in Google Cloud? The answer lies in Eventarc and Workflows integration.
Eventarc triggers
To recap, an Eventarc trigger enables you to read events from Google Cloud sources via Audit Logs and custom sources via Pub/Sub and direct them to Cloud Run services:
One limitation of Eventarc is that it currently only supports Cloud Run as targets. This will change in the future with more supported event targets. It’d be nice to have a future Eventarc trigger to route events from different sources to Workflows directly.
In absence of such a Workflows enabled trigger today, you need to do a little bit of work to connect Eventarc to Workflows. Specifically, you need to use a Cloud Run service as a proxy in the middle to execute the workflow.
Let’s take a look at a couple of concrete examples.
Eventarc Pub/Sub + Workflows integration
In the first example, imagine you want a Pub/Sub message to trigger a workflow.
Define and deploy a workflow
First, define a workflow that you want to execute. Here’s a sample workflows.yaml that simply decodes and logs the Pub/Sub message body:
main:params: [args]steps:- init:assign:- headers: ${args.headers}- body: ${args.body}...- pubSubMessageStep:call: sys.logargs:text: ${"Decoded Pub/Sub message data is " + text.decode(base64.decode(args.body.message.data))}severity: INFODeploy the workflow with a single command:gcloud workflows deploy ${WORKFLOW_NAME} --source=workflow.yaml --location=${REGION}
Deploy a Cloud Run service to execute the workflow
Next, you need a Cloud Run service to execute this workflow. Workflows has an execution API and client libraries that you can use for your favorite language. Here’s an example of the execution code from a Node app.js file. It simply passes the received HTTP request headers and body to the workflow and executes it:
const execResponse = await client.createExecution({parent: client.workflowPath(GOOGLE_CLOUD_PROJECT, WORKFLOW_REGION, WORKFLOW_NAME),execution: {argument: JSON.stringify({headers: req.headers, body: req.body})}});
Deploy the Cloud Run service with the Workflows name and region passed as environment variables:
gcloud run deploy ${SERVICE_NAME} \--image gcr.io/${PROJECT_ID}/${SERVICE_NAME} \--region=${REGION} \--allow-unauthenticated \--update-env-vars GOOGLE_CLOUD_PROJECT=${PROJECT_ID},WORKFLOW_REGION=${REGION},WORKFLOW_NAME=${WORKFLOW_NAME}
Connect a Pub/Sub topic to the Cloud Run service
With Cloud Run and Workflows connected, the next step is to connect a Pub/Sub topic to the Cloud Run service by creating an Eventarc Pub/Sub trigger:
gcloud eventarc triggers create ${SERVICE_NAME} \--destination-run-service=${SERVICE_NAME} \--destination-run-region=${REGION} \--location=${REGION} \--event-filters="type=google.cloud.pubsub.topic.v1.messagePublished"
This creates a Pub/Sub topic under the covers that you can access with:
export TOPIC_ID=$(basename $(gcloud eventarc triggers describe ${SERVICE_NAME} --format='value(transport.pubsub.topic)'))
Trigger the workflow
Now that all the wiring is done, you can trigger the workflow by simply sending a Pub/Sub message to the topic created by Eventarc:
gcloud pubsub topics publish ${TOPIC_ID} --message="Hello there"
In a few seconds, you should see the message in Workflows logs, confirming that the Pub/Sub message triggered the execution of the workflow:
Eventarc Audit Log-Storage + Workflows integration
In the second example, imagine you want a file creation event in a Cloud Storage bucket to trigger a workflow. The steps are similar to the Pub/Sub example with a few differences.
Define and deploy a workflow
As an example, you can use this workflow.yaml that logs the bucket and file names:
main:params: [args]steps:...- log:call: sys.logargs:text: ${"Workflows received event from bucket " + bucket + " for file " + file}severity: INFO
Deploy a Cloud Run service to execute the workflow
In the Cloud Run service, you read the CloudEvent from Eventarc and extract the bucket and file name in app.js using the CloudEvent SDK and the Google Event library:
const cloudEvent = HTTP.toEvent({ headers: req.headers, body: req.body });//"protoPayload" : {"resourceName":"projects/_/buckets/events-atamel-images-input/objects/atamel.jpg}";const logEntryData = toLogEntryData(cloudEvent.data);const tokens = logEntryData.protoPayload.resourceName.split('/');const bucket = tokens[3]
Executing the workflow is similar to the Pub/Sub example, except you don’t pass in the whole HTTP request but rather just the bucket and file name to the workflow:
const execResponse = await client.createExecution({parent: client.workflowPath(GOOGLE_CLOUD_PROJECT, WORKFLOW_REGION, WORKFLOW_NAME),execution: {argument: JSON.stringify({bucket: bucket, file: file})}});
Connect Cloud Storage events to the Cloud Run service
To connect Cloud Storage events to the Cloud Run service, create an Eventarc Audit Logs trigger with the service and method names for Cloud Storage:
gcloud eventarc triggers create ${SERVICE_NAME} \--destination-run-service=${SERVICE_NAME} \--destination-run-region=${REGION} \--location=${REGION} \--event-filters="type=google.cloud.audit.log.v1.written" \--event-filters="serviceName=storage.googleapis.com" \--event-filters="methodName=storage.objects.create" \--service-account=${PROJECT_NUMBER}-compute@developer.gserviceaccount.com
Trigger the workflow
Finally, you can trigger the workflow by creating and uploading a file to the bucket:
echo "Hello World" > random.txtgsutil cp random.txt gs://${BUCKET}/random.txt
In a few seconds, you should see the workflow log the bucket and object name.
Conclusion
In this blog post, I showed you how to trigger a workflow with two different event types from Eventarc. It’s certainly possible to do the opposite, namely, trigger a Cloud Run service via Eventarc with a Pub/Sub message (see connector_publish_pubsub.workflows.yaml) from Workflows or a file upload to a bucket from Workflows.
All the code mentioned in this blog post is in eventarc-workflows-integration. Feel free to reach out to me on Twitter @meteatamel for any questions or feedback.
The Fantastic Story of How BMG Enables a Micropayments Strategy So Music Artists Get Paid
7459
Of your peers have already read this article.
3:30 Minutes
The most insightful time you'll spend today!
The music industry is rapidly changing. Only 20 years ago, the availability of music and the infrastructure that was required to make an album a sales success were incredibly complex and expensive. With the decline of physical sales and a fundamental shift to digital, music streaming now accounts for more than half of all sales globally.
At the same time, technology has democratized music-making; in many ways, it has made the musical landscape more diverse. Artists can upload their music with the click of a button. But while it’s easier for creators to share their songs with audiences, getting paid has become more fragmented.
Although music is booming, people no longer buy it outright. Instead, listeners download music digitally or subscribe to streaming services to have their libraries with them at all times. To monetize digital content effectively, artists need to know when, where, and how often their songs are played on each service. To help them navigate this complicated royalties landscape and maximize their profits, Berlin-based international music company BMG provides customized, transparent, and fair services to songwriters and artists.
With publishing and recording divisions under one roof, the subsidiary of international media giant Bertelsmann works with both emerging artists and established stars, including John Legend, Kylie Minogue, Mick Jagger, and Keith Richards. With the MyBMG web and mobile application, clients can view and analyze their royalty details in real time and collect payment. When a new record is released, BMG uses data to maximize its impact and revenue for its creators.
“We make sure that everyone who uses our clients’ music knows who needs to be paid the associated royalties, then we collect these royalties and share them out quickly and transparently,” explains Sebastian Hentzschel, Chief Information Officer at BMG. “When our artists release new music, we make sure that it’s marketed and promoted effectively around the world.”
“We needed a scalable solution for our royalty workloads that was intuitive for our developers. We also wanted a partner, not a client-vendor relationship. With autoscaling via BigQuery, excellent customer support, and a clean and simple user interface, Google Cloud ticks every box for us.”
—Gaurav Mittal, Vice President Group Technology, BMG
Getting up to speed with a new way of paying artists
In this digital world, artists aren’t just paid every time a fan buys an album—they’re paid a small amount, or royalty, for each song downloaded or streamed by a listener. So, when the industry shifted to digital, the volume of data that BMG needed to handle grew exponentially. “One CD sale is equivalent to about 1,500 streamed songs or plays,” says Gaurav Mittal, Vice President Group Technology at BMG. “That means IT departments have to process 1,500 times the amount of data to calculate payments for artists, and this makes scalable micropayment processing very important.”
Until 2019, BMG’s infrastructure was entirely hosted on-premises. Hardware limitations made it challenging to scale on-demand, making it harder to handle the data peaks that royalty processing can bring. “With our on-premises infrastructure, we were going to hit a ceiling in a few years,” says Gaurav. “We still managed to process royalty payments for our clients, but it was increasingly time consuming and expensive. To keep focusing on our clients, rather than our infrastructure, we decided to migrate to Google Cloud.”
From the outset, Gaurav and his team had a clear vision for the partnership: “Most importantly, we needed a scalable solution for our royalty workloads that was intuitive for our developers. We also wanted a partner, not a client-vendor relationship,” he says. “With autoscaling via BigQuery, excellent customer support, and a clean and simple user interface, Google Cloud ticks every box for us.”
Keeping artists happy with business-as-usual payouts during migration
To move applications to the cloud while keeping payment cycles on track for its artists, BMG teamed up with Google Cloud partner Rackspace Technology. “We selected Rackspace Technology because it combines strong technical muscle and a global footprint, with the customer service of a local boutique firm,“ shares Gaurav.
BMG’s own technology team put together the outline for the Google Cloud architecture, which they passed on to Rackspace Technology for optimizations and the ultimate stamp of approval. Whenever Gaurav and his developers needed support, Rackspace Technology was ready to go. “So far, we’ve migrated 17 applications successfully, and Rackspace Technology has been 100% spot-on with each suggestion,” says Gaurav. “I can’t recall a single flaw in a Rackspace Technology-approved architecture, and that really says something.”
When BMG began its migration in August 2019, the team developed an ambitious two-year plan. Only 14 months later, however, the project is more than 75% complete. Among the solutions that BMG is using today are Cloud Storage to securely store 130 TB of data, and Cloud SQL as its standard database technology. The web applications run on Compute Engine, App Engine, and Google Kubernetes Engine.
“After successfully moving a few applications, it was clear that with the strong teamwork of BMG and Rackspace Technology, together with the ease of use of Google Cloud, we could speed up the project without sacrificing quality,” says Gaurav. “We’re set to complete our migration six months before schedule, helping us to quickly move out of our hybrid environment.”
“Our income-tracking teams are very savvy on the data, and the simplicity of Google Cloud empowers them to self-serve analytics, rather than wait for IT. Our teams are much more productive.”
—Gaurav Mittal, Vice President Group Technology at BMG
Royalty reporting and processing with BigQuery and Dataproc
So far, all of BMG’s critical workloads are up and running on Google Cloud. Royalty calculations, for example, which require incredible processing power and the collection of many micropayments to ensure full and timely payout, run entirely on Dataproc with output stored on BigQuery for downstream integration and reporting.
Enabling more harmonious workflows through self-serve analytics
As the new beating heart of BMG’s royalty reporting, BigQuery changed the rhythm of collaboration company-wide. In the past, income tracking teams had to contact IT departments if they needed deeper data insights for their work. By integrating Data Catalog with BigQuery, BMG has made the data more accessible to all teams. This helps them detect missing income and new revenue streams independently, maximizing profits for artists.
“Our income-tracking teams are very savvy on the data, and the simplicity of Google Cloud empowers them to self-serve analytics, rather than wait for IT,” says Gaurav. “Our teams are much more productive.”
“Google Cloud enables us to be more client focused and deliver better features faster. We believe it’s just the beginning. We offer rights and royalty services for music publishing, recorded music, neighboring rights, and books. Without scalability limitations, it’s absolutely conceivable to offer our platform as a service to other companies or industries, such as gaming. Google Cloud has opened a world of possibilities.”
—Sebastian Hentzschel, Chief Information Officer, BMG
With a leaner IT environment, BMG can focus its effort on the needs of its clients. Beyond improvements in royalty processing, it can concentrate on app development, releasing new features and enhancements more frequently. By hosting applications on Google Kubernetes Engine, App Engine, and Compute Engine, BMG has built a CI/CD pipeline with automated deployments and testing to significantly speed up workflows.
“In our old system, it could take several weeks to set up an environment,” says Gaurav. “With Google Kubernetes Engine, any of our developers can complete the process in a few clicks. Having that autonomy makes our developers more motivated and self-driven.”
“Google Cloud enables us to be more client focused and deliver better features faster,” adds Sebastian. “We believe it’s just the beginning. We offer rights and royalty services for music publishing, recorded music, neighboring rights, and books. Without scalability limitations, it’s absolutely conceivable to offer our platform as a service to other companies or industries, such as gaming. Google Cloud has opened a world of possibilities.”
With the migration almost complete, BMG is looking forward to its next technology project. It plans to leverage AutoML to further scale and automate royalty tracking with machine learning. On the marketing side, advanced analytics will help BMG determine the effectiveness of promotional campaigns around the world, further increasing profits for artists. By connecting Google Data Studio to BigQuery, BMG will increase the quality of its analyses, helping musicians better understand the reach of their music around the world.
In the end, Gaurav shares, helping musicians is what it all comes down to. “We’re a new kind of music company because we build our services around our artist, songwriter, and publisher clients, not the other way around,” he says. “Google Cloud is helping us maintain strong relationships with our clients, and that’s music to our ears.”
Why and How to Migrate to Google BigQuery
5380
Of your peers have already read this article.
6:30 Minutes
The most insightful time you'll spend today!
Over the past few decades, organizations have mastered the science of data warehousing. They have increasingly applied descriptive analytics to large quantities of stored data, gaining insight into their core business operations. Conventional Business Intelligence (BI), which focuses on querying, reporting, and Online Analytical Processing, might have been a differentiating factor in the past, either making or breaking a company, but it’s no longer sufficient.
Today, not only do organizations need to understand past events using descriptive analytics, they need predictive analytics, which often uses machine learning (ML) to extract data patterns and make probabilistic claims about the future. The ultimate goal is to develop prescriptive analytics that combine lessons from the past with predictions about the future to automatically guide real-time actions.
Traditional data warehouse practices capture raw data from various sources, which are often Online Transactional Processing (OLTP) systems. Then, a subset of data is extracted in batches, transformed based on a defined schema, and loaded into the data warehouse. Because traditional data warehouses capture a subset of data in batches and store data based on rigid schemas, they are unsuitable for handling real-time analysis or responding to spontaneous queries. Google designed BigQuery in part in response to these inherent limitations.
Innovative ideas are often slowed by the size and complexity of the IT organization that implements and maintains these traditional data warehouses. It can take years and substantial investment to build a scalable, highly available, and secure data warehouse architecture. BigQuery offers sophisticated software as a service (SaaS) technology that can be used for serverless data warehouse operations. This lets you focus on advancing your core business while delegating infrastructure maintenance and platform development to Google Cloud.
BigQuery offers access to structured data storage, processing, and analytics that’s scalable, flexible, and cost effective. These characteristics are essential when your data volumes are growing exponentially—to make storage and processing resources available as needed, as well as to get value from that data. Furthermore, for organizations that are just starting with big data analytics and machine learning, and that want to avoid the potential complexities of on-premises big data systems, BigQuery offers a pay-as-you-go way to experiment with managed services.
With BigQuery, you can find answers to previously intractable problems, apply machine learning to discover emerging data patterns, and test new hypotheses. As a result, you have timely insight into how your business is performing, which enables you to modify processes for better results. In addition, the end user’s experience is often enriched with relevant insights gleaned from big data analysis, as we explain later in this series.
The migration framework
Undertaking a migration can be a complex and lengthy endeavor. Therefore, we recommend adhering to a framework to organize and structure the migration work in phases:
- Prepare and discover: Prepare for your migration with workload and use case discovery.
- Assess and plan: Assess and prioritize use cases, define measures of success, and plan your migration.
- Execute: Iterate the following steps for each use case:
- Migrate (offload): Migrate only your data, schema, and downstream business applications.
- Migrate (full): Alternatively, migrate the use case fully end-to-end. The same as Migrate (offload), with the addition of the upstream data pipelines.
- Verify and validate: Test and validate the migration to assess return on investment.
The following diagram illustrates the recommended framework and shows how the different phases are connected:
For a deeper understanding, read Migrating data warehouses to BigQuery: Introduction and overview
5137
Of your peers have already watched this video.
15:30 Minutes
The most insightful time you'll spend today!
Analytics in a Multi-Cloud World with BigQuery Omni
Enterprises have more data at hand than they have ever had in the past. But are unable to leverage it fully.
“We have all of this data at our fingertips. But we just can’t quite get to it because we’re living in this world of data silos,” says Emily Rapp, Product Manager, Google Cloud.
Part of the problem is that over 80% of enterprises use more than one cloud provider. For many companies, leveraging a multi-cloud strategy is a competitive advantage, and sometimes a necessity. Multi-cloud allows organizations to meet users where they are and expedite their transformation journey to deliver a compelling customer experience.
So how can data teams manage? That’s why Google Cloud has introduced BigQuery Omni, a flexible, multi-cloud solution that lets you analyze data across clouds.
Watch the video to learn how to break down data silos in your environment and see how you can run analytics in a multi-cloud world.
Anthos for Manufacturing: Tackle DevOps Complexities and Drive Digital Transformation
DOWNLOAD E-BOOK6278
Of your peers have already downloaded this article
2:00 Minutes
The most insightful time you'll spend today!
Key Highlights on Data Analytics to Smooth Your Organization’s Data Journey
4868
Of your peers have already read this article.
2:00 Minutes
The most insightful time you'll spend today!
As the Olympics kicked off in Tokyo at the end of July, we found ourselves reflecting on the beauty of diverse countries and cultures coming together to celebrate greatness and sportsmanship. For this month’s blog, we’d like to highlight some key data and analytics performances that should help inspire you to reach new heights in your data journey.
Let’s review the highlights!
BigQuery ML Anomaly Detection: A perfect 10 for augmented analytics
Identifying anomalous behavior at scale is a critical component of any analytics strategy. Whether you want to work with a single frame of data or a time series progression, BigQuery ML allows you to bring the power of machine learning to your data warehouse.
In this blog released at the beginning of last month, our team walked through both non-time series and time-series approaches to anomaly detection in BigQuery ML:
- Non-time series anomaly detection
- Autoencoder model (now in Public Preview)
- K-means model (already in GA)
- Time-series anomaly detection
- ARIMA_PLUS time series model (already in GA)
These approaches make it easy for your team to quickly experiment with data stored in BigQuery to identify what works best for your particular anomaly detection needs. Once a model has been identified as the right fit, you can easily port that model into the Vertex AI platform for real-time analysis or schedule it in BigQuery for continued batch processing.
App Analytics: Winning the team event
Google provides a broad ecosystem of technologies and services aimed at solving modern day challenges. Some of the best solutions come when those technologies are combined with our data analytics offerings to surface additional insights and provide new opportunities.
Firebase has deep adoption in the app development community and provides the technology backbone for many organization’s app strategy. This month we launched a design pattern that shows Firebase customers how to use Crashlytics data, CRM, issue tracking, and support data in BigQuery and Looker to identify opportunities to improve app quality and enhance customer experiences.
Crux on BigQuery: Taking gold in the all-around data competition
Crux Informatics provides data services to many large companies to help their customers make smarter business decisions. While they were already operating on a modern stack and not on the hunt for a modern data warehouse, BigQuery became an enticing option due to performance and a more optimal pricing model. Crux also found advantages with lower-cost ingestion and processing engines like Dataflow that allow for streaming analytics.… when it came to building a centralized large-scale data cloud, we needed to invest in a solution that would not only suit our current data storage needs but also enable us to tackle what’s coming, supporting a massive ecosystem of data delivery and operations for thousands of companies.
Mark Etherington
Chief Technology Office, Crux Informatics
Technology is a team sport, and Crux found our support team responsive and ready to help. This decision to more deeply adopt Google Cloud’s data analytics offerings provides Crux with the flexibility to manage a constantly evolving data ecosystem and stay competitive.
You can read more about Crux’s decision to adopt BigQuery in this blog.
Google Trends: A classic emerges a champion
Following up on the launch of our Google Trends dataset in June, we delivered some examples of how to use that data to augment your decision making.
As a quick recap of that dataset, Google Cloud, and in particular BigQuery, provide access to the top 25 trending terms by Nielsen’s Designated Market Area® (DMA) with a weekly granularity. These trending terms are based on search patterns and have historically only been available on the Google Trends website.https://www.youtube.com/embed/9FJAXMF0ASc?enablejsapi=1&
The Google Trends design pattern addresses some common business needs, such as identifying what’s trending geographically near your stores and how to match trending terms to products to identify potential campaigns.
Dataflow GPU: More power than ever for those streaming sprints
Dataflow is our fully-managed data processing platform that supports both batch and streaming workloads. The ability of Dataflow to scale and easily manage unbounded data has made it the streaming solution of choice for large workloads with high-speed needs in Google Cloud.
But what if we could take that speed and provide even more processing power for advanced use cases? Our team, in partnership with NVIDIA, did just that by adding GPU support to Dataflow. This allows our customers to easily accelerate compute-intensive processing like image analysis and predictive forecasting with amazing increases in efficiency and speed.
Take a look at the times below:
Data Fusion: A play-by-play for data integration’s winning performance
Data Fusion provides Google Cloud customers with a single place to perform all kinds of data integration activities. Whether it’s ETL, ELT, or simply integrating with a cloud application, Data Fusion provides a clean UI and streamlined experience with deep integrations to other Google Cloud data systems. Check out our team’s review of this tool and the capabilities it can bring to your organization.
More Relevant Stories for Your Company
ESG Report: Economic Advantages of Google BigQuery OnDemand Serverless Analytics
Traditional big data solutions require a significant upfront investment, ongoing maintenance of hardware and software, and manual provisioning to match compute and storage resources with demand. Google’s serverless analytics warehouse does away with all that extra work, helping businesses focus on what matters most: getting value from their data. Enterprise
NVIDIA and Google Cloud Pave the Way for Single-cell Genomic Analysis
In the past decade, the Healthcare and Life Sciences industry has enjoyed a boon in technological and scientific advancement. New insights and possibilities are revealed almost daily. At Google Cloud, driving innovation in cloud computing is in our DNA. Our team is dedicated to sharing ways Google Cloud can be
Google Cloud’s Recommendation AI Helps Bazaarvoice with 60 Percent increase in CTR
Not long ago, building AI into recommendation engines was a daunting, expensive task that could take years to get off the ground. But as Bazaarvoice has shown, with the help of cloud services, the time from AI investment to business outcomes is shorter than ever. Bazaarvoice is the leading provider
Hospitals Can Offer Interconnected Patient Experiences Using Google’s Natural Language Services
Machine Learning (ML) in healthcare helps extract data from conversations, medical records, forms, research reports, insurance claims and other documents across the care value-chain to help care providers have a holistic view of their patients to draw insights for diagnoses and treatments. With Natural Language Processing(NLP), healthcare organizations can program
