Editor’s note: Technology today lets companies of any size take on entire industries simply with an innovative business model plus digital distribution. Take Shine, a French startup whose platform helps freelancers manage their finances — and their administrative commitments. Here, Raphael Simon, Shine’s CTO and co-founder, talks about why Shine built a new bank on Google Cloud Platform, and in particular Cloud Spanner.

More and more people are deciding to take the plunge and start a freelance career. Some of them by choice, others out of necessity. One of their biggest pain points is dealing with administrative tasks.

In some countries, especially in Europe, the administrative burden of being a freelancer is similar to what a company of 10 or more people deals with. A freelancer doesn’t necessarily have the time or skills to manage all this paperwork. So we are building a new bank for freelancers from the ground up that helps automate administrative tasks associated with their business.

Shine’s banking services and financial tools make it as easy to work as a freelancer as it is to work for a larger company. We deal with administrative tasks on behalf of the freelancer so that he or she can focus on their job: finding and wowing clients.

Building our infrastructure 

As a new bank, we had the opportunity to build our infrastructure from the ground up. Designing an infrastructure and choosing a database presents tough decisions, especially in the financial services world. Financial institutions come under tremendous scrutiny to demonstrate stability and security. Even a tiny leak of banking data can have tremendous consequences both for the bank and its clients, and any service interruption can trigger a banking license to be suspended or a transaction to be declined.

At the same time, it’s vital for us to optimize our resources so we can maximize the time we spend developing user-facing features. In our first six months, we iterated and validated a prototype app using Firebase, and secured our seed funding round (one of the largest in Europe in 2017).

Based on our positive experience with Firebase, plus the ease-of-use and attractive pricing that Google Cloud offered, we decided to build our platform on Google Cloud Platform (GCP).

We were drawn to GCP because it has a simple, consistent interface that is easy to learn. We chose App Engine flexible environment with Google Cloud Endpoints for an auto-scaling microservices API. These helped us reduce the time, effort, and cost in terms of DevOps engineers, so we could invest more in developing features, while maintaining our agility.

We use Cloud Identity and Access Management (Cloud IAM) to help control developer access to critical parts of the application such as customer bank account data. It was quite a relief to lean on a reliable partner like Google Cloud for this.

Database decisions 

Next came time to choose a database. Shine lives at the financial heart of our customers’ businesses and provides guidance on things like accounting and tax declaration. The app calculates the VAT for each invoice and forecasts the charges they must pay each quarter.

Due to the sensitivity of our customers’ data, the stakes are high. We pay careful attention to data integrity and availability and only a relational database with support for ACID transactions (Atomicity, Consistency, Isolation, Durability) can meet this requirement.

At the same time, we wanted to focus on the app and user experience, not on database administration or scalability issues. We’re trying to build the best possible product for our users, and administering a database has no direct value for our customers. In other words, we wanted a managed service.

Cloud Spanner combines a globally distributed relational database service with ACID transactions, industry-standard SQL semantics, horizontal scaling, and high availability. Cloud Spanner provided additional security, high-availability, and disaster recovery features out-of-the-box that would have taken months for us to implement on our own. Oh, and no need to worry about performance — Cloud Spanner is fast. Indeed, Cloud Spanner has been a real asset to the project, from the ease-of-use of creating an instance to scaling the database.

Cloud Spanner pro tips 

We began working with Cloud Spanner and have learned a lot along the way. Here are some technical notes about our deployment and some best practices that may be useful to you down the road:

• Cloud Spanner allows us to change a schema in production without downtime. We always use a NOT NULL constraint, because we generally think that using NULL leads to more errors in application code. We always use a default value when we create an entity through our APIs and we use Cloud Dataflow to set values when we change a schema (e.g., adding a field to an entity). 
• With microservices, it’s generally a good practice to make sure every service has its own database to ensure data isolation between the different services. However, we adopted a slightly different strategy to optimize our use of Cloud Spanner. We have an instance on which there are three databases — one for production, one for staging and one for testing our continuous integration (CI) pipeline. Each service has one or more interleaved tables that are isolated from others services’ tables (we do not use foreign-keys between tables from different services). This way our microservices data are not tightly “coupled”. 
• We created an internal query service that performs read-only queries to Cloud Spanner to generate a dashboard or do complex queries for analytics. It is the only service where we allow joins between tables across services. 
• We take advantage of Cloud Spanner’s scalability, and thus don’t delete any data that could one day be useful and/or profitable. 
• We store all of our business logs on Cloud Spanner, for example connection attempts to the application. We append the ‘-Logs’ suffix to them. 
• When possible, we always create an interleave. 

 In short, implementing Cloud Spanner has been a good choice for Shine:

• It’s saved us weeks, if not months, of coding. 
• We feel we can rely on it since it’s been battle-tested by Google. 
• We can focus on building a disruptive financial services product for freelancers and SMBs.

And because Cloud Spanner is fully managed and horizontally scalable, we don’t have to worry about hardware, security patches, scaling, database sharding, or the possibility of a long and risky database migration in the future. We are confident Cloud Spanner will grow with our business, particularly as we expand regionally and globally. I strongly recommend Cloud Spanner to any company looking for a complete database solution for business-critical, sensitive, and scalable data.

The music industry is rapidly changing. Only 20 years ago, the availability of music and the infrastructure that was required to make an album a sales success were incredibly complex and expensive. With the decline of physical sales and a fundamental shift to digital, music streaming now accounts for more than half of all sales globally.

At the same time, technology has democratized music-making; in many ways, it has made the musical landscape more diverse. Artists can upload their music with the click of a button. But while it’s easier for creators to share their songs with audiences, getting paid has become more fragmented.

Although music is booming, people no longer buy it outright. Instead, listeners download music digitally or subscribe to streaming services to have their libraries with them at all times. To monetize digital content effectively, artists need to know when, where, and how often their songs are played on each service. To help them navigate this complicated royalties landscape and maximize their profits, Berlin-based international music company BMG provides customized, transparent, and fair services to songwriters and artists.

With publishing and recording divisions under one roof, the subsidiary of international media giant Bertelsmann works with both emerging artists and established stars, including John Legend, Kylie Minogue, Mick Jagger, and Keith Richards. With the MyBMG web and mobile application, clients can view and analyze their royalty details in real time and collect payment. When a new record is released, BMG uses data to maximize its impact and revenue for its creators.

“We make sure that everyone who uses our clients’ music knows who needs to be paid the associated royalties, then we collect these royalties and share them out quickly and transparently,” explains Sebastian Hentzschel, Chief Information Officer at BMG. “When our artists release new music, we make sure that it’s marketed and promoted effectively around the world.”

“We needed a scalable solution for our royalty workloads that was intuitive for our developers. We also wanted a partner, not a client-vendor relationship. With autoscaling via BigQuery, excellent customer support, and a clean and simple user interface, Google Cloud ticks every box for us.”

—Gaurav Mittal, Vice President Group Technology, BMG

Getting up to speed with a new way of paying artists

In this digital world, artists aren’t just paid every time a fan buys an album—they’re paid a small amount, or royalty, for each song downloaded or streamed by a listener. So, when the industry shifted to digital, the volume of data that BMG needed to handle grew exponentially. “One CD sale is equivalent to about 1,500 streamed songs or plays,” says Gaurav Mittal, Vice President Group Technology at BMG. “That means IT departments have to process 1,500 times the amount of data to calculate payments for artists, and this makes scalable micropayment processing very important.”

Until 2019, BMG’s infrastructure was entirely hosted on-premises. Hardware limitations made it challenging to scale on-demand, making it harder to handle the data peaks that royalty processing can bring. “With our on-premises infrastructure, we were going to hit a ceiling in a few years,” says Gaurav. “We still managed to process royalty payments for our clients, but it was increasingly time consuming and expensive. To keep focusing on our clients, rather than our infrastructure, we decided to migrate to Google Cloud.”

From the outset, Gaurav and his team had a clear vision for the partnership: “Most importantly, we needed a scalable solution for our royalty workloads that was intuitive for our developers. We also wanted a partner, not a client-vendor relationship,” he says. “With autoscaling via BigQuery, excellent customer support, and a clean and simple user interface, Google Cloud ticks every box for us.”

Keeping artists happy with business-as-usual payouts during migration

To move applications to the cloud while keeping payment cycles on track for its artists, BMG teamed up with Google Cloud partner Rackspace Technology. “We selected Rackspace Technology because it combines strong technical muscle and a global footprint, with the customer service of a local boutique firm,“ shares Gaurav.

BMG’s own technology team put together the outline for the Google Cloud architecture, which they passed on to Rackspace Technology for optimizations and the ultimate stamp of approval. Whenever Gaurav and his developers needed support, Rackspace Technology was ready to go. “So far, we’ve migrated 17 applications successfully, and Rackspace Technology has been 100% spot-on with each suggestion,” says Gaurav. “I can’t recall a single flaw in a Rackspace Technology-approved architecture, and that really says something.”

When BMG began its migration in August 2019, the team developed an ambitious two-year plan. Only 14 months later, however, the project is more than 75% complete. Among the solutions that BMG is using today are Cloud Storage to securely store 130 TB of data, and Cloud SQL as its standard database technology. The web applications run on Compute Engine, App Engine, and Google Kubernetes Engine.

“After successfully moving a few applications, it was clear that with the strong teamwork of BMG and Rackspace Technology, together with the ease of use of Google Cloud, we could speed up the project without sacrificing quality,” says Gaurav. “We’re set to complete our migration six months before schedule, helping us to quickly move out of our hybrid environment.”

“Our income-tracking teams are very savvy on the data, and the simplicity of Google Cloud empowers them to self-serve analytics, rather than wait for IT. Our teams are much more productive.”

—Gaurav Mittal, Vice President Group Technology at BMG

Royalty reporting and processing with BigQuery and Dataproc

So far, all of BMG’s critical workloads are up and running on Google Cloud. Royalty calculations, for example, which require incredible processing power and the collection of many micropayments to ensure full and timely payout, run entirely on Dataproc with output stored on BigQuery for downstream integration and reporting.

Enabling more harmonious workflows through self-serve analytics

As the new beating heart of BMG’s royalty reporting, BigQuery changed the rhythm of collaboration company-wide. In the past, income tracking teams had to contact IT departments if they needed deeper data insights for their work. By integrating Data Catalog with BigQuery, BMG has made the data more accessible to all teams. This helps them detect missing income and new revenue streams independently, maximizing profits for artists.

“Our income-tracking teams are very savvy on the data, and the simplicity of Google Cloud empowers them to self-serve analytics, rather than wait for IT,” says Gaurav. “Our teams are much more productive.”

“Google Cloud enables us to be more client focused and deliver better features faster. We believe it’s just the beginning. We offer rights and royalty services for music publishing, recorded music, neighboring rights, and books. Without scalability limitations, it’s absolutely conceivable to offer our platform as a service to other companies or industries, such as gaming. Google Cloud has opened a world of possibilities.”

—Sebastian Hentzschel, Chief Information Officer, BMG

With a leaner IT environment, BMG can focus its effort on the needs of its clients. Beyond improvements in royalty processing, it can concentrate on app development, releasing new features and enhancements more frequently. By hosting applications on Google Kubernetes Engine, App Engine, and Compute Engine, BMG has built a CI/CD pipeline with automated deployments and testing to significantly speed up workflows.

“In our old system, it could take several weeks to set up an environment,” says Gaurav. “With Google Kubernetes Engine, any of our developers can complete the process in a few clicks. Having that autonomy makes our developers more motivated and self-driven.”

“Google Cloud enables us to be more client focused and deliver better features faster,” adds Sebastian. “We believe it’s just the beginning. We offer rights and royalty services for music publishing, recorded music, neighboring rights, and books. Without scalability limitations, it’s absolutely conceivable to offer our platform as a service to other companies or industries, such as gaming. Google Cloud has opened a world of possibilities.”

With the migration almost complete, BMG is looking forward to its next technology project. It plans to leverage AutoML to further scale and automate royalty tracking with machine learning. On the marketing side, advanced analytics will help BMG determine the effectiveness of promotional campaigns around the world, further increasing profits for artists. By connecting Google Data Studio to BigQuery, BMG will increase the quality of its analyses, helping musicians better understand the reach of their music around the world.

In the end, Gaurav shares, helping musicians is what it all comes down to. “We’re a new kind of music company because we build our services around our artist, songwriter, and publisher clients, not the other way around,” he says. “Google Cloud is helping us maintain strong relationships with our clients, and that’s music to our ears.”

In November 2021, we announced the general availability of Tau VMs. Since then, Google Cloud’s Tau VMs with Google Kubernetes Engine (GKE) have unlocked value for many customers who are now using Tau VMs for their production workloads, such as: Ascend, who achieved over 125% higher performance; Nylas, who gained over 40% higher price-performance; and OpenX, who achieved 40% better price-performance while at the same time reducing their application latency by 62%.

T2D is the first instance type in the Tau VM family and is built on the latest 3rd generation AMD EPYCTM processors, offering 42% higher price-performance compared to general-purpose VMs from any of the leading public cloud vendors. Tau VMs offer a leading combination of performance, price, and full x86 compatibility, offering customers the lowest cost solution for scale-out workloads. Tau VMs are available in predefined shapes, with up to 60vCPUs per VM, 4GB of memory per vCPU, networking up to 32 Gbps and a slew of storage options including Standard, Balanced and Performance PD. Tau VMs are also available as Spot VMs, offering an over 60% discount compared to on-demand pricing.

For customers looking for advanced container orchestration, GKE delivers high levels of reliability, security, and scalability, and has supported Tau VMs since the day they became available on Google Cloud. Tau VMs are ideal for CPU-bound workloads such as web-serving with encryption, video encoding, compression/decompression, image processing and horizontally-scaled applications. Using Tau VMs along with GKE’s cost-optimization best practices can help lower your total cost of ownership. You can add Tau VMs to new or existing GKE clusters by specifying the Tau T2D machine type in your GKE node-pools through the Cloud console or by using –machine-type in gcloud.

Here is what some of our customers have to say about Tau VMs:

Ascend provides a unified analytics and data engineering platform, and chose Tau VMs along with GKE to run their data-intensive workload — primarily because of Tau’s absolute performance and price-performance advantage.

“Our core capability at Ascend is bringing together data ingestion, transformation, delivery, orchestration and observability into a single platform. To operate at scale and keep pace with our telemetry data production rates, high single-threaded performance is critical. With Google Cloud’s Tau VMs with Google Kubernetes Engine (GKE), we are able to achieve over 125% higher performance than previous generation families. This has completely changed our ability to query historical metrics. Where previously metric queries against historical data over ranges longer than a couple hours were difficult, we can now easily query data ranges of multiple weeks.” – Joe Stevens, Tech Lead – Infrastructure, Ascend.io

Nylas is a pioneer and leading provider of productivity infrastructure solutions for modern software. In the past year, Nylas has been using GKE in their journey to reinvent their architecture and provide their enterprise customers with a bi-directional universal email sync, security compliance with the highest enterprise standards, and industry-specific machine learning services.

“For our core application, Google’s Tau VMs with Google Kubernetes Engine delivers over 40% better price-performance than Amazon’s Graviton-based VMs. Further, Tau VMs maintain x86 compatibility and eliminate the need to maintain a separate stack for ARM. We are moving our workload from Amazon Web Services to Google Cloud to take advantage of these benefits.” – David Ting, SVP of Engineering, Nylas

OpenX operates an independent ad exchange. Operating 100% on Google Cloud has enabled OpenX to achieve improved performance, scalability, speed and global reach.

“At OpenX, our ad-exchange services over 200 billion requests every day. Getting the best combination of performance and price from the infrastructure is critically important for us. We use multiple Google Kubernetes Engine (GKE) clusters across geographic regions with autoscaling to power our ad-delivery components. Running Google Cloud’s Tau VMs with GKE has enabled over 40% better price-performance and 62% latency reduction for our application as compared to the prior generation family. We have made the move to Tau VMs for our application to take advantage of these benefits.” – Paul T.Ryan, CTO, OpenX

We are excited to see Tau VMs adding value for so many of our customers by enabling industry leading price-performance for a variety of workloads.

If you haven’t tried Tau VMs yet, give them a try today in our Iowa, Netherlands and Singapore regions and move your production workloads to Tau VMs. Tau VMs will be arriving in additional regions and zones in the coming weeks. You can provision GKE node pools based on Tau VMs and explore how you can take advantage of improved price-performance for your scale-out containerized workloads.

To get started, go to the Google Cloud Console, select Google Kubernetes Engine, and choose Tau T2D for your GKE nodes. To learn more about Tau VMs or other Compute Engine VM options, check out our machine types and our pricing pages.

Communication services providers (CSPs) are experiencing a period of disruption. Overall revenue growth is decelerating and is projected to remain below 1 percent per year, following a trend that started even before the pandemic.¹ At the same time, driven by the pandemic, data consumption in 2020 increased by 30 percent relative to 2019, with some operators even reporting increases of 60 percent.² 

The combination of pressure on revenues with rising data traffic costs is forcing operators to innovate in three fundamental ways. First, operators are looking to establish new sources of revenue. Second, increased network utilization must be met with a reduction in network cost. And third, there is an opportunity to gain new customers by improving the customer experience.

Fortunately, 5G offers a path forward across each of these three areas. Concepts such as network slicing and private networks allow CSPs to offer differentiated network services to public sector and enterprise customers. The disaggregation of hardware and software allows new vendors with unique strengths to enter the market and to enable CSPs to build, deploy, and operate networks in fundamentally new ways. And the ability to place workloads at the edge permits CSPs to offer compelling experiences to consumers and businesses alike. In this blog, we will discuss how CSPs can create a solid foundation for their cloud networks. 

Understanding telecommunications networks

First, it is useful to consider the way telecommunications networks were traditionally built. Initially, networks were built using physical network functions (PNFs) — appliances that used a tight combination of hardware and software to perform a specific function. PNFs offered the benefit of being purpose-built for a specific application, but they were inflexible and difficult to upgrade. As an example, deploying new features frequently required replacing the entire PNF, i.e., deploying a new hardware appliance.

The first step in improving deployment agility came with the concept of virtualized network functions (VNFs), software workloads designed to operate on commercial off-the-shelf (COTS) hardware. Rather than utilizing an integrated hardware and software appliance, VNFs disaggregated the hardware from the software. As such, it became possible to procure the hardware from one vendor and the software from another. It also became possible to separate the hardware and software upgrade cycles. 

However, while VNFs offered advantages over PNFs, VNFs were still an intermediate step. First, they typically needed to be run within a virtual machine (VM), and as such required a hypervisor to interface between the host operating system (OS) and the guest OS inside the VM. The hypervisor consumed CPU cycles and added inefficiency. Second, the VNF itself was frequently designed as a monolithic function. This meant that while it was possible to upgrade the VNF separately from the hardware, such an upgrade, even for a feature that affected only a portion of the VNF, required deployment of the entire large VNF. This created risk and operational complexity, which in turn meant that upgrades were delayed just as they were with PNFs.

Creating the foundation for cloud networks

The trick to establishing your cloud based network resides in the challenge of moving from VNFs to containerized network functions (CNFs) — network functions organized as containers as a collection of small programs, each of which can be independently operated. 

The concept of containers is not new. In fact, Google has been using containerized workloads for over 15 years. Kubernetes, which Google developed and open-sourced, is the world’s most popular container orchestration system, and is based on Borg, Google’s internal container management system.³ There are lots of benefits to using containers, but fundamentally, it frees developers from worrying about resource scheduling, interprocess communication, security, self-healing, load balancing, and many other tedious (but important!) tasks. 

Consider just a couple examples of benefits that containerization brings to network functions. First, when upgrading the network function to implement new features, you no longer need to re-deploy the entire network function. Instead, you only need to re-deploy the containers that are affected by the upgrade. This improves developer velocity and reduces the risk of the upgrade because, rather than infrequent upgrades that each introduce substantial changes, you can now have frequent upgrades that each deploy small changes. Small changes are less risky because they are easier to understand and to roll back in case of anomaly. Incidentally, this also improves your security posture because it reduces the time between when a security vulnerability is discovered and when a patch is deployed.

Speaking of security, another example of the benefits that containerization brings to network functions is an automatic zero-trust security posture. In Kubernetes, the communication among microservices can be handled by a service mesh, which manages mundane aspects of inter-services communication such as retries in case of failure and providing observability into communication. It can also manage other essential aspects such as security. For example, Anthos Service Mesh, which is a fully-managed implementation of the open-source Istio service mesh (also co-developed by Google), includes the ability to authenticate and encrypt all communications using mutual TLS (mTLS) and to deploy fine-grained access control for each individual microservice.

Automation and orchestration for cloud networks

CNFs bring tremendous benefits, but they also bring challenges. In place of a relatively small number of network appliances, we now have a large number of containers, each of which requires configuration, management, and maintenance. In the past, many of these processes were accomplished using manual techniques, but this is impossible to accomplish economically and reliably at the scale required by CNFs.

Fortunately, there are cloud-native approaches to solving these challenges. First, consider the problem of autonomously deploying and maintaining CNFs. The ideal way is to use the concept of Configuration as Data. Unlike imperative techniques such as Infrastructure as Code, which provide a detailed description of a sequence of steps that need to be executed to achieve an objective, Configuration as Data is a declarative method whereby the user specifies the desired end state (i.e., the actual desired configuration) and relies on automated controllers to continuously drive the infrastructure to achieve that state. Kubernetes includes such automated controllers, and the great news is that this method can be used not just for infrastructure but also for the applications residing on top of it, including CNFs. This cloud-native technique frees you from the toil and associated risk of writing detailed configuration procedures, so you can focus on the business logic of your applications.

As another example, consider the problem of understanding your network performance, including anomaly detection, root cause analysis, and resolution. The cloud-native approach starts with creating a data platform where both infrastructure and CNF monitoring data can be ingested, regularized, processed, and stored. You can then correlate data sets against each other to detect anomalies, and with AI/ML techniques, you can even anticipate anomalies before they happen. AI/ML is likewise indispensable in gaining an understanding of why the anomaly is happening, i.e. performing root cause analysis, and automated closed-loop controllers can be developed to correct the problem, ideally before it even happens.

Architecting for the edge

The transition from VNFs to CNFs is a critical piece in addressing the challenge that CSPs face today, but it alone is not enough. CNFs need infrastructure to run on, and not all infrastructure is created equal. 

Consider a typical 5G network. There are some functions, such as those associated with an access network, that need to be deployed at the edge. These functions require low latency, high throughput, or even a combination of the two. In 5G networks, examples of such functions include the radio unit (RU), distributed unit (DU), centralized unit (CU), and the user plane function (UPF). The first three are components of the radio access network (RAN), while the last is a component of the 5G core. At the same time, there are some other control plane functions such as the session management function (SMF) or the authentication and mobility management function (AMF) that do not have such tight latency and high throughput requirements and can thus be placed in a more centralized data center. Furthermore, consider an AI/ML use case where a particular model (perhaps for radio traffic steering) needs to run at the network edge because of its latency requirements. While the model itself needs to run at the edge, model training (i.e., generating the model coefficients) is frequently a compute-intensive exercise that is latency-insensitive and is thus more optimal to run in a public cloud region.

All of these use cases have one thing in common: they call for a hybrid deployment environment. Some applications must be deployed at the edge as close to the user as possible. Others can be deployed in a more centralized environment. Still others can be deployed in a public cloud region to take advantage of the large amount of compute and economies of scale available therein. Wouldn’t it be convenient — if not transformational — if you could use a single environment for deploying at the edge, in a private datacenter, and in public cloud, with a consistent set of security, lifecycle management, policy, and orchestration resources across all such locations? This is indeed what Google Distributed Cloud, enabled by Anthos, brings to the table.

With Google Distributed Cloud, you can architect a 5G network deployment such as the one shown below.

Business benefits of cloud networks

Beyond the technical benefits, consider the business benefits of such an architecture. First, by following the best practices of hardware and software disaggregation, it permits the CSP to procure the infrastructure and the network functions from different vendors, spurring competition among vendors. Second, each workload is placed in precisely the right location, enabling efficient utilization of hardware resources and offering compelling low-latency, high-throughput services to users. Third, because the architecture utilizes a common hybrid platform (Anthos), it makes it easy to move workloads across infrastructure locations. Fourth, the separation of workloads into microservices accelerates time-to-market when developing new features or applications, such as those enabling enterprise use cases. And finally, the container management platform supports the simultaneous deployment of both network functions and edge applications on the same infrastructure, allowing the operator to deploy new experiences such as AR/VR directly on bare metal as close to the user as possible.

The next generation cloud network is now

There is a lot more we could say, but perhaps the most important takeaway is that this architecture is not a future dream. It exists today, and Google is working with leading CSPs and network vendor partners to deploy it, helping them realize the promise of 5G to deliver new revenues, reduce operating costs, and enable new customer experiences.

To learn more, watch the video series on the cloudification of CSP networks.

Discover what’s happening at the edge: How CSPs Can Innovate at the Edge.

^{1.Statista, Forecast growth worldwide telecom services spending from 2019 to 2024
2 PricewaterhouseCoopers, Global entertainment and media outlook 2021-2025
3. Borg: The Predecessor to Kubernetes}

When critical services fail, businesses risk losing revenue, productivity, and trust. That’s why Google Cloud customers running SAP applications choose to deploy high availability (HA) systems on Google Cloud.

In these deployments Linux operating system clustering provides application and guest awareness for the application state and automates recovery actions in case of failure — including cluster node, resource or node failover or failed action. 

Pacemaker is the most popular software Linux administrators use to manage their HA clusters, which includes automating notifications about events — including failover fencing and node, attribute, and resource events — and reporting on events. With automated alerts and reports, Linux administrators can not only learn about events as they happen, but they can also make sure other stakeholders are alerted to take action when critical events occur. They can even discover past events to assess the overall health of their HA systems. 

Here, we break down the steps to setting up automated alerts for HA cluster events and alert  reporting.

How to Deploy the Alert Script

To set up event-based alerts, you’ll need to take the following steps to execute the script. 

1. Download the script file ‘gcp_crm_alert.sh’ from

https://github.com/GoogleCloudPlatform/pacemaker-alerts-cloud-logging

2. Under root user, add exec flag for the script and execute deployment with:

  chmod +x ./gcp_crm_alert.sh
./gcp_crm_alert.sh -d

3. Confirm that the deployment runs successfully. If it does, you will see the following INFO log messages:

In the Red Hat Enterprise Linux (RHEL) system:

gcp_crm_alert.sh:2022-01-24T23:48:30+0000:INFO:'pcs alert recipient add gcp_cluster_alert value=gcp_cluster_alerts id=gcp_cluster_alert_recepient options value=/var/log/crm_alerts_log' rc=0

In the SUSE Linux Enterprise Server (SLES):

gcp_crm_alert.sh:2022-01-25T00:13:27+00:00:INFO:'crm configure alert gcp_cluster_alert /usr/share/pacemaker/alerts/gcp_crm_alert.sh meta timeout=10s timestamp-format=%Y-%m-%dT%H:%M:%S.%06NZ to { /var/log/crm_alerts_log attributes gcloud_timeout=5 gcloud_cmd=/usr/bin/gcloud }' rc=0

Now, in the event of a cluster node, resource, node failover, or failed action, Pacemaker will start the alert mechanism. For further details on the alerting agent, check out the Pacemaker Explained documentation.

How to Use Cloud Logging for Alert Reporting

Alerted events are published in Cloud Logging. Below is an example of the log record payload, where the cluster alert key-value pairs get recorded in the jsonPayload node.

{

 "insertId": "ktildwg1o3fbim",   "jsonPayload": {     "CRM_alert_recipient": "/var/log/crm_alerts_log",     "CRM_alert_attribute_name": "",     "CRM_alert_kind": "resource",     "CRM_alert_status": "0",     "CRM_alert_rsc": "STONITH-sapecc-scs",     "CRM_alert_rc": "0",     "CRM_alert_timestamp_usec": "",     "CRM_alert_interval": "0",     "CRM_alert_node_sequence": "21",     "CRM_alert_task": "start",     "CRM_alert_nodeid": "",     "CRM_alert_timestamp": "2022-01-25T00:17:06.515313Z",     "CRM_alert_timestamp_epoch": "",     "CRM_alert_desc": "ok",     "CRM_alert_target_rc": "0",     "CRM_alert_version": "1.1.15",     "CRM_alert_attribute_value": "",     "CRM_alert_node": "sapecc-ers",     "CRM_alert_exec_time": ""   },   "resource": {     "type": "global",     "labels": {       "project_id": "gcp-tse-sap-on-gcp-lab"     }   },   "timestamp": "2022-01-25T00:17:09.662557309Z",   "severity": "INFO",   "logName": "projects/gcp-tse-sap-on-gcp-lab/logs/sapecc-ers%2F%2Fvar%2Flog%2Fcrm_alerts_log", "receiveTimestamp": "2022-01-25T00:17:09.662557309Z" 

}

To get notified of a resource event — for example, when the HANA topology resource monitor fails — you can use the following filter for the alerting definition:

jsonPayload.CRM_alert_node=("hana-venus" OR "hana-mercury") -jsonPayload.CRM_alert_status="0" jsonPayload.CRM_alert_rsc="rsc_SAPHanaTopology_SBX_HDB00" jsonPayload.CRM_alert_task="monitor" To define an alert for a fencing event, your can apply this filter: jsonPayload.CRM_alert_node=("hana-venus" OR "hana-mercury") jsonPayload.CRM_alert_kind="fencing" The fencing log entry gets recorded with warning severity to give you deeper insight, and this additional information is also helpful for more specific filtering criteria: {   "insertId": "1plznskfjsxt82",   "jsonPayload": {     "CRM_alert_attribute_value": "",     "CRM_alert_recipient": "/var/log/crm_alerts_log",     "CRM_alert_rsc": "",     "CRM_alert_rc": "0",     "CRM_alert_timestamp_usec": "529261",     "CRM_alert_desc": "Operation reboot of hana-mercury by hana-venus for crmd.2361@hana-venus: OK (ref=2a9bf814-9adf-4247-af3f-94ac254fc3ca)", "CRM_alert_target_rc": "",     "CRM_alert_nodeid": "",     "CRM_alert_kind": "fencing",     "CRM_alert_node_sequence": "33",     "CRM_alert_task": "st_notify_fence",     "CRM_alert_status": "",     "CRM_alert_exec_time": "",     "CRM_alert_attribute_name": "",     "CRM_alert_timestamp_epoch": "1643072786",     "CRM_alert_version": "1.1.19",     "CRM_alert_timestamp": "2022-01-25T01:06:26.529261Z",     "CRM_alert_interval": "",     "CRM_alert_node": "hana-mercury"   },   "resource": {     "type": "global",     "labels": {       "project_id": "gcp-tse-sap-on-gcp-lab"     }   },   "timestamp": "2022-01-25T01:06:27.267017052Z",   "severity": "WARNING",   "logName": "projects/gcp-tse-sap-on-gcp-lab/logs/hana-venus%2F%2Fvar%2Flog%2Fcrm_alerts_log", "receiveTimestamp": "2022-01-25T01:06:27.267017052Z" 
} 

Alerts can be delivered through multiple channels, including text and email. Below is an example of an email notification for our earlier example, when we defined an alert for a HANA topology resource monitor failure:

You can write and apply filters to your log-based alerts to isolate certain types of incidents and analyze events over time. For example, the following script will surface a resource event occurring within a two-hour window on a specific date:

timestamp>="2022-01-25T00:00:00Z" timestamp<="2022-01-25T02:00:00Z"

jsonPayload.CRM_alert_kind="resource"

With the ability to analyze these logged alerts over time, determine whether event patterns warrant any action.

[SIDEBAR]

The alert script prints details in the standard output and in the log file /var/log/crm_alerts_log, and this can grow over time. We recommend that the log file is set with the Linux logrotate service in order to limit the file system space. Use the following command to create the necessary logrotate setting for the alerting log file:

cat > /etc/logrotate.d/crm_alerts_log << END-OF-FILE  /var/log/crm_alerts_log {   create 0660 root root   rotate 7   size 10M   missingok   compress   delaycompress   copytruncate   dateext   dateformat -%Y%m%d-%s   notifempty } END-OF-FILE 

[END SIDEBAR]

Tips for Troubleshooting When you first deploy your alert script, how can you tell for certain that you’ve done it correctly? Use the following commands to test it out:

In RHEL:

pcs alert show 

In SLES:

sudo crm config show | grep -A3 gcp_cluster_alert 

You should see the following if the script is correct:

In RHEL:

Alerts:  Alert: gcp_cluster_alert (path=/usr/share/pacemaker/alerts/gcp_crm_alert.sh)   Description: "Cluster alerting for hana-node-X"   Options: gcloud_cmd=/usr/bin/gcloud gcloud_timeout=5   Meta options: timeout=10s timestamp-format=%Y-%m-%dT%H:%M:%S.%06NZ   Recipients:    Recipient: gcp_cluster_alert_recepient (value=gcp_cluster_alerts)     Options: value=/var/log/crm_alerts_log In SLES:

alert gcp_cluster_alert "/usr/share/pacemaker/alerts/gcp_crm_alert.sh" \ meta timeout=10s timestamp-format="%Y-%m-%dT%H:%M:%S.%06NZ" \ to "/var/log/crm_alerts_log" attributes gcloud_timeout=5 gcloud_cmd="/usr/bin/gcloud" 

If the commands do not display the alerts properly, re-deploy the script.

In case there is an issue with the script, or if the Cloud Logging records are not presenting as expected, examine the script log file /var/log/crm_alerts_log. The errors and warning can be filtered with:

egrep '(ERROR|WARN)' /var/log/crm_alerts_log 

Any Pacemaker alert failures will be recorded in the messages and/or Pacemaker log. To examine recent alert failures, use the following command:

egrep '(gcp_crm_alert.sh|gcp_cluster_alert)' \   /var/log/messages /var/log/pacemaker.log 

Keep in mind, though, that the Pacemaker log location may be different in your system from the one in the example above.

From reactive to proactive

Your SAP applications are too critical to risk outages. The most effective way to manage high availability clusters for your SAP systems on Google Cloud is to take full advantage of Pacemaker’s alerting capabilities, so you can be proactive in ensuring your systems are healthy and available.

Learn more about running SAP on Google Cloud.