As the pace of digital transformation accelerates, our partnership with VMware continues to focus on helping customers successfully navigate their cloud journey and achieve their business objectives through seamless and rapid migration of business critical VMware workloads.

We announced the general availability of Google Cloud VMware Engine in May 2020. Since then we have worked closely with VMware to make it easier for customers to quickly migrate and run business-critical, VMware-based workloads on Google Cloud. Customers are already leveraging the service across a variety of use cases including application migration, datacenter exit, virtual desktop infrastructure, disaster recovery, and spinning up new capacity quickly to meet business needs.

For example, retailer Carrefour migrated its on-premises VMware workloads to Google Cloud without disruption to shoppers or employees while reducing operating costs by 40% and energy consumption by 45%. Once in the cloud, Carrefour was able to leverage its data and AI to deliver innovative customer experiences across online and in-store channels. Similarly, telecommunications provider Mitel migrated thousands of VMware instances across 30 data centers to Google Cloud in less than 90 days, quickly achieving increased stability, scale, and security.

Today, we announced the continued growth of the Google Cloud and VMware partnership with the addition of Google Cloud VMware Engine within VMware Cloud Universal. Google Cloud VMware Engine delivers a cloud-native VMware experience and enables you to rapidly migrate to the cloud without changes to your apps, policies, or tools. Once you migrate your VMware workloads to Google Cloud, you can accelerate digital transformation through seamless access to services such as BigQuery for real-time data analytics and cloud-native container-based architectures on Kubernetes.

With VMware Cloud Universal, you will be able to accelerate migrations of your workloads and applications to Google Cloud through purchase of Google Cloud VMware Engine from VMware and its partners, allowing you to flexibly purchase credits, and leverage existing spend and unused VMware Cloud Universal credits. The program will offer the following benefits:

• Financial flexibility by letting you redeem VMware Cloud Universal credits for Google Cloud VMware Engine
• Streamlined consumption by enabling you to burn down your Google Cloud commits while purchasing from VMware
• Use of existing VMware licensing investments through the VMware Cloud Universal program for Google Cloud VMware Engine

With Google Cloud VMware Engine, you can take advantage of Google Cloud’s highly performant, scalable infrastructure with fully redundant and dedicated 100 Gbps networking, providing 99.99% availability to meet the needs of the most demanding workloads at very low costs. By providing a consistent VMware environment natively in Google Cloud, you can quickly migrate your VMware workloads to Google Cloud without changes. Deep and unique networking integrations and capabilities such as multi-region and multi-VPC connectivity, further ease the migration of complex enterprise networking topologies to Google Cloud. With rapid provisioning of private clouds across 13 global regions, you can also take advantage of on-demand capacity to serve your infrastructure needs with a cloud-native VMware environment in Google Cloud.Once in the cloud, you can take advantage of other Google Cloud services such as BigQuery and Cloud Operations to gain data-driven insights and unify operations.

Getting started

With Google Cloud VMware Engine as part of VMware Cloud Universal, Google Cloud is a compelling cloud destination for your VMware workloads. You can learn more about how to get started with the service and get additional detail around use cases and pricing on our website.

When critical services fail, businesses risk losing revenue, productivity, and trust. That’s why Google Cloud customers running SAP applications choose to deploy high availability (HA) systems on Google Cloud.

In these deployments Linux operating system clustering provides application and guest awareness for the application state and automates recovery actions in case of failure — including cluster node, resource or node failover or failed action. 

Pacemaker is the most popular software Linux administrators use to manage their HA clusters, which includes automating notifications about events — including failover fencing and node, attribute, and resource events — and reporting on events. With automated alerts and reports, Linux administrators can not only learn about events as they happen, but they can also make sure other stakeholders are alerted to take action when critical events occur. They can even discover past events to assess the overall health of their HA systems. 

Here, we break down the steps to setting up automated alerts for HA cluster events and alert  reporting.

How to Deploy the Alert Script

To set up event-based alerts, you’ll need to take the following steps to execute the script. 

1. Download the script file ‘gcp_crm_alert.sh’ from

https://github.com/GoogleCloudPlatform/pacemaker-alerts-cloud-logging

2. Under root user, add exec flag for the script and execute deployment with:

  chmod +x ./gcp_crm_alert.sh
./gcp_crm_alert.sh -d

3. Confirm that the deployment runs successfully. If it does, you will see the following INFO log messages:

In the Red Hat Enterprise Linux (RHEL) system:

gcp_crm_alert.sh:2022-01-24T23:48:30+0000:INFO:'pcs alert recipient add gcp_cluster_alert value=gcp_cluster_alerts id=gcp_cluster_alert_recepient options value=/var/log/crm_alerts_log' rc=0

In the SUSE Linux Enterprise Server (SLES):

gcp_crm_alert.sh:2022-01-25T00:13:27+00:00:INFO:'crm configure alert gcp_cluster_alert /usr/share/pacemaker/alerts/gcp_crm_alert.sh meta timeout=10s timestamp-format=%Y-%m-%dT%H:%M:%S.%06NZ to { /var/log/crm_alerts_log attributes gcloud_timeout=5 gcloud_cmd=/usr/bin/gcloud }' rc=0

Now, in the event of a cluster node, resource, node failover, or failed action, Pacemaker will start the alert mechanism. For further details on the alerting agent, check out the Pacemaker Explained documentation.

How to Use Cloud Logging for Alert Reporting

Alerted events are published in Cloud Logging. Below is an example of the log record payload, where the cluster alert key-value pairs get recorded in the jsonPayload node.

{

 "insertId": "ktildwg1o3fbim",   "jsonPayload": {     "CRM_alert_recipient": "/var/log/crm_alerts_log",     "CRM_alert_attribute_name": "",     "CRM_alert_kind": "resource",     "CRM_alert_status": "0",     "CRM_alert_rsc": "STONITH-sapecc-scs",     "CRM_alert_rc": "0",     "CRM_alert_timestamp_usec": "",     "CRM_alert_interval": "0",     "CRM_alert_node_sequence": "21",     "CRM_alert_task": "start",     "CRM_alert_nodeid": "",     "CRM_alert_timestamp": "2022-01-25T00:17:06.515313Z",     "CRM_alert_timestamp_epoch": "",     "CRM_alert_desc": "ok",     "CRM_alert_target_rc": "0",     "CRM_alert_version": "1.1.15",     "CRM_alert_attribute_value": "",     "CRM_alert_node": "sapecc-ers",     "CRM_alert_exec_time": ""   },   "resource": {     "type": "global",     "labels": {       "project_id": "gcp-tse-sap-on-gcp-lab"     }   },   "timestamp": "2022-01-25T00:17:09.662557309Z",   "severity": "INFO",   "logName": "projects/gcp-tse-sap-on-gcp-lab/logs/sapecc-ers%2F%2Fvar%2Flog%2Fcrm_alerts_log", "receiveTimestamp": "2022-01-25T00:17:09.662557309Z" 

}

To get notified of a resource event — for example, when the HANA topology resource monitor fails — you can use the following filter for the alerting definition:

jsonPayload.CRM_alert_node=("hana-venus" OR "hana-mercury") -jsonPayload.CRM_alert_status="0" jsonPayload.CRM_alert_rsc="rsc_SAPHanaTopology_SBX_HDB00" jsonPayload.CRM_alert_task="monitor" To define an alert for a fencing event, your can apply this filter: jsonPayload.CRM_alert_node=("hana-venus" OR "hana-mercury") jsonPayload.CRM_alert_kind="fencing" The fencing log entry gets recorded with warning severity to give you deeper insight, and this additional information is also helpful for more specific filtering criteria: {   "insertId": "1plznskfjsxt82",   "jsonPayload": {     "CRM_alert_attribute_value": "",     "CRM_alert_recipient": "/var/log/crm_alerts_log",     "CRM_alert_rsc": "",     "CRM_alert_rc": "0",     "CRM_alert_timestamp_usec": "529261",     "CRM_alert_desc": "Operation reboot of hana-mercury by hana-venus for crmd.2361@hana-venus: OK (ref=2a9bf814-9adf-4247-af3f-94ac254fc3ca)", "CRM_alert_target_rc": "",     "CRM_alert_nodeid": "",     "CRM_alert_kind": "fencing",     "CRM_alert_node_sequence": "33",     "CRM_alert_task": "st_notify_fence",     "CRM_alert_status": "",     "CRM_alert_exec_time": "",     "CRM_alert_attribute_name": "",     "CRM_alert_timestamp_epoch": "1643072786",     "CRM_alert_version": "1.1.19",     "CRM_alert_timestamp": "2022-01-25T01:06:26.529261Z",     "CRM_alert_interval": "",     "CRM_alert_node": "hana-mercury"   },   "resource": {     "type": "global",     "labels": {       "project_id": "gcp-tse-sap-on-gcp-lab"     }   },   "timestamp": "2022-01-25T01:06:27.267017052Z",   "severity": "WARNING",   "logName": "projects/gcp-tse-sap-on-gcp-lab/logs/hana-venus%2F%2Fvar%2Flog%2Fcrm_alerts_log", "receiveTimestamp": "2022-01-25T01:06:27.267017052Z" 
} 

Alerts can be delivered through multiple channels, including text and email. Below is an example of an email notification for our earlier example, when we defined an alert for a HANA topology resource monitor failure:

You can write and apply filters to your log-based alerts to isolate certain types of incidents and analyze events over time. For example, the following script will surface a resource event occurring within a two-hour window on a specific date:

timestamp>="2022-01-25T00:00:00Z" timestamp<="2022-01-25T02:00:00Z"

jsonPayload.CRM_alert_kind="resource"

With the ability to analyze these logged alerts over time, determine whether event patterns warrant any action.

[SIDEBAR]

The alert script prints details in the standard output and in the log file /var/log/crm_alerts_log, and this can grow over time. We recommend that the log file is set with the Linux logrotate service in order to limit the file system space. Use the following command to create the necessary logrotate setting for the alerting log file:

cat > /etc/logrotate.d/crm_alerts_log << END-OF-FILE  /var/log/crm_alerts_log {   create 0660 root root   rotate 7   size 10M   missingok   compress   delaycompress   copytruncate   dateext   dateformat -%Y%m%d-%s   notifempty } END-OF-FILE 

[END SIDEBAR]

Tips for Troubleshooting When you first deploy your alert script, how can you tell for certain that you’ve done it correctly? Use the following commands to test it out:

In RHEL:

pcs alert show 

In SLES:

sudo crm config show | grep -A3 gcp_cluster_alert 

You should see the following if the script is correct:

In RHEL:

Alerts:  Alert: gcp_cluster_alert (path=/usr/share/pacemaker/alerts/gcp_crm_alert.sh)   Description: "Cluster alerting for hana-node-X"   Options: gcloud_cmd=/usr/bin/gcloud gcloud_timeout=5   Meta options: timeout=10s timestamp-format=%Y-%m-%dT%H:%M:%S.%06NZ   Recipients:    Recipient: gcp_cluster_alert_recepient (value=gcp_cluster_alerts)     Options: value=/var/log/crm_alerts_log In SLES:

alert gcp_cluster_alert "/usr/share/pacemaker/alerts/gcp_crm_alert.sh" \ meta timeout=10s timestamp-format="%Y-%m-%dT%H:%M:%S.%06NZ" \ to "/var/log/crm_alerts_log" attributes gcloud_timeout=5 gcloud_cmd="/usr/bin/gcloud" 

If the commands do not display the alerts properly, re-deploy the script.

In case there is an issue with the script, or if the Cloud Logging records are not presenting as expected, examine the script log file /var/log/crm_alerts_log. The errors and warning can be filtered with:

egrep '(ERROR|WARN)' /var/log/crm_alerts_log 

Any Pacemaker alert failures will be recorded in the messages and/or Pacemaker log. To examine recent alert failures, use the following command:

egrep '(gcp_crm_alert.sh|gcp_cluster_alert)' \   /var/log/messages /var/log/pacemaker.log 

Keep in mind, though, that the Pacemaker log location may be different in your system from the one in the example above.

From reactive to proactive

Your SAP applications are too critical to risk outages. The most effective way to manage high availability clusters for your SAP systems on Google Cloud is to take full advantage of Pacemaker’s alerting capabilities, so you can be proactive in ensuring your systems are healthy and available.

Learn more about running SAP on Google Cloud.