Every cloud system begins with high-quality hardware infrastructure. Sometimes, however, hardware breaks — and when it happens, our most important goal is to minimize the impact on our customers and their cloud workloads.

Memory errors are the most common type of hardware failure, and they’re also one of the most challenging in terms of their impact on production workloads and system reliability. That’s why we’re excited to share what Google Cloud has been doing to minimize the impact of memory errors. If your business runs SAP HANA in the cloud, this is an important innovation —  one that Google Cloud is proud to deliver to our customers.

Memory errors: A big problem with a long history

First things first: Memory errors are a high priority because they happen often. And when they happen, the disruption can have far-reaching effects on your customers and your business. 

In 2009, Google Cloud published the first major study on memory reliability. We found an average error rate of over 8% per year in DIMM modules installed in production systems. Given that each generation of DDR RAM packs more capacity into smaller packages, it’s safe to think that memory hardware has become less reliable since then.

Memory error impacts: They could be worse, but they’re far from good

What happens when a system detects a bad segment in a DIMM module? While data loss or corruption from memory errors is not common, some errors are correctable but some are not, potentially resulting in a critical system failure..  

Modern CPUs are equipped with error-correcting memory features and are very good at correcting simple errors with ECC (Error Correction Code). The challenge is that most of the software that runs on a host system — whether it’s a hypervisor, a virtual machine, an operating system, a database or an application — will crash instantly when it encounters an uncorrectable memory error. In a cloud environment, this kind of crash can take down cached data and even data saved to a local SSD. The crashed applications will recover, but the process means several minutes of downtime. The more data you have, the longer this process will take.

Sometimes, that’s merely an inconvenience. Other times, it’s a very big deal. A Google Cloud customer running business-critical SAP applications and an in-memory HANA database might measure downtime costs well over $10,000 per minute in lost revenue and other direct impacts. Many HANA databases load into terabytes of memory, and it can take an hour or longer to get everything restarted and back to normal after a crash. For SAP HANA, a fast recovery with up to 10 minutes of downtime requires a redundant replica provisioned all the time, doubling the cost.

And statistically speaking, when a HANA instance occupies almost all of the memory on a host system, it’s also the most likely application to stumble across a memory error. You can see why this would be a problem.

 The ‘victim neighbor’ VM challenge

There’s a final problem to consider when a memory error takes out production applications: what we call the “victim neighbor” issue.

In any cloud, a single physical host is a multi-tenant environment that might run dozens of VMs, potentially owned by dozens of different customers. A memory error won’t just crash the VM actually using the bad section, it will crash every VM running on the system. That’s a standard VM response to memory errors on a host system, and it will happen to any VM architecture available on the market today to avoid memory corruption. 

Overall, this “victim neighbor” effect accounts for more than 90% of the VMs that get knocked down by a memory error on a physical server. That’s a huge blast radius for such a common problem.

A practical solution to memory-error impacts

You can see why managing this problem is a big deal for Google Cloud. While we know that some failures are inevitable, we have developed another way to tackle the problem. Google Cloud already maintains some unique and valuable tools, such as Live Migration, that help our customers minimize unplanned downtime.When we integrate these tools with recent work that leverages error-handling capabilities built into CPUs (courtesy of Intel) and into certain applications (in particular, SAP HANA), we get a solution that dramatically reduces downtime and disruptions related to memory errors — in many cases, to the point where customers won’t even know there was a problem.

The Google Cloud solution: Memory poisoning recovery

At a big picture level, we refer to our solution as Memory Poisoning Recovery (MPR). It combines some existing Google Cloud capabilities, some new capabilities, and some important third-party capabilities at the CPU (Intel) and application (SAP HANA) levels. MPR can be broken down into two main processes:

Memory Error Isolation 

• Step 1: We hardened our VM technology to be more robust against memory errors. We intercept and analyse the memory error coming from the system. Then we flag the signaled region of a memory DIMM with an uncorrectable error as “poisoned”. 
• Step 2: Then we trigger processes to keep track of these “poisoned” regions and the VMs they affect so they can’t affect data integrity. 

Memory Error Recovery

• Step 3: Then we notify the Guest OS & the MCE-aware applications that a memory error has been recorded, in a manner that allows the applications to execute application relevant memory error handling.
• Step 4: At the same time we communicate with Google Cloud Live Migration to begin moving guest VMs off the affected host. This ensures customers are running on a healthy host which reduces the probability of more uncorrectable errors happening and avoids further downtime.

Below is a simple visual of how this all works:

How MPR makes life better for customers

Let’s look again at the different groups of Google Cloud customers involved in a memory error scenario and how we can help them achieve a happier ending after a crash — starting with the customer running the VM and application that actually triggered the memory error. 

Customer Group: MCE-Aware SAP HANA with Fast Restart enabled on a VM directly affected by a memory error.

Customer Group: Customers running other, non MCE Aware applications on a VM directly affected by a memory error

Next, our “victim neighbors” group probably won’t even know there was a problem with the host system. Google Cloud Live Migration will move them to a new host, instantly and automatically, and avoid the crash-and-restart scenario.

Customer Group: Customers running other, any application on a VM not directly affected by a memory error

Simple steps for taking advantage of MPR

Our MPR capabilities will be available on our Google Cloud memory-optimized Compute Engine second generation instances in Q4 of 2021. We’ll continue to roll out the capability during the months ahead to additional instances and look for new ways to work with applications that adopt a MCE Aware architecture.

Most customers in the “victim neighbor” category will not need to lift a finger to experience the benefits. By marrying our Live Migration feature to some awareness of those MCE signals, we ensure that it hears the alarm first and gets a critical head start on the migration process before issues begin with the guest VMs. Our customers land safely on a new host, and their applications keep running.

For our SAP customers running HANA, MPR is all about protecting against loss. Unplanned downtime for a HANA environment is incredibly expensive, the recovery process from a hard crash is extremely long, and the business disruptions can be truly damaging to the business. Thanks to MPR, all of that cost and worry can get compressed almost to nothing — with Fast Restart reducing what can be an hour or more of downtime to a matter of seconds.

But our SAP customers have to take a critical first step to claim these benefits. Fast Restart is a crucial piece of the MPR solution, and it is not enabled by default. Configuring your SAP HANA instance for Fast Restart involves changing a few configuration settings; the process is fast, easy, and doesn’t involve risk. 

Finally, if you’re not running your workloads — SAP or otherwise — on Google Cloud, consider the benefits of running on a cloud that mitigates a hardware reliability issue affecting businesses of every size and industry. And consider the value of tools like Live Migration that already help Google Cloud customers improve uptime and reduce risk.

Hardware failures happen, and they probably always will. But we’re proving how valuable it can be to avoid the bad things that usually happen when memory failures occur. Right now, only Google Cloud has a practical solution to this very difficult problem.    

Learn more about Fast Restart for SAP HANA, Live Migration and other key Google Cloud capabilities for your SAP environment.

About Tencent Africa

Tencent Africa is dedicated to making every day easier for customers and brands. Opening its doors in 2013 as WeChat Africa, the company is responsible for WeChat operations on the continent.

Industries: Technology

Location: South Africa

About Siatik Systems

Founded in 2008, South Africa-based Siatik Systems is a leading Google Cloud Platform solutions provider with a focus on cloud integration services, information technology consulting services, and bespoke software engineering solutions.

Industries:

Location:

As one of Africa’s leading technology companies, Tencent Africa is responsible for WeChat operations on the continent. WeChat Africa has successfully launched a number of features including the WeChat Wallet, a mobile payment service for smartphones, which enables seamless and secure transactions for friends to send money to one another, cash money out at Standard Bank ATMs, use at accredited retailers, and make purchases from any SnapScan merchant in South Africa. Under the new name, Tencent Africa recently launched two new mobile products, JOOX, a mobile music streaming application, and VOOV, a social live streaming application.

When China’s Tencent brought its hugely popular instant messaging service WeChat to South Africa, it faced a near saturated market. To gain traction, Tencent Africa had to showcase WeChat’s versatility as a platform that goes far beyond just instant messaging. Engaging with local financial partners helped WeChat launch a WeChat Wallet to facilitate electronic payments, while providing exclusive content from South African cultural figures helped bring the platform extra publicity.

As WeChat started to grow its customer base, its on-premises infrastructure was stretched beyond its limits and the company looked for a cloud-based solution. After evaluating all its options, Tencent Africa found Google Cloud Platform best suited its needs.

“We’re a growing business and constantly upgrading our infrastructure. Getting new servers was becoming quite expensive so we wanted a cloud-based solution to reduce costs,” says Christoff Albertyn, Head of Technology at Tencent Africa. “When we did some latency testing, Google came out on top, so it made sense to go with them. Everybody loves Google, it’s a great brand. Just knowing a company like Google is behind it gives you peace of mind.”

Building a platform to scale

Instant messaging services need a stable infrastructure, the ability to scale at speed, and lightning fast performance. When Tencent Africa built an Official Accounts API to run a number of services on WeChat, to help companies engage with users on the platform, latency was its main concern. The Official Accounts services had to deliver replies over WeChat in under five seconds or the connection would drop.

Between the API on-premises servers in South Africa and WeChat’s main servers in China, Tencent Africa’s services regularly hit the limits of the permissible response time. In addition, by 2015, WeChat was starting to garner more customers across South Africa and Tencent Africa had to provision more servers. The on-premises servers were simply being asked to do too much, and expansion was proving expensive, so the company looked for a cloud-based solution.

Before committing to anything, Tencent Services did a thorough latency test of several cloud-based solutions. Google Cloud Platform came top. Tencent also engaged the help of Siatik, a Google Partner, to gauge what solutions best fit the company’s needs and help implement them. Siatik customers span both the public and private sectors and its consultants are regularly engaged in the health care, education, financial and technology industries. In order to provide leading technology solutions, it partners with industry leaders, such as Google, to provide best-of-breed infrastructure and cloud platform solutions to its customers.

“Siatik was a great partner to work with,” says Christoff. “Any questions I had, I could just pick up the phone and whatever it was they’d help me out. They were also able to get some of the team from Google to come to our offices and see how we worked, which was really good.”

With a simple, clean interface in Google Compute Engine, the company was able to spin up virtual instances very quickly and begin migrating its stack to Google. When it became more comfortable with Google Cloud Platform features, Tencent used Google Cloud SQL and Google BigQuery to manage its data streams, and Google Cloud Storage to store them safely.

From day one, it was very smooth. The interface was amazing—Google must have spent a long time perfecting it because it was so easy to use says Christoff. “I set up the first virtual machine in no time without any struggles.”

In addition to powerful hardware from Google Compute Engine, the fast, global network from Google helped to speed up performance. With the infrastructure ported over to Google, Tencent also brought in G Suite as an office productivity solution, which eased the pressure on its servers even more. Throughout the process, Siatik were on hand to advise and provide technical guidance whenever any issues came up.

What I really like is how quickly you can spin servers up. There’s no messing about and the interface is really easy to use, says Christoff. “You can have a virtual machine running in just a few minutes. It’s very impressive.”

Cutting costs, raising expectations

Google Cloud Platform helped Tencent Africa to slash its latency. Before the migration, pings to the Chinese servers and back would regularly take 180 milliseconds. Afterwards, this fell to around 35 milliseconds, helping to ensure smooth service to Tencent’s South African customers. Google offers innovative pricing, such as per-minute billing and sustained use discounts, which helped significantly cut server costs compared to an equivalent on-premises solution.

As its customer base grew, and Tencent Africa began to release new products, the ease of use of Google Compute Engine meant that spinning up new servers took minutes instead of hours and allowed the company to scale up and down at speed. Exploring products such as Kubernetes for rapid deployment or TensorFlow for machine learning, Tencent Africa continues to evolve and experiment with new products to keep WeChat a growing success in South Africa.

“Initially, we just used per-minute billing, but when I saw how much work we were doing with Google BigQuery and Google Cloud SQL, we started to generate sustained-use discounts, which helped save us a lot of money,” says Christoff. “The other great thing is, Google Cloud Platform monitors how you’re using it and tells you what else you can do to reduce costs.”

With Google Cloud, Autonom8 can run a platform that accelerates and streamlines customer journeys in a scalable, reliable, cost-effective infrastructure, while using advanced optical character recognition to enable intelligent document processing.

About Autonom8

Headquartered in the United States and India, Autonom8 has built a low-code SaaS platform that allows businesses to digitize customer-facing workflows. The business aims to help clients reduce costs and improve interactions with their customers through automation and enablement of customer journeys.

Industries: Technology

Location: United States and India

Google Cloud results:

• Increased margins by up to 30% by switching from a home-grown OCR system to Cloud Vision AI
• Enables one DevOps team member to manage up to 30 customers
• Provides real-time information about customer journeys to enable businesses to respond quickly and accurately
• Ensures use of its platform with containerization in customers’ private data centers
• Reduced operating costs by up to 20% with localized scalability and architecture through GKE

Just as cars are evolving to become autonomous, smart and self-driving, enterprises can gain self-awareness, an ability to learn and an ability to adapt. This is the value proposition put forward by Autonom8, an India- and United States-based enterprise workflow management software business. “We provide a low-code, high-intelligence customer journey automation SaaS platform,” explains ​​Ranjit Padmanabhan, Co-founder and Chief Technology Officer (CTO), Autonom8.

The Autonom8 platform includes components, such as A8Studio, a drag and drop location from which clients can create customer journeys, a chat platform that enables clients to create chatbots, and an analytics module. “With our platform and services, businesses can reduce costs and improve interactions with their customers by applying automation to accelerate and provide better customer journeys,” adds Padmanabhan.

Demand for Autonom8 is being driven by the changing customer demands of enterprises, including the expectation to interact with them over multiple channels, and the rising cost of building software with experienced developers. These trends place enterprises under growing pressure to increase the productivity of the people they do have, particularly those who are less technically inclined. In addition, changing consumer habits, regulations and the emergence of new technologies mean customer journeys cannot remain static and need to evolve.

Developing a microservices-based SaaS platform

From the start, Autonom8 planned to deliver a SaaS platform and initially deployed on a multinational cloud service, chosen due to the team’s familiarity with its products and the availability of credits. However, the company’s decision to opt for a microservices architecture that enables individual services to scale independently while running in a containerized environment, demanded high-quality container orchestration. To optimize cost, scalability and performance, Autonom8 began evaluating Google Kubernetes Engine (GKE).

The business then completed a side-by-side comparison between Google Cloud and its incumbent provider of compute, storage and other services. Google Cloud fared favorably, with Vision AI in particular providing powerful machine learning and optical character recognition (OCR) functionality, supporting a key use case for Autonom8.

In addition, many of Autonom8’s clients at the time are financial institutions in India, and legally required to retain data within the country’s borders. Google Cloud’s global network and local presence means the business could fulfill this requirement easily.

“We decided to evaluate Google Cloud, particularly GKE, from two perspectives. One, from a security perspective, as we sell to banks that audit our platform, and two, as a failover between regions because downtime costs money. We found it a compelling solution.”

— Ranjit Padmanabhan, Co-founder and Chief Technology Officer (CTO), Autonom8

A seamless move to Google Cloud

Autonom8 began deploying on Google Cloud in 2018, with its architecture comprising storage, compute, serverless, container management and orchestration, and Vision AI. “We looked at our scripting with the previous provider, and using the Google Cloud documentation available online, educated ourselves over a few weeks before moving pieces of our architecture step by step to Google Cloud,” says Padmanabhan. “We did not run into any major issues. It was pretty simple, with our experienced engineers training others in the product.”

According to the CTO, the business had two options when moving to Google Cloud. Autonom8 could either install raw virtual machines and effectively create its own virtualized data center, or rely on managed services for functions such as memory store, registration and authentication to save time and resources over the long term. Autonom8 opted for the latter and has transitioned fully to Google Cloud, with the number of cloud products and services in its architecture rising from five to about 15. While each product and service performs a key role in the delivery of Autonom8’s products and services, Padmanabhan nominates GKE, Vision AI and Cloud SQL as providing the greatest value to the business.

Scalability, real-time monitoring and intelligent document processing at low cost

With GKE, the business can now scale the nodes or containers specific to each microservice in the event traffic to a particular client surges, due to a rebate or promotion. “Through the combination of the architecture and localized scalability we achieve with GKE, we are reducing our operating costs by up to 20%,” says Padmanabhan.

Running an open source TimescaleDB on Postgres in Cloud SQL enables Autonom8 to give its clients the ability to monitor customer journey information in real time. An example of a journey is applying for a bank loan. The customer must take steps including providing income, tax and other financial details that the bank then appraises to help make a decision on the application. “The moment someone applies for a loan, for example, a bank knows about it and can monitor for fraud, bottlenecks, or other abnormalities, and immediately route to a remediation workflow,” explains Padmanabhan. “Cloud SQL enables us to maintain transactional logging and provide real-time data to our dashboards.”

After evaluating alternative services, including developing a home-grown OCR engine, the business turned to Cloud Vision AI to manage the intelligent document processing that comprises much of its transactional volume. “Vision AI is significantly better than the alternatives and the cost of maintaining our version did not make sense, because Google Cloud continues to make improvements over time that enable us to deliver more and more accurate results to our customers,” says Padmanabhan. “Switching from our home-grown service to Vision AI has enabled us to increase our profit margins by up to 30%.”

“Through the combination of the architecture and localized scalability we achieve with Google Kubernetes Engine, we are reducing our operating costs by up to 20%.”

—Ranjit Padmanabhan, Co-founder and Chief Technology Officer (CTO), Autonom8

Supporting client demands and improving developer efficiency

Google Cloud also enables Autonom8 to meet the demands of businesses that want to run its platform within their own private data centers. “We can undertake the build within Google Cloud and ship our containers to compatible hosts within those clients’ data centers,” explains Padmanabhan. “With our previous provider, we could create containers, but these would not run properly within those data centers.”

Furthermore, Google Cloud documentation and online resources help Autonom8 reduce the training needed for new developers to become productive, with the Google Cloud learning curve taking up just 10% of the overall onboarding cycle.

The organization spends the equivalent of just 3% of its overall annual revenue on DevOps, measured as DevOps Utility Ratio, while the cloud cost of revenue is about USD 1 for every USD 6 in annual recurring revenue, measured as Cloud Utility Ratio. “These two metrics are about what we can do with the people we have,” explains Padmanabhan. “Our current ratio implies that one DevOps person can handle approximately 30 customers. This is made possible by the tools we have, and the comprehensive support from Google Cloud in terms of security patches, intelligent alerts, resource overloading, and more.”

Google Cloud also provides the flexibility for Autonom8 to accommodate the varying service levels required by individual customers based on factors, such as the impact of downtime, as the business can failover seamlessly between regions to mitigate the impact of any issues that may occur.

“Our current ratio implies that one DevOps person can handle approximately 30 customers. This is made possible by the tools we have, and the comprehensive support from Google Cloud in terms of security patches, intelligent alerts, resource overloading, and more.”

—Ranjit Padmanabhan, Co-founder and Chief Technology Officer (CTO), Autonom8

Integrating Google Workspace with Autonom8 to deliver new capabilities

Autonom8 relies on Google Workspace for communication, collaboration and other workplace productivity requirements, growing its footprint from Gmail when the employee population was four or five, to a range of products including Sheets and Drive as the business grew. “It became natural to use the capabilities in Google Workspace as we matured,” says Padmanabhan. “One of the most interesting capabilities was our ability to integrate Google Workspace into our platform. For example, when someone is running a workflow, they can add data from a Sheet. We’ve added Google Workspace authentication capabilities into our products as well.”

“Everyone is using shared links to Drive and I really like the granular permissions structure,” he adds. “I can open up folders to clients while keeping an internal space within the business to ensure security and privacy.”

With Google Cloud, Autonom8 is now poised to continue growing its business and adding new features and capabilities for clients. “We are extremely excited at the opportunity to step up our offering to clients with Google Cloud,” concludes Padmanabhan.