Architecting Multi-region Database Disaster Recovery for MySQL - Build What's Next
How-to

Architecting Multi-region Database Disaster Recovery for MySQL

4099

Of your peers have already read this article.

4:30 Minutes

The most insightful time you'll spend today!

Enterprises expect extreme reliability of the database infrastructure that’s accessed by their applications. Despite your best intentions and careful engineering, database errors happen. Find out how to deploy a database architecture that implements high availability and disaster recovery for MySQL on Compute Engine, using regional disks as well as load balancers.

Enterprises expect extreme reliability of the database infrastructure that’s accessed by their applications. Despite your best intentions and careful engineering, database errors happen, whether that’s machine crashes or network partitioning. Good planning can help you stay ahead of problems and recover more quickly when issues do occur.

This blog shows one approach of deploying a database architecture that implements high availability and disaster recovery for MySQL on Compute Engine, using regional disks as well as load balancers.

Any database architecture must provide approaches to tolerate errors and recover from those errors quickly without losing data. These approaches are expressed in RTO (recovery time objective) and RPO (recovery point objective), which offer ways to set and then measure how long a service can be unavailable, and how far back data should be saved.

After a database error, a database must recover as fast as possible with an RTO as small as possible, ideally in seconds. There must be as little data loss as possible—ideally, none at all. The desired RPO is the last consistent database state.

From a database architecture and deployment viewpoint, this can be accomplished with two distinct concepts: high availability and disaster recovery. Use both at the same time in order to achieve an architecture that’s prepared for the widest range of errors or incidents.

Creating a resilient database architecture

A high-availability database architecture has database instances in two or more zones. If a server on a zone fails, or the zone becomes inaccessible, the instances in other zones are available to continue the processing. The figure below shows two instances, one in zone zn1, and one in zone zn2. The load balancer in front supports directing traffic to a healthy database instance available for read and write queries.

A disaster recovery architecture adds a second high-availability database setup in a second region. If one of the regions becomes inaccessible or fails, the other region takes over. The figure below shows two regions, primary and DR. Data is replicated from the primary to the DR region so that the DR region can take over from the latest consistent database state. The load balancer in front of the regions directs traffic to the region in charge of the read and write traffic. Here’s how this architecture looks:

MySQL database architecture.jpg

In addition to the database instance setup, a regional disk is deployed so that data is written simultaneously in two zones, proving fail-safe in the event of zone failure. This is a huge advantage of Google Cloud, allowing you to skip MySQL-level replication within a region. Each write operation to disk is done in two zones synchronously. When the primary instance fails, a standby instance is mounted with regional persistent disk(s), and the database service (MySQL) is then started using the same. This brings the peace of mind of not worrying about replication lag or database state for high availability.

From a disaster recovery process view, the following happens over time during a failure situation:

  • Normal steady state database operation
  • A failure happens and a region becomes unavailable or the database instance inaccessible
  • A decision must be made to fail over or not (in case there is the expectation that the region becomes available soon enough or the instance becomes responsive again)
  • DNS is updated manually, therefore it redirects application traffic to a second region
  • Fallback to the primary region after it becomes available again is optional, as the second region is a fully built-out deployment

From a high-availability process view, the following happens over time during a failure situation:

  • Normal steady state database operation
  • Database instance fails or becomes unavailable
  • Launch the standby instance
  • Mount regional SSD and start database
  • Automatic redirection of application traffic to the standby via load balancer
  • After the failed or unavailable instance becomes available again, a fallback can take place or not

The database architecture shown demonstrates a highly available architecture supporting disaster recovery. With regional disks and load balancers, it is straightforward to provide a resilient database deployment.

Find out more about load balancers and regional disks. Check out general HA and DR processes and detailed steps in the initial part of the reference guide. Try it out to become familiar with the architecture as well as the two major failover processes.

6373

Of your peers have already watched this video.

2:15 Minutes

The most insightful time you'll spend today!

Explainer

What’s Google Cloud Firestore Database and What are its Benefits for Business and Developers?

Cloud Firestore is a NoSQL document database that simplifies storing, syncing, and querying data for your mobile and web apps at global scale.

Cloud Firestore is a fast, fully managed, serverless, cloud-native NoSQL document database that simplifies storing, syncing, and querying data for your mobile, web, and IoT apps at global scale.

Its client libraries provide live synchronization and offline support, while its security features and integrations with Firebase and Google Cloud Platform (GCP) accelerate building truly serverless apps.

Here’s other stuff it’s good at:

Sync data across devices, on or offline

With Cloud Firestore, your applications can be updated in near real time when data on the back end changes. This is not only great for building collaborative multi-user mobile applications, but also means you can keep your data in sync with individual users who might want to use your app from multiple devices.

With Firebase Realtime Database, we felt we had built the best force-plate testing software on the market. Thanks to Cloud Firestore, in only two weeks, we built a system that’s significantly better and includes features we never thought possible to ship on Day 1.

Chris Wales, CTO, Hawkin Dynamics

Cloud Firestore has full offline support, so you can access and make changes to your data, and those changes will be synced to the cloud when the client comes back online. Built-in offline support leverages local cache to serve and store data, so your app remains responsive regardless of network latency or internet connectivity.

Simple and effortless

Cloud Firestore’s robust client libraries make it easy for you to update and receive new data while worrying less about establishing network connections or unforeseen race conditions. It can scale effortlessly as your app grows. Cloud Firestore allows you to run sophisticated queries against your data. This gives you more flexibility in the way you structure your data and can often mean that you have to do less filtering on the client, which keeps your network calls and data usage more efficient.

Store and sync data between your users in realtime.

Enterprise-grade, scalable NoSQL

Cloud Firestore is a fast and fully managed NoSQL cloud database. It is built to scale and takes advantage of GCP’s powerful infrastructure, with automatic horizontal scaling in and out, in response to your application’s load. Security access controls for data are built in and enable you to handle data validation via a configuration language.

Blog

AI Solutions for Government Organizations: How to Get Started

4387

Of your peers have already read this article.

3:00 Minutes

The most insightful time you'll spend today!

As AI continues to advance, government organizations are starting to explore its potential uses and benefits. In this article, we discuss new offerings that can help government organizations get started with AI and take advantage of its capabilities.

Cloud-native features are helping public sector teams innovate faster than ever. Ideas discussed in a morning meeting can be a working proof of concept later that day. Managed services can remove administrative burden and reduce the steps needed to design and provision cloud infrastructure. Security can be built-in from the beginning with Identity and Access Management (IAM), Virtual Private Cloud Service Controls (VPC-SC), and Data Loss Prevention (DLP). Short-lived services and infrastructure-as-code allow rapid and cost-effective prototyping. These technologies can be used to architect a solution that follows the principle of least privilege and helps you secure your data.

So the question becomes: given the complex problems agencies face, where do you start? Google Public Sector now offers “Getting Started” and “Scaling” service offerings for CCAI, DocAI, and BigQuery to help you jumpstart your AI journey, based on where you are.

Complex problems, simpler AI-based solutions

Solving more challenging problems with cloud-native technology doesn’t have to be overwhelming. You can approach them the same way you might solve a puzzle: start with one piece that follows another until the larger picture takes shape. Though you can simplify the steps, solving these problems still requires powerful tools. Google Cloud’s AI/ML capabilities may be the answer for your team.

Google Public Sector is making it easier to get started with advanced technologies, beginning with artificial intelligence and machine learning (AI/ML) workloads for government organizations, and it’s something you can do now, one piece at a time.

Automating your FAQs with CCAI

Does your agency require a team to answer commonly asked questions? What if you could train an agent to answer questions immediately and operate 24/7? Contact Center AI (CCAI) can do this and more. Already using CCAI and need the agent to level up to address complex interactive dialogs? Getting Started with CCAI and Scaling with CCAI are new Google service offerings specifically designed to help public sector organizations tackle situations like these.

Automate data entry with DocAI

How many hours does your team spend manually reviewing or entering data from standardized forms? What if you could automatically pull data right from the page? Google Document AI (DocAI) specializes in exactly this—even if the form has handwritten text. Getting Started with DocAI and Scaling with DocAI are new service offerings that help you remove this burden from your team. DocAI automates data entry and makes that data available to other teams while prioritizing both security and ease of use.

Making data and insights accessible with BigQuery

Then there’s all your existing data. You may have years of it stored in many places, and you may not have a way to make use of it when you need it. BigQuery is Google’s enterprise data warehouse. It was designed for data analytics—looking back at historical data to make conclusions about it. But BigQuery’s analytics don’t stop there. It can also look forward in time to make predictions, often using the same datasets. Getting Started with BigQuery and Scaling with BigQuery are new service offerings that help you take your first steps toward AI/ML capabilities by starting with a single table or pipeline that can help make sense of all your data.

The best help is the kind that meets you where you are and gets you where you want to be. Google Public Sector’s new service offerings do just that: help you work through complex problems by meeting you wherever your starting line is, whether you’re ready to start or ready to scale. Let us know if you would like us to contact you about the services mentioned in this article. Let’s solve your highest impact problems together, one puzzle piece at a time.

How-to

An AI-Powered Cost Cutting Guide: 8 Strategies for Maximizing Profits

1710

Of your peers have already read this article.

3:30 Minutes

The most insightful time you'll spend today!

Want to stay ahead of the curve and keep your business thriving? It's time to start harnessing the power of data and AI. In this post, we'll share 8 actionable tips for cutting costs and driving profits, all while staying ahead of the competition.

We are increasingly seeing one question arise in virtually every customer conversation: How can the organization save costs and drive new revenue streams? 

Everyone would love a crystal ball, but what you may not realize is that you already have one. It’s in your data. By leveraging Data Cloud and AI solutions, you can put your data to work to achieve your financial objectives. Combining your data and AI reveals opportunities for your business to reduce expenses and increase profitability, which is especially valuable in an uncertain economy. 

Google Cloud customers globally are succeeding in this effort, across industries and geographies. They are improving ROI by saving money and creating new revenue streams. We have distilled the strategies and actions they are implementing—along with customer examples and tips—in our eBook, “Make Data Work for You.” In it, you’ll find ways you can pare costs, increase profitability, and monetize your data.  

Find money in your data 

Our Google Cloud teams have identified eight strategies that successful organizations are pursuing to trim expenses and uncover new sources of revenue through intelligent use of data and AI. These use cases range from scaling small efficiencies in logistics to accelerating document-based workflows, monetizing data, and optimizing marketing spend.

https://storage.googleapis.com/gweb-cloudblog-publish/images/1_Cost_Optimization.max-900x900.jpg

The results are impressive. They include massive cost savings and additional revenue. On-time deliveries have increased sharply at one company, and procure-to-pay processing costs have fallen by more than half at another. Other organizations have reaped big gains in ecommerce upselling and customer satisfaction.

We’ve found that businesses across every industry and around the globe are able to take action on at least one of these eight strategies. Contrary to common misperceptions, implementation does not require massive technology changes, crippling disruption to your business, or burdensome new investments. 

What success looks like 

If you worry your business is not ready or you need to gain buy-in from leadership, the success stories of the 15 companies in this report are helpful examples. Learning how organizations big and small, in different industries and parts of the world, have implemented these data and AI strategies makes the opportunities more tangible.

Carrefour 
Among the world’s largest retailers, Carrefour operates supermarkets, ecommerce, and other store formats in more than 30 countries. To retain leadership in its markets, the company wanted to strengthen its omnichannel experience.

Carrefour moved to Google Data Cloud and developed a platform that gives its data scientists secure, structured access to a massive volume of data in minutes. This paved the way for smarter models of customer behavior and enabled a personalized recommendation engine for ecommerce services. 

The company saw a 60% increase in ecommerce revenue during the pandemic, which it partly attributes to this personalization. 

ATB Financial 
ATB Financial, a bank in the Canadian province of Alberta, uses its data and AI to provide real-time personalized customer service, generating more than 20,000 AI-assisted conversations monthly. Machine learning models enable agents to offer clients real-time tailored advice and product suggestions. 

Moreover, marketing campaigns and month-end processes that used to take five to eight hours now run in seconds, saving over CA$2.24 million a year. 

Bank BRI
Bank BRI, which is owned by the Indonesian government, has 75.5 million clients. Through its use of digital technologies, the institution amasses a lot of valuable data about this large customer base. 

Using Google Cloud, the bank packages this data through more than 50 monetized open APIs for more than 70 ecosystem partners who use it for credit scoring, risk management, and other applications. Fintechs, insurance companies, and financial institutions don’t have the talent or the financial resources to do quality credit scoring and fraud detection on their own, so they are turning to Bank BRI. 

Early in the effort, the project generated an additional $50 million in revenue, showing how data can drive new sources of income. 

How to get going now

Make Data Work for You” will help you launch your financial resiliency initiatives by outlining the steps to get going. The process lays the groundwork for realizing your own cost savings and new revenue streams by leveraging data and AI.

Among these steps include building frameworks to operate cost efficiently, make informed decisions related to spending and optimize your data and AI budgets.

https://storage.googleapis.com/gweb-cloudblog-publish/images/2_Cost_Optimization.max-900x900.jpg

Operate: Billing that’s specific to your use-case
Control your costs by choosing data and analytics vendors who offer industry-leading data storage solutions and flexible pricing options. For example, multiple pricing options such as flat rate and pay-as-you-go allow you to optimize your spend for best price-performance.

Inform: make informed decisions based on usage
Use your cloud vendor’s dashboards or build a billing data report to gain insights on your spending over time. Make use of cost recommendations and other forecasting tools to predict what your future expenses are going to be.

Optimize: Never pay more than you use 
While planning data analytics capacity, organizations often overprovision and overpay than what they actually use. Consider migrating your workloads that have unpredictable demand to a data warehousing solution that offers granular level autoscaling features so that you never have to pay for more than what you use.

There are other key moves that will set your initiative up for success including how to shorten time to value in building AI models and measuring impact. You can find details in the report.

A brighter future

The teams at Google Cloud helped the companies in “Make Data Work for You,” along with many more organizations, use their data and AI to achieve meaningful results. Download the full report to see how you can too.

Case Study

Brazilian Insurer Uses Google BigQuery to Underwrite Critical Decisions

DOWNLOAD CASE STUDY

3568

Of your peers have already downloaded this article

1:30 Minutes

The most insightful time you'll spend today!

To stay competitive in the rapidly changing insurance marketplace, Mitsui Sumitomo Seguros (MSS) recognized the need to leverage its available data to stabilize and scale existing operations, improve sales decisions and better engage employees.

MSS managers required frequent and easy access to newly organized data (including performance metrics and market intelligence) to make timely decisions. MSS sales teams needed granular-level knowledge to create what-if scenarios, and analytical tools to diagnose problems and offer a range of policies on tight deadlines to their customers. All of these solutions had to be scalable and affordable, growing as Mitsui Sumitomo Seguros grew.

It used Google BigQuery and Compute Engine for secure and economical analytics data storage and, as a result, transformed its corporate culture by facilitating data-driven decisions, spurring 40% yearly sales growth.

Blog

Cloud Spanner & Bigtable Helps Sabre Build Consistency & Scalability to Serve 1 Billion Travellers

3636

Of your peers have already read this article.

1:30 Minutes

The most insightful time you'll spend today!

Sabre, an innovative tech company leverages databases and AI to power travel companies across the globe. The firm took its 10-year old association with Google a step further by leveraging Spanner and Bigtable to vest its focus on innovation.

Sabre is an innovative software and technology company that leverages highly scalable databases and artificial intelligence (AI) to power travel industry partners around the globe. Our company processes more than 12 billion shopping requests and serves over 1 billion travelers every year.

The companies that partner with us include airlines, travel websites, agencies, and hotels. We are distinguished by a long history of integrating cutting edge technology into our operations. Our earliest innovations include building one of the first transaction processing systems in existence and a travel distribution network that predates the Internet! Through our next generation of AI-powered solutions, we are focused on optimizing the retailing, distribution, and fulfillment experience of the travel industry.

In my role as chief architect for Sabre Labs, I spearhead the long-term technology choices we are making to enhance the development, deployment, and operation of our software. We’ve embarked on a multi-year strategy to transform our technology and solutions. Rearchitecting our infrastructure to become fully cloud native is a central tenet of this transformation. In these early iterations of our plan, we have seen the significant impact of machine learning and its potential in bringing truly personalized travel experiences to customers. As part of this effort, we’ve established a 10-year partnership with Google to help accelerate our transformation and bring innovation to the travel industry.

Database choices require tradeoffs

The complexity and scale of the travel industry places high demands on the cloud services we utilize. We tend to place more emphasis on how a particular cloud service will impact an application’s reliability, performance, or development time, rather than choosing a service purely for its functionality. When it comes to databases, this can often mean making a tradeoff between latency and consistency.

The tradeoff exists for any database that serves multiple copies of its data in different availability zones or geographic regions for reliability. A database designed to ensure that everyone sees a consistent view of the latest data might update those copies synchronously using a consensus algorithm, which affects how quickly the data can be served. On the other hand, a database that’s optimized for faster data serving might update each copy asynchronously and not guarantee consistent reads across records.

Cloud Spanner and Bigtable–two of Google Cloud’s managed databases–are both highly effective services, and each one could support many of our travel applications. But as you will see, the latency vs. consistency tradeoff made it clear which one was best suited for two of our most critical cases.

Google Cloud Spanner facilitates strong, global consistency for Sabre

An airline’s reservation database stores a passenger’s booking information, seat selection, tickets, special requests, and other critical information about their trip. As a result, this data sits at the consistency end of that consistency/latency spectrum. Sabre typically processes thousands of reservation updates per second on behalf of our carrier customers. An airline’s reservation database must be served from many availability zones (and data is replicated across these availability zones) so that it remains available in the event of an outage. It also requires ACID properties for transactional updates across records since airlines often make changes to multiple passengers and multiple flights at the same time.

We needed a system that can handle bursts of concurrent updates, as would occur during a snowstorm when hundreds of thousands of passengers might be automatically moved to alternate flights. Spanner is a great fit for the reservations case because of its unique consistency guarantees. It processes over 1 billion requests per second at peak and provides five 9s SLA (99.999 percent) to support our applications. Spanner also helps us maintain compliance, business continuity, redundancy, and reliability using the same secure-by-design infrastructure, built-in data protection and replication, and multi-layered security that are essential to our Google Cloud workloads.

Spanner’s client libraries also provide built-in mechanisms to handle retrying in the event of write conflicts with another transaction and allow developers to choose stale reads in read-only transactions for improved performance. Of course, consistency across multiple zones or regions doesn’t come for free. It means higher write latencies than if we wrote to a comparable database running in a single availability zone, but for an application that manages flight reservations, it’s a tradeoff that makes sense.

Bigtable provides predictable, low latency at scale

Our flight shopping systems sit at the other end of the latency/consistency spectrum. Sabre’s shopping engine generates millions of itineraries per second on behalf of travelers using mobile apps, third-party travel websites, and airline call centers. Each itinerary requires significant compute resources to calculate: We need to find which combinations of flights make sense and evaluate complex rules about their availability and pricing. Users are typically less patient while searching for a flight than they are when booking it, so we needed a low latency solution.  But we can cache many of these shopping results to reduce our compute usage. For instance we can decide how long to cache results based on factors like how far the flight results are from the departure. 

Bigtable makes an excellent choice for this shopping cache. It’s a NoSQL database service built to handle high-throughput, low-latency applications, with more than 10 exabytes of data under management. Bigtable’s unique latency properties—such as predictability and single-digit millisecond response time even for multi-petabyte tables—enable us to serve large volumes of shopping results cost effectively, while providing low response times to travelers.

Google Cloud supports a focus on innovation

Managed databases like Bigtable and Spanner are a significant part of Sabre’s cloud strategy. The combination of unique tools like Key Visualizer for Bigtable and Spanner as well as integration with other Google Cloud services like Cloud IAM, Cloud Monitoring, and now Datastream, make the experience of operating managed databases with Google Cloud much easier than it is with several of their self-hosted counterparts. As a result of their granular pricing models and ability to be deployed and automated by SREs, the managed databases we use also end up with a lower total cost of ownership.

We’re particularly excited about a few recent database-related announcements from Google Cloud. Bigtable’s SLA update gives us more concrete expectations in terms of multi-cluster, multi-region uptime. Spanner’s change to provisioning in Processing Units increases its cost efficiency when deploying in non-production environments where we may need many isolated instances, but won’t come close to the limits of a single node. In those cases, Spanner instances may now be configured in one-tenth of a node increments.

Our cloud transformation depends on having a choice of databases for different use cases, trade offs, and migration schedules. In addition to managed databases, we expect to use self-hosted databases, database solutions available in Cloud Marketplace, and transitional services like Cloud SQL for a few more years. In an industry as demanding as travel, accelerating our most critical applications using technology unique to Google Cloud means less time spent optimizing latency and consistency, and more time spent innovating.

Learn more about how your organization can use Bigtable and Spanner.

More Relevant Stories for Your Company

Case Study

Customer Voices: How Firms from Across Industries Leverage Google Cloud

From powering everyday operations and accelerating application innovation, to providing tools for specific business needs and executing on big ideas, to advancing the security of technology solutions, companies from across industries have leveraged Google Cloud for business benefits. Companies from across industries have turned to Google Cloud for transforming their

How-to

How to Move From Redshift to BigQuery Easily

Enterprise data warehouses are getting more expensive to maintain. Traditional data warehouses are hard to scale and often involve lots of data silos. Business teams need data insights quickly, but technology teams have to grapple with managing and providing that data using old tools that aren’t keeping up with demand.

Case Study

Daffodil Software Saves Hundreds of Hours Managing Databases with Google Cloud SQL

Daffodil Software is an India-based information technology services company with 180 employees and satellite offices in the US, Singapore, and the United Arab Emirates. Its product: A suite of web-based applications, including a business customer relationship management (CRM) program and an enterprise resource planning (ERP) app for schools. Daffodil’s Challenge

Blog

Spark on Google Cloud: How this Helps Customers with Agility, Cost Reduction and Time Spent on Spark

Apache Spark has become a popular platform as it can serve all of data engineering, data exploration, and machine learning use cases. However, Spark still requires the on-premises way of managing clusters and tuning infrastructure for each job. Also, end to end use cases require Spark to be used along with

SHOW MORE STORIES