A Recap on Google Cloud Databases and Storage Options- Part 2
3958
Of your peers have already read this article.
6:00 Minutes
The most insightful time you'll spend today!
Isaac Newton said, “If I have seen farther than others it is because I have stood on the shoulders of giants.” He meant that in order to explain the Law of Gravity, he used the work of major thinkers who came before him in order to make intellectual progress (as-in giving credit). In the 1640’s (yes, the time of the English Civil War and the start of the mini Ice Age), we can say the Age of Reason began and the word “data” was also (re)born in a way.
Why? Hard to assert a specific reason, it could be apropos of all those events of that decade, and somewhere amidst all that, the scientists and Churchmen (rather Churchman, Henry Hammond who really coined the term data) started to pen down their credit in books and there was a continued proliferation of data to reason their finding (and most times to reason why their work was better than that of the others). And thus comes to us the word datum from the Latin verb dare (it means “to give”, not the English dare).
Dare to Recap History?
Data is history captured through language (it has become the future as well, but that is for another day). Now we all like history (well, most of us). But it is highly likely the context gets lost in the complexity and style of definition. One way to mitigate that risk is to have a clear set of definitions (language), sustained hold of events (history), a clean process of capture (extract) and a scalable process for translation and aggregation (transform). If we want our data to be successful and rise to the occasion, then we need to keep these ways to mitigate the risk of complexity in mind.
And this is exactly what we discussed in the Part 1 of this blog series, Data Modeling Basics—the various business attributes, technical aspects, design questions, and considerations for designing your database model.
In this blog…
We will look into the different databases and storage options in Google Cloud, a brief note on each one of them, when to choose one over the other, interesting alternatives, exceptions and if you make it to the end of the blog, a fun challenge to make sure we put this little tech nugget to an ACID test (see what I did there?). If you are a cloud enthusiast, a database practitioner, a data geek, or a general wonderer of life with computing, you may find this engaging…
Google Cloud Storage Options
We at Google Cloud, have realized how hard it is to go through these laundry list assessment aspects and have made it simpler for you with a Decision Tree. (Of course, It ain’t Christmas if not for the tree):
If only the world was always “Structured”
In a structured world, you will know all the attributes on a first-name basis (I mean to say that you will have a well defined fixed set of attributes that can be modeled in a table of rows and columns), and the applications are transactional or analytical in orientation. Transactional Structured Data operate one row at a time generally and they need to adhere to ACID compliance. (Ah. Now you connect the dots, if not already.) ACID properties are Atomicity, Consistency, Isolation, and Durability. Cloud SQL and Cloud Spanner are our Google Cloud choices for Transactional Structured Data use cases.
Let’s look at the below aspects for each type and structure of data:
- Why that option? (highlights and key features)
- When to choose?
- When not to choose?
- Security aspects
Cloud SQL
- Fully Managed, cloud-native RDBMS (Relational DataBase Management System) that offers both MySQL, PostgreSQL, SQL Server engines
- Cloud SQL is accessible from apps running on App Engine, GKE, or Compute Engine
Note: A managed database is one that does not require as much administration and operational support (creating databases, performing backups, updating the operating system of database instances) as an unmanaged database.
When to use Cloud SQL?
- Typical online transaction processing (OLTP) workloads
- Lift and shift of on-premise SQL databases (or from anywhere else) to cloud
- Regional applications that do not need to store > 30 TB of data in a single instance
When not to use Cloud SQL?
Cloud SQL is not an appropriate storage system for online analytical processing (OLAP) workloads or data that requires dynamic schemas on a per-object basis.
Security
Data stored is encrypted both in transit and at rest. Have built-in support for access control, using network firewalls to manage database access.
Cloud Spanner
Relational, horizontally scalable, global database with strong consistency- Supports schemas, ACID transactions, and SQL queries (ANSI 2011)
- Scales horizontally in regions, but can also scale across regions for workloads that have more stringent availability requirements
When to use Cloud Spanner?
- For large amounts of data and when you require high transactional consistency
- When you require sharding for higher throughput, access and low latency
When not to use Cloud Spanner?
Cloud Spanner is not an appropriate storage system for online analytical processing (OLAP) workloads
Security
Security features in Spanner include data-layer encryption, audit logging, and Identity and Access Management (IAM) integration.
Analytical Structure is when we want the data to tell us an aggregated or enhanced story, for which we use limited columns and multiple rows and hence mostly use a Column-Oriented storage mechanism. Column-oriented storage is if we want to store the data in the tables by columns instead of by rows, and this column-oriented storage is done to efficiently access only a subset of columns for querying. BigQuery is the data warehouse option for analytics needs.
BigQuery
BigQuery is a fully managed Data Warehouse for analytics with built-in data transfer service- Peta-byte scale, low-cost warehouse that supports loading data through the web interface, command line tools, and REST API calls
- Incorporates features for machine learning, business intelligence, and geospatial analysis that are provided through BigQuery ML, BI Engine, and GIS.
Note: A data warehouse stores large quantities of data for query and analysis instead of transactional processing.
When to use BigQuery?
For use cases that cover process analytics and optimization, big data (Petabyte scale) processing and analytics, data warehouse modernization, machine learning-based behavioral analytics, and predictions
When not to use BigQuery?
BigQuery is not a Transactional database and is oriented on running analytical queries, not for simple CRUD operations and queries.
Security
BigQuery provides encryption at rest and in transit. Cloud Data Loss Prevention (Cloud DLP) can be used to scan the BigQuery tables and to protect sensitive data and meet compliance requirements. BigQuery supports access control of datasets and tables using Identity and Access Management (IAM).
And then we have the Semi-structured and the Unstructured world of data that we will address in the below sections.
Cloud Firestore (Cloud Datastore)
Firestore is the next major version of Datastore and a re-branding of the product. Taking the best of Datastore and the Firebase Realtime Database, Firestore is a NoSQL document database built for automatic scaling, high performance, and ease of application development.
- A fully managed, serverless NoSQL Google Cloud database designed for the development of serverless apps that stores JSON data
- Can be used to store, sync, and query data for web, mobile, and IoT applications
- Automatically handles sharding and replication making it highly available, durable, and scalable
- Provides ACID transactions, SQL-like queries, indexes, and more
- If a client does not have network connectivity, the Firestore API lets your app persist data to a local disk and synchronizes itself with the current server state once connectivity is reestablished
When to use?
For use cases of app development, live synchronization, offline support, multi-user collaborative applications, leader board, etc.
When not to use?
Not a relational database so not meant for relational structured data use cases.
Security
Firestore Security Rules support serverless authentication and authorization for the mobile and web client libraries. Identity and Access Management (IAM) manages database access.
Cloud Bigtable
- Bigtable is a wide-column, fully managed, high-performance NoSQL database service designed for terabyte- to petabyte-scale workloads
- Bigtable is battle tested on Google internal Bigtable database infrastructure that powers Google Search, Google Analytics, Google Maps, and Gmail
- Provides consistent, low-latency, and high-throughput storage for large-scale NoSQL data
When to use?
- For large amounts of single key data and is preferable for low-latency, high throughput workloads
- For real-time app serving workloads and large-scale analytical workloads
When not to use?
While Bigtable is considered an OLTP system, it doesn’t support multi-row transactions, SQL queries or joins. For those use cases, consider either Cloud SQL or Datastore.
Security
- All the data at rest in Cloud Bigtable is encrypted using Google’s default encryption, by default.
- Instead of Google managing the encryption keys that protect your data, your Bigtable instance can also be protected using a key that you manage (customer-managed encryption keys (CMEK)) in Cloud Key Management Service (Cloud KMS).
Cloud Storage
- Google Cloud Storage is an object storage system that is durable and highly available, persists unstructured data like images, videos, data files, videos, backup, and other data
- It is unstructured and so the files in the cloud storage are atomic that you read the entire file but you cannot access specific blocks in the files
- Cloud Storage is available in multiple classes, depending on the availability and performance required for apps and services
- Standard – Offers the highest levels of availability and is appropriate for storing data that requires low-latency access
Nearline – Low-cost, highly durable, fast-access storage service for storing data that you access less than once per month
Coldline – Very-low-cost, highly durable, fast-access storage service for storing data that you intend to access less than once per quarter
Archive – Lowest-cost, highly durable, fast-access storage service for storing data that you intend to access less than once per year
Security
Files in Cloud Storage are organized by project into individual buckets. These buckets can support either custom access control lists (ACLs) or centralized identity and access management (IAM) controls.
Firebase Realtime Database
- Firebase is a realtime, NoSQL, Google Cloud database that is a part of the Firebase platform that allows you to store and sync data in real-time and includes caching capabilities for offline use
- Data is stored as JSON and synchronized in real-time to every connected client and remains available when app goes offline
When to use?
For mobile and web app development, development of apps that work across devices
When not to use?
Not in relational dataset use cases. The Realtime Database is a NoSQL database and as such has different optimizations and functionality compared to a relational database. The Realtime Database API is designed to only allow operations that can be executed quickly.
Security
The Realtime Database provides a flexible, expression-based rules language, called Firebase Realtime Database Security Rules, to define how your data should be structured and when data can be read from or written to. When integrated with Firebase Authentication, developers can define who has access to what data, and how they can access it.
That’s a rather packed read. But I hope you find this useful to understand comprehensively the basics of data, storage options and databases in Google Cloud Platform.
Next Steps, before I go…
In the blog part 1 of the series, I ended with an action item – “How would you model a NoSQL solution for an application that needs to query the lineage between individual entities that are represented in pairs?”.
Well, my answer is Firestore. As part of this episode, why don’t you take some time to go over the options and key aspects that attribute to this.
3659
Of your peers have already watched this video.
3:10 Minutes
The most insightful time you'll spend today!
What is Google Cloud SQL?
Cloud SQL is a fully managed relational database for MySQL, PostgreSQL, and SQL Server.
It reduces maintenance cost and automates database provisioning, storage capacity management, replication, and backups. It offers quick setup, with standard connection drivers and built-in migration tools.
How Do You Set It Up?
Cloud SQL is easy to setup. You select the region and zone where you would like the instance to be, and it gets created there.
You also have a lot of configuration options, where you can select the machine type with the right number of CPU and amount of memory you need.
Choose storage each type between solid state and hard disk drives, depending on latency queries per second and cost requirements and set storage capacity.
Higher storage capacity leads to better performance.
What About Back Up?
Cloud SQL also offers automated backups and recovery options.
You can set time slots and locations for backups. For production applications, it is recommended to enable high availability, or HA.
By enabling this feature, the database instancewill automatically failover to another zone in your selected region in case of an outage.
You can also create cross-regional replicas to protect from regional failures.
In addition, you can enable automatic storage increase to add more storage when nearing capacity.
How Do You Migrate an Existing MySQL Database to Cloud SQL?
Cloud console makes it very easy by providing a migrate data button, which guides you through easy steps.
First, you provide your data source details, things like public IP address, port number,
and your replication credentials.
Second, you create a Cloud SQL read replica, just like we discussed in the creation process, using a SQL dump file.
Third, you sync the read replica with source. And, finally, you promote the read replica to primary instance with very low downtime.
Is Data Safe in Cloud SQL?
Like anything else in Google Cloud,the data in Cloud SQL is encrypted at rest and in transit.
External connections can be encrypted using SSL or Cloud SQL Proxy, which is a tool to help you connect to your Cloud SQL instance from your local machines.
You can use Cloud SQL as a relational database for your applications that are hosted within Google Cloud, like App Engine, Cloud Run, Compute Engine, Kubernetes Engine, or Cloud Functions.
You can also connect your Cloud SQL database with applications that are hosted outside of Google Cloud.
How Much Does It Cost and What Are Some Uses Cases?
Cloud SQL pricing varies depending on MySQL, PostgreSQL,or SQL Server.
Broadly speaking, though, it’s the combination of the type of instance, storage, or network you use. SQL Server also has some licensing costs.
Since Cloud SQL is a relational database, you can use it with any online transaction processing apps, such as order or payment processing apps, where you need to handle frequent queries with fast response times.
New Enterprise Capabilities and Changes to Cloud Spanner Reduces Cost of Running Workloads by 90 Percent
5402
Of your peers have already read this article.
3:00 Minutes
The most insightful time you'll spend today!
Customers love Cloud Spanner because it gives them the benefits of relational semantics and SQL while also delivering the scale and availability of non-relational databases. Many of these customers want to move even more of their work to Spanner, and have requested smaller instance sizes to support development, testing and small production workloads. We’re happy to announce that they’ll soon get what they asked for: more granular instance sizing is coming to Spanner.
Granular instance sizing will be available in Public Preview soon. With this feature, you can run workloads on Spanner at as low as 1/10th the cost of regular instances, equating to approximately $65/month.
In addition, we are launching new enterprise capabilities to break down operational silos for real-time insights and provide greater database observability to developers:
- Datastream, now in public preview, is a change data capture (CDC) service that allows enterprises to synchronize data across heterogeneous databases and applications. With Spanner support in Datastream, users will be able to stream data from MySQL or Oracle to Spanner reliably and with minimal latency.
- BigQuery federation to Spanner (coming soon) lets users query transactional data residing in Spanner, from BigQuery, without moving or copying data.
- Key Visualizer, available now in public preview, provides interactive monitoring so developers can quickly identify trends and usage patterns in Spanner.
Democratizing access with more granular instance sizing
Today, customers provision Spanner instances by specifying the number of nodes they need to run their workloads. Each node can provide up to 10,000 queries per second (QPS) of reads or 2,000 QPS of writes (writing single rows at 1 KB of data per row) and 2 TB of storage. A Spanner node is replicated across 3 zones for regional instances, and 5 or more zones for multi-regional instances. The choice of node count determines the amount of serving and storage resources that are available to the databases in a given instance.
Historically, the most granular unit for provisioning resources on Spanner has been one node. To enable more granular control, we are introducing Processing Units (PUs); one Spanner node is equal to 1,000 PUs. Customers can now provision in batches of 100 PUs, and get a proportionate amount of compute and storage resources. This will allow teams to run smaller workloads on Spanner at much lower cost. With this feature, customers can start at 100 PUs and scale up as needed in batches of 100 PUs, to up to 1,000 PUs (1 node), all with zero downtime. Subsequently, customers can continue to scale up by adding more nodes, just like what they do today. Customers do have the choice of using either PUs or nodes to provision resources within workloads, when those workloads occupy multiple nodes of capacity.
To illustrate with an example, let’s say a game developer creates a Spanner instance called “baseball-game” in us-central1, with 100 PU compute capacity at $65/month price.
As you can see below, proportional maximum storage of 205 GB is assigned to the 100 PU instance.
Once the game starts becoming popular and resource demands increase, the user edits the instance to increase the compute capacity to 500 PUs with proportional maximum 1 TB of storage.
The game grows in popularity, and the gaming company prepares to launch a much-awaited capability in the game. Anticipating a sharp increase in usage at launch day, the game developers increase the compute capacity to 3,000 PUs. Compute capacity assigned to the instance over a period of time can be viewed in Cloud Monitoring graphs:
Request Access
You can request early access to granular instance sizing feature by filling this form.
Breaking down operational silos with Datastream CDC and BigQuery federation
BigQuery Federation. BigQuery has made analytics easy by bringing together data from multiple sources for seamless analysis. Soon you’ll also be able to analyze data in Spanner directly in BigQuery. With Spanner’s BigQuery federation, you’ll be able to instantly query data residing in Spanner in real-time without moving or copying the data. Simply set up the Spanner as an external data source in BigQuery, as shown below.
Change Data Capture ingest. Now in public preview, Datastream lets you stream change data into Google Cloud from MySQL and Oracle databases. As of today, you can ingest this change data directly into Spanner using a built-in Dataflow template. This lets you migrate data from MySQL and Oracle databases into Spanner in near-real time.
Understanding performance and resource usage with Key Visualizer
Key Visualizer is a new interactive monitoring tool that lets developers and administrators analyze usage patterns in Spanner. It reveals trends and outliers in key performance and resource metrics for databases of any size, helping to optimize queries and reduce infrastructure costs. Designed for performance tuning and instance sizing, Key Visualizer is available today in public preview in the web-based Cloud Console for all Spanner databases at no additional cost. Learn more in the Key Visualizer blog.
Learn more
- To request early access to granular instance sizing in Spanner, fill out this form.
- To get started with Spanner today, create an instance or try it out with a Spanner Qwiklab.
Lending DocAI Shortens Borrowers’ Journey on Roostify
11825
Of your peers have already read this article.
3:00 Minutes
The most insightful time you'll spend today!
The home lending journey entails processing an immense number of documents daily from hundreds of thousands of borrowers. Currently, home lending document processing relies on some outdated digital models and a high dependency on manual labor, resulting in slow processing times and higher origination costs. Scaling a business that sorts through millions of documents daily, while increasing efficacy and accuracy, is no small feat. When it comes to applying for a mortgage loan, consumers expect a digital experience that’s as good as the in-person one. Roostify simplifies the home lending journey for lenders and their customers.
No time to spare: Overcoming document processing challenges with AI
Roostify provides enterprise cloud applications for mortgage and home lenders. In order to empower its customers to deliver a better, more personalized lending experience, they needed to automate and scale their in-house document parsing functionality.
As a key component of its document intelligence service, Roostify is leveraging Google Cloud’s Lending DocAI machine learning platform to automate processing documents required during a home loan application process, such as tax returns or bank statements with multi-language support. This partnership delivers data capture at scale, enabling Roostify customers to automatically identify document types from the uploaded file and to extract relevant entities such as wages, tax liabilities, names, and ID numbers for further processing, and make things move faster in the cumbersome lending process.
Roostify’s solutions leverage Google Cloud’s Lending DocAI, which is built on the recently announced Document AI platform, a unified console for document processing. Customers can easily create and customize all the specialized parsers (e.g., mortgage lending documents and tax returns parsers) on the platform without the need to perform additional data mapping or training. All Google Cloud’s specialized parsers are fine-tuned to achieve industry-leading accuracy, helping customers and partners confidently unlock insights from documents with machine learning. Learn more about the solution from the GA launch blog and the overview video.
Integrating Lending DocAI’s intelligent document processing capabilities into the Roostify platform means more innovation for their customers and tangible results: faster loan processing times, fewer document intake errors, and lower origination costs. Additional support in Google Lending DAI for other languages and more documents like global Know Your Customer (KYC) documents or payroll reports is in the near future.
Full integration of AI solutions
Working together with Roostify’s platform team, we were able to help them solve their document processing challenge through integration of various GCP products such as Lending DocAI (LDAI), Data Loss Prevention (DLP) for redacting sensitive data, BigQuery for data warehousing and analytics, and Firestore for API status. To make it very safe and secure, all data was encrypted end-to-end at Rest and in Transit. LDAI won’t require any training data to process. It is an easy plug and play API.
Here is a sneak peek in the high level deployment architecture for LDAI in Roostify environment:
Here are the steps for processing data:
- Receives document processing request from the client.
- API Function directs requests to the pre-processing service. For Async requests a processing ID is generated and returned to the caller.
- Pre-processing service sends the request for further processing (Long/short PDF conversion), calling other microservices and receives back the responses. Any error in the response received is then sent to the response processing service.
- If the response is synchronous, the pre-processing service directs it to the LDAI Invoker service.
- If the response is asynchronous, the pre-processing service feeds it into the Cloud Pub/Sub service.
- Cloud Pub/Sub service feeds the response back to the LDAI Invoker service.
- LDAI Invoker service routes the request to the Google LDAI API for classification if there are multiple pages in the document.
- Document will be split based on LDAI response and then saved in a GCS bucket for temporary storage.
- LDAI entity interface for single page processing and then LDAI Invoker sends LDAI results to LDAI Response Processing
- If a request is a synchronous request the LDAI Response Processor sends results to the API Function so that it can complete the synchronous call and respond to the rConnect caller.
- If the request is an asynchronous request the LDAI Response Processor will respond to the caller’s webhook and complete the transaction.
- Finally, Data stored in the GCP bucket will be deleted.
All the responses that come from the LDAI API can optionally feed into BigQuery via the Response Processor, after parsing it through Data Loss Prevention (DLP) API to redact the PII/sensitive information. Throughout the processing of both asynchronous and synchronous requests all transactions are logged using Cloud Logging. For asynchronous transactions, the state is maintained throughout the process using Cloud Firestore.
Roostify currently uses this technology to power two different solutions: Roostify Document Intelligence and Roostify Beyond™. Roostify Document Intelligence is a real-time document capture, classification, and data extraction solution built for home lenders. It ingests documents uploaded by borrowers and loan officers, identifies the relevant documents, and extracts and classifies key information. Roostify Document Intelligence is available as a standalone API service to any home lender with any digital lending infrastructure already in place.
Roostify Beyond™ is a robust suite of AI-powered solutions that enables home lenders to create intelligent experiences from start to close. It combines powerful data, insightful analytics, and meaningful visualization to streamline the underwriting process. Roostify Beyond™ is currently available only to Roostify customers as part of an Early Adopter program and will be rolled out to the market later this year.
Through this partnership, Roostify has enabled its customers to adopt a data-first approach to their home lending processes, which will lead to improved user experiences and significantly reduced loan processing times.
Fast track end-to-end deployment with Google Cloud AI Services (AIS)
Google AIS (Professional Services Organization), in collaboration with our partner Quantiphi, helped Roostify deploy this system into production and fast-tracked the development multifold to generate the final business value.
The partnership between Google Cloud and Roostify is just one of the latest examples of how we’re providing AI-powered solutions to solve business problems.
Simplifying Your Database Migration With Google Cloud
2871
Of your peers have already read this article.
3:30 Minutes
The most insightful time you'll spend today!
For several decades, before the rise of cloud computing upended the way we think about databases and applications, Oracle and Microsoft SQL Server databases were a mainstay of business application architectures. But today, as you map out your cloud journey, you’re probably reevaluating your technology choices in light of the cloud’s vast possibilities and current industry trends.
In the database realm, these trends include a shift to open source technologies (especially to MySQL, PostgreSQL, and their derivatives), adoption of non-relational databases, multi-cloud and hybrid-cloud strategies, and the need to support global, always-on applications. Each application may require a different cloud journey, whether it’s a quick lift-and-shift migration, a larger application modernization effort, or a complete transformation with a cloud-first database.
Google Cloud offers a suite of managed database services that support open source, third-party, and cloud-first database engines. At Next 2022, we published five new videos specifically for Oracle and SQL Server customers looking to either lift-and-shift to the cloud or fully free themselves from licensing and other restrictions. We hope you’ll find the videos useful in thinking through your options, whether you’re leaning towards a homogeneous migration (using the same database you have today) or a heterogeneous migration (switching to a different database engine).
Let’s dive into our five new videos.
#1 Running Oracle-based applications on Google Cloud
By Jagdeep Singh & Andy Colvin
Moving to the cloud may be difficult if your business depends on applications running on an Oracle database. Some applications may have dependencies on Oracle for reasons such as compatibility, licensing, and management. Learn about several solutions from Google Cloud, including Bare Metal Solution for Oracle, a hardware solution certified and optimized for Oracle workloads, and solutions from cloud partners such as VMware and Equinix. See how you can run legacy workloads on Oracle while adopting modern cloud technologies for newer workloads.
#2 Running SQL Server-based applications on Google Cloud
By Isabella Lubin
Microsoft SQL Server remains a popular commercial database engine. Learn how to run SQL Server reliably and securely with Cloud SQL, a fully-managed database service for running MySQL, PostgreSQL and SQL Server workloads. In fact, Cloud SQL is trusted by some of the world’s largest enterprises with more than 90% of the top 100 Google Cloud customers using Cloud SQL. We’ll explore how to select the right database instance, how to migrate your database, how to work with standard SQL Server tools, and how to monitor your database and keep it up to date.
#3 Choosing a PostgreSQL database on Google Cloud
By Mohsin Imam
PostgreSQL is an industry-leading relational database widely admired for its permissive open source licensing, rich functionality, proven track record in the enterprise, and strong community of developers and tools. Google Cloud offers three fully-managed databases for PostgreSQL users: Cloud SQL, an easy-to-use fully-managed database service for open source PostgreSQL; AlloyDB, a PostgreSQL-compatible database service for applications that require an additional level of scalability, availability, and performance; and Cloud Spanner, a cloud-first database with unlimited global scale, 99.999% availability and a PostgreSQL interface. Learn which one is right for your application, how to migrate your database to the cloud, and how to get started.
#4 How to migrate and modernize your applications with Google Cloud databases
By Sandeep Brahmarouthu
Migrating your applications and databases to the cloud isn’t always easy. While simple workloads may just require a simple database lift-and-shift, custom enterprise applications may benefit from more complete modernization and transformation efforts. Learn about the managed database services available from Google Cloud, our approach to phased modernization, the database migration framework and programs that we offer, and how we can help you get started with a risk-free assessment.
#5 Getting started with Database Migration Service
By Shachar Guz & Inna Weiner
Migrating your databases to the cloud becomes very attractive as the cost of maintaining legacy databases increases. Google Cloud can help with your journey whether it’s a simple lift-and-shift, a database modernization to a modern, open source-based alternative, or a complete application transformation. Learn how Database Migration Service simplifies your migration with a serverless, secure platform that utilizes native replication for higher fidelity and greater reliability. See how database migration can be less complex, time-consuming and risky, and how to start your migration often in less than an hour.
We can’t wait to partner with you
Whichever path you take in your cloud journey, you’ll find that Google Cloud databases are scalable, reliable, secure and open. We’re looking forward to creating a new home for your Oracle- and SQL Server-based applications.
Start your journey with a Cloud SQL or Spanner free trial, and accelerate your move to Google Cloud with the Database Migration Program.
Memorystore for Redis Read Replicas to Scale App Read Requests by 6X
5524
Of your peers have already read this article.
2:00 Minutes
The most insightful time you'll spend today!
Modern applications need to process large-scale data at millisecond latency to provide experiences like instant gaming leaderboards, fast analysis of streaming data from millions of IoT sensors, or real-time threat detection of malicious websites. In-memory datastores are a critical component to deliver the scale, performance, and availability required by these modern applications.
Memorystore makes it easy for developers building applications on Google Cloud to leverage the speed and powerful capabilities of the most loved in-memory store: Redis. Memorystore for Redis standard tier instances are a popular choice for applications requiring a highly available Redis instance. Standard tier provides a failover replica across zones for redundancy and provides fast failover with a 99.9% SLA. However, in some cases, your applications will require more read throughput from a standard tier instance. One of the common patterns customers use to scale read queries in Redis is leveraging read replicas.
Introducing Memorystore for Redis Read Replicas
Today we are excited to announce the public preview of Memorystore for Redis Read Replicas, which allows you to seamlessly scale your application’s read requests by 6X with the click of a button.
With read replicas, you can easily add up to five replicas and leverage the read endpoint to automatically load balance read queries across all the available replicas, increasing read performance linearly with each replica added. Additionally, Memorystore’s support for Redis 6 introduced multi-thread I/O, increasing performance significantly for M3 and higher configurations. Combined, you can achieve read requests of more than a million requests per second.
You will benefit from this new functionality in several ways. You will be able to scale on demand with up to five read replicas and use read endpoint with any redis client to easily load balance read queries across multiple replicas. This new functionality will also improve availability with automatic distribution of replicas across multiple zones. With read replicas, you will also be able to minimize application downtime with fast failover to the replica with the least replication lag. In the future, Memorystore will also easily enable read replicas on existing standard tier instances to increase read throughput.
You can learn more about how to configure and use read replicas in the Read Replicas Overview.
Improving performance with Read Replicas and Redis 6
With the launch of read replicas, you can easily increase the read throughput of a Memorystore instance. You can further enhance the read performance by leveraging Redis version 6 along with read replicas.
To understand why combining read replicas and Redis 6 can significantly improve your application’s read performance, let’s look at the various Memorystore configurations that you can use with your applications today. Memorystore Basic and Standard offerings provide different capacity tiers. The capacity tier determines the single node performance of a basic and standard tier instance.
The table below outlines the configuration of the various capacity tiers:
Up until version 5, Redis processed commands using a single thread. The processing involved reading the request, parsing the request, processing the request, and writing the response back to the socket. This approach means that all of the processing, which includes writing the response, was sequentially processed by a single vCPU regardless of the number of vCPUs available in the instance.
Redis 6 introduced I\O threading which allows writing the response using parallel threads. This functionality enables Redis 6 to more effectively leverage available vCPUs, thereby increasing the overall throughput compared to lower versions. Memorystore for Redis version 6 leverages I\O threading and automatically configures the optimal number of I\O threads to achieve the best possible performance for the various capacity tiers. We have also improved the overall network throughput for all redis versions by leveraging improvements in the Google Cloud infrastructure. Together these improvements deliver significant performance improvements and come at no additional cost to you.
So what can you expect from using Redis 6? As outlined in the table, Redis version 5 and lower uses a single thread to process the write requests for all tiers while Redis 6 uses a larger number of I\O threads at higher capacity tiers which provides substantially incremental throughput for higher capacity tiers.
For example, using a Redis 6 standard tier instance with a capacity of 101 GB (M5), we have observed up to a 200% improvement in read/write performance compared to version 5, though the actual value you’ll see is dependent on your workload. You can get the benefits of Redis 6 by upgrading an existing instance or by deploying a new instance.
By enabling read replicas on an instance using Redis 6, you can further improve the read performance. Read throughput scales linearly with the number of replicas so you can increase your read queries on an instance by up to 500% using five read replicas. By combining Redis 6 and read replicas, you can see a 10X increase in read performance compared to a version 5 standard tier instance with no read replicas.
We are excited about the launch of read replicas but this is just one step in our journey to deliver the scale you need at the best price-performance. You can learn more about read replicas pricing on our Memorystore pricing page and get started using the Read Replicas Overview.
More Relevant Stories for Your Company
BURGER KING Germany: Serving Up Marketing Insights and Supply Chain Visibility Easily
Do hamburgers really come from Hamburg? This may still be a matter of debate, but the popularity of American-style burger joints not just in Hamburg but all over Germany, is clear. Germany’s top two fast food companies are both burger chains. One of them is BURGER KING®, a global brand that welcomes more
Partnering with Google Cloud is the Key Behind Recent Healthcare Innovations
It’s simply amazing to witness how some of our systems integrators employ Google Cloud solutions to drive innovation in ways we at Google may never have considered—especially in healthcare. According to analyst firm MarketsandMarkets, the market for the Cloud in healthcare is projected to grow 43% between 2020 and 2025
Gartner Identifies Critical Capabilities for Data Management Solutions
Data management solutions for analytics offerings are consolidating, with major vendors able to address a range of use cases and smaller vendors addressing a subset of use cases. Data and analytics leaders can use this research to guide evaluation and initial vendor selection for DMSA offerings. For data and analytics
Bitly’s Big Move: A Story of Massive Data Migration to Google’s Cloud Bigtable
Editor’s note: Bitly, the link & QR Code management platform, migrated 80 billion rows of link data to Cloud Bigtable. Here’s how and why they moved this data from a MySQL database, all in just six days. Our goal at Bitly is simple: Make your life easier with connections. Whether you're sharing
