A Recap on Google Cloud Databases and Storage Options- Part 2
3951
Of your peers have already read this article.
6:00 Minutes
The most insightful time you'll spend today!
Isaac Newton said, “If I have seen farther than others it is because I have stood on the shoulders of giants.” He meant that in order to explain the Law of Gravity, he used the work of major thinkers who came before him in order to make intellectual progress (as-in giving credit). In the 1640’s (yes, the time of the English Civil War and the start of the mini Ice Age), we can say the Age of Reason began and the word “data” was also (re)born in a way.
Why? Hard to assert a specific reason, it could be apropos of all those events of that decade, and somewhere amidst all that, the scientists and Churchmen (rather Churchman, Henry Hammond who really coined the term data) started to pen down their credit in books and there was a continued proliferation of data to reason their finding (and most times to reason why their work was better than that of the others). And thus comes to us the word datum from the Latin verb dare (it means “to give”, not the English dare).
Dare to Recap History?
Data is history captured through language (it has become the future as well, but that is for another day). Now we all like history (well, most of us). But it is highly likely the context gets lost in the complexity and style of definition. One way to mitigate that risk is to have a clear set of definitions (language), sustained hold of events (history), a clean process of capture (extract) and a scalable process for translation and aggregation (transform). If we want our data to be successful and rise to the occasion, then we need to keep these ways to mitigate the risk of complexity in mind.
And this is exactly what we discussed in the Part 1 of this blog series, Data Modeling Basics—the various business attributes, technical aspects, design questions, and considerations for designing your database model.
In this blog…
We will look into the different databases and storage options in Google Cloud, a brief note on each one of them, when to choose one over the other, interesting alternatives, exceptions and if you make it to the end of the blog, a fun challenge to make sure we put this little tech nugget to an ACID test (see what I did there?). If you are a cloud enthusiast, a database practitioner, a data geek, or a general wonderer of life with computing, you may find this engaging…
Google Cloud Storage Options
We at Google Cloud, have realized how hard it is to go through these laundry list assessment aspects and have made it simpler for you with a Decision Tree. (Of course, It ain’t Christmas if not for the tree):
If only the world was always “Structured”
In a structured world, you will know all the attributes on a first-name basis (I mean to say that you will have a well defined fixed set of attributes that can be modeled in a table of rows and columns), and the applications are transactional or analytical in orientation. Transactional Structured Data operate one row at a time generally and they need to adhere to ACID compliance. (Ah. Now you connect the dots, if not already.) ACID properties are Atomicity, Consistency, Isolation, and Durability. Cloud SQL and Cloud Spanner are our Google Cloud choices for Transactional Structured Data use cases.
Let’s look at the below aspects for each type and structure of data:
- Why that option? (highlights and key features)
- When to choose?
- When not to choose?
- Security aspects
Cloud SQL
- Fully Managed, cloud-native RDBMS (Relational DataBase Management System) that offers both MySQL, PostgreSQL, SQL Server engines
- Cloud SQL is accessible from apps running on App Engine, GKE, or Compute Engine
Note: A managed database is one that does not require as much administration and operational support (creating databases, performing backups, updating the operating system of database instances) as an unmanaged database.
When to use Cloud SQL?
- Typical online transaction processing (OLTP) workloads
- Lift and shift of on-premise SQL databases (or from anywhere else) to cloud
- Regional applications that do not need to store > 30 TB of data in a single instance
When not to use Cloud SQL?
Cloud SQL is not an appropriate storage system for online analytical processing (OLAP) workloads or data that requires dynamic schemas on a per-object basis.
Security
Data stored is encrypted both in transit and at rest. Have built-in support for access control, using network firewalls to manage database access.
Cloud Spanner
Relational, horizontally scalable, global database with strong consistency- Supports schemas, ACID transactions, and SQL queries (ANSI 2011)
- Scales horizontally in regions, but can also scale across regions for workloads that have more stringent availability requirements
When to use Cloud Spanner?
- For large amounts of data and when you require high transactional consistency
- When you require sharding for higher throughput, access and low latency
When not to use Cloud Spanner?
Cloud Spanner is not an appropriate storage system for online analytical processing (OLAP) workloads
Security
Security features in Spanner include data-layer encryption, audit logging, and Identity and Access Management (IAM) integration.
Analytical Structure is when we want the data to tell us an aggregated or enhanced story, for which we use limited columns and multiple rows and hence mostly use a Column-Oriented storage mechanism. Column-oriented storage is if we want to store the data in the tables by columns instead of by rows, and this column-oriented storage is done to efficiently access only a subset of columns for querying. BigQuery is the data warehouse option for analytics needs.
BigQuery
BigQuery is a fully managed Data Warehouse for analytics with built-in data transfer service- Peta-byte scale, low-cost warehouse that supports loading data through the web interface, command line tools, and REST API calls
- Incorporates features for machine learning, business intelligence, and geospatial analysis that are provided through BigQuery ML, BI Engine, and GIS.
Note: A data warehouse stores large quantities of data for query and analysis instead of transactional processing.
When to use BigQuery?
For use cases that cover process analytics and optimization, big data (Petabyte scale) processing and analytics, data warehouse modernization, machine learning-based behavioral analytics, and predictions
When not to use BigQuery?
BigQuery is not a Transactional database and is oriented on running analytical queries, not for simple CRUD operations and queries.
Security
BigQuery provides encryption at rest and in transit. Cloud Data Loss Prevention (Cloud DLP) can be used to scan the BigQuery tables and to protect sensitive data and meet compliance requirements. BigQuery supports access control of datasets and tables using Identity and Access Management (IAM).
And then we have the Semi-structured and the Unstructured world of data that we will address in the below sections.
Cloud Firestore (Cloud Datastore)
Firestore is the next major version of Datastore and a re-branding of the product. Taking the best of Datastore and the Firebase Realtime Database, Firestore is a NoSQL document database built for automatic scaling, high performance, and ease of application development.
- A fully managed, serverless NoSQL Google Cloud database designed for the development of serverless apps that stores JSON data
- Can be used to store, sync, and query data for web, mobile, and IoT applications
- Automatically handles sharding and replication making it highly available, durable, and scalable
- Provides ACID transactions, SQL-like queries, indexes, and more
- If a client does not have network connectivity, the Firestore API lets your app persist data to a local disk and synchronizes itself with the current server state once connectivity is reestablished
When to use?
For use cases of app development, live synchronization, offline support, multi-user collaborative applications, leader board, etc.
When not to use?
Not a relational database so not meant for relational structured data use cases.
Security
Firestore Security Rules support serverless authentication and authorization for the mobile and web client libraries. Identity and Access Management (IAM) manages database access.
Cloud Bigtable
- Bigtable is a wide-column, fully managed, high-performance NoSQL database service designed for terabyte- to petabyte-scale workloads
- Bigtable is battle tested on Google internal Bigtable database infrastructure that powers Google Search, Google Analytics, Google Maps, and Gmail
- Provides consistent, low-latency, and high-throughput storage for large-scale NoSQL data
When to use?
- For large amounts of single key data and is preferable for low-latency, high throughput workloads
- For real-time app serving workloads and large-scale analytical workloads
When not to use?
While Bigtable is considered an OLTP system, it doesn’t support multi-row transactions, SQL queries or joins. For those use cases, consider either Cloud SQL or Datastore.
Security
- All the data at rest in Cloud Bigtable is encrypted using Google’s default encryption, by default.
- Instead of Google managing the encryption keys that protect your data, your Bigtable instance can also be protected using a key that you manage (customer-managed encryption keys (CMEK)) in Cloud Key Management Service (Cloud KMS).
Cloud Storage
- Google Cloud Storage is an object storage system that is durable and highly available, persists unstructured data like images, videos, data files, videos, backup, and other data
- It is unstructured and so the files in the cloud storage are atomic that you read the entire file but you cannot access specific blocks in the files
- Cloud Storage is available in multiple classes, depending on the availability and performance required for apps and services
- Standard – Offers the highest levels of availability and is appropriate for storing data that requires low-latency access
Nearline – Low-cost, highly durable, fast-access storage service for storing data that you access less than once per month
Coldline – Very-low-cost, highly durable, fast-access storage service for storing data that you intend to access less than once per quarter
Archive – Lowest-cost, highly durable, fast-access storage service for storing data that you intend to access less than once per year
Security
Files in Cloud Storage are organized by project into individual buckets. These buckets can support either custom access control lists (ACLs) or centralized identity and access management (IAM) controls.
Firebase Realtime Database
- Firebase is a realtime, NoSQL, Google Cloud database that is a part of the Firebase platform that allows you to store and sync data in real-time and includes caching capabilities for offline use
- Data is stored as JSON and synchronized in real-time to every connected client and remains available when app goes offline
When to use?
For mobile and web app development, development of apps that work across devices
When not to use?
Not in relational dataset use cases. The Realtime Database is a NoSQL database and as such has different optimizations and functionality compared to a relational database. The Realtime Database API is designed to only allow operations that can be executed quickly.
Security
The Realtime Database provides a flexible, expression-based rules language, called Firebase Realtime Database Security Rules, to define how your data should be structured and when data can be read from or written to. When integrated with Firebase Authentication, developers can define who has access to what data, and how they can access it.
That’s a rather packed read. But I hope you find this useful to understand comprehensively the basics of data, storage options and databases in Google Cloud Platform.
Next Steps, before I go…
In the blog part 1 of the series, I ended with an action item – “How would you model a NoSQL solution for an application that needs to query the lineage between individual entities that are represented in pairs?”.
Well, my answer is Firestore. As part of this episode, why don’t you take some time to go over the options and key aspects that attribute to this.
3745
Of your peers have already watched this video.
20:30 Minutes
The most insightful time you'll spend today!
Business Modernization with Oracle Workloads in Google Cloud
With today’s ever-changing business dynamics, business continuity is a key focus for all of us, and enterprise databases like Oracle play a critical role. We’ll explore how running Oracle workloads in Google Cloud creates a variety of solutions, from a Bare Metal Solution to open source and cloud-native databases, to help reduce your operational overhead and keep your business running smoothly.
This session is led by Paul Miller, Head of Solution Management for DB, Google Cloud. Paul is an accomplished data management leader with over 20 years experience in application development, data management, and solution engineering. He is an emphatic team builder who loves to figure things out. Paul has deep experience on databases from leading enterprise software companies such as Google Cloud and Oracle.
Google Leads the Database-as-a-Service Market: Forrester Research
DOWNLOAD RESEARCH REPORTS5475
Of your peers have already downloaded this article
3:00 Minutes
The most insightful time you'll spend today!
Database-as-a-service (DBaaS) has become critical for all businesses to build and support modern business applications and operational systems. It is changing the way companies build and support business applications and operational systems. With DBaaS, organizations can provision a relational or non-relational database of any size in minutes, without needing any technical expertise.
For app developers, it offers a database platform to build simple to sophisticated applications quickly, allowing them to focus on application logic rather than deal with database administration challenges. DBaaS automates the provisioning, administration, backup, recovery, availability, security, and scalability of the database without the need for a database administrator (DBA).
In addition, DBaaS helps enterprises migrate from their on-premises databases to the cloud to save money, support elastic scale, and deliver higher performance for expanding workloads.
Analyst firm Forrester Research in its recent report on the Database-as-a-Service market has named Google Cloud a leader in this space as it supports a broader set of use cases, automation, high-end scalability and performance, and security.
Download this Forrester Research report to understand why enterprises are turning to DBaaS and why Google Cloud is a leader in this space.
Google Spanner Wins SIGOPS Hall of Fame Award 2022
2917
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
Earlier this month, SIGOPS announced that it had selected the paper, “Spanner: Google’s Globally-Distributed Database” for the 2022 SIGOPS Hall of Fame Award, an honor bestowed on the most influential Operating Systems papers published by the organization.
In giving this award, the award committee stated:
“Spanner showed how to balance the requirements of consistency, availability, and partition tolerance for a globally replicated transactional database system. In doing so, Spanner simplified the implementation of a range of higher level services which could then scale across the planet through a familiar and powerful transactional API. The key insights of leveraging clock synchronization using TrueTime and a dedicated, redundant, high-capacity datacenter WAN distinct from the public Internet as the basis for scaling remain fundamental design patterns today.”
As one of the original authors of the paper, to say that I am humbled by this award is an understatement. It’s also taken me on a trip down memory lane, and gotten me thinking about all the ways — expected and unexpected — that Spanner has influenced not only how Google builds its applications, but also how it’s simplified application design and operations for organizations that aren’t running at hyperscale.
From very early on — think early 2000s — one of Google’s biggest challenges was scaling its software to keep up with the rapid growth of the internet. We needed a database that scaled, that could be built with a small team. Those needs led to Bigtable and the NoSQL movement: a feature-lean database that performed and scaled.
But as Google grew, and entered other lines of business, our priorities evolved. We needed to simplify the jobs of our ever-larger engineering staff and support business-critical services like the AdWords and Payments platforms, as well as then-new consumer products like Gmail. To do that, we knew we had to give our product teams database features like ACID transactions, multi-region consistency, schema-declared invariants, SQL, and more — without compromising on cost or scale. At the time it was an outrageous goal, but with our experience building Bigtable and Megastore, as well as new innovations like TrueTime, we knew it could be done – and it was clear talking to our internal customers that it needed to be done.
Like most good software, our plans for Spanner really came together when we found our first serious customer: AdWords. At the time, Google’s entire advertiser-facing interface was sharded across MySQL instances on specially acquired hardware. Ads had just completed a very challenging resharding project to accommodate Google’s growing business, and was resolute in its desire to never do it again. So the Spanner team and Ads partnered on a project, called F1, to transparently move the entire AdWords stack from MySQL to Spanner.
“Before Spanner, Google’s entire advertiser-facing interface was sharded across MySQL instances on specially acquired hardware.”
By 2012, when we published the Spanner paper, we had just completed the migration of AdWords frontend traffic to the new Spanner backend, and we knew we had the seed of something really interesting. Since then, we have been rapidly improving Spanner in every dimension. Google, like most businesses, needs databases that are fully managed, easy to deploy, highly available, and robust in the face of any workload. Google services set a high bar: Spanner stores a copy of the Internet (actually several, at different stages of the indexing pipeline) and is the bedrock of availability and durability for billions of users that depend on Google in their daily lives.
Then, in 2017, Google Cloud launched Cloud Spanner, a fully managed database service that brought the unique capabilities of Spanner to every organization, allowing them to deliver always-on experiences at any scale from thousands to millions of active users across the world without sacrificing consistency.
That took us down another road: innovating to make it easy for all enterprises to build data-driven applications using Cloud Spanner. This meant adding enterprise features such as CMEK and access approval, fine-grained access control, backup and restore, and point-in-time recovery (PITR). We also added VPC Service Controls support and compliance certifications and necessary approvals so that Spanner can be used for workloads requiring ISO 27001, 27017, 27018, PCI DSS, SOC1|2|3, HIPAA and FedRamp. More recently, we added granular instance sizing, a PostgreSQL interface, and free trial instances to lower the barrier to entry for Spanner, and make it accessible for any developer and for any application, big or small.
Speaking of small, one of the interesting things we discovered along the way is that Spanner isn’t only a great fit for huge mission-critical applications – we also use it internally for smaller applications and internal tools, secure in the knowledge that the data is safe, available, and nowhere near any limits. That’s the thing about infrastructure innovations: at their best, they aren’t only about scale, they’re about extracting complexity from applications and presenting the solution in an easy-to-use package.
As we have incorporated solutions to those problems into Spanner, customers have dramatically simplified their applications. For example, application teams often created a dedicated “storage” tier to provide higher-level features like indexes, transactions, or data synchronization needed to support their business logic. Spanner enables application teams to hollow out or even completely eliminate these tiers for significant cost and maintenance savings. We have also seen big reliability dividends: rather than each application discovering the corner cases of transactions, consistency or high availability architectures one at a time, Spanner’s customers are all able to benefit from a rigorously designed and battle-tested implementation out of the gate.
“At their best, infrastructure innovations aren’t only about scale, they’re about extracting complexity from applications and presenting the solution in an easy-to-use package.”
The tradeoff between the scale limitations of relational databases and the functionality limitations of NoSQL systems is really familiar, and the software world has learned to build great applications within those confines. But we never tire of seeing great applications transform as Spanner melts the distinction away.
Get started with Spanner
It has never been easier to try out Spanner. With a free trial instance, you can try out Spanner at no cost for 90 days. You can even prototype an entire application for free on Google Cloud by using the Spanner free trial along with the free tier offered by other Google Cloud products such as Compute Engine and BigQuery.
Create a 90-day Spanner free trial instance. Try Spanner free.
Take a deep dive into the new trial experience and learn more about Spanner.
10507
Of your peers have already watched this video.
2:15 Minutes
The most insightful time you'll spend today!
The True Story of How HotStar Broke a World-Record–Thanks to Firebase and Google BigQuery
Hotstar, India’s largest video streaming platform with 150 million monthly active users around the world, provides live-streaming of TV shows, movies, sports, and news on the go.
By using a combination of Firebase products together, Hotstar safely rolled out new features to its watch screen during a major live-streaming event without disrupting users, sacrificing stability, or releasing a new build. They also used Firebase with BigQuery to analyze their event data and reduce app startup time.
“We have an ambitious mission, but our engineering team is only a fraction of the size of most of our competitors. But we are still keeping up, and we are doing it with the help of Firebase,” says Ayushi Gupta, Android Engineer, Hotstar.
11208
Of your peers have already watched this video.
1:30 Minutes
The most insightful time you'll spend today!
Indian Retailer Figures Optimizes Hyperlocal Delivery to Increase Customer Experience
Anyone who follows the Indian e-commerce scene knows that one of the largest challenges these companies face is hyperlocal delivery.
That was a problem facing Wellness Forever, a retail chain of pharmacies with 150-plus stores across India.
“Exactly a year ago, we started our journey of hyperlocal deliveries. This optimization was a big time challenge for us to understand how to optimize this,” Palani Subbiah, CTO, Wellness Forever.
The problem in front of Wellness Forever was to identify which customer could can be sold from which store, so that a delivery could be made within 90 minutes.
“We handle a large amount of customer data and we wanted to use insights to help and improve the customer satisfaction index,” says Subbiah.
To do that Wellness Forever leveraged Google Big Query to run massive amount of data to come up with the operational insights. They also used Firebase and Google Maps.
“By 2021, we are going to have about 450 stores. Those stores are going to be not only a physical store, which is a digital store.
More Relevant Stories for Your Company
Google Data Cloud: The Catalyst for Modern App Development and Innovation
97 zettabytes was the estimated volume of data generated worldwide in 20221. This sort of explosion in data volume is happening in every enterprise. Now imagine being able to access all this data you own from anywhere, at any time, analyze it, and leverage its insights to innovate your products
How to Enhance Incremental Pipeline Performance while Ingesting Data into BigQuery
When you build a data warehouse, the important question is how to ingest data from the source system to the data warehouse. If the table is small you can fully reload a table on a regular basis, however, if the table is large a common technique is to perform incremental
Guide for Measuring Cloud Spanner Performance for Your Custom Workload
Database migration to a new database platform or technology can be daunting for various reasons. One of the common concerns is database performance. It is hard to evaluate database performance early in the evaluation cycle without performing actual data migration and application changes. This becomes even more important in the
How Kinguin Notched Up Shopping Experience with Google Recommendations AI
Over 2.14 billion people worldwide are expected to buy online this year, according to Statista. Online retail sales will account for 22% of all purchases by 2023. But in a competitive retail landscape, positive interactions can mean the difference between a sale and an abandoned shopping cart. One of the leading global
