Picking the right database for your application is not easy. The choice depends heavily on your use case—transactional processing, analytical processing, in-memory database, and so on—but it also depends on other factors. This post covers the different database options available within Google Cloud across relational (SQL) and non-relational (NoSQL) databases and explains which use cases are best suited for each database option. 

Relational databases 

In relational databases information is stored in tables, rows and columns, which typically works best for structured data. As a result they are used for applications in which the structure of the data does not change often. SQL (Structured Query Language) is used when interacting with most relational databases. They offer ACID consistency mode for the data, which means:

• Atomic: All operations in a transaction succeed or the operation is rolled back.
• Consistent: On the completion of a transaction, the database is structurally sound.
• Isolated: Transactions do not contend with one another. Contentious access to data is moderated by the database so that transactions appear to run sequentially.
• Durable: The results of applying a transaction are permanent, even in the presence of failures.

Because of these properties, relational databases are used in applications that require high accuracy and for transactional queries such as financial and retail transactions. For example: In banking when a customer makes a funds transfer request, you want to make sure the transaction is possible and it actually happens on the most up-to-date account balance, in this case an error or resubmit request is likely fine.

There are three relational database options in Google Cloud: Cloud SQL, Cloud Spanner, and Bare Metal Solution.

Cloud SQL: Provides managed MySQL, PostgreSQL and SQL Server databases on Google Cloud. It reduces maintenance cost and automates database provisioning, storage capacity management, back ups, and out-of-the-box high availability and disaster recovery/failover. For these reasons it is best for general-purpose web frameworks, CRM, ERP, SaaS and e-commerce applications.

Cloud Spanner: Cloud Spanner is an enterprise-grade, globally-distributed, and strongly-consistent database that offers up to 99.999% availability, built specifically to combine the benefits of relational database structure with non-relational horizontal scale. It is a unique database that combines ACID transactions, SQL queries, and relational structure with the scalability that you typically associate with non-relational or NoSQL databases. As a result, Spanner is best used for applications such as gaming, payment solutions, global financial ledgers, retail banking and inventory management that require ability to scale limitlessly with strong-consistency and high-availability. 

Bare Metal Solution: Provides hardware to run specialized workloads with low latency on Google Cloud. This is specifically useful if there is an Oracle database that you want to lift and shift into Google Cloud. This enables data center retirements and paves a path to modernize legacy applications. 

Non-relational databases

Non-relational databases (or NoSQL databases) store compex, unstructured data in a non-tabular form such as documents. Non-relational databases are often used when large quantities of complex and diverse data need to be organized. Unlike relational databases, they perform faster because a query doesn’t have to access several tables to deliver an answer, making them ideal for storing data that may change frequently or for applications that handle many different kinds of data. 

For example, an apparel store might have a database in which shirts have their own document containing all of their information, including size, brand, and color with room for adding more parameters later such as sleeve size, collars, and so on.

Qualities that make NoSQL databases fast:

• Eventual consistency: stores usually exhibit consistency at some later point (e.g., lazily at read time)
• Horizontal scaling, usually using hashed distributions
• Typically, they are optimized for a specific workload pattern (i.e., key-value, graph, wide-column)
• Typically, they don’t support cross shard transactions or flexible isolation modes.

Because of these properties, non-relational databases are used in applications that require large scale, reliability, availability, and frequent data changes.They can easily scale horizontally by adding more servers, unlike some relational databases, which scale vertically by increasing the machine size as the data grows. Although, some relations databases such as Cloud Spanner support scale-out and strict consistency.

Non-relational databases can store a variety of unstructured data such as documents, key-value, graphs, wide columns, and more. Here are your non-relational database options in Google Cloud: 

• Document databases: Store information as documents (in formats such as JSON and XML). For example: Firestore
• Key-value stores: Group associated data in collections with records that are identified with unique keys for easy retrieval. Key-value stores have just enough structure to mirror the value of relational databases while still preserving the benefits of NoSQL. For example: Datastore, Bigtable, Memorystore
• In-memory database: Purpose-built database that relies primarily on memory for data storage. These are designed to attain minimal response time by eliminating the need to access disks. They are ideal for applications that require microsecond response times and can have large spikes in traffic. For example: Memorystore
• Wide-column databases: Use the tabular format but allow a wide variance in how data is named and formatted in each row, even in the same table. They have some basic structure while preserving a lot of flexibility. For example: Bigtable
• Graph databases: Use graph structures to define the relationships between stored data points; useful for identifying patterns in unstructured and semi-structured information. For example: JanusGraph

There are three non-relational databases in Google Cloud:

• Firestore: Is a serverless document database which scales on demand and acts as a backend-as-a-service. It is DBaaS that increases the speed of building applications. It is perfect for all general purpose uses cases such as ecommerce, gaming, IoT and real time dashboards. With Firestore users can interact with and collaborate on live and offline data making it great for real-time application and mobile apps.  
• Cloud Bigtable: Cloud Bigtable is a sparsely populated table that can scale to billions of rows and thousands of columns, enabling you to store terabytes or even petabytes of data. It is ideal for storing very large amounts of single-keyed data with very low latency. It supports high read and write throughput at sub-millisecond latency, and it is an ideal data source for MapReduce operations. It also supports the open-source HBase API standard to easily integrate with the Apache ecosystem including HBase, Beam, Hadoop and Spark along with Google Cloud ecosystem.
• Memorystore: Memorystore is a fully managed in-memory data store service for Redis and Memcached at Google Cloud. It is best for in-memory and transient data stores and automates the complex tasks of provisioning, replication, failover, and patching so you can spend more time coding. Because it offers extremely low latency and high performance, Memorystore is great for web and mobile, gaming, leaderboard, social, chat, and news feed applications.

Conclusion

Choosing a relational or a non-relational database largely depends on the use case. Broadly, if your application requires ACID transactions and your data structure is not going to change much, select a relational database. 

In Google Cloud use Cloud SQL for any general-purpose SQL database and Cloud Spanner for large-scale globally scalable, strongly consistent use cases. In general, if your data structure may change later and if scale and availability is a bigger requirement than consistency then a non-relational database is a preferable choice.  Google Cloud offers Firestore, Memorystore, and Cloud Bigtable to support a variety of use cases across the document, key-value, and wide column database spectrum.

For more comparison resources on each database check out the overview. For more hands-on experience with Bigtable, check out our on-demand training here and learn about migrating databases to managed services check out this whitepaper.  

https://youtube.com/watch?v=2TZXSnCTd7E%3Fenablejsapi%3D1%26

For more #GCPSketchnote, follow the GitHub repo. For similar cloud content follow me on Twitter @pvergadia and keep an eye out on thecloudgirl.dev.

During uncertain times, ecosystem partnerships that leverage APIs have proven to help companies scale and address gaps in their businesses. Apigee customers, like CHAMP Cargosystems, have pursued API-first ecosystem models to enter adjacent markets, create new customer interaction models, and rapidly grow their brand reach and partner ecosystems. As a result of building their API ecosystem, they are transacting 300 million electronic exchanges and 20 million shipments per year.

CHAMP selected Apigee to provide an API platform and developer self-service portal option to all of its SaaS customers. Google Cloud’s Apigee API management platform and portal allows CHAMP and its customers to quickly connect to a variety of backend systems, including in-house, third party apps, marketplace portals and more– thereby accelerating their digitization strategies and opening up new markets through an API ecosystem.

Join this webcast and hear from this leading Enterprise company on how to:

• Identify new revenue sources and markets using an API management platform
• Improve time to market while still complying with all industry requirements
• How to create a proof of concept to grow API adoption throughout your organization
• Align internal business leaders and partners to see the importance of an API-first platform vision

Secret Manager is a Google Cloud service that provides a secure and convenient way to store API keys, passwords, certificates, and other sensitive data. It is the central place and single source of truth to manage, access, and audit secrets across Google Cloud. Since its launch, Secret Manager has helped secure millions of workloads and continues to provide industry-first features like replication policies and support for VPC perimeters. This blog post explores new Secret Manager capabilities and integrations that will help keep your secrets safer.

New tier, free of charge

No, you’re not dreaming – Secret Manager now has a tier that is free of charge! With this tier, each month per billing account you can have up to:

• 6 secret versions
• 3 rotation events
• 10,000 API calls

This enables you to experience the value of Secret Manager with minimal financial risk and pairs nicely with existing services that offer a free tier like Cloud Run and Cloud Functions. For existing Secret Manager customers, this change will go into effect next billing cycle. Learn more about the Secret Manager free tier in the documentation.

Increased SLA

To meet the growing availability and reliability requirements of our customers, the Secret Manager SLA is now 99.95%! With this update, Secret Manager guarantees that all valid requests will succeed 99.95% of the time. This means you can depend on Secret Manager for even your most critical workloads. Additional details are available in the updated Secret Manager SLA.

Geo-expansion

In addition to the free tier and increased SLA, Secret Manager is now available in all public Google Cloud regions! With Secret Manager’s replication policies, you can choose the specific regions in which to replicate your secret, which means you can store secret payloads in geographical proximity to your workloads or users to reduce latency. This is also very useful if you have legal or regulatory requirements to store data in a particular locality. For more information, check out the list of Secret Manager locations in the documentation.

Compliance certifications

For customers wishing to use Secret Manager to store and process regulated data, Secret Manager is validated for compliance use cases including ISO 27001, ISO 27017, ISO 27018, SOC 1, SOC 2, SOC 3, PCI DSS, and HIPAA. Combined with the increased SLA and geographical availability, this makes Secret Manager suitable for use with regulated workloads.

Customer-Managed Encryption Keys (CMEK)

Secret Manager has always encrypted payloads in transit with TLS and at rest with AES-256. For customers that want additional control over the keys used to encrypt their secret payloads, Secret Manager now supports Customer-Managed Encryption Keys (CMEK). Secret Manager CMEK supports software-backed keys via Cloud KMS, hardware-backed keys via Cloud HSM, and even externally-managed keys via Cloud EKM. Learn how to enable CMEK support for Secret Manager in our tutorial.

Expiration and TTLs

While it was previously possible to expire access to a secret using IAM conditions, the underlying secret would continue to exist. Secret Manager now supports auto-expiring secrets which permanently deletes a secret at a specified timestamp or TTL. Since it is also possible to update a secret’s TTL, services can “lease” a secret and renew their lease on a periodic basis. If the service does not extend the lease by updating the TTL, the secret is automatically deleted.

Expiring secrets can be used in combination with IAM conditions to more safely expire secrets. For more information on expiring secrets and safety measures, see the guide on creating and managing expiring secrets.

Etags and server-side filtering

For customers that create or manage Secret Manager secrets via the API or an SDK, concurrency controls and performance are extremely important. This is why Secret Manager now supports Etags and server-side filtering! Etags help prevent concurrent modifications to the same secret by providing optimistic concurrency controls, while server-side filtering can dramatically reduce payload size and client-side computational overhead. Together, these enable stronger consistency guarantees and performance improvements to your applications. Learn more about Secret Manager Etags and Secret Manager server-side filtering in the documentation.

Code, build, run, deploy, monitor, and orchestrate

Secrets – like API keys, passwords, and certificates – are an integral part of most modern software applications. It is crucial that developers, operators, and security teams are empowered to build, operate, and observe software securely. That is why Secret Manager is now integrated with popular tools and technologies used throughout the application development lifecycle:

• Code – Software engineers can create and access secrets directly from their preferred IDEs with Cloud Code. In VS Code, IntelliJ, or the Cloud Shell Editor, developers can browse secrets and insert code snippets for access secrets, all from the comfort of their local IDE.
• Build – Release engineers can access secrets as part of CI builds using the Cloud Build Secret Manager integration. This could be used, for example, to authenticate to a Docker registry or communicate with the GitHub API. For customers that use other CI systems, there is also a GitHub Action for accessing Secret Manager secrets.
• Run (on serverless) – Developers can mount secrets to be available as environment variables or via the filesystem through the native Cloud Run Secret Manager integration. Since the secrets are resolved in Cloud Run’s control plane, developers can use this integration to avoid a tight coupling between their applications and Secret Manager to enable hybrid cloud deployments or better local development experiences.
• Run (on Kubernetes) – Developers can mount secrets from GKE, Anthos, or any Kubernetes cluster using the Secret Manager CSI driver. This vendor-agnostic driver exposes secrets via environment variables or the filesystem and enables hybrid cloud deployments using the same interface as other public cloud providers and HashiCorp Vault.
• Deploy – To complement the existing Secret Manager Terraform integration, operators can now manage Secret Manager via Kubernetes Config Connector (KCC). KCC allows operators to manage Google Cloud resources through Kubernetes and the familiar Kubernetes APIs.
• Monitor – With the Secret Manager Cloud Asset Inventory (CAIS) integration, security teams can understand secret usage across specific projects, folders, or the entire organization.
• Orchestrate – Secret Manager Event Notifications enable DevOps and security teams to subscribe to Pub/Sub topics for when secrets or secret versions are changed. This enables customers to create deeply-integrated workflows, such as creating a ServiceNow ticket when a new secret version is added. Additionally, Secret Manager Rotation Scheduling enables DevOps teams to build automatic rotation flows like the ones described in the rotation guide.

Best practices

The Secret Manager best practices guide ensures customers get the maximum security benefits from Secret Manager. Security is non-binary, and this guide covers nuanced topics like access controls, coding practices, and secret administration. While not an exhaustive list, the Secret Manager best practices guide answers some of the most common questions and concerns around using Secret Manager in production deployments.

Towards seamless security

Secrets management is an important part of every organization’s security toolkit. With Secret Manager, you can easily manage, audit, and access secrets like API keys and credentials across Google Cloud, Anthos, and on-premises. These new features and integrations make it easy to adopt Secret Manager whether you are a hobbyist working on a side project or a large enterprise with thousands of employees.

To get started, check out the Secret Manager documentation.

In the past year, Google has worked to surface timely and reliable health information, amplify public health campaigns, and help nonprofits get urgent support to Indians in need. Now, we are continuing to focus on helping India’s businesses accelerate their digital transformation, deepening our commitment to India’s digitization and economic recovery. To support customers and the public sector in India and across Asia Pacific, we’re excited to announce that our new Google Cloud region in Delhi National Capital Region (NCR) is now open. 

Designed to help both Indian and global companies alike build highly available applications for their customers, the Delhi NCR region is our second Google Cloud region in India and 10th to open in Asia Pacific. 

What customers and partners are saying

Navigating this past year has been a challenge for companies as they grapple with changing customers demands and economic uncertainty. Technology has played a critical role, and we’ve been fortunate to partner with and serve people, companies, and government institutions around the world to help them adapt. The Google Cloud region in Delhi NCR will help our customers adapt to new requirements, new opportunities and new ways of working, like we’ve helped so many companies do in the region: 

• InMobi scaled a personalized AI platform to support 120+ million active users. “With the arrival of the Google Cloud Delhi NCR, InMobi Group sees the opportunity to continue closing the gap between our users and products,” says Mohit Saxena, Co-founder and Group CTO of Inmobi. “Glance, especially, has been serving AI-powered personalised content to over 120 million active users. We can’t wait to continue giving them truly meaningful experiences that are speedy, scale well, and are relevant to them, by expanding the use of our current tools working on Google Cloud with the opening of a new region.”
• Groww now supports a sizable user base. “Google Cloud provides great technology that enables us to build and scale infrastructure to millions of users, and the new Google Cloud region in Delhi NCR will continue to help more businesses and startups in India access powerful cloud-based infrastructure, products and services,” says Neeraj Singh, Co-founder and Chief Technology Officer, Groww.
• HDFC Bank is positioned for the future. “At HDFC Bank, we are harnessing technology platforms to both run and build the bank. As we progress to be future ready, the objective is to invest in future technologies that give us scale, efficiency and resiliency. Towards this the Google Cloud region in Delhi NCR will enable us to enhance our resiliency and help us in building an active-active design framework for our new generation applications on cloud,” says Ramesh Lakshminarayanan, CIO, HDFC Bank.  
• Dr. Reddy’s Lab built a modern data platform with Google Cloud. “At Dr Reddy’s, we pride ourselves in helping patients regain good health, acting quickly to provide innovative solutions to address patients’ unmet needs and in accelerating access to medicines to people worldwide. Our Google Cloud-powered data platform is helping us realize these objectives and we welcome Google’s investment in the new Delhi NCR region as helping us and other businesses in India make further contributions to our social and economic future,” says Mukesh Rathi, Senior Vice President & CIO, Dr. Reddy’s Laboratories.
• “To survive the disruption caused by the pandemic and to succeed in the long term, organizations need to become digital natives, so they can be more agile, explore new business models and build new capabilities that boost resilience. A cloud-first strategy plays a key role in enabling businesses to do this,” said Piyush N. Singh, Lead – India market unit & lead – Growth and Strategic Client Relationships, Asia Pacific and Latin America, Accenture. “Harnessing the potential of cloud requires the right data infrastructure and this expansion by Google Cloud will undoubtedly help Indian enterprises in their digital transformation journeys.”

A global network of regions

Delhi NCR joins 25 existing Google Cloud regions connected via our high-performance network, helping customers better serve their users and customers throughout the globe. As the second region in India, customers benefit from improved business continuity planning with distributed, secure infrastructure needed to meet IT and business requirements for disaster recovery, while maintaining data sovereignty.

With this new region, Google Cloud customers operating in India also benefit from low latency and high performance of their cloud-based workloads and data. Designed for high availability, the region opens with three availability zones to protect against service disruptions, and offers a portfolio of key products, including Compute Engine, App Engine, Google Kubernetes Engine, Cloud Bigtable, Cloud Spanner, and BigQuery. 

Supporting India’s recovery with training and education

Google and Google Cloud will also continue to support our customers with people and education programs. We’re investing in local talent and the local developer community to help enterprises digitally transform and support economic recovery. 

Through the India Digitization Fund, we expanded our efforts to support India’s recovery from COVID-19—in particular, through programs to support education and small businesses. In addition to expanding internet access, and investments to help start-ups accelerate India’s digital transformation, we’ve grown our Grow with Google efforts. Businesses can access digital tools to maintain business continuity, find resources like quick help videos, and learn digital skills—in both English and in Hindi.

Helping customers build their transformation clouds

Google Cloud is here to support businesses, helping them get smarter with data, deploy faster, connect more easily with people and customers throughout the globe, and protect everything that matters to their businesses. The cloud region in Delhi NCR offers new technology and tools that can be a catalyst for this change. To learn more, visit the Google Cloud locations page, and be sure to watch the region launch event here.