Today, more and more applications are being deployed in containers on Kubernetes—so much so that we’ve heard Kubernetes called the Linux of the cloud.

Despite all that growth on the application layer, the data layer hasn’t gotten as much traction with containerization. That’s not surprising, since containerized workloads inherently have to be resilient to restarts, scale-out, virtualization, and other constraints. So handling things like state (the database), availability to other layers of the application, and redundancy for a database can have very specific requirements. That makes it challenging to run a database in a distributed environment. 

However, the data layer is getting more attention, since many developers want to treat data infrastructure the same as application stacks.

Operators want to use the same tools for databases and applications, and get the same benefits as the application layer in the data layer: rapid spin-up and repeatability across environments. In this blog, we’ll explore when and what types of databases can be effectively run on Kubernetes.

Before we dive into the considerations for running a database on Kubernetes, let’s briefly review our options for running databases on Google Cloud Platform (GCP) and what they’re best used for.

• Fully managed databases. This includes Cloud Spanner, Cloud Bigtable and Cloud SQL, among others. This is the low-ops choice, since Google Cloud handles many of the maintenance tasks, like backups, patching and scaling. As a developer or operator, you don’t need to mess with them. You just create a database, build your app, and let Google Cloud scale it for you. This also means you might not have access to the exact version of a database, extension, or the exact flavor of database that you want.
• Do-it-yourself on a VM. This might best be described as the full-ops option, where you take full responsibility for building your database, scaling it, managing reliability, setting up backups, and more. All of that can be a lot of work, but you have all the features and database flavors at your disposal.
• Run it on Kubernetes. Running a database on Kubernetes is closer to the full-ops option, but you do get some benefits in terms of the automation Kubernetes provides to keep the database application running. That said, it is important to remember that pods (the database application containers) are transient, so the likelihood of database application restarts or failovers is higher. Also, some of the more database-specific administrative tasks—backups, scaling, tuning, etc.—are different due to the added abstractions that come with containerization.

Tips for running your database on Kubernetes

When choosing to go down the Kubernetes route, think about what database you will be running, and how well it will work given the trade-offs previously discussed.

Since pods are mortal, the likelihood of failover events is higher than a traditionally hosted or fully managed database. It will be easier to run a database on Kubernetes if it includes concepts like sharding, failover elections and replication built into its DNA (for example, ElasticSearch, Cassandra, or MongoDB). Some open source projects provide custom resources and operators to help with managing the database.

Next, consider the function that database is performing in the context of your application and business. Databases that are storing more transient and caching layers are better fits for Kubernetes. Data layers of that type typically have more resilience built into the applications, making for a better overall experience.  

Finally, be sure you understand the replication modes available in the database. Asynchronous modes of replication leave room for data loss, because transactions might be committed to the primary database but not to the secondary database(s). So, be sure to understand whether you might incur data loss, and how much of that is acceptable in the context of your application.

After evaluating all of those considerations, you’ll end up with a decision tree looking something like this:

How to deploy a database on Kubernetes

Now, let’s dive into more details on how to deploy a database on Kubernetes using StatefulSets.

With a StatefulSet, your data can be stored on persistent volumes, decoupling the database application from the persistent storage, so when a pod (such as the database application) is recreated, all the data is still there.

Additionally, when a pod is recreated in a StatefulSet, it keeps the same name, so you have a consistent endpoint to connect to. Persistent data and consistent naming are two of the largest benefits of StatefulSets. You can check out the Kubernetes documentation for more details.

If you need to run a database that doesn’t perfectly fit the model of a Kubernetes-friendly database (such as MySQL or PostgreSQL), consider using Kubernetes Operators or projects that wrap those database with additional features. Operators will help you spin up those databases and perform database maintenance tasks like backups and replication. For MySQL in particular, take a look at the Oracle MySQL Operator and Crunchy Data for PostgreSQL. 

Operators use custom resources and controllers to expose application-specific operations through the Kubernetes API. For example, to perform a backup using Crunchy Data, simply execute pgo backup [cluster_name]. To add a Postgres replica, use pgo scale cluster [cluster_name].

There are some other projects out there that you might explore, such as Patroni for PostgreSQL. These projects use Operators, but go one step further. They’ve built many tools around their respective databases to aid their operation inside of Kubernetes. They may include additional features like sharding, leader election, and failover functionality needed to successfully deploy MySQL or PostgreSQL in Kubernetes.

While running a database in Kubernetes is gaining traction, it is still far from an exact science. There is a lot of work being done in this area, so keep an eye out as technologies and tools evolve toward making running databases in Kubernetes much more the norm. 

When you’re ready to get started, check out GCP Marketplace for easy-to-deploy SaaS, VM, and containerized database solutions and operators that can be deployed to GCP or Kubernetes clusters anywhere.

Oden Technologies is using the Internet of Things (IoT) to improve the factories of today. The giant network of “things” (including people) connected to each other via the Internet has the potential to reduce waste, increase efficiency, and improve safety across all walks of life. Oden is leading IoT innovation in manufacturing by combining wireless connectivity, big data, and cloud computing.

The use of data to improve manufacturing is practically as old as manufacturing itself. But the computerization of manufacturing has resulted in broad and rapid changes to the way data is collected and processed, as well as the sheer volume of data available.

Oden’s goal is to help manufacturers tap into this data to quickly identify process trends and even warning signs of machine breakdown. Such visibility can reveal opportunities to improve manufacturing and maintenance processes that reduce waste and increase profit margins.

Oden designs and develops data collection devices that can plug into almost any kind of machine and can wirelessly transmit data with minimal complexity and setup time.

Once devices are installed, the Oden technology platform processes data to give manufacturers cutting-edge analytics that are easy to comprehend. Analysis produced by the platform provides factory engineers with data points such as detailed root-cause analysis down to the second, factory-wide performance in real-time, and trend analysis.

Improving cloud delivery

Oden’s previous cloud platform performed satisfactorily, but the company evaluated alternatives in search of potential reductions in cost and complexity and increases in performance.

When evaluating Google Cloud Platform, Oden discovered it would require fewer virtual machine (VM) instances for equivalent performance, which would cut costs. Furthermore, Oden could gain more sophisticated data analytics and machine learning capabilities compared with its existing cloud provider.

Today, Oden runs its entire platform on Google Cloud Platform including Google Compute Engine, Google Cloud Pub/Sub, Google Cloud Bigtable, Google Stackdriver, and Google Kubernetes Engine.

“In order to serve our customers, we need a cloud platform that can scale reliably while keeping costs low, perform under heavy loads, and consistently deliver sophisticated features such as machine learning,” says Willem Sundblad, CEO and Founder at Oden Technologies. “Google Cloud Platform is way ahead in all of these areas compared to our previous cloud provider.”

Capturing tens of millions of metrics a day

Using Google Cloud Platform, Oden can help an average factory capture and store approximately 10 million metrics on a single manufacturing line every day.

Metrics can include extremely granular detail, such as the amount of electricity going to machines, the amount of raw material consumed, and the volume of material produced. Sensors can also capture and transmit environmental information such as temperature, humidity, and dew point so that manufacturers can identify weather-related and seasonal impacts on production.

The updated Oden Cloud Platform uses Kubernetes Engine—powered by the open source Kubernetes system—to run application program interfaces (APIs) that capture data from Oden’s wireless devices on the factory floor.

Google Cloud Pub/Sub then sends the data in real time to Google Cloud Bigtable, where data is processed using Oden’s proprietary analytics tools. Google Stackdriver supports Google Cloud Platform monitoring, logging, and diagnostics, which help Oden deliver its cloud platform with confidence.

Oden Technologies builds dashboards powered by Kubernetes Engine, which pull analyzed data from Google Cloud Bigtable. The dashboards provide customers with real-time visibility into their manufacturing lines. Oden Factory Cloud dashboards allow customers to delve deeper into their data to fine-tune production processes or discover the root causes of production issues.

With the previous cloud provider, Oden required 80 VM instances to run the dashboards. With Google Cloud Platform that number has been cut to 45, which dramatically reduces costs and complexity.

“We migrated from our previous cloud provider to Google Cloud Platform in just one month,” says Willem. “Further, our storage and data analytics costs have decreased by 30%. Cost savings like these allow us to protect customers from rising expenses, keeping us focused on bringing the best products possible to market.”

Faster data access; more efficient factories

With Google Cloud Platform, Oden can now deliver a complete factory analytics picture to manufacturers. In environments where thousands of variables affect the bottom line, businesses can now automatically and perpetually record machine and performance measurement. Oden Factory Cloud gives customers access to comprehensive data insights and can eliminate reliance on onsite infrastructure investments to run their own analytics.

Because manufacturers have access to live data and can analyze production data quickly, they can troubleshoot and resolve problems in minutes rather than months. Such information helps improve product quality, minimize unplanned downtime, cut costs, and improve profitability.

“With the help of Google Cloud Platform, we are helping our customers to be data-driven, which wasn’t possible before,” adds Willem. “They now understand that data is their most important asset. That allows them to be more innovative and continually improve their production processes.”

Editor’s note: Renault, the French automaker, embarked on a wholesale migration of its information systems—moving 70 applications to Google Cloud. Here’s how they migrated from Oracle databases to Cloud SQL for PostgreSQL.

The Renault Group, known for its iconic French cars has grown to include four complementary brands, and sold nearly 3 million vehicles in 2020. Following our company-wide strategic plan, “Renaulution,” we’ve shifted our focus over the past year from a car company integrating tech, to a tech company integrating cars that will develop software for our business. For the information systems group, that meant modernizing our entire portfolio and migrating 70 in-house applications (our quality and customer information systems) to Google Cloud. It was an ambitious project, but it’s paid off. In two years we migrated our Quality and Customer Satisfaction information systems applications, optimized our code, and cut costs thanks to managed database services. Compared to our on-premises infrastructure, using Google Cloud services and open-source technologies comes to roughly one dollar per user per year, which is significantly cheaper.

An ambitious journey to Google Cloud

We began our cloud journey in 2016 with digital projects integrating a new way of working and new technologies. These new technologies included those for agility at scale, data capabilities and CI/CD toolchain. Google Cloud stood out as the clear choice for its data capabilities. Not only are we using BigQuery and Dataflow to improve scaling and costs, but we are also now using fully managed database services like Cloud SQL for PostgreSQL. Data is a key asset for a modern car maker because it connects the car maker to the user, allows car makers to better understand usage and better informs what decisions we should make about our products and services. After we migrated our data lake to Google Cloud, it was a natural next step to move our front-end applications to Google Cloud so they would be easier to maintain and we could benefit from faster response times. This project was no small undertaking. For those 70 in-house applications (e.g. vehicle quality evaluation, statistical process control in plants, product issue management, survey analysis), for our information systems landscape, we had a range of technologies—including Oracle, MySQL, Java, IBM MQ, and CFT—with some applications created 20 years ago.

Champions spearhead each migration

Before we started the migration, we did a global analysis of the landscape to understand each application and its complexity. Then we planned a progressive approach, focusing on the smallest applications first such as those with a limited number of screens or with simple SQL queries, and saving the largest for last. Initially we used some automatic tools for the migration, but we learned very quickly nothing can replace the development team’s institutional knowledge. They served as our migration champions.

The apps go marching one by one

When we migrated our first few Oracle databases to Cloud SQL for PostgreSQL we tracked our learnings in an internal wiki to share common SQL patterns, which helped us speed up the process. For some applications, we simplified the architecture and took the opportunity to analyze and optimize SQL queries during the rework. We also used monitoring tools like Dynatrace and JavaMelody to ensure we improved the user experience.

The approach we developed was very successful—where database migration was initially seen as insurmountable, the entire migration project was completed in two years.

With on-premises applications it was hard for our developers to separate code performance from infrastructure limitations. So as part of our migration to Google Cloud, we optimized our applications with monitoring services. With these insights our team has more control over resources, which has reduced our maintenance and operations activity and resulted in faster, more stable applications. Plus, migrating to Cloud SQL has made it much easier for us to change our infrastructure as needed, add more power when necessary or even reduce our infrastructure size.

A new regime on Cloud SQL

Now that we’re running on Cloud SQL, we’ve improved performance even on large databases with many connected users. Thanks to built-in tools in the Google Cloud environment, we can now easily understand performance issues and quickly solve them. For example, we were able to reduce the duration of a heavy batch processing by a factor of three from nine to three hours. And we don’t have to wait for the installation of a new server, so our team can move faster. Beyond speed, we’ve also been able to cut costs. We optimized our code based on insights from monitoring tools, which not only enabled a more responsive application for the user, but it also reduced our costs because we’re not overprovisioned.

Learn more about the Renault Group and try out Cloud SQL today.

In recent years, the grocery industry has had to shift to facilitate a wider variety of checkout journeys for customers. This has meant ensuring a richer transaction mix, including mobile shopping, online shopping, in-store checkout, cashierless checkout or any combination thereof like buy online, pickup in store (BOPIS).  

What’s more, in the past year and a half alone grocers have had to enable consumers new ways to shop for essentials. This has included needing to rapidly integrate or build on-demand delivery apps, offer curbside pickup with near-instant fulfillment as well as support touchless and cashless checkout experiences. Searches on Google Maps for retailers in the US with curbside pickup options have increased by 9000% since March 2020, and we believe these trends from 2020 will continue to define the future of grocery shopping.

The future of grocery will require agility and openness

Firstly, the need to rapidly adapt to changing consumer habits will be the new normal. Grocers will increasingly look to digitally transform legacy retail systems and modernize point of sale (POS) platforms to deliver and scale omnichannel experiences as quickly as possible. This necessitates a more agile and open architectural approach to technology – one built on microservices and leverages APIs so that new applications and experiences can be built, integrated and delivered faster.

Automation and data-driven retailing will be table stakes

In order for retailers to blend what they’re offering in the store with digital experiences more efficiently, they will also need to automate more. For example, with automation and business intelligence, grocers can take labor that might have been tied up with tender operations and checkout and redistribute those resources to restocking shelves, curbside pick-up or improving customer experiences. 

Automation and access to real-time in-store inventory & supply chain data can also help grocers avoid the supply chain challenges seen in the early days of COVID-19. Grocers will need to find ways to leverage automation to ingest, organize, and analyze data from physical store networks, digital channels, distribution centers to better forecast demand and manage future fluctuations.

How NCR and Google Cloud are helping grocers adapt to disruption with operational agility

Helping grocers improve operational agility to address changing consumer shopping habits and to thrive during times of disruption is something that NCR and Google Cloud have teamed up to do. NCR has over 135 years of experience in retail, having invented the cash register and are continuing to help grocers innovate. NCR Emerald builds upon the company’s leadership in POS software and has turned it into a unified platform that helps grocers operate the entire store from front to back. The solution supports cashier-led checkout, self-checkout, integrated payments, merchandising, and enables regional managers and corporate employees access to the analytics and tools needed to optimize loyalty programs and promotions.

NCR has invested in a comprehensive, agile, and API-led retail architecture that lets grocers continually innovate and design new experiences as customers and the industry evolve. By running Emerald on Google Cloud, NCR can offer the solution on a subscription basis, helping grocers lower upfront capital expenditures and ensuring scalability. What’s more, NCR can tap into Google Cloud’s strength in data, analytics, and openness to deliver three key imperatives. Let’s take a look at each of these below.

Run the way grocers need to while leveraging Google Cloud as a single source of logic

Traditionally the POS system lived in the store. If disaster strikes, people still need access to food and essentials so the grocery store still needs to operate. It hardly gets more mission-critical than that. NCR Emerald is built on microservices, leveraging Kubernetes for front-of-house compute, and VMs (See graphic 1 below). This makes it easy to support lightweight clients accessible by store employees via any range of mobile devices, computer terminals, self-service kiosks, peripheral devices like receipt printers as well as legacy applications.

What’s unique is that because Emerald runs on Google Cloud, it supports all those in-store and digital touchpoints mentioned above, but also allows grocers to run lean. Emerald leverages Google Cloud as a single source of truth and operates a lot of what it does out of logic. Every sales transaction coming from every channel, including e-commerce, can be logged via NCR’s Hosted Service and centralized in BigQuery and Bigtable as a transaction data master. This enables the grocer to manage any transactional use case very consistently, whether it e supporting customers who want to purchase in one store and return in another, offering digital receipts or the ability to exchange online purchases in store. Emerald on Google Cloud can help retailers extend capabilities through the power of the cloud but not need to live exclusively in the cloud. In other words, the solution allows grocers the ability to run the way they need to.

Enable data-driven and real-time decision making for grocers

Store managers, regional managers, category managers, and others all require different cuts of the data to do their jobs effectively. However, data silos persist and how data is formatted and arranged can still remain pretty static. Therefore allowing users with different roles the ability to view and analyze that data quickly and in different ways continues to be a challenge. 

As mentioned above, Emerald leverages Google Cloud data management solutions as the central repository for transactional, behavioral, and merchandising data. Every transaction from every store and every channel can be stored via NCR Hosted Service on BigQuery and Bigtable. NCR Analytics then harnesses the advanced analytical and data visualization capabilities of Looker to help grocers get a consolidated view of their business across all channels and then allow employees to slice and dice the data they way they need to. NCR Analytics also leverages the power of Google Cloud AI and machine learning to add another level of intelligence to the retailer’s data. For example, store managers can visualize how well they’re using their real estate and see how productive lanes 1-3 are compared with 7-10 or compare self-service versus manned lanes. By mapping to the retailer’s own catalog, they can also break down category-level performance and trends.

NCR Analytics takes advantage of Google Cloud’s data pipeline to reduce processing time, with scaling and resource management provided out of the box. By letting the cloud store and process the data, NCR is providing the ability for retailers to analyze their data in near real-time across all platforms – a real game changer in the grocery business.

Open APIs let grocers continually enrich the retail experience

Finally, Emerald is built on an API-first architecture managed through Apigee. It uses the power of Apigee as an open API platform to expose how Emerald can work with other NCR applications like loyalty and promotions, and third party applications like mobile ordering and order delivery to enrich the grocery experience for employees and customers. Every API that Emerald uses is available on Apigee, allowing them to share code samples and giving developers the ability to run scripts. This approach can allow retailers the ability to innovate in a fraction of the time and cost, speeding up 3rd party integrations up front and as businesses grow. 

Take, for example, Northgate Market, a chain of 40 stores in California, that were able to transform its digital operations and enable experiences that set it apart from competitors – quickly and simply with Emerald. It took less than 6 months to go from contract to live deployment in the first store. Since then, Northgate Market has been able to extend their intelligence by leveraging the power of Looker and NCR Analytics.

Learn more about how NCR has been able to leverage an open, cloud-enabled architecture to help customers innovate across the retail, hospitality, and banking industries on the webinar “Role of APIs in Digital Transformation”. You can also learn more about how Northgate uses e-commerce to transform customer experience and gain consumer insights.

97 zettabytes was the estimated volume of data generated worldwide in 2022¹. This sort of explosion in data volume is happening in every enterprise. Now imagine being able to access all this data you own from anywhere, at any time, analyze it, and leverage its insights to innovate your products and services? One of the biggest barriers to fulfilling this vision is the complexity inherent in dealing with data trapped across silos in an enterprise. Google Data Cloud offers a unified, open, and intelligent platform for innovation, allowing you to integrate your data on a common architectural platform. Across industries, organizations are able to reimagine their data possibilities in entirely new ways and quickly build applications that delight their customers.

Siloed data: The barrier to speed and innovation

Digital technologies ranging from transaction processing to analytics and AI/ML use data to help enterprises understand their customers better. And the pace of innovation has naturally accelerated as organizations learn, adapt, and race to build the next generation of applications and services to compete for customers and meet their needs. At Next 2022, we made a prediction that the barriers between transactional and analytical workloads will mostly disappear.

Traditionally, data architectures have separated transactional and analytical systems—and that’s for good reason. Transactional databases are optimized for fast reads and writes, and analytical databases are optimized for analyzing and aggregating large data sets. This has siloed enterprise data systems, leaving many IT teams struggling to piece together solutions. The result has been time consuming, expensive, and complicated fixes to support intelligent, data-driven applications. 

But, with the introduction of new technologies, a more time-efficient and cost-effective approach is possible. Customers now expect to see personalized recommendations and tailored experiences from their applications. With hybrid systems that support both transactional and analytical processing on the same data, without impacting performance, these systems now work together to generate timely, actionable insights that can be used to create better experiences and accelerate business outcomes.   

According to a 2022 research paper from IDC, unifying data across silos via a data cloud is the foundational capability enterprises need to gain new insights on rapidly changing conditions and to enable operational intelligence. The modern data cloud provides unified, connected, scalable, secure, extensible, and open data, analytics, and AI/ML services. In this platform, everything is connected to everything else.

Why reducing data barriers delivers more value 

The primary benefit of a unified data cloud is that it provides an intuitive and timely way to represent data and allows easy access to related data points. By unifying their data, enterprises are able to: 

1. Ingest data faster – for operational intelligence
2. Unify data across silos – for new insights
3. Share data with partners – for collaborative problem solving
4. Change forecasting models – to understand and prepare for shifting markets
5. Iterate with decision-making scenarios – to ensure agile responses
6. Train models on historical data – to build smarter applications

As generative AI applications akin to Bard, an early experiment by Google, become available in the workplace, it will be more important than ever for organizations to have a unified data landscape to holistically train and validate their proprietary large language models. 

With these benefits enterprises can accelerate their digital transformation in order to thrive in our increasingly complex digital environment. A survey of more than 800 IT leaders indicated that using a data cloud enabled them to significantly improve employee productivity, operational efficiency, innovation, and customer experience, among others. 

Build modern apps with a unified and integrated data cloud

Google Cloud technologies and capabilities reduce the friction between transactional and analytical workloads and make it easier for developers to build applications, and to glean real-time insights. Here are a few examples. 

1. AlloyDB for PostgreSQL, a fully managed PostgreSQL-compatible database, and AlloyDB Omni, the recently launched downloadable edition of AlloyDB, can analyze transactional data in real time. AlloyDB is more than four times faster for transactional workloads and up to a 100 times faster for analytical queries compared to standard PostgreSQL, according to our performance tests. This kind of performance makes AlloyDB the ideal database for hybrid transactional and analytical processing (HTAP) workloads.
2. Datastream for BigQuery, a serverless change data capture and replication service, provides simple and easy real time data replication from transactional databases like AlloyDB, PostgreSQL, MySQL and Oracle directly into BigQuery, Google Cloud’s enterprise data warehouse.
3. And, query federation with Cloud Spanner, Cloud SQL, and Cloud Bigtable, make data available right from the BigQuery console allowing customers to analyze data in real-time in transactional databases. 

Speed up deployments and lower costs

By reducing data barriers, we’re taking a fundamentally different approach that allows organizations to be more innovative, efficient, and customer-focused by providing: 

• Built-in industry leading AI and ML that helps organizations not only build improved insights, but also automate core business processes and enable deep ML-driven product innovation. 
• Best-in-class flexibility. Integration with open source standards and APIs ensures portability and extensibility to prevent lock-in. Plus, choice of deployment options means easy interoperability with existing solutions and investments.  
• The most unified data platform with the ability to manage every stage of the data lifecycle, from running operational databases to managing analytics applications across data warehouses and lakes to rich data-driven experiences. 
• Fully managed database services that free up DBA/DevOps time to focus on high-value work that is more profitable to the business. Organizations that switch from self-managed databases eliminate manual work, realize significant cost savings, reduce risk from security breaches and downtime and increase productivity and innovation. In an IDC survey, for example, Cloud SQL customers achieve an average three-year ROI of 246% because of the value and efficiencies this fully managed service delivers. 

Google Data Cloud improves the efficiency and productivity of your teams, resulting in increased innovation across your organization. This is the unified, open approach to data-driven transformation bringing unmatched speed, scale, security, and with AI built in. 

In case you missed it: Check out our Data Cloud and AI Summit that happened on March 29th, to learn more about the latest innovations across databases, data analytics, BI and AI. In addition, learn more about the value of managed database services like Cloud SQL in this IDC study.

^{1. How Big is Big? – Statista}