Are Open Banking Regulations an Effective Entry into the API Economy?
4289
Of your peers have already read this article.
3:30 Minutes
The most insightful time you'll spend today!
With stated goals of increasing competition, innovation, and financial inclusion, bank regulators across the world are mandating that banks open their systems, enabling consumers to share their financial data with third parties.
One touted benefit of this sharing is that it may create digital banking ecosystems that offer consumers more services than ever, provide banking information and capabilities in more useful and convenient contexts, expand the market reach of ecosystem participants, and boost financial participation among the unbanked and underbanked.
These benefits involve requiring that banks produce application programming interfaces (APIs) to make data and functionality easy to share with partners in a standardized way and to give consumers control over the services with which their data is shared. Because APIs enable developers to leverage and reuse software for new services and digital experiences, including by combining APIs from multiple providers, tech pundits and commentators often refer to the “API economy” — that is, to digital ecosystems in which companies symbiotically share and combine their software to create richer offerings, to complement their proprietary strengths with offerings from other organizations, to share innovation across enterprises, and to expand into new sectors.
Rather than being able to access and act on financial data only through specific channels, for example, consumers in an Open Banking world would theoretically be able to use their money across a constantly-expanding array of apps and digital experiences. Likewise, rather than being confined to one bank’s specific digital services, consumers would be able to opt into a range of services, such as better loan matching or debt reduction advice, to help them do more with their money. Banks, meanwhile, would not have to create all aspects of digital experiences themselves but could rely on external partners, which they can add at unprecedented scale via APIs, to shoulder some of the burden of attracting and creating value for customers.
The envisioned disruption is sweeping, but key questions for bank leaders and bank investors remain, notably the extent to which the Open Banking movement will exert significant, durable impacts on market dynamics — and the extent to which Open Banking compliance constitutes an effective market entry into digital ecosystems and the celebrated benefits of the API economy.
Put simply, is complying with Open Banking regulations adequate to advance a bank’s digital strategy?
Building compliant APIs vs. entering the API economy
Individual regulators are fundamentally constrained in their ability to give banks detailed instructions on how to behave in digital ecosystems. Mandatory regulations can be originally conceived with only one or two business models in mind, not the many thousands of business models that could be partially or fully enabled by the API economy. Detailed rules and specifications may threaten the functionality of services, as regulatory interventions that give highly specific guidance may not be system- and business model-agnostic.
In terms of policy, the willingness of regulators to force banks to adopt APIs is highly significant, but because these regulatory interventions do not offer a meaningful and durable market entry roadmap for banks to follow, the regulations may be an initial catalyst to prompt financial market evolution rather than a natural and enduring end-state for business activity.
Aside from the constraints at the level of the individual bank regulator, there is limited consistency among Open Banking regulations across the globe. Europe’s PSD2 was the regulatory intervention that kicked off this global wave, but there are small but significant variations in the ripple of regulatory actions around the world.
In Australia, Open Banking is part of the Consumer Data Right, an initiative to give customers the right to access their data in a machine-readable form — a right the country does not confine to banking. In Singapore, the Monetary Authority of Singapore is pushing for a lightweight regulatory framework regime. Japan’s Amended Banking Act introduced a registration system for third party providers. Korea’s Financial Services Commission has launched a Fintech Open Platform. Mexico’s recent law to regulate financial technology institutions lays groundwork for an Open Banking regime. The Central Bank of Brazil is aiming to implement the relevant regulatory reforms by the end of 2019. For the largest, internationally diversified banks, these regional differences further complicate any effort to regard mere compliance with these interventions as an effective and coherent market entry strategy.
Despite these complexities and ambiguities, I’ve observed in my work consulting with financial institutions that some banks nevertheless treat Open Banking compliance projects as a means of market entry into the API economy. This mindset could be a costly mistake and may leave these banks at a significant competitive disadvantage as we continue to accelerate into the era of digital ecosystems. In addition to compliance, banks should view a commercial entry into a foreign country as a useful reference for entering the API economy.
Entering the API economy is like entering a foreign market
Banks executing a commercial entry into a foreign country will encounter cultural differences, whether in the form of language, ethnicity, religion, social networks, values or norms. When these cultural differences are large, market entry into a foreign market is more difficult.
The practical implications of administering new business activity in a foreign country also impact the level of difficulty. The physical distance between the home country and the foreign country has to be managed. Border hardness, time zones, and climate also impact the administrative challenges posed by the foreign country. Additionally, the market landscape is foreign. New countries can significantly differ in their natural, financial, and human resources. Staff sent to build up a new division in a new country will have to cope with different levels of market Infrastructure, information, and knowledge. Finally, the economics of entering a foreign country can be heavily influenced by historical and political factors such as a shared colonial history, common memberships of trading blocs, and shared currency zones.
These patterns of differences and similarities can likewise be observed when a bank tries to enter the API economy.
Like entry into a new territory, entry into the API economy may pose sharp cultural and operational differences for banks to grapple with. In this world of APIs and software ecosystems, an API provider’s commercial aim is to make third-parties the dominant force for innovation — that is, to securely share data and services with external contributors who build new connected experiences that generate value for the provider, much as ridesharing companies have generated value by leveraging APIs such as Google Maps. Enterprises focused on ecosystem development market their APIs as products for developers so that new innovation can emerge organically, without necessarily being pre-planned. Investment decisions in the API economy are driven by a desire to help preferred ecosystems to evolve fastest. All of this may be a very sharp change in culture for many banks that historically have sought to be the dominant innovator shaping customer experiences.
The established approach to innovation in banks is highly deliberate, aiming to match bank financial products to specific customer needs and to beat the financial product offerings of peer banks. Compared to modern digital ecosystems, these legacy banks’ resources for, scope of, and receptivity to innovation may be considerably constrained. Banks that actively treat the culture of the API economy as very foreign to their traditional corporate culture have a far greater chance of acknowledging these challenges and making a successful market entry into the API economy.
Many banks may also face challenges transitioning to ecosystem administration models. In traditional banking models, the C-suite commands and controls the bank staff that distributes products. Typically, very few external business partners are involved, and those that are have generally been deliberately, if not laboriously, selected. In contrast, the API economy requires the orchestration of very large numbers of third parties that add value to a bank’s innovation and distribution capabilities. Literally thousands of partners may access a bank’s APIs to build services atop banking data, extend a bank’s functionality or insert the bank’s APIs into new business contexts. In many ways, the whole point is that partnerships don’t have to occur in slow-moving, methodically planned formal partnerships but rather can be achieved at Internet scale while preserving control over and visibility into customer data. This shift in strategy is no small departure for many banks — so again, thinking of the endeavor as entry into a foreign market, rather than something that can be jumpstarted via regulations, is wise.
Crucially, the market landscape in the API economy is also very different from what legacy financial institutions are used to. Traditionally, banks have segmented the market into very large market segments (e.g. consumer banking, corporate banking, private banking etc.). In sharp contrast, the API economy sees partners working together to serve market micro-segments that would not be reachable and profitable by each individual enterprise working in isolation. Simply put, addressing all customer needs, from mainstream use cases to niche applications, is beyond the capabilities of any single enterprise and can generally only be achieved through ecosystems and partnerships. Because partners working together in the API economy seek to serve the customer through the customer’s preferred digital interface (which may be different than bank’s preferred digital interface), ecosystem partnerships represent a massive shift in marketing management, with major implications for a bank’s business architecture, technical architecture, governance, and risk management.
Additionally, in the API economy, pricing decisions are generally driven by a desire for ecosystem growth. Partners work together to extend the customization and value that customers experience when consuming services. By design, the intellectual property in an ecosystem becomes dispersed. Banks have traditionally sought to protect the exclusivity of their intellectual property, and they have often sought economic returns via cost-plus pricing and minimal customization of financial products. Just as with other aspects of the API economy, these ecosystem dynamics and economic relationships typically fall outside a bank’s status quo operations and strategies — more akin to entering a foreign market than to simply satisfying regulatory requirements.
The C-suite should be involved
To approach entry into the API economy as they would approach market entry into a foreign country, banks should consider focusing on a small number of products or segments. They should seek complementary partners who can help them adapt their skills to the API economy and extend the reach of their core differentiating strengths. Just like entry into a foreign market requires combining elements of the home country business model with new local elements, banks will have to transform to increase the effectiveness of their ecosystem participation.
Because an Open Banking compliance project involves regulators specifying the scope and schedule for mandatory APIs, the C-suite may feel relegated to the role of budget provider, which may tempt executives to delegate the sponsorship and steering of the project. Such delegation is not an option for a foreign market entry, where all decisions on scope, cost and timetable are strategic and entirely in the hands of the bank itself — and the C-suite should not consider it an option for entry into the API economy. Bank leaders need to be involved, as the organization likely will not be able to make the necessary strategic and operational transformations if the vision is not defined and driven from the top.
What to Look for from Cloud CISO Perspective
6648
Of your peers have already read this article.
2:30 Minutes
The most insightful time you'll spend today!
May is a big month for the security industry. It’s been over a year since we gathered for RSA in San Francisco for one of 2020’s last major in-person events. While we likely won’t be together in person this year, it’s an important time for the security community to come together and reflect on many accomplishments, and to consider the challenges still ahead of us. As the world focuses on security incidents and all the risks that still need resolving, it is important to stand back, on occasion, and also note that immense progress has been made by large numbers of small, medium and large enterprises to protect themselves and their customers against increased threats. What is also amazing is to see organizations do this while accelerating their digital transformations, supporting and protecting customers and managing ongoing remote working challenges. We are privileged to play our part in supporting those great teams.
It’s also been a busy month for us here at Google Cloud since our inaugural CISO perspectives blog post in April. Today, I’ll recap our cloud security and industry highlights, a sneak peak of what’s ahead from Google at RSA and more.
Thoughts from around the industry
- Risk Governance of Digital Transformation in the Cloud – In our latest Office of the CISO whitepaper, we shared guidance on both the challenges and opportunities of cloud transformation for Chief Risk Officers, Chief Compliance Officers, Heads of Internal Audit and their teams. A misconception we sometimes see among these executives is that moving to the cloud creates more risk to manage. Having held these leadership positions in previous roles, I believe that the cloud is as much a means of managing security, resilience and other risks as it is a risk in its own right. The whitepaper dives deep into considerations for each of these leadership functions as their organization embarks on a digital transformation journey.
- The importance of meeting global compliance requirements – Compliance is critical for building trust with customers in regulated industries, especially the public sector. It is worth remembering that in any critical industry, where there can be material impact from incidents, strong industry practices and standards to protect customers are vital (I wrote about this last summer). At Google Cloud, we’re regularly adding new compliance and security certifications to meet our customers’ needs globally. Recently, we expanded our list of FedRAMP High-certified products to include Cloud DNS, and helped our customers in the Asia-Pacific region address various compliance requirements to meet new government regulations for security and data protections. Google Cloud was also the only cloud service provider to complete an annual pooled audit with the Collaborative Cloud Audit Group (CCAG), which is a syndicate of 39 leading European financial institutions and insurance companies who depend on cloud infrastructure and technologies to deliver innovative solutions and experiences for their customers. Having spent most of my career in the financial services industry, I know firsthand the importance of managing risk assessments for outsourced vendors to provide the necessary assurances customers need from their cloud providers.
RSA 2021
We have a great lineup of speaking sessions and keynotes from Googlers at RSA this year. Below are the highlights you don’t want to miss:
- I’ll be doing a session on May 20 about supply chain resilience, where a panel of experts will dive into how we can adjust risk and security initiatives to handle the next “punch to the supply chain.” Additionally, on May 18 I’ll join many of my esteemed CISO leaders from various industries and governments for a keynote discussion on our top security insights, lessons learned and best practices for how we move forward as an industry to address the next wave of challenges.
- Google’s Senior Director of Information Security Heather Adkins will deliver a session on how to build secure and reliable systems at scale, which will cover principles from Google’s Site Reliability Engineering book with the same title (available for free download here). I’m most looking forward to Heather’s advice for how we as an industry can reshape our security thinking, based on modern architectures and technologies that can help organizations design scalable and reliable systems that are fundamentally secure.
- Nelly Porter, Senior Product Manager at Google Cloud Security, will participate in a panel discussion with security experts on the importance of Confidential Computing technology, how it’s changing the security landscape and where it’s headed. Google Cloud has made great progress in delivering a Confidential Computing portfolio for our customers in regulated industries over the past year, and we’re excited for new milestones in 2021.
Google cloud security highlights
- Infrastructure and SRE spotlight – Before I joined Google Cloud, I always admired the infrastructure and benefits this organization delivers that are uniquely Google – from the subsea cable innovations to SRE inventions and principles. Security and resiliency are baked into every layer of our infrastructure. Many of the Googlers who build and support our platform have sat in the same seat as our customers, so they understand those needs intimately. Over the last few months it’s been amazing to watch our technical infrastructure team grow, and the direct reliability, operational resilience and security benefits that team brings to our customers. For example, we’ve opened a new region in Poland, announced the first subsea cable that will directly connect the U.S. to Singapore with fiber pairs over an express route, and released an SRE book focused on how organizations can complete a successful cloud migration.
- New security foundations blueprint guide – As part of our mission to deliver the industry’s most trusted cloud, we strive to operate in a shared-fate model for risk management in conjunction with our customers. This includes sharing opinionated step-by-step guidance with key decision points and focus areas for how our customers deploy workloads in Google Cloud. This is why we’ve updated our Google Cloud security foundations guide and corresponding Terraform blueprint scripts. These blueprints are tremendously helpful to many stakeholders within an enterprise, like a CISO that needs to understand our key principles for cloud security, or a C-Suite business leader that needs to quickly identify the skills their teams need to meet an organization’s security, risk, and compliance needs on Google Cloud.
When we think about the types of features to build into products, we have many principles we follow. But the two that I keep coming back to as crucial are:
- The need for secure products not just security products. All products should have security built in and while we do build great security products our security and other teams remain focused on constantly enhancing the base levels of security and the security features in all our products.
- Defense in Depth. We don’t just focus on defense in depth from attacks – for ourselves and our customers. We also prioritize defense in depth from configuration errors or other hazards.
As you see below in some of the highlights of new features and products, these represent our commitment to secure products and all forms of defense in depth.
- Workload identity federation – Service account keys are powerful credentials, and can represent a security risk if they are not managed correctly. A safer approach is to use workload identity federation, using IAM to grant external identities IAM roles, including the ability to impersonate service accounts. This lets you access resources directly and eliminates the maintenance and security burden associated with service account keys. We also offered related overall guidance on the best way to use and authenticate service accounts on Google Cloud.
- VPC-SC Directional Policies – With VPC Service Controls (VPC-SC), admins can define a security perimeter around Google-managed services to control communication to and between those services. Using VPC-SC, you can isolate your production GCP resources from unauthorized VPC networks or the internet. But what if you need to transfer data between isolated environments that you’ve set up? VPC-SC directional policies is a new secure data exchange feature that allows you to configure efficient, private, and secure data exchange between isolated environments.
- Anthos service mesh supports VMs as well as clusters – Most enterprise compute resources are still in VMs and many will remain there for a long time to come. In Anthos 1.7, your VM-based workloads can now take advantage of the same mesh functionality as your container-based workloads.
- Cloud Spanner CMEK and Access Approvals – Cloud Spanner is Google Cloud’s fully managed relational database that offers unlimited scale, high performance, strong consistency across regions and high availability. Spanner now supports customer-managed encryption keys (CMEK) and Access Approval, Google Cloud’s industry-leading controls to require approval before access to your content by Google support and engineering teams.
- External Key Manager enhancements – In early 2020 we launched Cloud External Key Manager (Cloud EKM), the industry’s leading Hold-Your-Own-Key (HYOK) product. Using Cloud EKM, the keys used to protect your data stored and processed in Google Cloud are completely hosted and managed outside of Google Cloud infrastructure. Cloud EKM initially launched with support for BigQuery and GCE/PD; we expanded support for Cloud SQL, GKE, Dataflow Shuffle, and Secret Manager, with CMEK support currently in beta. We also provided in-depth documentation on the functionality, architecture and use cases for Cloud EKM in a new whitepaper.
- Web App and API Protection solution – Web applications and public APIs are increasingly important to how organizations interface with their customers and partners, and we’ve seen increased investment in tools to protect these resources from fraud and abuse. Google Cloud’s new Web App and API protection solution is based on the same technology Google uses to protect its public-facing services against web application exploits, DDoS attacks, fraudulent bot activity, and API targeted threats. It provides protection across clouds and on-premises environments.
- Threat Intel for Chronicle – Most threat intelligence feeds require security teams to do the implementation and legwork. With our new Threat Intel for Chronicle offering, however, our intelligence insights are applied automatically across your security telemetry to present unique observations within your environment. Threat Intel for Chronicle is exclusively curated for enterprise customers by Uppercase, Google Cloud’s intelligence research and applications team to provide our perspective on threats across the internet and surface them as relevant alerts.
That wraps up another month of thoughts and highlights. If you’d like to have this Cloud CISO Perspectives post delivered every month to your inbox, click here to sign-up, and we’ll see you in June!
Quick Recap on Google Cloud: Latest News, Launches, Updates, Events and More
12943
Of your peers have already read this article.
2:00 Minutes
The most insightful time you'll spend today!
Want to know the latest from Google Cloud? Find it here in one handy location. Check back regularly for our newest updates, announcements, resources, events, learning opportunities, and more.
Tip: Not sure where to find what you’re looking for on the Google Cloud blog? Start here: Google Cloud blog 101: Full list of topics, links, and resources.
Week of May 24-May 28 2021
- Google Cloud for financial services: driving your transformation cloud journey–As we welcome the industry to our Financial Services Summit, we’re sharing more on how Google Cloud accelerates a financial organization’s digital transformation through app and infrastructure modernization, data democratization, people connections, and trusted transactions. Read more or watch the summit on demand.
- Introducing Datashare solution for financial services–We announced the general availability of Datashare for financial services, a new Google Cloud solution that brings together the entire capital markets ecosystem—data publishers and data consumers—to exchange market data securely and easily. Read more.
- Announcing Datastream in Preview–Datastream, a serverless change data capture (CDC) and replication service, allows enterprises to synchronize data across heterogeneous databases, storage systems, and applications reliably and with minimal latency to support real-time analytics, database replication, and event-driven architectures. Read more.
- Introducing Dataplex: An intelligent data fabric for analytics at scale–Dataplex provides a way to centrally manage, monitor, and govern your data across data lakes, data warehouses and data marts, and make this data securely accessible to a variety of analytics and data science tools. Read more.
- Announcing Dataflow Prime–Available in Preview in Q3 2021, Dataflow Prime is a new platform based on a serverless, no-ops, auto-tuning architecture built to bring unparalleled resource utilization and radical operational simplicity to big data processing. Dataflow Prime builds on Dataflow and brings new user benefits with innovations in resource utilization and distributed diagnostics. The new capabilities in Dataflow significantly reduce the time spent on infrastructure sizing and tuning tasks, as well as time spent diagnosing data freshness problems. Read more.
- Secure and scalable sharing for data and analytics with Analytics Hub–With Analytics Hub, available in Preview in Q3, organizations get a rich data ecosystem by publishing and subscribing to analytics-ready datasets; control and monitoring over how their data is being used; a self-service way to access valuable and trusted data assets; and an easy way to monetize their data assets without the overhead of building and managing the infrastructure. Read more.
- Cloud Spanner trims entry cost by 90%–Coming soon to Preview, granular instance sizing in Spanner lets organizations run workloads at as low as 1/10th the cost of regular instances, equating to approximately $65/month. Read more.
- Cloud Bigtable lifts SLA and adds new security features for regulated industries–Bigtable instances with a multi-cluster routing policy across 3 or more regions are now covered by a 99.999% monthly uptime percentage under the new SLA. In addition, new Data Access audit logs can help determine whether sensitive customer information has been accessed in the event of a security incident, and if so, when, and by whom. Read more.
- Build a no-code journaling app–In honor of Mental Health Awareness Month, Google Cloud’s no-code application development platform, AppSheet, demonstrates how you can build a journaling app complete with titles, time stamps, mood entries, and more. Learn how with this blog and video here.
- New features in Security Command Center—On May 24th, Security Command Center Premium launched the general availability of granular access controls at project- and folder-level and Center for Internet Security (CIS) 1.1 benchmarks for Google Cloud Platform Foundation. These new capabilities enable organizations to improve their security posture and efficiently manage risk for their Google Cloud environment. Learn more.
- Simplified API operations with AI–Google Cloud’s API management platform Apigee applies Google’s industry leading ML and AI to your API metadata. Understand how it works with anomaly detection here.
- This week: Data Cloud and Financial Services Summits–Our Google Cloud Summit series begins this week with the Data Cloud Summit on Wednesday May 26 (Global). At this half-day event, you’ll learn how leading companies like PayPal, Workday, Equifax, and many others are driving competitive differentiation using Google Cloud technologies to build their data clouds and transform data into value that drives innovation. The following day, Thursday May 27 (Global & EMEA) at the Financial Services Summit, discover how Google Cloud is helping financial institutions such as PayPal, Global Payments, HSBC, Credit Suisse, AXA Switzerland and more unlock new possibilities and accelerate business through innovation. Read more and explore the entire summit series.
- Announcing the Google for Games Developer Summit 2021 on July 12th-13th–With a surge of new gamers and an increase in time spent playing games in the last year, it’s more important than ever for game developers to delight and engage players. To help developers with this opportunity, the games teams at Google are back to announce the return of the Google for Games Developer Summit 2021 on July 12th-13th. Hear from experts across Google about new game solutions they’re building to make it easier for you to continue creating great games, connecting with players and scaling your business. Registration is free and open to all game developers. Register for the free online event at g.co/gamedevsummit to get more details in the coming weeks. We can’t wait to share our latest innovations with the developer community. Learn more.
How to Decide Whether to Run a Database on Kubernetes
5631
Of your peers have already read this article.
3:30 Minutes
The most insightful time you'll spend today!
Today, more and more applications are being deployed in containers on Kubernetes—so much so that we’ve heard Kubernetes called the Linux of the cloud.
Despite all that growth on the application layer, the data layer hasn’t gotten as much traction with containerization. That’s not surprising, since containerized workloads inherently have to be resilient to restarts, scale-out, virtualization, and other constraints. So handling things like state (the database), availability to other layers of the application, and redundancy for a database can have very specific requirements. That makes it challenging to run a database in a distributed environment.
However, the data layer is getting more attention, since many developers want to treat data infrastructure the same as application stacks.
Operators want to use the same tools for databases and applications, and get the same benefits as the application layer in the data layer: rapid spin-up and repeatability across environments. In this blog, we’ll explore when and what types of databases can be effectively run on Kubernetes.
Before we dive into the considerations for running a database on Kubernetes, let’s briefly review our options for running databases on Google Cloud Platform (GCP) and what they’re best used for.
- Fully managed databases. This includes Cloud Spanner, Cloud Bigtable and Cloud SQL, among others. This is the low-ops choice, since Google Cloud handles many of the maintenance tasks, like backups, patching and scaling. As a developer or operator, you don’t need to mess with them. You just create a database, build your app, and let Google Cloud scale it for you. This also means you might not have access to the exact version of a database, extension, or the exact flavor of database that you want.
- Do-it-yourself on a VM. This might best be described as the full-ops option, where you take full responsibility for building your database, scaling it, managing reliability, setting up backups, and more. All of that can be a lot of work, but you have all the features and database flavors at your disposal.
- Run it on Kubernetes. Running a database on Kubernetes is closer to the full-ops option, but you do get some benefits in terms of the automation Kubernetes provides to keep the database application running. That said, it is important to remember that pods (the database application containers) are transient, so the likelihood of database application restarts or failovers is higher. Also, some of the more database-specific administrative tasks—backups, scaling, tuning, etc.—are different due to the added abstractions that come with containerization.
Tips for running your database on Kubernetes
When choosing to go down the Kubernetes route, think about what database you will be running, and how well it will work given the trade-offs previously discussed.
Since pods are mortal, the likelihood of failover events is higher than a traditionally hosted or fully managed database. It will be easier to run a database on Kubernetes if it includes concepts like sharding, failover elections and replication built into its DNA (for example, ElasticSearch, Cassandra, or MongoDB). Some open source projects provide custom resources and operators to help with managing the database.
Next, consider the function that database is performing in the context of your application and business. Databases that are storing more transient and caching layers are better fits for Kubernetes. Data layers of that type typically have more resilience built into the applications, making for a better overall experience.
Finally, be sure you understand the replication modes available in the database. Asynchronous modes of replication leave room for data loss, because transactions might be committed to the primary database but not to the secondary database(s). So, be sure to understand whether you might incur data loss, and how much of that is acceptable in the context of your application.
After evaluating all of those considerations, you’ll end up with a decision tree looking something like this:
How to deploy a database on Kubernetes
Now, let’s dive into more details on how to deploy a database on Kubernetes using StatefulSets.
With a StatefulSet, your data can be stored on persistent volumes, decoupling the database application from the persistent storage, so when a pod (such as the database application) is recreated, all the data is still there.
Additionally, when a pod is recreated in a StatefulSet, it keeps the same name, so you have a consistent endpoint to connect to. Persistent data and consistent naming are two of the largest benefits of StatefulSets. You can check out the Kubernetes documentation for more details.
If you need to run a database that doesn’t perfectly fit the model of a Kubernetes-friendly database (such as MySQL or PostgreSQL), consider using Kubernetes Operators or projects that wrap those database with additional features. Operators will help you spin up those databases and perform database maintenance tasks like backups and replication. For MySQL in particular, take a look at the Oracle MySQL Operator and Crunchy Data for PostgreSQL.
Operators use custom resources and controllers to expose application-specific operations through the Kubernetes API. For example, to perform a backup using Crunchy Data, simply execute pgo backup [cluster_name]. To add a Postgres replica, use pgo scale cluster [cluster_name].
There are some other projects out there that you might explore, such as Patroni for PostgreSQL. These projects use Operators, but go one step further. They’ve built many tools around their respective databases to aid their operation inside of Kubernetes. They may include additional features like sharding, leader election, and failover functionality needed to successfully deploy MySQL or PostgreSQL in Kubernetes.
While running a database in Kubernetes is gaining traction, it is still far from an exact science. There is a lot of work being done in this area, so keep an eye out as technologies and tools evolve toward making running databases in Kubernetes much more the norm.
When you’re ready to get started, check out GCP Marketplace for easy-to-deploy SaaS, VM, and containerized database solutions and operators that can be deployed to GCP or Kubernetes clusters anywhere.
Develop for Compute Engine in your IDE with Cloud Code
2824
Of your peers have already read this article.
2:30 Minutes
The most insightful time you'll spend today!
When developing services with Compute Engine, our customizable compute service that lets you create and run virtual machines on Google’s infrastructure, you’ll likely find yourself frequently switching between your code editor, terminal, and the Google Cloud Console.
Cloud Code is a set of IDE plugins for popular IDEs like VS Code and IntelliJ that make it easier to develop applications that use Google Cloud services. And now, Cloud Code makes it easy to develop with Compute Engine by incorporating common workflows with your favorite IDE’s user interface.
Specifically, this new integration between Compute Engine and Cloud Code makes it easier to manage your commonly used virtual machines in the IDE, view details about them, connect to them over SSH, upload your application files to them, and view their logs.
Before you begin
Let’s demonstrate how the new integration works in Cloud Code for VS Code. Install Cloud Code for VS Code, and once installed, open its icon on the activity bar on the left and find “Compute Engine”:
Cloud Code for Jetbrains IDEs (such as IntelliJ) could be installed similarly, and you will find Compute Engine in the list of your IDE tool windows.
View your VMs
Cloud Code makes it easy to see all relevant VMs in your GCP project and view details needed to effectively work with the VM from the IDE. To start working with a Compute Engine VM in the IDE, navigate to Cloud Code’s new Compute Engine explorer. From there, you can see all the VMs in your current Cloud project. Clicking on a VM will display details such as machine type, boot image, IP address and more. You can also right click on a VM for a quick link to the Google Cloud Console where you can take additional action.
Connect to your VMs over SSH
Once you’ve found the VM you want to work with, Cloud Code makes it easy to connect to that VM over SSH. Again, find the VM you want to connect to in Cloud Code’s Compute Engine explorer, right click it, and select “Open SSH”. Cloud Code will then establish an SSH connection from your IDEs terminal into the VM. If there’s any difficulty establishing a connection, Cloud Code can run a troubleshooting diagnostic to help resolve the issue.
Many organizations maintain VMs that don’t have a public IP address, making it difficult to establish an SSH connection to them. For those VMs that use Identity-Aware Proxy, Cloud Code can still securely connect to them over SSH, even without a public IP address.
Upload files to your VMs
You might want to try a debug version of your application, run a script, or try a new code in an environment identical to production, in this case on a development VM instance which might not have access to full source code or is not a part of your CI/CD pipeline. Cloud Code provides an easy way to upload your code files into a VM instance.
Find the VM you want to connect to in Cloud Code’s Compute Engine explorer, right click it, and select “Upload File via SCP”. Choose a file from your local system and Cloud Code will upload it to a VM instance using SCP. Once upload completes, Cloud Code offers to open a new SSH connection to access the files and work with them on a remote VM instance. Again, if there’s any difficulty establishing a connection, Cloud Code can run a troubleshooting diagnostic to help resolve the issue.
View your VM logs
As you’re working with your VM, you can right click it and select to view the VM instance logs. From Visual Studio Code this will open a logs viewer in the IDE. From IntelliJ, the logs viewer in the Cloud Console will be opened. If you’ve configured application logs to be collected with Cloud Logging, you can also view those in these logs viewers as well.
Get Started
We invite you to try out Compute Engine with Cloud Code to better streamline your development workflow. To learn more, check out the Compute Engine documentation for Visual Studio Code and JetBrains IDEs. If you’re new to development with IDEs, you can take the first step by installing Visual Studio Code or IntelliJ.
3464
Of your peers have already downloaded this article
1:30 Minutes
The most insightful time you'll spend today!
Banking, insurance and fintech companies strive to deliver superior customer experience. By adopting a human centered design for the digital customer journey, platform providers can drive monetization by focusing on interactions, behaviors and events!
More Relevant Stories for Your Company
Secret Manager: Keeping Your Organization’s Secrets Safer!
Secret Manager is a Google Cloud service that provides a secure and convenient way to store API keys, passwords, certificates, and other sensitive data. It is the central place and single source of truth to manage, access, and audit secrets across Google Cloud. Since its launch, Secret Manager has helped secure
Smart Home Appliances Start-up Chooses Google Cloud to Wow European Customers
The internet of things is everywhere now. Almost everyone has at least one connected device at home, likely a virtual assistant and perhaps a smart programmable thermostat. Nearly all new televisions come preinstalled with streaming apps. Even electric vehicles send drivers communications when tire pressure is low or other problems
Plainsight Vision AI Available for Google Cloud Customers to Unlock Accurate, Actionable Insights
Data-savvy businesses increasingly rely on images and videos for critical functions, and yet are challenged by the sheer mass of information—more than 3.2 billion images and 720,000 hours of video are created daily. This explosion in visual data has paved the way for the growth of computer vision, a form of
Transport Platform’s Richly-detailed Geospatial Data Allows Commuters to Track Buses in Real-time!
Vinayak Bhavnani, Co-Founder and CTO of India-based bus transport technology company Chalo, shares how Google Maps Platform is used to improve visibility for commuters and bus operators across India by visualizing geospatial data. Effective public transport networks contribute to the local economy and help make cities safe, pleasant, and sustainable.
