APIs are the backbone of digital transformation. Via APIs, you can securely share data and functionality with developers both inside and outside of your organizational boundaries, letting you build applications faster, seamlessly connect and interact with partners, and drive new business revenue. 

Because APIs encompass business-critical information, any downtime or performance degradation can lead to significant loss in revenue, customers, and brand value. Therefore, there’s mounting pressure on operations teams to ensure that APIs are always available and performing as expected. If the APIs go down, so too do the services that fuel customer experiences and on which the organization relies for collaboration and business processes.

However, as you build and scale your API programs, it becomes practically impossible for API operators to manually monitor and manage all your APIs. To help, we brought the power of industry-leading AI and ML technologies to API operations via Apigee X, a major release of our API management platform. Apigee X seamlessly weaves together Google Cloud’s expertise in AI, security and networking to help you efficiently build and manage APIs at scale. 

Put your API data into action

Apigee applies machine learning to your API metadata and provides you the required tools that simplify various aspects of API operations. A great example of AI for APIs is anomaly detection: 

• AI-powered rules trigger alerts based on a set of predefined conditions that are determined by applying Google’s industry-leading machine learning models to your historical API data.
• Auto-thresholds adjust the monitoring criteria of your APIs and set them to pattern-based values. 
• Reduce overhead results because operators don’t have to manually monitor anomalies or adjust the monitoring thresholds on APIs.

“By applying AI and ML models to our historical API data, these advanced features are able to alert us about scenarios we haven’t thought of. Such automation capabilities significantly reduce our upfront efforts. And from a security perspective, the actionable insights help us ensure that our proxies are exposed only over secure HTTPs ports and adhere to compliance requirements. We’re also able to closely monitor user activity and quickly pull out reports during audits.” – Adam Brancato, Sr. Manager, Global Technology and Security at Citrix

As our customers scale their API programs, they find it extremely useful to harness AI-powered capabilities.  In our recent State of the API Economy 2021 report, we found a 230% increase in enterprises’ use of anomaly detection, bot protection, and security analytics features.

To learn more about Apigee X, and see AI and machine learning in action, check out this video, and to try Apigee X for free, click here.

Go is the world’s leading programming language for cloud-based development, used by millions of developers to build and scale their cloud applications and business-critical cloud infrastructure. Whether it’s building CLIs, web applications, or cloud and network services, developers find Go easy to learn, easy to maintain, and packed with useful features such as built in concurrency and a robust standard library.

And now, getting started with Go on Google Cloud is a little bit easier. Following Go’s recent announcement of gonew, an experimental tool for instantiating new projects in Go from predefined templates, Google Cloud is releasing four templates for gonew that developers can use to bootstrap their Go applications across several Google Cloud products. This includes common use cases such as creating a simple HTTP handler Cloud Function, subscribing to a Cloud Pub/Sub topic, or creating an HTTP Server on Cloud Run.

Google Cloud Go templates

• httpfn: A basic HTTP handler (Cloud Function)
• pubsubfn: A function that is subscribed to a PubSub topic handling a Cloud Event (Cloud Function)
• microservice: An HTTP server that can can be deployed to a serverless runtime (Cloud Run)
• taskhandler: An basic app that handles tasks from requests (App Engine)

Let’s get started

• Make sure you have Go installed on your machine, if not you need to download/install from here .
• Start by installing gonew using go install:
  
  $ go install golang.org/x/tools/cmd/gonew@latest
  To generate a basic HTTP Handler Cloud Function, instantiate the existing httpfn template by running gonew in your new project’s parent directory. As of now, gonew’s syntax expects two arguments: first, the path to the template you wish to invoke, and second, the module name of the project you are creating. For example:
  $ gonew github.com/GoogleCloudPlatform/go-templates/functions/httpfn yourdomain.com/httpfn
• Deploy this new go module to Cloud Functions by following the steps listed here, you can also try this in the Cloud Shell editor.

What’s next?

We have big plans for Go on Google Cloud, with several new enhancements on our roadmap. In the meantime, please try out these new templates, deploy them to Google Cloud using the instructions provided and let us know how we can make them better. Please use this to report any issues.

With stated goals of increasing competition, innovation, and financial inclusion, bank regulators across the world are mandating that banks open their systems, enabling consumers to share their financial data with third parties.

One touted benefit of this sharing is that it may create digital banking ecosystems that offer consumers more services than ever, provide banking information and capabilities in more useful and convenient contexts, expand the market reach of ecosystem participants, and boost financial participation among the unbanked and underbanked.

These benefits involve requiring that banks produce application programming interfaces (APIs) to make data and functionality easy to share with partners in a standardized way and to give consumers control over the services with which their data is shared. Because APIs enable developers to leverage and reuse software for new services and digital experiences, including by combining APIs from multiple providers, tech pundits and commentators often refer to the “API economy” — that is, to digital ecosystems in which companies symbiotically share and combine their software to create richer offerings, to complement their proprietary strengths with offerings from other organizations, to share innovation across enterprises, and to expand into new sectors.

Rather than being able to access and act on financial data only through specific channels, for example, consumers in an Open Banking world would theoretically be able to use their money across a constantly-expanding array of apps and digital experiences. Likewise, rather than being confined to one bank’s specific digital services, consumers would be able to opt into a range of services, such as better loan matching or debt reduction advice, to help them do more with their money. Banks, meanwhile, would not have to create all aspects of digital experiences themselves but could rely on external partners, which they can add at unprecedented scale via APIs, to shoulder some of the burden of attracting and creating value for customers.

The envisioned disruption is sweeping, but key questions for bank leaders and bank investors remain, notably the extent to which the Open Banking movement will exert significant, durable impacts on market dynamics — and the extent to which Open Banking compliance constitutes an effective market entry into digital ecosystems and the celebrated benefits of the API economy.

Put simply, is complying with Open Banking regulations adequate to advance a bank’s digital strategy?

Building compliant APIs vs. entering the API economy

Individual regulators are fundamentally constrained in their ability to give banks detailed instructions on how to behave in digital ecosystems. Mandatory regulations can be originally conceived with only one or two business models in mind, not the many thousands of business models that could be partially or fully enabled by the API economy. Detailed rules and specifications may threaten the functionality of services, as regulatory interventions that give highly specific guidance may not be system- and business model-agnostic.

In terms of policy, the willingness of regulators to force banks to adopt APIs is highly significant, but because these regulatory interventions do not offer a meaningful and durable market entry roadmap for banks to follow, the regulations may be an initial catalyst to prompt financial market evolution rather than a natural and enduring end-state for business activity.

Aside from the constraints at the level of the individual bank regulator, there is limited consistency among Open Banking regulations across the globe. Europe’s PSD2 was the regulatory intervention that kicked off this global wave, but there are small but significant variations in the ripple of regulatory actions around the world.

In Australia, Open Banking is part of the Consumer Data Right, an initiative to give customers the right to access their data in a machine-readable form — a right the country does not confine to banking. In Singapore, the Monetary Authority of Singapore is pushing for a lightweight regulatory framework regime. Japan’s Amended Banking Act introduced a registration system for third party providers. Korea’s Financial Services Commission has launched a Fintech Open Platform. Mexico’s recent law to regulate financial technology institutions lays groundwork for an Open Banking regime. The Central Bank of Brazil is aiming to implement the relevant regulatory reforms by the end of 2019. For the largest, internationally diversified banks, these regional differences further complicate any effort to regard mere compliance with these interventions as an effective and coherent market entry strategy.

Despite these complexities and ambiguities, I’ve observed in my work consulting with financial institutions that some banks nevertheless treat Open Banking compliance projects as a means of market entry into the API economy. This mindset could be a costly mistake and may leave these banks at a significant competitive disadvantage as we continue to accelerate into the era of digital ecosystems. In addition to compliance, banks should view a commercial entry into a foreign country as a useful reference for entering the API economy.

Entering the API economy is like entering a foreign market

Banks executing a commercial entry into a foreign country will encounter cultural differences, whether in the form of language, ethnicity, religion, social networks, values or norms. When these cultural differences are large, market entry into a foreign market is more difficult.

The practical implications of administering new business activity in a foreign country also impact the level of difficulty. The physical distance between the home country and the foreign country has to be managed. Border hardness, time zones, and climate also impact the administrative challenges posed by the foreign country. Additionally, the market landscape is foreign. New countries can significantly differ in their natural, financial, and human resources. Staff sent to build up a new division in a new country will have to cope with different levels of market Infrastructure, information, and knowledge. Finally, the economics of entering a foreign country can be heavily influenced by historical and political factors such as a shared colonial history, common memberships of trading blocs, and shared currency zones.

These patterns of differences and similarities can likewise be observed when a bank tries to enter the API economy.

Like entry into a new territory, entry into the API economy may pose sharp cultural and operational differences for banks to grapple with. In this world of APIs and software ecosystems, an API provider’s commercial aim is to make third-parties the dominant force for innovation — that is, to securely share data and services with external contributors who build new connected experiences that generate value for the provider, much as ridesharing companies have generated value by leveraging APIs such as Google Maps. Enterprises focused on ecosystem development market their APIs as products for developers so that new innovation can emerge organically, without necessarily being pre-planned. Investment decisions in the API economy are driven by a desire to help preferred ecosystems to evolve fastest. All of this may be a very sharp change in culture for many banks that historically have sought to be the dominant innovator shaping customer experiences.

The established approach to innovation in banks is highly deliberate, aiming to match bank financial products to specific customer needs and to beat the financial product offerings of peer banks. Compared to modern digital ecosystems, these legacy banks’ resources for, scope of, and receptivity to innovation may be considerably constrained. Banks that actively treat the culture of the API economy as very foreign to their traditional corporate culture have a far greater chance of acknowledging these challenges and making a successful market entry into the API economy.

Many banks may also face challenges transitioning to ecosystem administration models. In traditional banking models, the C-suite commands and controls the bank staff that distributes products. Typically, very few external business partners are involved, and those that are have generally been deliberately, if not laboriously, selected. In contrast, the API economy requires the orchestration of very large numbers of third parties that add value to a bank’s innovation and distribution capabilities. Literally thousands of partners may access a bank’s APIs to build services atop banking data, extend a bank’s functionality or insert the bank’s APIs into new business contexts. In many ways, the whole point is that partnerships don’t have to occur in slow-moving, methodically planned formal partnerships but rather can be achieved at Internet scale while preserving control over and visibility into customer data. This shift in strategy is no small departure for many banks — so again, thinking of the endeavor as entry into a foreign market, rather than something that can be jumpstarted via regulations, is wise.

Crucially, the market landscape in the API economy is also very different from what legacy financial institutions are used to. Traditionally, banks have segmented the market into very large market segments (e.g. consumer banking, corporate banking, private banking etc.). In sharp contrast, the API economy sees partners working together to serve market micro-segments that would not be reachable and profitable by each individual enterprise working in isolation. Simply put, addressing all customer needs, from mainstream use cases to niche applications, is beyond the capabilities of any single enterprise and can generally only be achieved through ecosystems and partnerships. Because partners working together in the API economy seek to serve the customer through the customer’s preferred digital interface (which may be different than bank’s preferred digital interface), ecosystem partnerships represent a massive shift in marketing management, with major implications for a bank’s business architecture, technical architecture, governance, and risk management.

Additionally, in the API economy, pricing decisions are generally driven by a desire for ecosystem growth. Partners work together to extend the customization and value that customers experience when consuming services. By design, the intellectual property in an ecosystem becomes dispersed. Banks have traditionally sought to protect the exclusivity of their intellectual property, and they have often sought economic returns via cost-plus pricing and minimal customization of financial products. Just as with other aspects of the API economy, these ecosystem dynamics and economic relationships typically fall outside a bank’s status quo operations and strategies — more akin to entering a foreign market than to simply satisfying regulatory requirements.

The C-suite should be involved

To approach entry into the API economy as they would approach market entry into a foreign country, banks should consider focusing on a small number of products or segments. They should seek complementary partners who can help them adapt their skills to the API economy and extend the reach of their core differentiating strengths. Just like entry into a foreign market requires combining elements of the home country business model with new local elements, banks will have to transform to increase the effectiveness of their ecosystem participation.

Because an Open Banking compliance project involves regulators specifying the scope and schedule for mandatory APIs, the C-suite may feel relegated to the role of budget provider, which may tempt executives to delegate the sponsorship and steering of the project. Such delegation is not an option for a foreign market entry, where all decisions on scope, cost and timetable are strategic and entirely in the hands of the bank itself — and the C-suite should not consider it an option for entry into the API economy. Bank leaders need to be involved, as the organization likely will not be able to make the necessary strategic and operational transformations if the vision is not defined and driven from the top.

The name Kitabisa means “we can” in Bahasa Indonesia, the official language of Indonesia, and captures our aspirational ethos as Indonesia’s most popular fundraising platform. Since 2013, Kitabisa has been collecting donations in times of crisis and natural disasters to help millions in need. Pursuing our mission of “channeling kindness at scale,” we deploy AI algorithms to foster Southeast Asia’s philanthropic spirit with simplicity and transparency.

Unlike e-commerce platforms that can predict spikes in demand, such as during Black Friday, Kitabisa’s mission of raising funds when disasters like earthquakes strike is by definition unpredictable. This is why the ability to scale up and down seamlessly is critical to our social enterprise.

In 2020, Indonesia’s COVID-19 outbreak coincided with Ramadan. Even in normal times, this is a peak period, as the holy month inspires charitable activity. But during the pandemic, the crush of donations pushed our system beyond the breaking point. Our platform went down for a few minutes just as Indonesia’s giving spirit was at its height, creating frustrations for users.

A new cloud beginning

That’s when we realized we needed to embark on a new cloud journey, moving from our monolithic system to one based on microservices. This would enable us to scale up for surges in demand, but also scale down when a wave of giving subsides. We also needed a more flexible database that would allow us to ingest and process the vast amounts of data that flood into our system in times of crisis.

These requirements led us to re-architect our entire platform on Google Cloud. Guided by a proactive Google Cloud team, we migrated to Google Kubernetes Engine (GKE) for our overall containerized computing infrastructure, and from Amazon RDS to Cloud SQL for MySQL and PostgreSQL, for our managed database services.

The result surpassed our expectations. During the following year’s Ramadan season, we gained a 50% boost in computing resources to easily handle escalating crowdfunding demands on our system. This was thanks to both the seamless scaling of GKE and recommendations from the Google Cloud Partnership team on deploying and optimizing Cloud SQL instances with ProxySQL to optimize our managed database instances.

A progressive journey to kindness at scale

While Kitabisa’s mission has never wavered, our journey to optimized performance took us through several stages before we ultimately landed on our current architecture on Google Cloud.

Origins on a monolithic provider
Kitabisa was initially hosted on DigitalOcean, which only allowed us to run monolithic applications based on virtual machines (VMs) and a stateful managed database. This meant manually adding one VM at a time, which led to challenges in scaling up VMs and core memory when a disaster triggered a spike in donations.

Conversely, when a fundraising cycle was complete, we could not scale down automatically from the high specs of manually provisioned VMs, which was a strain on manpower and budgetary resources.

Transition to containers
To improve scalability, Kitabisa migrated from DigitalOcean to Amazon Web Services (AWS), where we hoped deploying load balancers would provide sufficient automated scaling to meet our network needs. However, we still found manual configurations to be too costly and labor-intensive.

We then attempted to improve automation by switching to a microservices-based architecture. But on Amazon Elastic Container Service (Amazon ECS) we hit a new pain point: when launching applications, we needed to ensure that they were compatible with CloudFormation in deployment, which reduced the flexibility of our solution building due to vendor locking.

We decided it was “never too late” to migrate to Kubernetes, which is a more agile containerized solution. Given that we were already using AWS, it seemed natural to move our microservices to Amazon Elastics Kubernetes Service (Amazon EKS). But we soon found that provisioning Kubernetes clusters with EKS was still a manual process that required a lot of configuration work for every deployment.

Unlocking automated scalability
At the height of the COVID-19 crisis, faced with mounting demands on our system, we decided it was time to give Google Kubernetes Engine (GKE) a try. Since Kubernetes is a Google-designed solution, it seemed likeliest that GKE would provide the most flexible microservices deployment, alongside better access to new features.

Through a direct comparison with AWS, we discovered that everything from provisioning Kubernetes clusters to deploying new applications became fully automated, with the latest upgrades and minimal manual setups. By switching to GKE, we can now absorb any unexpected surge in donations, and add new services without expanding the size of our engineering team. The transformative value of GKE became apparent when severe flooding hit Sumatra in November 2021, affecting 25,000 people. Our system easily handled the 30% spike in donations.

Moving to Cloud SQL and ProxySQL

Kitabisa was also held back by its monolithic database system, which was prone to crashing under heavy demand. We started to solve the problem by moving from a stateful DigitalOcean database to a stateless Redis one, which freed us from relying on a single server, giving us better agility and scale.

But the strategy left a major pain point because it still required us to self-manage databases. In addition, we were experiencing high database egress costs due to the need to execute data transfers from a non-Google Cloud database into BigQuery.

In December 2021, we migrated our Amazon RDS to Cloud SQL for MySQL, and immediately saved 10% in egress costs per month. But one of the greatest benefits came when the Google Cloud team recommended using the open source proxy for MySQL to improve the scalability and stability of our data pipelines.

Cloud SQL’s compatibility allowed us to use connection pooling tools such as ProxySQL to better load balance our application. Historically, creating a direct connection to a monolithic database was a single point of failure that could end up in a crash. With Cloud SQL plus ProxySQL, we create layers in front of our database instances. It serves as a load balancer that allows us to connect simultaneously to multiple database instances, by creating a primary and a read replica instance. Now, whenever we have a read query, we redirect the query to our read replica instance instead of the primary instance.

This configuration has transformed the stability of our database environment because we can have multiple database instances running at the same time, with the load distributed across all instances. Since switching to Cloud SQL as our managed database, and using ProxySQL, we have experienced zero downtime on our fundraising platform even when a major crisis hits.

We are also saving costs. Rather than having a separate database for each different Kubernetes cluster, we’ve merged multiple database instances into one instance. We now group databases according to business units instead of per service, yielding database cost reductions of 30%.

Streamlining with Terraform deployment

There’s another key way in which Google Cloud managed services have allowed us to optimize our environment: using Terraform as an infrastructure-as-a-code tool to create new applications and upgrades to our platform.

We also managed to automate the deployment of Terraform code into Google Cloud with the help of Cloud Build, and no human intervention. That means our development team can focus on creative tasks, while Cloud Build deploys a continuous stream of new features to Kitabisa.

The combination of seamless scalability, resilient data pipelines, and creative freedom is enabling us to drive the future of our platform, expanding our mission to inspire people to create a kinder world in other Asian regions.

We believe that having Google Cloud as our infrastructure backbone will be a critical part of our future development, which will include adding exciting new insurtech features. Now firmly established on Google Cloud, we can go further in shaping the future of fundraising to overcome turbulent times.