Traveloka Scales Up Operations and Growth with APIs
3619
Of your peers have already read this article.
4:15 Minutes
The most insightful time you'll spend today!
Traveloka is an Indonesian tech company currently focusing on the travel and lifestyle markets. Our customers, the majority of which come from Southeast Asia, visit the site mainly to book flights, hotels, transportation, lifestyle experiences (e.g. theme parks, movie tickets, beauty and spa treatments, and restaurant vouchers), and much more. We aim to serve more customers across the globe as part of our giant expansion plans.
In past blog posts, my colleagues have written about how Traveloka is using Google Cloud for data provisioning and stream analytics. Today, I’m sharing how we use Apigee API Management Platform from Google Cloud, a key part of my role as an engineering manager in charge of the after-sales and affiliation domain.
Extending the global footprint with Apigee
Expanding to markets globally means dealing with a lot of unique and challenging factors in each country. Thinking about different government regulations, currencies, payment systems, and customer service expectations is only the beginning.
Problems arose when partners who had connected with our flights API wanted to add another service, like accommodation. They felt like they were working with a different company. We turned to Google Cloud and found an answer.
There’s a lot of potential obstacles given how differently each country’s marketplace operates, which is why we use APIs and a partner integration approach to better distribute our inventory, thus helping us to scale our business quickly in a competitive space.
When we started Traveloka, we were serious about building a product that would be better than our competitors. Better in terms of inventories, services, and also pricing.
We also knew that we wanted to have the best strategies for increasing our exposure within and outside of Southeast Asia. With all the right features needed (security, high performance, monitoring, and low development and maintenance costs), Apigee helps us execute on our business plan, by making it quick and easy to work with the partners that make the Traveloka platform a success.
API Gateway: Make Or Buy?
When we started, we built our own API gateway with our first version of API specifications. Before that, we looked into several API gateway products (including Apigee), did some research, some POCs, and had several meetings with vendors.
However, since we were just starting to dip our toes into the API business and didn’t have enough use cases to justify the purchase, we decided to build our own system with the bare minimum requirements.
It used to take up to three months to deploy a new API for a partner. Now that we have the Apigee developer portal, it just takes a few days.
After a while, other Traveloka products that also wanted to have their API exposed rebuilt the same thing. Problems arose when partners who had connected with our flights API wanted to add another service, like accommodation. They felt like they were working with a different company. Different standards, different security practices, different formats—it wasn’t an optimal way to work.
About a year and a half after the release, we started to understand the scale, needs, and the limitations of our own platform. We were convinced that a better solution was needed to support our business growth. We then met with Apigee team again, at the Google Cloud Summit 2018 in Jakarta.
Aside from standardizing our APIs, Apigee supports benchmarking, a fantastic developer portal, a sandbox environment, and great monitoring and analytics. Apigee gives us remarkable speed and agility. It enables us to scale quickly, which is vital to executing our business plan to offer more services in more regions.
I also like the rate-limiting feature that lets us limit the throughput rate from one API to another. The fact that we can do this not only per URL, but also per partner, gives us a lot of flexibility.
For example, if partner A has a throughput of 100 calls per second, but partner B needs 500, we can easily meet those differing needs and manage our traffic efficiently. When we evaluated API gateways, Apigee was the only one that enabled us to limit to that level of granularity.
Our APIs are meant for B2B, not for the public. Because of the requirements, we need a high level of security. Apigee helps us in managing the security with its SSL handshake features as well as the authentication and authorization methods.
Aggressive Growth with Apigee
Before we had Apigee, sharing APIs with partners was a laborious process. We exposed approximately 20 APIs through a shared document that listed our APIs with the types of the requests and responses they need to handle. Partners could see what was available and how they could integrate into our staging. More work was required before they could come in and try APIs in our sandbox.
It used to take up to three months to deploy a new API for a partner. Now that we have the Apigee developer portal, it just takes a few days.
The developer portal enables our partners to view all of our APIs and try them out in the sandbox. The simplicity in creating and managing proxies is also a huge time saver. Right now, we have two B2B partners onboarded to the developer platform, and a few more still working manually through our old, in-house platform.
We expect to transition them over and add 20 new partners to the Apigee platform this year. Because it now takes us as little as two weeks to onboard a new partner from first establishment to go live, we’re ramping up quickly. We estimate that we saved at least a year of development time by deploying the Apigee platform as opposed to building one in-house.
Rapid Learning, Rapid Onboarding
Our developers are very happy with Apigee and feel the portal makes Traveloka look more professional. Equally important, the Apigee learning curve for developers and partners has been short. In just one week our partners know how to use all our APIs. If you compare that to the previous process of referring to a Google Doc and building their own servers to connect to our staging, it’s incredibly easy.
We look forward to exploring other ways that the Apigee platform can help Traveloka; we see strong potential to use even more features to help Traveloka grow and stay competitive.
Why You Should Consider API-first Integration
3168
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
Enterprises need to move faster than ever to gain a competitive advantage in today’s customer-focused environment. Time-to-market for products and services has shortened dramatically, from years to days. IT teams must move fast, react fast, and enable business strategies via constant innovation.
All of this digital transformation is about more than adopting the latest technologies. It is also about maximizing the use of existing data and services to improve efficiency and productivity, drive engagement and growth, and ultimately make the lives of customers, partners, and staff better. Connecting existing data and services and making them easily accessible via APIs promises a path forward, empowering enterprises to extend the value they already possess with new technologies, managed services, ecosystems, and support.
In addition to the challenges of legacy data and systems, today’s organizations are overwhelmed by the variety of cloud applications to meet their business needs and deliver innovative services to their customers.
Managing all of this data, connecting sources, integrating applications, and surfacing them as easy-to-use APIs for development is a crucial competency for any IT organization.
Design APIs with an Outside-in Approach
Many IT organizations have focused on solving this challenge with an “inside-out” approach: starting with the integration layer, building the flow, and then developing the APIs. But this approach is inefficient and fundamentally flawed because it looks at the problem from an “exposure” model; rather than designing for the business use cases of developers and other API consumers. Leaders in the organization end up seeing all of this data, connectivity, and integration as the “table stakes plumbing”, and thus do not seek inputs regarding business value from key business stakeholders.
The key issue is not exposure but rather how you leverage your data, services, and systems to drive impact across your digital value chain. Goals such as meeting your adoption or sales targets, reducing costs across lines of business, speeding up time to market, and reducing time spent supporting your customers may all be within reach. Easy-to-use APIs, rather than crudely exposed systems, are foundational enablers of this impact, and they are almost always designed from the outside-in, from the perspective of teams that are consuming the APIs to achieve a business goal.
Embrace API-first Integration
To accelerate the speed of development, enterprise IT teams need to take an API-first approach to integration, starting with the consumers’ use cases rather than the structure of the data in their systems.
The notion of outside-in thinking should be familiar to product managers, who routinely have to demonstrate customer empathy and put themselves in their customers’ shoes. If your team has a product owner, be sure they are empowered to decide what functionality is needed from their data.
Maximize your APIs with the right technology enablers
An API-first strategy treats the API not as middleware but as a software product that empowers developers, enables partnerships, and accelerates innovation—a big shift from integration-first operations in which APIs are typically exposed and then forgotten.
Possessing APIs is only part of the equation. If a company is going to share valuable digital assets with outsiders, it needs API management tools to:
- apply security protections, such as authentication and authorization
- protect assets from malicious attacks
- monitor digital services to ensure availability and high performance
- measure and track usage of the assets
With the right tools in place, APIs can unlock incredible business opportunities—which is a reason for every enterprise to aspire to be API-first!
Visit our website to learn more about API management with Google Cloud.
Scaling with Breaking News: BBC’s Serverless Infrastructure on Google Cloud
1282
Of your peers have already read this article.
3:30 Minutes
The most insightful time you'll spend today!
Editors note: Today’s post is from Neil Craig at the British Broadcasting Corporation (BBC), the national broadcaster of the United Kingdom. Neil is part of the BBC’s Digital Distribution team which is responsible for building the services such as the public-facing www bbc.co.uk and .com websites and ensuring they are able to scale and operate reliably.
The BBC’s public-facing websites inform, educate, and entertain over 498 million adults per week across the world. Because breaking news is so unpredictable, we need a core content delivery platform that can easily scale in response to surges in traffic, which can be quite unpredictable.
To this end, we recently rebuilt our log-processing infrastructure on a Google Cloud serverless platform. We’ve found that the new system, based on Cloud Storage, Eventarc, Cloud Run and BigQuery, enables us to provide a reliable and stable service without us having to worry about scaling up during busy times. We’re also able to save license fee payers money by operating the service more cost effectively than our previous architecture. Not having to manually manage the scale of major components of the stack has freed up our time, allowing us to spend it on using, rather than creating the data.
A log in time
To operate the site and ensure our services run smoothly we continually monitor Traffic Manager and CDN access logs. Our websites generate more than 3B log lines per day, and handle large data bursts during major news events; on a busy day our system supports over 26B log lines in a single day.
As initially designed, we stored log data in a Cloud Storage bucket. But every time we needed to access that data, we had to download terabytes of logs down to a virtual machine (VM) with a large amount of attached storage, and use the ‘grep’ tool to search and analyze them. From beginning to end, this took us several hours. On heavy news days, the time lag made it difficult for the engineering team to do their jobs.
We needed a more efficient way to make this log data available, so we designed and deployed a new system that deals with logs and reacts to spikes more efficiently as they arrive, improving the timeliness of critical information significantly.
In this new system, we still leverage Cloud Storage buckets, but on arrival, each log generates an event using EventArc. That event triggers Cloud Run to validate, transform and enrich various pieces of information about the log file such as filename, prefix, and type, then processes it and outputs the processed data as a stream into BigQuery. This event-driven design allows us to process files quickly and frequently — processing a single log file typically takes less than a second. Most of the files that we feed into the system are small, fewer than 100 Megabytes, but for larger files, we automatically split those into multiple files and Cloud Run automatically creates additional parallel instances very quickly, helping the system scale almost instantly.
The nature of running a global website which provides news coverage means we see frequent, unpredictable large spikes of traffic. We learn from these and optimize our systems where necessary so we’re confident in the system’s ability to handle significant traffic. For example, around the time of the announcement of the Queen’s passing in September, we saw some huge traffic spikes. During the largest, within one minute, we went from running 150 – 200 container instances to over 1000…. and the infrastructure just worked. Because we engineered the log processing system to rely on the elasticity of a serverless architecture, we knew from the get-go that it would be able to handle this type of scaling.
Around the time of the announcement of the Queen’s passing in September, we saw some huge traffic spikes. During the largest, within one minute, we went from running 150 – 200 container instances to over 1000…. and the infrastructure just worked
Our initial concern about choosing serverless was cost. It turns out that using Cloud Run is significantly more cost-effective than running the number of VMs we would need for a system that could survive reasonable traffic spikes with a similar level of confidence.
Switching to Cloud Run also allows us to use our time more efficiently, as we no longer need to spend time managing and monitoring VM scaling or resource usage. We picked Cloud Run intentionally because we wanted a system that could scale well without manual intervention. As the digital distribution team, our job is not to do ops work on the underlying components of this system — we leave that to the specialist ops teams at Google.
Another conscious choice we made whilst rebuilding the system was to use the built-in service-to-service authentication in Google Cloud. Rather than implementing and maintaining the authentication mechanism ourselves, we add some simple configuration which instructs the client side to create and send a OIDC token for a service account we define and the server side to authenticate and authorize the client. Another example is pushing events into Cloud Run, where we can configure Cloud Run authorization to only accept events from specific EventArc triggers, so it is fully private.
Going forward, the new system has allowed us to make better use of our data safely. For example, BigQuery’s per-column permissions allow us to open up access to our logs to other engineering teams around the organization, without having to worry about sharing PII that’s restricted to approved users.
The goal of our team is to empower all teams within the BBC to get the content they want on the web when they want it, make it reliable, secure, and make sure it can scale. Google Cloud serverless products helped us to achieve these goals with relatively little effort and require significantly less management than previous generations of technology.
3364
Of your peers have already watched this video.
24:40 Minutes
The most insightful time you'll spend today!
A Guide to Anthos: the App Modernization Platform for Hybrid and Multi-cloud Deployments
Anthos gives you a consistent platform for all your application deployments, both legacy as well as cloud native, while offering a service-centric view of all your environments.
You can build enterprise-grade containerized applications faster with managed Kubernetes on cloud and on-premises environments. Create a fast, scalable software delivery pipeline with cloud-native tooling and guidance.
It also helps leverage a programmatic, outcome-focused approach to managing policies for apps across environments, and enable greater awareness and control with a unified view of your services’ health and performance.
Tune in to this brief explainer video for an overview of Anthos, deployment examples, and explore options to get started illustrated through a demo with Bradley Wong, Product Manager at Google Cloud. Learn to make the best first step in your journey of application modernization and gain a clear understanding of the value of Anthos for your hybrid and multi-cloud workloads.
4133
Of your peers have already downloaded this article
10:32 Minutes
The most insightful time you'll spend today!
While microservices are lauded as catalysts for speed and scale, the conversation is incomplete if it does not include APIs. Without APIs, microservices cannot be securely, reliably, and adaptably scaled across an organization or to outside partners. Moreover, because these APIs and their microservices are meant to be widely consumed and reused, it’s not enough for companies to merely create the APIs and move on. Rather, the APIs should be continually managed so the business can control how its microservices are accessed, generate insight into how they are used, and encourage reliability of its services.
As microservices grow beyond their original use cases and are being leveraged to share important functionality throughout the enterprises and even with external partners, the problem that arises is how this sharing creates challenges for teams trying to secure, monitor, understand usage of, and fully take advantage of microservices.
In this eBook we explore how leading enterprises are facing these challenges head-on and scaling their microservices strategies. Learn how APIs make it possible for microservices to be securely, reliably, and adaptably shared, and how an API management platform enables enterprises to ensure that they maintain control over their microservices as consumption grows.
Serverless for Startups, the Best Way to Succeed: Expert Says
7067
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
As Google Cloud has become a choice for more startups, I’ve experienced an increase in founders asking how they should think about cloud services. Though each startup is different and requirements may vary across industries and regions, I’ve seen a few core best practices that help startups to succeed—as well as several traps to avoid.
For example, If you go with a public cloud provider, you’re ideally not starting from ground zero like you would running your own data center, but it’s important not to introduce similar complexity in a virtualized environment. Just because you’re using public cloud infrastructure doesn’t mean you want to manage it.
Instead, you want to leverage platforms that abstract away complexity so your team can focus on delivering value to customers. That’s where serverless comes in. Serverless platforms are fully managed by the provider, offering automatic scaling for workloads, as well as provisioning, configuring, patching, and management of servers and clusters. Freed from these resource-intensive tasks, your technical talent can focus on the things that differentiate your business, not on IT curation.
This applies to not only running stateless applications, but also managing and analyzing data. You should be collecting data points about which features are most popular on your platform, what people are buying, the types of activities that help your customers get their jobs done, and so on. This information is essential to building and executing on a product roadmap that will serve your customers. However, it’s not enough to collect this data, you need to make your data accessible, secure, and easy for your team to analyze—so where do you run and host it?
On Google Cloud, this is where options like Spanner and Cloud SQL can play a large role, as can BigQuery for analysis. You won’t have to worry about standing up infrastructure or patching servers—you can just stream your data to our data management platforms, where it’s available whenever someone needs to run a query. By leveraging a serverless architecture, your startup can be data-driven without having to invest in the traditional complexity of database administration and management—and that can significantly change your playing field.
Serverless is just one of the factors you should consider as you build out your tech stack. To hear my thoughts on a range of other topics relevant to startups — such as security, cloud credits, and the differences among managed services — check out the below video or visit our Build and Grow page.
More Relevant Stories for Your Company
AI Features in Apigee X Helps Build and Manage APIs at Scale
APIs are the backbone of digital transformation. Via APIs, you can securely share data and functionality with developers both inside and outside of your organizational boundaries, letting you build applications faster, seamlessly connect and interact with partners, and drive new business revenue. Because APIs encompass business-critical information, any downtime or
How This Leading Trading Company Uses APIs to Build Fintech Apps Quickly and Cost-Effectively
Tradier uses the Apigee API management platform from Google to abstract the legacy complexities of capital markets so that developers can build FinTech applications in an agile, nimble, and quick fashion at minimal cost. The company embodies the evolution of what cloud technology can enable in the form of an API-first business delivered
Building a Software Delivery Platform with Anthos
Understand the architecture of a GitOps based CI/CD pipeline. In a CI/CD pipeline, there are three different personas -- developers, operators, and security engineers. This demo includes common developer, operator, and security engineer tasks to show how the patterns can improve your company’s software delivery performance.
