Architecting and building a data lake on GCP with open source tools - Build What's Next

3010

Of your peers have already watched this video.

36:30 Minutes

The most insightful time you'll spend today!

Explainer

Architecting and building a data lake on GCP with open source tools

Creating data lakes is often the first step towards maximizing value from data by generating insights for the business.

Hadoop data leaks, are today, the most common ones that are found on-premise and many of Google’s customers are moving these to the Google Cloud Platform. And the trend is accelerating.

In this video, Google Cloud Strategic Cloud Engineer, Roderick Yao, will teach you about the growing challenges in managing on-prem data lakes and what is driving the growth of open source implementations on the cloud.

He will walked you through how to architect, migrate, and secure your own open source data lake on Google Cloud using a mix of managed services and open source tools.

During the course of this presentation, Yao will go over:

  • Why run data lakes on Google Cloud
  • Designing and migrating data lakes
  • Security and governance
Case Study

Mid-Sized B2B Firm Achieves the Business Trifecta with a Single Strategy

6903

Of your peers have already read this article.

2:45 Minutes

The most insightful time you'll spend today!

How a single strategy allowed AfterShip to innovate, generate new revenue channels, and improve customer experience; shrink turnaround time, costs and downtime; and scale its business quickly—while keeping its team small. Find out.

Thirteen years’ experience in e-commerce has given Teddy Chan, Chief Executive Officer and Chief Technology Officer, AfterShip, a deep understanding of the challenges of shipping and tracking packages to customers worldwide.

“The key problem many merchants face is customers asking ‘where is my order?’ and ‘when I will get the package?’” Chan says. “When I considered this issue, I came up with the idea of AfterShip.” Chan helped found AfterShip in 2011 to enable merchants to keep track of packages sent to customers via a web portal or an API. AfterShip also allows merchants to notify customers of anticipated delivery times.

“Using AfterShip, merchants can provide the same experience to buyers regardless of which couriers they use,” Chan says. “Merchants can also improve their customer engagement by including up-selling or marketing content with their delivery notifications.”

Hong Kong-headquartered AfterShip continues to grow quickly and now has a 40-person team. Thirty members of this team are based on the China special administrative region and 10 are based in India.

“Google Cloud Platform has a network of global datacentres with deep connectivity that enables us to put our infrastructure close to our customers. In addition, the ability to horizontally scale our global database using tools such as Google Cloud Spanner eliminates any limits on our geographic expansion.”

Teddy Chan, Chief Executive Officer and Chief Technology Officer, AfterShip

By October 2017, the business was tracking about 30 million packages per month and had expanded its services to include label and rate calculation and self-service return. Revenue, package transaction numbers and team size have doubled every year for the past three years, while more than 300,000 merchants and 426 couriers are signed up to the service. Key customers include Wish, Etsy and Groupon.

Close to half AfterShip’s customers are based in the United States, about one third in Europe and the remainder are located in Asia. AfterShip had initially delivered its applications and services from an incumbent public cloud service. However, the company wanted to continue its growth trajectory while automating key infrastructure processes, implementing a continuous deployment model and controlling costs.

The business needed to achieve these objectives while maintaining a global presence and high-quality service. AfterShip started reviewing its options and decided to migrate to Google Cloud Platform (GCP). “Google Cloud Platform has a network of global datacentres with deep connectivity that enables us to place our infrastructure close to our customers,” Chan says.

“In addition, the ability to horizontally scale our global database using tools such as Google Cloud Spanner eliminates any limits to our geographic expansion. Furthermore, the managed services provided through GCP would allow us to focus on building better features for online merchants.” The reliability provided by GCP would also enable AfterShip to meet the stringent service level requirements of large digital marketplaces in the United States, Asia and elsewhere.

“Google Cloud Platform could manage the high volumes and enable us to deliver the service levels that would realise our ambition of becoming the number one tracking API platform in the world,” Chan says. “For example, with Google Cloud Platform, we can provide a 99.95% monthly uptime service level to our customers.” Finally, GCP provided managed solutions, including Google Kubernetes Engine powered by open source container orchestrator Kubernetes, that would enable AfterShip to automate processes such as scaling and enable its team to focus on developing applications.

AfterShip has moved its websites into GCP infrastructure in three datacentres around the world and anticipates completing the migration in Q4 2017. “Google provided a lot of assistance, particularly early in the project when we needed it,” Chan says. “They briefed us on several services we hadn’t known about that could replace the equivalents in the public cloud we were using previously.” The business then completed the migration using its own skilled team members. As well as Google Kubernetes Engine and Google Cloud Spanner, AfterShip is using Google BigQuery to store and analyse transaction information.

“Google Cloud Platform could manage the high volumes and enable us to deliver the service levels that would realise our ambition of becoming the number one tracking API platform in the world.”

Teddy Chan, Chief Executive Officer and Chief Technology Officer, AfterShip

Deployment times down from one hour to two minutes

With deployment times falling from up to one hour in its previous cloud environment to about two minutes in GCP, AfterShip has been able to adopt a continuous deployment model. “This has improved our service levels,” Chan says. “If there are any issues we can fix them quickly, while we can iterate faster to create new features in response to customer requests or changes in the market. “This enables us to continue to lead our competitors.”

Targeting a 30 percent reduction in costs

AfterShip is now targeting a 30% reduction in costs by optimising its use of Docker containerisation technology on GCP.

“By using Docker with Kubernetes, we have been able to fine-tune our use of compute resources and better control our costs,” Chan says. “We’re extremely pleased with Google Cloud Platform as it really is built for engineers,” he adds. “In addition, its documentation is extremely clear, allowing us to troubleshoot or carry out activities on the platform ourselves. “We look forward to continuing to grow and extend our package tracking and associated services with Google Cloud Platform.”

Case Study

How Do You Cut Costs and Improve Staff Productivity, and Business Visibility? Ascend Money Has an Answer

10699

Of your peers have already read this article.

6:30 Minutes

The most insightful time you'll spend today!

When fintech, Ascend, needed to pare costs and increase internal efficiency, it turned to G Suite and Google Cloud. The move saved the business about US$90,000 in licensing costs and saves about 20 hours per week for each worker across the company. In addition, automation tools reduced the time to complete infrastructure activities by 50 percent.

Hundreds of millions of residents of South East Asia have only limited access to banking and finance services. However, help is at hand. One of South East Asia’s largest fintech businesses, Ascend Money is using digital technologies to realize its mission of enabling as many people as possible to access innovative financial services and live better lives.

According to Ascend Money, 60 percent of the region’s 620 million residents do not have access to a full suite of financial services. Even a relatively small proportion of this market represents a big opportunity for the business.

Ascend Money’s offerings include the TrueMoney regional payment platform for underserved and digital consumers. The TrueMoney platform supports more than 40 million consumers across six countries: Cambodia, Indonesia, Myanmar, the Philippines, Thailand, and Vietnam.

“We found, based on value, ease of use, effectiveness, and the skills and adaptability of our own team members, BigQuery and other Google Cloud Platform services were the best fit for our business.”

– Abraham Jarrett, Head of Engineering, Ascend Money

TrueMoney also provides an e-wallet app that provides easy ways to top up mobile phones, undertake online shopping, and pay for products and services; a network of 65,000 branded shops; and international remittances, initially between Thailand and Myanmar. In addition, Ascend Money offers financial products to small to medium businesses.

Ascend Money is part of the South East Asian online business Ascend Group, which also operates e-commerce, e-procurement, data centers, cloud services, fulfilment, and digital marketing services.

Cost is a key criteria

Ascend Money initially started operations using physical infrastructure and on-premises workforce productivity applications. However, as the business grew its customer base and expanded into new markets, its technology leaders began to explore options to improve value for money; reduce the maintenance load on in-house team members; improve its data management and analysis; and collaborate more effectively.

“Operating our own on-premises infrastructure entails a higher cost of ownership and maintenance, as well as requiring us to scale up our hardware as needed,” says Abraham Jarrett, Head of Engineering, Ascend Money. “We conducted an evaluation and found that moving to the cloud would deliver a range of benefits.”

G Suite and Google Cloud Platform the best fit

The business evaluated solutions available in the market and opted to move to G Suite and Google Cloud Platform.

“Moving to Google Cloud Platform was an optimization exercise, with lowering our costs our primary goal, and achieving better performance an added benefit,” says Jarrett. Google Cloud Platform monitoring, diagnostic, and analytics tools have enabled Ascend Money to reduce its infrastructure spending, becoming more efficient and cost-effective in the process.

“Managing software is a big cost for us. Through G Suite, we are significantly reducing our software deployment, licensing, and repair costs.”

– Abraham Jarrett, Head of Engineering, Ascend Money

The Ascend Money team saw Google Kubernetes Engine as enabling a seamless way of transitioning from an on-premises data center to a cloud service. “We considered industry trends such as what people were adopting and who we could hire when making our decision,” says Jarrett. “Engineering teams are focusing on using Kubernetes open source container management at scale and as a way of doing business, which was attractive to us.

“Google Kubernetes Engine presented the easiest way to manage and orchestrate our containers, and drive us away from our existing infrastructure.”

BigQuery best for cost and ease of use

Ascend Money also reviewed data infrastructure solutions, with its business intelligence and data platform teams considering a range of options. The teams quickly ruled out an on-premises solution due to the capital investment required and opted for a cloud service. Following a rigorous evaluation of Google Cloud Platform against the services offered by another cloud provider, Ascend Money opted to run on Google Cloud.

“We found, based on value, ease of use, effectiveness, and the skills and adaptability of our own team members, BigQuery and other Google Cloud Platform services were the best fit for our business,” says Jarrett. The business is now running an architecture comprising a BigQuery analytics data warehouse; Cloud Storage to store raw and archive data; Cloud Dataflow to process stream and batch data, and Cloud Pub/Sub for event ingestion and delivery.

“We’re expanding our utilization of BigQuery and other services on a daily basis,” says Jarrett.

Ascend Money is also using a range of Google Cloud Platform services for the infrastructure outside its data platform, including Stackdriver logging and monitoring; Cloud KMS to manage cryptographic keys for its cloud services, Cloud Build to undertake continuous integration, delivery, and deployment; Cloud DNS to provide domain name system services; and Cloud Functions to enable its developers to run and scale code in the cloud.

With Google Cloud Platform, Ascend Money is now processing about 12GB of batch data per day and streaming data from about 200 data marts per day in Thailand alone.

Cost effective scalability and faster to market

Google Kubernetes Engine has enabled the business to scale and deliver to market faster. “We can build on the platform, take the application and container and deploy them to scale out,” says Jarrett. “The Google Kubernetes Engine orchestration mechanism that provides containerized, elastic scalability is extremely important in allowing us to manage costs and expend our effort efficiently. Through Kubernetes, we can write once and deploy everywhere.”

With Google Cloud Platform, the business has saved 3,000,000 THB (about US$90,000) in licensing costs and, through automation tools, reduced the time to complete infrastructure activities by 50 percent.

“We don’t have as many meetings since we deployed G Suite and we can work effectively in a distributed fashion.”

– Abraham Jarrett, Head of Engineering, Ascend Money

Ascend Money’s entire workforce is now using G Suite to collaborate and operate productively. The business is achieving a range of benefits including being able to get new team members up and running more quickly and reduced cost. “Managing software is a big cost for us,” says Jarrett. “Through G Suite, we are significantly reducing our software deployment, licensing, and repair costs.” Meanwhile, the shorter time to onboard team members to G Suite is paying off with improved productivity and an accelerated ability to collaborate.

Ascend Money team members primarily use SheetsSlides, and Docs to create internal materials, including product requirement documents in Docs and workforce and project planning spreadsheets in Sheets.

The organization also uses Hangouts Meet to collaborate when working from different locations. “We don’t have as many meetings since we deployed G Suite and we can work effectively in a distributed fashion,” says Jarrett. “People can work from home and we have five regions plus Thailand where they can collaborate live on a document, such as a slide deck for board meetings or meetings with our chief executive officer, head of engineering, or other senior managers.”

20 hours per week saved

Jarrett describes the time savings of using web browser-based productivity software and G Suite as saving them an average of 20 hours per week. This has improved productivity and the quality of life for Ascend Money’s hard-working team, increasing overall staff satisfaction.

“The ability to work remotely is a big win for us,” Jarrett says. “Our team members can keep one computer at work and one at home and access the same information, they can work in a plane while it sits on the tarmac, or update a Google Doc or a Google Sheet in real time on their phone. They can work anytime, from any location, as long as they have an internet connection. They have less dead time, meaning more uptime.”

With Google Cloud Platform and G Suite, Ascend Money is ideally positioned to support growing demand for its fintech services in South East Asia, particularly among people with limited access to banking. “We have the agility and dynamism to support rising demand and look forward to continuing to work with Google to realize our business ambitions,” says Jarrett.

Case Study

Rubin Observatory Leverages Google Cloud to Power Astronomical Research

6073

Of your peers have already read this article.

2:00 Minutes

The most insightful time you'll spend today!

The Vera C. Rubin Observatory launched its the first preview of new Rubin Science Platform (RSP) for an initial cohort of astronomers, powered by Cloud Storage, Google Kubernetes Engine (GKE), and Compute Engine to drive astronomical researches.

This week, the Vera C. Rubin Observatory is launching the first preview of its new Rubin Science Platform (RSP) for an initial cohort of astronomers. The observatory, which is located in Chile but managed by the U.S. National Science Foundation’s NOIRLab in Tucson, AZ and SLAC in California, is jointly funded by the NSF and the U.S. Department of Energy. The platform provides an easy-to-use interface to store and analyze the massive datasets of the Legacy Survey of Space and Time (LSST), which will survey a third of the sky each night for ten years, detecting billions of stars and galaxies, and millions of supernovae, variable stars, and small bodies in our Solar System.

The LSST datasets are unprecedented in size and complexity, and will be far too large for scientists to download to their personal computers for analysis. Instead, scientists will use the RSP to process, query, visualize, and analyze the LSST data archives through a mixture of web portal, notebook, and other virtual data analysis services. An initial launch with simulated data, called Data Preview 0, builds on the Rubin Observatory’s three-year partnership with Google to develop an Interim Data Facility (IDF) on Google Cloud to prototype hosting of the massive LSST dataset. This agreement marks the first time a cloud-based data facility has been used for an astronomy application of this magnitude.

Bringing the stars to the cloud

For Data Preview 0, the IDF leverages Cloud StorageGoogle Kubernetes Engine (GKE), and Compute Engine to provide the Rubin Observatory user community access to simulated LSST data in an early version of the RSP. The simulated data were developed over several years by the LSST Dark Energy Science Collaboration to imitate five years of an LSST-like survey over 300 square degrees of the sky (about 1,500 times the area of the moon). The resulting images are very realistic: they have the same instrumental characteristics, such as pixel size and sensitivity to photons, that are expected from the Rubin Observatory’s LSST Camera, and they were processed with an early version of the LSST Science Pipelines that will eventually be used to process LSST data. “This will be the first time that these workloads have ever been hosted in a cloud environment. Researchers will have an opportunity to explore an early version of this platform,” says Ranpal Gill, senior manager and head of communications at the Rubin Observatory.

Broadening access for more researchers

Over 200 scientists and students with Rubin Observatory data rights were selected to participate in Data Preview 0 from a pool of applicants that represents a wide range of demographic criteria, regions, and experience level. Participants will be supported with resources such as tutorials, seminars, communication channels, and networking opportunities—and they will be free to pursue their own science at their own pace using the data in the RSP. 

“The revolutionary nature of the future LSST dataset requires a commensurately innovative system for data access and analysis paired with robust support for scientists,” says Melissa Graham, lead community scientist for the Rubin Observatory and research scientist in the astronomy department at the University of Washington. “I’m personally excited to enhance my own skills by using the RSP’s tools for big data analysis, while also helping others to learn and to pursue their LSST-related science goals during Data Preview 0.” 

At the same time, the fact that the RSP is hosted in the cloud provides researchers at smaller institutions access to state-of-the-art astronomy infrastructure that is comparable to that of the largest national research centers.

The launch benefits the observatory too: the development team can learn what researchers are interested in while also testing and debugging the platform. Graham says that “the platform is still in active development so researchers using it will be able to follow along in the progress, and provide feedback on ways that we can optimize the development of the tools.”

Next steps

The LSST aims to begin the ten-year survey in 2023-24 and expects it to include 500 petabytes of data. Through the cloud, Google aims to help make this extraordinary project scalable and accessible to researchers everywhere. To learn more about Data Preview 0, watch this video.


Want to ramp up your own research in the cloud? We offer research credits to academics using Google Cloud for qualifying projects in eligible countries. You can find our application form on Google Cloud’s website or contact our sales team.

Case Study

What Swiggy and You Can Learn From This Company’s Use of ML to Engage Customers

9658

Of your peers have already read this article.

1:45 Minutes

The most insightful time you'll spend today!

Just Eat, which is similar to Swiggy, uses Google Cloud's machine learning to power sophisticated consumer recommendations on both its app and website. It enables them to create an “Adventurous Index”, for instance, something we haven't seen in Indian ordering apps.

The app economy has enabled a huge range of unique business models to flourish. One such model is online food ordering and delivery services, in which apps leverage geo-location data to aggregate local food choices and offer personalized options to consumers.

A leading company in this space is Just Eat. Launched in the UK in 2001 with a vision of ‘serving the world’s greatest menu. Brilliantly.’ The company has capitalized on the popularity of online food delivery and grown its presence across 12 markets. 

Just Eat acts as an intermediary between take-out food outlets and hungry customers, giving local restaurants access to a broader base of potential diners, while providing consumers with an easy and secure way to order and pay for food from their favourite restaurants.

Today the company helps 27 million customers find food from more than 112,000 restaurants—everything from homemade Italian pasta, to Chinese noodle bowls, to fish-and-chips. 

Data is the fuel of Just Eat’s rapid growth, but it wasn’t always looked at that way. In its early days, Just Eat struggled with the deluge of information and faced fragmentation across its systems. In fact, the company realized its legacy data vendor wasn’t capable of ingesting 90 percent of the data produced by its food platform. This was incredibly frustrating for Just Eat’s analysts and data scientists, who had to waste time cleaning up sources instead of leveraging the data to create a better user experience. 

Just Eat turned to Google Cloud, and now uses machine learning (ML) to power sophisticated consumer recommendations on both its app and website. It also makes heavy use of features offered by Google Cloud Platform, including BigQuery for running analytics on its customer data set and Cloud Pub/Sub for messaging app users with relevant offers in real-time. 

Having all of Just Eat’s data in one platform has translated into real value for its customers. With Google Cloud tools, Just Eat has created its own proprietary Customer Ontology framework, which today contains 5.5 billion features that better understand consumers’ behavior and food habits, and provides insights into previous visits.

Just Eat recently created an “Adventurous Index” to map its customers according to their ordering habits, enabling them to tailor their marketing and user experiences. For example, mid-adventurous customers are shown a choice of restaurants that serve their most ordered cuisine, while adventurous customers can choose from restaurants that serve a wider variety. This not only has prompted consumers to be more adventurous with their choices, but also has led to more business at a more diverse set of restaurants.

Matt Cresswell, Director of Customer Platforms at Just Eat said that Google Cloud has become integral to its product delivery: “Consumer food choice is a hugely nuanced topic. We know that individuals have their own unique journeys when they use Just Eat. We’ve sought to create a truly one-to-one relationship with every customer. The changes we’ve made to the platform mean they can access the dishes they enjoy at the touch of a fingertip, and find inspiration to discover new dishes they’ll love. We’re grateful to Google Cloud for helping us support our customers on their culinary explorations.”

Whitepaper

How Real IT Leaders Create a Machine Learning Strategy

DOWNLOAD WHITEPAPER

5135

Of your peers have already downloaded this article

7:30 Minutes

The most insightful time you'll spend today!

Sure, machine learning is becoming a business imperative, but how does it work in practice?

That’s the subject of a new step-by-step guide to solving business problems with artificial intelligence and ML, based on insights gathered by IDG Research Services.

Its publication comes at a time when technology leaders face growing pressure to embrace these emerging technologies, yet many have questions about how to get started.

It has real-life examples such as a health services company that used ML to reduce support ticket-resolution time from 48 minutes to six.

In another section, a financial services VP explains that cloud-based ML services enable his company to avoid spending money on computing resources that sit idle.

The guide also includes concrete tips for new ML adopters, provided by the CIOs and other IT leaders who participated in IDG’s research. For example, a real-estate CIO recommends the use of third-party tools that rely on AI and ML technologies, while a financial services VP highlights the challenge and potential of incorporating unstructured data into ML initiatives.

Download the guide now!

More Relevant Stories for Your Company

How-to

Recommendations for Modelling SAP Data inside BigQuery

Over the past few years, many organizations have experienced the benefits of migrating their SAP solutions to Google Cloud. But this migration can do more than reduce IT maintenance costs and make data more secure. By leveraging BigQuery, SAP customers can complement their SAP investments and gain fresh insights by consolidating enterprise

Blog

Google Cloud Celebrates Journey of 3 Inspiring Founders for the Asian Pacific American Heritage Month

May is Asian Pacific American Heritage Month —a time for us to come together to celebrate and remember the important people and history of Asian and Pacific Island heritage. This feature highlights three AAPI founders from the Google For Startups community. Read on to learn how these three founders built

Blog

An Expert’s Opinion on What Early-stage Startups Must Know

As lead for analytics and AI solutions at Google Cloud, my team works with startups building on Google Cloud. This puts us in the fortunate position to learn from founders and engineers about how early-stage startups’ investments can either constrain them or position them for success, even at the seed

Blog

Making Streaming Analytics Simpler and More Cost-Effective With Cloud Dataflow

Streaming analytics helps businesses to understand their customers in real time and adjust their offerings and actions to better serve customer needs. It’s an important part of modern data analytics, and can open up possibilities for faster decisions. For streaming analytics projects to be successful, the tools have to be

SHOW MORE STORIES