Google Cloud Results

• Improves application performance by 8x to 12x; customers can create increasingly complex optimized driving routes in single-digit seconds
• Improves customer satisfaction via increased reliability and greater application performance
• Focuses on adding value to customers by improving software and algorithms, not infrastructure management
• Saves 5x in infrastructure costs

In 2009, Dan Khasis needed to rent an apartment. His search had him driving around the greater New York City area in unfamiliar areas, scattershot-style, often ending up where he started. The frustrating experience led the serial entrepreneur to launch Route4Me, a smartphone navigation app to help consumers create driving routes optimized for multiple stops.

Soon, business users recognized Route4Me’s value and requested enhancements specifically for them. While route optimization apps for big businesses already existed, they were almost exclusively offline desktop programs that were expensive to purchase, deploy, and get trained on. Recognizing the opportunity, Route4Me developed an affordable route optimization solution across various devices, such as smartphones, smartwatches, and telematics devices. The software was tailored to logistics-intensive businesses such as last-mile delivery services and business units conducting field sales, field service, and field marketing functions.

As Route4Me grew its user base, it became clear that its infrastructure of rented, dedicated servers from various providers wasn’t sustainable. “The hardware costs seemed low, but there were many risks and hidden costs,” says Dan Khasis, Co-founder and CEO at Route4Me. For example, “Multi-zone disaster recovery, high availability, automated failover, and on-demand surging of many nodes was simply impossible,“ he adds.

Because under the hood Route4Me’s routing optimization platform requires complex computations, the company needed a globally scalable infrastructure capable of delivering low latency and high throughput. Route4Me also needed to stay competitive by developing and delivering new services as quickly and efficiently as possible.

For these and other reasons, Route4Me moved 100% into the cloud. “Like many entrepreneurial software companies, we test all the latest technologies we can find before upgrading. Typically we go with the fastest technology, with a strong bias towards open source and open standards,” Khasis says. Based on extensive testing, Route4Me selected Google Cloud Platform (GCP). Along with the scalability, flexibility, reliability, and low-cost structure of GCP, Route4Me had already migrated its entire platform to containerized microservices, which Khasis says “are extremely stable and reliable” on Google Kubernetes Engine. While Route4Me has proprietary routing and route optimization engines, it uses Google Maps for high-precision geocoding and as the frontend.

With GCP, Route4Me has reduced its IT infrastructure costs while delivering faster route optimizations and more reliable service to customers. Because of GCP, the company is also planning to add services that will deliver the fastest possible routing simulations and calculations to customers at a price that Khasis says is “impossible without a mature cloud-based platform like GCP.”

Unexpected savings, pleasant surprises

The migration to GCP and Kubernetes Engine required Route4Me to revamp its Service-Oriented Architecture (SOA) and convert millions of lines of code into containerized microservices running on Kubernetes Engine. With more than 150 microservices and thousands of add-on modules and features offered on the Route4Me platform, the migration took several months. But the transition, which began in May 2017 and concluded toward year’s end, went smoothly. “Thanks to the reliability and open source portability of Google Kubernetes Engine, Route4Me experienced one-tenth of the problems that we’ve had when onboarding to other cloud providers,” says Khasis.

Halfway into the migration, Route4Me engineers discovered an unexpected cost savings. The ability to run preemptible virtual machine (VM) instances with Kubernetes Engine resulted in a 90% savings in infrastructure costs, according to Khasis.

The engineering team was also pleasantly surprised by the improved intra-system latency and performance between the Google network and those of third-party systems and other data centers that Route4Me connects to. Overall latency dropped from 8x to 12x. “Where it used to take 8 to 14 seconds to plan a complicated route, now it takes as little as 2 seconds,” Khasis says. Route4Me is also running most of its transactional and operational data through Google BigQuery for a variety of business use cases, including complex machine learning tasks such as geospatial analytics, geospatial pattern detection, and synthetic density.

Scaling while delivering great performance

Route4Me algorithms take into account such data as driving distance, driving time, who’s driving, the day of the week, the vehicle being used, weather conditions, and dozens of other attributes. “All those scenarios and data have to be run in near real time,” Khasis explains. The Route4Me system must access multiple internal and external databases, aggregate all the information in parallel, and deliver it using a high-speed infrastructure platform.

“Our core services and algorithms work much faster on a Google architecture, bringing the total time to solve a complex route problem down to single-digit seconds.” “Many of those steps are resource-intensive,” Khasis adds. “With Kubernetes Engine clusters, we can do much more, scaling up and down as needed, and still deliver great performance to customers around the world.”

Because of its scale, Route4Me built its own automation system for marketing, support, and communications with its customers. “Since we moved our proprietary marketing automation system to GCP, we began delivering our omni-channel marketing communications more reliably, and the correct message reached customers faster and at just the right moment,” says Khasis. “That’s translated to happier customers and increased revenue.”

Customer satisfaction has increased, too, because Route4Me’s users experience far fewer slowdowns than before due to the reliability of GCP. The reliability also means the company spends less time worrying about certain clusters or servers going down for extended periods of time. “We have zero sysadmins, which was the Achilles heel of some of my previous startups,” says Khasis. “So we can focus on software development rather than infrastructure management.”

In order to scale as needed and develop new features, Khasis had expected the company would need to hire more SysAdmin, DevOps, and SecOps staff. “But once we migrated to the modern GCP environment, we didn’t have to make those hires. We saved a lot of money by not having to hire, train, and manage more people,” explains Khasis.

Flexible GCP pricing, in which customers only pay for what they use, has saved Route4Me money on its IT infrastructure. “Preemptible server pricing on GCP is so aggressive,” Khasis says. “If servers are automatically shut off for a certain time period, we don’t pay for them for that period. And if servers are on for a certain amount of time, we get an automatic 30% discount. We’re saving money on the platform with fixed and dynamic workloads.”

Per-second billing with GCP also helps Route4Me cut costs. “If it only takes 25 seconds to do something, we only pay for those 25 seconds,” Khasis says. For the same 25 seconds, other cloud providers might charge for 10 minutes usage or even an hour.”

Road map for the future

In the coming year, Route4Me plans to offer additional add-ons as part of its self-service marketplace, providing customers with transparent pricing on highly complex route optimizations. The service will be extremely valuable to heavy users. For instance, if an organization has to visit 50,000 locations by a certain time, it might wonder if it needs to add 20 people to make that happen and how much it’s going to cost. “Because we’re on GCP, our customer can run a variety of complicated routing scenarios to see which one is the most efficient in seconds instead of minutes,” says Khasis. “As far as I know, none of our competitors can offer that kind of service, giving us an edge as well as a new revenue stream.”

Going forward, Route4Me will begin migrating a huge portion of its core routing optimization platform to Google Google Cloud Spanner. “We want to take further advantage of Cloud Spanner, which comes closest to the CAP theorem and permits us to operate an infinitely scalable and nearly indestructible platform,” Khasis says.

As one example, Route4Me receives telematics data, such as GPS coordinates, from Internet of Things (IoT) devices in smartphones and vehicles, and performs complex algorithmic analysis running on Cloud Spanner. This provides real-time return on investment (ROI) information, so customers can see how much money they’re saving by using Route4Me routing optimization services.

“In order to help as many logistics-intensive businesses as possible, we intend to migrate our proprietary mapping, routing, and route optimization services to Cloud Spanner to take advantage of its extreme reliability and redundancy, and the multi-availability zones of Google Cloud Platform,” says Khasis.

Route4Me also plans to leverage Google machine learning technology, in part to make its routing solution available for use in autonomous and drone vehicles, as well as decentralized edge computing deployments. In addition, Google security and encryption technology will help the company expand its offerings to the heavily regulated medical industry.

Over 60 Route4Me team members use G Suite for almost everything. ”We’re interested in using everything possible with G Suite. We get inspiration from G Suite, too. A lot of thinking and effort went into improving G Suite, and we use that as inspiration to improve own products.”

Startups need to move quickly and focus their limited resources on areas where they can differentiate. If infrastructure isn’t your differentiator, don’t put a lot of energy into your infrastructure when someone else can do it for you. What’s more, time-to-market matters to startups more now than ever. Successful early stage companies know this, and leverage existing tools, libraries, frameworks and innovations whenever possible.  

Next week Google Cloud will be featured at TechCrunch Disrupt, the iconic annual event where “founders and investors shaping the future of disruptive technology” come together to share stories, network, and learn from each other. Google Cloud will host a session and Google Cloud engineers will be available in a virtual booth to answer startup questions. 

The session, “Demo Derby – How startups are disrupting the status quo with innovative data analytics, AI and modern app development” will be a fun, fast-paced set of presentations showing short demos of startups and startup projects built with Google Cloud. 

Demo Derby Presenters

• Andrea Le Vot, Chief Data Protection Officer with startup BlueZoo will deliver a demo showing how they are revolutionizing the property insurance industry with AI and Cloud – and doing so while actually protecting people’s privacy.
• Dale Markowitz and Zack Akil, Applied AI Engineers, will show how easy it is to use AI for everything from video search to editing to automated translation.
• Vidya Nagarajan, Group Product Manager from Google Cloud will deliver a fast paced demo that shows how startups can drive developer productivity with serverless innovations.

“Disrupt is iconic for its engagement with founders, investors, and the broad community of early stage companies,” said Andrea Le Vot, Chief Data Protection Officer at BlueZoo. “We are thrilled to share our insights with others in our community, and to show how we have partnered with Google Cloud to innovate faster than most of the larger, well funded insurance companies in our market.”

The series of snippet sized demos will be followed by a roundtable discussion with the demo developers focused on lessons learned, best practices, how to reduce time-to-market, and how to focus on where you can most effectively differentiate.

The session will air on Day One of the event: Tuesday September 21, 2021 at 1pm PT and will be available on demand after the event for all Disrupt attendees.

Editor’s note: This post was originally published in TechPulse Belgium, where Jack Buser, Google Cloud’s Director of Game Industry Solutions, shared his trends for the industry this year.

The year 2022 will hold many surprises (with a few already dropping!), but there’s one near certainty: By this time next year there will be millions more gamers worldwide. It’s thanks to the industry’s little-noted technology growth drivers, which are getting stronger. 

Games are no longer a niche hobby; it’s global, mainstream entertainment. There were 3.1 billion gamers worldwide in 2020. It’s estimated that by 2024 there will be another 500 million. With a global population of 7.7 billion, that’s nearly half the planet. Revenues in 2021 may reach $175 billion—likely more than movies, music, and books combined. 

It’s a strength unique to the games industry, driven by the way its creators deploy new technology. Advances we see in everything from pricing and emerging markets, to backend computing and delivery suggests it will grow from here, on things like artificial intelligence and planet-scale networks.

Games forever, game better

It’s no surprise that videogames have become a huge business. Playing games is one of our deepest human expressions. There is a 4,500 year old board game that people still play, a testimony to the human love of testing cleverness, honing reactions, building trust, or just enlivening a day.

Video games do all that and for decades game makers have taken the most cutting-edge computing tech to make things better. We’ve seen electronic ping pong become shooting asteroids, to simple (yet iconic) 2D sprites evolve into believable, expressive 3D characters, all leading to today’s immersive worlds hosting tens of millions of players worldwide.

There’s more to come. I work at technology’s cutting edge for games, offering the world’s leading game creators access to powerful cloud computing, a low-latency global network, Machine Learning and AI, and much more. As usual, game innovators are taking up all these tools, building new narratives, new ways to play, new markets, and new jaw-dropping moments.

And game creators aren’t just thinking about players; they’re also thinking about viewers. More and more people are watching eSports, and creators are using the games medium to engage their viewers in increasingly immersive ways.

Streaming like never before 

AI and ML are sharpening games and attracting new players in other ways. These technologies increasingly power everything from network performance to global player matching. The delicate but essential work of encouraging newcomers and discouraging bad actors, optimizing monetization, or building and scaling up new player environments.

Another important area we’re seeing cloud tools being put to use is in coordinating the workflows of global teams of developers. It’s not just a question of building faster anymore, but of collaborating and responding more effectively. The distributed work trends that featured so strongly during the pandemic had already featured in much of the game industry, increasingly it’s an industry standard. This will likely mean even faster and more diverse game development, addressing the needs of new markets where we’re seeing double-digit growth.

Tools like cloud streaming, 5G, and edge computing are likely to accelerate the use of Game Streaming, Augmented Reality, and other types of immersive gameplay. If my metaverse doesn’t have games, count me out! We’re looking at some exciting new developments in coming months with some major industry players. Stay tuned.

A big year for the games industry

As a premier provider of solutions, tools and services to games companies, our engineering job is complex and global. Fortunately, our mission is simple: Help game companies transform to meet new global opportunities with planet scale solutions. Technology has brought the gaming industry to astonishing heights, and is the perfect compliment to great storytelling and creative ingenuity. 

As a decades-long industry veteran, I’ve never been more impressed with what technology can do for our industry. Looking past 2022, when we’ll see more compute, more AI, more AR, and more innovation around healthy and exciting game play, there is just one near certainty: More growth.

Records are made to be broken. In 2019, we calculated 31.4 trillion digits of π — a world record at the time. Then, in 2021, scientists at the University of Applied Sciences of the Grisons calculated another 31.4 trillion digits of the constant, bringing the total up to 62.8 trillion decimal places. Today we’re announcing yet another record: 100 trillion digits of π.

This is the second time we’ve used Google Cloud to calculate a record number1 of digits for the mathematical constant, tripling the number of digits in just three years.

This achievement is a testament to how much faster Google Cloud infrastructure gets, year in, year out. The underlying technology that made this possible is Compute Engine, Google Cloud’s secure and customizable compute service, and its several recent additions and improvements: the Compute Engine N2 machine family, 100 Gbps egress bandwidth, Google Virtual NIC, and balanced Persistent Disks. It’s a long list, but we’ll explain each feature one by one.

Before we dive into the tech, here’s an overview of the job we ran to calculate our 100 trillion digits of π.

• Program: y-cruncher v0.7.8, by Alexander J. Yee
• Algorithm: Chudnovsky algorithm
• Compute node: n2-highmem-128 with 128 vCPUs and 864 GB RAM
• Start time: Thu Oct 14 04:45:44 2021 UTC
• End time: Mon Mar 21 04:16:52 2022 UTC
• Total elapsed time: 157 days, 23 hours, 31 minutes and 7.651 seconds
• Total storage size: 663 TB available, 515 TB used
• Total I/O: 43.5 PB read, 38.5 PB written, 82 PB total

Architecture overview

Calculating π is compute-, storage-, and network-intensive. Here’s how we configured our Compute Engine environment for the challenge.

For storage, we estimated the size of the temporary storage required for the calculation to be around 554 TB. The maximum persistent disk capacity that you can attach to a single virtual machine is 257 TB, which is often enough for traditional single node applications, but not in this case. We designed a cluster of one computational node and 32 storage nodes, for a total of 64 iSCSI block storage targets.

The main compute node is a n2-highmem-128 machine running Debian Linux 11, with 128 vCPUs and 864 GB of memory, and 100 Gbps egress bandwidth support. The higher bandwidth support is a critical requirement for the system as we adopted a network-based shared storage architecture.

Each storage server is a n2-highcpu-16 machine configured with two 10,359 GB zonal balanced persistent disks. The N2 machine series provides balanced price/performance, and when configured with 16 vCPUs it provides a network bandwidth of 32 Gbps, with an option to use the latest Intel Ice Lake CPU platform, which makes it a good choice for high-performance storage servers.

Automating the solution

We used Terraform to set up and manage the cluster. We also wrote a couple of shell scripts to automate critical tasks such as deleting old snapshots, and restarting from snapshots (we didn’t need to use this though). The Terraform scripts created OS guest policies to help ensure that the required software packages were automatically installed. Part of the guest OS setup process was handled by startup scripts. In this way, we were able to recreate the entire cluster with just a few commands.

We knew the calculation would run for several months and even a small performance difference could change the runtime by days or possibly weeks. There are also a number of combinations of parameters in the operating system, infrastructure, and application itself. Terraform helped us test dozens of different infrastructure options in a short time. We also developed a small program that runs y-cruncher with different parameters and automated a significant portion of the measurement. Overall, the final design for this calculation was about twice as fast as our first design. In other words, the calculation could’ve taken 300 days instead of 157 days!

The scripts we used are available on GitHub if you want to look at the actual code that we used to calculate the 100 trillion digits.

Choosing the right machine type for the job

Compute Engine offers machine types that support compute- and I/O-intensive workloads. The amount of available memory and network bandwidth were the two most important factors, so we selected n2-highmem-128 (Intel Xeon, 128 vCPUs and 864 GB RAM). It satisfied our requirements: high-performance CPU, large memory, and 100 Gbps egress bandwidth. This VM shape is part of the most popular general purpose VM family in Google Cloud.

100 Gbps networking

The n2-highmem-128 machine type’s support for up to 100 Gbps of egress throughput was also critical. Back in 2019 when we did our 31.4-trillion digit calculation, egress throughput was only 16 Gbps, meaning that bandwidth has increased by 600% in just three years. This increase was a big factor that made this 100-trillion experiment possible, allowing us to move 82.0 PB of data for the calculation, up from 19.1 PB in 2019.

We also changed the network driver from virtio to the new Google Virtual NIC (gVNIC). gVNIC is a new device driver and tightly integrates with Google’s Andromeda virtual network stack to help achieve higher throughput and lower latency. It is also a requirement for 100 Gbps egress bandwidth.

Storage design

Our choice of storage was crucial to the success of this cluster – in terms of capacity, performance, reliability, cost and more. Because the dataset doesn’t fit into main memory, the speed of the storage system was the bottleneck of the calculation. We needed a robust, durable storage system that could handle petabytes of data without any loss or corruption, while fully utilizing the 100 Gbps bandwidth.

Persistent Disk (PD) is a durable high-performance storage option for Compute Engine virtual machines. For this job we decided to use balanced PD, a new type of persistent disk that offers up to 1,200 MB/s read and write throughput and 15-80k IOPS, for about 60% of the cost of SSD PDs. This storage profile is a sweet spot for y-cruncher, which needs high throughput and medium IOPS.

Using Terraform, we tested different combinations of storage node counts, iSCSI targets per node, machine types, and disk size. From those tests, we determined that 32 nodes and 64 disks would likely achieve the best performance for this particular workload.

We scheduled backups automatically every two days using a shell script that checks the time since the last snapshots, runs the fstrim command to discard all unused blocks, and runs the gcloud compute disks snapshot command to create PD snapshots. The gcloud command returns and y-cruncher resumes calculations after a few seconds while the Compute Engine infrastructure copies the data blocks asynchronously in the background, minimizing downtime for the backups.

To store the final results, we attached two 50 TB disks directly to the compute node. Those disks weren’t used until the very last moment, so we didn’t allocate the full capacity until y-cruncher reached the final steps of the calculation, saving four months worth of storage costs for 100 TB.

Results

All this fine tuning and benchmarking got us to the one-hundred trillionth digit of π — 0. We verified the final numbers with another algorithm (Bailey–Borwein–Plouffe formula) when the calculation was completed. This verification was the scariest moment of the entire process because there is no sure way of knowing whether or not the calculation was successful until it finished, five months after it began. Happily, the Bailey-Borwein-Plouffe formula found that our results were valid. Woo-hoo! Here are the last 100 digits of the result:

4658718895 1242883556 4671544483 9873493812 1206904813
2656719174 5255431487 2142102057 7077336434 3095295560

You can also access the entire sequence of numbers on our demo site.

So what?

You may not need to calculate trillions of decimals of π, but this massive calculation demonstrates how Google Cloud’s flexible infrastructure lets teams around the world push the boundaries of scientific experimentation. It’s also an example of the reliability of our products – the program ran for more than five months without node failures, and handled every bit in the 82 PB of disk I/O correctly. The improvements to our infrastructure and products over the last three years made this calculation possible.

Running this calculation was great fun, and we hope that this blog post has given you some ideas about how to use Google Cloud’s scalable compute, networking, and storage infrastructure for your own high performance computing workloads. To get started, we’ve created a codelab where you can create and calculate pi on a Compute Engine virtual machine with step-by-step instructions. And for more on the history of calculating pi, check out this post on The Keyword. Here’s to breaking the next record!

1. We are actively working with Guinness World Records to secure their official validation of this feat as a “World Record”, but we couldn’t wait to share it with the world. This record has been reviewed and validated by Alexander J. Yee, the author of y-cruncher.