Records are made to be broken. In 2019, we calculated 31.4 trillion digits of π — a world record at the time. Then, in 2021, scientists at the University of Applied Sciences of the Grisons calculated another 31.4 trillion digits of the constant, bringing the total up to 62.8 trillion decimal places. Today we’re announcing yet another record: 100 trillion digits of π.

This is the second time we’ve used Google Cloud to calculate a record number1 of digits for the mathematical constant, tripling the number of digits in just three years.

This achievement is a testament to how much faster Google Cloud infrastructure gets, year in, year out. The underlying technology that made this possible is Compute Engine, Google Cloud’s secure and customizable compute service, and its several recent additions and improvements: the Compute Engine N2 machine family, 100 Gbps egress bandwidth, Google Virtual NIC, and balanced Persistent Disks. It’s a long list, but we’ll explain each feature one by one.

Before we dive into the tech, here’s an overview of the job we ran to calculate our 100 trillion digits of π.

• Program: y-cruncher v0.7.8, by Alexander J. Yee
• Algorithm: Chudnovsky algorithm
• Compute node: n2-highmem-128 with 128 vCPUs and 864 GB RAM
• Start time: Thu Oct 14 04:45:44 2021 UTC
• End time: Mon Mar 21 04:16:52 2022 UTC
• Total elapsed time: 157 days, 23 hours, 31 minutes and 7.651 seconds
• Total storage size: 663 TB available, 515 TB used
• Total I/O: 43.5 PB read, 38.5 PB written, 82 PB total

Architecture overview

Calculating π is compute-, storage-, and network-intensive. Here’s how we configured our Compute Engine environment for the challenge.

For storage, we estimated the size of the temporary storage required for the calculation to be around 554 TB. The maximum persistent disk capacity that you can attach to a single virtual machine is 257 TB, which is often enough for traditional single node applications, but not in this case. We designed a cluster of one computational node and 32 storage nodes, for a total of 64 iSCSI block storage targets.

The main compute node is a n2-highmem-128 machine running Debian Linux 11, with 128 vCPUs and 864 GB of memory, and 100 Gbps egress bandwidth support. The higher bandwidth support is a critical requirement for the system as we adopted a network-based shared storage architecture.

Each storage server is a n2-highcpu-16 machine configured with two 10,359 GB zonal balanced persistent disks. The N2 machine series provides balanced price/performance, and when configured with 16 vCPUs it provides a network bandwidth of 32 Gbps, with an option to use the latest Intel Ice Lake CPU platform, which makes it a good choice for high-performance storage servers.

Automating the solution

We used Terraform to set up and manage the cluster. We also wrote a couple of shell scripts to automate critical tasks such as deleting old snapshots, and restarting from snapshots (we didn’t need to use this though). The Terraform scripts created OS guest policies to help ensure that the required software packages were automatically installed. Part of the guest OS setup process was handled by startup scripts. In this way, we were able to recreate the entire cluster with just a few commands.

We knew the calculation would run for several months and even a small performance difference could change the runtime by days or possibly weeks. There are also a number of combinations of parameters in the operating system, infrastructure, and application itself. Terraform helped us test dozens of different infrastructure options in a short time. We also developed a small program that runs y-cruncher with different parameters and automated a significant portion of the measurement. Overall, the final design for this calculation was about twice as fast as our first design. In other words, the calculation could’ve taken 300 days instead of 157 days!

The scripts we used are available on GitHub if you want to look at the actual code that we used to calculate the 100 trillion digits.

Choosing the right machine type for the job

Compute Engine offers machine types that support compute- and I/O-intensive workloads. The amount of available memory and network bandwidth were the two most important factors, so we selected n2-highmem-128 (Intel Xeon, 128 vCPUs and 864 GB RAM). It satisfied our requirements: high-performance CPU, large memory, and 100 Gbps egress bandwidth. This VM shape is part of the most popular general purpose VM family in Google Cloud.

100 Gbps networking

The n2-highmem-128 machine type’s support for up to 100 Gbps of egress throughput was also critical. Back in 2019 when we did our 31.4-trillion digit calculation, egress throughput was only 16 Gbps, meaning that bandwidth has increased by 600% in just three years. This increase was a big factor that made this 100-trillion experiment possible, allowing us to move 82.0 PB of data for the calculation, up from 19.1 PB in 2019.

We also changed the network driver from virtio to the new Google Virtual NIC (gVNIC). gVNIC is a new device driver and tightly integrates with Google’s Andromeda virtual network stack to help achieve higher throughput and lower latency. It is also a requirement for 100 Gbps egress bandwidth.

Storage design

Our choice of storage was crucial to the success of this cluster – in terms of capacity, performance, reliability, cost and more. Because the dataset doesn’t fit into main memory, the speed of the storage system was the bottleneck of the calculation. We needed a robust, durable storage system that could handle petabytes of data without any loss or corruption, while fully utilizing the 100 Gbps bandwidth.

Persistent Disk (PD) is a durable high-performance storage option for Compute Engine virtual machines. For this job we decided to use balanced PD, a new type of persistent disk that offers up to 1,200 MB/s read and write throughput and 15-80k IOPS, for about 60% of the cost of SSD PDs. This storage profile is a sweet spot for y-cruncher, which needs high throughput and medium IOPS.

Using Terraform, we tested different combinations of storage node counts, iSCSI targets per node, machine types, and disk size. From those tests, we determined that 32 nodes and 64 disks would likely achieve the best performance for this particular workload.

We scheduled backups automatically every two days using a shell script that checks the time since the last snapshots, runs the fstrim command to discard all unused blocks, and runs the gcloud compute disks snapshot command to create PD snapshots. The gcloud command returns and y-cruncher resumes calculations after a few seconds while the Compute Engine infrastructure copies the data blocks asynchronously in the background, minimizing downtime for the backups.

To store the final results, we attached two 50 TB disks directly to the compute node. Those disks weren’t used until the very last moment, so we didn’t allocate the full capacity until y-cruncher reached the final steps of the calculation, saving four months worth of storage costs for 100 TB.

Results

All this fine tuning and benchmarking got us to the one-hundred trillionth digit of π — 0. We verified the final numbers with another algorithm (Bailey–Borwein–Plouffe formula) when the calculation was completed. This verification was the scariest moment of the entire process because there is no sure way of knowing whether or not the calculation was successful until it finished, five months after it began. Happily, the Bailey-Borwein-Plouffe formula found that our results were valid. Woo-hoo! Here are the last 100 digits of the result:

4658718895 1242883556 4671544483 9873493812 1206904813
2656719174 5255431487 2142102057 7077336434 3095295560

You can also access the entire sequence of numbers on our demo site.

So what?

You may not need to calculate trillions of decimals of π, but this massive calculation demonstrates how Google Cloud’s flexible infrastructure lets teams around the world push the boundaries of scientific experimentation. It’s also an example of the reliability of our products – the program ran for more than five months without node failures, and handled every bit in the 82 PB of disk I/O correctly. The improvements to our infrastructure and products over the last three years made this calculation possible.

Running this calculation was great fun, and we hope that this blog post has given you some ideas about how to use Google Cloud’s scalable compute, networking, and storage infrastructure for your own high performance computing workloads. To get started, we’ve created a codelab where you can create and calculate pi on a Compute Engine virtual machine with step-by-step instructions. And for more on the history of calculating pi, check out this post on The Keyword. Here’s to breaking the next record!

1. We are actively working with Guinness World Records to secure their official validation of this feat as a “World Record”, but we couldn’t wait to share it with the world. This record has been reviewed and validated by Alexander J. Yee, the author of y-cruncher.

Google Cloud Results

• Improves application performance by 8x to 12x; customers can create increasingly complex optimized driving routes in single-digit seconds
• Improves customer satisfaction via increased reliability and greater application performance
• Focuses on adding value to customers by improving software and algorithms, not infrastructure management
• Saves 5x in infrastructure costs

In 2009, Dan Khasis needed to rent an apartment. His search had him driving around the greater New York City area in unfamiliar areas, scattershot-style, often ending up where he started. The frustrating experience led the serial entrepreneur to launch Route4Me, a smartphone navigation app to help consumers create driving routes optimized for multiple stops.

Soon, business users recognized Route4Me’s value and requested enhancements specifically for them. While route optimization apps for big businesses already existed, they were almost exclusively offline desktop programs that were expensive to purchase, deploy, and get trained on. Recognizing the opportunity, Route4Me developed an affordable route optimization solution across various devices, such as smartphones, smartwatches, and telematics devices. The software was tailored to logistics-intensive businesses such as last-mile delivery services and business units conducting field sales, field service, and field marketing functions.

As Route4Me grew its user base, it became clear that its infrastructure of rented, dedicated servers from various providers wasn’t sustainable. “The hardware costs seemed low, but there were many risks and hidden costs,” says Dan Khasis, Co-founder and CEO at Route4Me. For example, “Multi-zone disaster recovery, high availability, automated failover, and on-demand surging of many nodes was simply impossible,“ he adds.

Because under the hood Route4Me’s routing optimization platform requires complex computations, the company needed a globally scalable infrastructure capable of delivering low latency and high throughput. Route4Me also needed to stay competitive by developing and delivering new services as quickly and efficiently as possible.

For these and other reasons, Route4Me moved 100% into the cloud. “Like many entrepreneurial software companies, we test all the latest technologies we can find before upgrading. Typically we go with the fastest technology, with a strong bias towards open source and open standards,” Khasis says. Based on extensive testing, Route4Me selected Google Cloud Platform (GCP). Along with the scalability, flexibility, reliability, and low-cost structure of GCP, Route4Me had already migrated its entire platform to containerized microservices, which Khasis says “are extremely stable and reliable” on Google Kubernetes Engine. While Route4Me has proprietary routing and route optimization engines, it uses Google Maps for high-precision geocoding and as the frontend.

With GCP, Route4Me has reduced its IT infrastructure costs while delivering faster route optimizations and more reliable service to customers. Because of GCP, the company is also planning to add services that will deliver the fastest possible routing simulations and calculations to customers at a price that Khasis says is “impossible without a mature cloud-based platform like GCP.”

Unexpected savings, pleasant surprises

The migration to GCP and Kubernetes Engine required Route4Me to revamp its Service-Oriented Architecture (SOA) and convert millions of lines of code into containerized microservices running on Kubernetes Engine. With more than 150 microservices and thousands of add-on modules and features offered on the Route4Me platform, the migration took several months. But the transition, which began in May 2017 and concluded toward year’s end, went smoothly. “Thanks to the reliability and open source portability of Google Kubernetes Engine, Route4Me experienced one-tenth of the problems that we’ve had when onboarding to other cloud providers,” says Khasis.

Halfway into the migration, Route4Me engineers discovered an unexpected cost savings. The ability to run preemptible virtual machine (VM) instances with Kubernetes Engine resulted in a 90% savings in infrastructure costs, according to Khasis.

The engineering team was also pleasantly surprised by the improved intra-system latency and performance between the Google network and those of third-party systems and other data centers that Route4Me connects to. Overall latency dropped from 8x to 12x. “Where it used to take 8 to 14 seconds to plan a complicated route, now it takes as little as 2 seconds,” Khasis says. Route4Me is also running most of its transactional and operational data through Google BigQuery for a variety of business use cases, including complex machine learning tasks such as geospatial analytics, geospatial pattern detection, and synthetic density.

Scaling while delivering great performance

Route4Me algorithms take into account such data as driving distance, driving time, who’s driving, the day of the week, the vehicle being used, weather conditions, and dozens of other attributes. “All those scenarios and data have to be run in near real time,” Khasis explains. The Route4Me system must access multiple internal and external databases, aggregate all the information in parallel, and deliver it using a high-speed infrastructure platform.

“Our core services and algorithms work much faster on a Google architecture, bringing the total time to solve a complex route problem down to single-digit seconds.” “Many of those steps are resource-intensive,” Khasis adds. “With Kubernetes Engine clusters, we can do much more, scaling up and down as needed, and still deliver great performance to customers around the world.”

Because of its scale, Route4Me built its own automation system for marketing, support, and communications with its customers. “Since we moved our proprietary marketing automation system to GCP, we began delivering our omni-channel marketing communications more reliably, and the correct message reached customers faster and at just the right moment,” says Khasis. “That’s translated to happier customers and increased revenue.”

Customer satisfaction has increased, too, because Route4Me’s users experience far fewer slowdowns than before due to the reliability of GCP. The reliability also means the company spends less time worrying about certain clusters or servers going down for extended periods of time. “We have zero sysadmins, which was the Achilles heel of some of my previous startups,” says Khasis. “So we can focus on software development rather than infrastructure management.”

In order to scale as needed and develop new features, Khasis had expected the company would need to hire more SysAdmin, DevOps, and SecOps staff. “But once we migrated to the modern GCP environment, we didn’t have to make those hires. We saved a lot of money by not having to hire, train, and manage more people,” explains Khasis.

Flexible GCP pricing, in which customers only pay for what they use, has saved Route4Me money on its IT infrastructure. “Preemptible server pricing on GCP is so aggressive,” Khasis says. “If servers are automatically shut off for a certain time period, we don’t pay for them for that period. And if servers are on for a certain amount of time, we get an automatic 30% discount. We’re saving money on the platform with fixed and dynamic workloads.”

Per-second billing with GCP also helps Route4Me cut costs. “If it only takes 25 seconds to do something, we only pay for those 25 seconds,” Khasis says. For the same 25 seconds, other cloud providers might charge for 10 minutes usage or even an hour.”

Road map for the future

In the coming year, Route4Me plans to offer additional add-ons as part of its self-service marketplace, providing customers with transparent pricing on highly complex route optimizations. The service will be extremely valuable to heavy users. For instance, if an organization has to visit 50,000 locations by a certain time, it might wonder if it needs to add 20 people to make that happen and how much it’s going to cost. “Because we’re on GCP, our customer can run a variety of complicated routing scenarios to see which one is the most efficient in seconds instead of minutes,” says Khasis. “As far as I know, none of our competitors can offer that kind of service, giving us an edge as well as a new revenue stream.”

Going forward, Route4Me will begin migrating a huge portion of its core routing optimization platform to Google Google Cloud Spanner. “We want to take further advantage of Cloud Spanner, which comes closest to the CAP theorem and permits us to operate an infinitely scalable and nearly indestructible platform,” Khasis says.

As one example, Route4Me receives telematics data, such as GPS coordinates, from Internet of Things (IoT) devices in smartphones and vehicles, and performs complex algorithmic analysis running on Cloud Spanner. This provides real-time return on investment (ROI) information, so customers can see how much money they’re saving by using Route4Me routing optimization services.

“In order to help as many logistics-intensive businesses as possible, we intend to migrate our proprietary mapping, routing, and route optimization services to Cloud Spanner to take advantage of its extreme reliability and redundancy, and the multi-availability zones of Google Cloud Platform,” says Khasis.

Route4Me also plans to leverage Google machine learning technology, in part to make its routing solution available for use in autonomous and drone vehicles, as well as decentralized edge computing deployments. In addition, Google security and encryption technology will help the company expand its offerings to the heavily regulated medical industry.

Over 60 Route4Me team members use G Suite for almost everything. ”We’re interested in using everything possible with G Suite. We get inspiration from G Suite, too. A lot of thinking and effort went into improving G Suite, and we use that as inspiration to improve own products.”

There are multiple reasons a growing number of database administrators, enterprise architects, application developers and other technology practitioners are moving to Google Cloud’s various database services.

Some are being driven by missing features in offering from other providers such as AWS. In Gartner’s Magic Quadrant for Operational Database Management Systems, the research and advisory firm points out that, “AWS’s surveyed reference customers scored its overall product capabilities one standard deviation (STD) below the mean. Their responses identified missing features such as multiregion writes and autosharding.”

Others are moving to database services on Google Cloud Platform driven by a few benefits. According to Gartner, “Reference customers repeatedly commented on Google’s ease of use and implementation, reliability and integration (with other services and other systems). Reference customers scored Google a full STD above the mean for satisfaction with GCP’s pricing; it received the second-highest satisfaction score of any vendor in this Magic Quadrant.

If you are looking to leverage the power of Google Cloud database offerings—but were unsure of which database services comes closest to the service you are currently using, here’s a handy map to find your way.