A Record Breaking Calculation: 100 Trillion Digits of π on Google Cloud!
3569
Of your peers have already read this article.
4:00 Minutes
The most insightful time you'll spend today!
Records are made to be broken. In 2019, we calculated 31.4 trillion digits of π — a world record at the time. Then, in 2021, scientists at the University of Applied Sciences of the Grisons calculated another 31.4 trillion digits of the constant, bringing the total up to 62.8 trillion decimal places. Today we’re announcing yet another record: 100 trillion digits of π.
This is the second time we’ve used Google Cloud to calculate a record number1 of digits for the mathematical constant, tripling the number of digits in just three years.
This achievement is a testament to how much faster Google Cloud infrastructure gets, year in, year out. The underlying technology that made this possible is Compute Engine, Google Cloud’s secure and customizable compute service, and its several recent additions and improvements: the Compute Engine N2 machine family, 100 Gbps egress bandwidth, Google Virtual NIC, and balanced Persistent Disks. It’s a long list, but we’ll explain each feature one by one.
Before we dive into the tech, here’s an overview of the job we ran to calculate our 100 trillion digits of π.
- Program: y-cruncher v0.7.8, by Alexander J. Yee
- Algorithm: Chudnovsky algorithm
- Compute node: n2-highmem-128 with 128 vCPUs and 864 GB RAM
- Start time: Thu Oct 14 04:45:44 2021 UTC
- End time: Mon Mar 21 04:16:52 2022 UTC
- Total elapsed time: 157 days, 23 hours, 31 minutes and 7.651 seconds
- Total storage size: 663 TB available, 515 TB used
- Total I/O: 43.5 PB read, 38.5 PB written, 82 PB total
Architecture overview
Calculating π is compute-, storage-, and network-intensive. Here’s how we configured our Compute Engine environment for the challenge.
For storage, we estimated the size of the temporary storage required for the calculation to be around 554 TB. The maximum persistent disk capacity that you can attach to a single virtual machine is 257 TB, which is often enough for traditional single node applications, but not in this case. We designed a cluster of one computational node and 32 storage nodes, for a total of 64 iSCSI block storage targets.
The main compute node is a n2-highmem-128 machine running Debian Linux 11, with 128 vCPUs and 864 GB of memory, and 100 Gbps egress bandwidth support. The higher bandwidth support is a critical requirement for the system as we adopted a network-based shared storage architecture.
Each storage server is a n2-highcpu-16 machine configured with two 10,359 GB zonal balanced persistent disks. The N2 machine series provides balanced price/performance, and when configured with 16 vCPUs it provides a network bandwidth of 32 Gbps, with an option to use the latest Intel Ice Lake CPU platform, which makes it a good choice for high-performance storage servers.
Automating the solution
We used Terraform to set up and manage the cluster. We also wrote a couple of shell scripts to automate critical tasks such as deleting old snapshots, and restarting from snapshots (we didn’t need to use this though). The Terraform scripts created OS guest policies to help ensure that the required software packages were automatically installed. Part of the guest OS setup process was handled by startup scripts. In this way, we were able to recreate the entire cluster with just a few commands.
We knew the calculation would run for several months and even a small performance difference could change the runtime by days or possibly weeks. There are also a number of combinations of parameters in the operating system, infrastructure, and application itself. Terraform helped us test dozens of different infrastructure options in a short time. We also developed a small program that runs y-cruncher with different parameters and automated a significant portion of the measurement. Overall, the final design for this calculation was about twice as fast as our first design. In other words, the calculation could’ve taken 300 days instead of 157 days!
The scripts we used are available on GitHub if you want to look at the actual code that we used to calculate the 100 trillion digits.
Choosing the right machine type for the job
Compute Engine offers machine types that support compute- and I/O-intensive workloads. The amount of available memory and network bandwidth were the two most important factors, so we selected n2-highmem-128 (Intel Xeon, 128 vCPUs and 864 GB RAM). It satisfied our requirements: high-performance CPU, large memory, and 100 Gbps egress bandwidth. This VM shape is part of the most popular general purpose VM family in Google Cloud.
100 Gbps networking
The n2-highmem-128 machine type’s support for up to 100 Gbps of egress throughput was also critical. Back in 2019 when we did our 31.4-trillion digit calculation, egress throughput was only 16 Gbps, meaning that bandwidth has increased by 600% in just three years. This increase was a big factor that made this 100-trillion experiment possible, allowing us to move 82.0 PB of data for the calculation, up from 19.1 PB in 2019.
We also changed the network driver from virtio to the new Google Virtual NIC (gVNIC). gVNIC is a new device driver and tightly integrates with Google’s Andromeda virtual network stack to help achieve higher throughput and lower latency. It is also a requirement for 100 Gbps egress bandwidth.
Storage design
Our choice of storage was crucial to the success of this cluster – in terms of capacity, performance, reliability, cost and more. Because the dataset doesn’t fit into main memory, the speed of the storage system was the bottleneck of the calculation. We needed a robust, durable storage system that could handle petabytes of data without any loss or corruption, while fully utilizing the 100 Gbps bandwidth.
Persistent Disk (PD) is a durable high-performance storage option for Compute Engine virtual machines. For this job we decided to use balanced PD, a new type of persistent disk that offers up to 1,200 MB/s read and write throughput and 15-80k IOPS, for about 60% of the cost of SSD PDs. This storage profile is a sweet spot for y-cruncher, which needs high throughput and medium IOPS.
Using Terraform, we tested different combinations of storage node counts, iSCSI targets per node, machine types, and disk size. From those tests, we determined that 32 nodes and 64 disks would likely achieve the best performance for this particular workload.
We scheduled backups automatically every two days using a shell script that checks the time since the last snapshots, runs the fstrim command to discard all unused blocks, and runs the gcloud compute disks snapshot command to create PD snapshots. The gcloud command returns and y-cruncher resumes calculations after a few seconds while the Compute Engine infrastructure copies the data blocks asynchronously in the background, minimizing downtime for the backups.
To store the final results, we attached two 50 TB disks directly to the compute node. Those disks weren’t used until the very last moment, so we didn’t allocate the full capacity until y-cruncher reached the final steps of the calculation, saving four months worth of storage costs for 100 TB.
Results
All this fine tuning and benchmarking got us to the one-hundred trillionth digit of π — 0. We verified the final numbers with another algorithm (Bailey–Borwein–Plouffe formula) when the calculation was completed. This verification was the scariest moment of the entire process because there is no sure way of knowing whether or not the calculation was successful until it finished, five months after it began. Happily, the Bailey-Borwein-Plouffe formula found that our results were valid. Woo-hoo! Here are the last 100 digits of the result:
4658718895 1242883556 4671544483 9873493812 1206904813
2656719174 5255431487 2142102057 7077336434 3095295560
You can also access the entire sequence of numbers on our demo site.
So what?
You may not need to calculate trillions of decimals of π, but this massive calculation demonstrates how Google Cloud’s flexible infrastructure lets teams around the world push the boundaries of scientific experimentation. It’s also an example of the reliability of our products – the program ran for more than five months without node failures, and handled every bit in the 82 PB of disk I/O correctly. The improvements to our infrastructure and products over the last three years made this calculation possible.
Running this calculation was great fun, and we hope that this blog post has given you some ideas about how to use Google Cloud’s scalable compute, networking, and storage infrastructure for your own high performance computing workloads. To get started, we’ve created a codelab where you can create and calculate pi on a Compute Engine virtual machine with step-by-step instructions. And for more on the history of calculating pi, check out this post on The Keyword. Here’s to breaking the next record!
- We are actively working with Guinness World Records to secure their official validation of this feat as a “World Record”, but we couldn’t wait to share it with the world. This record has been reviewed and validated by Alexander J. Yee, the author of y-cruncher.
3535
Of your peers have already watched this video.
1:30 Minutes
The most insightful time you'll spend today!
See How Rémy Cointreau Drives Customer Centricity with SAP on Google Cloud
Rémy Cointreau is a French, family-owned business group whose origins date back to 1724. Rémy Cointreau is working to be a more customer centric organization. In order to fulfill this goal and to modernize, they determined they needed to get away from infrastructure management and decided to move their SAP landscape to Google Cloud.
Additionally Rémy Cointreau wanted to become more data centric. Learn how operations that used to take five weeks now take five minutes. Learn how Rémy Cointreau is leveraging live data analysis and is preparing for the future with SAP on Google Cloud.
Gartner Names Google Cloud a Leader in the 2019 Cloud Infrastructure as a Service Market
DOWNLOAD WHITEPAPER3535
Of your peers have already downloaded this article
2:30 Minutes
The most insightful time you'll spend today!
As increasing amount of cloud IaaS is being bought for traditional IT, with an emphasis on cost reduction, safety and security, service providers are offering a high-quality service, with excellent availability, good performance, high security and good customer support.
In its Magic Quadrant for Cloud Infrastructure as a Service (IaaS), Gartner notes that the distinctions among providers are apparent in the market for cloud IaaS in terms of worldwide enterprise adoption, capabilities and service availability. Infrastructure and operations leaders should evaluate providers with broad capabilities and a positive track record for customer success.
Download the Gartner Magic Quadrant to understand why Google is a leader in the market.
3495
Of your peers have already watched this video.
2:30 Minutes
The most insightful time you'll spend today!
Google Cloud Helps DSW Engage Over 28M Shoe Lovers in Real Time
DSW Shoes turned to Google Cloud and Google Cloud Partner – Slalom to develop a realtime, engaging loyalty program that serves more than 28 million active members instantly.
The major move from on premise to a cloud-based, data-driven platform that offers speed, flexibility, and scalability has helped increase DSW’s new customer rate by 9% and allows the company to achieve a best-in-class retention rate.
Watch the video to understand how DSW Shoes did this.
Transformation Cloud: A New Wheel of Innovation and Digital Transformation
3361
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
Looking back on the past year, I see challenges—but also reinvention. Reinvention in how children are educated. Reinvention in how medical professionals provide care. Reinvention in how customers purchase products. This reinvention was made possible by all of the IT leaders around the world who had a vision about what could be possible with technology.
Technology has allowed people to work and complete critical activities safely outside of their standard locations. But, it has also enabled transformation in ways we have not seen before: in how people collaborate, in how businesses operate, and most important to me, in how organizations innovate.
Rethinking what it means to transform
The leading indicator for organizations that are accelerating their innovation during this time is how they are thinking about transformation. Instead of asking infrastructure questions about where their apps and services should run, they are asking transformation questions about how to build an environment that enables every person, process, and technology to adapt in order to bring the highest level of innovation to the business.
Innovative companies have moved beyond migrating their data centers to the cloud, changing not only where their business is done but, more importantly, how it is done. For example, Papa John’s recently announced they are building a digital platform that looks at real-time data across the business to improve its loyalty programs, website, and customer and partner experiences. Albertsons Companies is transforming itself by reinventing grocery shopping, both the digital and physical aisle, through shoppable maps, AI-powered conversational commerce, and predictive grocery list building. Airbus is reimagining their work environment to embrace the hybrid work reality. And Siemens is partnering with Google Cloud to reinvent industrial manufacturing with AI to empower employees, automate mundane tasks, and improve product quality.
Here, transformation is made possible with technologies that enable new innovations for their customers versus decisions about where infrastructure should be run. And this aligns with a recent study by Forrester that states the top two IT initiatives for the next 12 months are to increase innovation and to invest in technology that helps employees do their jobs better.1
Some of the best conversations I’ve had with customers are focused on how to:
- Accelerate transformation while also maintaining the freedom to adapt to market needs.
- Make every employee—data scientists to sales associates—smarter with real-time data to make the best decisions.
- Bring people together and enable them to communicate, collaborate, and share with each other when they can not meet in person.
- Protect everything that matters to us—our people, our customers, our data, our customers’ data, and each transaction we undertake.
This new customer thinking is driving new technology requirements—requirements that can be solved through a transformation cloud. A transformation cloud accelerates an organization’s digital transformation through app and infrastructure modernization, data democratization, people connections, and trusted transactions. The result is an organization – and its workers – that can take advantage of all of the benefits of cloud computing to drive innovation.
The new requirements for innovation
Organizations want to work with multiple cloud providers to choose the best technology for each of their apps and services while also mitigating against inevitable cloud outages and vendor lock in. They value the flexibility of open source based solutions and look to our multicloud platforms like Google Kubernetes Engine and Anthos to instill freedom in how they innovate and drive differentiated customer experiences. It is no surprise that, in a recent Google-commissioned IDG research study, 78% of Global IT leaders stated that multi/hybrid cloud support is a major consideration when selecting a cloud provider and 74% preferred open source cloud solutions.2 Customers like MLB, DenizBank, and Macquarie Bank understand the necessity of a multicloud strategy and are taking advantage of Google Cloud’s open, hybrid architecture to give them the maximum flexibility to run their business how and where they want.
These organizations also want to use data to better understand their customers, enhance their products, and improve inventory accuracy in order to make real-time decisions—and bring it together into a cohesive data cloud. They value how analytics solutions democratize access to data for all employees and how embedded AI helps them predict and automate the future. The Forrester study mentioned above also shows that companies focused on improving their use of data for better decision-making, are taking actions to improve data self-service capabilities and make access to data and insights more democratic.3 Customers like Twitter, PayPal, The Home Depot, HSBC, and Stanford Medicine, unify their data across their organizations to power deeper AI-driven business insights, make better real-time decisions, and build and run their data-driven applications.
And while technology is driving many of the transformations we see, so are an organization’s people. Workers are finding new ways to strengthen human connections, deepen their impact, and serve customers while transforming how work happens—as shown in the fact that 59% of organizations in the IDG study accelerated or newly introduced remote working and collaboration capabilities in 2020.4 Customers like Kia Motors, Cambridge Health Alliance, and PwC are using Google Workspace to enable teams of all sizes to connect, create, collaborate, and to drive innovation from any device, and any location.
Finally, the innovation that we see in every digital transaction is matched with new ways to protect and secure the business. Organizations want to protect their employees, customers, and partners against emerging threats, analyze massive amounts of data to secure infrastructure, and build a long term strategy for strategic governance of their assets regardless of their location. Getting this right is essential as organizations see security as a top pain point impeding innovation.5 Customers like Equifax and Evernote are using Google Cloud’s secure platform and security products to extend customer confidence anywhere their systems may operate.
Google Cloud technologies are already powering customers’ transformation clouds
Supporting our customers’ reinventions are our top priority and we believe that together, we can pave the way for what is next. With our investments in multicloud and AI/ML, to sustainable infrastructure, industry solutions, and technology that improves our communities, such as COVID-19 vaccine distribution, we are proud that our customers trust Google Cloud solutions to digitally transform their business.
We have lots more to tell you about in the coming months, starting with our Data Cloud Summit on May 26th. Between this event, and the multiple other events we have this summer, you will learn about how our industry leadership and collaboration with our partners are enabling our customers to build powerful transformation clouds to support their continued reinvention.
1. Forrester Analytics Business Technographics® Priorities And Journey Survey, 2021
2. IDG Communications, Inc: “No Turning Back: How the Pandemic Has Reshaped Digital Business Agendas”, 2021
3. Of the companies that prioritize data in decision making, 37% are improving data self-service capabilities and 30% are making access to data and insights more democratic (Forrester Analytics Business Technographics® Priorities And Journey Survey, 2021)
4. IDG Communications, Inc, “No Turning Back: How the Pandemic Has Reshaped Digital Business Agendas”, 2021
5. 33% of organizations stated Security risks & concerns as a top pain point impeding innovation (IDG Communications, Inc, “No Turning Back: How the Pandemic Has Reshaped Digital Business Agendas”, 2021)
3941
Of your peers have already downloaded this article
4:30 Minutes
The most insightful time you'll spend today!
In 2020, we’ve witnessed tremendous change not only to our daily lives, but also how businesses work.
A key lesson for CFOs and finance leaders: Those who digitized early found faster and most lasting success.
The growth we are likely to witness in 2021 is also going to need digitization, but of a different sort.
Indian enterprises need to create cloud foundations that enable their business with the speed, scale, flexibility and visibility to move quickly and be agile in order to make the most of the growth opportunities in 2021—without over-leveraging or over-burdening their organizations.
That means ensuring their businesses migrate and modernize their most important enterprise workloads—think SAP, Windows, and VMware—to Google Cloud, helping them achieve scale, flexibility and cost savings.
More Relevant Stories for Your Company
Indonesia moves towards advanced education with Cloud, ML & Mobile Development
Indonesia is leading the way for digital transformation in Southeast Asia. According to Google’s e-Conomy South East Asia report, the country’s 2030 Gross Merchandise Value - the value of online retailing to consumers - could be twice the value of the whole of Southeast Asia today. This growth means that
Guide to Google’s Underwater Infrastructure
From data centers and cloud regions to subsea cables, Google is committed to connecting the world. Our investments in infrastructure aim to further improve our network—one of the world’s largest—which helps improve global connectivity, supporting users and Google Cloud customers. Our subsea cables play a starring role in this work,
withVR Uses the Power of VR to Prep People with Speech Disorders for Real-life Speaking Situations
Editor’s note: Meet Gareth Walkom, an entrepreneur dedicated to helping others with speech disorders. Turning life experience into innovation Did you know that 3% of Americans have a speech disorder, while 1% of the world’s population have a stutter? Just getting what they need in everyday interactions can be stressful,
Multicloud Mindset: Thinking About Open Source and Security in a Multicloud World
There’s never been a better time to talk about multicloud, and the Google Cloud Multicloud Mindset series on Twitter Spaces was created to do just that! This series takes place once every two weeks and features live conversations with top experts about the latest multicloud topics. You can join the
