With more than 2.5 million customers, Italian utility company A2A is committed to delivering electricity, gas, clean water, and waste collection every day. More recently, the company made another significant commitment: To incorporate the principles of the “circular economy” into its way of doing business — part of the UN 2030 Agenda’s Sustainable-Development Goals — all while also aiming to double its client base by 2030. Growing rapidly but sustainably requires operating as efficiently as possible at every level of the organization, from operating smart meters to generating accurate demand projections. That’s why A2A chose to deploy its SAP S/4HANA ERP and the SAP BW/4HANA data warehouse on Google Cloud.

Roadblocks to innovation

Instead of a linear consumption model that starts with raw materials and ends with use and disposal, the circular economy is a continuous cycle that emphasizes repair, recycling, and the creation of materials rather than their disposal. To take an example from A2A’s own success story: The company keeps 99.7 percent of collected waste out of landfills.1 Of the UN’s sustainability goals, A2A is committing to the three most relevant to its industries:

• Ensuring availability and sustainable management of water and sanitation for all
• Ensuring sustainable consumption and production patterns
• Protecting, restoring, and promoting sustainable use of terrestrial ecosystems

Achieving A2A’s sustainability and customer-first strategies requires high scalability, rapid data ingestion, and rich, accurate analytics. None of this could be reliably supported with the company’s legacy on-premises SAP and Data Warehouse, especially given A2A’s projected growth and the increasing complexity of the data landscape, including IoT deployments and energy market liberalization.

Provisioning data infrastructure was also slow and complex. Simply adding a new metric could require increasing capacity by an order of magnitude. And analytical and transactional data lived in siloes, which created a fragmented and out-of-date view of each customer across sales and customer support teams. A2A’s fragmented data also made it difficult to take proactive action when changing priorities or processes required shifting focus from one data source to another.

With a data warehouse that refreshed only once every 24 hours, simple processes such as responding to a customer calling because their power has been cut off due to an unpaid bill became cumbersome.

Scalability was also a concern. With the on-premises solution, A2A needed to define the budget for its data warehouse over a two-year timeframe, but the rollout of new electricity meters — each sending data every 10 minutes — across Italy made those data requirements hard to predict.

The move to the cloud: From monolith to microservices

The move has been a giant step forward in A2A’s goal of meeting its data-driven, customer-centric strategy. In deploying its SAP systems to Google Cloud, A2A can take advantage of a highly flexible hybrid environment and powerful data management and analytics. It can replicate data from Salesforce, SAP, and other systems in BigQuery, which operates as a data lake with Google Cloud SQL, connected directly to Google Analytics and Google Ads for data-driven customer service, decision-making, and marketing.

“From BigQuery we can feed relevant information directly to the people who need it. Our customer operators work on Salesforce, so we use an OData protocol to embed real-time data in that platform. Elsewhere, we present the information through a dashboard, or with a BI component delivering one-page reports.” —Vito Martino, Head of CRM, Marketing and Sales B2C & B2B, A2A

By running SAP on Google Cloud, A2A can also count on an infrastructure platform that provides:

• Scalability. The robust data architecture on Google Cloud adapts to shifting and increasing demands without compromising on speed or availability, so A2A doesn’t have to worry about over- or under-provisioning as the rollout of smart meters proceeds.
• Speed. The new A2A data solution refreshes every five minutes instead of 24 hours, so the company can respond to its customers’ needs without delays. Customer operators working in Salesforce now receive real-time data from Google BigQuery so that, when a customer calls, operators can see accurate information in seconds. They can now offer value-added services and sustainable options tailored to the customer’s needs, from energy consumption to their preferred method of communication.
• Availability. With microservices orchestrated by Google Kubernetes Engine, the team can update the solution through continuous integration and delivery (CI/CD), eliminating the need for downtime when changes are required.
• Security and control. The A2A IT team uses Google Kubernetes Engine to orchestrate clusters of instances on Google Compute Engine, with Google Cloud Load Balancing and backups on Google Cloud Persistent Disk. Google Cloud Anthos ensures operational consistency across on-premises and cloud platforms.

Ready to grow the sustainable way

By moving to Google Cloud — the industry’s cleanest cloud, with zero net emissions — A2A is ready to grow quickly while locking down the efficiency it will need to meet its ambitious sustainability goals. “To bring sustainable utilities to market, we need to be both responsive to our customers and responsive to the internal needs of A2A,” explains Davide Rizzo, Head of IT Governance and Strategy at A2A. “Understanding what customers need in detail means we can improve their services and reduce their environmental impact at the same time.”

Learn more about the ways Google Cloud can transform your organization’s SAP solutions with scalability, speed, and advanced analytics capabilities.

^{1.  Circular Economy: one of the four founding pillars of A2A’s 2030 sustainability policy | Drupal}

Records are made to be broken. In 2019, we calculated 31.4 trillion digits of π — a world record at the time. Then, in 2021, scientists at the University of Applied Sciences of the Grisons calculated another 31.4 trillion digits of the constant, bringing the total up to 62.8 trillion decimal places. Today we’re announcing yet another record: 100 trillion digits of π.

This is the second time we’ve used Google Cloud to calculate a record number1 of digits for the mathematical constant, tripling the number of digits in just three years.

This achievement is a testament to how much faster Google Cloud infrastructure gets, year in, year out. The underlying technology that made this possible is Compute Engine, Google Cloud’s secure and customizable compute service, and its several recent additions and improvements: the Compute Engine N2 machine family, 100 Gbps egress bandwidth, Google Virtual NIC, and balanced Persistent Disks. It’s a long list, but we’ll explain each feature one by one.

Before we dive into the tech, here’s an overview of the job we ran to calculate our 100 trillion digits of π.

• Program: y-cruncher v0.7.8, by Alexander J. Yee
• Algorithm: Chudnovsky algorithm
• Compute node: n2-highmem-128 with 128 vCPUs and 864 GB RAM
• Start time: Thu Oct 14 04:45:44 2021 UTC
• End time: Mon Mar 21 04:16:52 2022 UTC
• Total elapsed time: 157 days, 23 hours, 31 minutes and 7.651 seconds
• Total storage size: 663 TB available, 515 TB used
• Total I/O: 43.5 PB read, 38.5 PB written, 82 PB total

Architecture overview

Calculating π is compute-, storage-, and network-intensive. Here’s how we configured our Compute Engine environment for the challenge.

For storage, we estimated the size of the temporary storage required for the calculation to be around 554 TB. The maximum persistent disk capacity that you can attach to a single virtual machine is 257 TB, which is often enough for traditional single node applications, but not in this case. We designed a cluster of one computational node and 32 storage nodes, for a total of 64 iSCSI block storage targets.

The main compute node is a n2-highmem-128 machine running Debian Linux 11, with 128 vCPUs and 864 GB of memory, and 100 Gbps egress bandwidth support. The higher bandwidth support is a critical requirement for the system as we adopted a network-based shared storage architecture.

Each storage server is a n2-highcpu-16 machine configured with two 10,359 GB zonal balanced persistent disks. The N2 machine series provides balanced price/performance, and when configured with 16 vCPUs it provides a network bandwidth of 32 Gbps, with an option to use the latest Intel Ice Lake CPU platform, which makes it a good choice for high-performance storage servers.

Automating the solution

We used Terraform to set up and manage the cluster. We also wrote a couple of shell scripts to automate critical tasks such as deleting old snapshots, and restarting from snapshots (we didn’t need to use this though). The Terraform scripts created OS guest policies to help ensure that the required software packages were automatically installed. Part of the guest OS setup process was handled by startup scripts. In this way, we were able to recreate the entire cluster with just a few commands.

We knew the calculation would run for several months and even a small performance difference could change the runtime by days or possibly weeks. There are also a number of combinations of parameters in the operating system, infrastructure, and application itself. Terraform helped us test dozens of different infrastructure options in a short time. We also developed a small program that runs y-cruncher with different parameters and automated a significant portion of the measurement. Overall, the final design for this calculation was about twice as fast as our first design. In other words, the calculation could’ve taken 300 days instead of 157 days!

The scripts we used are available on GitHub if you want to look at the actual code that we used to calculate the 100 trillion digits.

Choosing the right machine type for the job

Compute Engine offers machine types that support compute- and I/O-intensive workloads. The amount of available memory and network bandwidth were the two most important factors, so we selected n2-highmem-128 (Intel Xeon, 128 vCPUs and 864 GB RAM). It satisfied our requirements: high-performance CPU, large memory, and 100 Gbps egress bandwidth. This VM shape is part of the most popular general purpose VM family in Google Cloud.

100 Gbps networking

The n2-highmem-128 machine type’s support for up to 100 Gbps of egress throughput was also critical. Back in 2019 when we did our 31.4-trillion digit calculation, egress throughput was only 16 Gbps, meaning that bandwidth has increased by 600% in just three years. This increase was a big factor that made this 100-trillion experiment possible, allowing us to move 82.0 PB of data for the calculation, up from 19.1 PB in 2019.

We also changed the network driver from virtio to the new Google Virtual NIC (gVNIC). gVNIC is a new device driver and tightly integrates with Google’s Andromeda virtual network stack to help achieve higher throughput and lower latency. It is also a requirement for 100 Gbps egress bandwidth.

Storage design

Our choice of storage was crucial to the success of this cluster – in terms of capacity, performance, reliability, cost and more. Because the dataset doesn’t fit into main memory, the speed of the storage system was the bottleneck of the calculation. We needed a robust, durable storage system that could handle petabytes of data without any loss or corruption, while fully utilizing the 100 Gbps bandwidth.

Persistent Disk (PD) is a durable high-performance storage option for Compute Engine virtual machines. For this job we decided to use balanced PD, a new type of persistent disk that offers up to 1,200 MB/s read and write throughput and 15-80k IOPS, for about 60% of the cost of SSD PDs. This storage profile is a sweet spot for y-cruncher, which needs high throughput and medium IOPS.

Using Terraform, we tested different combinations of storage node counts, iSCSI targets per node, machine types, and disk size. From those tests, we determined that 32 nodes and 64 disks would likely achieve the best performance for this particular workload.

We scheduled backups automatically every two days using a shell script that checks the time since the last snapshots, runs the fstrim command to discard all unused blocks, and runs the gcloud compute disks snapshot command to create PD snapshots. The gcloud command returns and y-cruncher resumes calculations after a few seconds while the Compute Engine infrastructure copies the data blocks asynchronously in the background, minimizing downtime for the backups.

To store the final results, we attached two 50 TB disks directly to the compute node. Those disks weren’t used until the very last moment, so we didn’t allocate the full capacity until y-cruncher reached the final steps of the calculation, saving four months worth of storage costs for 100 TB.

Results

All this fine tuning and benchmarking got us to the one-hundred trillionth digit of π — 0. We verified the final numbers with another algorithm (Bailey–Borwein–Plouffe formula) when the calculation was completed. This verification was the scariest moment of the entire process because there is no sure way of knowing whether or not the calculation was successful until it finished, five months after it began. Happily, the Bailey-Borwein-Plouffe formula found that our results were valid. Woo-hoo! Here are the last 100 digits of the result:

4658718895 1242883556 4671544483 9873493812 1206904813
2656719174 5255431487 2142102057 7077336434 3095295560

You can also access the entire sequence of numbers on our demo site.

So what?

You may not need to calculate trillions of decimals of π, but this massive calculation demonstrates how Google Cloud’s flexible infrastructure lets teams around the world push the boundaries of scientific experimentation. It’s also an example of the reliability of our products – the program ran for more than five months without node failures, and handled every bit in the 82 PB of disk I/O correctly. The improvements to our infrastructure and products over the last three years made this calculation possible.

Running this calculation was great fun, and we hope that this blog post has given you some ideas about how to use Google Cloud’s scalable compute, networking, and storage infrastructure for your own high performance computing workloads. To get started, we’ve created a codelab where you can create and calculate pi on a Compute Engine virtual machine with step-by-step instructions. And for more on the history of calculating pi, check out this post on The Keyword. Here’s to breaking the next record!

1. We are actively working with Guinness World Records to secure their official validation of this feat as a “World Record”, but we couldn’t wait to share it with the world. This record has been reviewed and validated by Alexander J. Yee, the author of y-cruncher.

To fuel the breakthroughs of the future, today’s researchers need easy access to the most innovative cloud technologies. That’s why we are delighted to announce the latest expansion of Google Cloud research credits beyond its previous scope of government and academic research institutions to researchers at nonprofit institutions. This initiative is part of our ongoing efforts to make the advantages of cloud research available to more people everywhere–and to drive more discoveries faster. 

To serve researchers outside of academia, we are now offering nonprofit institutions access to Google’s research credit program. Leading centers of scientific knowledge like the New York Genome Center, Scripps Research Institute, the Vector Institute in Toronto, the Ontario Institute for Cancer Research, and the Allen Institute for AI (AI2) have already benefited from the efficiencies of managing and analyzing their datasets on Google Cloud.

Michael Schmitz, Director of Engineering at AI2, reports that “Google Cloud has been essential for our research program. While we have an on-premise installation for custom hardware, the ability to use Google Cloud for isolated workstations (where researchers always have guaranteed access to a development environment) and bursting with increased need (such as our summer intern surge or a hyperparameter tuning just before our deadline) has been a critical part of conducting our research.”

Also, to better support Ph.D. students, we are updating the terms of Google Cloud credits for use toward advanced research. Now, Ph.D. students can apply for up to $1,000 grants that are renewable over the five years of their programs. We know that the researchers of the future need the best resources now.

Hassaan Maan, Ph.D. student at the University of Toronto/Vector Institute, says “when the pandemic started, I began working on a web-based platform to help researchers analyze genome sequences of the COVID-19 virus in real time and compare with data from labs around the world. Deploying the Covid Genotyping Tool (CGT) platform was challenging, as I had minimal experience hosting web applications and we required a scalable and fine-tuned deployment. Google Cloud’s comprehensive user interface and thorough documentation made for a seamless deployment, and I was able to get the platform up and running in less than two weeks. Google Cloud was a huge factor in the success of this project, and I hope to be able to use it further during my Ph.D. studies.”

To ramp up your own research project with Google Cloud, apply now for free credits in selected countries. To supplement your own learning on Google Cloud, sign up to earn four Google Cloud skills badges that you can share on your social platforms.