Making Weather Predictions Easy with Weather Research and Forecasting (WRF) Models on Google Cloud!
3591
Of your peers have already read this article.
6:00 Minutes
The most insightful time you'll spend today!
Weather forecasting and climate modeling are two of the world’s most computationally complex and demanding tasks. Further, they’re extremely time-sensitive and in high demand — everyone from weekend travelers to large-scale industrial farming operators wants up-to-date weather predictions. To provide timely and meaningful predictions, weather forecasters usually rely on high performance computing (HPC) clusters hosted in an on-premises data center. These on-prem HPC systems require significant capital investment and have high long-term operational costs. They consume a lot of electricity, have largely fixed configurations, and the underlying computer hardware is replaced infrequently.
Using the cloud instead offers increased flexibility, constantly refreshed hardware, high reliability, geo-distributed compute and networking, and a “pay for what you use” pricing model. Ultimately, cloud computing allows forecasters and climate modelers to provide timely and accurate results on a flexible platform using the latest hardware and software systems, in a cost effective manner. This is a big shift compared with traditional approaches to weather forecasting, and can appear challenging. To help, weather forecasters can now run the Weather Research and Forecasting (WRF) modeling system easily on Google Cloud using the new WRF VM image from Fluid Numerics, and achieve the performance of an on-premises supercomputer for a fraction of the price. With this solution, weather forecasters can get a WRF simulation up and running on Google Cloud in less than an hour!
A closer look at WRF
Weather Research and Forecasting (WRF) is a popular open-source numerical weather prediction modeling system used by both researchers and operational organizations. While WRF is primarily used for weather and climate simulation, teams have extended it to support interactions with chemistry, forest fire modeling, and other use cases. WRF development began in the late 1990s through a collaboration between the National Center for Atmospheric Research (NCAR), National Oceanic and Atmospheric Administration (NOAA), U.S. Air Force, Naval Research Laboratory, University of Oklahoma, and the Federal Aviation Administration. The WRF community comprises more than 48,000 users spanning over 160 countries, with the shared goal of supporting atmospheric research and operational forecasting.
The Google Cloud WRF image is built using Google’s MPI best practices for HPC, with the exception that hyperthreading is not disabled by default, and is easily integrated with other HPC solutions on Google Cloud, including SchedMD’s Slurm-GCP. Normally, installing WRF and its dependencies is a time consuming process. With these new WRF VM images, deploying a scalable HPC cluster with WRF v4.2 pre-installed is quick and easy with our Codelab. OpenMPI 4.0.2 was used throughout this work. Google has had good success with Intel MPI, and we intend to study whether further performance gains can be achieved in this context.
Optimizing WRF
Determining the optimal architecture and build settings for performance and cost was a key part of the process in developing the WRF images. We evaluated how to select the ideal compiler, right CPU platform, and the best file system for handling file IO, so you don’t have to. As a test case for assessing performance, we used the CONUS 2.5km benchmark.
Below, the CONUS 2.5km runtime and cost figure shows the run time required for simulating WRF over a two-hour forecast using 480 MPI ranks (a way of numbering processes) for different machine types available on Google Cloud. For each machine type, we’re showing the lowest measured run time from a suite of tests that varied compiler, compiler optimizations, and task affinity.
We found that compute-optimized c2 instances provided the shortest run time. The Slurm job scheduler allows you to map the MPI tasks to compute hardware using task affinity flags. When optimizing the runtime and cost for each machine type, we compared using srun –map-by core –bind-to core to launch WRF, which maps each MPI process to a physical core (two vCPU per MPI rank), and srun –map-by thread –bind-to thread, which maps each MPI process to a single vCPU. Mapping by core and binding MPI ranks to cores is akin to disabling hyperthreading.
The ideal simulation cost and runtime for CONUS 2.5km for each platform is found when each MPI rank is subscribed to each vCPU. When binding to vCPUs, half as many compute resources are needed when compared to binding to physical cores lowering the per-second cost for the simulation. For CONUS 2.5km, we also found that although mapping MPI ranks to cores results in reduced runtime for the same number of MPI ranks, the performance gains are not significant enough to outweigh the cost savings. For this reason, the WRF-GCP solution does not disable hyperthreading by default.
Runtime and simulation cost can be further reduced by selecting an ideal compiler: the figure below (CONUS 2.5km Compiler Comparisons) shows the simulation runtime for the WRF CONUS 2.5km benchmark on eight c2-standard-60 instances, using GCC 10.30, GCC 11.2.0 and the Intel® OneAPI® compilers (v2021.2.0). In all cases, WRF is built using level 3 compiler optimizations and Cascade Lake target architecture flags. By compiling WRF with the Intel® OneAPI® compilers, the WRF simulation runs about 47% faster than the GCC builds, and at about 68% of the cost, on the same hardware. We’ve used OpenMPI 4.0.2 with each of the compilers as the MPI implementation in this work. With other applications, Google has seen good performance with Intel MPI 2018, and we intend to investigate performance comparisons with this and other MPI implementations.
File IO in WRF can become a significant bottleneck as the number of MPI ranks increases. Obtaining the optimal file IO performance requires using parallel file IO in WRF and leveraging a parallel file system such as Lustre.
Below, we show the speedup in file IO activities relative to serial IO on an NFS file system. For this example, we are running the CONUS 2.5km benchmark on c2-standard-60 instances with 960 MPI ranks. By changing WRF’s file IO strategy to parallel IO, we accelerate file IO time by a factor of 60.
We further speed up IO and reduce simulation costs by using a Lustre parallel file system deployed from open-source Lustre Terraform infrastructure-as-code from Fluid Numerics. Lustre is also available with support from DDN’s EXAScaler solution in the Google Cloud Marketplace. In this case, we use four n2-standard-16 instances for the Lustre Object Storage Server (OSS) instances, each with 3TB of Local SSD. The Lustre Metadata Server (MDS) is an n2-standard-16 instance with a 1TB PD-SSD disk. After mounting the Lustre file system to the cluster, we set the Lustre stripe count to 4 so that file IO can be distributed across the four OSS instances. By switching to the Lustre file system for IO, we speed up file IO by an additional factor of 193, which is orders of magnitude faster than a single NFS server with serial IO.
Adding compute resources and increasing the number of MPI ranks reduces the simulation run time. Ideally, with perfect linear scaling, doubling the number of MPI ranks would cut the simulation time in half. However, adding MPI ranks also increases communication overhead, which can increase the cost per simulation. The communication overhead is due to the increased amount of communication necessitated by splitting the problem more finely across more machines.
To assess the scalability of WRF for the CONUS 2.5km benchmark, we can execute a series of model forecasts where we successively double the number of MPI ranks. Below, we show two- hour forecasts on the c2-standard-60 instances with the Lustre file system, varying the number of MPI ranks from 480 to 1920. In all of these runs, MPI ranks are bound to vCPUs so that the number of vCPUs dedicated to each simulation increases with the increase in MPI ranks. While many HPC workloads run best with simultaneous multithreading (SMT) disabled, we find the best performance for CONUS 2.5km with SMT enabled. Thus, the number of MPI ranks in our runs equals the total number of vCPUs.
As you can see, the CONUS 2.5km Runtime & Cost Scaling figure shows that the run time (blue bars) decreases as the number of MPI ranks and the amount of compute resources increases, at least up to 1920 ranks. When transitioning from 480 to 960 MPI ranks, the run time drops, yielding a speedup of about 1.8x. Doubling again to 1920 MPI ranks, though, we obtained an additional speedup of just 1.5x. This declining trend in the speedup with increasing MPI ranks is a signature of MPI overhead, which increases with more MPI ranks.
Determining your best fit
Most tightly-coupled MPI applications such as WRF exhibit this kind of scaling behavior, where scaling efficiency decreases with increasing MPI ranks. This makes assessing cost-scaling alongside performance-scaling critical when considering Total Cost of Ownership (TCO). Thankfully, per-second billing on Google Cloud makes this kind of analysis a little bit easier. As shown above, a second doubling of the count from 960 cores to 1920 cores can provide an additional 1.5x speedup, but at a 32% higher cost. In some circumstances, this faster turnaround may be needed and worth the extra cost.
If you want to get started with WRF quickly and experiment with the CONUS 2.5km benchmark, we’ve encapsulated this deployment in Terraform scripts and prepared an accompanying codelab.
You can learn more about Google Cloud’s high performance computing offerings at https://cloud.google.com/hpc, and you can find out more about Google’s partner Fluid Numerics at https://www.fluidnumerics.com.
What You Can Learn from ‘Up or Out’ Framework for Cloud Adoption
3239
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
In times of significant disruption, organizations are faced with three choices: Retrench within legacy solutions, pause and do nothing while waiting for more data or different circumstances, or press ahead, potentially even accelerating to realize the desired outcome. In such an environment, it is critical to ensure you’re delivering the greatest possible impact to the business.
In Google Cloud’s Office of the CTO, or OCTO, we have the privilege of co-innovating with customers to explore what’s possible and how we can re-imagine and solve their most strategic challenges. These collaborative innovation engagements are often core to critical transformational projects, which often include the rehosting, evolution, and at times re-architecture of existing business solutions.
We are happy to offer this brand new white paper, where we have distilled the conversations we’ve had with CIOs, CTOs, and their technical staff into several frameworks that can help cut through the hype and the technical complexity, to help devise the strategy that empowers both the business and IT. We called one such framework “up or out.” (And we don’t mean some consulting firm’s hard-nosed career philosophy.)
One model that we found can help enterprises chart their cloud adoption journey delineates cloud migration along two axes—up and out, and we’ll cover this in much greater detail in the white paper itself.
As you can see, there isn’t a single path to the cloud—not for individual enterprises and not even for individual applications. The up or out framework can help an IT organization and its leadership characterize how they can best benefit from migrating their services or workloads. The framework acts as a general pattern that highlights the continuum of approaches to explore, and you can learn all about it by downloading this detailed white paper.
Or, if you’re really ready to jump start your migration today, you can take advantage of our current offer by signing up for a free discovery and assessment.
6470
Of your peers have already watched this video.
1:30 Minutes
The most insightful time you'll spend today!
Value Realization with Google Cloud for Retail SAP Data
Retail engagements have changed drastically over the last few years, and by the virtue of COVID-19 pandemic, retail data and customers’ expectations plummeted. To drive value and transformation across the entire value-chain retailers can make most of Google Cloud’s secure, reliable IaaS by migrating their SAP systems and taking advantage of integrations, insights and innovations. In times of change retail companies can gain maximum visibility of SAP data unlocking Google Cloud’s infrastructure modernization and Big Data and analytics capabilities. Watch the video to understand how Google Cloud and SAP partnership is a golden handshake for retail businesses’ future.
Speed Up Data-driven Innovation in Life Sciences with Google Cloud
4303
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
The last few years have underscored the importance of speed in bringing new drugs and medical devices to market, while ensuring safety and efficacy. Over this time, healthcare and life sciences organizations have transformed the way they research, develop, and deliver patient care by embracing agility and innovation.
Now, the industry is set to reap the benefits of cloud technology and overcome the existing barriers to innovation.
What’s holding back innovation?
Costly clinical trials: The process of trialing and developing new drugs and devices is still long and costly, with more than 1 in 5 clinical trials failing due to a lack of funding.1 The high failure rate comes as no surprise when you consider the average clinical trial costs $19 million and takes 10-15 years (through all 3 phases) to be approved.2
Stringent security requirements: Pre-clinical R&D and clinical trials use large volumes of highly sensitive patient data – making the life sciences industry one of the top sectors targeted by hackers.3 On top of this, the FDA and other regulatory bodies have strict requirements for medical device cybersecurity.
Unpredictable supply chains: Global supply chains are becoming increasingly complex and unpredictable. This can be brought on by anything from supply shortages, to geo-political events, and even bad weather. Making things worse is the lack of visibility into medical shipment disruptions – so when disaster strikes you’re often caught off guard.
Google Cloud for life sciences
At Alphabet, we’ve made significant investments in healthcare and life sciences, helping to tackle the world’s biggest healthcare problems, from chronic disease management, to precision medicine, to protein folding.
Together with Google, you can transform your life sciences organization and deliver secure, data-driven innovation across the value chain.
- Accelerate clinical trials to deliver life-saving treatments faster and at less cost. Clinical trials require relevant and equitable patient cohorts that can produce clinically valid data. Solutions like DocAI can enable optimal patient matching for clinical trials, helping organizations optimize clinical trial selection and increase time to value. How that patient data is collected is also important. Collection in a physician’s office captures a snapshot of the participant’s data at one point in time and doesn’t necessarily account for daily lifestyle variables. Fitbit, used in more than 1,500 published studies–more than any other wearable device–can enrich clinical trial endpoints with new insights from longitudinal lifestyle data, which can help improve patient retention and compliance with study protocols. We have introduced Device Connect for Fitbit, which empowers healthcare and life sciences enterprises with accelerated analytics and insights to help people live healthier lives. We are able to empower organizations to improve clinical trials in key ways:
- Enable clinical trial managers to quickly create and launch mobile and web RWE collection mechanism for patient reported outcomes
- Enable privacy controls with Cloud Healthcare Consent API and, as needed, remove PHI using Cloud Healthcare De-identification API
- Ingest RWE and data into BigQuery for analysis
- Leverage Looker to enable quick visualization and powerful analysis of a study’s progress and results
- Ensure security and privacy for a safe, coordinated, and compliant approach to digital transformation. Google Cloud offers customers a comprehensive set of services including pioneering capabilities such as BeyondCorp Enterprise for Zero Trust and VirusTotal for malicious content and software vulnerabilities; Chronicle’s security analytics and automation coupled with services such as Security Command Center to help organizations detect and protect themselves from cyber threats; as well as expertise from Google Cloud’s Cybersecurity Action Team. Google Cloud also recently acquired Mandiant, a leader in dynamic cyber defense, threat intelligence and incident response services.
- Optimize supply chains and enhance your data to prepare for the unpredictable. With a digital supply chain platform, we can empower supply chain professionals to solve problems in real time including visibility and advanced analytics, alert-based event management, collaboration between teams and partners, and AI-driven optimization and simulation.
Ready to learn more? We’ll be taking a deep dive into each of the challenges outlined above in our life sciences video series. Stay tuned.
- National Library of Medicine
- How much does a clinical trial cost?
- Life Sciences Industry Becomes Latest Arena in Hackers’ Digital Warfare
3951
Of your peers have already downloaded this article
4:30 Minutes
The most insightful time you'll spend today!
In 2020, we’ve witnessed tremendous change not only to our daily lives, but also how businesses work.
A key lesson for CFOs and finance leaders: Those who digitized early found faster and most lasting success.
The growth we are likely to witness in 2021 is also going to need digitization, but of a different sort.
Indian enterprises need to create cloud foundations that enable their business with the speed, scale, flexibility and visibility to move quickly and be agile in order to make the most of the growth opportunities in 2021—without over-leveraging or over-burdening their organizations.
That means ensuring their businesses migrate and modernize their most important enterprise workloads—think SAP, Windows, and VMware—to Google Cloud, helping them achieve scale, flexibility and cost savings.
Recent Updates on Google Cloud EKM to Meet Customers’ Cloud Data Security
3230
Of your peers have already read this article.
2:00 Minutes
The most insightful time you'll spend today!
Google Cloud External Key Manager (Cloud EKM) lets you protect your cloud data with encryption keys that are stored and managed in a third-party key management system outside Google Cloud’s infrastructure. This allows you to achieve full separation between your encryption keys and your data stored in the cloud, making you the ultimate arbiter of access to your data. We are continuously innovating and developing the functionality of Cloud EKM, so let’s explore some recent updates we’ve made.
New functionality
Available today, we have added several much-anticipated features to Cloud EKM to help meet customer requirements:
Cloud EKM over VPC
Many customers want to incorporate an additional layer of security and reliability when connecting their key manager to the cloud. To help meet this need, we are introducing Cloud EKM support for Virtual Private Cloud (VPC) networks. This support allows Cloud EKM to connect via a secured private network, giving customers stricter control over network access to their external key manager. For more information, see Using Cloud EKM with VPC.
Support for asymmetric keys
In addition to symmetric encryption keys, Cloud EKM now recognizes both RSA as well as Elliptic Curve asymmetric keys created in a supported external key manager. With support for asymmetric keys, you can sign approvals granted via Access Approval. Asymmetric keys can add a layer of assurance when granting administrative access to customer data. You can also use the external asymmetric keys to sign data just as you would a cloud native key. For more information, see Asymmetric signing keys.
Protection level organization policy
We’ve made a new organization policy available for Cloud KMS that allows for fine-grained control over what types of keys are used. By using this org policy, you can specify that only specified KMS key types, for example EXTERNAL or EXTERNAL_VPC, may be created. This function can help meet specific requirements for separation of data or data sovereignty, ensuring only externally-managed keys are used with certain workloads. For more information, see Organization policy constraints.
Cloud EKM supports the Google Cloud services which typically store customers’ most sensitive data assets, and we are constantly adding support for more services. For example, we recently added Cloud EKM support for Cloud Storage, allowing customers to leverage Google-scale storage while adhering to local regulations and holding their keys in their own key manager. For a complete list, see our currently supported services, and if you’re interested in using Cloud EKM with a GCP service that is not yet supported, you can make feature suggestions here.
Best practices for Cloud EKM
The newly published Reference architectures for reliable deployment of Cloud EKM services guide provides recommendations for running a highly available and reliable external key manager integrated with Cloud EKM. These recommendations answer some of the most common questions and concerns we’ve heard from customers. The recommendations are aimed at operators of an external key manager, meaning that if a supported partner operates your EKM, you might share some of these responsibilities with a partner, depending on the design of their product and how it integrates with Cloud EKM.
Take encryption into your own hands
Being deliberate about encryption is critical for securing your sensitive data on Google Cloud. We’re always evolving our encryption products to meet your needs and help you achieve your business goals, and we hope that the additional features mentioned in this blog will allow you to make better use of your key management infrastructure. To get started with Cloud EKM, check out our documentation to learn more or try it for yourself in the GCP console.
More Relevant Stories for Your Company
Earth Week: Google Cloud at the Heart of Sustainability
Today’s Google Doodle reminds us of the enormous changes our planet is experiencing due to climate change. Everyone, from businesses to governments to technologists, has the opportunity to meet this challenge — transforming themselves and their organizations to be more sustainable. For this Earth Day 2022, and indeed Earth Week,
Haaretz on Google Cloud Guarantees Faster, Reliable & Responsive Services to its Audience
Israeli centenarian newspaper, Haaretz relied on on-prem infrastructure to serve readers digitally. As the need for scalability, security and business intelligence grew alongside their readership, Haaretz was looking for more than just a cloud-based solution to replace their infrastructure. Inon Gershovitz, CTO, Haaretz takes us through the journey of recreating
How Ambrook and Google Cloud Ensure Sustainability and Profits for Farmers
Ambrook is one of the leading innovators in the agriculture sector that offers farm accounting and management software to help farmers grow finances and resources to be more sustainable and profitable. In this video session with Google Cloud, learn how Ambrook broke the gap between profitability and sustainability in the
Sky News Looks to Google Cloud to Live Stream Election Results
Sky News live streamed the results from 150 of the 650 constituency counts in the U.K. while competitors, who did not have live video from as many counts, had to wait for slower independent data services to report the results. Sky News also delivered all the live streams over the
