Making Weather Predictions Easy with Weather Research and Forecasting (WRF) Models on Google Cloud!
3588
Of your peers have already read this article.
6:00 Minutes
The most insightful time you'll spend today!
Weather forecasting and climate modeling are two of the world’s most computationally complex and demanding tasks. Further, they’re extremely time-sensitive and in high demand — everyone from weekend travelers to large-scale industrial farming operators wants up-to-date weather predictions. To provide timely and meaningful predictions, weather forecasters usually rely on high performance computing (HPC) clusters hosted in an on-premises data center. These on-prem HPC systems require significant capital investment and have high long-term operational costs. They consume a lot of electricity, have largely fixed configurations, and the underlying computer hardware is replaced infrequently.
Using the cloud instead offers increased flexibility, constantly refreshed hardware, high reliability, geo-distributed compute and networking, and a “pay for what you use” pricing model. Ultimately, cloud computing allows forecasters and climate modelers to provide timely and accurate results on a flexible platform using the latest hardware and software systems, in a cost effective manner. This is a big shift compared with traditional approaches to weather forecasting, and can appear challenging. To help, weather forecasters can now run the Weather Research and Forecasting (WRF) modeling system easily on Google Cloud using the new WRF VM image from Fluid Numerics, and achieve the performance of an on-premises supercomputer for a fraction of the price. With this solution, weather forecasters can get a WRF simulation up and running on Google Cloud in less than an hour!
A closer look at WRF
Weather Research and Forecasting (WRF) is a popular open-source numerical weather prediction modeling system used by both researchers and operational organizations. While WRF is primarily used for weather and climate simulation, teams have extended it to support interactions with chemistry, forest fire modeling, and other use cases. WRF development began in the late 1990s through a collaboration between the National Center for Atmospheric Research (NCAR), National Oceanic and Atmospheric Administration (NOAA), U.S. Air Force, Naval Research Laboratory, University of Oklahoma, and the Federal Aviation Administration. The WRF community comprises more than 48,000 users spanning over 160 countries, with the shared goal of supporting atmospheric research and operational forecasting.
The Google Cloud WRF image is built using Google’s MPI best practices for HPC, with the exception that hyperthreading is not disabled by default, and is easily integrated with other HPC solutions on Google Cloud, including SchedMD’s Slurm-GCP. Normally, installing WRF and its dependencies is a time consuming process. With these new WRF VM images, deploying a scalable HPC cluster with WRF v4.2 pre-installed is quick and easy with our Codelab. OpenMPI 4.0.2 was used throughout this work. Google has had good success with Intel MPI, and we intend to study whether further performance gains can be achieved in this context.
Optimizing WRF
Determining the optimal architecture and build settings for performance and cost was a key part of the process in developing the WRF images. We evaluated how to select the ideal compiler, right CPU platform, and the best file system for handling file IO, so you don’t have to. As a test case for assessing performance, we used the CONUS 2.5km benchmark.
Below, the CONUS 2.5km runtime and cost figure shows the run time required for simulating WRF over a two-hour forecast using 480 MPI ranks (a way of numbering processes) for different machine types available on Google Cloud. For each machine type, we’re showing the lowest measured run time from a suite of tests that varied compiler, compiler optimizations, and task affinity.
We found that compute-optimized c2 instances provided the shortest run time. The Slurm job scheduler allows you to map the MPI tasks to compute hardware using task affinity flags. When optimizing the runtime and cost for each machine type, we compared using srun –map-by core –bind-to core to launch WRF, which maps each MPI process to a physical core (two vCPU per MPI rank), and srun –map-by thread –bind-to thread, which maps each MPI process to a single vCPU. Mapping by core and binding MPI ranks to cores is akin to disabling hyperthreading.
The ideal simulation cost and runtime for CONUS 2.5km for each platform is found when each MPI rank is subscribed to each vCPU. When binding to vCPUs, half as many compute resources are needed when compared to binding to physical cores lowering the per-second cost for the simulation. For CONUS 2.5km, we also found that although mapping MPI ranks to cores results in reduced runtime for the same number of MPI ranks, the performance gains are not significant enough to outweigh the cost savings. For this reason, the WRF-GCP solution does not disable hyperthreading by default.
Runtime and simulation cost can be further reduced by selecting an ideal compiler: the figure below (CONUS 2.5km Compiler Comparisons) shows the simulation runtime for the WRF CONUS 2.5km benchmark on eight c2-standard-60 instances, using GCC 10.30, GCC 11.2.0 and the Intel® OneAPI® compilers (v2021.2.0). In all cases, WRF is built using level 3 compiler optimizations and Cascade Lake target architecture flags. By compiling WRF with the Intel® OneAPI® compilers, the WRF simulation runs about 47% faster than the GCC builds, and at about 68% of the cost, on the same hardware. We’ve used OpenMPI 4.0.2 with each of the compilers as the MPI implementation in this work. With other applications, Google has seen good performance with Intel MPI 2018, and we intend to investigate performance comparisons with this and other MPI implementations.
File IO in WRF can become a significant bottleneck as the number of MPI ranks increases. Obtaining the optimal file IO performance requires using parallel file IO in WRF and leveraging a parallel file system such as Lustre.
Below, we show the speedup in file IO activities relative to serial IO on an NFS file system. For this example, we are running the CONUS 2.5km benchmark on c2-standard-60 instances with 960 MPI ranks. By changing WRF’s file IO strategy to parallel IO, we accelerate file IO time by a factor of 60.
We further speed up IO and reduce simulation costs by using a Lustre parallel file system deployed from open-source Lustre Terraform infrastructure-as-code from Fluid Numerics. Lustre is also available with support from DDN’s EXAScaler solution in the Google Cloud Marketplace. In this case, we use four n2-standard-16 instances for the Lustre Object Storage Server (OSS) instances, each with 3TB of Local SSD. The Lustre Metadata Server (MDS) is an n2-standard-16 instance with a 1TB PD-SSD disk. After mounting the Lustre file system to the cluster, we set the Lustre stripe count to 4 so that file IO can be distributed across the four OSS instances. By switching to the Lustre file system for IO, we speed up file IO by an additional factor of 193, which is orders of magnitude faster than a single NFS server with serial IO.
Adding compute resources and increasing the number of MPI ranks reduces the simulation run time. Ideally, with perfect linear scaling, doubling the number of MPI ranks would cut the simulation time in half. However, adding MPI ranks also increases communication overhead, which can increase the cost per simulation. The communication overhead is due to the increased amount of communication necessitated by splitting the problem more finely across more machines.
To assess the scalability of WRF for the CONUS 2.5km benchmark, we can execute a series of model forecasts where we successively double the number of MPI ranks. Below, we show two- hour forecasts on the c2-standard-60 instances with the Lustre file system, varying the number of MPI ranks from 480 to 1920. In all of these runs, MPI ranks are bound to vCPUs so that the number of vCPUs dedicated to each simulation increases with the increase in MPI ranks. While many HPC workloads run best with simultaneous multithreading (SMT) disabled, we find the best performance for CONUS 2.5km with SMT enabled. Thus, the number of MPI ranks in our runs equals the total number of vCPUs.
As you can see, the CONUS 2.5km Runtime & Cost Scaling figure shows that the run time (blue bars) decreases as the number of MPI ranks and the amount of compute resources increases, at least up to 1920 ranks. When transitioning from 480 to 960 MPI ranks, the run time drops, yielding a speedup of about 1.8x. Doubling again to 1920 MPI ranks, though, we obtained an additional speedup of just 1.5x. This declining trend in the speedup with increasing MPI ranks is a signature of MPI overhead, which increases with more MPI ranks.
Determining your best fit
Most tightly-coupled MPI applications such as WRF exhibit this kind of scaling behavior, where scaling efficiency decreases with increasing MPI ranks. This makes assessing cost-scaling alongside performance-scaling critical when considering Total Cost of Ownership (TCO). Thankfully, per-second billing on Google Cloud makes this kind of analysis a little bit easier. As shown above, a second doubling of the count from 960 cores to 1920 cores can provide an additional 1.5x speedup, but at a 32% higher cost. In some circumstances, this faster turnaround may be needed and worth the extra cost.
If you want to get started with WRF quickly and experiment with the CONUS 2.5km benchmark, we’ve encapsulated this deployment in Terraform scripts and prepared an accompanying codelab.
You can learn more about Google Cloud’s high performance computing offerings at https://cloud.google.com/hpc, and you can find out more about Google’s partner Fluid Numerics at https://www.fluidnumerics.com.
LearningMate & Google Cloud Partnership to Aid Equitable Educational Opportunities
3452
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
A “one size fits all” approach to education no longer works in today’s classrooms. Using cloud-based technologies, schools and educators can take a more personalized approach to education–one that suits each student’s unique learning style, abilities, and needs.
Taking the lead toward more equitable educational opportunities worldwide, education technology pioneer, LearningMate, is partnering with Google Cloud to offer intelligent, personalized learning via Google Cloud’s Student Success Services.
In a student survey by EDUCAUSE, a nonprofit association whose mission is to advance higher education through the use of information technology, nearly all respondents asked for more digital learning and study options. Given a list of educational material types, such as study guides and recorded lessons, 93 percent said they would like to have online access to at least two options and more than half (56 percent) chose seven or more.
The trend towards student choice challenges educators to reconsider how they teach and support learners. Students expect the same qualities in their lessons as they encounter in their other, non-school related digital experiences: personalization; user-friendliness; and engagement. Educators who are used to a more top-down education model can struggle to meet these new expectations.
Google Cloud created Student Success Services to help education institutions meet learners where they are—in the digital age. This suite of tools and services uses Google’s advanced artificial intelligence (AI) and analytic tools to:
- Engage with students via an AI-powered learning platform and interactive tutor
- Help educators and learners collaborate more effectively
- Provide current, actionable data on student progress
To bring our Student Success Services to more schools and organizations worldwide, we’re partnering with LearningMate — a global leader in digital learning infrastructure. LearningMate is a key go-to-market partner, helping schools design, launch, and maintain their own learning infrastructure. To start, LearningMate is adding Google’s AI-powered learning platform to its Frost platform, a popular content management for education.
As the education model continues to change and digital learning becomes the norm, disadvantaged students risk getting left behind. For these learners, the “digital divide” is very real, and stands to hinder them further–not only in school, but in society and the workplace in later life.
“The gap will only widen between those with digital advantages and those who struggle to gain access to devices and network necessities,” EDUCAUSE states. To meet all these challenges, schools need a diverse mix of tools, methods, and partnerships.
To help close the digital divide among students, digital tools need to be able to scale up or down so institutions and districts of any size and budget can use them. Google Cloud designed Student Success Services with this ability in mind. By offering these services, LearningMate aims to ensure that even schools with smaller teams and fewer resources can offer individualized learning experiences to their students.
With LearningMate and Google’s nearly 40 years of combined experience in education, this partnership aims to help educators better understand their students’ engagement, performance, and preferences. To learn more about Student Success Services and our collaboration with LearningMate, watch this session from our Government and Education Summit.
Google Migration and BigQuery Brings PedidosYa Closer towards its Goal of Becoming Data-driven
7256
Of your peers have already read this article.
2:00 Minutes
The most insightful time you'll spend today!
Editor’s note: PedidosYa is the market leader for online food ordering in Latin America, serving 15 markets and over 400 cities. It’s also one of the largest brands within the German multinational company Delivery Hero SE. With over 20 million app downloads, PedidosYa provides the best online delivery experience through 71,000+ online partners, including restaurants, shops, drugstores, and specialized markets.
Having constant access to fresh customer data is a key requirement for PedidosYa to improve and innovate our customer’s experience. Our internal stakeholders also require faster insights to drive agile business decisions. Back in early 2020, PedidosYa’s leadership tasked the data team to make the impossible possible. Our team’s mission was to democratize data by providing universal and secure access while creating a comprehensive information ecosystem across PedidosYa. We also had to achieve this goal while keeping costs under control— even during the migration stage and removing operational bottlenecks.
Challenges with legacy cloud infrastructure
PedidosYa first built its data platform on top of AWS. Our data warehouse ran on Redshift, and our data lake was in S3. We used Presto and Hue as the user interfaces for our data analysts. However, maintaining this infrastructure was a daunting task. Our legacy platform couldn’t keep up with the increasing analytics demands. For example, the data stored on S3 complemented by Presto/Hue required high operational overhead. This was because Presto and our IAM (identity access management) didn’t integrate well in our legacy ecosystem. Managing individual users and mapping IAM roles with groups and Kerberos was operationally time-consuming and costly. Further, sharding access on the S3 files was far too complicated to enable seamless ACLs (access control lists).
There were also challenges with workload management. Our data warehouse had batch data loaded overnight. If one analyst scheduled a query to run during the overnight ETL (extract, transform, load) workload, it would disrupt the current ETL task. This could stop the entire data pipeline. We’d have to wait until data engineers intervened with a manual fix.
It was also difficult to understand whether a query error was due to performance issues or platform resource exhaustion. This lack of clarity affected our data analysts’ ability to autonomously improve querying efficiency. Data team members needed to manually inspect personal queries looking for performance issues. Also, the current architecture was prone to a ‘tragedy of the commons’ situation; it was seen as an unlimited and free resource. As a result, it was impossible to disentangle the infrastructure from different stakeholder teams, as all had very different needs.
The decision to modernize our data warehouse
Given the growing challenges from our legacy platform, our tech team decided to transform our analytics environment with a modern data warehouse. They required the following key criteria from their next data platform:
- Scalability – The ability to grow with elastic infrastructure.
- Cost control – Cost management and transparency. These factors promote efficiency and ownership—both key aspects of data democratization.
- Metadata management – Intuitive data platform focusing on users’ previous SQL knowledge. Plus, being able to enrich the informational ecosystem with metadata, to diminish data gatekeepers.
- Ease of management – The team needed to reduce operational costs with a serverless solution. Data engineers wanted to focus on their key roles rather than acting as database administrators and infrastructure engineers. The team also wanted much higher availability, and to reduce the impact of maintenance windows and vacuum/analysis.
- Data governance and access rights – With a growing employee base with varying data access requirements, the team needed a simple yet comprehensive solution to understand and track user access to data.
Migrating to Google Cloud
After exploring other alternatives, we concluded Google Cloud had an answer to each of our decision drivers. Google Cloud’s serverless, managed, and integrated data platform, coupled with its seamless integration across open-source solutions, was the perfect answer for our organization. In particular, the natural integration with Airflow as a job orchestrator and Kubernetes for flexible on-demand infrastructure was key.
We used Dataflow together with Pub/Sub and Cloud Functions for our data ingestion requirements, which has made our deployment process with Terraform seamless. Because we set up everything in our environment programmatically, operation time has diminished. Google Cloud reduced the deployment process from about 16 hours in our legacy platform to 4 hours. This is partly due to the friendliness of automating the deployment (such as schema check, load test, table creation, build.) process with Terraform, Cloud Functions, Pub/Sub, Dataflow, and BigQuery on GCP. Input messages processed with Dataflow allow us to abstract and plan the schema changes according to the needs of the functional team. For example, schema changes raise an alarm, and then we can modify the raw layer table schema. By doing this, we ensure that backend modifications that we don’t control do not affect upper layers.
A key reason why we picked Google Cloud was because of its advanced cost and workload management coupled with its transparent log analytics. This information gives us a complete view into any query performance issues to make improvements on the fly. Further, we achieved a significant amount of cost savings by consolidating multiple tools to BigQuery.With BigQuery, we’ve been able to reduce our total cost per query by 5x.
This was due to a number of reasons:
- Automating pipeline deployment made it much simpler to maintain the data processing processes.
- Analysts are conscious about what queries they’re running, resulting in running better, more optimized queries.
- Analysts use a Data Studio dashboard to see their queries and all the associated costs. As a result, there’s a lot more transparency for each persona.
With these changes, we can easily manage and assign costs associated with each workload with their own cost centers using specific Google Cloud projects.
Change management is always challenging. However, BigQuery is intuitive and doesn’t have a steep learning curve from Hue/Hive on SQL basics. BigQuery also allowed the team to expand its capabilities and enabled them to properly work with nested structures, avoiding unnecessary joins and improving query efficiency. Additionally, we now use Data Catalog as our unique point of truth for metadata management. This allows our team to break the data access barriers and enable federation of data across the organization. By using Airflow to orchestrate everything, we keep track of every data stream. With this information, each end user can see their regularly used data entities’ status via the dashboard. This also adds transparency to our everyday data processes.
Finally, with Google Cloud’s IAM rules applied across the different products, data sharing and access is close to a noOps experience. We have programmatically implemented access according to roles and level access within the company. This allows certain pre-validated roles to view more sensitive information. These solutions help drive a more automated data governance experience.
Up next: Google Cloud AI/ML
The new stack based on BigQuery has created significant productivity gains. Freed from the burden of operational management, PedidosYa’s data team can now focus on adding value through data tools and products.
- Our data engineers are better equipped to integrate constantly changing transactional and operational data.
- The dataOps team can automate the infrastructure and provide autonomy to the end user.
- Our data quality team can focus on bringing added value to data stakeholders.
- Data scientists and data analytics can spend more time analyzing data and less time asking data gatekeepers for data access.
PedidosYa can now democratize data access with a well-governed architecture. We are still at the beginning of our journey, but we are closer to achieving our vision of building a data-driven organization. Up next: expanding our artificial intelligence and machine learning capabilities.
Tune in to Google Cloud’s Applied ML Summit on June 10th, 2021, or listen on-demand later, to learn how to apply groundbreaking machine learning technology in your projects.
Go Green with Google’s Latest Tool and Pick the Most Sustainable Cloud Region
5678
Of your peers have already read this article.
1:30 Minutes
The most insightful time you'll spend today!
As a Google Cloud customer, your carbon footprint is already carbon neutral: Google first achieved carbon neutrality in 2007, and has been purchasing enough solar and wind energy to match 100% of its global electricity consumption since 2017. Now, Google is targeting a new sustainability goal: operating on carbon-free energy (CFE) 24/7, everywhere, by 2030.
We want to empower you to make more sustainable decisions and progress with us towards this 24/7 carbon-free future. Earlier this year, we published the carbon characteristics of our Google Cloud regions. Later, we introduced a simple tool to help you pick a Google Cloud region, taking variables like price, latency and sustainability into account. Our next question was: what’s the best way to surface that sustainability info when you’re actually picking a region for your cloud resources?
Starting today, we are indicating regions with the lowest carbon impact inside Cloud Console location selectors. Available today for Cloud Run and Datastream, you’ll see it roll out to more Google Cloud offerings over time:
Regions that feature the “Lowest CO2” and the leaf have a CFE% of at least 75%, or, if the information is not available for this region yet, a grid carbon intensity of maximum 200 gCO2eq/kWh. You can read more about how we calculate these metrics in our documentation.
Before releasing this feature, we ran experiments to measure its impact: Users who were exposed to the enhanced region picker were 19% more likely to select a “low carbon” region for their Cloud Run service—a significant lift. These results show that by displaying carbon information in context of when you make the decision of picking a region, we are helping you make more sustainable decisions.
By sharing and displaying carbon information of Google Cloud regions, together we’re making tangible progress towards our goal of a carbon-free future. Learn more about Carbon free energy for Google Cloud regions.
Recapping Google Cloud VMware Engine’s Latest Milestones
3476
Of your peers have already read this article.
3:00 Minutes
The most insightful time you'll spend today!
We’ve made several updates to Google Cloud VMware Engine in recent weeks—today’s post provides a recap of our latest milestones. Google Cloud VMware Engine delivers an enterprise-grade VMware stack running natively in Google Cloud. This cloud service is one of the fastest paths to the cloud for VMware workloads without making changes to existing applications or operating models across a variety of use-cases. These include rapid data center exit, application lift and shift, disaster recovery, virtual desktop infrastructure, or modernization at your own pace.
In fact, Mitel, a global provider of unified communications-as-a-service to 70 million business users across 100 countries, migrated 1,000 VMware instances to Google Cloud VMware Engine in less than 90 days and improved its monthly operational output four times.
In our last update, we focused on several innovative capabilities around networking, reach, and scale. Let us take a look at the highlights we released since our last installment.
Fast provisioning of a dedicated, intrinsically secure VMware private cloud
With Google Cloud VMware Engine, you can spin up a VMware private cloud in about 30 minutes. You can also scale your VMware-based infrastructure on-demand with dedicated hosts located in secure Google data centers. Let us look at what’s new:
Autoscale: The ability to elastically and programmatically manage infrastructure resources to align with business needs or what is called “right-sizing” is a core capability of an IaaS platform. With autoscale, Google Cloud VMware Engine users can leverage policy-driven automation to scale the nodes needed to meet the compute demands of the VMware infrastructure.
Autoscale:
- Addresses seasonal spikes in demand, gradual increases of utilization, or new projects being onboarded or expanded due to disaster recovery events.
- Analyzes the CPU, memory, and storage utilization to give you the controls to scale Google Cloud VMware Engine nodes up or down.
- Ensures that storage consumption does not exceed the recommended limits for maintaining the Google Cloud VMware Engine service-level agreement.
- Reduces overhead on IT teams by automating capacity monitoring and enabling sufficient availability of resources based on thresholds. Note that safeguards for maintaining minimum capacity and maximum capacity can be configured to ensure there are boundaries to the automation.
Learn how to set up Autoscale.
Mumbai region availability
Google Cloud VMware Engine is now available in the Mumbai region. This brings the availability of the service to 12 regions globally, enabling our multi-national and regional customers to leverage a VMware-compatible infrastructure-as-a-service platform on Google Cloud. For more details, please read the press release.
Enterprise-grade infrastructure
With 99.99% availability for a cluster in a single zone, fully dedicated 100 Gbps east-west networking with no oversubscription, and all nonvolatile memory express storage, Google Cloud VMware Engine provides the highest performance required for the most demanding workloads. Let us look at what’s new:
Preview – Google Cloud KMS integration: You already have the ability to bring your own keys to encrypt your vSAN datastores. With this new capability, organizations that want to eliminate the overhead of managing external key providers can leverage a Google managed key provider, using Cloud KMS. This brings increased flexibility in securing workloads and data by enabling vSAN encryption by default for newly instantiated VMware Private Clouds. This feature is currently in Preview.
HIPAA compliance: Since April, Google Cloud VMware Engine is Health Insurance Portability and Accountability Act (HIPAA) compliant. This opens the service up to healthcare organizations, that can now migrate and run their HIPAA-compliant VMware workloads in a fully compatible VMware Cloud Verified stack running natively in Google Cloud with Google Cloud VMware Engine, without changes or re-architecture to tools, processes, or applications. Read more in this blog.
NSX-T support for Active Directory: With NSX-T support for Active Directory, you can now leverage your on-premises Active Directory as one of the lightweight directory access protocol identity sources for user authentication into NSX-T manager. This extends the theme of being able to leverage your on-premises tools with Google Cloud VMware Engine. For more information, read the documentation on how to set up identity sources.
vSAN TRIM/UNMAP support: For space-efficiency, vSAN allows creating thin-provisioned disks that grow gradually as they are filled with data. However, files that are deleted within the guest operating system (OS) do not result in vSAN freeing up space allocated. To increase space efficiency, guest OS file systems have the ability to reclaim capacity that is no longer used, using TRIM/UNMAP commands. vSAN is fully aware of these commands that are sent from the guest OS and enables reclamation of previously allocated storage as free space. We have enabled TRIM/UNMAP for vSan by default in Google Cloud VMware Engine.
Simplicity in experience and operations
With Google Cloud VMware Engine, you only need to worry about your workloads—not patching, upgrading, and updating the solution layer, for fewer interoperability issues and infrastructure maintenance. IIn addition, we have pre-built service accounts to enable your third-party VMware-supported tools and solutions to work seamlessly in VMware Engine. Access to Google services privately over local connections is also natively supported, enabling enrichment of existing applications and modernization over time. Finally, this service brings the power of Google Cloud Virtual Private Cloud (VPC) design by natively providing multi-VPC, multi-region networking that’s unique. Let’s look at what’s new:
Dashboards for Day 2 operations: To speed up cloud transformation and enable efficiency, Google Cloud VMware Engine administrators can take advantage of Cloud Operations dashboards for the solution. In addition, administrators can create custom policies through cloud alerting and enable notifications via channels of their choice (SMS, email, Slack, and more). For more details on how to set up cloud monitoring, please refer to Setting up Cloud Monitoring.
For the latest updates, bookmark Google Cloud VMware Engine release notes.
Thanks to Manish Lohani, Product Management, Google Cloud; Nargis Sakhibova, Product Management, Google Cloud; and Wade Holmes, Solutions Management, Google Cloud; for their contributions to this blog post.
Neo4J & Google Cloud: Graph Data in Cloud to Address Challenges in FinServ Industry
6919
Of your peers have already read this article.
3:00 Minutes
The most insightful time you'll spend today!
Over the last decade, financial service organizations have been adopting a cloud-first mindset. According to InformationWeek, lower costs and enhanced scalability were the biggest drivers for cloud adoption in financial services, and cloud-native applications allow access to the latest technology and talent, enabling adopters to rebuild transaction processing systems capable of supporting very high volumes and low latency.
Both Neo4j and Google Cloud have been using relationship-based data representations since the beginning, and we’re dedicated to using this technology to help financial services customers drive business transformation. We are excited about the prospects of financial services (FinServ) cloud systems and believe that graph data in the cloud can help solve significant challenges in the industry.
Data Challenge #1: Risk Management and Compliance
First among the top concerns for any CIO moving to the cloud is risk management and compliance. Disconnected, uncontextualized, or stale data create opportunities for fraud and financial crimes to occur. The fact is when it comes to FinServ, the question is not “if” but rather how often an attack will occur. Unfortunately, incidents have been trending upward over the last decade, and COVID has only exacerbated this reality. Financial crimes affect the bottom line both in the remediation of these crimes and in intangibles like brand value.
Add to this the complexity of international banking, which makes “compliance” a moving target. Penalties due to noncompliance are a constant concern to any FinServ organization.
The tabular representation of information with a fixed number of columns that never change prevents a description of an ever changing world with changing characteristics. Relational databases are great if the world you describe does not move fast but have limitations when data structures are highly interlinked and not homogeneous.
Neo4j Aura on Google Cloud provides a foundation for creating dynamic, futureproof, scalable applications that adhere to the security standards and protocols today’s financial services organizations require to meet the challenges of finding and preventing bad actors. This also includes enterprise scalability; reaching over 1 Billion nodes and relationships to streamline queries and provide solutions that meet regulatory and privacy compliance across geographies. Neo4j has helped some organizations save billions of USD in fraud in the first year of deployment alone.
What makes graph technology the best choice for fraud detection use cases is that the relationships between the data-points are as important as the data-points themselves. Let’s take as an example, one John Smith approaches a multi-national banking institution to manage the primary account for his new holding corporation.
While no one has any record of John R Smith Holdings LLC, the bank’s application built on graph technology understands that there are several well-known entities owned by John Smith Holdings. The application also identifies several well-known board members who bank with this institution. Due to this relationship-driven approach, the bank now understands John R Smith is not “John Smith,” who previously attempted to open an account for his holding corporation, which had no information associated with it prior to two months ago.
Data Challenge #2 Manual Processes and Inefficiencies
The ubiquity of the cloud offers an opportunity to deploy automation at unprecedented levels to tackle the errors and inefficiencies that manual processing allows to creep into processes. When data comes from disparate, perhaps legacy systems – which may have become siloed and “untouchable” over the years – further complexity arises. As an example, if someone in sales types “John Smith” into a CRM system not knowing that John R Smith is the spelling in the customer data master, it may result in two separate and potentially conflicting records. Being able to join those records together in a mastered view helps to solve this problem. In addition, low data quality equates to an increase in risk, costs, and implementation times for new systems.
Neo4j Aura on Google Cloud provides automation and artificial intelligence (AI) that reduces manual processes and the errors that accompany them. In this graph architecture each node, which can represent a person, will have labels, relationships, and properties associated with it. This allows for the use of AI which can easily understand that John Smith in the CRM is the same John R Smith in the customer master. The information contained in Neo4j can be connected bi-directionally to ensure consistency across applications and data sources.
One of the benefits of this approach is that linking information allows organizations to keep the full value of the data, rather than forcing the data into predetermined tabular representations, with the risk of losing valuable information and insights.
Data Challenge #3: Customer Engagement and Insight
Another significant concern is the high expectations today’s customers have for every interaction. End users are accustomed to predictable experiences on their digital devices, and FinServ apps are no exception. Added to this, the “Covid economy” has driven digital adoption significantly across demographics; even among customers who might traditionally have used in-person services. This also equates to increased expectations for personalized, predictable experiences with every digital interaction. We know that latency has always been a key consideration for financial trading, but a recent ComputerWeekly study showed that every financial organization should ensure their visible latency is at 10 milliseconds or less. Customers no longer accept their broadband is at fault.
Finally, blind spots in the customer journey often result in dissatisfaction, which ultimately leads to increased churn. Without gaining actionable insights from your customers, there is no room to innovate and iterate on what they are looking for in your products and services. And this translates to losing market share and competitive advantage.
The NoSQL architecture, specifically the dynamic schema and structure of Neo4j Aura gives you the ability to take charge of your data and make changes according to your development cycles or newer data models. This equates to faster builds, more comprehensive releases and a wider, richer data-set that can be contextualized and understood instantly. Graph technology is the logical choice for building a Customer 360 application. Under this approach organizations not only get valuable insight into the individual client’s behavior and patterns, but also those of their family, friends and colleagues. This allows for stronger personalization, targeted campaigns and successful execution, resulting in increased customer satisfaction and retention levels.
Graph Technology on Google Cloud
Neo4j is a recognized leader in graph database technology and the only fully integrated graph solution on Google Cloud, helping to fill a common need for Google Cloud customers. Both Neo4j and Google Cloud are invested in continuing to grow our partnership and mutual product direction.
You can find and deploy the Neo4j graph database straight from the Google Cloud marketplace, whether you want to download the software for an on-premises deployment, use the virtual machine image, or use the hosted solution, Aura on Google Cloud, the graph database-as-a-service. In any deployment, you get the same enterprise-grade scalability, reliability, and connectivity along with successful, repeatable use cases you can rely on to resolve your particular challenges and integrated billing.
For a real-world example of how graph technology can optimize financial services, you can read our Case Study with fintech Current. Current, a leading U.S. financial technology platform with over three million members, used Neo4j Aura on Google Cloud to create a personalization engine based on client relationships.
To learn more about Neo4j Aura on Google Cloud for FinServ organizations, register for our webinar on Thursday, December 16 with Jim Webber, Chief Scientist, CTO Field Ops at Neo4j and Antoine Larmanjat, Technical Director, Office of the CTO, Google Cloud.
More Relevant Stories for Your Company
Simplify Your Modernization Journey from Windows
Microsoft and Windows on Google Cloud provides a first-class experience for Windows workloads. You can self-manage workloads or leverage managed services and use license-included images or bring your own licenses. Now, easily migrate, optimize, and modernize your Windows workloads for agility and scalability. Start with migration Migrate to increase IT
Cloud Migration and Modernization is ‘Easier Done than Said’ with Google Cloud RAMP!
The reliance on cloud compute, storage and network has accelerated with the growing usage of digital products and services. To keep up with the customer demands and deliver digital solutions, many companies are struggling through their cloud migration and modernization journeys. But not anymore, says Google Cloud with the Rapid
Confused about Cloud? Here is a Primer to get all Your Doubts Answered
When you have decided on moving workloads to the cloud, the task of choosing the right cloud platform can be a tough one with many questions looming in your mind. From which specific product to choose from the plethora of options available to how and where to store your data
Revolutionizing Cloud Computing: Introducing G2 VMs with NVIDIA L4 GPUs
Organizations across industries are looking to AI to turn troves of data into intelligence, powered by the latest advances in generative AI. Yet for many organizations, there is a barrier to adopting the latest models because they can be costly to train or serve. A new class of cloud GPUs
