Throughout the past couple of years, I have helped a good number of companies, big and small, migrate their systems to the Google Cloud Platform (aka GCP). During the course of these migrations, there are always a few of those moments where people look at a specific Google Cloud feature and say, “now, that’s cool!”.

More often than not, it is because, coming from other platforms, they have gotten used to some features requiring multiple steps, or some operations being complicated, etc. And often they find out that in GCP you can do this specific operation in a couple of clicks, or by setting up a simple text-based configuration. Then you see that light bulb turning on in their head, and there you go… happy customer.

A few of these happen so often that I compiled them in a list to share with others who might also benefit from these “aha!” moments. You could say these are the five things I wish they told me when I started using Google Cloud.

Projects: Naturally Group Resources Together

A project is a namespace where resources live. Every resource you instantiate in GCP, from load balancers to Kubernetes clusters to virtual machines, belongs to a single project, and has no access (by default) to resources in other projects. User roles and authorisations can be defined per-project and trickle down to everything in it. This has two immediate benefits: you can group things that belong together in neat logical units, and things that don’t belong together are isolated from each other (and isolation is a Good Thing)

This is powerful and quite simple, but it often takes new users off-guard. I’ve had many clients call me and ask me “How can I make sure my developers cannot access the production machines? What’s the best way to create access policies? ”

The answer to this is actually super-simple:

• have a project for development where your developers have rights,
• have a project for production where they don’t.
• That’s it.

Every machine/other resource in the production project won’t be accessible to developers.

Of course there is a lot more to it, and you can refine roles and permissions to a much greater degree using Organizations, Folders, etc. Not to mention all the crazy things you can do with per-project billing. But at least you can say “hey, if it’s a machine in the staging environment then it can be found in the “staging” project”.

Global Virtual Networks Are *Truly* Global

Imagine you are using a Cloud provider and that you have servers in the US, and servers in Singapore, and that they need to communicate.

So you create a VPC (Virtual Private Cloud) network in the US data center, another one in the Singapore data center, and then you will connect them by setting up inter-region VPC peering or a VPN (Virtual Private Network) or a transit VPC or other routing magic.

Lots of work, right? And many moving parts, so lots of opportunities for things to break.

With GCP, however, what makes my clients go “aha!” is when they realize that in GCP a single VPC network covers the entire planet. Only subnets are attached to a geographic location, and virtual machines communicate between subnets on private IPs (good old RFC1918 addresses) — no extra routing needed.

So, to make your server communicate across continents on GCP, here are the steps:

• create a VPC network
• create a subnet in the US, put your US servers in it
• create a subnet in Singapore, put your Singapore servers in it

That’s all there is to it. Your VPC network spanning 2 continents is ready to use. Below is a screenshot of how it looks on my account, for a VPC network called ‘my-global-network’ with 2 subnets. The first column (“us-central1” and “asia-southeast1”) contains the name of the GCP regions (read: data centers). The second column is the subnet name that I picked when I created them.

A machine in the US (on the “us-central” subnet) with IP 10.0.0.5 can communicate directly with a machine in Singapore (on the “singapore”) subnet with IP 10.10.0.8.

Nothing else to set up.

And thanks to the way these networks work, the Google Cloud Load Balancer can present a single IP to the world, and forward traffic to the instances that are the closest to you geographically without having to setup a tedious DNS-based load balancing. But that’s worth an entire blog. I’ll save it for another day.

Firewalls with Tags and (Almost) No IP Addresses

There is no network security without a firewall so unsurprisingly GCP comes with one built-in.

Now, I don’t know about you, but nothing makes my brain hurt like a list of firewall rules displaying IP ranges and addresses and ‘Allow/Deny’ directives. It looks a bit like this:

If you imagine a normal network with a few dozen (hundred?) servers, you can quickly see how this can get out of control. You’d better have a solid printout of your network layout to refer to when you start adding and changing rules. And good luck debugging things!

Wouldn’t it be nice if, instead, you could just tell the firewall: “the HTTP traffic from outside can only reach the HTTP servers and the MySQL database is only reachable by the HTTP server(s) on the same network?”

Turns out it’s pretty simple on GCP by using a little thing called network tags. As the documentation says:

“Network tags are text attributes you can add to Compute Engine virtual machine (VM) instances. Tags allow you to make firewall rules and routes applicable to specific VM instances.”

So let’s see how it works. Firewall rules in GCP are defined in terms of source and target (the traffic flows from the source to the target). You can define filtering rules that apply to the source or the target, and in both cases you can use tags.

This is simpler shown with an example. The rule below states that on the default network, the traffic to the VMs with the tag mysql-server can come from the VMs with the tag http-appserver. Any other traffic is “Deny”-ed by default.

All you have to do is to tag your machines properly, and they will automatically be covered by the rule. You don’t need to enter their IP range.

That’s neat if you ask me. It makes it a lot simpler to grasp what’s happening.

Of course, there’s a TON more to firewalls in GCP. Tags also apply to routes and you can mix and match IP-based rules with tag-based rules. Not to mention that thing called service accounts, but I’ll leave those for another day.

The bottom line is that you can create most rules by just expressing a business need and not having to remember complicated network layouts. I have no hard stats, but I’m pretty sure this has saved me hours of work.

Console Access to VMs from the Browser

Easily access virtual machines (VMs) from the Google Cloud console was one of my first “aha!” moments when I started using GCP.

This is a screen capture of my Google Cloud console, with a virtual machine and its internal IP.

The last column has a header that says “Connect” and when you click on the word “SSH” a separate windows pops up. You wait for a few seconds, and… this is what you get. Your personal shell access — in a browser popup no less.

You are connected through ssh to the virtual machine of your choice. You did not have to download ssh keys and put them in the ~/.ssh directory, do the correct chmod command and run a long-winded ssh -i ~/.ssh/somekey me@<it-took-me-forever-to-copy-paste-the-address-here>

In addition, you have access to a few nifty features such as uploading and downloading files, changing the user etc. Just use the menu behind the cog icon at the top right.

In truth, you should not need to connect directly that often, but when you have to, this is a godsend.

Your Personal Jumphost from the Google Cloud Console

The Google Cloud console has a cool trick: you can actually connect to a virtual environment that is managed by the Google Cloud console itself. It serves a bit as a jump host. You can access most resources from the projects from it, and you can activate it directly from the top menu with, no particular setup on your side. It’s called the Cloud Shell.

This is how it looks at the top right of the console:

When you activate the Cloud Shell, the session opens at the bottom of the console. You get a command line prompt and it’s fully configured with the gcloud command line tool (the jack-of-all-trades of Google Cloud scripting).

You can do a great many things from there, and this even includes uploading and downloading files, editing code or deploying it, a web preview for your AppEngine application, and more.

So you can get access to a fully configured shell environment in your project from any laptop where you can connect with your credentials. On top of this, it persists between connections so you can fine-tune it to your needs and have these changes available the next time you re-connect.

This has saved me many times during my previous life as a traveling consultant!

Live migration

Did I say 5 “Aha!” moments ? Well, you’ve been patient reading all the way to here, so here’s one more for free.

Google Cloud has an amazing way to literally “teleport” a running virtual machine between physical hosts without stopping it. It’s called Live Migration. It allows Google to move your virtual machine away from a defective host, or a host that needs a patch or an upgrade, or for any other infrastructure related reason.

It’s all done in the background, and is totally transparent, so you never really see it happening. Unless you look VERY closely. I once did a demo to a client, where a machine was live migrated while he was simulating a solid network load — and we did not lose a single packet, with no noticeable degradation in latency.

And that’s a wrap!

So there you go. These are 5+1 things that made me go “Aha!” when I became more familiar with the Google Cloud Platform, and that still make my clients do the same.

There is a lot of depth to the platform, and my examples above only scratch the surface of our features. I encourage you to try it yourself. There is a generous free tier, and when you are ready to take the plunge and create that new company, please contact us at Google Cloud for Startups. We’ll get you up and running in no time.

Jerome is a Startup Architect at Google Cloud. Based in Singapore, he helps startups make the most of the Google Cloud Platform.

Weather forecasting and climate modeling are two of the world’s most computationally complex and demanding tasks. Further, they’re extremely time-sensitive and in high demand — everyone from weekend travelers to large-scale industrial farming operators wants up-to-date weather predictions. To provide timely and meaningful predictions, weather forecasters usually rely on high performance computing (HPC) clusters hosted in an on-premises data center. These on-prem HPC systems require significant capital investment and have high long-term operational costs. They consume a lot of electricity, have largely fixed configurations, and the underlying computer hardware is replaced infrequently.

Using the cloud instead offers increased flexibility, constantly refreshed hardware, high reliability, geo-distributed compute and networking, and a “pay for what you use” pricing model. Ultimately, cloud computing allows forecasters and climate modelers to provide timely and accurate results on a flexible platform using the latest hardware and software systems, in a cost effective manner. This is a big shift compared with traditional approaches to weather forecasting, and can appear challenging. To help, weather forecasters can now run the Weather Research and Forecasting (WRF) modeling system easily on Google Cloud using the new WRF VM image from Fluid Numerics, and achieve the performance of an on-premises supercomputer for a fraction of the price. With this solution, weather forecasters can get a WRF simulation up and running on Google Cloud in less than an hour!

A closer look at WRF
Weather Research and Forecasting (WRF) is a popular open-source numerical weather prediction modeling system used by both researchers and operational organizations. While WRF is primarily used for weather and climate simulation, teams have extended it to support interactions with chemistry, forest fire modeling, and other use cases. WRF development began in the late 1990s through a collaboration between the National Center for Atmospheric Research (NCAR), National Oceanic and Atmospheric Administration (NOAA), U.S. Air Force, Naval Research Laboratory, University of Oklahoma, and the Federal Aviation Administration. The WRF community comprises more than 48,000 users spanning over 160 countries, with the shared goal of supporting atmospheric research and operational forecasting.

The Google Cloud WRF image is built using Google’s MPI best practices for HPC, with the exception that hyperthreading is not disabled by default, and is easily integrated with other HPC solutions on Google Cloud, including SchedMD’s Slurm-GCP. Normally, installing WRF and its dependencies is a time consuming process. With these new WRF VM images, deploying a scalable HPC cluster with WRF v4.2 pre-installed is quick and easy with our Codelab. OpenMPI 4.0.2 was used throughout this work. Google has had good success with Intel MPI, and we intend to study whether further performance gains can be achieved in this context.

Optimizing WRF

Determining the optimal architecture and build settings for performance and cost was a key part of the process in developing the WRF images. We evaluated how to select the ideal compiler, right CPU platform, and the best file system for handling file IO, so you don’t have to. As a test case for assessing performance, we used the CONUS 2.5km benchmark.

Below, the CONUS 2.5km runtime and cost figure shows the run time required for simulating WRF over a two-hour forecast using 480 MPI ranks (a way of numbering processes) for different machine types available on Google Cloud. For each machine type, we’re showing the lowest measured run time from a suite of tests that varied compiler, compiler optimizations, and task affinity.

We found that compute-optimized c2 instances provided the shortest run time. The Slurm job scheduler allows you to map the MPI tasks to compute hardware using task affinity flags. When optimizing the runtime and cost for each machine type, we compared using srun –map-by core –bind-to core to launch WRF, which maps each MPI process to a physical core (two vCPU per MPI rank), and srun –map-by thread –bind-to thread, which maps each MPI process to a single vCPU. Mapping by core and binding MPI ranks to cores is akin to disabling hyperthreading.

The ideal simulation cost and runtime for CONUS 2.5km for each platform is found when each MPI rank is subscribed to each vCPU. When binding to vCPUs, half as many compute resources are needed when compared to binding to physical cores lowering the per-second cost for the simulation. For CONUS 2.5km, we also found that although mapping MPI ranks to cores results in reduced runtime for the same number of MPI ranks, the performance gains are not significant enough to outweigh the cost savings. For this reason, the WRF-GCP solution does not disable hyperthreading by default.

Runtime and simulation cost can be further reduced by selecting an ideal compiler: the figure below (CONUS 2.5km Compiler Comparisons) shows the simulation runtime for the WRF CONUS 2.5km benchmark on eight c2-standard-60 instances, using GCC 10.30, GCC 11.2.0 and the Intel® OneAPI® compilers (v2021.2.0). In all cases, WRF is built using level 3 compiler optimizations and Cascade Lake target architecture flags. By compiling WRF with the Intel® OneAPI® compilers, the WRF simulation runs about 47% faster than the GCC builds, and at about 68% of the cost, on the same hardware. We’ve used OpenMPI 4.0.2 with each of the compilers as the MPI implementation in this work. With other applications, Google has seen good performance with Intel MPI 2018, and we intend to investigate performance comparisons with this and other MPI implementations.

File IO in WRF can become a significant bottleneck as the number of MPI ranks increases. Obtaining the optimal file IO performance requires using parallel file IO in WRF and leveraging a parallel file system such as Lustre.

Below, we show the speedup in file IO activities relative to serial IO on an NFS file system. For this example, we are running the CONUS 2.5km benchmark on c2-standard-60 instances with 960 MPI ranks. By changing WRF’s file IO strategy to parallel IO, we accelerate file IO time by a factor of 60.

We further speed up IO and reduce simulation costs by using a Lustre parallel file system deployed from open-source Lustre Terraform infrastructure-as-code from Fluid Numerics. Lustre is also available with support from DDN’s EXAScaler solution in the Google Cloud Marketplace. In this case, we use four n2-standard-16 instances for the Lustre Object Storage Server (OSS) instances, each with 3TB of Local SSD. The Lustre Metadata Server (MDS) is an n2-standard-16 instance with a 1TB PD-SSD disk. After mounting the Lustre file system to the cluster, we set the Lustre stripe count to 4 so that file IO can be distributed across the four OSS instances. By switching to the Lustre file system for IO, we speed up file IO by an additional factor of 193, which is orders of magnitude faster than a single NFS server with serial IO.

Adding compute resources and increasing the number of MPI ranks reduces the simulation run time. Ideally, with perfect linear scaling, doubling the number of MPI ranks would cut the simulation time in half. However, adding MPI ranks also increases communication overhead, which can increase the cost per simulation. The communication overhead is due to the increased amount of communication necessitated by splitting the problem more finely across more machines.

To assess the scalability of WRF for the CONUS 2.5km benchmark, we can execute a series of model forecasts where we successively double the number of MPI ranks. Below, we show two- hour forecasts on the c2-standard-60 instances with the Lustre file system, varying the number of MPI ranks from 480 to 1920. In all of these runs, MPI ranks are bound to vCPUs so that the number of vCPUs dedicated to each simulation increases with the increase in MPI ranks. While many HPC workloads run best with simultaneous multithreading (SMT) disabled, we find the best performance for CONUS 2.5km with SMT enabled. Thus, the number of MPI ranks in our runs equals the total number of vCPUs.

As you can see, the CONUS 2.5km Runtime & Cost Scaling figure shows that the run time (blue bars) decreases as the number of MPI ranks and the amount of compute resources increases, at least up to 1920 ranks. When transitioning from 480 to 960 MPI ranks, the run time drops, yielding a speedup of about 1.8x. Doubling again to 1920 MPI ranks, though, we obtained an additional speedup of just 1.5x. This declining trend in the speedup with increasing MPI ranks is a signature of MPI overhead, which increases with more MPI ranks.

Determining your best fit

Most tightly-coupled MPI applications such as WRF exhibit this kind of scaling behavior, where scaling efficiency decreases with increasing MPI ranks. This makes assessing cost-scaling alongside performance-scaling critical when considering Total Cost of Ownership (TCO). Thankfully, per-second billing on Google Cloud makes this kind of analysis a little bit easier. As shown above, a second doubling of the count from 960 cores to 1920 cores can provide an additional 1.5x speedup, but at a 32% higher cost. In some circumstances, this faster turnaround may be needed and worth the extra cost.

If you want to get started with WRF quickly and experiment with the CONUS 2.5km benchmark, we’ve encapsulated this deployment in Terraform scripts and prepared an accompanying codelab.

You can learn more about Google Cloud’s high performance computing offerings at https://cloud.google.com/hpc, and you can find out more about Google’s partner Fluid Numerics at https://www.fluidnumerics.com.

As Google Cloud has become a choice for more startups, I’ve experienced an increase in founders asking how they should think about cloud services. Though each startup is different and requirements may vary across industries and regions, I’ve seen a few core best practices that help startups to succeed—as well as several traps to avoid. 

For example, If you go with a public cloud provider, you’re ideally not starting from ground zero like you would running your own data center, but it’s important not to introduce similar complexity in a virtualized environment. Just because you’re using public cloud infrastructure doesn’t mean you want to manage it.

Instead, you want to leverage platforms that abstract away complexity so your team can focus on delivering value to customers. That’s where serverless comes in. Serverless platforms are fully managed by the provider, offering automatic scaling for workloads, as well as provisioning, configuring, patching, and management of servers and clusters. Freed from these resource-intensive tasks, your technical talent can focus on the things that differentiate your business, not on IT curation. 

This applies to not only running stateless applications, but also managing and analyzing data. You should be collecting data points about which features are most popular on your platform, what people are buying, the types of activities that help your customers get their jobs done, and so on. This information is essential to building and executing on a product roadmap that will serve your customers. However, it’s not enough to collect this data, you need to make your data accessible, secure, and easy for your team to analyze—so where do you run and host it? 

On Google Cloud, this is where options like Spanner and Cloud SQL can play a large role, as can BigQuery for analysis. You won’t have to worry about standing up infrastructure or patching servers—you can just stream your data to our data management platforms, where it’s available whenever someone needs to run a query. By leveraging a serverless architecture, your startup can be data-driven without having to invest in the traditional complexity of database administration and management—and that can significantly change your playing field. 

Serverless is just one of the factors you should consider as you build out your tech stack. To hear my thoughts on a range of other topics relevant to startups — such as security, cloud credits, and the differences among managed services — check out the below video or visit our Build and Grow page.

https://www.youtube.com/embed/gn4RjLbs8Hc?enablejsapi=1&

Editor’s note: In this blog post we learn how TapClicks migrated to Google Cloud to offer their marketing customers a unified platform for data management, operations, insights, and analysis.

TapClicks is a smart marketing cloud, powered by data, that unifies our customer’s marketing. By choosing to migrate our core applications last year to Google Cloud, we cut costs, solved data-sharing concerns for our customers, and opened our stack up to a new ecosystem of possibilities. 

The core problem that we’re solving for our customers is how to manage their marketing infrastructures data and operations. Life isn’t easy for marketers now. There are 7,000 different vendors servicing this space today – creating much complexity between digital agencies, media, and brands. Marketers face challenges in navigating all of these systems, logging in and out, understanding pacing goals, and managing the flow of marketing data so they can analyze and report internally as well as to their clients at scale.

We unify omnichannel campaign data (250 API connectors and 6000 Smart Connectors ™ ) from a plethora of marketing sources on an automated data warehousing solution, creating simplicity for organizations. Over 4,000 agencies, media companies, and brands use our Marketing Operations and Data Management Platform, which imports data at scale and creates an automatic data warehouse on Google Cloud. Teams can also leverage TapClicks, like our world class Facebook connector, to import data directly into Google Data Studios.  Beyond importing and storing, we also provide data exporting to other Google solutions like Google Data Studio and Google Sheets.  We also create interactive dashboards that let stakeholders and clients analyze their data, as well as automated, multi-channel reports that go out to clients at specified times. So channel comparisons, optimizations, attribution, and calculations are easily performed.  Some of our customers are able to generate hundreds of thousands of individual reports and dashboards for their clients.

Although we may be best known for our reporting and analytics, we also empower teams managing the marketing operations workflow from customers and internal stakeholders, especially at scale. Our user-friendly, configurable system helps manage their orders and campaigns. Through automation of this process, we deliver tremendous amounts of efficiency, time saving, cost savings, and reduction of errors. The combination of these solutions makes up our unified platform, with additional capabilities like marketing intelligence that offers competitive and brand-level analysis. This is a disruptive solution in use by all leading media companies, agencies and many brands.

Partnering for possibilities

We faced a few challenges with our original tech stack, which included a mix of the leader in web services revenue, leaders in high performance data warehousing, as well as vendors on bare metal servers. 

• One challenge was around costs, which were growing. 
• Second, many of our customers work with multiple brands, and are very hesitant to share their data with the leader in web services, who’s often viewed as their competitor. 
• Third, these vendors are more focused on their own revenue rather than a true long term partnership that would enable their customers to enjoy similar success as they have experienced.

When looking at other cloud providers, Google Cloud emerged for us as the front runner. They were competitive on costs, and their native Kubernetes support was superior— a big selling point for our DevOps team. There’s also a movement in the marketing and advertising industry away from AWS toward Google Cloud because of the data-sharing concern. Finally, most of our customers are already using Google Cloud tools, so there’s brand recognition and familiarity there, and easier integrations with their own systems.

Migrating to Google Cloud

Our migration, which took about five months, involved moving a significant chunk of our infrastructure, including our core applications, using Google Kubernetes Engine (GKE). In our legacy architecture, each of our clients was assigned to one of our virtual machines (VMs), and there was a lot of unused capacity because we had to provision for the max usage. We appreciated GKE’s cloud native capabilities, especially autoscaling, a huge benefit for our web application. We have varying usage patterns during the day, and though our application is mostly used during business hours, there are also days in the month of higher usage, and autoscaling saves us time and costs. GKE also makes deployments much easier, and we anticipate a lot of benefits there for our developer environments. We’ve moved some of our microservices into GKE and plan to move more in the future. All in all, we were able to migrate our core products and the bulk of our AWS spend successfully to Google Cloud. 

We also moved from our other vendors Relational Database Service (RDS) to running MySQL on our own VMs on Google Cloud, which gives us more flexibility in terms of settings and fine tuning. We’re still trying to find the best mix as we’re modernizing our infrastructure, and we took this opportunity to migrate from MySQL 5.7 to 8.0.  

Our next stage is exploring more of the capabilities and services of Google Cloud, including BigQuery, which we’re considering for our own data warehouse. The fact that we could also run Snowflake on Google Cloud, if needed, was another selling point for our migration. 

We’re especially interested in BigQuery ML’s machine learning and natural language processing capabilities, which enabled better predictive insights. Our customers want insights from their campaigns— which are working, which are paying off, where should they invest next? Using our platform, they’re looking not only to generate reporting, but also identify opportunities to improve campaign performance. We plan to use AI and ML to improve those capabilities, so that our customers can seamlessly unlock insight and intelligence from their marketing data and campaigns.

Double-clicking on Google Cloud

For us, being able to deeply leverage and partner with Google Cloud to deliver those solutions on a single stack is critical, and we think our customers will love it. We see TapClicks and Google Cloud partnering at a level beyond what you typically see in a cloud provider relationship. Already, fifty percent of our company is working with various Google Cloud solutions, and we envision TapClicks and Google Cloud as extensions of each other, providing a single, powerful platform solution. 

Google Cloud understands the partnership concept, and their team was able to shine a light on their services and what they could bring to the table. Compared to our previous experiences, dealing with the Google Cloud team has been a true pleasure. Now that we’ve migrated, we’re ready to take our next steps into the services available to us in the Google Cloud ecosystem, and the problems we’ll continue to solve for our customers. Learn more about TapClicks and BigQuery ML.