High Performance Computing (HPC) is prevalent today across many industries, including financial services, life sciences, higher education research, manufacturing, and energy. More and more businesses are deploying HPC workloads in the cloud to take advantage of its elasticity, scalability, and availability. Job schedulers are critical for HPC applications given the nature of these workloads that create, process, and tear down thousands, and sometimes millions, of vCPU and network resources with TB to PB of storage capacity. Job scheduling tools lead to improved operational efficiency and a degree of certainty that a particular HPC job, which can run for hours to weeks, will complete successfully. 

IBM Spectrum LSF is used extensively in the manufacturing and semiconductor industry to manage Electronic Design Automation (EDA) workloads. Dynamically running workloads on-premises and in the cloud, also known as cloud bursting, is becoming a more common practice to address capacity and provisioning time constraints within data centers and enable enterprises to take advantage of virtually unlimited resources. 

However, the challenge with cloud bursting is integrating and maintaining operational consistency across on-premises and cloud environments. IBM Spectrum LSF, in combination with Google Cloud, addresses this problem head on. 

Google Cloud is excited to announce, in collaboration with IBM, enhanced capabilities to IBM Spectrum LSF that enables organizations to integrate their on-premises job scheduling scripts with resources deployed in Google Cloud. Customers are now able to fully leverage Google Cloud’s highly scalable and secure Compute Engine, networking and storage infrastructure. 

The LSF-Google Cloud resource connector patch supports key Google Cloud differentiators including Local SSDs, GCE instance templates, Preemptible VMs, and more: 

• Bulk API support– Deploy large fleets of VM instances in a matter of seconds.
• Instance Templates – Simplify VM configuration by creating reusable templates. 
• All Machine Types – Supports all GCE VM families and machine types, including Custom Machine Types.
• GPUs – Attach up to 16 GPUs per instance, including the largest A2 instances with up to 16 NVIDIA A100 GPUs
• Preemptible VMs – Preemptible VMs are provisioned from excess Compute Engine capacity and is a significant way to save money on GCE resources.
• Local SSD – Attach up to 9TB of NVMe SSD per instance
• Hyperthreading – Supports the “threads-per-core” option in GCE, which allows per-VM hypervisor level Hyperthread configuration (when supported by Instance Templates)
• Images – Supports custom disk images, including full support for Windows, for all attached Persistent Disks
• Placement Policies – Control where the instances are physically located relative to each other within a zone for improved low-latency performance
• Labels – Supports GCE Labels, which can be used for management of firewall rules, tracking billing, etc
• Minimum CPU Platform – Supports the ability to specify a minimum CPU Platform for your Virtual Machines.

Getting Started

This improvement to the IBM LSF Resource Connector was developed by IBM in coordination with Google. Find out more about the new supported features and their operation in the official IBM Spectrum LSF Resource Connector Documentation. You can also find additional documentation and download the software in the IBM Spectrum Computing Community. If you have further questions, you can contact IBM, or Google Cloud Sales.

Special thanks to Annie Ma-Weaver, Mark Mims, and Wyatt Gorman for their contributions.

At Google Cloud, we want to provide you with the access to all the tools you need to grow your business. Through the Google Cloud Technical Guides for Startups, leverage industry leading solutions with how-to video guides and resources curated for startups.

This multi-series contains 3 chapters: Start, Build and Grow, which matches your startup’s journey:

• The Start Series: Begin by building, deploying and managing new applications on Google Cloud from start to finish.
• The Build Series: Optimize and scale existing deployments to reach your target audiences.
• The Grow Series: Grow and attain scale with deployments on Google Cloud.

Additionally, at Google we have the Google for Startups Cloud Program, which is designed to help your business get off the ground and enable a sustainable growth plan for the future. The start of the Build Series delineates the benefits of the program, the application process, and more to help your business get started on Google Cloud.

A quick recap of the Build Series

Once you have applied for the Google for Startups Cloud Program, there’s so much to explore and try out on Google Cloud.

Figuring out a rapid but solid application development process can be key to many businesses in reducing time to market. Furthermore, learning what database to use to handle application data can be tricky. Deep dive into our Firestore video which walks through how Firestore can help you unlock application innovation with simplicity and speed.

We then move on to a deep dive into BigQuery and how it can help businesses. BigQuery is designed to support analysis over petabytes of data regardless of whether it’s structured or unstructured. This video is the goto video for getting started on BigQuery!

If you are someone looking to run your Spark and Hadoop jobs faster and on the cloud, look to Dataproc. To learn more about Dataproc and how this has helped other customers with their Hadoop clusters, click the video below to learn all things Dataproc related.

Next, we find out what Dataflow can bring to your business; some advantages, sample architectures, demos on the console, and how other customers are using Dataflow.

We also talked about Machine Learning, starting from selecting the right ML solution to Machine Learning APIs on cloud to exploring Vertex AI. Following that we look into API management in Google Cloud and how Apigee helps operate your APIs with enhanced scale, security, and automation.

We ended the series with the last two episodes focusing around security deep-dive and using Cloud Tasks and Cloud Scheduler.

Coming up next – The Grow Series

Dive into the next chapter of this multi-series, with our upcoming Grow Series, where we will be focusing on growing and attaining scale with deployments on Google Cloud.

Check out our website and join us by checking out the video series on the Google Cloud Tech channel, and subscribe to stay up to date. 

See you in the cloud!

Editor’s note: In this blog post we learn how TapClicks migrated to Google Cloud to offer their marketing customers a unified platform for data management, operations, insights, and analysis.

TapClicks is a smart marketing cloud, powered by data, that unifies our customer’s marketing. By choosing to migrate our core applications last year to Google Cloud, we cut costs, solved data-sharing concerns for our customers, and opened our stack up to a new ecosystem of possibilities. 

The core problem that we’re solving for our customers is how to manage their marketing infrastructures data and operations. Life isn’t easy for marketers now. There are 7,000 different vendors servicing this space today – creating much complexity between digital agencies, media, and brands. Marketers face challenges in navigating all of these systems, logging in and out, understanding pacing goals, and managing the flow of marketing data so they can analyze and report internally as well as to their clients at scale.

We unify omnichannel campaign data (250 API connectors and 6000 Smart Connectors ™ ) from a plethora of marketing sources on an automated data warehousing solution, creating simplicity for organizations. Over 4,000 agencies, media companies, and brands use our Marketing Operations and Data Management Platform, which imports data at scale and creates an automatic data warehouse on Google Cloud. Teams can also leverage TapClicks, like our world class Facebook connector, to import data directly into Google Data Studios.  Beyond importing and storing, we also provide data exporting to other Google solutions like Google Data Studio and Google Sheets.  We also create interactive dashboards that let stakeholders and clients analyze their data, as well as automated, multi-channel reports that go out to clients at specified times. So channel comparisons, optimizations, attribution, and calculations are easily performed.  Some of our customers are able to generate hundreds of thousands of individual reports and dashboards for their clients.

Although we may be best known for our reporting and analytics, we also empower teams managing the marketing operations workflow from customers and internal stakeholders, especially at scale. Our user-friendly, configurable system helps manage their orders and campaigns. Through automation of this process, we deliver tremendous amounts of efficiency, time saving, cost savings, and reduction of errors. The combination of these solutions makes up our unified platform, with additional capabilities like marketing intelligence that offers competitive and brand-level analysis. This is a disruptive solution in use by all leading media companies, agencies and many brands.

Partnering for possibilities

We faced a few challenges with our original tech stack, which included a mix of the leader in web services revenue, leaders in high performance data warehousing, as well as vendors on bare metal servers. 

• One challenge was around costs, which were growing. 
• Second, many of our customers work with multiple brands, and are very hesitant to share their data with the leader in web services, who’s often viewed as their competitor. 
• Third, these vendors are more focused on their own revenue rather than a true long term partnership that would enable their customers to enjoy similar success as they have experienced.

When looking at other cloud providers, Google Cloud emerged for us as the front runner. They were competitive on costs, and their native Kubernetes support was superior— a big selling point for our DevOps team. There’s also a movement in the marketing and advertising industry away from AWS toward Google Cloud because of the data-sharing concern. Finally, most of our customers are already using Google Cloud tools, so there’s brand recognition and familiarity there, and easier integrations with their own systems.

Migrating to Google Cloud

Our migration, which took about five months, involved moving a significant chunk of our infrastructure, including our core applications, using Google Kubernetes Engine (GKE). In our legacy architecture, each of our clients was assigned to one of our virtual machines (VMs), and there was a lot of unused capacity because we had to provision for the max usage. We appreciated GKE’s cloud native capabilities, especially autoscaling, a huge benefit for our web application. We have varying usage patterns during the day, and though our application is mostly used during business hours, there are also days in the month of higher usage, and autoscaling saves us time and costs. GKE also makes deployments much easier, and we anticipate a lot of benefits there for our developer environments. We’ve moved some of our microservices into GKE and plan to move more in the future. All in all, we were able to migrate our core products and the bulk of our AWS spend successfully to Google Cloud. 

We also moved from our other vendors Relational Database Service (RDS) to running MySQL on our own VMs on Google Cloud, which gives us more flexibility in terms of settings and fine tuning. We’re still trying to find the best mix as we’re modernizing our infrastructure, and we took this opportunity to migrate from MySQL 5.7 to 8.0.  

Our next stage is exploring more of the capabilities and services of Google Cloud, including BigQuery, which we’re considering for our own data warehouse. The fact that we could also run Snowflake on Google Cloud, if needed, was another selling point for our migration. 

We’re especially interested in BigQuery ML’s machine learning and natural language processing capabilities, which enabled better predictive insights. Our customers want insights from their campaigns— which are working, which are paying off, where should they invest next? Using our platform, they’re looking not only to generate reporting, but also identify opportunities to improve campaign performance. We plan to use AI and ML to improve those capabilities, so that our customers can seamlessly unlock insight and intelligence from their marketing data and campaigns.

Double-clicking on Google Cloud

For us, being able to deeply leverage and partner with Google Cloud to deliver those solutions on a single stack is critical, and we think our customers will love it. We see TapClicks and Google Cloud partnering at a level beyond what you typically see in a cloud provider relationship. Already, fifty percent of our company is working with various Google Cloud solutions, and we envision TapClicks and Google Cloud as extensions of each other, providing a single, powerful platform solution. 

Google Cloud understands the partnership concept, and their team was able to shine a light on their services and what they could bring to the table. Compared to our previous experiences, dealing with the Google Cloud team has been a true pleasure. Now that we’ve migrated, we’re ready to take our next steps into the services available to us in the Google Cloud ecosystem, and the problems we’ll continue to solve for our customers. Learn more about TapClicks and BigQuery ML.