Build a CI Pipeline for Containerized GPU Accelerated Applications

Learn how to use Google Cloud Build, Container Registry, and a cloud native auto-scaling HPC cluster with Singularity to create a continuous integration pipeline that can execute build and run tests for applications with GPU acceleration. This tutorial-by-example builds the foundation of a templatized approach for automated HPC application testing by leveraging Google Cloud resources. This framework will allow you to test HPC applications that require 1000's of cores and multi-GPU platforms for CI testing. We will pick up on last week's livestream (    • Video   ) and extend our CI infrastructure to incorporate the auto-scaling HPC cluster.

This tutorial will use the Spectral Element Libraries in Fortran ( github.com/FluidNumerics/SELF) as an example GPU accelerated application that we will integrate into the CI platform. We will discuss a few modifications you may need to make to your application repository to incorporate CI with Cloud Build, Docker, Singularity, and an auto-scaling HPC cluster on Google Cloud.

For the auto-scaling HPC cluster, we will be using the latest release of the fluid-slurm-gcp solution on Google Cloud ( console.cloud.google.com/marketplace/product/fluid… )