Docker + GPUs in Google HPC toolkit?

[yaroslavvb]

Machine Learning often needs Docker + GPU. Any tips how to get this kind of configuration in the HPC toolkit?

Here's how Oracle does it: https://github.com/oracle-quickstart/oci-hpc

[tpdownes]

To provide an update on GitHub based on a direct discussion.

We've previously integrated enroot and Pyxis and had success running CPU jobs via Slurm on Debian 11. However, the CUDA Toolkit is not being initialized properly inside the containerized environment, making the GPU unavailable. I just tested an alternative approach using Docker and the NVIDIA Container Toolkit and saw initial success outside of Slurm job environment. I will update here when I have further news about execution within Slurm.

[issacg]

I've been doing a successful PoC based on GKE using the GKE modules to provision the infrastructure over the past few months, which I'm very happy with. However, I'll mention that I've been using a 3rd party (commercial) product to provide the interface to researchers to submit jobs to the cluster.

[cboneti]

@yaroslavvb I believe this can now be closed with the recommendation to create a docker.socket file that is writable by your users or adding the users to a docker group.

Please feel free to reopen this discussion if this is not the case.

Thanks!