Containers

Pyxis & Enroot

Pyxis is a SLURM plugin that integrates with Enroot to enable seamless container execution inside SLURM jobs.

You can think of it as:

srun --container-image=docker://... mycommand

This replaces needing to manually start docker or enroot.

Run a Simple Container Job

srun --partition=dgx-b200 \
     --container-image=docker://ubuntu:22.04 \
     cat /etc/os-release

This will:

Pull the Docker image ubuntu:22.04
Convert it to a SquashFS via Enroot
Run it via Pyxis in your SLURM job

Interactive Container Session

For an interactive shell inside a container:

srun --partition=dgx-b200 \
     --pty --container-image=docker://ubuntu:22.04 \
     bash

Useful for testing tools or exploring inside the container.

Accessing NVIDIA Containers

PARCC has access to the full NVIDIA NGC suite of software. To access this, you must setup proper authentication with their systems. Take note of when containers were created, we need CUDA 12.8 or later. A general rule of thumb is that the container should have been compiled in May 2025 or later.

Generating Keys

Open the link to the NVIDIA build page
Click the green “Get API Key” button at the top of the page
Enter your preferred email address and continue.

Already have an account:

Continue signing in until the green “Generate Key” button is displayed

Need an account:

Continue the sign-in process by creating a password and logging in. It will ask to authenticate your email. After signing in it will prompt you to create an account name, we recommend something specific to both you and the organization, i.e. “PARCC-<username>”. Finally, it will require a phone number to input an OTP used for authentication. Continue to the page with the green “Generate Key” button.

Choose a key name and expiration date, neither are required to be altered. When you are satisfied with the key press on “Generate Key”
The next pop-up should display “API Key Granted” and include your unique key. Save this key somewhere, it will be necessary to use NVIDIA tools.

Next step will be entering the key info to your Linux account for running NVIDIA applications

Implementing Keys

mkdir -p $HOME/.config/enroot

echo """
machine nvcr.io login $oauthtoken password <api-key>
machine authn.nvidia.com login $oauthtoken password <api-key>
""" >> $HOME/.config/enroot/.credentials

Edit the credentials file and replace <api-key> with the key nvidia provided. It starts with nvapi- and is specific to your user.

Containerized GPU Job with PyTorch

Pyxis also lets you use sbatch as normal. It will utilize the SBATCH parameters in the script to set your container specific options. Copy and paste the following into a file called pytorch_enroot.sbatch

#!/bin/bash
#SBATCH --job-name=container-pytorch
#SBATCH --partition=dgx-b200
#SBATCH --gpus=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=32G
#SBATCH --time=00:15:00
#SBATCH --output=out_%j.log

# This tells SLURM (via Pyxis) which container image to use
#SBATCH --container-image=docker://nvcr.io/nvidia/pytorch:25.06-py3

# Optional: bind your project directory into the container
#SBATCH --container-mounts=$HOME:/mnt/home

# Optional: automatically mount your home directory
#SBATCH --container-mount-home

# Run your PyTorch job
python3 -c "import torch; print('CUDA available:', torch.cuda.is_available()); print('Device:', torch.cuda.get_device_name(0))"

Submit this with

sbatch pytorch_enroot.sbatch

You will see the following output in your log file once the job runs

pyxis: importing docker image: docker://nvcr.io/nvidia/pytorch:25.06-py3
pyxis: imported docker image: docker://nvcr.io/nvidia/pytorch:25.06-py3
CUDA available: True
Device: NVIDIA B200

Notes

--container-image works because Pyxis parses this SBATCH directive.

--container-mounts can be used for custom path mapping (e.g., code, data).

If you want to mount additional dependencies (e.g., conda envs, project dirs), just add more paths to the --container-mounts line.

torch.cuda.is_available() is a great quick sanity check.

On This Page