Slurm Issues¶

Issues related to the Slurm job scheduler, including controller failures, node state problems, job submission errors, and GPU detection.

`slurmctld` not starting¶

Symptom

The Slurm controller daemon fails to start. Running systemctl status slurmctld shows the service as failed or inactive.

Cause

The slurm.conf file has syntax errors or references non-existent nodes.
The munge authentication service is not running.
File permissions on /var/spool/slurmctld/ are incorrect.
The Slurm database daemon (slurmdbd) is unreachable and AccountingStorageEnforce is set.

Resolution

Check the slurmctld log for specific errors:

Run on: Slurm controller node

tail -100 /var/log/slurm/slurmctld.log

Verify munge is running:

Run on: Slurm controller node

systemctl status munge

If munge is not running, start it:

Run on: Slurm controller node

systemctl start munge

Validate the Slurm configuration:

Run on: Slurm controller node

slurmd -C    # Show computed configuration
slurmctld -Dvvv    # Run in foreground with verbose logging

Fix spool directory permissions:

Run on: Slurm controller node

chown -R slurm:slurm /var/spool/slurmctld/
chmod 755 /var/spool/slurmctld/

If slurmdbd is the issue, see the slurmdbd connection issues section below.

Nodes Entering DRAINED State¶

Symptom

Slurm nodes enter DRAINED state unexpectedly. Error messages include:

Expected output

State=IDLE+DRAIN Reason=Kill task failed

or:

Expected output

State=DOWN+DRAIN Reason=Not responding

Cause

The epilog script (/etc/slurm/epilog.d/logout_user.sh) is not executable.

Resolution

Fix the epilog script permissions and reconfigure Slurm:

Run on: Slurm controller node

chmod 0755 /etc/slurm/epilog.d/logout_user.sh
scontrol reconfigure

Nodes stuck in DOWN state¶

Symptom

sinfo shows one or more nodes in down or down* state:

PARTITION  AVAIL  TIMELIMIT  NODES  STATE  NODELIST
normal*       up   infinite      1  down*  compute-03

Cause

The slurmd service on the compute node is not running.
Network connectivity between the control node and the compute node is broken.
The node was manually set to DOWN and not resumed.
Hardware issues (memory errors, disk failures) triggered an automatic drain.

Resolution

Check why the node is down:

Run on: Slurm controller node

scontrol show node compute-03 | grep -i reason

Verify slurmd is running on the compute node:

Run on: Slurm controller node

ssh compute-03 systemctl status slurmd

If not running:

Run on: Slurm controller node

ssh compute-03 systemctl start slurmd

Test network connectivity:

Run on: Slurm controller node

ping compute-03
ssh compute-03 hostname

Resume the node after fixing the underlying issue:

Run on: Slurm controller node

scontrol update NodeName=compute-03 State=RESUME

Verify the node returns to idle:

Run on: Slurm controller node

sinfo -n compute-03

Job submission failures¶

Symptom

Submitting a job with sbatch or srun fails with errors such as:

sbatch: error: Batch job submission failed: Invalid account or account/partition combination specified
srun: error: Unable to allocate resources: No partition specified or system default partition

Cause

The user's account is not configured in Slurm accounting.
No default partition is defined in slurm.conf.
The requested resources exceed what is available in the cluster.

Resolution

Check available partitions:
Run on: Slurm controller node
```
sinfo
```

Verify a default partition exists in slurm.conf:

File: /etc/slurm/slurm.conf

PartitionName=normal Nodes=compute-[01-10] Default=YES MaxTime=INFINITE State=UP

If resources are the issue, check available resources:
Run on: Slurm controller node
```
sinfo -N -l
squeue    # Check for jobs consuming resources
```

`slurmdbd` connection issues¶

Symptom

slurmctld logs show errors connecting to the Slurm database daemon:

error: slurmdbd: Sending PersistInit msg: CONNECTION REFUSED
error: slurmdbd: DBD_ID_REGISTER failed

Cause

The slurmdbd service is not running.
The MariaDB database backend is down.
Network or firewall issues between the controller and the database node.
Incorrect database credentials in slurmdbd.conf.

Resolution

Check slurmdbd status:

Run on: Slurm controller node

systemctl status slurmdbd

Check the database backend:

Run on: Slurm controller node

systemctl status mariadb

Verify slurmdbd.conf settings:

Run on: Slurm controller node

grep -i storage /etc/slurm/slurmdbd.conf

Test database connectivity:

Run on: Slurm controller node

mysql -u slurm -p -h localhost slurm_acct_db -e "SELECT 1;"

Check the slurmdbd log:

Run on: Slurm controller node

tail -100 /var/log/slurm/slurmdbd.log

If credentials changed, update slurmdbd.conf and restart:

Run on: Slurm controller node

systemctl restart slurmdbd
systemctl restart slurmctld

NVIDIA GPU, CUDA, and DCGM Issues¶

`nvidia-smi` not found or driver not communicating¶

Symptom

nvidia-smi: command not found or nvidia-smi exits with a non-zero return code.

Cause

NVIDIA driver installation failed during provisioning.
GPU hardware is absent on this node.

Resolution

Verify GPU hardware is present on the node:

Run on: GPU compute node

lspci | grep -i nvidia

If confirmed present, re-install the driver:

Run on: GPU compute node

dnf install -y cuda-drivers

Review the driver installation log for error details:

Run on: GPU compute node

cat /var/log/nvidia_install.log

CUDA toolkit not available on node¶

Symptom

nvcc: command not found or /usr/local/cuda is empty.

Cause

Toolkit installation did not complete on the designated installer node due to a repository or NFS error.
NFS mount for the CUDA toolkit was not established at provisioning time.

Resolution

Verify the NFS mount at /usr/local/cuda is present:

Run on: GPU compute node

mount | grep cuda

If absent, re-mount manually. If the toolkit is not installed on the NFS share, review the installation log on the installer node:

Run on: installer node (login/compiler or compute)

cat /var/log/cuda_toolkit_install.log

CUDA toolkit NFS mount failed¶

Symptom

/usr/local/cuda is empty or not mounted after provisioning.

Cause

NFS server was unreachable at provisioning time.
The NFS export is not configured with no_root_squash.

Resolution

Verify NFS server reachability from the node.
Verify the NFS export includes no_root_squash.
Re-mount manually:

Run on: GPU compute node

mount -t nfs <NFS_SERVER>:<path>/hpc_tools/cuda /usr/local/cuda

Verify the fstab entry is present for persistence.

`nvidia-dcgm` service inactive or failed¶

Symptom

systemctl status nvidia-dcgm shows inactive or failed state.

Cause

DCGM package installation failed due to an unavailable repository or a CUDA version mismatch.
The NVIDIA driver was not functional at the time DCGM attempted to start.

Resolution

Verify driver is functional:

Run on: GPU compute node

nvidia-smi

Identify the installed CUDA version:

Run on: GPU compute node

nvidia-smi | grep "CUDA Version"

Re-install the matching DCGM package and restart the service.
Review the DCGM setup log for errors:

Run on: GPU compute node

cat /var/log/dcgm_setup.log

DCGM not installed¶

Symptom

nvidia-dcgm service is not present on the Slurm node, and /var/log/dcgm_setup.log is missing.

Cause

dcgm.metrics_enabled is set to false under telemetry_sources in telemetry_config.yml, so Omnia intentionally skips DCGM installation during Slurm node cloud-init.

Resolution

Set dcgm.metrics_enabled: true under telemetry_sources in input/telemetry_config.yml.
Re-run provisioning for affected Slurm nodes.
Validate:

Run on: GPU compute node

systemctl status nvidia-dcgm
dcgmi discovery -l

DCGM package version mismatch¶

Symptom

DCGM package installation fails with No match for argument or No packages found.

Cause

The CUDA major version on the node does not have a matching datacenter-gpu-manager-4-cuda<N> package available in the configured local repository.

Resolution

Verify the CUDA version:

Run on: GPU compute node

nvidia-smi | grep "CUDA Version"

Confirm the corresponding DCGM package is present in the local Pulp repository.
Update local_repo_config.yml to include the correct DCGM package version and re-run local_repo.yml.

`nvidia-peermem` not loading¶

Symptom

lsmod does not show nvidia_peermem. Workloads requiring GPUDirect RDMA fail to initialize.

Cause

Kernel headers were not available at provisioning time, causing the DKMS build to fail.
Base NVIDIA kernel modules were not loaded prior to nvidia-peermem load attempt.

Resolution

Verify kernel headers:

Run on: GPU compute node

ls /lib/modules/$(uname -r)/build

Install if missing:

Run on: GPU compute node

dnf install -y kernel-devel-$(uname -r)

Load the module:

Run on: GPU compute node

modprobe nvidia-peermem

Review the installation log:

Run on: GPU compute node

cat /var/log/nvidia_peermem_install.log

Note

If RDMA is not required for any workload on this node, this warning is non-blocking.

Munge authentication failure¶

Symptom

Slurm commands fail with authentication errors such as:

srun: error: Munge encode failed: Failed to access"/var/run/munge/munge.socket.2"
slurmd: error: Munge authentication failed

Cause

The munge service is not running on one or more nodes.
The Munge key is not identical across all nodes in the cluster.

Resolution

Verify Munge is running on all nodes:

Run on: omnia_core container

ansible slurm_cluster -m shell -a "systemctl status munge"

Verify the Munge key is identical across all nodes:

Run on: omnia_core container

ansible slurm_cluster -m shell -a "md5sum /etc/munge/munge.key"

All nodes should report the same MD5 hash.

If keys differ, redistribute the key from the controller node and restart Munge on all affected nodes:

Run on: affected node

systemctl restart munge

MariaDB connection error¶

Symptom

slurmdbd fails to start or slurmctld logs show database connection errors. The sacctmgr command returns errors.

Cause

The MariaDB service is not running on the Slurm controller node.
Database credentials in slurmdbd.conf do not match the MariaDB configuration.
The Slurm database was not initialized.

Resolution

Check MariaDB is running on the control node:

Run on: Slurm controller node

systemctl status mariadb

Test database connectivity:

Run on: Slurm controller node

mysql -u slurm -p -e "SHOW DATABASES;"

If MariaDB is stopped, start it and restart Slurm services:

Run on: Slurm controller node

systemctl start mariadb
systemctl restart slurmdbd
systemctl restart slurmctld

`slurmctld` fails to start after config rollback¶

Symptom

After rolling back a Slurm configuration backup, slurmctld fails to start.

Cause

The backup may reference nodes that no longer exist in the cluster.

Resolution

Run cleanup and redeploy with provision.yml:

Run on: omnia_core container

ansible-playbook /opt/omnia/utils/slurm_config_util.yml --tags slurm_cleanup
ansible-playbook provision.yml

New nodes show DOWN after adding¶

Symptom

Newly added compute nodes appear as down in sinfo after running provision.yml and PXE booting.

Cause

The slurmd service failed to start on the new node.
Cloud-init did not complete successfully.

Resolution

Verify slurmd is running on the new node:

Run on: new compute node

systemctl status slurmd
journalctl -u slurmd --no-pager -n 20

Check cloud-init completed successfully:

Run on: new compute node

cat /var/log/cloud-init-output.log | tail -20

Resume the node from the controller:

Run on: Slurm controller node

scontrol update nodename=<node> state=resume reason="added"

Node still appears in sinfo after removal¶

Symptom

After removing a node, it still appears in sinfo output.

Cause

The provision.yml playbook did not complete successfully.
The Slurm controller has not been reconfigured.

Resolution

Verify that provision.yml completed successfully and check the Slurm controller logs:

Run on: Slurm controller node

journalctl -u slurmctld --no-pager -n 20

If jobs were running on the removed node, they may show as FAILED or NODE_FAIL in accounting:

Run on: Slurm controller node

sacct --starttime=today --state=FAILED,NODE_FAIL

Resubmit affected jobs as needed.

CUDA Toolkit and DCGM Setup Failure: Manual Recovery¶

Symptom

Automated GPU setup fails during provisioning.

Cause

Repository unavailability, NFS connectivity issues, or node initialization errors.

Resolution

Perform all recovery steps as root on the affected node. Verify that the shared NFS path is reachable and repositories are accessible before proceeding.

Step 1: Verify Prerequisites

Run on: affected GPU node

showmount -e <NFS_SERVER_IP>
lspci | grep -i nvidia
dnf repolist | grep -i cuda
df -h /usr/local

Step 2: Recover NVIDIA Driver

If nvidia-smi is missing or returning errors:

Run on: affected GPU node

dnf install -y cuda-drivers
nvidia-smi

Step 3: Recover CUDA Toolkit

The recovery procedure differs depending on the node topology.

Scenario A -- Login or compiler node present in the cluster:

The login/compiler node is the designated installer. It installs the toolkit to the shared NFS location at /hpc_tools/cuda. Compute nodes mount this path at /usr/local/cuda.

On the login or compiler node:

Run on: login/compiler node

ls /hpc_tools/cuda/bin/nvcc 2>/dev/null && echo "Toolkit present" || echo "Toolkit NOT present"

If not present, trigger the installation manually:

Run on: login/compiler node

CUDA_INSTALL_MANUAL=true /usr/local/bin/install_cuda_toolkit.sh

On a compute node (after toolkit is confirmed on NFS):

Run on: GPU compute node

mount | grep cuda

If absent, re-mount manually:

Run on: GPU compute node

mount -t nfs <NFS_SERVER>:<hpc_tools_path>/hpc_tools/cuda /usr/local/cuda

Scenario B -- No login or compiler node in the cluster:

Compute nodes install the toolkit themselves to /hpc_tools/cuda.

Run on: any compute node

ls /hpc_tools/cuda/bin/nvcc 2>/dev/null && echo "Toolkit present" || echo "Toolkit NOT present"

If not present:

Run on: any compute node

CUDA_INSTALL_MANUAL=true /usr/local/bin/install_cuda_toolkit.sh

Note

Run this only after confirming no active toolkit installation is already in progress. Review /var/log/cuda_toolkit_install.log to check current installation status.

Step 4: Recover DCGM

If the nvidia-dcgm service is inactive or failed:

Run on: GPU compute node

nvidia-smi | grep "CUDA Version"
dnf install -y datacenter-gpu-manager-4-cuda<N>
systemctl enable nvidia-dcgm
systemctl start nvidia-dcgm

Validate:

Run on: GPU compute node

systemctl status nvidia-dcgm
dcgmi discovery -l

Step 5: Recover nvidia-peermem (RDMA environments only)

Run on: GPU compute node

ls /lib/modules/$(uname -r)/build
dnf install -y kernel-devel-$(uname -r)
modprobe nvidia-peermem

Validate:

Run on: GPU compute node

lsmod | grep -E 'nv_peer_mem|nvidia_peermem'

Log file reference:

/var/log/nvidia_install.log -- NVIDIA driver installation output
/var/log/cuda_toolkit_install.log -- CUDA toolkit installation output and timing
/var/log/dcgm_setup.log -- DCGM package install, service startup, GPU discovery
/var/log/nvidia_peermem_install.log -- nvidia-peermem DKMS build and load output

Benchmark assets missing on Slurm nodes¶

Symptom

Benchmark tool directories are missing or incomplete under /hpc_tools.
Expected benchmark artifacts are not visible on login/compiler/compute nodes.

Cause

Shared NFS path (/hpc_tools) is not mounted or not accessible.
pull_benchmarks.sh or benchmark_tools.list is missing under /hpc_tools/scripts.
Pulp mirror endpoint is unreachable from the node.
Required benchmark content is not available in local repository/Pulp.
Tool directory already exists and contains files (script skips re-download by design).
Architecture mismatch (for example, msr-safe on aarch64, which is skipped by design).

Resolution

Verify NFS and scripts path:

Run on: affected node

ls -ld /hpc_tools
ls -l /hpc_tools/scripts

Expected files: /hpc_tools/scripts/pull_benchmarks.sh and /hpc_tools/scripts/benchmark_tools.list.

Run the runtime staging script and review output:

Run on: affected node

/hpc_tools/scripts/pull_benchmarks.sh

Review the runtime log:

Run on: affected node

tail -n 200 /var/log/pull_benchmarks.log

Validate staged benchmark directories:

Run on: affected node

ls -l /hpc_tools/osu-micro-benchmarks /hpc_tools/imb /hpc_tools/likwid /hpc_tools/papi /hpc_tools/geopm /hpc_tools/sionlib

If a tool was skipped as already present, remove that tool directory only if refresh is required, then re-run /hpc_tools/scripts/pull_benchmarks.sh.

Note

msr-safe is expected only on x86_64.

`sacct` erroring out or returning empty results¶

Symptom

The sacct command returns no output or empty results when querying job accounting information.

Cause

slurmdbd service is not running.
MariaDB service is not running (slurmdbd depends on MariaDB).
slurmdbd cannot communicate with the database.
Port 6819 (slurmdbd port) is not listening.

Resolution

Check if slurmdbd service is running:

Run on: Slurm controller node

systemctl status slurmdbd

Check if MariaDB service is running:

Run on: Slurm controller node

systemctl status mariadb

Check the slurmdbd logs:

Run on: Slurm controller node

tail -50 /var/log/slurm/slurmdbd.log

Check the slurmdbd port:

Run on: Slurm controller node

ss -tlnp | grep 6819

Restart the services accordingly:

Run on: Slurm controller node

systemctl restart slurmdbd
systemctl restart mariadb

Slurm RPM build failures¶

Symptom

rpmbuild fails when building Slurm RPMs from source.

Cause

Missing development dependencies on the build host.
Missing kernel headers on aarch64 build hosts.
CUDA toolkit not installed before building with GPU support.

Resolution

Install missing development packages:

Run on: build host

dnf install -y <missing-package>-devel

For aarch64 builds, install kernel headers:

Run on: aarch64 build host

dnf install -y kernel-devel kernel-headers

Verify CUDA is installed before running rpmbuild with GPU support:

Run on: build host

ls /usr/local/cuda/lib64/stubs/libnvidia-ml.so

Info

Setup Slurm -- Slurm cluster setup guide.
Slurm With Gpu -- GPU configuration for Slurm.
Add Remove Nodes -- Adding or removing Slurm nodes.

Slurm Issues¶

slurmctld not starting¶

Nodes Entering DRAINED State¶

Nodes stuck in DOWN state¶

Job submission failures¶

slurmdbd connection issues¶

NVIDIA GPU, CUDA, and DCGM Issues¶

nvidia-smi not found or driver not communicating¶

CUDA toolkit not available on node¶

CUDA toolkit NFS mount failed¶

nvidia-dcgm service inactive or failed¶

DCGM not installed¶

DCGM package version mismatch¶

nvidia-peermem not loading¶

Munge authentication failure¶

MariaDB connection error¶

slurmctld fails to start after config rollback¶

New nodes show DOWN after adding¶

Node still appears in sinfo after removal¶

CUDA Toolkit and DCGM Setup Failure: Manual Recovery¶

Benchmark assets missing on Slurm nodes¶

sacct erroring out or returning empty results¶

Slurm RPM build failures¶

`slurmctld` not starting¶

`slurmdbd` connection issues¶

`nvidia-smi` not found or driver not communicating¶

`nvidia-dcgm` service inactive or failed¶

`nvidia-peermem` not loading¶

`slurmctld` fails to start after config rollback¶

`sacct` erroring out or returning empty results¶