Add Slurm Nodes¶

Add new compute nodes to a running Slurm cluster. Omnia automatically updates the Slurm configuration and integrates new nodes.

Overview¶

Omnia supports dynamic addition of compute nodes to an existing Slurm cluster. The process involves updating the PXE mapping file, running provision.yml, and PXE booting the new nodes. Omnia handles Slurm configuration updates automatically.

Note

Only slurm_node additions are supported. Adding new controller or login nodes requires a full redeployment.

Prerequisites¶

A working Slurm cluster deployed via Set Up Slurm.
New server(s) are physically racked, cabled, and have BMC connectivity.

Procedure¶

Add new entries to the PXE mapping file:

Run on: omnia_core

vi /opt/omnia/input/project_default/pxe_mapping_file.csv

Add new rows with the slurm_node_x86_64 or slurm_node_aarch64 functional group:

File: /opt/omnia/input/project_default/pxe_mapping_file.csv

slurm_node_x86_64,slurm_cluster,NEWSVCTG1,,,aa:bb:cc:dd:ee:10,10.5.0.110,aa:bb:cc:dd:ff:10,10.3.0.110,,
slurm_node_x86_64,slurm_cluster,NEWSVCTG2,,,aa:bb:cc:dd:ee:11,10.5.0.111,aa:bb:cc:dd:ff:11,10.3.0.111,,

Run provision.yml:

Run on: omnia_core

cd /opt/omnia
ansible-playbook provision/provision.yml

PXE boot the newly added nodes.

Omnia automatically updates the Slurm configuration and integrates the new nodes into the cluster during provisioning.

Verification¶

Check that new nodes appear in the cluster:
Run on: Slurm controller node
```
sinfo
```
New nodes should appear in the normal(default) partition with idle state.
Run a test job on the new nodes:
Run on: Slurm controller node
```
srun -w <new-node-hostname> hostname
```
Check slurmd is running on the new nodes:
Run on: new compute node
```
systemctl status slurmd
```

Next Steps¶

Slurm with GPU -- Verify GPU support on the new nodes
Config Backup -- Back up the updated configuration

Troubleshooting¶

New nodes show DOWN after adding Verify slurmd is running and cloud-init completed on the new node:

Run on: new compute node

systemctl status slurmd
journalctl -u slurmd --no-pager -n 20
cat /var/log/cloud-init-output.log | tail -20

Resume the node from the controller:

Run on: Slurm controller node

scontrol update nodename=<node> state=resume reason="added"

For the complete list, see Slurm Issues.