Remove Slurm Nodes¶

Remove compute nodes from a running Slurm cluster. Omnia handles Slurm configuration updates and provides job protection for busy nodes.

Overview¶

Omnia supports dynamic removal of compute nodes from an existing Slurm cluster. The process involves removing entries from the PXE mapping file and running provision.yml. Omnia automatically updates the Slurm configuration.

If removed nodes have active running jobs, Omnia prompts you to either abort (remove only idle nodes) or force-remove (cancel all jobs on affected nodes).

Note

Only slurm_node removals are supported. Removing controller or login nodes requires a full redeployment.

Prerequisites¶

A working Slurm cluster deployed via Set Up Slurm.

Tip

Take a configuration backup before removing nodes. See Config Backup.

Procedure¶

Remove node entries from the PXE mapping file:
Run on: omnia_core
```
vi /opt/omnia/input/project_default/pxe_mapping_file.csv
```
Remove the rows for the nodes you want to decommission.
Run provision.yml:
Run on: omnia_core
```
cd /opt/omnia
ansible-playbook provision/provision.yml
```
If the removed nodes have active running jobs, Omnia prompts:
- Abort -- Removes only idle nodes, keeps busy nodes in the cluster
- Force-remove -- Cancels all jobs on affected nodes and removes them

Omnia automatically updates slurm.conf and reconfigures the Slurm controller.

Verification¶

Confirm the node is no longer in the cluster:
Run on: Slurm controller node
```
sinfo
```
The removed node should no longer appear.
Run a test job to confirm remaining nodes are functional:
Run on: Slurm controller node
```
srun -N 1 hostname
```
Verify no orphaned jobs reference the removed node:
Run on: Slurm controller node
```
squeue -t all
```

Next Steps¶

Add Slurm Nodes -- Add replacement nodes if needed
Config Backup -- Back up the updated configuration

Troubleshooting¶

Node still appears in sinfo after removal Verify that provision.yml completed and check the controller logs:

Run on: Slurm controller node

journalctl -u slurmctld --no-pager -n 20

Check for failed jobs on the removed node and resubmit as needed:

Run on: Slurm controller node

sacct --starttime=today --state=FAILED,NODE_FAIL

For the complete list, see Slurm Issues.