Path A: Slurm Quick Start

Deploy a Slurm HPC cluster using Omnia. This is the fastest path to a working Omnia environment and the recommended starting point for first-time users.

What you will build:

Role	Functional Group	Purpose
OIM (management)	--	Runs the omnia_core container; orchestrates the deployment. Does not join the Slurm cluster.
Head node	slurm_control_node_x86_64	Runs slurmctld (Slurm controller) and slurmdbd (accounting database). x86_64 only.
Compute node(s)	slurm_node_x86_64 / slurm_node_aarch64	Run slurmd; execute jobs submitted to the cluster.
Login node	login_node_x86_64 / login_node_aarch64	User-facing SSH gateway for job submission.
Login/compiler node	login_compiler_node_x86_64 / login_compiler_node_aarch64	Login gateway with compiler toolchains for building applications.

Note

This tutorial assumes you have completed every item on the Prerequisites Checklist. If you have not, stop here and finish that first.

Step 1 -- Deploy the omnia_core Container

Clone the Omnia artifacts repository, build the omnia_core container image, and deploy the container on the OIM. The container packages the complete Omnia codebase and Ansible engine.

For details, see Deploy Omnia Core.

1. Clone the Omnia Artifactory repository and build the container image:

   Run on: OIM host

   git clone https://github.com/dell/omnia-artifactory.git -b omnia-container-v2.2.0.0
   cd omnia-artifactory
   ./build_images.sh core omnia_branch=v2.2.0.0 core_tag=2.2
   

2. Download the omnia.sh script:

   Run on: OIM host

   wget https://raw.githubusercontent.com/dell/omnia/refs/tags/v2.2.0.0/omnia.sh
   chmod +x omnia.sh
   

3. Install the omnia_core container:

   Run on: OIM host

   ./omnia.sh --install
   

Caution

The password must not contain special characters such as , |, &, ;, \, <>, *, ?, !, $, (), {}, []`.

Verification

1. Verify the omnia_core container is running:

   Run on: OIM host

   podman ps --filter name=omnia_core --format "table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}"
   

   Expected output:

   Expected output

   NAMES        IMAGE                       STATUS       PORTS
   omnia_core   localhost/omnia_core:2.2     Up 1 day     2222/tcp
   

2. Access the omnia_core container:

   Run on: OIM host

   ssh omnia_core
   

   You will be automatically logged in to the omnia_core container.

Warning

• Do not delete any key pairs generated by Omnia from /root/.ssh -- this causes omnia_core.service execution failure.
• Do not manually delete files from the Omnia shared directory. Use ./omnia.sh --uninstall to safely remove.

Step 2 -- Create the PXE Mapping File

Omnia supports two methods for creating the PXE mapping file:

• Manual -- Collect PXE NIC information and fill in the pxe_mapping_file.csv manually.
• OME-based discovery (recommended) -- Use OpenManage Enterprise (OME) to discover cluster nodes and auto-generate the mapping file using discovery.yml.

Option A: Fill the PXE mapping file manually

Create a pxe_mapping_file.csv in /opt/omnia/input/project_default/ and set the pxe_mapping_file_path variable in provision_config.yml to point to it.

/opt/omnia/input/project_default/pxe_mapping_file.csv

FUNCTIONAL_GROUP_NAME,GROUP_NAME,SERVICE_TAG,PARENT_SERVICE_TAG,HOSTNAME,ADMIN_MAC,ADMIN_IP,BMC_MAC,BMC_IP,IB_NIC_NAME,IB_IP
slurm_control_node_x86_64,grp0,SVCTAG01,,head01,a1:b2:c3:d4:e5:f6,172.16.107.52,a2:b3:c4:d5:e6:f7,172.17.107.52,,
slurm_node_x86_64,grp1,SVCTAG02,,compute01,b1:c2:d3:e4:f5:a6,172.16.107.43,b2:c3:d4:e5:f6:a7,172.17.107.43,,
login_node_x86_64,grp2,SVCTAG03,,login01,c1:d2:e3:f4:a5:b6,172.16.107.44,c2:d3:e4:f5:a6:b7,172.17.107.44,,
login_compiler_node_x86_64,grp3,SVCTAG04,,login-compiler01,d1:e2:f3:a4:b5:c6,172.16.107.45,d2:e3:f4:a5:b6:c7,172.17.107.45,,

Warning

Replace all placeholder values (SVCTAG*, MAC addresses, IPs) with your actual hardware data.

Note

• All header fields are case-sensitive.
• Leave the PARENT_SERVICE_TAG column empty for Slurm-only deployments (without K8s).
• IB_NIC_NAME and IB_IP are optional. Leave them empty if InfiniBand is not used.
• The ADMIN_MAC and BMC_MAC addresses should refer to the PXE NIC and BMC NIC on the target nodes respectively.
• Target servers should be configured to boot in PXE mode with the appropriate NIC as the first boot device.
• Hostnames should not contain the domain name of the nodes.

For detailed information on PXE mapping file format and parameters, see PXE Mapping File.

Option B: Create PXE file using OME

Use the discovery.yml playbook to auto-generate the mapping file from an OME inventory. For detailed instructions including OME prerequisites, static group setup, and iDRAC hostname conventions, see Discover Nodes Using OME.

Run on: omnia_core container

cd /omnia/discovery
ansible-playbook discovery.yml -e "discovery_mechanism=ome"

The playbook generates a bmc_pxe_mapping_file_<timestamp>.csv in /opt/omnia/input/project_default/. Verify and edit the file as needed.

Step 3 -- Provide Inputs

Configure the input files that define your cluster's network, provisioning, and storage settings. For a Slurm deployment, update the following input files in /opt/omnia/input/project_default/. Click each file name to view the full parameter reference.

Input File	Purpose
network_spec.yml	Network CIDRs, interfaces, and IP ranges
provision_config.yml	OS provisioning and PXE settings
software_config.json	Software stack selections
omnia_config.yml	Slurm cluster configuration
storage_config.yml	NFS storage mount configuration
local_repo_config.yml	Repository mirror settings
telemetry_config.yml	Telemetry and monitoring settings
security_config.yml	OpenLDAP authentication settings

Slurm-specific guidance

software_config.json -- The slurm_custom entry is mandatory for Slurm deployments. Include openldap if LDAP authentication is required.

Minimum required entries

{
  "softwares": [
    {"name": "default_packages", "arch": ["x86_64"]},
    {"name": "slurm_custom", "arch": ["x86_64"]},
    {"name": "openldap", "arch": ["x86_64"]}
  ]
}

Note

The arch field accepts x86_64 or aarch64. If your cluster includes aarch64 nodes, add a separate entry for each software component with "arch": ["aarch64"].

For the full procedure and parameter reference, see Configure Inputs.

Note

For Slurm-only deployments, disable all telemetry metrics in telemetry_config.yml except DCGM, which can be enabled if GPU telemetry is required. For more information related to DCGM, see DCGM.

Tip

If you need to build custom Slurm RPMs from source or host them on a local server, complete those steps first:

• Build Slurm RPM Repository
• Host Slurm RPM Repository

Caution

If any Slurm nodes have an InfiniBand interface and ib_network is defined in network_spec.yml:

• The Slurm user repository must not include ucx, ucx-devel, openmpi, or openmpi-devel packages.
• Slurm must be compiled without UCX and OpenMPI support.
• DOCA-OFED provides its own UCX and OpenMPI stack, configured automatically during provisioning.

For InfiniBand network configuration details, see Configure InfiniBand.

Step 4 -- Configure Slurm

Omnia applies a default Slurm configuration optimized for HPC clusters:

• Default partition: A partition named normal is created with all compute nodes from the PXE mapping file
• Scheduler: sched/backfill with select/cons_tres and CR_Core_Memory
• GPU support: GresTypes=gpu with AutoDetect=nvml
• Configless mode: Compute nodes use --conf-server to fetch configuration from the controller

Note

The parameters ClusterName, SlurmctldHost, and AccountingStorageHost are managed by Omnia and cannot be overridden.

For custom Slurm configuration, see Configure Slurm.

Step 5 -- Prepare the OIM

Deploys the OIM infrastructure: OpenCHAMI provisioning stack, Pulp local repository, container registry, MinIO S3 storage, OpenLDAP authentication, and step-ca certificate authority.

For details, see Prepare OIM.

Run on: omnia_core container

cd /omnia/prepare_oim
ansible-playbook prepare_oim.yml

Verification -- OIM Infrastructure

After prepare_oim.yml completes, verify the OIM services on the OIM host (not inside the container):

1. Check omnia.target status:

   Run on: OIM host

   systemctl is-active omnia.target
   

   Expected output: active

2. Verify all service dependencies:

   Run on: OIM host

   systemctl list-dependencies omnia.target
   

   Expected output:

   Expected output

   omnia.target
   ● ├─minio.service
   ● ├─omnia_auth.service
   ● ├─omnia_core.service
   ● ├─pulp.service
   ● ├─registry.service
   ● ├─network-online.target
   ● │ └─NetworkManager-wait-online.service
   ● └─openchami.target
   ●   ├─acme-deploy.service
   ●   ├─acme-register.service
   ●   ├─bss-init.service
   ●   ├─bss.service
   ●   ├─cloud-init-server.service
   ●   ├─coresmd-coredhcp.service
   ●   ├─coresmd-coredns.service
   ●   ├─haproxy.service
   ●   ├─hydra-gen-jwks.service
   ●   ├─hydra-migrate.service
   ●   ├─hydra.service
   ●   ├─opaal-idp.service
   ●   ├─opaal.service
   ●   ├─openchami-cert-trust.service
   ●   ├─postgres.service
   ●   ├─smd-init.service
   ●   ├─smd.service
   ●   └─step-ca.service
   

3. Verify all containers are running:

   Run on: OIM host

   podman ps --format "table {{.Names}}\t{{.Status}}"
   

   Expected output:

   Expected output

   NAMES               STATUS
   bss                 Up 1 day
   cloud-init-server   Up 1 day
   coresmd-coredhcp    Up 1 day
   coresmd-coredns     Up 1 day
   haproxy             Up 1 day
   hydra               Up 1 day
   minio-server        Up 1 day
   omnia_auth          Up 1 day
   omnia_core          Up 1 day
   opaal               Up 1 day
   opaal-idp           Up 1 day
   postgres            Up 1 day
   pulp                Up 1 day
   registry            Up 1 day
   smd                 Up 1 day
   step-ca             Up 1 day
   

Note

• The minio-server container will not be present if you configured PowerScale as the S3 endpoint (s3_configurations.provider: "powerscale") in storage_config.yml. In that case, Omnia uses the external PowerScale S3 service instead of deploying a local MinIO container.
• The omnia_auth container will not be present if openldap is not included in software_config.json.

For detailed OIM verification procedures, see Verify OIM Services.

Step 6 -- Create Local Repositories

Downloads all required RPM packages, container images, and tarballs into Pulp based on software_config.json for air-gapped provisioning.

For details, see Create Local Repos.

Run on: omnia_core container

cd /omnia/local_repo
ansible-playbook local_repo.yml

Note

Expect 45--90 minutes depending on network speed. Total download size is typically 20--40 GB.

Verification -- Local Repository Status

After local_repo.yml completes, verify that all software components were downloaded successfully by checking the software.csv status file. The components listed in this file correspond directly to the software entries configured in software_config.json.

1. Verify x86_64 package status:

   Run on: omnia_core container

   cat /opt/omnia/log/local_repo/rhel/10.0/x86_64/software.csv
   

   Expected output:

   Expected output

   name,status
   default_packages,success
   openldap,success
   slurm_custom,success
   

2. Verify aarch64 package status (if aarch64 is included in software_config.json):

   Run on: omnia_core container

   cat /opt/omnia/log/local_repo/rhel/10.0/aarch64/software.csv
   

   Expected output:

   Expected output

   name,status
   default_packages,success
   openldap,success
   slurm_custom,success
   

Note

The software.csv output reflects the software components configured in software_config.json. Each component with "arch": ["x86_64"] appears in the x86_64 status file, and each component with "arch": ["aarch64"] appears in the aarch64 status file. All entries must show success status before proceeding.

Step 7 -- Build Node Images

Builds diskless OS images for each functional group in the PXE mapping file and uploads them to MinIO (S3) for PXE boot delivery.

For details, see Build Cluster Images.

Build x86_64 Images

Run on: omnia_core container

cd /omnia/build_image_x86_64
ansible-playbook build_image_x86_64.yml

Build aarch64 Images

If your PXE mapping file contains aarch64 functional groups, you must first prepare an aarch64 build node. See Prepare aarch64 Node for the complete procedure (manual RHEL 10 installation, inventory file creation, etc.).

Run on: omnia_core container

cd /omnia/build_image_aarch64
ansible-playbook build_image_aarch64.yml -i inventory

Example: inventory

[admin_aarch64]
10.0.0.1

Verification -- Boot Images in S3

After the build playbooks complete, verify the images are uploaded to MinIO (S3). Each functional group produces 3 image artifacts: rootfs (full OS root filesystem), vmlinuz (Linux kernel), and initramfs (initial RAM filesystem for PXE boot).

1. List all boot images in S3:

   Run on: OIM host

   s3cmd ls s3://boot-images/
   

   Expected output (one directory per functional group plus efi-images):

   Expected output

                       DIR  s3://boot-images/efi-images/
                       DIR  s3://boot-images/login_compiler_node_x86_64/
                       DIR  s3://boot-images/slurm_control_node_x86_64/
                       DIR  s3://boot-images/slurm_node_x86_64/
   

2. Verify individual image artifacts for a specific functional group:

   Run on: OIM host

   s3cmd ls -Hr s3://boot-images/slurm_control_node_x86_64/
   s3cmd ls -Hr s3://boot-images/efi-images/slurm_control_node_x86_64/
   

   Expected output:

   Expected output

   2026-06-26 11:42  1449M  s3://boot-images/slurm_control_node_x86_64/rhel-slurm_control_node_x86_64_omnia_2.2.0.0/rhel10.0-rhel-slurm_control_node_x86_64_omnia_2.2.0.0-10.0
   2026-06-26 11:42    78M  s3://boot-images/efi-images/slurm_control_node_x86_64/rhel-slurm_control_node_x86_64_omnia_2.2.0.0/initramfs-6.12.0-55.82.1.el10_0.x86_64.img
   2026-06-26 11:42    15M  s3://boot-images/efi-images/slurm_control_node_x86_64/rhel-slurm_control_node_x86_64_omnia_2.2.0.0/vmlinuz-6.12.0-55.82.1.el10_0.x86_64
   

Note

The directories listed in s3://boot-images/ correspond to the functional groups defined in your PXE mapping file. Each functional group will have exactly 3 image artifacts (rootfs, vmlinuz, initramfs). The efi-images/ directory contains the initramfs and vmlinuz boot files used during PXE network boot, while the root filesystem is stored directly under each functional group directory. If any artifacts are missing, re-run the corresponding build playbook.

Step 8 -- Provision Nodes

The provision.yml playbook provisions the cluster nodes. It configures boot scripts, cloud-init, and prepares nodes for Slurm deployment.

Run on: omnia_core container

cd /omnia/provision
ansible-playbook provision.yml

Verification -- nodes.yaml

After provision.yml completes, verify that all nodes from your PXE mapping file are present in the generated nodes.yaml file. Every node defined in pxe_mapping_file.csv should have a corresponding entry with its hostname, functional group, MAC address, and IP address.

Run on: omnia_core container

cat /opt/omnia/openchami/workdir/nodes/nodes.yaml

Expected output (one entry per node in the PXE mapping file):

Expected output

nodes:
- name: head01
  xname: x1000c0s0b0n0
  description: SVCTAG01
  nid: 1
  group: slurm_control_node_x86_64
  bmc_mac: a2:b3:c4:d5:e6:f7
  bmc_ip: 10.3.0.XXX
  interfaces:
  - mac_addr: a1:b2:c3:d4:e5:f6
    ip_addrs:
    - name: management
      ip_addr: 10.5.0.XXX
- name: compute01
  xname: x1000c0s0b1n0
  description: SVCTAG02
  nid: 2
  group: slurm_node_x86_64
  bmc_mac: b2:c3:d4:e5:f6:a7
  bmc_ip: 10.3.0.XXX
  interfaces:
  - mac_addr: b1:c2:d3:e4:f5:a6
    ip_addrs:
    - name: management
      ip_addr: 10.5.0.XXX
...

Note

Post execution of provision.yml, IPs and hostnames cannot be re-assigned by changing the mapping file.

Caution

• Do not run ssh-keygen post execution of provision.yml to avoid breaking the password-less SSH channel on the OIM.
• Do not delete the Omnia shared path or the NFS directory.

For troubleshooting boot issues, IP route conflicts, and cloud-init failures, see Provisioning Issues.

Step 9 -- PXE Boot Nodes

After provision.yml completes, PXE boot all Slurm-related nodes.

Option 1: Manual PXE Boot

Configure each node to boot from the network via iDRAC or BIOS settings.

Option 2: Automated PXE Boot

Sets PXE boot order on all nodes via iDRAC Redfish and reboots them. Nodes boot from the network, load their OS image from S3, and execute cloud-init to complete provisioning.

Run on: omnia_core container

cd /omnia/utils
ansible-playbook set_pxe_boot.yml

Warning

This playbook will restart your servers and power them on if they are off. Any unsaved data will be lost.

Verification -- Cloud-Init Provisioning Status

After the nodes PXE boot, verify that cloud-init has completed on all nodes. SSH from omnia_core into each node using its hostname from the PXE mapping file (HOSTNAME column):

Run on: omnia_core container (example for 2 nodes)

ssh head01 'cloud-init status'
ssh compute01 'cloud-init status'

Expected output on each node:

Expected output

status: done

Note

Check every node in your cluster. Open your PXE mapping file and run ssh <HOSTNAME> 'cloud-init status' for each entry. All nodes must report status: done before proceeding.

Verification -- Slurm Cluster

SSH into the slurm_control_node and verify all compute nodes are idle:

Run on: omnia_core container (example)

ssh head01 'sinfo'

Expected output:

Expected output

PARTITION  AVAIL  TIMELIMIT  NODES  STATE  NODELIST
normal*    up     infinite   2      idle   compute[01-02]

For detailed cluster verification procedures, see Verify Cluster.

Step 10 -- Verify the Cluster

After all nodes have booted and cloud-init has completed, verify the Slurm cluster is operational.

For detailed cluster verification procedures, see Verify Cluster.

Run on: Slurm controller node

systemctl status slurmctld
sinfo

Expected output

PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
normal*      up   infinite      1   idle compute01

Run on: Slurm controller node

srun -N 1 hostname

Run on: login node

srun -N 1 hostname

What's Next?

Your Slurm cluster is operational. Common next steps:

Configure GPU support If your compute nodes have NVIDIA GPUs, follow Slurm with GPU to enable GPU scheduling and GRES configuration.

Install NVIDIA HPC SDK Set up the NVIDIA HPC compilers (nvc, nvc++, nvfortran) on compiler and compute nodes using NVIDIA HPC SDK Setup.

Customize Slurm configuration Tune partitions, scheduling policies, and accounting settings using Configure Slurm.

Run HPC benchmarks Validate cluster performance with HPL, OSU Micro-Benchmarks, and other tools using Run HPC Benchmarks.

Scale the cluster Add more compute nodes with Add Slurm Nodes or remove nodes with Remove Slurm Nodes.

Back up Slurm configuration Create timestamped backups and rollback points using Config Backup.

Add K8s and telemetry later Follow Full Deployment to add a Kubernetes service cluster and telemetry pipeline to this existing Slurm deployment.

Info

• Set Up Slurm -- Detailed Slurm setup guide
• Full Deployment -- Add K8s to this Slurm deployment
• Prerequisites Checklist -- Master checklist
• Slurm Troubleshooting -- Troubleshoot Slurm issues