Skip to content

Path C: Full Deployment (Slurm + K8s + Telemetry)

Deploy a production-grade cluster with Slurm job scheduling, a highly available Kubernetes service cluster, and telemetry. This is the canonical Omnia deployment that exercises every major subsystem.

What you will build:

Role Functional Group Count Purpose
OIM (management) -- 1 Runs omnia_core; orchestrates the deployment.
K8s control plane service_kube_control_plane_x86_64 3 HA Kubernetes control plane (kube-apiserver, etcd, kube-scheduler, kube-controller-manager).
K8s worker node service_kube_node_x86_64 1 Runs the telemetry stack: iDRAC collector, LDMS aggregator, Kafka, VictoriaMetrics.
Slurm control node slurm_control_node_x86_64 1 Runs slurmctld (Slurm controller), slurmdbd (accounting), and MariaDB.
Slurm compute node(s) slurm_node_x86_64 / slurm_node_aarch64 1+ Run slurmd; execute jobs submitted to the cluster.
Login / compiler node login_compiler_node_x86_64 / login_compiler_node_aarch64 1 User-facing SSH gateway with compiler toolchains for job submission and building applications.

Estimated time: ~4 hours.

Note

Complete the Prerequisites Checklist before proceeding.

Step 1 -- Deploy the omnia_core Container

Clone the Omnia artifacts repository, build the omnia_core container image, and deploy the container on the OIM. The container packages the complete Omnia codebase and Ansible engine.

For details, see Deploy Omnia Core.

  1. Clone the Omnia Artifactory repository and build the container image:

    Run on: OIM host
    git clone https://github.com/dell/omnia-artifactory.git -b omnia-container-v2.2.0.0
cd omnia-artifactory
./build_images.sh core omnia_branch=v2.2.0.0 core_tag=2.2

  2. Download the omnia.sh script:

    Run on: OIM host
    wget https://raw.githubusercontent.com/dell/omnia/refs/tags/v2.2.0.0/omnia.sh
chmod +x omnia.sh

  3. Install the omnia_core container:

    Run on: OIM host
    ./omnia.sh --install

Caution

The password must not contain special characters such as , |, &, ;, \, <>, *, ?, !, $, (), {}, []`.

Verification

  1. Verify the omnia_core container is running:

    Run on: OIM host
    podman ps --filter name=omnia_core --format "table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}"

    Expected output:

    Expected output
    NAMES        IMAGE                       STATUS       PORTS
omnia_core   localhost/omnia_core:2.2     Up 1 day     2222/tcp

  2. Access the omnia_core container:

    Run on: OIM host
    ssh omnia_core

    You will be automatically logged in to the omnia_core container.

Warning

  • Do not delete any key pairs generated by Omnia from /root/.ssh -- this causes omnia_core.service execution failure.
  • Do not manually delete files from the Omnia shared directory. Use ./omnia.sh --uninstall to safely remove.

Step 2 -- Create the Mapping File

Omnia supports two methods for creating the PXE mapping file:

  • Manual -- Collect PXE NIC information and fill in the pxe_mapping_file.csv manually.
  • OME-based discovery (recommended) -- Use OpenManage Enterprise (OME) to discover cluster nodes and auto-generate the mapping file using discovery.yml.

Option A: Fill the PXE mapping file manually

Create a pxe_mapping_file.csv in /opt/omnia/input/project_default/ and set the pxe_mapping_file_path variable in provision_config.yml to point to it.

/opt/omnia/input/project_default/pxe_mapping_file.csv
FUNCTIONAL_GROUP_NAME,GROUP_NAME,SERVICE_TAG,PARENT_SERVICE_TAG,HOSTNAME,ADMIN_MAC,ADMIN_IP,BMC_MAC,BMC_IP
service_kube_control_plane_x86_64,kube,SVCTAG01,,kube-cp01,24:6E:96:BB:01:01,10.5.0.201,,10.3.0.201
service_kube_control_plane_x86_64,kube,SVCTAG02,,kube-cp02,24:6E:96:BB:01:02,10.5.0.202,,10.3.0.202
service_kube_control_plane_x86_64,kube,SVCTAG03,,kube-cp03,24:6E:96:BB:01:03,10.5.0.203,,10.3.0.203
service_kube_node_x86_64,kube,SVCTAG04,,kube-wk01,24:6E:96:BB:02:01,10.5.0.204,,10.3.0.204
slurm_control_node_x86_64,slurm,SVCTAG05,,head01,24:6E:96:BB:03:01,10.5.0.205,,10.3.0.205
slurm_node_x86_64,slurm,SVCTAG06,SVCTAG05,compute01,24:6E:96:BB:04:01,10.5.0.206,,10.3.0.206
slurm_node_x86_64,slurm,SVCTAG07,SVCTAG05,compute02,24:6E:96:BB:04:02,10.5.0.207,,10.3.0.207
login_compiler_node_x86_64,slurm,SVCTAG08,,login-compiler01,24:6E:96:BB:05:01,10.5.0.208,,10.3.0.208

Warning

Replace all placeholder values (SVCTAG*, MAC addresses, IPs) with your actual hardware data.

Note

  • All header fields are case-sensitive.
  • PARENT_SERVICE_TAG maps Slurm nodes to their parent service kube node for iDRAC telemetry grouping.
  • The ADMIN_MAC and BMC_MAC addresses should refer to the PXE NIC and BMC NIC on the target nodes respectively.
  • Target servers should be configured to boot in PXE mode with the appropriate NIC as the first boot device.
  • Hostnames should not contain the domain name of the nodes.

Note

The arch field accepts x86_64 or aarch64. If your cluster includes aarch64 nodes, add a separate entry for each software component with "arch": ["aarch64"].

For detailed information on PXE mapping file format and parameters, see PXE Mapping File.

Option B: Create PXE file using OME

Use the discovery.yml playbook to auto-generate the mapping file from an OME inventory. For detailed instructions including OME prerequisites, static group setup, and iDRAC hostname conventions, see Discover Nodes Using OME.

Run on: omnia_core container
cd /omnia/discovery
ansible-playbook discovery.yml -e "discovery_mechanism=ome"

The playbook generates a bmc_pxe_mapping_file_<timestamp>.csv in /opt/omnia/input/project_default/. Verify and edit the file as needed.

Warning

For a full deployment, ensure at least 3 rows use the service_kube_control_plane functional group and at least 1 row uses service_kube_node. HA requires a minimum of 3 control-plane nodes.

Step 3 -- Provide Inputs

Configure the input files that define your cluster's network, provisioning, telemetry, and storage settings. For a full deployment, update the following input files in /opt/omnia/input/project_default/. Click each file name to view the full parameter reference.

Input File Purpose
network_spec.yml Network CIDRs, interfaces, and IP ranges
provision_config.yml OS provisioning and PXE settings
high_availability_config.yml Kubernetes HA virtual IP configuration
telemetry_config.yml Telemetry sources, bridges, and sinks
telemetry_storage_config.yml Telemetry storage resources and retention
software_config.json Software stack (K8s, Slurm, telemetry components)
local_repo_config.yml Repository mirror settings
storage_config.yml NFS storage mount configuration
omnia_config.yml Slurm and service cluster K8s settings
security_config.yml OpenLDAP authentication settings
discovery_config.yml BMC discovery and OME integration
build_stream_config.yml BuildStreaM CI/CD pipeline settings (optional)
additional_cloud_init.yml Custom cloud-init scripts (optional)

For the full procedure and parameter reference, see Configure Inputs. For detailed Slurm-specific input configuration, see Set Up Slurm.

Tip

If you need to build custom Slurm RPMs from source or host them on a local server, complete those steps first:

InfiniBand deployments

If any Slurm nodes have an InfiniBand interface and ib_network is defined in network_spec.yml:

  • The Slurm user repository must not include ucx, ucx-devel, openmpi, or openmpi-devel packages.
  • Slurm must be compiled without UCX and OpenMPI support.
  • DOCA-OFED provides its own UCX and OpenMPI stack, configured automatically during provisioning.

For InfiniBand network configuration details, see Configure InfiniBand.

Step 4 -- Prepare the OIM

Deploys the OIM infrastructure: OpenCHAMI provisioning stack, Pulp local repository, container registry, MinIO S3 storage, OpenLDAP authentication, and step-ca certificate authority.

For details, see Prepare OIM.

Run on: omnia_core container
cd /omnia/prepare_oim
ansible-playbook prepare_oim.yml

Verification -- OIM Infrastructure

After prepare_oim.yml completes, verify the OIM services on the OIM host (not inside the container):

  1. Check omnia.target status:

    Run on: OIM host
    systemctl is-active omnia.target

    Expected output: active

  2. Verify all service dependencies:

    Run on: OIM host
    systemctl list-dependencies omnia.target

    Expected output:

    Expected output
    omnia.target
● ├─minio.service
● ├─omnia_auth.service
● ├─omnia_core.service
● ├─pulp.service
● ├─registry.service
● ├─network-online.target
● │ └─NetworkManager-wait-online.service
● └─openchami.target
●   ├─acme-deploy.service
●   ├─acme-register.service
●   ├─bss-init.service
●   ├─bss.service
●   ├─cloud-init-server.service
●   ├─coresmd-coredhcp.service
●   ├─coresmd-coredns.service
●   ├─haproxy.service
●   ├─hydra-gen-jwks.service
●   ├─hydra-migrate.service
●   ├─hydra.service
●   ├─opaal-idp.service
●   ├─opaal.service
●   ├─openchami-cert-trust.service
●   ├─postgres.service
●   ├─smd-init.service
●   ├─smd.service
●   └─step-ca.service

  3. Verify all containers are running:

    Run on: OIM host
    podman ps --format "table {{.Names}}\t{{.Status}}"

    Expected output:

    Expected output
    NAMES               STATUS
bss                 Up 1 day
cloud-init-server   Up 1 day
coresmd-coredhcp    Up 1 day
coresmd-coredns     Up 1 day
haproxy             Up 1 day
hydra               Up 1 day
minio-server        Up 1 day
omnia_auth          Up 1 day
omnia_core          Up 1 day
opaal               Up 1 day
opaal-idp           Up 1 day
postgres            Up 1 day
pulp                Up 1 day
registry            Up 1 day
smd                 Up 1 day
step-ca             Up 1 day

Note

  • The minio-server container will not be present if you configured PowerScale as the S3 endpoint (s3_configurations.provider: "powerscale") in storage_config.yml. In that case, Omnia uses the external PowerScale S3 service instead of deploying a local MinIO container.
  • The omnia_auth container will not be present if openldap is not included in software_config.json.

For detailed OIM verification procedures, see Verify OIM Services.

Step 5 -- Create Local Repositories

Downloads all required RPM packages, container images, and tarballs into Pulp based on software_config.json for air-gapped provisioning.

For details, see Create Local Repos.

Run on: omnia_core container
cd /omnia/local_repo
ansible-playbook local_repo.yml

Note

Expect 45--90 minutes depending on network speed. Total download size is typically 20--40 GB.

Verification -- Local Repository Status

After local_repo.yml completes, verify that all software components were downloaded successfully by checking the software.csv status file. The components listed in this file correspond directly to the software entries configured in software_config.json.

  1. Verify x86_64 package status:

    Run on: omnia_core container
    cat /opt/omnia/log/local_repo/rhel/10.0/x86_64/software.csv

    Expected output:

    Expected output
    name,status
default_packages,success
admin_debug_packages,success
openldap,success
service_k8s,success
slurm_custom,success
csi_driver_powerscale,success
ldms,success

  2. Verify aarch64 package status (if aarch64 is included in software_config.json):

    Run on: omnia_core container
    cat /opt/omnia/log/local_repo/rhel/10.0/aarch64/software.csv

    Expected output:

    Expected output
    name,status
default_packages,success
openldap,success
slurm_custom,success

Note

The software.csv output reflects the software components configured in software_config.json. Each component with "arch": ["x86_64"] appears in the x86_64 status file, and each component with "arch": ["aarch64"] appears in the aarch64 status file. Components such as service_k8s and csi_driver_powerscale are typically x86_64-only. All entries must show success status before proceeding.

Step 6 -- Build Node Images

Builds diskless OS images for each functional group in the PXE mapping file and uploads them to MinIO (S3) for PXE boot delivery.

For details, see Build Cluster Images.

Build x86_64 Images

Run on: omnia_core container
cd /omnia/build_image_x86_64
ansible-playbook build_image_x86_64.yml

Build aarch64 Images

If your PXE mapping file contains aarch64 functional groups, you must first prepare an aarch64 build node. See Prepare aarch64 Node for the complete procedure (manual RHEL 10 installation, inventory file creation, etc.).

Run on: omnia_core container
cd /omnia/build_image_aarch64
ansible-playbook build_image_aarch64.yml -i inventory

Verification -- Boot Images in S3

After the build playbooks complete, verify the images are uploaded to MinIO (S3). Each functional group produces 3 image artifacts: rootfs (full OS root filesystem), vmlinuz (Linux kernel), and initramfs (initial RAM filesystem for PXE boot).

  1. List all boot images in S3:

    Run on: OIM host
    s3cmd ls s3://boot-images/

    Expected output (one directory per functional group plus efi-images):

    Expected output
                        DIR  s3://boot-images/efi-images/
                    DIR  s3://boot-images/login_compiler_node_x86_64/
                    DIR  s3://boot-images/service_kube_control_plane_first_x86_64/
                    DIR  s3://boot-images/service_kube_control_plane_x86_64/
                    DIR  s3://boot-images/service_kube_node_x86_64/
                    DIR  s3://boot-images/slurm_control_node_x86_64/
                    DIR  s3://boot-images/slurm_node_x86_64/
                    DIR  s3://boot-images/slurm_node_aarch64/

  2. Verify individual image artifacts for a specific functional group:

    Run on: OIM host
    s3cmd ls -Hr s3://boot-images/slurm_control_node_x86_64/
s3cmd ls -Hr s3://boot-images/efi-images/slurm_control_node_x86_64/

    Expected output:

    Expected output
    2026-06-26 11:42  1449M  s3://boot-images/slurm_control_node_x86_64/rhel-slurm_control_node_x86_64_omnia_2.2.0.0/rhel10.0-rhel-slurm_control_node_x86_64_omnia_2.2.0.0-10.0
2026-06-26 11:42    78M  s3://boot-images/efi-images/slurm_control_node_x86_64/rhel-slurm_control_node_x86_64_omnia_2.2.0.0/initramfs-6.12.0-55.82.1.el10_0.x86_64.img
2026-06-26 11:42    15M  s3://boot-images/efi-images/slurm_control_node_x86_64/rhel-slurm_control_node_x86_64_omnia_2.2.0.0/vmlinuz-6.12.0-55.82.1.el10_0.x86_64

Note

The directories listed in s3://boot-images/ correspond to the functional groups defined in your PXE mapping file. Each functional group will have exactly 3 image artifacts (rootfs, vmlinuz, initramfs). The efi-images/ directory contains the initramfs and vmlinuz boot files used during PXE network boot, while the root filesystem is stored directly under each functional group directory. If any artifacts are missing, re-run the corresponding build playbook.

Step 7 -- Provision Nodes

The provision.yml playbook provisions the cluster nodes. It configures boot scripts, cloud-init, and prepares nodes for K8s and Slurm deployment.

Run on: omnia_core container
cd /omnia/provision
ansible-playbook provision.yml

Verification -- nodes.yaml

After provision.yml completes, verify that all nodes from your PXE mapping file are present in the generated nodes.yaml file. Every node defined in pxe_mapping_file.csv should have a corresponding entry with its hostname, functional group, MAC address, and IP address.

Run on: omnia_core container
cat /opt/omnia/openchami/workdir/nodes/nodes.yaml

Expected output (one entry per node in the PXE mapping file):

Expected output
nodes:
- name: scnode
  xname: x1000c0s0b0n0
  description: JS8MN34
  nid: 1
  group: slurm_control_node_x86_64
  bmc_mac: 6c:3c:8c:85:be:04
  bmc_ip: 10.3.0.XXX
  interfaces:
  - mac_addr: 04:32:01:DD:9D:F0
    ip_addrs:
    - name: management
      ip_addr: 10.5.0.XXX
- name: snode1
  xname: x1000c0s0b1n0
  description: 1T8MN34
  nid: 2
  group: slurm_node_x86_64
  ...
- name: kcp1
  xname: x1000c0s0b5n0
  description: H94M8F3
  nid: 6
  group: service_kube_control_plane_first_x86_64
  bmc_mac: b0:7b:25:d8:4a:f4
  bmc_ip: 10.3.0.XXX
  interfaces:
  - mac_addr: BC:97:E1:F0:94:F0
    ip_addrs:
    - name: management
      ip_addr: 10.5.0.XXX
...

Note

Post execution of provision.yml, IPs and hostnames cannot be re-assigned by changing the mapping file.

Caution

  • Do not run ssh-keygen post execution of provision.yml to avoid breaking the password-less SSH channel on the OIM.
  • Do not delete the Omnia shared path or the NFS directory.

For troubleshooting boot issues, IP route conflicts, and cloud-init failures, see Provisioning Issues.

Step 8 -- PXE Boot Nodes

After provision.yml completes, PXE boot all cluster nodes (K8s and Slurm).

Option 1: Manual PXE Boot

Configure each node to boot from the network via iDRAC or BIOS settings.

Option 2: Automated PXE Boot

Sets PXE boot order on all nodes via iDRAC Redfish and reboots them.

Run on: omnia_core container
cd /omnia/utils
ansible-playbook set_pxe_boot.yml

Warning

This playbook will restart your servers and power them on if they are off. Any unsaved data will be lost.

Verification -- Cloud-Init Provisioning Status

After the nodes PXE boot, verify that cloud-init has completed on all nodes. SSH from omnia_core into each node using its hostname from the PXE mapping file (HOSTNAME column):

Run on: omnia_core container (example for 2 nodes)
ssh scnode 'cloud-init status'
ssh kcp1 'cloud-init status'

Expected output on each node:

Expected output
status: done

Note

Check every node in your cluster. Open your PXE mapping file and run ssh <HOSTNAME> 'cloud-init status' for each entry. All nodes must report status: done before proceeding.

Verification -- Kubernetes Service Cluster

SSH into any service_kube_control_plane node and verify all nodes are Ready:

Run on: omnia_core container (example)
ssh kcp1 'kubectl get nodes'

Expected output:

Expected output
NAME   STATUS   ROLES           AGE   VERSION
kcp1   Ready    control-plane   1d    v1.35.1
kcp2   Ready    control-plane   1d    v1.35.1
kcp3   Ready    control-plane   1d    v1.35.1
kn     Ready    <none>          1d    v1.35.1

Verification -- Slurm Cluster

SSH into the slurm_control_node and verify all compute nodes are idle:

Run on: omnia_core container (example)
ssh scnode 'sinfo'

Expected output:

Expected output
PARTITION  AVAIL  TIMELIMIT  NODES  STATE  NODELIST
normal*    up     infinite   2      idle   snode[1-2]

For detailed cluster verification procedures, see Verify Cluster.

Step 9 -- Deploy iDRAC Telemetry (Optional)

The telemetry.yml playbook initiates the iDRAC telemetry service on the service cluster. For prerequisites, configuration details, and collecting telemetry from external nodes, see Configure iDRAC Telemetry.

Note

This step is required only when idrac: metrics_enabled is set to true in telemetry_config.yml. It is not required for other telemetry types.

Run on omnia_core container
cd /omnia/telemetry
ansible-playbook telemetry.yml

Important

If you want to enable additional telemetry components after the first successful deployment (by updating telemetry_config.yml), and Kubernetes is already up and running, execute the telemetry.sh script on kube-control-plane at path <K8s_NFS_mount_point>/telemetry/telemetry.sh.

Step 10 -- Verify the Telemetry Pipeline

After deploying telemetry, verify that all telemetry pods and services are operational. Refer to the topics in the following table for instructions on verifying each telemetry service.

Telemetry Service Description Topic
iDRAC Verify collection and ingestion of hardware telemetry metrics. iDRAC Telemetry -- Verification
LDMS Verify collection and routing of node-level telemetry metrics. LDMS Telemetry -- Verification
PowerScale Verify collection and ingestion of storage metrics and logs. PowerScale Telemetry -- Verification
UFM Verify collection and ingestion of fabric metrics and logs. UFM Telemetry -- Verification
VAST Verify collection and ingestion of storage metrics and logs. VAST Telemetry -- Verification
OpenManage Enterprise (OME) Verify collection and routing of OME metrics and logs. OME Telemetry -- Verification

What's Next?

Your cluster is fully operational with Slurm scheduling, Kubernetes service cluster, and telemetry monitoring. Common next steps:

Configure GPU support If your compute nodes have NVIDIA GPUs, follow Slurm with GPU to enable GPU scheduling and GRES configuration.

Customize Slurm configuration Tune partitions, scheduling policies, and accounting settings using Configure Slurm.

Enable additional telemetry sources Configure iDRAC, DCGM, PowerScale, UFM, VAST, or OME telemetry collection. See Telemetry Setup for an overview of all supported sources.

Deploy PowerScale CSI driver Enable persistent storage for Kubernetes workloads using Deploy PowerScale CSI.

Run HPC benchmarks Validate cluster performance with HPL, OSU Micro-Benchmarks, and other tools using Run HPC Benchmarks.

Scale the cluster Add or remove compute nodes with Add / Remove Nodes.

Back up Slurm configuration Create timestamped backups and rollback points using Config Backup.

Enable BuildStreaM for GitOps Automate image building and deployment via CI/CD pipelines using BuildStreaM Deployment.

Info