Skip to content

omnia_config.yml Reference

File path: /opt/omnia/input/project_default/omnia_config.yml

This file controls the deployment of Slurm and Kubernetes across cluster nodes.

Parameter reference

Slurm Configuration Parameters

Parameter

Details

slurm_cluster

list / elements=dict / required

List of slurm cluster configurations.

cluster_name

string / required

Unique name for the slurm cluster.

nfs_storage_name

string / required

Name of the nfs storage in storage_config.yml

vast_storage_name

string

Name of the vast storage in storage_config.yml

skip_merge

boolean

Variable indicates whether a specific configuration file path under config_sources should be used as-is without merging

config_sources

dict

Config can be a file path or inline mapping

Value can be: str (File path string) or dict (Inline configuration mapping).

node_discovery_mode

string

Node hardware discovery mode. 'homogeneous' for group-based discovery, 'heterogeneous' for individual node discovery. Default: heterogeneous

node_hardware_defaults

dict

<node_hardware_defaults_key>

dict

Hardware specifications for homogeneous node groups. Key is group name (grp0-grp100)

sockets

integer / required

Number of CPU sockets per node

Minimum: 1.

cores_per_socket

integer / required

Number of CPU cores per socket

Minimum: 1.

threads_per_core

integer / required

Number of CPU threads per core

Minimum: 1.

real_memory

integer / required

Memory in MB (exact value to use in Slurm)

Minimum: 1.

gres

string

GPU resources in format 'gpu:N' (optional)

Kubernetes Configuration Parameters

Parameter

Details

service_k8s_cluster

list / elements=dict / required

List of service Kubernetes cluster configurations.

cluster_name

string / required

Unique name for the service Kubernetes cluster.

deployment

boolean

Indicates if Kubernetes will be deployed or not. Accepted values: true or false.

etcd_on_local_disk

boolean / required

Determines whether ETCD is deployed on local disk or NFS storage. When true, ETCD is deployed on local disk on all master nodes prioritizing BOSS card if available. When false, ETCD storage is provisioned using NFS. Default: false.

k8s_cni

string / required

Kubernetes SDN network. Accepted values: calico. Default: calico.

pod_external_ip_range

string

IP range used by the loadbalancer for assigning external IPs to Kubernetes services. Ensure the IP range is not assigned to any node in the cluster.

k8s_service_addresses

string / required

Kubernetes internal network for services. This network must be unused in your network infrastructure. Default: 10.233.0.0/18.

k8s_pod_network_cidr

string

Kubernetes pod network CIDR for internal communication. This network must be unused in your network infrastructure. Default: 10.233.64.0/18.

nfs_storage_name

string

Name of the NFS client server as mentioned in storage_config.yml. Used to set up the NFS provisioner on the K8s service cluster for PV provisioning.

csi_powerscale_driver_secret_file_path

string

Absolute file path for the CSI PowerScale driver secret.yaml file. Required when using Dell PowerScale storage with the Kubernetes cluster.

csi_powerscale_driver_values_file_path

string

Absolute file path for the CSI PowerScale driver values.yaml file. Required when csi_powerscale_driver_secret_file_path is provided.

k8s_crio_storage_size

string / required

Storage size for CRI-O container runtime. Specify in Gigabytes (e.g., 10G, 15G, 100G). Default: 10G.

Usage example

File: /opt/omnia/input/project_default/omnia_config.yml
---
slurm_cluster:
  - cluster_name: slurm_cluster
    nfs_storage_name: nfs_slurm
    vast_storage_name: vast_storage
    # Optional: Override Slurm and cgroup configuration
    config_sources:
      slurm:
        SlurmctldTimeout: 60
        SlurmdTimeout: 150
        NodeName:
          - NodeName: newnode1
            CPUs: 16
            RealMemory: 64000
          - NodeName: newnode2
            CPUs: 16
            RealMemory: 64000
      cgroup:
        CgroupPlugin: autodetect
        ConstrainCores: True
        ConstrainDevices: True
        ConstrainRAMSpace: True
        ConstrainSwapSpace: True   
    # Optional: Override hardware specs for specific node groups
    node_hardware_defaults:
      grp1:
        sockets: 2
        cores_per_socket: 64
        threads_per_core: 2
        real_memory: 512000
        gres: "gpu:4"
      grp2:
        sockets: 2
        cores_per_socket: 32
        threads_per_core: 2
        real_memory: 256000


service_k8s_cluster:
  - cluster_name: service_cluster
    deployment: true
    etcd_on_local_disk: false
    k8s_cni: "calico"
    pod_external_ip_range: "172.16.107.170-172.16.107.200"
    k8s_service_addresses: "10.233.0.0/18"
    k8s_pod_network_cidr: "10.233.64.0/18"
    nfs_storage_name: "nfs_k8s"
    k8s_crio_storage_size: "20G"
    csi_powerscale_driver_secret_file_path: ""
    csi_powerscale_driver_values_file_path: ""

Info