Skip to content

Re-provision Cluster Nodes

Re-provisioning replaces the diskless image on existing cluster nodes. Since Omnia uses stateless (diskless) provisioning, nodes load their OS entirely from network-delivered images. Re-provisioning is performed by PXE booting the target nodes so they receive the latest image from the OIM.

Warning

If you deploy a fresh Slurm or Service Kubernetes cluster, ensure the NFS server share path used by the cluster is cleared manually. NFS details are configured in storage_config.yml.

Caution

Re-provisioning the Slurm Control Node or Kube Control Plane requires the entire cluster to be reprovisioned.

Prerequisites

  • The OIM is healthy and all OIM services are running (verify with Verify OIM Services).
  • NFS or PowerScale shared storage is accessible from the OIM and all cluster nodes.

Re-provision without modifications

If no changes have been made to the mapping file, software_config.json, or input configuration files, PXE boot the target nodes. The OS is automatically installed on every PXE boot. This can be done using the PXE Boot Playbook.

Run on: OIM host
ssh omnia_core
cd /omnia/utils
ansible-playbook set_pxe_boot.yml -i inventory

Re-provision with modifications

Use this procedure when you have updated the mapping file, software_config.json, or any input configuration files.

  1. Update the PXE mapping file (for mapping file discovery) or ensure nodes are configured in OME (for OME-based provisioning). Update software_config.json as required.

  2. If software_config.json has been modified, run local_repo.yml and then build new images:

    Run on: omnia_core container
    cd /omnia/local_repo
ansible-playbook local_repo.yml

cd /omnia/build_image_x86_64
ansible-playbook build_image_x86_64.yml

# For aarch64 images
cd /omnia/build_image_aarch64
ansible-playbook build_image_aarch64.yml -i inventory

  3. After images are created, run provision.yml:

    Run on: omnia_core container
    cd /omnia/provision
ansible-playbook provision.yml

  4. PXE boot the target nodes using the PXE Boot Playbook.

NFS share cleanup

When deploying a fresh cluster, you must clear the NFS share paths before re-provisioning. OIM Cleanup can be used to clear the NFS share paths. Choose one of the following approaches based on your requirements.

Reuse the same share paths

  1. Power off all servers except the OIM.

  2. From the OIM, delete all contents in the nfs share used by the cluster (identified in storage_config.yml):

    Run on: OIM host
    rm -rf <mounted_share_path>/*

  3. Run provision.yml:

    Run on: omnia_core container
    ssh omnia_core
cd /omnia/provision
ansible-playbook provision.yml

  4. PXE boot the target nodes using the PXE Boot Playbook.

Use new share paths

  1. Update mounts in storage_config.yml corresponding to nfs_storage_name under slurm_cluster or service_k8s_cluster in omnia_config.yml.

  2. Run provision.yml:

    Run on: omnia_core container
    cd /omnia/provision
ansible-playbook provision.yml

  3. PXE boot the target nodes using the PXE Boot Playbook.

Verification

Run on: Slurm control node
sinfo
Run on: Kubernetes control plane
kubectl get nodes

Info