Verify Cluster¶

Verify that the Slurm cluster and Kubernetes are deployed successfully on the service cluster after booting the nodes.

Overview¶

After booting the nodes, verify the following:

Required Slurm services are active and compute nodes report an idle state.
GPU-enabled Slurm nodes complete GPU driver installation.
The Slurm PAM feature restricts SSH access to compute nodes to the duration of an active job.
Kubernetes pods and nodes on the service cluster are healthy.

Prerequisites¶

Nodes are booted and provisioned.
Slurm and/or Kubernetes have been deployed (see Setup Slurm or Setup Service K8S).
For GPU verification: GPU-enabled Slurm nodes are configured (see Slurm With Gpu).
For PAM verification: External LDAP is deployed (see Deploy External LDAP).

Procedure¶

Verify Slurm Cluster¶

On the Slurm controller node, verify the required services are running:
Run on: Slurm control node
```
systemctl status munge
systemctl status slurmctld
systemctl status slurmdbd
systemctl status mariadb
```
Confirm that each service is active (running).
Verify the node status with sinfo:
Run on: Slurm control node
```
sinfo
```
Ensure that the compute nodes are listed and the node state is idle.

Verify Slurm Cluster with GPU¶

On Slurm nodes that have GPUs, it may take some time for Slurmd to start because of the GPU driver installation. To view the logs during this process, run:
Run on: GPU compute node
```
tail -f /var/log/cloud-init-output.log
```

Note

The CUDA installation path on the OIM and nodes must be {client_share_path}/slurm/cuda.
The client_share_path is the same as mentioned in storage_config.yml for nfs_slurm.

Verify PAM Feature for Slurm¶

Slurm PAM restricts SSH access to compute nodes for non-root users. You can log in only while their job is actively running on the node. After the job is completed, you are automatically logged out.

On the login node, switch to the LDAP user:
Run on: login node
```
ssh <ldap_user>@<login_node_hostname>
sbatch job.sh
```
While the job is running, SSH as <ldap_user> to the Slurm node where the job is running. After the job is completed, <ldap_user> is logged out.

Verify Kubernetes on the Service Cluster¶

Run the following commands on the Kubernetes controller node:
Run on: K8s control plane node
```
kubectl get pods -A -o wide
kubectl get nodes -o wide
```

Verification¶

Check	Command	Expected Result
Slurm services active	`systemctl status munge slurmctld slurmdbd mariadb`	All `active (running)`
Slurm nodes idle	`sinfo`	Compute nodes listed, state `idle`
PAM restricts SSH	`ssh <ldap_user>@<node>` outside job runtime	Access denied after job completes
K8s nodes ready	`kubectl get nodes -o wide`	All `Ready`
K8s pods running	`kubectl get pods -A -o wide`	All `Running`

Next Steps¶

Slurm With GPU -- Configure GPU support for Slurm.
Deploy External LDAP -- Set up centralized authentication for the PAM feature.