Skip to content

Build Cluster Image Issues

Issues related to building diskless cluster node images, S3/MinIO image storage, and architecture-specific build failures.

S3 upload fails

Symptom

The build_image_x86_64.yml or build_image_aarch64.yml playbook fails during the image upload stage. The built image is not visible in the boot-images S3 bucket.

Cause
  • The MinIO service on the OIM is not running or is unreachable.
  • S3 credentials are misconfigured.
  • Insufficient disk space on the MinIO storage volume.
Resolution
  1. Verify MinIO is running and accessible:
Run on: omnia_core container
s3cmd ls
  1. If MinIO is unreachable, restart it on the OIM host:
Run on: OIM host
systemctl restart minio.service
  1. Verify the boot-images bucket exists:
Run on: omnia_core container
s3cmd ls s3://boot-images
  1. After resolving the issue, re-run the build image playbook.

Kernel image not found in S3

Symptom
  • provision.yml fails with a kernel validation error.
  • The specified kernel_version_override is not found in S3.
Cause

The build image step did not complete successfully, or the kernel packages were not available in the Pulp repository when the image was built.

Resolution
  1. Verify that the build image step completed successfully and uploaded images to S3:
Run on: omnia_core container
s3cmd ls -Hr s3://boot-images
  1. Look for kernel and initramfs entries matching your functional group:
s3://boot-images/efi-images/<functional_group>/rhel-<functional_group>_omnia_<version>/vmlinuz-<kernel_version>
s3://boot-images/efi-images/<functional_group>/rhel-<functional_group>_omnia_<version>/initramfs-<kernel_version>.img

  1. If the expected kernel is missing, verify that the kernel packages were available in the Pulp repository before running build_image_x86_64.yml. The build process selects the latest kernel available across all configured repositories.

  2. Re-run the build image playbook to rebuild with the correct kernel:

Run on: omnia_core container
cd /omnia/build_image_x86_64
ansible-playbook build_image_x86_64.yml
  1. After the build completes, verify the new kernel image in S3 using s3cmd ls -Hr s3://boot-images and then re-run provision.yml.

Build image fails for aarch64 — missing inventory

Symptom

The build_image_aarch64.yml playbook fails with: "aarch64 functional groups detected in pxe_mapping_file but no hosts found in 'admin_aarch64' inventory group" or "The inventory group 'admin_aarch64' does not exist or has no hosts."

Cause

The PXE mapping file contains aarch64 functional groups, but the playbook was run without an inventory file containing the [admin_aarch64] group.

Resolution
  1. Create an inventory file with the [admin_aarch64] group containing exactly one ARM admin node:
[admin_aarch64]
<arm_admin_node_ip>
  1. Re-run the build image playbook with the inventory file:
Run on: omnia_core container
cd /omnia/build_image_aarch64
ansible-playbook build_image_aarch64.yml -i inventory

Note

The [admin_aarch64] group must have exactly one host. NFS must be configured on the OIM for aarch64 image building.

Repository sync issues

Symptom
  • local_repo.yml fails to sync the additional kernel repositories.
  • Kernel packages are not available in Pulp after sync.
  • build_image_x86_64.yml builds an image with an older kernel than expected.
Cause

Repository URLs in local_repo_config.yml are incorrect or unreachable, or RHEL subscription certificates are invalid or expired.

Resolution
  1. Verify repository URLs are correct and accessible from the omnia_core container:
Run on: OIM host
podman exec -it omnia_core curl -I <repository_url>
  1. For RHEL subscription (EUS) repositories, verify that the entitlement certificates are valid and correctly placed:
Run on: omnia_core container
ls -la /opt/omnia/rhel_repo_certs/
  1. Validate kernel packages are available in the synced Pulp repository. From within the omnia_core container, list the repository distributions:
Run on: omnia_core container
pulp rpm distribution list
  1. Query the Pulp content endpoint to check for kernel packages. Replace <oim_admin_ip> with the OIM admin IP and <repo_name> with the distribution name from the previous step:
Run on: omnia_core container
curl -k https://<oim_admin_ip>:2225/pulp/content/opt/omnia/offline_repo/cluster/x86_64/rhel/10.0/rpms/<repo_name>/Packages/k/ | grep kernel
  1. If no kernel packages are found, correct the repository URLs in local_repo_config.yml and re-run local_repo.yml.

Images not created for all functional groups

Symptom

After running the build image playbook, s3cmd ls -Hr s3://boot-images shows images for some functional groups but not all groups defined in the mapping file.

Cause
  • The mapping file contains functional groups that do not match the target architecture of the playbook being run.
  • local_repo.yml was not executed with software packages for all required functional groups.
Resolution
  1. Verify which functional groups are defined in the mapping file:
Run on: omnia_core container
cat /opt/omnia/input/project_default/pxe_mapping_file.csv

  1. Ensure local_repo.yml was executed with software_config.json that includes software for all required architectures and functional groups.

  2. Re-run the appropriate build image playbook and verify images:

Run on: omnia_core container
s3cmd ls -Hr s3://boot-images

Info