Skip to content

Provisioning Issues

Issues related to PXE booting, node discovery, cloud-init configuration, the discovery.yml playbook, and local repository (Pulp) operations.

PXE boot failures

Node hangs at nm-wait-online-initrd.service

Symptom

Node hangs during boot at the nm-wait-online-initrd.service stage.

Cause

IP address conflict with an old node.

Resolution
  1. Ensure the old node is powered off or disconnected.
  2. Verify the IP address is unused on the network.
  3. Re-run provision.yml.

PXE boot timeout (TFTP/service timeout)

Symptom

PXE boot process times out with TFTP or service timeout errors:

Expected output
PXE-E32: TFTP open timeout
PXE-T02: TFTP packet timeout
Cause
  • PXE NIC not configured in BIOS.
  • Extra NIC interfering with the boot process.
  • Multiple PXE servers on the same network.
Resolution
  1. Configure BIOS: navigate to Network Settings > PXE Device and assign the correct active NIC.
  2. Remove or disable any extra NIC until after boot completion.
  3. Verify no rogue PXE/DHCP servers exist on the admin network.

Target server unreachable after PXE boot

Symptom

Target server becomes unreachable after PXE boot completes.

Cause
  • POST errors on the target server.
  • F1 hardware prompts blocking boot.
  • Boot stalls due to hardware issues.
Resolution
  1. Log in to iDRAC and check console output.
  2. Clear errors or disable POST prompts.
  3. Hard reboot the server.
  4. Disable PXE temporarily if needed to bypass boot loops.

Root login fails after provisioning

Symptom

Unable to log in as root via SSH. Error messages include:

  • WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!
  • Permission denied (publickey,gssapi-keyex,gssapi-with-mic)
  • ssh: connect to host <ip> port 22: Connection refused
Cause
  • Outdated SSH key in ~/.ssh/known_hosts.
  • cloud-init not rendered on the target node.
Resolution

  1. Remove the stale SSH key:

    Run on: OIM host
    ssh-keygen -R <hostname>

  2. Retry login or reprovision the node.

cloud-init issues

Symptom

Nodes boot the OS successfully but post-boot configuration fails. The node is accessible via console but network settings, hostname, or SSH keys are not configured correctly.

Cause
  • The cloud-init data source is not configured.
  • The cloud-init configuration file has syntax errors.
  • The cloud-init service was disabled or removed from the OS image.
Resolution

  1. Check cloud-init status on the affected node:

    Run on: compute node
    cloud-init status --long

  2. Review cloud-init logs:

    Run on: compute node
    cat /var/log/cloud-init.log
cat /var/log/cloud-init-output.log

  3. If cloud-init was disabled, re-enable it:

    Run on: compute node
    systemctl enable cloud-init
cloud-init clean
reboot

Local repository and Pulp issues

local_repo.yml download failures

Symptom

The local_repo.yml playbook fails during package download, displaying errors such as "TASK [parse_and_download : Display Failed Packages]" or indicating that specific software packages could not be downloaded.

Cause
  • Incorrect URLs in software JSON configuration files.
  • Docker pull limit reached or invalid Docker credentials.
  • Insufficient disk space on Pulp NFS storage.
  • Unreachable software repositories.
Resolution
  1. Verify and correct URLs in the software JSON configuration files.
  2. Provide valid Docker credentials in input/omnia_config_credentials.yml.
  3. Ensure adequate disk space is available on Pulp NFS storage.
  4. Re-run the local_repo.yml playbook.

Log analysis for download failures:

  • Overall download status:

    Example
    /opt/omnia/log/local_repo/<cluster_os>/<cluster_os_version>/<arch>/software.csv

    Example: /opt/omnia/log/local_repo/rhel/10.0/x86_64/software.csv

  • Per-software task results:

    Example
    /opt/omnia/log/local_repo/rhel/10.0/x86_64/<sw>_task_results.log

    Example for OpenLDAP: /opt/omnia/log/local_repo/rhel/10.0/x86_64/openldap_task_results.log

  • Package-level status:

    Example
    /opt/omnia/log/local_repo/<cluster_os>/<cluster_os_version>/<arch>/<sw>/status.csv

    Example: /opt/omnia/log/local_repo/rhel/10.0/x86_64/openldap/status.csv

  • Detailed failure information. View the reason a job was unsuccessful in the package_status_<pid>.log file referenced in the <sw>_task_results.log:

    Example
    /opt/omnia/log/local_repo/rhel/10.0/x86_64/<sw>/logs/package_status_<pid>.log

    Example: /opt/omnia/log/local_repo/rhel/10.0/x86_64/openldap/logs/package_status_858667.log

Note

If local_repo.yml completes without any package download failures, a Successful message is displayed.

Failure when re-run multiple times

Symptom

The local_repo.yml playbook fails when re-run multiple times in quick succession.

Cause

Pulp container resource saturation.

Resolution

Allow the system to idle approximately 1 hour before re-running.

Pulp reset password failed

Symptom

Pulp reset password operation fails during prepare_oim.yml execution.

Cause
  • NFS Storage Export Configuration (PowerScale): Missing or incorrect settings for nfsv4-no-names, nfsv4-no-domain, nfsv4-no-domain-uids, and nfsv4-allow-numeric-ids.
  • Inconsistent UID and GID mappings between NFS server and client.
  • Missing no_root_squash option in NFS export configuration.
  • NFS server connectivity issues or firewall blocking ports 2049, 111, and 20048.
Resolution

Verify the NFS export configurations and settings mentioned above, then re-run the prepare_oim.yml playbook.

EPEL repository instability

Symptom

EPEL repository is unstable or unavailable during package installation.

Cause

EPEL repository server issues or network connectivity problems.

Resolution
  • If no packages depend on EPEL, remove the EPEL URL from the configuration.
  • If required, wait for repository stability or host EPEL packages locally.

Intermittent local repository sync failure due to non-persistent iptables rules

Symptom

Local repository sync fails intermittently due to blocked outbound internet access from containers.

Cause

iptables rules on the OIM node are not persistent. After OIM startup, restrictive iptables policies block outbound internet access from containers.

Resolution

As a workaround, relax the iptables default policies on the OIM node:

Run on: OIM host
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT

Connectivity issues

Symptom

The local_repo.yml playbook fails with connectivity errors.

Cause

The OIM was unable to reach a required online resource due to a network glitch.

Resolution

Verify all connectivity and re-run the playbook.

Software installation fails with checksum error

Symptom

Software installation fails with a checksum error.

Cause

A local repository for the software has not been configured by the local_repo.yml playbook.

Resolution
  1. Re-run the local_repo.yml playbook with proper inputs to download the software package to the Pulp repository.
  2. Once the local repository has been configured successfully, re-run the failed installation script.

Boot issues on provisioned nodes

Symptom

A provisioned node fails to boot correctly, or post-boot configuration (hostname, network, SSH keys) is incomplete or missing.

Cause
Resolution
  1. Check the cloud-init output log on the affected node:
cat /var/log/cloud-init-output.log
  1. Review the provisioning log on the OIM:
cat /opt/omnia/log/provision.log

  1. If cloud-init completed with errors, re-run provision.yml after fixing the root cause.

  2. If the hostname or root password is not configured because cloud-init was not loaded in time, wait 5 minutes and retry provisioning the node. If the issue persists, redeploy the cluster after running the oim_cleanup.yml playbook.

IP route conflict after provisioning

Symptom

After provisioning, nodes lose connectivity on the admin network or cannot reach the OIM, while the public/internet NIC works (or vice versa).

Cause

An IP route conflict exists between the admin network and an additional NIC (for example, an internet-facing NIC). Both NICs may have overlapping default routes.

Resolution
  1. List current routes on the affected node:
ip route show
  1. Delete the conflicting admin route or adjust route priority:
# Delete conflicting route
ip route del <conflicting_route>

# Or set metric to prioritize one route over another
ip route add <network> via <gateway> dev <nic> metric <priority>
  1. To make the change persistent, update the network configuration files for the appropriate NIC.

Info