Skip to content

Kubernetes Issues

Issues related to the Kubernetes service cluster, including image pulls, pod scheduling, DNS, storage, CSI drivers, control plane join, and networking.

ImagePullBackOff / ErrImagePull

Symptom

Pods fail to start with ImagePullBackOff or ErrImagePull status.

Cause
  • Docker rate limits exceeded.
  • Local repository missing required container images.
Resolution
  1. Add Docker credentials to omnia_config_credentials.yml.
  2. Ensure local_repo.yml completed successfully.

For more information, see Pull an Image from a Private Registry.

Pods not in running state

Symptom

Pods are not in Running state. Status values observed include Pending, CrashLoopBackOff, ImagePullBackOff, or OOMKilled (visible in kubectl describe pod Events section).

Cause

Pod startup failures due to resource constraints, image pull failures, or application errors.

Resolution

  1. Identify the failing pods:

    Run on: K8s control plane
    kubectl get pods --all-namespaces

  2. Delete and allow the controller to recreate:

    Run on: K8s control plane
    kubectl delete pod <pod-name> -n <namespace>

Cluster nodes reboot

Symptom

Cluster nodes reboot unexpectedly or require reboot after configuration changes.

Cause
  • Configuration changes requiring node restart.
  • Kernel updates.
  • System instability.
Resolution
  1. Wait 15 minutes for the node to rejoin the cluster.

  2. Verify cluster status:

    Run on: K8s control plane
    kubectl get nodes
kubectl cluster-info

DNS unresponsive / CoreDNS issues

Symptom

DNS resolution fails or CoreDNS is unresponsive in the cluster.

Cause
  • CoreDNS pod not running.
  • DNS configuration errors.
  • Network connectivity issues.
Resolution

Restart CoreDNS:

Run on: K8s control plane
kubectl rollout restart deployment coredns -n kube-system

PowerScale SmartConnect DNS resolution issues

Symptom

DNS resolution fails for PowerScale SmartConnect zone entries.

Cause

CoreDNS is unaware of external SmartConnect zone.

Resolution

  1. Edit the CoreDNS ConfigMap:

    Run on: K8s control plane
    kubectl -n kube-system edit configmap coredns

  2. Add a hosts block:

    Example
    hosts {
    10.x.x.x management.ps.com
    fallthrough
}

  3. Restart CoreDNS:

    Run on: K8s control plane
    kubectl rollout restart deployment coredns -n kube-system

Control-plane join fails due to certificate key expiry

Symptom

Control-plane node fails to join the cluster due to certificate key expiry.

Cause

The kubeadm certificate key expires after approximately 2 hours.

Resolution

  1. On a healthy control-plane, regenerate the join script:

    Run on: K8s control plane
    {{ k8s_client_mount_path }}/generate-control-plane-join.sh

  2. Reboot the failed node.

Static pods show stale running state after node shutdown

Symptom

After a control plane node is powered off or rebooted, static pods on the affected node may show 1/1 Running (stale) even though the node is NotReady. This is most commonly observed with kube-apiserver pods, but can affect etcd, kube-controller-manager, kube-scheduler, and kube-vip.

Note

This is an intermittent issue caused by a race condition. The behavior varies depending on shutdown timing, network conditions, and system load.

Cause

During graceful shutdown, all critical pods receive SIGTERM simultaneously. A circular dependency exists: kubelet needs the API server to update the API server's own status. When the VIP is released before kube-apiserver fully terminates, the container state remains stale.

Impact: No functional impact on cluster operations. The cluster continues to operate normally with remaining control planes. Pods are properly garbage collected based on --terminated-pod-gc-threshold.

Resolution

This behavior is expected and does not require action. When the node powers back on, pods restart automatically with incremented restart count.

Related Kubernetes issues:

  • Issue #110755 -- Kubelet doesn't finish killing pods before shutdown.
  • Issue #124448 -- GracefulNodeShutdown fails to update Pod status.
  • Issue #109531 -- Pods in Running/Terminating state after shutdownGracePeriod expiry.

Official Kubernetes documentation:

NFS-client provisioner CrashLoopBackOff

Symptom

NFS-client provisioner pod enters CrashLoopBackOff state.

Cause

NFS server not active at server_share_path.

Resolution

Ensure NFS server is active and reachable from the Kubernetes worker nodes.

PowerScale CSI controller issues

Symptom

PowerScale (Isilon) CSI controller pod in CrashLoopBackOff after node reboot.

Cause
  • CSI controller fails to reconnect to PowerScale storage after node reboot.
  • Storage connectivity issues or configuration problems.
  • PowerScale (Isilon) service unavailability.
Resolution

  1. Inspect recent logs from the controller deployment:

    Run on: K8s control plane
    kubectl logs deploy/isilon-controller -n isilon --all-containers=true | tail -n 60

  2. Restart the Isilon controller deployment:

    Run on: K8s control plane
    kubectl rollout restart deployment isilon-controller -n isilon

  3. Restart the Isilon node daemonset:

    Run on: K8s control plane
    kubectl rollout restart daemonset isilon-node -n isilon

Missing PowerScale CSI driver

Symptom

PowerScale CSI driver is not deployed or available in the cluster.

Cause

Driver not listed in software_config.json.

Resolution

  1. Add the required entry to software_config.json:

    Example
    {
  "name": "csi_driver_powerscale",
  "version": "v2.17.0",
  "arch": ["x86_64"]
}

  2. Re-run the playbook.

Info