Kubernetes
⦾ Why do static pods show “Running” status after a control plane node is powered off or rebooted?
Potential Causes: This is a known Kubernetes limitation with graceful node shutdown. When a control plane node is powered off, shut down, or rebooted, kubelet may not have enough time to update pod status to the API server before the node shuts down. This creates a race condition where the behavior is intermittent and unpredictable:
The node is marked as
NotReady.Static pods (
kube-apiserver,etcd,kube-controller-manager,kube-scheduler,kube-vip) show stale “Running” status.Container state remains as “running” even though containers are stopped.
This occurs due to a race condition during shutdown. Kubelet attempts to update pod status to the API server, but the API server itself is shutting down or the VIP (kube-vip) becomes unavailable before the status update completes, creating a circular dependency. The behavior varies - sometimes pods show correct status, sometimes they show stale “Running” status, depending on timing and network conditions.
Impact:
No functional impact on cluster operations
Cluster continues to operate normally with remaining control planes
Pods are automatically garbage collected based on
--terminated-pod-gc-threshold=5When node powers back on, pods restart automatically
Resolution: This is expected behavior and does not require action. For detailed information, see Static Pods Show Stale “Running” State After Node Shutdown in the Troubleshooting Guide.
Related Kubernetes Issues:
Official Documentation: Kubernetes Node Shutdowns
If you have any feedback about Omnia documentation, please reach out at omnia.readme@dell.com.