Troubleshooting nodes

Overview

This page shows typical problems with nodes and cluster behavior when a node fails.

Behavior in case of node failure or unavailability

The cluster begins to redistribute the load from the non-working node after 30–60 seconds. This is due to the kube-controller-manager and kubelet settings.

How Kubernetes defines a dead node

  • Kubelet sends status to the API server with an interval of --node-status-update-frequency (default 10 sec).
  • Controller manager checks the status every --node-monitor-period (default 5 seconds).
  • If the Kubelet does not respond within --node-monitor-grace-period (default 40 sec), the node is marked as NotReady.

Total: from the moment of refusal to NotReady - about 45–50 seconds. Until then, the pods on the node continue to receive traffic, and errors may occur when accessing them.

Speed up reaction (optional)

You can reduce nodeStatusUpdateFrequency, node-monitor-period, node-monitor-grace-period in kubelet and kube-controller-manager configuration.

Note: Warning: values that are too small increase the risk of false positives (for example, during high load or short-term network unavailability). This may result in unnecessary load redistribution.

Common problems

Node in NotReady status

Causes: Kubelet not responding, network unavailable, lack of resources (memory, disk).

Actions:

  1. Check Node Conditions on the node page or using the command:
kubectl describe node <name>
  1. Check the kubelet logs on the node (if you have access).
  2. Check Cilium/CNI: The NetworkUnavailable condition is often associated with CNI.
  3. If necessary, delete the node with re-creation or add taint node.kubernetes.io/out-of-service=NoExecute for urgent eviction of pods.

Node in Unknown status

Causes: API server does not receive heartbeat from Kubelet (network, node is down).

Action: Same as NotReady. Check network connectivity and node status in the infrastructure.

Pods are not scheduled per node

Possible reasons:

  • Node in Cordon (Scheduling: Disabled).
  • Taints on the node - pods need appropriate Tolerations.
  • Insufficient resources (CPU, memory).
  • nodeSelector or nodeAffinity of the pod does not match the node.

Actions: on the node page, look at the Events tab; check kubectl describe node <name>, kubectl describe pod <name> (events).

Drain freezes

Reasons: DaemonSet pods, PDB blocks eviction, pods with local storage.

Actions:

  • Enable Ignore DaemonSets in the Drain settings.
  • Check PodDisruptionBudget: kubectl get pdb -A.
  • For pods with local storage, forced Drain may be required (be careful with data).

NCI is not applied to the node

Causes: invalid node selector, NCI configuration error, node not rebooted (if required).

Check the presence and status of NodeConfig on the Node Configuration tab of the node page.

Selectors, priorities and examples of NCI sections are on the Node Configuration (NCI) page.

Actions:

  1. Check the node’s labels on the Labels and Annotations tab of the node page of the node page or using the command:
kubectl get node <name> --show-labels
  1. Check the service logs shturval-node-config in the cluster where the nodes are located (in the client cluster): it is installed in each cluster and applies NCI to the nodes of this cluster. Primary preparation of nodes (distribution of NCI resources among nodes) is performed by the shturval-cluster-manager service in the management cluster; further application of the configuration on the nodes is performed by this service in the corresponding cluster.

Changes do not apply

If Cordon/Drain actions, template parameter changes, or other changes do not apply, check the status of infrastructure reconciliation on the cluster. When reconciliation is paused, changes are planned but not applied.

No scaling occurs

Check availability of infrastructure provider. If you are using shturvalv2 provider, check for free hosts with the labels specified in the host group in the selected provider instance.

MachineHealthCheck does not restore the node

Causes: the number of unhealthy nodes exceeds maxUnhealthy. The same happens when maxUnhealthy is 0, MHC objects have been deleted, or cluster reconciliation is paused. Those are ways to stop automatic recovery on purpose.

Action: check the MHC settings. If recovery is needed, set maxUnhealthy above 0, create the checks again, or resume reconciliation. Otherwise, reduce the number of unhealthy nodes (restore some manually).

Reference

← Node Configuration (NCI)
Protect nodes from automatic restart →