Protect nodes from automatic restart

Shturval provides a mechanism for automatically restarting nodes in cases of changes in node configuration (NCI) or cluster updates. The drain and reboot policy is set by the shturval-node-config operator. When the node is automatically restarted, the following actions are performed:

  • Cordon;
  • Drain;
  • Reboot.

If necessary, you can protect cluster nodes from automatic restart using special labels. This is applicable for cases when some operations must be performed before draining or rebooting the cluster node.

The sequence of actions will be as follows:

  1. Set a ban on drain nodes.
  2. Make the necessary changes.
  3. Check that the node configuration has received the condition reboot required.
  4. Set the permission to apply changes without a drain node over the prohibition on drain.
  5. Wait until the node reboots or the configuration is applied.
  6. Remove permission to apply changes without drain node.

Commands must be executed using the node name instead of my-special-node.

  1. To disable the Drain node, set the node.shturval.tech/disable-drain label using the command:
Command — illustration
kubectl label node my-special-node node.shturval.tech/disable-drain=
Screenshot

Node page: disable-drain label applied

  1. To check the installation of the label, run the command:
Command — illustration
kubectl get node my-special-node -o jsonpath='{.metadata.labels}' | jq -r
Screenshot

Node labels output after disable-drain is set

Please note! Complete the necessary steps to prepare the node for restart. After completing the steps, perform drain. The `disable-drain` label only blocks the node's automatic drain.
  1. Set the permission to apply the changes using the node.shturval.tech/apply-as-is label.
Please note! The `apply-as-is` label allows changes to be applied immediately without checking whether the node has been drained.
Command — illustration
kubectl label node my-special-node node.shturval.tech/apply-as-is=
Screenshot

apply-as-is label set on the node page

  1. Check that the resource NodeConfig of the node (nodeconfig) is applied and has the status ready=true:
Command — illustration
kubectl get nodeconfig my-special-node -o jsonpath='{.status.ready}'
Screenshot

RebootRequired condition in node status

  1. Be sure to remove the node.shturval.tech/apply-as-is label from the node to disable permission to unconditionally apply changes to the node by running the command:
Command — illustration
kubectl label node my-special-node node.shturval.tech/apply-as-is-
Screenshot

Removing disable-drain label on the node page

Please note! The ```node.shturval.tech/disable-drain``` label remains installed on your node, blocking the Drain of the node.