Recovery when the IP addresses of all Control Plane nodes change

Before proceeding with recovery, install module etcd-helper.

To restore the functionality of the etcd cluster if the IP addresses on all Control Plane nodes change, follow these steps:

1. On the first Control Plane node, run the command to prepare the configuration using your parameter values:

/usr/local/bin/etcdhelper prepare --nodeip=ENTER-NEW-NODE-IP-ADDRESS

2. Add the --force-new-cluster option to the /etc/kubernetes/manifests/etcd.yaml manifest.

3. Reboot the first Control Plane node.

4. Edit the node entry in the etcd cluster.

# View the node ID
/usr/local/bin/etcdhelper exec -- member list -w table
# Update the node record
/usr/local/bin/etcdhelper exec -- member update ENTER-NODE-ID --peer-urls=https://ENTER-NEW-NODE-IP-ADDRESS:2380

5. Remove the --force-new-cluster option from the /etc/kubernetes/manifests/etcd.yaml manifest.

6. Reboot the first node.

7. Verify that the kubernetes cluster is running.

8. When restoring the management cluster, make sure that all pods in namespace shturval-backend are running. If shturval-backend-redis or shturval-backend-valkey is not running, force delete to restart.

9. When restoring a cluster (client, management) with the Shturval v2 provider, prepare new hosts and add them to the provider instance.

10. In the cluster graphical interface, open the Administration section and go to the Node Management page, click on Group Configuration Control Plane. On the ClusterAPI Configuration tab, set the Replicas Requested field to 1.

# Change the number of replicas in KubeadmControlPlane
kubectl edit KubeadmControlPlane -n CLUSTER-NAME CLUSTER-NAME-control-plane  

11. Return to the Manage Nodes page and copy the names of the nodes in the NotReady status.

# Get cluster node names
kubectl get node

12. Get the list of Machines and delete the Machines corresponding to the Control Plane nodes in the NotReady status.

# Get all cluster Machines
kubectl get machines -n CLUSTER-NAME
# Delete Machines corresponding to Control Plane nodes in `NotReady` status one by one
kubectl delete machines -n CLUSTER-NAME MACHINE-NAME

Removal takes some time. As a result, Machines, nodes and VMs will be deleted if the cluster is deployed with a provider other than Shturval v2.

Please note! In case of long-term removal, for example, more than 15 minutes, you can remove Machines manually:

• Edit the Machine resource to be deleted by removing the finalizer.
• Remove the node from the cluster and the VM in the virtualization platform.

13. When all Machines have been deleted, in the cluster graphical interface, open the Administration section and go to the Node Management page, click on Group Configuration Control Plane. On the ClusterAPI Configuration tab, set the Replicas Requested field to 3.

# Change the number of replicas in KubeadmControlPlane
kubectl edit KubeadmControlPlane -n CLUSTER-NAME CLUSTER-NAME-control-plane