Defragmentation ETCD

If the ETCD is full, you must defragment it. This can only be done from the console.

1. Determining the ETCD leader in the cluster

This must be done first, since the leader must be defragmented last!

First get a list of ETCD pods.

Command
kubectl -n kube-system get pod -l component=etcd -o wide
Output Example
NAME                 READY   STATUS    RESTARTS   AGE     IP              NODE            NOMINATED NODE   READINESS GATES
etcd-10-XX-XXX-112   1/1     Running   0          4h17m   10.XX.XXX.112   10-XX-XXX-112   <none>           <none>
etcd-10-XX-XXX-144   1/1     Running   0          4h17m   10.XX.XXX.144   10-XX-XXX-144   <none>           <none>
etcd-10-XX-XXX-97    1/1     Running   0          4h17m   10.XX.XXX.97    10-XX-XXX-97    <none>           <none>

Contact any of the pods (etcd-10-XX-XXX-112 as an example) for a list of ETCD cluster members.

Command
kubectl -n kube-system exec -it etcd-10-XX-XXX-112 -- etcdctl \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  endpoint status --cluster -w table
Output Example
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|          ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://10.XX.XXX.144:2379 | 9b99fe49341767e1 |   3.5.9 |   84 MB |     false |      false |         2 |     182220 |             182220 |        |
|  https://10.XX.XXX.97:2379 | c27fc518bba10abd |   3.5.9 |   84 MB |      true |      false |         2 |     182222 |             182222 |        |
| https://10.XX.XXX.112:2379 | e6fa42dc23d4ac7d |   3.5.9 |   85 MB |     false |      false |         2 |     182222 |             182222 |        |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

From this output it is clear (IS LEADER=true) that at the moment the leader is the node https://10.XX.XXX.97:2379

2. Defragmentation of ETCD cluster members.

Run the command to defragment the database.

Command
kubectl -n kube-system exec -it etcd-10-XX-XXX-112 -- etcdctl \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --command-timeout=30s --endpoints=https://localhost:2379 defrag
Output Example
Finished defragmenting etcd member[https://localhost:2379]

If a timeout error occurs, increase the value of –command-timeout until the command executes successfully.

Make sure the database size is reduced.

Command
kubectl -n kube-system exec -it etcd-10-XX-XXX-112 -- etcdctl \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  endpoint status --cluster -w table
Output Example
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|          ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://10.XX.XXX.144:2379 | 9b99fe49341767e1 |   3.5.9 |   84 MB |     false |      false |         2 |     198808 |             198808 |        |
|  https://10.XX.XXX.97:2379 | c27fc518bba10abd |   3.5.9 |   84 MB |      true |      false |         2 |     198808 |             198808 |        |
| https://10.XX.XXX.112:2379 | e6fa42dc23d4ac7d |   3.5.9 |   42 MB |     false |      false |         2 |     198808 |             198808 |        |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Initially, the database on node 10.XX.XXX.112 was 85 MB, but has now decreased to 42 MB.

After defragmenting the database, it is recommended to wait at least one minute so that the etcd module can recover.

Repeat the steps in this paragraph with the following ETCD cluster members. We remind you that the leader must be defragmented last!

3. Check and clear triggered NOSPACE alarms.

Check if there are any NOSPACE alarms.

Command
kubectl -n kube-system exec -it etcd-10-XX-XXX-112 -- etcdctl \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  alarm list
Output Example
memberID:12345678912345678912 alarm:NOSPACE

Reset alarms.

Command
kubectl -n kube-system exec -it etcd-10-XX-XXX-112 -- etcdctl \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  alarm disarm
← CPU redundancy in a cluster
Create a cluster with external ETCD →