ETCD defragmentation

Defragment etcd members from the console; defragment the leader last.

If etcd runs out of space, defragment it from the console.

1. Determining the ETCD leader in the cluster

This must be done first, since the leader must be defragmented last!

First get a list of ETCD pods.

kubectl -n kube-system get pod -l component=etcd -o wide
Output example
NAME                 READY   STATUS    RESTARTS   AGE     IP              NODE            NOMINATED NODE   READINESS GATES
etcd-10-XX-XXX-112   1/1     Running   0          4h17m   10.XX.XXX.112   10-XX-XXX-112   <none>           <none>
etcd-10-XX-XXX-144   1/1     Running   0          4h17m   10.XX.XXX.144   10-XX-XXX-144   <none>           <none>
etcd-10-XX-XXX-97    1/1     Running   0          4h17m   10.XX.XXX.97    10-XX-XXX-97    <none>           <none>

Contact any pod (for example, etcd-10-XX-XXX-112) to list etcd cluster members.

kubectl -n kube-system exec -it etcd-10-XX-XXX-112 -- etcdctl \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  endpoint status --cluster -w table
Output example
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|          ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://10.XX.XXX.144:2379 | 9b99fe49341767e1 |   3.5.9 |   84 MB |     false |      false |         2 |     182220 |             182220 |        |
|  https://10.XX.XXX.97:2379 | c27fc518bba10abd |   3.5.9 |   84 MB |      true |      false |         2 |     182222 |             182222 |        |
| https://10.XX.XXX.112:2379 | e6fa42dc23d4ac7d |   3.5.9 |   85 MB |     false |      false |         2 |     182222 |             182222 |        |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

The leader is https://10.XX.XXX.97:2379 (IS LEADER=true).

2. Defragment etcd members

Run the command to defragment the database.

kubectl -n kube-system exec -it etcd-10-XX-XXX-112 -- etcdctl \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --command-timeout=30s --endpoints=https://localhost:2379 defrag
Output example
Finished defragmenting etcd member[https://localhost:2379]

If a timeout error occurs, increase the value of –command-timeout until the command executes successfully.

Make sure the database size is reduced.

kubectl -n kube-system exec -it etcd-10-XX-XXX-112 -- etcdctl \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  endpoint status --cluster -w table
Output example
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|          ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://10.XX.XXX.144:2379 | 9b99fe49341767e1 |   3.5.9 |   84 MB |     false |      false |         2 |     198808 |             198808 |        |
|  https://10.XX.XXX.97:2379 | c27fc518bba10abd |   3.5.9 |   84 MB |      true |      false |         2 |     198808 |             198808 |        |
| https://10.XX.XXX.112:2379 | e6fa42dc23d4ac7d |   3.5.9 |   42 MB |     false |      false |         2 |     198808 |             198808 |        |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Initially, the database on node 10.XX.XXX.112 was 85 MB, but has now decreased to 42 MB.

After defragmenting the database, it is recommended to wait at least one minute so that the etcd module can recover.

Repeat for the remaining members. Defragment the leader last.

3. Check and clear NOSPACE alarms

Check if there are any NOSPACE alarms.

kubectl -n kube-system exec -it etcd-10-XX-XXX-112 -- etcdctl \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  alarm list
Output example
memberID:12345678912345678912 alarm:NOSPACE

Reset alarms.

kubectl -n kube-system exec -it etcd-10-XX-XXX-112 -- etcdctl \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  alarm disarm
← CPU reservation in a cluster
ETCD Backup and Restore →