CPU redundancy in a cluster

CPU pinning for etcd in a cluster can be configured in the GUI or software interface.

CPU reservation results in fewer available (Allocatable) CPUs on the node. This can be seen on the node page or if you describe the node (sections Capacity and Allocatable). The reservation itself does not limit the CPU in any way, but only sets “weights” (prioritization). Those. A pod without a limit will always be able to consume 100% of the CPU if no one else is claiming the CPU at that moment. Allocatable is also taken into account at the planning stage of a pod with the specified requests: if there are not enough resources for the requests, the pod will not be launched or lower-priority pods will be preempted.

Please note! We recommend that as a result of the reservation, at least 6 free CPUs should remain on the node.

CPU redundancy ingraphical interface #

In the cluster interface, go to the “Node Configuration” section, create a new NCI object. Select the “Kubelet” section. In the node selector, select the key “node-role.kubernetes.io/control-plane”.

Fill in the fields:

• CPU management policy (cpuManagerPolicy) with the value static. This value tells kubelet to bind pods to reserved CPUs.
• List of reserved CPUs (reservedSystemCPUs) in the form of a range or separated by commas, for example, “0-1”.

In the “Feature Gates” block, specify the parameters in key-value format:

• CPUManagerPolicyOptions: true
• CPUManagerPolicyAlphaOptions: true
• CPUManagerPolicyBetaOptions: true

In the “CPU Management Options” block (cpuManagerPolicyOptions), specify the parameters in key-value format:

• full-pcpus-only: “true”
• distribute-cpus-across-numa: “true”
• align-by-socket: “true”

Save the created object. Kubelet will restart automatically.

CPU reservation in the software interface #

Connect to the cluster from consoleli.

apiVersion: node.shturval.tech/v1beta2
kind: NodeConfigItem
metadata:
  name: cpu-pinning-test
spec:
  kubelet:
    cpuManagerPolicy: static
    featureGates:
      CPUManagerPolicyOptions: true
      CPUManagerPolicyAlphaOptions: true
      CPUManagerPolicyBetaOptions: true 
    cpuManagerPolicyOptions:
      align-by-socket: "true"
      distribute-cpus-across-numa: "true"
      full-pcpus-only: "true"
    reservedSystemCPUs: 0-1
  nodeconfigselector:
    node-role.kubernetes.io/control-plane: ""
  priority: 100

Apply the generated manifest.

kubectl apply -f cpu-pinning-test.yaml

Checking CPU reservation status #

To check the status of Kubelet CPU Manager.

cat /var/lib/kubelet/cpu_manager_state

{"policyName":"static","defaultCpuSet":"0-1,4-7","entries":{"3f6868ef501db41abc9cc55c40ab1020":{"etcd":"2-3"}},"checksum":4106740397}

Here you can see that:

• static policy is used
• by default, all pods are placed in a distributed pool of processors 0-1,4-7
• 2-3 CPUs are allocated for the etcd pod

If the CPU reservation is successful, there will be a message in the kubelet log.

policy_static.go:175] "Reserved CPUs not available for exclusive assignment" reservedSize=2 reserved="0-1" reservedPhysicalCPUs="0-1"

Please note! There is a possibility that socket alignment will not work on virtual machines if you set more than 1 socket per VM, since there is no way to influence the number of NUMA nodes. In this case, you need to remove the parameter: align-by-socket: “true”.

I0613 19:29:30.525539    5408 cpu_manager.go:172] "Detected CPU topology" topology={"NumCPUs":8,"NumCores":8,"NumSockets":2,"NumNUMANodes":1,"CPUDetails":{"0":
{"NUMANodeID":0,"SocketID":0,"CoreID":0}
,"1":{"NUMANodeID":0,"SocketID":0,"CoreID":1},"2":{"NUMANodeID":0,"SocketID":0,"CoreID":2},"3":{"NUMANodeID":0,"SocketID":0,"CoreID":3},"4":{"NUMANodeID":0,"SocketID":1,"CoreID":4},"5":{"NUMANodeID":0,"SocketID":1,"CoreID":5},"6":{"NUMANodeID":0,"SocketID":1,"CoreID":6},"7":{"NUMANodeID":0,"SocketID":1,"CoreID":7}}}
Jun 13 19:29:30 shturval-mycluster-cpu-master-qc2rv kubelet[5408]: E0613 19:29:30.525580    5408 container_manager_linux.go:329] "Failed to initialize cpu manager" err="new static policy error: Align by socket is not compatible with hardware where number of sockets are more than number of NUMA"
Jun 13 19:29:30 shturval-mycluster-cpu-master-qc2rv kubelet[5408]: E0613 19:29:30.525610    5408 run.go:74] "command failed" err="failed to run Kubelet: new static policy error: Align by socket is not compatible with hardware where number of sockets are more than number of NUMA"

Conditions for using reserved resources for pods #

CPU reservation results in no pods being allocated on selected nodes reserved for CPU redundancy. The exception is those pods that meet all the conditions for using guaranteed resources:

• must be present as requests and limits;
• requests and CPU limits must be integers;
• requests and limits must match.

spec:
  containers:
    - name: etcd
      resources:
        limits:
          memory: "500Mi"
          cpu: "2"
          example.com/device: "1"
        requests:
          memory: "500Mi"
          cpu: "2"
          example.com/device: "1"

If all conditions are met, then after the pod is launched, the status will indicate the QoS Class: Guaranteed type.

Please note! On the reserved processors in the cluster, pods could have been reserved/launched before the CPU was reserved. Make sure the nodes restart after applying the Kubelet settings. CPU reservation does not mean that OS processes will not run on these resources. This ensures that the kubelet does not host other pods on the reserved CPUs.