Labels, annotations, and taints in Cluster API

Annotations #

For annotations, Cluster API synchronizes from Machine to Node only annotations from the node.cluster.x-k8s.io domain by default. All other annotations must be explicitly allowed with the --additional-sync-machine-annotations argument.

Synchronization argument #

• --additional-sync-machine-annotations — a list of regular expressions for annotations that must be additionally synchronized from Machine to Node.

An annotation is synchronized if it matches at least one regular expression from the list. Values in the list are separated by commas:

args:
  - --additional-sync-machine-annotations=^example\.com/owner$,^ops\.shturval\.tech/maintenance-window$

To allow synchronization of all annotations, use the .* regular expression.

To configure the argument, use ShturvalServicePatch for the shturval-capi service.

General scenario #

1. Prepare a ShturvalServicePatch manifest for shturval-capi.
2. In the management cluster, upload the manifest through manifest import.
3. Wait for ShturvalServicePatch to be applied. On the shturval-capi service page, the service status changes to Patched, and the uploaded ShturvalServicePatch appears on the Applied ShturvalServicePatch tab.

Manifest example:

apiVersion: ops.shturval.tech/v1beta2
kind: ShturvalServicePatch
metadata:
  name: shturval-capi-sync-annotations
spec:
  shturvalServiceConfigName: shturval-capi
  patchOrder: 11000
  customvalues:
    capiControllerManager:
      extraArgs:
        - --additional-sync-machine-annotations=^example\.com/owner$

Where to set annotations #

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: <machine-deployment-name>
  namespace: <namespace>
spec:
  template:
    metadata:
      annotations:
        example.com/owner: platform-team

apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
  name: <kubeadm-control-plane-name>
  namespace: <namespace>
spec:
  machineTemplate:
    metadata:
      annotations:
        example.com/owner: platform-team

Application chain:

MachineDeployment.spec.template.metadata.annotations
  or KubeadmControlPlane.spec.machineTemplate.metadata.annotations
    -> Machine.metadata.annotations
      -> Node.metadata.annotations

At the last step, only annotations from the node.cluster.x-k8s.io domain or annotations allowed by the --additional-sync-machine-annotations argument are copied from Machine to Node.

Annotation verification commands #

Check annotations on Machine objects:

kubectl get machines -n <namespace> -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations}{"\n"}{end}'

Check annotations on nodes:

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations}{"\n"}{end}'

Taints #

Worker-group taints are set on the Taints tab or in spec.template.spec.taints of the MachineDeployment. Cluster API copies them from Machine to the linked Node according to the propagation field.

The MachineTaintPropagation feature gate of the Cluster API controller is enabled in the platform by default. There is nothing extra to turn on. In Cluster API 1.13 this propagation works for Machine, MachineSet, and MachineDeployment, and does not cover KubeadmControlPlane (Experimental features, Cluster API 1.13).

Two modes are supported:

• Always — the taint is continuously maintained on the Node. If you remove it from the node by hand, Cluster API adds it again during reconciliation. If you remove the taint from MachineDeployment, the controller removes it from Node. If the mode was OnInitialization and you change it to Always, the taint is applied to nodes that already exist.
• OnInitialization — the taint is added to Node once, when the node is created. Nodes that are already running do not receive it. If it is later removed from Node, Cluster API does not put it back. If you delete this taint from MachineDeployment, it stays on nodes that were already created.

Taints from MachineDeployment are passed to the managed Machine objects without recreating nodes, and from there to Node according to the propagation rules.

Where to set taints #

For a Worker group, add taints to spec.template.spec.taints of the MachineDeployment resource:

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: <machine-deployment-name>
  namespace: <namespace>
spec:
  clusterName: <cluster-name>
  template:
    spec:
      taints:
        - key: dedicated
          value: db
          effect: NoSchedule
          propagation: Always

Application chain:

MachineDeployment.spec.template.spec.taints
  -> Machine.spec.taints
    -> Node.spec.taints

Limits and reserved keys #

CAPI taints have API limits:

• the list can contain no more than 64 taints;
• key is required, value is optional;
• effect is required and can be NoSchedule, PreferNoSchedule, or NoExecute;
• specify propagation explicitly: Always or OnInitialization.

Avoid using system keys and prefixes managed by Kubernetes:

• node.cluster.x-k8s.io/uninitialized;
• node.cluster.x-k8s.io/outdated-revision;
• the node.kubernetes.io/ prefix, except node.kubernetes.io/out-of-service;
• the node.cloudprovider.kubernetes.io/ prefix;
• node-role.kubernetes.io/control-plane on Worker nodes;
• node-role.kubernetes.io/master.

These limits do not replace normal Kubernetes rules: for a pod to be scheduled on a node with taints, the pod specification must include matching tolerations. For details, see Taints, Tolerations, and Affinity.

Taint verification commands #

Check taints on Machine objects:

kubectl get machines -n <namespace> -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.taints}{"\n"}{end}'

Check taints on nodes:

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.taints}{"\n"}{end}'