Labels, annotations, and taints in Cluster API

In the Shturval platform, the node page allows assigning labels and annotations for Cluster API, infrastructure, and the node. The Taints tab allows managing taints on a specific Node.

To apply labels or taints to all nodes in a group, set them not manually on each node, but in the Cluster API resources of the node group in the management cluster:

  • for Worker groups, in MachineDeployment;
  • for the Control Plane group, in KubeadmControlPlane.

The parameters are passed to Machine, and then, if Cluster API rules allow it, synchronized to the corresponding Kubernetes Node objects.

Some labels require no additional configuration. For example, node group roles in the platform are set with the node-role.kubernetes.io/<role> label, and the node-role.kubernetes.io/ prefix is included in the list of labels that Cluster API always synchronizes from Machine to Node (Cluster API documentation). When configuring custom labels, annotations, and taints through CAPI resources, note the separate synchronization rules: not every value is automatically copied from Machine to Node, and taints require an additional parameter.

Manual changes to labels or taints on a Node can be overwritten by controllers or lost when the node is recreated.

    Labels

    Cluster API does not copy every label from Machine to Node; it only synchronizes labels allowed by its synchronization rules.

    Labels are always synchronized when they match at least one of these conditions:

    • they have the node-role.kubernetes.io/ prefix;
    • they belong to the node-restriction.kubernetes.io domain;
    • they belong to the node.cluster.x-k8s.io domain.

    For example, the node-role.kubernetes.io/infra label is copied from Machine to Node without additional configuration. For the same reason, node group roles set as node-role.kubernetes.io/<role> match the standard Cluster API rule.

    Custom labels are not synchronized by default. To allow their propagation, add the --additional-sync-machine-labels argument to the Cluster API controller manager.

    Synchronization argument

    • --additional-sync-machine-labels — a list of regular expressions for labels that must be additionally synchronized from Machine to Node.

    A label is synchronized if it matches at least one regular expression from the list. Values in the list are separated by commas:

    args:
  - --additional-sync-machine-labels=^env$,^role$,^topology\.kubernetes\.io/zone$

    To allow synchronization of all labels, use the .* regular expression.

    Use the synchronization argument only after assessing the risks, because service or temporary labels that must not affect workload scheduling can be copied to nodes.

    To configure the argument, use ShturvalServicePatch for shturval-capi.

    General scenario

    1. Prepare a ShturvalServicePatch manifest for shturval-capi.
    2. In the management cluster, upload the manifest through manifest import.
    3. Wait for ShturvalServicePatch to be applied. On the shturval-capi service page, the service status changes to Patched, and the uploaded ShturvalServicePatch appears on the Applied ShturvalServicePatch tab.

    Manifest example:

    apiVersion: ops.shturval.tech/v1beta2
kind: ShturvalServicePatch
metadata:
  name: shturval-capi-sync-labels
spec:
  shturvalServiceConfigName: shturval-capi
  patchOrder: 11000
  customvalues:
    capiControllerManager:
      extraArgs:
        - --additional-sync-machine-labels=.*

    Where to set labels

      For Worker group nodes, add labels to spec.template.metadata.labels of the MachineDeployment resource:

      apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: <machine-deployment-name>
  namespace: <namespace>
spec:
  template:
    metadata:
      labels:
        env: prod
        role: app
        topology.kubernetes.io/zone: az1

      For Control Plane group nodes, add labels to spec.machineTemplate.metadata.labels of the KubeadmControlPlane resource:

      apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
  name: <kubeadm-control-plane-name>
  namespace: <namespace>
spec:
  machineTemplate:
    metadata:
      labels:
        env: prod
        role: control-plane
        topology.kubernetes.io/zone: az1

      Application chain:

      MachineDeployment.spec.template.metadata.labels
  or KubeadmControlPlane.spec.machineTemplate.metadata.labels
    -> Machine.metadata.labels
      -> Node.metadata.labels

      At the last step, only labels allowed by Cluster API rules or by the --additional-sync-machine-labels argument are copied from Machine to Node.

      Label verification commands

      Check labels on Machine objects in the management cluster:

      kubectl get machines -n <namespace> --show-labels

      Check labels on cluster nodes:

      kubectl get nodes --show-labels

      Annotations

      For annotations, Cluster API synchronizes from Machine to Node only annotations from the node.cluster.x-k8s.io domain by default. All other annotations must be explicitly allowed with the --additional-sync-machine-annotations argument.

      Synchronization argument

      • --additional-sync-machine-annotations — a list of regular expressions for annotations that must be additionally synchronized from Machine to Node.

      An annotation is synchronized if it matches at least one regular expression from the list. Values in the list are separated by commas:

      args:
  - --additional-sync-machine-annotations=^example\.com/owner$,^ops\.shturval\.tech/maintenance-window$

      To allow synchronization of all annotations, use the .* regular expression.

      Do not expand the annotation list without necessity: other controllers and integrations can read Node annotations.

      To configure the argument, use ShturvalServicePatch for the shturval-capi service.

      General scenario

      1. Prepare a ShturvalServicePatch manifest for shturval-capi.
      2. In the management cluster, upload the manifest through manifest import.
      3. Wait for ShturvalServicePatch to be applied. On the shturval-capi service page, the service status changes to Patched, and the uploaded ShturvalServicePatch appears on the Applied ShturvalServicePatch tab.

      Manifest example:

      apiVersion: ops.shturval.tech/v1beta2
kind: ShturvalServicePatch
metadata:
  name: shturval-capi-sync-annotations
spec:
  shturvalServiceConfigName: shturval-capi
  patchOrder: 11000
  customvalues:
    capiControllerManager:
      extraArgs:
        - --additional-sync-machine-annotations=^example\.com/owner$

      Where to set annotations

        For a Worker group, add annotations to spec.template.metadata.annotations of the MachineDeployment resource:

        apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: <machine-deployment-name>
  namespace: <namespace>
spec:
  template:
    metadata:
      annotations:
        example.com/owner: platform-team

        For Control Plane, add annotations to spec.machineTemplate.metadata.annotations of the KubeadmControlPlane resource:

        apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
  name: <kubeadm-control-plane-name>
  namespace: <namespace>
spec:
  machineTemplate:
    metadata:
      annotations:
        example.com/owner: platform-team

        Application chain:

        MachineDeployment.spec.template.metadata.annotations
  or KubeadmControlPlane.spec.machineTemplate.metadata.annotations
    -> Machine.metadata.annotations
      -> Node.metadata.annotations

        At the last step, only annotations from the node.cluster.x-k8s.io domain or annotations allowed by the --additional-sync-machine-annotations argument are copied from Machine to Node.

        Annotation verification commands

        Check annotations on Machine objects:

        kubectl get machines -n <namespace> -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations}{"\n"}{end}'

        Check annotations on nodes:

        kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations}{"\n"}{end}'

        Taints

        In Cluster API, taints are set in the Machine specification. The controller copies taints from Machine to the linked Node and applies them according to the propagation field.

        Two modes are supported:

        • Always — the taint is continuously maintained on the Node. If such a taint is manually removed from the node, Cluster API adds it again during reconciliation. If the taint is removed from Machine, the controller removes it from Node.
        • OnInitialization — the taint is added to Node once during initialization. If it is later removed from Node, Cluster API does not add it again.

        Taints set in MachineDeployment or KubeadmControlPlane are passed to managed Machine objects without a node rollout. After that, they are synchronized to Node according to the propagation rules.

        How to enable taint synchronization

        Taint synchronization requires enabling the MachineTaintPropagation feature gate for the Cluster API controller.

        To enable the feature gate, use ShturvalServicePatch for the shturval-capi service.

        General scenario

        1. Prepare a ShturvalServicePatch manifest for shturval-capi.
        2. In the management cluster, upload the manifest through manifest import.
        3. Wait for ShturvalServicePatch to be applied. On the shturval-capi service page, the service status changes to Patched, and the uploaded ShturvalServicePatch appears on the Applied ShturvalServicePatch tab.

        Manifest example:

        apiVersion: ops.shturval.tech/v1beta2
kind: ShturvalServicePatch
metadata:
  name: shturval-capi-taint-propagation
spec:
  shturvalServiceConfigName: shturval-capi
  patchOrder: 11000
  customvalues:
    capiControllerManager:
      featureGates:
        MachineTaintPropagation: true

        Where to set taints

          For a Worker group, add taints to spec.template.spec.taints of the MachineDeployment resource:

          apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: <machine-deployment-name>
  namespace: <namespace>
spec:
  clusterName: <cluster-name>
  template:
    spec:
      taints:
        - key: dedicated
          value: db
          effect: NoSchedule
          propagation: Always

          For Control Plane in CAPI v1beta1, taints can be set in spec.machineTemplate.taints of the KubeadmControlPlane resource if this field is supported by the installed CRD:

          apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
  name: <kubeadm-control-plane-name>
  namespace: <namespace>
spec:
  clusterName: <cluster-name>
  version: <kubernetes-version>
  machineTemplate:
    metadata: {}
    taints:
      - key: custom-taint-key
        effect: NoSchedule
        propagation: Always
  kubeadmConfigSpec: {}

          The taints field can also be located at spec.machineTemplate.spec.taints. Before applying, check the CRD schema in the management cluster:

          kubectl explain kubeadmcontrolplane.spec.machineTemplate --api-version controlplane.cluster.x-k8s.io/v1beta1

          For KubeadmControlPlane, the kubeadm mechanism kubeadmConfigSpec.initConfiguration.nodeRegistration.taints and kubeadmConfigSpec.joinConfiguration.nodeRegistration.taints is also available. It sets taints during node registration through kubeadm and is closer in meaning to OnInitialization: it is not the same continuous CAPI reconciliation mechanism as Machine.spec.taints.

          Managed CAPI taint application chain:

          MachineDeployment.spec.template.spec.taints
  or KubeadmControlPlane.spec.machineTemplate.taints
    -> Machine.spec.taints
      -> Node.spec.taints

          Limits and reserved keys

          CAPI taints have API limits:

          • the list can contain no more than 64 taints;
          • key is required, value is optional;
          • effect is required and can be NoSchedule, PreferNoSchedule, or NoExecute;
          • specify propagation explicitly: Always or OnInitialization.

          Avoid using system keys and prefixes managed by Kubernetes:

          • node.cluster.x-k8s.io/uninitialized;
          • node.cluster.x-k8s.io/outdated-revision;
          • the node.kubernetes.io/ prefix, except node.kubernetes.io/out-of-service;
          • the node.cloudprovider.kubernetes.io/ prefix;
          • node-role.kubernetes.io/control-plane on Worker nodes;
          • node-role.kubernetes.io/master.

          These limits do not replace normal Kubernetes rules: for a pod to be scheduled on a node with taints, the pod specification must include matching tolerations. For details, see Taints, Tolerations, and Affinity.

          Taint verification commands

          Check taints on Machine objects:

          kubectl get machines -n <namespace> -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.taints}{"\n"}{end}'

          Check taints on nodes:

          kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.taints}{"\n"}{end}'

          See also

          ← Node Conditions
          Taints, Tolerances and Affinity →