NCI Object (NodeConfigItem)

Overview #

NodeConfigItem (NCI) is a Kubernetes cluster custom resource that describes a reusable configuration template for Linux nodes. In NCI, the administrator specifies what needs to be configured on the nodes, to which nodes the configuration should be distributed, and with what priority to merge it with other NCIs.

NCI is used to declaratively manage OS settings and Kubernetes components on nodes: NTP, containerd, sysctl, packages, repositories, files, users, systemd units, kubelet, kube-apiserver and other sections. Configuring NCI in the GUI, form fields, and advanced example manifests are described on the Node Configuration (NCI) page.

NCI is not applied to the node directly. The shturval-node-config module selects suitable NCI objects by selectors, writes their names in NodeConfig (NC) of specific nodes, and the std (shturvald) daemon on the node combines the configuration, applies it and reflects the result in the NodeConfig status.

When to use NCI #

Use NCI when needed:

• uniformly configure a group of nodes by label, for example, all Worker nodes or only nodes with GPUs;
• connect package repositories or private registries for containerd;
• install, remove or block OS packages;
• create files, directories, local users and SSH keys;
• configure sysctl, systemd units, NTP, certificates, kernel parameters, SELinux, swap or iptables;
• manage kubelet and kube-apiserver parameters on selected nodes;
• separate the basic configuration and specialized overrides via priority.

To view the application status of a specific NCI on a node, use the node page or the corresponding NodeConfig status.

Resource identification #

For a general overview of API groups and custom platform resources, see the Custom resources in the platform section.

Lifecycle #

1. The administrator creates or modifies NodeConfigItem.
2. Mutating webhook adds service annotations username and updatedDate for the external user. The operator’s service account is skipped without mutation.
3. NodeConfigItemReconciler selects NodeConfig whose labels match spec.nodeconfigselector.
4. The NCI name is added to the NodeConfig.spec.nodeconfigitems of each selected node.
5. The std daemon on the node reads the local init NCI and the NCI cache from the cluster, stores the cache in /etc/std/caches/<name>.json, merges the configurations in /etc/std/merged.json and applies the result.
6. The result of the application is recorded in NodeConfig.status.ncistate.
7. NodeConfigItem.status shows the aggregated picture: which NodeConfig have been selected, which ones have already been updated and whether the NCI as a whole is ready.

The relationship between NCI, NodeConfig, drain and reboot is described in detail in the module life cycle shturval-node-config.

Structure #

Minimal example:

apiVersion: node.shturval.tech/v1beta2
kind: NodeConfigItem
metadata:
  name: nodeconfigitem-sample
spec:
  nodeconfigselector:
    kubernetes.io/os: linux
  priority: 100
  files:
    - path: /etc/shturval/test
      type: directory
      mode: "0600"

metadata.name is the NCI name. Once created, the object’s name does not change. To replace an NCI with a different name, create a new object and delete the old one, subject to deletion and rollback restrictions.

Fields spec #

For a complete list of form fields and examples for the spec sections, see the Node Configuration (NCI) page.

Node selector #

spec.nodeconfigselector is a required card key: value. The controller selects those NodeConfig that have all the labels specified in the NCI selector with the same values. The format is close to nodeSelector Kubernetes.

Example of selecting all Linux Worker nodes:

spec:
  nodeconfigselector:
    kubernetes.io/os: linux
    node-role.kubernetes.io/worker: ""

If a selector matches multiple nodes, one NCI will be added to the NodeConfig.spec.nodeconfigitems of each matching node. To select nodes, use standard Kubernetes labels or custom labels. Examples of labels settings are available on the Node Configuration (NCI) page.

Precedence and merge #

There can be several NCIs on one node. Before use, the daemon combines them into the final configuration /etc/std/merged.json.

merge algorithm:

1. NCIs are sorted in ascending order by spec.priority.
2. If priority is equal, NCI comes first with an earlier CreationTimestamp.
3. If the timestamp matches, the order is determined by the name.
4. Merge is executed from lower priority to higher priority, so NCI with a large priority overwrites conflicting values.

Practical conclusion: set basic NCIs with a lower priority, and specialized overrides with a higher one. Given the same priority, a later created NCI may override conflicting values of a previously created NCI; if the creation time is the same, the last one will be the NCI with the lexicographically larger name.

nodeconfigselector and priority are not included in the final node configuration. They are used only to select nodes and join order.

NodeConfigItem status #

NodeConfigItem.status is controlled by the operator and shows the aggregated result of NCI distribution across selected nodes.

NodeConfigItem.status.ready means that the NCI has been brought to the state of applied in NodeConfig.status.ncistate. This is not an independent rollback mechanism and not a separate check of each file, package or parameter outside the agent logic. For node-specific diagnostics, see NodeConfig.status.ncistate; its fields are described in the reference book NodeConfig (NC).

Application order on the node #

After merge, the daemon applies the configuration sections in a fixed order:

1. swap
2. selinux
3. iptables
4. users
5. files
6. kubeapi
7. sysctl
8. kernelarguments
9. kubecerts 10.certificates 11.repositories 12.packages 13.runtime 14.kernelmodules 15.systemdunits
10. ntp
11. kubelet

kubeversion is applied by separate Kubernetes upgrade logic: N firstCI is added only to the bootstrap/init node, after updating it - to the remaining selected nodes.

Main configuration sections #

Repositories and packages #

repositories manages network repositories of OS packages. The repository is identified by address; state: present connects the repository, state: absent removes it from the package manager configuration.

packages installs, removes, and blocks packages. The packet is identified by name; to delete, use state: absent, to disable auto-update - setlock: true.

spec:
  repositories:
    - name: internal
      state: present
      address: https://repo.example.com/os
      keyurl: https://repo.example.com/RPM-GPG-KEY
      checkgpg: true
      checkssl: true
  packages:
    - name: nginx
      state: present
      setlock: true
    - name: old-agent
      state: absent
      ignorelock: true

Files, certificates and users #

files creates or modifies files and directories. The merge identifier is a pair of path and type. Deleting files and directories through NCI is not possible.

certificates adds trusted OS certificates. To remove a certificate, use exist: false.

users creates or modifies Linux users, groups and SSH keys. SSH keys are merged and deduplicated. There is no field to delete a user or group via NCI.

spec:
  files:
    - path: /etc/shturval/config.env
      type: file
      mode: "0640"
      owner: root
      group: root
      content: |
                ENABLED=true
  certificates:
    - name: corp-ca
      url: https://ca.example.com:443
      exist: true
  users:
    - username: admin
      homedir: /home/admin
      shell: /bin/bash
      group: admin
      sshkeys:
        - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAExample admin@example

Container Runtime #

runtimecfg controls containerd: registry hosts, config.toml, sandbox image, cgroup driver and custom sections.

For mirror authorization, use runtimecfg.registries[].license: this field is written as an Authorization header in hosts.toml.

For the source private registry login and password, add runtimecfg.configSections with the path plugins."io.containerd.grpc.v1.cri".registry.configs."<registry-name>".auth and the username and password parameters. Set exist: true on the section and on the parameters. The <registry-name> in the path must match runtimecfg.registries[].name — then the GUI shows them as one configuration. Several such sections for different registries appear as several list items.

If login and password are empty, do not create the auth section. To remove previously set credentials, send the same section with exist: false. Leave other configSections unchanged. In the GUI this section is set in the Container registry configuration block; it is not shown in Plugins.

For a detailed scenario for connecting a private registry, see How to connect a private registry.

spec:
  runtimecfg:
    registries:
      - name: registry.example.com
        host: https://registry.example.com
        capabilities: [pull, resolve]
        license: "Basic dXNlcjpwYXNzd29yZA=="
        insecure: false

For HTTP registry, be sure to specify insecure: true:

spec:
  runtimecfg:
    registries:
      - name: registry.local
        host: http://registry.local:5000
        capabilities: [pull, resolve]
        insecure: true

Source registry login and password:

spec:
  runtimecfg:
    registries:
      - name: myprivateregistry.domain.ltd
        capabilities: [pull, resolve]
    configSections:
      - exist: true
        path: plugins."io.containerd.grpc.v1.cri".registry.configs."myprivateregistry.domain.ltd".auth
        params:
          - exist: true
            key: username
            value: myLogin
          - exist: true
            key: password
            value: myPassword

To configure config.toml, use runtimecfg.configSections rather than the deprecated fields pauseimage and cgroupdriver:

spec:
  runtimecfg:
    configSections:
      - path: 'plugins."io.containerd.grpc.v1.cri"'
        params:
          - key: sandbox_image
            value: registry.k8s.io/pause:3.10
      - path: 'plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options'
        params:
          - key: SystemdCgroup
            value: true

Sections and parameters of configSections can be deleted using exist: false, but root section with path: "" cannot be deleted. To remove unnecessary registry directories that are not in spec, use runtimecfg.removeunused: true.

Sysctl, systemd and OS settings #

sysctl specifies kernel parameters as string values. They are stored in /etc/sysctl.d/9999-shturval-items.conf.

systemdunits starts, stops, enables or disables autostart of systemd units. If active or enabled is not specified, the value is treated as true.

kernelarguments controls kernel bootloader options; to remove a parameter use exist: false. kernelmodules enables autoloading of modules or places them in the blacklist. If autoload: true and blacklist: true are used simultaneously, blacklist takes precedence.

spec:
  sysctl:
    param:
      net.ipv4.ip_forward: "1"
      vm.max_map_count: "262144"
  systemdunits:
    - name: chronyd.service
      active: true
      enabled: true
  kernelmodules:
    - name: br_netfilter
      autoload: true
    - name: nouveau
      autoload: false
      blacklist: true

ntp specifies NTP servers and timezone. To apply NTP settings, the ntpd or chronyd service must be installed and enabled on the node; An example of service installation via packages and systemdunits is given on the Node Configuration (NCI) page.

Kubernetes components #

kubecerts specifies the rotation period for Kubernetes system certificates. The value of rotationperiod must be between 7 and 365 days, the default value is 30.

kubeapi controls individual kube-apiserver parameters: Admission Plugins, Feature Gates, OIDC (oidc — one issuer, still supported), AuthenticationConfiguration (authenticationConfiguration.jwt — several JWT issuers), Audit Policy, requests/limits, priority and Secrets encryption. Apply this section to Control Plane nodes. Several issuers, merge, and the switch to the file from Kubernetes 1.36 are described in shturval-node-config.

spec:
  nodeconfigselector:
    node-role.kubernetes.io/control-plane: ""
  kubeapi:
    admissionplugins:
      AlwaysPullImages: true
      PodSecurityPolicy: false

kubelet contains a subset of KubeletConfiguration parameters: CPU and memory policies, reservedSystemCPUs, systemReserved, maxPods, featureGates, TLS, synchronization frequencies, registry pull limits and other fields. For CPU Pinning scenarios, see CPU Pinning for etcd.

spec:
  kubelet:
    cpuManagerPolicy: static
    memoryManagerPolicy: Static
    topologyManagerPolicy: single-numa-node
    reservedSystemCPUs: "0-1"
    maxPods: 100

Limits and important behavior #

Removing NCI does not roll back changes #

NCI is not a rollback mechanism. If NCI is deleted or is no longer suitable for selector, the operator removes its name from NodeConfig, but already applied OS changes remain on the node: created users, files, repositories, packages, runtime settings and other elements are not rolled back automatically.

To change the state, create a new NCI or edit an existing NCI with the desired parameters. If there is a conflict, use a higher priority.If a change requires a reboot, it is coordinated by shturval-node-config; manual control is described in the section Protect nodes from automatic restart.

Removal is only available through explicit commands #

Validated ways to describe the target state “object should not exist”:

• packages[].state: absent;
• runtimecfg.removeunused: true for extra registry directories.

No confirmed delete command found for Linux user or group via users, file or directory via files, systemd unit file via systemdunits and arbitrary registry entry without removeunused.

Empty strings often don’t clear values #

In merge for repositories, packages, files, users, kernelmodules and runtime registries, many string fields are updated only if the new value is non-empty. Therefore, an empty string usually cannot clear a previously set value via a higher priority NCI.

Validation does not cover all errors #

The main validation is specified by the OpenAPI CRD schema. Certain business restrictions may only appear when applied on a host, for example:

• incorrect runtimecfg.configSections.path;
• attempt to remove root section containerd;
• HTTP registry without insecure: true;
• conflict of kubelet fields, for example maxParallelImagePulls together with serializeImagePulls: true.

View and diagnostics #

# List all NCIs in the cluster
kubectl get nodeconfigitem
kubectl get nci

# Short output with priority, applied/ready and selector
kubectl get nodeconfigitem -o wide

# Complete NCI manifest
kubectl get nodeconfigitem <nci-name> -o yaml

# List of NodeConfig, where you can see the application at the node level
kubectl get nodeconfig

# Specific node status
kubectl get nodeconfig <node-name> -o yaml

For general diagnostics of the operator and daemon, use the section Diagnostics of the shturval-node-config module. In the GUI, the list of NCIs and the ratio of applied nodes to the selected ones are displayed on the Node Configuration (NCI) page, and the NCI application status on a specific node is displayed on the Node Page.

Quick reminder #

• Use apiVersion: node.shturval.tech/v1beta2.
• Always set spec.nodeconfigselector: this is a required node selection field.
• Separate basic and custom configuration by priority.
• For containerd, prefer runtimecfg.configSections over deprecated pauseimage and cgroupdriver.
• For mirror authorization, use the YAML field license, not authorization.
• For private registry login and password, use runtimecfg.configSections with the path plugins."io.containerd.grpc.v1.cri".registry.configs."<registry-name>".auth.
• For HTTP registry, specify insecure: true.
• Do not expect that deleting NCI will rollback changes to thenode.
• To delete, use only the explicit state: absent, exist: false or removeunused: true commands if supported by the relevant section.
• For node-specific diagnostics, see NodeConfig.status.ncistate, not just kubectl get nci.