Node problem detection (shturval-npd)
About the service
| Parameter | Value |
|---|---|
| GUI name | shturval-npd |
| System name (identifier) | shturval-npd |
| Cluster assignment | A daemon on each Linux node publishes node problems as Node conditions and Events, and separately exposes Prometheus metrics. |
| Critical | No |
| Public repository | Node Problem Detector |
| Dependencies | — |
Purpose
shturval-npd reports node problems on cluster nodes to Kubernetes. On each Linux node, the node-problem-detector daemon finds the problem and writes it to the apiserver, as conditions and Events on the Node. The daemon runs as a DaemonSet. There is no CRD: after installation you do not create objects.
Until the problem is written to the Node, new pods are still scheduled there. Once the daemon writes it to the Node, Kubernetes can see the problem. The daemon can detect, for example:
- NTP on the node is unavailable;
- the CPU, memory, or disk has failed;
- the kernel has hung or the filesystem is corrupt;
- the container runtime does not respond.
By default the daemon reads kernel messages from /dev/kmsg and the /var/log journal and writes node conditions. The monitor files are built into the image, in /config. The set includes kernel-monitor.json (kernel hang, XFS shutdown, hardware error), readonly-monitor.json (the filesystem was remounted read-only), and docker-monitor.json (the dockerd journal). The Shturval platform runs containers with containerd and does not use Docker, so the docker monitor is on but has no separate signal to report. Metrics are available on port 20257. ServiceMonitor is disabled.
The service is not in a quick-install group. Set the configuration in customvalues when installing the service, or through ShturvalServicePatch (SSP).
The Pod must be privileged (securityContext.privileged: true): without that, kernel messages from the /dev/kmsg device cannot be read.
While the service is running, a periodic health-checker returns a condition to False when the check succeeds again. A condition from a log monitor stays after the log line matched and does not return to False on its own. After the service is removed, conditions remain on the Node until they are cleared by editing the node status (kubectl patch).
Where to configure
Services and repositories → Available charts → chart shturval-npd → Install.
If the service is already installed: Installed services → shturval-npd → Manage.
The namespace is chosen at install time. Values have no separate key for it.
Node conditions
When shturval-npd is installed, the daemon starts on Linux nodes. It reads local signals and adds conditions to the Node:
| Condition | Meaning |
|---|---|
ReadonlyFilesystem | The filesystem was remounted read-only |
KernelDeadlock | The Linux kernel is in a deadlock |
XfsShutdown | The XFS filesystem has shut down |
CperHardwareErrorFatal | A hardware error with fatal severity |
CorruptDockerOverlay2 | overlay2 storage is corrupt |
This is the default set. It comes from the kernel monitor, the read-only filesystem monitor, and the docker monitor. The Node still has the built-in Kubernetes conditions: Ready, MemoryPressure, DiskPressure, PIDPressure, and NetworkUnavailable.
To monitor frequent kubelet, docker, and containerd restarts on nodes, container runtime and kubelet availability, and the kernel waiting for a network device to be released, set the service customvalues or an SSP. The example below adds these conditions to the Node:
| Condition | Meaning |
|---|---|
FrequentKubeletRestart | kubelet restarts often |
FrequentDockerRestart | docker restarts often |
FrequentContainerdRestart | containerd restarts often |
FrequentUnregisterNetDevice | The kernel often waits for a network device to be released |
ContainerRuntimeUnhealthy | The container runtime does not respond |
KubeletUnhealthy | kubelet does not pass the health check |
Parameters (customvalues)
| Parameter | Default | Description |
|---|---|---|
settings.log_monitors | kernel-monitor.json, readonly-monitor.json, docker-monitor.json | JSON log-monitor list from the image directory /config. The controller consolidates extra paths from customvalues with the default set |
settings.custom_plugin_monitors | [] | JSON list of health-checker and log-counter checks. An empty list adds no conditions |
settings.extraArgs | unset | Extra flags of the node-problem-detector process. For example, --config.system-stats-monitor |
hostNetwork | false | Pod network. With true, the kubelet check opens port 10248 on the node. With false, that port belongs to the Pod, and kubelet is not reachable there |
extraVolumes, extraVolumeMounts | unset | Volumes from the node: the containerd socket and the crictl binary for the CRI check |
settings.prometheus_address | 0.0.0.0 | Address where the process serves /metrics |
| Prometheus port | 20257 | Metrics Service port |
settings.heartBeatPeriod | 5m0s | How often conditions on the Node are refreshed even when they did not change |
metrics.enabled, serviceMonitor.enabled | false | Metrics Service and ServiceMonitor |
securityContext.privileged | true | Privileged Pod, access to /dev/kmsg |
Do not pass --system-log-monitors together with --config.system-log-monitor in settings.extraArgs: the process exits. --port=0 and --prometheus-port=0 with an empty Stackdriver path stop the process as well: no exporter is left.
Example: extra monitors
By default the containerd socket is not mounted, hostNetwork is off, and no custom plugin is passed. The customvalues block below can go into the service specification at install time or into an SSP. It turns on checks from files in the image and /proc metrics collection. Enable only the files you need: do not assemble the whole /config directory at once.
hostNetwork: true is required for the kubelet check: health-checker calls localhost:10248. Without the node network that address is the Pod, not kubelet.
ShturvalServicePatch example: extra monitors
apiVersion: ops.shturval.tech/v1beta2
kind: ShturvalServicePatch
metadata:
name: shturval-npd-monitors
spec:
shturvalServiceConfigName: shturval-npd
customvalues:
extraVolumeMounts:
- mountPath: /var/run/containerd/containerd.sock
name: containerd
readOnly: true
- mountPath: /bin/crictl
name: crictl
readOnly: true
extraVolumes:
- name: containerd
hostPath:
path: /var/run/containerd/containerd.sock
type: Socket
- name: crictl
hostPath:
path: /bin/crictl
type: File
hostNetwork: true
settings:
custom_plugin_monitors:
- /config/health-checker-containerd.json
- /config/systemd-monitor-counter.json
- /config/kernel-monitor-counter.json
- /config/network-problem-monitor.json
- /config/health-checker-kubelet.json
extraArgs:
- --config.system-stats-monitor=/config/system-stats-monitor.json
log_monitors:
- /config/kernel-monitor.json
- /config/readonly-monitor.json
- /config/systemd-monitor.json
Files from the example are written to the Node as follows:
| File | What it adds to the Node |
|---|---|
kernel-monitor.json, readonly-monitor.json | The same conditions as at install, except CorruptDockerOverlay2 |
systemd-monitor.json | No permanent conditions: temporary Events about kubelet, docker, and containerd starts |
systemd-monitor-counter.json | FrequentKubeletRestart, FrequentDockerRestart, FrequentContainerdRestart |
kernel-monitor-counter.json | FrequentUnregisterNetDevice |
health-checker-kubelet.json | KubeletUnhealthy |
health-checker-containerd.json | ContainerRuntimeUnhealthy. The socket volume and the node crictl binary are required to call CRI |
network-problem-monitor.json | No permanent condition |
system-stats-monitor.json | Does not write conditions. It is passed through settings.extraArgs |
docker-monitor.json is not in this list. It stays in the default set. The platform does not run Docker, so this monitor has no separate signal to report. A condition already written by a log monitor is not cleared by itself.
The constructor builds a customvalues block for the service specification or for an SSP. Lists in settings are entered as space-separated values. Each listed value has a note of what the file does. Enable only the files you need.
Build settings
The customvalues block is built in the browser and is not sent to a server. Values are cleared when you leave the page.
Paste the block into the service customvalues or into spec of a ShturvalServicePatch.
How to deploy this service — How to deploy a new service.