Provider Shturval v2 (shturval-capsm)

About the service #

For reference: For a list of Shturval* objects, see the section Custom node life cycle resources - Shturval v2.

Contents #

1. Review
2. Architecture
3. Service installation
   • Where to configure
4. Diagnostics
5. Lifecycle
6. Operator parameters
7. CRD/API reference

1. Review #

shturval-manual is a Kubernetes operator that implements the CAPI infrastructure provider role for the Shturval platform. The operator manages bare-metal and SSH nodes: automatically registers physical hosts, monitors their readiness, associates with Cluster API objects and performs initial initialization (bootstrap) of Kubernetes nodes.

Place in the chain of command #

Key Features #

Note: The shturval-capsm provider (Shturval v2 / manual) is designed for the x86_64 architecture. Before deployment, ensure that the target nodes conform to the supported architecture.

Controllers and webhook #

The operator registers four controllers and one mutating webhook:

Operating constraints #

2. Architecture #

Workload Components #

Pod Security Settings #

Under deploymentThis is in accordance with the Pod Security Standards: restricted policy:

RBAC #

The operator operates on behalf of ServiceAccount controller-manager, which is assigned ClusterRole manager-role through ClusterRoleBinding.

Additionally, aggregated roles are provided for each CRD type with the suffixes -admin, -editor, -viewer.

Interaction with external systems #

The operator connects via SSH to physical nodes whose addresses are specified in ShturvalProvider.spec.hosts. Kubernetes Secrets are used for authentication, referenced by ShturvalProvider.spec.credentials.secretRef and, optionally, ShturvalHost.spec.credentials.ssh.

A secret of type Opaque with data for connecting via SSH is created in the management cluster in namespace shturval-capsm. The name is formed from the name of the Shturval v2 provider instance with the suffix capsm-v2-provider-secret.

Example Secret:

apiVersion: v1
kind: Secret
metadata:
  name: provider-name-capsm-v2-provider-secret
  namespace: shturval-capsm
type: Opaque
data:
  sshKey: LS0tLS1CRUdJTiBPUEVOU1NIIFBSSVZBVEUgS0VZLS0tLS0KdXNlcnNzaGtleQotLS0tLUVORCBPUEVOU1NIIFBSSVZBVEUgS0VZLS0tLS0=
  sshUser: c2h0dXJ2YWw=
  sshPassword: bXlwYXNzd29yZA==

The private key must be unencrypted, without passphrase: password protection of the key is not supported.

3. Installing the service #

The CAPSM provider is an infrastructure provider Cluster API (infrastructure.cluster.x-k8s.io). In the Shturval platform, the connection and life cycle of such a provider in the management cluster is ensured by cluster-manager; ShturvalServiceConfig (SSC) objects for provider charts are created and maintained as part of the platform configuration, rather than as an arbitrary manual installation of a service chart.

In the production environment, the provider is represented by two SSCs:

• shturval-capsm — operator’s main chart;
• shturval-capsm-crds — CRD chart of the provider module.

The CRD chart is allocated to a separate SSC so that the helm.sh/resource-policy: keep annotation protects the CRD from being deleted when the main chart is updated.

For correct operation, the CRD of the main Cluster API (shturval-capi-crds and the associated component shturval-capi) must be available in the management cluster.

Below are the chart names and the correspondence of spec.customvalues to the helm/main/values.yaml file of the main chart - for comparison with the release and configuration via ShturvalServicePatch (SSP). This does not duplicate the full deployment scenario via the cluster-manager UI/API.

If the service was disabled during platform deployment, the provider’s SSCs could receive an absence mode (mode: absent). In order for the provider to be available to work with the Shturval v2 infrastructure, first switch the shturval-capsm-crds mode to auto, then shturval-capsm to auto. You can change the mode in the platform GUI on the Providers page.

In the GUI, this action is performed by the service mode switch. The screenshot shows the selection of the Auto mode for the provider service.

3.1 Customvalues parameters #

The structure of the main chart values is described by the spec.customvalues field of the corresponding ShturvalServiceConfig and coincides with helm/main/values.yaml of the shturval-capsm chart. Full list of fields specified through this file:

Recommendation: do not edit the ShturvalServiceConfig object manually to change limits, nodeSelector, operator image, etc.Other release parameters - such an object relates to the platform configuration and can be overwritten during updates. Make all changes to the behavior of the service through ShturvalServicePatch (SSP), which overlays spec.customvalues on top of the base SSC.

Webhook #

By default, webhook ShturvalCluster is enabled. To disable, set the environment variable ENABLE_WEBHOOKS=false in the container manager. Webhook requires a TLS certificate; the path is specified via --webhook-cert-path (see section 6).

Creating ShturvalProvider #

ShturvalProvider — global (cluster-scoped) provider configuration object. When creating ShturvalProvider, the controller automatically creates ShturvalHost objects for each address from spec.hosts. The object must exist before clusters and machines are created.

Example of minimum ShturvalProvider:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: ShturvalProvider
metadata:
  name: shturval-demo
spec:
  systemRegistry: registry.example.com
  releaseVersion: "2.14"
  licenseToken: "<license token>"
  credentials:
    secretRef:
      name: shturval-demo-provider-secret
      namespace: shturval-capsm
  hosts:
    - address: 192.168.1.10
      labels:
        role: worker
        zone: dc1
    - address: 192.168.1.11
      labels:
        role: master
        zone: dc1
  paused: false

spec.releaseVersion sets the platform version for host bootstrap (downloading stc-<releaseVersion>). For a single host the version is set in spec.hosts[].releaseVersion; if the field is empty, the provider version is used. In the interface this is the Version field of the provider instance or the host. Specify the full version, for example 2.13.1.

After creating ShturvalProvider, the controller creates ShturvalHost with a name equal to the host’s IP address and a label provider_config_name: <provider-name>. spec.systemRegistry and spec.licenseToken are copied from the provider. The host spec.releaseVersion is taken from spec.hosts[].releaseVersion, or from the provider spec.releaseVersion if that field is not set.

It is not recommended to create ShturvalHost manually, bypassing ShturvalProvider, if the hosts are intended to be shared with the provider: the controller may overwrite such objects or fix an ownership conflict.

Where to configure #

Services and repositories → Installed services → shturval-capsm → Manage. The service is hosted in the management cluster.

Configuring provider instances in the platform - Shturval v2. General overview - How to work with providers in the platform works.

4. Diagnostics #

Operator performance check #

Diagnostics ShturvalHost #

kubectl get shturvalhost
kubectl describe shturvalhost <hostname>

Flags in the output of kubectl get shturvalhost:

Conditions for ShturvalHost #

status.readyformachine = true is set when the following are executed simultaneously: CanConnectToTheNode=True, CheckSudo=True, ShturvalNodeConfigIsCorrect=True and at least one of: StcCompleted=True or StdDeployed=True.

Conditions for ShturvalMachine #

Conditions for ShturvalCluster #

Common failures #

View node information #

After successfully passing all checks, the operator fills in status.nodeInfo with ShturvalHost:

kubectl get shturvalhost <hostname> -o jsonpath='{.status.nodeInfo}' | python -m json.tool

5. Life cycle #

ShturvalProvider controller #

The controller fully manages the life cycle of ShturvalHost objects based on spec.hosts.

Host creation: when adding an address to spec.hosts, the controller creates ShturvalHost with a name equal to the IP address, assigns it labels from spec.hosts[].labels, label provider_config_name: <provider-name>, SSH credentials and version fields (systemRegistry, releaseVersion, licenseToken). Sets the OwnerReference of the provider to the host.

Host update: when labels change in spec.hosts[].labels the controller synchronizes them in ShturvalHost. Labels from the system list (architecture, memory, cpu_cores, ethernet_interface, operating_system, root_volume_size, hostname, provider_config_name) are not affected by synchronization. systemRegistry and licenseToken are always synchronized. releaseVersion is updated only while the host is not bound to a machine. If the host has spec.hosts[].releaseVersion set, changing the provider version does not change it. Labels are not updated while the host is bound to the machine (MachineAttached=True).

Host Removal: When an address is removed from spec.hosts, the controller triggers the removal of the corresponding ShturvalHost. Waits for finalizers to finish cleaning up.

Deleting a provider: A provider can only be deleted if none of its hosts are bound to the cluster (no MachineAttached=True). When deleting, the operator removes the shturval/<provider-name> finalizer from the Secret with the credentials.

Host Credentials: If spec.hosts[].credentials is set to individual host credentials, they are used instead of the global spec.credentials.

ShturvalHost controller #

The controller prepares the physical node to join the Kubernetes cluster.

Finalizer #

When a new ShturvalHost is first encountered, the controller adds the shturvalhost.infrastructure.cluster.x-k8s.io finalizer and ends the iteration with requeue after 1 second.

Preparing a new host #

1. SSH connection — establishes a connection using spec.address. If there is a network error - CanConnectToTheNode=False, requeue 1 s.
2. Check sudo (one time, until CheckSudo is not True) - if there is an error CheckSudo=False.
3. Deployment of the shturvald daemon (one time, until StdDeployed or StcCompleted is not True) - call EnsureSTDDeployed().
4. Receiving information about the node - metadata is requested via shturvald (architecture, hostname, memory, CPU, machine-id, Kubernetes status). If the daemon is manually removed, StdDeployed is reset.
5. Busy check - if the Kubernetes service is active on the node or kubeconfig is present, ShturvalNodeConfigIsCorrect=False (HostOccupied) is set.
6. Uniqueness of machine-id - the operator maintains an in-memory map of machine-id (initialized at startup for all existing hosts). If there is a duplicate, ResetMachineID() is called and the iteration is restarted.
7. Result - when all steps are completed, ShturvalNodeConfigIsCorrect=True and MachineIDUnique=True are set; status.readyformachine becomes true.

Ready condition IsReadyForMachine #

The host is considered ready to bind a machine (readyformachine=true) when simultaneously executing:

• CanConnectToTheNode=True
• StcCompleted=True OR StdDeployed=True

Linking a car #

When readyformachine=true, the controller looks for ShturvalMachine with status.selectedhost == host.name. From the found candidates, one is selected by priority:

priority = 10 × len(matchExpressions) + 1 × len(matchLabels)

If the priority is the same, the machine with the lexicographically smaller name is selected. Machines in Deleting state are skipped. After selection, status.linkedmachine and MachineAttached=True are set.

Binding consistency #

While the host is bound (machineAttachedAndConfirmed=true), each iteration checks that ShturvalMachine of status.linkedmachine exists and that the machine is not removed. If there is a mismatch, the binding is reset.

Triggers reconcile #

Removal #

With deletionTimestamp != nil, the controller detaches the linked machine, cleans host service state, and removes finalizer shturvalhost.infrastructure.cluster.x-k8s.io.

ShturvalMachine controller #

The controller selects a host and bootstraps the Kubernetes node.

Host selection: if status.selectedhost is empty, the controller looks for a free (not busy) ShturvalHost by the label provider_config_name (taken from the spec.providerConfigName machine or from ShturvalCluster.spec.providerConfigName). Additionally, spec.hostSelector is applied. If the host is found but not ready (readyformachine=false) - wait. If the host is ready, but has not confirmed the choice, wait for confirmation from ShturvalHost.

Checking the installation status of components: The sudo std node status command is executed. If the status is synched, it goes to bootstrap. At syncInProgress orwaitingRebootApprove - wait (requeue in 1 minute). When syncFailed - an error is recorded, reconcile stops.

Bootstrap (version ≥ 2.14): Called node.PreBootstrapWithCloudInit(). If shturvald requires a restart, restart and requeue. Then node.BootstrapWithCloudInit() with data from the bootstrap CAPI secret.

Bootstrap (version < 2.14): removes /etc/std/nci.json, restarts shturvald, waits for synched, then executes BootstrapWithCloudInit.

Setting providerID: After a successful bootstrap, the controller connects to the target cluster, finds a Kubernetes node with a matching machine-id and sets spec.providerID in the format shturval://xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.

Delete: If deletionTimestamp is present, the controller calls node.Reset() on the node to clean it up. If IgnoreResetError=true - reset errors are ignored and deletion continues. After successful cleaning, the finalizer is removed.

Reconcile trigger: ShturvalHost status change (update or delete) queues all related machines.

ShturvalCluster controller #

reconcileNormal: adds the finalizer shturvalcluster.infrastructure.cluster.x-k8s.io, sets status.ready=true and condition ShturvalClusterReconciliationReady=True. If the CAPI cluster has paused the cluster, it sets Ready=false.

reconcileDelete: expects to delete all ShturvalMachine labeled cluster.x-k8s.io/cluster-name: <cluster name>. While the machines exist - requeue every 5 seconds. After removing all the machines, the finalizer is removed.

6. Operator parameters #

Parameters are set through container launch arguments manager. To change, use the customization mechanism via cluster-manager or ShturvalServicePatch.

Environment Variables #

7. CRD/API reference #

All resources belong to the API group infrastructure.cluster.x-k8s.io, version v1beta1.

ShturvalHost #

A cluster-scoped object representing a physical SSH node. Created and managed by the ShturvalProvider controller.

Short name: smh
Scope: Cluster
Finalizer: shturvalhost.infrastructure.cluster.x-k8s.io

spec #

status #

ShturvalProvider #

Cluster-scoped provider global configuration object: list of hosts, image registry, Helm version, license, default SSH credentials. Is the source of truth for everyone ShturvalHost.

Short name: smp
Scope: Cluster
Finalizer: shturvalprovider.infrastructure.cluster.x-k8s.io

spec #

status #

ShturvalCluster #

Namespaced Kubernetes cluster object in terms of the Cluster API.

Short name: smc
Scope: Namespace
Finalizer: shturvalcluster.infrastructure.cluster.x-k8s.io

spec #

status #

| Field | Type | Description | |——|—–|———|| status.ready | bool | Cluster readiness | | status.conditions | array | Cluster Conditions (ShturvalClusterReconciliationReady, ShturvalClusterDeleting) | | status.failureDomains | array ([]FailureDomain) | Fault domains |

ShturvalMachine #

A Namespaced object representing a cluster node in terms of the Cluster API.

Short name: smm
Scope: Namespace
Finalizer: shturvalmachine.infrastructure.cluster.x-k8s.io

spec #

status #

ShturvalMachineTemplate #

Namespaced template for ShturvalMachine. Used by CAPI to automatically create ShturvalMachine objects within MachineDeployment or MachineSet.

Short name: smt
Scope: Namespace

spec #