Anatomy of Shturval: how cluster management works under the hood

Why Cluster API alone is not enough for the cluster lifecycle, and how Cluster Manager, Init Job, and the cert-manager ↔ webhooks ↔ CNI chain fit together.

Greetings, colleagues. We continue our technical series. In How Shturval controls the clouds, we looked at how Cluster API (CAPI) and its infrastructure providers provision virtual machines and turn them into nodes. Today we move to the next act — what happens under the hood of the Shturval platform after the cloud has delivered resources.

Let’s see why standard CAPI is not enough on its own and why, without the right initialization sequence, your cluster remains a pile of running binaries with no usable network.

Introduction: NotReady state and the skeleton without a nervous system #

Imagine: CAPI has worked, virtual machines have been created, five standard binaries have been launched on them (kubelet, apiserver, controller-manager, etcd and scheduler). If you do kubectl get nodes, you will see the coveted master node, but in the status NotReady.

The problem is that the cluster has no network. We deliberately do not install CNI through a proxy during basic deployment. CAPI gives us only the skeleton. Shturval acts as the finisher that adds muscle and a nervous system to that skeleton. The main conductor here is Cluster Manager.

Cluster Manager and Init Job: how to bypass metadata limits #

As soon as the base cluster reaches a minimal ready state, the Cluster Manager in the management cluster wakes up. Its job is to start the first deployment wave through the ClusterConfig resource.

The Init Job mechanism handles this:

• What it is: a pod in the new cluster with an image that includes Helm and kubectl.
• Why not Cloud-init? Cloud-init metadata is size-limited. Providers such as Basis or vDirector cap metadata at around 10 KB. Pushing heavy monitoring chart manifests or CRDs (megabytes) causes metadata overflow — the VM cannot accept that config.

Init Job avoids this limit and rolls out the base stack in a loop. At this stage the platform installs:

• Cert-manager (critical for survival);
• CNI (so that the nodes finally come to life);
• SSC (Shturval Service Config) and NCI (Node Config Interface);
• SSP (Shturval Service Patch) - for subsequent additional configuration if the parameters change.

Why cert-manager comes first, or a deadly deadlock #

People often ask: “Why do we need cert-manager in the first seconds?” Without it, you hit a classic architectural deadlock.

Modern Kubernetes relies on webhooks (Validating and Mutating). For example, when you create Ingress or AcmeConfig, the API server must ask the operator: “Is this a valid config?” To prevent anyone from intercepting or tampering with that exchange (for example, unauthorized services mutating pod specs on the fly), Kubernetes requires strict TLS for webhooks.

The dependency chain looks like this:

• No cert-manager — no valid certificates for webhooks.
• No TLS — the API server refuses to call mutating webhooks.
• Without webhooks, operator resources (including CNI) are rejected as invalid.

Result: CNI never starts, nodes stay NotReady, and the cluster is unusable.

CCM and Provider ID: connecting virtual entities with reality #

For CAPI to work, the management cluster must know which CAPI Machine in its manifests corresponds to the real Node in the client cluster. The link is Provider ID.

We follow the standard path through CCM (Cloud Controller Manager).

• Security first: we run CCM in the management cluster. That way we do not store cloud credentials (virtualization passwords) on client nodes. If an attacker compromises one node, they do not gain access to your entire cloud.
• Mapping: CCM reads the ID from the cloud and writes it to the node status. If this mapping breaks or the provider does not return an ID, CAPI loses track of whether the machine is alive.

ClusterInfo and ClusterProfile: control and deep customization #

To avoid drowning in finalizer loops, we provide ClusterInfo — a single pane of glass for installation status, keys, worker CPU counts, and system service health. The Shturval frontend reads this resource for its dashboards.

To standardize the cluster fleet, we use ClusterProfile. This is our defaulting mechanism (Defaulting Webhook). Its most powerful feature is the ability to attach secrets that contain arbitrary manifests to a profile.

Using Match Expressions and labels, you can say: “For all clusters in the RU-CENTRAL zone, apply these patches via SSP and add these secrets.” That lets you customize infrastructure without changing the core operator code.

The art of teardown: how to avoid orphaned cloud resources #

Deleting a cluster is more delicate than creating one. If you simply delete the namespace, you end up with orphaned resources in the cloud that you will keep paying for.

The Shturval deletion algorithm is built to prevent mistakes:

• SSC in Manual mode: stop controllers from trying to “heal” resources you are deleting.
• LoadBalancer cleanup: remove services with external IPs first so the cloud releases those addresses.
• Ignoring PVs: we deliberately do not touch Persistent Volumes. Data safety matters more than a clean cloud account. We would rather have an admin delete a disk manually than let a script destroy three years of production because of a typo in a Retain policy.
• CAPI Cluster deletion trigger: only after that do we tell CAPI to remove the virtual machines.
• Garbage Collector and secrets: the namespace holds cloud secrets and SSH keys needed to delete resources. If the namespace is deleted first, you lose the keys needed to remove the VMs. Our GC waits for the garbage-collect label and CAPI finalizers, and only after the cloud confirms cleanup does it delete the namespace.

Conclusion: replicators and a bright future #

The platform is evolving. In 2.14 we moved from simple resource handoff to multidirectional replication. Two management clusters can exchange data, and automation through capture decides which one is primary for a given set of client clusters.

Our near-term plan is to replace Init Job with an even more elegant bootstrap process. Until then, check ClusterConfig statuses manually — they reveal a lot about the life of your nodes!