Anatomy of “Shturval”: how the magic of managing clusters from the inside works

Why Cluster API alone is not enough for the life cycle of a cluster and what role do Cluster Manager, Init Job and the cert-manager ↔ webhooks ↔ CNI combination play.

Greetings, colleagues. We continue our technical series. In How Shturval controls the clouds, we looked at how Cluster API (CAPI) and its infrastructure providers provision virtual machines and turn them into nodes. Today we move to the next chapter — what happens under the hood of the Shturval platform after the cloud has delivered resources.

Let’s see why standard CAPI is not enough on its own and why, without the right initialization sequence, your cluster stays a pile of running binaries with no usable network.

Introduction: NotReady state and the skeleton without a nervous system #

Imagine: CAPI has worked, virtual machines have been created, five standard binaries have been launched on them (kubelet, apiserver, controller-manager, etcd and scheduler). If you do kubectl get nodes, you will see the coveted master node, but in the status NotReady.

The problem is that there is no network in the cluster. We deliberately do not proxy CNI during the basic deployment phase. CAPI gives us only the “skeleton”. The role of the “closer”, who pulls the muscles and nervous system onto this skeleton, is taken on by Shturval. And the main conductor here is the Cluster Manager.

Cluster Manager and Init Job: how to bypass metadata limits #

As soon as the base cluster reaches a minimal ready state, the Cluster Manager in the management cluster wakes up. Its job is to start the first deployment wave through the ClusterConfig resource.

The Init Job mechanism handles this:

• What it is: a pod in the new cluster with an image that includes Helm and kubectl.
• Why not Cloud-init? Cloud-init metadata is size-limited. Providers such as Basis or vDirector cap metadata at around 10 KB. Pushing heavy monitoring chart manifests or CRDs (megabytes) causes metadata overflow — the VM cannot accept that config.

Init Job avoids this limit and rolls out the base stack in a loop. At this stage the platform installs:

• Cert-manager (critical for survival);
• CNI (so that the nodes finally come to life);
• SSC (Shturval Service Config) and NCI (Node Config Interface);
• SSP (Shturval Service Patch) - for subsequent additional configuration if the parameters change.

Why Cert-manager is the “head”, or a deadly deadlock #

People often ask: “Why do we need cert-manager first?” Without it, you hit a classic deadlock.

Modern Kubernetes is tied to webhooks (Validating and Mutating). For example, when you create Ingress or AcmeConfig, the API server should ask the operator: “Is this a valid config?” To prevent anyone from replacing the answer and getting into the middle of the process (protection from “left” services that edit pod specs on the fly), Kubernetes requires a strict TLS connection for webhooks.

The dependency chain looks like this:

• No Cert-manager - no valid certificates for webhooks.
• No TLS - The API server refuses to communicate with mutating webhooks.
• Without webhooks, operator resources (including CNI) will simply be rejected as incorrect.

Result: CNI never starts, nodes stay NotReady, and the cluster is dead in the water.

CCM and Provider ID: connecting virtual entities with reality #

For the CAPI magic to work, the managing cluster must clearly understand which CAPI Machine in its manifests corresponds to the real Node in the client cluster. Communication goes through Provider ID.

Here we go the “orthodox” way through CCM (Cloud Controller Manager).

• Security First: We run CCM in the Management Cluster. Why? To avoid storing cloud credentials (virtualization passwords) on client nodes. If an attacker takes over one node, he will not have access to manage your entire cloud.
• Mapping: CCM pulls the ID from the cloud and assigns it to the node status. If this mapping breaks or the provider does not issue an ID, CAPI becomes depressed and does not understand whether the machine is alive.

ClusterInfo and ClusterProfile: control and deep customization #

To avoid drowning in finalization hell, we provide ClusterInfo — a single pane of glass for installation status, keys, worker CPU counts, and system service health. The Shturval frontend reads this resource for its dashboards.

To standardize the cluster fleet, ClusterProfile is used. This is our default mechanism (Defaulting Webhook). The most powerful feature here is the ability to attach secrets containing arbitrary manifests to a profile.

Using Match Expressions and Labels, you can say: “For all clusters in the RU-CENTRAL zone, apply these patches via SSP and add these secrets.” This allows you to customize the infrastructure without touching the main operator code.

The art of destruction: how not to leave “ghosts” in the cloud #

Deleting a cluster is a more subtle process than creating it. If you simply trash Namespace, you will end up with a pile of unowned resources in the cloud, for which you will continue to pay.

The deletion algorithm in Shturval is built on foolproof protection:

• SSC in Manual mode: stop controllers from trying to “heal” resources you are deleting.
• Clearing LoadBalancer: First of all, we demolish services with external IPs so that the cloud frees up the addresses.
• Ignoring PV: we deliberately do not touch Persistent Volumes. Data security is more important than cleanliness. We’d rather force the admin to delete the disk manually than for our script to destroy three years of production due to a typo in the Retain policy.
• CAPI Cluster deletion trigger: only after this a command is given to demolish the virtual machines.
• Garbage Collector and secrets: Namespace contains cloud secrets and SSH keys needed to delete resources. If Namespace is deleted first, that’s it, you’ve lost the keys to the apartment where the money (and virtual machines) are located. Our GC waits for the garbage-collect label and CAPI finalizers, and only when the cloud confirms “clean” does it demolish the Namespace to zero.

Conclusion: replicators and a bright future #

The platform is evolving. In 2.14 we moved from simple resource handoff to multidirectional replication. Two management clusters can exchange data, and automation through leader election decides which one is primary for a given set of client clusters.

Our immediate plans are to finally bury Init Job for an even more elegant launch process. In the meantime, don’t hesitate to look into the ClusterConfig statuses with your hands, there’s a lot of interesting information about the life of your nodes!