Hybrid clusters

Overview #

Hybrid cluster is a Kubernetes cluster in which nodes are created by more than one infrastructure provider Cluster API (any provider + Shturval v2 provider).

Typical scenario: the main infrastructure (Control Plane nodes, Worker groups default and infra, application Workers) is hosted on virtualization or cloud (VMware vSphere, oVirt, OpenStack, etc.), and provider hosts Shturval v2 are added to the cluster by an additional Worker group.

This scheme is needed, in particular, for ML workloads: calculations on physical servers with GPUs, while the API server, etcd, Ingress and regular applications remain on the VM.

Constraints and model #

How to make a cluster hybrid #

Suitable for clusters on: VMware vSphere, zVirt, oVirt, HostVM, ED Virtualization, ROSA Virtualization, OpenStack, Selectel, VK Cloud, Basis Dynamix, VMware vCloud Director, Yandex Cloud (any provider except Shturval v2 as the primary one).

General scheme #

1. Create a cluster with a single infrastructure provider, not Shturval v2:
   • client cluster;
   • management cluster.
2. Create an instance of the Shturval v2 provider and add hosts — in Providers → Provider configuration: provider instance and adding hosts. Hosts must be in Free status. GPU (if needed): install drivers on target hosts; plan Nvidia runtime configuration on Shturval v2 nodes.
3. Add a Worker group with the Shturval v2 provider to the existing cluster — see Add the Shturval v2 Worker group below.

Add the Shturval v2 Worker group to a cluster with any other provider #

1. Select the cluster → Administration → Manage Nodes.
2. Click Add Worker Node Group.
3. Enter group name.
4. In the form Creating a group of Worker nodes:
   • In the Provider Type field, the cluster provider is selected by default. Select Shturval v2 (available if the platform has at least one Shturval v2 provider instance) to open the Shturval v2 group creation form.
5. Set the required configuration: manual or automatic scaling, replicas requested = number of free hosts that should be included in the group, update strategy, add Node Roles.
6. In the InfraMachineTemplate block:
   • In the Provider Template field, select the instance of the Shturval v2 provider from which the hosts will be taken. If there is only one instance, it is substituted automatically.
   • Specify the host role or in advanced mode configure the host selector by label (see ShturvalMachineTemplate).
7. Save the group configuration.

You can track the creation of nodes in a group on the Manage Nodes page.

If necessary, you can add hosts and increase the number of nodes or Reduce the number of nodes in the created Worker group of nodes with Shturval v2 in a hybrid cluster.

Scheduling ML and GPU workloads #

After Shturval v2 nodes appear in the cluster:

• Set taints and tolerations and nodeAffinity so ML pods schedule only on GPU nodes — see Taints and Tolerations.
• Install and configure NVIDIA GPU Operator and runtime on the nodes of the Worker group (see node configuration).

Related topics #

• Node groups - general group model, Cluster API, InfraMachineTemplate.
• Shturval v2 - provider instance and hosts.
• Provider Shturval v2 (shturval-capsm) - CAPSM operator.
• How providers work — Cluster API terms and provider instances.