Requirements

Infrastructure, license, mirror, nodes.

Closed loop requirements #

All cluster nodes and shtil machine must have network access to the mirror (usually TCP 443 or the port specified when the mirror was initialized).

Requirements for administrator qualifications #

The administrator must have:

• basic Linux administration skills;
• skills in working with Kubernetes and automation processes for managing container applications in clusters.

Supported providers #

• Shturval v2 - bare metal / hosts via SSH
• oVirt (including zVirt, HostVM, ED Virtualization, ROSA Virtualization)
• VMware vSphere

Public clouds and providers that require nodes to access the Internet are not used in a closed loop.

More details: How work with providers works.

Requirements for the mirror host #

• OS: Linux. The stm utility must be run on a host running Linux OS.
• Resources: minimum 2 vCPU, 4 GiB RAM, 150 GiB disk; 4 vCPUs, 16 GiB RAM, 300 GiB disk are recommended.
• SELinux: permissive or disabled.
• Nginx ports (parameters http-port and https-port, default 80 and 443): must be free.
• Container runtime: on the host do not install Docker manually. Use Docker from the bundle; operation with another container runtime instance is not guaranteed.
• Certificate: when importing - PEM (CA chain and certificate), subjectAltName with the mirror FQDN matching the --fqdn parameter.

Mirror Host Resources {#Mirror Host Resources} #

Resource requirements #

The management cluster must consist of at least one Control Plane node. For fault tolerance - at least three Control Planes and three Workers.

When installed on only one Control Plane, only critical services are installed on a node; lLogging, monitoring and backup are not installed in this configuration.

Note: A cluster on one Control Plane is not recommended for production. Up to 5 client clusters.

The license specifies limits on Worker nodes and vCPUs; When exceeded, a notification is displayed in the GUI.

When the load is over 80% it is recommended to add nodes. Metrics are stored for 7 days; logs are rotated based on completion.

Minimum configuration (1 Control Plane node) #

For test installations only

Recommended configuration (1 CP + 3 Workers) #

Not recommended for production installations

Fault-tolerant configuration (3 CP + 3 Worker) #

Stand-alone cluster (without client clusters) #

For stand-alone in a closed perimeter, use the Mirror line from the minimal configuration.

More information about the mode: Management cluster operating modes.

Machine resources shtil #

The shtil utility must be run from a host with Linux OS. Minimum 3 vCPU, 3 GiB RAM, 20 GiB disk; Docker from the bundle; network access to the mirror via HTTPS (port according to the mirror settings, usually 443). If the mirror has a self-signed or enterprise CA, configure mirror trust at the Docker daemon level before running shtil: the --bootstrap-insecure and --bootstrap-registry-certs-dir flags do not affect the primary docker pull. Checks are on the Preparation page.

Supported operating systems for the shtil startup machine #

Requirements for hosts and OS #

Supported cluster node operating systems #

All cluster nodes must use the same kernel OS version.

Required packages on nodes: openssh-server and dependencies, package manager (yum or apt), sudo, iptables, gnupg.

Infrastructure: physical servers of x86 architecture and/or virtual machines running oVirt version no lower than 4.4 and/or VMware vSphere (hypervisor no lower than 6.7 update 3).

Cluster nodes #

• Nodes must resolve FQDN mirrors and other network names used by the selected scenario.
• SELinux - permissive or disabled mode.
• Inaccessible OS package repositories should be disabled or removed.
• A home directory must be created for the Linux user under whose name the installation is performed (for example, /home/shturval).
• The installation user must be able to run sudo without a password on all target nodes.

Remote access via SSH during installation on pre-created hosts #

On each server:

1. The OpenSSH service accepts connections on port TCP 22.
2. An identical service account for installation was created on all nodes.
3. Authentication using the same public key is configured on all nodes.
4. The service account is allowed to execute sudo without a password.

Local storage #

RawFile Provisioner is used as local storage; ZFS is not supported. Valid file systems: ext4, btrfs, xfs.

Network requirements #

Firewalls should not block necessary connections.

Main ports #

* or the port specified when deploying the mirror.

LDAP and OIDC (Keycloak, Blitz): only inside a closed perimeter.

S3 (Velero): Internal S3-compatible storage only.

Addressing and access rules #

1. IP addresses of nodes do not change during cluster operation.
2. The shtil machine and all cluster nodes must have access to the mirror via HTTPS (port as configured for the mirror, usually 443).
3. Management cluster mustIt is not possible to have access to the API of the infrastructure providers used (oVirt Engine, vCenter, etc. - according to the deployment scenario).

Control Plane subnet and virtual IPs (VIPs) #

The IP addresses of the Control Plane nodes must belong to the same subnet and the same broadcast segment. When using internal balancers for KubeAPI and Ingress, one free VIP is required for each.

If external balancers are used for KubeAPI and/or Ingress, VIP allocation is not required. Requirements and an example of setting up an external balancer are given below.

Additional rules for domestic VIP:

1. VIP should not be configured on host interfaces prior to installation.
2. The VIP for the API server must belong to the same subnet as the IP addresses of the Control Plane nodes.
3. VIP for Ingress is used on the network load balancer and must be accessible from the admin workstation if the internal Ingress scenario is selected.
4. For a management cluster with one Control Plane, IP addresses must be assigned in ascending order: first the node IP address, then the VIP API server, then the VIP Ingress.

External load balancer #

When deploying a cluster with multiple Control Plane nodes, KubeAPI requires a load balancer. The platform can use a built-in (VIP via Kube-VIP) or external balancer.

An external load balancer is also used for Ingress when traffic must go through a common entry point such as HAProxy in front of the cluster.

Requirements:

1. Balancing is performed at level L4 (TCP).
2. The balancer must support storing host selection by source IP address (sticky session / source IP affinity), for example balance source in HAProxy.
3. An external balancer for KubeAPI when installing a cluster with the Shturval v2 provider must be configured before the start of the cluster installation.

HAProxy configuration example (static configuration) {#haproxy-configuration example-static-configuration} #

#---------------------------------------------------------------------
# myclustername cluster
#---------------------------------------------------------------------

frontend myclustername-api-server
    bind 10.11.12.100:6443
    default_backend myclustername-api-server
    mode tcp

backend myclustername-api-server
    balance source
    mode tcp
    server master1 10.11.13.1:6443 check
    server master2 10.11.13.2:6443 check
    server master3 10.11.13.3:6443 check

frontend myclustername-ingress-http
    bind 10.11.12.100:80
    default_backend myclustername-ingress-http
    mode tcp
    option tcplog

backend myclustername-ingress-http
    balance source
    mode tcp
    server worker1 10.11.13.4:80 check
    server worker2 10.11.13.5:80 check

frontend myclustername-ingress-https
    bind 10.11.12.100:443
    default_backend myclustername-ingress-https
    mode tcp
    option tcplog

backend myclustername-ingress-https
    balance source
    mode tcp
    server worker1 10.11.13.4:443 check
    server worker2 10.11.13.5:443 check

Dynamically adding backends (CoreHA) #

The shturval-coreha module allows HAProxy to dynamically obtain a list of nodes from the Ingress Controller via DNS. This is useful when scaling or migrating pods.

Setting up CoreHA:

1. Install and enable Module for searching nodes with running hostport services (shturval-coreha) in the Services and repositories section.
2. Configure chart parameters, for example:

   config:
     zoneName: cluster-prod01-dc1.shturval
   

3. Apply ShturvalServicePatch to shturval-ingress-controller with label ingress.cluster/name and hostPort enabled.
4. In HAProxy use resolvers and server-template:

resolvers coredns-cluster
    nameserver ns1 <api-server-vip>:1053
    accepted_payload_size 8192

backend cluster-prod01-dc1-ingress-http
    balance source
    mode tcp
    server-template daemonset 2 <label_value>.ingress.<zoneName>:80 check resolvers coredns-cluster init-addr none

Where <api-server-vip> is the address of the cluster API server, <zoneName> is the value of config.zoneName in shturval-coreha, <label_value> is the value of the ingress.cluster/name label.

Check: dig @<api-server-vip> -p 1053 <label_value>.ingress.<zoneName>.

Ports and traffic directions #

When using CoreHA, the balancer connects to the module’s DNS server on port 1053 (VIP Control Plane) to dynamically resolve backend names.

By provider #

DNS and network names #

• Each cluster node must have a unique network name.
• The full network name of the host must be a valid DNS name no longer than 63 characters.
• The cluster name must be unique in combination with the organization’s DNS zone and form a valid DNS subdomain.
• A zone must be created in the DNS; For each cluster node, a record of type A must be created in this zone.
• An A-record of the form api.<cluster_name>.<base_domain> is created for the management cluster API server.
• A wildcard record of the form *.<cluster_name>.<base_domain> is created for the Ingress of the management cluster.
• The FQDN of the mirror must resolve to the IP address on the machine shtil andon cluster nodes.

Firewalls should not block network connections. This is required to support interaction between platform components.

In a closed loop, external cloud APIs in matrices are not used; instead of an external Registry - a mirror in your perimeter.

Internetworking schemes #

Shturval v2 with internal balancer for KubeAPI and external balancer for Ingress and CoreHA for client cluster #

Shturval v2 with internal balancer for KubeAPI and external balancer for Ingress #

Shturval v2 with internal balancer for KubeAPI and Ingress #

Shturval v2 with external balancer for KubeAPI and Ingress #

Shturval v2 stand-alone cluster with internal balancer for KubeAPI and Ingress #

Shturval v2 stand-alone withinternal balancer for KubeAPI and external for Ingress #

Shturval v2 DR platform with internal balancer for KubeAPI and external for Ingress #

oVirt, VMware vSphere with internal balancer for KubeAPI and external for Ingress #

oVirt, VMware vSphere with internal balancer for KubeAPI, external for Ingress and CoreHA in client cluster #

oVirt, VMware vSphere with internal balancer for KubeAPI and Ingress #

Additional requirements #

Browser requirements #

Recommended monitor resolution: 1498×906.

Supported #

Browsers from which the GUI is not available #

• IE;
• Opera Mini;
• UC Browser for Android;
• QQ Browser;
• Baidu Browser;
• KaiOS Browser;
• Opera Mobile;
• Android Browser;
• Firefox for Android;
• Samsung Internet Browser;
• Safari on iOS.

Checklist before installation #

• mirror expanded; there is a bundle and Enterprise license
• Nodes and machine shtil have access to the mirror via HTTPS
• If Docker is used on the shtil machine and the mirror has a self-signed or corporate CA, the CA is added to /etc/docker/certs.d/<registry-host>/ca.crt or the mirror is configured to insecure-registries
• Resources and OS meet requirements
• DNS, VIP or balancers prepared according to the script
• Firewalls are configured according to matrices

Checks with the line command - Prepare.