Cluster update module (shturval-update)

About the service

ParameterDescription
GUI nameCluster update module
System name (identifier)shturval-update
Cluster AssignmentCluster update management (Kubernetes, platform components); coordination of the process of updating units and components. Installed in the management cluster.
CriticalYes
Public repository— (component of the Shturval platform)
Dependenciesshturval-update-crds, shturval-networking
Namespaceshturval-update-system

Contents

  1. Review
  2. Architecture
  3. Installation/deployment
  4. Diagnostics
  5. Lifecycle
  6. Operator parameters
  7. CRD/API reference

1. Review

Purpose: the shturval-update operator updates the Shturval platform to the version specified in ShturvalUpdate.spec.version, checks the availability of versions through the update channel and can launch the update on a cron schedule.

Features

ContourIntended use
ShturvalUpdatestores the target version, the current platform version and the execution status of the update phases
ShturvalUpdateChannelretrieves update matrix from registry and publishes available versions to status.availableupdates
ShturvalUpdateSchedulecalculates the update window according to a cron schedule and initiates the update via the default ShturvalUpdate
Webhook ShturvalUpdatechecks resource operations and prohibits dangerous changes during active update

Initiation scripts

EventInitiatorObserved effect
Manual platform updateadministrator or automation changes ShturvalUpdate.spec.versionthe operator checks the version, builds a set of phases and sequentially performs the update
Checking the update channeloperator ShturvalUpdateChannel periodically or after the default ShturvalUpdate is readyin status.availableupdates a sorted list of available versions appears
Scheduled updateShturvalUpdateSchedule in the active cron windowdefault ShturvalUpdate gets new spec.version and spec.followSchedule
Admission validationKubernetes API Server on create/update/delete ShturvalUpdatechanging the version and deleting a resource are blocked if the update is not yet ready

Limitations and implications

LimitConsequencefor administrator
The default name for ShturvalUpdate and ShturvalUpdateChannel is shturval-updatechannel with a different name fails; scheduling and channel checks are focused on default resources
ShturvalUpdateChannel patches the entire object, not just the status subresourcewhen making external changes to a channel resource, keep in mind that the controller regularly updates the object itself
Registry URL is collected as https://<SYSTEM_REGISTRY>/repository/shturval_bin_publicfor a private registry you need to correctly set SYSTEM_REGISTRY and, if necessary, insecure mode
Webhook is enabled by default if ENABLE_WEBHOOKS is not equal to falsewith a standard installation of Kubernetes API Server will access the operator’s admission webhook
Extensive RBAC rights have been declared for ShturvalUpdate and ShturvalUpdateChannelchanging the access model must undergo a separate security and update compatibility check
The algorithm for selecting a minor version in schedule requires confirmation#QUESTION: confirm expected version selection behavior with spec.allowMinorVersion=true

Place in the chain of command

Update control chain

Update management chain

In the operational chain, the administrator or automation sets the target version via ShturvalUpdate. ShturvalUpdateChannel prepopulates the list of available versions from the registry. If a schedule is used, ShturvalUpdateSchedule selects the appropriate version from the channel resource status and patches the default update resource. ShturvalUpdate then performs the update phases and reflects the progress of the process in status.ready and status.conditions.

Key Features

CharacteristicsDescription
API group/versionupdate.shturval.tech/v1beta1
Main namespaceshturval-update-system; Helm namespace is set by the installation command #QUESTION: do I need to add namespace to Helm values
ControllersShturvalUpdate, ShturvalUpdateChannel, ShturvalUpdateSchedule
Deploymentshturval-update-controller-manager
Container imageset when installing chart shturval-update
Leader electionenabled by the --leader-elect flag; in the manager-manifest --leader-elect is passed
Metricsbind address by defaultyu 0; Kustomize includes metrics service and authn/authz patch
health/readinessdefault probe endpoint address :8081
Webhooksvalidating/defaulting webhook for ShturvalUpdate; turns off ENABLE_WEBHOOKS=false
Pod SecurityrunAsNonRoot: true, allowPrivilegeEscalation: false, drop ALL
HTTP/2disabled by default for metrics and webhook servers

Controllers

ControllerPurpose
ShturvalUpdateReconcilerupdates the cluster to spec.version, runs finalizer and conditions
ShturvalUpdateChannelReconcilerreads the current version, gets the update matrix from the registry and publishes the available versions
ShturvalUpdateScheduleReconcilercalculates the cron window and starts the update using the default ShturvalUpdate

2. Architecture

Workload Components

ComponentKubernetes typeScopeDescription
ShturvalUpdateCRD/custom resourceNamespacedtarget version and platform update status
ShturvalUpdateChannelCRD/custom resourceNamespacedupdate channel and list of available versions
ShturvalUpdateScheduleCRD/custom resourceNamespacedcron schedule and next update interval
shturval-update-controller-managerDeploymentNamespacedruns /shturval-update with leader election
webhook-serviceServiceNamespacedendpoint for admission webhook
ServiceMonitorPrometheus resourceNamespacedmonitoring metrics endpoint; installation depends on the presence of Prometheus CRD in the cluster
Certificatecert-manager resourceNamespacedwebhook server certificate; installation depends on the presence of cert-manager CRD in the cluster

Deployment architecture

Illustration: Deployment architecture

Deployment architecture

Interaction with external systems

SystemHow to use
Kubernetes APIstoring CRD, running reconcile, applying changes to platform resources
Update registrymatrix file source at <registry>/update/<channel>/<currentVersion>
Shturval service configurationmanaging system services during update phases
Child clustersaccess viakubeconfig Secret <clusterId>-kubeconfig in namespace <clusterId>, key value
Illustration: Update Resource Links

Update Resource Links

Access to child clusters

If the update phase accesses a child cluster, the operator expects a Secret with kubeconfig:

apiVersion: v1
kind: Secret
metadata:
  name: <clusterId>-kubeconfig
  namespace: <clusterId>
type: Opaque
stringData:
  value: |
    apiVersion: v1
    kind: Config
    clusters:
    - name: <cluster>
      cluster:
        server: https://<api-server>
        certificate-authority-data: <base64-ca>
    contexts:
    - name: <cluster>
      context:
        cluster: <cluster>
        user: <user>
    current-context: <cluster>
    users:
    - name: <user>
      user:
        token: <token>    

SSH keys and passphrase-protected SSH keys are not read by this circuit: kubeconfig from the value key is used for access.


3. Installation / deployment

The delivery includes two Helm charts: shturval-update-crds for the CRD and shturval-update for the controller. The installation must first deliver the CRD, then the main chart with Deployment, RBAC, webhook and related objects.

Note: any changes to installation parameters must be made in concert with the platform via ShturvalServicePatch (SSP) or other approved service management mechanism, without manual changes that bypass the platform lifecycle.

Minimum installation order

StepWhat is installedResult
1chart shturval-update-crdsCRDs ShturvalUpdate, ShturvalUpdateChannel, ShturvalUpdateSchedule appear in Kubernetes API
2chart shturval-updatecontroller manager is launched, RBAC, Service and webhook integration are created
3default CRresources are created through which the operator determines the current version, channel and schedule

4. Diagnostics

Diagnostics are based on the CR and conditions status fields. The original technical manual does not have a ready set of operational kubectl get/logs commands, so the test commands are not duplicated here.

General symptoms

SymptomPossible reasonAction
ShturvalUpdate.status.ready=false does not change for a long timethe target version is not available, the phases are not built, or one of the phases returned an errorcheck status.conditions resource ShturvalUpdate
ShturvalUpdateChannel.status.ready=falsethe current version has not been read or is unavailable update matrixcheck status.currentversion, status.lastchecked, registry parameters
There is no expected version in status.availableupdateschannel, current version or matrix semver restrictions do not give the desired transitioncheck channel and current platform version
Scheduled updatee startscron window has not yet arrived, update is already in progress or channel does not contain available versionscheck ShturvalUpdateSchedule.status.nextUpdateStart/End and status.conditions
Removal ShturvalUpdate rejectedwebhook prevents deletion at status.ready=falsewait for the update to complete or confirm the emergency procedure separately

Conditions ShturvalUpdate

ConditionTrueFalse/reasons
VersionIsCorrectversion available and phases builtSpecVersionErr, SpecVersionResourceNotFound, SpecVersionNotAvailable, BuildUpdatePhasesErr
ControllerUpdatedinit/controller phase completedreason depends on the specific update
PostInitUpdatedpost-init phase completedreason depends on the specific update
CRDUpdatedCRD phase completedCRDGetFailed, CRDPatchFailed
PreKubeUpdatedpre-kube phase completedreason depends on the specific update
NCIUpdatedkube/NCI phase completedreason depends on kube phase
StdUpdatedSTD phase completedreason depends on the STD phase
SystemServicesUninstallSSC uninstall phase completedSSCUpdateFail, SSCUninstallFail
PreSystemServiceUpdatedpre-SSC phase completedreason depends on the release command
SystemServicesUpdatedsystem services phase completedSSCGetFail, SSCAddFail, SSCUpdateFail, SSCUpdateWait, SSCServices*
ValidateKubeApiVersionAPI Kubernetes check completedreason depends on the command phase
UpdatePausedupdate paused due to schedule windowcondition is removed by the controller after resumption
PostSSCUpdatepost-SSC phase completedreason depends on the release command
DefineClusterVersioncurrent cluster version definedUndefinedBaseSscVersions

Conditions ShturvalUpdateChannel

ShturvalUpdateChannel has no separate conditions defined. The fields status.ready, status.currentversion, status.lastchecked and status.availableupdates are used for diagnostics.

Conditions ShturvalUpdateSchedule

ConditionTrueFalse/reasons
TriggerUpdateShturvalUpdate successfully patched by schedulePatchShturvalUpdateError if there is an error when launching update according to schedule

5. Life cycle

End-to-end update lifecycle

Illustration: End-to-end update lifecycle

End-to-end update lifecycle

Update via ShturvalUpdate

StepResource/ComponentAction
1ShturvalUpdateoperator receives the resource and completes reconcile without error if the resource has already been deleted
2ShturvalUpdate.statusif the current version is not yet defined, the operator sets in status.currentversion
3ShturvalUpdate.status.clusterinfoif the cluster type is not yet defined, the operator fills in information about the cluster
4ShturvalUpdatefinalizer update.shturval.tech/finalizer is added
5ShturvalUpdate.spec.versionif the target version is empty, the operator sets status.ready=true and does not trigger the update
6ShturvalUpdate.status.conditionswhen a new update starts, conditions are reset, ready is transferred to false
7Update phasesoperator checks version availability, builds phases and executes update commands
8ShturvalUpdate.status.currentversionafter successful execution and in the absence of a pause, the operator again determines the cluster version

Finalizer: update.shturval.tech/finalizer. When deleted, the operator removes the finalizer and completes the reconcile.

Reconcile ShturvalUpdate

Illustration: Reconcile ShturvalUpdate

Reconcile ShturvalUpdate

Checking the update channel

StepResource/componentAction
1ShturvalUpdateChannelthe operator accepts for processing only the default resource with the name shturval-update
2ShturvalUpdateChannel.statusready is translated into false, updated lastchecked
3Default ShturvalUpdateoperator reads current version
4Update registryloading matrix at URL <registry>/update/<channel>/<currentVersion>
5ShturvalUpdateChannel.status.availableupdatesmatrix is ​​filtered by semver rules, the result is sorted and written to status
6ShturvalUpdateChannel.status.readyafter a successful check, true is set

Finalizer: update.shturval.tech/finalizer. When deleting a channel resource, the operator removes the finalizer.

Reconcile ShturvalUpdateChannel

Illustration: Reconcile ShturvalUpdateChannel

Reconcile ShturvalUpdateChannel

Scheduled update

StepResource/componentAction
1ShturvalUpdateSchedulethe operator receives the schedule and calculates the window using spec.schedule
2status.nextUpdateStart/Endthe beginning and end of the nearest or current window are recorded
3Default ShturvalUpdateif update is already in progress, schedule waits for the next attempt
4UpdatePausedif the update was paused by this schedule, the operator clears pause-condition
5ShturvalUpdateChannel.status.availableupdatesselecting the version to launch
6Default ShturvalUpdatespec.version and spec.followSchedule are patched, condition TriggerUpdate=True

When deleting ShturvalUpdateSchedule, the operator prohibits deletion if the default ShturvalUpdate.spec.followSchedule points to the schedule to be deleted; otherwise it removes the finalizer.

Reconcile ShturvalUpdateSchedule

Illustration: Reconcile ShturvalUpdateSchedule

Reconcile ShturvalUpdateSchedule

Resource Negotiation Protocol

StepMemberAction
1ShturvalUpdateChannelreads ShturvalUpdate.status.currentversion, gets matrix and writes status.availableupdates
2ShturvalUpdateSchedulereads ShturvalUpdateChannel.status.availableupdates into the cron window and patches the default ShturvalUpdate
3ShturvalUpdatechecks spec.version against ShturvalUpdateChannel.status.availableupdates, builds and executes phases
4ShturvalUpdateChannelafter the default one is ready, ShturvalUpdate starts a re-check channel

6. Operator parameters

Process flags

FlagDefaultPurpose
--debugfalseenables debug mode
--metrics-bind-address0metrics endpoint address; :8443 for HTTPS, :8080 for HTTP, 0 disables metrics service
--health-probe-bind-address:8081address health/readiness probe endpoint
--leader-electfalseincludes leader election; in the manager-manifest --leader-elect is passed
--metrics-securetrueenables HTTPS for metrics endpoint; false enables HTTP
--enable-http2falseenables HTTP/2 for metrics and webhook servers

Environment Variables

| Variable | Default/Source | Purpose | |————|———————————|————|| WATCH_NAMESPACE | shturval-update-system if env is missing | namespace of reconciler work | | SYNC_PERIOD | 60m | retry interval for ShturvalUpdateChannel | | SCHEDULE_SYNC_PERIOD | at runtime 60m, Helm default 10m | retry interval for ShturvalUpdateSchedule | | SYSTEM_REGISTRY | r.shturval.tech | registry host; operator adds schema and path /repository/shturval_bin_public | | SYSTEM_REGISTRY_INSECURE | false; auto true for non-default registry without env | insecure HTTP client mode for registry | | ENABLE_WEBHOOKS | enabled if not false | webhook registration | | GODEBUG | tlsmlkem=0 | runtime/build Go option |


7. CRD/API reference

API group/version for all resources: update.shturval.tech/v1beta1.

ShturvalUpdate (su, shup, shupdate) — Namespaced

spec fieldTypeRequiredDescription
spec.versionstringNtarget platform version; semver is allowed with optional v, prerelease and build metadata
spec.followSchedule*stringNname of ShturvalUpdateSchedule that initiated the update; empty for manual start
status fieldTypeDescription
status.readyboolreadiness of the update process
status.currentversionstringcurrent platform version
status.clusterinfo.typeclusterstringcluster type: base or mgmt
status.clusterinfo.insecureboolrepository connection type from ShturvalRepoConfig.spec.insecure
status.conditions[]metav1.Conditionupdate progress and phase errors

Conditions ShturvalUpdate are listed in the Diagnostics section.

ShturvalUpdateChannel (suc, shupch, shupdatechannel) — Namespaced

spec fieldTypeRequiredDescription
spec.channelstringNupdate channel; an empty value at runtime is interpreted as stable
status fieldTypeDescription
status.readyboolchannel check readiness
status.currentversionstringcurrent version from default ShturvalUpdate
status.lastcheckedmetav1.Timelast check time
status.availableupdates[]stringsorted list of available versions from update matrix

Conditions for ShturvalUpdateChannel are not defined.

ShturvalUpdateSchedule — Namespaced

There is no ShortName marker for ShturvalUpdateSchedule. #QUESTION: is a short alias necessary?for kubectl.

spec fieldTypeRequiredDescription
spec.schedulestringYcron-expression
spec.updatePeriodmetav1.DurationYupdate window duration; default 24h0m0s
spec.pauseIfUpdatePeriodExceedboolYif true, update is paused when leaving the window
spec.allowMinorVersionboolYif true, schedule can select the latest available version from channel status
status fieldTypeDescription
status.nextUpdateStart*metav1.Timestart of the nearest or current window
status.nextUpdateEnd*metav1.Timeend of window
status.nextUpdateVersion*stringversion selected by schedule
status.conditions[]metav1.Conditionplanning progress

Conditions ShturvalUpdateSchedule are listed in the Diagnostics section.


Where to configure

Services and repositories → Installed services → shturval-update → Manage. More details: Update Kubernetes.