Background information

Backup and snapshot #

The backup copy does not replace a snapshot of the entire cluster: only those resources that were included in the backup are involved in the recovery.

Backup behavior (Velero and platform) #

• Selective selection — include and exclude lists of namespaces and resource types (included/excluded by scope), plus labels. Velero treats empty lists as “everything except what is excluded”. * in a list means every resource or namespace in that scope; excludedNamespaces: ["*"] is invalid. How the form maps to these fields — creating backups. An individual object can be marked with velero.io/exclude-from-backup=true (Backup Reference (v1.17), Resource filtering (v1.17)).
• Automatic CSI snapshot exclusion — from Velero v1.17.0 (platform 2.13.1) the controller forcibly excludes volumesnapshots.snapshot.storage.k8s.io and volumesnapshotcontents.snapshot.storage.k8s.io from regular resource collection. Details — shturval-backup.
• Namespace velero is excluded automatically.
• defaultVolumesToFsBackup - if the value is “yes”, pod volumes are copied via file system backup (the general scheme is described in File System Backup (v1.17)). When disabled, volumes use provider-level mechanisms (CSI snapshots, etc.) if available in the cluster. For a full backup of a cluster with volumes, it is recommended to disable file copying for all volumes to reduce the risk of errors during recovery.
• BackupRepository and kopia — when backing up from PV, a structure with backup data and a kopia directory are created in object storage. Repositories in kopia are tied to namespaces with backed up PVs, and not to individual Backup objects. In the namespace velero resources BackupRepository are created (including those with the namespace name prefix).
• Deleting a backup — deleting only the Backup CR in Kubernetes does not remove data from storage; use velero backup delete for complete removal (Backup Reference (v1.17)). Deleting the CR does not immediately remove kopia repositories; cleanup runs via maintenance (quick maintenance frequency is set on BackupRepository, for example maintenanceFrequency, default 1 hour; full maintenance runs automatically). Complete removal of traces can take weeks after deleting the backup. Check maintenance logs in Job resources in namespace velero (only the last few jobs are retained).
• Object storage with immutability - object immutability modes for different providers prevent updating backup metadata in the storage; behavior depends on the policy type and provider (Backup Reference (v1.17)).

Restore behavior #

• Asynchrony - status in Restore.
• Restore source — check status.phase: restore from a named backup after it finishes (Completed or PartiallyFailed); from a scheduler Velero uses the latest backup created by that scheduler with status Completed, see When restore is possible.
• Policy for existing resources — none (do not overwrite) or update (update on conflict); restore filters are includedResources/excludedResources and includedNamespaces/excludedNamespaces, see the restore page.
• PV in use - volumes mounted in running pods are not restored (Deployment, StatefulSet, Pod).
• Order - resources are restored in the order calculated by Velero taking into account dependencies (Restore Reference (v1.17)).
• Resources outside the backup are not affected.

Recommendations #

1. Connect S3-compatible storage before the first backups.
2. For regular copies, use Schedule (cron or CRON_TZ=).
3. Separate backups according to purpose (critical namespaces, “full” cluster, DR).
4. For a full backup of a cluster with PV, disable defaultVolumesToFsBackup for the corresponding backups.
5. Set TTL based on storage capacity and storage policy.
6. Periodically check restore on the test cluster or test namespace.
7. Do not delete backup data directories (restic/kopia) in object storage manually without the agreed deletion of BackupRepository - otherwise errors like Error checking repository for stale locks are possible.
8. For pods with hostPath, if necessary, set the annotation backup.velero.io/backup-volumes-excludes with a list of volume names.
9. management cluster recovery scenarios - see Cluster recovery.

Documentation Velero v1.17 #

Reference for CLI, CRD fields and restrictions: Velero v1.17 Docs.