Migrating Ceph CSI to shturval-ceph-csi

This page describes how to move the services shturval-ceph-cephfs and shturval-ceph-rbd, already running in the cluster, to shturval-ceph-csi. Data in Ceph is not lost. The drivers keep their names (cephfs.csi.ceph.com and rbd.csi.ceph.com), so existing volumes remain attached to the same disks.

Save the command output before step 2 and compare it after step 7.

The migration tool only reads the cluster and writes YAML to disk. It does not change cluster objects. A person changes the cluster: disabling the services, kubectl apply, and the Helm install are separate steps below. Do not reorder them.

Before you start #

Save the command output to a text file in the tool directory, for example pre-migration.txt. Do this before you turn the services off. The file is only for you: it is not applied to the cluster and does not belong in manifests/. After the services are set to Off, the chart StorageClass is removed. Keep the Secret in the old driver namespaces (ceph-fs, ceph-rbd). Use the file to check the Secret name and namespace, to confirm the old StorageClass is gone, and to see which pods must not be restarted until the operator CSI pods are Running.

• Secret name (spec.customvalues.secret.name) and namespace (spec.namespace) of each SSC.

  kubectl get shturvalserviceconfig shturval-ceph-cephfs shturval-ceph-rbd -o custom-columns=NAME:.metadata.name,NAMESPACE:.spec.namespace,SECRET:.spec.customvalues.secret.name
  

• StorageClass name (spec.customvalues.storageClass.name), if it is set.

  kubectl get shturvalserviceconfig shturval-ceph-cephfs shturval-ceph-rbd -o custom-columns=NAME:.metadata.name,STORAGECLASS:.spec.customvalues.storageClass.name
  

• PVCs that are already mounted. Do not restart their pods until the new operator CSI pods are Running.

  kubectl get pvc -A --field-selector=status.phase=Bound
  

If only one of the two modules is installed, the tool exits with code 1 and reports an error for the missing SSC. Manifests for the ready module are still saved. For a full run, both services must be ready before you start.

Step 1. Build the tool and generate manifests #

On this step the tool only writes YAML into the manifests/ directory. It applies nothing to the cluster: no Secret, no ConfigMap, no StorageClass. The image file does not exist until the tool has run, so there is nothing to edit beforehand.

go build -o ceph-operator-migration .
./ceph-operator-migration --kubeconfig /path/to/kubeconfig.conf

The kubeconfig path is not hardcoded; the flag is required. The same run without a binary: go run . --kubeconfig /path/to/kubeconfig.conf.

saved ... lines go to stderr. Exit code 0 means both SSCs were read. If stderr contains error:, do not disable the services: fix the cause and run the tool again.

After a successful run the working directory must contain:

manifests/
├── step0/
│   └── namespace.yaml
├── backup/
│   ├── ceph-fs-csi-secret.yaml
│   └── ceph-rbd-csi-secret.yaml
├── step1/
│   ├── operator-images-configmap.yaml
│   ├── cephfs-driver.yaml
│   ├── rbd-driver.yaml
│   ├── ceph-fs-connection.yaml
│   ├── ceph-rbd-connection.yaml
│   ├── ceph-fs-client-profile.yaml
│   ├── ceph-rbd-client-profile.yaml
│   ├── ceph-fs-operator-secret.yaml
│   └── ceph-rbd-operator-secret.yaml
└── step2/
    ├── ceph-fs-storageclass.yaml
    └── ceph-rbd-storageclass.yaml

Images #

The CSI image file appears only after the tool has run: manifests/step1/operator-images-configmap.yaml. These images are not in the chart. The tool writes the default addresses there. If the pods must come from your registry, replace them in this file and apply it at step 7. You do not rebuild the tool.

The Driver manifests cephfs-driver.yaml and rbd-driver.yaml already point at this file:

spec:
  imageSet:
    name: operator-images

Check this before you disable the SSCs:

• In backup/, each Secret has the same metadata.name as the SSC, and metadata.namespace is the original CSI namespace (ceph-fs or ceph-rbd).
• In step1/, the same Secrets are placed in the operator namespace.
• In step1/*-connection.yaml, spec.monitors matches the monitors of the corresponding SSC.
• In step2/, the classes are named shturval-cephfs and shturval-rbd. The tool sets these names itself, even if the chart customvalues used another class name.
• parameters.clusterID, fsName (CephFS), and pool (RBD) match that SSC’s customvalues.

The tool writes Secret files with mode 0600. The manifests/ directory is gitignored: it is a snapshot of cluster credentials.

Step 2. Turn off both SSCs #

Two drivers with the same name cannot run at once. Remove the chart-based CSI first, then start the operator.

For Ceph CSI Module - Ceph File System and Ceph CSI Module - Ceph Block Device:

1. Services and repositories → Installed services.
2. Open the service card → Manage.
3. Set the management mode to Off.
4. Click Save.

The same action with kubectl. The services are not critical, so you do not clear iscritical:

kubectl patch shturvalserviceconfig shturval-ceph-cephfs --type merge -p '{"spec":{"mode":"absent"}}'
kubectl patch shturvalserviceconfig shturval-ceph-rbd --type merge -p '{"spec":{"mode":"absent"}}'

Wait until the chart pods are gone:

kubectl get pods -n ceph-fs
kubectl get pods -n ceph-rbd

Turning a service off removes the controller and node plugin pods and the StorageClass created by the chart. Do not delete the Secret in the old driver namespaces (ceph-fs, ceph-rbd): existing volumes refer to it. PersistentVolume and PersistentVolumeClaim objects are not deleted. Data in Ceph stays. A volume that is already mounted on a node keeps working: the kernel mount or ceph-fuse mount is already in place.

The SSC object and its customvalues stay in the cluster. Do not switch these two services back to Auto after the migration: the chart would create drivers with the same names and collide with the operator.

Step 3. Confirm the old StorageClasses are gone #

kubectl get storageclass shturval-cephfs shturval-rbd

If customvalues used another class name, check that name too. Empty output is the expected result.

If the StorageClass is still there, delete it:

kubectl delete storageclass shturval-cephfs shturval-rbd

Do not delete the Secret in the old driver namespaces. It must stay in ceph-fs and ceph-rbd (or in the SSC spec.namespace). Do not delete PersistentVolumes or PersistentVolumeClaims either.

Step 4. Restore Secrets to the original namespaces #

Secrets for existing volumes must stay in the old driver namespaces. If they are gone after the services are turned off, put the copies from manifests/backup/ back:

kubectl apply -f manifests/backup/

Secrets return to the namespaces the tool read them from (ceph-fs, ceph-rbd, or the actual spec.namespace). Existing PVCs reference the Secret in that namespace. You cannot change the Secret namespace on an existing PVC, so this step is required: without it the volume will not mount again after a pod restart.

Check:

kubectl get secret -n ceph-fs
kubectl get secret -n ceph-rbd

The names must match metadata.name in the backup/ files.

Step 5. Create the operator namespace #

kubectl apply -f manifests/step0/

This creates the ceph-csi-operator namespace (or the --operator-namespace value). You do not need a separate kubectl create namespace: step0/namespace.yaml does the same thing.

Step 6. Install the operator chart #

Chart ceph-csi-operator/ceph-csi-operator from https://ceph.github.io/ceph-csi-operator/. In values.yaml, change only the operator image:

controllerManager:
  manager:
    image:
      repository: quay.io/cephcsi/ceph-csi-operator
      tag: v1.0.5

helm install ceph-csi-operator ceph-csi-operator/ceph-csi-operator \
  -n ceph-csi-operator --create-namespace=false \
  --values values.yaml

The operator namespace then has one pod: ceph-csi-operator-controller-manager. The CSI pods are not running yet. The operator creates them after you apply the manifests from step1.

Step 7. Apply step1 and wait for the pods #

kubectl apply -f manifests/step1/
kubectl get pods -n ceph-csi-operator -w

The operator receives Driver, CephConnection, ClientProfile, the operator-images ConfigMap, and Secrets in its namespace, and starts the controller plugin and node plugin for cephfs.csi.ceph.com and rbd.csi.ceph.com.

Until this step, ConfigMap operator-images existed only as the file manifests/step1/operator-images-configmap.yaml. It now appears in the operator namespace (ceph-csi-operator, or the --operator-namespace value). The operator reads it when it creates the driver pods: in the Driver manifest, spec.imageSet.name is operator-images. To see it: kubectl get configmap operator-images -n ceph-csi-operator -o yaml.

Go to step 8 only when every pod in the namespace is Running, including pods on each node where volumes must be mounted. ContainerCreating and CrashLoopBackOff mean the new plugin is not ready to remount. Check pod events: the usual causes are an image from the operator-images ConfigMap that cannot be pulled, or a bad ClientProfile (monitors, clusterID).

At this point you can restart a pod that uses an existing PVC and confirm the volume mounts again. That mount uses the Secret from backup, not the Secret in the operator namespace.

Step 8. Apply the StorageClasses #

kubectl apply -f manifests/step2/
kubectl get storageclass shturval-cephfs shturval-rbd

The classes shturval-cephfs and shturval-rbd reference the Secret in the operator namespace. They are for new PVCs. Existing PVCs keep using the Secret from step 4 and the same driver name on the PV. You do not change their storageClassName.

To check a new volume, create a PVC with storageClassName: shturval-cephfs or shturval-rbd, and wait until it is Bound and the pod starts. If the PVC stays Pending, read kubectl describe pvc and the provisioner logs in the ceph-csi-operator namespace.

Where manifest fields come from #