CEPH CSI module

In the Shturval platform, Ceph Container Storage Interface (CSI) drivers for RBD, CephFS are available for installation in the cluster:

  • Ceph CSI module - CephFS (shturval-ceph-cephfs);
  • Ceph CSI module - CephRBD (shturval-ceph-rbd).

In Kubernetes clusters, modules allow you to use a virtual hard disk (RBD) or a network file system (FS) to store data.

Ceph CSI modules are not critical services for the operation of the Shturval platform. For Ceph CSI Modules to work correctly, the Cluster Network Management Module (shturval-networking) must be running in the cluster.

Please note! Before configuring shturval-ceph-cephfs in the Shturval platform, when configuring Storage on the CephFS side, a subvolumegroup with the name “csi” must be created.

Command for creating subvolumegroup
ceph fs subvolumegroup create ENTER-FILE-SYSTEM-NAME csi

If Ceph CSI Modules were installed when creating the cluster, then manual configuration will not be required in the created cluster.

To enable modules in the created cluster, in the platform GUI, go to the Services and Repositories section and open the Installed Services page and find the Ceph CSI Module.

To install, select the required chart (shturval-ceph-cephfs or shturval-ceph-rbd) and click Manage. In the window that opens, select the desired version. It is recommended to create a new namespace for the chart being installed.

After selecting versions, the available chart values ​​will appear on the right side of the screen. Below are the recommended configuration options.

##CEPHFS

Customvalues example
csiConfig:
  - clusterID: <your-parameter-value>
    monitors:
      - <your-parameter-value>
      - <your-parameter-value>
      - <your-parameter-value>
secret:
  # Specifies whether the secret should be created
  create: true
  name: <your-parameter-value>
  # Key values correspond to a user name and its key, as defined in the
  # ceph cluster. User ID should have required access to the 'pool'
  # specified in the storage class
  adminID: <your-parameter-value>
  adminKey: <your-parameter-value>
cephconf: |
    <your Ceph cluster configuration>

    # Workaround for http://tracker.ceph.com/issues/23446
    fuse_set_user_groups = false

    # ceph-fuse which uses libfuse2 by default has write buffer size of 2KiB
    # adding 'fuse_big_writes = true' option by default to override this limit
    # see https://github.com/ceph/ceph-csi/issues/1928
    fuse_big_writes = true    
storageClass:
  # Specifies whether the Storage class should be created
  create: true
  name: <your-parameter-value>
  # Annotations for the storage class
  # Example:
  # annotations:
  #   storageclass.kubernetes.io/is-default-class: "true"
  annotations: {}

  # String representing a Ceph cluster to provision storage from.
  # Should be unique across all Ceph clusters in use for provisioning,
  # cannot be greater than 36 bytes in length, and should remain immutable for
  # the lifetime of the StorageClass in use.
  clusterID: <your-parameter-value>
  # (required) CephFS filesystem name into which the volume shall be created
  # eg: fsName: myfs
  fsName: <your-parameter-value>
  # (optional) Ceph pool into which volume data shall be stored
  # pool: <cephfs-data-pool>
  # For eg:
  # pool: "replicapool"
  pool: ""
  # (optional) Comma separated string of Ceph-fuse mount options.
  # For eg:
  # fuseMountOptions: debug
  fuseMountOptions: ""
  # (optional) Comma separated string of Cephfs kernel mount options.
  # Check man mount.ceph for mount options. For eg:
  # kernelMountOptions: readdir_max_bytes=1048576,norbytes
  kernelMountOptions: ""
  # (optional) The driver can use either ceph-fuse (fuse) or
  # ceph kernelclient (kernel).
  # If omitted, default volume mounter will be used - this is
  # determined by probing for ceph-fuse and mount.ceph
  # mounter: kernel
  mounter: ""
  # (optional) Prefix to use for naming subvolumes.
  # If omitted, defaults to "csi-vol-".
  # volumeNamePrefix: "foo-bar-"
  volumeNamePrefix: <your-parameter-value>
  # The secrets have to contain user and/or Ceph admin credentials.
  provisionerSecret: <your-parameter-value>
  # If the Namespaces are not specified, the secrets are assumed to
  # be in the Release namespace.
  provisionerSecretNamespace: ""
  controllerExpandSecret: <your-parameter-value>
  controllerExpandSecretNamespace: ""
  nodeStageSecret: <your-parameter-value>
  nodeStageSecretNamespace: ""
  reclaimPolicy: <your-parameter-value>
  allowVolumeExpansion: true
  mountOptions: []
  # Mount Options
  # Example:
  # mountOptions:
  #   - discard
nodeplugin:
  tolerations:
  - key: <your-parameter-value>
    operator: <your-parameter-value>
    value: <your-parameter-value>
    effect: <your-parameter-value>
ParameterDescriptionData typeExample
clusterIDUnique Ceph cluster identifierstringmy-cluster-id
monitorsCeph Monitoring IP Listarray192.168.1.1
secret.nameName of the secret to be createdstringceph-secret
secret.adminIDService account ID for accessing Cephstringclient.admin
secret.adminKeyService account key for accessing Cephstringsdfdff
cephconfCeph configuration in text formatstring[global] auth cluster required = cephx auth service required = cephx auth client required = cephx
storageClass.nameName of the created Storage Classstringcsi-cephfs-sc
storageClass.clusterIDCeph Cluster IDstringcluster-id
storageClass.fsNameName of the CephFS file system into which the PV will be createdstringmyfs
storageClass.volumeNamePrefixPrefix for volume namesstringshturval-
storageClass.provisionerSecretSecret for Provisioning with Ceph User and/or Admin Credentialsstringcephfs-secret
storageClass.provisionerSecretNamespaceNamespace containing the secret for Provisioning with user and/or Ceph administrator credentials. If the namespace is not specified, the secret is assumed to be in the “Release” namespacestringnamenamespace
storageClass.controllerExpandSecretThe Secret to Extending the Controllerstringceph-expand-secret
storageClass.controllerExpandSecretNamespaceNamespace, which contains the secret for expanding the controller. If the namespace is not specified, the secret is assumed to be in the “Release” namespacestringnamenamespace
storageClass.nodeStageSecretSecret for mounting a nodestringcsi-cephfs-secret
storageClass.controllerExpandSecretNamespaceNamespace in which the secret is located. The secret for mounting the node. If the namespace is not specified, the secret is assumed to be in the “Release” namespacestringnamenamespace
storageClass.reclaimPolicyVolume lifecycle management policy (possible values: Retain, Delete, Recycle)stringDelete
nodeplugin.tolerations.keyToleration keystringrole
nodeplugin.tolerations.operatorToleration condition check operatorstringEqual
nodeplugin.tolerations.valueToleration key valuestringinfra
nodeplugin.tolerations.effectToleration effectstringNoSchedule

Service account permission requirements

client.kubecephfs mon ‘allow r’ mgr ‘allow rw’ mds ‘allow rws’ osd ‘allow rw pool=cephfs_data, allow rw pool=cephfs_metadata’

CEPH RBD

Customvalues example
csiConfig:
  - clusterID: <your-parameter-value>
    monitors:
      - <your-parameter-value>
      - <your-parameter-value>
      - <your-parameter-value>
storageClass:
  # Specifies whether the storageclass should be created
  create: true
  name: <your-parameter-value>

  # Use Thick Provisioning for this storage class. Default: false. Enabled if not false.
  thickProvision: false

  # Annotations for the storage class
  # Example:
  # annotations:
  #   storageclass.kubernetes.io/is-default-class: "true"
  annotations: {}

  # (required) String representing a Ceph cluster to provision storage from.
  # Should be unique across all Ceph clusters in use for provisioning,
  # cannot be greater than 36 bytes in length, and should remain immutable for
  # the lifetime of the StorageClass in use.
  clusterID: <your-parameter-value>

  # (optional) If you want to use erasure coded pool with RBD, you need to
  # create two pools. one erasure coded and one replicated.
  # You need to specify the replicated pool here in the `pool` parameter, it is
  # used for the metadata of the images.
  # The erasure coded pool must be set as the `dataPool` parameter below.
  # dataPool: <ec-data-pool>
  dataPool: ""

  # (required) Ceph pool into which the RBD image shall be created
  # eg: pool: replicapool
  pool:  <your-parameter-value>

  # (optional) RBD image features, CSI creates image with image-format 2 CSI
  # RBD currently supports `layering`, `journaling`, `exclusive-lock`,
  # `object-map`, `fast-diff`, `deep-flatten` features.
  # Refer https://docs.ceph.com/en/latest/rbd/rbd-config-ref/#image-features
  # for image feature dependencies.
  # imageFeatures: layering,journaling,exclusive-lock,object-map,fast-diff
  imageFeatures: ""

  # (optional) Specifies whether to try other mounters in case if the current
  # mounter fails to mount the rbd image for any reason. True means fallback
  # to next mounter, default is set to false.
  # Note: tryOtherMounters is currently useful to fallback from krbd to rbd-nbd
  # in case if any of the specified imageFeatures is not supported by krbd
  # driver on node scheduled for application pod launch, but in the future this
  # should work with any mounter type.
  # tryOtherMounters: false

  # (optional) uncomment the following to use rbd-nbd as mounter
  # on supported nodes
  # mounter: rbd-nbd
  mounter: ""

  # (optional) ceph client log location, eg: rbd-nbd
  # By default host-path /var/log/ceph of node is bind-mounted into
  # csi-rbdplugin pod at /var/log/ceph mount path. This is to configure
  # target bindmount path used inside container for ceph clients logging.
  # See docs/rbd-nbd.md for available configuration options.
  # cephLogDir: /var/log/ceph
  cephLogDir: ""

  # (optional) ceph client log strategy
  # By default, log file belonging to a particular volume will be deleted
  # on unmap, but you can choose to just compress instead of deleting it
  # or even preserve the log file in text format as it is.
  # Available options `remove` or `compress` or `preserve`
  # cephLogStrategy: remove
  cephLogStrategy: ""

  # (optional) Prefix to use for naming RBD images.
  # If omitted, defaults to "csi-vol-".
  # volumeNamePrefix: "foo-bar-"
  volumeNamePrefix: ""

  # (optional) Instruct the plugin it has to encrypt the volume
  # By default it is disabled. Valid values are "true" or "false".
  # A string is expected here, i.e. "true", not true.
  # encrypted: "true"
  encrypted: ""

  # (optional) Use external key management system for encryption passphrases by
  # specifying a unique ID matching KMS ConfigMap. The ID is only used for
  # correlation to configmap entry.
  encryptionKMSID: ""
  

  # Add topology constrained pools configuration, if topology based pools
  # are setup, and topology constrained provisioning is required.
  # For further information read TODO<doc>
  # topologyConstrainedPools: |
  #   [{"poolName":"pool0",
  #     "dataPool":"ec-pool0" # optional, erasure-coded pool for data
  #     "domainSegments":[
  #       {"domainLabel":"region","value":"east"},
  #       {"domainLabel":"zone","value":"zone1"}]},
  #    {"poolName":"pool1",
  #     "dataPool":"ec-pool1" # optional, erasure-coded pool for data
  #     "domainSegments":[
  #       {"domainLabel":"region","value":"east"},
  #       {"domainLabel":"zone","value":"zone2"}]},
  #    {"poolName":"pool2",
  #     "dataPool":"ec-pool2" # optional, erasure-coded pool for data
  #     "domainSegments":[
  #       {"domainLabel":"region","value":"west"},
  #       {"domainLabel":"zone","value":"zone1"}]}
  #   ]
  topologyConstrainedPools: []

  # (optional) mapOptions is a comma-separated list of map options.
  # For krbd options refer
  # https://docs.ceph.com/docs/master/man/8/rbd/#kernel-rbd-krbd-options
  # For nbd options refer
  # https://docs.ceph.com/docs/master/man/8/rbd-nbd/#options
  # Format:
  # mapOptions: "<mounter>:op1,op2;<mounter>:op1,op2"
  # An empty mounter field is treated as krbd type for compatibility.
  # eg:
  # mapOptions: "krbd:lock_on_read,queue_depth=1024;nbd:try-netlink"
  mapOptions: ""

  # (optional) unmapOptions is a comma-separated list of unmap options.
  # For krbd options refer
  # https://docs.ceph.com/docs/master/man/8/rbd/#kernel-rbd-krbd-options
  # For nbd options refer
  # https://docs.ceph.com/docs/master/man/8/rbd-nbd/#options
  # Format:
  # unmapOptions: "<mounter>:op1,op2;<mounter>:op1,op2"
  # An empty mounter field is treated as krbd type for compatibility.
  # eg:
  # unmapOptions: "krbd:force;nbd:force"
  unmapOptions: ""

  # The secrets have to contain Ceph credentials with required access
  # to the 'pool'.
  provisionerSecret: <your-parameter-value>
  # If Namespaces are left empty, the secrets are assumed to be in the
  # Release namespace.
  provisionerSecretNamespace: <your-parameter-value>
  controllerExpandSecret: <your-parameter-value>
  controllerExpandSecretNamespace: <your-parameter-value>
  nodeStageSecret: <your-parameter-value>
  nodeStageSecretNamespace: <your-parameter-value>
  # Specify the filesystem type of the volume. If not specified,
  # csi-provisioner will set default as `ext4`.
  fstype: <your-parameter-value>
  reclaimPolicy: <your-parameter-value>
  allowVolumeExpansion: true
  mountOptions: []
  # Mount Options
  # Example:
  # mountOptions:
  #   - discard

# Mount the host /etc/selinux inside pods to support
# selinux-enabled filesystems
selinuxMount: true

secret:
  # Specifies whether the secret should be created
  create: true
  name: <your-parameter-value>
  # Key values correspond to a user name and its key, as defined in the
  # ceph cluster. User ID should have required access to the 'pool'
  # specified in the storage class
  userID: <your-parameter-value>
  userKey: <your-parameter-value>
  # Encryption passphrase
  # encryptionPassphrase: test_passphrase

# This is a sample configmap that helps define a Ceph configuration as required
# by the CSI plugins.
# Sample ceph.conf available at
# https://github.com/ceph/ceph/blob/master/src/sample.ceph.conf Detailed
# documentation is available at
# https://docs.ceph.com/en/latest/rados/configuration/ceph-conf/
cephconf: |
    <your Ceph cluster configuration>
nodeplugin:
  tolerations:
  - key: <your-parameter-value>
    operator: <your-parameter-value>
    value: <your-parameter-value>
    effect: <your-parameter-value>
ParameterDescriptionData typeExample
clusterIDUnique Ceph cluster identifierstringmy-cluster-id
monitorsCeph Monitoring IP Listarray192.168.1.1
secret.nameName of the secret to be createdstringcephrbd-secret
secret.userIDService account ID for accessing Cephstringclient.admin
secret.userKeyService account key for accessing Cephstringsdfdff
cephconfCeph configuration in text formatstring[global] auth cluster required = cephx auth service required = cephx auth client required = cephx
storageClass.nameName of the created Storage Classstringcsi-cephrbd-sc
storageClass.clusterIDCeph Cluster IDstringcluster-id
storageClass.poolCeph pool which should contain cosgiven RBD imagestringkuberbd
storageClass.provisionerSecretSecret for Provisioning with Ceph User and/or Admin Credentialsstringcsi-rbd-secret
storageClass.provisionerSecretNamespaceNamespace containing the secret for Provisioning with user and/or Ceph administrator credentials. If the namespace is not specified, the secret is assumed to be in the “Release” namespacestringnamenamespace
storageClass.controllerExpandSecretThe Secret to Extending the Controllerstringceph-expand-secret
storageClass.controllerExpandSecretNamespaceNamespace, which contains the secret for expanding the controller. If the namespace is not specified, the secret is assumed to be in the “Release” namespacestringnamenamespace
storageClass.nodeStageSecretSecret for mounting a nodestringcsi-cephrbd-secret
storageClass.controllerExpandSecretNamespaceNamespace in which the secret is located. The secret for mounting the node. If the namespace is not specified, the secret is assumed to be in the “Release” namespacestringnamenamespace
storageClass.fstypeThe volume’s file system type. Default ext4stringext4
storageClass.reclaimPolicyVolume lifecycle management policy (possible values: Retain, Delete, Recycle)stringDelete
nodeplugin.tolerations.keyToleration keystringrole
nodeplugin.tolerations.operatorToleration condition check operatorstringEqual
nodeplugin.tolerations.valueToleration key valuestringinfra
nodeplugin.tolerations.effectToleration effect. Valid values: NoSchedule, PreferNoSchedule, NoExecutestringNoSchedule

Service account permission requirements

client.kuberbd mon ‘profile rbd’ osd ‘profile rbd pool=kuberbd’

Customization features

If you need to connect several instances of CEPH FS or CEPH RBD in one cluster:

  1. Install service instances in different namespaces of the cluster.
  2. In the driverName parameter, enter arbitrary values ​​that differ between service instances.
  3. Make the port value in the nodePlugin.httpMetrics.ContainerPort parameter unique for each service instance.