CEPH CSI module
In the Shturval platform, Ceph Container Storage Interface (CSI) drivers for RBD, CephFS are available for installation in the cluster:
- Ceph CSI module - CephFS (
shturval-ceph-cephfs); - Ceph CSI module - CephRBD (
shturval-ceph-rbd).
In Kubernetes clusters, modules allow you to use a virtual hard disk (RBD) or a network file system (FS) to store data.
Ceph CSI modules are not critical services for the operation of the Shturval platform. For Ceph CSI Modules to work correctly, the Cluster Network Management Module (shturval-networking) must be running in the cluster.
Please note! Before configuring shturval-ceph-cephfs in the Shturval platform, when configuring Storage on the CephFS side, a subvolumegroup with the name “csi” must be created.
Command for creating subvolumegroup
ceph fs subvolumegroup create ENTER-FILE-SYSTEM-NAME csi
If Ceph CSI Modules were installed when creating the cluster, then manual configuration will not be required in the created cluster.
To enable modules in the created cluster, in the platform GUI, go to the Services and Repositories section and open the Installed Services page and find the Ceph CSI Module.
To install, select the required chart (shturval-ceph-cephfs or shturval-ceph-rbd) and click Manage. In the window that opens, select the desired version. It is recommended to create a new namespace for the chart being installed.
After selecting versions, the available chart values will appear on the right side of the screen. Below are the recommended configuration options.
##CEPHFS
Customvalues example
csiConfig:
- clusterID: <your-parameter-value>
monitors:
- <your-parameter-value>
- <your-parameter-value>
- <your-parameter-value>
secret:
# Specifies whether the secret should be created
create: true
name: <your-parameter-value>
# Key values correspond to a user name and its key, as defined in the
# ceph cluster. User ID should have required access to the 'pool'
# specified in the storage class
adminID: <your-parameter-value>
adminKey: <your-parameter-value>
cephconf: |
<your Ceph cluster configuration>
# Workaround for http://tracker.ceph.com/issues/23446
fuse_set_user_groups = false
# ceph-fuse which uses libfuse2 by default has write buffer size of 2KiB
# adding 'fuse_big_writes = true' option by default to override this limit
# see https://github.com/ceph/ceph-csi/issues/1928
fuse_big_writes = true
storageClass:
# Specifies whether the Storage class should be created
create: true
name: <your-parameter-value>
# Annotations for the storage class
# Example:
# annotations:
# storageclass.kubernetes.io/is-default-class: "true"
annotations: {}
# String representing a Ceph cluster to provision storage from.
# Should be unique across all Ceph clusters in use for provisioning,
# cannot be greater than 36 bytes in length, and should remain immutable for
# the lifetime of the StorageClass in use.
clusterID: <your-parameter-value>
# (required) CephFS filesystem name into which the volume shall be created
# eg: fsName: myfs
fsName: <your-parameter-value>
# (optional) Ceph pool into which volume data shall be stored
# pool: <cephfs-data-pool>
# For eg:
# pool: "replicapool"
pool: ""
# (optional) Comma separated string of Ceph-fuse mount options.
# For eg:
# fuseMountOptions: debug
fuseMountOptions: ""
# (optional) Comma separated string of Cephfs kernel mount options.
# Check man mount.ceph for mount options. For eg:
# kernelMountOptions: readdir_max_bytes=1048576,norbytes
kernelMountOptions: ""
# (optional) The driver can use either ceph-fuse (fuse) or
# ceph kernelclient (kernel).
# If omitted, default volume mounter will be used - this is
# determined by probing for ceph-fuse and mount.ceph
# mounter: kernel
mounter: ""
# (optional) Prefix to use for naming subvolumes.
# If omitted, defaults to "csi-vol-".
# volumeNamePrefix: "foo-bar-"
volumeNamePrefix: <your-parameter-value>
# The secrets have to contain user and/or Ceph admin credentials.
provisionerSecret: <your-parameter-value>
# If the Namespaces are not specified, the secrets are assumed to
# be in the Release namespace.
provisionerSecretNamespace: ""
controllerExpandSecret: <your-parameter-value>
controllerExpandSecretNamespace: ""
nodeStageSecret: <your-parameter-value>
nodeStageSecretNamespace: ""
reclaimPolicy: <your-parameter-value>
allowVolumeExpansion: true
mountOptions: []
# Mount Options
# Example:
# mountOptions:
# - discard
nodeplugin:
tolerations:
- key: <your-parameter-value>
operator: <your-parameter-value>
value: <your-parameter-value>
effect: <your-parameter-value>
| Parameter | Description | Data type | Example |
|---|---|---|---|
clusterID | Unique Ceph cluster identifier | string | my-cluster-id |
monitors | Ceph Monitoring IP List | array | 192.168.1.1 |
secret.name | Name of the secret to be created | string | ceph-secret |
secret.adminID | Service account ID for accessing Ceph | string | client.admin |
secret.adminKey | Service account key for accessing Ceph | string | sdfdff |
cephconf | Ceph configuration in text format | string | [global] auth cluster required = cephx auth service required = cephx auth client required = cephx |
storageClass.name | Name of the created Storage Class | string | csi-cephfs-sc |
storageClass.clusterID | Ceph Cluster ID | string | cluster-id |
storageClass.fsName | Name of the CephFS file system into which the PV will be created | string | myfs |
storageClass.volumeNamePrefix | Prefix for volume names | string | shturval- |
storageClass.provisionerSecret | Secret for Provisioning with Ceph User and/or Admin Credentials | string | cephfs-secret |
storageClass.provisionerSecretNamespace | Namespace containing the secret for Provisioning with user and/or Ceph administrator credentials. If the namespace is not specified, the secret is assumed to be in the “Release” namespace | string | namenamespace |
storageClass.controllerExpandSecret | The Secret to Extending the Controller | string | ceph-expand-secret |
storageClass.controllerExpandSecretNamespace | Namespace, which contains the secret for expanding the controller. If the namespace is not specified, the secret is assumed to be in the “Release” namespace | string | namenamespace |
storageClass.nodeStageSecret | Secret for mounting a node | string | csi-cephfs-secret |
storageClass.controllerExpandSecretNamespace | Namespace in which the secret is located. The secret for mounting the node. If the namespace is not specified, the secret is assumed to be in the “Release” namespace | string | namenamespace |
storageClass.reclaimPolicy | Volume lifecycle management policy (possible values: Retain, Delete, Recycle) | string | Delete |
nodeplugin.tolerations.key | Toleration key | string | role |
nodeplugin.tolerations.operator | Toleration condition check operator | string | Equal |
nodeplugin.tolerations.value | Toleration key value | string | infra |
nodeplugin.tolerations.effect | Toleration effect | string | NoSchedule |
Service account permission requirements
client.kubecephfs mon ‘allow r’ mgr ‘allow rw’ mds ‘allow rws’ osd ‘allow rw pool=cephfs_data, allow rw pool=cephfs_metadata’
CEPH RBD
Customvalues example
csiConfig:
- clusterID: <your-parameter-value>
monitors:
- <your-parameter-value>
- <your-parameter-value>
- <your-parameter-value>
storageClass:
# Specifies whether the storageclass should be created
create: true
name: <your-parameter-value>
# Use Thick Provisioning for this storage class. Default: false. Enabled if not false.
thickProvision: false
# Annotations for the storage class
# Example:
# annotations:
# storageclass.kubernetes.io/is-default-class: "true"
annotations: {}
# (required) String representing a Ceph cluster to provision storage from.
# Should be unique across all Ceph clusters in use for provisioning,
# cannot be greater than 36 bytes in length, and should remain immutable for
# the lifetime of the StorageClass in use.
clusterID: <your-parameter-value>
# (optional) If you want to use erasure coded pool with RBD, you need to
# create two pools. one erasure coded and one replicated.
# You need to specify the replicated pool here in the `pool` parameter, it is
# used for the metadata of the images.
# The erasure coded pool must be set as the `dataPool` parameter below.
# dataPool: <ec-data-pool>
dataPool: ""
# (required) Ceph pool into which the RBD image shall be created
# eg: pool: replicapool
pool: <your-parameter-value>
# (optional) RBD image features, CSI creates image with image-format 2 CSI
# RBD currently supports `layering`, `journaling`, `exclusive-lock`,
# `object-map`, `fast-diff`, `deep-flatten` features.
# Refer https://docs.ceph.com/en/latest/rbd/rbd-config-ref/#image-features
# for image feature dependencies.
# imageFeatures: layering,journaling,exclusive-lock,object-map,fast-diff
imageFeatures: ""
# (optional) Specifies whether to try other mounters in case if the current
# mounter fails to mount the rbd image for any reason. True means fallback
# to next mounter, default is set to false.
# Note: tryOtherMounters is currently useful to fallback from krbd to rbd-nbd
# in case if any of the specified imageFeatures is not supported by krbd
# driver on node scheduled for application pod launch, but in the future this
# should work with any mounter type.
# tryOtherMounters: false
# (optional) uncomment the following to use rbd-nbd as mounter
# on supported nodes
# mounter: rbd-nbd
mounter: ""
# (optional) ceph client log location, eg: rbd-nbd
# By default host-path /var/log/ceph of node is bind-mounted into
# csi-rbdplugin pod at /var/log/ceph mount path. This is to configure
# target bindmount path used inside container for ceph clients logging.
# See docs/rbd-nbd.md for available configuration options.
# cephLogDir: /var/log/ceph
cephLogDir: ""
# (optional) ceph client log strategy
# By default, log file belonging to a particular volume will be deleted
# on unmap, but you can choose to just compress instead of deleting it
# or even preserve the log file in text format as it is.
# Available options `remove` or `compress` or `preserve`
# cephLogStrategy: remove
cephLogStrategy: ""
# (optional) Prefix to use for naming RBD images.
# If omitted, defaults to "csi-vol-".
# volumeNamePrefix: "foo-bar-"
volumeNamePrefix: ""
# (optional) Instruct the plugin it has to encrypt the volume
# By default it is disabled. Valid values are "true" or "false".
# A string is expected here, i.e. "true", not true.
# encrypted: "true"
encrypted: ""
# (optional) Use external key management system for encryption passphrases by
# specifying a unique ID matching KMS ConfigMap. The ID is only used for
# correlation to configmap entry.
encryptionKMSID: ""
# Add topology constrained pools configuration, if topology based pools
# are setup, and topology constrained provisioning is required.
# For further information read TODO<doc>
# topologyConstrainedPools: |
# [{"poolName":"pool0",
# "dataPool":"ec-pool0" # optional, erasure-coded pool for data
# "domainSegments":[
# {"domainLabel":"region","value":"east"},
# {"domainLabel":"zone","value":"zone1"}]},
# {"poolName":"pool1",
# "dataPool":"ec-pool1" # optional, erasure-coded pool for data
# "domainSegments":[
# {"domainLabel":"region","value":"east"},
# {"domainLabel":"zone","value":"zone2"}]},
# {"poolName":"pool2",
# "dataPool":"ec-pool2" # optional, erasure-coded pool for data
# "domainSegments":[
# {"domainLabel":"region","value":"west"},
# {"domainLabel":"zone","value":"zone1"}]}
# ]
topologyConstrainedPools: []
# (optional) mapOptions is a comma-separated list of map options.
# For krbd options refer
# https://docs.ceph.com/docs/master/man/8/rbd/#kernel-rbd-krbd-options
# For nbd options refer
# https://docs.ceph.com/docs/master/man/8/rbd-nbd/#options
# Format:
# mapOptions: "<mounter>:op1,op2;<mounter>:op1,op2"
# An empty mounter field is treated as krbd type for compatibility.
# eg:
# mapOptions: "krbd:lock_on_read,queue_depth=1024;nbd:try-netlink"
mapOptions: ""
# (optional) unmapOptions is a comma-separated list of unmap options.
# For krbd options refer
# https://docs.ceph.com/docs/master/man/8/rbd/#kernel-rbd-krbd-options
# For nbd options refer
# https://docs.ceph.com/docs/master/man/8/rbd-nbd/#options
# Format:
# unmapOptions: "<mounter>:op1,op2;<mounter>:op1,op2"
# An empty mounter field is treated as krbd type for compatibility.
# eg:
# unmapOptions: "krbd:force;nbd:force"
unmapOptions: ""
# The secrets have to contain Ceph credentials with required access
# to the 'pool'.
provisionerSecret: <your-parameter-value>
# If Namespaces are left empty, the secrets are assumed to be in the
# Release namespace.
provisionerSecretNamespace: <your-parameter-value>
controllerExpandSecret: <your-parameter-value>
controllerExpandSecretNamespace: <your-parameter-value>
nodeStageSecret: <your-parameter-value>
nodeStageSecretNamespace: <your-parameter-value>
# Specify the filesystem type of the volume. If not specified,
# csi-provisioner will set default as `ext4`.
fstype: <your-parameter-value>
reclaimPolicy: <your-parameter-value>
allowVolumeExpansion: true
mountOptions: []
# Mount Options
# Example:
# mountOptions:
# - discard
# Mount the host /etc/selinux inside pods to support
# selinux-enabled filesystems
selinuxMount: true
secret:
# Specifies whether the secret should be created
create: true
name: <your-parameter-value>
# Key values correspond to a user name and its key, as defined in the
# ceph cluster. User ID should have required access to the 'pool'
# specified in the storage class
userID: <your-parameter-value>
userKey: <your-parameter-value>
# Encryption passphrase
# encryptionPassphrase: test_passphrase
# This is a sample configmap that helps define a Ceph configuration as required
# by the CSI plugins.
# Sample ceph.conf available at
# https://github.com/ceph/ceph/blob/master/src/sample.ceph.conf Detailed
# documentation is available at
# https://docs.ceph.com/en/latest/rados/configuration/ceph-conf/
cephconf: |
<your Ceph cluster configuration>
nodeplugin:
tolerations:
- key: <your-parameter-value>
operator: <your-parameter-value>
value: <your-parameter-value>
effect: <your-parameter-value>
| Parameter | Description | Data type | Example |
|---|---|---|---|
clusterID | Unique Ceph cluster identifier | string | my-cluster-id |
monitors | Ceph Monitoring IP List | array | 192.168.1.1 |
secret.name | Name of the secret to be created | string | cephrbd-secret |
secret.userID | Service account ID for accessing Ceph | string | client.admin |
secret.userKey | Service account key for accessing Ceph | string | sdfdff |
cephconf | Ceph configuration in text format | string | [global] auth cluster required = cephx auth service required = cephx auth client required = cephx |
storageClass.name | Name of the created Storage Class | string | csi-cephrbd-sc |
storageClass.clusterID | Ceph Cluster ID | string | cluster-id |
storageClass.pool | Ceph pool which should contain cosgiven RBD image | string | kuberbd |
storageClass.provisionerSecret | Secret for Provisioning with Ceph User and/or Admin Credentials | string | csi-rbd-secret |
storageClass.provisionerSecretNamespace | Namespace containing the secret for Provisioning with user and/or Ceph administrator credentials. If the namespace is not specified, the secret is assumed to be in the “Release” namespace | string | namenamespace |
storageClass.controllerExpandSecret | The Secret to Extending the Controller | string | ceph-expand-secret |
storageClass.controllerExpandSecretNamespace | Namespace, which contains the secret for expanding the controller. If the namespace is not specified, the secret is assumed to be in the “Release” namespace | string | namenamespace |
storageClass.nodeStageSecret | Secret for mounting a node | string | csi-cephrbd-secret |
storageClass.controllerExpandSecretNamespace | Namespace in which the secret is located. The secret for mounting the node. If the namespace is not specified, the secret is assumed to be in the “Release” namespace | string | namenamespace |
storageClass.fstype | The volume’s file system type. Default ext4 | string | ext4 |
storageClass.reclaimPolicy | Volume lifecycle management policy (possible values: Retain, Delete, Recycle) | string | Delete |
nodeplugin.tolerations.key | Toleration key | string | role |
nodeplugin.tolerations.operator | Toleration condition check operator | string | Equal |
nodeplugin.tolerations.value | Toleration key value | string | infra |
nodeplugin.tolerations.effect | Toleration effect. Valid values: NoSchedule, PreferNoSchedule, NoExecute | string | NoSchedule |
Service account permission requirements
client.kuberbd mon ‘profile rbd’ osd ‘profile rbd pool=kuberbd’
Customization features
If you need to connect several instances of CEPH FS or CEPH RBD in one cluster:
- Install service instances in different namespaces of the cluster.
- In the driverName parameter, enter arbitrary values that differ between service instances.
- Make the port value in the nodePlugin.httpMetrics.ContainerPort parameter unique for each service instance.