ACME #

ACME Certificate automates the receipt of SSL certificates through the ACME server. To configure automatic issuance of certificates through the ACME server, you need to obtain the root CA certificate that the ACME server uses, encode its data and add the certificate to the cluster manually or through ClusterConfig.

Prepare a backup #

In the backup, include resources associated with the ACME server, cert-manager and system services for which certificates will be issued.

Non-namespace resources:

• shturvalserviceconfigs.

Namespaces:

For any cluster type:

• shturval-cd, if Argo CD is used;
• cert-manager.

For the management cluster:

• shturval-backend;
• shturval-dashboards if Grafana is used.

Namespace resources:

• secrets;
• clusterissuers.

Step 1: Prepare ACME #

If your organization already has an ACME server, request the root CA certificate for that server from your administrator.

If there is no ACME server, you can install step-ca and obtain a root CA certificate. Using a certificate from step-ca in the root CA platform will ensure trust in all certificates that will be issued by a certificate from step-ca.

step-ca can be set in several ways. Below is an example of installation via a Docker container. If necessary, select a different installation method from step-ca official documentation.

1. Create a directory for configuration and certificates:

mkdir data
chown 1000:1000 data

2. Create a file docker-compose.yaml:

services:
  step-ca:
    container_name: step-ca
    image: smallstep/step-ca:0.27.2
    restart: always
    ports:
        - "9443:9000"
    environment:
       DOCKER_STEPCA_INIT_DNS_NAMES: issuer.ip-XX-XX-XXX-XX.shturval.link # the hostname(s) or IPs that the CA will accept requests on
       DOCKER_STEPCA_INIT_NAME: shturval # the name of your CA - this will be the issuer of your CA certificates
       DOCKER_STEPCA_INIT_REMOTE_MANAGEMENT: true # enable remote provisioner management
       DOCKER_STEPCA_INIT_ACME: true # also create an initial ACME provisioner for the CA
       # DOCKER_STEPCA_INIT_PROVISIONER_NAME: admin # a label for the initial admin (JWK) provisioner.
       # DOCKER_STEPCA_INIT_SSH: "" # set this to a non-empty value to enable SSH certificate support
       # DOCKER_STEPCA_INIT_PASSWORD_FILE: "" # the location of a password file to be used for both private keys and the default CA provisioner.
       # DOCKER_STEPCA_INIT_PASSWORD: shturval # Normally, CA passwords will be generated for you. With this option, you can specify a password for the encrypted CA keys and the default CA provisioner.
    volumes:
        - ./data:/home/step # 1000:1000
    logging:
      driver: "json-file"
      options:
        max-size: "200m"

In the DOCKER_STEPCA_INIT_DNS_NAMES parameter, specify the DNS name by which the server responds.

3. Run step-ca:

# Start step-ca server
docker compose up -d
# Startup check
docker compose ps

If the container is in UP state, the server is ready to issue certificates.

4. Obtain the root CA certificate of the server step-ca:

cat data/certs/root_ca.crt

Sample answer:

-----BEGIN CERTIFICATE-----
MIIBoDCCAUagAwIBAgIRAJV9+1S0q+d9+V/UmBXhR+4wCgYIKoZIzj0EAwIwLjER
MA8GA1UEChMIc2h0dXJ2YWwxGTAXBgNVBAMTEHNodHVydmFsIFJvb3QgQ0EwHhcN
MjQxMTIyMTUyMzQ5WhcNMzQxMTIwMTUyMzQ5WjAuMREwDwYDVQQKEwhzaHR1cnZh
bDEZMBcGA1UEAxMQc2h0dXJ2YWwgUm9vdCBDQTBZMBMGByqGSM49AgEGCCqGSM49
AwEHA0IABNhNCtkBTfs+HXA6h1Ty8Qlg+zKQNu
-----END CERTIFICATE-----

Step 2. Add to cluster #

apiVersion: cert-manager.io/v1
kind:ClusterIssuer
metadata:
  name: corp-acme
spec:
  acme:
    caBundle: <root_CA_certificate_ACME_server_base64>
    email: cluster@shturval.tech
    privateKeySecretRef:
      name: corp-acme
    server: https://issuer.ip-XX-XX-XX-XX.shturval.link:9443/acme/acme/directory
    solvers:
    - http01:
        ingress:
          class:nginx

apiVersion: ops.shturval.tech/v1beta2
kind: ShturvalServicePatch
metadata:
  name: <resource name>
spec:
  shturvalServiceConfigName: shturval-backend
  customvalues:
    global:
      tls:
        enabled: true
        backend_tls_secretname: <your parameter value>
        cluster_issuer_name: <your parameter value>

apiVersion: ops.shturval.tech/v1beta2
kind: ShturvalServicePatch
metadata:
  name: <resource name>
spec:
  shturvalServiceConfigName: shturval-frontend
  customvalues:
    tls:
      enabled: true
      front_tls_secretname: <your parameter value>
      docs_tls_secretname: <your parameter value>
      cluster_issuer_name: <your parameter value>

apiVersion: ops.shturval.tech/v1beta2
kind: ShturvalServicePatch
metadata:
  name: <resource name>
spec:
  shturvalServiceConfigName: shturval-auth
  customvalues:
    tls:
      enabled: true
      auth_tls_secretname: <your parameter value>
      cluster_issuer_name: <your parameter value>

apiVersion: ops.shturval.tech/v1beta2
kind: ShturvalServicePatch
metadata:
  name: <resource name>
spec:
  shturvalServiceConfigName: shturval-dashboards
  customvalues:
    grafana.ini:
      security:
        cookie_secure: true
      auth.generic_oauth:
        tls_skip_verify_insecure: true
    ingress:
      annotations:
        cert-manager.io/cluster-issuer: <your parameter value>
      enabled: true
      tls:
      - hosts:
        - <your parameter value>
        secretName: <your parameter value>

apiVersion: ops.shturval.tech/v1beta2
kind: ShturvalServicePatch
metadata:
  name: <resource name>
spec:
  shturvalServiceConfigName: shturval-cd
  customvalues:
    server:
      ingress:
        annotations:
          cert-manager.io/cluster-issuer: <your parameter value>
        enabled: true
        hostname: <your parameter value>
        path: /
        pathType: Prefix
        tls: true

base64 -w0 acme-ca-bundle.crt
base64 -w0 acme-root-ca.crt

export CLUSTER_NAMESPACE=<cluster_namespace>
export CLUSTER_NAME=<cluster_name>
kubectl -n "$CLUSTER_NAMESPACE" get clusterconfig "$CLUSTER_NAME" -o yaml > "clusterconfig-$CLUSTER_NAME-backup.yaml"

kubectl -n "$CLUSTER_NAMESPACE" edit clusterconfig "$CLUSTER_NAME"

apiVersion: cluster.shturval.tech/v1beta1
kind: ClusterConfig
metadata:
  name: <cluster_name>
  namespace: <cluster_namespace>
spec:
  clusterName: <cluster_name>
  systemServicesConfig:
    ingress:
      acmeConfig:
        acmerootcabundle: <CA_bundle_base64>
        acmeserverurl: https://issuer.ip-XX-XX-XX-XX.shturval.link:9443/acme/acme/directory
        acmerootca: <root_CA_certificate_base64>
        email: cluster@shturval.tech

Let’s Encrypt #

To create certificates using Let’s Encrypt, the ingress controller must be accessible from the Internet. At least one condition must be met:

• the external balancer has a white (public) IP address;
• for IngressVIP a white (public) IP address is specified.

Prepare a backup #

In the backup, include resources associated with cert-manager and system services for which Let’s Encrypt will issue certificates.

Non-namespace resources:

• shturvalserviceconfigs.

Namespaces:

For any cluster type:

• shturval-cd, if Argo CD is used;
• cert-manager.

For the management cluster:

• shturval-backend;
• shturval-dashboards if Grafana is used.

Namespace resources:

• secrets;
• clusterissuers.

apiVersion: cert-manager.io/v1
kind:ClusterIssuer
metadata:
  name: <resource name>
  namespace: cert-manager
spec:
  acme:
    server: <Let's Encrypt server URL>
    email: <Let's Encrypt account email>
    privateKeySecretRef:
      name: <secret name for account key>
    solvers:
    - http01:
        ingress:
          class:nginx

kubectl apply -f acme-issuer.yaml

apiVersion: ops.shturval.tech/v1beta2
kind: ShturvalServicePatch
metadata:
  name: <resource name>
spec:
  shturvalServiceConfigName: shturval-backend
  customvalues:
    global:
      tls:
        enabled: true
        backend_tls_secretname: <your parameter value>
        cluster_issuer_name: <your parameter value>

apiVersion: ops.shturval.tech/v1beta2
kind: ShturvalServicePatch
metadata:
  name: <resource name>
spec:
  shturvalServiceConfigName: shturval-frontend
  customvalues:
    tls:
      enabled: true
      front_tls_secretname: <your parameter value>
      docs_tls_secretname: <your parameter value>
      cluster_issuer_name: <your parameter value>

apiVersion: ops.shturval.tech/v1beta2
kind: ShturvalServicePatch
metadata:
  name: <resource name>
spec:
  shturvalServiceConfigName: shturval-auth
  customvalues:
    tls:
      enabled: true
      auth_tls_secretname: <your parameter value>
      cluster_issuer_name: <your parameter value>

apiVersion: ops.shturval.tech/v1beta2
kind: ShturvalServicePatch
metadata:
  name: <resource name>
spec:
  shturvalServiceConfigName: shturval-dashboards
  customvalues:
    grafana.ini:
      security:
        cookie_secure: true
      auth.generic_oauth:
        tls_skip_verify_insecure: true
    ingress:
      annotations:
        cert-manager.io/cluster-issuer: <your parameter value>
      enabled: true
      tls:
      - hosts:
        - <your parameter value>
        secretName: <your parameter value>

apiVersion: ops.shturval.tech/v1beta2
kind: ShturvalServicePatch
metadata:
  name: <resource name>
spec:
  shturvalServiceConfigName: shturval-cd
  customvalues:
    server:
      ingress:
        annotations:
          cert-manager.io/cluster-issuer: <your parameter value>
        enabled: true
        hostname: <your parameter value>
        path: /
        pathType: Prefix
        tls: true

https://acme-v02.api.letsencrypt.org/directory

export CLUSTER_NAMESPACE=<cluster_namespace>
export CLUSTER_NAME=<cluster_name>
kubectl -n "$CLUSTER_NAMESPACE" get clusterconfig "$CLUSTER_NAME" -o yaml > "clusterconfig-$CLUSTER_NAME-backup.yaml"

kubectl -n "$CLUSTER_NAMESPACE" edit clusterconfig "$CLUSTER_NAME"

apiVersion: cluster.shturval.tech/v1beta1
kind: ClusterConfig
metadata:
  name: <cluster_name>
  namespace: <cluster_namespace>
spec:
  clusterName: <cluster_name>
  systemServicesConfig:
    ingress:
      acmeConfig:
        acmerootcabundle: <CA_base64>
        acmeserverurl: https://acme-v02.api.letsencrypt.org/directory
        email: name@example.com

Corporate certificate #

A corporate certificate is a ready-made end-entity certificate that the company CA has already issued.

How you add it to the cluster in step 2 depends on the certificate:

• A wildcard (the name contains *, for example *.clustername.corp.domain) can be added manually or via ClusterConfig.
• A certificate for an explicit list of names (no *) can be added only manually: a Secret and ShturvalServicePatch. Via ClusterConfig, cluster-manager writes the certificate from spec.systemServicesConfig.ingress.wildcardCert, and that field is for a wildcard certificate.

Prepare a backup copy #

In the backup, include the resources associated with the Ingress enterprise certificate.

Non-namespace resources:

• shturvalserviceconfigs.

Namespaces:

For any cluster type:

• ingress;
• cert-manager.

Namespace resources:

• secrets;
• configmaps.

Step 1. Prepare the certificate #

1. Create a configuration file openssl-san.cnf. In the alt_names block, specify the Ingress domain name:

cat > openssl-san.cnf <<EOF
[req]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn

[ dn ]
C=RU
ST=Moscow
L=Moscow
O=My Company
OU=Department
emailAddress=admin@example.com
CN = example.com

[ req_ext ]
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = *.clustername.corp.domain

[ v3_ext ]
authorityKeyIdentifier=keyid,issuer:always
basicConstraints=CA:FALSE
keyUsage=keyEncipherment,dataEncipherment
extendedKeyUsage=serverAuth,clientAuth
subjectAltName=@alt_names
EOF

Where corp.domain is the Ingress domain the platform uses. The example above is a wildcard: you can add it manually or via ClusterConfig.

If you need a certificate for a list of names, list them explicitly in alt_names, without *. Then use only the Manually tab.

[ alt_names ]
DNS.1 = ingress.clustername.corp.domain
DNS.2 = dashboards.clustername.corp.domain

2. Create a certificate request (CSR):

openssl req -newkey rsa:2048 -sha256 -days 365 -nodes -keyout tls.key -out tls.csr -extensions req_ext -config openssl-san.cnf

3. Send tls.csr to your Certificate Authority (CA) for signature.
4. Obtain the signed certificate in PEM format and rename it to tls.crt.

To add a corporate certificate to the cluster you will need:

• tls.crt — the corporate certificate in PEM format: the end-entity certificate, intermediate CA certificates, and the root CA. The certificate must be issued by a certificate authority or be self-signed for the Ingress domain name. This is the certificate data in Secret corp-tls;
• tls.key — the private key of the corporate Ingress certificate;
• the root certificate of your certificate authority, or the chain up to it. The administrator of that authority issues it together with the signed certificate. Save the PEM you received under a name such as root-ca.crt.

Check the certificate files before loading them into the cluster: Check the files before loading into the cluster.

The check of an Ingress that is already running is in Verify the certificate.

Check the files before loading into the cluster #

Before loading into the cluster, check:

1. The Ingress certificate is signed by your certificate authority: from the certificate in tls.crt, through the intermediates, to the root certificate.

openssl verify -show_chain -CAfile root-ca.crt -untrusted tls.crt tls.crt

If you put the root certificate as the last block of tls.crt and have no separate root-ca.crt, copy that last block into root-ca.crt and keep it in -CAfile. Without -CAfile the command does not confirm the certificate. OpenSSL uses the intermediates in tls.crt only as steps and does not treat them as trusted.

The output must contain the line tls.crt: OK. OpenSSL prints the chain from top to bottom: the Ingress certificate, then the intermediates, and at the end the root certificate from root-ca.crt. If the line says error instead of OK: the root certificate is a different one, or tls.crt does not contain the intermediate that signed the Ingress certificate.

2. Which certificates are in the file, who issued (signed) the certificate, and to whom it was issued.

openssl crl2pkcs7 -nocrl -certfile tls.crt \
  | openssl pkcs7 -print_certs -noout

In the output, each certificate has two lines: subject is whose name it is, issuer is who signed it.

The Ingress certificate has the intermediate’s name in issuer. That intermediate has the same name in subject. Each pair works the same way: the issuer of the current certificate matches the subject of the next one. The issuer of the last certificate in the list matches the subject of the root certificate in root-ca.crt. If the root certificate is also written at the end of tls.crt, that last entry has the same subject and issuer: the certificate authority signed the certificate itself.

3. Certificate validity periods:

openssl x509 -in root-ca.crt -noout -subject -issuer -dates

In the output, notBefore is the date from which the certificate is valid, notAfter is the date until which it is valid. Today’s date must fall inside this range.

openssl x509 -in tls.crt -noout -subject -issuer -dates

Only the first certificate in tls.crt. notAfter must be later than today. If that date has already passed, you need a new certificate from the certificate authority before loading it into the cluster.

Step 2. Add to cluster #

apiVersion: ops.shturval.tech/v1beta2
kind: ShturvalServicePatch
metadata:
  name: <resource name>
spec:
  shturvalServiceConfigName: shturval-ingress-controller
  customvalues:
    controller:
      extraArgs:
        default-ssl-certificate: "ingress/corp-tls"

base64 -w0 tls.crt
base64 -w0 tls.key
base64 -w0 root-ca.crt

export CLUSTER_NAMESPACE=<cluster_namespace>
export CLUSTER_NAME=<cluster_name>
kubectl -n "$CLUSTER_NAMESPACE" get clusterconfig "$CLUSTER_NAME" -o yaml > "clusterconfig-$CLUSTER_NAME-backup.yaml"

kubectl -n "$CLUSTER_NAMESPACE" edit clusterconfig "$CLUSTER_NAME"

apiVersion: cluster.shturval.tech/v1beta1
kind: ClusterConfig
metadata:
  name: <cluster_name>
  namespace: <cluster_namespace>
spec:
  clusterName: <cluster_name>
  systemServicesConfig:
    ingress:
      wildcardCert:
        ingresscert: <corporate_ingress_certificate>
        ingresskey: <private_key_of_corporate_certificate_ingress>
        rootingresscert: <root_ingress_certificate>

Verify the certificate #

Run the check after step 2, when Ingress is already using the certificate from Secret corp-tls.

Run the command on a machine that can reach Ingress. It opens the site and checks that root-ca.crt validates the site certificate. The file is the certificate or chain from ConfigMap root-ca, if you added the certificate manually, or from rootingresscert, if you set it via ClusterConfig.

openssl s_client -connect <ingress-host>:443 \
  -servername <ingress-host> \
  -CAfile root-ca.crt \
  -no-CApath \
  -no-CAstore \
  -verify_return_error \
  -brief </dev/null

On the management cluster, after the certificate for shturval-auth is replaced, check that the authentication client received the same certificate. Synchronization can take several minutes.

kubectl get authoidcclients.auth.shturval.tech -n shturval-backend backend -o jsonpath='{.status.clientInfo.caFileBase64}' | base64 -d

The output must match the data in tls.crt. If it does not, wait a few minutes and run the command again.