Skip to main content

Automated certificate management

One of the options for configuring TLS/HTTPS connectivity for MintPress is to use cert-manager to automatically manage the certificates for you.

This guide walks you through setting up cert-manager and the considerations you should be aware of when using it.

Considerations​

By default, when using cert-manager, a certificate authority (CA) will be created and used to sign the certificates used by MintPress. Alternatively, cert-manager can be configured to issue certificates from an external certificate authority (e.g. Let's Encrypt, Vault, Venafi) - see the cert-manager documentation for more information.

Whichever CA is used, its certificate will need to be trusted by the container runtime on your Kubernetes nodes, and by any systems from which you will access the MintPress API, UI and secret vault (if you use cert-manager to manage these certificates). This can be done after installing MintPress, as described in the trusting the CA certificate section.

Installation​

Ensure Helm is installed

Please ensure you have installed and validated Helm as described in the installing K3s guide.

You can easily install cert-manager in your Kubernetes cluster using Helm. This will install the cert-manager CRDs and the cert-manager controller in the cert-manager namespace.

helm upgrade --install cert-manager oci://quay.io/jetstack/charts/cert-manager --namespace cert-manager --create-namespace --version v1.20.2 --set "crds.enabled=true" --set "featureGates=AdditionalCertificateOutputFormats=true" --set "webhook.extraArgs={--feature-gates=AdditionalCertificateOutputFormats=true}"

Verify that the cert-manager controller is running:

kubectl get pods -n cert-manager

You should see the cert-manager pods running.

NAME                                      READY   STATUS    RESTARTS   AGE
cert-manager-574b55f49b-7845v             1/1     Running   0          10m
cert-manager-cainjector-5d69f57b7-h57jg   1/1     Running   0          10m
cert-manager-webhook-6b7d7f6f5-d6fjw      1/1     Running   0          10m

Configuration​

By default, MintPress relies on cert-manager to manage all of its certificates, but you can still configure which hostnames you want to use for each service and cert-manager will issue the appropriate certificates matching these hostnames.

tip

Once MintPress is installed, cert-manager will automatically create the CA and use it to issue the certificates for the services it manages.

There is no additional configuration required pre-installation if you're using cert-manager to manage all of your certificates. However, if you opted to provide your own certificates for some components, refer to the manual certificate management guide to configure these certificates before installing MintPress.

Trusting the CA certificate - post installation​

After installing MintPress with cert-manager, you will need to trust the CA certificate on each client machine and in the one where MintPress is hosted in order to access the MintPress API, UI and secret vault via HTTPS.

Once MintPress is installed, you can extract the CA certificate created by cert-manager from the opschain-ca-key-pair secret by running the following command:

kubectl -n ${KUBERNETES_NAMESPACE} get secret opschain-ca-key-pair -o jsonpath="{.data.ca\.crt}" | base64 -d > opschain-ca.pem

Each platform has a different way of trusting a certificate. Follow your platform's documentation to trust the certificate so you're able to access the MintPress API, UI and secret vault via HTTPS.

Forcing certificate renewal​

cert-manager automatically renews each certificate before it expires - by default, once two-thirds of its validity period (duration) has elapsed. opschain-ca and the certificates it issues are all configured with a 10 year duration, so this happens rarely. If you need to force a renewal sooner - for example, immediately after an upgrade that changes a certificate's duration, or if a certificate needs to be replaced - MintPress does not install the cmctl CLI, so rather than cmctl renew, trigger a renewal by deleting the Kubernetes secret backing the certificate. cert-manager detects that the secret is missing and reissues it immediately.

You can check a certificate's current status, including its expiry and next scheduled renewal, at any time:

kubectl -n ${KUBERNETES_NAMESPACE} describe cert opschain-ca

Renewing the CA​

kubectl -n ${KUBERNETES_NAMESPACE} delete secret opschain-ca-key-pair

Wait for cert-manager to reissue it before continuing:

kubectl -n ${KUBERNETES_NAMESPACE} get certificate opschain-ca -w

Once it shows READY True, re-extract and re-trust the CA certificate as described in trusting the CA certificate above, and if you use K3s, follow the K3s registry trust steps to pick up the change there too.

Renew the CA before its leaf certificates

If you're also renewing the certificates the CA issues (below), always renew the CA first and wait for it to finish before renewing any of the others. Each leaf certificate is signed using whatever is currently in the opschain-ca-key-pair secret at the moment it's reissued - renewing a leaf certificate before the CA has finished rotating just signs it with the CA that's about to be replaced, and it will need to be renewed again afterwards. If this happens, there's no lasting harm - once the CA has finished rotating, just delete that leaf certificate's secret again to reissue it correctly.

Renewing the leaf certificates​

Each certificate cert-manager issues for MintPress's services is stored in a secret of the same name: opschain-api-cert, opschain-image-registry-cert, opschain-secret-vault-external-cert, opschain-secret-vault-cert, and opschain-build-service-cert. Force any of them to be renewed the same way:

kubectl -n ${KUBERNETES_NAMESPACE} delete secret opschain-api-cert opschain-image-registry-cert opschain-secret-vault-external-cert opschain-secret-vault-cert opschain-build-service-cert

Wait for all of them to show READY True before continuing:

kubectl -n ${KUBERNETES_NAMESPACE} get certificates.cert-manager.io

Restarting services after renewing a leaf certificate​

Renewing a certificate updates its Kubernetes secret, but not every service notices the change on its own:

  • The API, secret vault UI, and image registry are all served through the Kong ingress, which watches its certificate secrets directly and reloads them automatically - no restart needed.

  • The build service reads its certificate from disk once, when it starts, and nothing rolls it for you, so renewing opschain-build-service-cert needs a manual restart to take effect:

    kubectl -n ${KUBERNETES_NAMESPACE} rollout restart deployment/opschain-build-service

  • The secret vault also reads its certificate from disk once per pod, but it picks up a renewed opschain-secret-vault-cert or opschain-secret-vault-external-cert on its own - a change to either secret is detected automatically and the vault's pods are rolled within a few minutes, one at a time, so no manual action is required. If you want the renewed certificate to take effect straight away rather than waiting, you can roll every replica yourself:

    kubectl -n ${KUBERNETES_NAMESPACE} delete pod -l app.kubernetes.io/name=openbao
    info

    The secret vault runs as a multi-replica StatefulSet whose pods are only replaced when deleted, not restarted via a rollout. Rolling the pods to pick up a renewed certificate therefore happens by deleting them, which MintPress does automatically after the certificate changes; deleting them by label yourself simply rolls every replica at once instead of waiting.

  • After renewing opschain-secret-vault-cert or opschain-secret-vault-external-cert, also restart the API so it stops using its existing connection to the secret vault:

    kubectl -n ${KUBERNETES_NAMESPACE} rollout restart deployment/opschain-api deployment/opschain-api-worker

What to do next​