Skip to main content

Introduction

MintPress requires TLS certificates for its internal communication with its services, such as the image registry and the build service. It can also be configured to use TLS/HTTPS connectivity for its API and web UI, ensuring that all communication is encrypted and secure.

This guide will walk you through the necessary information to understand how to configure TLS/HTTPS connectivity for MintPress and the options available to you.

Networking​

MintPress uses Kong Ingress proxy to route traffic to its services, this means that the hostname on which MintPress and its services are accessible defaults to the hostname of the VM it is installed on. It is your responsibility to ensure that this server's hostname is reachable from your machine/network by creating the appropriate DNS entries for it.

Add two DNS entries pointing to the server's hostname, one with the API hostname and the other with the secret vault's external hostname.

Port mapping​

All services sit behind the Kong Ingress proxy, which listens on port 3000 for HTTP and port 3443 for HTTPS connections. The requests are then forwarded to the appropriate services based on the request hostname.

MintPress services listen on the following ports by default:

ServicePort(s)
API and UI3000 for HTTP, 3443 for HTTPS
Build service50000
Database5432
Image registry8000
LDAP389
Log aggregator24220, 24224
Secret vault8200, 8201
info

These ports are only accessible internally from the Kubernetes cluster and the machine it is hosted on. Unless you expose them via an external service, they won't be accessible from outside the cluster. See the Kubernetes topology reference for the services these ports belong to.

Additionally, when enabling external services for the secret vault or the database (for high availability setups), these ports will need to be opened on the firewall to allow incoming connections from the other instances.

Database connections​

The OpsChain API, its workers, the pods that run changes and the secret vault connect to the database only over TLS, and only with SCRAM password authentication.

The default database authentication rules use SCRAM for MintPress's connections. If you replace them with db.cnpg.security.pgHba.customRules, the rules for the opschain user must use scram-sha-256 or md5, or MintPress cannot connect to its database.

Disabling the insecure HTTP listener​

You can disable the Kong Ingress proxy's insecure HTTP listener on port 3000 by setting kong.proxy.http.enabled to false in your values.yaml file.

kong:
proxy:
http:
enabled: false
warning

Only disable the insecure HTTP listener after ensuring you can reach the API, UI and secret vault via HTTPS.

Enforcing HTTPS in browsers​

MintPress can send a Strict-Transport-Security (HSTS) header on every response from the API, the web UI and the secret vault's external hostname. A browser that receives it connects to that hostname over HTTPS only, and keeps doing so until the header's maximum age has passed since its last visit.

Enable it in your values.yaml file:

api:
hsts:
enabled: true
maxAge: 31536000
SettingDefaultDescription
api.hsts.enabledfalseSend the header.
api.hsts.maxAge31536000The number of seconds a browser keeps using HTTPS only for the hostname. 31536000 is one year.
api.hsts.includeSubDomainsfalseApply the header to every subdomain of each hostname as well. Only enable this if every one of those subdomains serves HTTPS with a certificate your users' browsers trust.

After the upgrade, confirm the header is sent:

curl -sI https://opschain.my-company.com/ | grep -i strict-transport-security
warning

Only enable HSTS once your users' browsers trust the API certificate, for example one issued by your organisation's certificate authority. Browsers ignore the header from a hostname whose certificate they don't trust. Once a browser has recorded the header from a trusted certificate, it no longer lets the user continue past a certificate error on that hostname, such as an expired certificate, until maxAge has passed.

tip

Start with a short maxAge, such as 300 (five minutes), and raise it once you have confirmed the API, UI and secret vault work over HTTPS. Disabling HSTS or lowering maxAge only takes effect in a browser the next time it visits.

Certificates​

MintPress requires the following certificates for its services to function correctly:

  • Image registry certificate: This certificate is for the internal communication between the image registry and the API, using the hostname configured via env.OPSCHAIN_IMAGE_REGISTRY_HOST.
  • Build service certificate: This certificate is for the internal communication between the build service and the API. The build service hostname is opschain-build-service and cannot be changed.

For the OpsChain secret vault, you will also need the following certificates:

  • Secret vault internal certificate: This certificate is for the internal communication between the secret vault and the API. The secret vault internal hostname is opschain-secret-vault-0.opschain-secret-vault-internal and cannot be changed.
  • Secret vault external certificate: This certificate is for accessing the secret vault's UI from the client machines.
*.local.gd does not reliably resolve to 127.0.0.1

The provided self-signed certificates use *.local.gd hostnames on the assumption that this domain's DNS automatically resolves any subdomain to 127.0.0.1. This is not guaranteed — local.gd's DNS is controlled by a third party outside LimePoint, and it does not currently resolve mintpress-image-registry.local.gd (or any other *.local.gd subdomain) to anything at all. Do not rely on it. Instead, add an explicit entry for each *.local.gd hostname you use to the server's hosts file:

echo "127.0.0.1 mintpress-image-registry.local.gd" >> /etc/hosts

If your /etc/hosts file is managed by a configuration management tool such as Ansible or Chef, ensure that it is updated to include the entry. See image registry hostname resolution for the equivalent guidance when using a hostname of your own choosing instead.

Optional certificates​

Optionally, you can also use HTTPS for MintPress's API and web UI, which will require an additional API certificate, used exclusively for external access to the API and UI.

tip

You can first install MintPress without HTTPS connectivity for the API, and then configure it to use HTTPS later.

Trusting additional certificate authorities​

Two of the vault's certificates are involved, and they do not have to come from the same authority:

  • the internal certificate, presented on the connection MintPress opens to the vault inside the cluster
  • the external certificate, presented by the ingress at global.secretVaultExternalHostName. In a high availability setup, a request that reaches a cluster whose secret vault is not the active one is redirected there, so MintPress reaches this certificate during normal operation, not only from a browser

If those two come from different authorities, MintPress must trust both. The API contacts the vault as it starts, so an authority it does not trust stops the API from starting rather than failing a single request. See secret vault requests fail with certificate verify failed if that has already happened.

MintPress keeps the certificate authorities you trust in its trust store, and every MintPress workload uses them - the connections the API makes to the secret vault included. Adding an authority to the trust store is all that is required; no vault setting needs to name it.

Supplying an authority at install or upgrade​

Set trustStore.certificates in your values.yaml file, keyed by the filename to hold each authority under:

trustStore:
certificates:
corp-root-ca.pem: |
-----BEGIN CERTIFICATE-----
MIIDDzCCAfegAwIBAgIUccZDqIbWK49Sr9wEovijlne0XXowDQYJKoZIhvcNAQEL
-----END CERTIFICATE-----

To read an authority from a file rather than pasting it, add --set-file to the installation command:

--set-file trustStore.certificates.corp-root-ca\.pem=/path/to/corp-root-ca.pem

Supplied this way, the authority is in place before the OpsChain API pod is created, so it is trusted the first time the API starts. Use this route for an authority the API needs in order to start at all.

Certificates must be PEM encoded. A certificate exported from Windows is DER encoded by default, so convert it first:

openssl x509 -inform der -in corp-root-ca.crt -out corp-root-ca.pem
warning

Supply your values with -f values.yaml on every upgrade. helm upgrade reuses the previous release's values when it is given none, so an authority removed from values.yaml stays in place until the file is supplied again.

Uploading an authority to a running instance​

Upload the certificate in the CA certificates section of the system configuration page. Certificates uploaded there and certificates supplied in values.yaml are held alongside each other, and an upgrade leaves the uploaded ones untouched.

Uploading needs a running API, so it cannot be used to trust an authority the API itself needs in order to start. Supply that one in values.yaml.

In a high availability setup, an uploaded certificate is trusted by every site in the topology. The site that handles the upload trusts it straight away, and the other sites within 10 minutes, or as soon as their API restarts. Changing or deleting an uploaded certificate reaches every site the same way, so an authority you stop trusting is removed everywhere.

How uploaded and supplied certificates combine​

MintPress trusts both the certificates supplied in values.yaml and the ones uploaded through the system configuration page:

  • A certificate supplied in values.yaml takes precedence. MintPress rejects an upload that uses the same filename, so choose a different filename or change the certificate in values.yaml.
  • A certificate supplied in values.yaml changes only when you upgrade with a new values.yaml, and applies to the site you deploy it to.
  • MintPress keeps the rest of the trust store in line with the uploaded certificates, so a certificate added to the opschain-trust-store config map by hand is removed within 10 minutes. Supply it in values.yaml or upload it instead.

OpsChain certificate authorities​

Every MintPress site creates its own certificate authority, which signs the certificates of its internal services, the secret vault's included. MintPress trusts the authority of every site in the topology, so the sites of a high availability setup trust each other's secret vault without any configuration.

These authorities are listed in the CA certificates section as opschain-system-ca-<site>.pem, where <site> is the site's db.cnpg.clusterName. MintPress maintains them itself, so they cannot be uploaded, changed or deleted, and filenames starting with opschain-system-ca are reserved.

Where the trust store is mounted​

The trust store is the opschain-trust-store config map, mounted at /etc/pki/ca-trust/source/anchors/ in the MintPress pods that connect to other services on your behalf. Each of those pods rebuilds its trusted certificate bundle whenever the config map changes, so an authority added to a running instance is picked up without restarting anything - allow a minute or two for Kubernetes to publish the change into each pod.

The same certificates are built into the images that run your steps, so your actions trust the authorities the rest of MintPress does. See step runner for how that image is assembled.

Hostname configuration​

You can configure the hostnames that your certificates use (or should be issued for) by each service in your values.yaml file. After configuring your hostnames, follow one of the available options guide to configure your certificates.

warning

When using your own certificates or provided self-signed certificates, it is your responsibility to ensure that the certificates include a DNS subjectAlternativeName that matches the hostnames you defined in your values.yaml file.

API and web UI hostname​

By default, the OpsChain API will use the opschain-api hostname, which points to its Kubernetes service, meaning it will be accessible at your server's external IP address/hostname. You can change the hostname used by the API to something more friendly by setting the api.hostName and the env.OPSCHAIN_API_HOST_NAME values in your values.yaml file.

api:
hostName: "opschain.my-company.com"

env:
OPSCHAIN_API_HOST_NAME: "opschain.my-company.com"
info

The OpsChain API and UI always share the same hostname.

You must also configure the env.OPSCHAIN_GUI_BASE_URL setting in your values.yaml file to ensure that links within notifications are valid. For example:

env:
OPSCHAIN_GUI_BASE_URL: "https://opschain.my-company.com"
Port and scheme configuration

If the API and UI are accessible via a non-standard port, like 443 for HTTPS and 80 for HTTP, you must include the port in the URL. For example: https://opschain.my-company.com:3443. You should also include the scheme in the URL, either https or http, depending on the protocol you opted to use.

Build service hostname​

The default build service hostname is opschain-build-service and cannot be changed.

Image registry hostname​

Unlike the API and web UI hostname, the image registry hostname is not set to a working value out of the box — env.OPSCHAIN_IMAGE_REGISTRY_HOST and the related trow settings are blank by default. You must explicitly set the trow.trow.domain, trow.ingress.hosts[0].host, trow.ingress.tls[0].hosts[0] and the env.OPSCHAIN_IMAGE_REGISTRY_HOST values in your values.yaml file, either to mintpress-image-registry.local.gd (the hostname used by the provided self-signed certificates) or to a hostname of your own choosing. For example:

trow:
trow:
domain: "image-registry.my-company.com"
ingress:
hosts:
- paths: [ "/" ]
host: "image-registry.my-company.com"
tls:
- secretName: "opschain-image-registry-cert"
hosts:
- "image-registry.my-company.com"

env:
OPSCHAIN_IMAGE_REGISTRY_HOST: "image-registry.my-company.com"
Image registry hostname resolution

If env.OPSCHAIN_IMAGE_REGISTRY_HOST is left unset, the opschain-image-registry-cert certificate will fail to render and the installation will fail.

Once set, you must also ensure the internal image registry hostname resolves to the Kubernetes node's IP address so that MintPress can access it. It's recommended you do this mapping via your company's DNS server, but you can also achieve this by adding the following entry to the server's hosts file:

echo "127.0.0.1 image-registry.my-company.com" >> /etc/hosts

If your /etc/hosts file is managed by a configuration management tool such as Ansible or Chef, ensure that it is updated to include the entry.

The hostname you use may vary if you have used a different domain name for the internal image registry in your values.yaml file.

Secret vault hostname​

The OpsChain secret vault has two hostnames:

  • Internal hostname: opschain-secret-vault.opschain-secret-vault-internal - This hostname cannot be changed.
  • External hostname: the hostname you'll use to access the secret vault's UI.

The external hostname must be configured in your values.yaml file by setting the global.secretVaultExternalHostName value:

global:
secretVaultExternalHostName: "vault.my-company.com"
warning

The OpsChain secret vault uses the same ingress as the API, ensure that the host name you provide for the secret vault is different than the API host name.

For the clients that need to access the secret vault UI, you must configure your DNS to map the secret vault hostname to the server where MintPress is installed.

In a high availability setup, the external hostname must also resolve inside the cluster

In a high availability setup, each cluster runs its own secret vault and only one of them is active. A request from MintPress — the API, workers, and runner pods — that reaches a standby is forwarded or redirected to the active secret vault at its external hostname. For that to succeed, the external hostname must resolve from inside the Kubernetes cluster, not only from client machines.

You can test this by running the following command with your own vault hostname on a running MintPress instance:

kubectl -n ${KUBERNETES_NAMESPACE} exec -it deploy/opschain-api-worker -- getent hosts vault.my-company.com

If the command returns a result, the hostname resolves from within the cluster. If it does not, you need to configure your DNS to resolve the hostname either by adding it to your organisation's DNS, or, on K3s, adding it to CoreDNS with a coredns-custom ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
name: coredns-custom
namespace: kube-system
data:
opschain-vault.server: |
vault.my-company.com {
hosts {
192.168.0.10 vault.my-company.com
fallthrough
}
}

Apply it and restart CoreDNS:

kubectl apply -f coredns-custom.yaml
kubectl -n kube-system rollout restart deployment coredns

In a high availability setup, each cluster's CoreDNS must resolve the external hostname of every cluster's secret vault, since the active vault can move between clusters.

Available options​

There are a few options available for configuring TLS/HTTPS connectivity for MintPress, each with their own advantages and disadvantages:

  1. Using cert-manager to automatically manage certificates for you. This is the recommended approach if you want to quickly setup your instance and don't want to manage certificates manually.
  2. Using the provided self-signed certificates. This option is useful if you want to trial MintPress without cert-manager and don't have your own certificates yet.
  3. Bringing your own certificates. This option is the most flexible if you are running in an enterprise environment and can only trust your own certificates.
  4. Using a combination of cert-manager and your own certificates. You can use cert-manager to automatically manage certificates for the internal services while providing your own certificates for the API, UI and secret vault, for example. Refer to the cert-manager guide to install cert-manager and then refer to the manual certificate management guide to configure your own certificates.
Considerations

Remember that TLS certificates expire and need to be renewed. Depending on your chosen option, you will need to manually renew your certificates before they expire to ensure that your instance continues to function properly. Check the relevant guide for more information on how to renew your certificates.

What to do next​

  • Follow the guide for your chosen option to configure your certificates and proceed with the installation of MintPress.