K3s Installation guide

This guide takes you through steps for installing self hosted K3s platform in order to host MintPress.

Linux Certification

K3s for MintPress is tested and supported on RHEL/OEL/AlmaLinux 8.x and 9.x on a VM.

Access to software media and licence

You must have acquired a license from LimePoint before you begin to install. If you don't have one, please reach out to your account manager or drop a note on support@limepoint.com. As part of license acquisition you would also get credentials to download the installers.

You must have access to the following URLs from your VM network either directly or via a proxy.

• https://hub.docker.com/
• https://get.k3s.io/
• https://charts.jetstack.io/
• https://api.github.com
• https://github.com
• https://rpm.rancher.io
• https://raw.githubusercontent.com/
• https://kubernetes.github.io/

Infrastructure requirements

The infrastructure requirements includes minumum configuration required for K3s along with requirements for MintPress. The requirements for MintPress are calculated based on the number of parallel executions you wish to do.

# of Parallel Steps	Memory (GB)	CPU	Storage (GB)
10	4	2	100
100	8	4	100
1000	32	8	300

You can add more storage if you intend to save logs for longer within MintPress. We recommend externalizing logs to Splunk or a file system for better management.

Kernel version

To support native rootless image builds, you must have a Kernel version newer than 5.11. In systems older than 5.11, you can have rootless image builds by enabling the FUSE device plugin in the settings.

More details are provided later in the image build settings documentation.

Supported Platforms

MintPress needs a container platform to run. It can be deployed on any of the following container platforms:

• Azure Kubernetes Services (AKS)
• Elastic Kubernetes Services (EKS)
• OpenShift Container Platform
• Self hosted Kubernetes (k8s or k3s) either on bare metal or a VM

The following setup guides will assume that you are installing MintPress on a self hosted K3s cluster.

Endpoint Security Software Compatibility

MintPress is deployed on K3s with containerd as the container runtime. Endpoint security products running on the host including Antivirus (AV), Endpoint Protection Platform (EPP), Endpoint Detection and Response (EDR), and Extended Detection and Response (XDR) solutions are known to interfere with the correct operation and performance of K3s and containerd.

This is a prerequisite for installation. MintPress cannot function if the paths described below are subject to active filesystem scanning. Configuring the exclusions in this section is mandatory before proceeding.

Background

K3s manages temporary mounts as part of normal container lifecycle operations. Active scanning of these mount points by endpoint security software causes failures in containerd that cannot be mitigated at the application level. This behaviour is documented in the upstream containerd issue tracker (containerd#5538).

Required Path Exclusions

The following directories must be excluded from real-time or on-access scanning in your endpoint security product. Ensure there are no trailing spaces in the configured paths.

Path	Purpose
/limepoint/k3s/	MintPress K3s root
/limepoint/k3s/agent/containerd/	containerd data directory
/var/lib/rancher/k3s/	K3s runtime state

Example: Microsoft Defender for Endpoint (Linux):

sudo mdatp exclusion folder add --path "/limepoint/k3s/"
sudo mdatp exclusion folder add --path "/limepoint/k3s/agent/containerd/"
sudo mdatp exclusion folder add --path "/var/lib/rancher/k3s/"

The exact procedure for configuring exclusions will differ depending on your endpoint security product. Consult your vendor's documentation for the equivalent configuration steps.

Performance Baseline

If MintPress is operational but exhibiting degraded performance, endpoint security software should be considered a potential contributing factor. To establish a valid performance baseline, temporarily disable your endpoint security product and retest. If performance improves, re-enable the product and work with your security team to tune the exclusion configuration.

Privilege model

OpsChain itself does not require root. The only privileged steps are host-level tasks needed to install and run K3s. To support enterprise environments that prohibit unrestricted (NOPASSWD:ALL) sudo, this guide splits the work into two phases:

• Host provisioning (performed once as root, or by your platform team) — creating the installation user, applying kernel/ulimit settings, creating the data directories, configuring the firewall, and installing K3s and Helm. These are described in the host provisioning section below.
• Installation and operation (performed by the unprivileged opschain user) — everything from installing OpsChain onwards uses only helm, kubectl and the OpsChain CLI against the cluster's kubeconfig, and needs no sudo at all.

A small number of ongoing operations (restarting the K3s service, reloading the firewall, inspecting containers) still require root. Rather than granting blanket sudo, grant the installation user a scoped sudoers allowlist covering only those commands (see scoped sudo for the installation user).

Managed Kubernetes

If you are deploying onto a managed Kubernetes platform (AKS, EKS, OpenShift) instead of self-hosted K3s, this entire host-provisioning phase does not apply — there is no host sudo requirement at all. You only need Helm and kubectl with permission to install the chart and the CNPG operator. Skip to the configuration guide.

Host provisioning

The commands in this section must be run as root (or by your platform team during VM provisioning). Once they are complete, all remaining steps are performed by the unprivileged installation user.

Installation user

Create a user named opschain on the Linux VM. The user does not need to be called opschain, it can be whatever name you want. For the purpose of this guide, we will assume the Linux user is opschain. If you decide to use any other username, please replace all occurrences of the opschain user in this guide to the name of your choice.

groupadd --gid 1001 opschain
useradd --uid 1001 --gid opschain --create-home opschain

UID & GID

If there are any existing users or groups with the same UID or GID, please change these to unique values.

Scoped sudo for the installation user

K3s service control and a few container-inspection tasks require root on an ongoing basis. Grant the installation user a scoped, passwordless allowlist containing only the control commands. Create the sudoers file as root:

visudo -f /etc/sudoers.d/opschain

And paste in the following (adjust binary paths to match your distribution if needed):

opschain ALL=(ALL) NOPASSWD: /usr/local/bin/k3s-killall.sh, \
  /usr/bin/systemctl start k3s, /usr/bin/systemctl stop k3s, \
  /usr/bin/systemctl restart k3s, \
  /usr/local/bin/k3s crictl *, /usr/local/bin/k3s ctr *, \
  /usr/bin/firewall-cmd

Then grant the installation user read-only access to the system journal so it can inspect K3s logs and service status without sudo. The recommended way is to add it to the systemd-journal group:

usermod -aG systemd-journal opschain

After this, journalctl -u k3s and systemctl status k3s work for the opschain user with no sudo.

If the systemd-journal group isn't honoured

Where identities are managed centrally (LDAP/SSSD) and local membership in systemd-journal is not applied, the usermod above may not take effect. Instead, add a scoped, pager-disabled journalctl entry to the sudo allowlist:

opschain ALL=(ALL) NOPASSWD: /usr/bin/journalctl --no-pager *

The command will then be available for the opschain user via sudo journalctl --no-pager <arguments>.

Every entry in the sudo allowlist is a security decision

Commands granted via NOPASSWD run as root, so each must be safe against privilege escalation before you add it.

The allowlist above covers everything OpsChain's documented operations need. If you require additional root access for some commands, add these specific commands to the allowlist at your discretion.

Installation user

After the host-provisioning steps below are complete, switch to the opschain user with su - opschain. From the installation guide onwards, most steps run as that unprivileged user — but a few later operations (for example the post-deploy CA-trust setup and backups) still require root, and are flagged where they occur.

Kernel & Ulimit settings

Update the kernel and ulimit settings to ensure the system has enough resources to run MintPress.

tip

The opschain in the lines below refer to the installation user.

Update the limits by running the following command and pasting in the following lines:

vi /etc/security/limits.d/opschain.conf

root soft nofile 131072
root hard nofile 131072
opschain soft nofile 131072
opschain hard nofile 131072

Either log out and back in with the root user, or reboot the system to apply the changes:

reboot

Update the kernel settings by running the following command and pasting in the following lines:

vi /etc/sysctl.d/99-k3s-inotify.conf

# for k3s
fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=1048576
fs.inotify.max_queued_events=32768
fs.file-max = 2097152

# for fluentd
net.core.somaxconn = 1024
net.core.netdev_max_backlog = 5000
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_wmem = 4096 12582912 16777216
net.ipv4.tcp_rmem = 4096 12582912 16777216
net.ipv4.tcp_max_syn_backlog = 8096
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 10240 65535
net.ipv4.ip_local_reserved_ports = 24224

Load the updated kernel settings:

sysctl -p /etc/sysctl.d/99-k3s-inotify.conf

Installing K3s and Helm

These steps complete the host-provisioning phase: they create the data directories, configure the firewall, and install the K3s and Helm binaries. They write to system locations (/, /etc, /usr/local/bin) and register a systemd service, so they must be run as root. After Helm is installed, switch to the unprivileged opschain user for the steps that follow.

Setup proxy

If you are using a proxy, please set the following variables on the shell before starting the next steps. This would be used only during the installation process. See Proxy Configuration later in this document for persisting proxy settings.

export http_proxy=<your_proxy_address>
export https_proxy=<your_proxy_address>

Directory setup

All K3s configuration and data is stored under /limepoint. Create the directories and the /etc/rancher symlink as root, and leave them root-owned — K3s and its embedded containerd run as root and manage this tree as root.

mkdir -p /limepoint/k3s /limepoint/rancher
ln -s /limepoint/rancher /etc/rancher

Keep the K3s data directory root-owned

The unprivileged opschain user does not need to own this directory. It reads the cluster's kubeconfig (world-readable at mode 644) via its own ~/.kube/config copy, and the few operations that write under /limepoint (trusting the registry CA, editing registries.yaml) are tasks that should be performed by the root user, not the installation user.

Firewall setup

If you are using the local Linux firewall, the following rules need to be added. First, check if the Linux firewall is running:

ps -ef|grep -i firewall

If you see a process, run the following commands, otherwise skip this step.

sudo firewall-cmd --permanent --add-port=6443/tcp # API server - # skip if no firewall is running
sudo firewall-cmd --permanent --add-port=30432/tcp # database replication - # skip if no firewall is running
sudo firewall-cmd --permanent --zone=trusted --add-source=10.42.0.0/16 # pods - # skip if no firewall is running
sudo firewall-cmd --permanent --zone=trusted --add-source=10.43.0.0/16 # services - # skip if no firewall is running
sudo firewall-cmd --reload # skip if no firewall is running

Download & install K3s

K3s version

We recommend installing K3s version v1.35.3+k3s1 or later, but this requires that your operating system uses cgroups v2, given cgroups v1 is deprecated in Kubernetes versions beyond v1.35. If your operating system does not support cgroups v2, you can proceed with installing K3s version v1.34.6+k3s1 instead.

In most Linux distributions, you can check if your system uses cgroups v2 by running the following command:

stat -fc %T /sys/fs/cgroup/

If the output contains cgroup2fs, your system uses cgroups v2. If the command does not work, refer to your operating system's documentation for how to check if your system uses cgroups v2.

Learn more about cgroups.

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="v1.35.3+k3s1" INSTALL_K3S_EXEC="--disable traefik --write-kubeconfig-mode 644 --data-dir /limepoint/k3s --secrets-encryption true" sh -

Validate K3s:

k3s --version

Tuning K3s

With K3s installed, you can configure it to your needs. See the K3s configuration documentation for more details on how to configure it.

A recommended configuration is to tune the image garbage collection settings to your needs. See the container image cleanup guide for more details.

Download Helm

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | DESIRED_VERSION="v3.20.2" bash

Validate Helm:

helm version

Download Stern

Stern is a CLI tool for Kubernetes that allows you to view logs from multiple pods at once. Head to Stern's webpage to download and install it with your preferred method.

Setup shell

Switch to the unprivileged opschain user (su - opschain). The kubeconfig that K3s generates at /etc/rancher/k3s/k3s.yaml is owned by root, so the installation user can read it but cannot write to it. Give the user its own copy so that kubectl commands that update local configuration, such as setting a default namespace, work:

mkdir -p ~/.kube
cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
chmod 600 ~/.kube/config

Then update the login shell to point at that copy and add a couple of aliases to make life simpler.

vi ~/.bash_profile

export KUBECONFIG=$HOME/.kube/config
# Replace with your preferred installed editor.
export KUBE_EDITOR=vim
alias crictl='sudo /usr/local/bin/k3s crictl --config /limepoint/k3s/agent/etc/crictl.yaml'
alias ctr='sudo /usr/local/bin/k3s ctr'

Your copy of the kubeconfig

~/.kube/config is a snapshot taken now. If you later regenerate the cluster's kubeconfig (for example after rotating the cluster CA), copy /etc/rancher/k3s/k3s.yaml again to refresh it.

Working as root instead? Point KUBECONFIG straight at /etc/rancher/k3s/k3s.yaml - root can write to it directly.

Scoped sudo

The crictl and ctr aliases call the K3s-bundled subcommands (k3s crictl / k3s ctr), which talk to the root-owned container runtime socket and work via the scoped sudoers allowlist configured during host provisioning.

And then source the file for changes to take effect:

source ~/.bash_profile

Proxy Configuration

MintPress requires outbound network access in two distinct contexts: during installation, to pull images from the MintPress container registry; and at runtime, when MintPress executes changes that reach external networks (this is optional). If the host VM does not have direct internet access, a proxy must be configured to serve both contexts.

Note: If your environment requires a more complex proxy topology (e.g. selective proxy bypass), refer to Advanced Proxy Setup before proceeding.

K3s Service Proxy Configuration

Note: These steps are only required if you need proxy during runtime (from within the pod network). Otherwise, setting the proxy variables on the command line before running helm would suffice during the install time.

Once K3s is installed, proxy settings are configured via the K3s service environment file at /etc/systemd/system/k3s.service.env. These settings govern both host-level traffic during installation and pod-level traffic at runtime.

The example below assumes the proxy is reachable at proxy-host.com:4125 and the host subnet is 10.100.22.0/24. Adjust these values to match your environment.

# /etc/systemd/system/k3s.service.env

HTTPS_PROXY="http://proxy-host:4125"
HTTP_PROXY="http://proxy-host:4125"
https_proxy="http://proxy-host:4125"
http_proxy="http://proxy-host:4125"

NO_PROXY="127.0.0.1,localhost,10.42.0.0/16,10.43.0.0/16,10.100.22.0/24,kubernetes,kubernetes.default,.svc,.cluster.local"
no_proxy="127.0.0.1,localhost,10.42.0.0/16,10.43.0.0/16,10.100.22.0/24,kubernetes,kubernetes.default,.svc,.cluster.local"

Both the uppercase and lowercase variants of each variable must be set, as different tools and runtimes observe different conventions.

The NO_PROXY / no_proxy values are required and must not be omitted. They ensure that traffic to internal cluster addresses including the pod CIDR (10.42.0.0/16), service CIDR (10.43.0.0/16), host subnet, and Kubernetes DNS names bypasses the proxy entirely.

Applying the Configuration

After saving the file, reload the systemd daemon and restart K3s:

sudo systemctl daemon-reload
sudo systemctl restart k3s

What to do next

• Proceed with the TLS configuration guide to configure TLS/HTTPS connectivity for MintPress.