Memory and CPU usage

MintPress records how much memory and CPU each change and each of its steps used while they ran. The dashboard shows the figures for the changes in the time range you choose, so you can see which changes need the most memory, how that varies from run to run, and whether a pod's CPU limit is slowing your changes down. A change's own page shows its figures and those of each step; see memory and CPU.

The figures are also available from the API: request a change with include=resource_usage,steps.resource_usage to add its figures and each step's to the response. A change or step that was not measured has none.

After reading this guide, you should understand:

• which changes have figures
• what each figure includes and leaves out
• how to read the percentiles the dashboard shows
• how to use the figures to size your pods

Which changes are measured

MintPress measures a change however it runs:

• in a change worker pod, whether the pod runs that change alone or is shared with other changes through the executor pool
• with each step in a pod of its own, with pod_per_change_step
• on a remote runner

The figures are recorded each time one of its steps finishes, so they are in place as soon as the change ends. When each step runs in a container of its own, the change's figures are added up from its steps; see changes that run each step in a container of its own.

A step that fails has figures like any other step. A change that stops before any of its steps finishes, for example because its pod ran out of memory, has no figures. A change that stops partway through has the figures recorded up to the last step that finished. Look at the change's log to find out why it stopped.

The change's figures

These figures cover the processes that run the change: the process that runs its actions, each step's process, and the child processes the steps start. A child process is any process a step starts: a command run with exec_command, a shell script, or a copy of the step process doing work in parallel. The figures do not include the OpsChain runner itself, so a change gets the same figures whether its pod runs it alone or shares it with other changes.

Figure	What it means
Memory peak	The most memory the change's processes held at the same time. If steps run in parallel, their memory is added together.
CPU time	The total time the change's processes spent running on a CPU. Two processes that each run on a CPU for 3 seconds add up to 6 seconds of CPU time, however long the change took.
User / system CPU	The CPU time split in two. User time is spent running the processes' own code. System time is spent by the operating system working for them, for example reading files or using the network.

MintPress checks the change's processes once a second, so:

• a spike in memory that lasts less than a second can be missed, and the memory peak can be lower than the real peak
• a process that ends between two checks still counts in full once the process that started it has waited for it to finish, as exec_command does. A process that nothing waits for, such as one left running in the background, can lose up to its last second, so the CPU time can then be a little lower than the real total

The memory peak counts the memory that holds your processes' own data. It does not count the operating system's cache of the files they read and write, which the operating system frees when memory runs short. A process started by copying another process shares that process's memory until either of them changes it, and the shared memory counts once.

The step's figures

Each step that runs an action has its own figures:

Figure	What it means
Memory peak	The most memory the step process and its child processes held at the same time. Memory the step shares with the process that started it counts once.
Step process alone	How much the step's own process grew while it ran. It leaves out the commands and other child processes the step started, so it is never more than the memory peak.
CPU time	The CPU time of the step process, and of the child processes it waited for to finish. CPU time is the time spent running on a CPU, so it can be longer than the step took if it ran child processes in parallel, or much shorter if the step spent most of its time waiting.

The step process's own memory is measured exactly. The memory of its child processes is checked once a second, so a child process that lasts less than a second may not be counted. A step that only groups other steps records just the small cost of starting them.

MintModel steps

A step that runs an action from a MintModel has the same figures, for the process that runs mintpress_ctl.rb and the commands it runs. MintModel steps always run in the change worker pod, never in a pod of their own.

By default that process is a copy of the change worker's own process, which has already loaded MintPress, so the memory it shares counts once and its step process alone counts only what the step added. When runner.use_fork_for_mintpress_ctl_rb is false, the process starts from scratch and its figures also include starting Ruby and loading MintPress, so the same step reads higher.

The pod's figures

When a change runs in a pod of its own, MintPress also records the pod's own figures. They are measured by the operating system for the whole pod, so they include the OpsChain runner, which uses about 100 MiB of memory and a few seconds of CPU time to start. Changes that run in a pod shared through the executor pool have no pod figures, because those figures would cover every change in the pod. Changes that run on a remote runner have no pod figures either, because there the operating system's figures would cover everything the remote runner runs.

Figure	What it means
Memory peak	The most memory the pod used, leaving out the file cache. This is the figure to compare with the pod's memory limit. It is checked once a second.
Memory peak with file cache	The most memory the pod used, including the operating system's cache of the files it read and wrote. The operating system frees the cache when memory runs short, so this figure can be close to the pod's memory limit without putting the change at risk.
CPU time	The total CPU time of every process in the pod, from the moment the pod started.
CPU limit	Whether the pod's CPU limit slowed it down, and for how long in total. A pod that tries to use more CPU than its limit allows is made to wait. A pod with no CPU limit is never slowed down.

tip

The time a pod was slowed down is added up across its processes, like CPU time. If two processes are made to wait for the same second, that counts as two seconds, so a change that runs several processes at once can show more time slowed down than the change took to run.

Changes that run each step in a container of its own

A change runs each step in a container of its own when pod_per_change_step is true, whether the steps run in your cluster or on a remote runner. Nothing watches such a change as a whole, so its figures work differently.

In a pod per step, each step also has its container's figures. They are measured by the operating system for the whole container, so they count every process the step started, including processes nothing waited for, and they also include the OpsChain runner and the log forwarder that share the container:

Figure	What it means
Container memory peak	The most memory the container used, including the operating system's cache of the files it read and wrote. Because nothing checks the step's child processes once a second, this is also the step's memory peak.
Container CPU time	The total CPU time of every process in the container.
CPU limit	How long the container's CPU limit slowed it down in total.

The same step therefore shows a higher memory peak in a pod of its own than in a change pod: about 100 MiB more for the OpsChain runner, plus the file cache it used.

A step on a remote runner has no container figures, for the same reason a change on a remote runner has no pod figures. Nothing measures the commands it runs either, so its memory peak is not measured. Its CPU time and the step process alone still are.

The change's figures are added up from its steps:

• CPU time is the sum of the CPU time of its steps, each with the child processes it waited for.
• The step containers' CPU time and CPU limit add up the containers of all its steps. A change on a remote runner has neither. The dashboard's pod figures leave these changes out, since a sum over many containers does not compare with one change pod.
• The change has no memory peak. Its steps run in separate containers, some of them at the same time, and MintPress does not add their memory together. The dashboard's memory panels leave these changes out. Use the steps' container memory peaks to size the step pods.

A step whose container ran out of memory has no figures, so the change's CPU time leaves it out. Look at the step's log to find out why it stopped.

The container figures need the node to use cgroup v2, and the container memory peak also needs Linux 5.19 or later. Where either is missing, those figures are not measured, and the other figures still are.

Reading the percentiles

The dashboard shows each figure's spread over the changes in the time range:

Label	Meaning
p50	Half of the changes used less than this, and half used more.
p95	95 out of 100 changes used less than this.
p99	99 out of 100 changes used less than this.
max	The most any change used.
avg	The average over the changes.

The percentiles are more useful than the largest value when you size a pod, because one unusual run can make the largest value much higher than what your changes normally need.

Sizing your pods

Use the pod's figures to set the change worker's pod_templates.<pod type>.resources:

• Set the memory limit comfortably above the p99 of the pod's memory peak. A pod that goes over its memory limit is stopped, and its change fails.
• If many pods with a CPU limit are slowed down, the limit is holding your changes back. Raise the limit, or accept the longer run times.

For changes that share a pod through the executor pool, add up the memory peaks of the changes you expect to run in the pod at the same time, and add about 100 MiB for the OpsChain runner.

For changes that run each step in a pod of its own, set the step_runner pod's memory limit comfortably above the largest container memory peak of their steps.