Changes
This guide covers MintPress changes, creating them, managing their execution and limitations to be aware of. After reading this guide you should understand:
- how to create a change
- how a change is executed
- configuration options available to control change execution
- adding and using change metadata
Overview
A change is the application of an action from a specific commit in a Git repository, to a particular project, environment or asset. The action(s) defined in the repository's actions.rb file allow you to structure your changes in a variety of ways and will be influenced by the tools you use with MintPress. You may structure your changes using "desired state" techniques, or by applying explicit actions (e.g. upgrading a single package in response to a security vulnerability).
Creating a change
MintPress changes can be created via the MintPress CLI, GUI, or by directly POSTing to the API changes endpoint. To create a MintPress change, the following information is required:
- A MintPress project, environment or asset
- If running the change on a project or environment, a Git remote and Git revision (tag/branch/SHA). Assets are already linked to a Git remote and Git revision
- The MintPress action to execute
For more information on using the GUI to create a change, see creating a new change. To create several changes at once, see run bulk changes.
To learn how to create changes via the API, see MintPress's API reference by accessing the API documentation on the MintPress host with your browser (e.g. http://<host>/api-docs).
Change properties
To unlock the true power of MintPress, your actions should be constructed to take advantage of the MintPress properties framework. This allows the actions to dynamically source hostnames, credentials and other project/environment specific information at runtime rather than being hard-coded into the actions.
Static properties
The change's Git reference identifies the static repository properties that will be supplied to the change. As detailed in the OpsChain properties guide, repository properties can be overridden by project, environment, asset and change properties.
Dynamic properties
As each step in your change is constructed, MintPress will supply it with the latest version of the change's project, environment, asset and change database properties. This ensures any modifications made to the properties in prior change steps (or other changes) are available to the action.
Change execution
A step will be created for the change action, with additional steps created for each child action. The child execution strategy specified by each action will determine whether its child actions are executed serially or in parallel.
The number of OpsChain worker pods configured when the MintPress server is deployed provides a hard limit on the number of steps that MintPress can execute at a time. For example, with 3 worker pods, MintPress can run:
- 3 parallel steps from a single change
- 3 individual steps from 3 distinct changes
- 2 parallel steps from one change, and one step from another
Change & step lifecycle
Changes, and the steps that make them up, transition between states as they execute.
When a change is created, its state is set to initializing whilst the Git revision is resolved and validated. Once the Git revision details are validated, the change moves to the pending state. A step remains in the pending state until its prerequisites are complete. If a step fails, the on_failure: and always: arms of each branch step containing it still run, and any other steps in the same change that are still pending are set to the aborted state. When a branch step chooses an arm, the steps in its other arms are set to skipped.
When a change starts executing it enters the queued state. Changes and steps stay in the queued state while they are waiting for an OpsChain worker to start executing them (e.g. if all workers are already busy, a change will wait until a worker is available). A change or step may also wait here when the cluster has reached its runner pod concurrency cap, starting once a running pod completes and frees capacity.
Whilst a change or step is actively executing it is in the running state.
If the change/step succeeds it transitions to the success state. If the change/step fails it transitions to the error state.
If the change/step succeeds only because a failure was ignored, or because a group recovered from one, it transitions to the warning state instead of success — see ignoring an action failure and ignoring child step failure. A step whose child finished warning also finishes warning, so the warning is passed up to the change. warning is a successful outcome: the change carried on, but its log records a failure you may want to look at. The GUI shows it as completed with warnings, and the CLI treats it as a success, so --wait-for-completion and attach exit zero. A notification rule or event filter that matches success does not match it, but a change's own success notification does.
A running change can be cancelled by anyone with permission to execute its action — the same permission required to start it. If a change is cancelled by a user, all finalised steps (i.e. in the success, warning or error state) remain in their existing state, steps that are queued or running are set to cancelled, and steps that have not started (pending) are set to aborted. There is no rollback of any kind, steps that have not yet started will not start, and steps that are in progress are stopped immediately. The only steps that then run are the on_cancel: steps of a group that was running when the change was cancelled; their actions are stopped before those steps start.
A change may also remain in the pending state while it waits for other changes in the same project, environment or asset to finish. By default it only waits for changes with the same action — see change execution options.
Behaviour when a child step fails
Configuring a step's children to run sequentially or in parallel not only impacts how they are executed but also effects how MintPress processes them in case of a failure. If a child step fails:
- Sequential: MintPress terminates the change at the completion of the failed child step and any remaining steps will not run
- Parallel: MintPress allows all siblings of the failed child step to complete and then terminates the change
The change status will transition to error when MintPress terminates the change.
If the failed step is inside a group with on_failure: steps, those steps run before the change is terminated, followed by any always: steps. A group with recover: true lets the change carry on once its on_failure: steps have succeeded.
Approving a change
A change matched by a requires_approval_from rule does not start until its approvers approve it.
The change's step tree opens with an approval step as its root, and the action you asked for runs as that step's child. Approving the step releases the change and the action starts; rejecting it aborts the change, which then reports the rejected status rather than aborted. Every requirement in the matched rules must be approved before the action runs — a rule listing several users or LDAP groups is satisfied by the first of them to approve, so requiring two people means writing two rules. A rule that the person who created the change satisfies is dropped rather than applied, because nobody is asked to approve their own change, so a change whose only matching rules name its creator starts without waiting.
While a change is held at an approval step — this one, or one declared in its actions — it reports the waiting_for_approval status. The change still reports the action you requested. The activities list, the change's action and the authorisation rules that govern who can run it are unaffected by the approval step sitting above it.
While a change waits for this approval it holds no runner slot, builds no image and starts no pod, and it does not wait behind older changes to reach the approval. Those happen once it is approved. The exception is a change that reads its steps from your repository as it runs, such as a change on a project or environment, when a requires_approval_from rule that applies to it names actions other than the one it was created for. That change builds its change worker and discovers its steps first, so that every matching rule is applied, and it keeps its pod and runner slot while it waits.
Approve and reject from the change page's buttons, from the approval tab of manage activity, or via POST /api/steps/{id}/approve and POST /api/steps/{id}/reject.
A change can carry two gates: this one, configured by an administrator in the settings, and an approval step declared in your actions.rb for the action itself. The change stops at each in turn, and the two approver lists are kept separate — someone named in the setting cannot approve the declared step unless they are named there too.
Pausing and resuming a change
A running change can be paused by anyone with permission to execute its action — the same permission required to start it — using POST /api/changes/{id}/pause. Unlike cancelling, pausing does not stop anything already in progress: steps that are running or queued when the change is paused continue to completion as normal. Pausing only stops the change from admitting further work once those steps finish — no further steps are started, and the change stops waiting to run if it had not started yet. Pausing a change before it has built its worker pod prevents that build from happening at all, the same way maintenance mode does.
A paused change reports its pause_state as one of:
none— the change is not pausedpausing— the change is paused, but still has steps actively running or waitingpaused— the change is paused and has fully drained; nothing is still executing
POST /api/changes/{id}/resume lets the change continue admitting steps again, picking up from wherever it was left. Both requests take an optional reason — see the API reference for the request format.
Pausing and resuming are recorded as audit:changes:pause/audit:changes:resume events, returned as the change's paused_by attribute alongside who paused or resumed it and any reason supplied. Each is also written to the change's own log, naming who asked and why, so the log shows why the change stopped starting steps.
The activities endpoints report paused, paused_at, pause_state and paused_by for every change and workflow run, and can be filtered on them: filter[paused_eq]=true returns the activities that are pausing or paused, and filter[pause_state_eq]=paused only those that have fully drained.
A change that has already finished cannot be paused, and a change that is already paused cannot be paused again.
The same pause/resume behaviour is available for workflow runs. See pausing and resuming a workflow run.
An instance-wide equivalent exists too: putting MintPress into maintenance mode stops every change and workflow run from starting, without needing to pause each one individually.
Retrying changes
Changes that have failed or been cancelled can be retried. To rerun one failed step while the rest of a running change carries on, retry the step rather than the change. See retrying a failed step.
When retrying a change, the existing change is duplicated as a new change and started from where the existing change ended. Any successfully completed steps are not rerun - they will stay in the success or warning state with their original started and finished times. The exception is a step with a failed descendant step, such as a step marked warning because it ignored its child's failure - the step is reset and the failed step is run again. Steps that were being run by a worker when the change ended are restarted from the start as MintPress does not track step internals.
As steps are rerun from the start we suggest only retrying changes/steps that are idempotent.
A change's settings overrides are fixed when it is created and cannot be edited afterwards. When retrying a change you can supply a new set of override settings, which replaces the overrides copied from the change being retried rather than merging into them. This makes it possible to correct or adjust a setting to get the change through, instead of only being able to rerun it with the settings it already had. Supplying an empty set of overrides clears the copied overrides entirely, returning each affected setting to the value it inherits; omitting them altogether keeps the copy and prune behaviour described below.
- When MintPress retries a change, it will retry it using the code from the resolved Git SHA stored with the original change, so a retry reproduces the code the original change ran rather than picking up later commits. To run the latest code instead, ask for the Git revision to be re-resolved when retrying (
refresh_sha) — for a change on an asset this also moves the retry onto the asset's current template version, and the retry's steps follow the new revision: steps the action now declares are added, unfinished steps it no longer declares are dropped, and steps that already succeeded are kept but not rerun. When the steps it carries over ran at a different Git SHA, the retry's log names both SHAs. For a change on an asset, arefresh_sharetry is refused if the template version's actions have not yet been generated for its current commit, or if the action is no longer in the template version or has been renamed. Wait for action generation to finish, or retry withoutrefresh_shato rerun the original step tree. - The logs for the original change are not included on the new change. However, they can still be seen in the original change page, which the new change's Original change link opens.
- A retried change runs with the settings the original change resolved, so the values it ran with do not move underneath it. Settings that did not exist when the original ran are resolved from the current settings, as are the settings that determine where a change is built and deployed, the pod template settings and the
requires_approval_fromapproval requirements. A retry cannot target a deployment target that has moved, skip an approval requirement added since, or keep demanding approval from an approver who has since been removed, and it runs in pods with the current resources, volumes and security context, so a change that failed because its pods ran out of memory can be retried after the pod template settings are corrected. - Override settings copied from the original change that are no longer valid change level settings are dropped, and the dropped settings are recorded in the change's events.
- A retried change asks for its approvals again — both the change level gate and any approval step in the tree that had not already been approved. Steps that completed successfully are not rerun, so an approval step that was approved and whose children finished is not presented again.
- A retried change keeps the branches of any input step that fanned out, with the same targets, even when the Git revision is re-resolved.
- A retried change keeps the arm each branch step chose, unless
reset_on_retryistruefor that branch step (the default forchoices:) and a step that failed is inside it or in one of its siblings. A group whoseon_failure:arm ran is retried from its first step, rerunning steps that succeeded, unless it setsretry_from: :failed_step.
Skipping steps
When creating or retrying a change, you can supply a skip_steps array of glob patterns. Steps whose identifier matches a pattern are automatically skipped at runtime — full_path is matched for change steps; MintModel steps are matched by their hierarchical step name as it appears in the step tree (e.g. **/Install jdk Binaries). Patterns are matched without regard to case, so **/install jdk binaries matches the same step.
Inside an input step that fans out, a pattern written against the input step's child steps applies in every branch — see skipping steps in a fan out.
The arms of a branch step are part of the path of the steps inside them, so a pattern can skip a whole arm. For example, **/on_failure runs a change without any of its rollback steps. A pattern that ends in /**, such as **/on_failure/**, skips only the steps inside the arm: the arm itself still runs, empty, and is recorded as taken. Steps already in a group keep their paths when it gains handlers. Steps inside a new arm get the arm's name in their path. A branch whose chosen arm has been skipped completes without running anything.
The skip_steps mask is carried forward automatically on retry — previously-skipped steps remain skipped without needing to be resubmitted. The mask can be overridden at retry time.
Approval steps are always exempt from skipping — a step with requires_approval_from set will run regardless of any matching pattern. The same holds for starting_step: starting a change inside one of an approval step's choices still waits for that approval. Creating or retrying a change where skip_steps would skip the root step is rejected with a validation error.
A plain wait or input step named in skip_steps is skipped, so the change does not stop at it. Only approval steps are never skipped.
A skip_steps glob that names a stage, group or arm skips every step inside it, as a literal entry does.
A run_if condition or arm selection on the way to an approval is still evaluated, even though its group or stage was skipped, and evaluating it runs that step's own action. An approval-gated action inside a group skipped on request still runs its own code once approved.
In the GUI, Mark steps to skip in the run change dialog opens the chosen action's step tree and turns the steps you tick into patterns for you. It is available when starting a change, when scheduling one, and when retrying or rerunning one — a rerun opens with the previous run's selection already ticked.
The same skip_steps feature is available on workflow runs. See skipping steps for more information.
Starting a change partway through
When a change's step tree is known up front — a templated or asset change whose actions come from a MintModel or an actions.rb action with child steps — you can begin execution at a nominated step rather than at the top of the tree. Supply a starting_step naming the step to start from: the steps that precede it (its ancestors) are skipped, and the nominated step and its descendants run as usual.
starting_step can only be supplied for a change that has a generated step tree, and must name a step within it. It can name a step inside an arm of a branch step, which makes that arm the one taken. An approval step above the starting step is still presented for approval. When the starting step is inside one of the approval step's choices, only that choice is offered, so the arm runs only once it is approved. It is matched without regard to case. It cannot name a step inside an input step that fans out, though it can name the input step itself.
If starting_step names an actions.rb action that collides with a same-named MintModel action, the actions.rb action's own precedence still applies even though it is now the root step of the change — see name collisions between actions.rb and MintModel actions.
In the GUI, this is available from a non-root step's play button in an asset's action tree via Run <action> from here — see running an activity.
A change started partway through an action tree cannot be scheduled.
Limitations
The OpsChain properties guide highlights a number of limitations that must be taken into account when changing properties in concurrent steps.
Change execution options
By default, MintPress runs changes with different actions at the same time in a project, environment or asset, but runs changes with the same action one after the other. Only changes that target the same project, environment or asset are compared, so a change on a project does not wait for changes in its environments or assets, and the reverse. This reduces the chance that the limitations described above affect running changes.
Two settings control this, and can be set on a project, environment or asset:
allow_parallel.runs_of_same_change— set totrueif your actions can safely run alongside another run of the same action.allow_parallel.changes— set tofalseto run every change one at a time, whatever its action.
Change metadata
When creating a change, MintPress allows you to associate additional metadata with a change. This metadata can then be used:
- when searching for changes (via the web UI)
- when reporting on and searching the change history (via the API)
- from within your
actions.rbactions
Adding metadata to a change
On the GUI, you can add metadata to a change by filling in the metadata fields in the metadata tab of the change creation form.
To add metadata to a change via the CLI, see the changes CLI guide for more information.
Query changes by metadata
You can query changes by metadata via the search field in the activity page.
When querying changes via the change API, you can use MintPress's API filtering feature to limit the response to changes whose metadata matches the value we specified in the metadata, e.g.:
curl -G --user "{{username}}:{{password}}" 'http://<host>/api/changes' --data-urlencode 'filter[metadata_text_cont]=CR921'
Update the username, password, host and port to reflect your MintPress server configuration.
For more information on filtering the change list output, see the API filtering & sorting guide.
Metadata MintPress records
MintPress adds its own entries under the opschain key of a change's metadata, which is reserved for this:
| Key | Recorded when |
|---|---|
parent_change_id | The change was created by another change's action through the OpsChain API, with that change's API key. Holds the ID of the change that created it. |
asset_action | A change on an asset starts partway through an action. Holds the name of the action the starting step belongs to. |
commit_file_patterns | A scheduled change was created because commit file patterns matched. |
The activity details screen links to the parent change and to the asset action.
Using metadata in actions
Using the metadata example from above, the change metadata can also be accessed from within your actions.rb via OpsChain context. The following action would output the change request key into the change log:
action :print_change_request_key do
log.info "The change request key is: #{OpsChain.context.change.metadata.change_request}"
end
Secrets
You can securely access your secret vault from within your actions. See the secrets guide for more information.