Events
This guide provides an overview of the event tracking system in MintPress.
After following this guide you should know:
- how events are handled in MintPress
- which events are created by MintPress automatically
- how to query events via the API
- how to create custom events using the API
All the examples in this guide assume the OpsChain API server is running on your local machine. Replace <host> with your MintPress server name if connecting to a remote MintPress server.
Overview
Any API request that modifies data in MintPress will be tracked for auditing and reporting purposes, this includes creating and updating data, interacting with activities and more. In future versions, MintPress will track more events, please let us know if there are particular events you would like tracked.
MintPress does not track API requests to the api/events API itself.
The data provided within the attributes section of the event API response varies depending on the type of event, but it contains more detailed information about the event context.
All automatically created API events start with the api: prefix and are then followed by the API controller, and then the API method. When a failure occurs, MintPress will create an event with the error: prefix.
System created events
Events created internally by MintPress can be identified by the system property. If system is true then the event was created by MintPress, if it is false then the event was created by a user using the api/events endpoint - the username field identifies the user that initiated the request.
Further down we present a list of the event types created by MintPress.
Viewing events
You can view events in the MintPress web UI by navigating to the audit history page. Click an event there to open its details page, which shows its message, progress logs and data.
On the API, the MintPress api/events endpoint can be queried to see events in the MintPress system.
curl -u "{{username}}:{{password}}" http://<host>/api/events
The response is a JSON:API payload containing a list of the most recent events. The response will include the relevant events from oldest to newest in the response data array, i.e. data[0] will be the oldest event in the result set.
Example change start event
Below is an example of a change start event returned by the api/events endpoint.
{
"id": "43404b06-e265-4d4e-a387-4fc83320a778",
"type": "event",
"attributes": {
"username": "opschain",
"system": true,
"type": "api:changes:start",
"created_at": "2021-01-01T01:00:00.000000Z"
},
"relationships": {
"source": {
"links": {
"source": "/api/projects/hello_world/changes/d57edfd7-7536-40fc-9c5b-9492faa0a6fd"
}
}
},
"links": {
"self": "/api/events/43404b06-e265-4d4e-a387-4fc83320a778"
}
}
Filtering events on the API
The query to the api/events endpoint can be filtered by providing the relevant query parameters.
For example, the following query will return up to 100 events that were created after 2021-01-01.
By default, the response is limited to only 10 events, and there is a hard limit of 1000 events. The response status code will be 206 Partial Content when the response has been truncated by the limit.
curl --globoff --user "{{username}}:{{password}}" 'http://<host>/api/events?filter[created_at_gt]=2021-01-01T01:00:00.000000Z&limit=100'
The --globoff argument is required when using the filtering queries using curl.
Filtering examples with the API
The API filtering and sorting guide includes a variety of examples that highlight MintPress's filtering feature and how it can be used to find specific events.
More complex examples using the event API endpoint
Below are some more complex examples of querying the api/events API.
The examples require the jq and curl utilities, and have been tested with Zsh and Bash 4.
Waiting for an event to occur
The following is an example of watching the events API waiting for a change start event to occur.
user='{{username}}:{{password}}'
since="$(date --iso-8601=ns)"
event='api:changes:start'
while true; do
response="$(curl -s -G --user "${user}" http://<host>/api/events --data-urlencode "filter[created_at_gt]=${since}" --data-urlencode "filter[type_eq]=${event}")"
if jq -e '.data | length > 0' <<<"${response}" >/dev/null; then
echo "${response}"
break
fi
sleep 1
done
Don't forget to add --data-urlencode "filter[system_true]=yes" if you wish to wait for a system event.
Paginating through events
The following is an example of paginating backwards through the events API. It will output the newest event to the oldest.
user='{{username}}:{{password}}'
while true; do
response="$(curl -s -G --user "${user}" http://<host>/api/events --data-urlencode "filter[created_at_lt]=${before}")"
before="$(jq -r '.data[0].attributes.created_at // empty' <<<"${response}")"
if [[ -z "${before}" ]]; then
break
fi
jq '.data | reverse[]' <<<"${response}"
done
Creating custom events
Events can be created in the MintPress events framework by sending a POST request to the api/events endpoint.
The request needs to be a valid JSON:API request. E.g.
curl --fail --user {{username}}:{{password}} http://<host>/api/events -H 'content-type: application/vnd.api+json' -d '{ "data": { "type": "Event", "attributes": { "type": "custom", "some": "value", "nesting": { "also": "works" } } } }'
curl --fail --user {{username}}:{{password}} http://<host>/api/events -H 'content-type: application/vnd.api+json' -d @event-file.json
MintPress responds with a 201 status code and no response body when the event is created successfully.
Every attribute other than type is stored as the event's data, except an attribute that links the event, such as project_path or change_id. To store data whose keys could be mistaken for a link, send it as a data object instead — its contents are stored exactly as supplied and never used as a link:
curl --fail --user {{username}}:{{password}} http://<host>/api/events -H 'content-type: application/vnd.api+json' -d '{ "data": { "type": "Event", "attributes": { "type": "custom", "project_path": "/projects/bank", "data": { "change_id": "CHG-1234" } } } }'
The api:, audit:, error:, info:, opschain:, warn: and warning: prefixes are reserved for system event types. Give custom event types a different prefix: a trigger on a type with a reserved prefix matches only events MintPress raises, so it never fires for a custom event.
Action code can raise an event without making this request itself — see raising events.
Linking events
Events can be linked to data within MintPress. Below is an example of linking a project with the path /projects/bank (this is the same as a project with the code bank) to a custom event.
curl --fail --user {{username}}:{{password}} http://<host>/api/events -H 'content-type: application/vnd.api+json' -d '{ "data": { "type": "Event", "attributes": { "type": "linked:to:project:example", "project_path": "/projects/bank" } } }'
To keep the link apart from the event's data, send it as a source object naming exactly one resource instead, for example "source": { "node_path": "/projects/bank" }. A source that names more than one resource, or uses a key that cannot link an event, is rejected, and the error lists every key that can be used.
Events can be linked to many kinds of resource, including:
- Projects via a path or an ID, e.g.
"project_path": "/projects/bank", or"node_path": "/projects/bank", or"project_id": "ff1bf781-4fe0-4b14-b0d2-20ef8cb1be80" - Environments via a path or an ID, e.g.
"node_path": "/projects/bank/environments/dev", or"node_id": "969a2b4c-a700-40d2-a25c-1f4f68cf6d54" - Assets via a path or an ID, e.g.
"node_path": "/projects/bank/environments/dev/assets/obp", or"node_id": "2f988308-325d-4a41-bdab-4cf0b8c3103a" - In addition, the following models can be linked via an ID, (e.g.
"{{model_type_id}}": "ff1bf781-4fe0-4b14-b0d2-20ef8cb1be80"):- Scheduled changes (via
scheduled_change_id) - Changes (via
change_id) - Steps (via
step_id) - Workflow steps (via
workflow_change_step_id,workflow_noop_step_idorworkflow_wait_step_id) - Workflow runs (via
workflow_run_id) - Bookmarks (via
bookmark_id) - Templates (via
template_id) - Template versions (via
template_version_id) - Git remotes (via
git_remote_id) - Properties (via
properties_id) - Properties versions (via
properties_version_id) - Settings (via
settings_id) - Settings versions (via
settings_version_id)
- Scheduled changes (via
System event types
Every system event type is made up of a level, a subject and an action, separated by colons. The level tells you how to react to the event:
api:— a normal operation was carried out, usually via the API.audit:— a user made an approval decision. These are kept as a record of who approved or rejected what.info:— MintPress did something in the background worth knowing about.warn:andwarning:— something was not right, but MintPress carried on.error:— an operation failed. The reason is in the event'serrordata key, and abacktraceis often included.
Some events are throttled, meaning MintPress records at most one of them in the stated period no matter how often the underlying condition occurs. This keeps a repeating failure from filling the event history.
Changes
api:changes:create— a change was created.api:changes:start— a change was started.api:changes:success— every step in the change completed successfully.api:changes:warning— the change completed successfully, but only because a failure was ignored or a group withrecover: truerecovered from one. Its description reads completed with warnings. A filter onapi:changes:successdoes not match it.api:changes:error— the change stopped because one of its steps failed.api:changes:cancel— the change was cancelled.api:changes:abort— the change was aborted.audit:changes:pause— a user paused a running change. Themessagedata key holds the reason given, if any.audit:changes:resume— a user resumed a paused change. Themessagedata key holds the reason given, if any.api:changes:retry_settings_pruned— an incomplete change was retried and some of its settings overrides were dropped because they are no longer valid for a change. Thedropped_settingsdata key lists them.api:changes:destroy— a change was deleted.api:change_listener:error— the listener that picks up change cancellations failed.error:change_worker:delete— the change's runner pod could not be removed after the change finished. The pod may need to be removed by hand.error:change_worker:oom_killed— a step failed because the pod running it, a change's own runner pod or a pooled executor pool pod, ran out of memory. One is recorded for each step the pod took down, against that step. The data keys includepod_name,pod_uid,memory_limit,cluster_name,step_idandchange_id. When MintPress has a memory sample from the pod, taken every few seconds while the change runs,last_memory_sampleholds the last one: when it was taken (sampled_at), the pod's memory use and limit in bytes (used_bytesandlimit_bytes), and the name, title and private memory in bytes (private_bytes) of each of the change's processes and the OpsChain runner. Each step's process is namedstep-followed by the last eight characters of the step's ID. The same figures are written to the failed step's log.warning:change_worker:memory_pressure— the pod running a change was short of memory: a step of the change waited to start because the pod was using too much of its memory limit (reasonisstep_start_held), or processes of the change were stopped to keep the pod under its limit (reasonissteps_stopped). See when a pooled pod runs low on memory. It is recorded against the change's root step and throttled to one every 10 minutes for each change and reason. When a memory sample is available,memory_sampleholds it, in the same form aslast_memory_sampleabove.info:change_worker:memory_usage— the memory a running change was using, recorded at the interval set bydiagnostics.memory_event_interval, which records none by default. It is recorded against the change's root step, andmemory_sampleholds the sample, in the same form aslast_memory_sampleabove. To see a running change's latest sample without waiting for an event, requestGET /api/changes/:id/memory_snapshot. It is refreshed every few seconds, and returnsdata: nullwhen no sample is available: before the first sample is taken, and shortly after the change finishes.
Steps
api:steps:start— a step was started.api:steps:approve— a step was approved.api:steps:reject— a step was rejected.api:steps:continue— a waiting step was continued.api:steps:success— the step's action completed successfully.api:steps:warning— the step completed successfully, but only because a failure was ignored or a group withrecover: truerecovered from one.api:steps:error— a step transition could not be applied.api:steps:cancel— the step was cancelled.api:steps:abort— the step was aborted.info:steps:arm_taken— a branch step took an arm. Records the step, the arm, why it was taken and, for a choice, who chose it.warning:steps:arm_defaulted— a branch step's block returned a value that is not one of its arms, so itsdefault:arm ran. Records the value returned.error:step:action— the step's action raised an error.error:step:processing— MintPress could not process the step at all, so it was failed.error:step:aasm_failure— a step status transition raised an error.error:step:resource_usage_not_recorded— the step's memory and CPU figures could not be recorded. The step carried on regardless. Throttled to one event per hour for each step.error:steps:transient_error— a temporary database error stopped a step transition being applied. MintPress retries, and moves the step tosystem_errorif the retries are exhausted. Throttled to one event per hour.api:assign_system_error_job:error— the job that moves a step tosystem_errorfailed.
Step approvals
audit:steps:approve— a user approved a step that was waiting for approval.audit:steps:reject— a user rejected a step that was waiting for approval, aborting it.audit:steps:continue— a user continued a step that was waiting. For a step with choices, records the choice made, asaudit:steps:approvedoes for an approval step.audit:steps:approve:denied— a user tried to approve a step they are not an approver for.audit:steps:reject:denied— a user tried to reject a step they are not an approver for.audit:steps:continue:denied— a user tried to continue a step that was waiting for approval. Such a step can only be released by approving it.audit:workflow_steps:approve— a user approved a workflow step that was waiting for approval.audit:workflow_steps:reject— a user rejected a workflow step that was waiting for approval, aborting it.audit:workflow_steps:continue— a user continued a workflow step that was waiting.audit:workflow_steps:approve:denied— a user tried to approve a workflow step they are not an approver for.audit:workflow_steps:reject:denied— a user tried to reject a workflow step they are not an approver for.audit:workflow_steps:continue:denied— a user tried to continue a workflow step that was waiting for approval. Such a step can only be released by approving it.warn:approval_identity_unresolved— approvers named in arequires_approval_fromsetting, or in an approval step declared in anactions.rb, do not exist as MintPress users or LDAP groups. Recorded when the actions are loaded, and by an LDAP directory refresh that finds a step already waiting on approvers that no longer resolve. Steps waiting on them cannot be approved until the approvers are corrected. Throttled to one event per hour.warn:approval_identity_validation_skipped— approver names were accepted without being checked because the LDAP directory could not be reached. Throttled to one event per hour.
Workflows
api:workflows:create— a workflow was created.api:workflows:update— a workflow was updated.api:workflow_runs:create— a workflow run was created.api:workflow_runs:start— a workflow run was started.api:workflow_runs:success— every step in the workflow run completed successfully.api:workflow_runs:warning— the workflow run completed successfully, but only because a failure was ignored or a stage withrecover: truerecovered from one. Its description reads completed with warnings. A filter onapi:workflow_runs:successdoes not match it.api:workflow_runs:error— the workflow run stopped because one of its steps failed.api:workflow_runs:cancel— the workflow run was cancelled.api:workflow_runs:abort— the workflow run was aborted.audit:workflow_runs:pause— a user paused a running workflow run. Themessagedata key holds the reason given, if any.audit:workflow_runs:resume— a user resumed a paused workflow run. Themessagedata key holds the reason given, if any.api:workflow_runs:destroy— a workflow run was deleted.api:workflow_steps:start— a workflow step was started.api:workflow_steps:approve— a workflow step was approved.api:workflow_steps:reject— a workflow step was rejected.api:workflow_steps:continue— a waiting workflow step was continued.api:workflow_steps:success— the workflow step completed successfully.api:workflow_steps:warning— the workflow step completed successfully, but only because a failure was ignored or a stage withrecover: truerecovered from one.api:workflow_steps:error— the workflow step failed.api:workflow_steps:cancel— the workflow step was cancelled.api:workflow_steps:abort— the workflow step was aborted.info:workflow_steps:arm_taken— a branching stage, or a wait step with choices, took an arm. Records the step by its path in the run (the stages and arms above it and its own name, joined with/), the arm, why it was taken and, for a choice, who chose it.warning:workflow_steps:arm_defaulted— no arm of a branching stage matched, so itsdefaultarm ran.error:workflow_step:aasm_failure— a workflow step status transition raised an error.error:workflow_step:run_failure— running the workflow step raised an error.
Scheduled changes and workflows
api:scheduled_changes:create— a scheduled change was created.api:scheduled_changes:destroy— a scheduled change was deleted.api:scheduled_workflows:create— a scheduled workflow was created.api:scheduled_workflows:destroy— a scheduled workflow was deleted.api:scheduled_changes:change_creation:create— a scheduled change came due and started creating its change.api:scheduled_changes:change_creation:success— a scheduled change created its change successfully.api:scheduled_workflows:workflow_run_creation:create— a scheduled workflow came due and started creating its workflow run.api:scheduled_workflows:workflow_run_creation:success— a scheduled workflow created its workflow run successfully.error:scheduled_changes:change_creation— a scheduled change could not create its change.error:scheduled_changes:git_sha— the Git SHA for a scheduled change could not be resolved, so no change was created.error:scheduled_workflows:workflow_run_creation— a scheduled workflow could not create its workflow run.warning:scheduled_activity:skipped— a scheduled change or workflow was skipped because one it created earlier was still running and it does not allow parallel execution, because it allows parallel execution andscheduled_activity_max_waiting_runsof its runs are already waiting to start, or because the user who created it is disabled in the LDAP directory. The first two are throttled to one event per hour; a skip for a disabled user is recorded every time.warning:scheduled_changes:commit_file_patterns— a scheduled change with commit file patterns either could not evaluate them, in which case it created its change anyway, or reached the end of its schedule without them ever being satisfied. Throttled to one event per hour.
Projects, environments and assets
api:projects:create— a project was created.api:projects:update— a project was updated.api:projects:destroy— a project was deleted.api:nodes:create— an environment, asset or agent was created.api:nodes:update— an environment, asset or agent was updated.api:nodes:destroy— an environment, asset or agent was deleted.api:bookmarks:create— a bookmark was created.api:bookmarks:update— a bookmark was updated.api:bookmarks:destroy— a bookmark was deleted.
Properties and settings
api:properties:update— properties were updated.api:settings:update— settings were updated. Any of MintPress's own service credentials in the request, such asldap.password, are recorded as{{hidden}}.info:maintenance_mode:enabled— themaintenance_modeglobal setting was turned on.info:maintenance_mode:disabled— themaintenance_modeglobal setting was turned off.warn:settings:override— a setting was overridden by anOPSCHAIN_OVERRIDE_environment variable supplied by the deployment, so it takes precedence over the value stored in MintPress.
Templates and template versions
api:templates:create— a template was created.api:templates:update— a template was updated.api:template_versions:update— a template version was updated.api:action_refresh:warn— an asset's actions could not be refreshed after its template version changed. The asset keeps its previous actions, and theerrordata key holds the reason.info:template_version:fetch_initialize— a template version was created and queued for its first fetch.info:template_version:fetch— the fetch of a template version's Git repository started. Itsprogressdata key is updated as the fetch produces output, so this event doubles as the live fetch log.info:template_version:fetch_complete— the fetch finished. Thesuccessdata key records whether it worked.info:template_version:refresh_cancelled— an in-progress actions refresh was cancelled.info:template_version:refresh_superseded— a Git revision refresh finished after a newer refresh had replaced it, or after it was cancelled, so the revision it resolved was discarded. Nothing failed - the refresh that replaced it decides the version's revision.warn:template_version:float_refresh_failed— a template version that follows its Git revision could not be refreshed. The version keeps serving the commit it already had, and the reason is reported against the version until it successfully follows its revision again.error:template_version:fetch— the template version's Git repository could not be fetched. Thefetch_outputdata key holds the Git output.error:template_version:commit_verification— the template version's Git remote and revision could not be verified.error:template_version:broken— the template version was marked broken and cannot be used until it is corrected.error:template_version:build_agent_image— the agent image for the template version could not be built.error:template_version:refresh_actions— the template version's actions could not be refreshed.error:template_version:refresh_cancelled— cancelling an in-progress actions refresh failed.
MintModel
api:generate_actions_request:create— a request to generate an asset's actions was created. Thetriggerdata key records what prompted it.error:generate_actions_request:generate— an actions generation request failed.error:mintmodel:generate— the MintModel API could not generate the asset's MintModel. The converged properties sent to the API are included to help diagnose the failure.error:mintmodel:derive_actions— the asset's actions could not be derived from its MintModel.error:mintmodel:actions_cache_not_persisted— the generated actions could not be cached, so they will be regenerated every time they are needed until this is resolved. Throttled to one event per hour.error:mint_model_concretise_task:concretise— a MintModel concretisation task failed.warn:mintmodel:actions_unavailable— an asset's MintModel actions could not be determined while its actions were being generated, so anactions.rbreading them withmintmodel_actionssaw an empty catalog. The generation itself continues. Throttled to one event per hour.
Agents
api:agent:start— an agent was asked to start and the task to create its container was created.api:agent:started— the agent's container is running.api:agent:stop— an agent was asked to stop and the task to remove its container was created.api:agent:stopped— the agent's container was removed.api:agent:failed— the site running the agent found that its container had stopped for good or was gone, so it stopped the agent and is removing what is left of the container. Thefailure_reasondata key records why.error:agent:start— the agent's container could not be created.error:agent:stop— the agent's container could not be removed.error:agent:verify— the site running the agent could not check whether its container had failed. Throttled to one event per hour.warn:agent:image_missing— the agent could not be started because its image was not in the registry of the site handling the start. A build was started and the agent will automatically start in that site once the build finishes.error:build_agent_image_task:build— building the agent's image failed.
Image builds
info:image_build:queued— an image build was held back because the configured number of concurrent image builds was already running. Thequeued_behinddata key lists the builds ahead of it.info:image_build:started— an image build started.info:image_build:completed— an image build finished successfully.info:image_build:failed— an image build failed.warn:image_build:content_key_error— the key used to recognise an equivalent existing image could not be calculated, so the image was built again instead of being reused.warn:image_build:required_check_error— MintPress could not work out whether an image build was needed, so it built one to be safe.
Runner base image
info:runner_base_image_warm:completed— the runner base image was pulled into the build service cache ahead of time, so the next runner image build does not have to wait for it. Theduration_secondsdata key records how long it took.warn:runner_base_image_warm:failed— the runner base image could not be pulled into the build service cache. Runner image builds still work, but the first one pays the download cost. Throttled to one event per hour.
Image registry
info:registry_reconcile:summary— the image registry was reconciled against the images MintPress still needs, and unused images were removed.error:registry_reconcile:failure— reconciling the image registry failed.
Git remotes
api:git_remotes:create— a Git remote was created.api:git_remotes:update— a Git remote was updated.api:git_remotes:destroy— a Git remote was deleted.api:git_remote_pushes:create— agit_cloneresource pushed commits to the Git remote through MintPress. The data records the branch, the pushed and expected commit SHAs, the project code, the Git remote's name and the size of the pushed bundle. Rejected pushes are not recorded.error:git_remote:create— the Git remote could not be reached when it was created. It is still created, but changes using it will fail until it is reachable.error:git_remote:update— the Git remote could not be reached after it was updated.error:git_remote:fetch— fetching from the Git remote failed.warn:git_remote:mountable_fetch_failed— MintPress could not refresh its copy of the Git remote before mirroring it into a runner pod for thegit_cloneresource. The pod is still started, so the mirror the step checks out from may be out of date, or missing altogether if the remote has never been fetched successfully. Throttled to one event per hour.warn:git_remote:push_refresh_failed— MintPress pushed commits from agit_cloneresource to the Git remote, but could not refresh its copy of the remote afterwards. The push still succeeded; a latergit_clonemay not see the pushed commits until the next refresh. Throttled to one event per hour.warn:git_remote:periodic_fetch:failed— the scheduled background fetch that keeps MintPress's copy of the Git remote up to date failed, or did not finish because the worker running it stopped. The remote is retried progressively less often while it keeps failing, as described ingit_remote.periodic_fetch_interval. Throttled to one event per hour.info:git_remote:periodic_fetch:recovered— the scheduled background fetch of a Git remote succeeded again after previously failing. The message records how many consecutive failures preceded it.info:git_remote:periodic_fetch:summary— a periodic summary of background Git remote fetching across all active Git remotes: how many there are, how many have been fetched at least once, how many are currently failing, when the least recently fetched one was last fetched, and the fastest, median, 95th percentile and slowest fetch durations. Recorded once a day by default — seegit_remote.periodic_fetch_summary.enabled.
Secrets and the secret vault
api:secrets:encrypt— a value was encrypted and stored in the secret vault.api:secrets:resolve— a stored secret was decrypted and returned.error:secret_vault:ldap_auth— setting up LDAP sign-in to the secret vault, or applying superuser and block assignments to it, did not complete. Themessagedata key says why. Throttled to one event per hour for the same message.api:external_vaults:create,api:external_vaults:updateandapi:external_vaults:destroy— an external secret vault was registered, edited or deleted. Its address, credentials and client options are hidden.api:vault_runner_credentials:create,api:vault_runner_credentials:updateandapi:vault_runner_credentials:destroy— a runner credential was created, changed or deleted.api:user_system_credentials:create,api:user_system_credentials:updateandapi:user_system_credentials:destroy— a user registered, replaced or deleted a secret vault credential. The token is hidden.api:secret_vault_policies:create,api:secret_vault_policies:updateandapi:secret_vault_policies:destroy— a vault policy was created, changed or deleted.error:secret_vault:user_session— signing in could not give an LDAP user access to the bundled secret vault. The MintPress sign-in still succeeds.error:secret_vault:api_key— an API key's login to the bundled secret vault could not be obtained, renewed or revoked.info:secret_vault:api_key:revoked— an API key lost its own login to the bundled secret vault.warn:secret_vault:cache_invalidation_failed— a secret was stored or a vault policy changed, but MintPress could not clear its cached copies of the affected values. Themountandnode_pathdata keys say which.error:secret_vault:internal_accounts— MintPress could not set up its own accounts in the bundled secret vault, or mount a store a project, environment or asset uses.
Authorisation
api:authorisation_policies:create— an authorisation policy was created.api:authorisation_policies:update— an authorisation policy was updated.api:authorisation_policies:destroy— an authorisation policy was deleted.api:authorisation_rules:create— an authorisation rule was created.api:authorisation_rules:update— an authorisation rule was updated.api:authorisation_rules:destroy— an authorisation rule was deleted.api:policy_assignments:create— a policy was assigned to a user or group.api:policy_assignments:destroy— a policy assignment was removed.api:policy_rules:create— a rule was added to a policy.api:policy_rules:destroy— a rule was removed from a policy.
Users and LDAP
api:ldap_refresh:user_start— a refresh of the cached LDAP user entries started. The search base and filter used are included.api:ldap_refresh:group_start— a refresh of the cached LDAP group entries started.api:ldap_refresh:success— the LDAP refresh completed, and the entry counts, including how many LDAP users the refresh disabled and how many it re-enabled, are recorded in the event data.api:ldap_refresh:warn— the LDAP refresh completed but something was not right. The cached entries were left as they were, unlessmass_disable_guardisoverridden_by_age: a drop in users held back by the mass-disable guard persisted long enough to be applied.api:ldap_refresh:error— the LDAP refresh failed, or completed with problems serious enough that stale cached entries were removed. Withmass_disable_guardset totripped, the refresh would have disabled more than half of the enabled LDAP users, so nothing was applied.
Events and notifications
api:event_filter:error— an event filter could not be evaluated, so its subscribers were not notified.api:event_subscriber:error— an event subscriber could not be notified. Throttled to one event per hour.api:event_processor:error— an event could not be processed. Throttled to one event per hour.api:notification:error— a notification could not be delivered. Throttled to one event per hour.warn:notification_channel:disabled— a subscription would have been delivered through a notification channel that is disabled, so it was not delivered. Throttled to one event per hour per channel.info:email:sent— an email was sent, recording the sender, recipients and subject, and the message id the mail server returned. Only recorded whilelog_email_eventsis enabled.error:email:send_failed— an email could not be sent, recording the sender, recipients and subject, and the reason in theerrordata key. Only recorded whilelog_email_eventsis enabled.
Data cleaning
info:data_cleanup:activities— a data cleaning rule removed finished changes and workflow runs. Theremoved_countandfiltersdata keys record how many were removed and the rule that removed them.info:data_cleanup:events— a data cleaning rule removed old events.info:data_cleanup:jobs— a data cleaning rule removed old job history.info:data_cleanup:agent_images— a data cleaning rule removed unused agent images.info:data_cleanup:node_background_tasks— a data cleaning rule removed superseded node background tasks and their logs.info:data_cleanup:statistics— a data cleaning rule refreshed the database's query planner statistics. Theanalyzed_tablesdata key lists each table analysed with its estimated row count before and after, and is empty when nothing was stale. Theskipped_reasondata key is set instead when another site was already refreshing them.info:data_cleanup:artifacts— a data cleaning rule removed old artifacts.error:data_cleanup:activities— removing finished changes and workflow runs failed.error:data_cleanup:events— removing old events failed.error:data_cleanup:jobs— removing old job history failed.error:data_cleanup:agent_images— removing unused agent images failed.error:data_cleanup:node_background_tasks— removing superseded node background tasks failed.error:data_cleanup:statistics— refreshing the query planner statistics failed.error:data_cleanup:artifacts— removing old artifacts failed.
Executor pool
error:executor_pool:step_hung— a step of a pooled change was failed because its process had been missing from its pod for longer thanexecutor_pool.hung_step_timeout. It is recorded against the step, and themessage,change_idandstep_iddata keys say why and which step. See when a pooled step's process has gone.warning:executor_pool:outdated_pods_busy— after an upgrade, some pooled change pods on the previous runner image were still running changes, so they were not deleted. Thepodsdata key lists each pod'spod_nameand thechange_idsof the changes using it. Each pod is torn down once it has been idle forexecutor_pool.idle_timeout. See after an upgrade.
Resource slots
warn:resource_slot_pool:full— every slot in a resource slot pool was in use, so work that needed one had to wait. Thepoolandconfigured_limitdata keys identify which pool and its limit. Throttled per cluster and pool.api:resource_slot_freed_listener:error— the listener that wakes work waiting on a freed resource slot failed. Waiting work still runs, but starts later than it could have.
API autoscaler
info:api_autoscaler:grow— an API worker was added because the request backlog stayed high.info:api_autoscaler:shrink— an API worker was removed because the request backlog had cleared.warn:api_autoscaler:grow_blocked_connections— the autoscaler wanted to add an API worker but did not, because too few database connections were spare. Throttled to one event every five minutes.warn:api_autoscaler:grow_blocked_memory— the autoscaler wanted to add an API worker but did not, because the pod had too little spare memory. Throttled to one event every five minutes.warn:api_autoscaler:worker_killed_memory— an API worker was stopped and replaced because it was using at least twice as much memory as the pod's other workers while the pod was close to its memory limit. The requests that worker was serving failed; the rest of the pod kept running. Theworker_rss,baseline_rss,container_anonandcontainer_limitdata keys give, in bytes, the worker's memory, what it was compared with, and the pod's memory use and limit. Indry_runmode it records what would have happened without stopping the worker. See API worker memory protection.error:api_autoscaler:tick_failed— an autoscaler evaluation failed. Throttled to one event every five minutes.
Sites and database clusters
info:database_cluster:registered— a site's database cluster reported itself for the first time and now appears on the database screen.info:database_cluster:role_changed— a site's database cluster changed between primary and replica, so a failover is visible in the audit history.info:database_cluster:removed— a site was removed from the database screen, along with any cordon on it, by a reset after a recovery.info:site:cordoned— a site was cordoned, so no new work starts there. Thecluster_nameandcordon_reasondata keys give the site and the reason given, and the event records who cordoned it.info:site:uncordoned— a cordoned site was uncordoned and takes new work again.warn:trust_store:reconcile_failed— a site's API started without bringing its trust store in line with the CA certificates stored in MintPress. Theerrordata key says why. The site tries again within 10 minutes.error:trust_store:reconcile— a site could not bring its trust store in line with the CA certificates stored in MintPress. The message names the site and the error. Throttled to one event per hour for each site.
Logs
See log ingestion and log forwarding for more about these events.
warning:log_lines:dropped— a pod dropped or lost log lines before they were stored, usually because its log buffer filled while MintPress was unavailable, or because a step wrote output faster than it could be stored. Raisinglog_buffer_sizehelps. Thedropped_countdata key records how many.warning:log_lines:legacy_pods_running— pods created by an earlier version of MintPress are still running, and their log output is not stored. Thepod_namesandcluster_namedata keys identify them.error:log_ingest:rejected_batch— MintPress rejected a batch of log lines that could never be stored, so they were dropped. Throttled to one event per hour for each pod and reason.error:step:log_not_stored— one of MintPress's own log messages for a step could not be stored. The step carried on regardless. Throttled to one event per hour for each step.error:log_forwarding:failed— stored log lines could not be sent to the log aggregator for your additional outputs. MintPress keeps them queued and retries. Throttled to one event per hour for each site.warning:log_forwarding:dropped— log lines waiting to be sent to the log aggregator were discarded because more thanlog_forwarding.max_queued_lineswere waiting. Thedropped_countdata key records how many. The lines remain stored in MintPress.
Platform and background jobs
error:api:controller:unhandled— an API request failed with an error MintPress did not expect. Therequest_methodandrequest_pathdata keys identify the request.api:job:error— a recurring background job could not be re-queued, so it may not run until the next attempt.
Custom (i.e. user created) events can have any type as it is specified when the event is created.
Removing events
Older MintPress events can be removed to free up space, see the MintPress data cleaning API reference guide for more details.