Properties

The MintPress properties framework provides a location to store:

key value pairs
environment variables and values (that will be available in the Unix environment running a change action)
files (that will be written before running a change action)

MintPress properties can be stored in your Git remote and also at the project, environment and asset levels in the MintPress database. For the properties stored in the database, MintPress maintains a complete version history of each change made to the MintPress properties JSON, enabling you to view and compare the properties used for any change. Similarly, Git version history can be used to identify changes made to the Git repository properties.

After following this guide you should understand:

how to incorporate MintPress properties into your Git remote
how to view project, environment and asset MintPress properties from the GUI and API server
the various types of values that can be stored in MintPress properties
the difference between MintPress properties and OpsChain context values

MintPress properties

Within each action, MintPress properties are available via OpsChain.properties (which will behave like a Hashie Mash). The values available are the result of a deep merge of the change's Git repository properties with the project, environment and asset level properties. If a property exists at multiple levels, project values will override repository values, environment values will override project and repository values, and asset values will override all of them.

Properties can be accessed using dot or square bracket notation with string or symbol keys. These examples are equivalent:

require 'opschain'

OpsChain.properties.server.setting
OpsChain.properties[:server][:setting]
OpsChain.properties['server']['setting']

NOTES

You will not be able to use dot notation to access a property with the same name as a method on the properties object (for example keys). In this case you must use square bracket notation instead.
Any arrays in the properties will be overwritten during a deep merge (use JSON objects with keys instead to ensure they are merged).
The OpsChain.properties structure is read only. Please see modifiable properties below for information on making changes to the project, environment or asset properties.

Comparison with OpsChain context

In addition to the user defined and maintained MintPress properties available in a MintPress change, MintPress also provides OpsChain context values.

The OpsChain context is automatically populated by MintPress with information about the context in which a change is run, for example the project, environment or asset name or the action being executed by the change.

Rather than manually putting change related values - e.g. the environment code, project code, action name, etc. - into your properties, consider whether you could use the OpsChain context instead.

tip

See the OpsChain context guide if you would like to learn more about the OpsChain context framework.

Storage options

Git repository

The root folder used to store the repository properties is configurable via the OPSCHAIN_REPO_FOLDER setting. We'll assume the default value of .mintpress for the rest of this guide.

MintPress will look for properties stored in your template's Git remote under the .mintpress/properties folder. MintPress will navigate all the files with a json, yaml, and toml extension within the target directory tree.

Property files will be loaded starting from the top down to the bottom of the folder structure. Properties at the bottom of the folder structure will override the parent property if they define the same property.

For example, given the below property folder structure in your repository:

└── .mintpress
    └── properties
        └── projects
            ├── common-for-all-projects.json
            ├── bank
            │   ├── bank-project-properties.json
            │   └── environments
            │       ├── common-for-all-envs-in-bank-project.json
            │       ├── dev
            │       │   ├── dev-env-properties.json
            │       │   └── assets
            │       │       ├── common-for-all-assets-in-dev.json
            │       │       ├── soa
            │       │       │   ├── soa-1.json
            │       │       │   └── soa-2.json
            │       │       └── obp
            │       └── staging
            ├── another_project

Here are some of the possible scenarios:

When your target is the soa asset, the property files will be loaded and merged as follows:
1. load the common-for-all-projects.json properties
2. load the bank-project-properties.json properties, merging them with the common-for-all-projects.json properties
3. load the common-for-all-envs-in-bank-project.json properties, merging them with the result of the previous step
4. load the dev-env-properties.json properties, merging them with the result of the previous step
5. load the common-for-all-assets-in-dev.json properties, merging them with the result of the previous step
6. load the soa-1.json properties, merging them with the result of the previous step
7. load the soa-2.json properties, merging them with the result of the previous step
When your target is in another_project, you will share the common-for-all-projects.json properties file with other projects
When your target is the staging environment in the bank project, you will perform the first three load and merge steps described for the soa asset as these files are common to the soa and staging environments

Within each action, the result of merging the properties in these files will be available via OpsChain.repository_properties.

NOTES

Property files within the same folder are loaded alphabetically (including their extension name). For example, if the following property files are within a single folder they will be loaded and merged in the following order:

abc.json
abc.toml
abc.yaml
common-1.json
common.json
common1.json
common2.json

NOTES

Files with extensions other than .json, .yaml, or .toml will not be processed.
File extensions must be in lowercase (e.g. .json instead of .JSON). Files with non-lowercase extensions will not be processed.
Files without extensions will not be processed.

NOTES

The repository properties are read only within each action (as MintPress cannot modify the underlying Git repository to store any changes).

Parent specific repository properties

The project, environment or asset specific Git repository properties can be accessed via OpsChain.repository_properties_for(:project), OpsChain.repository_properties_for(:environment), OpsChain.repository_properties_for(:asset) respectively.

note

If multiple files exist for a given project, environment or asset code (e.g. .mintpress/environments/dev.json and .mintpress/environments/dev.toml) then OpsChain.repository_properties_for(:environment) will merge the properties in these files and return the result.

Access to the individual file contents is not available via the repository_properties_for API. Requiring access to the individual files may indicate a brittle properties implementation. If access is required then the individual files can be read manually using normal Ruby APIs, e.g. File.read etc.

Database

Properties stored in the database are encrypted prior to being written to disk such that they are encrypted-at-rest. Within each action, project, environment and asset properties are available via OpsChain.properties_for(:project), OpsChain.properties_for(:environment) and OpsChain.properties_for(:asset) respectively. Actions can modify these properties at runtime and any changes will be persisted to the database (see modifiable properties below).

Editing properties

You can edit properties at the project, environment or asset level in their respective properties page in the GUI.

Properties content

Key value pairs

You can use MintPress key value properties from anywhere in your actions.rb to provide project, environment or asset specific values to your resource actions. E.g.

database :my_database do
  host OpsChain.properties.database.host_name
  source_path OpsChain.properties.database.source_path
end

Modifiable properties

In addition to the read only values available from OpsChain.properties, the project, environment and asset specific database properties are available via:

OpsChain.properties_for(:project)
OpsChain.properties_for(:environment)
OpsChain.properties_for(:asset)

These are exposed to allow you to add, remove and update properties, with any modifications saved on step completion. The modified project, environment or asset properties are then available to any subsequent steps or changes.

The object returned by OpsChain.properties is the merged set of properties and is regenerated every time the method is called. This means that if the result of OpsChain.properties is assigned to a variable - or passed to a resource - it won't reflect updates.

log.info OpsChain.properties.example # ''
props = OpsChain.properties
OpsChain.properties_for(:project).example = 'hello'
log.info OpsChain.properties.example # 'hello'
log.info props.example # '' - this value was not updated

Creating / updating properties within actions

The following code will set the project server_name property, creating or updating it as applicable:

OpsChain.properties_for(:project).server_name = 'server1.limepoint.com'

note

As properties behave like a Hashie::Mash, creating multiple levels of property nesting in a single command requires you to supply a hash as the value. E.g.

OpsChain.properties_for(:project).parent = { child: { grandchild: 'value' } }

Once created, nested properties can be updated as follows:

OpsChain.properties_for(:project).parent.child.grandchild = 'new value'

Deleting properties

To delete the grandchild property described above, use the following command:

OpsChain.properties_for(:project).parent.child.delete(:grandchild)

note

This would leave the parent and child keys in the project properties. To delete the entire tree, use the following command:

OpsChain.properties_for(:project).delete(:parent)

To delete all database properties at the project, environment or asset level for the currently running step, use the following command:

OpsChain.properties_for(<level>).clear       # where <level> can be :project, :environment or :asset

caution

Using clear on your properties from within an action should be used with caution. This will remove all database properties at the specified level. To restore the properties, retrieve an older version via the GUI or API and then apply them as the latest version.

Example

An example of setting properties can be seen in the Confluent example. The provision action in actions.rb modifies the environment properties to change settings for broker1.

Changing properties in concurrent steps

Changes that take advantage of the :parallel change execution strategy will cause MintPress to run multiple steps concurrently. Similarly, starting multiple changes at once will also lead to steps executing concurrently.

When each step starts, the current state of the project, environment and asset properties (in the MintPress database) is supplied to the step's action(s). This means steps that run concurrently will start with the same set of properties. At the completion of each step, any changes made to the project and/or environment properties by the action, are reflected in a JSON Patch applicable to the relevant source properties. The JSON Patch(es) are returned from the step runner to the OpsChain API and applied to the current state of the database properties. It is up to the action developer to ensure any changes made to properties by concurrent steps are compatible with each other.

caution

MintPress recommends that you do not modify properties from within concurrent steps. However, if this is a requirement, ensuring the modifications apply to unrelated sections of the MintPress properties will mitigate the risk. The following sections describe various types of properties changes and the possible errors you may encounter. For simplicity, the examples all show concurrent steps created within a single change using the :parallel child step execution strategy. Steps executing from changes that have been submitted concurrently can run into similar limitations.

Modifying different properties

Using a JSON Patch to apply changes made by actions to the MintPress properties ensures concurrent steps can modify independent properties successfully. For example:

# Sets up an initial set of values for the MintPress project properties, then calls the foo and bar child actions in parallel
action :default, steps: [:foo, :bar], run_as: :parallel do
  OpsChain.properties_for(:project) = { foo: 'old_foo', bar: 'old_bar' }
end

action :foo do
  OpsChain.properties_for(:project).foo = 'new_foo'
end

action :bar do
  OpsChain.properties_for(:project).bar = 'new_bar'
end

At the completion of the child steps, the MintPress project properties will be:

{ foo: 'new_foo', bar: 'new_bar' }

Race conditions

Modifying the same property in concurrent steps will produce unexpected results. In the example below, at the completion of the child steps, the final value of the race property will be the value assigned by the child step that completes last.

# Sets up an initial set of values for the MintPress project properties, then calls the foo and bar child actions in parallel
action :default, steps: [:foo, :bar], run_as: :parallel do
  OpsChain.properties_for(:project) = { race: 'initial value' }
end

action :foo do
  OpsChain.properties_for(:project).race = 'possible value 1'
end

action :bar do
  OpsChain.properties_for(:project).race = 'possible value 2'
end

Conflicting changes

In addition to the race conditions example above, changes to MintPress properties made by concurrent steps can create JSON Patch conflicts that will result in a change failing. The following scenarios are example of parallel steps that will generate conflicting JSON Patches.

Scenario 1: Deleting a property in one child, while modifying that property's elements in the other.

action :default, steps: [:foo, :bar], run_as: :parallel do
  OpsChain.properties_for(:project).parent = { child: 'value' }
end

action :foo do
  OpsChain.properties_for(:project).delete(:parent)
end

action :bar do
  OpsChain.properties_for(:project).parent.child = 'new value'
  sleep(10)
end

Scenario 2: Modifying the data type of a property in one child, while generating a patch based on the original data type in the other.

action :default, steps: [:foo, :bar], run_as: :parallel do
  OpsChain.properties_for(:project).parent = { child: 'value' }
end

action :foo do
  OpsChain.properties_for(:project).parent = 'I am now a string'
end

action :bar do
  OpsChain.properties_for(:project).parent.child = 'new value'
  sleep(10)
end

In both scenarios, the default action will fail running child step bar. As the child steps start with the properties defined by the default action, the logic within each child will complete successfully. However, as bar (with its included sleep) will finish last, the JSON Patch it produces will fail when MintPress attempts to apply it as foo has changed the parent property to be incompatible with the patch made by bar. In both cases, the child element no longer exists and cannot be modified.

Resolving conflicts

If a step's JSON Patches fail to apply, the change will error at the failing step and the logs will provide the following information for each failed patch:

ERROR: Updates made to the project properties in step "[c5556d54-d98f-415e-9198-4134848fb93f] bar" could not be applied.

Original project properties supplied to the step:
{
  "parent": {
    "child": "value"
  }
}

JSON Patch reflecting the updates made to the properties in the step (that cannot be applied):
[{
  "op": "replace",
  "path": "/parent/child",
  "value": "new value"
}]

Patched original properties - that could not be saved because the project properties were modified outside this step:
{
  "parent": {
    "child": "new value"
  }
}

Current value of project properties (that the JSON Patch fails to apply to):
{}

Please resolve this conflict manually and correct the project properties via the `mintpress project set-properties` command. If applicable, retry the change to complete any remaining steps.

Use the four JSON documents from the change log, and your knowledge of the actions being performed by the conflicting steps, to:

construct a version of the properties that incorporates the required updates
use the GUI to manually update the relevant properties.

If there are no further steps in the change to run, there is no need to retry the failed change and you can continue using MintPress as normal.

If there are further steps in the change to run, and the failed step is idempotent, you can retry the change from the GUI to restart the change from the failed step. It is important to note that MintPress will re-run the failed step in its entirety.

If there are further steps in the change to run, and the failed step is NOT idempotent, you will need to create change(s) to perform the incomplete actions.

File properties

MintPress file properties are written to the working directory prior to the step action being initiated. Any property under opschain.files is interpreted as a file property and will be written to disk.

{
  "opschain": {
    "files": {
      "/full/path/to/file1.txt": {
        "mode": "0600",
        "content": "contents of the file"
      },
      "~/path/to/file2.json": {
        "content": {
          "json": "file",
          "values": "here"
        },
        "format": "json"
      }
    }
  }
}

Each file property key is an absolute path (or will be expanded to one) and represents the location the file will be written to. Each file property value can include the following attributes:

Attribute	Description
mode	The file mode, specified in octal (optional)
content	The content of the file (optional)
format	The format of the file (optional)

File formats

The file format attribute provides MintPress with information on how to serialise the file content (for storage in MintPress properties), and de-serialise the content (before writing to the Opschain runner filesystem). The following formats are currently supported:

base64
json
raw (default)

Please contact LimePoint if you require other file formats.

Storing & removing files

The project, environment or asset properties can be edited directly to add, edit or remove file properties (using a combination of a text editor, the show-properties and set-properties commands). In addition, MintPress enables you to store and remove files from within your actions.

Project, environment or asset file properties

To store a file in the project:

  OpsChain.store_file!(:project, '/file/to/store.txt')

To remove a file from the project:

  OpsChain.remove_file!(:project, '/file/to/store.txt')

To do the same for the environment or asset, replace :project with :environment or :asset respectively.

Optional file format

The store_file! method accepts an optional format: parameter, allowing you to specify the file format MintPress should use when adding the file into the file properties. For example:

  OpsChain.store_file!(:environment, '/file/to/store.txt', format: :base64)

Storing files examples

Examples of storing files can be seen in the Ansible example.

The save_known_hosts action in actions.rb uses this feature to store the SSH known_hosts file in the environment properties - to ensure the host is trusted in future steps and actions

Environment variables

MintPress environment variable properties allow you to configure the process' Unix environment prior to running your step actions. Any property under opschain.env will be interpreted as an environment variable property.

{
  "opschain": {
    "env": {
      "VARIABLE_NAME": "variable value",
      "DIFF_VARIABLE": "different variable value"
    }
  }
}

Action environment

Each step action is executed using the opschain-action command. This will define an environment variable for each of the MintPress environment variable properties prior to executing the action.

Bundler credentials

Bundler Gem source credentials can be configured via environment variables. Defining a MintPress environment variable with the relevant username/password (e.g. "BUNDLE_BITBUCKET__ORG": "username:password") will make this available to bundler.

Setting environment variables example

An example of setting environment variables can be seen in the Ansible example. The project_properties.json contains the credentials to be able to successfully login to your AWS account.

Secrets

OpsChain secret vault

In addition to Kubernetes secrets, OpsChain also allows you to use the OpsChain secret vault to store secure property information. Any MintPress property can reference a secret vault path as its value (including the content for file properties and the values for environment variables).

Secret vault values must use the following format:

{
  "my_password": "secret-vault://vault/path/to/secrets/secret_key"
}

In the above example, when the OpsChain.properties.my_password property is accessed, OpsChain will attempt to retrieve the secret_key secret, from the vault/path/to/secrets key value store in the secret vault. If required, the keyword arguments described in customising the secret get can be used to customise the secret generation. These must be supplied as query params in the secret vault path. E.g.

{
  "my_password": "secret-vault://vault/path/to/secrets/secret_key?length=20&include_numbers=false"
}

MintPress properties​

Comparison with OpsChain context​

Storage options​

Git repository​

Parent specific repository properties​

Database​

Editing properties​

Properties content​

Key value pairs​

Modifiable properties​

Creating / updating properties within actions​

Deleting properties​

Example​

Changing properties in concurrent steps​

Modifying different properties​

Race conditions​

Conflicting changes​

Resolving conflicts​

File properties​

File formats​

Storing & removing files​

Project, environment or asset file properties​

Optional file format​

Storing files examples​

Environment variables​

Action environment​

Bundler credentials​

Setting environment variables example​

Secrets​

OpsChain secret vault​

MintPress properties

Comparison with OpsChain context

Storage options

Git repository

Parent specific repository properties

Database

Editing properties

Properties content

Key value pairs

Modifiable properties

Creating / updating properties within actions

Deleting properties

Example

Changing properties in concurrent steps

Modifying different properties

Race conditions

Conflicting changes

Resolving conflicts

File properties

File formats

Storing & removing files

Project, environment or asset file properties

Optional file format

Storing files examples

Environment variables

Action environment

Bundler credentials

Setting environment variables example

Secrets

OpsChain secret vault