Enforced State Management#

To remain idempotent, Idem keeps track of the resources that it manages. For example, if you only want a resource to be created once, repeated present operations must know whether or not the resource already exists.

Enforced state management (ESM) is how Idem tracks a resource from run to run, using the ID, reference, and name. See Cache Keys below.

ESM plays a role in drift correction, where parameter values are enforced according to the following priority.

  1. Resource parameters defined in the SLS file are used first.

  2. If there is no SLS parameter, the cached ESM value from the most recent run is used.

  3. If there is no SLS parameter and no cached value, the default from the Python function header is used.

Local Cache#

The default ESM plugin keeps a local cache of the enforced state. The local cache is based on CLI arguments:

  • –root-dir

  • –cache-dir

  • –run-name

Alternatively, you can add variables in your Idem config file.

  • root_dir

    The default root_dir for non-root users is ~/.idem.

    The default is / when running Idem as root.

  • cache_dir

    The default cache_dir is root_dir/var/cache.

  • run_name

    The default run_name is cli and is part of the cache filename.

Using defaults, the local ESM cache is found at:

  • Linux/Mac default

    .idem/var/cache/idem/esm/local/cli.msgpack

  • Windows default

    C:\Users\<username>\AppData\Local\idem\esm\local\cli.msgpack

Cache Keys#

The cache contains current values from Idem state runs. Every key in the cache has a tag based on state ID, state reference, and state name. For an SLS state of the following:

state_id:
    cloud.resource.present:
       name: state_name

The tag in the cache becomes the following:

cloud.resource_|-state_id_|-state_name_|-

Idem States#

State modules that return old_state and new_state will have new_state available in the ctx of future runs.

# my_project_root/my_project/state/my_plugin.py

__contracts__ = ["resource"]


def present(hub, ctx, name):
    # ctx.old_state contains the new_state from the previous run
    # When ctx.test is True, there should be no changes to the resource, but old_state and new_state should reflect changes that would be made.
    new_state = ...

    return {
        "result": True,
        "comment": "",
        "old_state": ctx.old_state,
        "new_state": new_state,
    }


def absent(hub, ctx, name):
    # ctx.old_state contains the new_state from the previous run
    return {"result": True, "comment": "", "old_state": ctx.old_state, "new_state": {}}


def describe(hub, ctx, name):
    ...

ESM CLI#

Validating the ESM#

This CLI validates the access to the ESM and that its version is up to date.

$ idem exec esm.validate provider=<...> profile=<...> --acct-file=<...> --acct-profile=<...> --acct-key=<...>

  • provider is the esm_plugin, “local” by default

  • profile is the esm_profile, “default” by default

Unlocking an Idem State Run#

Sometimes a state is run using an ESM provider other than the local default (such as AWS). If such a state run is prematurely canceled, the provider might lock ESM. To force an unlock, use an Idem command based on the following pattern.

$ idem exec esm.unlock provider=<...> profile=<...> --acct-file=<...> --acct-profile=<...> --acct-key=<...>

  • provider is the esm_plugin, “local” by default

  • profile is the esm_profile, “default” by default

Display the ESM content#

To display the content of the ESM, either from an ESM provider or a local ESM cache, use Idem command based on the following pattern

$ idem exec esm.show provider=<...> profile=<...> --acct-file=<...> --acct-profile=<...> --acct-key=<...>

  • provider is the esm_plugin, “local” by default

  • profile is the esm_profile, “default” by default

Remove a resource from ESM#

To remove a single resource from ESM by its tag use the following Idem command

$ idem exec esm.remove tag=<...> provider=<...> profile=<...> --acct-file=<...> --acct-profile=<...> --acct-key=<...>

  • tag is the resource identified in ESM. Use idem exex esm.show to see the possible tags

  • provider is the esm_plugin, “local” by default

  • profile is the esm_profile, “default” by default

Context#

The context feature allows only one instance of an Idem state run for a given context. It also exposes a state dictionary that can be managed by an arbitrary plugin. The context is managed for you by Idem when you write an ESM plugin.

The following code example shows how context works and how to use it.

async def my_func(hub):
    # Retrieve the context manager
    context_manager = hub.idem.managed.context(
        run_name=hub.OPT.idem.run_name,
        cache_dir=hub.OPT.idem.cache_dir,
        esm_plugin="my_esm_plugin",
        esm_profile=hub.OPT.idem.esm_profile,
        acct_file=hub.OPT.acct.acct_file,
        acct_key=hub.OPT.acct.acct_key,
        serial_plugin=hub.OPT.idem.serial_plugin,
    )

    # Enter the context and lock the run.
    # This calls 'hub.esm.my_esm_plugin.enter()' and 'hub.esm.my_esm_plugin.get_state()' with the appropriate ctx
    async with context_manager as state:
        # The output of get_state() is now contained in the "state" variable
        # Changes to 'state' will persist when we exit the context and 'hub.esm.my_esm_plugin.set_state()' is called with the appropriate ctx
        state.update({})
    # After exiting the context, 'hub.esm.my_esm_plugin.exit_()' is called with the appropriate ctx

Writing an ESM Plugin#

An ESM plugin follows this basic format:

# my_project_root/my_project/esm/my_plugin.py
from typing import Any
from typing import Dict


def __init__(hub):
    hub.esm.my_plugin.ACCT = ["my_acct_provider"]


async def enter(hub, ctx):
    """
    :param hub:
    :param ctx: A namespace addressable dictionary that contains the 'acct' credentials
        "acct" contains the esm_profile from "my_acct_provider"

    Enter the context of the enforced state manager
    Only one instance of a state run will be running for the given context.
    This function enters the context and locks the run.

    The return of this function will be passed by Idem to the "handle" parameter of the exit function
    """


async def exit_(hub, ctx, handle, exception: Exception):
    """
    :param hub:
    :param ctx: A namespace addressable dictionary that contains the 'acct' credentials
        "acct" contains the esm_profile from "my_acct_provider"
    :param handle: The output of the corresponding "enter" function
    :param exception: Any exception that was raised while inside the context manager or None

    Exit the context of the Enforced State Manager
    """


async def get_state(hub, ctx) -> Dict[str, Any]:
    """
    :param hub:
    :param ctx: A dictionary with 3 keys:
        "acct" contains the esm_profile from "my_acct_provider"

    Use the information provided in ctx.acct to retrieve the enforced managed state.
    Return it as a python dictionary.
    """


async def set_state(hub, ctx, state: Dict[str, Any]):
    """
    :param hub:
    :param ctx: A namespace addressable dictionary that contains the 'acct' credentials
        "acct" contains the esm_profile from "my_acct_provider"

    Use the information provided in ctx.acct to upload/override the enforced managed state with "state".
    """

The following extends the ESM dyne in your project for a plugin:

# my_project_root/my_project/conf.py
DYNE = {"esm": ["esm"]}

Refresh#

Idem includes a refresh command that can update the ESM cache by bringing resources, as detected from a describe operation, into the ESM context:

$ idem refresh aws.ec2.*

The preceding refresh is equivalent to these commands:

$ idem describe aws.ec2.* --output=yaml > ec2.sls
$ idem state ec2.sls --test
$ rm ec2.sls

A refresh only returns resource attributes and should not make changes to resources as long as the resources implement the ctx.test flag properly in their present state.

Restore#

ESM keeps a cache of the local run state. The cache is a JSON file that’s generated on every Idem state run and is based on run_name and cache_dir:

$ idem state my_state --run-name=my_run_name --cache-dir=/var/cache/idem

The cache file for the preceding run is:

/var/cache/idem/my_run_name.json

Idem includes a restore command for the ESM cache. The command calls the ESM plugin set_state method with the contents of the saved JSON file.

$ idem restore my_run_name.json

Disable ESM#

To disable ESM, run idem with --esm-plugin=null or specify the null ESM plugin in your idem config:

# idem.cfg

idem:
  esm_plugin: "null"