Enforced State Management

Enforced state management (ESM) lets Idem track resources across runs. ESM makes it possible for resources that aren’t natively idempotent to become idempotent through their unique present state name.

In the given context, the previous state (old_state) will be enforced with the following logic:

• Parameters for a resource in the given SLS file will have the highest priority

• If a parameter is not defined in the SLS, it will be pulled from the old_state of the previous run

• If there is no old_state or the parameter is not in old_state, then the default from the python function header will be used.

Local cache

The default ESM plugin keeps a local cache of the enforced state. The local cache is based on the --root-dir, --cache-dir, and --run-name cli arguments. Alternatively the root_dir, cache_dir, and run_name variables can be set in idem’s config. The default root_dir is / when running idem as root, otherwise it is ~/.idem . The default cache_dir is cache under root_dir/var; see below for examples:

If the run_name is cli (the default), then the local ESM plugin will store it’s cache in ~/.idem/var/cache/idem/esm/local/cli.mspgack. The ESM cache would be in ~/.idem/var/cache/idem/esm/cache/cli.mspgack.

The cache contains the “new_state” data of state runs. Every key in the cache is a tag based on the state name, the state id, and the the state’s name.

For a state that looks like this:

state_id:
    cloud.resource.present:
       name: state_name

The tag generated to track that state’s “new_state” in the cache would look like this:

``cloud.resource_|-state_id_|-state_name_|-``

Idem states

State modules that return “old_state” and “new_state” will have “new_state” available in the ctx of future runs.

# my_project_root/my_project/state/my_plugin.py

__contracts__ = ["resource"]


def present(hub, ctx, name):
    # ctx.old_state contains the new_state from the previous run
    # When ctx.test is True, there should be no changes to the resource, but old_state and new_state should reflect changes that `would` be made.
    new_state = ...

    return {
        "result": True,
        "comment": "",
        "old_state": ctx.old_state,
        "new_state": new_state,
    }


def absent(hub, ctx, name):
    # ctx.old_state contains the new_state from the previous run
    return {"result": True, "comment": "", "old_state": ctx.old_state, "new_state": {}}


def describe(hub, ctx, name):
    ...

Unlock Idem state run

When a state is run using an esm provider other than the local default (such as AWS) a lock may be left behind when the state is prematurely canceled. To force an unlock in this situation use a command line such as the following:

idem exec esm.unlock provider=aws profile=<...> --acct-file=<...> --acct-key=<...>

context

The context feature allows only one instance of an Idem state run for a given context. It also exposes a state dictionary that can be managed by an arbitrary plugin. The context is managed for you by Idem when you write an ESM plugin. The following shows how context works and how to use it:

async def my_func(hub):
    # Retrieve the context manager
    context_manager = hub.idem.managed.context(
        run_name=hub.OPT.idem.run_name,
        cache_dir=hub.OPT.idem.cache_dir,
        esm_plugin="my_esm_plugin",
        esm_profile=hub.OPT.idem.esm_profile,
        acct_file=hub.OPT.acct.acct_file,
        acct_key=hub.OPT.acct.acct_key,
        serial_plugin=hub.OPT.idem.serial_plugin,
    )

    # Enter the context and lock the run.
    # This calls `hub.esm.my_esm_plugin.enter()` and `hub.esm.my_esm_plugin.get_state()` with the appropriate ctx
    async with context_manager as state:
        # The output of get_state() is now contained in the "state" variable
        # Changes to `state` will persist when we exit the context and `hub.esm.my_esm_plugin.set_state()` is called with the appropriate ctx
        state.update({})
    # After exiting the context, `hub.esm.my_esm_plugin.exit_() is called with the appropriate ctx

Writing an ESM plugin

An ESM plugin follows this basic format:

# my_project_root/my_project/esm/my_plugin.py
from typing import Any
from typing import Dict


def __init__(hub):
    hub.esm.my_plugin.ACCT = ["my_acct_provider"]


async def enter(hub, ctx):
    """
    :param hub:
    :param ctx: A namespace addressable dictionary that contains the `acct` credentials
        "acct" contains the esm_profile from "my_acct_provider"

    Enter the context of the enforced state manager
    Only one instance of a state run will be running for the given context.
    This function enters the context and locks the run.

    The return of this function will be passed by Idem to the "handle" parameter of the exit function
    """


async def exit_(hub, ctx, handle, exception: Exception):
    """
    :param hub:
    :param ctx: A namespace addressable dictionary that contains the `acct` credentials
        "acct" contains the esm_profile from "my_acct_provider"
    :param handle: The output of the corresponding "enter" function
    :param exception: Any exception that was raised while inside the context manager or None

    Exit the context of the Enforced State Manager
    """


async def get_state(hub, ctx) -> Dict[str, Any]:
    """
    :param hub:
    :param ctx: A dictionary with 3 keys:
        "acct" contains the esm_profile from "my_acct_provider"

    Use the information provided in ctx.acct to retrieve the enforced managed state.
    Return it as a python dictionary.
    """


async def set_state(hub, ctx, state: Dict[str, Any]):
    """
    :param hub:
    :param ctx: A namespace addressable dictionary that contains the `acct` credentials
        "acct" contains the esm_profile from "my_acct_provider"

    Use the information provided in ctx.acct to upload/override the enforced managed state with "state".
    """

Extend the ESM dyne in your project for a plugin:

# my_project_root/my_project/conf.py
DYNE = {"esm": ["esm"]}

refresh

Idem includes a refresh command that can bring resources from describe into the ESM context.

The refresh command:

$ idem refresh aws.ec2.*

Is functionally equivalent to these commands:

$ idem describe aws.ec2.* --output=yaml > ec2.sls
$ idem state ec2.sls --test
$ rm ec2.sls

An idem refresh only returns resource attribtues and should not make any changes to resources as long as the resources implement the ctx.test flag properly in their present state.

restore

ESM keeps a cache of the local run state. The restore command calls an ESM plugin’s set_state method with the contents of a json file.

$ idem restore esm_cache_file.json

The cache file is generated on every Idem state run and is based on Idem’s run_name and cache_dir:

$ idem state my_state --run-name=my_run_name --cache-dir=/var/cache/idem
# Cache file for this run will be located in /var/cache/idem/my_run_name.json