Troubleshooting Idem#

This section describes issues that you might see while using Idem.

Idem command output is empty#

In some cases, Idem commands return empty braces:

idem describe aws.ec2.instance
{}

Unexpected empty output occurs because Idem can’t retrieve data from the endpoint. Configuration or connection issues are often the cause:

  • Idem is installed, but not the endpoint plug-in.
  • The endpoint plug-in needs to be updated.
  • The endpoint plug-in doesn’t support the resource type that you’re trying to manage.
  • Account file or key environment variables haven’t been added.
  • The credentials file is missing, or has incorrect or missing keys.

Note that an empty result might be correct when everything is in order but there are no results to be found.

Authentication errors when connecting to AWS#

Per-session credentials for AWS require an additional entry in the Idem credentials file, to account for the session token:

aws:
  default:
    use_ssl: true
    aws_access_key_id: xxxxxxxxxxxxxxxxxxxx
    aws_secret_access_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    aws_session_token: IQoJb3JpZ2luX2VjEMT//////////wEaCXVzLWVhc3QsMSJIMEYCIQDYCDX9mqho....
    region_name: us-west-2

Throttling errors#

When the number of AWS service requests exceeds the allowed limit, AWS throttles additional requests. Throttled requests receive a 429 HTTP response code, and you could see RequestLimitExceeded, ThrottlingException, or similar messages when running Idem commands. For example:

ClientError: An error occurred (Throttling) when calling the
<operation-name> operation (reached max retries: 4): Rate
exceeded

Changing the number and type of retries might help. Edit your Idem account credentials file to add a config retries block:

aws:
  default:
    use_ssl: true
    aws_access_key_id: xxxxxxxxxxxxxxxxxxxx
    aws_secret_access_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    region_name: us-west-2
    config:
      retries:
        max_attempts: <N>
        mode: <retry-mode>

The retry mode can be legacy, standard, or adaptive. For more information, see the AWS retries documentation.

ESM cache issues#

To remain idempotent, Idem keeps track of the resources that it manages. For example, repeated present operations must know whether or not the resource already exists when you only want the resource to be created once.

Idem tracks resources in a compressed Enforced State Management (ESM) cache file:

  • Linux/Mac default

    .idem/var/cache/idem/esm/local/cli.msgpack

  • Windows default

    C:\Users\<username>\AppData\Local\idem\esm\local\cli.msgpack

You can change the ESM cache directory with the --cache_dir CLI option.

Be careful to keep actual resources and the ESM cache in sync. For example, you might delete a resource by directly using the AWS interface, which leaves the resource listed only in the cache. When that happens, Idem operations might fail and post an error about not finding the resource:

Comment: ["Get aws.ec2.vpc 'vpc-idem-test' result is empty",
"ClientError: An error occurred (InvalidVpcID.NotFound) when
calling the DescribeVpcs operation: The vpc ID
'vpc-08388a2fbec8c6c22' does not exist"]

In some cases, you might want to delete the ESM cache and start over with a fresh describe operation. A fresh describe lets you define a new starting point, an SLS file that’s in sync with actual resources.