Contributing Guide#

Contributions are what make the open source community such an amazing place to learn, inspire, and create in. Any contributions you make are greatly appreciated!

TL;DR Quickstart#

  1. Have pre-requisites installed:

  2. Fork the project

  3. git clone your fork locally

  4. Create your feature branch (ex. git checkout -b amazing-feature)

  5. Run pre-commit install

  6. Test out nox and build docs with nox -e 'docs-html(clean=True)'. The first time this is ran will take some time, but later generating of docs can be done with nox -e 'docs-html(clean=False)'

  7. Hack away!

  8. Run all unit tests with nox -e tests-3 -- -vv tests/unit/idem_aws

    • tests-3 ensures that the currently available version of Python 3 is ran (if supported)

    • Run targeted unit tests by calling a file directly. Example: nox -e tests-3 -- -vv tests/unit/idem_aws/tool/aws/test_client.py

  9. Commit your changes (ex. git commit -m 'Add some amazing-feature')

  10. Push to the branch (ex. git push origin amazing-feature)

  11. Open a merge request

For the full details, see below.

Ways to contribute#

We value all contributions, not just contributions to the code. In addition to contributing to the code, you can help the project by:

  • Writing, reviewing, and revising documentation, modules, and tutorials

  • Opening issues on bugs, feature requests, or docs

  • Spreading the word about how great this project is

The rest of this guide will explain our toolchain and how to set up your environment to contribute to the project.

Overview of how to contribute to this repository#

To contribute to this repository, you first need to set up your own local repository:

After this initial setup, you then need to:

Once a merge request gets approved, it can be merged!

Prerequisites#

For local development, the following prerequisites are needed:

This project makes use of POP, or Plugin Oriented Programming, in addition to idem. If you are new to projects making use of these, the following reference material should be useful:

Windows 10/11 users#

For the best experience, when contributing from a Windows OS to projects using Python-based tools like pre-commit, we recommend setting up Windows Subsystem for Linux (WSL), with the latest version being WSLv2.

The following gists on GitHub have been consulted with success for several contributors:

We recommend Installing Chocolatey on Windows 10 via PowerShell w/ Some Starter Packages. This installs git, microsoft-windows-terminal, and other helpful tools via the awesome Windows package management tool, Chocolatey.

choco install git easily installs git for a good Windows-dev experience. From the git package page on Chocolatey, the following are installed:

  • Git BASH

  • Git GUI

  • Shell Integration

Fork, clone, and branch the repo#

This project uses the fork and branch Git workflow. For an overview of this method, see Using the Fork-and-Branch Git Workflow.

  • First, create a new fork into your personal user space.

  • Then, clone the forked repo to your local machine.

    # SSH or HTTPS
git clone <forked-repo-path>/idem_aws.git

Note

Before cloning your forked repo when using SSH, you need to create an SSH key so that your local Git repository can authenticate to the GitLab remote server. See GitLab and SSH keys for instructions, or Connecting to GitHub with SSH.

Configure the remotes for your main upstream repository:

# Move into cloned repo
cd idem_aws

# Choose SSH or HTTPS upstream endpoint
# git@gitlab.com:vmware/idem/idem-aws.git
# https://gitlab.com/vmware/idem/idem-aws.git
# Example using SSH:
git remote add upstream git@gitlab.com:vmware/idem/idem-aws.git

Create new branch for changes to submit:

git checkout -b amazing-feature

Set up your local preview environment#

If you are not on a Linux machine, you need to set up a virtual environment to preview your local changes and ensure the prerequisites are met for a Python virtual environment.

From within your local copy of the forked repo:

# Install required python packages to venv
# This may require pip3 instead of pip in commands
pip install --user --upgrade pip
pip install --user --upgrade nox pre-commit

# Setup pre-commit by running this in root of project dir
pre-commit install

pre-commit and nox Setup#

This project uses pre-commit and nox to make it easier for contributors to get quick feedback, for quality control, and to increase the chance that your merge request will get reviewed and merged.

nox handles Sphinx requirements and plugins for you, always ensuring your local packages are the needed versions when building docs. You can think of it as Make with superpowers.

nox also manages dynamic virtual environments for running tests, instead of requiring contributors to setup their own environments to run pytest within.

What is pre-commit?#

pre-commit is a tool that will automatically run local tests when you attempt to make a git commit. To view what tests are run, you can view the .pre-commit-config.yaml file at the root of the repository.

One big benefit of pre-commit is that auto-corrective measures can be done to files that have been updated. This includes Python formatting best practices, proper file line-endings (which can be a problem with repository contributors using differing operating systems), and more.

If an error is found that cannot be automatically fixed, error output will help point you to where an issue may exist.

Sync local master branch with upstream master#

If needing to sync feature branch with changes from upstream master, do the following:

Note

This will need to be done in case merge conflicts need to be resolved locally before a merge to master in the upstream repo.

git checkout master
git fetch upstream
git pull upstream master
git push origin master
git checkout my-new-feature
git merge master

Preview docs changes locally#

To ensure that the changes you are implementing are formatted correctly, you should preview a local build of your changes first. To preview the changes:

# Generate HTML documentation with nox
nox -e 'docs-html(clean=True)' # First run
nox -e 'docs-html(clean=False)' # Subsequent runs

# Sphinx website documentation is dumped to docs/_build/html/*
# View locally
# Use xdg-open instead of firefox when on Linux and MacOS systems
firefox docs/_build/html/index.html # Firefox is just an example

# Optional: Runs an interactive docs site while modifying
nox -e docs

Note

If you encounter an error, Sphinx may be pointing out formatting errors that need to be resolved in order for nox to properly generate the docs.

URL validation with brok#

brok is used in the testing pipeline as a way to verify URLs.

We went with brok instead of the built-in Sphinx link validator because brok can scan files of any kind across a repository to do validations, as opposed to only files within scope of Sphinx docs. This makes the tool more versatile across our repositories.

The output in the CI/CD logs should show which URLs are causing any problems. The links can be verified in your personal browser, and links can then be updated in the appropriate files if they have become out-dated.

  • If a URL or IP address is being used to provide an example, refer to the Google Style Guide on best practices (ex. use example.com, example.org, IP addresses that don’t exist on the internet, etc.). This ensures that URL validation is simplified, with less exceptions, and isn’t hitting an actual endpoint. For more information: Google Style Guide: Example domains and names

  • If a URL is valid, but can’t be easily verified (due to requiring login, or the site is preventing automated link calls, or the link is an example dummy link, etc.), then the link should be added to the .brokignore file at the root of the repo.

It is recommended that brok isn’t ran locally, and is only done via pipeline, as otherwise your personal IP address will be reaching out to websites in a way that may be seen as behavior to be blocked.

Running tests#

Unit tests can be run with only the prerequisite of nox.

# Example: Run all unit tests
nox -e tests-3 -- -vv tests/unit/idem_aws

Integration tests will require LocalStack.

LocalStack setup#

LocalStack can be used to test idem-aws on your local machine without needing legitimate AWS credentials. It can be used for running the idem-aws tests or for testing your states locally.

Note

To test with LocalStack CLI, docker is required.

Install LocalStack with pip in a venv:

# Create a new virtualenv
# May require python3 instead of python
python -m venv .venv
source .venv/bin/activate
pip install -U pip setuptools wheel

# Install localstack
pip install "localstack [full]"

Start the LocalStack infrastructure:

localstack infra start

In a separate terminal, you should now be able to run integration tests.

Run Integration Tests#

In order to run the tests you must have a profile called test_development_idem_aws in your acct provider information. This can use LocalStack (recommended), otherwise testing does support valid production AWS credentials at your own risk.

Create a file named .localstack.yml with the following contents. This file will be ignored by the .gitignore file in order to prevent any accidental commits of your localstack configuration.

aws:
  test_development_idem_aws:
    endpoint_url: http://localhost:4566
    use_ssl: False
    aws_access_key_id: localstack
    aws_secret_access_key: _
    region_name: us-west-1

For tests to properly reference the profile, you need to define the ACCT_FILE environment variable. Integration tests can then be ran as expected.

# ACCT_FILE is referenced by idem-aws to know what profile you are using
export ACCT_FILE='.localstack.yml'

# Example: Run all integration tests
nox -e tests-3 -- -vv tests/integration/idem_aws

# Example: Run all integration AND unit tests
nox -e tests-3 -- -vv tests/integration/idem_aws tests/unit/idem_aws