Exec Modules#
An Exec Module provides imperative functions to directly interact with your cloud or API. All python functions in an Exec module will be made available on the CLI at exec.<your module>.<function name>. These functions should imperatively take an action such as creating something or deleting something.
You can create as many nested directories to group functions as needed. For example, an Idem exec function for a cloud might have exec.<my cloud>.network.vpc.create. So all functions related to networking within the cloud are in the network directory or file under the my cloud directory. In this case “network” is the sub-directory and “vpc” is the file.
Create Sqlite Exec module#
We’re going to add a simple Idem Exec module to our idem-sqlite-demo Provider plugin to interact with a small sqlite database. We’ll be using python functions to interact with this sqlite database, but the same principles hold if you’re communicating with your cloud’s API through a python module or directly making http requests to the API.
Create Sqlite module#
Our idem_sqlite_demo/exec/demo directory should already exist. Let’s create our sqlite python module.
touch idem_sqlite_demo/exec/demo/sqlite.py
Add Setup function#
Note
In a production provider plugin, a separate “db” module to handle creating and managing the sqlite db file would be recommended. In the interest of brevity, we’re just going to have our sqlite module handle setting up the database for us.
Edit idem_sqlite_demo/exec/demo/sqlite.py with the following contents
"""Demo Idem Provider plugin for interacting with an sqlite database."""
import os
import sqlite3
async def setup_db(hub, ctx, db_path="~/idem-sqlite-demo.db"):
"""Create a new sqlite database.
Args:
db_path (str, Optional):
The path to the desired location of the sqlite file.
Examples:
Calling from the CLI:
.. code-block:: bash
$ idem exec demo.sqlite.setup_db
"""
result = dict(comment=[], ret=None, result=True)
con = sqlite3.connect(os.path.expanduser(db_path))
cur = con.cursor()
# check if table already exists
tables = cur.execute(
"""SELECT name FROM sqlite_master WHERE type='table'
AND name='people'; """
).fetchall()
if not tables:
cur.execute(
"""CREATE TABLE people
(resource_id integer primary key AUTOINCREMENT,
name varchar(20) NOT NULL,
age varchar(20) NOT NULL)"""
)
cur.execute(
"""
INSERT INTO people VALUES
(null, 'Sarah', 28),
(null, 'David', 45)
"""
)
con.commit()
vals = cur.execute("SELECT resource_id, name, age FROM people").fetchall()
result["ret"] = list(vals)
con.close()
return result
We’ll use this function to set up our sqlite database. We’ll specify a default location, but you can specify a different path if you desire.
Let’s initialize our database.
idem exec demo.sqlite.setup_db
You should see this output:
|_
- 1
- Sarah
- 28
|_
- 2
- David
- 45
Add List function#
Now we’re going to create a list function to list all our database entries. You would do something similar to list all VMs in your cloud or all the VPCs.
See idem-aws exec functions for examples.
First we need to add a function alias at the top of our sqlite.py file. The word list is a reserved word in python so we need to define our function as list_. Notice the underscore. This will allow us to use list on the CLI and when calling the function directly, but not overwrite the built in Python list.
Make the top of your file look like this:
"""Demo Idem Provider plugin for interacting with an sqlite database."""
import os
import sqlite3
__func_alias__ = {"list_": "list"}
Now let’s add our list function. We’ll add this above the existing setup_db function. Notice the underscore in the function definition. The __funct_alias__ above remaps that to list.
"""Demo Idem Provider plugin for interacting with an sqlite database."""
import os
import sqlite3
__func_alias__ = {"list_": "list"}
async def list_(hub, ctx, db_path="~/idem-sqlite-demo.db"):
"""List all people database entries.
Args:
db_path (str, Optional):
The path to the desired location of the sqlite file.
Examples:
Calling from the CLI:
.. code-block:: bash
$ idem exec demo.sqlite.list
"""
result = dict(comment=[], ret=None, result=True)
con = sqlite3.connect(os.path.expanduser(db_path))
con.row_factory = sqlite3.Row
cur = con.cursor()
vals = cur.execute("SELECT resource_id, name, age FROM people").fetchall()
result["ret"] = [list(row) for row in vals]
con.close()
return result
Let’s list our database entries:
idem exec demo.sqlite.list
We should get the following result:
|_
- 1
- Sarah
- 28
|_
- 2
- David
- 45
I mentioned “Tool plugins” in the Plugin overview. We’re going to create our first utility function now. This function is going to reformat the sqlite return. We’ll make this a Tool function so that we can easily reuse this elsewhere.
First we’ll create the demo directory in the tool subdirectory and create the sqlite file:
mkdir idem_sqlite_demo/tool/demo
touch idem_sqlite_demo/tool/demo/sqlite.py
Now add the following contents to your sqlite file:
"""Utility functions for the idem-sqlite-demo project"""
async def sqlite_to_dict(hub, ctx, cur):
"""Convert sqlite cursor to a python dictionary.
Args:
cur (str):
Sqlite cursor
"""
return [dict(row) for row in cur]
Now we can use this function to change the format of the sqlite return anywhere we want. If we decide to change the output we can make that change in this one location.
Here’s the updated list function:
async def list_(hub, ctx, db_path="~/idem-sqlite-demo.db"):
"""List all people database entries.
Args:
db_path (str, Optional):
The path to the desired location of the sqlite file.
Examples:
Calling from the CLI:
.. code-block:: bash
$ idem exec demo.sqlite.list
"""
result = dict(comment=[], ret=None, result=True)
con = sqlite3.connect(os.path.expanduser(db_path))
con.row_factory = sqlite3.Row
cur = con.cursor()
vals = cur.execute("SELECT resource_id, name, age FROM people").fetchall()
if vals:
result["ret"] = await hub.tool.demo.sqlite.sqlite_to_dict(ctx, vals)
con.close()
return result
Let’s list our database entries again:
idem exec demo.sqlite.list
We should now get the following result that includes the column names:
|_
----------
resource_id:
1
name:
Sarah
age:
28
|_
----------
resource_id:
2
name:
David
age:
45
Let’s update the setup_db function to also use our new sqlite Tool function:
Update your setup_db function to look like this:
async def setup_db(hub, ctx, db_path="~/idem-sqlite-demo.db"):
"""Create a new sqlite database.
Args:
db_path (str, Optional):
The path to the desired location of the sqlite file.
Examples:
Calling from the CLI:
.. code-block:: bash
$ idem exec demo.sqlite.setup_db
"""
result = dict(comment=[], ret=None, result=True)
con = sqlite3.connect(os.path.expanduser(db_path))
con.row_factory = sqlite3.Row
cur = con.cursor()
# check if table already exists
tables = cur.execute(
"""SELECT name FROM sqlite_master WHERE type='table'
AND name='people'; """
).fetchall()
if not tables:
cur.execute(
"""CREATE TABLE people
(resource_id integer primary key AUTOINCREMENT,
name varchar(20) NOT NULL,
age varchar(20) NOT NULL)"""
)
cur.execute(
"""
INSERT INTO people VALUES
(null, 'Sarah', 28),
(null, 'David', 45)
"""
)
con.commit()
vals = cur.execute("SELECT resource_id, name, age FROM people").fetchall()
if vals:
result["ret"] = await hub.tool.demo.sqlite.sqlite_to_dict(ctx, vals)
con.close()
return result
Add Get function#
Now we’re going to create a get function that will retrieve information about a specific entry in our sqlite database. This would be similar to requesting information about a specific VM from our cloud.
Add the following function to your file at idem_sqlite_demo/exec/demo/sqlite.py. You can put this above or below your list_ function.
async def get(hub, ctx, resource_id, db_path="~/idem-sqlite-demo.db"):
"""Retrieve information for a specific entry in our database.
Args:
resource_id(str):
The resource_id of the database entry.
db_path (str, Optional):
The path to the desired location of the sqlite file.
Examples:
Calling from the CLI:
.. code-block:: bash
$ idem exec demo.sqlite.get 2343
"""
result = dict(comment=[], ret=None, result=True)
con = sqlite3.connect(os.path.expanduser(db_path))
con.row_factory = sqlite3.Row
cur = con.cursor()
val = cur.execute(
"SELECT resource_id, name, age FROM people WHERE resource_id = ?",
(resource_id,),
).fetchone()
if val:
result["ret"] = dict(val) # convert a valid response to a dictionary
con.close()
return result
In this get function we have a required position argument resource_id. We then query for resource_id in our database and return the results. Use a resource_id from one of the entries we viewed by using the list function.
Let’s run our get command.
idem exec demo.sqlite.get 2
We should see the following result:
resource_id:
2
name:
David
age:
45
Add Create function#
Now let’s add a create function. This function will take a name and age as arguments and add them to the database. In the interest of brevity we’re not going to add functionality for maintaining uniqueness in this database. The reader can add this functionality if desired.
Add this create function anywhere in the sqlite.py file we’ve been working on.
async def create(hub, ctx, name, age, db_path="~/idem-sqlite-demo.db"):
"""Create a new database entry.
Args:
name (str):
The person's name.
age (int):
The person's age.
db_path (str, Optional):
The path to the desired location of the sqlite file.
Examples:
Calling from the CLI:
.. code-block:: bash
$ idem exec demo.sqlite.create name="Sumit" age=35
.. code-block:: bash
$ idem exec demo.sqlite.create Sarah 36
"""
result = dict(comment=[], ret=None, result=True)
con = sqlite3.connect(os.path.expanduser(db_path))
cur = con.cursor()
cur.execute("INSERT INTO people VALUES(null, ?, ?)", (name, age))
con.commit()
con.close()
return result
Now we can use our new create function to add a new entry to our database.
idem exec demo.sqlite.create 'Joe' 22
You should see output of just None.
None
We’re not going to spend too much time perfecting this example, but just seeing None doesn’t leave me confident that our data successfully made it into our database. Let’s query the database for our info and return that. Instead of creating a new sql query, we’ll reuse our get function that we’ve already added. This will introduce us into how we can use Idem Exec functions from within any Idem function.
You might have noticed that all the functions we’ve created so far have had hub as their first argument. The hub is a pop feature. You can dive into the details of pop’s hub here.
The quick explanation is that all of the functions we’ve created so far are available inside ANY Idem function at hub.exec.demo.sqlite.<FUNCTION NAME>. So in this case we’re going to use hub.exec.demo.sqlite.get to query our database.
Here’s our updated sqlite.py:
async def create(hub, ctx, name, age, db_path="~/idem-sqlite-demo.db"):
"""Create a new database entry.
Args:
name (str):
The person's name.
age (int):
The person's age.
db_path (str, Optional):
The path to the desired location of the sqlite file.
Examples:
Calling from the CLI:
.. code-block:: bash
$ idem exec demo.sqlite.create name="Sumit" age=35
.. code-block:: bash
$ idem exec demo.sqlite.create Sarah 36
"""
result = dict(comment=[], ret=None, result=True)
con = sqlite3.connect(os.path.expanduser(db_path))
cur = con.cursor()
cur.execute("INSERT INTO people VALUES(null, ?, ?)", (name, age))
con.commit()
# Let's query our database to ensure our data is there.
val = await hub.exec.demo.sqlite.get(ctx, resource_id=cur.lastrowid)
result["ret"] = val["ret"]
con.close()
return result
Let’s try running create again.
idem exec demo.sqlite.create 'Sonya' 23
You should see that our data was entered and then queried from out database. We were able to reuse our get function we already created.
resource_id:
5
name:
Sonya
age:
23
Add Update function#
Finally, let’s add an update function. This function will require the resource_id and optionally any of the data that we want to update.
async def update(
hub, ctx, resource_id, name=None, age=None, db_path="~/idem-sqlite-demo.db"
):
"""Update a database entry.
Args:
resource_id(int):
The person's unique id
name (str):
The person's name.
age (int):
The person's age.
db_path (str, Optional):
The path to the desired location of the sqlite file.
Examples:
Calling from the CLI:
.. code-block:: bash
$ idem exec demo.sqlite.update name="Sumit" age=35 resource_id=3
.. code-block:: bash
$ idem exec demo.sqlite.update resource_id=5 age=39
"""
result = dict(comment=[], ret=None, result=True)
con = sqlite3.connect(os.path.expanduser(db_path))
cur = con.cursor()
# Get existing values
val = await hub.exec.demo.sqlite.get(ctx, resource_id=resource_id)
vals = val["ret"]
# Update existing values with new values, if any are supplied
if name:
vals["name"] = name
if age:
vals["age"] = age
cur.execute(
"UPDATE people SET name=?, age=? WHERE resource_id=?",
(vals["name"], vals["age"], resource_id),
)
con.commit()
# Let's query our database to ensure our data is there.
val = await hub.exec.demo.sqlite.get(ctx, resource_id=resource_id)
result["ret"] = val["ret"]
con.close()
return result
Let’s try updating our previous entry.
idem exec demo.sqlite.update resource_id=5 age=24
You should see that Sonya’s age was updated.
resource_id:
5
name:
Sonya
age:
24
Complete Sqlite Exec module#
For the sake of completeness here’s the complete sqlite Exec module, including a delete function. Note that I’ve added detailed docstrings for each function. Idem uses this information for presenting documentation to the user. The Idem project uses Google style docstrings as a standard.
"""Demo Idem Provider plugin for interacting with an sqlite database."""
import os
import sqlite3
__func_alias__ = {"list_": "list"}
async def update(
hub, ctx, resource_id, name=None, age=None, db_path="~/idem-sqlite-demo.db"
):
"""Update a database entry.
Args:
resource_id(int):
The person's unique id
name (str):
The person's name.
age (int):
The person's age.
db_path (str, Optional):
The path to the desired location of the sqlite file.
Examples:
Calling from the CLI:
.. code-block:: bash
$ idem exec demo.sqlite.update name="Sumit" age=35 resource_id=3
.. code-block:: bash
$ idem exec demo.sqlite.update resource_id=5 age=39
"""
result = dict(comment=[], ret=None, result=True)
con = sqlite3.connect(os.path.expanduser(db_path))
cur = con.cursor()
# Get existing values
val = await hub.exec.demo.sqlite.get(ctx, resource_id=resource_id)
vals = val["ret"]
# Update existing values with new values, if any are supplied
if name:
vals["name"] = name
if age:
vals["age"] = age
cur.execute(
"UPDATE people SET name=?, age=? WHERE resource_id=?",
(vals["name"], vals["age"], resource_id),
)
con.commit()
# Let's query our database to ensure our data is there.
val = await hub.exec.demo.sqlite.get(ctx, resource_id=resource_id)
result["ret"] = val["ret"]
con.close()
return result
async def delete(hub, ctx, resource_id, db_path="~/idem-sqlite-demo.db"):
"""Delete a specific entry in our database.
Args:
resource_id(str):
The resource_id of the database entry.
db_path (str, Optional):
The path to the desired location of the sqlite file.
Examples:
Calling from the CLI:
.. code-block:: bash
$ idem exec demo.sqlite.delete 2343
"""
result = dict(comment=[], ret=None, result=True)
con = sqlite3.connect(os.path.expanduser(db_path))
con.row_factory = sqlite3.Row
cur = con.cursor()
val = cur.execute(
"Delete FROM people WHERE resource_id = ?",
(resource_id,),
)
con.commit()
result["ret"] = f"{cur.rowcount} row(s) deleted"
con.close()
return result
async def create(hub, ctx, name, age, db_path="~/idem-sqlite-demo.db"):
"""Create a new database entry.
Args:
name (str):
The person's name.
age (int):
The person's age.
db_path (str, Optional):
The path to the desired location of the sqlite file.
Examples:
Calling from the CLI:
.. code-block:: bash
$ idem exec demo.sqlite.create name="Sumit" age=35
.. code-block:: bash
$ idem exec demo.sqlite.create Sarah 36
"""
result = dict(comment=[], ret=None, result=True)
con = sqlite3.connect(os.path.expanduser(db_path))
cur = con.cursor()
cur.execute("INSERT INTO people VALUES(null, ?, ?)", (name, age))
con.commit()
# Let's query our database to ensure our data is there.
val = await hub.exec.demo.sqlite.get(ctx, resource_id=cur.lastrowid)
result["ret"] = val["ret"]
con.close()
return result
async def get(hub, ctx, resource_id, db_path="~/idem-sqlite-demo.db"):
"""Retrieve information for a specific entry in our database.
Args:
resource_id(str):
The resource_id of the database entry.
db_path (str, Optional):
The path to the desired location of the sqlite file.
Examples:
Calling from the CLI:
.. code-block:: bash
$ idem exec demo.sqlite.get 2343
"""
result = dict(comment=[], ret=None, result=True)
con = sqlite3.connect(os.path.expanduser(db_path))
con.row_factory = sqlite3.Row
cur = con.cursor()
val = cur.execute(
"SELECT resource_id, name, age FROM people WHERE resource_id = ?",
(resource_id,),
).fetchone()
if val:
result["ret"] = dict(val) # convert a valid response to a dictionary
con.close()
return result
async def list_(hub, ctx, db_path="~/idem-sqlite-demo.db"):
"""List all people database entries.
Args:
db_path (str, Optional):
The path to the desired location of the sqlite file.
Examples:
Calling from the CLI:
.. code-block:: bash
$ idem exec demo.sqlite.list
"""
result = dict(comment=[], ret=None, result=True)
con = sqlite3.connect(os.path.expanduser(db_path))
con.row_factory = sqlite3.Row
cur = con.cursor()
vals = cur.execute("SELECT resource_id, name, age FROM people").fetchall()
if vals:
result["ret"] = await hub.tool.demo.sqlite.sqlite_to_dict(ctx, vals)
con.close()
return result
async def setup_db(hub, ctx, db_path="~/idem-sqlite-demo.db"):
"""Create a new sqlite database.
Args:
db_path (str, Optional):
The path to the desired location of the sqlite file.
Examples:
Calling from the CLI:
.. code-block:: bash
$ idem exec demo.sqlite.setup_db
"""
result = dict(comment=[], ret=None, result=True)
con = sqlite3.connect(os.path.expanduser(db_path))
con.row_factory = sqlite3.Row
cur = con.cursor()
# check if table already exists
tables = cur.execute(
"""SELECT name FROM sqlite_master WHERE type='table'
AND name='people'; """
).fetchall()
if not tables:
cur.execute(
"""CREATE TABLE people
(resource_id integer primary key AUTOINCREMENT,
name varchar(20) NOT NULL,
age varchar(20) NOT NULL)"""
)
cur.execute(
"""
INSERT INTO people VALUES
(null, 'Sarah', 28),
(null, 'David', 45)
"""
)
con.commit()
vals = cur.execute("SELECT resource_id, name, age FROM people").fetchall()
if vals:
result["ret"] = await hub.tool.demo.sqlite.sqlite_to_dict(ctx, vals)
con.close()
return result