Using local library in Ansible filter_plugin

I am creating a bunch of filter plugins for Ansible (2.9/2.13) and I would like to have a tools library that can be used by these filter plugins.

I have the directory filter_plugins under my playbook directory and in there, I have a bunch of python scripts that provide Ansible with various filters. This works fine.

Now, some of these python scripts have the same functions. Naturally, I would like to move these shared function to a separate library file. So here is what I tried:

Option 1: have the library file (tools.py) in the directory filter_plugins
- This does not work well. Python can't find the module tools and Ansible gives me this error:

[WARNING]: Skipping plugin (/<REDACTED>/playbook_dir/filter_plugins/tools.py) as it seems to be invalid: module 'ansible.plugins.filter.966997458576792353_tools' has no attribute 'FilterModule'

Option 2: Create a library folder and put the tools.py script there (import library.tools or from library.tools import shared_function)
- This doesn't work either (with or without an empty __init__.py). Ansible gives me this error:

[WARNING]: Skipping plugin (/<REDACTED>/playbook_dir/filter_plugins/check.py) as it seems to be invalid: No module named 'tools'

All the filter scripts look basically the same (some have more filters, some have less):

#!/usr/bin/python

class FilterModule(object):
    def filters(self):
        return {
            'filter1': self.filter1,
            'filter2': self.filter2
        }

    def shared_function(data_to_process):
        processed_data = <process_data>
        return processed_data

    def filter1(self, in_data):
        """some code to process in_data to out_data"""
        process_data = <do_some_stuff>
        out_data = shared_function(process_data)
        return out_data

    def filter2(self, in_data):
        """some other code to process in_data to out_data"""
        process_data = <do_some_different_stuff>
        out_data = shared_function(process_data)
        return out_data

As shared_function is used in more than one filter script, I want to move it out of there to the tools.py library

Can anybody help me find the correct way to include local library files in filter plugins for Ansible?

Thanx for your help.

Solution

So after a couple of months, leaving this issue to the side, I finally found a useful (at least for myself) solution.

next to the filter_plugins directory, I create the module_utils directory. In here, I put everything I want to import from a filter:

├── roles
│   └── my_role
│       └── tasks
│       └── filter_plugins
│       └── module_utils
│           └── my_python_module
│       └── ...

Now, in if there is anything I need to import, I use the following piece of code:

#!/usr/bin/python

import sys

class FilterModule(object):

    def filters(self):
        return {
            "my_filter": self.my_filter
        }

    def my_filter(self, argument):
        current_file = __file__
        current_path = os.path.dirname(current_file)
        parent_path = os.path.dirname(current_path)
        module_path = parent_path+"/module_utils"

        sys.path.append(module_path)
        import my_python_module

You can treat the module_utils directory just like the lib/python3.x/site-packages directory. So if you install packages via pip, you can copy them from lib/python3.x/site-packages to your module_utils directory.

For instance, I created a fresh venv, activated it and ran the following command:

pip3 install jdcal openpyxl

This installed the following files in my ~/venv/lib/python3.9/site-packages:

$ ls -ln ~/venv/lib/python3.9/site-packages
...
drwxr-xr-x 1 4096 4096     0 30 okt 18:09 et_xmlfile
drwxr-xr-x 1 4096 4096     0 30 okt 18:09 et_xmlfile-2.0.0.dist-info
-rw-r--r-- 1 4096 4096 12462 30 okt 18:10 jdcal.py
drwxr-xr-x 1 4096 4096     0 30 okt 18:10 jdcal-1.4.1.dist-info
drwxr-xr-x 1 4096 4096     0 30 okt 18:09 openpyxl
drwxr-xr-x 1 4096 4096     0 30 okt 18:09 openpyxl-3.1.5.dist-info
...

Next, I copied the directories et_xmlfile and openpyxl and the file jdcal.py to the module_utils directory of my role and used the above piece of code to succesfully import the openpyxl module in my filter.