Including your Airflow plugins in Python path just like it works in a production environment

It’s common to face import errors when creating your own custom plugins and dags while developing for Airflow, mainly because when the Airflow webserver starts, it creates references for the plugins within the plugins folder to its own python path.

Import errors when developing the DAG script

To solve this problem, I built a simple shell script to simulate how airflow access your custom plugins just by linking your local plugins repositories to both python and airflow installation paths.

Utilization of the script

The script, when ran, will ask some information:

• The name of your plugin (the same you set in the __init__.py file).
• The absolute path to your plugin repository folder.
• The modules from your plugin you want to load (separated by commas).

Obs.: you MUST set the environment variable AIRFLOW_HOME before running the script, pointing to the root directory of your airflow installation. It normally is at:

/home/<USERNAME>/.local/lib/<PYTHON_FOLDER>/site-packages/airflow

The result is that your plugin will now be imported normally with no errors in any python IDE.

No import errors

The script source code can be found here.

Feel free to access my other repositories, I post a lot of snippets and personal projects that could help you!

Thanks for reading :)