pythonairflowmwaa

Python doesn't see added module to the same dir


I added to my utils dir a new file that contains dict variable which I want to import into another file.

mapping.py

config = [
    {
        "filter_column": "SHOW_NAME",
        "horizons": [
            3,
            4,
            5,
            6
        ],
        "windows": [
            52
        ],
        "metrics": [
            "mean",
            "std",
            "min",
            "max",
            "quantile_25",
            "quantile_75",
            "trend"
        ]
    },.....
  ]

On deploy, my airflow stage2.py dag throws an error:

ModuleNotFoundError: No module named 'utils'

Odd things is that env_config is recognized

Folder structure

README.rst
LICENSE
...
dags/utils/env_config.py
dags/utils/mapping.py
dags/utils/__init__.py
dags/stage2.py
...

stage2.py

import os
import sys
from datetime import datetime, timedelta

from airflow import DAG
from airflow.operators.dummy_operator import DummyOperator
from airflow.providers.amazon.aws.operators.glue import GlueJobOperator
from airflow.utils.task_group import TaskGroup


sys.path.insert(0, os.path.abspath(os.path.dirname(__file__)))

from utils.env_config import EnvironmentConfig  # noqa: E402
from utils.mapping import * # returns error


config = EnvironmentConfig(__file__)

Solution

  • Will leave tihs solution even if it's trivial.

    1. Restart youre IDE
    2. push your changes to Airflow's env and dag directory
    3. Give Airflow few minutes to upload and refresh