I added to my utils
dir a new file that contains dict
variable which I want to import into another file.
mapping.py
config = [
{
"filter_column": "SHOW_NAME",
"horizons": [
3,
4,
5,
6
],
"windows": [
52
],
"metrics": [
"mean",
"std",
"min",
"max",
"quantile_25",
"quantile_75",
"trend"
]
},.....
]
On deploy, my airflow stage2.py
dag throws an error:
ModuleNotFoundError: No module named 'utils'
Odd things is that env_config
is recognized
Folder structure
README.rst
LICENSE
...
dags/utils/env_config.py
dags/utils/mapping.py
dags/utils/__init__.py
dags/stage2.py
...
stage2.py
import os
import sys
from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.dummy_operator import DummyOperator
from airflow.providers.amazon.aws.operators.glue import GlueJobOperator
from airflow.utils.task_group import TaskGroup
sys.path.insert(0, os.path.abspath(os.path.dirname(__file__)))
from utils.env_config import EnvironmentConfig # noqa: E402
from utils.mapping import * # returns error
config = EnvironmentConfig(__file__)
Will leave tihs solution even if it's trivial.
push
your changes to Airflow's env and dag
directory