mongodbairflowlocal-database

Connect query local mongodb database from dag airflow


question from beginner in Airflow:

I can't query my local mongodb database from a task in a DAG airflow.

the dag is correctly imported in Airflow but when i start it the task fail because of timeout servor: enter image description here

Here the connection I set into Airflow: enter image description here

Here my pyhton function to initialize code:

from airflow import DAG
from airflow.utils.dates import days_ago
from airflow.operators.python import PythonOperator
from airflow.providers.mongo.hooks.mongo import MongoHook
from pymongo.mongo_client import MongoClient

def connecter():
    hook = MongoHook(mongo_conn_id='mongo_default')
    client = hook.get_conn()
    my_db = client['Immo']
    return my_db


mydag = DAG(
    dag_id='my_dag',
    default_args={
        'owner': 'airflow',
        'start_date': days_ago(0),
    }
)

def get_one(c: str):

    a=connecter().find({"id_transaction": c})[0]['_id_transaction']
    print(a)
    return a

task_1_0 = PythonOperator(
    task_id='test_connection_my_bd',
    python_callable=db.get_one,
    dag=mydag,
    op_kwargs={'c': 107332}
)

I run airflow inside containers using the docker-compose built by the community airflow: https://airflow.apache.org/docs/apache-airflow/stable/docker-compose.yaml

When I run my function get_one(c:str) outside of dag airflow with the function connecter_local it works fine:

def connecter_local():
    uri= "mongodb://localhost:27017"
    client = MongoClient(uri)
    my_db = client['Immo']
    return my_db

def get_one_local(c: str):

    a=connecter_local().find({"id_transaction": c})[0]['_id_transaction']
    print(a)
    return a

print(get_one_local('1'))

Thanks in advance.


Solution

  • Your airflow is running on a docker and the mongo-db on your machine (not inside the airflow docker).

    when you configure host=127.0.0.1 its inside the docker (internal network) and its not the host ip. you should change host to your computer-name or ip