pythonhookcredentialsairflowdirected-acyclic-graphs

What is the best way to store login credentials on Airflow?

I found out there are lot of ways to store it as variables, hooks and other ways using encryption. I would like to know what's the best way to do it.

Solution

  • Currently there 2 ways of storing secrests:

    1) Airflow Variables: Value of a variable will be hidden if the key contains any words in (‘password’, ‘secret’, ‘passwd’, ‘authorization’, ‘api_key’, ‘apikey’, ‘access_token’) by default, but can be configured to show in clear-text as shown in the image below.

    enter image description here

    However, there is a known-bug where anyone with an access to UI can export all the variables which will expose the secrets.

    2) Airflow Connections:

    You can use the Passwords field in Airflow connections which will encrypt that field if you had installed the crypto package (pip install apache-airflow[crypto]). The password field would just appear as blank in the UI as shown in the screenshot.

    More on Securing connections: https://airflow.apache.org/howto/secure-connections.html

    enter image description here

    I recommend the 2nd approach as even if someone gets access to the UI, he/she won't be able to get your secrets. Keep in mind though that you need to install the crypto package for this.

    You can then access the secrets as below:

    from airflow.hooks.base_hook import BaseHook

connection = BaseHook.get_connection(CONN_ID)
slack_token = connection.password

    You can set the CONN_ID as the name of your connection.