I am using airflow 1.10.10 with Vault and I am trying to retrieve variables stored in Vault. I am able to retrieve connections but not variables.
The airflow configuration file contains:
airflow.cfg
[secrets]
backend = airflow.contrib.secrets.hashicorp_vault.VaultBackend
backend_kwargs = {"connections_path": "connections", "variables_path": "variables", "mount_point": "secret", "url": "http://127.0.0.1:8200", "auth_type":"token", "token":"token"}
I run the following command on vault side:
vault login token=token
vault secrets enable -path=secret -version=2 kv
vault kv put secret/connections/smtp_default conn_uri=smtps://user:host@relay.example.com:465
vault kv put secret/variables/auth_KEY_vault test=test123
vault secrets list
vault auth list
I run the following dag to test the integration of connections and variables:
from airflow import DAG
from airflow.models import Variable
from airflow.operators.python_operator import PythonOperator
from datetime import datetime
from airflow.hooks.base_hook import BaseHook
def get_secrets():
conn = BaseHook.get_connection('smtp_default')
var2 = Variable.get("auth_KEY_vault")
print(var2)
print(f"Password: {conn.password}, Login: {conn.login}, URI: {conn.get_uri()}, Host: {conn.host}")
with DAG('example_secrets_dags', start_date=datetime(2020, 1, 1), schedule_interval=None) as dag:
test_task = PythonOperator(
task_id='test-task',
python_callable=get_secrets
)
Which result to the following error:
*** Reading local file: /usr/local/airflow/logs/example_secrets_dags/test-task/2022-03-25T07:42:44.764744+00:00/1.log
[2022-03-25 07:42:54,964] {{taskinstance.py:669}} INFO - Dependencies all met for <TaskInstance: example_secrets_dags.test-task 2022-03-25T07:42:44.764744+00:00 [queued]>
[2022-03-25 07:42:55,018] {{taskinstance.py:669}} INFO - Dependencies all met for <TaskInstance: example_secrets_dags.test-task 2022-03-25T07:42:44.764744+00:00 [queued]>
[2022-03-25 07:42:55,022] {{taskinstance.py:879}} INFO -
--------------------------------------------------------------------------------
[2022-03-25 07:42:55,023] {{taskinstance.py:880}} INFO - Starting attempt 1 of 1
[2022-03-25 07:42:55,024] {{taskinstance.py:881}} INFO -
--------------------------------------------------------------------------------
[2022-03-25 07:42:55,089] {{taskinstance.py:900}} INFO - Executing <Task(PythonOperator): test-task> on 2022-03-25T07:42:44.764744+00:00
[2022-03-25 07:42:55,098] {{standard_task_runner.py:53}} INFO - Started process 1132 to run task
[2022-03-25 07:42:55,439] {{logging_mixin.py:112}} INFO - Running %s on host %s <TaskInstance: example_secrets_dags.test-task 2022-03-25T07:42:44.764744+00:00 [running]> 5d5d845cb830
[2022-03-25 07:42:55,547] {{logging_mixin.py:112}} INFO - [2022-03-25 07:42:55,546] {base_hook.py:87} INFO - Using connection to: id: smtp_default. Host: relay.example.com, Port: 465, Schema: , Login: user, Password: XXXXXXXX, extra: None
[2022-03-25 07:42:55,573] {{taskinstance.py:1145}} ERROR - 'Variable auth_KEY_vault does not exist'
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 983, in _run_raw_task
result = task_copy.execute(context=context)
File "/usr/local/lib/python3.6/site-packages/airflow/operators/python_operator.py", line 113, in execute
return_value = self.execute_callable()
File "/usr/local/lib/python3.6/site-packages/airflow/operators/python_operator.py", line 118, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "/usr/local/airflow/dags/vault_integration_test/vault_integration.py", line 10, in get_secrets
var2 = Variable.get("auth_KEY_vault")
File "/usr/local/lib/python3.6/site-packages/airflow/models/variable.py", line 118, in get
raise KeyError('Variable {} does not exist'.format(key))
KeyError: 'Variable auth_KEY_vault does not exist'
[2022-03-25 07:42:55,586] {{taskinstance.py:1202}} INFO - Marking task as FAILED.dag_id=example_secrets_dags, task_id=test-task, execution_date=20220325T074244, start_date=20220325T074254, end_date=20220325T074255
[2022-03-25 07:43:04,875] {{logging_mixin.py:112}} INFO - [2022-03-25 07:43:04,872] {local_task_job.py:103} INFO - Task exited with return code 1
Any idea about how can I get the variables from vault please ?
Thanks in advance for your help
I found the error, and while adding the variables you must respect the value parameter. So instead of adding a variable that way
vault kv put secret/variables/auth_KEY_vault test=test123
you should pass value to the value instead of test
vault kv put secret/variables/auth_KEY_vault value=test123