In Airflow, you're suppose to raise an AirflowException if you want a task to be marked as a failure. But the raised error doesn't seem to be caught in the top-level Airflow module, and so it results in the entire stacktrace being dumped into the logs. If you do your error handling properly, it should be possible to print a helpful error message and then fail the task, without gumming up the logs. How can this be done?
Yes, you can do it by using Python’s exception chaining syntax. Instead of writing:
raise AirflowException("Your error message")
you write:
raise AirflowException("Your error message") from None
This “from None” tells Python not to include the exception’s context (i.e. the chained traceback) in the logs. So, when your task fails, you’ll just see your helpful error message instead of a long traceback, making the logs much cleaner.
For example:
from airflow.exceptions import AirflowException
def some_task():
try:
# some code that might raise an error
pass
except Exception as e:
# Log a neat error message
error_msg = "Something went wrong, please check your configuration."
# Now raise the AirflowException without the full traceback
raise AirflowException(error_msg) from None