I am using Apify Client to scrape information in a cron job written in airflow dag. When I try to execute the task file manually, it works fine, but when I do so from the airflow UI, it throws this error. I tried to upgrade the version of httpx to 0.24 but Apify client support 0.23. I don't know if this is the issue with Apify or Airflow
APPIFY_CLIENT = ApifyClient(config('APIFY_CLIENT_TOKEN'))
File "/usr/share/pyshared/venv/airflow/lib/python3.8/site-packages/apify_client/client.py", line 126, in __init__
self.http_client = _HTTPClient(
File "/usr/share/pyshared/venv/airflow/lib/python3.8/site-packages/apify_client/_http_client.py", line 59, in __init__
self.httpx_client = httpx.Client(headers=headers, follow_redirects=True, timeout=timeout_secs)
TypeError: __init__() got an unexpected keyword argument 'follow_redirects'
You probably have some old version of httpx installed in the environment Airflow uses - the follow_redirects argument is supported since httpx version 0.20.0: https://github.com/encode/httpx/releases/tag/0.20.0.
Try updating Airflow along with its dependencies - Airflow updated httpx to 0.20.0 in its dependency constraints October 2021: https://github.com/apache/airflow/commit/7cda696975aaa271b5e941173ca754ea6505b95b