Unable to create new SSH connection in Apache Airflow (AWS managed MWAA)
I followed the AWS' airflow tutorial here for creating an SSH connection using the Airflow SSHOperator. However, after carefully following the steps, when I go to the UI and go to Admin->Connections->Add a New Connection the connection type SSH is not populating in the drop-down list.
I initially thought this had something to do with my requirements.txt file which I copied from the tutorial. The SSH example DAG did not render because of an import error and this was fixed by changing the below outdated requirements provided by the MWAA tutorial:
-c https://raw.githubusercontent.com/apache/airflow/constraints-2.2.2/constraints-3.7.txt
apache-airflow-providers-ssh
This was not compatible with my Airflow version 2.6.3 so I changed it to the below:
-c https://raw.githubusercontent.com/apache/airflow/constraints-2.6.3/constraints-3.10.txt
apache-airflow-providers-ssh
But even after more playing around with the requirements.txt file and getting the SSH DAG to render I still have no option available for SSH when creating a new connection in the GUI.
Other things I've tried: (1) waiting several hours for the Airflow environment to update, (2) booting up a new Airflow environment with the requirements available from the start and (3) rolling back to 2.4.3. The DAG rendering works but getting that SSH connection option to show up does not :(
So it turns out that when you're setting up Airflow MWAA with the "private" option it does not install pip dependencies on the webserver layer. This means that although the scheduler and workers have dependencies, the webserver is not able to download the SSH provider so it will not show up in the UI. You can get around this by packaging all of your own dependencies and making them available on S3 or you can reprovision your airflow instance as Public.
It is strange that this occurs as it is not mentioned in the tutorials AWS provides around setting up the SSH operator. I found this information here: Airflow webserver not installing python requirements
- WSGIServerException: [Errno 8] nodename nor servname provided, or not known
- Python/Flask Login form throws 500 error on IIS
- Matplotlib plt.show() isn't showing graph
- How to avoid duplicate processing in python API server?
- Extract points/coordinates from a polygon in Shapely
- How do I detect collision in pygame?
- Typehint a method that returns new instance using a superclass classmethod
- How to process requests from multiiple users using ML model and FastAPI?
- How can I clear the interpreter console?
- What are the reasons for the differences in output results when using the groupby function in the Python pandas package?
- How to antialias picture to get rid of the jerkiness in the image
- Is there any difference between list(array) and array.tolist() in Python?
- How to move x-axis on top of the plot in plotnine?
- Pulling Zillow Rent Data from Zillow API
- Safest way to incrementally append to a file?
- Unable to install mysqlclient package on EC2 instance
- FastAPI - How to get app instance inside a router?
- SQLAlchemy: engine, connection and session difference
- Generating binary arrays with alternating values based on change indices in NumPy
- How can i solve this error: ValueError: The feature names should match those that were passed
- 'super' object has no attribute '__sklearn_tags__'
- Python: How to copy all attibutes from base class to derived one
- mitm proxy is closed, but the port is still occupied
- How to find pg_config path
- Why does "pip install" inside Python raise a SyntaxError?
- Plotting a matplotlib pie chart using 2 columns
- ImportError: No module named langchain.llms
- How do I disable the security certificate check in Python requests
- How can I import Psycopg2 into a Lambda Function on AWS?
- Django built in Logout view `Method Not Allowed (GET): /users/logout/`