google-cloud-platformgoogle-kubernetes-enginegoogle-cloud-composer

Is it possible to amend environment config for GCP Cloud Composer?


I am trying to use GCP Cloud Composer to connect to SQL Server 2012, encountered this error.

pyodbc.OperationalError: ('08001', '[08001] [Microsoft][ODBC Driver 17 for SQL Server]SSL Provider: [error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol] (-1) (SQLDriverConnect)')

I am trying to implement this fix, would like to ask for help for the method to set up the config above in Cloud Composer.

I have thought about ssh into each container, since Cloud Composer is deployed as GKE cluster, and manually implement the fix, but since Cloud Composer allows for autoscaling, it will not be feasible to manually do this.


Solution

  • To solve this, you have to use Composer correctly. Most of Composer/Airflow user do not use it for what it is designed for.

    Airflow is a great orchestrator, with multiple connector. But it's above all an orchestrator, thus orchestrate! And perform the processing OUTSIDE Airflow, in container for example, running on Cloud Run Job (I patched the operator earlier this year) or directly on GKE (kubernetesPodOperator).

    By doing this, you will avoid a lot of problem with composer when you scale, and you will also solve your database connection error that will be put in a container and not on Airflow workers.