[SOLVED] AWS Glue and SSL connection

AWS Glue and SSL connection

I'm using AWS Glue to write data into PostgreSQL. I'm using DynamicFrame to do it and I write using Glue Catalog. But I face with “SSL error: handshake_failure” when writing.

But when I testes connection I created in Glue Console (without specifying any SSL details) it shows that the test was successful. Even Glue Crawler was successful.

But when I start the job that writes data into PostgreSQL it fails due to “handshake_failure”. I tried to add different JDBC string parameters to avoid it (like ssl=true&sslmode=allow, etc), added the path to certificate in S3 bucket, but it didn’t help.

Could someone help with this issue?

Solution

I resolved this issue using next approach.

Glue JDBC connection setup: Require SSL connection - false

Connection URL string - "jdbc:postgresql://host-name.rds.amazonaws.com:5432/database-name?ssl=true&sslmode=allow"

When writing to database I used next approach:

PostgreSQLtable_node3 = glueContext.write_dynamic_frame.from_options(
    frame=ChangeSchema_node2,
    connection_type="postgresql",
    connection_options={
    "useConnectionProperties": "true", 
    "connectionName": "postgresql_prod",
    "dbtable": f"{landing_schema}.{table}",
    "bulksize": 10
},
    transformation_ctx="PostgreSQLtable_node3",
)

Also, this issue would be possible to resolve just using native spark ways to push data into PostgreSQl.