Running below code in AWS glue. Job is able to read the Data from DB but failing while writing.
An error occurred while calling o102.pyWriteDynamicFrame. Command failed with error 301: 'Retryable writes are not supported' on server :. The full response is {"ok": 0.0, "code": 301, "errmsg": "Retryable writes are not supported", "operationTime": {"$timestamp": {"t": 1647921685, "i": 1}}}
Used the catalogue DocumentDB connection in Job Details section
Tried using retryWrite=false in connection string but still getting the error
documentdb_uri = "mongodb://<host name>:27017"
documentdb_write_uri = "mongodb://<host name>:27017"
read_docdb_options = {
"uri": documentdb_uri,
"database": "test",
"collection": "profiles",
"username": "<username>",
"password": "<password>",
"ssl": "true",
"ssl.domain_match": "false"
}
write_documentdb_options = {
"uri": documentdb_write_uri,
"database": "test",
"collection": "collection1",
"username": "<username>",
"password": "<password>",
"ssl": "true",
"ssl.domain_match": "false"
}
# Get DynamicFrame from DocumentDB
dynamic_frame2 = glueContext.create_dynamic_frame.from_options(connection_type="documentdb",
connection_options=read_docdb_options)
# Write DynamicFrame to DocumentDB
glueContext.write_dynamic_frame.from_options(dynamic_frame2, connection_type="documentdb",
connection_options=write_documentdb_options)
job.commit()
Solved it by downgrading Glue version from 3.0 to 2.0. In 3.0 there is no way to set retryWrite setting while using dynamic frame.
A ticket has been created in their board and it was not resolved yet. Issue in AWS board for reference - https://github.com/awslabs/aws-glue-libs/issues/111 [An error occurred while calling o365.pyWriteDynamicFrame. Command failed with error 301: 'Retryable writes are not supported' on server ****.*****.docdb.amazonaws.com:27017.]