I have been using AWS aurora with python lambda function for executing queries for our application. The lambda functions works great, but returns only top 1000 results not all. I tried increasing the limit to 5000 by using paginators, couldn't find an appropriate solution to that :
import boto3
def lambda_handler(event, context):
client = boto3.client('rds-data')
readParam = event['query'] # readParam = 'select * from table;'
database1 = event['database'] # Database name
response = client.execute_sql(
awsSecretStoreArn='arn:aws:secretsmanager:us-east-1:xxxxx:secret:abc/read-XXXX',
database=database1,
dbClusterOrInstanceArn='arn:aws:rds:us-east-1:xxxxx:cluster:abcd-abc',
sqlStatements=readParam
)
return {
'statusCode': 200,
'headers': {
'Content-Type': 'application/json',
"Access-Control-Allow-Origin": "*",
"Access-Control-Allow-Headers": "Content-Type",
"Access-Control-Allow-Methods": "OPTIONS,POST"
},
'body': response
}
`
I have tried using SQLAlchemy and pydataapi and deploying the AWS development package to lambda, that doesn't work. The lambda function doesn't read the appropriate python file having the lambda_handler. The code is as follows:
import pymysql.cursors
from sqlalchemy.engine import create_engine
def lambda_handler(event, context):
readParam = event['query']
database1 = event['database']
engine = create_engine(
'mysql+pydataapi://',
connect_args={
'resource_arn': 'arn:aws:rds:us-east-1:xxxx:cluster:abcd-abc',
'secret_arn': 'arn:aws:secretsmanager:us-east-1:xxxx:secret:abc/read-XXXX',
'database': 'mimic_dev'
}
)
result: ResultProxy = engine.execute(readParam)
return {
'statusCode': 200,
'headers': {
'Content-Type': 'application/json',
"Access-Control-Allow-Origin": "*",
"Access-Control-Allow-Headers": "Content-Type",
"Access-Control-Allow-Methods": "OPTIONS,POST"
},
'body': result.fetchall
}
Is there any better alternate solution to what I have been trying? Any help appreciated. Thank You
This issue has been solved. Resources: AWS lambda deployment Package in Python
And the following code:
import pymysql.cursors
import json
from sqlalchemy.engine import create_engine
def lambda_handler(event,context):
engine = create_engine(
'mysql+pydataapi://',
connect_args={
'resource_arn': 'arn:aws:rds:us-east-1:xxxxx:cluster:xxxx',
'secret_arn': 'arn:aws:secretsmanager:us-east-1:xxxx:secret:decima/abcd-abc',
'database': 'mimic_dev'
}
)
result: ResultProxy = engine.execute("select * from person limit 10000")
resultValue = result.fetchall()
return json.dumps([dict(r) for r in resultValue])