pythonamazon-web-servicesamazon-s3aws-lambdaterraform-provider-aws

Get files from S3 with lambda

I'm trying to retrieve files from an AWS S3 bucket using a Lambda function, but my script keeps timing out, and I can't figure out why. "errorMessage": "2025-05-16T14:37:13.093Z fdb6***4165 Task timed out after 30.03 seconds"

The code I'm using is a basic script I got from the Doc:

import boto3

def lambda_handler(event, context):
    bucket_name = "<myS3_bucket"  
    file_key = "<path_to/file.csv>" 

    s3 = boto3.resource('s3')
    for bucket in s3.buckets.all():
        print(bucket.name)

I create my Lambda with terraform, here is my Lambda policy:

resource "aws_iam_role" "lambda_role" {
  name = "${var.lambda_name}_role"
  assume_role_policy = jsonencode({
    Statement = [
        {
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = {
        Service = "lambda.amazonaws.com"
      }
    }]
  })
}

resource "aws_iam_role_policy_attachment" "S3_read_only" {
  role       = aws_iam_role.lambda_role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonS3FullAccess"
}
resource "aws_iam_role_policy_attachment" "lambda_logs" {
  role       = aws_iam_role.lambda_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}
resource "aws_iam_role_policy_attachment" "lambda_vpc_access" {
  role       = aws_iam_role.lambda_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole"
}

Security group:

data "aws_vpc" "default" {
  default = true
}
resource "aws_security_group" "lambda_sg" {
  name        = "sg_${var.lambda_name}"
  description = "Allow all the ports needed for lambda"
  vpc_id      = data.aws_vpc.default.id

  # allow all inbound traffic
  ingress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  # allow all outbound traffic
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}
}

I also add bucket policy:

resource "aws_s3_bucket_policy" "lambda_s3_access_policy" {
  bucket = "superset-dockerfiles" 
  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect = "Allow",
        Principal = {
          AWS = aws_iam_role.lambda_role.arn
        },
        Action = [
          "s3:GetObject",
          "s3:ListBucket"
        ],
        Resource = [
          "arn:aws:s3:::superset-dockerfiles",
          "arn:aws:s3:::superset-dockerfiles/*"
        ]
      }
    ]
  })
}

What am i missing ?

Solution

  • A Lambda function in a VPC does not have Internet access, unless you have configured the Lambda function to only run in private subnets that have route to a NAT Gateway. The S3 API exists on the Internet, and your Lambda function running in your VPC currently does not have a network route to connect to the S3 API.

    You have the following options: