Fix uploading error on @aws-sdk/client-s3 on nestjs

I’m trying to upload attachments to Amazon S3 using the AWS SDK v3 (@aws-sdk/client-s3) within a NestJS middleware, but I keep getting this timeout error:

RequestTimeout: Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.
It happens after a few seconds, and I’m not sure why. I’ve tried using different AWS accounts, adjusting package versions, and verifying my credentials, but none of that has solved it.

Here’s my middleware code (s3-upload.middleware-factory.ts):

import { BadRequestException } from '@nestjs/common';
import { Request, Response, NextFunction } from 'express';
import * as multer from 'multer';
import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3';

const s3Client = new S3Client({
  region: process.env.AWS_REGION,
  credentials: {
    accessKeyId: process.env.AWS_ACCESS_KEY,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
  },
});

interface S3UploadMiddlewareOptions {
  fields: { name: string; maxCount?: number }[];
  fileFilter?: (
    req: Request,
    file: Express.Multer.File,
    cb: multer.FileFilterCallback,
  ) => void;
}

/**
 * A factory that returns an Express middleware for handling file uploads
 * with Multer (.fields) and then uploading each file to S3.
 *
 * @param fields  An array of { name, maxCount } for Multer .fields()
 * @returns       An Express middleware function
 */
export function createS3UploadMiddleware(options: S3UploadMiddlewareOptions) {
  console.log('inside createS3UploadMiddleware');
  console.log('options.fields', options.fields);
  console.log('options.fileFilter', options.fileFilter);

  const upload = multer({
    storage: multer.memoryStorage(),
    fileFilter: options.fileFilter ?? ((req, file, cb) => cb(null, true)),
  });

  return async (req: Request, res: Response, next: NextFunction) => {
    console.log('middleware called');
    upload.fields(options.fields)(req, res, async (err: any) => {
      console.log('upload.fields called');
      if (err) {
        console.log('error', err);
        return next(err);
      }

      console.log('req.files', req.files);
      if (!req.files || Object.keys(req.files).length === 0) {
        return next();
      }

      const filesMap = req.files as Record<string, Express.Multer.File[]>;
      const allFiles: Express.Multer.File[] = Object.values(filesMap).flat();

      const folderName = process.env.AWS_FOLDER_NAME || 'default-folder';

      try {
        console.log('uploading files to s3');
        const uploadPromises = allFiles.map(async (file) => {
          const key = `${folderName}/${Date.now()}-${file.originalname}`;
          await s3Client.send(
            new PutObjectCommand({
              Bucket: process.env.AWS_BUCKET_NAME,
              Key: key,
              Body: file.buffer,
              ContentType: file.mimetype,
            }),
          );

          const fileUrl = `https://${process.env.AWS_BUCKET_NAME}.s3.${process.env.AWS_REGION}.amazonaws.com/${key}`;
          return {
            fieldname: file.fieldname,
            originalname: file.originalname,
            mimetype: file.mimetype,
            size: file.size,
            location: fileUrl,
          };
        });

        const uploadedFiles = await Promise.all(uploadPromises);
        console.log('uploadedFiles', uploadedFiles);

        req.body.uploadedFiles = uploadedFiles;
        next();
      } catch (uploadErr) {
        console.log('upload error', uploadErr);
        return next(uploadErr);
      }
    });
  };
}

What I’ve tried so far:

Verified AWS credentials in my .env file, even switching to a different AWS account. Double-checked my region and bucket name (us-east-2 for Ohio, for example). Tested with smaller/larger files. Tried different package versions of @aws-sdk/client-s3. Despite these efforts, I still get a RequestTimeout after a few seconds. Has anyone else encountered a similar error when uploading to S3 with the AWS SDK v3 in NestJS or Express?

Any suggestions on what else I can check or do to debug this? I’d really appreciate any pointers.

Solution

OK I was likely able to solve it on our side (at least it looks like it, but need to wait 1-2 days if it happens again).

Reason is more or less described in this issue: https://github.com/aws/aws-sdk-js-v3/issues/6763

tldr:

in SDK V2

new S3({
  httpOptions: {
    timeout: 10 * 1000,
    connectTimeout: 10 * 1000,
  }
});

was used to configure the timeouts of the S3 client.

This was somehow supported for some time also in SDK V3 but suddenly was not supported anymore (around version 3.709 somewhere).

The correct way now is to configure the timeouts via

new S3({
  requestHandler: {
    requestTimeout: 10 * 1000,
    connectTimeout: 10 * 1000,
  }
});

in the S3 client.