is there a reason why multer won't let me upload files greater than 1MB?
import { Request, Response } from 'express';
import { v4 as uuidv4 } from 'uuid';
import multer from 'multer';
import multerS3 from "multer-s3"
import s3 from "../config/amazons3"
const uploadController = {
transferFile: async (req, res) => {
try {
const s3upload = multer({
storage: multerS3({
s3: s3,
acl: 'public-read',
bucket: 'filetransferawsbucket1',
metadata: (req, file, cb) => {
cb(null, {fieldName: file.fieldname})
},
key: (req, file, cb) => {
try {
console.log("File", file)
const key: string = `medium_zip_file_${uuidv4()}`
cb(null, key)
} catch (error) {
cb(error) // Pass the error to multer
}
}
}),
limits: {
fileSize: 5 * 1024 * 1024 * 1024, // 5 GB limit
fieldSize: 52428800
},
fileFilter: (req, file, cb) => {
if (file.mimetype === 'application/zip') {
// Allow only zip files
cb(null, true)
} else {
cb(new Error('Invalid file type. Only zip files are allowed.'))
}
},
})
console.log('Before Multer Middleware')
s3upload.single('file')(req, res, (err) => {
console.log('After Multer Middleware')
if (err) {
console.log('Multer error:', err)
return res.status(500).json({ error: "Error processing file upload" })
}
})
res.status(201).send("Success")
} catch (error) {
console.log("TRANSFER FILE ERROR", error)
res.status(500).json({ error: "Error transferring file" })
}
}
}
If I run it with files <1MB they are successfully uploaded to my aws s3 bucket. But when I attempt uploading files >1MB it doesn't run this block of code. ( s3upload.single('file') works but the bit after doesn't )
s3upload.single('file')(req, res, (err) => {
console.log('After Multer Middleware')
if (err) {
console.log('Multer error:', err)
return res.status(500).json({ error: "Error processing file upload" })
}
})
It doesn't give an error message or anything (I get a response of 201 success). I've tried adjusting the CORS settings and the bucket policy but that's not worked.
When I upload a 1.2MB file using aws cli it works. Just can't seem to get it to work with node js using multer-s3
UPDATE:
I have moved some functions around and it seems to be working now. This is the updated code:
uploadRouter.ts
import express from "express"
import uploadController, { s3upload } from "../controllers/uploadContoller"
const router = express.Router()
router.post("/transferFile", s3upload.single("file"), uploadController.transferFile)
export default router
uploadController.ts (original file with uploadS3 moved into uploadRouter.ts)
import { Request, Response } from 'express';
import { TransferModel } from "../models/transferModel";
import { v4 as uuidv4 } from 'uuid';
import multer from 'multer';
import multerS3 from "multer-s3"
import s3 from "../config/amazons3"
interface IUploadContoller {
transferFile: (req: Request, res: Response) => Promise<void>
transferFileBusBoy?: (req: Request, res: Response) => Promise<void>
}
export const s3upload = multer({
storage: multerS3({
s3: s3,
acl: 'public-read',
bucket: 'filetransferawsbucket1',
metadata: (req, file, cb) => {
cb(null, {fieldName: file.fieldname})
},
key: (req, file, cb) => {
try {
console.log("***FILE***", file)
const key: string = `medium_zip_file_${uuidv4()}` // replace this with uuid -> https://filetransferawsbucket1.s3.eu-west-2.amazonaws.com/{uuid}
cb(null, key)
} catch (error) {
cb(error) // Pass the error to multer
}
}
}),
limits: {
fileSize: 5 * 1024 * 1024 * 1024, // 5 GB limit
fieldSize: 5 * 1024 * 1024 * 1024,
},
fileFilter: (req, file, cb) => {
if (file.mimetype === 'application/zip') {
// Allow only zip files
cb(null, true)
} else {
cb(new Error('Invalid file type. Only zip files are allowed.'))
}
},
})
const uploadController: IUploadContoller = {
transferFile: async (req: Request, res: Response) => {
try {
console.log('Before Multer Middleware')
res.status(201).send("Success")
} catch (error) {
console.log("TRANSFER FILE ERROR", error)
res.status(500).json({ error: "Error transferring file" })
}
},
}
export default uploadController
My guess is when I moved the s3upload.single("file") middleware directly into the route definition, it could have ensured that the file was processed and uploaded to S3 before proceeding with the rest of the route logic(?)
As a side note: it's also working for files of ~1GB but its taking quite long (up to 5 mins) - any suggestions on how to speed it up would be welcome
My guess is when I moved the s3upload.single("file") middleware directly into the route definition, it could have ensured that the file was processed and uploaded to S3 before proceeding with the rest of the route logic(?)
I'm not sure how your middlewares were set up before the rewrite, but you are mixing async, sync and callbacks, all within a single function.
Your controller method is async, but your code that is uploading S3 is calling these two methods:
s3upload.single('file')(req, res, (err) => {
console.log('After Multer Middleware')
if (err) {
console.log('Multer error:', err)
return res.status(500).json({ error: "Error processing file upload" })
}
});
res.status(201).send("Success");
and returns right away, without so much as an await.
Since you only have one shot at setting a response status in ExpressJS, this function will only ever send a 201 Success.
Technically, the closed response should not affect callbacks, but since multer is a middleware as well, it might be smart and not bother to call the callbacks if it sees that the request is closed.
If you are using the async pattern, make sure to wrap your controllers in express-async-handler and actually await for the results of your uploads and other asynchronous operations.
If you want to use callbacks or middlewares, make sure to stack them in the right way (it looks like you did that already), or move your processing logiс inside the callbacks:
s3upload
.single('file')(req, res,
(err) => err
? req.status(500).send("Error")
: req.status(201).send("Success")))
, so that the 201 only gets sent after the upload has completed.