I have a strange situation regarding http server and piping request.
From my past experience, when piping the request object of a http server to a writable stream of some sort, it does not include the headers, just the payload.
Today however, I wrote some very simple code, and from some reason, I'm spending the past 2 hours trying to figure out why it writes the headers to the file (super confusing!)
Here's my code:
server = http.createServer((req, res) => {
f = '/tmp/dest'
console.log(`writing to ${f}`)
s = fs.createWriteStream(f)
req.pipe(s)
req.on('end', () => {
res.end("done")
})
})
server.listen(port)
I test this with the following curl
command:
curl -XPOST -F 'data=@test.txt' localhost:8080
And this is what I'm getting when I'm reading /tmp/dest
:
--------------------------993d19e02b7578ff
Content-Disposition: form-data; name="data"; filename="test.txt"
Content-Type: text/plain
hello - this is some text
--------------------------993d19e02b7578ff--
Why am I seeing the headers here? I expected it to only write the payload
I have a code I wrote about a year ago that streams directly to a file without the headers, I don't understand what's different, but this one did the trick:
imageRouter.post('/upload', async(req, res) => {
if(!req.is("image/*")) {
let errorMessage = `the /upload destination was hit, but content-type is ${req.get("Content-Type")}`;
console.log(errorMessage);
res.status(415).send(errorMessage);
return;
}
let imageType = req.get("Content-Type").split('/')[1];
let [ err, writeStream ] = await getWritableStream({ suffix: imageType });
if (err) {
console.log("error while trying to write", err);
return res.status(500).end();
}
let imageName = writeStream.getID();
req.on('end', () => {
req.unpipe();
writeStream.close();
res.json({
imageRelativeLink: `/images/${imageName}`,
imageFullLink: `${self_hostname}/images/${imageName}`
});
});
req.pipe(writeStream);
});
What's different? Why does my code from a year ago (last block) writes without the form-data/headers? The resulting file is only an image, without text, but this time (the first block) shows http headers in the resulting file
As it turns out, I had a mistake in my understanding, and therefore made a mistake in my question.
What I thought were the headers, were actually http multipart specification. This is how curl
uploads a file when used with this syntax.
What I actually needed was to change the way I test my code with curl to one of the following:
cat /path/to/test/file | curl -T - localhost:8080
# or
curl -T - localhost:8080 < /path/to/test/file
# or
curl -T /path-/to/test/file localhost:8080 < /path/to/test/file
Using the -T
(or --upload-file
) flag, curl
uploads the file (or stdin) without wrapping it in an http form.