node.jshttp-streaming

Why does the request pipe includes the headers in this code?


I have a strange situation regarding http server and piping request.

From my past experience, when piping the request object of a http server to a writable stream of some sort, it does not include the headers, just the payload.

Today however, I wrote some very simple code, and from some reason, I'm spending the past 2 hours trying to figure out why it writes the headers to the file (super confusing!)

Here's my code:

server = http.createServer((req, res) => {
  f = '/tmp/dest'
  console.log(`writing to ${f}`)
  s = fs.createWriteStream(f)
  req.pipe(s)
  req.on('end', () => {
    res.end("done")
  })
})

server.listen(port)

I test this with the following curl command:

curl -XPOST  -F 'data=@test.txt' localhost:8080

And this is what I'm getting when I'm reading /tmp/dest:

--------------------------993d19e02b7578ff
Content-Disposition: form-data; name="data"; filename="test.txt"
Content-Type: text/plain

hello - this is some text


--------------------------993d19e02b7578ff--

Why am I seeing the headers here? I expected it to only write the payload

I have a code I wrote about a year ago that streams directly to a file without the headers, I don't understand what's different, but this one did the trick:

imageRouter.post('/upload', async(req, res) => {
  if(!req.is("image/*")) {
    let errorMessage = `the /upload destination was hit, but content-type is ${req.get("Content-Type")}`;
    console.log(errorMessage);
    res.status(415).send(errorMessage);
    return;
  }

  let imageType = req.get("Content-Type").split('/')[1];
  let [ err, writeStream ] = await getWritableStream({ suffix: imageType });
  if (err) {
    console.log("error while trying to write", err);
    return res.status(500).end();
  }

  let imageName = writeStream.getID();
  req.on('end', () => {
    req.unpipe();
    writeStream.close();
    res.json({
      imageRelativeLink: `/images/${imageName}`,
      imageFullLink: `${self_hostname}/images/${imageName}`
    });
  });
  req.pipe(writeStream);
});

What's different? Why does my code from a year ago (last block) writes without the form-data/headers? The resulting file is only an image, without text, but this time (the first block) shows http headers in the resulting file


Solution

  • As it turns out, I had a mistake in my understanding, and therefore made a mistake in my question.

    What I thought were the headers, were actually http multipart specification. This is how curl uploads a file when used with this syntax.

    What I actually needed was to change the way I test my code with curl to one of the following:

    cat /path/to/test/file | curl -T - localhost:8080
    # or
    curl -T - localhost:8080 < /path/to/test/file
    # or
    curl -T /path-/to/test/file localhost:8080 < /path/to/test/file
    

    Using the -T (or --upload-file) flag, curl uploads the file (or stdin) without wrapping it in an http form.