My API needs to handle file uploads as well as attached files in emails. Emails are routed to my API via the mail parse function at Sendgrid.
Both methods results in a multipart form. In the email case, fields are "to", "from", etc. In the ordinary upload form I decide the fields.
I don't want to handle local temp files, so ideally I'll just pipe the file streams to S3. I have implemented this and it works great. But there are edge cases..
Before I pipe the files to S3 I want to ensure that the email is legit. I do this by checking both the from and to addresses. Unfortunately multipart forms does not specify a specific order for fields/files, rather the spec says to preserve order of original form.
Problem
Occationally I have received emails from Sendgrid parse where the fields arrive AFTER the files.
So, in order to validate the sender via the form fields I might need to buffer the file while waiting for the field data. This seems to be a quite stupid way of handling incoming files, i.e. start to process/store files before I even know if I should handle them.
All emails sent to a specified subdomain will be forwarded from Sendgrid to my server. This means that I could potentially be flooded with emails which I'd have to handle completely (including buffering the files) before I can decide if the email is legit.
Question
Have I misunderstood something?
Some email clients postpones downloading of attachments. How is this done?
The tools I'm using
node.js
multiparty
Sendgrid Parse
At last I grasped this. The simple answer is that a multipart form is a single stream. If files comes before fields in the stream I have to buffer the files while awaiting the fields. At least if I need the information from fields in order to process the files.
I guess that email clients which are able to postpone downloading of attachments must rely on a server that parses the email and buffers the files.
Thank you Doug Wilson at https://github.com/pillarjs/multiparty for clarifying this.