node.jsaws-lambdastreamurllibdailymotion-api

Upload a file from URL to another server


I'm trying to upload a public video file from Amazon S3 to Dailymotion using a NodeJS Lambda function. AWS Lambda functions have a storage limitation: I can't download the whole video file locally on the serverless function and then send it to Dailymotion.

I searched for a way to read the file on parts and send them with HTTPS requests to Dailymotion, and I found the urllib library that seems helpful!

To upload a video file to Dailymotion, you can send a POST request to an UPLOAD_URL that should be generated using another authenticated request:

curl -X POST \
     -F 'file=@/path/to/your/video.mp4' \
     '<UPLOAD_URL>'

The steps are clearly mentioned on this document page.

What I need is a way to stream the file from a URL like this https://cdn.example.com/videos/filename.mp4 to Dailymotion through my Lambda function.

Anyone with good experience can help with a clear example of how to do it.


Solution

  • I found a solution using request and form-data libraries. Just:

    1. Created a new form.
    2. Appended the file to the form as a stream using request.
    3. Submit the form.
    4. Wait for the upload to end to send the callback back to the caller with the response.

    Here's a working copy of my Lambda function code:

    import request from 'request';
    import FormData from 'form-data';
    
    export const handler = (event, context, callback) => {
      const form = new FormData();
      form.append('file', request('https://cdn.mysite.com/video.mp4'));
      form.submit({
        host: 'upload-XX.dcX.dailymotion.com',
        path: '/upload?uuid=XXXXX&seal=XXXXX&extra=XXXXX'
      }, (error, response) => {
        if (error) throw error;
        let body = '';
        response.on('data', chunk => {body += chunk.toString()});
        response.on('end', () => {callback(null, JSON.parse(body))});
        response.resume();
      });
    };
    

    Update

    Another solution using the https native module of Nodejs instead of the deprecated request library:

    import https from 'https';
    import FormData from 'form-data';
    
    export const handler = (event, context, callback) => {
      const form = new FormData();
      https.get('https://cdn.mysite.com/video.mp4', response => {
        form.append('file', response);
        form.submit(
          {
            host: 'upload-XX.dcX.dailymotion.com',
            path: '/upload?uuid=XXXXX&seal=XXXXX&extra=XXXXX'
          },
          (error, response) => {
            if (error) callback(error);
            let body = '';
            response.on('data', chunk => body += chunk.toString());
            response.on('end', () => callback(null, JSON.parse(body)));
            response.resume();
          }
        );
      });
    };