[SOLVED] OVH Object Storage, nothing happens when I try to upload large file (more than 100 Ko)

OVH Object Storage, nothing happens when I try to upload large file (more than 100 Ko)

I try to upload files with OVH object storage. But I have three different behavior according to the heavy file.

With a weight file which less than 100Ko, everything is ok
With a weight file which more than 100Ko, I have this error: Error: write after end, but the file is uploaded on ovh object storage
With a weight file which more than 250Ko, nothing happens, and the file is not uploaded. The fs ReadStream is open, but the write stream piped (with the read stream) not finish.

This is my code:

  var client = require('pkgcloud').storage.createClient({
    provider: 'openstack',
    username: myusername,
    password: mypassword,
    region: 'GRA',
    authUrl: 'https://auth.cloud.ovh.net/'
  });

  const fsReadStream = fs.createReadStream(path.resolve(__dirname, fileLocation))

  let writeStream = client.upload({
    container: myOvhStorageContainer,
    remote: 'fileName.jpg',
  });

  writeStream.on('error', function (err) {

    console.log(err)
  });

  writeStream.on('success', async function (file) {
    console.log(file)
  });

  fsReadStream.on('open', function () {
    console.log('open!!')

    fsReadStream.pipe(writeStream);
  });

Solution

The problem comes from a bug in pkgcloud in how it streams files for OpenStack Storage.

The solution is exposed, and a fix is proposed, in https://github.com/pkgcloud/pkgcloud/pull/673

There are forks of pkgcloud that include the proposed fix, and they can be used while waiting for the fix to be officially accepted:

To use such a repository in your project/application, edit your package.json to modify the dependencies like this:

"dependencies": {
    …
    "pkgcloud": "https://github.com/madarche/pkgcloud.git#fe2701eb6eb984e3d067d5df7610ca0751528dbd",
    …
  },

You can also simply create your own fork of pkgclould, so you don't have to trust a random Git repository.