javascriptnode.jstypescriptamazon-web-servicesamazon-s3

invalid checksum when using multipart upload with FULL_OBJECT checksum in NodeJS


Problem

I am trying to upload a file into AWS (since I am testing I am using a dockerized minio).

  1. I am calculating the checksum of the file like this:
    const fileContent = await readFile(fileName, {
      encoding: "binary",
    });
    const checksum = checksums.crc32(fileContent);
    
  2. Then I create a multipart upload, this part of my code.
  3. I store the UploadId for future use here.
  4. Then I start uploading each chunk inside a for loop.
  5. I also store all the parts responses in an array (here).
  6. Finally I try to complete the whole file upload by sending a CompleteMultipartUploadCommand.

If I remove line 77 and 78:

ChecksumType: "FULL_OBJECT",
ChecksumCRC32: checksum.toString(),

It will upload the file but that is not what I want.

Desired Outcome

  1. I want send the calculated checksum of the enter file (from the first byte to the last byte) to AWS S3 when I am sending the CompleteMultipartUploadCommand.
  2. So that AWS S3 can check data integrity of the uploaded part when they are being assembled back.

Side Notes

Questions

First, thanks in advance for you're answer.

  1. Please if possible add a link to a repo or share some example.

  2. Explain what I do not know about this holy grail of checksum & data integrity check.

  3. I am not exactly familiar with how AWS S3 is generating those CRC32 checksums since the ones I was able to generate are all numbers and nothing like things AWS S3 returns as your checksum. You can look at the logs of Parts, here is one of them:

    ChecksumCRC32: 'VG/A4w=='
    

    Whereas the one I generate from the entire file is 209188370, a number!

    So maybe someone out there know how in NodeJS I can generate the same CRC32 as AWS folks do since I feel like my code is broken somehow.


Solution

  • So here is the thing:

    1. I was generating the checksum correctly but not hashing it properly. So here I've explained it how it should be done.
    2. Minio right now (16.02.2025) has some internal issue with checksum for the entire object as I've explained here. Meaning that AWS S3 works just fine with this code.

    And make sure to watch my YouTube video about this: https://youtu.be/Pgl_NmbxPUo.