node.jsazureazure-blob-storageazure-node-sdk

Invalid Blob or Block content error, Node.js staging blocks


I have been banging my head against the wall on this one, it just doesn't make any sense for it not to work. So I am uploading a file to Azure Blobs using the stage and commit method. For some reason when uploading the last block, it fails.

Code for uploading a single block

app.post("/upload/block", async (req, res) => {
    const { fileName, block } = req.body;
    const blobName = fileName;
    const blockBlobClient = containerClient.getBlockBlobClient(blobName);

    // receive the blocks from client in JSON
    const { blockChecksum, blockNumber } = block; //
    const blockDataBytes = convertStringToBytes(block.blockData);


    const blockDataBytesChecksum = crypto.createHash('md5').update(blockDataBytes).digest('hex');
    if (blockChecksum !== blockDataBytesChecksum) {
        console.log("Block checksum mismatch");
        res.json({
            status: "FAILURE"
        });
        return;
    }
    const blockId = createBlockId(block);
    console.log({ blockIdLength: blockId.length, blockNumber, blockSize });

    let isError = false;

    await blockBlobClient.stageBlock(blockId, blockDataBytes, blockDataBytes.length)
        .catch((err) => {
            console.log(err);
            isError = true;
        })

    if (isError) {
        console.log("Error staging block");
        res.json({
            status: "FAILURE"
        });
        return;
    }
    console.log("Staged block");
    res.status(200).json({
        status: "SUCCESS"
    })
});

I am uploading the blocks from clients sequentially, when one gets uploaded (success response received), I upload the next one.
Server Logs

{ blockIdLength: 64, blockNumber: 0, blockSize: 10485760 }
Staged block
{ blockIdLength: 64, blockNumber: 1, blockSize: 10485760 }
Staged block
{ blockIdLength: 64, blockNumber: 2, blockSize: 10485760 }
Staged block
{ blockIdLength: 64, blockNumber: 3, blockSize: 10485760 }
Staged block
{ blockIdLength: 64, blockNumber: 4, blockSize: 8306344 }
The specified blob or block content is invalid.
RequestId:8e343600-98d9-4c58-aadc-b5e707ccd749
Time:2022-04-27T15:24:32.598Z
Error staging block

The last block's size is the only difference, it's checksum matches with what the client sent and everything. I am clueless as to what is wrong here. As you can see I have added the block ID length in the logs, to show that it is also constant. If any more information is required I am happy to provide it.

Edit Creating block Ids here:

const createBlockId = (block) => {
    const paddedBlockNumber = block.blockNumber.toString().padStart(6, "0");
    const blockString = `${paddedBlockNumber}_${block.blockSize}_${block.blockChecksum}`;
    return Buffer.from(blockString).toString('base64');
}

Solution

  • Found the issue. I was making a mistake in creating a block Id. The block Id's description on Azure is:

    Required. A valid Base64 string value that identifies the block. Prior to encoding, the string must be less than or equal to 64 bytes in size. For a given blob, the length of the value specified for the blockid parameter must be the same size for each block. Note that the Base64 string must be URL-encoded.

    Now, while creating the block Id, I was padding the number of block but not the size of block, and that was the issue.

    I was checking the length of base64 encoded block Id, which was still the same as can be seen in the server logs.

    Finally: The length of the block Id before conversion should be the same.