I have been banging my head against the wall on this one, it just doesn't make any sense for it not to work. So I am uploading a file to Azure Blobs using the stage and commit method. For some reason when uploading the last block, it fails.
Code for uploading a single block
app.post("/upload/block", async (req, res) => {
const { fileName, block } = req.body;
const blobName = fileName;
const blockBlobClient = containerClient.getBlockBlobClient(blobName);
// receive the blocks from client in JSON
const { blockChecksum, blockNumber } = block; //
const blockDataBytes = convertStringToBytes(block.blockData);
const blockDataBytesChecksum = crypto.createHash('md5').update(blockDataBytes).digest('hex');
if (blockChecksum !== blockDataBytesChecksum) {
console.log("Block checksum mismatch");
res.json({
status: "FAILURE"
});
return;
}
const blockId = createBlockId(block);
console.log({ blockIdLength: blockId.length, blockNumber, blockSize });
let isError = false;
await blockBlobClient.stageBlock(blockId, blockDataBytes, blockDataBytes.length)
.catch((err) => {
console.log(err);
isError = true;
})
if (isError) {
console.log("Error staging block");
res.json({
status: "FAILURE"
});
return;
}
console.log("Staged block");
res.status(200).json({
status: "SUCCESS"
})
});
I am uploading the blocks from clients sequentially, when one gets uploaded (success response received), I upload the next one.
Server Logs
{ blockIdLength: 64, blockNumber: 0, blockSize: 10485760 }
Staged block
{ blockIdLength: 64, blockNumber: 1, blockSize: 10485760 }
Staged block
{ blockIdLength: 64, blockNumber: 2, blockSize: 10485760 }
Staged block
{ blockIdLength: 64, blockNumber: 3, blockSize: 10485760 }
Staged block
{ blockIdLength: 64, blockNumber: 4, blockSize: 8306344 }
The specified blob or block content is invalid.
RequestId:8e343600-98d9-4c58-aadc-b5e707ccd749
Time:2022-04-27T15:24:32.598Z
Error staging block
The last block's size is the only difference, it's checksum matches with what the client sent and everything. I am clueless as to what is wrong here. As you can see I have added the block ID length in the logs, to show that it is also constant. If any more information is required I am happy to provide it.
Edit Creating block Ids here:
const createBlockId = (block) => {
const paddedBlockNumber = block.blockNumber.toString().padStart(6, "0");
const blockString = `${paddedBlockNumber}_${block.blockSize}_${block.blockChecksum}`;
return Buffer.from(blockString).toString('base64');
}
Found the issue. I was making a mistake in creating a block Id. The block Id's description on Azure is:
Required. A valid Base64 string value that identifies the block. Prior to encoding, the string must be less than or equal to 64 bytes in size. For a given blob, the length of the value specified for the blockid parameter must be the same size for each block. Note that the Base64 string must be URL-encoded.
Now, while creating the block Id, I was padding the number of block but not the size of block, and that was the issue.
I was checking the length of base64 encoded block Id, which was still the same as can be seen in the server logs.
Finally
: The length of the block Id before conversion should be the same.