node.jsbinary-data

NodeJS Buffer Size Logic: why 8KB as a default and not just the data size?

If you read a file smaller than 8KB in size NodeJS allocates 8 KB for it.

This isn't intuitive or expected, we are getting "more" bytes than we put. But it isn't a bug either at least in the sense that it is documented.

Minimal NodeJS Example

const { Buffer } = require("node:buffer");

// Create a buffer smaller than `Buffer.poolSize`.
const nodeBuffer = Buffer.from(new Uint8Array([0, 1, 2, 3, 4]));

// size is 8KB, not 5 bytes.
console.log(nodeBuffer.buffer);

Question

Files over 8KB would have an allocated space that matches the number of bytes in the file.

Other Details

I'm aware that we can use the offsets for reading:

// 1 KB
const buffer = fs.readFileSync(fileName, buffer.byteOffset, buffer.byteLength)

but the question isn't this.

Solution

  • The clue is in the name of the property you linked, poolSize.

    Elsewhere on the same page:

    The Buffer module pre-allocates an internal Buffer instance of size Buffer.poolSize that is used as a pool for the fast allocation of new Buffer instances created using Buffer.allocUnsafe(), Buffer.from(array), and Buffer.concat() only when size is less than Buffer.poolSize >>> 1 (floor of Buffer.poolSize divided by two).

    In other words, it's for performance reasons, since allocating and deallocating memory can be slow; instead, there's a pool of memory the module dips into for suitably small allocations. If you want a smaller buffer, it'd probably still be less efficient to allocate it from out of the pool than to have a buffer that's a bit bigger than what you need.

    (At the time of writing) you can use the buf.buffer and buf.byteOffset properties to see that two small allocations were indeed done off the same ArrayBuffer:

    > const { Buffer } = require("node:buffer");
> a = Buffer.from([1,2,3,4])
<Buffer 01 02 03 04>
> a.buffer
ArrayBuffer {
  [Uint8Contents]: <2f 00 00 00 00 00 00 00 01 02 03 04 00 00 00 00 00 00 00 00 00 ...>,
  byteLength: 8192
}
> b = Buffer.from([5,6,7,8])
<Buffer 05 06 07 08>

> b.buffer
ArrayBuffer {
  [Uint8Contents]: <2f 00 00 00 00 00 00 00 01 02 03 04 00 00 00 00 05 06 07 08 00 ...>,
  byteLength: 8192
}
> a.byteOffset
8
> b.byteOffset
16
>