If you read a file smaller than 8KB in size NodeJS allocates 8 KB for it.
This isn't intuitive or expected, we are getting "more" bytes than we put. But it isn't a bug either at least in the sense that it is documented.
Minimal NodeJS Example
const { Buffer } = require("node:buffer");
// Create a buffer smaller than `Buffer.poolSize`.
const nodeBuffer = Buffer.from(new Uint8Array([0, 1, 2, 3, 4]));
// size is 8KB, not 5 bytes.
console.log(nodeBuffer.buffer);
Question
Files over 8KB would have an allocated space that matches the number of bytes in the file.
Other Details
I'm aware that we can use the offsets for reading:
// 1 KB
const buffer = fs.readFileSync(fileName, buffer.byteOffset, buffer.byteLength)
but the question isn't this.
The clue is in the name of the property you linked, poolSize
.
The
Buffer
module pre-allocates an internal Buffer instance of sizeBuffer.poolSize
that is used as a pool for the fast allocation of new Buffer instances created usingBuffer.allocUnsafe()
,Buffer.from(array)
, andBuffer.concat()
only when size is less thanBuffer.poolSize >>> 1
(floor ofBuffer.poolSize
divided by two).
In other words, it's for performance reasons, since allocating and deallocating memory can be slow; instead, there's a pool of memory the module dips into for suitably small allocations. If you want a smaller buffer, it'd probably still be less efficient to allocate it from out of the pool than to have a buffer that's a bit bigger than what you need.
(At the time of writing) you can use the buf.buffer
and buf.byteOffset
properties to see that two small allocations were indeed done off the same ArrayBuffer
:
> const { Buffer } = require("node:buffer");
> a = Buffer.from([1,2,3,4])
<Buffer 01 02 03 04>
> a.buffer
ArrayBuffer {
[Uint8Contents]: <2f 00 00 00 00 00 00 00 01 02 03 04 00 00 00 00 00 00 00 00 00 ...>,
byteLength: 8192
}
> b = Buffer.from([5,6,7,8])
<Buffer 05 06 07 08>
> b.buffer
ArrayBuffer {
[Uint8Contents]: <2f 00 00 00 00 00 00 00 01 02 03 04 00 00 00 00 05 06 07 08 00 ...>,
byteLength: 8192
}
> a.byteOffset
8
> b.byteOffset
16
>