I'm trying to decode the dataset from this source: http://yann.lecun.com/exdb/mnist/
There is a description of the "very simple" IDX file type in the bottom, but I cannot figure it out.
What I'm trying to achieve is something like:
var imagesFileBuffer = fs.readFileSync(__dirname + '/train-images-idx3-ubyte');
var labelFileBuffer = fs.readFileSync(__dirname + '/train-labels-idx1-ubyte');
var pixelValues = {};
Do magic
pixelValues are now like:
// {
// "0": [0,0,200,190,79,0... for all 784 pixels ... ],
// "4": [0,0,200,190,79,0... for all 784 pixels ... ],
etc for all image entries in the dataset. I've tried to figure out the structure of the binary files, but failed.
I realized there would be duplicate keys in my structure of the pixelValues object, so I made an array of objects of it instaed. The following code will create the structure I'm after:
var dataFileBuffer = fs.readFileSync(__dirname + '/train-images-idx3-ubyte');
var labelFileBuffer = fs.readFileSync(__dirname + '/train-labels-idx1-ubyte');
var pixelValues = [];
// It would be nice with a checker instead of a hard coded 60000 limit here
for (var image = 0; image <= 59999; image++) {
var pixels = [];
for (var x = 0; x <= 27; x++) {
for (var y = 0; y <= 27; y++) {
pixels.push(dataFileBuffer[(image * 28 * 28) + (x + (y * 28)) + 15]);
}
}
var imageData = {};
imageData[JSON.stringify(labelFileBuffer[image + 8])] = pixels;
pixelValues.push(imageData);
}
The structure of pixelValues is now something like this:
[
{5: [28,0,0,0,0,0,0,0,0,0...]},
{0: [0,0,0,0,0,0,0,0,0,0...]},
...
]
There are 28x28=784 pixel values, all varying from 0 to 255.
To render the pixels, use my for loops like I did above, rendering the first pixel in the upper left corner, then working towards the right.