Memory Limitation when Extracting VLAD from SIFT Descriptors in VLFeat with Matlab

I recently asked how to extract VLAD from SIFT descriptors in VLFeat with Matlab here.

However, I am running up against memory limitations. I have 64GB RAM and 64GB Swap.

 all_descr = single([sift_descr{:}]);

... produces a memory error:

Requested 128x258438583 (123.2GB) array exceeds maximum array size preference. Creation of arrays greater than this limit may take a long time and cause MATLAB to become unresponsive. See array size limit or preference panel for more information.

What is the correct way to extract VLAD when we have a very large training dataset? For example, I could subset the SIFT descriptors before running vl_kmeans, like this:

all_descr = single([sift_descr{1:100000}]);

This works within memory, but how will it affect my VLAD features? Thank you.

The main issue here is not how to extract the VLAD for such a matrix: you calculate the VLAD vector of each image by loop'ing over all images and calculating the VLAD one-by-one, i.e. the memory problem doesn't appear there.

You run out of memory when you try to concatenate the SIFT descriptors of all images, in order to cluster them into visual words using a nearest neighbor search:

all_descr = single([sift_descr{:}]);
centroids = vl_kmeans(all_descr, 64);

The simplest way I can think of is to switch to MATLAB's own kmeans function, which is included in the Statistics and Machine Learning toolbox. It comes with support for Tall Arrays, i.e. MATLAB's datatype for arrays which don't fit into the memory.

To do that, you can save the SIFT descriptors of each image into a CSV file using csvwrite:

for k = 1:size(filelist, 1)
    I = imread([repo filelist(k).name]) ;
    I = single(rgb2gray(I)) ;
    [f,d] = vl_sift(I) ;
    csvwrite(sprintf('sift/im_%d.csv', k), single(d.'));
end

Then, you can load the descriptors as a Tall Array by using the datastore function and converting it to tall. As the result will be a table, you can use table2array to convert it to a tall array.

ds = datastore('sift/*.csv');
all_descriptors = table2array(tall(ds));

Finally, you can call thekmeans function on that, and get your centroids

[~, centroids] = kmeans(all_descriptors, 64);

Now you can proceed with calculating the VLAD vector as usual.