I have some unit tests for code that is doing some very minor image manipulation (combining several small images into a larger image). When I was running the tests, I noticed that three out of four of them failed on the line where they are reading an image from a directory (fails with an index out of bounds error).
However, if I run it again, they all pass. When I was writing the code as well, I noticed that whenever I set a breakpoint in my code, I'd have to run the unit tests twice because (after the very first time) it would run through the tests without hitting any breakpoints.
I have my repo organized like this:
src/
/* source code .m files are in here */
unit_tests/
images/
squares/
- img1.png
- img2.png
...
- imgn.png
- unit_tests.m
And I have a line in my setup (inside unit_tests.m) to generate and add paths for all of the code:
function tests = unit_tests()
addpath(genpath('..'));
tests = functiontests(localfunctions);
end
The unit tests all have this format:
function testCompositeImage_2x3(testCase)
squares = dir('images/squares/*.png');
num_images = length(squares);
img = imread([squares(1).folder filesep squares(1).name]); % all same size squares
rows = 2;
cols = 3;
buffer = 2;
for idx = 1:num_images - (rows*cols)
imarray = cell(1,(rows*cols));
n = 1;
for ii = idx:idx +(rows*cols) -1
imarray{n} = imread([squares(ii).folder filesep squares(ii).name]);
n = n + 1;
end
newimg = createCompositeImage(rows,cols,imarray, buffer);
expCols = cols*size(img,1) + (cols+1)*2*buffer;
expRows = rows*size(img,2) + (rows+1)*2*buffer;
assert(checksize(newimg, expRows, expCols, 3) == true);
end
end
("checksize" is just a helper I wrote that returns a boolean b/c assert doesn't compare matrices)
When I launch a fresh matlab session and run the unit tests (using the "Run Tests" button in the editor tab), they pass with this output:
>> runtests('unit_tests\unit_tests.m')
Running unit_tests
.......
Done unit_tests
__________
ans =
1×7 TestResult array with properties:
Name
Passed
Failed
Incomplete
Duration
Details
Totals:
7 Passed, 0 Failed, 0 Incomplete.
0.49467 seconds testing time.
Running it a second time (again, by pressing the button):
>> runtests('unit_tests')
Running unit_tests
..
================================================================================
Error occurred in unit_tests/testCompositeImage_2x2 and it did not run to completion.
---------
Error ID:
---------
'MATLAB:badsubscript'
--------------
Error Details:
--------------
Index exceeds array bounds.
Error in unit_tests>testCompositeImage_2x2 (line 47)
img = imread([squares(1).folder filesep squares(1).name]); % all same size
================================================================================
/*similar error info for the other two failing tests...*/
...
Done unit_tests
__________
Failure Summary:
Name Failed Incomplete Reason(s)
==================================================================
unit_tests/testCompositeImage_2x2 X X Errored.
------------------------------------------------------------------
unit_tests/testCompositeImage_2x3 X X Errored.
------------------------------------------------------------------
unit_tests/testCompositeImage_3x2 X X Errored.
ans =
1×7 TestResult array with properties:
Name
Passed
Failed
Incomplete
Duration
Details
Totals:
4 Passed, 3 Failed (rerun), 3 Incomplete.
0.0072287 seconds testing time.
The fact that it's failing on basically the first line because it's not reading anything from the folder leads me to suspect that even when the other 4 tests supposedly pass, they are in fact not running at all. And yet, if I run the tests again, they all pass. Run it a 4th time, and they fail once again.
At first, I thought that perhaps the unit tests were executing too quickly (only on even-numbered runs somehow?) and it was running the unit tests before the addpath/genpath functions in the setup had finished, so I added a pause statement and re-ran the tests, but I had the same issue only this time it would wait for the requisite number of seconds before going ahead and failing. If I run it again, no problem - all my tests pass.
I am completely at a loss as to why this is happening; I am using vanilla matlab (R2018a) running on a Win10 machine and don't have anything fancy going on. I feel like you should be able to run your unit tests as many times as you like and expect the same result! Is there something I've just somehow overlooked? Or is this some bizarre feature?
Adding my fix just in case someone else runs into the same issue.
As pointed out by Cris, something about the line
addpath(genpath('..'));
causes the GUI to go into a weird state where pressing the "Run Tests" button alternates between calling runtests('unit_tests\unit_tests.m') and runtests('unit_tests') which in turn causes the tests to alternately pass and fail. It does not seem to be an issue with the path variable itself (as it always contains - at a minimum - the necessary directories), but rather something intrinsic to matlab itself. The closest I could get to the root of the issue was the call to the (compiled) dir function inside the genpath function.
The "correct" solution was to remove the line from the unit_tests function entirely and add it to a setupOnce function:
function tests = unit_tests()
tests = functiontests(localfunctions);
end
function setupOnce(testCase)
addpath(genpath('..'));
end
A hack (not recommended) which doesn't require a setupOnce function is the following:
function tests = unit_tests()
pth = fullfile(fileparts(fileparts(mfilename('fullpath'))),'src');
paths = regexp(genpath(pth), ';', 'split');
for idx = 1:length(paths) - 1 % last element is empty
addpath(paths{idx});
end
end
I needed to relaunch matlab for the changes to take effect. This worked for my setup using r2018a running on Win10.