matlabdeep-learningneural-networklstmbilstm

Error forming mini-batch for network input


I am trying to predict vehicle trajectory, using t-30 data sequence to predict until t+30 trajectory. The data I have created is an neighbour occupancy matrix for each frames for each cars, this data is 13x3x1x30x138634 where 13x3 the grid, 1 is the channel, 30 is sequence length and 138634 is the observation number. For the target vehicle I have 2 arrays of 30x3x138634 where 30 is the sequence length, 3 is (x,y,v) for the target vehicle and 138634 is the observation number. So neighbour data is SSCTB and target data is TCB. The error I am getting is:

Error forming mini-batch for network input "sequenceinput". Data interpreted with format "SSCTB". To specify a different format, use the InputDataFormats option.
Error in 
main (line 488)
trainnedNet = trainnet(TrainDS,net,"l2loss",options);
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Caused by:
    Batch dimension of datastore must match the format batch dimension (5).

I have attached the code but can't attach *.mat files they are too large even when zipped. You can simply assume in the code

input_neigh_odd = zeros(13x3x1x30x138634);
input_target_odd_centered = zeros(30x3x138634);
input_target_even_centered = zeros(30x3x138634);

The code part is,

load("neighInputOdd.mat")
load("targetInputOddCentered.mat")
load("targetInputEvenCentered.mat")

numSamples = 138634;
shuffledIdx = randperm(numSamples);

nTrain = floor(0.70 * numSamples);
nVal   = floor(0.15 * numSamples);
nTest  = numSamples - nTrain - nVal;

trainIdx = shuffledIdx(1:nTrain);
valIdx   = shuffledIdx(nTrain+1:nTrain+nVal);
testIdx  = shuffledIdx(nTrain+nVal+1:end);

X1Train = {permute(input_neigh_odd(:,:,:,:,trainIdx), [1 2 3 5 4])};
X1Val   = {permute(input_neigh_odd(:,:,:,:,valIdx), [1 2 3 5 4])};
X1Test  = {permute(input_neigh_odd(:,:,:,:,testIdx), [1 2 3 5 4])};

X2Train = {input_target_odd_centered(:,:,trainIdx)};
X2Val   = {input_target_odd_centered(:,:,valIdx)};
X2Test  = {input_target_odd_centered(:,:,testIdx)};

YTrain = {input_target_even_centered(:,:,trainIdx)};
YVal   = {input_target_even_centered(:,:,valIdx)};
YTest  = {input_target_even_centered(:,:,testIdx)};

save("TrainValTest\X1Train.mat","X1Train","-v7.3")
save("TrainValTest\X2Train.mat","X2Train","-v7.3")
save("TrainValTest\YTrain.mat","YTrain","-v7.3")

save("TrainValTest\X1Val.mat","X1Val","-v7.3")
save("TrainValTest\X2Val.mat","X2Val","-v7.3")
save("TrainValTest\YVal.mat","YVal","-v7.3")

save("TrainValTest\X1Test.mat","X1Test","-v7.3")
save("TrainValTest\X2Test.mat","X2Test","-v7.3")
save("TrainValTest\YTest.mat","YTest","-v7.3")

matFiles = dir(fullfile('TrainValTest', '*.mat'));

for i = 1:numel(matFiles)
    filePath = fullfile(matFiles(i).folder, matFiles(i).name);
    
    [~, name, ~] = fileparts(matFiles(i).name);
    varName = [name 'DS'];

    ds = signalDatastore(filePath);
    
    assignin('base', varName, ds);
end

TrainDS = combine(X1TrainDS,X2TrainDS,YTrainDS);
ValDS = combine(X1ValDS,X2ValDS);

alpha = 0.001;
options = trainingOptions( ...
    "adam", ...
    "InputDataFormats", {'SSCTB', 'TCB'},...
    Plots="training-progress", ...
    ValidationData = {ValDS,YValDS}, ...
    ValidationFrequency = 100, ...
    ValidationPatience = 5, ...
    ExecutionEnvironment = "parallel-auto",...
    MaxEpochs = 15,...
    InitialLearnRate=alpha,...
    MiniBatchSize=256,...
    L2Regularization=0.0001,...
    Verbose=false);

net=dlnetwork;
numHiddenUnits=64;
numNeurons=128;

layers1=[
    sequenceInputLayer([13 3 1])
    convolution2dLayer([3 3],16,'Padding','same')
    batchNormalizationLayer
    reluLayer
    convolution2dLayer([3 3],32,'Padding','same')
    batchNormalizationLayer
    reluLayer
    bilstmLayer(64,'OutputMode','sequence','name','bilstm1')
    concatenationLayer(1,2,'Name','cat')
    fullyConnectedLayer(3)
    ];

layers2=[
    sequenceInputLayer(3)
    bilstmLayer(32,'OutputMode','sequence','name','bilstm2')
    ];

net=addLayers(net,layers1);
net=addLayers(net,layers2);
net=connectLayers(net,'bilstm2','cat/in2');

trainnedNet = trainnet(TrainDS,net,"l2loss",options);

Solution

  • The error happens because you're wrapping your entire dataset in a single cell, so MATLAB thinks you have only one observation. For training, MATLAB needs to treat each sample as a separate item along the batch dimension , 5th for SSCTB and 3rd for TCB. To fix it, use arrayDatastore with the correct iteration dimensions:

    X1 = permute(input_neigh_odd, [1 2 3 5 4]);  % SSCTB
    X2 = input_target_odd_centered;             % TCB
    Y  = input_target_even_centered;            % TCB
    X1TrainDS = arrayDatastore(X1(:,:,:,:,trainIdx), 'IterationDimension', 5);
    X2TrainDS = arrayDatastore(X2(:,:,trainIdx),     'IterationDimension', 3);
    YTrainDS  = arrayDatastore(Y(:,:,trainIdx),      'IterationDimension', 3);
    X1ValDS = arrayDatastore(X1(:,:,:,:,valIdx), 'IterationDimension', 5);
    X2ValDS = arrayDatastore(X2(:,:,valIdx),     'IterationDimension', 3);
    YValDS  = arrayDatastore(Y(:,:,valIdx),      'IterationDimension', 3);
    TrainDS = combine(X1TrainDS, X2TrainDS, YTrainDS);
    ValDS   = combine(X1ValDS, X2ValDS, YValDS);
    

    Now trainnet will correctly interpret the batch dimensions, and the error should go away.