In PyTorch should you shuffle the validation and test datasets?
According to this pytorch discussion thread I believe the answer is no, it should be False.
The official docs/tutorial states While training a model, we typically want to pass samples in “minibatches”, reshuffle the data at every epoch to reduce model overfitting; which suggest just training should be shuffle=True However their immediate example has both set to True.
Any guidance would be helpful.
The reason I ask is I have mine set to True; but my test accuracy is terrible.
Epoch: 1 | Train Loss: 2.1650 | Train Acc: 0.4683 | Test Loss: 6.2742 | Test Acc: 0.0349
Epoch: 2 | Train Loss: 1.2008 | Train Acc: 0.6624 | Test Loss: 6.9259 | Test Acc: 0.0359
Epoch: 3 | Train Loss: 0.9474 | Train Acc: 0.7325 | Test Loss: 7.3948 | Test Acc: 0.0365
Epoch: 4 | Train Loss: 0.7897 | Train Acc: 0.7808 | Test Loss: 7.7263 | Test Acc: 0.0352
Epoch: 5 | Train Loss: 0.6799 | Train Acc: 0.8186 | Test Loss: 8.0628 | Test Acc: 0.0377
Epoch: 6 | Train Loss: 0.5953 | Train Acc: 0.8374 | Test Loss: 8.3828 | Test Acc: 0.0351
Epoch: 7 | Train Loss: 0.5269 | Train Acc: 0.8647 | Test Loss: 8.5854 | Test Acc: 0.0351
Epoch: 8 | Train Loss: 0.4693 | Train Acc: 0.8791 | Test Loss: 8.9052 | Test Acc: 0.0337
Epoch: 9 | Train Loss: 0.4234 | Train Acc: 0.8955 | Test Loss: 9.1290 | Test Acc: 0.0354
train_ds = ImageFolder(
root=TRAIN_DIR,
transform=pretrained_weights.transforms(),
)
test_ds = ImageFolder(
root=TEST_DIR,
transform=pretrained_weights.transforms(),
)
train_dl = DataLoader(
dataset=train_ds, batch_size=BATCH_SIZE, shuffle=True
)
test_dl = DataLoader(
dataset=test_ds, batch_size=BATCH_SIZE, shuffle=True
)
Typically you want to set it to True for train. This allows the model to see more diverse batches across different epochs. At test time, it does not really matter.
I would say the reason for your low test loss is most likely unrelated to the shuffle parameter.