Context:
I am trying to train a image classifier on kaggle cell dataset of hopefully 0.95 val_acc. I have tried many model architectures and number of epochs, among several other hyperparameters arriving at a promising set that yields a 0.9 val_acc.
Things I tried:
BatchNormalization(), Dropout() to reduce overfitting (now model is underfitting)Problem:
The set of hyperparameters that gave the best val_acc still plateus at 0.9. I have tried many permutations, is there anything I am missing/doing wrong?
Model:
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 120, 160, 8) 224
_________________________________________________________________
batch_normalization (BatchNo (None, 120, 160, 8) 32
_________________________________________________________________
activation (Activation) (None, 120, 160, 8) 0
_________________________________________________________________
dropout (Dropout) (None, 120, 160, 8) 0
_________________________________________________________________
conv2d_1 (Conv2D) (None, 60, 80, 8) 584
_________________________________________________________________
batch_normalization_1 (Batch (None, 60, 80, 8) 32
_________________________________________________________________
activation_1 (Activation) (None, 60, 80, 8) 0
_________________________________________________________________
conv2d_2 (Conv2D) (None, 60, 80, 8) 584
_________________________________________________________________
batch_normalization_2 (Batch (None, 60, 80, 8) 32
_________________________________________________________________
activation_2 (Activation) (None, 60, 80, 8) 0
_________________________________________________________________
dropout_1 (Dropout) (None, 60, 80, 8) 0
_________________________________________________________________
conv2d_3 (Conv2D) (None, 30, 40, 8) 584
_________________________________________________________________
batch_normalization_3 (Batch (None, 30, 40, 8) 32
_________________________________________________________________
activation_3 (Activation) (None, 30, 40, 8) 0
_________________________________________________________________
conv2d_4 (Conv2D) (None, 30, 40, 8) 584
_________________________________________________________________
batch_normalization_4 (Batch (None, 30, 40, 8) 32
_________________________________________________________________
activation_4 (Activation) (None, 30, 40, 8) 0
_________________________________________________________________
dropout_2 (Dropout) (None, 30, 40, 8) 0
_________________________________________________________________
conv2d_5 (Conv2D) (None, 15, 20, 8) 584
_________________________________________________________________
batch_normalization_5 (Batch (None, 15, 20, 8) 32
_________________________________________________________________
activation_5 (Activation) (None, 15, 20, 8) 0
_________________________________________________________________
conv2d_6 (Conv2D) (None, 15, 20, 16) 3216
_________________________________________________________________
batch_normalization_6 (Batch (None, 15, 20, 16) 64
_________________________________________________________________
activation_6 (Activation) (None, 15, 20, 16) 0
_________________________________________________________________
dropout_3 (Dropout) (None, 15, 20, 16) 0
_________________________________________________________________
conv2d_7 (Conv2D) (None, 8, 10, 16) 6416
_________________________________________________________________
batch_normalization_7 (Batch (None, 8, 10, 16) 64
_________________________________________________________________
activation_7 (Activation) (None, 8, 10, 16) 0
_________________________________________________________________
conv2d_8 (Conv2D) (None, 8, 10, 16) 6416
_________________________________________________________________
batch_normalization_8 (Batch (None, 8, 10, 16) 64
_________________________________________________________________
activation_8 (Activation) (None, 8, 10, 16) 0
_________________________________________________________________
dropout_4 (Dropout) (None, 8, 10, 16) 0
_________________________________________________________________
conv2d_9 (Conv2D) (None, 4, 5, 16) 6416
_________________________________________________________________
batch_normalization_9 (Batch (None, 4, 5, 16) 64
_________________________________________________________________
activation_9 (Activation) (None, 4, 5, 16) 0
_________________________________________________________________
flatten (Flatten) (None, 320) 0
_________________________________________________________________
dense (Dense) (None, 240) 77040
_________________________________________________________________
batch_normalization_10 (Batc (None, 240) 960
_________________________________________________________________
dropout_5 (Dropout) (None, 240) 0
_________________________________________________________________
dense_1 (Dense) (None, 162) 39042
_________________________________________________________________
batch_normalization_11 (Batc (None, 162) 648
_________________________________________________________________
dropout_6 (Dropout) (None, 162) 0
_________________________________________________________________
dense_2 (Dense) (None, 84) 13692
_________________________________________________________________
batch_normalization_12 (Batc (None, 84) 336
_________________________________________________________________
dropout_7 (Dropout) (None, 84) 0
_________________________________________________________________
dense_3 (Dense) (None, 4) 340
=================================================================
Total params: 158,114
Trainable params: 156,918
Non-trainable params: 1,196
visualization of activations and val_acc, val_loss
Note:
The optimization was done using talos which can be found here. I edited and added some modules here.
Edit 1:
The optimizer I used is Nadam with learning rate 0.0002. Full notebook.
TLDR:
Trained on kaggle cell dataset using best hyperparameters from a test run of trying around 200 different hyperparameters. Plateaus at 0.9. Why not higher?
From what I can find, I was using too low a learning rate. Increasing it seems to help.
