Tensorflow image classification model returns same prediction every time, despite good training results

I have been trying to build an image classification model using Tensorflow and keras. The model is supposed to classify how many fingers are being shown.

I have tried training the model various different ways, but right now despite getting like 95+% training and validation accuracy, the model returns the same prediction with the same confidence every time. Even when I try to predict the class on an image the model was trained on, it still returns the same result.

Data sample:

My last training attempt:

train_ds = tf.keras.utils.image_dataset_from_directory(
  dataset_path,
  image_size=(300,300),
  seed=123,
  batch_size=batch_size,
  color_mode='grayscale')

val_ds = tf.keras.utils.image_dataset_from_directory(
  dataset_pathh,
  image_size=(300,300),
  seed=123,
  batch_size=batch_size,
  color_mode='grayscale')

AUTOTUNE = tf.data.AUTOTUNE

train_ds = train_ds.cache().shuffle(1000).prefetch(buffer_size=AUTOTUNE)
val_ds = val_ds.cache().prefetch(buffer_size=AUTOTUNE)

num_classes = len(class_names)

model = Sequential([
    layers.Rescaling(1./255),
    layers.Conv2D(32, 3, activation='relu',  input_shape=(300, 300, 1)),
    layers.MaxPooling2D(),
    layers.Conv2D(64, 3, activation='relu'),
    layers.MaxPooling2D(),
    layers.Conv2D(128, 3, activation='relu'),
    layers.MaxPooling2D(),
    layers.Conv2D(128, 3, activation='relu'),
    layers.MaxPooling2D(),
    layers.Flatten(),
    layers.Dropout(0.5),
    layers.Dense(512, activation='relu'),
    layers.Dense(num_classes, activation='softmax')
])

model.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])

epochs=5
history = model.fit(
  train_ds,
  validation_data=val_ds,
  epochs=epochs
)

Training results:

Getting the prediction:

img = tf.keras.utils.load_img(
    img_path, target_size=(300, 300)
)
img_array = tf.keras.utils.img_to_array(img)
img_array = tf.expand_dims(img_array, 0)
img_array = tf.image.rgb_to_grayscale(img_array)
img_array = img_array / 255
predictions = model.predict(img_array)
score = tf.nn.softmax(predictions[0])

Prediction being returned:

tf.Tensor([0.1295672  0.1295672  0.1295672  0.1295672  0.35214293 0.12958823], shape=(6,), dtype=float32)

I am a beginner to this, and mostly followed Tensorflow tutorials.

EDIT: Forgot to mention the dataset is balanced so the problem is not there.

It's difficult to pinpoint the exact reasons without having access to the data, but there are a few mistakes in the original code. Fixing these mistakes may improve model performance.

First, the loss function expects unscaled logits. These are the values that are returned by a layer, prior to any activation. In the network architecture, however, there is a softmax activation on the last layer. To fix this, set activation to None so the layer returns unscaled logits.

layers.Dense(num_classes, activation=None)

Second, your model implementation incorporates image scaling, but you also scale the images in your evaluation code. In the evaluation code, this has the effect of scaling the images twice. Remove the following line in your evaluation code:

img_array = img_array / 255

In addition to these errors in implementation, an imbalance in the dataset could also lead to performance issues. Consider adding metrics like AUROC and AUPR, which can help give a better view of performance when it comes to imbalanced data.