python-3.xdeep-learningresnetimage-classificationimagedatagenerator

AttributeError: 'NoneType' object has no attribute 'items' when training DL dataset made using 'ImageDataGenerator'


I am trying to train a resnet50 model using transfer learning and a dataset containing 40,000 images. I used ImageDataGenerator to prepare the dataset and then used flow_from_directory to make the training and validation datasets(validation_split=0.2). Optimizer is Adam() for model compiling.

when training the model later I got the error:

AttributeError: 'NoneType' object has no attribute 'items'

I used shuffle=True, repeat function, manual filtering, but none seemed to work.

The code is:

# Import Library
import numpy as np 
import pandas as pd
import matplotlib.pyplot as plt
from glob import glob
import tensorflow as tf
from tensorflow.keras.preprocessing.image import ImageDataGenerator, img_to_array, load_img
from tensorflow.keras.models import save_model
from keras.models import Sequential
from keras.layers import Dense
import os
import cv2

from tensorflow.keras.applications import ResNet50
from tensorflow.keras.layers import Dense, Flatten
from tensorflow.keras.applications.imagenet_utils import preprocess_input  # Assuming TensorFlow 2.x
from tensorflow.keras.models import Model  # To define your model architecture
from tensorflow.keras.optimizers import Adam  # Or any other optimizer you choose
from tensorflow.keras.losses import categorical_crossentropy  # Or any other loss function

from keras.models import Model
from keras.callbacks import EarlyStopping

# Test and Train path

# Define the path to the dataset and batch size
path = r"C:\Users\Rajarshi\Downloads\Compressed\Concrete Crack Images for Classification"
batch_size = 32

# Step 1: Set up data generators
#image_generator = ImageDataGenerator(validation_split=0.2)  # Remove rescale argument
image_generator = ImageDataGenerator(horizontal_flip=True,
                                     rescale=1./255,
                                     zoom_range=0.2, 
                                     validation_split=0.2)
                                     
#image_generator.preprocessing_function = custom_preprocessing  # Apply custom preprocessing
try:
 train_data = image_generator.flow_from_directory(batch_size=batch_size,
                                                 directory=path,
                                                 shuffle=True,
                                                 target_size=(224, 224),
                                                 subset="training",
                                                 class_mode="categorical")
                                                
 validation_data = image_generator.flow_from_directory(batch_size=batch_size,
                                                      directory=path,
                                                      shuffle=True,
                                                      target_size=(224, 224),
                                                      subset="validation",
                                                      class_mode="categorical")

except OSError as e:
  print(f"Error encountered while generating data: {e}")
  print("Please check your data directory path and structure.")

print (train_data.shape) and (validation_data.shape) #to verify data generation.

# model compile
model.compile(loss='categorical_crossentropy',  # Adjust loss function based on your problem
              optimizer=Adam(),  # Adjust optimizer based on preference
              #optimizer='rmsprop',
              metrics=['accuracy'])


# Training Model

model.fit(
     train_data,
     steps_per_epoch=train_data.samples // train_data.batch_size,  # Calculate steps per epoch
     epochs=num_epochs,
     validation_data=validation_data,
     validation_steps=validation_data.samples // validation_data.batch_size  # Calculate validation steps
 )

Model Traing Error Dataset Generation Error Check

** I used shuffle=True, repeat function, manual filtering, but none seemed to work. I tried with 'rmsprop' optimizer as well.**


Solution

  • The issue occurs in these lines:

    steps_per_epoch=train_data.samples // train_data.batch_size
    validation_steps=validation_data.samples // validation_data.batch_size
    

    Because the ImageDataGenerator is deprecated in the latest version (2.16.1) of Tensorflow (as seen here).

    I ran into the same problem, and the solution I found was to downgrade Tensorflow to a previous version through this command in Jupyter:

    %pip install tensorflow==2.15