I've created a mixed model in Keras, creating weights for metadata and image data and then combining them for the classification. Here's the model:
Model: "model_1"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_5 (InputLayer) [(None, 80, 120, 3)] 0
__________________________________________________________________________________________________
xception (Functional) (None, 3, 4, 2048) 20861480 input_5[0][0]
__________________________________________________________________________________________________
input_4 (InputLayer) [(None, 10)] 0
__________________________________________________________________________________________________
conv2d_9 (Conv2D) (None, 3, 4, 8) 409608 xception[0][0]
__________________________________________________________________________________________________
dense_3 (Dense) (None, 4) 44 input_4[0][0]
__________________________________________________________________________________________________
global_average_pooling2d_1 (Glo (None, 8) 0 conv2d_9[0][0]
__________________________________________________________________________________________________
concatenate_1 (Concatenate) (None, 12) 0 dense_3[0][0]
global_average_pooling2d_1[0][0]
__________________________________________________________________________________________________
dense_4 (Dense) (None, 4) 52 concatenate_1[0][0]
__________________________________________________________________________________________________
dense_5 (Dense) (None, 1) 5 dense_4[0][0]
==================================================================================================
Total params: 21,271,189
Trainable params: 21,216,661
Non-trainable params: 54,528
__________________________________________________________________________________________________
I decided to augment the images due to imbalance. I used the following ImageDataGenerator:
aug = ImageDataGenerator(rescale=1/255.,
rotation_range=180,
height_shift_range=0.2,
width_shift_range=0.2,
brightness_range=[0.5,1.5],
channel_shift_range=100.0,
horizontal_flip=True,
vertical_flip=True,
shear_range=45.0)
I then compiled and attempted to train the model using ImageDataGenerator().flow():
epochs = 10
BATCH_SIZE = 20
flow = aug.flow(img_train, y_train, batch_size=BATCH_SIZE)
history = model.fit([meta_train, flow], y_train, epochs=epochs, batch_size=100, validation_data=([meta_test, img_test], y_test), class_weight=class_weight)
This gives me an error:
ValueError: Failed to find data adapter that can handle input: (<class 'list'> containing values of
types {"<class 'pandas.core.frame.DataFrame'>", "<class 'tensorflow.python.keras.preprocessing.image.NumpyArrayIterator'>"}),
<class 'numpy.ndarray'>
I've tried multiple versions of the code, but I'm just not familiar enough with the backend to correctly diagnose the problem. Can anyone help me with this?
LEARNING_RATE = 0.001
# Define inputs
meta_inputs = Input(shape=(10,))
img_inputs = Input(shape=(80,120,3,))
# Model 1
meta_layer1 = Dense(4, activation='relu')(meta_inputs)
# Model 2
xception_layer = Xception(include_top=False, input_shape=(80,120,3,))(img_inputs)
img_conv_layer1 = Conv2D(8, kernel_size=(5,5), padding='same', activation='relu')(xception_layer)
img_gap_layer = GlobalAveragePooling2D()(img_conv_layer1)
# img_sdense_layer = Dense(4, activation='relu')(img_gap_layer)
# Merge models
merged_layer = Concatenate()([meta_layer1, img_gap_layer])
merged_dense_layer = Dense(4, activation='relu')(merged_layer)
merged_output = Dense(1, activation='sigmoid')(merged_dense_layer)
# Define functional model
model = Model(inputs=[meta_inputs, img_inputs], outputs=merged_output)
# Compile model
auc = AUC(name = 'auc')
model.compile(Adam(learning_rate=LEARNING_RATE), loss='binary_crossentropy', metrics=[auc])
model.summary()
age_approx Unknown female male head/neck lower extremity \
11655 45 0 0 1 0 0
24502 60 0 0 1 0 1
2524 50 0 1 0 0 1
13894 60 0 1 0 0 0
29325 45 0 1 0 0 1
oral/genital palms/soles torso upper extremity
11655 0 0 1 0
24502 0 0 0 0
2524 0 0 0 0
13894 0 0 1 0
29325 0 0 0 0
Array too large, see code here.
(23188, 1)
first of all if you generate the following arrays your models works with no error;
import tensorflow as tf
import numpy as np
LEARNING_RATE = 0.001
# Define inputs
meta_inputs = tf.keras.layers.Input(shape=(10,))
img_inputs = tf.keras.layers.Input(shape=(80,120,3,))
# Model 1
meta_layer1 = tf.keras.layers.Dense(4, activation='relu')(meta_inputs)
# Model 2
xception_layer = tf.keras.applications.Xception(include_top=False, input_shape=(80,120,3,))(img_inputs)
img_conv_layer1 = tf.keras.layers.Conv2D(8, kernel_size=(5,5), padding='same', activation='relu')(xception_layer)
img_gap_layer = tf.keras.layers.GlobalAveragePooling2D()(img_conv_layer1)
# img_sdense_layer = Dense(4, activation='relu')(img_gap_layer)
# Merge models
merged_layer = tf.keras.layers.Concatenate()([meta_layer1, img_gap_layer])
merged_dense_layer = tf.keras.layers.Dense(4, activation='relu')(merged_layer)
merged_output = tf.keras.layers.Dense(1, activation='sigmoid')(merged_dense_layer)
# Define functional model
model = tf.keras.models.Model(inputs=[meta_inputs, img_inputs], outputs=merged_output)
# Compile model
auc = tf.keras.metrics.AUC(name = 'auc')
model.compile(tf.keras.optimizers.Adam(learning_rate=LEARNING_RATE), loss='binary_crossentropy', metrics=[auc])
model.summary()
Let's generate arrays as below;
y_array = np.zeros((23188, 1))
train_array = np.zeros((23188, 80, 120,3))
meta_array = np.zeros((23188, 10))
Now test your model;
model.fit(x = [meta_array, train_array], y = y_array, epochs = 1)
as you can see it runs
>>> model.fit(x = [meta_array, train_array], y = y_array, epochs = 1)
2021-05-17 10:30:03.430303: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
2021-05-17 10:30:03.447843: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 3093120000 Hz
47/725 [>.............................] - ETA: 6:52 - loss: 0.6818 - auc: 0.0000e+00
Now the problem is in your inputs:
(1) first convert all of your meta data in an array let's say it has batch of 1000 as a result it has to have a shape of meta_train.shape is (1000,10)
(2) your img_train.shape is (1000,80,120,3)
(3) your y_train.shape is (1000,1)
here 1000 can be 23188 as well. But let's assume you 1000 images, y_train (target) and 1000 meta data.
As you want to augment your image train you have to be careful here. Use as below;
import tensorflow as tf
import numpy as np
import time
here I created empty array as example but your origal data has to be in array as shapes of below, except instead of 1000, it has to be number of images you have.
y_train = np.zeros((1000, 1))
img_train = np.zeros((1000, 80, 120,3))
meta_train= np.zeros((1000, 10))
aug = tf.keras.preprocessing.image.ImageDataGenerator(rescale=1/255.,
rotation_range=180,
height_shift_range=0.2,
width_shift_range=0.2,
brightness_range=[0.5,1.5],
channel_shift_range=100.0,
horizontal_flip=True,
vertical_flip=True,
shear_range=45.0)
aug.fit(img_train)
# NOTE; change batch_size to 32 if your img_train is too high or your device has less memory
# set shuffle to false, you have to concatenate at each loop the meta_train and y_train for each image, so do not shuffle the images!
imagenerator = aug.flow(img_train, batch_size = img_train.shape[0], shuffle=False, sample_weight=None, seed=123, save_to_dir=None, save_prefix="", save_format="png", subset=None)
new_meta = meta_train.copy() # concatenate meta_train at each loop to new_meta
new_y = y_train.copy() # concatenate y_train at each loop to new_y
batches = 0
#let's iterate 5 times.
for x_batch in imagenerator:
print(batches, time.strftime("%Y:%m%d-%H:%M:%S"))
batches += 1
img_train = np.concatenate((img_train, x_batch), axis = 0)
new_meta = np.concatenate((new_meta, meta_train), axis = 0) # concatenate corresponding meta data
new_y = np.concatenate((new_y, y_train), axis = 0) #concatenate corresponding label/target data!
if batches >= 5:
break
model.fit(x = [new_meta, img_train], y = new_y, epochs = 1, batch_size = 32)
NOTE: you can shuffle your data simply as below before using them in model.fit
idxs = np.array([x for x in range(img_train.shape[0])])
np.random.shuffle(idxs)
img_train = img_train[idx]
new_y = new_y[idx]
new_meta = new_meta[idx]