I am building a neural network. I couldn't load all the training data into memory at once, so I am using TensorFlow's tf.data.Dataset.from_generator function to load data incrementally. However, it throws an error saying it does not accept a list of tensors as a type.
TypeError: `output_signature` must contain objects that are subclass of
`tf.TypeSpec` but found <class 'list'> which is not.
The input to my neural network is a list of 151 separate tensors. How can I represent this in the generator? My code is below:
def generator(file_paths, batch_size, files_per_batch, tam, value):
return tf.data.Dataset.from_generator(
lambda: data_generator(file_paths, batch_size, files_per_batch, tam, value),
output_signature=(
[tf.TensorSpec(shape=(batch_size, tam), dtype=tf.float32) for _ in range(tam+1)], # Lista de 151 tensores
tf.TensorSpec(shape=(batch_size, tam), dtype=tf.float32) # RĂ³tulos
)
)
inputArray = [Input(shape=(tam,)) for _ in range(tam + 1)]
train_dataset = generator(file_paths, batch_size, files_per_batch, tam, False)
train_dataset = train_dataset.prefetch(tf.data.AUTOTUNE)
model.fit(train_dataset, epochs=1000, validation_split=0.2, verbose=1)
I tried to use tf.data.Dataset.from_generator to feed data into my neural network in batches, since I can't load all the data into memory at once. However, I encountered an error:
TypeError: output_signature must contain objects that are subclass of tf.TypeSpec but found <class 'list'> which is not.
I solved the problem using a dictionary instead of a list.
def generator(file_paths, batch_size, files_per_batch, size, value):
return tf.data.Dataset.from_generator(
lambda: data_generator(file_paths, batch_size, files_per_batch, size, value),
output_signature=(
{f"input_{i}": tf.TensorSpec(shape=(batch_size, size), dtype=tf.float32) for i in range(size + 1)}, # Inputs
tf.TensorSpec(shape=(batch_size, size), dtype=tf.float32) # Labels
)
)
To achieve this, I adjusted the input layer to:
inputArray = [Input(shape=(size,), name=f"input_{i}") for i in range(size + 1)]
This adjustment ensures that the keys from the generator match the keys expected by the model at the input.