I have a model that I generated using AutoKeras and I want to replicate the model so that I can construct it with keras tuner to do further hyperparameter tuning. But I am running into issues replicating the model. The model summary of the autokeras model is:
Model: "model"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_1 (InputLayer) [(None, 11)] 0
_________________________________________________________________
multi_category_encoding (Mul (None, 11) 0
_________________________________________________________________
normalization (Normalization (None, 11) 23
_________________________________________________________________
dense (Dense) (None, 16) 192
_________________________________________________________________
re_lu (ReLU) (None, 16) 0
_________________________________________________________________
dense_1 (Dense) (None, 32) 544
_________________________________________________________________
re_lu_1 (ReLU) (None, 32) 0
_________________________________________________________________
dense_2 (Dense) (None, 3) 99
_________________________________________________________________
classification_head_1 (Softm (None, 3) 0
=================================================================
Total params: 858
Trainable params: 835
Non-trainable params: 23
Layer config
{'batch_input_shape': (None, 11), 'dtype': 'string', 'sparse': False, 'ragged': False, 'name': 'input_1'}
{'name': 'multi_category_encoding', 'trainable': True, 'dtype': 'float32', 'encoding': ListWrapper(['int', 'int', 'int', 'int', 'int', 'int', 'int', 'int', 'int', 'int', 'int'])}
{'name': 'normalization', 'trainable': True, 'dtype': 'float32', 'axis': (-1,)}
{'name': 'dense', 'trainable': True, 'dtype': 'float32', 'units': 16, 'activation': 'linear', 'use_bias': True, 'kernel_initializer': {'class_name': 'GlorotUniform', 'config': {'seed': None}}, 'bias_initializer': {'class_name': 'Zeros', 'config': {}}, 'kernel_regularizer': None, 'bias_regularizer': None, 'activity_regularizer': None, 'kernel_constraint': None, 'bias_constraint': None}
{'name': 're_lu', 'trainable': True, 'dtype': 'float32', 'max_value': None, 'negative_slope': array(0., dtype=float32), 'threshold': array(0., dtype=float32)}
{'name': 'dense_1', 'trainable': True, 'dtype': 'float32', 'units': 32, 'activation': 'linear', 'use_bias': True, 'kernel_initializer': {'class_name': 'GlorotUniform', 'config': {'seed': None}}, 'bias_initializer': {'class_name': 'Zeros', 'config': {}}, 'kernel_regularizer': None, 'bias_regularizer': None, 'activity_regularizer': None, 'kernel_constraint': None, 'bias_constraint': None}
{'name': 're_lu_1', 'trainable': True, 'dtype': 'float32', 'max_value': None, 'negative_slope': array(0., dtype=float32), 'threshold': array(0., dtype=float32)}
{'name': 'dense_2', 'trainable': True, 'dtype': 'float32', 'units': 3, 'activation': 'linear', 'use_bias': True, 'kernel_initializer': {'class_name': 'GlorotUniform', 'config': {'seed': None}}, 'bias_initializer': {'class_name': 'Zeros', 'config': {}}, 'kernel_regularizer': None, 'bias_regularizer': None, 'activity_regularizer': None, 'kernel_constraint': None, 'bias_constraint': None}
{'name': 'classification_head_1', 'trainable': True, 'dtype': 'float32', 'axis': -1}
My training data is a dataframe that's converted to string type with both numerical and categorical data. Since the output is softmax i used LabelBinarizer to convert the target classes.
To make sure the model was replicated properly, i used keras.clone_model to create a copy of the model and try training it myself. But when I tried to train it myself the accuracy does not improve despite hitting 500 epochs.
Is there something that I am missing when it comes to training the model from scratch?
I finally was able to solve my issue. It's weird but even though the custom multicategory layer did not have params it contained its own mapping for the data. To extend the model and to examine the effect of layer depths I created a new model by adding the multicategory layer from the existing model. Once I did this training accuracy matched AutoKeras.
Edit: Adding code below:
from tensorflow import keras
inputs = keras.layers.Input(shape=(11,), dtype='string')
x = base_model.layers[1](inputs)
x = base_model.layers[2](x)
x = keras.layers.Dense(176)(x)
x = keras.layers.ReLU()(x)
x = keras.layers.Dense(400)(x)
x = keras.layers.ReLU()(x)
x = keras.layers.Dense(464)(x)
x = keras.layers.ReLU()(x)
x = keras.layers.Dense(3)(x)
x = keras.layers.Softmax()(x)
layer_3 = keras.Model(inputs, x)
where the index is for the multicategorical layer.