How to optimize keras model with batchnorm layers with Intel inference engine (OpenVINO)?

Failed to optimize keras model with Intel inference engine (OpenVINO toolkit R.5)

I freeze my model just like following tutorial suggests. The keras model is trained and tested. I need to optimize it for inference. However I get an error while running model optimizer (mo.py script) on custom model.

[ ERROR ] shapes (128,9) and (0,) not aligned: 9 (dim 1) != 0 (dim 0)

Last few layers of my model (9 is number of output of classes) are:

conv2d_4 (Conv2D) (None, 4, 4, 128) 204928 batch_normalization_3[0][0]
__________________________________________________________________________________________________
activation_4 (Activation) (None, 4, 4, 128) 0 conv2d_4[0][0]
__________________________________________________________________________________________________
batch_normalization_4 (BatchNor (None, 4, 4, 128) 512 activation_4[0][0]
__________________________________________________________________________________________________
average_pooling2d_2 (AveragePoo (None, 1, 1, 128) 0 batch_normalization_4[0][0]
__________________________________________________________________________________________________
dropout_2 (Dropout) (None, 1, 1, 128) 0 average_pooling2d_2[0][0]
__________________________________________________________________________________________________
flatten (Flatten) (None, 128) 0 dropout_2[0][0]
__________________________________________________________________________________________________
dense (Dense) (None, 128) 16512 flatten[0][0]
__________________________________________________________________________________________________
activation_5 (Activation) (None, 128) 0 dense[0][0]
__________________________________________________________________________________________________
batch_normalization_5 (BatchNor (None, 128) 512 activation_5[0][0]
__________________________________________________________________________________________________
dropout_3 (Dropout) (None, 128) 0 batch_normalization_5[0][0]
__________________________________________________________________________________________________
dense_1 (Dense) (None, 9) 1161 dropout_3[0][0]
__________________________________________________________________________________________________
color_prediction (Activation) (None, 9) 0 dense_1[0][0]
__________________________________________________________________________________________________

Model optimizer fails due to presence of BatchNormalization layers. When I remove them it runs successfully. However I freeze graph with

tf.keras.backend.set_learning_phase(0) 

So nodes like BatchNormalization and Dropout must be removed in freezed graph, I can't figure out why they don't removed.

Thanks a lot!

I managed to run OpenVINO model optimizer on Keras model with Batch Normalization layers. The model also seemed to converge little faster. Though test classification rate was lower for about 5-7% (and a gap between classification rate on testing and training datasets was bigger) than one of the model without BN. I am not sure if BatchNormalization is properly removed from model in my solution (but openVINO model file doesn't include one so it's removed).

Remove BN and Dropout layers:

#Clear any previous session.
tf.keras.backend.clear_session()
#This line must be executed before loading Keras model.
tf.keras.backend.set_learning_phase(0) 
model = tf.keras.models.load_model(weights_path)

for layer in model.layers:
    layer.training = False
    if isinstance(layer, tf.keras.layers.BatchNormalization):
        layer._per_input_updates = {}
    elif isinstance(layer, tf.keras.layers.Dropout):
        layer._per_input_updates = {}

And than freeze session:

def freeze_session(session, keep_var_names=None, output_names=None, clear_devices=True):
"""
Freezes the state of a session into a pruned computation graph.

Creates a new computation graph where variable nodes are replaced by
constants taking their current value in the session. The new graph will be
pruned so subgraphs that are not necessary to compute the requested
outputs are removed.
@param session The TensorFlow session to be frozen.
@param keep_var_names A list of variable names that should not be frozen,
                    or None to freeze all the variables in the graph.
@param output_names Names of the relevant graph outputs.
@param clear_devices Remove the device directives from the graph for better portability.
@return The frozen graph definition.
"""
from tensorflow.python.framework.graph_util import convert_variables_to_constants
graph = session.graph
with graph.as_default():
    freeze_var_names = list(set(v.op.name for v in tf.global_variables()).difference(keep_var_names or []))
    output_names = output_names or []
    output_names += [v.op.name for v in tf.global_variables()]
    # Graph -> GraphDef ProtoBuf
    input_graph_def = graph.as_graph_def()
    if clear_devices:
        for node in input_graph_def.node:
            node.device = ""
    frozen_graph = convert_variables_to_constants(session, input_graph_def,
                                                output_names, freeze_var_names)
    return frozen_graph