QAT output nodes for Quantized Model got the same min max range
Recently, I have worked on quantization aware training on tf1.x to push the model to Coral Dev Board. However, when I finished training the model, why is my min max of my 2 outputs fake quantization is the same?
Should it be different when one's maximum target is 95 and one is 2pi?
I have figured out the problem. It is the problem when that part of the model is not really trained QAT. This happens for the output node that somehow forgets to QAT when training. The -6 and 6 values come from the default source of the quantization of tf1.x as mention here
To overcome the problem, we should provide some op to trigger the QAT for the output nodes. In my regression case, I add a dummy op: tf.maximum(output,0) in the model to make the node QAT. If your output is strictly between 0-1, applying "sigmoid" activation at output instead of relu can also solve the problems.
- How do I initialize weights in PyTorch?
- Does one convolutional filter always have different coefficients for each of the channels of the previous layer?
- Obtain the output of intermediate layer (Functional API) and use it in SubClassed API
- Optuna Hyperband Algorithm Not Following Expected Model Training Scheme
- Broadcasting multiple versions of X_data that pair with the same y_data
- Low validation accuracy during data training using lfw datasets
- How to make TensorFlow use 100% of GPU?
- Contrastive Loss from Scratch
- Error loading Pytorch model checkpoint: _pickle.UnpicklingError: invalid load key, '\x1f'
- RuntimeError: Trying to backward through the graph a second time on loss tensor
- InceptionResnetV2 STEM block keras implementation mismatch the one in the original paper?
- How to correctly apply LayerNorm after MultiheadAttention with different input shapes (batch_first vs default) in PyTorch?
- Debugging GAN covergence
- Facing this error while classifying Images, containing 10 classes in pytorch, in ResNet50. My code is:
- Loss is increasing from first epoch itself
- Does pytorch CNN care about image size?
- Failed to use OnnxRuntime on Windows 7
- Handling Absence of Color Data in 3D Mesh Neural Network Input
- Questions about training LLMs on large text datasets for text generation from scratch
- What's the difference between "hidden" and "output" in PyTorch LSTM?
- HuggingFace Transformers For Text Generation with CTRL with Google Colab's free GPU
- Batch normalisation during testing
- "AttributeError: 'str' object has no attribute 'decode' " while Loading a Keras Saved Model
- Not all points are within the bounds of the space error in Scikit-Optimize
- How to Implement Softmax, in python, whereby the input are signed 8 integers
- "NotImplementedError: Could not run 'aten::add.out' with arguments from the 'QuantizedCPU' backend" while implementing QAT on resnet18 using pytorch
- Does peft train newly initialized weights?
- On which device is a python dictionary containing pytorch tensors that are loaded on cuda?
- Unexpected '__mul__' call during dot product
- Early stopping in Bert Trainer instances