Mismatch between computational complexity of Additive attention and RNN cell
According to
Attention is all you needpaper: Additive attention (The classic attention use in RNN by Bahdanau) computes the compatibility function using a feed-forward network with a single hidden layer. While the two are similar in theoretical complexity, ...
Indeed, we can see here that the computational complexity of additive attention and dot-prod (transformer attention) are both n²*d.
However, if we look closer at additive attention, it is in fact a RNN cell which have a computational complexity of n*d² (according to the same table).
Thus, shouldn't the computational complexity of additive attention be n*d² instead of n²*d ?
Your claim that additive attention is in fact a RNN cell is what is leading you astray. Additive attention is implemented using a fully-connected shallow (1 hidden layer) feedforward neural network "between" the encoder and decoder RNNs as shown below and described in the original paper by Bahdanau et al. (pg. 3) [1]:
... an alignment model which scores how well the inputs around position
jand the output at positionimatch. The score is based on the RNN hidden states_i − 1(just before emittingy_i, Eq. (4)) and thej-th annotationh_jof the input sentence.We parametrize the alignment model
aas a feedforward neural network which is jointly trained with all the other components of the proposed system...
Figure 1: Attention mechanism diagram from [2].
Thus, the alignment scores are calculated by adding the outputs of the decoder hidden state to the encoder outputs. So the additive attention is not a RNN cell.
References
[1] Bahdanau, D., Cho, K. and Bengio, Y., 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
[2] Arbel, N., 2019. Attention in RNNs. Medium blog post.
- How does one set the pad token correctly (not to eos) during fine-tuning to avoid model not predicting EOS?
- text categorization classifiers
- Why does AWS SageMaker run a web server for batch transform?
- Q-learning with a state-action-state reward structure and a Q-matrix with states as rows and actions as columns
- What are the common techniques for feature engineering and presenting the ML model?
- xgboost in R: how does xgb.cv pass the optimal parameters into xgb.train
- Statistical learning confusion table variable
- KMeans clustering in PySpark
- Is there a way to extract information from contracts using ML with including contract files and targeted strings as inputs and outputs?
- Training different stage of model with different loss
- How can I properly split imbalanced dataset to train and test set?
- How can I get the confidence variable from a CoreML prediction
- Deep learning for inferences in sequences
- How is log_softmax() implemented to compute its value (and gradient) with better speed and numerical stability?
- Is a neural network a lazy or eager learning method?
- bnlearn wrong dimensions for node
- Why is the accuracy for my Keras model always 0 when training?
- Can we use Pydantic models (BaseModel) directly inside model.predict() using FastAPI, and if not ,why?
- How to use MFCCs in Weka for audio classification?
- Creating a train, test split for data nested in multiple folders
- Custom Encoder and Decoder Layers within Keras Model Show As Unbuilt
- Bee swarm plot with SHAP values for Random forest
- Multiclass classifier using Keras from 'Deep Learning with Python' yields very different accuracy compared to what's in the textbook
- `TypeError: Unsupported integer size (0)` when attempted to save custom Keras model
- MLmetrics F1_Score function
- tensorflow TypeError: cannot unpack non-iterable float object
- AttributeError : 'tuple' has no attribute 'to'
- Understanding How YOLO is trained
- Confused about YOLO Process
- Columns are missing after imputing and creating dummy variables. How should I fix this?