why is gradient descent with momentum an exponentially weighted average?
I recently watched Andrew Ng's video on SGDM. I understand that the momentum term updates the gradient by weighting the last gradient and using a small component of V_dw. I don't understand why momentum is also known as exponentially weighted average. Also, in Ng's video at 6:37 he says using Beta = 0.9 effectively means using an average of the last 10 gradients. Can someone explain how that works? To me, it's just a scalar weighting of 1-0.9 to all the gradients in the vector dW.
Appreciate any insight! I feel like I'm missing something fundamental.
You just have to think about what is in your last gradient. The last gradient is already a weighted gradient, due to the momentum term.
In the first step you will just do a gradient descent. In the second step you will have a momentum gradient of m_grad_2 = grad_2 + 0.9 m_grad_1. In the third step you will again have a momentum gradient m_grad_3 = grad_3 + 0.9 m_grad_2, but the old gradient is composed of a momentum term. Therefore 0.9*m_grad_2 = 0.9 * (grad_2 + 0.9 grad_1), which is 0.9 grad_2 + 0.81 grad_1. Therefore the impact of a gradient on the kth step will be 0.9^k. After 10 steps the impact will be quite small.
- Can we use GNN on graphs with only edge features?
- How do I initialize weights in PyTorch?
- Does one convolutional filter always have different coefficients for each of the channels of the previous layer?
- How to use a Dense layer with an input that has a dynamically sized dimension?
- Should we do learning rate decay for adam optimizer
- InceptionResnetV2 STEM block keras implementation mismatch the one in the original paper?
- TypeError: '_IncompatibleKeys' object is not callable
- Debugging GAN covergence
- Loss is increasing from first epoch itself
- Handling Absence of Color Data in 3D Mesh Neural Network Input
- Can a neural network be trained while it changes in size?
- unexpected transformer's dataset structure after set_transform or with_transform
- Using scikit-learn's MLPClassifier in AdaBoostClassifier
- Do I fit t-SNE the two sets of embeddings from two different models at the same time or do I fit each separately then visualize and compare?
- Autograd error caused by ReLU in Pytorch?
- Early stopping in Bert Trainer instances
- Google Colab is very slow compared to my PC
- torch.nn.functional vs torch.nn - Pytorch
- Is a neural network a lazy or eager learning method?
- Why is the accuracy for my Keras model always 0 when training?
- Does calling the model.fit method again reinitialize the already trained weights?
- Why do we divide by batch_size in the backward pass of Softmax with Cross-Entropy Loss?
- Why do BF16 models have slower inference on Mac M-series chips compared to F16 models?
- Efficient way to compare two similar images
- why do we need a y variable in keras model.fit()?
- How to create a neural network for regression?
- training neural network for user recognition in MATLAB
- Values of weights and outputs in Convolutional Neural Networks
- How should I determine the input size for layers following the initial layer in a CNN?
- Assertion Failure in Eigen Matrix Broadcasting: Dimension Mismatch in Neural Network Forward Pass