neural-networkdeep-learningbackpropagationmomentumfeed-forward

Adding momentum term in online back propagation weight update?


I have implemented the ANN for two-layer network, I need to modify my weight update code with momentum, but i need to know how can i update it. below is the code snap of only weight update. The below code updates weight for each example it has seen, The hiddenWights are the hidden layer weight and the outputWeights are output layer weights.

 for examplen = 1: nTrainingExamples
           inputVector = inputs(:,examplen);
           HiddenLayerOutput = sigmoid( hiddenWeights * inputVector);
           OutputLayerOutput = sigmoid( outputWeights * HiddenLayerOutput);

           l2_error = OutputLayerOutput - targets(:, examplen);
           l2_delta = learningRates(1, i) .* (OutputLayerOutput .* (1 - OutputLayerOutput)) .* l2_error;
           l2_v = mu * l2_v - l2_delta * t;

           l1_delta = learningRates(1, i) .* (HiddenLayerOutput .* (1 - HiddenLayerOutput)) .* (outputWeights' * l2_delta);
           l1_v = mu * l1_v - l1_delta * t;
           % weights = weights + v
           outputWeights = outputWeights + (l2_v*HiddenLayerOutput');
           hiddenWeights = hiddenWeights + (l1_v*inputVector');
       end

Solution

  • From our conversation it turn out that you want to implement a classic momentum method (not Nestrov). So you need an additional parameter - the speed v and time t and momentum mu constants.

    At the beginning of learning you should set your speed to 0 :

    v = 0
    

    and choose some value of t (e.g. set it to 0.1) and mu (e.g. 0.9).

    During each iterations of training you should change your weights according to the following rule :

    v = mu * v - delta * t
    weights = weights + v
    

    Parameters mu and t are meta parameters and you can find good values of them by a grid or random search.