neural-networklayeractivation-function

Neural network - exercise


I am currently learning for myself the concept of neural networks and I am working with the very good pdf from http://neuralnetworksanddeeplearning.com/chap1.html

There are also few exercises I did, but there is one exercise I really dont understand, at least one step

Task: There is a way of determining the bitwise representation of a digit by adding an extra layer to the three-layer network above. The extra layer converts the output from the previous layer into a binary representation, as illustrated in the figure below. Find a set of weights and biases for the new output layer. Assume that the first 3 layers of neurons are such that the correct output in the third layer (i.e., the old output layer) has activation at least 0.99, and incorrect outputs have activation less than 0.01. enter image description here

I found also the solution, as can be seen on the second image enter image description here

I understand why the matrix has to have this shape, but I really struggle to understand the step, where the user calculates

0.99 + 3*0.01
4*0.01

I really don't understand these two steps. I would be very happy if someone can help me understand this calculation

Thank you very much for help


Solution

  • Output of previous layer is 10x1(x). Weight matrix is 4x10. New output layer will be 4x1. There are two assumption first:

    0.010 + 0.991 + 0.010 + 0.011 + 0.010 + 0.011 + 0.010 + 0.011 + 0.010 + 0.011 = 1.03

    So I believe there is a typo, because author probably assume 4 ones at first row of W, which is not true, because there is 5 ones. Because if there was 4 ones at first first row, than really results will be 0.04 for 0.99 at first row of x and 1.02 for 0.99 at second row of x.

    (Sorry, different author here): The original answer should be correct, notice how the author of the answer notes the 0th column, which means that he/she indicates that the rows and columns are zero indexed. So yes, there are 4 ones in the first row of W (second row in the perspective above).