I'm a bit confused by activation functions and blogs/posts that continuously mention that neurons are not activated or not fired.
But mathematically speaking, if whatever activation function (whether it's sigmoid, tanh, relu) calculates an output of 0, isn't that value still given to all connected neurons in the next layer? And if so, doesn't that mean that this neuron still firing/activating?
Or am I simply wrong and the neuron is really not firing and it really doesn't provide any value at all to any connected neurons in the next layer? And how does this work mathematically?
Expressions such as not activated and not fired, as well as the term neuron itself, are just metaphorical depictions, and they should not be taken at face value. They are used just to verbally describe the (very) loose analogy between the (artificial) neural networks used in machine learning and the actual neuronal networks of living beings, but that's all.
As you correctly suspect, in such cases an output value of 0 is indeed produced by the "neuron" and propagated in the net. Because, actually, there are not any neurons there, just variables in a computer program, which must have a value at all times, for both mathematical and computational reasons.