I'm trying to calculate the entropy of the weights while training my graph and to use it for regularization. This of course involves w*tf.log(w), and as my weights are changing some of them are bound to get into a region which results in NaNs being returned.
Ideally I would include a line in my graph setup:
w[tf.is_nan(w)] = <number>
but tensorflow doesn't support assigning like that. I could of course create an operation, but that wouldn't work because I need for it to happen during the execution of the entire graph. I can't wait for the graph to execute and then 'fix' my weights, is has to be part of the graph execution.
I haven't been able to find an equivalent to np.nan_to_num in the docs.
Anybody have an idea?
(For obvious reasons, adding an epsilon doesn't work)
I think you need to use tf.select.
w = tf.select(tf.is_nan(w), tf.ones_like(w) * NUMBER, w); #if w is nan use 1 * NUMBER else use element in w
Update: TensorFlow 1.0 has deprecated tf.select in favor of Numpy compatible tf.where.