I am creating a model in tensorflow with all layers having relu as the activation layer. However, when the batch size increases to 500, I want to change the model such that the second last layer to the output layer has sigmoid activation layer.
What I am confused about is that d I need to re-initialize all the variables since I am replacing the optimizer in the middle? Or do I keep the old variables?
This is a very interesting question. I think it depends on your datasets and models.
Yes: Perhaps, you can use weights (before batch size 500) as pre-trained like what Deep Belief Networks (with RBM) do.
No: Perhaps, these pre-trained weights hurt your model and may not better than other good initializers such as xavier initializer https://www.tensorflow.org/versions/r0.8/api_docs/python/contrib.layers.html#xavier_initializer
I think it's worth try both options.