pythontensorflowkerasreproducible-research

How to get reproducible weights initializaiton in Keras?


  1. I set both numpy and tensorflow random seeds as suggested
  2. Generate some data - this part is reproducible, gives same results always
  3. Create a simple network and make a prediction (without training, just with random weights) - prediction is different every time
import numpy as np
from tensorflow.keras.layers import Dense, Dropout
from tensorflow.keras import Sequential, optimizers
import tensorflow as tf

np.random.seed(32)
tf.set_random_seed(33)

random_data = np.random.rand(10, 2048)
print(random_data[:,0])

def make_classifier():
    model = Sequential()
    model.add(Dense(1024, activation='relu', input_dim=2048))
    model.add(Dropout(0.5))
    model.add(Dense(1, activation='sigmoid'))
    model.compile(optimizer=optimizers.RMSprop(lr=2e-4),
              loss='binary_crossentropy')
    return model
model = make_classifier()
# model.summary()
model.predict(random_data)

When I re-run the whole cell, the print statement always outputs [0.85888927 0.23846818 0.17757634 0.07244977 0.71119893 0.09223853 0.86074647 0.31838194 0.7568638 0.38197083]. However the prediction is different every time:

array([[0.5825965 ],
       [0.8677979 ],
       [0.70151913],
       [0.64572096],
       [0.78101623],
       [0.76483005],
       [0.7946336 ],
       [0.6281612 ],
       [0.8208673 ],
       [0.8273002 ]], dtype=float32)
array([[0.51012236],
       [0.6562015 ],
       [0.5593666 ],
       [0.686155  ],
       [0.6488372 ],
       [0.5966359 ],
       [0.6236731 ],
       [0.58099884],
       [0.68447435],
       [0.58886844]], dtype=float32)

And so on.

  1. How do I get reproducible prediction for a just-initialized network?
  2. When does the weights initialization happen exactly? when I compile the model or?..

Solution

  • tf.keras.initializers objects have a seed argument for reproducible initialization.

    import tensorflow as tf
    import numpy as np
    
    initializer = tf.keras.initializers.GlorotUniform(seed=42)
    
    for _ in range(10):
        print(np.round(initializer((4,)), 3))
    
    [-0.377 -0.003  0.373 -0.831]
    [-0.377 -0.003  0.373 -0.831]
    [-0.377 -0.003  0.373 -0.831]
    [-0.377 -0.003  0.373 -0.831]
    [-0.377 -0.003  0.373 -0.831]
    [-0.377 -0.003  0.373 -0.831]
    [-0.377 -0.003  0.373 -0.831]
    [-0.377 -0.003  0.373 -0.831]
    [-0.377 -0.003  0.373 -0.831]
    [-0.377 -0.003  0.373 -0.831]
    

    In a Keras layer, you can use it like this:

    tf.keras.layers.Dense(1024, 
                          activation='relu', 
                          input_dim=2048,
                          kernel_initializer=tf.keras.initializers.GlorotUniform(seed=42))