Problem : What is the optimal initial weights for the convolutions networks.

Answer :  One should generally initialize initial weights with small noise for symmetry breaking and to prevent 0 gradient. In case of Relu (Rectifier Neural networks), it is also advised to use slightly positive values to prevent dead neurons.

initial = tf.truncated_normal(shape, stddev=0.1)

 

Ref:

https://www.tensorflow.org/get_started/mnist/pros

Advertisements