As a reference model I used a comparably small network:
|1||Convolution||10 filters 3x3|
|2||Convolution||20 filters 3x3, pooling 2x2|
|3||Convolution||64 filters 3x3, pooling 2x2|
|4||Convolution||64 filters 3x3, pooling 2x2|
|5||Convolution||128 filters 3x3, pooling 2x2|
Then I trained six models: a reference one, then I fixed first layer, first and second and so forth. The idea is illustrated on the picture (grey colour means, that the weights are fixed):
I plotted training and validation error during training.
We can see, that models with fixed parameters output reasonable results. Also it is interesting, that fixing weights sometimes gives regularizing effect (like on blue, magenta and yellow lines). It is not so amazing, because we are decreasing the capacity of the model.