Measuring Saturation in Neural Networks
2015
In the neural network context, the phenomenon of saturation refers to the state in which a neuron predominantly outputs values close to the asymptotic ends of the bounded activation function. Saturation damages both the information capacity and the learning ability of a neural network. The degree of saturation is an important neural network characteristic that can be used to understand the behaviour of the network itself, as well as the learning algorithm employed. This paper suggests a measure of saturation for bounded activation functions. The suggested measure is independent of the activation function range, and allows for direct comparisons between different activation functions.
Keywords:
- Degree of saturation
- Recurrent neural network
- Types of artificial neural networks
- Time delay neural network
- Physical neural network
- Machine learning
- Saturation (chemistry)
- Artificial intelligence
- Activation function
- Rectifier (neural networks)
- Computer science
- Stochastic neural network
- Topology
- Artificial neural network
- Probabilistic neural network
- Correction
- Source
- Cite
- Save
- Machine Reading By IdeaReader
16
References
18
Citations
NaN
KQI