Characterising neutrality in neural network error landscapes

The characterisation of topographical features of fitness landscapes can provide significant insight into the nature of underlying optimisation problems and the behaviour of metaheuristic search algorithms. Neutrality as a landscape feature is often overlooked in continuous problems, but researchers have theorised that the presence of neutral regions on neural network error surfaces may be an impediment to current population-based search algorithms for training neural networks. An empirical approach to measuring the amount of neutrality would provide a stepping stone for future studies on the effects of neutrality. To date, there is no offline technique to achieve this in continuous domains. This paper proposes two normalised measures of neutrality based on a progressive random walk algorithm. Measurements are shown to agree with visual analysis of two-dimensional benchmark problems, and are shown to scale well to higher dimensions. The measures are ultimately applied to neural network classification problems where saturation-induced neutrality is confirmed.