Lowering variance of decisions by using artificial neural network portfolios

Artificial neural networks (ANNs) usually take a long time to train. Experimenters often find that a number of parameters have to be tuned manually before a network that manifests reasonable performance is obtained. For example, in the case of backpropagation (Rumelhart et al. 19861, these parameters include the learning rate, number of units at each layer, the connectivity among the nodes, the function to be executed at each node, offline vs. online error propagation, and the momentum term. The choice of the learning algorithm (Hebbian learning, standard backpropagation, etc.) adds another degree of freedom in this model-building exercise. Instead of hand-picking (by trial and error) a good point in the multidimensional parameter space described above, it is desirable to automate the entire training process, including the choice of values for parameters such as topology and learning rate. However, automating the entire process can be a very difficult task. An attractive, albeit indirect, way of dealing with this problem is to train a number of ANNs (on the same data, using different learning algorithms, momentum rates, learning rates, topologies, etc.) and to use this portfolio of ANNs to make the final decision. Portfolio theory (starting with Markowitz 1952) provides the rationale for such an approach and the purpose of this note is to draw attention to this fact. Consider a portfolio of ANNs with the individual network decisions denoted by d, and the variance of the decisions denoted by a2(d;). The expected decision of a portfolio of N nets is given by