Hamming Clustering: Improving Generalization in Binary Neural Networks

An important reason that gives rise to current increasing interest in neural networks is undoubtedly their ability to generalize the information contained in a given set of patterns. In fact, in a large number of real applications a proper neural network is able to provide correct outputs for patterns not belonging to its training set. The comprehension of this phenomenon has been the object of many papers because of its practical consequences (Denker et al., 1987; Baum & Haussler, 1989). With regard to classification problems it has been recently observed that simple statistical methods, like nearest neighbor algorithm, show a high generalization ability in many real situations (Bottou & Vapnik, 1992). Consequently, corresponding functional dependences between inputs and outputs have some locality properties that can be used to accelerate and/or improve the learning process. If we limit ourselves to binary neural networks, then a locality property can be formulated in the following way: it is more likely that patterns at a low Hamming distance belong to the same class (i.e. have the same outputs). In the following section a new method, called Hamming Clustering (HC), is presented, that allows the construction of a given neural network in such a way to satisfy this locality property. This technique can be directly inserted in a particular constructive method, such as the upstart algorithm (Frean, 1990) or the procedure of sequential learning (Marchand et al., 1990). Moreover, the Hamming clustering considerably reduces the number of connections in the input layer by neglecting redundant weights.