residual neural network

A residual neural network (ResNet) is an artificial neural network (ANN) of a kind that builds on constructs known from pyramidal cells in the cerebral cortex. Residual neural networks do this by utilizing skip connections, or short-cuts to jump over some layers. Typical ResNet models are implemented with double- or triple- layer skips that contain nonlinearities (ReLu) and batch normalization in between. An additional weight matrix may be used to learn the skip weights; these models are known as HighwayNets. Models with several parallel skips are referred to as DenseNets. In the context of residual neural networks, a non-residual network may be described as a plain network. A residual neural network (ResNet) is an artificial neural network (ANN) of a kind that builds on constructs known from pyramidal cells in the cerebral cortex. Residual neural networks do this by utilizing skip connections, or short-cuts to jump over some layers. Typical ResNet models are implemented with double- or triple- layer skips that contain nonlinearities (ReLu) and batch normalization in between. An additional weight matrix may be used to learn the skip weights; these models are known as HighwayNets. Models with several parallel skips are referred to as DenseNets. In the context of residual neural networks, a non-residual network may be described as a plain network. One motivation for skipping over layers is to avoid the problem of vanishing gradients, by reusing activations from a previous layer until the adjacent layer learns its weights. During training, the weights adapt to mute the upstream layer, and amplify the previously-skipped layer. In the simplest case, only the weights for the adjacent layer's connection are adapted, with no explicit weights for the upstream layer. This works best when a single non-linear layer is stepped over, or when the intermediate layers are all linear. If not, then an explicit weight matrix should be learned for the skipped connection (a HighwayNet should be used). Skipping effectively simplifies the network, using fewer layers in the initial training stages. This speeds learning by reducing the impact of vanishing gradients, as there are fewer layers to propagate through. The network then gradually restores the skipped layers as it learns the feature space. Towards the end of training, when all layers are expanded, it stays closer to the manifold and thus learns faster. A neural network without residual parts explores more of the feature space. This makes it more vulnerable to perturbations that cause it to leave the manifold, and necessitates extra training data to recover. The brain has structures similar to residual nets, as cortical layer VI neurons get input from layer I, skipping intermediary layers. In the figure this compares to signals from the apical dendrite (3) skipping over layers, while the basal dendrite (2) collects signals from the previous and/or same layer. Similar structures exists for other layers. How many layers in the cerebral cortex compare to layers in an artificial neural network is not clear, nor whether every area in the cerebral cortex exhibits the same structure, but over large areas they appear similar. For single skips, the layers may be indexed either as ℓ − 2 { extstyle ell -2} to ℓ { extstyle ell } or as ℓ { extstyle ell } to ℓ + 2 { extstyle ell +2} . (Script ℓ { extstyle ell } used for clarity, usually it is written as a simple l.) The two indexing systems are convenient when describing skips as going backward or forward. As signal flows forward through the network it is easier to describe the skip as ℓ + k { extstyle ell +k} from a given layer, but as a learning rule (back propagation) it is easier to describe which activation layer you reuse as ℓ − k { extstyle ell -k} , where k − 1 { extstyle k-1} is the skip number. Given a weight matrix W ℓ − 1 , ℓ { extstyle W^{ell -1,ell }} for connection weights from layer ℓ − 1 { extstyle ell -1} to ℓ { extstyle ell } , and a weight matrix W ℓ − 2 , ℓ { extstyle W^{ell -2,ell }} for connection weights from layer ℓ − 2 { extstyle ell -2} to ℓ { extstyle ell } , then the forward propagation through the activation function would be (aka HighwayNets)