Towards a unified theory of efficient, predictive and sparse coding
4
Citation
35
Reference
10
Related Paper
Citation Trend
Abstract:
A central goal in theoretical neuroscience is to predict the response properties of sensory neurons from first principles. Several theories have been proposed to this end. “Efficient coding” posits that neural circuits maximise information encoded about their inputs. “Sparse coding” posits that individual neurons respond selectively to specific, rarely occurring, features. Finally, “predictive coding” posits that neurons preferentially encode stimuli that are useful for making predictions. Except in special cases, it is unclear how these theories relate to each other, or what is expected if different coding objectives are combined. To address this question, we developed a unified framework that encompasses these previous theories and extends to new regimes, such as sparse predictive coding. We explore cases when different coding objectives exert conflicting or synergistic effects on neural response properties. We show that predictive coding can lead neurons to either correlate or decorrelate their inputs, depending on presented stimuli, while (at low-noise) efficient coding always predicts decorrelation. We compare predictive versus sparse coding of natural movies, showing that the two theories predict qualitatively different neural responses to visual motion. Our approach promises a way to explain the observed diversity of sensory neural responses, as due to a multiplicity of functional goals performed by different cell types and/or circuits.Keywords:
Predictive coding
Neural coding
ENCODE
Decorrelation
A central goal in theoretical neuroscience is to predict the response properties of sensory neurons from first principles. To this end, "efficient coding" posits that sensory neurons encode maximal information about their inputs given internal constraints. There exist, however, many variants of efficient coding (e.g., redundancy reduction, different formulations of predictive coding, robust coding, sparse coding, etc.), differing in their regimes of applicability, in the relevance of signals to be encoded, and in the choice of constraints. It is unclear how these types of efficient coding relate or what is expected when different coding objectives are combined. Here we present a unified framework that encompasses previously proposed efficient coding models and extends to unique regimes. We show that optimizing neural responses to encode predictive information can lead them to either correlate or decorrelate their inputs, depending on the stimulus statistics; in contrast, at low noise, efficiently encoding the past always predicts decorrelation. Later, we investigate coding of naturalistic movies and show that qualitatively different types of visual motion tuning and levels of response sparsity are predicted, depending on whether the objective is to recover the past or predict the future. Our approach promises a way to explain the observed diversity of sensory neural responses, as due to multiple functional goals and constraints fulfilled by different cell types and/or circuits.
Predictive coding
Neural coding
ENCODE
Deep Neural Networks
Cite
Citations (155)
Predictive coding
Neural coding
Neurophysiology
Computational neuroscience
Cite
Citations (1,013)
Decorrelation
SIGNAL (programming language)
Spatial correlation
Cite
Citations (1)
A computer model based on the impulse response method to simulate the ultrasound field of a 30 MHz transducer was developed for studying the decorrelation properties of echo signals from an intravascular scanner. The decorrelation due to (1) lateral displacement, (2) axial displacement, (3) noise level, (4) velocity distribution and (5) variance in correlation estimate were investigated. The results show that the lateral decorrelation function can be approximated by a linear model, which provides a basis for flow quantitation by decorrelation processing. Decorrelation sources unrelated to the motion of blood particles such as noise can be ignored at intravascular imaging conditions. In conclusion, this study provides insight into the parameters that affect decorrelation-based flow assessment and predicts the feasibility of such an approach using IVUS techniques.
Decorrelation
Intravascular Ultrasound
Impulse response
Cite
Citations (1)
The goal of predictive sparse coding is to learn a representation of examples as sparse linear combinations of elements from a dictionary, such that a learned hypothesis linear in the new representation performs well on a predictive task. Predictive sparse coding has demonstrated impressive performance on a variety of supervised tasks, but its generalization properties have not been studied. We establish the first generalization error bounds for predictive sparse coding, in the overcomplete setting, where the number of features k exceeds the original dimensionality d. The learning bound decays as O(√dk/m) with respect to d, k, and the size m of the training sample. It depends intimately on stability properties of the learned sparse encoder, as measured on the training sample. Consequently, we also present a fundamental stability result for the LASSO, a result that characterizes the stability of the sparse codes with respect to dictionary perturbations.
Neural coding
Predictive coding
Lasso
K-SVD
Cite
Citations (31)
The proposal that cortical activity in the visual cortex is optimized for sparse neural activity is one of the most established ideas in computational neuroscience. However, direct experimental evidence for optimal sparse coding remains inconclusive, mostly due to the lack of reference values on which to judge the measured sparseness. Here we analyze neural responses to natural movies in the primary visual cortex of ferrets at different stages of development and of rats while awake and under different levels of anesthesia. In contrast with prediction from a sparse coding model, our data shows that population and lifetime sparseness decrease with visual experience, and increase from the awake to anesthetized state. These results suggest that the representation in the primary visual cortex is not actively optimized to maximize sparseness.
Neural coding
Predictive coding
Neural Activity
Representation
Cite
Citations (30)
A central goal in theoretical neuroscience is to predict the response properties of sensory neurons from first principles. Several theories have been proposed to this end. “Efficient coding” posits that neural circuits maximise information encoded about their inputs. “Sparse coding” posits that individual neurons respond selectively to specific, rarely occurring, features. Finally, “predictive coding” posits that neurons preferentially encode stimuli that are useful for making predictions. Except in special cases, it is unclear how these theories relate to each other, or what is expected if different coding objectives are combined. To address this question, we developed a unified framework that encompasses these previous theories and extends to new regimes, such as sparse predictive coding. We explore cases when different coding objectives exert conflicting or synergistic effects on neural response properties. We show that predictive coding can lead neurons to either correlate or decorrelate their inputs, depending on presented stimuli, while (at low-noise) efficient coding always predicts decorrelation. We compare predictive versus sparse coding of natural movies, showing that the two theories predict qualitatively different neural responses to visual motion. Our approach promises a way to explain the observed diversity of sensory neural responses, as due to a multiplicity of functional goals performed by different cell types and/or circuits.
Predictive coding
Neural coding
ENCODE
Decorrelation
Cite
Citations (4)
Neural circuits reliably encode and transmit signals despite the presence of noise at multiple stages of processing. The efficient coding hypothesis, a guiding principle in computational neuroscience, suggests that a neuron or population of neurons allocates its limited range of responses as efficiently as possible to best encode inputs while mitigating the effects of noise. Previous work on this question relies on specific assumptions about where noise enters a circuit, limiting the generality of the resulting conclusions. Here we systematically investigate how noise introduced at different stages of neural processing impacts optimal coding strategies. Using simulations and a flexible analytical approach, we show how these strategies depend on the strength of each noise source, revealing under what conditions the different noise sources have competing or complementary effects. We draw two primary conclusions: (1) differences in encoding strategies between sensory systems—or even adaptational changes in encoding properties within a given system—may be produced by changes in the structure or location of neural noise, and (2) characterization of both circuit nonlinearities as well as noise are necessary to evaluate whether a circuit is performing efficiently.
ENCODE
Neural coding
Generality
Biological neural network
Limiting
Cite
Citations (51)
A new blind separation algorithm without nonlinear functions is proposed. The algorithm is derived through the decorrelation process near an equilibrium point. The algorithm utilizes the correlation between its outputs and the sensor signals in the decorrelation process. The validity and performance of the algorithm are confirmed through computer simulations.
Decorrelation
Separation (statistics)
Cite
Citations (0)
The perceptual effects of interchannel decorrelation, using a method proposed by Lauridsen, have been investigated subjectively, looking specifically at the frequency dependency of decorrelation. Twelve subjects graded the perceived auditory image width of a pink noise sample that had been decorrelated by a Lauridsen decorrelator algorithm, varying the frequency-band, time-delay and decorrelation factor for each sample. The same test has been carried out in both the horizontal and vertical planes. Results generally indicate that decorrelation is more effective horizontally than vertically. For horizontal decorrelation, the higher the frequency, the more effective the decorrelation, with a longer time-delay required for lower frequencies. In contrast, the vertical width produced by vertical decorrelation is better perceived at lower frequencies than higher ones.
Decorrelation
Cite
Citations (6)