Bruno Olshausen

Olshausen's research focuses on understanding the information processing strategies employed by the visual system for tasks such as object recognition and scene analysis. His approach combines studying neural response properties with mathematical modeling to develop functional theories of vision. This work aims to both advance understanding of brain function and develop new algorithms for image analysis based on biological principles. He has also contributed to technological applications, including image and signal processing, alternatives to backpropagation for unsupervised learning, memory storage and computation, analog data compression systems, etc.

One of Olshausen's most significant contributions is demonstrating how the principle of sparse coding can explain response properties of neurons in visual cortex. His 1996 paper in Nature with David J. Field showed how simple cells in the V1 cortex receptive field properties could emerge from learning a sparse code for natural images.^[3] This paper is based on two previous reports that gave additional technical details.^[4][5]

The paper argued that simple cells have Gabor-like, localized, oriented, and bandpass receptive fields. Previous methods, such as generalized Hebbian algorithm, obtains Fourier-like receptive fields that are not localized or oriented. But with sparse coding, such receptive fields do emerge.

Specifically, consider an image ${\displaystyle I}$ and some receptive fields ${\displaystyle \phi _{1},\dots ,\phi _{m}}$. An image can be approximately represented as a linear sum of the receptive fields: ${\displaystyle I\approx \sum _{i}a_{i}\phi _{i}}$. If so, then the image can be coded as ${\displaystyle (a_{1},\dots ,a_{m})}$, a code which may have better properties than directly coding for the pixel values of the image.

The algorithm proceeds as follows:^[4]

• Initialize ${\displaystyle \sigma _{i}=1}$ for all ${\displaystyle i}$, initialize ${\displaystyle \lambda }$ to a good value.
• Choose a bell-shaped function ${\displaystyle S(x)}$. Examples include ${\textstyle S(x)=|x|,\ln(1+x^{2}),-e^{-x^{2}}}$

• Loop
  • Sample a batch of images ${\displaystyle I}$.
    • For each image ${\displaystyle I}$ in the batch, solve for the coefficients ${\displaystyle a(I)=(a_{1}(I),a_{2}(I),\dots )}$ that minimize the loss function$${\displaystyle L(a):=\|I-\sum _{i}a_{i}\phi _{i}\|^{2}+\lambda \sum _{i}S(a_{i}/\sigma _{i})}$$
    • Define the reconstructed image ${\displaystyle {\hat {I}}(I):=\sum _{i}a_{i}(I)\phi _{i}}$.
  • Update each feature ${\displaystyle \phi _{i}}$ by Hebbian learning: ${\textstyle \phi _{i}\leftarrow \phi _{i}+\eta E[a_{i}(I-{\hat {I}})]}$. Here, ${\displaystyle \eta }$ is the learning rate and the expectation is over all images ${\displaystyle I}$ in the batch.
  • Update each ${\displaystyle \sigma _{i}^{2}}$ by ${\textstyle \sigma _{i}^{2}\leftarrow E[a_{i}^{2}]}$. Adjust learning rate.

The key part of the algorithm is the loss function$${\displaystyle L(a):=\|I-\sum _{i}a_{i}\phi _{i}\|^{2}+\lambda \sum _{i}S(a_{i}/\sigma _{i})}$$where the first term is image reconstruction loss, and the second term is the sparsity loss. Minimizing the first term leads to accurate image reconstruction, and minimizing the second term leads to sparse linear coefficients, that is, a vector ${\displaystyle (a_{1},\dots ,a_{m})}$ with many almost-zero entries. The hyperparameter ${\displaystyle \lambda }$ balances the importance of image reconstruction vs sparsity.

Based on the 1996 paper, he worked out a theory that the Gabor filters appearing in the V1 cortex performs sparse coding with overcomplete basis set, such that it is optimal for images occurring in the natural habitat of humans.^[6][7]

• Homepage at Redwood Center for Theoretical Neuroscience
• Google Scholar profile