Mathematics Colloquia and Seminars

Return to Colloquia & Seminar listing

Humans and other animals exhibit learning abilities and understanding of the world far beyond the capabilities of current AI and machine learning systems. Such abilities are driven largely by intrinsic objectives without external supervision. Unsupervised representation learning (aka self-supervised representation learning) aims to build models that find patterns in data automatically and reveal the patterns underlying data explicitly with a representation. Two fundamental goals in unsupervised representation learning are to model natural signal statistics and to model biological sensory systems. These are intertwined because many properties of the sensory system are adapted to the statistical structure of natural signals.

First, we can formulate unsupervised representation learning from neural and statistical principles. Sparsity provides a good account of neural response properties at the early stages of sensory processing, and low-rank spectral embedding can model the essential degrees of freedom of high-dimensional data. This approach leads to the sparse manifold transform (SMT), and offers a way to exploit the structure in a sparse code to straighten natural non-linear transformations and learn higher-order structure at later stages of processing. Second, we can use reductionism to demonstrate that the success of state-of-the-art joint-embedding self-supervised learning methods can be unified and explained by a distributed representation of image patches. These two seemingly unrelated approaches have a surprising convergence --- we can show that they share the same learning objective, and their benchmark performance can also be closed significantly. The evidence outlines an exciting direction for building a theory of unsupervised hierarchical representation and explains how the visual cortex performs hierarchical manifold disentanglement. The tremendous advancement in this field also provides new tools for modeling emerging signal modalities, and promises unparalleled scalability for future data-driven machine learning. However, these innovations are only steps on the path to building an autonomous machine intelligence tha.t can learn as efficiently as humans and animals. In order to achieve this grand goal, we must venture far beyond classical notions of representation learning.