Mathematics Colloquia and Seminars

Return to Colloquia & Seminar listing

In this talk I will present new results on neural networks with linear threshold activation functions. The class of functions that are representable by such neural networks can be fully characterized, and two hidden layers are necessary and sufficient to represent any function in the class. This is a surprising result in the light of recent exact representability investigations for neural networks using other popular activation functions like rectified linear units. I will present nearly tight bounds on the sizes of the neural network, as well as an algorithm to solve the empirical risk minimisation (ERM) problem to global optimality for these neural networks with a fixed architecture. The algorithm’s running time is polynomial in the size of the data sample, if the input dimension and the size of the network architecture are considered fixed constants. Finally I will present a new type of architecture, the shortcut linear threshold networks, a strict superclass of the rectified linear units (ReLU) neural networks, which has several desirable theoretical properties. In particular, the ERM problem can also be solved to global optimality with a similar algorithm.