Open Collections

(with Paul Wilson) Categorical Foundations of Gradient-Based Learning

Banff International Research Station for Mathematical Innovation and Discovery

We propose a categorical foundation of gradient-based machine learning algorithms in terms of lenses, parametrised maps, and reverse derivative categories. This foundation provides a powerful explanatory and unifying framework: it encompasses a variety of gradient descent algorithms such as ADAM, AdaGrad, and Nesterov momentum, as well as a variety of loss functions such as as MSE and Softmax cross-entropy, shedding new light on their similarities and differences. Our approach also generalises beyond neural networks (modelled in categories of smooth maps), accounting for other structures relevant to gradient-based learning such as boolean circuits. Finally, we also develop a novel implementation of gradient-based learning in Python, informed by the principles introduced by our framework.

Mathematics; Category Theory; Homological Algebra; Algebraic Topology; Algebraic Structures

video/mp4

Author affiliation: University of Strathclyde

2023-10-28

Attribution-NonCommercial-NoDerivatives 4.0 International

http://hdl.handle.net/2429/86371

Unreviewed

http://creativecommons.org/licenses/by-nc-nd/4.0/