Open Collections

Mohamadi, Mohamad Amin

University of British Columbia

Studying neural networks in the limit of infinite-width has provided us with numerous valuable theoretical and practical insights about the initialization of NNs, their training dynamics and properties of the learnt functions. One of the theoretical tools emerged from this study is the empirical Neural Tangent Kernel (eNTK). The eNTK can provide a good understanding of a given network’s representation: they are often far less expensive to compute and applicable more broadly than infinite-width NTKs. In this work, we use eNTKs to predict the local dynamics of neural networks in an active learning setup to propose a new method for approximating active learning acquisition strategies that are based on retraining with hypothetically-labeled candidate data points. Furthermore, to tackle the notorious space and computational complexity of calculating eNTKs, we propose a fast approximation of eNTK using a block-diagonal kernel resulting from eNTK with respect to only one (or average) of the output neurons of a network. We further use this approximation in our proposed “look-ahead” strategies in deep active learning. We finally present empirical evidence that our querying strategy beats other look-ahead strategies by large margins, and achieves equal or better performance compared to state-of-the-art methods on several benchmark datasets in pool-based active learning.

Text

2023-05-02

Attribution-NonCommercial-NoDerivatives 4.0 International

http://hdl.handle.net/2429/84542

Computer Science

University of British Columbia

UBCV

http://creativecommons.org/licenses/by-nc-nd/4.0/