Open Collections

Abstract

Deep Neural Networks (DNNs) have achieved remarkable success in various machine learning domains, but their training remains computationally intensive and susceptible to architectural choices. Residual Networks (ResNets) mitigate some of these problems. However, they still face challenges with rapid weight and bias changes between layers, leading to unstable training, slow convergence, and over-fitting, especially with limited data. This thesis addresses these limitations by exploring Neural Ordinary Differential Equation (NODE) model architecture, which offer a continuous perspective on network depth. We propose and investigate a regularization method that promotes smoother transitions between layers of network parameters. Our work presents both theoretical foundations and empirical validations. We derive the Euler-Lagrange equations to characterize the optimal continuum limit of regularized NODEs and analyze the asymptotic behaviour of network parameters under strong regularization. Experimentally, we conduct a comparison between regularized NODEs and standard ResNets on image classification tasks using the MNIST and CIFAR-10 datasets. Our findings demonstrate that regularized NODEs, particularly when combined with a soft-restart training strategy that progressively decreases the regularization parameter, exhibit superior training stability, faster convergence with limited data, and increased robustness to varying network depths compared to ResNets. We demonstrate the usage of regularized NODEs on real life biomedical data and compare to existing models. This research highlights the advantages of regularized NODEs as a more stable and efficient architecture for deep learning, offering significant benefits in scenarios with limited computational resources or scarce data.