Learning from imbalanced data : a geometric study on over-parameterized models
department or school: Applied Science, Faculty of
department or school: Electrical and Computer Engineering, Department of
data provider: DSpace
degree campus: UBCV
creator: Behnia, Tina
issued: 2023-08-24T22:23:25Z
issued: 2023
related degree: Master of Applied Science - MASc
degree grantor: University of British Columbia
description: Machine learning has trended toward training overparameterized networks that can exactly fit the training set. In this regime, when the training data is class-imbalanced, traditional methods for mitigating imbalances are not effective and yield poor performance on minorities. Tailored to the over-parameterized setting, one successful technique to combat imbalances involves adjusting the logits during training by introducing several hyper-parameters in the cross-entropy loss. The idea behind these adjustments is rooted in the implicit bias analysis which, for linear models, explains why they successfully induce bias on the optimization path towards solutions that favor minorities. However, the impact of these adjustments is not well-understood for deep, non-linear models that simultaneously learn features from the data and fit classifiers on them. In this work, we take a step towards formalizing the impact of data imbalances and the choice of loss function when learning in over-parameterized setups. At the core of our analysis is the unconstrained features model (UFM), which recently provided partial theoretical justification for the empirical finding known as Neural-collapse. Limited to the balanced setting, Neural-collapse suggests that over-parameterized models learn features and classifiers that are arranged in a perfectly symmetric and maximally-separated geometry. Leveraging the UFM in our analysis, we analytically characterize the changes in the learned geometry as data becomes imbalanced or the loss function is modified. Our analysis characterizes the properties of the learned features and classifiers, justifying previously observed empirical findings on imbalanced training. To verify the accuracy of our theoretical predictions, we conduct experiments on benchmark networks and vision datasets. Additionally, we adopt the UFM to conduct a preliminary study on supervised contrastive loss, a recently proposed alternative to the cross-entropy. We observe that thes…
aggregated CHO: https://circle.library.ubc.ca/rest/handle/2429/85620
note: Learning from Imbalanced Data:A Geometric Study onOver-parameterized ModelsbyTina BehniaB.Sc., Sharif University of Technology, 2021A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFMASTER OF APPLIED SCIENCEinThe Faculty of Graduate and Postdoctoral Studies(Electrical and Computer Engineering)THE UNIVERSITY OF BRITISH COLUMBIA(Vancouver)August 2023© Tina Behnia, 2023The following individuals certify that they have read, and recommend to the Faculty ofGraduate and Postdoctoral Studies for acceptance, the thesis entitled:Learning from Imbalanced Data: A Geometric Study on Over-parameterizedModelssubmitted by Tina Behnia in partial fulfillment of the requirements for the degree ofMaster of Applied Science in Electrical and Computer Engineering.Examining Committee:Christos Thrampoulidis, Assistant Professor, Department of Electrical and Computer En-gineering, UBCSupervisorPurang Abolmaesumi, Professor, Department of Electrical and Computer Engineering, UBCSupervisory Committee MemberMark Schmidt, Associate Professor, Department of Computer Science, UBCSupervisory Committee MemberiiAbstractMachine learning has trended toward training overparameterized networks that can exactlyfit the training set. In this regime, when the training data is class-imbalanced, traditionalmethods for mitigating imbalances are not effective and yield poor performance on minori-ties. Tailored to the over-parameterized setting, one successful technique to combat imbal-ances involves adjusting the logits during training by introducing several hyper-parametersin the cross-entropy loss. The idea behind these adjustments is rooted in the implicitbias analysis which, for linear models, explains why they successfully induce bias on theoptimization path towards solutions that favor minorities. However, the impact of theseadjustments is not well-understood for deep, non-linear models that simultaneously learnfeatures from the data and fit classifiers on them. In this work, we…
has type: Thesis/Dissertation
date issued: 2023-11
is shown at: 10.14288/1.0435573
language: eng
degree discipline: Electrical and Computer Engineering
provider: Vancouver : University of British Columbia Library
publisher: University of British Columbia
rights: Attribution-NonCommercial-NoDerivatives 4.0 International
rights URI: http://creativecommons.org/licenses/by-nc-nd/4.0/
scholar level: Graduate
contributor: Thrampoulidis, Christos
type: Text
identifier URI: http://hdl.handle.net/2429/85620