Efficient online uncertainty management : Conformal-in-the-Loop for measuring and actioning uncertainty at training time
department or school: Science, Irving K. Barber Faculty of (Okanagan)
department or school: Computer Science, Mathematics, Physics and Statistics, Department of (Okanagan)
data provider: DSpace
degree campus: UBCO
creator: Graham-Knight, John Brandon
issued: 2025-05-29T22:04:58Z
issued: 2025
related degree: Doctor of Philosophy - PhD
degree grantor: University of British Columbia
description: Focus on Neural Network research has largely been on statistical performance, as Deep Learning is an optimization problem. This often overshadows properties needed for adoption, such as robustness and fairness. Early adversarial attacks showed minor input changes could cause incorrect predictions. Fairness is obscured by the black-box nature of neural networks, which are typically evaluated by a single performance statistic. This thesis explores uncertainty quantification motivated by improving fairness and robustness. A use case on fair artificial intelligence (AI) explores the Breast Screening Program at the BC Cancer Agency with the evaluation of a commercial AI. Model performance was evaluated using the area under the curve of receiver-operating characteristic. Compared to radiologists, it underperforms on follow-up periods up to two years but overperforms for longer periods. It performs well overall on a Canadian population, but performs poorly in certain subgroups including BI-RADS densities, patients with a self-reported family history of cancer, and mammograms showing calcifications. A second use case evaluates a car detection algorithm using the KITTI dataset, where the model shows a lack of robustness for small perturbations within sensor error. These use cases motivate the thesis. Uncertainty is evaluated by comparing the performance and diversity of 144 models from a grid search of network parameters for semantic segmentation of the CityScapes dataset. Many models show no significant performance difference, and performance and diversity across models can be predicted. This diversity can be used a measure of uncertainty in model ensembles. Uncertainty quantification during training is often used for online labeling and pruning mislabeled examples. However, these methods are usually computationally expensive and require multiple stages or training runs. The final chapter applies conformal prediction for efficient online data curation in a single training r…
aggregated CHO: https://circle.library.ubc.ca/rest/handle/2429/91243
note: Efficient Online Uncertainty ManagementConformal-In-The-Loop for Measuring and ActioningUncertainty at Training TimebyJohn Brandon Graham-KnightB.A., The University of Victoria, 2005B.Eng., The University of Victoria, 2010M.Sc., The University of British Columbia, 2021A DISSERTATION SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFDOCTOR OF PHILOSOPHYinThe College of Graduate Studies(Computer Science)THE UNIVERSITY OF BRITISH COLUMBIA(Okanagan)May 2025© John Brandon Graham-Knight, 2025The following individuals certify that they have read, and recommend tothe College of Graduate Studies for acceptance, a dissertation entitled:Efficient Online Uncertainty Management:Conformal-In-The-Loop for Measuring andActioning Uncertainty at Training Timesubmitted by John Brandon Graham-Knight in partial fulfilment of therequirements of the degree of Doctor of PhilosophyDr. Patricia Lasserre, Irving K. Barber Faculty of ScienceCo-SupervisorDr. Homayoun Najjaran, University of VictoriaCo-SupervisorDr. Yves Lucet, Irving K. Barber Faculty of ScienceSupervisory Committee MemberDr. Mohamed Sami Shehata, Irving K. Barber Faculty of ScienceSupervisory Committee MemberDr. Apurva Narayan, University of Western OntarioSupervisory Committee MemberDr. Julian Cheng, Faculty of Applied ScienceUniversity ExaminerDr. Amber Simpson, Queen’s UniversityExternal ExamineriiAbstractFocus on Neural Network research has largely been on statistical perfor-mance, as Deep Learning is an optimization problem. This often overshadowsproperties needed for adoption, such as robustness and fairness. Early adver-sarial attacks showed minor input changes could cause incorrect predictions.Fairness is obscured by the black-box nature of neural networks, which aretypically evaluated by a single performance statistic.This dissertation explores uncertainty quantification motivated by im-proving fairness and robustness. A use case on fair artificial intelligence (AI)explores the Breast Screening Prog…
has type: Thesis/Dissertation
date issued: 2025-09
is shown at: 10.14288/1.0448981
language: eng
degree discipline: Computer Science
provider: Vancouver : University of British Columbia Library
publisher: University of British Columbia
rights: Attribution-NonCommercial-ShareAlike 4.0 International
rights URI: http://creativecommons.org/licenses/by-nc-sa/4.0/
scholar level: Graduate
contributor: Lasserre, P.
contributor: Najjaran, Homayoun
type: Text
identifier URI: http://hdl.handle.net/2429/91243