Open Collections

Graham-Knight, John Brandon

University of British Columbia

Focus on Neural Network research has largely been on statistical performance, as Deep Learning is an optimization problem. This often overshadows properties needed for adoption, such as robustness and fairness. Early adversarial attacks showed minor input changes could cause incorrect predictions. Fairness is obscured by the black-box nature of neural networks, which are typically evaluated by a single performance statistic. This thesis explores uncertainty quantification motivated by improving fairness and robustness. A use case on fair artificial intelligence (AI) explores the Breast Screening Program at the BC Cancer Agency with the evaluation of a commercial AI. Model performance was evaluated using the area under the curve of receiver-operating characteristic. Compared to radiologists, it underperforms on follow-up periods up to two years but overperforms for longer periods. It performs well overall on a Canadian population, but performs poorly in certain subgroups including BI-RADS densities, patients with a self-reported family history of cancer, and mammograms showing calcifications. A second use case evaluates a car detection algorithm using the KITTI dataset, where the model shows a lack of robustness for small perturbations within sensor error. These use cases motivate the thesis. Uncertainty is evaluated by comparing the performance and diversity of 144 models from a grid search of network parameters for semantic segmentation of the CityScapes dataset. Many models show no significant performance difference, and performance and diversity across models can be predicted. This diversity can be used a measure of uncertainty in model ensembles. Uncertainty quantification during training is often used for online labeling and pruning mislabeled examples. However, these methods are usually computationally expensive and require multiple stages or training runs. The final chapter applies conformal prediction for efficient online data curation in a single training run. It proposes a data-centric method for dynamically weighting examples using prediction set size. Evaluated on CIFAR-10 for classification and CityScapes for segmentation, results show improved accuracy and improved Jaccard Index with small computational overhead.

Text

2025-05-29

Attribution-NonCommercial-ShareAlike 4.0 International

http://hdl.handle.net/2429/91243

Computer Science

University of British Columbia

UBCO

http://creativecommons.org/licenses/by-nc-sa/4.0/