Science, Irving K. Barber Faculty of (Okanagan)
Computer Science, Mathematics, Physics and Statistics, Department of (Okanagan)
DSpace
UBCO
Osman, Islam
Doctor of Philosophy - PhD
University of British Columbia
Current state-of-the-art foreground segmentation models have achieved great success when trained with massive labeled datasets. However, these models have limitations including 1) they can not capture the fine details of the exterior silhouette of objects. 2) The performance of these models degrades when trained with limited data. 3) The performance of these models is limited in the scope of the problems they were trained on. When those models are trained using a broader scope of problems (i.e., multiple datasets), their performance degrade due to the catastrophic forgetting. This thesis proposes novel deep-learning models that overcome the problems mentioned above. For the first problem, deep-learning models are proposed: MODY-Net, REFNet, and TransBlast. The results show that REFNet solved the problem of missing fine details and achieved state-of-the-art performance. The f-measure performance of REFNet is higher than other foreground segmentation models by approximately 4%. To overcome the second problem of learning with limited data, three novel few-shot learning techniques KTNet, ORIF-TR, and FeSh-Net, are proposed. KTNet combines few-shot and self-supervised learning in both in-domain and out-of-domain image classification. Then, ORIF-TR and FeSh-Net reformulated the few-shot to foreground segmentation. FeSh-Net uses few-shot learning to learn from an exemplar frame to segment the current frame, boosting the current state-of-the-art performance by 7.7%. To solve the last problem (catastrophic forgetting), three novel lifelong-learning techniques with server-client architecture are proposed: TBPI, KDNet, and DGT. These learning techniques allow a large model (server) to learn from continuous data without forgetting. In contrast, during inference time, a small model (client) is instantiated from the larger model (server) to segment foreground objects with high performance and low inference time. DGT performance is higher than state-of-the-art (REFNet) by 4.65% and inference time of 11.4 FPS. Finally, a single model called UFSeg is proposed to overcome all four problems. UFSeg is a combination of the best model proposed in each chapter. These models are REFNet, FeSh-Net, and DGT. UFSeg outperforms state-of-the-art foreground segmentation models by 1.34%. Also, UFSeg can be used on out-of-domain real-world examples such as frames from webcams or public IP cameras.
Continual Foreground Segmentation with Limited DatabyIslam OsmanM.Sc., Ain Shams University, 2018A THESIS SUBMITTED IN PARTIAL FULFILLMENTOF THE REQUIREMENTS FOR THE DEGREE OFDOCTOR OF PHILOSOPHYinThe College of Graduate StudiesComputer ScienceTHE UNIVERSITY OF BRITISH COLUMBIA(Okanagan)April 2023© Islam Osman, 2023The following individuals certify that they have read, and recommend to the College of Graduate Studiesfor acceptance, a thesis/dissertation entitled:Continual Foreground Segmentation with Limited Datasubmitted by Islam Osman in partial fulfillment of the requirements ofthe degree of Doctor of Philosophy .Mohamed S. ShehataSupervisorRamon LawrenceSupervisory Committee MemberKhalad HasanSupervisory Committee MemberSuliman GargoumUniversity ExaminerYang WangExternal ExamineriiAbstractCurrent state-of-the-art foreground segmentation models have achieved great success when trained withmassive labeled datasets. However, these models have limitations including 1) they can not capture thefine details of the exterior silhouette of objects. 2) The performance of these models degrades when trainedwith limited data. 3) The performance of these models is limited in the scope of the problems they weretrained on. When those models are trained using a broader scope of problems (i.e., multiple datasets), theirperformance degrade due to the catastrophic forgetting.This thesis proposes novel deep-learning models that overcome the problems mentioned above. For thefirst problem, deep-learning models are proposed: MODY-Net, REFNet, and TransBlast. The results showthat REFNet solved the problem of missing fine details and achieved state-of-the-art performance. Thef-measure performance of REFNet is higher than other foreground segmentation models by approximately4%.To overcome the second problem of learning with limited data, three novel few-shot learning techniquesKTNet, ORIF-TR, and FeSh-Net, are proposed. KTNet combines few-shot and self-supervised learningin both in-domain and out-of-domain image classification. Then, ORIF-TR and FeSh-Net reformulated thefew-shot to foreground segmentation. FeSh-Net uses few-shot learning to learn from an exemplar frame tosegment the current frame, boosting the current state-of-the-art performance by 7.7%.To solve the last problem (catastrophic forgetting), three novel lifelong-learning techniques with server-client architecture are proposed: TBPI, KDNet, and DGT. These learning techniques allow a large model(server) to learn from continuous data without forgetting. In contrast, during inference time, a small model(client) is instantiated from the larger model (server) to segment foreground objects with high performanceand low inference time. DGT performance is higher than state-of-the-art (REFNet) by 4.65% and inferenceiiitime of 11.4 FPS.Finally, a single model called UFSeg is proposed to overcome all four problems. UFSeg is a combinationof the best model proposed in each chapter. These models are REFNet, FeSh-Net, and DGT. UFSegoutperforms state-of-the-art foreground segmentation models by 1.34%. Also, UFSeg can be used onout-of-domain real-world examples such as frames from webcams or public IP cameras.ivLay SummaryIn this thesis, we presented multiple novel deep-learning models. These models solve the problems oflearning with limited data and forgetting previously learned knowledge that exists in state-of-the-art fore-ground segmentation models while achieving state-of-the-art performance. Multiple novel foreground seg-mentation models MODSiam, MODY-Net, REFNet, and TransBlast are proposed. REFNet outperformsstate-of-the-art foreground segmentation models in the detection accuracy by approximately 4%. Addition-ally, to overcome the problem of learning with limited data, a novel few-shot learning image classificationnetwork called KTNet is proposed, and two novel few-shot learning techniques for foreground segmenta-tion, namely ORIF-TR and FeSh-Net, are proposed. The proposed techniques can learn with as little asone labeled frame while achieving good performance with an average of 6% improved detection accuracy.After that, in an attempt to solve the problem of learning from continuous data sequentially without forget-ting, three novel lifelong-learning techniques are proposed, namely TBPI, KDNet, and DGT. TBPI improvedthe performance of REFNet by 2%. DGT has a performance higher than REFNet-TBPI with an average of2.65%. Finally, a single model called UFSeg is proposed that achieves our research objective. UFSeg is acombination of the best model proposed in each chapter. These models are REFNet, FeSh-Net, and DGT.UFSeg outperforms DGT by 1.34%. Also, UFSeg solves the training with limited labeled data and catas-trophic forgetting problems. UFSeg can be used on out-of-domain real-world examples such as framesfrom webcams or public IP cameras.vPrefaceThe work in this thesis is performed by Islam Osman under the supervision of Dr. Mohamed Sami Shehata.In total, the research in my Ph.D. resulted in 11 papers as follows: 5 journals and 6 conferences. Allpublished papers in this thesis are used with the permission of IEEE. The results from Chapter 3 arepublished in four papers:• Islam I. Osman, and Mohamed S. Shehata. ”MODSiam: Moving Object Detection using SiameseNetworks.” IEEE Canadian Conference on Electrical and Computer Engineering. (2020)• Islam I. Osman, and Mohamed S. Shehata., ”MODY-Net: Moving Object Detection using multi-scaleoutput ensemble Y-Network.” Canadian Journal of Electrical and Computer Engineering (CJECE).(2020)• Islam I. Osman, Agwad Eltantawy, and Mohamed S. Shehata., ”Task-Based Parameter Isolationfor Foreground Segmentation without Catastrophic Forgetting using Multi-Scale Region and EdgesFusion Network.” Image and Vision Computing: 104248. (2021)• Islam I. Osman, Mohamed Abdelpakey, and Mohamed S. Shehata., ”TransBlast: Self-SupervisedLearning Using Augmented Subspace with Transformer for Background/Foreground Separation.” InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 215-224 (2021)The results from Chapter 4 are published in three papers:• Islam I. Osman, and Mohamed S. Shehata., ”Few-Shot Learning Network for Out-of-Distribution Im-age Classification.” IEEE Transactions on Artificial Intelligence (TAI) (2021)vi• Islam I. Osman, and Mohamed S. Shehata., ”Self-supervised Learning for Foreground Segmentationwith a Few Amount of Labeled Images Using Transformers.” International Conference on AdvancedIntelligent Systems and Informatics, 14-25 (2023)• Islam I. Osman, and Mohamed S. Shehata., ”Few-Shot Learning Network for Moving Object De-tection Using Exemplar-Based Attention Map.” IEEE International Conference on Image Processing(ICIP), 1056-1060. (2022)The results from Chapter 5 are documented in three papers:• Islam I. Osman, Agwad Eltantawy, and Mohamed S. Shehata, ”Task-Based Parameter Isolation forForeground Segmentation without Catastrophic Forgetting using Multi-Scale Region and Edges Fu-sion Network.” Image and Vision Computing: 104248. (2021)• Islam I. Osman, Islam Adel, and Mohamed S. Shehata ”KD-Net: a knowledge distillation network forlifelong foreground segmentation learning” submitted to Conference on Computer and Robot Vision(CRV) (2023)• Islam I. Osman, and Mohamed S. Shehata., ”Lifelong Learning using a Dynamically Growing Tree ofSub-networks for Out-of-Domain Foreground Segmentation.” submitted to IEEE transaction on imageprocessing (TIP) (2022)Other discoveries have led to publications not included in this thesis:• Reece Walsh, Islam I. Osman, and Mohamed S. Shehata., ”Masked Autoencoders with Rapid Do-main Adjustment for Few-Shot Image Classification.” submitted to IEEE transaction on image pro-cessing (TIP) (2022)• Reece Walsh, Islam I. Osman, and Mohamed S. Shehata., ”Fully Self-Supervised Out-of-DomainFew-Shot Learning with Masked Autoencoders.” submitted to IEEE International Conference on Im-age Processing (ICIP) (2023)viiContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiiList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xivList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xixList of Abbreviations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xxiiiAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xxiv1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Thesis Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.3 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 Background and Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.1 Foreground segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.1.1 Problem definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.1.2 Foreground segmentation related work . . . . . . . . . . . . . . . . . . . . . . . . . 72.1.3 Foreground segmentation datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.1.4 The evaluation metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9viii2.2 Few-shot learning techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102.2.1 Problem definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102.2.2 Few-shot learning related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.2.3 Few-shot learning datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.3 Lifelong learning techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.3.1 Problem definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.3.2 Lifelong learning related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 Fine Detail Foreground segmentation using deep learning . . . . . . . . . . . . . . . . . . . 153.1 Moving Object Detection using multi-scale output ensemble Y-shaped Network (MODY-Net) 163.1.1 MODY-Net Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163.1.2 Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193.1.3 Experiments and Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203.1.3.1 Ablation study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233.1.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253.2 Multi-scale Region and Edges Fusion Network (REFNet) . . . . . . . . . . . . . . . . . . . 253.2.1 Multi-scale Region and Edges Fusion Network (REFNet) . . . . . . . . . . . . . . . 253.2.1.1 REFNet Input . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253.2.1.2 REFNet Encoder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263.2.1.3 REFNet multi-scale dense block . . . . . . . . . . . . . . . . . . . . . . 263.2.1.4 REFNet Decoder and Scalar block . . . . . . . . . . . . . . . . . . . . . 273.2.1.5 REFNet Multi-scale region and edges fusion . . . . . . . . . . . . . . . . 283.2.2 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293.2.2.1 Ablation study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303.2.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323.3 Self-supervised learning using augmented subspace with transformer for foreground segmen-tation (TransBlast) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333.3.1 TransBlast Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33ix3.3.2 TransBlast training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363.3.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373.3.3.1 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383.3.3.2 Ablation study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404 Few-shot learning models under limited data . . . . . . . . . . . . . . . . . . . . . . . . . . . 414.1 Few-Shot Learning Network for Image Classification . . . . . . . . . . . . . . . . . . . . . 414.1.1 Knowledge transfer network (KTNet) architecture . . . . . . . . . . . . . . . . . . 414.1.2 Training of KTNet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424.1.3 Classification of in-distribution and out-of-distribution classes . . . . . . . . . . . . 454.1.3.1 Self-supervised learning for support set initialization (S4I) . . . . . . . . . 464.1.3.2 Robust and effective support image selector technique (RESIST) . . . . . 484.1.3.3 Testing of KTNet-S4I-RESIST . . . . . . . . . . . . . . . . . . . . . . . 504.1.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 544.1.4.1 Experiment setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 544.1.4.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 554.1.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 584.2 Self-supervised learning for foreground segmentation with limited labeled data . . . . . . . 604.2.1 Model Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 614.2.2 ORIF-TR Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 624.2.2.1 Pretext phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 624.2.2.2 Downstream phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 644.2.3 Experiments and results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 644.2.3.1 Experiments pre-training . . . . . . . . . . . . . . . . . . . . . . . . . . 654.2.3.2 Standard foreground segmentation experiment . . . . . . . . . . . . . . . 654.2.3.3 Ablation study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 674.2.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69x4.3 Meta-learning for few-shot foreground segmentation . . . . . . . . . . . . . . . . . . . . . 704.3.1 FeSh-Net Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 704.3.2 FeSh-Net Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 724.3.2.1 Meta-dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 724.3.2.2 Meta-training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 734.3.2.3 FeSh-Net testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 744.3.3 Experiments and results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 754.3.3.1 Experiment 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 754.3.3.2 Experiment 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 754.3.3.3 Ablation study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 794.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 795 Lifelong Learning models without catastrophic forgetting . . . . . . . . . . . . . . . . . . . . 815.1 Lifelong learning using Task-Based Parameter Isolation (TBPI) . . . . . . . . . . . . . . . . 815.1.1 Training REFNet using TBPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 825.1.2 Testing of REFNet-TBPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 855.1.3 Characteristics of TBPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 865.1.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 875.1.4.1 Experiment 1: segmentation of foreground objects . . . . . . . . . . . . . 875.1.4.2 Experiment 2: continual foreground segmentation . . . . . . . . . . . . . 935.1.4.3 Effectiveness of TBPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . 955.1.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 955.2 Lifelong learning using knowledge distillation (KDNet) . . . . . . . . . . . . . . . . . . . . 975.2.1 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 975.2.2 Training and testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 995.2.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1015.2.3.1 Traditional foreground segmentation experiment . . . . . . . . . . . . . . 1025.2.4 Domain generalization experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . 105xi5.2.4.1 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1075.3 Lifelong learning using Dynamically Growing Tree of sub-networks (DGT) . . . . . . . . . 1075.3.1 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1085.3.2 A Novel lifelong learning procedure . . . . . . . . . . . . . . . . . . . . . . . . . . 1095.3.3 DGT Foreground segmentation procedure . . . . . . . . . . . . . . . . . . . . . . . 1135.3.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1155.3.4.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1155.3.4.2 Evaluations and results . . . . . . . . . . . . . . . . . . . . . . . . . . . 1155.3.4.3 Traditional foreground segmentation using in-domain videos . . . . . . . 1165.3.4.4 Out-of-domain foreground segmentation under few-shot learning and catas-trophic forgetting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1165.3.4.5 DGT generalization over multiple in-domain and OOD videos without re-training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1205.3.5 DGT analysis and ablation study . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1255.3.5.1 Model size analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1255.3.5.2 Runtime analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1295.3.5.3 Different configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . 1295.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1316 A Unified foreground segmentation model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1336.1 UFSeg architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1336.2 UFSeg training and testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1346.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1366.3.1 Traditional foreground segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . 1366.3.2 Lifelong learning foreground segmentation . . . . . . . . . . . . . . . . . . . . . . 1376.3.3 Generalized foreground segmentation . . . . . . . . . . . . . . . . . . . . . . . . . 1406.3.4 UFSeg Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1406.3.5 Unified foreground segmentation experiment . . . . . . . . . . . . . . . . . . . . . 141xii7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148xiiiList of Tables3.1 Details of MODY-Net architecture, showing all layers and the output size of each layer. . . . 193.2 Results of MODY-Net on CDNet dataset versus current state-of-the-art methods, the valuein Bold means the best value, ↓ means the less the value, the better, ↑ means the higher thevalue, the better. * means our previous work. . . . . . . . . . . . . . . . . . . . . . . . . . 223.3 Results of different versions of MODY-Net on CDNet dataset, the value in Bold means thebest value, ↓ means the less the value, the better, ↑ means the higher the value, the better. . . 223.4 Results of REFNet compared with state-of-the-art methods on CDNet, methods with * areour previous work. Methods that have # at the end mean that the results are reproduced. . . . 303.5 Different versions of the REFNet results on CDNet . . . . . . . . . . . . . . . . . . . . . . 313.6 Results of TransBlast compared to state-of-the-art methods on CDNet, methods with * meansour previous work. Methods that have # at the end mean that the results are reproduced. . . . 383.7 Effect of each component in TransBlast using SegTrackV2 dataset. . . . . . . . . . . . . . . 394.1 Classification accuracy results for few-shot learning using 5-ways with 1-shot and 5-shot onmini-ImageNet. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 524.2 Classification accuracy results for few-shot learning using 5-way with 1-shot and 5-shot onfewshot-CIFAR. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 534.3 Classification accuracy for 1-shot, 2-way and 3-way with presence of 3-OOD and 2-OOD.Total accuracy is calculated for all five classes (ID and OOD). The 0 in the first 2 rows meansthe methods completely miss-classified the OOD images. . . . . . . . . . . . . . . . . . . . 574.4 KTNet against clustering techniques for 0shot, 0-way, 5-OOD, and 15-testing. . . . . . . . . 594.5 KTNet against few-shot learning models for 0shot, 0-way, 5-OOD, and 15-testing. . . . . . . 60xiv4.6 Results of ORIF-TR compared to state-of-the-art models in DAVIS16. Methods that have #at the end mean that the results are reproduced. . . . . . . . . . . . . . . . . . . . . . . . . 664.7 Results of ORIF-TR compared to state-of-the-art models in SegTrackV2. Methods that have# at the end mean that the results are reproduced. . . . . . . . . . . . . . . . . . . . . . . . 664.8 Results of ORIF-TR against the selected three models. k represents the number of trainingimages in each video. The values in the table are the F-Measure. Methods that have # at theend mean that the results are reproduced. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 684.9 Different versions of the ORIF-TR results on the SegTrackV2 dataset using the standardforeground segmentation configurations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 694.10 Results of FeSh-Net against state-of-the-art foreground segmentation methods on CDNet.The last four rows are the results of FeSh-Net with extra online fine-tuning using few-labeledframes. Methods with * are our previous work. Methods that have # at the end mean that theresults are reproduced. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 764.11 Results of FeSh-Net against three foreground segmentation methods on CDNet. k representsthe number of training frames in each video. The values in the table are the F-Measure.Methods that have # at the end mean that the results are reproduced. . . . . . . . . . . . . . 774.12 Results of FeSh-Net against ORIF-TR. k represents the number of training images in eachvideo. The values in the table are the F-Measure. . . . . . . . . . . . . . . . . . . . . . . . 784.13 Results of four versions of FeSh-Net on CDNet using one training frame. . . . . . . . . . . 795.1 Characteristics of the proposed continual learning technique compared to the state-of-the-arttechniques. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 865.2 Results of REFNet and TBPI compared to state-of-the-art methods on CDNet, methods with* means our work. Methods with * are our previous work. Methods that have # at the endmean that the results are reproduced. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 885.3 Results of REFNet and TBPI compared to state-of-the-art methods on DAVIS16. (PT) meansrequire pre-training, and (OF) means apply online fine-tuning. Methods that have # at theend mean that the results are reproduced. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89xv5.4 Results of REFNet and TBPI compared to state-of-the-art methods on SegTrackV2. Methodswith * are our previous work. Methods that have # at the end mean that the results arereproduced. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 915.5 Comparison between different learning methods on CDNet, DAVIS16, and SegTrackV2 ascontinual tasks using the proposed REFNet. The down arrow ↓ means the performance de-graded. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 945.6 Comparison of TBPI against state-of-the-art methods including REFNet using CDNet dataset.Methods that have # at the end mean that the results are reproduced. . . . . . . . . . . . . . 965.7 Comparison of TBPI against state-of-the-art methods including REFNet using DAVIS16dataset. Methods that have # at the end mean that the results are reproduced. . . . . . . . . . 965.8 Comparison of TBPI against state-of-the-art methods including REFNet using SegTrackV2dataset. Methods that have # at the end mean that the results are reproduced. . . . . . . . . . 975.9 Traditional foreground segmentation results of KDNet compared with state-of-the-art modelson the DAVIS16 dataset. Methods with * are ours. Methods that have # at the end mean thatthe results are reproduced. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1035.10 Traditional foreground segmentation results of KDNet compared with state-of-the-art modelson the SegTrackV2 dataset. Methods with * are ours. Methods that have # at the end meanthat the results are reproduced. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1045.11 Comparison of KDNet against state-of-the-art methods using the three datasets CDNet, DAVIS16,SegTrackV2. The values in the table represent the F-measure. KDNet (Server) is only theserver network without clients. On the other hand, KDNet (Server) means both the serverand clients are used in training. methods with * are our previous work. Methods that have #at the end mean that the results are reproduced. . . . . . . . . . . . . . . . . . . . . . . . . 1065.12 Traditional foreground segmentation results of DGT in comparison with state-of-the-art mod-els on the DAVIS16 dataset. * means our previous work. Methods that have # at the end meanthat the results are reproduced. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117xvi5.13 Traditional foreground segmentation results of DGT compared with state-of-the-art modelson the SegTrackV2 dataset. * means our previous work. Methods that have # at the end meanthat the results are reproduced. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1185.14 Foreground segmentation results of DGT compared to state-of-the-art methods on YOVOS. *means our previous work. Methods that have # at the end mean that the results are reproduced.1215.15 Lifelong learning foreground segmentation results of DGT compared to state-of-the-art meth-ods on CDNet. Methods are categorized as fine-tuning-based for traditional methods, andlifelong learning-based for methods trained using a lifelong learning technique. * means ourprevious work. Methods that have # at the end mean that the results are reproduced. . . . . . 1225.16 Few-shot foreground segmentation results of DGT compared to state-of-the-art methods onDAVIS17. Methods are categorized as fine-tuning-based for traditional methods, and lifelonglearning-based for methods trained using a lifelong learning technique. * means our previouswork. Methods that have # at the end mean that the results are reproduced. . . . . . . . . . . 1235.17 Generalized foreground segmentation results of DGT compared to state-of-the-art methodson all three datasets. * means our previous work. Methods that have # at the end mean thatthe results are reproduced. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1245.18 Model size analysis of proposed work against other models. The number of parameters is inmillions. * means our previous work. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1285.19 Runtime analysis of proposed work against other models. The reported runtime is in seconds.* means our previous work. DGT-NetS,S means testing on seen videos (searching for a nodeby video ID), and DGT-NetS,U means testing on unseen videos (searching for a node bygreedy algorithm). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1305.20 Different configurations of DGT-NetS using DAVIS16 dataset. . . . . . . . . . . . . . . . . 1316.1 Traditional foreground segmentation results of UFSeg compared with state-of-the-art modelson the DAVIS16 dataset. * means our previous work. . . . . . . . . . . . . . . . . . . . . . 136xvii6.2 Traditional foreground segmentation results of UFSeg compared with state-of-the-art modelson the SegTrackV2 dataset. * means our previous work. Methods that have # at the end meanthat the results are reproduced. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1376.3 Foreground segmentation results of UFSeg compared to state-of-the-art methods on CDNet.* means our previous work. Methods that have # at the end mean that the results are reproduced.1386.4 Foreground segmentation results of DGT compared to state-of-the-art methods on YOVOS. *means our previous work. Methods that have # at the end mean that the results are reproduced.1386.5 Lifelong learning foreground segmentation results of UFSeg compared to state-of-the-artmethods on CDNet. * means our previous work. Methods that have # at the end meanthat the results are reproduced. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1396.6 Few-shot foreground segmentation results of UFSeg compared to state-of-the-art methods onDAVIS17. * means our previous work. Methods that have # at the end mean that the resultsare reproduced. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1396.7 Generalized foreground segmentation results of UFSeg compared to state-of-the-art methodson all three datasets. * means our previous work. Methods that have # at the end mean thatthe results are reproduced. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1406.8 Model size and runtime analysis of UFSeg against other models. The number of parametersis in millions. * means our previous work. . . . . . . . . . . . . . . . . . . . . . . . . . . . 1416.9 Unified foreground segmentation experiment results for UFSeg. . . . . . . . . . . . . . . . 142xviiiList of Figures1.1 Performance of foreground segmentation models over the past decade. . . . . . . . . . . . . 22.1 Sample input and output of foreground segmentation. . . . . . . . . . . . . . . . . . . . . . 62.2 Spliting Mini-ImageNet into training/testing sets based on classes instead of images per class.Each set is split into episodes with each episode having Strain and Stest [29]. . . . . . . . . . 102.3 Spliting MNIST dataset into a sequence of tasks [117]. . . . . . . . . . . . . . . . . . . . . 133.1 MODY-Net architecture. The two encoders have shared weights. Their output is merged byconcatenation and used as input to the decoder. The decoder predicts the moving objectsbased on the average of all decoder outputs. . . . . . . . . . . . . . . . . . . . . . . . . . . 173.2 Residual block, three convolutional layers followed by concatenation with a convolutionallayer on the skip connection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173.3 Results of MODY-Net on CDNet dataset. The first row shows the RGB frames of differentsequences, the second row is the ground truth, and the third row shows the proposed modelMODY-Net results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213.4 Top row is the results of each block of the decoder, (a) is the output of the first block in thedecoder, (b) the second block output, (c) is the third block output, and (d) is the output of thelast block. The bottom row is the result of ensembling using different numbers of layers, (e)the output of the last layer without the ensemble, (f) the ensemble of the last two outputs, (g)the ensemble of the last three blocks, and (h) is the ensemble of all outputs. Yellow circleshighlight the false detection in each output. . . . . . . . . . . . . . . . . . . . . . . . . . . 243.5 The proposed multi-scale region and edges fusion network multi-scale Region and EdgesFusion Network (REFNet) architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26xix3.6 Multi-scale dense block that captures dense multi-scale feature maps from the encoder output. 273.7 Comparison between the network trained to extract region only versus region and boundaryedges. In the output image, a white pixel means ground truth, and a red pixel means predicted. 293.8 Outputs at different feature scales. Each decoder block captures different details. The finaloutput is the output of the REF block which fuses all three decoder outputs. . . . . . . . . . 293.9 Visual comparison between REFNet’s output and other models’ output. The top row is anRGB frame and corresponding ground truth from a video sequence. The bottom row is theoutput of different foreground segmentation models. . . . . . . . . . . . . . . . . . . . . . . 313.10 TransBlast architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323.11 TransBlast architecture during training using Barlow twins. . . . . . . . . . . . . . . . . . . 353.12 Results of TransBlast against three other models in SegTrackV2 using different thresholdvalues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404.1 The proposed hybrid meta-learning network (KTNet) architecture. The green arrow repre-sents the training phase (supervised learning using the support set). The yellow arrow repre-sents the testing phase (classifying the query set). The purple arrow represents the selectionof samples from the query set and adding them to the support set (to increase the number ofshots). The process is repeated until no more samples from the query set are added to thesupport set. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444.2 Images belonging to 5 different classes are drawn using T-SNE[118]. The number of imagesper class is 15 images. (a) The images are drawn in the image space, (b) the images are drawnin the task-agnostic feature space, and (c) the images are drawn in the task-specific featurespace learned using self-supervised learning on top of the task-agnostic feature space. . . . . 464.3 ORIF-TR architecture in the pretext phase. . . . . . . . . . . . . . . . . . . . . . . . . . . . 624.4 Visual results for both experiments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 684.5 The architecture of FeSh-Net. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 724.6 Sample results from FeSh-Net against three other models using 0 and 15 training frames. . . 785.1 The process of TBPI. (a) the pre-TBPI training phase, and (b) the TBPI training phase. . . . 83xx5.2 An example of applying TBPI on REFNet of a single convolutional layer, each circle is afilter with 3×3 parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 835.3 An example of creating a task-specific REFNet of a single convolutional layer for task t(2). . 845.4 Sample results from videos in CDNet dataset, the first row is the RGB frame, the last row isthe ground truth, and the rest are the result of different methods. . . . . . . . . . . . . . . . 905.5 Sample results from videos in DAVIS16 dataset. . . . . . . . . . . . . . . . . . . . . . . . . 905.6 Sample results from videos in SegTrackV2 dataset. . . . . . . . . . . . . . . . . . . . . . . 915.7 The effect of different threshold values on the performance of REFNet and REFNet-TBPI onCDNet dataset. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 925.8 Fine-tuning versus the proposed TBPI in continual tasks. In fine-tuning, a huge degradationin F-measure due to catastrophic forgetting. On the other hand, the proposed TBPI has 0degradations in performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 955.9 The architecture of KDNet. On the left is the server network with multi-head outputs, and onthe right are the client networks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 985.10 Training procedure of KDNet. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 995.11 The average loss of clients of all three datasets after each federated learning cycle. . . . . . 1075.12 Visualization of building a base DGT given 6 videos and a decoder with 4 layers. On the leftside is coarse to fine clustering of videos. On the right side is the produced DGT. . . . . . . 1115.13 DGT in the testing phase. First, DGT finds a suitable node for the new video using a labeledframe. Then, DGT generates a task-specific network by stacking sub-networks from nodesin the path from the root node to the selected node. Finally, the generated network is used tosegment the new video frames. The color of the network’s blocks represents which node isused to initialize the parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1145.14 Visual results of generalized foreground segmentation of our DGT against other models. . . 1255.15 Visual results of generalized foreground segmentation of our DGT-NetL. . . . . . . . . . . . 126xxi5.16 Visualization of DGT-NetL as a circular tree after training using YOVOS, CDNet, and DAVIS17.The node in the middle is the tree’s root, and the nodes in the first inner circle are childrenof the root node. The node color defines at which stage the node was added. Blue nodes areadded during the initial phase using YOVOS. Red nodes are added during the lifelong learn-ing phase using CDNet. Finally, black nodes are added during the few-shot learning phaseusing DAVIS17. The visualization is made using the ETE toolkit [35]. . . . . . . . . . . . . 1275.17 The increase in the number of parameters of the model versus the number of videos duringeach depth in the ”Building base tree” phase of DGT using the YOVOS dataset. (a) showsthe increase in the number of parameters of DGT-NetS. While (b) shows the increase in thenumber of parameters of DGT-NetL. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1286.1 Architecture of UFSeg. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1346.2 sample results of UFSeg on in-domain videos. GT means the ground truth. . . . . . . . . . . 1426.3 sample results of UFSeg on out-of-domain videos. GT means the ground truth. . . . . . . . 143xxiiList of AbbreviationsCNN Convolutional Neural NetworkMOD Moving Object DetectionMAML Model Agnostic Meta LearningMTL Meta Transfer LearningResNet Residual NetworkYOVOS YOutube Video Object SegmentationECSSD Extended Complex Scene Saliency DatasetDAVIS Densely Annotated VIdeo SegmentationFPS Frames Per SecondREFNet multi-scale Region and Edges Fusion NetworkGAP Global Average PoolingORIF Object Representation and Invariant FeaturesFeSh-Net Few-shot NetworkEAM Exemplar-based Attention MapTBPI Task-Based Parameter IsolationKDNet Knowledge Distillation NetworkDGT Dynamically Growing Tree of sub-networksUFSeg Ultimate Foreground Segmentation modelxxiiiAcknowledgementsPrimarily, I wish to express my gratitude to my supervisor, Dr. Mohamed Shehata, for his unwaveringsupport, motivation, and forbearance during my academic journey. Additionally, I would like to convey mydeepest appreciation to Mohamed H. Abdelpakey. Without your assistance and direction, this work couldnot have been accomplished.xxivChapter 1IntroductionForeground segmentation identifies the objects and regions of interest from the background. This is oftenused in image processing and computer vision applications to separate the foreground from the backgroundand to allow the foreground objects to be processed or analyzed independently. Foreground segmentationis vital for several reasons: 1) It extracts only the relevant information to a specific task, making it easierto analyze and understand the content of an image [31]. 2) It can be used to enable object recognition,tracking [98, 126], and other computer vision tasks that rely on the ability to distinguish foreground frombackground. 3) It can be used in image-based search and retrieval systems to help users find imagesthat contain specific objects or scenes [67]. With the advances in deep learning, the recent foregroundsegmentation models have achieved great success in segmenting the objects of interest from images withhigh accuracy. As shown in Figure 1.1, the performance of the statistical methods ranges between 50%and 70% in f-measure, while deep-learning models have performance between 70% and 84%. However,these models have limitations, including 1) they can not capture the fine details of the exterior silhouetteof the object due to the optimization process where the model learns an approximation of the mappingfunction between the input image and output image. This approximation leads to miss-detecting the finedetails as they have a negligible effect on the loss function. However, these fine details are essential inidentifying the object. 2) They require a large number of labeled data (more than 200 labeled images pervideo [63]) to learn how to segment the foreground object. This large number of labeled data is unavailablein some domains (e.g., medical images and live camera streams) because foreground segmentation’s1Figure 1.1: Performance of foreground segmentation models over the past decade.labeling process is costly. 3) The performance of these models is limited in the scope of problems (i.e.,the models are limited to learning from a single dataset). When those models are trained using a broaderscope of problems (i.e., multiple datasets), their performance can severely degrade due to a problem calledcatastrophic forgetting.Traditionally few-shot learning is used in the context of image classification [29] to overcome the problemof learning with limited data. Few-shot learning is a machine learning paradigm in which a model is trainedto classify new objects based on a small number of examples. Few-shot learning aims to learn meta-information to generalize to new domains with very little training data.Some researchers addressed the problem of catastrophic forgetting by introducing lifelong learningtechniques. These techniques allow a deep learning model to learn continuously from multiple datasetswithout forgetting the knowledge gained from previously learned individual datasets. Lifelong learning is alearning paradigm in which a model is trained to learn continually throughout its life rather than being trainedon all the data it will encounter in advance. Lifelong learning was traditionally introduced in the context ofimage classification [60]. To the best of our knowledge, we are the first to introduce lifelong learning to thecontext of foreground segmentation.Although some of the existing problems in foreground segmentation are addressed individually in differ-2ent research efforts, such as few-shot learning and lifelong learning in image classification, no single modelcan handle all these problems together.1.1 Thesis ObjectivesThe overall goal of this thesis is to propose a novel deep-learning model that can address the challengesdescribed above. This model has the following objectives:• outperforms state-of-the-art foreground segmentation models’ accuracy (i.e., f-measure).• can learn new foreground segmentation tasks (datasets) under limited labeled data.• can learn continually throughout its lifetime without forgetting what it has learned previously.1.2 MotivationIn a real-world application, we would use a foreground segmentation model to detect the objects of interestin frames captured by a camera. Since these frames are sampled from a distribution completely differentfrom the existing foreground segmentation datasets, they are considered out-of-domain (OOD) frames. Theperformance of existing foreground segmentation models degrades massively when used to segment out-of-domain frames due to their poor generalization. To improve the performance of these models, they mustbe fine-tuned using approximately 200 labeled frames from each camera. However, the frames capturedby a camera are not labeled, and it is so hard to manually label 200 frames from each camera. This iswhy these models need to be able to learn using limited labeled data (as few as 1 or 5 labeled frames).Also, suppose a new camera is added to the system. In that case, there are two choices to make thesemodels segment the frames from a new camera 1) fine-tune the models using labeled frames from thenew camera, but in this case, the models will forget what they learned from the previous cameras (i.e.,catastrophic forgetting), or 2) re-train the models using labeled frames from all cameras (both the previouscameras and the new cameras), this requires a long time to entirely re-train the models for each new cameraadded, and needs to store the labeled frames from all previous cameras. To overcome this problem, the3models need to utilize lifelong learning to allow them to learn continuously from new data without re-trainingor forgetting the previously learned ones.1.3 Thesis OutlineThe rest of the thesis has six chapters. In Chapter 2, the background and literature review of foregroundsegmentation modes, few-shot learning, and lifelong learning are presented. After that, four technicalchapters present our proposed work. The first technical chapter is fine detail foreground segmentationusing deep learning. In this chapter, we proposed multiple novel deep-learning models that overcomethe problem of miss-detecting the fine detail of the objects’ exterior silhouette and achieving state-of-the-art performance. The multiple deep-learning models proposed are MODY-Net, REFNet, and TransBlast.The results show that REFNet solved the problem of missing fine details and achieved state-of-the-artperformance. REFNet uses a loss function that learns to detect both the region and edges of the objectand combines them to produce high-definition output. Hence, the f-measure performance of REFNet ishigher than that of other foreground segmentation models by approximately 4%. In the second technicalchapter, which is named few-shot learning models under limited data, the goal is to overcome the problem oflearning with limited data. This chapter proposes three novel few-shot learning techniques, namely KTNet,ORIF-TR, and FeSh-Net. First KTNet combines few-shot and self-supervised learning in both in-domainand out-of-domain image classification. Then, ORIF-TR and FeSh-Net reformulated and customized thefew-shot and self-supervised learning (used in KTNet) to foreground segmentation. ORIF-TR uses self-supervised learning to leverage unlabeled data to learn useful information that can be transferred to newdomains using a few examples. On the other hand, FeSh-Net does not use self-supervised but rather usesfew-shot learning to learn an exemplar to segment the current frame. Hence, given any frame from thesame video, the model will accurately segment the foreground without needing more labeled data. whileboosting the current state-of-the-art performance from 70.3% to 78.0%.The third technical chapter is aboutlifelong learning models without catastrophic forgetting. This chapter proposes three novel lifelong-learningtechniques with server-client architecture To solve the problem of catastrophic forgetting. The proposedmodels are TBPI, KDNet, and DGT. These learning techniques allow a large model (server) to learn from4continuous data without forgetting. In contrast, during inference time, a small model (client) is instantiatedfrom the larger model (server) to segment foreground objects with high performance and low inferencetime. DGT performance is higher than state-of-the-art (REFNet) by 4.65% and inference time of 11.4 FPS.Finally, the fourth and last technical chapter that is called a unified foreground segmentation model.In this chapter, a single model called UFSeg is proposed to overcome all four problems. UFSeg is acombination of the best model proposed in each chapter. These models are REFNet, FeSh-Net, andDGT. UFSeg outperforms state-of-the-art foreground segmentation models by 1.34%. Also, UFSeg canbe used on out-of-domain real-world examples such as frames from webcams or public IP cameras. Afterthe technical chapters, there is one more chapter which is the conclusion. This chapter concludes thepresented research in the thesis and highlights future directions.5Chapter 2Background and Related Work2.1 Foreground segmentation2.1.1 Problem definitionForeground segmentation is a technique used in computer vision to separate objects from the backgroundin a given sequence of frames. The output of foreground segmentation is a binary image. Each pixel inthis image has a value of 0 or 1, such that 0 means background pixel, and 1 means foreground pixel (i.e.,pixel belongs to a foreground object). An example of foreground segmentation input and output is shown inFigure 2.1. The most common deep learning architecture for foreground segmentation is an auto-encoderbased on Convolutional Neural Network (CNN).Figure 2.1: Sample input and output of foreground segmentation.62.1.2 Foreground segmentation related workThe foreground segmentation problem witnessed many improvements over the past decades. The existingmethods in this domain are categorized into statistical and Convolutional Neural Network (CNN) methods.Statistical foreground segmentation methods target to create a model representing the background usingthe background’s intensity and motion vector (optical flow [9]). Then, the foreground is detected as pixelsthat have different intensities and motion vectors from the created model. For example, Ego-motion com-pensation with background modelling [2], Feature-based [95], Motion-based [6], Image-cues based [102],subspace-based [27, 26]. However, these methods suffer from high computational loads and a degradedperformance with camera motion and challenging environments [77, 92]. On the other side, CNN proposesan accurate segmentation of foreground [90] in challenging environments (such as [122]). One of the ear-liest CNN-based methods is ConvNets [13]. It trains LeNet-5 [49] on CDnet2014 to perform a subtractionbetween the input image and a static background image. This idea is extended in [7] by introducing pre-processing and post-processing steps for the CNN input and output, respectively. The pre-processing steptargets generating a background model adaptable to dynamic environment changes. The post-processingstep, i.e., spatial-median filtering [106], is to segment camouflaged foreground accurately. Although thesemethods show great potential for using CNN to solve foreground segmentation problems, they do not offeran end-to-end solution, i.e., they still depend on engineered, crafted techniques to detect foregrounds. Cas-caded CNN [123] presents an end-to-end CNN to distinguish between background and foreground objectsbased on appearance differences. A multi-resolution CNN network with cascaded architecture is proposed.The multi-resolution network detects different sizes of foregrounds, while the cascaded architecture en-sures spatial coherence between foreground pixels. In [41], Johnander et al. presents a CNN that utilizes aone-shot learning technique to differentiate between foreground and background. To this end, a probabilis-tic generative model is introduced to learn foreground and background feature distributions. Zhang et al.[134] propose a CNN model based on the encoder-decoder structure network to detect foregrounds. Theencoder part locates foreground regions of interest even if a foreground occlusion occurs. The Decoderpart employs the output from CNN-based optical flow [36] to refine the generated mask of the foreground.In [78], Oh et al. extract the foreground using spatial and temporal information. Therefore, the encoderconsists of two sub-networks that extract features from previous and current frames. Multi sub-network7encoders are presented in [88] to detect multi-scale features from consecutive frames and feature mapsof optical flow. Moreover, the edge extraction mechanism module is integrated into the encoder part to al-low sharing of features between the sub-network encoders, hence, detecting the most important foregroundfeatures. In [62], FgSegNet uses three sub-network encoders to embed an image in multiple scales into thefeature space. The underlying structure of each encoder sub-network is a simplified version of VGG-16 Net[136]. In FgSegNetV2 [63], Lim et al. extend FgSegNet to handle the camera motion. FgSegNetV2 intro-duces a feature pooling module to allow fusion features of the extracted multi-scale features within images.Li et al. in [55] introduce a multi-task motion-guided video salient object detection network that consistsof two sub-networks of encoders and two sub-networks of decoders. Regarding the two sub-networks ofencoders, one is trained to detect the salient objects in still images, while the other detects motion saliencyfrom optical flow images. These two sub-networks are connected via motion-guided attention modulesto enhance appearance features with motion features. The two sub-networks of decoders generate twomasks for appearance and motion saliency. The generated masks are fused to generate a final mask usingthe motion-guided attention module.2.1.3 Foreground segmentation datasetsThe benchmark datasets in foreground segmentation are:ChangeDetection.Net (CDNet) 2014 [122] is a benchmark dataset that detects changes between videoframes. These changes are the moving objects. The dataset consists of 11 different challenges. Eachchallenge has a number of videos ranging from 4 to 6, and the number of frames in each video ranges from1000 to 8000. The total number of videos in the dataset is 53 videos.Densely Annotated VIdeo Segmentation (DAVIS) [91] is a video object segmentation dataset. DAVISconsists of 50 videos. Each video has several frames ranging from 50 to 104. In each video, a single objectis annotated, which is the object of interest in this video.SegTrackV2 [53] is a video multiple objects segmentation dataset. The number of videos in the datasetis 14, and the number of frames ranged from 21 to 279 in each video. In this dataset, each frame hasmultiple ground-truth corresponding to each object in the video.YOutube Video Object Segmentation (YOVOS) [128] is a large-scale dataset collected from YouTube8video clips. The dataset has 94 different object categories and annotations for more than 190,000 objects.To use this dataset, we customized it to fit in the foreground segmentation context. The customization isdone by assigning one label for all foreground objects instead of different labels for different objects. Inother words, we care only about assigning a label for the foreground objects vs. the background in a frame,and we do not care about the class of an instance, whether it is, for example, a cat or a dog. Then, thedataset is split frame-wise into training and testing sets.2.1.4 The evaluation metricsThe evaluation metrics used in foreground segmentation:Pr =TPTP+FP(2.1)Re=TPTP+FN(2.2)FPR=FPFP+TN, (2.3)FNR=FNTP+FN, (2.4)FM =2×Pr×RePr+Re, (2.5)TP,TN,FP,andFN are the true positive, true negative, false positive, and false negative, respectively.Pr,Re are Precision and Recall, FM is the F1-Measure, FPR,FNR are false-positive rate, and false-negative rate. Precision is the percentage of pixels that are correctly classified as moving objects. Therecall is the percentage of pixels that belong to the object and are correctly predicted. The false-positive rateis the percentage of pixels predicted as foregrounds while it belongs to the background. The false-negativerate is the percentage of pixels predicted as background while it belongs to the foreground. F1-measure isthe most commonly used metric in foreground segmentation, defined as the harmonic mean of precisionand recall. Another important metric is the number of Frames Per Second (FPS).9Figure 2.2: Spliting Mini-ImageNet into training/testing sets based on classes instead of images perclass. Each set is split into episodes with each episode having Strain and Stest [29].2.2 Few-shot learning techniques2.2.1 Problem definitionFew-shot learning consists of two main phases: meta-training and meta-testing. During the meta-trainingphase, the model is trained to learn how to adapt to new tasks using a large number of tasks. A task Tis a classification task sampled from the distribution p(T ). T is called an episode. Each episode includesT (tr) the training set (a.k.a support set) and T (te) the testing set (a.k.a query set). During the meta-testingphase, an unseen task Tunseen (i.e., a task with classes that are not seen during training) is used to evaluatethe model such that the model is fine-tuned using T (tr)unseen, then classifies the T(te)unseen. An example of splittinga dataset into episodes is shown in Figure 2.2. Both T and Tunseen are sampled from the same dataset inK-shots and N-ways, where K is the number of training images per class, and N is the number of classesin the task. The most used settings in few-shot learning are the 1-shot 5-ways and 5-shots 5-ways.102.2.2 Few-shot learning related workFew-shot learning literature can be divided into five categories: 1) Data augmentation, 2) metric learning, 3)memory-based networks, 4) optimization-based methods, and 5) self-supervised learning-based methods.The main idea in Data augmentation is to increase the support set by applying different transformationssuch as random cropping, channel rotation, translation, rotation, blurring, additive noise, color solarization,mixup (combining two images), and randomly masking out part of the image. Another data augmentationapproach is using generative models to generate new images (e.g., [133, 20]). The newly generated datais still related to the original data (i.e., the diversity of the support set is not large).Metric learning is the process of learning embedding function that separates different classes in theembedding space and then uses a distance metric to measure the similarity between images. Classificationin metric learning is mainly based on the nearest neighbor (i.e., the class with the highest similarity to a givenimage is assigned to it). Metric learning is widely used in few-shot learning (e.g., [110, 58, 61, 125, 56]).Memory-based networks have extra memory to store the experience gained (i.e., contextual informa-tion) from each task during the learning process. The accumulated experience can allow the networkto work on unseen tasks. Examples of memory-based networks are task-dependent adaptive metric(TADAM)([79]) and simple neural attentive meta-learner (SNAIL)([75]).Optimization-based methods usually have a model with a task-agnostic classifier that can be adaptedto a new task. The idea is to learn parameters that can easily be adapted to another task using a fewiterations. This method is widely used in ([29, 76, 97, 4, 51, 109]). An early attempt in optimization-basedmethods is Model Agnostic Meta Learning (MAML) [29]. MAML learns task-agnostic parameters that canbe used to initialize any new task. An example of optimization-based is Meta Transfer Learning (MTL)([109]), introducing scale and shifting (SS) parameters that facilitate adapting the base classifier parameterstowards another task.Self-supervised learning-based methods leverage large unlabeled data to learn useful information thatcan be transferred to new tasks using a few examples. Self-supervised learning is a paradigm in which amodel learns to perform a task without explicitly being provided with labeled training examples. Instead, themodel is given a large amount of unlabeled data and learns to perform the task using the structure in thatdata. Some research has been done on self-supervised learning techniques to improve few-shot learning11performance. One approach is to pre-train a model on a large dataset using self-supervised learning andthen fine-tune it on the few-shot task using the labeled examples [18]. This can help the model learnuseful representations that generalize well to the few-shot task and can improve performance compared totraining from scratch on the few-shot task. Another approach is to use self-supervised learning to generatesynthetic examples for the few-shot task. These synthetic examples can augment the few labeled examples[59], enabling the model to learn from more data and improve performance.2.2.3 Few-shot learning datasetsCaltech-UCSD Birds 200 (CUB) ([119]) is an image dataset consisting of 200 different bird species, withoverall 6,033 colored images with about 30 per class. Each image has semantic attributes. In this work,we do not use annotations; we use images only. One hundred classes are grouped for training, 50 forvalidation, and 50 for testing. There is no overlap between training, validation, and testing sets.Mini-ImageNet is a subset from ImageNet ([96]). The dataset consists of 100 classes, with overall60,000 colored images with size 84 × 84, 600 per class. The dataset is divided into three uncorrelatedsets training, validation, and testing sets with 64, 16, and 20 classes, respectively. Tasks are sampled fromthese sets. Each task is K-shot and N-ways.Fewshot-Cifar100 is constructed from Cifar100 ([45]). The dataset consists of 100 classes, with 60,000colored images with low resolution 32× 32 600 per class. One hundred classes are grouped into 20superclasses. The superclasses divide the dataset to prevent overlapping between training, validation,and testing sets with 12, 4, and 4 superclasses, respectively. Tasks are sampled the same way as in themini-Imagenet.2.3 Lifelong learning techniques2.3.1 Problem definitionIn the context of this thesis, we define lifelong learning (a.k.a continual learning and incremental learning)as training a model using a sequence of N tasks {T0, T1,..., TN−1} as shown in Figure 2.3. Each task Tis a classification problem with a different set of classes. The training objective is to minimize the risk of12Figure 2.3: Spliting MNIST dataset into a sequence of tasks [117].all tasks. However, the model has access to one task at a time. To account for this, the training objectiveshould have an empirical risk on the current task Ti to prevent forgetting the knowledge gained by previoustasks {T0,..., Ti−1}, which is a problem known as catastrophic forgetting. The term task is defined as agroup of videos sampled from a dataset.2.3.2 Lifelong learning related workThe literature on lifelong learning can be divided into three categories: 1) replay-based, 2) regularization-based, and 3) parameter isolation.Replay-based approaches store examples of the previous tasks to be used during the model’s trainingon a new task. The stored examples are a reminder of prior tasks to the model [93]. In [103], they used twomodels, one for doing the task and the other one to generate examples from previous tasks. So, instead ofstoring examples of earlier tasks, they used a generator model to generate examples from previous tasks.Regularization-based is mainly based on adding a regularization term to the loss function. The purposeof this regularization term is to constrain updating parameters when training on a new task. In [44], a cloneof the model F is created when training on a new task. After training on the new task, the squared differencebetween the parameters of the model F and the cloned model F ′ is a regularization term in the loss function.This regularization term will force model parameters to be updated to minimize the difference between theparameters learned by a new task and those learned by previous tasks. Another work introduced in [3]computed the sensitivity of each parameter to changes in a model (i.e., if a parameter changed, how it willaffect the network output). The parameters’ sensitivity is used as a regularization term to allow the modelonly to update parameters with low sensitivity.Parameter isolation isolates parameters updated by task ti from parameters updated by task t j, where13i 6= j. [70], added a trainable mask layer mi for each task ti to the network. The parameters of the masklayer have values of either 0 or 1, one means the corresponding parameter in the network will contributeto the current task, and 0 means the parameter will not contribute. During training, the parameters of themask layer mi will be updated to decide which of the network parameters will contribute to task ti. Anotherwork introduced in [94] used a dynamic network (i.e., the size of the network increases as the number oftasks increases). A clone of the network is generated for each task, and all clones are concatenated into asingle network (e.g., if a network initially has 10 layers, for each task, another 10 layers will be generated,the generated layers for task ti will be updated by task ti only. Then, all clones are concatenated in a singlenetwork. If the number of tasks is 4, the network size will be 40 layers).14Chapter 3Fine Detail Foreground segmentation usingdeep learningMoving object detection is a specific foreground segmentation that is concerned with the segmentation offoreground moving objects. In this chapter, we use foreground segmentation and moving object detectioninterchangeably to refer to the same task. We present three deep-learning networks for foreground seg-mentation. The first deep learning model is called MODY-Net. This model uses an ensemble of multipleoutputs to reduce the uncertainty of the final output. Hence, reduces the false positive and boosts the per-formance. The second model is a novel deep-learning architecture called REFNet. This network solves theproblems of 1) detecting tiny objects, and 2) detecting the object fine-details in MODY-Net, and achievingstate-of-the-art foreground segmentation performance. REFNet outperforms state-of-the-art models witha noticeable increase in performance, up to 4%. The third model uses vision transformers and is calledTransBlast. However, the results show that REFNet still has the highest performance, more details on therationale behind this are provided in the discussion section of this chapterThis chapter is organized as follows: Section 3.1 presents MODY-Net a model that uses an ensembleto reduce the model uncertainty. After that, Section 3.2.1 shows the novel deep-learning architecture calledREFNet that outperforms state-of-the-art foreground segmentation models. Finally, Section 3.3 shows atransformer-based foreground segmentation model.153.1 Moving Object Detection using multi-scale output ensemble Y-shapedNetwork (MODY-Net)MODY-Net proposed in [86] is an end-to-end deep learning network for foreground segmentation. MODY-Net is a Y-shaped network, which is a network that can learn useful features from both background andcurrent frames to detect the moving object in a fixed scene (i.e., the camera is not moving). The learnedfeatures are invariant to challenges like illumination change, dynamic background, low resolution, and shad-ows. Then the features from both frames are used to extract objects that exist in one frame and do not existin the other. The network extracts moving objects on multiple scales to enhance detection. The final outputis the average of network outputs at different scales to reduce the uncertainty of the network that causes itto detect false objects.3.1.1 MODY-Net ArchitectureAs shown in Figure 3.1, the network consists of two main parts: two branch encoders and a decoder. Theencoder is used to learn features that are invariant to different Moving Object Detection (MOD) challengesfrom a given input. The encoder has two branches with the same weights to extract features from thebackground and the current frame. There are two separate branches because each video’s backgroundframe is processed only once. This is faster than combining and processing both frames for each videoframe. It is important to mention that the background is a single frame for each video (i.e., the backgroundframe does not change with time, even if there are some dynamic objects in the background, like trees orwater). The architecture of the encoder is a Residual Network (ResNet) [32] with four residual blocks. Eachresidual block (ResBlock) has three convolutional layers with skip connections between the first and lastlayers, as shown in Figure 3.2. Skip connection was introduced in [32] to solve the vanishing gradients indeep networks. In the residual block, the output has dimensions equal to half of the input dimension, andthis is done by using a max-pooling layer at the end of each residual block. The input of each branch is animage with size 256× 256 with 3 channels for red, green, and blue (RGB). The output feature map fromeach branch has a size of 16× 16× 512. The number of filters, activation functions, batch normalization,and output feature map size of all layers is shown in Table 3.1.16Figure 3.1: MODY-Net architecture. The two encoders have shared weights. Their output is mergedby concatenation and used as input to the decoder. The decoder predicts the moving objects basedon the average of all decoder outputs.Figure 3.2: Residual block, three convolutional layers followed by concatenation with a convolutionallayer on the skip connection.17The output feature map of both branches is concatenated to form a single feature map. This featuremap holds the features extracted from the background and current frames. The decoder uses the combinedfeature maps to perform two important operations, 1) detect objects that exist in the feature map of a framethat does not exist in the feature map of the other frame, 2) reverse the embedding operation done bythe encoder to transform the detected objects from feature space to pixel space. The architecture of thedecoder is the same as the encoder but in reverse order (i.e., the first block in the encoder is the last layer inthe decoder). The decoder has one extra convolutional layer added to each ResBlock to output N×N×1feature map representing the moving objects detected by this block. The output of all ResBlocks in thedecoder are re-scaled to 256× 256× 1 and are combined using the average operation to form the finaloutput. The following equations represent the whole process of the network:Encoder(z,x) = ϕ(z)⊕ϕ(x) = v (3.1)Decoder(v) = AVG(B1(v),B2(v),B3(v),B4(v)) (3.2)where z is the background frame, and x is the current frame, ϕ is the encoder function of each branch, ⊕ isthe concatenation of both branches, v is the combined output. The decoder decodes v and produces fouroutputs B1,B2,B3 and B4, where Bi is the output of ResBlocki of the decoder. Finally, the four outputs arecombined by average operation AVG to form one output.18Table 3.1: Details of MODY-Net architecture, showing all layers and the output size of each layer.EncoderBranch 1(Background frame)Branch 2(Current frame)Output size Layer Output size Layer256x256x3 Input 256x256x3 Input128x128x64 ResBlock1 128x128x64 ResBlock164x64x128 ResBlock2 64x64x128 ResBlock232x32x256 ResBlock3 32x32x256 ResBlock316x16x512 ResBlock4 16x16x512 ResBlock4Concatenate both branches (output size: 16x16x1024)DecoderOutput size Layer Side layer Side output size32x32x256 ResBlock1 Conv1 256x256x164x64x128 ResBlock2 Conv2 256x256x1128x128x64 ResBlock3 Conv3 256x256x1256x256x32 ResBlock4 Conv4 256x256x1Final outputAVG(Conv1/2/3/4)256x256x13.1.2 Implementation DetailsThe inputs are background and current frames with dimensions 256× 256× 3. The encoder architectureis four residual blocks. Each block has four convolutional layers with the same number of filters followedby max-pooling to reduce the input size to half, and the size of filters is fixed to 3× 3 in all layers of thenetwork. The number of filters in each convolutional layer in the first block is 64. The output of the firstlayer is 128×128×64. The next layer has a number of filters equal to the 2× filters in the previous block.19Therefore, the second block output is 64× 64× 128. The third block output is 32× 32× 256. The fourthblock output is 16× 16× 512. Then the output of the fourth block in both branches is merged into onefeature map with dimensions 16× 16× 1024. The decoder has the same architecture as the encoder butis in reverse order. The only difference is the convolutional layer added to each block to produce outputat the scale of the current block. This convolutional layer is a 1× 1 with a number of filters equal to thenumber of channels in the current block to produce N×N× 1 output at each block, where N is the sizeof the current block (e.g., the first block in decoder has size 16×16, this means N = 16 at this block) andthen it is re-scaled to 256× 256. Finally, all outputs are combined with the average operation to form thenetwork output. For the hyperparameters, the learning rate is 10−5 and uses a mini-batch of size 8. Theloss function is a weighted sum of the loss of all network outputs:`(y,B4,B3,B2,B1) = L4(y,B4)+αL3(y,B3)+βL2(y,B2)+ γL1(y,B1),(3.3)where Bi is the output of block i of the decoder. α , β and γ are weights. Outputs at early blocks have verylow resolution. This is why weight is used to reduce the effect of this output on the final network output.Only the output of the final layer is not weighted because it has the highest resolution, which is the mosteffective part of the final output. Each Li is the average loss of all pixels in the output of Bi:Li(y,Bi) =1|D| ∑u∈Dl(y[u],Bi[u]), (3.4)where D is the dimensions of the output and y[u] ∈ {1,−1} in each pixel of the ground truth, such that 1means foreground pixel and −1 means background pixel. Finally, l is a logistic regression loss function.3.1.3 Experiments and ResultsSample output of MODY-Net from CDNet videos is shown in Figure 3.3. As seen, the MODY-Net resultis clear and so close to the ground truth. Snow falls in the two frames, yet MODY-Net ignored the snowand detected only the actual moving objects. The same goes for the figure’s second and third two frames,where the network ignored the fountain’s shadows and the moving water.Table 3.2 compares the proposed model and current state-of-the-art methods in moving object detec-tion. It is shown that MODY-Net, the proposed model outperforms all other methods concerning almost20Figure 3.3: Results of MODY-Net on CDNet dataset. The first row shows the RGB frames of differ-ent sequences, the second row is the ground truth, and the third row shows the proposed modelMODY-Net results.all evaluation metrics. MODY-Net outperforms all other methods by a huge gap in recall (Re) and false-negative rate (FNR) evaluation metrics, as shown in the table, the highest recall after the proposed modelis 0.82 from BSUV-net [113], which is less than the proposed model by 0.08. Also, in FNR, the lowest valueafter the proposed model is 0.17, worse than the proposed model by 0.05. For the F1-measure, it is shownthat the proposed model has the highest values. Computing the average of multiple outputs from differentfeature maps significantly improved recall and FNR. This is because when averaging all outputs, only pix-els with high confidence (i.e., intensity level) exist in the final output. In contrast, other pixels detected asmoving objects falsely due to uncertainty are removed in the final output, as these pixels are not detectedin all outputs. The precision (Pr) decreased slightly due to adding the output of low resolution from the firstblocks in the decoder. This caused some pixels from the foreground objects to not be detected.21Table 3.2: Results of MODY-Net on CDNet dataset versus current state-of-the-art methods, the valuein Bold means the best value, ↓ means the less the value, the better, ↑ means the higher the value,the better. * means our previous work.Method Pr ↑ Re ↑ F1 ↑ FPR ↓ FNR ↓IUTIS-5 [11] 0.80 0.78 0.77 0.005 0.21SemanticBGS [12] 0.83 0.78 0.78 0.003 0.21BSUV-Net [113] 0.81 0.82 0.78 0.005 0.17DeepBS [7] 0.83 0.75 0.74 0.009 0.24WisenetMD [52] 0.76 0.81 0.75 0.009 0.18PAWCS [107] 0.78 0.77 0.74 0.005 0.22WeSamBE [39] 0.76 0.79 0.74 0.007 0.20SuBSENSE [106] 0.75 0.81 0.74 0.009 0.18FTSG [121] 0.76 0.76 0.72 0.007 0.23MODSiam* [85] 0.69 0.82 0.71 0.010 0.18MODY-Net(Proposed Model)0.82 0.9 0.86 0.005 0.12Table 3.3: Results of different versions of MODY-Net on CDNet dataset, the value in Bold means thebest value, ↓ means the less the value, the better, ↑ means the higher the value, the better.Method Pr ↑ Re ↑ F1 ↑ FPR ↓ FNR ↓MODY-Net +1L 0.83 0.82 0.82 0.005 0.18MODY-Net +2L 0.84 0.85 0.84 0.005 0.13MODY-Net +3L 0.83 0.87 0.85 0.005 0.12MODY-Net+4L(proposed)0.82 0.9 0.86 0.005 0.12223.1.3.1 Ablation studyAn experiment is conducted to show the importance of learning with multiple losses and the multi-scaleoutput ensemble. Learning with multiple losses of different blocks in the decoder forces the network tolearn features that are useful for detecting moving objects in different scales instead of relying on the lastlayer to do all the detection. Table 4.13 shows different network versions with a different number of lossfunctions. Where MODY-Net+1L is the proposed model with loss function on the last block of the decoderonly, and the network output is the final block output (i.e., no averaging). MODY-Net+2L is the model witha loss function on the last two blocks of the decoder, and the network output is the average of the last twodecoder outputs. The same with MODY-Net+3L, loss function on the last three blocks, and the output is theaverage of the last three blocks of the decoder. Finally, MODY-Net+4L is the full proposed model, wherethe loss function is computed for all blocks of the decoder, the total loss is the weighted sum of the lossfunction at each block, and the final output is the average of all outputs of the decoder blocks.In Table 3.3, increasing the number of the loss function and averaging multi-scale output is effective inimproving recall, F1-measure, and false-negative rate. But the precision degraded slightly. This is becauseaveraging outputs from the early blocks of the decoder causes the final result to be slightly pixelized. Thisis due to the low output resolution in the decoder’s early blocks.23Figure 3.4: Top row is the results of each block of the decoder, (a) is the output of the first block inthe decoder, (b) the second block output, (c) is the third block output, and (d) is the output ofthe last block. The bottom row is the result of ensembling using different numbers of layers, (e)the output of the last layer without the ensemble, (f) the ensemble of the last two outputs, (g) theensemble of the last three blocks, and (h) is the ensemble of all outputs. Yellow circles highlightthe false detection in each output.Figure 3.4 presents the reason for improving the moving object detection by reducing false detection.As shown in the figure, each decoder block outputs the pixels of the moving objects along with different falsedetected pixels. The falsely detected pixels are due to the uncertainty of the network for pixels that have aslight change in intensity, as the presence of shadow and illumination change. To reduce the uncertainty ofthe network ensemble of different outputs is used. Ensemble multiple outputs, remove the detected pixelswith low certainty and keep only those with high certainty in all outputs. On the other hand, the true positivepixels are reduced slightly due to combining outputs with different resolutions. The result of the first blockin the decoder has a very low resolution. In contrast, the output of the last layer is very smooth (i.e., highresolution). Combining these two outputs causes some pixels detected in high resolution to vanish fromthe final output. The second row of the figure shows that as the number of layers ensembled increases, theresult has less false detection and lower resolution.243.1.4 DiscussionMODY-Net detects moving objects using multi-scale outputs and combines them to produce an output withless uncertainty. MODY-Net is evaluated on a benchmark dataset CDNet, and the results are compared tocurrent state-of-the-art methods in moving object detection. MODY-Net outperforms all other methods withrespect to almost all evaluation metrics. MODY-Net has the best recall, F1-measure, FPR, and FNR valuescompared to all other methods.Although the F-measure and the false-negative rate have achieved state-of-the-art performance, thefine details of the exterior silhouette of the object are missing in the output of MODY-Net and other state-of-the-art models (i.e., the fine details of the boundary of objects are not detected). During the optimizationprocess of the model, they learn an approximation of the mapping function between the input image andoutput image. This approximation leads to miss-detecting the fine details as they have a negligible effecton the loss function. Another limitation of MODY-Net is that small objects are not detected accurately dueto the down-sampling process that occurs in the encoder.3.2 Multi-scale Region and Edges Fusion Network (REFNet)3.2.1 Multi-scale Region and Edges Fusion Network (REFNet)REFNet proposed in [81] is an end-to-end novel multi-scale region and edges fusion network that capturesthe moving objects’ boundary details. The architecture of REFNet is shown in Figure 3.5. REFNet is basedon the encoder-decoder architecture with the following novelties, 1) A multi-scale dense block is placedbetween the encoder and decoder to calculate a multi-scale feature map from the output of the encoder, 2)An output is produced from each block in the decoder to capture the object’s details at different scales, and3) a multi-scale region and edge fusion block is used at the end to fuse the different scales output from thedecoder and produce a high-detailed moving object.3.2.1.1 REFNet InputAt the network’s input, an RGB frame from a video is concatenated depth-wise with the RGB backgroundframe. The background frame is computed as the median of the last 20 frames before the current frame.25Figure 3.5: The proposed multi-scale region and edges fusion network REFNet architecture.Each frame has fixed dimensions 240× 320× 3. After concatenation, the input layer of the network hasdimensions 240×320×6.3.2.1.2 REFNet EncoderThe architecture of the encoder is four residual blocks (ResBlocks) and four transition blocks. Each residualblock has three convolutional layers. The output of the first convolutional layer is added to the output of thethird layer (skip connection [33]). REFNet uses skip connections to overcome the problem of vanishinggradients. The transition block that follows each ResBlock is a convolutional layer followed by max-poolingto reduce the feature map’s dimensions to half its size.3.2.1.3 REFNet multi-scale dense blockThe encoder’s output is inputted to the multi-scale dense block to compute a dense and rich multi-scalefeature map. As seen in Figure 3.6, the input is passed to two branches. The first is a 2× 2 max-poolinglayer with stride 1× 1 followed by a 1× 1 convolutional layer. The second branch is a convolutional layerwith 64 filters of size 3× 3. This number of filters is the same for all convolutional layers in this dense26Figure 3.6: Multi-scale dense block that captures dense multi-scale feature maps from the encoderoutput.block. Both branches’ output is concatenated with the next layer’s input, a convolutional layer with a filtersize 3× 3 and a dilation rate equal to 4. The dilation rate allows the convolutional layer to have a largerreceptive field with the same number of parameters. After this layer, each convolutional layer’s input andoutput are concatenated and used as input for the next layer. The final output of this dense block is adepth-wise concatenation of all the previous five parts, followed by the instance normalization layer andspatial dropout layer. The instance normalization layer is introduced in [116], which performs better thanthe traditional batch normalization layer. The spatial dropout is used to improve the network generalization.3.2.1.4 REFNet Decoder and Scalar blockThe dense feature map computed from the multi-scale dense block is used as an input to the decoder,consisting of three blocks. The first decoder block is a convolutional layer followed by instance normalizationand scaled by a coefficient vector. This coefficient vector is computed as the Global Average Pooling (GAP)of the convolutional layer’s output in the second transition block of the encoder. Considering the output ofthe convolutional layer in the second transition is ft,2, the coefficient vector v2 is the global average poolingof ft,2. Let the output of the convolutional layer in the first decoder block be fd,1. The scaled version ofit is f ′d,1 = fd,1 ∗ v2, the output of the decoder block is d1 which is computed as the addition of fd,1 and27the scaled version f ′d,1. Then, up-sampling is applied on d1. The second decoder block is the same butuses the convolutional layer’s output in the encoder’s first transition block in the encoder ft,1 instead ofthe second transition block. The third block is a convolutional layer followed by instance normalization andup-sampling to produce a feature map d3 with dimensions 240×320×64.3.2.1.5 REFNet Multi-scale region and edges fusionMost deep learning techniques cannot detect the objects’ boundary details because they are trained toextract the region of the moving object (i.e., pixels belonging to the moving object). This causes the networkoutput to approximate the actual output, which means high detection accuracy but low overlap (F-measure).REFNet is trained to extract both the region and the boundary edges of a moving object, forcing the networkto predict the actual object shape and not an approximation of it. If the network predicted an approximationof the boundary edges, the detection accuracy, in this case, will be zero, as shown in Figure 3.7. Threeversions of the output are extracted at different scales of feature maps to detect most of the object’s finedetails. A residual block is subsequently trained to merge the three decoder outputs to produce the finalnetwork output. The decoder outputs are produced from each block in the decoder. After each decoderblock, an up-sampling layer is added to scale up the feature map from its size to the target size, which is240×320. After scaling, a convolutional layer with only two filters is applied on the feature map to producetwo binary images. The first binary image is the object’s detected region, where 1 represents a pixel inthe moving object, and 0 represents the background. The second binary image is the detected object’sboundary edges, where 1 is a boundary edge pixel, and 0 is not an edge pixel. We get three differentoutput versions as we have three decoder blocks. It is noted that the output of each decoder block capturesdifferent details of the moving object, as shown in Figure 3.8. Therefore, a residual block combines thesethree outputs to produce a single output with all fine details detected at each decoder block. We named thisresidual block multi-scale region/edges fusion (REF) as it fuses different regions and edges extracted ateach decoder block. The input to REF is the last decoder block feature map of size 240×320×64 and thesix binary images produced from each decoder block, which makes the input size for REF 240×320×70.The six binary images act as semantic feature maps and fine boundary details of the decoder feature map.The output of REF is a binary image with 240× 320× 1 that represents the moving objects in the current28Figure 3.7: Comparison between the network trained to extract region only versus region and boundaryedges. In the output image, a white pixel means ground truth, and a red pixel means predicted.Figure 3.8: Outputs at different feature scales. Each decoder block captures different details. The finaloutput is the output of the REF block which fuses all three decoder outputs.frame.3.2.2 ExperimentsThe dataset used in this experiment is CDNet. The results of REFNet are compared with state-of-the-artmodels in foreground segmentation. The training set for CDNet is 200 frames from each video and testedon the frames specified for testing for each video sequence. The evaluation metrics used are false-positiverate (FPR), false-negative rate (FNR), recall (Re), precision (Pr), and F-measure. As shown in Table 3.4,The F-measure of state-of-the-art methods ranges from 74.0% to 84.2%, while REFNet outperforms thebest method by 3.9% in F-measure. A sample visual output of REFNet versus other models is shown in29Table 3.4: Results of REFNet compared with state-of-the-art methods on CDNet, methods with * areour previous work. Methods that have # at the end mean that the results are reproduced.Method FPR FNR Re Pr F-MeasuresEnDec# [1] 0.004 0.076 0.903 0.798 0.842FgSegNetV2# [63] 0.005 0.080 0.893 0.741 0.801CascadeCNN# [123] 0.007 0.097 0.821 0.771 0.786IUTIS-5 [11] 0.005 0.215 0.789 0.808 0.771SemanticBGS [12] 0.004 0.211 0.789 0.830 0.789BSUV-Net [114] 0.005 0.179 0.820 0.811 0.786BSUV-Net-SBGS [114] 0.005 0.182 0.817 0.831 0.798DeepBS [7] 0.009 0.245 0.754 0.833 0.745SuBSENSE [106] 0.009 0.187 0.812 0.751 0.741WisenetMD [52] 0.009 0.182 0.817 0.766 0.753PAWCS [107] 0.005 0.228 0.771 0.785 0.740MODSiam* [85] 0.010 0.181 0.823 0.692 0.714MODY-Net* [86] 0.005 0.124 0.903 0.822 0.865REFNet (ours) 0.004 0.038 0.961 0.827 0.881Figure 3.9.3.2.2.1 Ablation studyTo show the effectiveness of the proposed REFNet, Three different versions of the network are testedusing CDNet. Each version has different components of the network. The first version is without multi-scaleoutputs or the fusion block. The second version of the network uses the multi-scale region and edges, andthe output is the average of all three regions extracted at each scale. The third version is the same asthe second version, but instead of the average of the different regions, a residual block is added to learnthe best possible way to fuse all the regions and edges. As shown in Table 3.5, the first version of thenetwork achieved 0.831 F-measure. The multi-scale region and edges greatly improved the precision and30Figure 3.9: Visual comparison between REFNet’s output and other models’ output. The top row is anRGB frame and corresponding ground truth from a video sequence. The bottom row is the outputof different foreground segmentation models.Table 3.5: Different versions of the REFNet results on CDNetComponents of REFNet Evaluation MetricsMulti-ScaleRegion/EdgesRegion/EdgesFusion BlockFPR FNR Re Pr F-MeasureNo No 0.0049 0.088 0.911 0.797 0.831Yes No 0.0039 0.115 0.884 0.870 0.863Yes Yes 0.0044 0.038 0.961 0.827 0.881false-positive rate but decreased the recall. The second version’s result was expected, as the average ofdifferent outputs will decrease the network uncertainty, which by default increases the precision. On theother hand, the recall metric decreased due to the region’s low resolution at a small scale (i.e., early blocksin the decoder). To find the best trade-off between precision and recall using different regions and edges tomaximize the F-measure, a residual block is added in the third version, which increases both the recall andF-measure while decreasing the precision slightly.31Figure 3.10: TransBlast architecture.3.2.3 DiscussionREFNet has achieved state-of-the-art foreground segmentation performance by detecting the fine detailsof the exterior silhouette of the object that is usually missed in the output of other foreground segmentationmodels. However, it is well-known in other computer vision tasks that the vision transformer performs betterthan the convolutional neural networks. In the next section, an attempt to use vision transformers in thecontext of foreground segmentation is proposed.323.3 Self-supervised learning using augmented subspace with transformerfor foreground segmentation (TransBlast)3.3.1 TransBlast ArchitectureThe proposed TransBlast in [80] is a novel Transformer-based network architecture for background/foregroundseparation. The loss function of TransBlast is designed to be augmented by subspace learning. The ar-chitecture of TransBlast is shown in Figure 3.10. TransBlast is an encoder-decoder architecture. Theencoder is divided into a convolutional encoder and a Transformer encoder. The same goes for the de-coder. TransBlast uses a transformer because its learning capacity is better than the convolutional neuralnetwork (CNN). Hence, it can learn a stronger object representation than a CNN.TransBlast input is a single RGB frame with fixed dimensions 256× 256× 3. A convolutional encoderis used to transform the input from the image space to the feature space. The CNN encoder consists of3 residual blocks with a max-pooling layer between every two residual blocks. Each residual block has2 convolutional layers with a ReLU activation function followed by a skip connection from the input to theoutput of the second convolutional layer. Then, batch normalization is added after skip connection.Thenumber of filters and the filter size is fixed in all convolutional layers of the network, which are 64 and 3×3,respectively. The output feature maps of the CNN encoder have dimensions 64× 64× 64 correspondingto height, width, and channel (i.e., H×W ×C). The feature maps are divided into patches with the size4×4×64. This will result in 16×16 = 256 patches. Each patch is flattened to form a 1−D vector of size1024. All flattened patches are used as input to the transformer encoder.The transformer encoder consists of a 6 encoder block. In each encoder block, a layer normalization isapplied to the input patches x. Then, the input is transformed to three inputs q,k, and v as following:q= x+ p,k = x+ p,v= x (3.5)where q,k, and v are the query, key, and value, respectively. p is the position encoding which is a learnablevector. An multi-head attention layer Amh is applied on q,k, and v, such that:A(q,k) =q× kT√dk, (3.6)33A(q,k) = so f tmax(A(q,k)), (3.7)where dk is a scaling factor which is the dimension of k.Amh(q,k,v) = pro j(A(q,k)× v), (3.8)where pro j is a linear projection layer. A skip connection from before the Amh to after it is added. Then, alayer normalization followed by feed-forward network f f n with another skip connection is applied as follows:x= x+Amh(q,k,v), (3.9)x= layernorm(x), (3.10)x= x+ f f n(x). (3.11)where f f n is the feed-forward network consisting of two linear projection layers. Then x is fed as an inputto the next encoder block along with the position encoding p.The transformer decoder also consists of 6 decoder blocks. Each decoder block has the same structureas the encoder block but has two Amh layers. The decoder output is a set of flattened patches with thesame dimensions as the input flattened patches of the encoder. These output patches are unflattened toform 4×4×64 patches. Then, the unflattened patches are reshaped to form feature maps of dimensions64×64×64.The convolutional decoder uses the reshaped transformer output as an input and decodes it to output abinary image where the black pixel means the background pixel and the white pixel means the foregroundobject pixel. The architecture of the CNN decoder is 2 residual blocks with an up-sampling layer betweenevery two residual blocks. The two residual blocks have the same number of filters, 64, and the ReLUactivation function. A convolutional layer with a 2 filter and sigmoid activation function is added to eachblock. This convolutional layer is used to output 2 binary images from each block—one binary image for34Figure 3.11: TransBlast architecture during training using Barlow twins.the segmented foreground and another one for the predicted boundaries of the foreground. An extra up-sampling layer is added before the convolutional layer of the first block to make the output match the finaloutput size.The multiple-output combiner block (MOC-block) consists of a residual block that uses the output featuremaps of the convolutional decoder and the 2 outputs as input. This residual block learns to combine thefeature maps and different outputs to produce feature maps that highlight the moving objects and theirboundaries. At the end of the MOC block, a single convolutional layer with one filter and the sigmoidactivation function produces the final output, a binary image with dimensions 256×256×1.The subspace module augments the cross-entropy loss function by calculating the SVD for the outputtensor of the MOC block. Then, the decomposed matrix S1 is extracted to produce the low-rank approxi-mation matrix S2, as shown in Figure 3.10. The low-rank approximation matrix S2 is calculated from S1 asfollows:S1 =UΣV T (3.12)S2 =UkΣkV Tk (3.13)whereU andV are orthogonal matrices, and Σ is a diagonal matrix with non-negative entries. Note that thediagonal in the Σ matrix is ordered descending with respect to the singular values. The k is the top singularvalue (i.e., low-rank). The low-rank approximation matrix S2 is fed into a multi-layer perception (MLP), a4-layers network. The output of the MLP is two Eigenvalues. The objective is to maximize the Eigenvaluesduring the training when added to the cross-entropy described in the next section.353.3.2 TransBlast trainingTransBlast is trained using a self-supervised learning technique that allows the model to learn a strongobject representation for the input image using unlabeled data. The learning process is divided into twophases: 1) Pre-text task (learning representation using Barlow twins [131]) and 2) Down-stream task (usethe trained model to learn to segment foreground objects).In the pre-text task phase, the network architecture uses Barlow twins as a representation learningtechnique. The CNN decoder is replaced by 3 linear projection layers. Each projection layer has 4096neurons. The CNN decoder is removed because it is a task-specific part (i.e., related to the foregroundsegmentation task only). In self-supervised learning, we focus on learning a strong object representationindependent of any task (i.e., task-agnostic representation). Then, TransBlast is transformed to Siamesearchitecture [14], where two branches of the same network have shared weights but use different inputs.The modified TransBlast for using Barlow twins is shown in Figure 3.11. The different inputs are augmentedfrom the original input image. TransBlast uses data augmentation techniques such as random image crop,horizontal flipping, color jitter, grayscale, gaussian blur, and solarization. Each data augmentation techniqueis applied with a probability. The two branches extract embedding vectors zA and zB for the different inputsxA and xB. A cross-correlation is applied on these two embedding vectors, which is computed as:ci j =∑bzAb,izBb, j√∑b(zAb,i)2√∑b(zBb, j)2. (3.14)If the result from the cross-correlation c is an identity matrix I, this means both embedding vectors arethe same, and this is the target of the Barlow twins’ technique to force the network to extract embeddingvector that is invariant to the different data augmentation techniques. This is done by computing a lossfunction that minimizes the difference between the computed cross-correlation and the identity matrix. Thisloss function `BT is computed as:`BT =∑i(Iii− cii)2+α∑i∑i 6= j(Ii j− ci j)2. (3.15)Since the identity matrix I has ones on the diagonal Iii and zeros on the rest of the matrix, the loss function36can be rewritten as:`BT =∑i(1− cii)2+α∑i∑i6= jci j2. (3.16)In the downstream task phase, the CNN decoder is added to the network trained by the Barlow twins.All parameters in the network that have been updated using Barlow twins will be kept unchanged (i.e.,frozen parameters) during this phase. Only the parameters in the CNN decoder will be updated. This CNNdecoder transforms the embedding representation that is learned by the Barlow twins to segment objectsin the foreground domain. The input in this phase is a single image, and the output is the binary image, asmentioned in the previous section. The loss function in this phase is computed as:`BFS = `(y, yˆ)+2∑i=1(`(y, yˆi)+ `(yb, yˆib)). (3.17)where `BFS is the loss function for the background/foreground separation, ` is a cross-entropy loss function,y is the ground truth, yˆ is the output of the network, yˆi is the output of ith decoder block, yb is the boundariesground truth, and yˆib is the predicted boundaries of the ith decoder block.The loss function of the subspace learning module (SLM) that augments the cross-entropy is given by:`SLM =−β2∑i=1(λi). (3.18)Where β is an empirical strength value that is set to 0.2, λ is the Eigenvalue that MLP produces. The finalaugmented loss function is calculated as follows:` f inal = `BFS− `SLM. (3.19)3.3.3 ExperimentsTwo different versions of the proposed model are evaluated in this section. The first version is TransBlast-LA, which is the network without loss augmentation. The purpose of this version is to show the strengthof the proposed network alone and how loss augmentation will affect the performance of the network. Thesecond version is TransBlast, which is the full proposed model. Evaluation of the proposed TransBlast is37Table 3.6: Results of TransBlast compared to state-of-the-art methods on CDNet, methods with *means our previous work. Methods that have # at the end mean that the results are reproduced.Method FPR FNR Re Pr F-MeasureFgSegNetV2# [63] 0.005 0.080 0.893 0.741 0.801CascadeCNN# [123] 0.007 0.097 0.821 0.771 0.786MODSiam* [85] 0.010 0.181 0.823 0.692 0.714MODY-Net* [86] 0.005 0.124 0.903 0.822 0.865REFNet* [81] 0.004 0.038 0.961 0.827 0.881TransBlast-LA (ours) 0.004 0.086 0.841 0.797 0.815TransBlast (ours) 0.004 0.083 0.867 0.811 0.831done using CDNet. The dataset used to train the model Mini-ImageNet dataset in the Barlow twins trainingphase.3.3.3.1 ResultsThe proposed model TransBlast is pre-trained using Barlow twins to learn strong object representation fromthe Mini-ImageNet dataset. Then, the convolutional decoder is added in the downstream phase, and itsparameters are updated using the foreground segmentation datasets. In this phase, the rest of the networkparameters is also fine-tuned.In Table 3.6, the proposed model is trained on 200 frames from each video, and testing is done onthe rest of the frames. The proposed model outperforms state-of-the-art models even without loss aug-mentation. This proves that a transformer-based network with self-supervised initialization outperformsthe CNN-based networks when the model is trained using the augmented loss function, the performanceincrease even more, achieving 0.831 in terms of F-measure. Although it is well-known that transformersoutperform CNNs in many other computer vision tasks, the performance of the transformer was less thanthat of REFNet by 5%. This is because transformers require a massive labeled dataset to achieve goodperformance, while the dataset CDNet has limited labeled data.38Table 3.7: Effect of each component in TransBlast using SegTrackV2 dataset.Transformer MOC-block Boundary optimization Self-supervised Loss augmentation F-MeasureNo No No No No 0.702Yes No No No No 0.836Yes Yes No No No 0.857Yes Yes Yes No No 0.873Yes Yes Yes Yes No 0.882Yes Yes Yes Yes Yes 0.9043.3.3.2 Ablation studyA study is made to show the Effect of different threshold values on the proposed model against othermodels. As shown in Figure 3.12, the performance of all CNN-based models is affected by the thresholdvalue. The performance keeps increasing until a certain threshold value then starts to decrease again. Onthe other hand, the performance of the proposed TransBlast does not get affected by the threshold value.This is due to the solid learned object representation and also because the TransBlast is based on thetransformer, which is well-known for its ability to learn a representation that is better than the CNN in manydifferent computer vision tasks (e.g., Object detection in [15]).The Effect of different components in TransBlast is shown in Table 3.7. The results in this table areproduced using the SegTrackV2 dataset. The components presented in the table are transformer, MOC-Block, boundary optimization (i.e., optimized to extract the boundaries), self-supervised learning usingBarlow twins, and loss augmentation. As shown in Table 3.7, when the network had only the convolutionalencoder/decoder, the performance was 0.702 in F-measure. After adding the transformer, the performancebecomes 0.836. Adding MOC-block, which combines multiple outputs at different scales, improved theperformance by 2.1%. When the network is trained to extract boundaries using the proposed loss functionin Eq. 3.17 in Section 3.3.2, the performance improved by almost 1.6%. Pre-training the network self-supervised learning improved the performance by 0.9%. Finally, the full proposed TransBlast achieved0.904 F-Measure, which is the first place in this dataset.39Figure 3.12: Results of TransBlast against three other models in SegTrackV2 using different thresholdvalues.3.4 ConclusionREFNet successfully captures the fine details of the exterior silhouette of the object. These fine details areusually absent in the output of other foreground segmentation models. Hence, the output of REFNet is theclosest to the ground truth from any other foreground segmentation model.Although REFNet outperforms state-of-the-art models, REFNet is still following the traditional machinelearning paradigm, which means it has the same limitations as the other foreground segmentation models.REFNet requires massive labeled data and cannot learn from multiple domains without massive perfor-mance degradation. In the next chapter, multiple few-shot learning techniques are introduced to addressthe problem of learning using a few examples.40Chapter 4Few-shot learning models under limiteddataThis chapter tackles the problem of training a deep learning model with limited labeled data. Since mostfew-shot learning has been introduced in the context of image classification, a novel few-shot technique isproposed for image classification. Then, two novel few-shot learning techniques are proposed in foregroundsegmentation.4.1 Few-Shot Learning Network for Image ClassificationThe proposed hybrid meta-learning Network in [83] is called Knowledge Transfer Network (KTNet). KTNetleverages optimization-based and metric-based learning (i.e., hybrid meta-learning) to 1) transform a givendataset of In-Domain (ID) and Out-Of-Domain (OOD) classes to a dataset of classes that are ID only and2) autonomously increase the size of the support set.4.1.1 Knowledge transfer network (KTNet) architectureThe architecture of KTNet consists of a backbone followed by two branches that use two different meta-learning techniques: an optimization-based learning branch (OBB), and the metric-based learning branch(MBB), as shown in Figure 4.1. The backbone learns to extract features from the input image that areuseful for both branches. In KTNet, we use a modified version of ResNet12 instead of NAS as it is widely41used in few-shot learning. The backbone consists of four blocks. Each block contains R−residual blocks,where R = 1 at the beginning and linearly increases in later blocks until R = 4 in the fourth block. Eachresidual block has a skip connection from the output of the first convolutional layer to the output of the lastlayer in the block. A meta-layer (ML) is attached to the last residual block in each R− residual block in thebackbone.The optimization-based learning branch (OBB) is used to classify the feature maps extracted by thebackbone to one of the target classes. The OBB consists of one fully connected (FC) layer. The FC layerhas a number of neurons equal to the number of target classes.The metric-based learning branch (MBB) is used to learn a feature space that can separate imagesthat belong to different classes and group images that belong to the same class. The MBB consists ofone 4−residual block followed by a single 1× 1 convolutional meta-layer (CML) to transform the featureslearned from task-agnostic to task-specific.4.1.2 Training of KTNetThe training process is decomposed into two phases, a pre-training phase to train KTNetbase, which is thenetwork without the meta-layers, and a meta-training phase to train full model KTNet.Pre-training phase: In this phase, the backbone and OBB are pre-trained on a large, labeled dataset.The point of this phase is to learn to extract strong features and accelerate the convergence during themeta-training phase. Let Θ denote the backbone base parameters (i.e., without the meta-layer), and θdenotes the OBB parameters. The parameters are updated using Stochastic Gradient Descent (SGD) asfollows:[Θ,θ ] = [Θ,θ ]−α∇[Θ,θ ]`OBB(KTNetbase(x),y) (4.1)where ∇[Θ,θ ]`OBB(KTNetbase(x),y) is the partial derivative of the OBB’s loss function with respect to [Θ,θ ],and `OBB(KTNetbase(x),y) is a cross-entropy loss between the predicted label KTNetbase(x) and the actual42label y, and α is the learning rate. The KTNetbase(x) is computed as following:z(x) = fΘ(x)KTNetbase(x) = fθ (z(x))(4.2)where z(x) and fΘ(x) are the output feature maps of the base backbone using the input x, and fθ (z(x))is the output of OBB for the input feature maps. The term f[parameters](x) is used in this work as a partialfunction of the network that is defined by the set of parameters (e.g., fΘ(x) is the output of the backboneusing input x because Θ are the parameters of backbone, and fθ (x) means the output of OBB part of thenetwork only on the input x).On the other hand, the MBB is pre-trained using the same large dataset used in the KTNetbase pre-training phase. The main target of the MBB is to learn strong discriminative features from inputs, such thatsimilar inputs (i.e., inputs belonging to the same class) have similar features and different inputs (i.e., inputsbelonging to different classes) have different features. The MBB is divided into a 4−residual block and a1×1 convolutional meta-layer. The 4−residual block parameters are defined as Φ, and the convolutionalmeta-layer parameters are defined as φ . Similar to the backbone and OBB pre-training, both parameters ofthe MBB Φ and φ , are updated. The loss function used to train these parameters is contrastive loss, wheredissimilar feature maps generate a high loss and similar feature maps generate a low loss. The functionapplied by MBB for input x is g(x) which is computed as follows:g(x) = f[Φ,φ ](z(x)) (4.3)The similarity functions SIM between two given images x1 and x2 is the cosine similarity between the twofeature maps extracted by MBB from x1 and x2. The distance metric is computed as follows:SIM(x1,x2) = cos(g(x1),g(x2)) =g(x1)g(x2)‖g(x1)‖ ‖g(x2)‖ (4.4)The contrastive loss function is computed as follows:`MBB(SIM(x1,x2),y) = y∗SIM(x1,x2)+(1− y)∗max(0,c−SIM(x1,x2))(4.5)where l is the loss function, y is the label such that y ∈ [0,1], where 0 signifies a negative pair (two imagesfrom two different classes), 1 means positive pair (two images from the same class), and c is the margin43Figure 4.1: The proposed hybrid meta-learning network (KTNet) architecture. The green arrow repre-sents the training phase (supervised learning using the support set). The yellow arrow representsthe testing phase (classifying the query set). The purple arrow represents the selection of samplesfrom the query set and adding them to the support set (to increase the number of shots). Theprocess is repeated until no more samples from the query set are added to the support set.which is set to 0.5. Finally, this loss is used to update [Φ,φ ] using:[Φ,φ ] = [Φ,φ ]−α∇[Φ,φ ]`MBB(SIM(x1,x2),y) (4.6)Meta-training phase: In this phase, the large labeled dataset is divided into N, small, and few-labeleddatasets. Each few-labeled dataset has support set s, which is few-labeled images per class, and the queryset q. Generally, the support set is the training set, and the query set is the testing set. However, both setsin each few-labeled dataset are used for updating the network parameters during the meta-training phase.In this phase, only meta-parameters and parameters of the OBB in KTNet are updated using the supportset of a given dataset. The operation performed by the meta-layer is scaling and shifting the filters of thebase network’s residual blocks.ML(X ;W,b;M1,M2) = (W M1)X+(b+M2) (4.7)M1 and M2 are the meta-parameters, and each one of them is a single value. These two parameters aredefined for each filter in the residual block, where W and b are the filter parameters. In other words, foreach 10 parameter (i.e., 3×3 = 9 filter weights and 1 bias), 2 meta-parameters are defined.In this phase, the OBB’s parameters θ are replaced by task-specific parameters θ ′, which initially have44the same value as θ . The parameters θ ′ are fine-tuned using few-labeled images (i.e., the support set S)with the following equation:θ ′ = θ ′− γ∇θ ′`OBB(KTNet(xS),yS) (4.8)where xS and yS are the input image and label from the support set S. and KTNet(x) is the output of thenetwork with meta-layers, which is computed as follows:zm(x) = f[Θ,M1,M2](x)KTNet(x) = fθ (zm(x))(4.9)where zm(x) and fΘ,M1,M2(x) are the output feature maps of the backbone (with meta-layers) using the inputx. The fine-tuned θ ′ are subsequently used to classify the testing set (i.e., the query set Q) and update thebase task-agnostic θ and meta-parameters as follows:θ = θ − γ∇θ `OBB(KTNet(xQ),yQ) (4.10)Mi =Mi− γ∇Mi`OBB(KTNet(xQ),yQ) (4.11)where xQ and yQ are the input image and label from the query set. The same support set and query setare used to update the parameters φ of the meta-layer at the end of the MBB.After the meta-training phase is complete, the proposed network, KTNet, has a huge base of knowledge(which provides a good generalization and meta-layers that transfer this knowledge to any few-labeleddataset. The trained KTNet can converge to the optimal task-specific parameters using few-labeled imagesand fewer iterations without over-fitting while achieving high accuracy.4.1.3 Classification of in-distribution and out-of-distribution classesThis section first explains how to transform a dataset with in-distribution (ID) and out-of-distribution (OOD)classes to a dataset with all images inside the distribution. To facilitate this, we propose a technique thatallows KTNet to autonomously increase the number of labeled images in the support set.45Figure 4.2: Images belonging to 5 different classes are drawn using T-SNE[118]. The number of im-ages per class is 15 images. (a) The images are drawn in the image space, (b) the images aredrawn in the task-agnostic feature space, and (c) the images are drawn in the task-specific featurespace learned using self-supervised learning on top of the task-agnostic feature space.4.1.3.1 Self-supervised learning for support set initialization (S4I)First, we construct a few-labeled dataset divided into a support set and a query set, where the query sethas a number of classes T larger than the number of classes N in the support set. In doing so, the queryset has E classes out-of-distribution compared to the support set classes, such as E = T −N. The goal isto initialize new support set S f with one image for each class of the query set. Such that no more out-of-distribution classes exist in the query set; this way, we transformed the dataset with ID and OOD classesinto one with all classes ID.To initialize the support set covering the entire distribution, we must select at least one image from eachclass in the E out-of-distribution classes and add it to the current support set.This subsection proposes a self-supervised learning technique for the support set initialization (S4I)technique. This technique is divided into three phases: 1) data augmentation, 2) self-supervised learning,and 3) selection of a single image for each out-of-distribution class.Different data augmentation techniques are applied to the query set to generate more images in thedata augmentation phase. The data augmentation techniques used are random crop, rotation, translation,color channel shifting, and contrast.The second phase involves the application of self-supervised learning. Two different tasks are defined:a pretext task and a downstream task. The pretext task is usually a task that is generated using an unlabeleddataset (e.g., given a group of unlabeled images, convert them to grayscale, then the input for the network46is the grayscale images and the output is the RGB images. The network will learn image colorization usingan unlabeled dataset. This is an example of a pretext task generated using a group of unlabeled images). Inthe pretext task, we maximize the similarity between images in the query set and corresponding images inthe augmented set to minimize the similarity between different images. This way, the network will implicitlylearn a feature space that separates the images from each other, as shown in Figure 4.2. During thepretext-task, a set of pairs Qpaired is used, which is defined as follows:Qpaired = {{(x11,x12),y1}, ...,{(xZ1 ,xZ2 ),yZ}} (4.12)where the first pair has x11 as an image from the query set Q, x12 as an image from the augmented setQaugmented , and y1 as the label of the pair, such that y1 = 1 for positive pairs (i.e., the image from Q andcorresponding image in Qaugmented), and yi = 0 for negative pairs (i.e., the image from Q and a differentimage from Qaugmented). Finally, Z is the size of the set of pairs and equals the number of images in thequery set.For the network, we use KTNet with the MBB. KTNet parameters are kept unchanged. For the MBB,a clone of the trained MBB parameters is created as Φ′ and φ ′. Then, the cloned parameters are updatedusing Eq. 4.6 with the generated dataset Qpairs. After training with the pretext task, the cloned, updatedparameters are used in the downstream task, which is the support set initialization.The last phase is image selection. This phase aims to select a single image from each out-of-distributionclass. These images will be used as a support set for the given unlabeled dataset. The Steps of this phaseare as follows: 1) the MBB (with modified φ ′) is used to compute the similarity between the image and allother images in Q, 2) the image with the least similarity is selected (q). This image q is most likely to belongto a different class. The process is repeated E-times because we know the number of classes in Q, whichis T , and the number of classes in S, which is N. By applying this process, we have images belonging to Tdifferent classes in the support set. The whole process of S4I for a given dataset is shown in Algorithm 1.47Algorithm 1 S4I.Require: Support set S that has N classes, query set Q that has T classes, similarity function SIM, networkKTNet, and number of updates UEnsure: Support set that covers the full distribution S f1: [Φ′,φ ′]← [Φ,φ ]2: Qaugmented ← data augmentation(Q)3: Qpairs← generate pairs(Q,Qaugmented)4: for u in U do5: update [Φ,φ ′] using `MBB of Qpairs6: end for7: %Convert from unlabeled dataset to few-labeled dataset8: E ← T −N9: S f ← S10: for i← 1 to E do11: x← sample with least similarity(SIM(S f ,Q))12: y← Assign pseudo label(x)13: Add {x,y} to S f14: end for15: Return S f4.1.3.2 Robust and effective support image selector technique (RESIST)During testing, the dataset has few-labeled images (i.e., one image for each class). These few amountsof labeled images limit the classification performance of KTNet. To improve the performance of KTNet,we need to increase the number of labeled images. This is why, we propose RESIST, a technique thatleverages the advantages of hybrid learning within KTNet to autonomously increase the number of labeledimages.The process of RESIST is repeated in a loop. In each iteration, KTNet attempts to select a portion ofimages from the query set and add them to the support set. The selection process is divided into threesteps: 1) Fine-tuning both branches of KTNet, the OBB, and the MBB. 2) Classifying the query set usingboth updated branches. 3) Select images with high confidence from the query set and add them to thesupport set.48As part of the first fine-tuning step, a clone of the CML parameters within the MBB is fine-tuned usingthe support set. An additional clone of the fully connected layer parameters within the OBB is fine-tuned.This fine-tuning is applied to transform the parameters from task-agnostic to task-specific.In the second step, the network classifies the query set using both branches, the OBB and MBB. Forthe last step (i.e., the image selection step), an image in the query set is classified using both branchesto the same class with high confidence. This image can be selected as a support image and added tothe support set with the predicted class label. The same is done for each image in the query set. Thenew support set S′ has more images per class (i.e., the number of labeled images has increased). Thenew support set S′ is used as a new few-labeled dataset for the next iteration of RESIST. The selectionof new images is robust because it is based on hybrid meta-learning (i.e., metric- and optimization-basedlearning). Additional efficiency was also observed due to image feature extraction and distance matrixcalculation performed in one forward pass of KTNet. The process of RESIST is illustrated in Algorithm 2.49Algorithm 2 RESIST.Require: support set S, query set Q, similarity function SIM, network KTNet, number of classes N, andnumber of updates UEnsure: Improved support set S′1: S′← S2: while new images are added to S’ do3: θ ′← θ4: φ ′← φ5: for u in U do6: update θ ′ using `OBB of KTNet(S′)7: update φ ′ using `MBB of SIM(S′,Q)8: end for9: Od ← SIM(S′,Q)10: OKTNet ← KTNet(Q)11: Qsubset ← images with high confidence(Q,Od ,OKTNet)12: if |Qsubset |> 0 then13: Add Qsubset to S′14: end if15: end while16: Return S′4.1.3.3 Testing of KTNet-S4I-RESISTFor a given few-labeled test dataset (i.e., a dataset not seen during pre-training and meta-training phases),few-labeled images in the support set belong to N classes, and images in the query set belong to Tclasses. The classification process of this dataset is as follows: 1) initialize new support set S f that coversthe distribution of all T classes in the query set (i.e., transform the dataset to a dataset without out-of-distribution classes), 2) improve the initialized support set by creating new support set S′ that has increasedthe number of labeled images. 3) KTNet is fine-tuned using the improved support set S′, 4) The fine-tunedKTNet is used to classify the query set. The full classification process of a few-labeled test dataset is shownin Algorithm. 3.50Algorithm 3 The classification process of a few-labeled dataset using KTNet-S4I-RESIST.Require: Support set S, number of classes in support set N, query set Q, number of classes in query set T ,similarity function SIM, network KTNet, and number of updates UEnsure: Classification accuracy Acc1: if N = 0 then2: x← random image from(Q)3: y← Assign pseudo label(x)4: Add {x,y} to S5: N ← 16: end if7: if T > N then8: S← S4I(S,Q,SIM,KTNet,U)9: end if10: S← RESIST(S,Q,SIM,KTNet,T,U)11: θ ′← θ12: for u in U do13: update θ ′ using `OBB of KTNet(S)14: end for15: OKTNet ← KTNet(Q)16: Acc← compute accuracy(OKTNet)17: Return Acc51Table 4.1: Classification accuracy results for few-shot learning using 5-ways with 1-shot and 5-shot onmini-ImageNet.Model Transductive 1-shot 5-shotData augmentationSalNet Inter. Hal [133] No 57.45 ± 0.88 72.01 ± 0.67IDeMe-Net [20] No 59.14 ± 0.86 74.63 ± 0.74Optimization-based learningMAML [29] No 48.70 ± 1.75 63.15 ± 0.92adaResNet [76] No 56.88 ± 0.62 71.94 ± 0.57LEO [97] No 61.76 ± 0.8 77.59 ± 0.12MAML++ [4] Yes 62.86 ± 0.79 77.64 ± 0.4MetaOptNet-SVM [51] No 64.09 ± 0.62 78.63 ± 0.46MTL[109] No 61.2 ± 1.8 75.5 ± 0.8SIB [34] Yes 70.0 ± 0.6 79.2 ± 0.4BD-CSPN [64] Yes 70.31 ± 0.93 81.89 ± 0.60BD-CSPN+ESFR [50] Yes 73.98 82.32Metric-based learningCompareNets [110] No 50.44 ± 0.82 65.32±0.7DN4 [58] No 51.24 ± 0.74 71.02 ± 0.84PN+fsL+CP [125] No 49.64 ± 0.31 69.45 ± 0.28DC [61] No 62.53 ± 0.19 79.77 ± 0.19relation net M, CTM [56] No 62.05 ± 0.55 78.63 ± 0.06RENet [42] No 67.60 ± 0.44 82.58 ± 0.30SSFormers [19] No 67.25 ± 0.24 82.75 ± 0.20MSML [40] Yes 72.41 ± 1.72 84.33 ± 1.14Memory-based NetworkSNAIL [75] No 55.71 ± 0.99 68.88 ± 0.92TADAM [79] No 58.5 ± 0.3 76.7 ± 0.3LAWMP [135] No 65.91 ± 0.53 82.91 ± 0.49(Ours) KTNetbase No 57.62 ± 1.8 67.42 ± 1.7Hybrid learning KTNet No 65.13 ± 1.1 78.51 ± 0.92(Metric+optimization) KTNet-RESIST Yes 74.43 ± 0.8 82.64 ± 0.352Table 4.2: Classification accuracy results for few-shot learning using 5-way with 1-shot and 5-shot onfewshot-CIFAR.Model 1-shot 5-shotOptimization-based learningMAML [29] 38.1 ± 1.75 48.6 ± 0.6MAML++* [4] 38.7 ± 1.6 52.9 ± 0.4MetaOptNet-SVM [51] 47.2 ± 0.6 62.5 ± 0.6MTL [109] 45.1 ± 1.8 60.8 ± 0.9MTL w/ LCC [66] 46.8 ± 1.7 62.1 ±0.7SIB* [34] 52.2 ± 0.5 63.41 ± 0.4Metric-based learningDC [61] 42.04 ± 0.17 57.05 ± 0.16Prototypical Networks [104] 35.3 ± 0.6 48.6 ± 0.6MatchNets [111] 44.2 ± 1.9 58.0 ± 0.8EASY [19] 48.07 ± 0.19 64.74 ± 0.19SSFormers [10] 43.72 ± 0.21 58.92 ± 0.18Memory-based Network TADAM [79] 40.1 ± 0.4 56.1 ± 0.4(Ours) KTNetbase 41.82 ± 1.3 54.6 ± 1.3Hybrid learning KTNet 49.34 ± 0.6 61.35 ± 0.5(Metric+Optimization) KTNet-RESIST 57.91 ± 0.4 65.13 ± 0.4534.1.4 Experiments4.1.4.1 Experiment setupGiven a large, labeled dataset D, which is divided into two uncorrelated sets, we define a training set Dtrainand a testing set Dtest . The division process is done based on classes, not images per class (i.e. if a datasethas 20 classes and 100 images per class, the dataset will be divided into two sets based on class. As anexample, 15 classes for training and five classes for testing. Each class in each set has the full number ofimages, which is 100). After dividing the dataset, the training set Dtrain will be divided again into two sets:a support set Spre−train and a query set Qpre−train. The division this time is done based on the images perclass (i.e., a class with 100 images will be split into 70 images for the support set and 30 images for thequery set).The proposed network KTNetbase is pre-trained using the Spre−train. Subsequently, Dtrain is split into Lsmall datasets to simulate few-labeled datasets, such that:Dtrain = {d1,d2, ...,di, ...,dL} (4.13)Each small dataset di has K-shot (i.e., the number of labeled images per class in the support set Si of di),N-way (i.e., the number of classes in the small dataset), and Z-testing (i.e., the number of testing imagesper class in the query set Qi of di). The proposed network KTNet is meta-trained using all L small datasets.In the testing phase, the Dtest is split into P small datasets, which act as multiple new few-labeleddatasets. The split process is the same as that in Dtrain. Each small dataset has a support set S and aquery set Q. The goal is to use the trained KTNet to fine-tune some parameters using S (i.e., few-labeledimages that belong to classes not seen during training) for a few iterations and classify Q of this dataset.The reported accuracy is the average accuracy of classifying the Q for all P small datasets. This is the casewhere all images in Q are ID, which all state-of-the-art few-shot learning models use. We define this asimage classification without OOD classes.A second situation is explored where OOD images exist in Q. In this case, Q has more classes thanthat in S. The number of classes in Q is T , and the number of classes in S is N, such that T > N. Thedifference between the number of classes E = T −N is the number of OOD classes. We define this asimage classification with OOD classes.54Finally, a third situation is explored where there are no images in S (i.e., there are no labeled images,and all images in Q are OOD by default). In this case, the process applied is the same as in the secondcase. The only difference is that S is initialized by a random image from Q, as shown in lines 1-6 inAlgorithm. 3. In doing so, the problem is transformed from the third case to the second case. The initializedS has an image that belongs to one class, and Q has T −1 OOD classes. We define this as a pure OODImage classification.4.1.4.2 ResultsImage classification without OOD classes:The proposed work is evaluated using 1000 generated small datasets from the Dtest . Each small datasetis 1-shot or 5-shot 5-way, 15-testing, and 0-OOD. These are the standard configuration for the evaluationof a few-shot learning model. The evaluation metric used is average classification accuracy over all 1000datasets. The numbers after ± are the 95% confidence interval, which is computed as:CI95 =1.96∗ stds(Acc)√1000(4.14)where Acc is the list of accuracies for each small dataset, and stds is the standard deviation. The results of1-shot and 5-shot learning are shown in Tables. 4.1 and 4.2.In Table. 4.1, KTNet-RESIST achieved the best performance with an accuracy of 74.43% on 1shotmini-ImageNet, which is higher than the second-best model LaplacianShots by 3.69%. It is noticeable thatthe meta-layers improved the performance of the base model KTNetbase by 7.51%. Additionally, RESISTgreatly improved the performance of KTNet greatly by 9.3% at the expense of the run-time. A study of therun-time is shown in Table. 4.5. On 5-shot mini-ImageNet, KTNet-RESIST places second after DPGN witha small difference which is 1.43% accuracy.Table. 4.2 shows the results on the fewshot-CIFAR100 dataset. The proposed work KTNet-RESIST isthe number one in both 1-shot and 5-shot classification accuracy with 57.91% and 65.13%, respectively.KTNet without RESIST achieved third place in classification accuracy.As shown in both Table. 4.1 and 4.2, optimization-based techniques usually outperform the metric-based techniques in classification accuracy.55Image classification with OOD classes: The proposed work is evaluated using 1000 generated,small datasets from Dtest . Each small dataset is 1-shot, 2-way, 3-OOD (i.e., the number of classes in Qis T = 5, where 2 classes are ID and 3 are classes OOD), and 15-testing. An additional configurationis tested, which uses 3-way and 2-OOD. To make sure that the OOD classes are completely OOD, theOOD classes are sampled from a different dataset. The ID classes are sampled from mini-ImageNet, andOOD classes are sampled from fewshot-CIFAR100. The evaluation metric used is the respective averageclassification accuracy of ID classes, OOD classes, and both of them (total accuracy) for all 1000 datasets.The classification accuracy of ID classes is computed by the number of correctly classified ID samples overthe total number of ID samples. The classification accuracy of OOD classes is the same. Finally, the totalaccuracy is the number of correctly classified samples over the total number of samples (including ID andOOD samples).This experiment compares results against other few-shot learning models using our proposed S4I andRESIST. As shown in Table. 4.3, the performance of few-shot learning models degrades significantly dueto the presence of OOD classes. This degradation is due to the tested few-shot learning models classifyingOOD classes as ID classes. This causes the classification accuracy sharply decreases because all OODclasses are misclassified. The performance of other few-shot learning models increased significantly whenour proposed S4I and RESIST were adopted. However, the accuracy of ID classes decreased because theproblem now is 5-way instead of 3-way after using S4I. The performance of MTL increased by approximately14% when S4I was adopted. On the other hand, KTNet-S4I-RESIST outperforms the other few-shot learningmodels even when they adopted S4I and RESIST by approximately 4% in both cases.The accuracy of S4I to detect and select an image for each OOD class is 76.2% and 82.9% for 3 and2 OOD classes, respectively. This accuracy is computed with the average number of correctly detectedclasses divided by the number of OOD classes for all 1000 small datasets. This accuracy is not 100%,which means in some cases, the initialized support set will not have all T classes in the query set. This iswhy the performance of KTNet is less than that in the previous experiment in Table. 4.1.Pure OOD image classification: The proposed network is evaluated by classifying 1000 small, gener-ated datasets from Dtest . Each dataset has 0-shot, 0-way, 5-OOD, and 15-testing (i.e., there is no supportset, all images are in the query set, and these images belong to 5 different classes). Since it is pure OOD,56Table 4.3: Classification accuracy for 1-shot, 2-way and 3-way with presence of 3-OOD and 2-OOD.Total accuracy is calculated for all five classes (ID and OOD). The 0 in the first 2 rows means themethods completely miss-classified the OOD images.MethodAccuracy2-way/3-OOD 3-way/2-OODID OOD Total ID OOD TotalMAML++ 68.7 0 27.42 65.45 0 39.27MTL 62.82 0 25.13 62.68 0 37.61MAML++-S4I 61.47 47.67 53.19 59.73 56.53 58.45MTL-S4I 58.41 48.06 52.2 57.48 58.63 57.94MAML++-S4I-RESIST 67.63 55.33 60.25 65.06 62.78 64.15MTL-S4I-RESIST 64.16 55.49 58.96 62.27 65.32 63.49KTNet-S4I 63.23 48.71 54.52 58.66 57.16 58.06KTNet-S4I-RESIST 71.33 60.36 64.75 69.15 66.2 67.9757the 5 OOD classes are sampled from fewshot-CIFAR100, while the network is meta-trained using mini-ImageNet. The evaluation metric used is the accuracy of classifying images into 5 clusters (i.e., pseudolabels). This experiment also shows the training and testing time of methods, which is the average trainingand testing time required to cluster a small dataset. The time metric is presented in seconds.In Table. 4.4, we compared our results against clustering techniques and clustering techniques followedby our network KTNet. The clustering techniques used are KMeans, KMeans++, Mini-Batch KMeans,Spectral, and Birch. As shown in Table. 4.4, our full, proposed work KTNet-S4I-RESIST outperforms otherclustering techniques with respect to the classification accuracy by 23.72%. KTNet-S4I-RESIST takes 5.368seconds of training, while other clustering techniques have no training.The performance dropped when a clustering technique replaces S4I. As shown in Table. 4.4, when thebest clustering technique called (Spectral) is used instead of S4I, the performance degraded by 3.89%.In Table. 4.5, Spectral is used as initialization for the support set, and a few-shot learning model is usedto classify the images. The few-shot learning models used in this experiment are MAML++ and MTL. Thesetwo models were selected because they are CNN-based models, have their training weights uploaded,and are easy to adopt our RESIST and S4I without needing a major change in the network architecture.As shown in Table. 4.5, the performance of Spectral is improved by 9.54% classification accuracy whenclassification is done using a few-shot learning model. Still, the performance of the few-shot learning modelis affected greatly by the support set initialization (i.e., if the initial support set is not correct, then thefew-shot learning model will perform poorly). This is why, when the few-shot learning models were used,proposed S4I performance improved by 2.44%. This is due to our proposed S4I clustering images in atask-specific feature space learned from the given images. The proposed KTNet-S4I-RESIST outperformsother few-shot learning models even when using our proposed S4I and RESIST. This is proof that KTNetoutperforms other few-shot learning models.4.1.5 DiscussionThe proposed KTNet showed promising results in both datasets and is capable of working even without anylabeled data. Unlike other few-shot learning techniques that require at least one labeled image to classifythe query set. Although KTNet is used for image classification, still it gave us a deep understanding of few-58Table 4.4: KTNet against clustering techniques for 0shot, 0-way, 5-OOD, and 15-testing.Method AccuracyTrainTimeTestTimeKMeans [69] 33.34 0.359KMeans++ [5] 33.91 0.348Mini-Batch K-Means [99] 35.62 0.507Spectral [23] 35.97 0.138Birch [115] 34.59 0.087KMeans-KTNet 45.61 1.803 0.611KMeans++-KTNet 47.97 1.792 0.611Mini-Batch KMeans-KTNet 48.33 1.951 0.611Birch-KTNet 45.24 1.531 0.611Spectral-KTNet 49.13 1.582 0.611Spectral-KTNet-RESIST 54.42 5.217 0.611KTNet-S4I 52.46 1.843 0.611KTNet-S4I-RESIST 58.31 5.368 0.61159Table 4.5: KTNet against few-shot learning models for 0shot, 0-way, 5-OOD, and 15-testing.Method AccuracyTrainTimeTestTimeSpectral-MAML++ 45.51 1.194 0.529Spectral-MTL 44.07 1.427 0.582MAML++-S4I 47.95 1.351 0.529MAML++-S4I-RESIST 54.72 4.875 0.529MTL-S4I 46.31 1.689 0.582MTL-S4I-RESIST 54.11 5.094 0.582KTNet-S4I 52.46 1.843 0.611KTNet-S4I-RESIST 58.31 5.368 0.611shot learning techniques. Hence, we can introduce few-shot learning to the foreground segmentation task.In the next two sections, we propose two novel few-shot learning techniques for foreground segmentation.The first is a self-supervised learning-based technique, and the other is a meta-learning-based technique.4.2 Self-supervised learning for foreground segmentation with limited la-beled dataSelf-supervised learning leverages large unlabeled datasets to learn helpful information that can be easilytransferred to a new dataset with limited labeled data. The first useful information that any foreground seg-mentation model should learn is the object representation (i.e., high-level representation of each differentobject unaffected by the object’s deformation, color, and texture). The second useful information is invariantfeatures (i.e., features robust against image transformation, such as blurring, noise, illumination, rotation,and scale). In this work, two self-supervised techniques are combined to learn Object Representation andInvariant Features (ORIF) using unlabeled data [84]. The combined self-supervised techniques train Vi-sion Transformers [25]. We refer to the transformer trained by this self-supervised learning technique as60ORIF-TR. It is important to mention that the transformer is used in this experiment instead of CNN as it hasa higher learning capacity (i.e., the transformer is better than CNN when training using a large amount ofdata). A transformer is better because a massive unlabeled dataset is used to train the model.4.2.1 Model ArchitectureThe model architecture is an auto-encoder network comprising an encoder and a decoder. The encoder isused to extract high-level features from the input image. The decoder decodes the features to produce theoutput image of the same size as the input image.The encoder architecture is based on Vision Transformers. First, the input image is divided into patches.Each patch of size 16× 16. Then, A positional encoding is added to the patches and passed through alinear projection layer to produce the input for the encoder. The reason for adding the positional encodingis to keep track of the original position for each patch. Four transformer blocks process the produced input.Each transformer block has a layer normalization followed by three parallel linear projection layers, namelyq, k, and v for query, key, and value, respectively. Then, a multi-head attention layer is applied on q, k, andv as follows:A(q,k) =q× kT√sk(4.15)A(q,k) = so f tmax(A(q,k)) (4.16)Amh(q,k,v) = pro j(A(q,k)× v) (4.17)where sk is the embedding size (i.e, size of k), pro j is a linear projection layer. The exact process isrepeated for each transformer block. After the last transformer block in the encoder, a linear projection layerpro je,d is added to transform the output of the encoder e to the input of decoder d. Part of the output of thisprojection layer will be masked out (i.e., hidden from the decoder). The goal of the decoder is to predict themissing parts.The decoder architecture is the same as the encoder. The decoder input is part of the encoder output,and the other part is filled with random numbers, as shown in Figure 4.3. The output of the last block of the61Figure 4.3: ORIF-TR architecture in the pretext phase.decoder is passed through a linear projection layer to produce patches of the same size as the encoder’sinput. This output includes the seen patches from the encoder and the predicted hidden patches.4.2.2 ORIF-TR TrainingThe network is trained using our proposed self-supervised learning technique. Any self-supervised learningtechnique is divided into two phases, the pretext phase and the downstream phase. In the pretext phase,the network is trained on unlabeled data (unsupervised learning). The network is trained in the downstreamphase using labeled data (supervised learning). The task performed by the network in both phases isdifferent.4.2.2.1 Pretext phaseWe combined two different tasks for the network in the pretext phase. The network performs both taskssimultaneously, and the total loss is a combination of the loss of each task. The first task is applying arandom image transformation to the input image. Then, force the network to produce similar embeddingfor both inputs (the original and the transformed images). Let xo be the original input image and xt be thetransformed image. The embedding for both images is zo and zt , where the embedding is the output ofpro je,d layer after the encoder. We calculate the cross-correlation matrix c between the zo and zt . Thematrix c is a square matrix with the size of the input vector because the size of both vectors is the same.62Each index i and j of the matrix is defined as follows:ci, j =∑bzob,iztb, j√∑b(zob,i)2√∑b(ztb, j)2(4.18)If both vectors zo and zt are identical. Then, the cross-correlation matrix c will be an identity matrix I.We aim to minimize the difference between the matrix c and identity matrix I to force the network to producea similar vector for both images. The network will learn to produce an embedding vector invariant to imagetransformation by doing this. The transformation-agnostic loss function is defined as:`TA(I,c) =∑i(1− cii)2+α∑i∑i 6= jci j2 (4.19)where the first term of the equation calculates the sum of squared error between the diagonal of c andthe diagonal of I, which is 1 for all indices. On the other hand, the second term calculates the sum ofsquared error between the off-diagonal of c and I (values of I off-diagonal is 0 for all indices, this is why itis not in the equation). Finally, α is a learning parameter used to combine the loss of both diagonal andoff-diagonal of the matrix.The second pretext task is hiding parts of the embedding vector and letting the decoder predict thehidden parts of the input image. Let po be the original patches before it is passed through a linear projectlayer pro jin,e to form the encoder input pe. Then, pe is processed by the encoder and linear projection layerpro je,d to output zo. Then, zo is split into ps (seen patches) and ph (hidden patches). The seen patchesps are concatenated with randomly generated patches pr to form the decoder input pd . The randomlygenerated patches pr have the same size as the hidden patches ph (i.e., the size of pd is the same aspe). The decoder input pd is processed by the decoder blocks followed by linear projection layer pro jd,outto produce the final output pˆo, which is an approximation of the original patches po. The masked-decoderloss function is computed as follows:`(pom, pˆom) =1N2N∑i=1N∑j=1(pom,i, j− pˆom,i, j)2 (4.20)`MD(po, pˆo) =1‖po‖‖po‖∑m=1`(pom, pˆom) (4.21)63where N is the patch size 16, and ‖po‖ is the number of patches defined as the image size divided bythe patch size.The final loss function that is used to update all network parameters is defined as a linear combinationof both loss functions, transformation-agnostic loss `TA and masked-decoder loss `MD. The loss coefficientβ is 0.8. This value is obtained by trying different values between [0,1] over a small dataset sample.4.2.2.2 Downstream phaseIn this phase, the network is trained to perform the targeted task (i.e., Foreground segmentation). Thedecoder of the network will be replaced with a simple Convolutional-based decoder to produce a binaryimage. The reason for using a convolutional-based decoder instead of the transformer-based decoder is toallow fast adaptation using a small number of labeled images. This is because transformers’ parametersrequire a massive amount of labeled data. The convolutional-based decoder consists of three convolutionallayers. Each layer is followed by a ReLU activation function, batch-normalization layer, and up-samplinglayer (linearly increasing the embedding dimensions until it reaches the input image size). The input tothe decoder is pe after reshaping it to match the input of the first convolutional layer. During this phase,the parameters of the encoder are kept unchanged to prevent catastrophic forgetting [30] (i.e., accidentallyforgetting knowledge gained from the previous phase). Only the decoder parameters are updated.The training in this phase is simple for a given input image x with the corresponding ground truthy. The input image is processed by the network (i.e., transformer encoder, linear projection layer, andconvolutional-based decoder) to produce the output yˆ. Then, a binary cross-entropy loss function is appliedto the predicted output yˆ and the ground truth y. Finally, the loss is back-propagated to update the decoderparameters only. Since the network encoder has already learned to extract robust features and objectrepresentation, the network needs only to fine-tune the decoder parameters for a few iterations and use asmall number of labeled images on the target dataset to achieve good performance.4.2.3 Experiments and resultsThe performance of the proposed ORIF-TR is evaluated using two benchmark foreground segmentationdatasets with limited labeled data, namely DAVIS16 and SegTrackV2. The two datasets consist of images64from different scenes with one or more foreground objects in each image. Results are compared withstate-of-the-art foreground segmentation models. Two experiments are conducted on each dataset. Thefirst experiment is the standard foreground segmentation experiment, where the model is trained on labeleddata of some scenes and tested on other scenes. The second experiment is called the few-shot experiment.In this experiment, the model is trained using a dataset and fine-tuned on another dataset using a tinyamount of labeled data (few-shots). The number of shots that we used in this experiment is 1-shot and5-shots (i.e., two different configurations, the first one uses only a 1 labeled image for each scene. Thesecond configuration uses 5 labeled images for each scene. The evaluation metric used in all experimentsis the F-Measure (F-M), which combines precision and recall.4.2.3.1 Experiments pre-trainingThe pretext phase uses a huge unlabeled dataset of images, namely ImageNet. This phase pre-trains theencoder of the network using the two tasks mentioned in Section. 4.2.2.1 before conducting the foregroundsegmentation experiments. Once the pretext phase ends, the transformer decoder is removed, and aconvolutional-based decoder is added to the existing trained encoder.4.2.3.2 Standard foreground segmentation experimentThe dataset is split in this experiment into two sub-datasets (training and testing sets). The split is donebased on scenes in the dataset (e.g., the DAVIS16 dataset is split into 33 training scenes and 20 testingscenes. and SegTrack is divided into 8 training scenes and 6 testing scenes). The convolutional-baseddecoder is trained using all labeled images in the training set, and testing is performed using the testingset.Table. 4.6 and Table. 4.7 show the results of our ORIF-TR compared to state-of-the-art models. Asshown in the tables, our ORIF-TR outperforms other models by 3.2% and 0.8% on DAVIS16 and SegTrack,respectively. We can improve the performance even more if we increase the complexity of the convolutional-based decoder (i.e., add more layers and filters). But we designed the decoder to be simple to maximize theperformance when training is done using a small number of labeled images. When the decoder is simple,it can be adapted easily using a small number of labeled images and training iterations. The effectiveness65Table 4.6: Results of ORIF-TR comparedto state-of-the-art models in DAVIS16.Methods that have # at the end meanthat the results are reproduced.Model F-MeasureCNIM [8] 0.850LUCID [43] 0.820STRCF [48] 0.816DSRFCN3D [47] 0.766DCL [54] 0.711DHS [65] 0.758RFCN [120] 0.740UOVOS [137] 0.772FgSegNet# [63] 0.847CascadeCNN# [123] 0.814ORIF-TR (ours) 0.882Table 4.7: Results of ORIF-TR comparedto state-of-the-art models in Seg-TrackV2. Methods that have # at theend mean that the results are repro-duced.Model F-MeasureDCL [54] 0.730UOVOS [137] 0.643SSAV [28] 0.801DSRFCN3D [47] 0.878RFCN [120] 0.737DHS [65] 0.762GDHF [46] 0.868STRCF [48] 0.899FgSegNet# [63] 0.880CascadeCNN# [123] 0.867ORIF-TR (ours) 0.90766of this design is shown in the second experiment.Few-shot foreground segmentation experiment In this experiment, the convolutional-based decoder ofORIF-TR is trained using labeled images of a dataset. Then, fine-tuned to segment foreground objects fromanother dataset using k-shots from each scene (e.g., ORIF-TR is trained using DAVIS. Then, fine-tunedusing k-shots from each scene in SegTrack dataset). This experiment shows that the proposed model caneasily adapt the learned object representation using few-shots from one dataset to another. We selectedthree state-of-the-art models and trained them using the same configuration as ORIF-TR. So that we canshow the strength of the proposed self-supervised learning technique and its efficiency in fast adaptationfrom one dataset to another compared to existing models. The three selected models are FgSegNetV2[63], CascadeCNN [123], and sEnDec [1]. All models are tested using 1-shot, 5-shots, and 10 trainingiterations for fine-tuning. For the three selected models, all parameters are fine-tuned. For our ORIF-TR,only the parameters of the decoder are updated.Table. 4.8 shows the results of our ORIF-TR against the other models. The table is split into twoparts. The first part is when models are trained using SegTrackV2 and fine-tuned using DAVIS, and thesecond part is when models are trained using DAVIS16 and fine-tuned using SegTrackV2. This experimenthighlights the effectiveness of using self-supervised learning and the simple decoder design. As shownin the table, the performance of ORIF-TR is higher than other models by a significant amount (around 6%and 10% in 1-shot). In the second part of the table, when the models are trained on SegTrackV2 andtested on DAVIS, we can see that the performance of the other models is inferior. This is because themodels try to transfer knowledge from a small dataset to a large one using only one labeled image. On theother hand, our ORIF-TR does not rely only on transferring the knowledge as the model already has vastknowledge from ImageNet in the pre-training phase. Our model outperforms other models by around 10%in this specific case. Some visual results are shown in Figure 4.6. If the value of k is higher than 5, theperformance will keep increasing until it reaches the same performance from Table. 4.6 and 4.7.4.2.3.3 Ablation studyWe tested the performance of different versions of the proposed ORIF-TR to show the importance of eachcomponent. We trained different versions of ORIF-TR in this experiment on the SegTrackV2 dataset using67Table 4.8: Results of ORIF-TR against the selected three models. k represents the number of trainingimages in each video. The values in the table are the F-Measure. Methods that have # at the endmean that the results are reproduced.Train - Test Method k=1 k=5CascadeCNN# 0.6275 0.7152DAVIS16 - SegTrack FgSegNetV2# 0.6517 0.7419sEnDec 0.6742 0.7828ORIF-TR 0.7290 0.8437CascadeCNN# 0.5616 0.6510SegTrack - DAVIS16 FgSegNetV2 0.6073 0.7095sEnDec 0.6143 0.7504ORIF-TR 0.7101 0.8189Figure 4.4: Visual results for both experiments.68Table 4.9: Different versions of the ORIF-TR results on the SegTrackV2 dataset using the standardforeground segmentation configurations.Components of ORIF-TR Evaluation MetricTransformation-Agnostic lossMaskedDecoder lossF-MeasureNo No 0.7716No Yes 0.8703Yes No 0.8221Yes Yes 0.9072the standard foreground segmentation configurations. Results of different versions of ORIF-TR are shownin Table. 4.9. The first row in the table shows the model’s results when normally trained like other fore-ground segmentation models (i.e., without self-supervised learning). The ORIF-TR performance is poorcompared to other models as the transformers’ encoder requires a considerable amount of data to trainaccurately. In the second row, self-supervised learning is applied only using the masked-decoder loss func-tion. The performance is significantly improved by approximately 10% because the transformer encoder istrained using a vast unlabeled dataset (ImageNet). On the other hand, when the model is trained using thetransformation-agnostic loss function only. The performance is improved by 4.5%. This is expected as ob-ject representation in foreground segmentation is more critical than invariance to different transformations.Finally, when both are combined, the performance is improved by approximately 13%.4.2.4 DiscussionA transformer-based network for foreground segmentation is proposed. This network is trained using anovel self-supervised learning technique that leverages unlabeled data to learn robust features and objectrepresentation. Hence, the network can be easily adapted to any dataset using a small number of labeledimages and a small number of update iterations. The conducted experiments show that ORIF-TR outper-forms existing foreground segmentation models in both standard foreground segmentation and few-shotforeground segmentation experiments. In the next section, we propose a novel technique based on meta-69learning to improve performanceThis meta-learning technique outperforms the proposed self-supervisedlearning technique.4.3 Meta-learning for few-shot foreground segmentationThe proposed meta-learning in [82] consists of two main parts a network architecture and a meta-trainingalgorithm. The network architecture uses an exemplar image (i.e., a frame with the corresponding groundtruth) as an input to the network along with the current frame. Hence, the network learns how to use theexemplar image to produce an attention map to segment the current frame accurately. On the other hand,the meta-training algorithm optimizes the network parameters to find a suitable set of parameters that canbe used as a starting point for any dataset with limited data. This differs from traditional training algorithms,where the network parameters are optimized to find a suitable set of parameters that can segment a specificdataset.An end-to-end network called Few-shot Network (FeSh-Net) is proposed to segment the foregroundobjects from unseen frame sequences (i.e., not seen during the training phase) using few-labeled images.FeSh-Net is trained using meta-learning, which means the FeSh-Net is trained to be task-agnostic (i.e.,learned to learn) This is why we use an extra input for FeSh-Net, which is the exemplar. FeSh-Net can usethe exemplar as a guide to segment foreground objects in the current frame. This means that FeSh-Netdoes not need to rely on remembering which frame sequence in the training set the current frame belongsto. This allows FeSh-Net to accurately segment the foreground objects from unseen frame sequences,unlike other foreground segmentation methods.4.3.1 FeSh-Net ArchitectureFeSh-Net consists of four major components as shown in Figure 4.5, 1) Frame encoder, 2) Exemplarencoder, 3) Exemplar-based Attention Map (EAM) extractor, and 4) Decoder.Frame encoder is used to extract feature maps from the current frame. The encoder consists of fourblocks with a max-pooling layer between every two blocks. The first block has two convolutional layers with32×3×3 filters in each layer. A ReLU layer and a batch normalization layer follow each convolutional layer.70The second and third blocks are the same as the first block, but each convolutional layer has 64 and 128filters, respectively. The last block has three convolutional layers. Each convolutional layer has 256 filters.The output of the encoder is feature maps Mx.Exemplar encoder is used to extract feature maps with the foreground object being highlighted fromthe exemplar. This exemplar is a frame Xe with the corresponding ground truth Ye. The exemplar encoderhas the same architecture as the frame encoder. The only difference is that between every two consecutiveblocks, the output feature maps are added to themselves multiplied by the ground truth. This processhighlights parts of the feature maps corresponding to the foreground objects. The output of the exemplarencoder is feature maps Me with the same dimensions as the feature maps extracted by the frame encoderMx.Exemplar-based attention map extractor is used to extract attention maps that highlight the fore-ground objects in the current frame. These attention maps are based on the highlighted feature maps fromthe exemplar encoder. Firstly, the output feature maps from both encoders (i.e., Mx and Me) are concate-nated to form Mc. Then, Mc is used as an input to a convolutional layer with one filter. The output of thiscomponent is the exemplar-based attention map (EAM), which approximates the final output of the cur-rent frame. The output EAM highlights the foreground objects in the feature maps extracted by the frameencoder. The highlighted feature maps Mh are used as input to the decoder.Mh =Mc+YEAM×Mc (4.22)Decoder is used to produce the final output is a binary image representing the foreground objects inwhite pixels and the background in black pixels. The architecture of the decoder reflects the architecture ofthe frame encoder with a slight difference. In each block, the input is a concatenation of the feature mapsfrom the previous block and the feature maps from the corresponding block in the frame encoder. This kindof skip connection from the encoder to the decoder is to combine the high-level features of the decoder withthe low-level features of the encoder. The final layer of the decoder is a convolutional layer with one filterand the sigmoid activation function to produce the final output Y ′.71Figure 4.5: The architecture of FeSh-Net.4.3.2 FeSh-Net TrainingThe proposed network is trained using meta-learning to be task-agnostic (i.e., not optimized to segmentforeground objects from a specific type of video). To use meta-learning, a meta-dataset must be created,and a meta-training algorithm is applied to train the network.4.3.2.1 Meta-datasetGiven a dataset of videos D, this dataset is used to generate z small datasets. The small datasets are calledepisodes (Ep). Each episode has n-tasks (different videos). Each task has 1-exemplar frame (Xe,Y e), k-training frames (X tr,Y tr), and m-testing frames (X te,Y te).Dmeta = {Ep1,Ep2, ...,Epz} (4.23)Epi = {T1,T2, ...,Tn} (4.24)Tj = {(Xej ,Y ej ),(X trj ,Y trj ),(X tej ,Y tej )} (4.25)where Tj is the jth task, X trj is input images, and Ytrj is the ground truth images of the training frames.724.3.2.2 Meta-trainingThe network is trained by iterating all generated episodes for C-epochs. Two steps define the processapplied in each episode: 1) fast adaptation and 2) updating network parameters.Fast adaptation: In this step, the network adapts the parameters of the EAM extractor and decodertoward each task in the episode. For each task, a clone of the EAM extractor and decoder parameters iscreated (φ ′). Then, the clone parameters φ ′ are updated using the training frames as follows:φ ′ = φ ′−α∇φ ′LTj(Y tr′j ,Ytrj ) (4.26)where α is the learning rate, LTj is the loss function for task Tj, Y tr′j is the output of the network using theinputs X tr, Xe, and Y e.The loss function L is computed as follows:LTj = `Tj(Y′j ,Yj)+β`Tj(YEAMj ,Yresizedj ) (4.27)where Y ′j is the output of the network, β is a constant value between [0,1], YEAMj is the output of the EAMextractor, and Y resizedj is the ground truth after resizing to match the size of YEAMj . Finally, `Tj is a binarycross-entropy loss function (as each pixel in the output is binary).After φ ′ is updated using the training frames, the network segment the foreground objects in the testingframes, and the task loss LTj is computed using the testing frames.Updating network parameters: In this step, the original network’s parameters are updated by mini-mizing the loss for all n-tasks in a single episode. All network parameters θ are updated (i.e., updating theencoder, EAM extractor, and decoder parameters). The parameters θ are updated as follows:θ = θ −λ∇θn∑j=1LTj(Yte′j ,Ytej ) (4.28)The same two steps are applied for each episode in Dmeta. The entire meta-training process is shown inAlgorithm. 4.73Algorithm 4 FeSh-Net meta-training.Require: Meta-dataset Dmeta, network f , all network parameters θ , EAM extractor and decoder parametersφ , number of epochs C, fast adaptation iterations U , learning rates α and λEnsure: θ the trained parameters1: for c in C do2: for Ep in Dmeta do3: for Tj in Ep do4: Xej ,Yej ,Xtrj ← GetTrainInputs(Ep,Tj)5: Y trj ← GetTrainGroundtruth(Ep,Tj)6: φ ′← φ7: %Fast adaptation8: for u in U do9: Y tr′j ← f (Xej ,Y ej ,X trj )10: φ ′← φ ′−α∇φ ′LTj(Y tr′j ,Ytrj )11: end for12: Xej ,Yej ,Xtej ← GetTestInputs(Ep,Tj)13: Y te′j ← f (Xej ,Y ej ,X tej )14: end for15: θ ← θ −λ∇θ ∑nj=1LTj(Y te′j ,Ytej )16: end for17: end for18: return θ4.3.2.3 FeSh-Net testingIn testing, the dataset of videos is treated as one big episode with n-tasks. Each task has m frames. Asubset from the m frames is selected as 1-exemplar frame and k-training frames. The network adaptsthe EAM-extractor and decoder parameters (φ ) using the k-training frames. Then, the network segmentsthe foreground objects in all m-frames, and evaluation metrics are computed to evaluate the network’sperformance.744.3.3 Experiments and resultsIn this section, we evaluate the proposed FeSh-Net using a foreground segmentation benchmark datasetcalled ChangeDetection.Net (CDNet) [122]. CDNet is used because FeSh-Net requires a large labeledforeground segmentation dataset during the training phase. FeSh-Net can learn from any dataset withlimited labeled data while maintaining high performance in the testing phase. To evaluate the model per-formance, two experiments are conducted. The first experiment is the traditional foreground segmentation,where the network is trained using some frames from each scene in the dataset. This experiment is to com-pare the performance of FeSh-Net against state-of-the-art foreground segmentation methods. The secondexperiment tests the generalization and capability of the proposed FeSh-Net to learn new scenes using afew labeled frames (1,5,10, and 15 frames) from different datasets, namely DAVIS16 and SegTrackV2.4.3.3.1 Experiment 1In this experiment, all models are trained using 200 frames from each video in CDNet. The rest of theframes are used for testing. The evaluation metrics used are false-positive rate (FPR), false-negative rate(FNR), and F-Measure (F-M).Table.4.10 shows the results of the proposed FeSh-Net against state-of-the-art foreground segmenta-tion methods using CDNet dataset. The number after FeSh-Net represents the number of training framesused during testing (i.e., online fine-tuning). If the number after FeSh-Net is 0, then we do not update theparameters φ . We use only one labeled frame as an exemplar. The table shows that FeSh-Net, withoutextra fine-tuning, outperforms other techniques by 0.9% in the F-M. This shows the importance of usingthe attention mas created based on the exemplar frame. Additionally, fine-tuning using few-labeled framesimproves the performance even more. Using 15 training frames for each video improved the performanceof FeSh-Net by 5% in the F-M.4.3.3.2 Experiment 2In this experiment, we test the proposed FeSh-Net generalization and the learning capability of new scenesusing a few labeled frames. We trained FeSh-Net on group videos and tested on another group of un-seen videos (videos not seen during training). The results are compared with three methods to show the75Table 4.10: Results of FeSh-Net against state-of-the-art foreground segmentation methods on CDNet.The last four rows are the results of FeSh-Net with extra online fine-tuning using few-labeledframes. Methods with * are our previous work. Methods that have # at the end mean that theresults are reproduced.Method FPR FNR F-MsEnDec# [1] 0.004 0.076 0.842FgSegNetV2# [63] 0.005 0.080 0.801CascadeCNN# [123] 0.007 0.097 0.786IUTIS-5 [11] 0.005 0.215 0.771SemanticBGS [12] 0.004 0.211 0.789BSUV-Net [114] 0.005 0.179 0.786BSUV-Net-SBGS [114] 0.005 0.182 0.798DeepBS [7] 0.009 0.245 0.745SuBSENSE [106] 0.009 0.187 0.741WisenetMD [52] 0.009 0.182 0.753PAWCS [107] 0.005 0.228 0.740MODSiam* [85] 0.010 0.181 0.714MODY-Net* [86] 0.005 0.124 0.865REFNet* [81] 0.004 0.038 0.881FeSh-Net0 (Ours) 0.006 0.030 0.851FeSh-Net1 0.005 0.054 0.874FeSh-Net5 0.005 0.062 0.883FeSh-Net10 0.002 0.066 0.897FeSh-Net15 0.003 0.070 0.90176Table 4.11: Results of FeSh-Net against three foreground segmentation methods on CDNet. k repre-sents the number of training frames in each video. The values in the table are the F-Measure.Methods that have # at the end mean that the results are reproduced.Method k=0 k=1 k=5 k=10 k=15CascadeCNN# [123] 0.490 0.635 0.672 0.700 0.762FgSegNetV2# [63] 0.578 0.687 0.753 0.802 0.824sEnDec# [1] 0.647 0.703 0.782 0.828 0.851REFNet* [81] 0.675 0.714 0.797 0.839 0.866FeSh-Net 0.725 0.780 0.816 0.856 0.889effectiveness of FeSh-Net and the proposed meta-training algorithm. The other three methods are Cas-cadeCNN [123], FgSegNet [63], and sEnDec [1]. During this experiment, the parameters of these modelsare online fine-tuned using k-frames for each video. On the other hand, The parameters φ of FeSh-Net areonline fine-tuned using the fast adaptation step in the proposed meta-training algorithm.The results of CDNet dataset are shown in Table. 4.11. In this experiment, all methods are trainedusing 32 videos and tested using 21 videos. The reported F-Measures are that of the testing videos. Inthis experiment, FeSh-Net outperforms other models by a huge gap in the F-Measure. This is becausethe other models are designed to be task-specific. In the Table, when k = 0, FeSh-Net outperforms thesecond-best model by 7.8%. When online fine-tuning is applied, FeSh-Net outperforms other methods byan average of 4.4%. Figure 4.6 shows the visual results of this experiment. To further evaluate the modelgeneralization, Fesh-Net is evaluated using new scenes that are out-of-domain (i.e. scenes belonging to adifferent dataset) with a few labeled frames. The new scenes are sampled from DAVIS16 and SegTrackV2.The results are compared with the previous few-shot learning technique ORIF. As shown in Table. 4.12,FeSh-Net outperforms ORIF-TR by a huge gap when k= 1 due to usage of EAM module. Even at a highervalue for k, FeSh-Net outperforms ORIF-TR because the meta-training algorithm is more effective thanself-supervised learning when massive labeled data exists.77Table 4.12: Results of FeSh-Net against ORIF-TR. k represents the number of training images in eachvideo. The values in the table are the F-Measure.Train - Test Method k=1 k=5ORIF-TR 0.748 0.851CDNet - SegTrack FeSh-Net 0.819 0.873ORIF-TR 0.729 0.815CDNet - DAVIS FeSh-Net 0.801 0.838Figure 4.6: Sample results from FeSh-Net against three other models using 0 and 15 training frames.78Table 4.13: Results of four versions of FeSh-Net on CDNet using one training frame.ComponentsF-MeasureEAM moduleUpdatingdecoderUpdatingEAM moduleNo No No 0.663Yes No No 0.725Yes Yes No 0.769Yes Yes Yes 0.7804.3.3.3 Ablation studyIn this study, we show the importance of each component in our proposed FeSh-Net. We used four versionsof FeSh-Net: 1) FeSh-Net without exemplar encoder and EAM extractor, 2) FeSh-Net without updatingparameters using the exemplar image, 3) FeSh-Net with updating decoder parameters only, and 4) Fullversion of FeSh-Net. Table. 4.13 shows the results of the four different versions of FeSh-Net when evaluatedon CDNet dataset with one training image. As shown in the table, the base network (traditional encoder-decoder network) has a performance of 0.663. When the exemplar encoder and EAM extractor are addedto the network, the performance increases by almost 6%. Moreover, fine-tuning the decoder parametersto adapt to new scenes improved the performance by approximately 4.5%. Finally, updating the exemplarencoder and EAM extractor parameters improved the performance slightly by 1.1%.4.4 ConclusionThis chapter presented two few-shot learning techniques for foreground segmentation, a self-supervisedlearning technique called ORIF-TR, and a meta-learning technique called FeSh-Net. The experimentsshowed that training the meta-learning technique is more effective than the self-supervised learning tech-nique in segmenting the foreground objects using a few training data. However, in the initial training ofFeSh-Net, a large labeled foreground segmentation dataset is required to perform the meta-training algo-79rithm. While in ORIF-TR the initial training uses any large dataset that is unlabeled (i.e., the dataset doesnot have to be a foreground segmentation dataset. It can be any dataset of images).Both techniques solve the problem of learning with limited data but they can not learn continuouslywithout forgetting. In each episode (i.e., a dataset with limited labeled data), the initial model is fine-tuned using the labeled data. This means each episode is a completely isolated task (i.e., episodes arenot accumulated). Therefore, if the model revisited a previous episode, the model will have to be fine-tuned again. In other words, the model does not store the knowledge gained from each episode in thetesting phase. To solve this problem, lifelong learning techniques are presented in the next chapter. Thesetechniques allow the model to incrementally learn new episodes without forgetting the previous ones.80Chapter 5Lifelong Learning models withoutcatastrophic forgettingIn this chapter, we propose three lifelong learning techniques to overcome the problem of catastrophicforgetting. A novel deep-learning architectural design is proposed that consists of a server and multipleclient networks. Each client learns from local data. These local data are unbalanced, non-identically,and independently distributed (non-IID) across the clients. The knowledge gained by each client networkfrom its local data is stored in the server network. The idea behind this architecture is that each client iscompletely isolated from others. Hence, for a client to learn a new task will not affect the tasks learned byother clients. While the goal of the server is to combine the knowledge gained by all clients efficiently (i.e.,with minimal loss of knowledge). This is different from the traditional lifelong learning techniques where asingle network is used to train from multiple datasets sequentially. These techniques lead to either moreloss of knowledge gained or less capability to learn new tasks. In this chapter, the term task is defined asa group of videos sampled from a dataset.5.1 Lifelong learning using Task-Based Parameter Isolation (TBPI)Task-Based Parameter Isolation (TBPI) proposed in [81] uses a parameter isolation strategy that stores theknowledge gained by each task in a different set of parameters in the server network. The deep learningnetwork used for the client and server is REFNet, introduced in Chapter 3.815.1.1 Training REFNet using TBPIPre-training phase: First, REFNet undergoes a pre-training phase where it is trained normally on all thevideos in a given dataset D. The resulting network is called Pre-TBPI REFNet. Also, in the pre-trainingphase, the original dataset of videos D is divided into n tasks. A randomly equal number of videos isassigned to each task. Finally, in the pre-training phase, one clone of the pre-TBPI REFNet is created foreach task. Figure 5.1-a shows an example of the pre-training phase. The dataset is divided into 4 tasks,each containing the same number of videos. In this example, 4 videos. There are 4 clones of pre-TBPIREFNet to match the number of tasks.Training phase: The first step is to categorize the parameters (θ (l)) in each layer l in each pre-TBPIREFNet clone into updatable (θ (l)u ) and fixed (θ(l)f ) parameters. Let the number of parameters in a givenlayer l is k(l), hence,θ (l) = θ (l)u ∪θ (l)f = {θ (l)1 ,θ (l)2 , ...,θ (l)k } (5.1)The number of updatable parameters in this layer is s(l) and is computed as:s(l) =k(l)n(5.2)Then, the set of updatable parameters θ (l)u and fixed parameters θ(l)f are:θ (l)u = {θ (l)1 , ...,θ (l)s(l)} (5.3)θ (l)f = {θ (l)s(l)+1, ...,θ(l)k(l)} (5.4)Meaning that the updatable parameters are selected as the first s(l) parameters in θ (l) and the rest ofparameters in this layer, {θ (l)s(l)+1, ...θ (l)k(l)}, are fixed.The second step in this phase is to fine-tune the updatable parameters in each clone network usingthe training frames x(i)train of the corresponding task. As an example, Figure 5.1-b has 4 tasks. The first 1/4of the parameters in each clone network will be updated during the training phase, while the rest are keptunchanged (fixed). It is worth mentioning that the 4 clone networks are independent of each other and donot share parameters (i.e., isolated).82Figure 5.1: The process of TBPI. (a) the pre-TBPI training phase, and (b) the TBPI training phase.Figure 5.2: An example of applying TBPI on REFNet of a single convolutional layer, each circle is afilter with 3×3 parameters.83Figure 5.3: An example of creating a task-specific REFNet of a single convolutional layer for task t(2).The last step in the TBPI training after all clone networks have fine-tuned their updatable parameters isto concatenate them into a single network (trained-TBPI REFNet) that holds all updatable parameters forall the tasks.Since the number of updatable parameters in each clone equals the total number of parameters pre-TBPI REFNet divided by the number of tasks, after concatenating all updatable parameters together, thetotal number of parameters in the resulting trained-TBPI REFNet is the same as the number of parametersin the original REFNet. Only the pre-TBPI REFNet and trained-TBPI REFNet are kept to be used in thetesting phase. To illustrate this part, Figure 5.1-b shows how the updatable parameters are concatenatedto form the trained-TBPI REFNet. The combination of pre-TBPI REFNet and trained-TBPI REFNet is thefull REFNet-TBPI.To further illustrate the learning process in TBPI, assume that REFNET has only one convolutionallayer. In Figure 5.2, an example is shown of applying the TBPI training and fine-tuning on the 4 clones. Inthis example, assume the convolutional layer has 20 filters, each with 3× 3 parameters. This means thatthe layer has 180 parameters. Assume we have 4 learning tasks D = {t(1), t(2), t(3), t(4)}. In this example,the number of updatable parameters in the layer is 180/4 = 45 parameters (The colored circles), and theother 135 parameters are fixed (i.e., not updatable). The updatable parameters in each clone network arethe first 45 parameters, and these parameters are Task-specific. Each clone network will be fine-tuned by84training on the corresponding task. After all clone networks are fine-tuned, all updatable parameters in eachclone network are concatenated to form a single network that holds all updatable parameters for all tasks.By concatenating 45 updatable parameters from each clone network, a single network with 180 parametersis constructed and trained-TBPI-network.Pre-training and TBPI training use the same optimizer, learning rate, number of epochs, and loss func-tion. The differences between the two phases are the input and the parameters to be updated. In thepre-training, the input is from the entire dataset D, and all the network parameters are updated to result inthe pre-TBPI network (Figure 5.1-a). In the TBPI training, the input to each clone network is from the videosassigned to the corresponding task t(i), and only a portion of the clone network parameters are updated.During both the pre-training and the TBPI-training phases, the loss function is calculated as a weightedsum of all outputs losses, defined as:L= `re f +3∑j=1α j ∗ ` j (5.5)where `re f is the loss of the network output, ` j is the loss of output at decoder block number j, and α is theweight of each output loss. The suitable values of α are found by applying a grid search on a small portionof the dataset. There are four outputs for the network, an output from each decoder block and the output ofthe REF (i.e., the final output).Hyperparameters selection: The loss of each output ` is a binary cross-entropy. The learning rate is1e−4, the number of epochs is 50, and the optimizer used is Root Mean Square Propagation (RMSprop).The values of these hyperparameters are taken from the state-of-the-art model [63].5.1.2 Testing of REFNet-TBPIThe testing phase is when the proposed REFNet-TBPI is evaluated by extracting the objects from unseenframes. The testing phase is divided into two steps, 1) creating a task-specific REFNet for each task. 2)Segment foreground objects from each task’s test frames using the corresponding task-specific REFNet. Itis worth noting that the number of tasks in testing is the same as in training.Creating task-specific REFNet-TBPI: The task-specific REFNet-TBPI is a clone of pre-TBPI REFNetwith the first portion of parameters in each layer replaced by the task-specific parameters from the trained-TBPI REFNet corresponding to the current task.85Table 5.1: Characteristics of the proposed continual learning technique compared to the state-of-the-arttechniques.MethodAvoidForgettingFixedMemoryEnable ForwardTransferEnable BackwardTransferNo storedExamplesLwF [60] Yes No Yes No YesEWC [44] Yes No Yes No YesGEM [68] Yes No Yes Yes NoPackNet [71] Yes Yes Yes No YesTBPI (ours) Yes Yes Yes No YesWe continue using the same example from the previous subsection 5.1.1 that used a network with oneconvolutional layer with 20 filters, each filter has 3× 3 = 9 parameters, and the number of tasks is 4. Tocreate a task-specific network for testing in task t(2) as shown in Figure 5.3, we replace the first 45 parame-ters in the convolutional layer with the t(2) 45 parameters from the trained-TBPI Network. This results in thetask-specific REFNet-TBPI, then, the task t(2) test frames x(2)test are fed to it to segment foreground objects.Segmenting foreground objects: The created task-specific REFNet for t(i) is used to segment theforeground objects from each frame in x(i)test . The frames in x(i)test are unseen frames in task t(i) during trainingas mentioned before x(i)train ∩ x(i)test = φ . After segmenting the foreground objects from all frames in a giventask, the evaluation metrics are calculated to show the performance of REFNet-TBPI on this task. Finally,the reported performance is the average performance of REFNet-TBPI on all tasks in a given dataset.5.1.3 Characteristics of TBPIIt is important to mention the features of the TBPI technique compared to other continual learning tech-niques, 1) it prevents the occurrence of catastrophic forgetting, 2) network size is fixed despite the numberof tasks, 3) the order of tasks in training does not matter (i.e., training on a task does not negatively affectthe training on previous or next tasks), and 4) there is no need to store examples to remind the networkof previous tasks while training on the current task. The only limitation is that the network does not en-hance the performance of previous tasks while training on the current task, which is known as ”Enable86Backward Transfer” in continual learning. Table 5.1 summarizes the features of the proposed TBPI versusother continual learning techniques.5.1.4 ExperimentsThree benchmark datasets are used to evaluate the proposed REFNet-TBPI method: CDNet, DAVIS16,and SegTrackV2. Two experiments are conducted. The first one is the same as the experiment used inChapter 3 Section 5.1.4. This experiment shows the difference between training with and without TBPI.The second experiment shows the difference between training REFNet with different learning techniques,traditional learning (also known as joint training where the network is trained using all three datasets asone big dataset), fine-tuning (where the network is trained using three datasets sequentially, each datasetfine-tune the network parameters), and the proposed TBPI.5.1.4.1 Experiment 1: segmentation of foreground objectsAll methods, including ours, are first trained on all video sequences at once, and then each trained methodis then tested to detect the objects in each video sequence separately. For REFNet trained using TBPI, thenumber of video sequences in the dataset is divided into 4 tasks equally.The results of REFNet-TBPI on the CDNet dataset are compared with state-of-the-art models in seg-menting foreground objects. The training set for CDNet is 200 frames from each video and tested on theframes specified for testing for each video sequence. The evaluation metrics used are false-positive rate(FPR), false-negative rate (FNR), recall (Re), precision (Pr), and F-measure. As shown in Table 5.2, TheF-measure of state-of-the-art methods ranges from 74.0% to 84.2%, while REFNet outperforms the bestmethod by 3.9% in F-measure. Adding TBPI to REFNet has improved the performance concerning allevaluation metrics. The F-measure has increased by 2%, which is 5.9% more than the best method in thestate-of-the-art, which is sEnDec [1].In DAVIS16 and SegTrackV2 datasets, the authors report only the F-Measure as the evaluation metricas in [54, 48, 137, 72]. For the DAVIS16 dataset, the model is trained on 25 frames from each video. If avideo has a number of frames less than 25, half of the frames are used for training. The testing is doneon the rest of the frames. The two datasets’ results are shown in Table 5.3, achieving 0.927 f-measure in87Table 5.2: Results of REFNet and TBPI compared to state-of-the-art methods on CDNet, methods with* means our work. Methods with * are our previous work. Methods that have # at the end meanthat the results are reproduced.Method FPR FNR Re Pr F-MeasuresEnDec# [1] 0.004 0.076 0.903 0.798 0.842 3©FgSegNetV2# [63] 0.005 0.080 0.893 0.741 0.801CascadeCNN# [123] 0.007 0.097 0.821 0.771 0.786IUTIS-5 [11] 0.005 0.215 0.789 0.808 0.771SemanticBGS [12] 0.004 0.211 0.789 0.830 0.789BSUV-Net [114] 0.005 0.179 0.820 0.811 0.786BSUV-Net-SBGS [114] 0.005 0.182 0.817 0.831 0.798DeepBS [7] 0.009 0.245 0.754 0.833 0.745SuBSENSE [106] 0.009 0.187 0.812 0.751 0.741WisenetMD [52] 0.009 0.182 0.817 0.766 0.753PAWCS [107] 0.005 0.228 0.771 0.785 0.740MODSiam* [85] 0.010 0.181 0.823 0.692 0.714MODY-Net* [86] 0.005 0.124 0.903 0.822 0.865REFNet* [81] 0.004 0.038 0.961 0.827 0.881REFNet-TBPI (ours) 0.003 0.030 0.969 0.850 0.90188Table 5.3: Results of REFNet and TBPI compared to state-of-the-art methods on DAVIS16. (PT)means require pre-training, and (OF) means apply online fine-tuning. Methods that have # atthe end mean that the results are reproduced.Model PT OF F-MeasureCNIM [8] Yes Yes 0.850OSVOS [72] Yes Yes 0.873LUCID [43] Yes No 0.820RANet [124] No No 0.876UOVOS [137] No No 0.772sEnDec# [1] No No 0.856FgSegNetV2# [63] No No 0.847CascadeCNN# [123] No No 0.814ORIF-TR* [84] Yes No 0.882REFNet* [81] No No 0.874REFNet-TBPI (ours) Yes No 0.927DAVIS. The results are compared to state-of-the-art models, and the top three models in CDNet dataset,namely sEnDec [1], FgSegNetV2 [63], and CascadeCNN [123]. The results of the top three models arereproduced by training them using the same strategy as REFNet. As shown in Table 5.3, the proposedREFNet-TBPI outperforms the second-best model RANet [124] by 5.1% in F-measure. In Table 5.3, PTrefers to the model using extra training as a pre-training phase, and OF refers to the model applying onlinefine-tuning (i.e., the model updates its parameters during the testing phase).For the SegTrackV2 dataset, the models are trained using the same strategy as the DAVIS16 dataset.As shown in Table 5.4, REFNet-TBPI outperforms the second-best model STRCF [48] by 2.2% in the F-Measure. The overall improvement of the proposed REFNet-TBPI is 4.4% on state-of-the-art foregroundsegmentation, which is the average improvement on all three datasets.A sample of REFNet-TBPI results on CDNet against five other models and the associated ground truthis shown in Figure 5.4. Also, sample results from videos in the DAVIS16 dataset are shown in Figure 5.5,89Figure 5.4: Sample results from videos in CDNet dataset, the first row is the RGB frame, the last rowis the ground truth, and the rest are the result of different methods.Figure 5.5: Sample results from videos in DAVIS16 dataset.90Table 5.4: Results of REFNet and TBPI compared to state-of-the-art methods on SegTrackV2. Meth-ods with * are our previous work. Methods that have # at the end mean that the results are repro-duced.Model F-MeasureDCL [54] 0.730UOVOS [137] 0.643STRCF [48] 0.899sEnDec# [1] 0.848FgSegNetV2# [63] 0.780CascadeCNN# [123] 0.767ORIF-TR* [84] 0.907REFNet* [81] 0.904REFNet-TBPI (ours) 0.921Figure 5.6: Sample results from videos in SegTrackV2 dataset.91Figure 5.7: The effect of different threshold values on the performance of REFNet and REFNet-TBPIon CDNet dataset.92and the SegTrackV2 dataset in Figure 5.6, showing a comparison between the output of REFNet-TBPIand FgSegNetV2. Finally, Figure 5.7 shows the effect of different threshold values on the performanceof REFNet and REFNet-TBPI on the CDNet dataset. The proposed model is robust against a variety ofthreshold values. The F-measure ranges from 0.72 to 0.901 on threshold values from 0.1 to 0.9.5.1.4.2 Experiment 2: continual foreground segmentationThe three datasets in this experiment are considered continual tasks for segmenting the foreground ob-jects. Each one of these datasets is naturally different from the other with respect to the objective andenvironment. For example, in CDNet each video sequence has multiple objects that go in and out of thescene while the scene itself is fixed. In DAVIS16, the scene is not fixed, but only one object of interest isneeded to be extracted. For SegTrackV2, the scene is almost fixed, with multiple objects of interest, but theobjects never enter or leave the scene. Generally, each dataset requires a different feature space to detectthe target output correctly. With joint learning or fine-tuning, it is impossible to have a single network thatworks perfectly on all three datasets simultaneously.In Table 5.5, a comparison between different learning methods is conducted to show the limitationsof each learning method and how the proposed continual learning method overcomes these limitations.Traditional learning means training the proposed REFNet on all three datasets as a single dataset. Thenetwork’s performance is poor on all three datasets and biased to CDNet as it is the largest dataset in allthree datasets and has the lowest performance on SegTrackV2 because it is the smallest dataset. Thisshows that there are two problems in joint learning. The first is poor performance on all datasets becausethe network can’t learn a feature space that solves all three tasks. The second problem is the network’sbias towards the largest dataset due to the imbalance in the size of the datasets.Another learning method is fine-tuning. Starting with the network trained using traditional learning asa pre-trained network, the network is fine-tuned on each dataset separately. After fine-tuning on CDNet,it is noted that this dataset’s performance is improved dramatically. However, a problem appears whenfine-tuning the second dataset, which is DAVIS16, after the first fine-tuning on CDNet. The problem is thatthe network’s performance on CDNet is degraded when fine-tuned on DAVIS. As shown in Table 5.5, theperformance degraded by 45% in the F-measure. This problem is called catastrophic forgetting, where the93Table 5.5: Comparison between different learning methods on CDNet, DAVIS16, and SegTrackV2 ascontinual tasks using the proposed REFNet. The down arrow ↓ means the performance degraded.TrainingMethodTrainingdatasetTestingdatasetFPR FNR Re Pr F-MeasureJointLearningCDNet, DAVISand SegTrackV2CDNet 0.011 0.066 0.933 0.642 0.737DAVIS16 0.030 0.316 0.683 0.600 0.630SegTrackV2 0.026 0.488 0.511 0.527 0.505Fine-tuningCDNet CDNet 0.004 0.08 0.911 0.791 0.831DAVISCDNet 0.008 ↓ 0.687 ↓ 0.312 ↓ 0.646 ↓ 0.381 ↓DAVIS 0.010 0.104 0.895 0.839 0.862SegTrackV2CDNet 0.02 ↓ 0.449 ↓ 0.550 ↓ 0.447 ↓ 0.422 ↓DAVIS16 0.05 ↓ 0.342 ↓ 0.657 ↓ 0.497 ↓ 0.529 ↓SegTrackV2 0.005 0.069 0.930 0.822 0.917Task-BasedParameterIsolationCDNet CDNet 0.004 0.176 0.823 0.804 0.805DAVISCDNet 0.004 0.176 0.823 0.804 0.805DAVIS 0.013 0.230 0.769 0.781 0.768SegTrackV2CDNet 0.004 0.176 0.823 0.804 0.805DAVIS16 0.013 0.230 0.769 0.781 0.768SegTrackV2 0.009 0.125 0.874 0.848 0.860network accidentally forgets important features learned from the previous dataset. After fine-tuning the thirddataset, the network has good performance on the current dataset only, while the performance degradedtremendously on all previous datasets.The proposed task-based parameter isolation solves the problem of poor performance and unbalancingin joint learning and the problem of catastrophic forgetting in fine-tuning. In this learning method, thenetwork starts with the same weights as in fine-tuning method. Then, 33% of the network parameters areassigned for each dataset. After training on all three datasets, the network can perform well on all threedatasets without any degradation in the performance. In fine-tuning, the performance is better than TBPI ina specific task, and this is expected, as in fine-tuning, the whole network parameters that are updated onthe current task versus the 33% of network parameters are updated in TBPI on the current task. Figure 5.8shows a visualization of results in Table 5.5 for fine-tuning and the proposed TBPI.94Figure 5.8: Fine-tuning versus the proposed TBPI in continual tasks. In fine-tuning, a huge degrada-tion in F-measure due to catastrophic forgetting. On the other hand, the proposed TBPI has 0degradations in performance.5.1.4.3 Effectiveness of TBPITo prove the effectiveness of the proposed TBPI and that it can be applied to any CNN-based model andenhance its performance, we applied TBPI on the top three models in performance on the CDNet dataset,which are sEnDec[1], FgSegNetV2[63], and CascadeCNN[123]. As shown in Table 5.6, 5.7, and 5.8, whenTBPI is applied to other models, it improves the results with respect to all evaluation metrics. This shows thatthe proposed TBPI improves the result of any CNN-based model. However, The proposed REFNet-TBPIoutperforms other models even after applying TBPI. This is proof that the proposed REFNet architecture isbetter than other models.5.1.5 DiscussionThe proposed REFNet-TBPI is the first network to handle the catastrophic forgetting problem in foregroundsegmentation via the proposed lifelong learning. The results show the superiority of training a deep-learningmodel with TBPI as it amplifies the network performance by reducing catastrophic forgetting and makes thenetwork output slightly more robust, as shown in Fig 5.7.The limitation of TBPI is that the number of tasks is limited as the parameters of the network are splitamong tasks. The number of tasks is limited to the number of parameters in the smallest layer of thenetwork.95Table 5.6: Comparison of TBPI against state-of-the-art methods including REFNet using CDNetdataset. Methods that have # at the end mean that the results are reproduced.Dataset Model FPR FNR Re Pr F-MsEnDec# [1] 0.004 0.076 0.903 0.798 0.842sEnDec-TBPI# 0.004 0.031 0.918 0.811 0.855FgSegNetV2# [63] 0.005 0.080 0.893 0.741 0.801CDNet FgSegNetV2-TBPI# 0.004 0.033 0.924 0.784 0.839CascadeCNN# [123] 0.007 0.097 0.821 0.771 0.786CascadeCNN-TBPI# 0.005 0.051 0.882 0.793 0.812REFNet (ours) 0.004 0.038 0.961 0.827 0.881REFNet-TBPI (ours) 0.003 0.030 0.969 0.850 0.901Table 5.7: Comparison of TBPI against state-of-the-art methods including REFNet using DAVIS16dataset. Methods that have # at the end mean that the results are reproduced.Dataset Model FPR FNR Re Pr F-MsEnDec# [1] 0.011 0.084 0.916 0.820 0.856sEnDec-TBPI# 0.006 0.068 0.931 0.902 0.917FgSegNetV2# [63] 0.019 0.075 0.924 0.797 0.847DAVIS16 FgSegNetV2-TBPI# 0.010 0.064 0.937 0.850 0.887CascadeCNN# [123] 0.028 0.124 0.856 0.780 0.814CascadeCNN-TBPI# 0.010 0.110 0.889 0.849 0.867REFNet (ours) 0.008 0.100 0.899 0.854 0.874REFNet-TBPI (ours) 0.004 0.074 0.926 0.931 0.92796Table 5.8: Comparison of TBPI against state-of-the-art methods including REFNet using SegTrackV2dataset. Methods that have # at the end mean that the results are reproduced.Dataset Model FPR FNR Re Pr F-MsEnDec# [1] 0.006 0.160 0.829 0.871 0.848sEnDec-TBPI# 0.005 0.140 0.879 0.903 0.889FgSegNetV2# [63] 0.009 0.223 0.776 0.797 0.780SegTrackV2 FgSegNetV2-TBPI# 0.007 0.171 0.816 0.869 0.840CascadeCNN# [123] 0.010 0.248 0.751 0.788 0.767CascadeCNN-TBPI# 0.009 0.188 0.807 0.821 0.812REFNet (ours) 0.004 0.107 0.892 0.919 0.904REFNet-TBPI (ours) 0.004 0.086 0.913 0.932 0.9215.2 Lifelong learning using knowledge distillation (KDNet)To overcome the limitation of the number of tasks, a novel lifelong learning technique and a multi-headnetwork are proposed. In this lifelong learning technique, a regularization-based technique is proposed tooptimize the server parameters to learn new tasks while maintaining the same performance as the earliertasks. The main idea is to instantiate multiple client networks from the server network and train eachclient on a different dataset. Then, the server gathers all client networks and applies knowledge distillationto learn the knowledge gained by each client in an unsupervised fashion. The proposed model (server,clients, and lifelong learning technique)is referred to as Knowledge Distillation Network (KDNet).5.2.1 ArchitectureThe network used for both the server and clients is the same. This network is an encoder-decoder archi-tecture. The encoder is ResNet50, and the decoder is four blocks. Each decoder block consists of threeconvolutional layers followed by batch normalization and up-sampling layers. After the four blocks, a con-volutional layer is added with a sigmoid activation function to produce the output mask. The architectureof the server is slightly different from the clients as it has multiple heads (multiple output layers), and each97Figure 5.9: The architecture of KDNet. On the left is the server network with multi-head outputs, andon the right are the client networks.98Figure 5.10: Training procedure of KDNet.head is responsible for learning from a single client. Figure 5.9 shows the server and client architecture.5.2.2 Training and testingAs shown in Figure 5.10, the training procedure is a repeated cycle of four main steps, 1) instantiating theclients, 2) training the clients, 3) gathering the clients, and 4) training the server.Instantiating the client In this step, a client is instantiated from the server for each dataset (i.e., ifwe have three training datasets, there will be three clients instantiated from the server). The instantiationprocess copies the server network weights to the client network. It is important to mention that only theencoder and decoder parameters are copied to the client (i.e., the heads’ parameters are not copied).Training the clients, the clients’ training is straightforward. Each client is trained on the dataset as-signed to it (local dataset) using supervised learning. The goal is that each client successfully learns tosegment the foreground objects in the local dataset.Gathering the clients, the server gathers only the trained client networks (i.e., the local dataset ofeach client is not included)Training the server, this is the biggest step where the server network learns the knowledge gained by99all client networks. The server uses a large unlabeled dataset. Then, for each batch in this dataset, onlyone client will process this batch to produce the output mask. The clients are selected in a round-robin order(i.e., client1, client2, ..., clientM, client1, ..., etc.). After that, the server applies the knowledge distillation lossfunction to learn to produce an output similar to the client’s output. However, the server should learn fromeach client without affecting the knowledge gained by other clients. Hence, each client updates only onehead of the server. Also, a term in the loss function is added to regularize the update of shared parametersbetween clients (i.e., the encoder and decoder parameters). This term minimizes the distance between theoutput of other clients after the update and before the current client’s update. The loss function used isdefined as follows:`i = l(yi,yin)+1MM∑j 6=il(y jo,yjn) (5.6)the `i is the loss function for updating the server with the ith client. the first loss term is reducing thedistance between the server’s output of ith after the update yin and the output of the ith client yi. The secondterm is reducing the distance between the output of all other heads of the server before y jo and after yjn theupdate and M represents the number of clients. The second term acts as a regularizer to force the networkto update the parameters in a way that does not cause a huge change in the output of other heads. Asshown in the loss function, the ground truth (label) of the batches in the dataset is not used. This is whythe server is trained by the clients using unsupervised learning. A step-by-step illustration of the trainingprocedure is shown in Algorithm 5.100Algorithm 5 KDNet training procedure.Require: DL the local datasets, KDNet the server network, DU the unlabeled dataset, epochs number oftraining iterationsEnsure: KDNet the trained server1: for e in epochs do2: clients← instantiate Clients(KDNet)3: for client in clients do4: D← get Local Dataset(DL)5: Client← fit model(Client, D)6: end for7: clients← gather Clients()8: i← 09: for batch in DU do10: x← get images(batch)11: yo← predict(KDNet,x)12: client← clients[i]13: y← predict(client,x)14: KDNet← update using input(KDNet,x,y)15: yn← predict(KDNet,x)16: `← loss(y,yn[i])+ 1MM∑j 6=iloss(yo[ j],yn[ j])17: KDNet← update using loss(KDNet,`)18: i← i + 119: end for20: end for21: Return KDNetIn the testing phase, the dataset number must be known so that the evaluation metrics will be calculatedfrom only the output of the corresponding head in the server network.5.2.3 ExperimentsTo evaluate KDNet, two experiments are conducted. The first experiment is training the model using individ-ual datasets. This experiment compares the results of KDNet with other foreground segmentation models.In this experiment, KDNet uses split a single dataset into sub-datasets. Then, a client is instantiated to learn101from each sub-dataset. After that, the multi-heads of the server learn from all clients. During testing, eachhead of the server is used to segment foreground objects from the corresponding videos (i.e., each groupof videos in the dataset is segmented by a specific head of the server). On the other hand, the secondexperiment is to test the model generalization by training using multiple datasets (multiple domains). In thisexperiment, KDNet is trained the same way as in the previous experiment, but each client learns from adifferent dataset instead of a sub-dataset.5.2.3.1 Traditional foreground segmentation experimentIn this experiment, KDNet is evaluated using two datasets, DAVIS16 and SegTrackV2. The model is trainedand tested using each dataset individually. For this experiment, we show the result of KDNet multi-headwith federated learning, referred to as KDNet(Server-Client). Also, the result of another version of KDNetmulti-head without federated learning is shown, referred to as KDNet(Server). The number of heads forboth versions is 5. Hence, the videos in each dataset are split into 5 sub-datasets.Table 5.10 shows the results of KDNet against other foreground segmentation models using the Seg-TrackV2 dataset. The default evaluation metric for this dataset is the f-measure only. As shown in Ta-ble 5.10, the performance of both versions of KDNet is in the top places. KDNet outperforms the second-best model (REFNet) in literature by 1.4%. While the second version of KDNet outperforms REFNet by0.5%. This is due to the fact that catastrophic forgetting can occur even within a single dataset becauseeach video requires a different set of parameters to segment it. Although the chance of catastrophic forget-ting is small within a single dataset, it slightly degrades performance.For the DAVIS16 dataset, Table 5.9 shows the results of KDNet against state-of-the-art foregroundsegmentation models. The reported metrics are the f-measure (F ), the Jaccard index (J ), and theharmonic mean of the f-measure and Jaccard index (F&J ) as these are the default evaluation metricused for this dataset. The results show that KDNet is in second place by a small gap of 0.05% in averagef-measure and Jaccard index less than the top performing model.102Table 5.9: Traditional foreground segmentation results of KDNet compared with state-of-the-art mod-els on the DAVIS16 dataset. Methods with * are ours. Methods that have # at the end mean thatthe results are reproduced.Model F J F&JCINM [8] 0.850 0.834 0.842LUCID [43] 0.820 0.839 0.829STRCF [48] 0.816 0.774 0.795DSRFCN3D [47] 0.782 0.726 0.766DHS [65] 0.815 0.701 0.758STM [78] 0.901 0.887 0.894RMNet [127] 0.887 0.889 0.888MHP-VOS[129] 0.895 0.876 0.885RANet+[124] 0.876 0.866 0.871CFBI [130] 0.905 0.883 0.894CFBI+ [130] 0.911 0.887 0.899FgSegNetV2# [63] 0.847 0.805 0.826CascadeCNN# [123] 0.814 0.751 0.782STCN [22] 0.930 0.904 0.917ORIF-TR* [84] 0.896 0.868 0.882TransBlast* [80] 0.863 0.855 0.859REFNet-TBPI* [81] 0.927 0.891 0.909KDNet(Server) 0.891 0.879 0.885KDNet(Server-Client) 0.921 0.903 0.912103Table 5.10: Traditional foreground segmentation results of KDNet compared with state-of-the-art mod-els on the SegTrackV2 dataset. Methods with * are ours. Methods that have # at the end meanthat the results are reproduced.Model FDCL [54] 0.730UOVOS [137] 0.643SSAV [28] 0.801DSRFCN3D [47] 0.878RFCN [120] 0.737DHS [65] 0.762GDHF [46] 0.868UFO [108] 0.863TransVOS [74] 0.800ReuseVOS [87] 0.832FSNet [37] 0.772LTSI [17] 0.862CAS [38] 0.847PDB [105] 0.864MBNM [57] 0.716STRCF [48] 0.899FgSegNetV2# [63] 0.880CascadeCNN# [123] 0.867ORIF-TR* [84] 0.907TransBlast* [80] 0.904REFNet-TBPI* [81] 0.921KDNet(Server) 0.909KDNet(Server-Client) 0.9181045.2.4 Domain generalization experimentTo highlight the effectiveness of KDNet, we conducted a multi-domain experiment, where the model isevaluated using three benchmark datasets simultaneously: CDNet, DAVIS16, and SegTrackV2. The re-sults are compared with state-of-the-art foreground segmentation models. Four models are selected thathave their source code publicly available from the previous experiment to compare KDNet’s generalizationagainst them. These models are REFNet, sEnDec, FgSegNetV2, and CascadeCNN. In this experiment,all models are trained using all three datasets. The four selected models are trained using two differenttraining procedures. The first training procedure is joint training (i.e., all three datasets are used as onemassive dataset). The second training procedure is called fine-tuning (i.e., training is done using the threedatasets sequentially. Each dataset fine-tunes the model after training using the previous dataset). For ourproposed model, the same two training procedures are used in addition to the proposed training procedure.In training with the proposed procedure, an actual federated learning process is simulated such that eachclient acts as a remote device and has its local dataset. On the other hand, the server has no access to theclient’s local data. The server training session uses an entirely different unlabeled dataset called YOVOS.This training session aims to make the server see the input image the same way the other clients see it.Hence, the server will be able to learn the mapping function learned by each client. It is one of the essentialfeatures of a federated learning technique for the server not to have access to the clients’ local dataset.This feature is called privacy.As shown in Table 5.11, training using lifelong learning techniques is much better than training witheither joint or fine-tuning when using multiple datasets. KDNet outperforms other methods when trainedusing joint and fine-tuning. This is due to the multi-head in the architecture of KDNet. However, REFNettrained with TBPI still outperforms KDNet by a noticeable gap %9 in F-measure. The reason is that TBPIhas zero chance of catastrophic forgetting as the parameter used for a dataset are completely isolated fromparameters used for the other datasets. On the other hand, most of the parameters in KDNet are sharedbetween all three datasets. Although, KDNet’s loss function minimizes catastrophic forgetting. Still, thereis a chance for catastrophic forgetting to occur.During the training of KDNet in this experiment, it was noted that after each federated learning cycle,the clients instantiated from the server converged faster toward their local dataset. Figure 5.11 shows that105Table 5.11: Comparison of KDNet against state-of-the-art methods using the three datasets CDNet,DAVIS16, SegTrackV2. The values in the table represent the F-measure. KDNet (Server) is onlythe server network without clients. On the other hand, KDNet (Server) means both the server andclients are used in training. methods with * are our previous work. Methods that have # at the endmean that the results are reproduced.Training Model CDNet DAVIS16 SegTrackV2 AverageJointsEnDec# [1] 0.657 0.571 0.526 0.584FgSegNetV2# [63] 0.639 0.567 0.495 0.567CascadeCNN# [123] 0.582 0.535 0.460 0.525REFNet* [81] 0.737 0.630 0.505 0.624KDNet (Server) 0.627 0.706 0.580 0.637Fine-tuningsEnDec# [1] 0.451 0.562 0.829 0.614FgSegNetV2# [63] 0.419 0.518 0.755 0.564CascadeCNN# [123] 0.398 0.503 0.746 0.549REFNet* [81] 0.422 0.529 0.917 0.622KDNet (Server) 0.421 0.590 0.858 0.623Lifelong LearningREFNet-TBPI* [81] 0.805 0.768 0.860 0.811KDNet(Server) 0.511 0.638 0.852 0.667KDNet(Server-Client) 0.622 0.747 0.867 0.745106Figure 5.11: The average loss of clients of all three datasets after each federated learning cycle.the average loss of all clients gets lower and converges faster after each federated cycle. Hence, the trainedserver has good knowledge of different domains and can adapt agilely to new domains.5.2.4.1 DiscussionThe multi-head in KDNet architecture allows the network to perform better when using multiple datasetsdespite the training method. The proposed lifelong learning method improved the performance of KDNetby %10.8 in F-measure over the joint training. Although REFNet-TBPI outperforms KDNet, REFNet-TBPIhas a limited number of tasks. However, KDNet has no limit to the number of tasks. The number of headsand clients in KDNet can increase as the number of tasks increases.5.3 Lifelong learning using Dynamically Growing Tree of sub-networks (DGT)To take advantage of TBPI’s high performance and overcome the limitation on a fixed number of tasks, wepropose DGT, which combines the idea of TBPI with parts from KDNet to allow the network to learn from anynumber of tasks while maintaining high performance. Dynamically Growing Tree of sub-networks (DGT) is anovel hybrid lifelong learning technique based on parameter isolation and regularization-based techniques.107The server architecture is a tree of sub-networks (i.e., each node is a part of the network). In contrast,the client is a task-specific network constructed from stacking multiple tree nodes. DGT benefits from theserver-client architecture as the server size keeps growing while the client used for inference is fixed (i.e.,the inference time is fixed no matter how big the server is).It is important to define what is considered in-domain and what is an out-of-domain video in this chapter.An in-domain video is defined as one that is sampled from the same distribution of the training set (e.g. ifa dataset is split by videos into training and testing, then, both training and testing videos are consideredin-domain). On the other hand, An out-of-domain video is defined as one that is sampled from a distributionthat is different from the distribution of the training set (i.e., videos that are sampled from a dataset that isdifferent from the training dataset are considered out-of-domain).5.3.1 ArchitectureThe proposed model is a server-client architecture. The server is the DGT that holds the parameters thatsolve all tasks (i.e., segment all videos). The client is the task-specific network that is instantiated fromDGT to solve a specific task (i.e., segment one or more videos).DGT is a tree of sub-networks, representing each sub-network as a tree node. Each tree node holdsa list of videos assigned to it (Vid) and a number of parameters φ based on the depth of the node in thetree. The root node has all of the network parameters. A node at depth d has parameters of a number ofdecoders blocks N such that N = l− d+ 1 where l is the number of decoder blocks in the network. Forexample, if the number of decoder blocks is 4, a node at the first depth (i.e., a child node to the root node)has parameters from all four decoder blocks. A node at the second depth has the parameters of the lastthree blocks of the decoder, and so on. This means each node in the tree has the parameters of a group ofdecoder blocks (i.e., sub-network). It is important to note that DGT is applied on the decoder only while theencoder is kept unchanged (i.e., encoder parameters are not updated) to keep the encoder task-agnostic.The architecture of the task-specific network consists of an encoder which is a ResNet50 [33] pre-trained using ImageNet [24], and a simple decoder to segment a small group of videos (task-specific). BothThe decoder consists of four blocks with a 1×1 convolutional layer to produce the output. Each block hasthree convolutional layers. The parameters of the task-specific network are instantiated from DGT based108on the selected node (the suitable node to segment the current task). The instantiating process is doneby stacking the parameters stored in each node in the path from the root node to the selected node. Forexample, if the selected node is at the depth d = 3, the encoder parameters are instantiated from the rootnode, the parameters of the first decoder block are instantiated from a node at depth d = 1, the parametersof the second decoder block are instantiated from a node at depth d = 2, and the rest of parameters areinstantiated from the selected node. It is important to mention that the nodes at d = 1 and 2 are in the pathbetween the root node and the selected node.This novel architecture is built to overcome catastrophic forgetting even when only a few labels exist. Toachieve this, a novel lifelong learning procedure is proposed to train the proposed model, which is presentedin the next subsection.5.3.2 A Novel lifelong learning procedureThe training of DGT is based on a novel lifelong learning procedure and is divided into two phases, 1)building the base DGT and 2) growing the base DGT.Building the base DGT: Building the base DGT has two main steps: 1) initialization and 2) saturation.The initialization step: The tree is initialized with a single root node with all network parameters coming fromboth the encoder and the decoder. The encoder parameters are initialized with a pre-trained ResNet50,and the decoder parameters are initialized with random values. Then, a task-specific network is generatedusing the parameters in the root node, and the decoder parameters are updated using all N seen videos inthe training set. Finally, the updated parameters are stored in the root node, replacing the initial parameters,and the ID of all training videos is stored in the list of videos in the root node.The saturation step: In this step, the N seen videos are repeatedly clustered coarse to fine. Startingwith the first clustering iteration, a child node is instantiated from the root node that has a copy of thedecoder parameters. Then, a task-specific network is generated by stacking the decoder parameters of thechild node on top of the encoder parameters from the root node.After that, the decoder parameters are updated using the first video in the list of videos in the parentnode. Then, the updated parameters are stored in the child node. At this point, a child node with updatedparameters is created based on the first video in the parent node. To assign the remaining videos to the109parent node, we need to check if they belong to the newly created node or if new nodes need to be created.Hence, for the remaining videos in the parent node list, a task-specific network is generated from the childnode and the parent node, and the f-measure is calculated. If the highest f-measure is the parent node,a new child is instantiated from the parent node, and the current video is assigned to it (i.e., the currentvideo will be used to update the parameters of this newly instantiated node). On the other hand, if thehighest f-measure is the child node, the current video is assigned to it. Since this node already exists (i.e.,other videos are assigned to it), the parameters updating process is weighted by a factor such that theparameters sensitive to changes (i.e., important to previous videos) has small weight and other parametershave higher weights. This is done to reduce catastrophic forgetting. The parameter sensitivity is calculatedusing the fisher information matrix [112]. The Fisher information matrix measures how much a change inthe parameter’s value would change the model’s prediction and is calculated as follows:ExDKL(Pφ (y|x)||Pφ+δ (y|x)) = δ TFφδ +O(δ 3) (5.7)where δ is a small perturbation vector with the size |φ |, and Fφ is the fisher information matrix with size|φ |× |φ |. Since we need to calculate Fφ in each iteration, we approximate Fφ as a diagonal matrix to beeasily computed as follows:Fˆφ =1NN∑i=1Ey∼Pφ (y|xi)(∇φ logPφ (y|xi))2 (5.8)Then, Fˆφ is normalized to be ∈ [0,1]. If a parameter has the value of 0 in Fˆφ means that this parameterhas no effect on the model’s output (not an important parameter) and vice versa. Finally, each parameterupdate is weighted by its inverted Fˆφ :φ = φ −λ (1− Fˆφ )∇φ `(yˆ,y) (5.9)By doing this, updating parameters that are important to previous videos will be minimal as the correspond-ing value in Fˆφ is close to 0. On the other hand, the parameters that are not important will be updatednormally as their corresponding value in Fˆφ is close to 1.After assigning a video to a node, and the node is updated using the new video, a pruning step isapplied. In this step, if the f-measure of the new video is still higher than the newly assigned node, this110Figure 5.12: Visualization of building a base DGT given 6 videos and a decoder with 4 layers. On theleft side is coarse to fine clustering of videos. On the right side is the produced DGT.video is removed from the new node, and if it was the only video in this node, then, this node is pruned fromthe tree.In the next clustering iterations, the same process is repeated for each terminal node as a root for a sub-tree. The only difference is that the nodes added at a deeper depth have a fewer number of parametersthan the nodes added at a shallower depth (i.e., each node is a sub-network of the parent node). Themaximum number of clustering iterations is l-iterations where l is the number of decoder blocks.The result of this phase (building the base DGT) is an unbalanced tree of sub-networks. The tree isunbalanced due to the pruning step that may terminate the clustering process at an early depth for somenodes. The full process of building a base tree is shown in Algorithm. 6An example of building a base DGT for a dataset of videos is visualized in Figure 5.12. The blue arearepresents the tree initialization step where the network is trained using all videos. The orange, green, andcyan areas represent three iterations of coarse to fine clustering. Each circle represents a node in the tree,such that φ0 is the set of parameters of the root node, φ1 and φ2 are parameters of nodes created at depthd = 1, φ3,φ4, and φ5 are parameters of nodes created at depth d = 2, and so on. Since the number ofdecoder layers is 4, then the maximum depth of the tree is 4. The location of vi in the figure represents theoptimal parameters to solve it, and the colored arrows represent training steps.Growing of DGT: In this phase, the base tree grows to learn new unseen OOD videos. For each new111video, DGT searches the tree for the most suitable node in a greedy breadth-first search fashion. Startingfrom the root node as a parent node, the F-measure is calculated for the parent and all children in the nextdepth. If the f-measure of the parent is higher than all children, the search stops and returns the parentas the suitable node. Else, if at least one child has an f-measure higher than the parent, the child withmaximum f-measure will act as a root to a sub-tree. This process is repeated until a parent node with ahigher f-measure than all its children is found or a terminal node is reached. Finally, based on the outcomeof the search, if the returned node by the search is a terminal node, DGT assigns the new video to thisnode and updates the parameters as mentioned before in Eq. 5.9. On the other hand, if the node is aninternal node, a new child node is instantiated as a copy of the parent node (e.g., building on the previousknowledge of the tree).112Algorithm 6 Building the base DGT.Require: V the foreground segmentation videos, Net the NetworkEnsure: Tree the base tree1: %tree initialization2: Net ← fit model(Net, V )3: Tree← create tree node(Net, depth= 0)4: Depth← size of decoder(Net)5: %coarse to fine clustering6: for D in Depth do7: Tree← generate deeper tree(Tree)8: %clustering videos to different nodes9: for v in V do10: Node, Reuse← create or reuse node(Tree)11: Net ← load weights(Net, node)12: if Reuse then13: Net ← fit model with FIM(Net, v)14: else15: Net ← fit model(Net, v)16: end if17: %store updated node or prune18: NodeF ,ParentF ←f measure(Net,Node,v)19: if NodeF > ParentF then20: Node← store weights(Net)21: else22: if Node has only one video then23: Tree← prune(Tree,Node)24: else25: Node← remove video(Node,v)26: end if27: end if28: end for29: end for30: Return Tree5.3.3 DGT Foreground segmentation procedureOnce the training is complete, as described above, DGT is ready to segment foreground objects from in-domain or out-of-domain videos. If the video is in-domain, DGT requires the video ID to find the suitablenode. For example, in Figure 5.12, if the video ID is 4 then node 7 is selected. If the video is OOD, onelabeled frame from the new video is required for DGT to select a suitable node. The selection is also madeusing a greedy breadth-first search algorithm to find the node with the highest f-measure. The next step isgenerating a task-specific network. In this step, the network is generated by stacking parameters from each113Figure 5.13: DGT in the testing phase. First, DGT finds a suitable node for the new video using alabeled frame. Then, DGT generates a task-specific network by stacking sub-networks fromnodes in the path from the root node to the selected node. Finally, the generated network is usedto segment the new video frames. The color of the network’s blocks represents which node isused to initialize the parameters.node in the path from the root node to the selected node. Figure 5.13 shows an example of a completecycle of testing using a new OOD video. The video frame with a camel represents one labeled frame froma new OOD video, which is fed to the DGT tree. Next step, DGT searches the tree to find a suitable node,the yellow node in the figure. Then, a task-specific network is generated by stacking the parameters ineach node in the path between the root (blue node) and the selected node (yellow node). The output of thisstep is a task-specific encoder-decoder network that has 1) encoder parameters from the root node, 2) firstdecoder block parameters from the second node in the path (orange node), 3) the second decoder blockparameters from the third node in the path (green node), and 4) the last two decoder blocks parameters arefrom the selected node (yellow node). Finally, the generated network is used to segment the foregroundobjects in all frames of the new OOD video (VID-OOD-1 in Figure 5.13.1145.3.4 Experiments5.3.4.1 DatasetsThe used datasets are DAVIS16 and SegTrackV2 to compare the results of the DGT with state-of-the-artmodels. In the lifelong learning and few-shot learning experiment, the used datasets are YOVOS, CDNet,and DAVIS17. YOVOS is used to build the base knowledge for DGT, REFNet-TBPI, and two state-of-the-artmodels. CDNet is used to test the capability of these models to learn from a new dataset without forgettingthe previous dataset. Finally, DAVIS17 is used to evaluate the performance of these models when only alimited number of labeled data is available.5.3.4.2 Evaluations and resultsTo show the strength of the proposed DGT, three different experiments are used to evaluate its performanceusing in-domain and out-of-domain videos: 1) traditional foreground segmentation using in-domain videos,2) out-of-domain foreground segmentation under few-shot learning and catastrophic forgetting, and 3) DGTgeneralization over multiple in-domain and OOD videos without re-training.In the first experiment, DGT and state-of-the-art methods are trained using a group of videos and testedusing another group of unseen videos within the same domain (i.e., the same dataset). The datasets usedin this experiment are DAVIS16[89] and SegTrackV2[53].In the second experiment, we address the problem of catastrophic forgetting and the lack of labeleddata. This experiment has three phases 1) building a base knowledge model using a large-scale dataset,YOVOS [128], 2) applying lifelong learning on the trained model from phase one using an out-of-domaindataset, CDNet [122], 3) applying lifelong learning but under few labeled samples from the OOD dataset,DAVIS17 [91] (same as phase two but using a few amounts of labeled data). The purpose of the firstphase is to build a base knowledge that can be transferred to new videos. The second phase assessesthe amount of catastrophic forgetting that occurs when the models are trained using out-of-domain videos.The third phase assesses the ability of the models to learn new videos using a few labeled frames (1 or 5labeled frames per video).The last experiment shows the strength of DGT to generalize over multiple in-domain and out-of-domain115videos without re-training. In this experiment, the generalization of the models is evaluated by testing on allthree datasets (YOVOS, CDNet, and DAVIS17) without applying re-training or fine-tuning to DGT.In all three experiments, the segmentation performance is calculated using two DGT networks withdifferent decoders denoted as NetS and NetL to estimate the scalability of DGT with respect to the modelsize. NetS represents DGT with a small number of parameters, while (NetL) is a DGT network with a largernumber of parameters. The difference between the two decoders is the number of filters in the convolutionallayers. Since the architecture of DGT has four blocks as described above, NetS has the size of filters ineach block as 64, 64, 32, and 32, respectively, whereas, NetL filters in each block have sizes of 256, 128,64, and 32, respectively.Finally, an analysis of the results from the three experiments and an ablation study are presented withrespect to runtime, growth of the model size, and the performance of different configurations of DGT.5.3.4.3 Traditional foreground segmentation using in-domain videosIn this experiment, the datasets DAVIS16 and SegTrackV2 are split video-wise into training and testing,such that videos in the testing set are different from videos in the training set. In traditional foregroundsegmentation evaluation, the models are trained using the training videos. Then, in the testing phase, themodels have access to one labeled frame from each video in the testing set to fine-tune the model. Forthe evaluation metrics, we used the standard metrics region similarityJ , f-measure F and their averageJ&F for DAVIS16. In SegTrackV2, the standard metric is f-measureF only. Table 5.12, and 5.13 showthe results of the proposed DGT against state-of-the-art methods. DGT outperforms other models in bothdatasets using both networks, NetS and NetL. This enhanced performance is largely attributed to the DGT’scapability of reducing catastrophic forgetting.5.3.4.4 Out-of-domain foreground segmentation under few-shot learning and catastrophic forgettingThe three phases of this experiment are 1) building a base knowledge model, 2) applying lifelong learningon the base knowledge model to learn using an out-of-domain dataset to overcome catastrophic forgetting,and 3) same as phase two but using a few labeled frames to show that the proposed model overcomes theproblem of lack of labeled data. In this experiment, three models are selected from the state-of-the-art to116Table 5.12: Traditional foreground segmentation results of DGT in comparison with state-of-the-artmodels on the DAVIS16 dataset. * means our previous work. Methods that have # at the endmean that the results are reproduced.Model F J F&JCINM [8] 0.850 0.834 0.842LUCID [43] 0.820 0.839 0.829STRCF [48] 0.816 0.774 0.795DSRFCN3D [47] 0.782 0.726 0.766DHS [65] 0.815 0.701 0.758MiVOS [21] 0.924 0.897 0.910KMN [100] 0.915 0.895 0.905STM [78] 0.901 0.887 0.894RMNet [127] 0.887 0.889 0.888MHP-VOS[129] 0.895 0.876 0.885RANet+[124] 0.876 0.866 0.871CFBI [130] 0.905 0.883 0.894CFBI+ [130] 0.911 0.887 0.899HMMN [101] 0.920 0.896 0.908STCN [22] 0.930 0.904 0.917FgSegNetV2# [63] 0.847 0.805 0.826CascadeCNN# [123] 0.814 0.751 0.782ORIF-TR* [84] 0.896 0.868 0.882TransBlast* [80] 0.863 0.855 0.859REFNet-TBPI* [81] 0.927 0.891 0.909KDNet* 0.921 0.903 0.912DGT-NetS (ours) 0.940 0.902 0.921DGT-NetL (ours) 0.949 0.910 0.929117Table 5.13: Traditional foreground segmentation results of DGT compared with state-of-the-art modelson the SegTrackV2 dataset. * means our previous work. Methods that have # at the end mean thatthe results are reproduced.Model FDCL [54] 0.730UOVOS [137] 0.643SSAV [28] 0.801DSRFCN3D [47] 0.878RFCN [120] 0.737DHS [65] 0.762GDHF [46] 0.868UFO [108] 0.863TransVOS [74] 0.800ReuseVOS [87] 0.832FSNet [37] 0.772LTSI [17] 0.862CAS [38] 0.847PDB [105] 0.864MBNM [57] 0.716STRCF [48] 0.899FgSegNetV2# [63] 0.880CascadeCNN# [123] 0.867ORIF-TR* [84] 0.907TransBlast* [80] 0.904REFNet-TBPI* [81] 0.921KDNet* 0.918DGT-NetS (ours) 0.948DGT-NetL (ours) 0.954118compare our results with their results.In the first phase (building a base knowledge model), the models are trained using the YOVOS dataset.The results of this phase are shown in Table 5.14. As shown in Table 5.14, we reported the results ofdifferent versions of our proposed work, the two networks NetS and NetL with/out DGT. The performance ofboth networks has massively increased when DGT is used, which leads to outperforming other models bya noticeable amount inF . DGT-NetL outperforms the second best model by approximately 9% inF .In the second phase (lifelong learning using an out-of-domain dataset), the models are trained usingthe CDNet dataset, which is an out-of-domain dataset. We compare our results with the three selectedmodels and five lifelong learning techniques. These lifelong learning techniques are applied to the networkNetL. We used the lifelong learning framework FACIL[73] to implement these lifelong learning techniques.The model is trained on videos from CDNet sequentially (one video at a time). The reported metrics arethe average f-measure for all videosF , and catastrophic forgetting factor CF . In more detail, after trainingon each video, the f-measure is calculated for all previous videos, andF is the average of all f-measures:Fv =1vv∑i=1Fv,i (5.10)F =1NN∑i=1Fi (5.11)whereFv,i is the f-measure of video i after training using v videos, and N is the total number of videos. TheforgettingCF is calculated as the average forgetting after each task. The forgetting after task t is calculatedas the average loss in f-measure for all tasks ∈ [1, t]:CFv =1vv∑i=1Fi,i−Ft,i (5.12)CF =1NN∑i=1CFi (5.13)Table 5.15 shows the results of the lifelong learning phase. The models are split into two categories,fine-tuning-based and lifelong learning-based. As shown in the table, DGT-NetL outperforms other modelsby more than 14% in F , and in the second place with respect to CF with the value 0.015, which meansDGT almost prevented the occurrence of catastrophic forgetting when compared with the base network119NetL with the value of 0.281 inCF . The reason why DGT-NetL is better than DGT-NetS is that the utilizationof a fisher information matrix with a larger number of parameters is more effective. It is noticeable that theperformance of MiVOS on the CDNet dataset is poor. This is because it is a memory-based network, andthe memory part of the network should be initialized with a frame that has a foreground object. However,CDNet is different from other datasets where some frames do not have any foreground objects (the maskis empty). This causes MiVOS to fail to segment foreground objects when initialized with an empty mask.Still, MiVOS has a good catastrophic forgetting factor compared to other fine-tuning-based models. This isbecause memory-based networks use a labeled frame from each video as a reminder during the testingphase. On the other hand, REFNet-TBPI has zero percent of catastrophic forgetting, but its performanceis bad compared to its performance in the pre-training. This is because REFNet-TBPI was designed to betrained on a finite set of videos. When REFNet is exposed to new videos after training is done, only a fewparameters can be fine-tuned, and these parameters are isolated from parameters used for other videos(i.e., knowledge transfer is not applicable), which leads to poor performance. Finally, all lifelong learning-based models have good performance with respect to bothF andCF . The conclusion of this phase is thattraining using a lifelong learning technique improves performance when trained on data sequentially.The last phase is lifelong learning with few-shot learning. We start with the trained models from theprevious phase (i.e., trained using YOVOS and CDNet). Then, for each video in DAVIS17, the models aretrained using a few labeled frames and tested to segment the foreground objects in all frames in this video.We used the standard configurations for the number of labeled frames in the few-shot learning, which are1-shot (i.e., one labeled frame per video) and 5-shot. As shown in Table 5.16, MiVOS is in second placein both 1-shot and 5-shot. This is because the DAVIS17 dataset has a foreground object in all frames ofeach video (i.e., MiVOS is initialized correctly, unlike the previous phase). However, DGT-NetL outperformsMiVOS by approximately 9% in 1-shot and 13% in 5-shot.5.3.4.5 DGT generalization over multiple in-domain and OOD videos without re-trainingAfter training is finished in the previous experiment, DGT and state-of-the-art models are evaluated usinga testing set that consists of all three datasets, namely, YOVOS, CDNet, and DAVIS17, without re-trainingor fine-tuning. This experiment tests the generalization of DGT (i.e., its performance on multiple datasets).120Table 5.14: Foreground segmentation results of DGT compared to state-of-the-art methods on YOVOS.* means our previous work. Methods that have # at the end mean that the results are reproduced.Model FMiVOS# [21] 0.628FgSegNet# [63] 0.729REFNet-TBPI* [81] 0.736NetS (ours) 0.515DGT-NetS (ours) 0.751NetL (ours) 0.607DGT-NetL (ours) 0.825Additionally, this experiment evaluates the catastrophic forgetting that occurs when moving from one domainto another (i.e., from one dataset to another). The reported evaluation metric is the f-measure of eachdataset individually, the average f-measure of all three datasets, the catastrophic forgetting of each dataset,and the average catastrophic forgetting. To report the catastrophic forgetting of each dataset individually,we define partial catastrophic forgetting factor CFp as:CFp =1(b−a)b∑i=aCFi (5.14)where b is the last video of a dataset, and a is the first video of the same dataset. Table 5.17 shows theresults of this experiment. Each dataset has two columns, one for the f-measureF and one for the partialcatastrophic forgetting that occurred in this dataset. The last two columns are the average performanceon all three datasets. As shown in the table, the proposed DGT model significantly outperforms state-of-the-art models by around 18% improvement in average F of all three datasets. Comparing the proposedDGT versus state-of-the-art models in Table 5.17, 1) the proposed DGT lifelong learning technique leadsto significant improvement in performance and reduces catastrophic forgetting, 2) the memory networkin MiVOS [21] is good in dealing with catastrophic forgetting but performs poorly when initialized with anempty mask with no foreground objects, 3) fine-tuning based models that are not memory networks (e.g.,FgSegNet [63]) has bad performance and a high catastrophic forgetting, and 4) the proposed DGT has the121Table 5.15: Lifelong learning foreground segmentation results of DGT compared to state-of-the-artmethods on CDNet. Methods are categorized as fine-tuning-based for traditional methods, andlifelong learning-based for methods trained using a lifelong learning technique. * means ourprevious work. Methods that have # at the end mean that the results are reproduced.Training Model F CFFine-tuning-basedMiVOS# [21] 0.165 0.068FgSegNet# [63] 0.481 0.253NetS (ours) 0.432 0.316NetL (ours) 0.520 0.281Lifelonglearning-basedEWC-NetL# [44] 0.569 0.092iCarL-NetL# [93] 0.590 0.078MAS-NetL# [3] 0.616 0.061PathIntegral-NetL# [132] 0.564 0.096RWalk-NetL# [16] 0.571 0.091REFNet-TBPI* [81] 0.394 0.00DGT-NetS (ours) 0.660 0.057DGT-NetL (ours) 0.751 0.015122Table 5.16: Few-shot foreground segmentation results of DGT compared to state-of-the-art methodson DAVIS17. Methods are categorized as fine-tuning-based for traditional methods, and lifelonglearning-based for methods trained using a lifelong learning technique. * means our previouswork. Methods that have # at the end mean that the results are reproduced.Training Model 1-shot 5-shotFine-tuning-basedMiVOS# [21] 0.655 0.676FgSegNet# [63] 0.621 0.640NetS (ours) 0.477 0.493NetL (ours) 0.559 0.590Lifelonglearning-basedEWC-NetL# [44] 0.583 0.625iCarL-NetL# [93] 0.610 0.640MAS-NetL# [3] 0.633 0.675PathIntegral-NetL# [132] 0.584 0.627RWalk-NetL# [16] 0.580 0.625REFNet-TBPI* [81] 0.403 0.413DGT-NetS (ours) 0.716 0.755DGT-NetL (ours) 0.742 0.806123Table 5.17: Generalized foreground segmentation results of DGT compared to state-of-the-art methodson all three datasets. * means our previous work. Methods that have # at the end mean that theresults are reproduced.ModelYOVOS CDNet DAVIS17 AverageF CFp F CFp F CFp F CFpMiVOS# [21] 0.539 0.088 0.137 0.027 0.645 0.031 0.441 0.0486FgSegNet# [63] 0.502 0.226 0.388 0.092 0.640 0.196 0.510 0.171EWC-NetL# [44] 0.515 0.091 0.468 0.101 0.625 0.079 0.571 0.090iCarL-NetL# [93] 0.536 0.070 0.538 0.051 0.640 0.063 0.561 0.061MAS-NetL# [3] 0.562 0.041 0.573 0.039 0.675 0.035 0.603 0.038PathIntegral-NetL# [132] 0.450 0.156 0.487 0.076 0.627 0.091 0.521 0.107RWalk-NetL# [16] 0.465 0.141 0.502 0.068 0.625 0.084 0.530 0.097REFNet-TBPI* [81] 0.736 0.000 0.394 0.000 0.413 0.000 0.514 0.000NetS (ours) 0.367 0.147 0.279 0.152 0.493 0.147 0.380 0.148DGT-NetS (ours) 0.724 0.026 0.639 0.020 0.755 0.030 0.719 0.025NetL (ours) 0.398 0.209 0.285 0.234 0.590 0.147 0.424 0.196DGT-NetL (ours) 0.805 0.018 0.733 0.017 0.806 0.021 0.782 0.018best generalization and can be used for a wide range of different videos, which means that DGT has theability to learn effectively from OOD videos with minimal forgetting of previous videos even with few labeledframes.Visualization. Sample results of DGT compared against other models in the generalized foregroundsegmentation experiment are shown in Figure 5.14, and additional results are shown in Figure 5.15. Thefull DGT-NetL trained on all three datasets are visualized in Figure 5.16. DGT-NetL is visualized as a circulartree, starting from the root node in the middle to the leaf nodes at the outermost circle. The written textbeside each node is the node name which is formatted as the path of this node from the root node (e.g.,a node name 0 0|1 3|2 2 means the second node in second depth branched from node three in the firstdepth branched from node zero (root) at depth zero). In the figure, nodes have three different colors blue,124Figure 5.14: Visual results of generalized foreground segmentation of our DGT against other models.red, and black. Blue nodes are the nodes created during training on the first dataset (YOVOS). The rednodes are the new ones created during training on the second dataset (CDNet). The black nodes are thenodes added to DGT during training on the last dataset (DAVIS17).5.3.5 DGT analysis and ablation study5.3.5.1 Model size analysisTo analyze the proposed model size, we calculated the number of parameters of the model after the lastphase in the previous Sub-section. 5.3.4.2. Since our model is divided into two parts, the tree (DGT) andthe task-specific network (NetS or NetL). We report the number of parameters in each part separately. InTable 5.18, the number of parameters of the full tree is reported as ”Full model (Param. #)”. While thenumber of parameters of the task-specific network is reported as an ”Inference model (Param. #)”. Thevalues in the table are in millions of parameters. For state-of-the-art models, the number of parameters isthe same in the full model and inference model except for REFNet-TBPI, as it has two parts as our model.As shown in Table 5.18, DGT-NetS started with 2 million parameters and, during training, expanded to 63.3million parameters. Still, the number of parameters in DGT-NetS close to MiVOS after training on threedatasets (YOVOS, CDNet, and DAVIS16) while achieving better performance as shown in Table 5.17.In Figure 5.17, The growth in the number of parameters of DGT against the number of videos is shown125Figure 5.15: Visual results of generalized foreground segmentation of our DGT-NetL.126Figure 5.16: Visualization of DGT-NetL as a circular tree after training using YOVOS, CDNet, andDAVIS17. The node in the middle is the tree’s root, and the nodes in the first inner circle arechildren of the root node. The node color defines at which stage the node was added. Bluenodes are added during the initial phase using YOVOS. Red nodes are added during the lifelonglearning phase using CDNet. Finally, black nodes are added during the few-shot learning phaseusing DAVIS17. The visualization is made using the ETE toolkit [35].127Table 5.18: Model size analysis of proposed work against other models. The number of parameters isin millions. * means our previous work.ModelFull model(Param. #)Inference model(Param. #)MiVOS [21] 53 53FgSegNet [63] 9.3 9.3REFNet-TBPI* [81] 94 5.8NetS (ours) 2 2DGT-NetS (ours) 67.5 2NetL (ours) 4.6 4.6DGT-NetL (ours) 207.6 4.6Figure 5.17: The increase in the number of parameters of the model versus the number of videos duringeach depth in the ”Building base tree” phase of DGT using the YOVOS dataset. (a) shows theincrease in the number of parameters of DGT-NetS. While (b) shows the increase in the numberof parameters of DGT-NetL.128in different depths of the ”Building Base Tree” training phase. As shown in Figure 5.17, the model sizeincreases by a noticeable amount during the training at depths 1 and 2 for both models DGT-NetS andDGT-NetL. Then, the model size increases slightly at depth 3. On the other hand, at depths 4 and 5, themodel size is almost the same (i.e., the model gets saturated after depth 3). We calculated the averagegrowth rate of model size as the total number of parameters added divided by the number of videos. Thegrowth rate for DGT-NetS and DGT-NetL is 18,236 and 50,590 parameters, respectively.5.3.5.2 Runtime analysisOur proposed model runtime is analyzed and compared with the state-of-the-art model. We split the runtimeinto initialization time and inference time, as our model has two phases during testing, the task-specificnetwork generation phase and the Inference phase. The initialization time is the time taken to generate atask-specific network, and this is done only once per video. The inference time is the time taken to producethe output of an input frame. Finally, frame per second (FPS) is reported to show how many frames canbe processed by our model within the same video. As shown in Table 5.19, DGT-NetS has the fastestinference time that can process 25 frames per second. When DGT is used with NetL, the FPS droppedto almost half the FPS when NetS on the expense of the performance gained as shown in Tables. 5.12-5.17. However, the initialization time for both DGT-NetS and DGT-NetL are approximately the same. Thisis because initialization time is affected by the number of nodes in DGT, while inference time is affected bythe number of parameters in the network. On the other hand, REFNet-TBPI has a huge initialization timebecause there is no greedy way to search for a suitable set of parameters, and the model has to go throughall existing sets of parameters to find the suitable one.5.3.5.3 Different configurationsThe proposed work is evaluated using different configurations to show the importance of each component.The components that we used in this experiment are pruning clustering criteria and the fisher informationmatrix. Pruning is the process of deleting unwanted nodes and stopping some videos from further cluster-ing. While the clustering criteria is the metric that we used to decide whether a video will be assigned to anexisting node or a new one. The metric we used is the f-measure F , but we tried a different metric in this129Table 5.19: Runtime analysis of proposed work against other models. The reported runtime is in sec-onds. * means our previous work. DGT-NetS,S means testing on seen videos (searching for a nodeby video ID), and DGT-NetS,U means testing on unseen videos (searching for a node by greedyalgorithm).ModelInitialization time(per video)Inference time(per frame) FPSMiVOS [21] 0.0 0.089 11.2FgSegNet [63] 0.00 0.045 22.1REFNet-TBPI* [81] 45.13 0.116 8.6NetS (ours) 0.00 0.039 25.1DGT-NetS,S (ours) 0.00 0.039 25.1DGT-NetS,U (ours) 8.75 0.039 25.1NetL (ours) 0.00 0.087 11.4DGT-NetL,S (ours) 0.00 0.087 11.4DGT-NetL,U (ours) 9.69 0.087 11.4130Table 5.20: Different configurations of DGT-NetS using DAVIS16 dataset.PruningClusteringCriteriaFisher InformationMatrix F Param. #Yes F Yes 0.940 3.3No F Yes 0.907 7.2Yes MSE Yes 0.926 27.5Yes F No 0.903 4.5experiment which is the mean squared error (MSE). Finally, the Fisher Information Matrix (FIM) is used toupdate some of the existing parameters to reduce the forgetting of important parameters. In Table 5.20, wereported the f-measure and number of parameters for each configuration. The dataset used in this exper-iment is DAVIS16, and the model used is DGT-NetS. As shown in the table, When pruning is not used thenumber of parameters increases, and performance decreases as a result of unwanted clustering. This clus-tering causes the network to be more task-specific, which leads to over-fitting on a specific video. Hence,the performance degrades during testing. A similar case happened when MSE was used instead ofF . AsMSE is more sensitive to the output, using MSE leads to over-clustering. The downside of over-clustering isthat each node in the tree will be assigned to one video. Hence, forward transfer (i.e., the ability to transferprevious knowledge to learn from a new video) is prevented. This also leads to over-fitting on the trainingframes of the given video. Additionally, leads to a massive increase in the number of parameters. On theother hand, when FIM is not used, the performance is degraded because the occurrence of catastrophicforgetting increases.5.4 ConclusionIn this chapter, three lifelong learning techniques are proposed namely, TBPI, KDNet, and DGT. DGT is acombination of TBPI with KDNet to take advantage of each technique to solve the problem of catastrophicforgetting while outperforming state-of-the-art lifelong learning techniques and foreground segmentationmodels. The source code is available at https://github.com/islamosmanubc/DGT.131In the next chapter, we propose a unified foreground segmentation model (UFSeg) that combines thebest method in each chapter to solve all the problems and outperform DGT, especially, in learning withlimited data.132Chapter 6A Unified foreground segmentation modelIn this chapter, we propose the Ultimate Foreground Segmentation model (UFSeg) that solves all the prob-lems mentioned in Chapter 1. Since we solved each problem in a separate model in previous chapters. TheUFSeg combines these separated models in one model. The selected models that are used in UFSeg areREFNet from Chapter 3, FeSh-Net from Chapter 4, and DGT from Chapter 5. These models are selectedas they are the best models introduced in each chapter.6.1 UFSeg architectureThe encoder-decoder architecture is the same as REFNet. Then, the Exemplar-based attention map ex-tractor (EAM) from FeSh-Net is added to the model. In more detail, the inputs are the current frame andexemplar frame (a frame with the corresponding ground truth). The encoder is the REFNet encoder whichconsists of four residual blocks and four transition blocks. A multi-scale dense block follows the encoder.This block is used to compute a dense and rich multi-scale feature map for the decoder. After the multi-scale dense block, the decoder is attached to three components: the encoder through a scalar block, themulti-scale dense block, and the EAM. The decoder consists of three blocks. The input for the first de-coder block is the output of the multi-scale dense block that is highlighted by the output of EAM. Then, thescalar block from the first encoder block highlights the output of the first decoder block. On the other hand,the output of the second block is highlighted by both the scalar block from the second encoder block. Aconvolutional layer processes the output of each decoder block to produce the output mask (binary image)133Figure 6.1: Architecture of UFSeg.and the edges, as illustrated in Section 3.2. Finally, all decoder outputs are combined with a region/edgesfusion block to produce the final output mask. The UFSeg model is shown in Figure 6.16.2 UFSeg training and testingUFSeg is trained using a lifelong meta-learning technique, combining FeSh-Net’s Meta-training and DGT.It is almost the same as DGT. The differences are that the dataset is divided into episodes (i.e., a group oftasks), and in each iteration of DGT the model process the dataset episode by episode instead of task bytask. and the update is done on the average loss of all tasks in an episode. The full training technique isshown in Algorithm 7.The testing procedure of UFSeg is the same as DGT. First, a labeled image from a frame sequenceis used as input, and a set of nodes from the tree is selected using a greedy search algorithm. Then, atask-specific network is created using the selected nodes. Finally, the created network segments all framesin the frame sequence.134Algorithm 7 Building base UFSeg.Require: Dmeta the foreground segmentation videos, Net the NetworkEnsure: Tree the base tree1: UFSeg← fit model(Net, Ep)2: Tree← create tree node(Net, depth= 0)3: Depth← size of decoder(Net)4: for D in Depth do5: Tree← generate deeper tree(Tree)6: for Ep in Dmeta do7: Node, Reuse← create or reuse node(Tree,Ep)8: loss← 09: for v in Ep do10: vtr,vte← get training and testing frames(v)11: Net ← load weights(Net, Node)12: if Reuse then13: Net ← fit model with FIM(Net, vtr)14: else15: Net ← fit model(Net, vtr)16: end if17: loss← loss + calculate loss(Net,vte)18: end for19: Net ← load weights(Net, Node)20: loss← loss/|Ep|21: Net ← Update with loss(Net,loss)22: NodeF ,ParentF ←f measure(Net,Node,Ep)23: if NodeF > ParentF then24: Node← store weights(Net)25: end if26: end for27: end forReturn Tree135Table 6.1: Traditional foreground segmentation results of UFSeg compared with state-of-the-art mod-els on the DAVIS16 dataset. * means our previous work.Model F J F&JMiVOS [21] 0.924 0.897 0.910KMN [100] 0.915 0.895 0.905HMMN [101] 0.920 0.896 0.908STCN [22] 0.930 0.904 0.917TransBlast* [80] 0.863 0.855 0.859REFNet-TBPI* [81] 0.927 0.891 0.909KDNet* 0.921 0.903 0.912ORIF-TR* [84] 0.896 0.868 0.882DGT-NetL* 0.949 0.910 0.929UFSeg 0.951 0.916 0.9336.3 ExperimentsUFSeg is evaluated using three experiments, 1) segment foreground objects in each dataset individually(the traditional foreground segmentation), and the datasets used are CDNet, SegTrackV2, YOVOS, andDAVIS16. 2) lifelong learning with/out few-shot learning using CDNet and DAVIS17 datasets, and 3) gen-eralized foreground segmentation by testing on all datasets without fine-tuning. The results are comparedwith top state-of-the-art models and all proposed models in this thesis. After that, the analysis of UFSeg isdiscussed. Finally, some visual results are shown6.3.1 Traditional foreground segmentationTables 6.1, 6.2, 6.3, and 6.4 shows the results of UFSeg in comparison with the state-of-the-art foregroundsegmentation models and all proposed models in this thesis. As shown in the tables, UFSeg outperformsall other models with respect to all evaluation metrics except for Table 6.2. The reason that DGT-NetLoutperforms UFSeg in SegTrackV2 dataset is that this dataset is very small. Hence, a smaller model would136Table 6.2: Traditional foreground segmentation results of UFSeg compared with state-of-the-art mod-els on the SegTrackV2 dataset. * means our previous work. Methods that have # at the end meanthat the results are reproduced.Model FDSRFCN3D [47] 0.878STRCF [48] 0.899FgSegNet# [63] 0.880TransBlast* [80] 0.904REFNet-TBPI* [81] 0.921KDNet* 0.918ORIF-TR* [84] 0.907DGT-NetL* 0.954UFSeg 0.948perform better on this dataset, and DGT-NetL is smaller than UFSeg with respect to the number of learnableparameters. On the other hand, we can see a noticeable gap in performance between UFSeg and DGT-NetL in both Tables 6.3 and 6.4 when testing the models using large datasets CDNet and YOVOS.6.3.2 Lifelong learning foreground segmentationTable 6.5 shows the result of UFSeg when trained using CDNet after it was trained using YOVOS to build thebase tree. The results show that UFSeg is better than DGT-NetL in both the f-measure and the catastrophicforgetting. Table 6.6 shows the result of lifelong learning with few-shot learning using the DAVIS17 dataset.The reported results are one and five shots (one and five training frames for each video). Training usingDAVIS17 is applied after the models are trained on both YOVOS and CDNet (accumulated on the resultof the previous table). As shown in Table 6.6, UFSeg outperforms the other models by a massive gap,especially in the 1-shot configuration. This shows the effectiveness of using the EAM module and themeta-training algorithm along with DGT.137Table 6.3: Foreground segmentation results of UFSeg compared to state-of-the-art methods on CDNet.* means our previous work. Methods that have # at the end mean that the results are reproduced.Method FPR FNR Re Pr F-MeasuresEnDec# [1] 0.004 0.076 0.903 0.798 0.842FgSegNetV2# [63] 0.005 0.080 0.893 0.741 0.801BSUV-Net-SBGS [114] 0.005 0.182 0.817 0.831 0.798MODSiam* [85] 0.010 0.181 0.823 0.692 0.714MODY-Net* [86] 0.005 0.124 0.903 0.822 0.865REFNet* [81] 0.004 0.038 0.961 0.827 0.881REFNet-TBPI* [81] 0.003 0.030 0.969 0.850 0.901UFSeg 0.003 0.021 0.971 0.861 0.916Table 6.4: Foreground segmentation results of DGT compared to state-of-the-art methods on YOVOS.* means our previous work. Methods that have # at the end mean that the results are reproduced.Model FMiVOS# [21] 0.628FgSegNet# [63] 0.729REFNet-TBPI* [81] 0.736DGT-NetL* 0.825UFSeg 0.842138Table 6.5: Lifelong learning foreground segmentation results of UFSeg compared to state-of-the-artmethods on CDNet. * means our previous work. Methods that have # at the end mean that theresults are reproduced.Training Model F CFFine-tuning-basedMiVOS# [21] 0.165 0.068FgSegNet# [63] 0.481 0.253Lifelonglearning-basediCarL-NetL# [93] 0.590 0.078MAS-NetL# [3] 0.616 0.061REFNet-TBPI* [81] 0.394 0.00DGT-NetL* 0.751 0.015UFSeg 0.759 0.013Table 6.6: Few-shot foreground segmentation results of UFSeg compared to state-of-the-art methodson DAVIS17. * means our previous work. Methods that have # at the end mean that the results arereproduced.Training Model 1-shot 5-shotFine-tuning-basedMiVOS# [21] 0.655 0.676FgSegNet# [63] 0.621 0.640Lifelonglearning-basediCarL-NetL# [93] 0.610 0.640MAS-NetL# [3] 0.633 0.675REFNet-TBPI* [81] 0.403 0.413DGT-NetL* 0.742 0.806UFSeg 0.794 0.831139Table 6.7: Generalized foreground segmentation results of UFSeg compared to state-of-the-art methodson all three datasets. * means our previous work. Methods that have # at the end mean that theresults are reproduced.ModelYOVOS CDNet DAVIS17 AverageF CFp F CFp F CFp F CFpMiVOS# [21] 0.539 0.088 0.137 0.027 0.645 0.031 0.441 0.0486FgSegNet# [63] 0.502 0.226 0.388 0.092 0.640 0.196 0.510 0.171iCarL-NetL# [93] 0.536 0.070 0.538 0.051 0.640 0.063 0.561 0.061MAS-NetL# [3] 0.562 0.041 0.573 0.039 0.675 0.035 0.603 0.038REFNet-TBPI* [81] 0.736 0.000 0.394 0.000 0.413 0.000 0.514 0.000DGT-NetL* 0.805 0.018 0.733 0.017 0.806 0.021 0.782 0.018UFSeg 0.826 0.016 0.744 0.016 0.831 0.018 0.801 0.0176.3.3 Generalized foreground segmentationThe models are tested using all three datasets without fine-tuning in this experiment. The reported resultsare the f-measure and the catastrophic forgetting of each dataset. As shown in Table 6.7, UFSeg is slightlybetter than DGT-NetL in the average catastrophic forgetting but has approximately 2% improvement inaverage f-measure.6.3.4 UFSeg AnalysisTable 6.8 shows the model size and runtime analysis. Although the number of parameters in UFSeg islarger than that of DGT-NetL, the full tree of UFSeg has 147.2 million parameters which are less than thefull tree of DGT by 60 million parameters. This is due to the training algorithm used in UFSeg updatingnodes using an episode of tasks, not a single task (i.e., each node in UFSeg has more tasks than eachnode in DGT. Hence, leading to less number of nodes in the tree). Also, UFSeg architecture is morecomplex than DGT (i.e., more parameters), meaning the learning capacity of each node of UFSeg is higherthan the learning capacity of each node of DGT. Hence, each node can segment foreground objects from140Table 6.8: Model size and runtime analysis of UFSeg against other models. The number of parametersis in millions. * means our previous work.ModelFull model(Param. #)Inference model(Param. #)Initialization time(per video) FPSMiVOS [21] 53 53 0.00 11.2FgSegNet [63] 9.3 9.3 0.00 22.1REFNet-TBPI* [81] 94 5.8 45.13 8.6DGT-NetL* 207.6 4.6 9.69 11.4UFSeg 147.2 6.2 6.52 8.3a larger number of tasks. The average initialization time (i.e., time to find a suitable node and create atask-specific network) for UFSeg is less than that of DGT-NetL by around 3 seconds, as UFSeg has lessnumber of nodes, the greedy searching algorithm requires less time. However, the inference time of UFSegis slower than that of DGT. Because the inference time is affected by the model architecture, UFSeg hasmore layers and parameters to process the input.6.3.5 Unified foreground segmentation experimentIn this experiment, UFSeg is trained sequentially (continually) on all existing foreground segmentationdatasets. These datasets are YOVOS, CDNet, DAVIS16, DAVIS17, SegtrackV2, DUTS, and ExtendedComplex Scene Saliency Dataset (ECSSD). The trained UFSeg is evaluated by testing the model usingall these datasets without fine-tuning. The results of UFSeg are shown in Table 6.9. The trained UFSeg isalso tested using out-of-domain videos from different sources, such as webcams, public IP cameras, andrandom YouTube videos. The visual results of both in- and out-of-domain videos are displayed in Figure 6.2and Figure 6.3.141Table 6.9: Unified foreground segmentation experiment results for UFSeg.Dataset F CFpYOVOS 0.820 0.022CDNet 0.734 0.026SegtrackV2 0.854 0.031DAVIS16 0.863 0.021DAVIS17 0.832 0.019DUTS 0.861 0.017ECSSD 0.882 0.011Average 0.835 0.021Figure 6.2: sample results of UFSeg on in-domain videos. GT means the ground truth.142Figure 6.3: sample results of UFSeg on out-of-domain videos. GT means the ground truth.143Chapter 7ConclusionIn this thesis, we presented multiple novel deep-learning models. These models solve the problems thatexist in state-of-the-art foreground segmentation models. In chapter 3, multiple novel foreground segmen-tation deep-learning models are proposed. The first model is MODY-Net, a Y-shaped network that usesthe current frame and background frame as inputs. During training, the network learns to apply back-ground subtraction in the feature space. MODY-Net also implements an ensemble of multiple outputs ofthe network to reduce network uncertainty. However, this model outputs a low-resolution image becauseit is the average of low and high-resolution outputs. The performance of MODY-Net is 86%. The seconddeep-learning model is REFNet, a network designed to overcome some segmentation issues that exist inmost other foreground segmentation models. These problems are network uncertainty, low detailed output,and the inability of detecting tiny objects. REFNet’s parameters are optimized to detect both region andedges of the foreground objects. Hence, REFNet is encouraged to detect the object’s exterior silhouette,which leads to highly detailed output. REFNet modified the ensemble method introduced in MODY-Netsuch that a convolutional layer is used to combine all three outputs instead of applying average operation.This convolutional layer solved the problem of low-resolution output in MODY-Net. Additionally, REFNetused a multi-scale dense block and a scalar block to allow the network to detect all foreground objects inthe input despite their size. REFNet performance is 88.1% which is better than state-of-the-art foregroundsegmentation models in the f-measure by approximately 4%. The third deep-learning model is TransBlast,a transformer-based network that uses an augmented loss function with a subspace learning module as a144regularizer. The subspace learning module uses the first k eigenvectors of the SVD of the output featuremap from the last decoder block as an input. Then, this module maximizes the first two eigenvalues tomaximize the separation between foreground and background. The performance of TransBlast is 83.1%f-measure using the CDNet dataset.In chapter 4, a novel few-shot learning technique called KTNet for few-shot image classification andtwo novel few-shot learning techniques, ORIF-TR and FeSh-Net, are proposed to solve the problem oftraining with limited labeled data. These techniques allow the deep-learning model to learn with as little asone labeled frame while boosting the current state-of-the-art performance from 70.3% to 78.0% f-measure.KTNet is a hybrid meta-learning network that utilizes both metric-learning and optimization-based learning.The aim of KTNet is to autonomously increase the number of training images in the support set. This isdone by iteratively selecting images from the query set and adding them to the support set. Also, KTNethas the ability to detect whether the query set has unknown classes or not. Then, KTNet will clusterthe unknown classes into different classes and selects a support image from each different class, andadd them to the support set. The second few-shot learning technique is ORIF-TR which combines twoself-supervised learning techniques called masked auto-encoder and Barlow twins. ORIF-TR uses thesetechniques to leverage unlabeled data to learn useful information that can be transferred to new datasetsusing a small amount of labeled data. The performance of ORIF-TR is 72.9% and 84.3% using 1 and 5shots, respectively. This is better than the second-best model by 5.5% and 6% for 1 and 5 shots. Thelast few-shot learning technique is FeSh-Net, a meta-learning network designed specifically for few-shotlearning in foreground segmentation. FeSh-Net is training using a meta-learning algorithm to allow thenetwork to learn task-agnostic knowledge and how to transfer them to new data by fine-tuning the task-specific parameters. Also, an exemplar-based attention map is proposed to allow the network to knowwhich object is foreground and which is background through the input. Hence, the network does not haveto memorize all foreground objects in a given dataset. The performance of FeSh-Net is 81.9% and 87.3%in 1 and 5 shots, which is better than ORIF-TR by 7.1% and 2.2% in 1 and 5 shots.After that, in chapter 5, three novel lifelong learning techniques are proposed: TBPI, KDNet, and DGT.These techniques solve the problem of catastrophic forgetting that occurs when training on a large amountof data sequentially (i.e., training using one task at a time). In TBPI, the network parameters are split into145groups such that each group is assigned to segment a specific group of videos. By doing so, the parametersupdated by a group of videos are isolated from the parameters of other videos. Hence, catastrophic forget-ting does not occur, leading to increased performance. TBPI improves the performance of any foregroundsegmentation deep-learning model by an average of 2.4% in individual datasets. However, the strengthof TBPI appears when training using multiple datasets. The performance of TBPI using multiple datasetsis 18.7% better than traditional supervised learning. The second lifelong learning technique is KDNet, anetwork that utilizes a knowledge distillation loss function to learn the knowledge gained by other networkswithout having access to their training datasets. The performance of KDNet is 10.8% better than traditionalsupervised training but less than TBPI by 6.6%. The last technique is DGT, a dynamically growing tree ofsub-networks. The results showed that DGT solves the problem of catastrophic forgetting and outperformsthe other two techniques. DGT can also learn using limited labeled data and outperforms existing fore-ground segmentation models. DGT performance is higher than state-of-the-art (REFNet-TBPI) by 4.65%,and the inference time is 11.4 FPS.In the last chapter 6, a single model called UFSeg is proposed to achieve our research objective.UFSeg is a combination of the best model proposed in each chapter. These models are REFNet, FeSh-Net, and DGT. UFSeg outperforms state-of-the-art foreground segmentation models. Also, UFSeg solvesthe training with limited labeled data and catastrophic forgetting problems. Additionally, UFSeg can be usedon out-of-domain real-world examples such as frames from webcams or public IP cameras.The possible future directions for the work proposed in this thesis, 1) modifying UFSeg to be utilizedin a more complicated computer vision task, such as scene understanding, multiple object tracking andunderstanding their behavior, security surveillance, and anomaly behavior detection. 2) allowing UFSegto learn useful tasks from unlabeled videos (any online videos such as videos from youtube) using a self-supervised technique. This will allow the model to learn from many videos compared to the limited existingdatasets. The possible tasks that UFSeg can perform by learning from unlabeled data are video captioning,video summarizing, and video generation. Also, the trained model can be easily fine-tuned using supervisedlearning to perform any video-related computer vision tasks. 3) using UFSeg to learn multiple computervision tasks simultaneously (i.e., having a single unified model that can perform different tasks such asvideo colorization, foreground segmentation, object tracking, and video summarizing). 4) Optimizing the146architecture design of UFSeg to speed up the inference time. 5) Improving the routing algorithm of DGT toreduce the time taken to generate the task-specific network.147Bibliography[1] T. Akilan and Q. J. Wu. sendec: An improved image to image cnn for foreground localization. IEEETransactions on Intelligent Transportation Systems, 21(10):4435–4443, 2019.[2] S. Ali and M. Shah. Cocoa: tracking in aerial imagery. In Airborne Intelligence, Surveillance, Recon-naissance (ISR) Systems and Applications III, volume 6209, page 62090D. International Society forOptics and Photonics, 2006.[3] R. Aljundi, F. Babiloni, M. Elhoseiny, M. Rohrbach, and T. Tuytelaars. Memory aware synapses:Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision(ECCV), pages 139–154, 2018.[4] A. Antoniou and A. J. Storkey. Learning to learn by self-critique. In Advances in Neural InformationProcessing Systems, pages 9936–9946, 2019.[5] D. Arthur and S. Vassilvitskii. k-means++: The advantages of careful seeding. Technical report,Stanford, 2006.[6] S. Aslani and H. Mahdavi-Nasab. Optical flow based moving object detection and tracking for trafficsurveillance. International Journal of Electrical, Computer, Energetic, Electronic and CommunicationEngineering, 7(9):1252–1256, 2013.[7] M. Babaee, D. T. Dinh, and G. Rigoll. A deep convolutional neural network for video sequencebackground subtraction. Pattern Recognition, 76:635–649, 2018.148[8] L. Bao, B. Wu, and W. Liu. Cnn in mrf: Video object segmentation via inference in a cnn-basedhigher-order spatio-temporal mrf. In Proceedings of the IEEE conference on computer vision andpattern recognition, pages 5977–5986, 2018.[9] J. L. Barron, D. J. Fleet, and S. S. Beauchemin. Performance of optical flow techniques. Internationaljournal of computer vision, 12(1):43–77, 1994.[10] Y. Bendou, Y. Hu, R. Lafargue, G. Lioi, B. Pasdeloup, S. Pateux, and V. Gripon. Easy: Ensembleaugmented-shot y-shaped learning: State-of-the-art few-shot classification with simple ingredients.arXiv preprint arXiv:2201.09699, 2022.[11] S. Bianco, G. Ciocca, and R. Schettini. Combination of video change detection algorithms by geneticprogramming. IEEE Transactions on Evolutionary Computation, 21(6):914–928, 2017.[12] M. Braham, S. Pie´rard, and M. Van Droogenbroeck. Semantic background subtraction. In 2017 IEEEInternational Conference on Image Processing (ICIP), pages 4552–4556. IEEE, 2017.[13] M. Braham and M. Van Droogenbroeck. Deep background subtraction with scene-specific convolu-tional neural networks. In 2016 international conference on systems, signals and image processing(IWSSIP), pages 1–4. IEEE, 2016.[14] J. Bromley, I. Guyon, Y. LeCun, E. Sa¨ckinger, and R. Shah. Signature verification using a” siamese”time delay neural network. Advances in neural information processing systems, 6:737–744, 1993.[15] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end objectdetection with transformers. In European Conference on Computer Vision, pages 213–229. Springer,2020.[16] A. Chaudhry, P. K. Dokania, T. Ajanthan, and P. H. Torr. Riemannian walk for incremental learn-ing: Understanding forgetting and intransigence. In Proceedings of the European Conference onComputer Vision (ECCV), pages 532–547, 2018.149[17] C. Chen, G. Wang, C. Peng, X. Zhang, and H. Qin. Improved robust video saliency detection basedon long-term spatial-temporal information. IEEE transactions on image processing, 29:1090–1100,2019.[18] D. Chen, Y. Chen, Y. Li, F. Mao, Y. He, and H. Xue. Self-supervised learning for few-shot imageclassification. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and SignalProcessing (ICASSP), pages 1745–1749. IEEE, 2021.[19] H. Chen, H. Li, Y. Li, and C. Chen. Sparse spatial transformers for few-shot learning. arXiv preprintarXiv:2109.12932, 2021.[20] Z. Chen, Y. Fu, Y.-X. Wang, L. Ma, W. Liu, and M. Hebert. Image deformation meta-networks for one-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,pages 8680–8689, 2019.[21] H. K. Cheng, Y.-W. Tai, and C.-K. Tang. Modular interactive video object segmentation: Interaction-to-mask, propagation and difference-aware fusion. In Proceedings of the IEEE/CVF Conference onComputer Vision and Pattern Recognition, pages 5559–5568, 2021.[22] H. K. Cheng, Y.-W. Tai, and C.-K. Tang. Rethinking space-time networks with improved memory cov-erage for efficient video object segmentation. Advances in Neural Information Processing Systems,34:11781–11794, 2021.[23] N. Cristianini, J. Shawe-Taylor, and J. S. Kandola. Spectral kernel methods for clustering. In NIPS,volume 14, pages 649–655, 2001.[24] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchicalimage database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.[25] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min-derer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognitionat scale. arXiv preprint arXiv:2010.11929, 2020.150[26] A. ElTantawy and M. S. Shehata. Krmaro: Aerial detection of small-size ground moving objects usingkinematic regularization and matrix rank optimization. IEEE Transactions on Circuits and Systemsfor Video Technology, 29(6):1672–1686, 2018.[27] A. Eltantawy and M. S. Shehata. An accelerated sequential pcp-based method for ground-movingobjects detection from aerial videos. IEEE Transactions on Image Processing, 28(12):5991–6006,2019.[28] D.-P. Fan, W. Wang, M.-M. Cheng, and J. Shen. Shifting more attention to video salient objectdetection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pages 8554–8564, 2019.[29] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks.In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1126–1135. JMLR. org, 2017.[30] R. M. French. Catastrophic forgetting in connectionist networks. Encyclopedia of cognitive science,2006.[31] K. Guo, P. Ishwar, and J. Konrad. Action recognition using sparse representation on covariancemanifolds of optical flow. In 2010 7th IEEE international conference on advanced video and signalbased surveillance, pages 188–195. IEEE, 2010.[32] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedingsof the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.[33] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedingsof the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.[34] S. X. Hu, P. Moreno, Y. Xiao, X. Shen, G. Obozinski, N. Lawrence, and A. Damianou. Empiricalbayes transductive meta-learning with synthetic gradients. In International Conference on LearningRepresentations (ICLR), 2020.151[35] J. Huerta-Cepas, F. Serra, and P. Bork. Ete 3: reconstruction, analysis, and visualization of phyloge-nomic data. Molecular biology and evolution, 33(6):1635–1638, 2016.[36] E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox. Flownet 2.0: Evolution of opticalflow estimation with deep networks. In Proceedings of the IEEE conference on computer vision andpattern recognition, pages 2462–2470, 2017.[37] G.-P. Ji, K. Fu, Z. Wu, D.-P. Fan, J. Shen, and L. Shao. Full-duplex strategy for video object seg-mentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages4922–4933, 2021.[38] Y. Ji, H. Zhang, Z. Jie, L. Ma, and Q. J. Wu. Casnet: A cross-attention siamese network for videosalient object detection. IEEE transactions on neural networks and learning systems, 32(6):2676–2690, 2020.[39] S. Jiang and X. Lu. Wesambe: A weight-sample-based method for background subtraction. IEEETransactions on Circuits and Systems for Video Technology, 28(9):2105–2115, 2017.[40] W. Jiang, K. Huang, J. Geng, and X. Deng. Multi-scale metric learning for few-shot learning. IEEETransactions on Circuits and Systems for Video Technology, 31(3):1091–1102, 2020.[41] J. Johnander, M. Danelljan, E. Brissman, F. S. Khan, and M. Felsberg. A generative appearancemodel for end-to-end video object segmentation. In Proceedings of the IEEE/CVF Conference onComputer Vision and Pattern Recognition, pages 8953–8962, 2019.[42] D. Kang, H. Kwon, J. Min, and M. Cho. Relational embedding for few-shot classification. In Proceed-ings of the IEEE/CVF International Conference on Computer Vision, pages 8822–8833, 2021.[43] A. Khoreva, R. Benenson, E. Ilg, T. Brox, and B. Schiele. Lucid data dreaming for video objectsegmentation. International Journal of Computer Vision, 127(9):1175–1197, 2019.[44] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan,T. Ramalho, A. Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks.Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.152[45] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.[46] H. Le, V. Nguyen, C.-P. Yu, and D. Samaras. Geodesic distance histogram feature for video segmen-tation. In Asian Conference on Computer Vision, pages 275–290. Springer, 2016.[47] T.-N. Le and A. Sugimoto. Deeply supervised 3d recurrent fcn for salient object detection in videos.In BMVC, volume 1, page 3, 2017.[48] T.-N. Le and A. Sugimoto. Video salient object detection using spatiotemporal deep features. IEEETransactions on Image Processing, 27(10):5002–5015, 2018.[49] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recog-nition. Proceedings of the IEEE, 86(11):2278–2324, 1998.[50] D. H. Lee and S.-Y. Chung. Unsupervised embedding adaptation via early-stage feature reconstruc-tion for few-shot classification. In International Conference on Machine Learning, pages 6098–6108.PMLR, 2021.[51] K. Lee, S. Maji, A. Ravichandran, and S. Soatto. Meta-learning with differentiable convex optimiza-tion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages10657–10665, 2019.[52] S.-h. Lee, G.-c. Lee, J. Yoo, and S. Kwon. Wisenetmd: Motion detection using dynamic backgroundregion analysis. Symmetry, 11(5):621, 2019.[53] F. Li, T. Kim, A. Humayun, D. Tsai, and J. M. Rehg. Video segmentation by tracking many figure-ground segments. In Proceedings of the IEEE International Conference on Computer Vision, pages2192–2199, 2013.[54] G. Li and Y. Yu. Deep contrast learning for salient object detection. In Proceedings of the IEEEconference on computer vision and pattern recognition, pages 478–487, 2016.[55] H. Li, G. Chen, G. Li, and Y. Yu. Motion guided attention for video salient object detection. In Pro-ceedings of the IEEE/CVF International Conference on Computer Vision, pages 7274–7283, 2019.153[56] H. Li, D. Eigen, S. Dodge, M. Zeiler, and X. Wang. Finding task-relevant features for few-shot learningby category traversal. In Proceedings of the IEEE Conference on Computer Vision and PatternRecognition, pages 1–10, 2019.[57] S. Li, B. Seybold, A. Vorobyov, X. Lei, and C.-C. J. Kuo. Unsupervised video object segmentationwith motion-based bilateral networks. In Proceedings of the European conference on computer vision(ECCV), pages 207–223, 2018.[58] W. Li, L. Wang, J. Xu, J. Huo, Y. Gao, and J. Luo. Revisiting local descriptor based image-to-classmeasure for few-shot learning. In Proceedings of the IEEE Conference on Computer Vision andPattern Recognition, pages 7260–7268, 2019.[59] Y. Li, F. Xu, and C.-G. Lee. Self-supervised meta learning generative adversarial network for few-shot fault diagnosis of hoisting system with limited data. IEEE Transactions on Industrial Informatics,2022.[60] Z. Li and D. Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machineintelligence, 40(12):2935–2947, 2017.[61] Y. Lifchitz, Y. Avrithis, S. Picard, and A. Bursuc. Dense classification and implanting for few-shotlearning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,pages 9258–9267, 2019.[62] L. Lim and H. Keles. Foreground segmentation using a triplet convolutional neural network for multi-scale feature encoding. arxiv 2018. arXiv preprint arXiv:1801.02225.[63] L. A. Lim and H. Y. Keles. Learning multi-scale features for foreground segmentation. Pattern Analysisand Applications, 23(3):1369–1380, 2020.[64] J. Liu, L. Song, and Y. Qin. Prototype rectification for few-shot learning. In European Conference onComputer Vision (ECCV), 2020.154[65] N. Liu and J. Han. Dhsnet: Deep hierarchical saliency network for salient object detection. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 678–686,2016.[66] Y. Liu, Q. Sun, A.-A. Liu, Y. Su, B. Schiele, and T.-S. Chua. Lcc: Learning to customize and combineneural networks for few-shot learning. arXiv preprint arXiv:1904.08479, 2019.[67] Y. Liu, H. Zhang, L. Chai, and Y. Qi. A foreground segmentation method for mobile image retrievalsystem. In 2012 3rd IEEE International Conference on Network Infrastructure and Digital Content,pages 492–497. IEEE, 2012.[68] D. Lopez-Paz and M. Ranzato. Gradient episodic memory for continual learning. arXiv preprintarXiv:1706.08840, 2017.[69] J. MacQueen et al. Some methods for classification and analysis of multivariate observations. InProceedings of the fifth Berkeley symposium on mathematical statistics and probability, volume 1,pages 281–297. Oakland, CA, USA, 1967.[70] A. Mallya, D. Davis, and S. Lazebnik. Piggyback: Adapting a single network to multiple tasks bylearning to mask weights. In Proceedings of the European Conference on Computer Vision (ECCV),pages 67–82, 2018.[71] A. Mallya and S. Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning.In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7765–7773, 2018.[72] K.-K. Maninis, S. Caelles, Y. Chen, J. Pont-Tuset, L. Leal-Taixe´, D. Cremers, and L. Van Gool. Videoobject segmentation without temporal information. IEEE transactions on pattern analysis and ma-chine intelligence, 41(6):1515–1530, 2018.[73] M. Masana, X. Liu, B. Twardowski, M. Menta, A. D. Bagdanov, and J. van de Weijer. Class-incremental learning: survey and performance evaluation. arXiv preprint arXiv:2010.15277, 2020.155[74] J. Mei, M. Wang, Y. Lin, Y. Yuan, and Y. Liu. Transvos: Video object segmentation with transformers.arXiv preprint arXiv:2106.00588, 2021.[75] N. Mishra, M. Rohaninejad, X. Chen, and P. Abbeel. A simple neural attentive meta-learner. ICLR,2018.[76] T. Munkhdalai, X. Yuan, S. Mehri, and A. Trischler. Rapid adaptation with conditionally shifted neu-rons. ICML, 2018.[77] Y. Nakaya and H. Harashima. Motion compensation based on spatial transformations. IEEE Trans-actions on circuits and systems for video technology, 4(3):339–356, 1994.[78] S. W. Oh, J.-Y. Lee, N. Xu, and S. J. Kim. Video object segmentation using space-time memorynetworks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages9226–9235, 2019.[79] B. Oreshkin, P. R. Lo´pez, and A. Lacoste. Tadam: Task dependent adaptive metric for improvedfew-shot learning. In Advances in Neural Information Processing Systems, pages 721–731, 2018.[80] I. Osman, M. Abdelpakey, and M. S. Shehata. Transblast: Self-supervised learning using augmentedsubspace with transformer for background/foreground separation. In Proceedings of the IEEE/CVFInternational Conference on Computer Vision, pages 215–224, 2021.[81] I. Osman, A. Eltantawy, and M. S. Shehata. Task-based parameter isolation for foreground segmen-tation without catastrophic forgetting using multi-scale region and edges fusion network. Image andVision Computing, 113:104248, 2021.[82] I. Osman and M. S. Shehata. Few-shot learning network for moving object detection using exemplar-based attention map. In 2022 IEEE International Conference on Image Processing (ICIP), pages1056–1060. IEEE, 2022.[83] I. Osman and M. S. Shehata. Few-shot learning network for out-of-distribution image classification.IEEE Transactions on Artificial Intelligence, 1(01):1–13, 2022.156[84] I. Osman and M. S. Shehata. Self-supervised learning for foreground segmentation with a fewamount of labeled images using transformers. In Proceedings of the 8th International Conference onAdvanced Intelligent Systems and Informatics 2022, pages 14–25. Springer, 2022.[85] I. I. Osman and M. S. Shehata. Modsiam: Moving object detection using siamese networks. In 2020IEEE Canadian Conference on Electrical and Computer Engineering (CCECE), pages 1–6. IEEE,2020.[86] I. I. Osman and M. S. Shehata. Mody-net: Moving object detection using multiscale output ensembley-network. IEEE Canadian Journal of Electrical and Computer Engineering, 44(4):491–496, 2021.[87] H. Park, J. Yoo, S. Jeong, G. Venkatesh, and N. Kwak. Learning dynamic network using a reuse gatefunction in semi-supervised video object segmentation. In Proceedings of the IEEE/CVF Conferenceon Computer Vision and Pattern Recognition, pages 8405–8414, 2021.[88] P. W. Patil, K. M. Biradar, A. Dudhane, and S. Murala. An end-to-end edge aggregation network formoving object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision andPattern Recognition, pages 8149–8158, 2020.[89] F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine-Hornung. A bench-mark dataset and evaluation methodology for video object segmentation. In Proceedings of the IEEEconference on computer vision and pattern recognition, pages 724–732, 2016.[90] P. I. Pierre-Marc Jodoin, Janusz Konrad and F. Porikli. Results for cd.net 2014.[91] J. Pont-Tuset, F. Perazzi, S. Caelles, P. Arbela´ez, A. Sorkine-Hornung, and L. Van Gool. The 2017davis challenge on video object segmentation. arXiv preprint arXiv:1704.00675, 2017.[92] M. Pouzet, P. Bonnin, J. Laneurit, and C. Tessier. A robust real-time image algorithm for moving targetdetection from unmanned aerial vehicles (uav). In 2014 11th International Conference on Informaticsin Control, Automation and Robotics (ICINCO), volume 1, pages 266–273. IEEE, 2014.157[93] S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert. icarl: Incremental classifier and represen-tation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition,pages 2001–2010, 2017.[94] A. Rosenfeld and J. K. Tsotsos. Incremental learning through deep adaptation. IEEE transactionson pattern analysis and machine intelligence, 42(3):651–663, 2018.[95] E. Rosten and T. Drummond. Machine learning for high-speed corner detection. In European con-ference on computer vision, pages 430–443. Springer, 2006.[96] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla,M. Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of com-puter vision, 115(3):211–252, 2015.[97] A. A. Rusu, D. Rao, J. Sygnowski, O. Vinyals, R. Pascanu, S. Osindero, and R. Hadsell. Meta-learningwith latent embedding optimization. ICLR, 2019.[98] V. Saligrama and Z. Chen. Video anomaly detection based on local statistical aggregates. In 2012IEEE Conference on Computer Vision and Pattern Recognition, pages 2112–2119. IEEE, 2012.[99] D. Sculley. Web-scale k-means clustering. In Proceedings of the 19th international conference onWorld wide web, pages 1177–1178, 2010.[100] H. Seong, J. Hyun, and E. Kim. Kernelized memory network for video object segmentation. InEuropean Conference on Computer Vision, pages 629–645. Springer, 2020.[101] H. Seong, S. W. Oh, J.-Y. Lee, S. Lee, S. Lee, and E. Kim. Hierarchical memory matching network forvideo object segmentation. In Proceedings of the IEEE/CVF International Conference on ComputerVision, pages 12889–12898, 2021.[102] H. Shen, S. Li, C. Zhu, H. Chang, and J. Zhang. Moving object detection in aerial video based onspatiotemporal saliency. Chinese Journal of Aeronautics, 26(5):1211–1217, 2013.[103] H. Shin, J. K. Lee, J. Kim, and J. Kim. Continual learning with deep generative replay. arXiv preprintarXiv:1705.08690, 2017.158[104] J. Snell, K. Swersky, and R. S. Zemel. Prototypical networks for few-shot learning. In NIPS, 2017.[105] H. Song, W. Wang, S. Zhao, J. Shen, and K.-M. Lam. Pyramid dilated deeper convlstm for videosalient object detection. In Proceedings of the European conference on computer vision (ECCV),pages 715–731, 2018.[106] P.-L. St-Charles, G.-A. Bilodeau, and R. Bergevin. Subsense: A universal change detection methodwith local adaptive sensitivity. IEEE Transactions on Image Processing, 24(1):359–373, 2014.[107] P.-L. St-Charles, G.-A. Bilodeau, and R. Bergevin. A self-adjusting approach to change detectionbased on background word consensus. In 2015 IEEE winter conference on applications of computervision, pages 990–997. IEEE, 2015.[108] Y. Su, J. Deng, R. Sun, G. Lin, and Q. Wu. A unified transformer framework for group-based seg-mentation: Co-segmentation, co-saliency detection and video salient object detection. arXiv preprintarXiv:2203.04708, 2022.[109] Q. Sun, Y. Liu, T.-S. Chua, and B. Schiele. Meta-transfer learning for few-shot learning. In Proceed-ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 403–412, 2019.[110] F. Sung, Y. Yang, L. Zhang, T. Xiang, P. H. Torr, and T. M. Hospedales. Learning to compare:Relation network for few-shot learning. In Proceedings of the IEEE Conference on Computer Visionand Pattern Recognition, pages 1199–1208, 2018.[111] F. Sung, Y. Yang, L. Zhang, T. Xiang, P. H. Torr, and T. M. Hospedales. Learning to compare: Relationnetwork for few-shot learning. In Proceedings of the IEEE conference on computer vision and patternrecognition, pages 1199–1208, 2018.[112] Y.-L. Sung, V. Nair, and C. A. Raffel. Training neural networks with fixed sparse masks. Advances inNeural Information Processing Systems, 34:24193–24205, 2021.[113] O. Tezcan, P. Ishwar, and J. Konrad. Bsuv-net: a fully-convolutional neural network for backgroundsubtraction of unseen videos. In The IEEE Winter Conference on Applications of Computer Vision,pages 2774–2783, 2020.159[114] O. Tezcan, P. Ishwar, and J. Konrad. Bsuv-net: A fully-convolutional neural network for backgroundsubtraction of unseen videos. In Proceedings of the IEEE/CVF Winter Conference on Applications ofComputer Vision, pages 2774–2783, 2020.[115] Z. Tian, R. Ramakrishnan, and L. M. Birch. An efficient data clustering method for very largedatabases. In Proc of the ACM SIGMOD International Conference on Management of Data. Mon-tre—al, Canada, pages 103–114, 1996.[116] D. Ulyanov, A. Vedaldi, and V. Lempitsky. Instance normalization: The missing ingredient for faststylization. arXiv preprint arXiv:1607.08022, 2016.[117] G. M. Van de Ven, H. T. Siegelmann, and A. S. Tolias. Brain-inspired replay for continual learningwith artificial neural networks. Nature communications, 11(1):4069, 2020.[118] L. van der Maaten and G. E. Hinton. Visualizing data using t-sne. Journal of Machine LearningResearch, 9:2579–2605, 2008.[119] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-ucsd birds-200-2011dataset. 2011.[120] L. Wang, L. Wang, H. Lu, P. Zhang, and X. Ruan. Saliency detection with recurrent fully convolutionalnetworks. In European conference on computer vision, pages 825–841. Springer, 2016.[121] R. Wang, F. Bunyak, G. Seetharaman, and K. Palaniappan. Static and moving object detection usingflux tensor with split gaussian models. In Proceedings of the IEEE Conference on Computer Visionand Pattern Recognition Workshops, pages 414–418, 2014.[122] Y. Wang, P.-M. Jodoin, F. Porikli, J. Konrad, Y. Benezeth, and P. Ishwar. Cdnet 2014: An expandedchange detection benchmark dataset. In Proceedings of the IEEE conference on computer visionand pattern recognition workshops, pages 387–394, 2014.[123] Y. Wang, Z. Luo, and P.-M. Jodoin. Interactive deep learning method for segmenting moving objects.Pattern Recognition Letters, 96:66–75, 2017.160[124] Z. Wang, J. Xu, L. Liu, F. Zhu, and L. Shao. Ranet: Ranking attention network for fast video objectsegmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages3978–3987, 2019.[125] D. Wertheimer and B. Hariharan. Few-shot learning with localization in realistic settings. In Pro-ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6558–6567,2019.[126] N. Wojke, A. Bewley, and D. Paulus. Simple online and realtime tracking with a deep associationmetric. In 2017 IEEE international conference on image processing (ICIP), pages 3645–3649. IEEE,2017.[127] H. Xie, H. Yao, S. Zhou, S. Zhang, and W. Sun. Efficient regional memory network for video ob-ject segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and PatternRecognition, pages 1286–1295, 2021.[128] N. Xu, L. Yang, Y. Fan, D. Yue, Y. Liang, J. Yang, and T. Huang. Youtube-vos: A large-scale videoobject segmentation benchmark. arXiv preprint arXiv:1809.03327, 2018.[129] S. Xu, D. Liu, L. Bao, W. Liu, and P. Zhou. Mhp-vos: Multiple hypotheses propagation for videoobject segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and PatternRecognition, pages 314–323, 2019.[130] Z. Yang, Y. Wei, and Y. Yang. Collaborative video object segmentation by foreground-backgroundintegration. In European Conference on Computer Vision, pages 332–348. Springer, 2020.[131] J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny. Barlow twins: Self-supervised learning viaredundancy reduction. arXiv preprint arXiv:2103.03230, 2021.[132] F. Zenke, B. Poole, and S. Ganguli. Continual learning through synaptic intelligence. In InternationalConference on Machine Learning, pages 3987–3995. PMLR, 2017.161[133] H. Zhang, J. Zhang, and P. Koniusz. Few-shot learning via saliency-guided hallucination of samples.In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2770–2779, 2019.[134] L. Zhang, Z. Lin, J. Zhang, H. Lu, and Y. He. Fast video object segmentation via dynamic targetingnetwork. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages5582–5591, 2019.[135] L. Zhang, L. Zuo, Y. Du, and X. Zhen. Learning to adapt with memory for probabilistic few-shotlearning. IEEE Transactions on Circuits and Systems for Video Technology, 31(11):4283–4292, 2021.[136] X. Zhang, J. Zou, K. He, and J. Sun. Accelerating very deep convolutional networks for classificationand detection. IEEE transactions on pattern analysis and machine intelligence, 38(10):1943–1955,2015.[137] T. Zhuo, Z. Cheng, P. Zhang, Y. Wong, and M. Kankanhalli. Unsupervised online video object seg-mentation with motion property understanding. IEEE Transactions on Image Processing, 29:237–249, 2019.162
Thesis/Dissertation
eng
Computer Science
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Shehata, Mohamed S.
Continual foreground segmentation using limited data
Text