Science, Irving K. Barber Faculty of (Okanagan)
Computer Science, Mathematics, Physics and Statistics, Department of (Okanagan)
DSpace
UBCO
Abdelfattah, Islam
Master of Science - MSc
University of British Columbia
Video segmentation involves isolating specific objects or the foreground from the background in a video, for example, separating people or vehicles from the rest of the scene. This task is crucial in many applications such as video editing, surveillance, autonomous vehicles, and augmented reality. Advanced methods often utilize Deep Learning (DL) to manage the complexities of video sequences, including motion and occlusion. Despite recent progress, current state-of-the-art methods face challenges such as catastrophic forgetting, poor domain generalization, and the loss of object details when objects are occluded by other objects. This thesis addresses these critical challenges in video segmentation by proposing innovative methodologies to enhance segmentation accuracy, robustness, and domain generalization. We introduce a Knowledge Distillation Network (KDNet) and a new training technique to combat catastrophic forgetting and improve domain generalization by leveraging advanced learning paradigms such as federated learning, continual learning, knowledge distillation, and few-shot learning. This approach yields significant improvements across diverse datasets while ensuring data privacy. Additionally, we propose a Mask Enhancement Model (MEM) to mitigate the loss of object details by integrating a foundational model output and employing mask fusion techniques, resulting in better segmentation masks. Experimental results demonstrate KDNet’s ability to achieve state-of-the art domain generalization performance on multiple datasets in foreground segmentation and adapt to new unseen domains using as few as one labelled image (one-shot learning). MEM also proves effective, versatile, and robust performance, achieving state-of-the-art results in Semi-Supervised Video Object Segmentation (S-SVOS).
Advancing Video Segmentation byIntegrating Advanced LearningParadigmsbyIslam AbdelfattahM.Eng., University of Ottawa, 2022B.Sc., Hons., Arab Academy for Science, Technology & Maritime Transport , 2019A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFMASTER OF SCIENCEinThe College of Graduate Studies(Computer Science)THE UNIVERSITY OF BRITISH COLUMBIA(Okanagan)October 2024© Islam Abdelfattah 2024The following individuals certify that they have read, and recommend tothe College of Graduate Studies for acceptance, a thesis entitled:Advancing Video Segmentation by Integrating Advanced Learning Paradigmssubmitted by Islam Abdelfattah in partial fulfillment of the requirementsof the degree of Master of ScienceExamining Committee:Mohamed S. Shehata, The University of British Columbia, Okanagan CampusSupervisorFatemeh Fard, The University of British Columbia, Okanagan CampusSupervisory Committee MemberGema Rodríguez-Pérez, The University of British Columbia, Okanagan CampusSupervisory Committee MemberShahria Alam, The University of British Columbia, Okanagan CampusUniversity ExamineriiAbstractVideo segmentation involves isolating specific objects or the foreground fromthe background in a video, for example, separating people or vehicles fromthe rest of the scene. This task is crucial in many applications such as videoediting, surveillance, autonomous vehicles, and augmented reality. Advancedmethods often utilize Deep Learning (DL) to manage the complexities ofvideo sequences, including motion and occlusion. Despite recent progress,current state-of-the-art methods face challenges such as catastrophic forget-ting, poor domain generalization, and the loss of object details when objectsare occluded by other objects.This thesis addresses these critical challenges in video segmentation byproposing innovative methodologies to enhance segmentation accuracy, ro-bustness, and domain generalization. We introduce a Knowledge DistillationNetwork (KDNet) and a new training technique to combat catastrophic for-getting and improve domain generalization by leveraging advanced learningparadigms such as federated learning, continual learning, knowledge distilla-tion, and few-shot learning. This approach yields significant improvementsacross diverse datasets while ensuring data privacy. Additionally, we proposea Mask Enhancement Model (MEM) to mitigate the loss of object details byintegrating a foundational model output and employing mask fusion tech-niques, resulting in better segmentation masks.Experimental results demonstrate KDNet’s ability to achieve state-of-the-art domain generalization performance on multiple datasets in foregroundsegmentation and adapt to new unseen domains using as few as one labelledimage (one-shot learning). MEM also proves effective, versatile, and robustperformance, achieving state-of-the-art results in Semi-Supervised Video Ob-ject Segmentation (S-SVOS).iiiLay SummaryVideo segmentation separates important objects, such as people or cars, fromthe background in videos, which is essential for tasks such as video editing,security monitoring, and self-driving cars. Current methods that are usingDeep Learning face challenges like forgetting learned information, poor per-formance on new videos, and losing object details when occluded.This thesis introduces KDNet and a new training technique to preventforgetting and enhance domain generalization performance using federatedlearning, knowledge distillation, continual learning, and few-shot learning,ensuring better results and data privacy. It also presents MEM, which pre-serves object details by combining outputs from different state-of-the-artmodels for superior segmentation masks.Experiments show that KDNet achieves state-of-the-art performance indomain generalization on multiple datasets in foreground segmentation andexcels in adapting to new unseen video types with minimal labelled data,and MEM achieves state-of-the-art results in semi-supervised video objectsegmenting.ivPrefaceChapters 3 and 4 of this thesis contain content from work submitted toconferences in which Islam Abdelfattah was both primary and co-author.Other co-authors for published or submitted work include Islam Osman andMohamed S. Shehata.Chapter 3 contains a version of content from "Domain Generalization forForeground Segmentation Using Federated Learning" which was published in"Bebis, G., et al. Advances in Visual Computing. ISVC 2023. Lecture Notesin Computer Science, vol 14361. Springer, Cham". This work has been acollaborative work between Islam Osman and Islam Abdelfattah. MohamedS. Shehata supervised all research and technical contributions and reviewedthe final manuscript before submission. The initial concept for this researchwas proposed by Islam Osman, and the entirety of the work—including ex-periments, code development, and paper writing—was carried out throughequal collaboration.Chapter 4 contains a version of content from "MEM: Mask EnhancementModel for Video Object Segmentation" which was submitted and acceptedby the International Symposium on Visual Computing (ISVC). Islam Ab-delfattah led all research for this work. Mohamed S. Shehata oversaw thesupervision of the research and reviewed the final manuscript before submis-sion.Grammarly and ChatGPT were occasionally used to improve writing dur-ing the preparation of this thesis (e.g., for grammar and typo checks). Noother generative AI tools or technologies were employed for data analysis,writing, editing, or any other part of this document’s development.vTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . viList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiGlossary of Notation . . . . . . . . . . . . . . . . . . . . . . . . . . xiiAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . xivDedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Research Motivation . . . . . . . . . . . . . . . . . . . . . . . 21.2 Research Objective . . . . . . . . . . . . . . . . . . . . . . . 31.3 Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.4 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . 52 Background and Related Work . . . . . . . . . . . . . . . . . 62.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.2 Video Segmentation Tasks . . . . . . . . . . . . . . . . . . . 72.2.1 Foreground Segmentation . . . . . . . . . . . . . . . . 82.2.1.1 Problem definition . . . . . . . . . . . . . . 82.2.1.2 Foreground Segmentation Related Work . . 82.2.1.3 Foreground Segmentation Datasets . . . . . 92.2.1.4 Evaluation Metrics . . . . . . . . . . . . . . 102.2.2 Video Object Segmentation . . . . . . . . . . . . . . . 11vi2.2.2.1 Problem definition . . . . . . . . . . . . . . 112.2.2.2 Video Object segmentation Related Work . 112.2.2.3 Video Object Segmentation Datasets . . . . 132.2.2.4 Evaluation Metrics . . . . . . . . . . . . . . 143 KDNet: Domain Generalization for Foreground Segmenta-tion Using Federated Learning . . . . . . . . . . . . . . . . . 153.1 Overview and Motivation . . . . . . . . . . . . . . . . . . . . 153.1.1 Domain Generalization . . . . . . . . . . . . . . . . . 153.1.2 Continual Learning . . . . . . . . . . . . . . . . . . . 163.1.3 Federated Learning . . . . . . . . . . . . . . . . . . . 163.1.4 Knowledge Distillation . . . . . . . . . . . . . . . . . 173.1.5 Few-Shot Learning . . . . . . . . . . . . . . . . . . . . 183.2 Proposed Work . . . . . . . . . . . . . . . . . . . . . . . . . . 193.2.1 Model Architecture . . . . . . . . . . . . . . . . . . . 193.2.2 Training Technique . . . . . . . . . . . . . . . . . . . 203.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . 223.3.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . 223.3.2 Implementation Details . . . . . . . . . . . . . . . . . 223.3.3 Traditional Foreground Segmentation Results . . . . . 233.3.4 Domain Generalization Results . . . . . . . . . . . . . 243.3.5 Few-shot Results . . . . . . . . . . . . . . . . . . . . . 263.4 Limitations and possible improvements . . . . . . . . . . . . 293.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304 MEM: Mask Enhancement Model . . . . . . . . . . . . . . . 314.1 Overview and Motivation . . . . . . . . . . . . . . . . . . . . 314.2 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . 324.2.1 VOS with SAM . . . . . . . . . . . . . . . . . . . . . 324.2.2 Architecture . . . . . . . . . . . . . . . . . . . . . . . 334.2.2.1 MEM . . . . . . . . . . . . . . . . . . . . . . 334.2.2.2 Modified Res-Block . . . . . . . . . . . . . . 344.2.2.3 Stem Stage . . . . . . . . . . . . . . . . . . . 344.2.2.4 Mask Enhancer . . . . . . . . . . . . . . . . 344.2.2.5 Fusion Block . . . . . . . . . . . . . . . . . . 354.2.2.6 Prediction Heads . . . . . . . . . . . . . . . 354.2.2.7 Post Processing Block . . . . . . . . . . . . 354.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . 364.3.1 Implementation Details . . . . . . . . . . . . . . . . . 364.3.1.1 Optimizeation . . . . . . . . . . . . . . . . . 36vii4.3.1.2 Loss . . . . . . . . . . . . . . . . . . . . . . 364.3.1.3 Training . . . . . . . . . . . . . . . . . . . . 364.3.1.4 Testing . . . . . . . . . . . . . . . . . . . . . 364.3.2 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . 374.3.3 Unsupervised VOS SAM Results . . . . . . . . . . . . 374.3.4 MEM Results . . . . . . . . . . . . . . . . . . . . . . 384.3.5 Ablation Study . . . . . . . . . . . . . . . . . . . . . 404.3.5.1 Bounding Box Regression Head and ObjectTracking Model . . . . . . . . . . . . . . . . 404.3.5.2 Get Bounding Box From VOS Model Output 404.3.5.3 Modified SAM Results . . . . . . . . . . . . 414.4 Limitations and Possible Improvements . . . . . . . . . . . . 414.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425 Conclusions and Future Work . . . . . . . . . . . . . . . . . . 43Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45AppendixA Implementation and Reproducibility Details . . . . . . . . . 54viiiList of Tables3.1 Traditional foreground segmentation results of KDNet com-pared with state-of-the-art models on the DAVIS16 dataset.The number inside the circle defines the order of the modelperformance concerning each evaluation metric. . . . . . . . . 243.2 Traditional foreground segmentation results of KDNet com-pared with state-of-the-art models on the SegTrackV2 dataset.The number inside the circle defines the order of the modelperformance concerning the evaluation metric. . . . . . . . . . 243.3 Comparison of KDNet against state-of-the-art methods whentrained using the three datasets CDNet, DAVIS16, Seg-TrackV2 jointly (i.e., all three datasets are combined andconsidered as one large dataset). The values in the table rep-resent the F-measure. . . . . . . . . . . . . . . . . . . . . . . . 253.4 Comparison of KDNet against state-of-the-art methods whentrained using the three datasets CDNet, DAVIS16, Seg-TrackV2 sequentially (i.e., the models are fine-tuned ondatasets sequentially. After the model is trained using the lastdataset, it is evaluated using the last dataset and all previousones). The values in the table represent the F-measure. . . . . 263.5 Few-shot foreground segmentation results of KDNet comparedto state-of-the-art methods on DAVIS17. . . . . . . . . . . . . 294.1 Comparison between unsupervised VOS J&F scores for thestate-of-the-art models and SAM for VOS. . . . . . . . . . . 374.2 Comparison of VOS state-of-the-art models on two differentdatasets DAVIS-17 and YouTube VOS . . . . . . . . . . . . . 39ix4.3 The highest IOU results are reported in this table to show thedifference between all three tested methods. Several experi-ments have been conducted to develop a better bounding boxregression model and the reported result was the best achievedresults for this method. For the OSTrack model, it has beenused without any training from our side. These results arereported after using the training set of DAVIS-17 . . . . . . . 41xList of Figures2.1 Foreground segmentation vs video object segmentation. Im-ages are from DAVIS 17 Dataset [1] . . . . . . . . . . . . . . . 73.1 The architecture of KDNet. On the left is the server networkwith multi-head outputs, and on the right are the client net-works. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193.2 Training procedure of KDNet. . . . . . . . . . . . . . . . . . 203.3 The average loss of clients of all three datasets after each fed-erated learning cycle. . . . . . . . . . . . . . . . . . . . . . . 273.4 Sample results of KDNet from domain generalization experi-ment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284.1 Two different frames are selected from two different video se-quences that show our model’s mask in comparison to SegmentAnything Model (SAM) and Cutie . . . . . . . . . . . . . . . 324.2 Getting SAM response after extracting bounding box fromVOS model. The process is performed in an iterative wayoverall video sequence frames. . . . . . . . . . . . . . . . . . . 334.3 MEM model architecture. Each Mask Enhancer block takesthe current frame and model mask as input. The Fusion Blocktakes both Mask Enhancer features and both models’ masksas inputs. y1, y2, and y3 are used only during training timewhile MEM-Mask and MEM-ORAND are the model outputduring inference time. . . . . . . . . . . . . . . . . . . . . . . 334.4 Detailed view of Mask Enhancer components. The Stem stagehas shared weights between both Mask Enhancers. . . . . . . 344.5 Bounding Box Regression Head architecture. Both SAM En-coders have shared weights. Both Encoder embeddings arefed to a fully connected layer followers by ReLU then anotherfully connected layer followed by a Sigmoid to calculate a nor-malized bounding box corner coordinates values. . . . . . . . 40xiGlossary of NotationKDNet Knowledge Distillation NetworkMEM Mask Enhancement ModelAI Artificial IntelligenceML Machine LearningDL Deep LearningGMM Gaussian Mixture ModelMRFs Markov Random FieldsCRFs Conditional Random FieldsSVMs Support Vector MachinesCNNs Convolutional Neural NetworksFCNs Fully Convolutional NetworksSTCN Space-Time Correspondence NetworkDAVIS16 Densely Annotated VIdeo Segmentation 2016DAVIS17 Densely Annotated VIdeo Segmentation 2017CDNet ChangeDetection.NetVOS Video Object SegmentationIOU Intersection Over UnionSVOS Supervised Video Object SegmentationRNNs Recurrent Neural NetworksS-SVOS Semi-Supervised Video Object SegmentationxiiUVOS Unsupervised Video Object SegmentationGANs Generative Adversarial NetworksMAE Maske Auto EncodersSAM Segment Anything ModelxiiiAcknowledgementsFirst and foremost, I would like to thank my supervisor, Dr. Mohamed She-hata, for his support, encouragement, and patience throughout my studies.I would also like to extend my sincerest thanks to Islam Osman, my labmate and dear friend. Your valuable feedback, support and guidance madeit possible for this work to fruit.xivDedicationI dedicate this work to my loving family and dear friends. To my mom, dad,and my two sisters, your love and support have given me a calm heart andunwavering determination. To my close and dear friends Moataz, Mokhtar,Omar, Ali, and Leandra, your constant support helped me through the mostchallenging moments of my studies and life when things seemed bleak. I willalways be grateful for having you as part of my life.xvChapter 1IntroductionOver the past few decades, video segmentation tasks have advanced signif-icantly due to progress in computer vision and machine learning. Thesetechniques are crucial for various applications, including video surveillance,autonomous driving, human-computer interaction, and video editing. Videosegmentation encompasses multiple tasks, such as foreground segmentation,video object segmentation, video instance segmentation and more. The com-mon goal among these tasks is to analyze a video sequence and extract valu-able information from it. This could involve segmenting the entire foregroundas a single entity from the background, e.g., foreground segmentation, iso-lating each object within the foreground, e.g., video object segmentation, orisolating each object based on its class, e.g., video instance segmentation.Initially, video segmentation techniques depended on statistical methodsin the early 2000s. However, a revolution occurred in the late 2000s withthe emergence of Artificial Intelligence (AI) and Machine Learning (ML).Although these advancements addressed numerous problems, they also in-troduced new challenges.Current state-of-the-art methods face challenges like catastrophic forget-ting, poor domain generalization, and the loss of object details when ob-jects are close to or occluded by other objects. Catastrophic forgetting isa phenomenon in machine learning, particularly in neural networks, wherea model abruptly and significantly loses performance on previously learnedtasks when trained on new tasks. This occurs because the learning processfor the new task interferes with the neural pathways established for the oldtasks, leading the model to "forget" what it had previously learned. Thisis a significant issue in fields like continual learning, where models need tolearn new information over time without losing the ability to perform previ-ously learned tasks. Poor domain generalization refers to the inability of amachine learning model to perform well on data that is significantly differentfrom the data it was trained on. In other words, a model with poor domaingeneralization performs well on the training data (or data similar to it) butfails to generalize its knowledge to new, unseen data from different domainsor distributions. This is a critical issue because real-world applications of-11.1. RESEARCH MOTIVATIONten involve deploying models in varied environments, and a lack of domaingeneralization limits the model’s usefulness and robustness in such scenarios.In video segmentation, occlusion refers to the situation where one object ina video frame overlaps or hides another object, making it challenging to ac-curately segment and track the obscured object. Occlusion can significantlycomplicate the task of identifying and following objects over time becausethe object may be partially or entirely covered by other objects or scenes,leading to gaps or ambiguities in the segmentation process. Effective videosegmentation algorithms need to handle occlusion by predicting or inferringthe presence of the occluded parts of objects and maintaining consistenttracking despite the partial visibility. Techniques to manage occlusion ofteninvolve leveraging temporal information from previous frames, using contextfrom surrounding pixels, or employing sophisticated models that can inferthe obscured regions based on learned patterns.This thesis addresses these critical challenges in video segmentation byproposing innovative methodologies to enhance segmentation accuracy, ro-bustness, and domain generalization in video segmentation tasks. We in-troduce KDNet and a new training technique to combat catastrophic for-getting and improve domain generalization by leveraging advanced learningparadigms such as federated learning, continual learning, knowledge distil-lation, and few-shot learning. These approaches yield significant improve-ments across diverse datasets while ensuring data privacy. Additionally, wepropose a novel model, MEM, to mitigate the loss of object details by inte-grating foundational model outputs and employing mask fusion techniques,resulting in superior segmentation masks.1.1 Research MotivationThe motivation for this research stems from the ongoing challenges in deeplearning, particularly in video segmentation tasks. Existing video segmenta-tion methods often struggle when trained on diverse datasets from varyingdistributions/domains or when subject to unseen data. Moreover, they facedifficulties in accurately segmenting objects in the presence of occlusions orsudden changes in appearance within video sequences. This research is fueledby a strong interest in investigating innovative and unconventional learningparadigms and combining them with current techniques to tackle these chal-lenges. Our aim is to contribute to this field of study and pave the way fornew and exciting research directions.21.2. RESEARCH OBJECTIVE1.2 Research ObjectiveTo build models that achieve state-of-the-art performances while overcomingthe existing problems in video segmentation tasks.1. Propose a novel deep-learning model and training technique to learnfrom single or multiple datasets effectively. Hence, achieves state-of-the-art domain generalization performance on multiple datasets.2. Propose a training technique that allows the model to adapt to newdomains using as few as one labelled image (One-shot).3. Develop an approach that adapts the foundational model SAM forvideo Object Segmentation tasks to make it a zero-shot VOS model,This approach produces state-of-the-art performance in UnsupervisedVideo Object Segmentation (UVOS)4. Propose a model that combines the strengths of both SAM (Foun-dational model) and Cutie (current state-of-the-art in Video ObjectSegmentation (VOS)), producing state-of-the-art results in VOS.31.3. PUBLICATIONS1.3 PublicationsThis thesis is based on the following published/submitted work:1. Osman, I., Abdelfattah, I., Shehata, M.S. (2023). Domain General-ization for Foreground Segmentation Using Federated Learning. In:Bebis, G., et al. Advances in Visual Computing. ISVC 2023. LectureNotes in Computer Science, vol 14361. Springer, Cham.2. MEM: Mask enhancement model for Video Object Segmentation. Ab-delfattah, I., Shehata, M.S., International Symposium on Visual Com-puting (ISVC), 2024. (Accepted)41.4. THESIS OUTLINE1.4 Thesis OutlineThe remainder of this thesis is organized as follows:Chapter 2 provides a comprehensive background and literature reviewof video segmentation tasks, including foreground segmentation and videoobject segmentation. It explores mainstream approaches such as supervised,semi-supervised, and unsupervised methods and lays a foundation for thefollowing technical chapters.Chapter 3 introduces a novel deep-learning model and training techniquethat effectively learns from single or multiple datasets, achieving state-of-the-art out-of-domain performance across multiple datasets. This chapter isbased on the published paper "Domain Generalization for Foreground Seg-mentation Using Federated Learning.".Chapter 4 proposes a model that integrates the strengths of SAM (afoundational model) and Cutie (the current state-of-the-art in video objectsegmentation), resulting in state-of-the-art VOS performance. This chapteris based on the submitted paper "MEM: Mask Enhancement Model for VideoObject Segmentation.".Finally, Chapter 5 concludes the thesis and suggests avenues for futureresearch in this area.5Chapter 2Background and Related Work2.1 OverviewVideo segmentation began with basic image processing techniques. In theearly days, background subtraction [2] was the most common approach. Thismethod is a gaussian mixture-based background/ foreground segmentationalgorithm that involves comparing each frame of a video with a backgroundmodel to identify moving objects. Techniques such as frame differencing andmedian filtering were employed to create a static background model. How-ever, these methods struggled with dynamic backgrounds, lighting changes,and camera movements. To address the limitations of simple background sub-traction, more sophisticated statistical models were introduced. One notableadvancement was the Gaussian Mixture Model (GMM) [3]. GMM modelsthe background as a mixture of multiple gaussian distributions, allowing it toadapt to gradual changes in the scene. This approach significantly improvedthe robustness of video segmentation tasks in real-world scenarios.In the early 2000s, the use of Markov Random Fields (MRFs) [4] andConditional Random Fields (CRFs) [5] became popular. These probabilisticmodels consider spatial and temporal dependencies between pixels, leading tomore accurate segmentation results. By modelling the relationships betweenneighbouring pixels, MRFs and CRFs can better handle noise and preserveobject boundaries.The advancements in machine learning in the late 2000s brought newpossibilities for video segmentation tasks like foreground segmentation, whereeach frame of the video is separated into a background and a foreground thatis seen as a single object. Support Vector Machines (SVMs) [6] and otherclassifiers were used to distinguish foreground objects from the backgroundbased on features such as colour, texture, and motion. These methods lever-aged training data to learn discriminative models, improving the accuracyand generalization of segmentation algorithms.The advent of deep learning in the 2010s revolutionized the field of videosegmentation. Convolutional Neural Networks (CNNs) [7] and Fully Convo-lutional Networks (FCNs) [8] demonstrated unprecedented performance in62.2. VIDEO SEGMENTATION TASKSvarious computer vision tasks, including video segmentation tasks. Methodssuch as DeepLab [9], Mask R-CNN [10], and U-Net [11] set new benchmarksby leveraging deep hierarchical features and end-to-end learning frameworks.2.2 Video Segmentation TasksWith the continuous advancements in video segmentation, various special-ized tasks have emerged, each addressing unique challenges and applications.Foreground segmentation focuses on distinguishing foreground objects as asingle object from the background. Video object segmentation aims to seg-ment and track multi-objects of interest throughout the video. Video instancesegmentation extends this by differentiating between multiple instances ofthe same object class, providing detailed and instance-specific object track-ing necessary for complex scene understanding. Moving object segmentationspecifically targets the detection and segmentation of objects in motion, es-sential for dynamic environments and activity recognition. In this thesis, themain focus relies on foreground and video object segmentation. These taskssaw significant advancements with deep learning.Figure 2.1: Foreground segmentation vs video object segmentation. Imagesare from DAVIS 17 Dataset [1]72.2. VIDEO SEGMENTATION TASKS2.2.1 Foreground Segmentation2.2.1.1 Problem definitionForeground segmentation is a method in computer vision that distinguishesobjects from the background in a sequence of frames. This segmentationresults in a binary image in which each pixel is assigned a value of 0 or 1,indicating whether it is part of the background (0) or the foreground (1)as seen in Figure 2.1. The most used model architecture for this task is aCNN-based Encoder-Decoder architecture.2.2.1.2 Foreground Segmentation Related WorkForeground segmentation models are showing outstanding performance nowa-days for the segmentation task. Foreground segmentation models couldbe categorized into statistical-based models and deep-learning-based mod-els. Statistical-based foreground segmentation models depend mainly on thechange in the pixel intensity of the background of any image and also dependon the reasons behind this change to be able to separate the objects fromthe background. Furthermore, statistical methods also depend on motionvectors to separate the object pixels from the foreground. Optical flow isa widely used statistical method that depends on motion vectors. Opticalflow depends on processing a sequence of images to approximate the 3-Dvelocities of surface points. Then, these vectors could be used for varioustasks, including object segmentation. Optical flow has been used for fore-ground segmentation as seen in [12] and [13]. Statistical models struggle toisolate the background when environmental challenges exist in the image,like illumination change, moving camera, and dynamic background.Recently, deep neural networks have been used for the task of foregroundsegmentation, and they have been showing a significant improvement inthe task of foreground segmentation. Deep-learning networks are dividedinto CNNs and transformers. One of the earlier CNN models that used anencoder-decoder type neural network is FgSegNet [14]. Several models useneural network-based architectures like CascadeCNN [15], which is basedon a multi-resolution CNNs with a cascaded architecture. RANet+[16],which is Ranking Attention Network, uses an encoder-decoder frameworkto learn pixel-level similarity and segmentation in an end-to-end manner.The most recent model, Space-Time Correspondence Network (STCN) [17],uses a Space-Time Correspondence Network and negative squared Euclideandistance to compute affinities. On the other hand, transformer-based modelslike TransBlast [18] are also being used and showing promising results. Trans-82.2. VIDEO SEGMENTATION TASKSBlast is trained using self-supervised learning to leverage the limited data andlearn a strong object representation while using the augmented subspace lossfunction. However, all these models suffer from catastrophic forgetting whensubject to domain shifts. This thesis proposes a federated learning techniqueto mitigate the effect of catastrophic forgetting and achieve domain gener-alization, even though federated learning is being used mainly for domaingeneralization on multiple mobile devices.2.2.1.3 Foreground Segmentation DatasetsDensely Annotated VIdeo Segmentation 2016 (DAVIS16) [1] is abenchmark for evaluating video object segmentation methods, featuring 50high-resolution Full HD videos with pixel-level ground truth annotations foreach frame. The primary task is to segment and track foreground objectsacross complex scenes, which include challenges like dynamic backgrounds,occlusions, and varying object motions. It provides dense, consistent anno-tations and is commonly used to assess performance through metrics like theJaccard Index (IoU) and F-measure, making it a key dataset for video objectsegmentation, tracking, and temporal consistency evaluation.SegTrackV2 [19] is a benchmark used for video object segmentation, con-sisting of 14 video sequences with pixel-level ground truth annotations, andthe number of frames ranged from 21 to 279 in each video. These sequencesinclude complex challenges like occlusions, overlapping objects, and changesin object appearance. Some videos contain multiple objects, making thetask more difficult by requiring accurate multi-object segmentation. Thedataset is widely used to evaluate segmentation methods based on both spa-tial accuracy and temporal consistency, making it suitable for benchmarkingalgorithms in video object segmentation and tracking.ChangeDetection.Net (CDNet) 2014 [20] is a benchmark dataset thatdetects changes between video frames. These changes are the moving ob-jects. The dataset consists of 11 different challenges. Each challenge has anumber of videos ranging from 4 to 6, and the number of frames in eachvideo ranges from 1000 to 8000. The total number of videos in the dataset is53 videos. It covers diverse challenges like dynamic backgrounds, camera jit-ter, illumination changes, and various object motions. The dataset includesground truth masks for foreground objects and provides different categories,such as shadows, night videos, and weather changes, to assess robustness92.2. VIDEO SEGMENTATION TASKSacross scenarios. CDNet is widely used to benchmark methods for changedetection, foreground-background segmentation, and object tracking.2.2.1.4 Evaluation MetricsThe evaluation metrics used in foreground segmentation:Precision =TPTP + FP(2.1)Recall =TPTP + FN(2.2)FPR =FPFP + TN(2.3)FNR =FNTP + FN(2.4)FM =2 ∗ Precision ∗RecallPrecision+Recall=2 ∗ TP2 ∗ TP + FP + FN (2.5)TP, TN, FP, and FN denote true positives, true negatives, false posi-tives, and false negatives, respectively. FM is the F1-Measure. FPR andFNR are the false-positive rate and false-negative rate. Precision indicatesthe percentage of pixels correctly classified as moving objects, while Recallrepresents the percentage of object pixels correctly identified. The false-positive rate shows the percentage of background pixels incorrectly classifiedas foreground, and the false-negative rate indicates the percentage of fore-ground pixels mistakenly identified as background. The F1-measure, a keymetric in foreground segmentation, is the harmonic mean of precision andrecall.102.2. VIDEO SEGMENTATION TASKS2.2.2 Video Object Segmentation2.2.2.1 Problem definitionVideo Object segmentation is a method in computer vision that isolatesobjects from the background and each other in a sequence of frames. Thissegmentation results in a multi-class image in which each pixel is assigneda value of 0 to n, indicating whether it is part of the background (0) or theobjects (n) as seen in Figure 2.1. The most used model architecture for thistask is an Encoder-Decoder architecture.2.2.2.2 Video Object segmentation Related WorkVOS has seen significant advancements over recent years, driven by innova-tions in deep learning and computer vision. This section reviews the threemain types of VOS and the latest and most impactful models in each type.Supervised Video Object Segmentation (SVOS) involves trainingmodels with a large dataset of annotated videos where the objects of in-terest are labelled frame by frame. Traditional techniques in SVOS reliedon methods like optical flow [21] and graph-based segmentation [22], whichoften struggled with issues of scalability and generalization. The rise ofdeep learning brought about CNNs and Recurrent Neural Networks (RNNs),which improved the accuracy and robustness of SVOS. However, these meth-ods require extensive labelled data and are prone to catastrophic forgetting,where the model forgets previously learned information when trained on newdata. Recently, models such as Mask R-CNN [10] and the development oftemporal convolutional networks have further enhanced SVOS, yet the needfor large, annotated datasets remains a significant bottleneck.Semi-supervised Video Object Segmentation (S-SVOS) aims to re-duce the reliance on extensive annotated data by leveraging a few annotatedframes and propagating this information across the video. Early techniquesemployed keyframe selection and interpolation methods, which often resultedin suboptimal segmentations due to error accumulation over frames. Withthe advent of AI, S-SVOS approaches began to utilize deep learning modelsthat combine both spatial and temporal information to improve segmentationaccuracy. Current techniques could be categorized into memory-based meth-ods, transformer-based approaches, and object-level reasoning techniques.Memory-based S-SVOS techniques utilize past frames to inform the seg-mentation of current frames, enhancing temporal coherence and robustness.112.2. VIDEO SEGMENTATION TASKSFEELVOS [23] achieves real-time performance by utilizing pixel-wise embed-dings from previous frames. FEELVOS employs a distance-based matchingmechanism between the embeddings of the current frame and a referenceframe, allowing it to segment objects efficiently without extensive compu-tational resources. STM[24] introduced a novel memory read-write mecha-nism, where previous frames are stored in a memory bank and matched withthe current frame to propagate the segmentation mask. This approach sig-nificantly improves the accuracy of VOS by maintaining detailed temporalinformation through frames. MiVOS [25] extends the STM framework byincorporating multi-level memory features. This model not only stores high-level semantic information but also retains low-level pixel details, enablingit to handle complex scenes with intricate object interactions. XMem [26]introduces multi-scale memory representations to better handle occlusionsand rapid object movements. By leveraging both local and global memoryfeatures, XMem can robustly track and segment objects even in challengingscenariosTransformers have brought significant improvements to S-SVOS by em-ploying attention mechanisms to model long-range dependencies between pix-els across frames. AOT [27] employs an associative attention mechanism todynamically link object instances across frames. This approach ensures thatthe model maintains consistent object identities throughout a video sequence,providing robust performance in complex environments. An adaptation ofDETR for VOS [28], this model extends object detection capabilities to han-dle the temporal dimension. SST [29] focuses on reducing the computationalcomplexity of transformers by applying sparse attention to regions of interest.Although this approach can sometimes compromise accuracy, it significantlyimproves processing speed, making it suitable for real-time applications.Object-level reasoning approaches focus on modelling and tracking in-dividual objects, providing a more holistic understanding of video content.ISVOS [30] injects instance-level features from a pre-trained instance seg-mentation network into a memory-based VOS framework. This integrationenhances the performance of VOS by leveraging detailed instance-level in-formation, although it requires separate instance segmentation steps. Cutie[31] introduces object-level memory reading and query-based transformers,achieving state-of-the-art performance while maintaining computational effi-ciency. This model can robustly segment objects even in complex scenarios,avoiding the need for computationally expensive operations between high-resolution feature maps.The robustness and efficiency of modern VOS methods facilitate their in-tegration into broader video segmentation pipelines, addressing tasks such as122.2. VIDEO SEGMENTATION TASKSopen-vocabulary segmentation and unsupervised segmentation. Track Any-thing [32] combines robust object tracking with segmentation capabilities,enabling seamless tracking and segmentation across diverse video sequences.Track Anything leverages advancements in VOS to handle complex videoscenes in real-time. DEVA [33] utilizes VOS techniques to automaticallyannotate video events, demonstrating the applicability of VOS in real-worldscenarios. This model integrates VOS into comprehensive video understand-ing tasks, enhancing the automation of video analysis.The field of VOS continues to evolve with innovative approaches thatpush the boundaries of accuracy and efficiency. Memory-based methods,transformer architectures, and object-level reasoning techniques are pivotalin addressing the inherent challenges of S-SVOS. Future research will likelybuild on these foundations, exploring new ways to integrate and optimizethese techniques for even more robust and versatile video segmentation so-lutions.Unsupervised Video Object Segmentation (UVOS) is the most chal-lenging task, which involves segmenting objects without any labelled data.Traditional methods relied on motion segmentation and clustering techniques,which were limited by their inability to differentiate between objects withsimilar motion patterns effectively. The introduction of deep learning hasled to the development of more sophisticated models that utilize GenerativeAdversarial Networks (GANs) [34] and self-supervised learning techniques.These models can learn robust features from vast amounts of unlabeled data,yet they often suffer from generalization issues and the problem of distin-guishing between objects in complex scenes. Recent approaches have fo-cused on leveraging spatiotemporal coherence and attention mechanisms toimprove the accuracy and reliability of UVOS.2.2.2.3 Video Object Segmentation DatasetsDensely Annotated VIdeo Segmentation 2017 (DAVIS17)[1] is abenchmark for video object segmentation, offering high-resolution videoswith pixel-level annotations for diverse and challenging sequences. It in-cludes 60 videos for training, 30 for validation, and 60 for testing (30 in thetest-dev set and 30 in the test-challenge set). It supports semi-supervisedand unsupervised segmentation tasks, providing detailed evaluation metricslike the Jaccard index.132.2. VIDEO SEGMENTATION TASKSYouTube-VOS 2019 dataset [35] is a large-scale benchmark for videoobject segmentation, offering high-resolution videos with detailed annota-tions. It includes 3,471 videos for training, 507 for validation, and 541for testing. Supporting both semi-supervised and unsupervised tasks, thisdataset provides extensive data for training and benchmarking segmentationmodels.2.2.2.4 Evaluation MetricsVideo object segmentation evaluation metrics:J (U, V ) = |U ∩ V ||U ∪ V | (2.6)J&F = J + F2(2.7)J is Jaccard index, F is the F measure. The Jaccard index is calculatedby calculating the intersection over union between the detected object pixelsand the object ground truth pixels similar to Intersection Over Union (IOU).J&F is the average of both the Jaccard index and F measure and is consid-ered the main metric in Video object segmentation.In this thesis, we employ and integrate advanced learning paradigms withthe proposed models to create models that are robust, generalized, and ca-pable of achieving state-of-the-art performance in video segmentation tasksas demonstrated in the technical chapters 3 and 4.14Chapter 3KDNet: DomainGeneralization for ForegroundSegmentation Using FederatedLearning3.1 Overview and MotivationIn this chapter, we propose a deep learning model and a novel training pro-cedure to learn from multiple domains while achieving domain generalizationand overcoming catastrophic forgetting by employing continual learning tech-niques. The proposed model is an encoder with a multi-head decoder thatcombines shared and isolated parameters for different domains. The proposedtraining procedure is inspired by federated learning, where client models arecreated such that each client learns from a specific domain. After that, themain model learns the knowledge gained by each client using knowledge dis-tillation with regularization to prevent the main network from forgetting thepreviously gained knowledge. Experiments demonstrate the effectiveness ofour proposed work in comparison with state-of-the-art models as well as themodel’s ability to adapt to new domains with few-shot learning.3.1.1 Domain GeneralizationDomain generalization seeks to improve a model’s ability to perform well onunseen domains, addressing the problem of domain shift where the trainingand testing data come from different distributions. Traditional models oftenstruggle with generalization beyond their training domain, leading to poorperformance in real-world applications. Domain generalization techniques,such as invariant feature learning, data augmentation, and adversarial train-ing, aim to create robust models that can generalize across various domainswithout requiring access to target domain data during training.153.1. OVERVIEW AND MOTIVATION3.1.2 Continual LearningContinual learning, also known as lifelong learning, aims to develop mod-els capable of learning from a continuous stream of data without forgettingpreviously acquired knowledge. Traditional machine learning models suf-fer from catastrophic forgetting, where new information overwrites existingknowledge, a problem known as catastrophic forgetting. Continual learningaddresses this by employing strategies such as rehearsal, regularization, andarchitectural approaches to retain past knowledge while integrating new in-formation, making it crucial for real-world applications where data evolvesover time. The approaches that address continual learning could be dividedinto three main categories, replay-based approaches, regularization-based ap-proaches, and parameter isolation approaches. Replay-based approaches relyon storing some examples from previous training sessions to reuse them dur-ing training on new tasks [36, 37]. In regularization-based techniques, themain focus is on changing the loss function by adding regularization terms torestrain overfitting to new tasks [38, 39]. Furthermore, in parameter isolationtechniques, the model parameters are separated and isolated for each taskand training session. Freezing the desired weights allows the model to trainon multiple tasks without forgetting the knowledge learned from previoustasks [40], Another approach is the add more weights and increase the modelsize with each new task [41].3.1.3 Federated LearningFederated learning is a decentralized approach to training machine learningmodels across multiple devices or organizations without centralizing the data.This method enhances privacy and security by keeping the data localized andonly sharing model updates. Federated learning leverages techniques like se-cure aggregation, differential privacy, and homomorphic encryption to ensuredata confidentiality while collaboratively training models. It is particularlyvaluable in privacy-sensitive applications like healthcare and finance, wheredata centralization is not feasible.In federated learning, the model is trained locally on each device usingthe data on that device, and the updated weights are shared with a centralserver. The central server then updates its weights using the aggregated up-dates from all the devices and releases the updated model to the devices.This process is repeated to keep the model up to date with new data. Asdifferent settings for federated learning have been used in networking andIoT like Cross-silo [42] and Cross-device [43]. Cross-silo federated learning163.1. OVERVIEW AND MOTIVATIONrefers to the collaborative training of machine learning models across mul-tiple organizational silos, where each silo represents a distinct entity withits private dataset. This approach enables the pooling of data while ensur-ing data privacy and security. In contrast, cross-device federated learninginvolves training models directly on decentralized devices, such as smart-phones or IoT devices, without the need for data transfer to a central server.The focus is on leveraging the computational power of diverse devices whilepreserving user privacy. Both approaches contribute to privacy-preservingmachine learning, but their distinction lies in the data distribution and col-laboration mechanisms employed. It has been found that Cross-silo is thebest setting for the proposed solution in this paper, as the main focus is noton data privacy but on domain generalization. Federated learning has alsobeen used for some computer vision tasks in medical imaging like [44], and[45] for decentralized learning. To the best of our knowledge, federated learn-ing has yet to be used to overcome the problem of catastrophic forgetting inthe context of foreground segmentation.3.1.4 Knowledge DistillationKnowledge distillation is a technique used to transfer knowledge from a large,complex model (the teacher) to a smaller, more efficient model (the student).This process involves training the student model to mimic the behaviourof the teacher model, often using the teacher’s soft predictions as a guide.Knowledge distillation helps in deploying models on resource-constrained de-vices by reducing the model size and computational requirements withoutsignificantly sacrificing performance, making it a key approach in optimizingdeep learning models for practical use. Knowledge distillation is particularlypromising as it achieves performances close to those of larger models, despitehaving significantly fewer parameters and lower computational complexity.For instance, Mobile-SAM [46] exemplifies the power of knowledge distilla-tion, being 100 times smaller and 50 times faster than SAM [47], with onlya 20% performance drop.173.1. OVERVIEW AND MOTIVATION3.1.5 Few-Shot LearningFew-shot learning focuses on enabling models to generalize from a limitednumber of training examples. Unlike traditional learning paradigms that re-quire large datasets, few-shot learning is designed to work effectively withminimal data, which is particularly useful in scenarios where data collec-tion is expensive or time-consuming. Techniques like metric learning, self-supervised learning-based methods, and data augmentation are commonlyused to enhance the model’s ability to recognize new classes or tasks fromjust a few examples, thereby improving efficiency and reducing the needfor extensive labelled data. Metric learning is a learning technique that fo-cuses on clustering and identifying similar features in the embedding space bylearning an embedding function that identifies similarities and separates sim-ilar features in the embedding space [48–50]. Self-supervised learning meth-ods primarily aim to extract knowledge from unlabeled data by generatingtheir own labels through techniques like random masking or applying randomtransformations to images. This process enables the model to learn abstractand high-level features from the data [51–53]. Subsequently, the model un-dergoes fine-tuning on a smaller, labelled dataset using transfer learning,enhancing its performance on specific tasks. Implementing this approachenhances learning efficiency and improves model generalization, advancingAI and ML capabilities. One of the commonly used self-supervised modelsfor few-shot and domain generalization is Maske Auto Encoders (MAE) [51].Applying augmentation techniques on limited training examples results inincreasing the number of the labelled data points which helps the model tolearn more features from the provided data. Some of the commonly used aug-mentation techniques are rotation, scaling, translation, flipping (Horizontaland Vertical), cropping, colour jittering (brightness, contrast, saturation),Gaussian noise, Cutout [54], Mixup [55], CutMix [56].183.2. PROPOSED WORK3.2 Proposed WorkThe proposed work comprises two main components: 1) model architectureconsisting of an encoder with a multi-head decoder. 2) training techniquethat combines federated learning with continual learning.3.2.1 Model ArchitectureFigure 3.1: The architecture of KDNet. On the left is the server networkwith multi-head outputs, and on the right are the client networks.The proposed model is an encoder-decoder architecture. The encoder isResNet50, and the decoder is three blocks. Each decoder block consists ofthree convolutional layers. Each convolutional layer is followed by a nor-malization layer, and an up-sampling layer follows each decoder block. Thenumber of filters of all convolutional layers in each block is the same butdiffers from one block to another. Such that convolution layers in the firstblock have 256 filters, in the second block have 128 filters, and in the lastblock have 64 filters. This size of all filters in all decoder blocks is 3 × 3.After the three blocks, a multi-head is added to produce the outputs for dif-ferent domains. The head is defined as two convolutional layers. The firstone has 3×3×64 filters followed by a batch normalization layer. The second193.2. PROPOSED WORKconvolutional layer has a 1× 1× 1 filter with a sigmoid activation function.This architecture is for the main model (server), while the client’s modelis the same but has a single head. Figure 3.1 shows the server and clientarchitecture.3.2.2 Training TechniqueAs shown in Figure 3.2, the training procedure is a repeated cycle of fourmain steps: 1) instantiating the clients, 2) training the clients, 3) gatheringthe clients, and 4) training the main model (server).Figure 3.2: Training procedure of KDNet.Instantiating the client: In this step, all clients are instantiated from theserver (i.e., a clone of the shared weights of the server is sent to the client).The clients initialize their network parameters with the cloned weights of theserver. It is important to mention that only the encoder and decoder pa-rameters are copied to the client (i.e., the heads’ parameters are not copied).After that, a head is initialized with random weights added at the end of the203.2. PROPOSED WORKclient network.Training the clients: The clients’ training is straightforward. Each clientis trained on its local dataset using supervised learning. The goal is thateach client successfully learns to segment the foreground objects in the localdataset.Gathering the clients: The server gathers only the trained client net-works (i.e., the local dataset of each client is not included)Training the server: This is the most significant step where the servernetwork learns the knowledge gained by all client networks. The server usesa large unlabeled dataset. Then, for each batch in this dataset, only oneclient will process this batch to produce the output mask. The clients areselected in a round-robin order (i.e., client1, client2, ..., clientM , client1, ...,etc.). After that, the server applies the knowledge distillation loss functionto learn to produce an output similar to the client’s output. However, theserver should learn from each client without affecting the knowledge gainedby other clients. Hence, each client updates only one head of the server.Also, a term in the loss function is added to regularize the updates of sharedparameters between clients (i.e., the encoder and decoder parameters). Thisterm minimizes the distance between the output of other clients after theupdate and before the current client’s update. The loss function used isdefined as follows:Li = ℓ(yi, yin) +1MM∑j ̸=iℓ(yjo, yjn) (3.1)Li = KDL(yi, yin) +1MM∑j ̸=iMSE(yjo, yjn) (3.2)the Li is the loss function for updating the server with the ith client.The first loss term is reducing the distance between the server’s output ofith after the update yin and the output of the ith client yi. The second termis reducing the distance between the output of all other heads of the serverbefore yjo, and after yjn, the update and M represents the number of clients.The second term acts as a regularization term to force the network to updatethe parameters in a way that does not cause a huge change in the output ofother heads. As shown in the loss function, the ground truth (label) of thebatches in the dataset is not used.213.3. EXPERIMENTS3.3 ExperimentsTo evaluate KDNet, Three experiments are conducted. The first experimentis traditional foreground segmentation, where the model is trained usingdatasets individually (separated training). This experiment compares theresults of KDNet with other foreground segmentation models. The secondexperiment tests the model generalization by training using multiple datasets(multiple domains). In this experiment, there are two training methods,namely joint and fine-tuning (sequential). Joint training means multipledatasets are combined to form one large dataset. Then, the model is trainedusing this large dataset. In sequential training, the model is trained usingthe dataset one after the other. After the model is trained using the lastdataset, the models are evaluated using all datasets. The last experimentis few-shot learning. In this experiment, we test the ability of the model toadapt to new domains using a few labeled images.3.3.1 DatasetsDensely Annotated VIdeo Segmentation 2016 (DAVIS16) [1] is avideo object segmentation dataset. DAVIS consists of 50 videos. Each videohas several frames ranging from 50 to 104. In each video, a single object isannotated, which is the object of interest in this video.SegTrackV2 [19] is a video multiple objects segmentation dataset. Thenumber of videos in the dataset is 14, and the number of frames ranged from21 to 279 in each video. In this dataset, each frame has multiple ground-truthcorresponding to each object in the video.ChangeDetection.Net (CDNet) 2014 [20] is a benchmark datasetthat detects changes between video frames. These changes are the movingobjects. The dataset consists of 11 different challenges. Each challenge hasa number of videos ranging from 4 to 6, and the number of frames in eachvideo ranges from 1000 to 8000. The total number of videos in the datasetis 53 videos.ImageNet [57] is a large-scale dataset used as a benchmark for imageclassification tasks. In this paper, we use the images in the dataset withoutlabels to train the main model to learn the knowledge gained by clients usingknowledge distillation.3.3.2 Implementation DetailsIn all experiments, we report the results of five different versions of KDNet.These versions 1) KDNet-base is the proposed model with a single head223.3. EXPERIMENTSdecoder, 2) KDNet-FL is the proposed model trained with the federatedlearning settings. In this version, the dataset is into sub-datasets. Then, aclient is instantiated to learn from each sub-dataset. After that, the multi-heads of the main model learn from all clients using knowledge distillationloss function without the regularization term in equation (3.2). 3) KDNet-CL is the same as KDNet-FL, but the main model learns directly from thesub-datasets instead of learning from the trained clients. The loss functionused is a mean squared error (MSE) with the regularization term in equation(3.2). 4) KDNet-FL-CL is the combination of both federated learning andcontinual learning. The loss function used in this version is the proposedfunction in equation (3.2). 5) KDNet-FL-CL-U is the same as KDNet-FL-CL, but the knowledge distillation between the main model and the clientsis applied using a completely different unseen dataset (ImageNet) from theone used to train the clients. The point of adding the last version is to showthat the proposed work does not need to access the datasets used in trainingthe clients (data privacy).During the training of KDNet, if the training data is a single dataset,the dataset is randomly split into five groups of videos to simulate multipledataset training. In this case, the main model has five heads, each oneresponsible for learning from one of the five groups of videos. On the otherhand, if the training data is multiple datasets, then the number of heads ofthe main model will be the same as the number of different datasets.In the testing phase, for all versions of KDNet with multi-head, we useone labelled frame for each video to find which head is suitable for segmentingthis video. We do this by feeding the frame as an input to the model. Then,the f-measure of each head of KDNet is calculated. The head with the highestf-measure is used to segment foreground objects from all frames of this video.3.3.3 Traditional Foreground Segmentation ResultsThis experiment evaluates KDNet using two datasets, DAVIS16 and Seg-TrackV2. The model is trained and tested using each dataset individually.For this experiment, we show the results of the five versions of KDNet men-tioned in the previous subsection. The reported metric for all experiments isthe f-measure (F).Table 3.1 shows the results of KDNet against state-of-the-art foregroundsegmentation models. The results show that KDNet is in second place by asmall gap of 0.06% in f-measure less than the top-performing model.For the SegTrackV2 dataset, Table 3.2 shows the results of KDNet againstother foreground segmentation models using the SegTrackV2 dataset. As233.3. EXPERIMENTSshown in Table 3.2, the performance of three versions of KDNet is in the topplaces. KDNet outperforms the second-best model (REFNet) in literatureby 1.8%. While the second version of KDNet outperforms REFNet by 1.2%.These tables show the effectiveness of the proposed work in learning fromindividual datasets, even though the proposed work is mainly designed tolearn effectively from multiple datasets.Table 3.1: Traditional foregroundsegmentation results of KDNetcompared with state-of-the-artmodels on the DAVIS16 dataset.The number inside the circledefines the order of the modelperformance concerning eachevaluation metric.Model FSTM [24] 0.901MHP-VOS[58] 0.895CFBI [59] 0.905CFBI+ [59] 0.911FgSegNet [60] 0.847CascadeCNN [15] 0.814STCN [17] 0.930 1○REFNet [61] 0.883TransBlast [18] 0.863KDNet-base 0.891KDNet-FL 0.909KDNet-CL 0.897KDNet-FL-CL 0.924 2○KDNet-FL-CL-U 0.921 3○Table 3.2: Traditional foregroundsegmentation results of KDNetcompared with state-of-the-artmodels on the SegTrackV2dataset. The number inside thecircle defines the order of themodel performance concerning theevaluation metric.Model FDSRFCN3D [62] 0.878GDHF [63] 0.868UFO [64] 0.863PDB [65] 0.864STRCF [66] 0.899FgSegNet [60] 0.880CascadeCNN [15] 0.867REFNet [61] 0.904TransBlast [18] 0.902KDNet-base 0.905KDNet-FL 0.911 3○KDNet-CL 0.909KDNet-FL-CL 0.920 1○KDNet-FL-CL-U 0.914 2○3.3.4 Domain Generalization ResultsTo highlight the effectiveness of the proposed work, we conducted multi-domain experiments. The model is trained in these experiments using threebenchmark datasets: CDNet, DAVIS16, and SegTrackV2. We show the re-sults of two training procedures. The first training procedure is Joint train-ing, where the models are trained using the three datasets simultaneously asif they are one large dataset. The second training procedure is Fine-tuning,243.3. EXPERIMENTSwhere the models are fine-tuned using one dataset after the other (sequen-tially). After training is done in both scenarios, the model is evaluated usingthe testing set of all three datasets.The results are compared with state-of-the-art foreground segmentationmodels. Four models with their source code publicly available from the pre-vious experiment are selected to compare KDNet’s generalization againstthem. These models are REFNet, sEnDec, FgSegNetV2, and CascadeCNN.Tables 3.3 and 3.4 show the results of the models when trained usingmultiple datasets either simultaneously or sequentially. In these tables, itis clear that all models degraded in performance tremendously compared totheir performance in the separated training. For example, REFNet has anaverage of 26.3% drop in performance when trained using multiple datasets.In Table 3.3, most of the models have higher performance in the first dataset(CDNet) as it is the largest dataset, which made the models biased toward it.However, our model solves this problem by using isolated parameters for eachdataset (multi-head decoder). The performance gain in our model is 14.1%compared to REFNet, which is a massive gain in performance. In Table3.4, the performance of all models in the last dataset is the highest. This isbecause the models tend to forget the knowledge gained by previous datasetsdue to sequential training (catastrophic forgetting). However, our proposedmodel overcomes this problem by the isolated parameters and the continuallearning regularization term in the loss function. Hence, the performance ofour proposed model is 13% higher than the second-best model.Table 3.3: Comparison of KDNet against state-of-the-art methods whentrained using the three datasets CDNet, DAVIS16, SegTrackV2 jointly(i.e., all three datasets are combined and considered as one large dataset).The values in the table represent the F-measure.Model CDNet DAVIS16 SegTrackV2 AveragesEnDec [67] 0.657 0.571 0.526 0.584FgSegNetV2 [60] 0.639 0.567 0.495 0.567CascadeCNN [15] 0.582 0.535 0.460 0.525REFNet [61] 0.737 1○ 0.630 0.505 0.624KDNet-base 0.627 0.706 0.580 0.637KDNet-FL 0.649 0.735 3○ 0.721 3○ 0.701 3○KDNet-CL 0.633 0.719 0.658 0.670KDNet-FL-CL 0.662 2○ 0.756 1○ 0.867 1○ 0.765 1○KDNet-FL-CL-U 0.659 3○ 0.751 2○ 0.816 2○ 0.742 2○253.3. EXPERIMENTSTable 3.4: Comparison of KDNet against state-of-the-art methods whentrained using the three datasetsCDNet, DAVIS16, SegTrackV2 sequen-tially (i.e., the models are fine-tuned on datasets sequentially. After themodel is trained using the last dataset, it is evaluated using the last datasetand all previous ones). The values in the table represent the F-measure.Model CDNet DAVIS16 SegTrackV2 AveragesEnDec [67] 0.451 0.562 0.829 0.614FgSegNetV2 [60] 0.419 0.518 0.755 0.564CascadeCNN [15] 0.398 0.503 0.746 0.549REFNet [61] 0.422 0.529 0.917 1○ 0.622KDNet-base 0.421 0.590 0.858 0.620KDNet-FL 0.476 0.643 0.872 0.664KDNet-CL 0.638 3○ 0.716 3○ 0.874 0.743 3○KDNet-FL-CL 0.655 1○ 0.721 1○ 0.880 2○ 0.752 1○KDNet-FL-CL-U 0.647 2○ 0.716 2○ 0.876 3○ 0.746 2○To show the output of the trained model using some example videosfrom the three datasets, some visual results from this experiment are shownin Fig. 3.4.The effectiveness of the proposed federated learning technique is shownin Fig. 3.3. In this figure, the average loss of all clients gets lower andconverges faster after each federated cycle. Hence, the trained server hasgood knowledge of different domains and can adapt agilely to new domains.3.3.5 Few-shot ResultsWe conducted a few-shot learning experiment to prove the trained KDNet’sversatility and agility to learn new domains with limited data. In this ex-periment, the KDNet trained using CDNet, SegTrackV2, and DAVIS16 isfine-tuned using a few labelled frames from the DAVIS17 dataset. The num-ber of labelled frames per video is either 1 or 5 frames, which is the defaultconfiguration for few-shot learning. Other models are trained jointly us-ing the same three datasets and fine-tuned using the labelled frames fromDAVIS17. In Table 3.5, KDNet (Server) is fine-tuning the jointly trainedmodel, while KDNet (Server-Client) is fine-tuning the model trained usingfederated learning. The results show the superiority of the KDNet trainedwith federated learning in the few-shot learning performance. In 1-shot (i.e.,one labelled frame per video), the performance of the proposed model is263.3. EXPERIMENTSFigure 3.3: The average loss of clients of all three datasets after each federatedlearning cycle.273.3. EXPERIMENTSFigure 3.4: Sample results of KDNet from domain generalization experiment.283.4. LIMITATIONS AND POSSIBLE IMPROVEMENTSaround 6.8% above the second model REFNet, which is expected as shownin Fig. 3.3 the trained KDNet makes the convergence towards a new taskfaster and easier. Hence, only a few fine-tuning iterations are required forthe trained KDNet to converge.Table 3.5: Few-shot foreground segmentation results of KDNet compared tostate-of-the-art methods on DAVIS17.Model 1-shot 5-shotCascadeCNN [15] 0.624 0.661FgSegNetV2 [60] 0.676 0.742sEnDec [67] 0.681 0.759REFNet [61] 0.692 0.767KDNet-base 0.714 0.785KDNet-FL-CL 0.760 0.8143.4 Limitations and possible improvementsForeground segmentation focuses on distinguishing the whole foreground asa single object from a static or minimally changing background, often leadingto simpler binary masks and struggles with complex or dynamic backgroundsand facing major challenges for multi-object segmentation. Therefore, thenatural progression in this case is to adapt VOS methods as demonstratedin the next technical chapter 4.293.5. CONCLUSION3.5 ConclusionIn this Chapter, we proposed a novel federated learning technique and anencoder-decoder network with multi-heads using multi-domain foregroundsegmentation datasets. The proposed model is called KDNet. KDNet in-stantiates a number of client networks corresponding to the number of dif-ferent datasets. Each client is responsible for training using one of thedatasets. Then, KDNet learns the knowledge gained by each client usingknowledge distillation while minimizing the catastrophic forgetting of knowl-edge gained by previous clients. The results show that the proposed KDNetoutperforms state-of-the-art foreground segmentation models by 1.8% whentrained/tested using the SegTrackV2 dataset and in the second place with asmall gap of 0.06% using DAVIS16. On the other hand, When the modelsare trained using multiple datasets to achieve domain generalization, KDNetoutperforms other models by a noticeable amount of 14.1% and 13% in Jointtraining and fine-tuning, respectively.However, This work faces challenges when it comes to multi-object seg-mentation in the foreground since the focus of this work is to isolate thewhole foreground as a single object. In this case, the future work focus shiftsto a multi-object segmentation capable video segmentation task such as VOSas demonstrated in the next technical chapter 4.30Chapter 4MEM: Mask EnhancementModel4.1 Overview and MotivationIn this chapter, we propose a deep-learning model that achieves state-of-the-art metrics in video object segmentation task by leveraging two maintechniques such as foundational model integration and mask fusion. We em-ploy two different models with two different functions. The first model is theCutie model [31], which represents the state-of-the-art in VOS and is utilizedin this chapter. Memory-based models such as Cutie operate by maintaininga record of previously segmented frames within the video sequence, enablingthe retrieval of features to enhance the segmentation of future frames. Onthe other hand, Cutie often fails when highly similar objects move in closeproximity or occlude each other. In these cases, neither the pixel memorynor the object memory can pick up sufficiently discriminative features for theobject transformer to operate on. The second model is SAM [47], which is alarge foundational model that is trained on the SA-1B dataset, which is oneof the largest segmentation datasets, containing over 1 billion multi-objectmasks across 11 million images. This diverse dataset covers a wide range ofobjects, lighting conditions, and environments, allowing SAM to generalizeacross various segmentation tasks. SAM works by combining a Vision Trans-former (ViT) backbone for feature extraction with a prompt-based approach.Users can provide different types of prompts, such as points, bounding boxes,or free-form text, to guide the model in identifying objects of interest. Themodel processes the prompt and image, generating a high-quality mask forthe object specified. This flexibility allows SAM to perform zero-shot seg-mentation without fine-tuning on new datasets, making it highly adaptableacross domains. However, SAM is not designed for video sequences or videoobject segmentation tasks. To use SAM for segmenting an object in a video,each frame and the object’s location in each frame must be provided indi-vidually and iteratively along the video sequence. The strength that SAMprovides in this work is that it doesn’t need knowledge from previous frames314.2. METHODOLOGYto segment an object like Cutie.Given the mentioned strengths and weaknesses of both models, fusingthe output masks of these two models will produce a model with bettergeneralization and segmentation accuracy. Additionally, we investigated vi-sually the output masks of both models. As shown in Figure 4.1, there areinstances where SAM performs better than Cutie, while Cutie outperformsSAM in other instances. Hence, fusing these models will produce overallbetter performance.Figure 4.1: Two different frames are selected from two different video se-quences that show our model’s mask in comparison to SAM and Cutie4.2 MethodologyWe propose MEM that leverages the strengths of both existing models. MEMcombines the current frame, SAMmask, and Cutie mask as inputs to generatean improved output mask, ultimately surpassing the individual performanceof each original model. First, we illustrate how SAM is used to segment thetarget object in each video. Then, we explain the proposed MEM architec-ture.4.2.1 VOS with SAMIt was observed that VOS models naturally follow objects and typically createmasks that cover most, if not all, of the object. The approach involvedfinding the bounding box coordinates of all non-zero elements in the VOSmodel output masks, as illustrated in Figure 4.2. These coordinates werethen used to generate the SAM response.324.2. METHODOLOGYFigure 4.2: Getting SAM response after extracting bounding box from VOSmodel. The process is performed in an iterative way overall video sequenceframes.4.2.2 Architecture4.2.2.1 MEMFigure 4.3: MEM model architecture. Each Mask Enhancer block takes thecurrent frame and model mask as input. The Fusion Block takes both MaskEnhancer features and both models’ masks as inputs. y1, y2, and y3 areused only during training time while MEM-Mask and MEM-ORAND arethe model output during inference time.MEM consists of two main components. The first component is the Mask En-hancer, which learns to produce an output better than that of one of the twoprimary models. The second component is the Fusion Block, which combines334.2. METHODOLOGYthe embeddings from both Mask Enhancer blocks to generate an improvedmask as seen in figure 4.3. The main building block of all components is amodified Res-block.4.2.2.2 Modified Res-BlockThe Modified Res-Block comprises three convolution layers with a kernelsize of 3 × 3 and ’same’ padding. A batch normalization layer and a ReLUactivation function follow each convolution layer.4.2.2.3 Stem StageThe Stem Stage extracts a feature map from the image to prepare it forthe Mask Enhancer component. A single Stem Stage is employed to extractimage features for both Mask Enhancer components. This stage consists ofone convolution layer, followed by a batch normalization layer and a ReLUactivation function. The resulting feature map size is 480× 480× 32.4.2.2.4 Mask EnhancerThe Mask Enhancer is utilized twice in this model, each with distinct sets ofweights. Each Mask Enhancer stage is constructed using three consecutiveModified Res-Blocks. The Mask Enhancer takes two inputs: the feature mapgenerated from the Stem Stage and the corresponding model mask scaled by32. These inputs are concatenated along the channel dimension, creating aninput size of 480 × 480 × 64. Due to the ’same’ padding style, the outputdimensions remain unchanged, maintaining the same size as the input.Figure 4.4: Detailed view of Mask Enhancer components. The Stem stagehas shared weights between both Mask Enhancers.344.2. METHODOLOGY4.2.2.5 Fusion BlockThe Fusion Block architecture mirrors that of the Mask Enhancer, comprisingthree consecutive Modified Res-Blocks. The key difference lies in its fourinputs, which increase the number of input channels. These inputs includethe final feature maps from both Mask Enhancer stages, the Cutie maskrepeated 32 times along the channel dimension, and the SAM mask similarlyrepeated 32 times along the channel dimension. This results in an input sizeof 480× 480× 1284.2.2.6 Prediction HeadsThe Prediction Heads are at the end of each component, heads are usedto generate a mask from the feature maps produced by each component.The heads used for the Mask Enhancer stages consist of a single convolutionlayer with a kernel size of 3 × 3 and ’same’ padding, followed by a sigmoidactivation layer. The prediction head for the fusion block is composed of twoconvolution layers with the same kernel size and padding as the other heads,followed by a sigmoid activation layer.4.2.2.7 Post Processing BlockThe Post Processing Block refines the model output by removing noise andapplying a threshold to the output from the sigmoid function, resulting intwo masks: MEM-Mask and MEM-ORAND. MEM-Mask is generated byapplying a threshold of 0.5 to the sigmoid values, converting any pixel valueabove 0.5 to 1 and any value below 0.5 to 0. This threshold has been selectedafter inspecting different threshold values. However, inspection of the MEM-Mask output revealed occasional false positives from the background part ofthe image as noise. To address this issue, the output should be restricted tothe scope of the object in the mask. Given that both Cutie and SAM masksisolate the object with minimal noise, the solution involves performing alogical OR operation on both masks, followed by a logical AND operationbetween the OR-mask and MEM-Mask. This process significantly reducesbackground noise, resulting in the MEM-ORAND Mask.354.3. EXPERIMENTS4.3 ExperimentsSeveral experiments have been conducted to validate the proposed method touse SAM for VOS as well as evaluate the proposed MEM model performanceon standard benchmark datasets.4.3.1 Implementation Details4.3.1.1 OptimizeationThe Adam optimizer is used with a learning rate of 1e-5 and a batch size of1.4.3.1.2 LossThe loss is calculated on each of the three masks (y1,y2,y3). The final lossis the summation of all three losses, as shown in Equation 4.1. ℓ is MeanSquared Error Loss (MSE), yi is the mask, y is the Ground Truth, and L isthe sum of all losses over the three masksL =M∑iℓ(yi, y) (4.1)4.3.1.3 TrainingThe model is trained on the training sets of both DAVIS17 and youTube-VOS individually. We train the model on each object separately. Afterisolating each object in a separate mask, both forward and backpropagationare performed in an n-stream manner, where n is the number of objects inthe mask.4.3.1.4 TestingThe model has been evaluated locally and on datasets’ competition servers.For DAVIS-17 val, the model has been evaluated locally and both the Jaccardindex and F-score have been calculated locally. For both the DAVIS-17 testand YouTube VOS-2019 val, the model output has been evaluated on thecompetition servers. The main evaluation metric is J&F [68], which is theaverage of both the Jaccard index and F-score.364.3. EXPERIMENTS4.3.2 DatasetsDensely Annotated VIdeo Segmentation 2016 (DAVIS16) [68] is aforeground segmentation dataset. DAVIS consists of 50 videos. Each videohas several frames ranging from 50 to 104. In each video, a single object isannotated, which is the object of interest in this video.Densely Annotated VIdeo Segmentation 2017 (DAVIS17)[1] isa benchmark for video object segmentation, offering high-resolution videoswith pixel-level annotations for diverse and challenging sequences. It includes60 videos for training, 30 for validation, and 60 for testing (30 in the test-dev set and 30 in the test-challenge set). It supports semi-supervised andunsupervised segmentation tasks, providing detailed evaluation metrics likethe Jaccard index.YouTube-VOS 2019 dataset [35] is a large-scale benchmark for videoobject segmentation, offering high-resolution videos with detailed annota-tions. It includes 3,471 videos for training, 507 for validation, and 541for testing. Supporting both semi-supervised and unsupervised tasks, thisdataset provides extensive data for training and benchmarking segmentationmodels.4.3.3 Unsupervised VOS SAM ResultsTable 4.1: Comparison between unsupervised VOS J&F scores for the state-of-the-art models and SAM for VOS.DAVIS-16 val DAVIS-17 val DAVIS-17 testMethod J&F J&F J&FRTNet [69] 85.2 - -PMN [70] 85.9 - -UnOVOST [71] - 67.9 58.0Propose-Reduce [72] - 70.4 -DEVA [33] 88.9 1○ 73.4 2○ 62.1 2○SAM [47](Modified for VOS) 87.1 2○ 86.6 1○ 78.4 1○As seen in Table 4.1, The SAM model for video object segmentation (VOS)achieved top J&F scores of 86.6% on the DAVIS-17 validation set and 78.4%on the DAVIS-17 test set. However, on the DAVIS-16 validation set, SAMscored 87.1%, which is the second-best score after DEVA’s 88.9%. Unlike374.3. EXPERIMENTSVOS datasets like DAVIS-17 and YouTube-VOS, DAVIS-16 focuses on seg-menting all foreground objects as a single entity. This difference explains thedrop in SAM’s performance on DAVIS-16, as it was trained on datasets thatinvolve multiple objects and parts segmentation.4.3.4 MEM ResultsMEM is compared to state-of-the-art models on DAVIS 2017 validation/testdevand YouTubeVOS validation standard benchmarks. Both Jaccard index J ,F1-score F , and their average J&F are reported for all benchmark datasets.As observed in Table 4.2, MEM model is achieving the state-of-the-artJ&F scores on both DAVIS-17 val and DAVIS-17 test with 89.0% and 86.0%respectively. However, observing the results on Youtube-VOS, MEM modelstill achieves a state-of-the-art overall G score of 86.6% despite achievinglower Fu and Fs scores than Cutie.384.3. EXPERIMENTSTable 4.2: Comparison of VOS state-of-the-art models on two different datasets DAVIS-17 and YouTube VOSDAVIS-17 val DAVIS-17 test YouTube VOS-2019 valMethod J&F J F J&F J F G Js Fs Ju FuSTCN [73] 85.4 82.2 88.6 76.1 72.7 79.6 82.7 81.1 85.4 78.2 85.9AOT-R50 [27] 84.9 82.3 87.5 79.6 75.9 83.3 85.3 83.9 88.8 79.9 88.5RDE [74] 84.2 80.8 87.5 77.4 73.6 81.2 81.9 81.1 85.5 76.2 84.8JointFormer [75] - - - 65.6 61.7 69.4 73.3 75.2 78.5 65.8 73.6XMem [26] 86.0 82.8 89.2 79.6 76.1 83.0 85.6 84.1 88.5 81.0 88.9 2○DeAOT-R50 [76] 86.0 83.1 88.9 82.8 79.1 86.5 85.3 84.2 89.0 79.9 88.2DEVA [33] 87.0 83.8 90.2 82.6 78.9 86.4 85.4 84.9 89.4 3○ 79.6 87.8Cutie-base [31] 88.8 3○ 85.6 3○ 91.9 1○ 85.3 2○ 81.4 3○ 89.3 2○ 86.5 2○ 85.4 3○ 90.0 1○ 81.3 2○ 89.3 1○SAM [47](Modified for VOS) 86.6 83.4 89.7 78.4 75.2 81.7 78.0 80.0 82.5 71.1 78.3MEM (Ours) 88.9 2○ 86.7 2○ 91.1 3○ 85.2 3○ 81.6 2○ 88.7 3○ 86.3 3○ 85.7 2○ 89.4 3○ 81.2 3○ 88.7 3○MEM-ORAND (Ours) 89.0 1○ 86.8 1○ 91.2 2○ 86.0 1○ 82.6 1○ 89.5 1○ 86.6 1○ 86.1 1○ 89.7 2○ 81.6 1○ 88.9 2○394.3. EXPERIMENTS4.3.5 Ablation StudyIn this section, we show other methods that have been tested to employ SAMfor VOS tasks.4.3.5.1 Bounding Box Regression Head and Object TrackingModelAs illustrated in Figure 4.5, a bounding box regression head was imple-mented. This head took SAM embeddings for two consecutive frames asinput and output the bounding box coordinates in the form of (X1, Y1, X2,Y2).Figure 4.5: Bounding Box Regression Head architecture. Both SAM En-coders have shared weights. Both Encoder embeddings are fed to a fullyconnected layer followers by ReLU then another fully connected layer fol-lowed by a Sigmoid to calculate a normalized bounding box corner coordi-nates values.The OStrack [77] model was evaluated and showed acceptable results andIOU scores but to use SAM for the VOS task, higher IOU scores were needed.4.3.5.2 Get Bounding Box From VOS Model OutputAs seen in figure 4.2, the bounding box generation algorithm is applied tothe output masks from the Cutie model and then provided to SAM alongwith the corresponding image. This method shows the best results amongother methods as it leverages the tracking nature of VOS models.404.4. LIMITATIONS AND POSSIBLE IMPROVEMENTS4.3.5.3 Modified SAM ResultsTable 4.3: The highest IOU results are reported in this table to show thedifference between all three tested methods. Several experiments have beenconducted to develop a better bounding box regression model and the re-ported result was the best achieved results for this method. For the OSTrackmodel, it has been used without any training from our side. These resultsare reported after using the training set of DAVIS-17Method Bounding Box Regression OS Track [77] IOU from VOSIOU Score 56.1 78.4 97.5For evaluation, a standard metric Intersection Over Union (IOU) is usedto measure how accurate is the predicted bounding box in comparison tothe ground truth bounding box. Ground truth bounding boxes have beengenerated by applying non-zero value analysis on ground truth masks similarto what is shown in Figure 4.2.As seen in Table 4.3, getting the bounding box from the VOS model’soutput masks resulted in the highest IOU of 97.5% while the bounding boxregression model resulted in an IOU of 56.1% which is not suitable for ourapplication as the bounding boxes need to be as accurate as possible to avoidmissed or partial segmentation with SAM.4.4 Limitations and Possible ImprovementsUpon reviewing the results, we notice that MEM achieves less-than-idealF1 scores on certain benchmarks. This shortcoming arises from the model’sdifficulty in accurately capturing the fine details of the outer silhouettes ofsome objects. This issue could be tackled in future work, considering recentadvancements in fine edge detection.414.5. CONCLUSION4.5 ConclusionThis chapter introduces MEM as a novel approach to video object segmenta-tion by combining the strengths of the Segment Anything Model (SAM) andthe Cutie model. Our results show that MEM enhances segmentation accu-racy, achieving top J&F scores on standard benchmark datasets. The inte-gration of SAM’s zero-shot segmentation capabilities with Cutie’s memory-based approach allows MEM to effectively handle complex video sequenceswith occlusions and highly similar objects. Despite MEM’s limitations, theexperiments conducted validate the robustness and MEM’s ability to achievehigher J and J&F scores than the state-of-the-art model due to the ef-fectiveness of foundational model integration techniques proposed in thischapter, making it a valuable contribution to the field of video object seg-mentation. Future work will explore further optimization and enhancementof MEM.42Chapter 5Conclusions and Future WorkThis thesis begins by presenting the motivation and objectives of the research,aiming to enhance video object segmentation through innovative techniques.The primary contributions of the introduction chapter 1, include definingthe problem space and setting the stage for the approaches explored in tech-nical chapters. The introduction highlights the problems in today’s videosegmentation as well as the importance of accurate video segmentation invarious applications, such as autonomous driving and video surveillance, andoutlines the potential impact of the proposed solutions on these fields.In chapter 2, an overview of existing video segmentation tasks and tech-niques is provided with the main focus on Foreground segmentation andvideo object segmentation. The key contribution here is the presentation ofcurrent methods and state-of-the-art methods, identifying their implementedmethods. This chapter serves as the foundation for understanding the ad-vancements proposed later in the thesis. The discussion on various segmen-tation approaches, including supervised, semi-supervised, and unsupervisedmethods, sets the context for the need for improved models like KDNet andMEM.Chapter 3 introduces a novel approach to foreground segmentation throughdomain generalization using federated learning. The main contribution is thedevelopment and evaluation of the KDNet architecture, which enables effec-tive learning across different domains without sharing raw data. The resultsdemonstrate significant improvements in segmentation accuracy and robust-ness across multiple datasets, showcasing the potential of federated learningin enhancing generalization capabilities when paired with knowledge distil-lation and continual learning.Chapter 4 presents the Mask Enhancement Model (MEM) which lever-ages the strengths of existing state-of-the-art models (SAM and Cutie) byfusing their outputs to generate superior segmentation masks. The key con-tributions include the design of the MEM architecture and its components,such as the Mask Enhancer and Fusion Block, which collectively improvesegmentation accuracy. Experimental shows the model’s ability to achievestate-of-the-art performance.43In conclusion, this thesis has explored innovative approaches and learningparadigms to advancing video segmentation, addressing critical challengesthrough the development of KDNet and MEM. By leveraging federatedlearning, continual learning, knowledge distillation, and few-shot learning,we demonstrated significant improvements in foreground segmentation ac-curacy and robustness across diverse datasets without compromising dataprivacy as a bonus. The MEM model, which integrates the strengths of ex-isting segmentation techniques, showed remarkable effectiveness in enhancingmask quality, particularly in out-of-domain scenarios. These contributionsunderscore the potential of the proposed methodologies to impact real-worldapplications, such as autonomous driving and video surveillance. The find-ings of this research help advance video segmentation and lay new pathsfor future explorations aimed at further enhancing segmentation accuracyand generalization capabilities through the integration of advanced learningparadigms.The possible future work proposed in this thesis is, 1) improving MEM’sability to capture fine details of object silhouettes. This involves leverag-ing advancements in fine edge detection techniques to enhance segmentationaccuracy. 2) further optimizing and enhancing the MEM architecture toimprove performance and efficiency. This could include refining the MaskEnhancer and Fusion Block components to better handle complex video se-quences with occlusions and highly similar objects as well as adapting sometechniques from KDNet to enhance MEM’s generalization.44Bibliography[1] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, AlexSorkine-Hornung, and Luc Van Gool, “The 2017 davis challenge on videoobject segmentation,” arXiv preprint arXiv:1704.00675, 2017.[2] Pakorn KaewTraKulPong and Richard Bowden, “An improved adaptivebackground mixture model for real-time tracking with shadow detec-tion,” Video-based surveillance systems: Computer vision and distributedprocessing, pp. 135–144, 2002.[3] Chris Stauffer and W Eric L Grimson, “Adaptive background mixturemodels for real-time tracking,” in Proceedings. 1999 IEEE computersociety conference on computer vision and pattern recognition (Cat. NoPR00149). IEEE, 1999, vol. 2, pp. 246–252.[4] S. Z. Li, “Markov random field models in computer vision,” in ComputerVision — ECCV ’94, Jan-Olof Eklundh, Ed., Berlin, Heidelberg, 1994,pp. 361–370, Springer Berlin Heidelberg.[5] John Lafferty, Andrew McCallum, Fernando Pereira, et al., “Condi-tional random fields: Probabilistic models for segmenting and labelingsequence data,” in Icml. Williamstown, MA, 2001, vol. 1, p. 3.[6] Marti A. Hearst, Susan T Dumais, Edgar Osuna, John Platt, and Bern-hard Scholkopf, “Support vector machines,” IEEE Intelligent Systemsand their applications, vol. 13, no. 4, pp. 18–28, 1998.[7] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner,“Gradient-based learning applied to document recognition,” Proceedingsof the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.[8] Jonathan Long, Evan Shelhamer, and Trevor Darrell, “Fully convolu-tional networks for semantic segmentation,” in Proceedings of the IEEEconference on computer vision and pattern recognition, 2015, pp. 3431–3440.45Bibliography[9] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Mur-phy, and Alan L Yuille, “Deeplab: Semantic image segmentation withdeep convolutional nets, atrous convolution, and fully connected crfs,”IEEE transactions on pattern analysis and machine intelligence, vol. 40,no. 4, pp. 834–848, 2017.[10] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick, “Maskr-cnn,” in Proceedings of the IEEE international conference on computervision, 2017, pp. 2961–2969.[11] Olaf Ronneberger, Philipp Fischer, and Thomas Brox, “U-net: Convo-lutional networks for biomedical image segmentation,” in Medical imagecomputing and computer-assisted intervention–MICCAI 2015: 18th in-ternational conference, Munich, Germany, October 5-9, 2015, proceed-ings, part III 18. Springer, 2015, pp. 234–241.[12] Suyog Dutt Jain and Kristen Grauman, “Supervoxel-consistent fore-ground propagation in video,” in Computer Vision–ECCV 2014: 13thEuropean Conference, Zurich, Switzerland, September 6-12, 2014, Pro-ceedings, Part IV 13. Springer, 2014, pp. 656–671.[13] Hajer Fradi and Jean-Luc Dugelay, “Robust foreground segmentationusing improved gaussian mixture model and optical flow,” in 2012 In-ternational Conference on Informatics, Electronics & Vision (ICIEV).IEEE, 2012, pp. 248–253.[14] Long Ang Lim and Hacer Yalim Keles, “Foreground segmentation usingconvolutional neural networks for multiscale feature encoding,” PatternRecognition Letters, vol. 112, pp. 256–262, 2018.[15] Yi Wang, Zhiming Luo, and Pierre-Marc Jodoin, “Interactive deep learn-ing method for segmenting moving objects,” Pattern Recognition Letters,vol. 96, pp. 66–75, 2017.[16] Ziqin Wang, Jun Xu, Li Liu, Fan Zhu, and Ling Shao, “Ranet: Rankingattention network for fast video object segmentation,” in Proceedingsof the IEEE/CVF International Conference on Computer Vision, 2019,pp. 3978–3987.[17] Ho Kei Cheng, Yu-Wing Tai, and Chi-Keung Tang, “Rethinking space-time networks with improved memory coverage for efficient video objectsegmentation,” Advances in Neural Information Processing Systems, vol.34, pp. 11781–11794, 2021.46Bibliography[18] Islam Osman, Mohamed Abdelpakey, and Mohamed S Shehata, “Trans-blast: Self-supervised learning using augmented subspace with trans-former for background/foreground separation,” in Proceedings of theIEEE/CVF International Conference on Computer Vision, 2021, pp.215–224.[19] Fuxin Li, Taeyoung Kim, Ahmad Humayun, David Tsai, and James MRehg, “Video segmentation by tracking many figure-ground segments,”in Proceedings of the IEEE International Conference on Computer Vi-sion, 2013, pp. 2192–2199.[20] Yi Wang, Pierre-Marc Jodoin, Fatih Porikli, Janusz Konrad, YannickBenezeth, and Prakash Ishwar, “Cdnet 2014: An expanded change de-tection benchmark dataset,” in Proceedings of the IEEE conference oncomputer vision and pattern recognition workshops, 2014, pp. 387–394.[21] James J Gibson, “The perception of the visual world.,” 1950.[22] Pedro F Felzenszwalb and Daniel P Huttenlocher, “Efficient graph-basedimage segmentation,” International journal of computer vision, vol. 59,pp. 167–181, 2004.[23] Paul Voigtlaender, Yuning Chai, Florian Schroff, Hartwig Adam, Bas-tian Leibe, and Liang-Chieh Chen, “Feelvos: Fast end-to-end embed-ding learning for video object segmentation,” in Proceedings of theIEEE/CVF conference on computer vision and pattern recognition, 2019,pp. 9481–9490.[24] Seoung Wug Oh, Joon-Young Lee, Ning Xu, and Seon Joo Kim, “Videoobject segmentation using space-time memory networks,” in Proceedingsof the IEEE/CVF International Conference on Computer Vision, 2019,pp. 9226–9235.[25] Seoung Wug Oh, Joon-Young Lee, Kalyan Sunkavalli, and Seon JooKim, “Fast video object segmentation by reference-guided mask propa-gation,” in Proceedings of the IEEE conference on computer vision andpattern recognition, 2018, pp. 7376–7385.[26] Ho Kei Cheng and Alexander G Schwing, “Xmem: Long-term video ob-ject segmentation with an atkinson-shiffrin memory model,” in EuropeanConference on Computer Vision. Springer, 2022, pp. 640–658.47Bibliography[27] Zongxin Yang, Yunchao Wei, and Yi Yang, “Associating objects withtransformers for video object segmentation,” Advances in Neural Infor-mation Processing Systems, vol. 34, pp. 2491–2502, 2021.[28] Wenyu Lv, Yian Zhao, Qinyao Chang, Kui Huang, Guanzhong Wang,and Yi Liu, “Rt-detrv2: Improved baseline with bag-of-freebies for real-time detection transformer,” 2024.[29] Brendan Duke, Abdalla Ahmed, Christian Wolf, Parham Aarabi, andGraham W Taylor, “Sstvos: Sparse spatiotemporal transformers forvideo object segmentation,” in Proceedings of the IEEE/CVF Confer-ence on Computer Vision and Pattern Recognition, 2021, pp. 5912–5921.[30] Junke Wang, Dongdong Chen, Zuxuan Wu, Chong Luo, Chuanxin Tang,Xiyang Dai, Yucheng Zhao, Yujia Xie, Lu Yuan, and Yu-Gang Jiang,“Look before you match: Instance understanding matters in video objectsegmentation,” in Proceedings of the IEEE/CVF conference on computervision and pattern recognition, 2023, pp. 2268–2278.[31] Ho Kei Cheng, Seoung Wug Oh, Brian Price, Joon-Young Lee, andAlexander Schwing, “Putting the object back into video object seg-mentation,” in Proceedings of the IEEE/CVF Conference on ComputerVision and Pattern Recognition (CVPR), June 2024, pp. 3151–3161.[32] Jinyu Yang, Mingqi Gao, Zhe Li, Shang Gao, Fangjing Wang, andFeng Zheng, “Track anything: Segment anything meets videos,” arXivpreprint arXiv:2304.11968, 2023.[33] Ho Kei Cheng, Seoung Wug Oh, Brian Price, Alexander Schwing, andJoon-Young Lee, “Tracking anything with decoupled video segmenta-tion,” in Proceedings of the IEEE/CVF International Conference onComputer Vision, 2023, pp. 1316–1326.[34] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, DavidWarde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio, “Gen-erative adversarial networks,” Communications of the ACM, vol. 63, no.11, pp. 139–144, 2020.[35] Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jian-chao Yang, and Thomas Huang, “Youtube-vos: A large-scale video ob-ject segmentation benchmark,” arXiv preprint arXiv:1809.03327, 2018.48Bibliography[36] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, andChristoph H Lampert, “icarl: Incremental classifier and representationlearning,” in Proceedings of the IEEE conference on Computer Visionand Pattern Recognition, 2017, pp. 2001–2010.[37] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim, “Continuallearning with deep generative replay,” Advances in neural informationprocessing systems, vol. 30, 2017.[38] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guil-laume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ra-malho, Agnieszka Grabska-Barwinska, et al., “Overcoming catastrophicforgetting in neural networks,” Proceedings of the national academy ofsciences, vol. 114, no. 13, pp. 3521–3526, 2017.[39] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, MarcusRohrbach, and Tinne Tuytelaars, “Memory aware synapses: Learn-ing what (not) to forget,” in Proceedings of the European conference oncomputer vision (ECCV), 2018, pp. 139–154.[40] Arun Mallya, Dillon Davis, and Svetlana Lazebnik, “Piggyback: Adapt-ing a single network to multiple tasks by learning to mask weights,”in Proceedings of the European conference on computer vision (ECCV),2018, pp. 67–82.[41] Amir Rosenfeld and John K Tsotsos, “Incremental learning throughdeep adaptation,” IEEE transactions on pattern analysis and machineintelligence, vol. 42, no. 3, pp. 651–663, 2018.[42] Yansheng Wang, Yongxin Tong, Dingyuan Shi, and Ke Xu, “An efficientapproach for cross-silo federated learning to rank,” in 2021 IEEE 37thInternational Conference on Data Engineering (ICDE), 2021, pp. 1128–1139.[43] Muhammad Habib ur Rehman, Ahmed Mukhtar Dirir, Khaled Salah,Ernesto Damiani, and Davor Svetinovic, “Trustfed: A framework for fairand trustworthy cross-device federated learning in iiot,” IEEE Transac-tions on Industrial Informatics, vol. 17, no. 12, pp. 8485–8494, 2021.[44] Dong Yang, Ziyue Xu, Wenqi Li, Andriy Myronenko, Holger R Roth,Stephanie Harmon, Sheng Xu, Baris Turkbey, Evrim Turkbey, Xiaosong49BibliographyWang, et al., “Federated semi-supervised learning for covid region seg-mentation in chest ct using multi-national data from china, italy, japan,”Medical image analysis, vol. 70, pp. 101992, 2021.[45] Bernardo Camajori Tedeschini, Stefano Savazzi, Roman Stoklasa, LucaBarbieri, Ioannis Stathopoulos, Monica Nicoli, and Luigi Serio, “De-centralized federated learning for healthcare networks: A case study ontumor segmentation,” IEEE Access, vol. 10, pp. 8693–8708, 2022.[46] Chaoning Zhang, Dongshen Han, Yu Qiao, Jung Uk Kim, Sung-HoBae, Seungkyu Lee, and Choong Seon Hong, “Faster segment any-thing: Towards lightweight sam for mobile applications,” arXiv preprintarXiv:2306.14289, 2023.[47] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rol-land, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C.Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick, “Segment any-thing,” in Proceedings of the IEEE/CVF International Conference onComputer Vision (ICCV), October 2023, pp. 4015–4026.[48] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, andTimothy M Hospedales, “Learning to compare: Relation network forfew-shot learning,” in Proceedings of the IEEE conference on computervision and pattern recognition, 2018, pp. 1199–1208.[49] Wenbin Li, Lei Wang, Jinglin Xu, Jing Huo, Yang Gao, and Jiebo Luo,“Revisiting local descriptor based image-to-class measure for few-shotlearning,” in Proceedings of the IEEE/CVF conference on computervision and pattern recognition, 2019, pp. 7260–7268.[50] Yann Lifchitz, Yannis Avrithis, Sylvaine Picard, and Andrei Bursuc,“Dense classification and implanting for few-shot learning,” in Proceed-ings of the IEEE/CVF conference on computer vision and pattern recog-nition, 2019, pp. 9258–9267.[51] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, andRoss Girshick, “Masked autoencoders are scalable vision learners,” inProceedings of the IEEE/CVF conference on computer vision and pat-tern recognition, 2022, pp. 16000–16009.[52] Da Chen, Yuefeng Chen, Yuhong Li, Feng Mao, Yuan He, and HuiXue, “Self-supervised learning for few-shot image classification,” in50BibliographyICASSP 2021-2021 IEEE International Conference on Acoustics, Speechand Signal Processing (ICASSP). IEEE, 2021, pp. 1745–1749.[53] Yang Li, Feiyun Xu, and Chi-Guhn Lee, “Self-supervised metalearninggenerative adversarial network for few-shot fault diagnosis of hoistingsystem with limited data,” IEEE Transactions on Industrial Informat-ics, vol. 19, no. 3, pp. 2474–2484, 2022.[54] Terrance DeVries and Graham W Taylor, “Improved regulariza-tion of convolutional neural networks with cutout,” arXiv preprintarXiv:1708.04552, 2017.[55] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz, “mixup: Beyond empirical risk minimization,” arXiv preprintarXiv:1710.09412, 2017.[56] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Jun-suk Choe, and Youngjoon Yoo, “Cutmix: Regularization strategy totrain strong classifiers with localizable features,” in Proceedings ofthe IEEE/CVF international conference on computer vision, 2019, pp.6023–6032.[57] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, SanjeevSatheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla,Michael Bernstein, et al., “Imagenet large scale visual recognition chal-lenge,” International journal of computer vision, vol. 115, no. 3, pp.211–252, 2015.[58] Shuangjie Xu, Daizong Liu, Linchao Bao, Wei Liu, and Pan Zhou, “Mhp-vos: Multiple hypotheses propagation for video object segmentation,”in Proceedings of the IEEE/CVF Conference on Computer Vision andPattern Recognition, 2019, pp. 314–323.[59] Zongxin Yang, Yunchao Wei, and Yi Yang, “Collaborative video ob-ject segmentation by foreground-background integration,” in EuropeanConference on Computer Vision. Springer, 2020, pp. 332–348.[60] Long Ang Lim and Hacer Yalim Keles, “Learning multi-scale featuresfor foreground segmentation,” Pattern Analysis and Applications, vol.23, no. 3, pp. 1369–1380, 2020.[61] Islam Osman, Agwad Eltantawy, and Mohamed S Shehata, “Task-basedparameter isolation for foreground segmentation without catastrophic51Bibliographyforgetting using multi-scale region and edges fusion network,” Imageand Vision Computing, vol. 113, pp. 104248, 2021.[62] Trung-Nghia Le and Akihiro Sugimoto, “Deeply supervised 3d recurrentfcn for salient object detection in videos.,” in BMVC, 2017, vol. 1, p. 3.[63] Hieu Le, Vu Nguyen, Chen-Ping Yu, and Dimitris Samaras, “Geodesicdistance histogram feature for video segmentation,” in Asian Conferenceon Computer Vision. Springer, 2016, pp. 275–290.[64] Yukun Su, Jingliang Deng, Ruizhou Sun, Guosheng Lin, and QingyaoWu, “A unified transformer framework for group-based segmentation:Co-segmentation, co-saliency detection and video salient object detec-tion,” arXiv preprint arXiv:2203.04708, 2022.[65] Hongmei Song, Wenguan Wang, Sanyuan Zhao, Jianbing Shen, and Kin-Man Lam, “Pyramid dilated deeper convlstm for video salient objectdetection,” in Proceedings of the European conference on computer vision(ECCV), 2018, pp. 715–731.[66] Trung-Nghia Le and Akihiro Sugimoto, “Video salient object detectionusing spatiotemporal deep features,” IEEE Transactions on Image Pro-cessing, vol. 27, no. 10, pp. 5002–5015, 2018.[67] Thangarajah Akilan and QM Jonathan Wu, “sendec: An improvedimage to image cnn for foreground localization,” IEEE Transactions onIntelligent Transportation Systems, vol. 21, no. 10, pp. 4435–4443, 2019.[68] Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool,Markus Gross, and Alexander Sorkine-Hornung, “A benchmark datasetand evaluation methodology for video object segmentation,” in Proceed-ings of the IEEE conference on computer vision and pattern recognition,2016, pp. 724–732.[69] Sucheng Ren, Wenxi Liu, Yongtuo Liu, Haoxin Chen, Guoqiang Han,and Shengfeng He, “Reciprocal transformations for unsupervised videoobject segmentation,” in Proceedings of the IEEE/CVF conference oncomputer vision and pattern recognition, 2021, pp. 15455–15464.[70] Minhyeok Lee, Suhwan Cho, Seunghoon Lee, Chaewon Park, and Sangy-oun Lee, “Unsupervised video object segmentation via prototype mem-ory network,” in Proceedings of the IEEE/CVF winter conference onapplications of computer vision, 2023, pp. 5924–5934.52[71] Jonathon Luiten, Idil Esen Zulfikar, and Bastian Leibe, “Unovost: Unsu-pervised offline video object segmentation and tracking,” in Proceedingsof the IEEE/CVF winter conference on applications of computer vision,2020, pp. 2000–2009.[72] Huaijia Lin, Ruizheng Wu, Shu Liu, Jiangbo Lu, and Jiaya Jia, “Videoinstance segmentation with a propose-reduce paradigm,” in Proceedingsof the IEEE/CVF International Conference on Computer Vision, 2021,pp. 1739–1748.[73] Ho Kei Cheng, Yu-Wing Tai, and Chi-Keung Tang, “Rethinking space-time networks with improved memory coverage for efficient video objectsegmentation,” Advances in Neural Information Processing Systems, vol.34, pp. 11781–11794, 2021.[74] Mingxing Li, Li Hu, Zhiwei Xiong, Bang Zhang, Pan Pan, and DongLiu, “Recurrent dynamic embedding for video object segmentation,”in Proceedings of the IEEE/CVF Conference on Computer Vision andPattern Recognition, 2022, pp. 1332–1341.[75] Jiaming Zhang, Yutao Cui, GangshanWu, and LiminWang, “Joint mod-eling of feature, correspondence, and a compressed memory for videoobject segmentation,” arXiv preprint arXiv:2308.13505, 2023.[76] Zongxin Yang and Yi Yang, “Decoupling features in hierarchical propa-gation for video object segmentation,” Advances in Neural InformationProcessing Systems, vol. 35, pp. 36324–36336, 2022.[77] Botao Ye, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen,“Joint feature learning and relation modeling for tracking: A one-streamframework,” in European Conference on Computer Vision. Springer,2022, pp. 341–357.53Appendix AImplementation andReproducibility DetailsAll experiments in this work use PyTorch 1.13 and Python 3.9. Training,testing, and inference on all studies conducted were performed using twoNVIDIA A6000 GPUs and a single NVIDIA GV100 GPU.54
Thesis/Dissertation
eng
Computer Science
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Shehata, Mohamed S.
Advancing video segmentation by integrating advanced learning paradigms
Text
Still Image