Applied Science, Faculty of
Electrical and Computer Engineering, Department of
DSpace
UBCV
Arya, Atrin
Master of Applied Science - MASc
University of British Columbia
Federated learning (FL) has gained prominence in the medical domain for enabling collaborative machine learning across decentralized datasets while preserving patient privacy. However, key challenges such as catastrophic forgetting and local overfitting hinder the effectiveness of FL models in heterogeneous healthcare environments. This thesis addresses these challenges through two complementary studies. The first study introduces Federated Impressions, a synthetic data generation method that extracts the knowledge of the global model without compromising privacy. By training the local model on these impressions, the method mitigates catastrophic forgetting, allowing the model to retain critical knowledge while generalizing effectively across diverse datasets. This approach ensures robustness and adaptability to evolving medical data distributions. The second study enhances FL for medical imaging by leveraging Vision Transformers (ViTs) in a DualPrompt architecture. Prompts are divided into shared and group prompts: shared prompts capture generalized medical features and are trained exclusively on Federated Impressions alongside local data to prevent overfitting, while group prompts adapt to specific client tasks using local data. This method effectively mitigates catastrophic forgetting, preserving personalization through group prompts while maintaining global generalization via shared prompts. It strikes a balance between retaining global medical knowledge and adapting to localized data distributions. Experimental evaluations on benchmark medical datasets demonstrate that the proposed approaches improve global generalization by avoiding local overfitting. These advancements make FL more robust and effective in heterogeneous medical environments, addressing critical challenges in privacy, scalability, and catastrophic forgetting. This thesis lays the groundwork for scalable and adaptive FL frameworks tailored to medical applications, fostering innovation in privacy-preserving healthcare AI.
Overcoming Catastrophic Forgettingin Heterogeneous Federated LearningbyAtrin AryaA THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFMASTER OF APPLIED SCIENCEinThe Faculty of Graduate and Postdoctoral Studies(Electrical and Computer Engineering)THE UNIVERSITY OF BRITISH COLUMBIA(Vancouver)December 2024© Atrin Arya 2024The following individuals certify that they have read, and recommend tothe Faculty of Graduate and Postdoctoral Studies for acceptance, the thesisentitled:Overcoming Catastrophic Forgetting in Heterogeneous FederatedLearningsubmitted by Atrin Arya in partial fulfillment of the requirements for thedegree of Master of Applied Science in Electrical and ComputerEngineering.Examining Committee:Purang Abolmaesumi, Professor, Electrical and Computer Engineering, UBCCo-supervisorXiaoxiao Li, Professor, Electrical and Computer Engineering, UBCCo-supervisorRobert Rohling, Professor, Electrical and Computer Engineering, UBCSupervisory Committee MemberiiAbstractFederated learning (FL) has gained prominence in the medical domain forenabling collaborative machine learning across decentralized datasets whilepreserving patient privacy. However, key challenges such as catastrophicforgetting and local overfitting hinder the effectiveness of FL models in het-erogeneous healthcare environments. This thesis addresses these challengesthrough two complementary studies.The first study introduces Federated Impressions, a synthetic data genera-tion method that extracts the knowledge of the global model without com-promising privacy. By training the local model on these impressions, themethod mitigates catastrophic forgetting, allowing the model to retain crit-ical knowledge while generalizing effectively across diverse datasets. Thisapproach ensures robustness and adaptability to evolving medical data dis-tributions.The second study enhances FL for medical imaging by leveraging VisionTransformers (ViTs) in a DualPrompt architecture. Prompts are dividedinto shared and group prompts: shared prompts capture generalized medicalfeatures and are trained exclusively on Federated Impressions alongside localdata to prevent overfitting, while group prompts adapt to specific client tasksusing local data. This method effectively mitigates catastrophic forgetting,preserving personalization through group prompts while maintaining globalgeneralization via shared prompts. It strikes a balance between retainingglobal medical knowledge and adapting to localized data distributions.Experimental evaluations on benchmark medical datasets demonstrate thatthe proposed approaches improve global generalization by avoiding localoverfitting. These advancements make FL more robust and effective in het-erogeneous medical environments, addressing critical challenges in privacy,scalability, and catastrophic forgetting. This thesis lays the groundworkfor scalable and adaptive FL frameworks tailored to medical applications,fostering innovation in privacy-preserving healthcare AI.iiiLay SummaryMedical research increasingly relies on artificial intelligence to analyze dataand develop insights, but patient privacy often limits data sharing. Fed-erated learning (FL) addresses this challenge by enabling AI models to betrained collaboratively across multiple hospitals or institutions without shar-ing sensitive patient data. However, current FL methods struggle to bal-ance learning from different medical datasets without forgetting previouslylearned information. This thesis proposes new approaches to improve FLin medical applications. The first method uses synthetic data to help AImodels retain knowledge while learning from new data. The second methodleverages advanced techniques from Vision Transformers to ensure the modelcaptures both general medical knowledge and specific details unique to eachhospital’s data. These innovations make FL more reliable and effective inhealthcare, paving the way for better AI solutions that respect patient pri-vacy and improve medical outcomes.ivPrefaceThis thesis is based on the research I conducted as part of my graduate stud-ies in the Department of Electrical and Computer Engineering at the Univer-sity of British Columbia (UBC). I was primarily responsible for the identifi-cation and design of the research program, developing the core methodologyfor data generation in FedImpres , performing the experiments on the Retinadataset, and analyzing the data. The development of the federated learningapproach, data generation techniques, and evaluation using medical imagingdatasets were carried out under the supervision and implementation of meand my co-author Sana Ayromlou.In addition to FedImpres, the second study in this thesis focuses on enhanc-ing federated learning using Vision Transformers (ViTs) and DualPromptarchitecture. I was responsible for conceptualizing this novel approach, de-signing and implementing the methodology to mitigate catastrophic forget-ting, and conducting experiments to evaluate its effectiveness in heteroge-neous medical imaging datasets.Several collaborators contributed to this research. My co-author, Sana Ay-romlou, assisted with methodology design, carrying out experiments, andwriting the paper. My other co-author, Armin Saadat, provided feedbackon experimental designs and contributed to discussions during the analysisphase and writing the paper. Dr. Purang Abolmaesumi and Dr. Xiaox-iao Li offered critical guidance throughout the project, particularly duringthe conceptualization and refinement of the methodologies. The work pre-sented in Chapter 2 and Chapter 3 has led to a publication titled “AtrinArya, Sana Ayromlou, Armin Saadat, Purang Abolmaesumi, and XiaoxiaoLi. Federated impression for learning with distributed heterogeneous data,2024. arXiv preprint arXiv:2409.07351” and was presented at Medical Im-age Computing and Computer Assisted Intervention (MICCAI) WorkshopsOctober 2024 in Morocco. The work in Chapter 4 on Vision Transform-ers and DualPrompt builds upon and extends these contributions. I wasresponsible for approximately 60% of the research and writing, while thevPrefaceremaining 40% was contributed by my collaborators.Ethics approval was not required for this research, as it involved publiclyavailable datasets and did not involve direct human or animal subjects.viTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiList of Abbreviations . . . . . . . . . . . . . . . . . . . . . . . . . xiiList of Symbols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiiAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . xvDedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvi1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . 21.2 Scope of Research . . . . . . . . . . . . . . . . . . . . . . . . 32 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.1 Federated Learning . . . . . . . . . . . . . . . . . . . . . . . 42.1.1 Overview of Federated Learning . . . . . . . . . . . . 42.1.2 Advantages of Federated Learning in Healthcare . . . 52.1.3 Challenges in Federated Learning . . . . . . . . . . . 62.1.4 Related Work . . . . . . . . . . . . . . . . . . . . . . 82.2 Catastrophic Forgetting in Heterogeneous Federated Learn-ing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15viiTable of Contents2.2.1 Overview of Catastrophic Forgetting . . . . . . . . . 152.2.2 Related Work . . . . . . . . . . . . . . . . . . . . . . 172.3 Recent Advancements in Federated Learning . . . . . . . . . 212.3.1 Emergence of Foundation Models . . . . . . . . . . . 212.3.2 Visual Prompt Tuning in Federated Learning . . . . . 222.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243 Federated impression for learning with distributed hetero-geneous data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263.1 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263.1.1 Problem Setting . . . . . . . . . . . . . . . . . . . . . 263.1.2 Overview . . . . . . . . . . . . . . . . . . . . . . . . . 273.1.3 Federated Impression . . . . . . . . . . . . . . . . . . 303.1.4 Forgetting-Penalized Local Training . . . . . . . . . . 323.2 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . 343.2.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . 343.2.2 Experimental Settings . . . . . . . . . . . . . . . . . . 363.2.3 Comparison with Baselines . . . . . . . . . . . . . . . 383.2.4 Ablation Studies . . . . . . . . . . . . . . . . . . . . . 403.2.5 Analysis and Discussion . . . . . . . . . . . . . . . . . 423.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444 Federated Prompt Learning with Dual-Prompt Optimiza-tion and Federated Impression Alignment . . . . . . . . . . 474.1 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 474.1.1 Problem Setting . . . . . . . . . . . . . . . . . . . . . 474.1.2 Overview . . . . . . . . . . . . . . . . . . . . . . . . . 494.1.3 Dual-Prompt Framework . . . . . . . . . . . . . . . . 504.1.4 Federated Impression Integration . . . . . . . . . . . 524.2 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . 554.2.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . 554.2.2 Experimental Settings . . . . . . . . . . . . . . . . . . 554.2.3 Comparison with Baselines . . . . . . . . . . . . . . . 574.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 595 Summary and Future Work . . . . . . . . . . . . . . . . . . . 615.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 615.1.1 Study 1: FedImpress . . . . . . . . . . . . . . . . . . 615.1.2 Study 2: PromptFusion . . . . . . . . . . . . . . . . . 625.1.3 Key Contributions . . . . . . . . . . . . . . . . . . . . 62viiiTable of Contents5.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . 635.2.1 Enhanced Synthetic Data Generation . . . . . . . . . 635.2.2 Scalability and Communication Efficiency . . . . . . . 635.2.3 Dynamic Client Participation . . . . . . . . . . . . . 635.2.4 Extension to Multimodal and Sequential Tasks . . . . 645.2.5 Privacy and Security Enhancements . . . . . . . . . . 645.2.6 Clinical Translation and Real-World Applications . . 645.2.7 Theoretical Analysis of Federated Prompts . . . . . . 645.2.8 Continuous and Lifelong Learning in FL . . . . . . . 655.2.9 Broader Benchmarking and Evaluation . . . . . . . . 655.2.10 Open-Source and Community Contribution . . . . . . 655.3 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . 65Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67ixList of Tables3.1 Classification accuracies on BloodMNIST and Retina datasetcompared with the state-of-the-art methods. We reportedFedImpres results using random, CIFAR-10, and medical un-labeled data of the same modality for initialization. Whilemedical initialization consistently performed better, FedImpresoutperformed baselines across all configurations. . . . . . . . 383.2 Classification accuracies on the Retina dataset comparing FedImpreswith the CE loss variant. Both synthesis methods use randomnoise initialization. . . . . . . . . . . . . . . . . . . . . . . . . 414.1 Classification accuracies on BloodMNIST and Retina datasets.PromptFusion demonstrates significant improvements over base-lines, particularly under severe label imbalance and domainshift conditions. . . . . . . . . . . . . . . . . . . . . . . . . . . 59xList of Figures2.1 Catastrophic forgetting occurs when server weights are over-written during local training, causing a loss of previous knowl-edge. To investigate the effect of catastrophic forgetting dur-ing local training in FL, we conducted experiments on Blood-MNIST using the same experimental settings described inSec. 3.2. Specifically, we plot each client’s local model accu-racy over other clients’ data during local training. The ac-curacy drops drastically using FedAvg; however, FedImpresmaintains stable accuracy across clients. . . . . . . . . . . . . 173.1 Our proposed approach, FedImpres, aims to capture the globaldistribution learned by the aggregated server model and dis-till it into a dataset that can be shared with clients. Theapproach consists of two steps: a) First, we perform pixel-wise optimization by starting from unlabeled public data andusing the server model’s predicted pseudo-labels to backprop-agate using Eq.((3.5)). b) Second, to improve local training,we add the synthesized data as a regularizer to the local datausing Eq.((3.6)). This allows us to share the global distilleddistribution with clients and leverage it to improve local train-ing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28xiList of AbbreviationsADMM Alternating Direction Method of MultipliersCE Cross-EntropyCNN Convolutional Neural NetworkDP Differential PrivacyFL Federated LearningGAN Generative Adversarial NetworkGDPR General Data Protection RegulationGP Group PromptsHIPAA Health Insurance Portability and Accountability ActIID Independent and Identically DistributedLDA Latent Dirichlet AllocationSGD Stochastic Gradient DescentSP Shared PromptsVAE Variational AutoencoderViT Vision TransformerxiiList of Symbols{C1, C2, ..., CN} Set of clients participating in the Federated Learning processN Total number of clientsXn Private dataset belonging to client nxni Input sample from client n’s datasetyni Label corresponding to input xniθG Weights of the global model’s feature extractorϕG Weights of the global model’s classifierJ(θG, ϕG) Global classification loss function to minimizeℓ General loss function (e.g., cross-entropy loss)f(x; θG) Feature extraction function of the global modelg(f(x);ϕG) Classification function of the global modelθrG Global model parameters at communication round rθrn Locally updated model parameters of client n at round rV Synthetic dataset generated on the server (federated impressions)vi Synthetic sample from the synthetic dataset Vyˆi Pseudo-label for the synthetic sample viLlocal Cross-entropy loss on the client’s local datasetLglobal Cross-entropy loss on the synthetic dataset (federated impressions)β Weight parameter balancing Llocal and LglobalΛ Lagrangian dual variable matrixxiiiList of Symbolsρ Penalty hyperparameter in the augmented Lagrangian formulation∇(θG,ϕG) Gradient with respect to the global model parameters∇(ϕG) Gradient with respect to the classifier parameters onlyE[·] Expectation operator, used to compute average loss over datasetsxivAcknowledgementsThroughout my graduate studies, I have been fortunate to receive the sup-port and guidance of many individuals. First and foremost, I would like toexpress my deepest gratitude to my supervisors, Purang Abolmaesumi andXiaoxiao Li, for their constant mentorship, encouragement, and insightfulfeedback. Their guidance has been crucial to my academic and professionaldevelopment, and this work would not have been possible without theirsupport.I would also like to extend my heartfelt thanks to Sana Ayromlou and ArminSaadat, whose collaboration and contributions significantly enhanced thiswork. The discussions we had and the feedback they provided were instru-mental in shaping the direction of this research.I am grateful to my colleagues at UBC’s Electrical and Computer Engineer-ing Department for fostering a stimulating research environment and offer-ing continuous support. Special thanks to the Vector Institute for providingresources and tools that were critical to the completion of this project.Finally, I would like to thank my family and friends for their unwaveringsupport and encouragement throughout this journey. Their belief in me hasbeen a source of strength during challenging times.xvDedicationTo my family, for their unconditional love and support throughout this jour-ney.xviChapter 1IntroductionThe rapid advancement of deep learning in recent years has led to itswidespread application in various fields, including healthcare. Medical imag-ing, in particular, has benefited from the powerful capabilities of deep learn-ing models to analyze and classify complex data. However, traditional deeplearning methods typically rely on centralized data collection, where alltraining data is aggregated into a single repository. This poses significantchallenges in healthcare, where sensitive patient data cannot be easily sharedacross institutions due to privacy concerns, data security risks, and regula-tory barriers [1, 2].Federated Learning (FL) has emerged as a promising solution to addressthese issues. FL enables multiple institutions to collaboratively train mod-els without sharing their local datasets, thereby preserving patient privacyand complying with data protection regulations [1, 3, 4]. By keeping datadecentralized and on-site, FL mitigates the risks of data breaches and unau-thorized access, addressing critical privacy concerns prevalent in fields suchas healthcare [5]. Additionally, this decentralized approach reduces the needfor extensive data transfer across networks, which helps maintain data se-curity and optimizes bandwidth usage [6, 7].FL’s structure also leverages the computational resources of individual in-stitutions, making it scalable and adaptable to large-scale projects withoutoverloading any central server [8]. This approach enables the creation ofmodels that can generalize better by training on a diverse set of distributeddata sources. As a result, FL facilitates the development of more robustmodels that reflect the variations present in different datasets, ultimatelyleading to more reliable and adaptable solutions that benefit a wider rangeof applications [9].11.1. Problem Statement1.1 Problem StatementDespite its advantages, FL faces significant challenges when applied to health-care data, which are often heterogeneous. Data heterogeneity arises from dif-ferences in data collection protocols, patient demographics, and disease pro-files across institutions, leading to imbalanced and non-IID (non-independentand identically distributed) data. These discrepancies can degrade modelperformance and hinder convergence, making it difficult for FL systems toachieve consistent results across diverse datasets [3, 7].A major challenge posed by data heterogeneity is catastrophic forgetting.In the context of FL, catastrophic forgetting occurs when local models losepreviously learned knowledge while training on new data distributions. Thisis particularly problematic in FL settings, where models are trained on dis-tinct local datasets before updates are aggregated into a global model. Thevariability in local datasets causes models to overfit to local data, resultingin poor generalization to broader distributions and a lack of robustness inthe global model [10, 11].Traditional FL methods, such as FedAvg, struggle to address these chal-lenges effectively. While FedAvg aggregates local updates to create a globalmodel, it does not account for the diversity and imbalance in client datadistributions, often leading to suboptimal convergence and inconsistent per-formance across clients [1, 12]. This limitation is especially critical in health-care applications, where data variability is inherent, and model reliability isvital for clinical decision-making.This thesis seeks to address the problem of catastrophic forgetting in fed-erated learning through two complementary studies. The first study intro-duces Federated Impressions (FedImpres), a novel synthetic data generationtechnique that captures global knowledge from federated data. By trainingthe global model on these Federated Impressions, catastrophic forgetting ismitigated, enabling robust generalization across heterogeneous clients [13–15].The second study builds upon this by integrating Vision Transformers (ViTs)with DualPrompt [16] architecture in the FL setting. Shared prompts,trained exclusively on Federated Impressions, capture generalizable featuresacross clients, while group prompts, trained on local data, preserve task-specific information. This dual approach minimizes overfitting and furtheraddresses catastrophic forgetting, resulting in a more stable and effectiveglobal model [17, 18].21.2. Scope of Research1.2 Scope of ResearchThis thesis focuses on improving federated learning for medical image clas-sification by addressing the challenge of data heterogeneity during training.The scope of the research includes:• Design and Implementation of FedImpres [13]: Developing asynthetic data generation method that captures global knowledge fromfederated models, mitigating catastrophic forgetting and enhancingrobustness during training.• Integration of Vision Transformers with DualPrompt andFederated Impressions: Utilizing Vision Transformers with prompt-based tuning to create shared and group prompts. This approachensures generalization across clients while maintaining personalizedperformance, further mitigating catastrophic forgetting when utilizedwith federated impressions [16, 19].• Evaluation on Medical Datasets: Assessing the proposed methodsusing benchmark medical imaging datasets, such as BloodMNIST andRetina, which feature significant label imbalance and domain shifts[20, 21].• Comparative Analysis: Comparing the proposed methods with ex-isting FL approaches, such as FedAvg and FedProx, to demonstrateimprovements in classification accuracy, robustness, and the reductionof catastrophic forgetting [1, 22].By addressing the critical challenge of catastrophic forgetting in FL, thisthesis contributes to the development of scalable and effective solutions forprivacy-preserving AI in the medical domain. These advancements pave theway for more robust and reliable federated learning frameworks capable ofhandling the complexities of heterogeneous medical data.3Chapter 2Background2.1 Federated Learning2.1.1 Overview of Federated LearningFederated Learning (FL) is a distributed machine learning paradigm de-signed to enable collaborative model training across multiple decentralizeddata sources while preserving data privacy [1, 7]. Unlike traditional ma-chine learning systems, which require raw data to be collected in a centralrepository for training, FL allows data to remain localized on the client de-vices. Model training is conducted locally, and only model updates, such asgradients or parameter weights, are exchanged with a central server [12].The FL workflow typically begins with a global model initialized by a cen-tral server. This model is sent to participating clients, which independentlytrain the model using their local datasets. Local training involves minimiz-ing an objective function, such as a loss function, over the client’s privatedata for several iterations [3]. Once training is completed, the clients sendtheir updated model parameters to the server. The server aggregates theseupdates, often using weighted averaging techniques like Federated Averag-ing (FedAvg) [1], to produce an updated global model. The updated modelis then redistributed to the clients for further rounds of training. This it-erative process continues until the global model converges to an acceptablelevel of performance [22].Several adaptations have been introduced to handle variations in client par-ticipation, data availability, and computational capabilities. For example,asynchronous FL techniques allow clients to update the global model atdifferent times, accommodating scenarios where some clients have limitedavailability [23]. Client selection strategies are also widely used to prioritizeparticipants with the most relevant data or computational resources for eachtraining round [24].42.1. Federated LearningAggregation is a critical component of FL, and various techniques have beendeveloped to address its limitations, particularly in handling heterogeneousclient data distributions. While FedAvg remains the standard aggregationalgorithm [1], modifications such as Federated Proximal (FedProx) introduceproximal terms to better align local updates with global objectives, improv-ing stability in non-IID scenarios [3]. Scaffold further addresses the varianceintroduced by data heterogeneity by using control variates to correct biasedupdates [23].To ensure data security, FL incorporates techniques such as differential pri-vacy and secure aggregation [1, 7]. Differential privacy introduces controllednoise into the shared model updates, ensuring that individual client contri-butions cannot be reconstructed [25]. Secure aggregation protocols enhanceconfidentiality by allowing the central server to aggregate updates withoutaccessing individual client data directly [12]. Together, these techniquesform a robust framework for enabling privacy-preserving, collaborative ma-chine learning.2.1.2 Advantages of Federated Learning in HealthcareFederated Learning has proven to be particularly advantageous in health-care, a domain where privacy concerns and data sensitivity are paramount[2]. In traditional centralized learning systems, aggregating sensitive medicaldata in a central repository poses significant risks and challenges, includingcompliance with regulations such as the General Data Protection Regula-tion (GDPR) and the Health Insurance Portability and Accountability Act(HIPAA) [9]. FL provides a solution by enabling institutions to collabora-tively train machine learning models without sharing raw data.A major application of FL in healthcare is medical imaging, where modelstrained on diagnostic images such as X-rays, CT scans, and MRIs haveshown significant promise [20]. Medical imaging datasets are often dis-tributed across multiple institutions, each of which may use different imagingequipment, protocols, and patient populations. FL enables these institutionsto jointly train models that benefit from the diversity of their datasets, re-sulting in robust and generalizable models [21]. For example, FL has beenemployed to develop models for detecting diseases such as pneumonia andcancer, segmenting tumors, and identifying diabetic retinopathy in retinalimages [26, 27].The ability of FL to aggregate knowledge across institutions without expos-52.1. Federated Learninging sensitive data is critical in advancing research on rare diseases, wheredatasets are inherently limited and distributed [22]. By pooling insightsfrom multiple institutions, FL facilitates the development of predictive mod-els that would otherwise be infeasible with data from a single source. Thiscapability is particularly important in rare disease diagnostics, where a lackof sufficient data often hinders the development of accurate machine learningsystems [5].Another significant advantage of FL in healthcare is its role in personal-ized medicine. By training models on patient-specific data while leveragingglobal knowledge from other participants, FL enables the creation of predic-tive models that are both accurate and tailored to individual patients [16].For example, predictive models for disease progression can combine an in-dividual’s medical history, genetic profile, and lifestyle factors with broaderpopulation data to provide personalized recommendations for treatment andcare [4].In addition to improving model accuracy, FL also addresses practical chal-lenges in healthcare data management. Its decentralized architecture allowsinstitutions to retain control over their data, fostering trust and collabo-ration among stakeholders [9]. Furthermore, by leveraging local computa-tional resources, FL reduces the need for centralized infrastructure, makingit a cost-effective and scalable solution for healthcare applications [2].2.1.3 Challenges in Federated LearningFederated Learning, while transformative, is not without significant chal-lenges. The decentralized and collaborative nature of FL introduces com-plexities that are not present in traditional centralized machine learningsystems. These challenges are particularly pronounced in real-world appli-cations where heterogeneity among clients, resource constraints, and privacyconsiderations play critical roles. Among these, the challenge of data het-erogeneity emerges as the most critical, as it directly impacts the ability ofthe global model to generalize effectively across clients [1].One of the foundational issues in FL is the inherent non-IID (non-independentand identically distributed) nature of client data. Unlike centralized train-ing, where all data is assumed to come from a unified distribution, FL op-erates under conditions where each client may have a dataset that reflectsits unique operational context, user demographics, or data collection proce-dures [3]. For instance, in a healthcare setting, different hospitals may have62.1. Federated Learningvarying patient populations, medical equipment, and diagnostic protocols,leading to significant variability in their datasets. This heterogeneity canresult in local models that are heavily biased toward the specific data oftheir respective clients, causing challenges during global aggregation [12].The aggregation of locally trained models is a cornerstone of FL, yet itis particularly vulnerable to the effects of data heterogeneity. Standardalgorithms like Federated Averaging (FedAvg) rely on the assumption thatlocal updates are representative of the overall data distribution. However, inthe presence of non-IID data, these updates can diverge significantly, leadingto conflicting gradients during the aggregation process [23]. This divergencenot only slows down convergence but also compromises the performanceof the global model, as it may overfit to the dominant patterns in certainclients’ data while underperforming on others [3].Another manifestation of data heterogeneity is the imbalance in class distri-butions across clients. In many real-world scenarios, certain classes may beoverrepresented in one client’s dataset while being completely absent in an-other. For example, a hospital specializing in cardiology may have extensivedata on heart conditions but little to no data on neurological disorders [22].Such imbalances can cause the global model to skew its predictions towardthe more prevalent classes, reducing its ability to perform well across thefull range of expected inputs.Resource constraints also compound the challenges in FL. Clients often op-erate under varying levels of computational power, storage capacity, andnetwork bandwidth. This variability creates disparities in clients’ ability toparticipate effectively in the training process [23]. For example, resource-constrained clients may drop out of training due to hardware limitationsor network instability, leading to incomplete model updates. Furthermore,clients with smaller or less diverse datasets may contribute updates that areless informative, further exacerbating the impact of heterogeneity on theglobal model.Communication efficiency is another critical issue in FL, particularly inlarge-scale deployments involving hundreds or thousands of clients. The it-erative nature of FL requires frequent exchanges of model updates betweenclients and the central server, which can introduce significant communica-tion overhead [12]. This overhead is especially problematic in environmentswith limited network bandwidth, as it can lead to delays in training or evenexclusion of certain clients [23].72.1. Federated LearningSecurity and privacy concerns, while core to the design of FL, remain per-sistent challenges. Although FL minimizes the need to share raw data, theexchange of model updates is not entirely immune to attacks. Maliciousclients can introduce poisoned updates to manipulate the global model, aphenomenon known as adversarial poisoning [25]. Similarly, gradient inver-sion attacks can reconstruct sensitive information from the shared updates,compromising the privacy of participating clients [4]. While techniques suchas differential privacy and secure aggregation offer promising solutions, theyoften come at the cost of increased computational and communication over-head, limiting their scalability [12].Fairness in FL is another emerging concern. In real-world scenarios, certainclients may dominate the training process due to their larger datasets orhigher computational resources. This dominance can marginalize smalleror resource-constrained clients, leading to a global model that dispropor-tionately benefits the more influential participants [24]. Ensuring equitablecontributions from all clients is essential for maintaining trust and collabo-ration in FL systems, particularly in sensitive domains like healthcare whereoutcomes must be fair and unbiased.Despite these multifaceted challenges, data heterogeneity remains the mostcritical issue, as it underpins many of the difficulties encountered in FL. Thevariability in data distributions across clients directly impacts the qualityof local updates, the efficiency of the aggregation process, and the gener-alizability of the global model. Addressing data heterogeneity is thereforeessential for unlocking the full potential of FL, particularly in applicationslike healthcare, where robustness and fairness are paramount [1].2.1.4 Related WorkFederated Learning (FL) has spurred a vast body of research to addressthe challenges outlined above, including data heterogeneity, resource con-straints, communication efficiency, security, and fairness. This section pro-vides an in-depth review of baseline and advanced methods proposed inthe literature to mitigate these challenges, diving deeper into techniquesdiscussed in earlier sections and introducing additional contributions fromrecent works [3, 23].82.1. Federated LearningMitigating Data Heterogeneity in Federated LearningData heterogeneity is a defining challenge in Federated Learning (FL), char-acterized by the decentralized and non-IID (non-independent and identicallydistributed) nature of client datasets. This variability introduces biases inlocal models, creates conflicts during aggregation, and undermines the gen-eralization capacity of the global model [1, 3]. Researchers have proposednumerous strategies to address data heterogeneity, which can be categorizedinto server-side, client-side, and hybrid methods. Among these, syntheticdata generation methods, particularly client-side solutions, have emerged asa highly effective and promising direction, leveraging local data generationto combat heterogeneity without compromising privacy [7].Server-Side Methods Server-side methods primarily address heterogene-ity by refining the aggregation process, clustering clients, or generating syn-thetic data centrally. These approaches aim to ensure that the global modelremains robust to diverse updates from clients.Federated Averaging (FedAvg), while foundational, performs poorly in non-IID settings due to its simplistic averaging mechanism [1]. Federated Prox-imal (FedProx) addresses this limitation by introducing a proximal term inthe global objective that penalizes local updates deviating from the globalmodel, reducing divergence and improving stability in non-IID environments[3]. SCAFFOLD goes further by leveraging control variates at the server tocorrect client drift, improving alignment between local and global optimiza-tion objectives [23]. These server-side adjustments mitigate the varianceintroduced by client-specific data, ensuring more stable aggregation.Clustered Federated Learning (CFL) groups clients based on the similarity oftheir data distributions and trains separate models for each cluster [5]. Thismethod reduces conflicts during aggregation and allows tailored trainingfor subsets of clients. Similarly, FedSEM dynamically partitions clients intoclusters and refines the cluster-specific models, providing a robust frameworkfor managing diverse datasets [22].Data-sharing strategies also play a role in addressing heterogeneity. FedDF(Federated Distillation Framework) aggregates client updates into a teachermodel and refines it using a small public dataset shared among clients. Thisaligns client updates with global objectives, although its effectiveness relieson the availability of a representative public dataset. FedNova introduces anormalization strategy to adjust client updates based on their computational92.1. Federated Learningefforts, ensuring equitable contributions to the global model and addressingimbalances caused by resource variability [12].Synthetic data generation techniques on the server side create representa-tive datasets to guide aggregation. FedSynth uses generative models, such asGANs and VAEs, to generate synthetic data approximating global patterns[14]. These datasets are used to align updates across clients, ensuring ro-bust global training without requiring raw data exchange. While effective,such server-side approaches are constrained by the central server’s abilityto produce sufficiently representative data, particularly in privacy-sensitivedomains [25].Client-Side Methods Client-side methods tackle heterogeneity at itssource by modifying local training processes, enabling personalization, oremploying synthetic data generation. These approaches aim to align clientupdates with global objectives while accommodating the unique character-istics of client data.Personalized Federated Learning (PFL) frameworks offer a significant client-side solution. Per-FedAvg trains a shared global model that serves as aninitialization for client-specific fine-tuning, balancing the benefits of collab-orative learning and personalization [22]. FedMeta integrates meta-learningprinciples, optimizing the global model for rapid adaptation to diverse clientdata distributions, ensuring effective personalization without compromisingthe global model’s robustness [28].Synthetic data generation on the client side has emerged as one of the mosttransformative approaches to mitigating heterogeneity. By generating syn-thetic datasets reflecting local distributions, clients can balance class rep-resentation, amplify underrepresented features, and align local models withglobal objectives. Techniques such as FedSynth utilize generative modelsto produce synthetic data locally, enabling privacy-preserving data augmen-tation [14]. Synthetic Minority Oversampling Technique (SMOTE) and itsderivatives have been adapted to FL settings, allowing clients to create syn-thetic examples for underrepresented classes, reducing data imbalances andenhancing local training robustness [29].Our method extends this line of work, focusing exclusively on syntheticdata generation at the client level. By preserving privacy and addressingheterogeneity effectively, client-side synthetic data generation ensures thatlocal training captures global patterns without requiring data sharing. This102.1. Federated Learningapproach is particularly effective in domains such as medical imaging, wheredata heterogeneity and privacy concerns are paramount [2].Hybrid Methods Hybrid methods integrate server-side and client-side ef-forts to address heterogeneity. These approaches leverage coordinated mech-anisms to ensure consistency between local updates and global objectives.Clustered methods like FedSEM rely on client contributions to identify clus-ters and train cluster-specific models [5]. FedDyn adapts the global modeldynamically to reflect evolving client data distributions, balancing local andglobal objectives [3]. Synthetic data generation also operates in hybrid se-tups, where the server produces representative synthetic data for client use.This enables alignment between client training and server aggregation whilepreserving data privacy [25].Synthetic Data Generation as a Prominent Solution Synthetic datageneration stands out as a highly effective solution for mitigating data het-erogeneity, particularly in privacy-sensitive settings. Unlike methods re-lying on shared datasets or extensive server-client coordination, syntheticdata generation operates independently within clients, producing privacy-preserving datasets that enhance local training. By simulating global pat-terns locally, synthetic data aligns client updates with global objectives,mitigating biases introduced by non-IID distributions [17].The central advantage of synthetic data generation lies in its adaptabilityand privacy-preserving nature. Clients can generate synthetic datasets tai-lored to their local distributions, addressing class imbalances and enhancingfeature representation without exposing raw data. This capability makessynthetic data generation a scalable and flexible solution for addressingheterogeneity, particularly in applications such as healthcare, where datasensitivity is critical [2].Despite its promise, synthetic data generation faces challenges, includingensuring the quality of generated data and managing the computationaloverhead of generative models. However, advancements in generative ar-chitectures, such as GANs and VAEs, continue to improve the fidelity andutility of synthetic datasets, addressing these limitations [30]. As researchprogresses, synthetic data generation is likely to play an increasingly centralrole in FL, providing a robust framework for tackling heterogeneity whileadhering to privacy constraints.112.1. Federated LearningDiscussion The division of methods into server-side, client-side, and hy-brid categories highlights the breadth of strategies employed to mitigatedata heterogeneity in FL. Among these, client-side synthetic data generationemerges as a particularly impactful approach due to its privacy-preservingnature and ability to address heterogeneity at its source. By enabling clientsto generate datasets that balance local and global objectives, synthetic datageneration methods bridge the gap between privacy and performance, offer-ing a scalable solution for diverse applications [14].As FL continues to evolve, the prominence of synthetic data generation isexpected to grow, particularly in domains where data heterogeneity andprivacy concerns intersect. These methods provide a robust foundation foradvancing FL systems, ensuring fairness, scalability, and robustness in het-erogeneous environments [2].Addressing Resource ConstraintsThe variability in client resources introduces significant challenges in main-taining uniform participation in FL. Adaptive optimization methods havebeen proposed to balance resource disparities while ensuring robust contri-butions from all clients [31].Adaptive Federated Optimization (AFO) dynamically adjusts the number oflocal epochs and client participation based on resource availability [12]. Thisflexibility allows resource-constrained clients to contribute without droppingout, ensuring a more inclusive training process. FedNova, an extensionof FedAvg, normalizes client updates by local learning rates and epochs,accounting for the varying computational capabilities of participants [3].These approaches improve convergence and reduce biases caused by unequalclient contributions.To further address resource limitations, methods such as gradient sparsifi-cation and quantization have been employed to reduce the computationaland communication burden. SignSGD transmits only the sign of the gradi-ents, significantly lowering bandwidth requirements without substantial lossin accuracy [7]. Similarly, top-k sparsification selects the most importantgradient updates for transmission, optimizing resource usage [23].Asynchronous FL techniques, such as FedAsync, provide additional solutionsby enabling clients to update the global model independently of one another.This removes the dependency on synchronized communication rounds, al-lowing slower clients to contribute without delaying the overall process [1].122.1. Federated LearningHowever, asynchronous updates introduce the risk of stale gradients, whichmust be mitigated through decay mechanisms or prioritization strategies.Enhancing Communication EfficiencyEfficient communication is critical in FL, particularly in large-scale deploy-ments where bandwidth is limited. Several techniques have been proposedto reduce communication overhead while maintaining model accuracy.Periodic aggregation, as implemented in FedPAQ (Periodic Averaging andQuantization), reduces the frequency of client-to-server communication byaggregating updates only after several local training iterations [12]. Bycombining this with gradient quantization, FedPAQ achieves significant re-ductions in communication overhead.Hierarchical FL frameworks offer another avenue for optimizing communi-cation. Hierarchical Federated Averaging (Hier-FedAvg) introduces inter-mediary aggregation nodes that collect updates from local clients beforeforwarding them to the central server [5]. This layered structure reduces thecommunication burden on the central server, improving scalability.Decentralized FL approaches, such as peer-to-peer training, further reducereliance on a central server by allowing clients to communicate directly withone another [32]. While this method minimizes server-side bottlenecks, itintroduces challenges in achieving global model consistency.Ensuring Security and PrivacyThe decentralized nature of FL introduces vulnerabilities that necessitaterobust security and privacy-preserving mechanisms. Differential privacy andsecure aggregation are two widely adopted techniques in this domain.Differential privacy enhances confidentiality by introducing noise into modelupdates, ensuring that individual contributions remain obscured [1]. DP-FedAvg applies this principle to FL, balancing privacy guarantees withmodel performance [8]. Similarly, PATE-FL leverages teacher-student mod-els to enable privacy-preserving training, providing additional safeguardsagainst data leakage [33].Secure aggregation protocols, such as SecAgg, allow the server to aggregateclient updates without accessing individual contributions. These protocolsrely on cryptographic techniques to maintain confidentiality, ensuring thatonly the aggregated model is visible. Homomorphic encryption extends this132.1. Federated Learningapproach by enabling computations to be performed directly on encrypteddata, preserving privacy throughout the training process.Byzantine-resilient aggregation methods address adversarial threats by fil-tering malicious updates. Techniques like Krum and Bulyan evaluate thesimilarity of updates across clients and exclude outliers indicative of adver-sarial behavior [12]. While effective, these methods often involve additionalcomputational complexity, highlighting the trade-offs between security andscalability.Achieving Fairness in Federated LearningFairness in FL is critical for ensuring that models perform equitably acrossdiverse clients. Disparities in client data distributions and resource capabil-ities often lead to models that disproportionately benefit dominant clients,marginalizing those with less representation [24].Agnostic Federated Learning (AFL) optimizes for the worst-case client per-formance, ensuring that the global model is fair across all participants [34].This approach is particularly relevant in applications like healthcare, whereequitable outcomes are paramount. Multi-gradient descent algorithms, asimplemented in FedMGDA+, further address fairness by balancing clientcontributions during aggregation [35].Reweighting strategies have also been employed to improve fairness. Byadjusting the weight of client updates based on data representativeness orresource constraints, these methods ensure that all participants have anequitable influence on the global model [17].Open Research DirectionsDespite significant advancements, FL faces unresolved challenges that war-rant further investigation. Addressing extreme data heterogeneity, particu-larly in highly non-IID scenarios, remains an open problem [36]. Communi-cation efficiency and resource optimization continue to be critical for scalingFL to large networks of clients [7]. Finally, achieving robust security andfairness without compromising performance requires innovative approachesthat balance these competing objectives [37].The methods reviewed in this section form the foundation for addressingFL’s core challenges. Building on these approaches, this thesis proposesnovel strategies to enhance the robustness and scalability of FL systems,142.2. Catastrophic Forgetting in Heterogeneous Federated Learningparticularly in applications involving medical imaging datasets [2].2.2 Catastrophic Forgetting in HeterogeneousFederated Learning2.2.1 Overview of Catastrophic ForgettingCatastrophic forgetting, also known as catastrophic interference, refers tothe phenomenon where a machine learning model loses previously acquiredknowledge upon learning new tasks or data distributions. This issue isparticularly prominent in neural networks, as their parameters are optimizediteratively and often overwrite features critical for earlier tasks when exposedto new data [10, 34]. The challenge of catastrophic forgetting is not unique tofederated learning; it has been extensively studied in the broader context ofcontinual learning, where models are trained sequentially on multiple taskswithout retaining all previously seen data [36, 38].Catastrophic Forgetting in Continual LearningIn continual learning, the primary goal is to enable models to adapt to newtasks while retaining knowledge from previously learned ones. However, tra-ditional neural networks are inherently prone to catastrophic forgetting dueto their shared parameter structure. When exposed to new tasks, gradient-based updates adjust the network’s parameters to optimize performance onthe current task, often at the expense of degrading performance on ear-lier tasks [39]. This happens because the same set of parameters is reusedto encode new knowledge, and there is no explicit mechanism to preserveimportant features for past tasks.Catastrophic forgetting is particularly severe in sequential task learning,where tasks are presented one after the other, and the model cannot revisitprevious data. For instance, in a continual learning scenario involving imageclassification, a model trained first on animals and then on vehicles mightlose its ability to recognize animals altogether if not explicitly designed topreserve earlier knowledge [11]. Techniques such as Elastic Weight Consoli-dation (EWC), memory-based replay, and dynamic architectures have beenproposed to address this issue. EWC, for example, penalizes changes to pa-rameters critical for prior tasks by incorporating a regularization term basedon the Fisher information matrix [10]. Memory-based approaches retain asubset of prior data or generate pseudo-data to rehearse earlier tasks, mit-152.2. Catastrophic Forgetting in Heterogeneous Federated Learningigating forgetting [40]. Dynamic architecture methods, such as ProgressiveNeural Networks, allocate new resources for new tasks, preventing interfer-ence with previously learned knowledge [41].The severity of catastrophic forgetting depends on factors such as task simi-larity, the capacity of the neural network, and the training dynamics. Whilethe problem is well-documented in centralized and sequential settings, its im-plications become even more pronounced in decentralized environments likefederated learning, especially with heterogeneous data distributions [38, 42].Catastrophic Forgetting in Federated Learning withHeterogeneous DataIn federated learning (FL), catastrophic forgetting is exacerbated by the de-centralized and iterative nature of training, particularly in settings wheredata across clients is non-independent and non-identically distributed (non-IID) [1, 7]. Each client in an FL system has its own unique dataset, oftenreflecting specific operational contexts, demographic distributions, or insti-tutional practices. This heterogeneity results in local models that are highlyspecialized to their respective data distributions, creating challenges whenthese updates are aggregated into a global model.The iterative nature of FL amplifies catastrophic forgetting because theglobal model must continuously adapt to updates from clients with varyingand potentially conflicting data distributions [8]. For instance, a hospitalspecializing in cardiology may provide updates focused on cardiac condi-tions, while another hospital with data on respiratory diseases provides up-dates that prioritize those conditions. When the global model aggregatesthese updates, it risks overwriting knowledge learned from one client to ac-commodate another. This dynamic leads to a degradation in the model’sability to generalize across clients and maintain robust performance on ear-lier distributions [3, 43].This phenomenon is illustrated in Fig. 2.1, where we investigate the effect ofcatastrophic forgetting during local training using BloodMNIST under theexperimental settings described in Sec. 3.2. Specifically, we plot each client’slocal model accuracy over other clients’ data during local training. The ac-curacy drops drastically with FedAvg, highlighting the loss of knowledge dueto conflicting updates from heterogeneous datasets. In contrast, FedImpresdemonstrates stable accuracy across clients, emphasizing its effectiveness inmitigating catastrophic forgetting.162.2. Catastrophic Forgetting in Heterogeneous Federated LearningLocal Epoch (E)0 4002000.00.20.40.60.8Accuracy FedAvg FedImpresClient 1Client 2Client 3Figure 2.1: Catastrophic forgetting occurs when server weights are over-written during local training, causing a loss of previous knowledge. Toinvestigate the effect of catastrophic forgetting during local training in FL,we conducted experiments on BloodMNIST using the same experimentalsettings described in Sec. 3.2. Specifically, we plot each client’s local modelaccuracy over other clients’ data during local training. The accuracy dropsdrastically using FedAvg; however, FedImpres maintains stable accuracyacross clients.Moreover, catastrophic forgetting in FL is influenced by the frequency ofmodel updates and the degree of data heterogeneity. Clients with moredistinct or imbalanced datasets contribute updates that diverge significantlyfrom the global model’s objective, resulting in conflicting gradients [23].Over time, this divergence can destabilize the global model, leading to poorretention of knowledge and suboptimal generalization [5].The challenge is further complicated by the privacy-preserving nature ofFL, which prohibits the sharing of raw data across clients. This constrainteliminates traditional remedies for catastrophic forgetting, such as rehearsalor memory replay, which rely on access to previously seen data [17]. Con-sequently, addressing catastrophic forgetting in FL requires innovative ap-proaches that operate within the decentralized and privacy-preserving con-straints of the framework [2, 9].2.2.2 Related WorkCatastrophic forgetting has been a long-standing challenge in machine learn-ing, particularly in settings involving sequential learning and heterogeneousdata distributions. In the context of federated learning (FL), this issue isamplified due to the decentralized and iterative nature of the training pro-172.2. Catastrophic Forgetting in Heterogeneous Federated Learningcess, which introduces unique challenges related to client heterogeneity andnon-IID data distributions [10, 34]. Various methods have been proposedto address catastrophic forgetting in FL, each with distinct mechanismsand trade-offs. This section provides a comprehensive review of these ap-proaches, incorporating insights from key studies.Regularization-Based ApproachesRegularization-based methods have played a pivotal role in addressing catas-trophic forgetting in federated learning by penalizing parameter updatesthat deviate significantly from prior knowledge. One of the foundationaltechniques in this category is Federated Curvature (FedCurv), inspired byElastic Weight Consolidation (EWC) from continual learning [10]. FedCurvincorporates a penalty term in the loss function to preserve the critical pa-rameters of the global model across clients. By computing the diagonal of theFisher information matrix, it identifies and protects important parameters,thereby mitigating the risk of catastrophic forgetting. FedCurv has demon-strated effectiveness in non-IID settings, particularly on tasks like imageclassification using MNIST datasets, where it outperforms baseline methodslike FedAvg and FedProx in both accuracy and convergence stability [1, 3].Building on this concept, Federated Regularization (FedReg) introducespseudo-data generated by the global model to provide a regularization mech-anism for local updates [44]. These pseudo-data points act as proxies forprior knowledge, ensuring that local models do not deviate significantly fromthe global model’s learned representations. FedReg addresses the dual chal-lenges of catastrophic forgetting and model drift, especially in scenarios withhigh client heterogeneity. Experimental results on CIFAR-10 and CIFAR-100 show that FedReg achieves superior performance compared to FedAvgand other baseline methods, highlighting its robustness in diverse federatedenvironments [1, 5].Gradient-Based MethodsGradient masking and regularization approaches have also been explored toaddress catastrophic forgetting in FL. Federated Neural Tangent Distance(FedNTD) is a notable example that minimizes the distance between localmodel updates and a reference model in the parameter space. This alignmentreduces the divergence introduced by non-IID data, ensuring that globalmodel parameters remain consistent with prior knowledge [45].182.2. Catastrophic Forgetting in Heterogeneous Federated LearningGradMA (Gradient Mask Aggregation) offers a complementary approachby selectively updating global model parameters based on gradient align-ment across clients [46]. Parameters with conflicting gradients are maskedto prevent updates that could compromise previously learned knowledge.GradMA has shown promising results in retaining global knowledge whileenabling local adaptability, making it particularly suitable for federated sys-tems with high heterogeneity.Knowledge Distillation and Pseudo-LabelingKnowledge distillation techniques have gained traction in federated learningas a means to mitigate catastrophic forgetting. In these approaches, theglobal model serves as a teacher to guide the training of local models. FedDF(Federated Distillation Framework) aggregates client models into a globalteacher model, which is then used to refine local models during subsequenttraining rounds. This hierarchical learning process ensures that local modelsretain global knowledge while adapting to their specific datasets.CFeD (Catastrophic Forgetting Elimination through Distillation) extendsthis idea by utilizing auxiliary unlabeled data for knowledge distillation [25].By leveraging data-agnostic distillation, CFeD addresses the challenge ofheterogeneous data distributions without relying on raw client data. Thesemethods highlight the potential of knowledge distillation as a scalable solu-tion to catastrophic forgetting in federated environments.Experience Replay and Memory-Based MethodsExperience replay, a widely used technique in continual learning, has beenadapted to federated settings despite the privacy constraints that precludedirect data sharing. Federated Weighted Interpolation Training (FedWeIT)employs sparse task-specific parameters to emulate the effects of experiencereplay [14]. By selectively reusing parameters associated with prior tasks,FedWeIT ensures that the global model retains diverse knowledge whileadapting to new client updates. This method has been particularly effectivein federated systems with dynamic client participation.Other methods in this category utilize pseudo-replay mechanisms to over-come the limitations of data sharing. For example, pseudo-replay involvesgenerating synthetic data that approximates previously seen distributions[47]. While these methods are less common in federated learning comparedto centralized settings, they hold promise for addressing catastrophic forget-192.2. Catastrophic Forgetting in Heterogeneous Federated Learningting in privacy-preserving frameworks.Hybrid Approaches Combining Continual and FederatedLearningThe intersection of federated learning and continual learning has given rise tohybrid approaches that leverage techniques from both domains. FedSpeECH(Federated Sparse Continual Hypernetworks) employs pruning masks to dy-namically allocate model parameters for new tasks while preserving criticalones for prior knowledge. This approach ensures that the global model canadapt to new clients without overwriting previously learned information [48].FedNTD, which aligns local models in a shared parameter space, also demon-strates the potential of hybrid methods. By combining parameter alignmentwith continual learning-inspired regularization, FedNTD achieves a balancebetween local adaptability and global knowledge retention [45]. These meth-ods underscore the synergies between federated and continual learning inaddressing catastrophic forgetting.Challenges and Open QuestionsDespite the advancements in mitigating catastrophic forgetting in federatedlearning, several challenges remain. Regularization methods, while effec-tive, often impose additional computational overhead and require carefulparameter tuning to balance local and global objectives. Gradient-basedapproaches, such as GradMA and FedNTD, rely on gradient alignment,which may be computationally intensive for large-scale deployments [23].Knowledge distillation and pseudo-labeling are contingent on the quality ofgenerated data, which can vary significantly across clients [9].Furthermore, experience replay and memory-based methods face limitationsin privacy-preserving frameworks, as they require synthetic data genera-tion or parameter sharing [8]. Hybrid approaches combining federated andcontinual learning hold promise but are still in their infancy, with limitedreal-world deployments and scalability challenges [42].Overall, the diversity of methods reviewed here highlights the complexityof addressing catastrophic forgetting in federated learning. Each approachoffers unique strengths and trade-offs, emphasizing the need for continuedresearch to develop robust, scalable, and privacy-preserving solutions. Thisthesis builds on these foundations, proposing novel methods that leverage202.3. Recent Advancements in Federated Learningsynthetic data generation and pretrained models to enhance global modelrobustness and mitigate forgetting in heterogeneous federated systems.2.3 Recent Advancements in Federated LearningRecent advancements in Federated Learning (FL) have been driven by the in-tegration of powerful machine learning paradigms, such as foundation mod-els and prompt tuning techniques. These innovations aim to address keychallenges in FL, such as data heterogeneity, scalability, and resource effi-ciency, while enhancing model performance and adaptability across diverseclient environments. This section explores two major areas of progress: theemergence of foundation models in FL and the adoption of visual prompttuning as a novel adaptation mechanism.2.3.1 Emergence of Foundation ModelsFoundation models represent a transformative shift in machine learning,characterized by their large-scale, pre-trained architectures and ability togeneralize across diverse downstream tasks [9, 49]. These models, such asBERT, GPT, and Vision Transformers (ViTs), are pre-trained on extensivedatasets and then fine-tuned for specific applications, leveraging their rich,pre-existing knowledge [19]. Their versatility and adaptability have madethem an appealing option for FL, where data is decentralized and heteroge-neous.Foundation models, due to their pre-trained nature, bring a significant ad-vantage to FL by reducing the reliance on extensive local training, a commonlimitation in resource-constrained clients [1, 12]. In FL, these models areoften deployed as a starting point, and client-specific updates are appliedto adapt them to local data distributions. This strategy mitigates the needfor exhaustive local training, addressing computational constraints whileensuring robust generalization.One notable advantage of foundation models in FL is their capacity to han-dle diverse data distributions. Pre-trained on large and diverse datasets,these models capture generalized patterns that are transferable to a widerange of applications. This generalization capability makes them particu-larly effective in non-IID (non-independent and identically distributed) FLscenarios, where client data distributions differ significantly [3].The utilization of foundation models in FL is also facilitated by transfer212.3. Recent Advancements in Federated Learninglearning techniques. By fine-tuning only a subset of model parameters orusing prompt tuning mechanisms, FL systems can leverage the pre-trainedknowledge of foundation models without requiring extensive computationalresources [9, 50]. For instance, lightweight fine-tuning techniques, such asLoRA (Low-Rank Adaptation), have been applied to efficiently adapt foun-dation models in decentralized settings, ensuring scalability while maintain-ing performance.Recent studies highlight the potential of foundation models to bridge thegap between global model generalization and local personalization in FL[33]. Their hierarchical architectures, such as those in Vision Transformers,allow for modular updates, where specific layers are adapted to client datawhile preserving the integrity of pre-trained representations. This modular-ity is particularly beneficial in healthcare applications, where models needto accommodate diverse medical imaging datasets without compromisingprivacy or performance [2].Despite their promise, integrating foundation models into FL presents chal-lenges, including communication efficiency and model size. The transmissionof large model updates between clients and the central server can introducesignificant communication overhead. To address this, techniques such asmodel compression, parameter-efficient fine-tuning, and federated pruninghave been explored, ensuring that foundation models remain practical in FLdeployments [9].In summary, the emergence of foundation models in FL represents a signifi-cant step forward, combining the adaptability and scalability of pre-trainedarchitectures with the privacy-preserving principles of FL. These modelsenable robust performance across heterogeneous data distributions, reduc-ing the computational burden on clients and enhancing the generalizationcapacity of FL systems.2.3.2 Visual Prompt Tuning in Federated LearningVisual Prompt Tuning (VPT) has recently gained traction as a novel adap-tation technique in FL, leveraging the pre-trained capabilities of vision mod-els such as Vision Transformers (ViTs) [18, 19]. By introducing lightweight,learnable prompts that adapt pre-trained models to specific tasks, VPT of-fers an efficient alternative to full fine-tuning. This subsection reviews theliterature on VPT in FL, with a detailed focus on the latest advancements.VPT involves appending learnable prompt tokens to the input sequence of222.3. Recent Advancements in Federated Learninga pre-trained model. These prompt tokens act as task-specific adapters,guiding the model to learn new tasks without modifying its pre-trained pa-rameters [17, 33]. This approach significantly reduces the computational andmemory requirements of fine-tuning, making it ideal for resource-constrainedFL environments.In FL, VPT enables clients to personalize a shared global model while pre-serving its core knowledge [17]. Instead of updating the entire model, clientsonly update the prompt tokens, which are lightweight and easily transfer-able. This reduces communication costs and computational overhead, ad-dressing two critical challenges in FL.One of the key contributions to VPT in FL is the SGPT (Scalable Gen-erative Prompt Tuning) framework, which introduces task-specific promptsin a federated setting. SGPT employs a dual-prompt mechanism, whereshared prompts capture global patterns, and client-specific prompts adaptthe model to local data distributions [18]. By separating global and localadaptation, SGPT effectively balances generalization and personalization,achieving robust performance across heterogeneous clients. The frameworkalso incorporates techniques to mitigate prompt overfitting, ensuring thatthe shared prompts remain representative of the global data distribution.The potential of VPT extends to complex FL scenarios, such as medicalimaging. For example, studies have demonstrated the effectiveness of VPTin adapting pre-trained ViTs to diverse imaging modalities, including X-rays, MRIs, and CT scans [2]. By introducing domain-specific prompts,these models achieve high accuracy while requiring minimal client-side com-putation. Furthermore, the modularity of VPT allows for incremental up-dates, enabling models to adapt to new imaging protocols or datasets with-out retraining the entire architecture [18].The SGPT framework exemplifies the benefits of VPT in FL. It employsa hierarchical prompt design, where initial layers capture general featuresand deeper layers adapt to task-specific nuances [33]. This design alignswell with the hierarchical structure of ViTs, leveraging their self-attentionmechanisms for efficient adaptation. Experimental results on benchmarkdatasets highlight the superiority of SGPT over traditional fine-tuning meth-ods, demonstrating improved accuracy, communication efficiency, and ro-bustness in non-IID settings [17, 18].In conclusion, VPT represents a paradigm shift in FL, enabling efficientadaptation of foundation models to diverse client data distributions. By232.4. Summaryleveraging lightweight prompts, VPT addresses critical challenges in FL, in-cluding computational constraints, communication efficiency, and data het-erogeneity. As research progresses, VPT is expected to play a pivotal rolein advancing FL systems, particularly in applications requiring scalable andpersonalized learning.2.4 SummaryThis chapter provided a comprehensive exploration of Federated Learning(FL), its applications, challenges, and recent advancements, forming thefoundational context for the studies presented in this thesis. FederatedLearning was introduced as a decentralized machine learning paradigm thatenables collaborative model training across distributed clients while pre-serving data privacy. The principles of FL, including its iterative trainingprocess, aggregation mechanisms, and privacy-preserving features, were dis-cussed, with a particular focus on its transformative role in domains suchas healthcare.FL’s advantages in healthcare, especially in medical imaging, were high-lighted. By facilitating cross-institutional collaborations without requiringraw data sharing, FL allows the development of robust models that gen-eralize across diverse datasets. This capability is particularly critical inaddressing challenges such as data scarcity in rare disease research and thedevelopment of personalized medicine applications.Despite these benefits, FL faces significant challenges, including data hetero-geneity, resource variability, communication efficiency, and fairness. Dataheterogeneity, which arises from non-independent and non-identically dis-tributed (non-IID) client data, was identified as the most critical issue. Thisvariability impacts the global model’s ability to generalize effectively acrossclients, necessitating innovative solutions. Various approaches to mitigatingdata heterogeneity were reviewed, categorized into server-side, client-side,and hybrid methods. Among these, synthetic data generation emerged asa particularly promising strategy. Client-side synthetic data generation, inparticular, provides a privacy-preserving mechanism to align local trainingobjectives with global model goals, addressing class imbalances and enhanc-ing feature representation.The chapter also examined the phenomenon of catastrophic forgetting, whichoccurs when models lose previously acquired knowledge while learning newtasks or data distributions. In FL, this issue is exacerbated by the decen-242.4. Summarytralized and iterative nature of training with heterogeneous client data. Acomprehensive review of methods to mitigate catastrophic forgetting waspresented, including regularization-based techniques, gradient stabilizationmethods, knowledge distillation, and hybrid approaches. These strategiesaddress the dual challenges of knowledge retention and adaptability in fed-erated systems.Recent advancements in FL were also explored, focusing on the integra-tion of foundation models and visual prompt tuning. Foundation modelsleverage their pre-trained architectures to enhance scalability and adaptabil-ity, addressing data heterogeneity and computational constraints. Visualprompt tuning was highlighted as an innovative approach for lightweightmodel adaptation in resource-constrained environments, offering a balancebetween global generalization and local personalization.The insights and discussions from this chapter set the stage for the twomain studies in this thesis. The first study presents a novel client-side syn-thetic data generation method, Federated Impressions, aimed at addressingdata heterogeneity in FL. The second study explores a prompt-based fed-erated learning framework to mitigate catastrophic forgetting by leveragingadvancements in visual prompt tuning and foundation models. Together,these contributions aim to advance the state of the art in Federated Learn-ing, particularly in applications with significant privacy and heterogeneityconstraints, such as healthcare.25Chapter 3Federated impression forlearning with distributedheterogeneous data3.1 Method3.1.1 Problem SettingFederated Learning (FL) is a distributed machine learning paradigm thatenables collaborative training across a group of clients {C1, C2, ..., CN}, eachpossessing its own private dataset Xn. Here, N denotes the total number ofclients. The core objective in this setup is to train a global model that gener-alizes effectively across all clients, ensuring high classification performanceon the union of their datasets. However, this must be achieved withoutsharing raw data due to stringent privacy constraints, making the problemboth challenging and impactful [1, 3].Each dataset Xn on client n consists of labeled samples (xni , yni ), where xni isthe input (e.g., an image) and yni is the corresponding class label. We denotethe global model’s feature extractor and classifier weights by θG and ϕG,respectively. The optimization problem in FL is formulated as minimizingthe global classification loss, which is expressed as:J(θG, ϕG) =N∑n=1E(xni ,yni )∈Xn ℓ(g(f(xni ; θG);ϕG), yni ), (3.1)where ℓ is the loss function, specifically the cross-entropy (CE) loss LCEin this case. The functions f(; θG) and g(;ϕG) correspond to the featureextractor and classifier components of the global model, respectively.263.1. MethodThe FL training process involves iterative local and global updates. At thestart of each communication round r, the central server initializes and broad-casts the current global model parameters θrG to all participating clients.Each client n uses these parameters as a starting point to train its localmodel on its private dataset Xn, performing several steps of gradient-basedoptimization. This local training generates updated model parameters θrn,which reflect the specific data distribution of client n [12].After completing local training, the clients send their updated parametersback to the server, where they are aggregated to form the updated globalmodel θr+1G . One of the most common aggregation strategies is FederatedAveraging (FedAvg) [1], which computes the weighted average of the clientupdates:θr+1G =1NN∑n=1θrn. (3.2)This iterative process continues until the global model converges to a sat-isfactory level of performance. However, several challenges arise in thissetup, especially in heterogeneous data settings where client datasets arenon-independent and non-identically distributed (non-IID) [8]. These chal-lenges include local model overfitting, conflicting gradients during aggrega-tion, and slower convergence rates. Such issues demand innovative solutionsto ensure effective model training in real-world FL deployments.In the context of this work, we extend the standard FL framework by in-troducing a synthetic data generation approach to mitigate the challengesposed by heterogeneous data [14, 51]. Our method synthesizes data at theserver, capturing a distilled representation of the global data distribution,and shares this data with clients to regularize their local training. Thisensures that local updates align more effectively with the global objective,improving both convergence and generalization. The detailed steps of thisapproach are outlined in the following sections.3.1.2 OverviewAs discussed in the background section (Sec. 2.2.1), catastrophic forgettingduring local training presents a critical challenge in heterogeneous FederatedLearning (FL). This issue arises primarily from the decentralized nature ofFL, where local models are optimized independently using client-specific273.1. MethodServergFedImpres Loss BackpropagateSynthesizedData. . .SynthesizedData Local Data 1CE Loss SynthesizedData Local Data NCE LossfClient’s NetworkServer’s Network(a) (b)Figure 3.1: Our proposed approach, FedImpres, aims to capture the globaldistribution learned by the aggregated server model and distill it into adataset that can be shared with clients. The approach consists of two steps:a) First, we perform pixel-wise optimization by starting from unlabeled pub-lic data and using the server model’s predicted pseudo-labels to backprop-agate using Eq.((3.5)). b) Second, to improve local training, we add thesynthesized data as a regularizer to the local data using Eq.((3.6)). Thisallows us to share the global distilled distribution with clients and leverageit to improve local training.data that often diverges significantly in distribution [3, 10]. Such diver-gence leads to feature drift, which complicates the aggregation process anddegrades the performance of the global model across diverse datasets [1].Addressing this challenge requires innovative solutions that can mitigate for-getting while preserving privacy and maintaining computational efficiency.In this work, we identify two fundamental challenges that must be addressedto develop a robust FL framework capable of handling heterogeneous data.The first challenge is alleviating catastrophic forgetting during local training.This can be achieved by introducing a unifying regularization mechanismthat counteracts the divergence caused by data heterogeneity. Specifically,we propose the use of synthetic datasets, referred to as federated impressions,to serve as a regularizer. These synthetic datasets act as anchors for local283.1. Methodtraining, ensuring that the server model’s global knowledge is retained andpropagated across clients [15, 51].The second challenge lies in the generation of these synthetic datasets. Un-like traditional data-sharing approaches that violate privacy constraints, wesynthesize federated impressions directly from the server model. This pro-cess captures a global representation of the underlying data distributionwithout requiring access to raw client data [14]. The federated impressionsare optimized to preserve the server model’s critical information, ensuringthat training on these datasets aligns with the global model’s objectives.This approach not only mitigates catastrophic forgetting but also enhancesthe consistency and generalization of the global model.The overall paradigm of our method is illustrated in Fig. 3.1. The methodoperates in two key stages: (1) generating federated impressions on theserver using a pixel-wise optimization process guided by the server model’sknowledge, and (2) leveraging these synthetic datasets as a regularizationterm during local training on client devices. In the first stage, we optimizesynthetic data to minimize cross-entropy loss while ensuring that the gradi-ents with respect to the server model’s parameters remain close to zero [50].This guarantees that the generated data faithfully captures the global dis-tribution encoded by the server model. In the second stage, these syntheticdatasets are distributed to clients and incorporated into their local train-ing processes, effectively mitigating feature drift and ensuring alignmentbetween local and global objectives.Our approach offers several advantages. By synthesizing federated impres-sions on the server, we ensure that critical global information is retainedand shared across clients without exposing sensitive data. This mechanismenhances the robustness of local training, reduces the impact of heterogene-ity, and addresses the challenges of catastrophic forgetting [8]. Furthermore,the integration of synthetic data into the local training pipeline allows forimproved generalization across diverse client datasets, resulting in a globalmodel that performs well on a wide range of tasks.The following sections provide a detailed description of each componentof our paradigm, including the optimization process for generating feder-ated impressions and the formulation of the regularization strategy for localtraining. These components collectively contribute to a scalable and privacy-preserving solution for addressing catastrophic forgetting in heterogeneousFL.293.1. Method3.1.3 Federated ImpressionThe challenge of addressing catastrophic forgetting and data heterogeneityin Federated Learning (FL) has led to the exploration of methods such asVHL [15], which introduced the use of global synthetic data to regularizelocal training. While this approach marked an important step, the syntheticdata generated by VHL fails to adequately preserve the server model’s crit-ical task-specific knowledge. Consequently, VHL’s synthetic data lacks theability to accurately represent the global distribution of the server model,limiting its effectiveness in mitigating feature drift and enhancing local train-ing in heterogeneous settings.To address these limitations, we propose an adaptive global data genera-tion paradigm designed to synthesize synthetic data, referred to as feder-ated impressions, based on the server model in each communication round.Unlike static synthetic data approaches, our method dynamically capturesthe evolving global distribution of the server model, ensuring the relevanceand utility of the generated data throughout the training process [9]. Thisadaptive approach not only strengthens the alignment between local andglobal objectives but also mitigates catastrophic forgetting by providing aconsistent reference point for local training.The primary objective of our method is to generate high-fidelity syntheticdata with an information-preserving property. This ensures that if a modelis trained from scratch using only the synthesized data, it achieves a perfor-mance comparable to the original server model. To achieve this, we formu-late the problem as an optimization task in the image space V, where pixelvalues are iteratively refined to minimize the cross-entropy (CE) loss of theserver model. Simultaneously, an equality constraint is imposed to keep thegradient of the CE loss with respect to the server model’s weights, θrG, closeto zero [52]. The optimization problem can be expressed as follows:minV∑(vi,yˆi)∈VLCE(g(f(vi; θrG);ϕrG), yˆi) s.t. ∇(θrG,ϕrG)LCE = 0, (3.3)where yˆi represents the pseudo-labels initialized using the predictions of theserver model for vi. This equality constraint ensures that the synthesizeddata captures the critical decision boundaries of the server model withoutintroducing biases or inaccuracies.Given the computational complexity of optimizing Eq. (3.3), we introduce303.1. Methoda relaxation according to [52] by applying the equality constraint only tothe classifier’s weights, ϕrG. This relaxed formulation significantly reducesthe computational burden while preserving the essential properties of thesynthetic data. The relaxed optimization problem is given as:minV∑(vi,yˆi)∈VLCE(g(f(vi; θrG);ϕrG), yˆi) s.t. ∇(ϕrG)LCE = 0. (3.4)This relaxation maintains the information-preserving property by focusingon latent representations instead of generating exact replicas of input data.By aligning the latent representations of the synthetic data with the globaldistribution, the method ensures that the synthesized data remains effectiveas a regularizer during local training.To solve the relaxed optimization problem efficiently, we employ an aug-mented Lagrangian formulation, which integrates the equality constraintinto the objective function as a penalty term. This approach allows foriterative optimization using the Alternating Direction Method of Multipli-ers (ADMM) [6], balancing computational efficiency with solution accuracy.The augmented Lagrangian objective is defined as:maxΛminVLFedImpres =∑(vi,yˆi)∈V[LCE(g(f(vi; θrG);ϕrG), yˆi)+ tr(ΛT∇ϕrGLCE)+ρ2||∇ϕrGLCE ||2], (3.5)where Λ denotes the Lagrangian dual variable matrix associated with theequality constraint, and ρ is the penalty hyperparameter that governs thestrength of the constraint [12]. By iteratively updating the synthetic dataand the dual variables, the ADMM approach refines the data to ensurefidelity and information preservation.The resulting synthetic data, termed federated impressions, encapsulatesthe global distribution and critical task-specific knowledge of the servermodel. These federated impressions are distributed to clients at the be-ginning of each communication round, serving as a robust regularizer dur-ing local training. By integrating this synthesized data into their trainingprocesses, clients can effectively counteract the biases introduced by theirheterogeneous datasets and align their updates with the global objective.313.1. MethodIn summary, the federated impression paradigm offers a scalable and adap-tive solution to the challenges of catastrophic forgetting and data hetero-geneity in FL. By dynamically generating high-fidelity synthetic data thatpreserves the global model’s knowledge, our method enhances the stabil-ity and robustness of the training process while adhering to the privacy-preserving principles of FL.3.1.4 Forgetting-Penalized Local TrainingTo effectively train the local model for client n in a Federated Learning(FL) environment, where data heterogeneity is a significant challenge, weemploy a forgetting-penalized training approach. This method builds uponthe federated impressions generated at the server and utilizes them to miti-gate catastrophic forgetting during local training. At the start of each localtraining round, the server transmits the optimized synthetic dataset V toeach client. These synthetic data act as a critical regularizer, complementingthe client’s local data and ensuring that the local model maintains alignmentwith the global model’s learned knowledge [1, 15].The essence of the forgetting-penalized training lies in augmenting the localdataset with synthetic data and jointly optimizing two key components: theloss on the client’s local data (Llocal) and the loss on the synthetic dataset(Lglobal). The optimization objective for the client model is expressed as:min(θrn,ϕrn)Llocal(θrn, ϕrn) + βLglobal(θrn, ϕrn), (3.6)where Llocal is the cross-entropy (CE) loss computed over the client’s privatedataset, and Lglobal is the CE loss evaluated on the shared synthetic dataset.The parameter β acts as a balancing factor, controlling the relative influenceof the synthetic data regularizer. By incorporating both local and globalcomponents in this manner, the training process effectively mitigates thebiases induced by the non-IID nature of local data distributions [3, 51].The local loss term, Llocal, ensures that the client model adapts to its specificdataset, enabling it to learn features that are locally relevant. This is crucialfor personalization and performance on the client’s tasks. However, with-out any form of regularization, the local optimization process can lead tocatastrophic forgetting, where the model overwrites critical knowledge fromthe global model in favor of local patterns [10]. Such forgetting adverselyimpacts the global model’s generalization ability after aggregation.323.1. MethodTo counteract this issue, the global loss term, Lglobal, is introduced as a regu-larization component. By training the client model on the federated impres-sions, Lglobal serves two critical purposes: first, it reinforces the knowledgeencapsulated in the synthetic data, ensuring that the local model retainsglobal patterns during training; second, it aligns the client’s updates withthe global objectives, thereby reducing the divergence between local andglobal models during aggregation [8].Unlike traditional regularization methods that directly minimize the param-eter distances between the local model and the global model, our approachleverages the information-preserving property of the synthetic data. Thisproperty ensures that the synthetic dataset encapsulates the global model’scritical features and decision boundaries. Training on this data helps thelocal model not only retain these features but also effectively incorporatethem into its own training process. This enables the local model to cap-ture both local-specific patterns and globally relevant knowledge, ensuringimproved generalization during aggregation [14].The choice of cross-entropy loss for both Llocal and Lglobal is particularlysuited to classification tasks, as it ensures robust training on both localand synthetic datasets. The synthetic data, by design, represent a distilledversion of the global knowledge, making the CE loss an effective metricfor aligning the local model’s output distribution with the global model’slearned representations [12].The parameter β plays a pivotal role in this optimization process. A highervalue of β increases the emphasis on global regularization, which can bebeneficial in scenarios where the local dataset is highly imbalanced or un-representative of the global distribution. Conversely, a lower β value allowsthe local model to prioritize learning from its dataset, which is advantageousfor tasks requiring high personalization. Selecting an appropriate value for βis thus crucial and may vary depending on the degree of data heterogeneityand the specific application requirements.This forgetting-penalized local training approach differs fundamentally fromprior methods that rely on directly regularizing parameter distances. Suchparameter-based methods often fail to capture the nuanced relationship be-tween the local model and the global model’s decision boundaries, leading tosuboptimal retention of global knowledge. By focusing on the informationencoded in the federated impressions, our method achieves a more robustbalance between local adaptability and global coherence [34].333.2. ExperimentsIn summary, the forgetting-penalized local training paradigm leverages theunique properties of federated impressions to address the challenges posedby data heterogeneity and catastrophic forgetting in FL. By incorporat-ing both local and global loss terms, this approach ensures that the clientmodels retain critical global knowledge while effectively learning from theirlocal datasets. This leads to improved convergence during aggregation andenhances the overall performance and robustness of the global model.3.2 Experiments3.2.1 DatasetsTo rigorously evaluate the performance of FedImpres, we conduct exper-iments using two well-established public medical image datasets: Blood-MNIST and the Retina dataset. These datasets are selected for theirrelevance to real-world medical imaging applications and their ability tosimulate two prominent challenges in Federated Learning (FL): label imbal-ance and domain shift.BloodMNIST: BloodMNIST is a benchmark dataset included in MedM-NIST [20], a comprehensive suite of medical imaging datasets designed forevaluating machine learning models on various medical tasks. BloodMNISTfocuses on the classification of blood cell images and contains eight distinctclasses, including eosinophils, basophils, and lymphocytes. This dataset isparticularly suitable for assessing FL algorithms under heterogeneous set-tings due to its multi-class nature and its relevance to medical diagnosis,such as detecting blood disorders or anomalies.The images in BloodMNIST are standardized to a resolution of 32 × 32pixels to maintain consistency in data preprocessing and model training.For simulating label imbalance in the FL setting, we partition the datasetamong eight clients using a Latent Dirichlet Allocation (LDA) partitioningscheme [4, 53]. The LDA parameter α controls the degree of heterogeneityin label distribution among clients. We utilize two settings: α = 0.01 formoderate label imbalance and α = 0.005 for severe label imbalance. Thesesettings mimic real-world scenarios where certain medical institutions orsources have overrepresented or underrepresented classes in their data.Additionally, while acquiring labeled medical data is often expensive andtime-consuming, accessing unlabeled public data of the same modality ismore feasible in many scenarios. To simulate such a setting, we randomly343.2. Experimentsselect a subset of BloodMNIST images without labels to initialize our syn-thetic data generation process, further aligning our experimental setup withreal-world constraints.Retina Dataset: The Retina dataset is a collection of retina images sourcedfrom four independent medical sites, originally curated for glaucoma detec-tion [21, 26, 27, 54]. Each image has a resolution of 256×256 pixels, provid-ing sufficient detail for detecting subtle pathological features. The datasetpresents two primary challenges for FL: label imbalance and domain shift.Label imbalance arises due to variations in the prevalence of Glaucomatousand normal samples across different sites, while domain shift stems fromdifferences in imaging equipment, protocols, and patient demographics ateach site.For our experiments, we focus on a binary classification task: distinguish-ing Glaucomatous retina images from normal ones. This task is clinicallysignificant, as early and accurate detection of glaucoma can prevent visionloss. To emulate a realistic FL scenario, we assign three sites as clients,each having 82 labeled samples. The fourth site, consisting of 100 unlabeledsamples, is utilized as a source of auxiliary data for initializing our syntheticdata generation algorithm. This approach reflects the common situation inmedical imaging where labeled data is scarce, but unlabeled data is availablein larger quantities.The Retina dataset’s inherent domain shift makes it an ideal testbed forevaluating the robustness of FL algorithms like FedImpres. Differences inimaging protocols and population demographics across sites necessitate thedevelopment of methods that generalize well across varying data distribu-tions. Moreover, the small dataset size emphasizes the need for effective dataaugmentation or synthetic data generation techniques to enhance model per-formance.Dataset Summary: Both datasets pose unique challenges that align withreal-world applications of FL in medical imaging. BloodMNIST providesa controlled environment to study the effects of label imbalance, while theRetina dataset introduces additional complexities of domain shift and lim-ited labeled data. Samples from both datasets, along with their respectivelabel distributions across clients, are detailed in the supplementary materi-als. These datasets enable comprehensive evaluation of FedImpres, show-casing its capability to address key challenges in heterogeneous FL settings.353.2. Experiments3.2.2 Experimental SettingsTo evaluate the performance of FedImpres in heterogeneous Federated Learn-ing (FL) scenarios, we conducted extensive experiments designed to addresstwo critical challenges: label imbalance and domain shift. These challengesare prevalent in real-world applications, particularly in medical imaging,where diverse data distributions across clients lead to significant variationsin model performance. The experimental settings, including datasets, ini-tialization strategies, and implementation details, are outlined below.• Data Heterogeneity: To assess the robustness of FedImpres underheterogeneous conditions, we simulated two distinct scenarios:– Label Imbalance: Using the BloodMNIST dataset, we simu-lated varying degrees of label imbalance by partitioning the dataacross eight clients via Latent Dirichlet Allocation (LDA) [4, 53].The LDA parameter α was set to 0.01 for moderate imbalanceand 0.005 for severe imbalance. Additionally, 10% of the datawas randomly withheld and used as unlabeled public data forsynthesis initialization, simulating realistic scenarios where la-beled data is scarce but unlabeled data is accessible. The taskrequired eight-way classification across clients.– Domain Shift: The Retina dataset, which contains data fromfour domains representing different demographics and equipmentsettings, was used to study domain shift. One domain was desig-nated as publicly available unlabeled data for synthesis initializa-tion, while the remaining three domains were distributed amongclients for binary classification (Glaucomatous vs. normal retina).This setup reflects real-world scenarios where medical data dis-tributions vary significantly across institutions.• Initialization Methods: We explored the impact of three synthesisinitialization strategies:– Random Initialization: Synthetic data was initialized withrandom noise, serving as a baseline to evaluate the effectivenessof the synthesis algorithm.– CIFAR-10 Initialization: Publicly available natural imagesfrom the CIFAR-10 dataset [55] were used to initialize the syn-thetic data. Despite being from a different domain, this initial-ization provided diverse representations to guide the synthesis363.2. Experimentsprocess.– Medical Initialization: Synthetic data was initialized with un-labeled images from the same modality as the client data. ForBloodMNIST, 10% of the dataset was used, while for Retina, onedomain was allocated as public data. This initialization repre-sents realistic scenarios in medical imaging.• Implementation Details: The implementation was standardized toensure consistency:– Model Architecture: A simple Convolutional Neural Network(CNN) was used for classification tasks. Detailed architecturespecifications are provided in the supplementary material.– Optimization Stages: Each communication round comprisedtwo key stages:∗ Synthesis Stage: Synthetic data was optimized using theSGD optimizer for 5 ADMM epochs for BloodMNIST and 32epochs for Retina. The server model’s weights were frozenduring this stage to capture its global distribution.∗ Local Training Stage: Clients updated their local mod-els using both private and synthetic data. The SGD opti-mizer was employed, and hyperparameters were tuned foreach dataset.– Hardware and Software: All experiments were implementedin PyTorch and executed on an NVIDIA Tesla V100 GPU with16 GB memory.• Experimental Configurations: Two configurations were designedto evaluate FedImpres:– 80 Communication Rounds: Clients performed 5 local updateepochs (E) per communication round, with 15 initial warm-uprounds using FedAvg.– 40 Communication Rounds: Clients performed 10 local up-date epochs (E) per communication round, with 10 warm-uprounds using FedAvg.Both configurations were trained for 400 total epochs to ensure con-sistency.373.2. ExperimentsDataset BloodMNIST Retinaα 0.01 0.005 NALocal update epochs (E) 5 10 5 10 5 10FedAvg [1] 83.1 82.4 39.0 37.6 55.7 52.0FedProx [3] 82.8 83.1 35.1 34.9 68.2 61.9VHL [15] 84.9 83.3 50.3 43.0 80.8 78.8FedVSS [51] 82.9 82.8 38.1 36.7 62.3 68.3FedCurv [56] 68.5 61.7 26.2 25.9 79.9 78.1FedReg [44] 20.1 16.9 18.9 16.8 62.5 62.1FedImpres (Random init) 83.9 82.6 52.6 51.4 78.1 80.6FedImpres (CIFAR-10 init) 84.1 83.6 60.2 53.8 81.5 79.8FedImpres (Medical init) 94.2 93.1 70.2 65.1 80.6 81.1Table 3.1: Classification accuracies on BloodMNIST and Retina datasetcompared with the state-of-the-art methods. We reported FedImpres resultsusing random, CIFAR-10, and medical unlabeled data of the same modalityfor initialization. While medical initialization consistently performed better,FedImpres outperformed baselines across all configurations.3.2.3 Comparison with BaselinesIn this section, we compare the performance of FedImpres with both com-mon Federated Learning (FL) algorithms and state-of-the-art (SOTA) meth-ods that address challenges similar to those targeted by FedImpres. Theprimary goal of this comparison is to evaluate how well FedImpres addresseskey issues such as data heterogeneity, label imbalance, and domain shift inFL settings.Among the common FL algorithms, we include FedAvg [1] and FedProx[3] as baseline methods. FedAvg, one of the foundational FL algorithms, av-erages the updates from all participating clients to update the global model.However, it is well-documented that FedAvg performs poorly under Non-IID(non-independent and identically distributed) data distributions, as it failsto address the divergence between client updates. To address this limitation,FedProx introduces a regularization term during local training, penalizingsignificant deviations of client models from the global model. While thisapproach mitigates the instability of FedAvg in Non-IID settings, its re-liance on penalizing divergence can hinder its ability to adapt effectively tohighly heterogeneous data distributions, particularly in cases of severe labelimbalance or domain shift.383.2. ExperimentsIn addition to these common methods, we compare FedImpres with SOTAFL algorithms that incorporate techniques for improving global model gen-eralization through synthetic data or enhanced local training. VHL [15]employs virtual global data generated using an untrained StyleGAN [30]to regularize client training. However, VHL’s synthetic data remains staticthroughout the training process, limiting its adaptability to dynamicallychanging data distributions during FL. Furthermore, the reliance on un-trained generative models can lead to synthetic data that fails to representcritical aspects of the global distribution.FedVSS [51], on the other hand, generates synthetic data locally by ad-versarially perturbing the client data using the global model. This methodaims to enhance the generalizability of individual client updates but doesnot focus on maintaining consistency across clients or capturing a unifiedglobal distribution. Consequently, while FedVSS achieves notable improve-ments in local generalization, it lacks a mechanism to directly address thechallenge of catastrophic forgetting in the global model, especially in casesof severe Non-IID data.We also include two SOTA methods specifically designed to tackle catas-trophic forgetting in FL: FedCurv [56] and FedReg [44]. FedCurv ex-tends the idea of Elastic Weight Consolidation (EWC) from continual learn-ing to FL by penalizing updates to parameters that are critical for priortasks. While FedCurv performs well on relatively balanced datasets, itseffectiveness diminishes under severe label imbalance, as observed in theBloodMNIST dataset. FedReg, which aligns local models with the globalmodel through a regularization mechanism, exhibits limited performanceon architectures involving batch normalization layers, further reducing itsapplicability to diverse FL scenarios.The comparison results are summarized in Table 3.1. FedImpres demon-strates significant improvements over all baselines and SOTA methods, par-ticularly in experiments involving severe heterogeneity, such as with α =0.005 on the BloodMNIST dataset and the Retina dataset with domainshift. Notably, FedImpres achieves consistent gains across all initializationsettings, including random noise, CIFAR-10, and medical data. This robust-ness highlights the strength of our synthesis approach in generating unifiedsynthetic data that effectively regularizes local training.For instance, in the Retina dataset, FedImpres with medical initializationachieves the highest accuracy in both local update epochs (E = 5 and E= 10), surpassing all baselines and SOTA methods. Even with CIFAR-10393.2. Experimentsor noise initialization, FedImpres outperforms competing methods, demon-strating the adaptability of our approach to varying initialization conditions.This adaptability ensures that FedImpres remains effective even in practicalscenarios where modality-specific unlabeled data may not be readily avail-able.Compared to VHL, FedImpres benefits from its adaptive synthesis mecha-nism, which updates synthetic data dynamically during each communicationround. This enables FedImpres to better capture evolving global distribu-tions and align local models with the server model. Similarly, the unifiedsynthesis approach of FedImpres ensures that all clients benefit from sharedsynthetic data, overcoming the limitations of FedVSS, which focuses solelyon local generalization.While FedCurv achieves competitive results on the Retina dataset, its re-liance on parameter-specific penalties limits its ability to handle the com-plexities of label imbalance in datasets like BloodMNIST. FedReg’s un-derperformance across both datasets highlights the challenges of applyingregularization-based approaches in scenarios involving diverse model archi-tectures and highly heterogeneous data distributions.Overall, FedImpres consistently outperforms all baselines and SOTA meth-ods across various datasets and heterogeneity settings. The superior per-formance of FedImpres can be attributed to its adaptive synthesis strategyand information-preserving loss function, which collectively address the dualchallenges of catastrophic forgetting and Non-IID data distributions in FL.3.2.4 Ablation StudiesTo comprehensively evaluate the contributions of our synthesis approach, weconducted ablation studies on the Retina dataset. The primary objective ofthese experiments was to isolate the impact of the proposed synthesis mech-anism, particularly the FedImpres loss function introduced in Eq. 3.4, com-pared to a simplified variant that employs only the standard cross-entropy(CE) loss for synthetic data generation.The Retina dataset, characterized by its inherent domain shifts and imbal-anced class distributions, serves as an ideal benchmark to assess the efficacyof the synthetic data generation process. The experiments were designedto determine how the inclusion of the information-preserving constraint inFedImpres impacts the synthetic data’s ability to encode global knowledgefrom the server model and subsequently enhance local client training.403.2. Experiments• Experimental Setup: Both FedImpres and the CE loss variant ini-tialized the synthetic data generation process with random noise, en-suring a uniform starting point for comparison. The same optimizationpipeline was applied across both methods to isolate the impact of theloss function. For local training, the number of update epochs (E) percommunication round was set to either 5 or 10, allowing us to evaluatethe models under varying levels of local training effort.• Evaluation Metrics: The primary evaluation metric was classifica-tion accuracy on the Retina dataset. This metric reflects the models’ability to generalize across heterogeneous data distributions, a criticalfactor in federated learning scenarios.• Findings: Table 3.2 presents the classification accuracies for bothsynthesis methods. FedImpres, which incorporates the FedImpres loss,consistently outperformed the CE loss variant across all configurations.Specifically, FedImpres achieved accuracies of 78.1% and 80.6% for5 and 10 local update epochs, respectively, compared to 73.9% and75.4% for the CE loss variant.Table 3.2: Classification accuracies on the Retina dataset comparingFedImpres with the CE loss variant. Both synthesis methods use randomnoise initialization.Method RetinaLocal update epochs (E) 5 10Data synthesis with CE loss 73.9 75.4Data synthesis with FedImpres loss 78.1 80.6The results underscore the critical role of the information-preserving con-straint in the FedImpres loss function. Unlike the CE loss, which optimizessynthetic data solely for classification accuracy, the FedImpres loss incorpo-rates a gradient constraint to ensure that the synthesized data retains theserver model’s knowledge. This alignment between the synthetic data andthe server model significantly enhances the quality of local training.• Importance of Information Preservation: The FedImpres lossfunction ensures that the synthetic data captures the server model’slatent knowledge, acting as a regularization mechanism during localtraining. This property mitigates catastrophic forgetting and improvesgeneralization, particularly in heterogeneous federated settings wherelocal client data distributions diverge significantly.413.2. Experiments• Impact on Local Training: By incorporating the FedImpres con-straint, FedImpres effectively transfers global knowledge encoded inthe server model to local clients. This transfer ensures that local train-ing not only adapts to the specific client data but also retains globalpatterns learned from the federated data distribution.• Practical Implications: The superior performance of FedImpreshighlights the importance of incorporating additional constraints intosynthetic data generation. These constraints ensure that the synthe-sized data serves as a robust bridge between the global and local train-ing processes, enabling federated learning systems to achieve higheraccuracy and better generalization across diverse client environments.In conclusion, the ablation studies clearly demonstrate the advantages ofFedImpres over simpler synthesis methods. The incorporation of the Fed-Impres loss function plays a pivotal role in enhancing the quality of syntheticdata, ensuring that it aligns with the server model’s global knowledge andcontributes to more effective local training. These findings validate the de-sign choices underlying FedImpres and highlight its potential for addressingkey challenges in federated learning, particularly in heterogeneous and im-balanced settings.3.2.5 Analysis and DiscussionOur experimental results provide strong evidence that FedImpres effectivelymitigates the challenges posed by label imbalance and domain shift in fed-erated learning (FL), both of which are pervasive in real-world applicationsinvolving heterogeneous data. By leveraging a shared synthetic dataset thatcaptures the global distribution, FedImpres achieves a delicate balance be-tween preserving global knowledge and enabling client-specific local adapta-tion. This dual capability addresses catastrophic forgetting while improvingmodel robustness in diverse data settings.Effectiveness Against Label Imbalance and Domain Shift Labelimbalance, where certain classes dominate local datasets, and domain shift,arising from variations in client-specific data distributions, represent majorbarriers to effective FL. These issues lead to biased local updates, poor gen-eralization, and reduced global model performance. FedImpres tackles theseproblems by synthesizing a dataset that encapsulates the global distributionwhile complementing local data during training. The shared synthetic data423.2. Experimentsserves as a regularizer, reducing the adverse effects of data heterogeneityand promoting consistency across clients.Compared to methods like FedAvg and FedProx, FedImpres exhibits supe-rior performance due to its data-driven approach and information-preservingproperties. FedAvg assumes that local updates are uniformly representative,which is invalid in non-IID scenarios, while FedProx focuses on constraininglocal updates but lacks mechanisms to address global knowledge retention.By contrast, FedImpres integrates synthetic data generation and regular-ization into the training process, resulting in more stable and robust modelupdates.Impact of Synthesis Initialization A significant aspect of our findingsis the influence of the initialization method for synthetic data generation.The results demonstrate that initializing with a medical dataset of the samemodality yields the best outcomes, highlighting the importance of domain-specific alignment. Such initialization provides a better foundation for gen-erating synthetic data that aligns with the global distribution, enhancingperformance in downstream tasks.Despite the superior results with medical initialization, FedImpres remainsrobust when initialized with CIFAR-10 or random noise. This adaptabil-ity underscores its practical utility, particularly in scenarios where domain-specific public datasets may be unavailable. Even with non-optimal initial-ization, FedImpres consistently outperforms baseline methods, demonstrat-ing its resilience and versatility.Role of the FedImpres Loss Function The ablation studies confirmthe critical importance of the FedImpres loss function. Unlike standardcross-entropy (CE) loss, which optimizes for synthetic data accuracy withoutexplicit regard for global knowledge retention, the FedImpres loss incorpo-rates an information-preserving constraint. This constraint ensures that thesynthesized data retains the server model’s latent knowledge, significantlyenhancing local training. Experimental comparisons show that FedImpresachieves higher accuracy and robustness when employing the FedImpres loss,validating its role as a core component of the synthesis process.Comparison with Alternative Methods In addition to surpassing Fe-dAvg and FedProx, FedImpres demonstrates advantages over recent meth-ods like VHL and FedVSS. VHL employs static synthetic data as regular-433.3. Conclusionization anchors, limiting its adaptability to evolving global distributions.Similarly, FedVSS focuses on variance reduction but does not directly ad-dress knowledge retention or catastrophic forgetting. In contrast, FedImprescombines adaptive synthetic data generation with an information-preservingobjective, offering a more comprehensive solution to the challenges of dataheterogeneity in FL.Insights for Real-World Deployment The results from FedImpreshave significant implications for practical FL applications, particularly insensitive domains like healthcare. By synthesizing high-fidelity, privacy-preserving data that aligns with global distributions, FedImpres enablescollaborative training without violating data privacy or fairness principles.Furthermore, its robustness to initialization methods ensures adaptability tovarious deployment environments, from resource-rich settings with access todomain-specific public data to more constrained scenarios relying on genericdatasets or random noise.Discussion of Limitations and FutureWork While FedImpres demon-strates exceptional performance, some limitations remain. The computa-tional cost of synthetic data generation, particularly when using the FedIm-pres loss, highlights the need for optimization to scale to larger datasets andmore complex models. Additionally, exploring the integration of FedImpreswith multi-modal FL tasks, where data spans diverse formats such as text,images, and time-series, presents an interesting avenue for future research.Summary of Key Contributions Overall, the results reaffirm the ef-ficacy of FedImpres in addressing critical challenges in FL. By combininginnovative synthetic data generation techniques with robust regularizationmechanisms, FedImpres establishes itself as a powerful tool for enhancingthe performance and robustness of FL systems in heterogeneous data envi-ronments.3.3 ConclusionFederated Learning (FL) has emerged as a pivotal machine learning paradigmfor decentralized training, particularly in privacy-sensitive domains such ashealthcare. Despite its potential, FL faces significant challenges arising fromthe heterogeneous nature of client data. This heterogeneity, characterizedby domain shift and label imbalance, exacerbates catastrophic forgetting443.3. Conclusionduring local training, particularly in medical datasets where data scarcityand variability are intrinsic. Our study addresses these challenges throughthe introduction of a novel method, FedImpres, designed to mitigate catas-trophic forgetting by leveraging server-generated synthetic data during localtraining.The central innovation of FedImpres lies in its synthesis of federated impres-sions—data generated by the server model at each communication round topreserve global knowledge during local updates. This synthetic data servesas an information-rich regularizer, enabling local models to balance client-specific learning with the retention of global knowledge. By incorporatingthe FedImpres loss function, FedImpres ensures that the generated data re-flects the server model’s distribution while maintaining information criticalfor generalization. This adaptive synthesis process eliminates the need foradditional generative models, reducing computational overhead and increas-ing the method’s practicality in real-world FL applications.Our comprehensive experimental evaluation demonstrates the efficacy ofFedImpres across two benchmark medical datasets: BloodMNIST and Retina.These datasets exemplify typical FL challenges, including extreme label im-balance and domain shift. The results unequivocally show that FedImpressignificantly outperforms traditional baselines such as FedAvg and FedProx,as well as state-of-the-art methods like VHL and FedVSS. Notably, the ad-vantages of FedImpres are most pronounced under conditions of severe het-erogeneity, such as when the LDA partition parameter α is set to 0.005 orwhen the Retina dataset, with its inherent domain shifts, is used. Thesefindings highlight FedImpres’ robustness in diverse and challenging FL sce-narios.The ablation studies further underscore the importance of the FedImpresloss function, which ensures that the synthesized data captures and preservesglobal knowledge. Comparisons with alternative synthesis approaches, suchas those using only cross-entropy loss, reveal the critical role of the information-preserving constraint in enhancing local training and overall performance.These studies validate our design choices and reinforce the theoretical un-derpinnings of FedImpres.An additional strength of FedImpres lies in its adaptability to various ini-tialization schemes for the synthesis process. While the use of medical datafrom the same modality yields the best results, experiments show that evenwith CIFAR-10 or random noise initialization, FedImpres consistently out-performs other methods. This flexibility demonstrates the robustness of the453.3. Conclusionsynthesis algorithm and its ability to adapt to different scenarios withoutsacrificing performance.From a practical perspective, FedImpres is well-suited for deployment inhealthcare applications, where privacy and data heterogeneity are paramountconcerns. By enabling collaborative training across institutions withoutsharing raw data, FedImpres aligns with the stringent privacy requirementsof healthcare regulations such as GDPR and HIPAA. Furthermore, its abil-ity to handle extreme heterogeneity makes it particularly valuable in multi-center studies, where data distributions often vary significantly due to dif-ferences in demographics, equipment, and protocols.Despite its strengths, FedImpres is not without limitations. The currentsynthesis algorithm solves the optimization problem approximately, leavingroom for further refinement. Future research could explore more advancedoptimization techniques to enhance the fidelity and efficiency of the gener-ated data. Additionally, while FedImpres has been validated on benchmarkdatasets, its performance on more complex and diverse medical datasetsremains an avenue for further investigation. Integrating FedImpres intoreal-world clinical workflows and adapting it to meet the rigorous standardsof clinical practice will also be critical next steps.In conclusion, FedImpres represents a significant advancement in FL byaddressing the dual challenges of catastrophic forgetting and data hetero-geneity through a novel synthetic data generation approach. Its superiorperformance, efficiency, and adaptability position it as a promising solutionfor decentralized learning in healthcare and other privacy-sensitive domains.Future work will focus on expanding the applicability of FedImpres, im-proving its synthesis algorithm, and exploring its integration into practicalmedical applications to fully realize its potential.46Chapter 4Federated Prompt Learningwith Dual-PromptOptimization and FederatedImpression Alignment4.1 Method4.1.1 Problem SettingFederated Learning (FL) is a distributed machine learning framework de-signed to collaboratively train a global model across a set of clients {C1, ..., CN},each possessing private data Xn. Here, N is the total number of clients, andeach client’s data is a collection of labeled samples (xni , yni ), where xni de-notes an input sample, and yni its corresponding label. The aim is to train aglobal model that generalizes well across all clients while maintaining dataprivacy by keeping Xn decentralized and private.In the context of this study, we employ a pretrained Vision Transformer(ViT) model, leveraging its modular structure to adapt to the FL paradigm.To address the heterogeneity of client data, we introduce a dual-level promptarchitecture, consisting of shared prompts and group prompts. Sharedprompts are designed to capture generalizable global patterns across clients,while group prompts provide client-specific adaptations for task-specific in-formation.The overall objective in this FL setting is to minimize the global classificationloss, formulated as:474.1. MethodJ(θG, ϕG) =N∑n=1E(xni ,yni )∈Xn ℓ(g(f(xni ; θG);ϕG), yni ), (4.1)where ℓ denotes the cross-entropy (CE) loss LCE . The pretrained ViT modelcomprises a feature extractor f(; θG) with parameters θG and a classifierg(;ϕG) with parameters ϕG.During each communication round r, the server broadcasts the global modelparameters θrG to all participating clients. Each client initializes its localmodel using these parameters and trains both the group prompts and itsclassifier using its private dataset Xn. The shared prompts are updatedexclusively at the server using globally synthesized data, ensuring they gen-eralize across all clients.The local training process yields client-specific model updates θrn, whichare aggregated at the server to update the global model parameters θr+1G .The Federated Averaging (FedAvg) [1] algorithm is typically used for thispurpose:θr+1G =1NN∑n=1θrn. (4.2)This process is iteratively repeated until the global model achieves satis-factory performance. However, several challenges arise in FL settings withheterogeneous client data, such as:• Local Overfitting: Client models tend to overfit to their local datadistribution, especially when the data is non-independent and non-identically distributed (non-IID).• Catastrophic Forgetting: Global knowledge may be lost duringlocal training, particularly in non-IID environments where data distri-butions differ significantly across clients.• Conflicting Gradients: The aggregation of client updates may leadto conflicting gradients, slowing convergence and degrading the globalmodel’s performance.To address these issues, we propose a novel dual-prompt architecture com-bined with a federated impression synthetic dataset. The key componentsof our approach include:484.1. Method1. Shared Prompts: Trained using globally synthesized federated im-pressions and the local client data, these prompts ensure generalizationacross clients and prevent overfitting to local data.2. Group Prompts: Adapted using client-specific data, these promptscapture task-specific local information, enabling personalization whilemaintaining alignment with the global model.This dual-prompt architecture effectively mitigates catastrophic forgettingand local overfitting, achieving robust performance in FL settings with het-erogeneous client data. The detailed methodology is outlined in the followingsections.4.1.2 OverviewAs discussed in the introduction (Sec. 2.2.1), the dual challenges of het-erogeneous data distribution and catastrophic forgetting present significantbarriers to effective Federated Learning (FL). These challenges stem fromthe non-IID nature of client datasets, which causes local models to overfit toclient-specific distributions, leading to feature drift during global aggrega-tion. Addressing these challenges requires innovative strategies that balanceglobal generalization with local adaptation while preserving computationaland privacy constraints.In this study, we propose a novel FL framework based on the Dual-Promptarchitecture, which leverages a pretrained Vision Transformer (ViT) model.The architecture introduces two types of prompts—shared prompts andgroup prompts—that operate synergistically to mitigate catastrophic for-getting and enhance model performance. Shared prompts are positioned inthe earlier layers of the ViT to capture generalizable global knowledge, whilegroup prompts are attached to deeper layers to learn task-specific local in-formation. This modular design ensures that global information is preservedacross clients while enabling personalized adaptation to client-specific data.To further enhance the generalization of the shared prompts and preventoverfitting, we incorporate federated impressions generated at the serverlevel. These impressions, as described in the FedImpress study, provide asynthesized representation of the global data distribution and are used to ex-clusively train the shared prompts. By leveraging federated impressions, theshared prompts retain critical global knowledge while remaining robust tothe heterogeneity of local datasets. Group prompts, on the other hand, aretrained on client-local data, allowing them to adapt to task-specific nuances494.1. Methodwhile ensuring alignment with the shared prompts’ generalization capabili-ties.Our method operates in three key stages. First, the Dual-Prompt frame-work is introduced, detailing the roles and interactions of shared and groupprompts in the federated setting. Second, federated impressions, generatedas per the methodology described in the FedImpress study, are integratedinto the training pipeline to regulate the shared prompts. Lastly, the train-ing and optimization process is detailed, explaining how shared and groupprompts are iteratively trained to mitigate catastrophic forgetting and bal-ance global and local learning objectives.This structured approach ensures a scalable, privacy-preserving solution forFL in heterogeneous settings. By integrating the Dual-Prompt architecturewith federated impressions, our method strikes a balance between globalgeneralization and local personalization, addressing critical challenges inFL. The following sections provide a detailed description of the Dual-Promptframework, the integration of federated impressions, and the training processthat underpins our methodology.4.1.3 Dual-Prompt FrameworkThe Dual-Prompt Framework is the cornerstone of our proposed methodol-ogy, leveraging a pretrained Vision Transformer (ViT) to address the chal-lenges of catastrophic forgetting and data heterogeneity in Federated Learn-ing (FL). This framework introduces two distinct types of prompts—sharedprompts and group prompts—each with unique roles in balancing globalgeneralization and local adaptation.Shared Prompts: Capturing Global Knowledge Shared prompts aredesigned to encapsulate generalizable knowledge that is common across allclients. These prompts are appended to the earlier layers of the pretrainedViT, which are responsible for extracting high-level, task-agnostic featuresfrom input data. By situating shared prompts at these layers, we ensurethat the information they capture remains invariant to client-specific datavariations.The shared prompts are trained exclusively using federated impressions, asdescribed in the FedImpress study. This ensures that they are aligned withthe global data distribution, preventing overfitting to any single client’s data.By using synthetic federated impressions instead of raw data, we adhere to504.1. Methodprivacy constraints while maintaining high generalization performance. Theshared prompts are kept consistent across all clients, effectively acting as aunifying mechanism that integrates global knowledge into the local trainingprocess.Group Prompts: Enabling Local Adaptation Group prompts, incontrast, are designed to learn client-specific information. These promptsare attached to the deeper layers of the ViT, which are better suited forencoding task-specific features. During local training, the group promptsadapt to the unique data distribution of each client, capturing task-relevantnuances and variations. Unlike shared prompts, which are uniform acrossclients, group prompts are personalized and optimized independently foreach client.To enhance personalization, the selection of group prompts is input-dependent.A dynamic routing mechanism assigns specific group prompts to individualinput samples, allowing the model to leverage different prompts based onthe characteristics of the input. This input-specific adaptability furtherenhances the ability of group prompts to capture local information whilemaintaining alignment with the shared prompts.Interaction Between Shared and Group Prompts The dual-promptstructure ensures a seamless interplay between shared and group prompts.Shared prompts provide a stable foundation of global knowledge, preventingcatastrophic forgetting by preserving information that is consistent acrossclients. Group prompts, on the other hand, add a layer of adaptability,allowing the model to fine-tune its representations based on local data.This division of roles mitigates the risk of overfitting that typically arises inheterogeneous FL settings. The shared prompts act as an anchor, ensuringthat local updates do not deviate significantly from the global objective.Meanwhile, group prompts ensure that the model remains sensitive to client-specific variations, facilitating effective personalization.Integration into the Federated Learning Pipeline The Dual-PromptFramework is integrated into the FL pipeline in a way that balances com-putational efficiency with performance. During each communication round,the shared prompts are updated using the federated impressions generatedby the server, while the group prompts are trained on the client’s local data.This dual optimization process ensures that shared prompts remain robust514.1. Methodto client-specific overfitting, while group prompts effectively capture localtask-specific information.At the end of each round, only the shared prompts are aggregated at theserver, ensuring that the global model benefits from generalized knowledgeacross all clients. The group prompts remain client-specific, preserving theiradaptability and avoiding interference during aggregation.Advantages of the Dual-Prompt Framework The Dual-Prompt Frame-work offers several advantages in addressing the challenges of FL. By sep-arating the roles of shared and group prompts, it achieves a balance be-tween global generalization and local personalization. This structure miti-gates catastrophic forgetting by preventing the shared prompts from overfit-ting to client data, while simultaneously improving local model performancethrough the adaptability of group prompts.Moreover, the use of federated impressions to train the shared prompts en-sures privacy preservation while maintaining high performance. The modu-lar nature of the framework makes it scalable and adaptable to a wide rangeof FL scenarios, particularly those involving heterogeneous client data dis-tributions.The next subsection outlines how federated impressions are leveraged withinthis framework, followed by a detailed explanation of the training and opti-mization process that underpins our methodology.4.1.4 Federated Impression IntegrationThe integration of federated impressions into the Dual-Prompt Frameworkplays a pivotal role in addressing the challenges of catastrophic forgettingand data heterogeneity in Federated Learning (FL). Federated impressions,introduced in the FedImpress study, are synthetic datasets generated at theserver to approximate the global data distribution. These synthetic datasetsprovide a privacy-preserving means to share knowledge across clients with-out exposing raw data, and their incorporation into our framework ensureseffective alignment between local and global objectives.Purpose of Federated Impressions In the Dual-Prompt Framework,federated impressions are used exclusively to train the shared prompts. Theshared prompts are responsible for capturing generalizable knowledge thatis invariant across client datasets. By training these prompts on federated524.1. Methodimpressions rather than client data, we ensure that the shared prompts donot overfit to the idiosyncrasies of individual clients, thereby preserving theirability to encode global knowledge effectively.Federated impressions also mitigate the feature drift caused by the inherentnon-IID nature of client datasets. By providing a synthetic representationof the global data distribution, federated impressions anchor the sharedprompts to a consistent feature space, reducing the risk of divergence duringaggregation. This ensures that the global model retains robustness andgeneralization across diverse clients.Generation of Federated Impressions The federated impressions aregenerated at the server using the FedImpress method. This involves opti-mizing synthetic data points to match the global model’s predictions whileminimizing cross-entropy loss. The optimization process ensures that thegenerated data reflects the global knowledge encoded in the server model,making it suitable for training shared prompts. The use of synthetic dataavoids privacy concerns associated with sharing raw client data, making ita secure and efficient approach for FL.Incorporating Federated Impressions into Training During eachcommunication round, federated impressions are distributed from the serverto all participating clients. These synthetic datasets are incorporated intothe local training pipeline in a controlled manner to achieve specific objec-tives:1. Shared Prompt Training: The shared prompts are trained exclu-sively on federated impressions. This decouples their training fromclient-specific data, ensuring that they remain aligned with the globaldata distribution. By doing so, the shared prompts avoid overfittingto the local data and maintain their generalizability.2. Group Prompt Training: While federated impressions are not di-rectly used for training group prompts, their influence is indirectlyfelt. The global knowledge encoded in the shared prompts serves asa foundation upon which group prompts can build their client-specificadaptations. This interplay ensures that group prompts benefit fromthe robustness of the shared prompts while remaining flexible to localvariations.3. Regularization Mechanism: Federated impressions act as a regu-534.1. Methodlarizer during local training, guiding the optimization process to pre-vent overfitting to local data. This regularization ensures that localupdates align more effectively with the global objective, improving thestability and convergence of the FL process.Optimization Process The training process for federated impressionsfollows a two-stage optimization strategy:1. Server-Side Training: Federated impressions are generated and re-fined on the server using the global model. This involves optimizingsynthetic data points to minimize cross-entropy loss while ensuringthat the gradients with respect to the model parameters remain closeto zero. The resulting synthetic datasets faithfully capture the globaldata distribution.2. Client-Side Training: Upon receiving the federated impressions,clients use them to train their shared prompts alongside their localdata. The shared prompts are optimized using federated impressionsto maintain consistency with the global model, while group promptsare trained on local data to capture client-specific information.Impact on the Dual-Prompt Framework The integration of feder-ated impressions into the Dual-Prompt Framework addresses two criticalchallenges in FL: mitigating catastrophic forgetting and improving modelgeneralization. By using federated impressions to train the shared prompts,we ensure that global knowledge is preserved across communication rounds,reducing the risk of forgetting. At the same time, the regularization providedby federated impressions improves the alignment between local updates andthe global objective, enhancing the overall performance of the FL system.Moreover, the use of federated impressions allows us to leverage the pre-trained ViT effectively. By aligning shared prompts with the global datadistribution, federated impressions maximize the utility of the pretrainedmodel’s representations, ensuring that both shared and group prompts op-erate in a consistent and optimized feature space.The next subsection details the joint optimization strategy for trainingshared and group prompts, highlighting how these components work to-gether to achieve our objectives.544.2. Experiments4.2 Experiments4.2.1 DatasetsTo comprehensively evaluate the performance of PromptFusion, we em-ploy the BloodMNIST and Retina datasets, as described in FedImpress.These datasets are particularly suitable for analyzing Federated Learning(FL) frameworks due to their ability to simulate critical challenges such aslabel imbalance and domain shift. BloodMNIST focuses on multi-class clas-sification of blood cell images, where the distribution of labels is controlledusing a Latent Dirichlet Allocation (LDA) scheme to simulate varying de-grees of heterogeneity. The Retina dataset, on the other hand, introducesdomain shift by combining images from different medical imaging sites withdiverse demographic and equipment characteristics. This binary classifica-tion dataset also incorporates a scarcity of labeled data, a common scenarioin real-world medical applications.For a detailed description of the datasets, including the preprocessing steps,label distributions, and partitioning strategies, please refer to the experi-mental setup section in FedImpress (Section 3.2). Here, we focus on howthese datasets are employed within the context of PromptFusion’s uniqueevaluation framework.4.2.2 Experimental SettingsThe experimental configurations largely follow those outlined in FedIm-press, ensuring a direct comparison between PromptFusion and existingapproaches. These configurations are designed to rigorously test FL al-gorithms under heterogeneous conditions. However, additional emphasis isplaced on evaluating PromptFusion’s capacity to preserve global knowledgeand adapt locally without overfitting. Below, we detail the experimentalsettings and enhancements specific to this work.Data Heterogeneity: Two distinct challenges are simulated using Blood-MNIST and Retina datasets:• Label Imbalance: For BloodMNIST, the LDA-based partitioningstrategy assigns data to eight clients, with α = 0.01 representing mod-erate imbalance and α = 0.005 representing severe imbalance. Thepartitioning results in varying levels of label distribution skew, mim-icking realistic FL scenarios in medical imaging where certain institu-554.2. Experimentstions have overrepresented or underrepresented classes.• Domain Shift: In the Retina dataset, images from four distinct sitesare distributed across three clients, with one site serving as a source ofunlabeled auxiliary data. This configuration reflects the diversity inimaging protocols and patient demographics typically encountered inmulti-site medical studies. Differences between client domains makethis dataset particularly challenging for FL algorithms.Evaluation Metrics: In addition to classification accuracy, we evaluatethe extent of catastrophic forgetting by tracking performance across all com-munication rounds. Metrics such as retention of prior knowledge and con-sistency of predictions over time are computed, providing deeper insightsinto PromptFusion’s ability to maintain a balance between global general-ization and local personalization. These metrics are especially crucial inFL settings where non-IID data distributions exacerbate the forgetting ofpreviously learned tasks.Implementation Details: PromptFusion experiments are implementedusing PyTorch, with the backbone model being a Vision Transformer (ViT)pretrained on ImageNet. Shared prompts are attached to earlier layers tocapture global knowledge, while group prompts are attached to deeper layersto enable personalized adaptations for individual clients. The optimizationprocess involves:• Shared Prompt Training: Shared prompts are updated using fed-erated impressions at the server, ensuring global relevance withoutoverfitting to specific client data.• Group Prompt Training: Group prompts are fine-tuned on localdata by individual clients, capturing domain-specific features whileleveraging global knowledge encoded in shared prompts.The SGD optimizer is used for all training stages, with a carefully tunedlearning rate and momentum to balance the optimization of shared andgroup prompts. Each communication round consists of prompt trainingfollowed by local model updates, ensuring synchronization across clients.Computational Configurations: Two configurations are evaluated totest PromptFusion under varying computational constraints:564.2. Experiments• 80 Communication Rounds: Clients perform five local updateepochs per round. The first 15 rounds are dedicated to warming up theglobal model using FedAvg, after which prompt training is introduced.• 40 Communication Rounds: Clients perform ten local updateepochs per round, with a shorter warm-up phase of ten rounds usingFedAvg. This configuration allows for more intensive local training,testing PromptFusion’s ability to adapt under fewer communicationrounds.All experiments are conducted on an NVIDIA Tesla V100 GPU with 16 GBmemory, ensuring consistency and reproducibility.4.2.3 Comparison with BaselinesTo thoroughly evaluate the performance of PromptFusion, we comparedit against multiple baselines, including standard Federated Learning (FL)methods like FedAvg [1] and FedProx [3], as well as advanced approachessuch as DualPrompt and FedImpress. The comparisons provide insights intohow PromptFusion handles the challenges of label imbalance, domain shift,and catastrophic forgetting, which are critical in real-world FL settings.Table 4.1 summarizes the results.Performance on BloodMNIST: For BloodMNIST, PromptFusion con-sistently outperforms all baselines across different levels of label imbalance.In the α = 0.01 configuration (moderate imbalance), PromptFusion achievesan accuracy of 95.0% with 5 local update epochs and 94.6% with 10 epochs,demonstrating its superior capability to generalize across clients despite im-balanced data distributions. This is a significant improvement over FedAvg(83.1% and 82.4%, respectively) and FedProx (82.8% and 83.1%), whichboth struggle in handling heterogeneity effectively.In the α = 0.005 configuration (severe imbalance), the gap widens further.PromptFusion achieves 72.6% and 71.4% accuracies for 5 and 10 localupdate epochs, respectively. Comparatively, FedAvg drops to 39.0% and37.6%, while FedProx fares only slightly better at 35.1% and 34.9%. Theseresults highlight the limitations of standard FL algorithms in addressingsevere heterogeneity. DualPrompt, even without the integration of federatedimpressions, performs well, achieving 68.5% and 67.2%. This improvementover FedImpress demonstrates the advantage of utilizing a pretrained VisionTransformer (ViT) backbone, which provides robust feature extraction and574.2. Experimentsgeneralization capabilities.Performance on Retina Dataset: On the Retina dataset, which poseschallenges due to domain shift, PromptFusion again demonstrates its ro-bustness. For 5 local update epochs, PromptFusion achieves a global ac-curacy of 81.0%, compared to 55.7% and 68.2% for FedAvg and FedProx,respectively. Similarly, for 10 epochs, PromptFusion achieves 82.5%, out-performing both FedAvg (52.0%) and FedProx (61.9%). The integration offederated impressions in PromptFusion ensures better alignment betweenlocal and global objectives, enabling the model to generalize across diversedomains.DualPrompt, with accuracies of 74.6% and 78.3% for 5 and 10 epochs, re-spectively, also outperforms FedImpress in this setting. This can be at-tributed to its use of pretrained features, which provide a strong initializa-tion for both shared and group prompts. However, the absence of federatedimpressions means it cannot fully address the domain shift challenges, re-sulting in lower performance compared to PromptFusion.Advantages over Baselines: The results clearly demonstrate the advan-tages of PromptFusion over the baselines:• Superior Generalization: PromptFusion consistently achieves higheraccuracies, particularly under severe heterogeneity and domain shiftconditions. Its dual-prompt architecture, combined with federatedimpressions, enables it to capture both global and local knowledgeeffectively.• Resilience to Label Imbalance: PromptFusion significantly out-performs FedAvg and FedProx in BloodMNIST experiments, high-lighting its robustness in scenarios with highly skewed label distribu-tions.• Enhanced Domain Adaptation: On the Retina dataset, Prompt-Fusion’s ability to handle domain shift is evident, with consistent im-provements over FedAvg, FedProx, and even DualPrompt.• Pretrained Model Benefits: The standalone DualPrompt frame-work showcases the value of using a pretrained ViT backbone, achiev-ing higher accuracy than FedImpress in many cases. This indicatesthat even without federated impressions, pretrained models can offersubstantial benefits in FL settings. However, integrating federated584.3. ConclusionDataset BloodMNIST Retinaα 0.01 0.005 NALocal update epochs (E) 5 10 5 10 5 10FedAvg [1] 83.1 82.4 39.0 37.6 55.7 52.0FedProx [3] 82.8 83.1 35.1 34.9 68.2 61.9FedImpres (Medical init) 94.2 93.1 70.2 65.1 80.6 81.1DualPrompt 92.7 91.9 68.5 67.2 74.6 78.3PromptFusion 95.0 94.6 72.6 71.4 81.0 82.5Table 4.1: Classification accuracies on BloodMNIST and Retina datasets.PromptFusion demonstrates significant improvements over baselines, par-ticularly under severe label imbalance and domain shift conditions.impressions with DualPrompt further enhances performance, as evi-denced by PromptFusion’s superior results.Forgetting Mitigation: While standard baselines like FedAvg and Fed-Prox suffer from catastrophic forgetting, especially under non-IID condi-tions, PromptFusion demonstrates minimal performance degradation acrosscommunication rounds. The dual-prompt architecture effectively isolatesglobal knowledge from local updates, preventing interference and retainingcritical information.Summary: PromptFusion establishes itself as a robust framework for het-erogeneous FL scenarios, outperforming traditional and advanced baselinesacross diverse datasets and configurations. The integration of federated im-pressions into the Dual-Prompt architecture leverages the strengths of bothapproaches, providing a holistic solution to the challenges of label imbalance,domain shift, and catastrophic forgetting. These results validate the designchoices underlying PromptFusion and highlight its potential for broader ap-plications in FL settings.4.3 ConclusionIn this study, we present PromptFusion, a novel Federated Learning (FL)framework that extends the Dual-Prompt architecture to address the criticalchallenges of catastrophic forgetting, data heterogeneity, and domain shift.By integrating federated impressions with pretrained Vision Transformers(ViTs), PromptFusion achieves a robust balance between global generaliza-594.3. Conclusiontion and local personalization, overcoming limitations inherent in existingFL methods.The proposed framework demonstrates significant performance improve-ments across diverse datasets, including BloodMNIST and Retina, whichencompass real-world challenges such as label imbalance and domain shifts.Experimental results show that PromptFusion consistently outperforms state-of-the-art FL baselines, including FedAvg, FedProx, and even standaloneDualPrompt, particularly in scenarios with severe heterogeneity. The syn-ergy between federated impressions and prompt-tuning ensures efficient knowl-edge transfer across clients while preserving privacy, a cornerstone of FL.Ablation studies confirm the pivotal roles of shared and group prompts in en-abling global and local adaptation, respectively. The integration of federatedimpressions further enhances the retention of global knowledge, preventingcatastrophic forgetting and ensuring robust performance across communica-tion rounds. Notably, the use of pretrained ViTs provides a strong founda-tion for feature extraction, enabling DualPrompt alone to surpass existingmethods like FedImpress in several cases, while PromptFusion’s completearchitecture achieves even greater gains.This work establishes the feasibility of combining federated impressionswith prompt-based learning architectures, opening new avenues for privacy-preserving and adaptable FL solutions. Future research directions includeextending PromptFusion to domains such as healthcare and finance, whereprivacy and data heterogeneity are critical. Additionally, refining the feder-ated impression generation process and exploring advanced prompt-tuningstrategies could further enhance its scalability and effectiveness. PromptFu-sion represents a significant step forward in the integration of prompt-basedmethods into FL, setting the stage for more adaptable and robust distributedlearning systems.60Chapter 5Summary and Future Work5.1 SummaryThis thesis explored innovative approaches to address critical challenges inFederated Learning (FL), particularly focusing on data heterogeneity andcatastrophic forgetting. Through two distinct but complementary studies,we proposed novel methodologies that significantly enhance the effectivenessand robustness of FL in diverse and non-IID data settings.5.1.1 Study 1: FedImpressThe first study introduced FedImpress, a novel framework designed totackle the challenges posed by data heterogeneity in FL. In conventionalFL, the lack of direct access to client data and the non-IID nature of localdatasets often lead to misaligned updates and suboptimal global models.To address this, FedImpress employed a server-driven synthetic data gen-eration approach to distill global knowledge into ”federated impressions.”These impressions are synthetic datasets optimized to capture the globaldata distribution while preserving privacy by avoiding direct use of clientdata.FedImpress synthesized federated impressions using pixel-wise optimizationguided by the server model’s predictions and gradient constraints. Thesesynthetic datasets were then shared with clients to serve as a regularizationterm during local training. This approach mitigated the impact of featuredrift and prevented local overfitting, resulting in improved alignment be-tween local updates and the global objective. The framework was evaluatedon heterogeneous datasets, demonstrating significant performance improve-ments over baseline FL methods, particularly under extreme non-IID con-ditions. FedImpress not only showcased the efficacy of leveraging syntheticdata for FL but also established a scalable, privacy-preserving paradigm forheterogeneous data settings.615.1. Summary5.1.2 Study 2: PromptFusionBuilding on the insights from FedImpress, the second study, PromptFu-sion, focused on mitigating catastrophic forgetting within a novel FL frame-work utilizing a pretrained Vision Transformer (ViT). The method capital-ized on the flexibility of prompt-tuning, adapting the Dual-Prompt archi-tecture to FL by introducing shared and group prompts. Shared prompts,attached to the earlier layers of the ViT, captured generalizable knowledgeacross clients, while group prompts, attached to deeper layers, encoded task-specific information unique to each client.PromptFusion integrated the concept of federated impressions to furtherenhance its effectiveness. Shared prompts were exclusively trained on feder-ated impressions, ensuring their alignment with the global data distributionand preventing overfitting to client-specific data. Group prompts, on theother hand, were trained using both local client data and federated impres-sions, enabling personalized adaptation while maintaining consistency withthe shared prompts.Through this design, PromptFusion effectively mitigated catastrophic for-getting by decoupling global and local knowledge representation. The sharedprompts ensured robust generalization across clients, while the group promptsallowed for task-specific adaptation without compromising global consis-tency. This dual mechanism demonstrated substantial improvements in FLperformance, particularly in scenarios involving significant data heterogene-ity and dynamic task requirements.5.1.3 Key ContributionsThe contributions of this thesis can be summarized as follows:• Developed FedImpress, a framework leveraging federated impressionsto align local training with global objectives, addressing data hetero-geneity in FL.• Introduced PromptFusion, a prompt-tuning-based FL framework thatintegrates federated impressions to mitigate catastrophic forgettingand improve task-specific adaptation.• Proposed a novel application of synthetic data generation in FL, en-suring privacy-preserving knowledge transfer and robust global modelperformance.625.2. Future Work• Demonstrated the efficacy of the proposed methods through extensiveexperimentation on benchmark datasets, showcasing their scalabilityand generalizability in real-world FL scenarios.By addressing fundamental challenges in FL through innovative designs,this thesis contributes to advancing the state-of-the-art in federated learn-ing, paving the way for more effective and privacy-conscious collaborativelearning systems.5.2 Future WorkWhile this thesis introduced significant advancements in Federated Learn-ing (FL) through the development of FedImpress and PromptFusion, thereremain several directions for future exploration to further enhance the ap-plicability, robustness, and generalizability of the proposed methods. Thesedirections are outlined below:5.2.1 Enhanced Synthetic Data GenerationThe federated impressions synthesized in FedImpress rely on an optimization-based approach to capture the global data distribution effectively. However,this process may be further improved by exploring more advanced generativemodels, such as diffusion models or adversarial training techniques, whichcould enhance the fidelity and diversity of the generated data. Addition-ally, leveraging domain-specific priors in the synthesis process may improveperformance in specialized applications, such as medical imaging.5.2.2 Scalability and Communication EfficiencyAs FL systems scale to include hundreds or thousands of clients, the com-putational and communication overhead of sharing synthetic datasets andmodel updates becomes significant. Future work could investigate lightweightversions of federated impressions, potentially using compressed representa-tions or model distillation techniques to reduce communication costs whilemaintaining performance. Similarly, exploring asynchronous or hierarchicalFL architectures could improve scalability in large-scale deployments.5.2.3 Dynamic Client ParticipationThe proposed frameworks were evaluated under the assumption of consistentclient participation across communication rounds. However, real-world FL635.2. Future Workscenarios often involve dynamic client availability and varying levels of dataquality. Adapting FedImpress and PromptFusion to handle such dynamicenvironments, potentially through adaptive learning rates or client selectionstrategies, would increase their robustness and practicality.5.2.4 Extension to Multimodal and Sequential TasksWhile both studies primarily focused on single-modality classification tasks,many real-world applications require multimodal learning or the ability tohandle sequential data. Extending the prompt-based framework of Prompt-Fusion to incorporate multimodal data, such as text and images, or to ad-dress sequential learning tasks, such as time-series forecasting, could signif-icantly broaden its applicability.5.2.5 Privacy and Security EnhancementsAlthough federated impressions provide a privacy-preserving alternative todirect data sharing, additional security measures could be explored to safe-guard against adversarial attacks and data leakage. Techniques such asdifferential privacy or secure multiparty computation could be integratedinto the synthesis and training processes to enhance privacy guarantees.5.2.6 Clinical Translation and Real-World ApplicationsThe promising results of the proposed methods in experimental settingswarrant further investigation into their deployment in real-world scenarios.This includes addressing regulatory challenges, ensuring compliance withdata-sharing regulations, and adapting the frameworks to meet the specificneeds of clinical practice. Collaborative studies with domain experts inhealthcare and other fields could help tailor these methods for impactfulapplications.5.2.7 Theoretical Analysis of Federated PromptsWhile PromptFusion demonstrated empirical success, a deeper theoreticalunderstanding of prompt-tuning in FL settings remains an open question.Future work could focus on establishing theoretical guarantees regardingconvergence, generalization, and stability when using prompt-based methodsin federated environments.645.3. Concluding Remarks5.2.8 Continuous and Lifelong Learning in FLPromptFusion introduced mechanisms to mitigate catastrophic forgetting;however, extending these approaches to continuous and lifelong learning set-tings in FL remains an exciting avenue for exploration. Future research couldinvestigate methods to dynamically adapt prompts as new clients and tasksare introduced, ensuring sustained performance without requiring retrainingon past data.5.2.9 Broader Benchmarking and EvaluationWhile the proposed methods were tested on well-established datasets, ex-panding the evaluation to include more diverse and challenging benchmarkscould provide deeper insights into their strengths and limitations. This in-cludes considering tasks with extreme data imbalance, rare-event detection,or highly unbalanced client participation.5.2.10 Open-Source and Community ContributionTo promote reproducibility and encourage community engagement, releasingopen-source implementations of FedImpress and PromptFusion would be anessential next step. This could enable researchers and practitioners to buildupon the proposed methods and explore their applications across a widerange of domains.By addressing these directions, future work can further advance the capabil-ities and applicability of federated learning systems, ensuring they remainrobust, scalable, and impactful in solving complex real-world challenges.5.3 Concluding RemarksThis thesis introduced two novel methods, FedImpress and PromptFusion,to address key challenges in Federated Learning (FL), including data hetero-geneity, catastrophic forgetting, and privacy preservation. These methodsprovide innovative approaches to enhance FL’s performance and applicabil-ity in diverse real-world scenarios.FedImpress leveraged synthetic datasets, termed federated impressions, tocapture the global data distribution in a privacy-preserving manner. Byincorporating these impressions during local training, it mitigated the effectsof non-IID data and local overfitting while preserving global knowledge.655.3. Concluding RemarksThe method demonstrated significant improvements in generalization acrossdiverse clients, especially in settings with label imbalance and domain shift.PromptFusion extended this work by introducing prompt-based learningwith pretrained Vision Transformers (ViTs). By separating shared promptsfor global knowledge and group prompts for client-specific adaptations, iteffectively mitigated catastrophic forgetting and leveraged the power of pre-trained representations. This approach showcased the potential of integrat-ing prompt tuning and federated learning for robust performance in hetero-geneous environments.Together, these contributions address fundamental FL challenges, offeringmethods that are both theoretically grounded and practically impactful.The evaluations on diverse datasets confirmed their effectiveness and adapt-ability, highlighting their potential for privacy-sensitive applications likehealthcare.This work lays the groundwork for future advancements in FL, includingimproving synthetic data generation, extending to multimodal tasks, andsupporting dynamic client participation. By addressing these directions,the proposed methods can evolve into even more robust and scalable FLsolutions.In conclusion, this thesis contributes to the advancement of FL by address-ing its most pressing challenges and demonstrating practical solutions thatensure both privacy and performance. These contributions pave the wayfor further innovation and broader adoption of FL across industries anddisciplines.66Bibliography[1] H Brendan McMahan, Eider Moore, Daniel Ramage, et al.Communication-efficient learning of deep networks from decentralizeddata. In Proceedings of the 20th International Conference on ArtificialIntelligence and Statistics (AISTATS), pages 1273–1282. PMLR, 2017.[2] Nicola Rieke, Jonny Hancox, Wenqi Li, et al. The future of digitalhealth with federated learning. NPJ Digital Medicine, 3(1):1–7, 2020.[3] Tian Li, Anit Kumar Sahu, Ameet Talwalkar, et al. Federated optimiza-tion in heterogeneous networks. In Proceedings of Machine Learningand Systems (MLSys), volume 2, pages 429–450, 2020.[4] Chaoyang He, Songze Li, Jinhyun So, et al. Fedml: A research li-brary and benchmark for federated machine learning. arXiv preprintarXiv:2007.13518, 2020.[5] Felix Sattler, Klaus-Robert Mu¨ller, and Wojciech Samek. Clusteredfederated learning: Model-agnostic distributed multitask optimizationunder privacy constraints. In Proceedings of the AAAI Conference onArtificial Intelligence (AAAI), 2020.[6] Stephen Boyd, Neal Parikh, Eric Chu, et al. Distributed optimizationand statistical learning via the alternating direction method of multi-pliers. Foundations and Trends® in Machine Learning, 2011.[7] Jakub Konecˇny´, Brendan McMahan, and Daniel Ramage. Federatedoptimization: Distributed optimization beyond the data center. arXivpreprint arXiv:1511.03575, 2015.[8] Tian Li, Anit Kumar Sahu, Ameet Talwalkar, et al. Federated learning:Challenges, methods, and future directions. IEEE Signal ProcessingMagazine, 37(3):50–60, 2020.67Bibliography[9] Rishi Bommasani, Drew A. Hudson, and Ehsan Adeli annd others. Onthe opportunities and risks of foundation models, 2022.[10] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, et al. Over-coming catastrophic forgetting in neural networks. Proceedings of theNational Academy of Sciences, 114(13):3521–3526, 2017.[11] Zhizhong Li and Derek Hoiem. Learning without forgetting.IEEE Transactions on Pattern Analysis and Machine Intelligence,40(12):2935–2947, 2017.[12] Sashank Reddi, Zachary Charles, Manzil Zaheer, et al. Adaptive feder-ated optimization. In Proceedings of the 9th International Conferenceon Learning Representations (ICLR), 2021.[13] Atrin Arya, Sana Ayromlou, Armin Saadat, Purang Abolmaesumi, andXiaoxiao Li. Federated impression for learning with distributed hetero-geneous data, 2024. arXiv preprint arXiv:2409.07351.[14] Sana Ayromlou, Purang Abolmaesumi, Teresa Tsang, et al. Classimpressions: Data-free incremental learning via data impressions. InProceedings of the 19th International Conference on Machine Learning(ICML), 2022.[15] Zhenheng Tang, Yonggang Zhang, Shaohuai Shi, et al. Virtual ho-mogeneity learning: Defending against data heterogeneity in federatedlearning, 2022.[16] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, et al. Dualprompt: Com-plementary prompting for rehearsal-free continual learning, 2022.[17] Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, et al. Learning to promptfor continual learning. In Proceedings of the IEEE/CVF Conferenceon Computer Vision and Pattern Recognition (CVPR), pages 151–161,2022.[18] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Be-longie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning,2022.[19] Alexey Dosovitskiy et al. An image is worth 16x16 words: Transformersfor image recognition at scale. In International Conference on LearningRepresentations (ICLR), 2021.68Bibliography[20] Jiancheng Yang, Rui Shi, Donglai Wei, et al. Medmnist v2 - a large-scalelightweight benchmark for 2d and 3d biomedical image classification.Scientific Data, 10(1):41, 2023.[21] Jayanthi Sivaswamy, S. R. Krishnadas, Gopal Datt Joshi, et al. Drishti-gs: Retinal image dataset for optic nerve head (onh) segmentation.In Proceedings of the IEEE International Symposium on BiomedicalImaging (ISBI), 2014.[22] Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, et al. Federatedmulti-task learning. Advances in Neural Information Processing Sys-tems (NeurIPS), 30:4424–4434, 2017.[23] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, et al. Scaffold:Stochastic controlled averaging for federated learning. In Proceedingsof the 37th International Conference on Machine Learning, 2020.[24] Tian Li, Maziar Sanjabi, Ahmad Beirami, et al. Fair resource allocationin federated learning. In Advances in Neural Information ProcessingSystems, pages 5691–5701, 2020.[25] Geoffrey E. Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowl-edge in a neural network. arXiv preprint arXiv:1503.02531, 2015.[26] Jose´ Ignacio Orlando, Huazhu Fu, Joa˜o Barbosa Breda, et al. Refugechallenge: A unified framework for evaluating glaucoma detection al-gorithms. In Proceedings of the IEEE/CVF Conference on ComputerVision and Pattern Recognition (CVPR), 2020.[27] Andres Diaz-Pinto, Sandra Morales, Valery Naranjo, et al. Cnns forautomatic glaucoma assessment using fundus images: an extensive val-idation. BioMedical Engineering OnLine, 18(1):29, 2019.[28] Khurram Javed and Martha White. Meta-learning representations forcontinual learning. In Advances in Neural Information Processing Sys-tems (NeurIPS), volume 32, 2019.[29] Prannay Khosla, Piotr Teterwak, Chen Wang, et al. Supervised con-trastive learning. In Advances in Neural Information Processing Sys-tems, pages 18661–18673, 2020.[30] Tero Karras, Samuli Laine, and Timo Aila. A style-based generatorarchitecture for generative adversarial networks. In Proceedings of the69BibliographyIEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), pages 4401–4410, 2019.[31] Shiqiang Wang, Tiffany Tuor, Theodoros Salonidis, et al. Adaptive fed-erated learning in resource-constrained edge computing systems. IEEETransactions on Parallel and Distributed Systems, 33(3):568–584, 2022.[32] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring theeffects of non-identical data distribution for federated visual classifica-tion, 2019.[33] Xiao Liu, Kaixuan Ji, Yicheng Fu, et al. P-tuning v2: Prompt tuningcan be comparable to fine-tuning universally across scales and tasks.arXiv preprint arXiv:2110.07602, 2021.[34] Michael McCloskey and Neal J. Cohen. Catastrophic interference inconnectionist networks: The sequential learning problem. Psychologyof Learning and Motivation, 24:109–165, 1989.[35] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, et al. Gradient surgeryfor multi-task learning. In Advances in Neural Information ProcessingSystems, pages 5825–5837, 2020.[36] David Lopez-Paz and Marc’Aurelio Ranzato. Gradient episodic mem-ory for continual learning. In Advances in Neural Information Process-ing Systems (NeurIPS), volume 30, pages 6470–6479, 2017.[37] Seyed Iman Mirzadeh, Arslan Chaudhry, Dong Yin, et al. Architec-ture matters in continual learning. In Advances in Neural InformationProcessing Systems (NeurIPS), volume 33, 2020.[38] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learn-ing through synaptic intelligence. In Proceedings of the InternationalConference on Machine Learning (ICML), pages 3987–3995, 2017.[39] Raia Hadsell, Dushyant Rao, Andrei A Rusu, et al. Embracing change:Continual learning in deep neural networks. Trends in Cognitive Sci-ences, 24(12):1024–1035, 2020.[40] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, et al.icarl: Incremental classifier and representation learning. In Proceedingsof the IEEE Conference on Computer Vision and Pattern Recognition(CVPR), pages 2001–2010, 2017.70Bibliography[41] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, et al. Pro-gressive neural networks. In Advances in Neural Information ProcessingSystems (NeurIPS), pages 5998–6008, 2016.[42] Zheda Mai, Ruixuan Li, Jinwoo Jeong, et al. Online continual learn-ing in image classification: An empirical survey. arXiv preprintarXiv:2101.10423, 2021.[43] Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federatedlearning. In Proceedings of the IEEE/CVF Conference on ComputerVision and Pattern Recognition (CVPR), pages 10713–10722, 2021.[44] Chencheng Xu, Zhiwei Hong, Minlie Huang, et al. Acceleration offederated learning with alleviated forgetting in local training, 2022.[45] Gihun Lee, Minchan Jeong, Yongjin Shin, Sangmin Bae, and Se-YoungYun. Preservation of the global knowledge by not-true distillation infederated learning, 2022.[46] Kangyang Luo, Xiang Li, Yunshi Lan, and Ming Gao. Gradma: Agradient-memory-based accelerated federated learning with alleviatedcatastrophic forgetting, 2023.[47] Tingting Zhao, Zifeng Wang, Aria Masoomi, et al. Deep bayesian un-supervised lifelong learning. Neural Networks, 149:95–106, 2022.[48] Jaehong Yoon, Eunho Yang, Jaekyun Lee, et al. Lifelong learningwith dynamically expandable networks. In International Conferenceon Learning Representations (ICLR), 2018.[49] Tom B. Brown, Benjamin Mann, Nick Ryder, et al. Language mod-els are few-shot learners. Advances in Neural Information ProcessingSystems (NeurIPS), 33, 2020.[50] Edward J. Hu, Yelong Shen, Phillip Wallis, et al. Lora: Low-rankadaptation of large language models. arXiv preprint arXiv:2106.09685,2021.[51] Wei Zhu and Jiebo Luo. Federated medical image analysis with virtualsample synthesis. In Medical Image Computing and Computer AssistedIntervention – MICCAI 2022, pages 728–738, Cham, 2022. SpringerNature Switzerland.71Bibliography[52] H Yao, Yuhong Guo, and Chunsheng Yang. Source-free unsuperviseddomain adaptation with surrogate data generation. In Proceedings ofNeurIPS 2021 Workshop on Distribution Shifts: Connecting Methodsand Applications, 2021.[53] Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, et al. Federated learningwith matched averaging. arXiv preprint arXiv:2002.06440, 2020.[54] Francisco Jose´ Fumero Batista, Tinguaro Diaz-Aleman, Jose Sigut,et al. Rim-one dl: A unified retinal image database for assess-ing glaucoma using deep learning. Image Analysis and Stereology,39(3):161–167, Nov. 2020.[55] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of fea-tures from tiny images. Technical report, University of Toronto, 2009.[56] Nadav Shoham, Simon Wiesler, and Lukasz Kaiser. Overcoming for-getting in federated learning on non-iid data. In Proceedings of theWorkshop on Federated Learning for Data Privacy and Confidentiality,2019.72
Thesis/Dissertation
eng
Electrical and Computer Engineering
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Abolmaesumi, Purang
Li, Xiaoxiao
Overcoming catastrophic forgetting in heterogeneous federated learning
Text