Applied Science, Faculty of
Non UBC
Electrical and Computer Engineering, Department of
DSpace
Electronics 14 (7): 1419 (2025)
Mukherjee, Debasmita
Najjaran, Homayoun
Cohesive human–robot collaboration can be achieved through seamless communication between human and robot partners. We posit that the design aspects of human–robot communication (HRCom) can take inspiration from human communication to create more intuitive systems. A key component of HRCom systems is perception models developed using machine learning. Being data-driven, these models suffer from the dearth of comprehensive, labelled datasets while models trained on standard, publicly available datasets do not generalize well to application-specific scenarios. Complex interactions and real-world variability lead to shifts in data that require domain adaptation by the models. Existing domain adaptation techniques do not account for incommensurable modes of communication between humans and robot perception systems. Taking into account these challenges, a novel framework is presented that leverages existing domain adaptation techniques off-the-shelf and uses statistical measures to start and stop the training of models when they encounter domain-shifted data. Statistically informed multimodal (domain adaptation by transfer) learning (SIMLea) takes inspiration from human communication to use human feedback to auto-label for iterative domain adaptation. The framework can handle incommensurable multimodal inputs, is mode and model agnostic, and allows statistically informed extension of datasets, leading to more intuitive and naturalistic HRCom systems.
Academic Editors: Kostas Karpouzis,Bipin Indurkhya, Eleuda Nuñez,Marie-Monique Schaper and CecilioAnguloReceived: 14 February 2025Revised: 22 March 2025Accepted: 28 March 2025Published: 31 March 2025Citation: Mukherjee, D.; Najjaran, H.Statistically Informed Multimodal(Domain Adaptation by Transfer)Learning Framework: A DomainAdaptation Use-Case for IndustrialHuman–Robot Communication.Electronics 2025, 14, 1419.https://doi.org/10.3390/electronics14071419Copyright: © 2025 by the authors.Licensee MDPI, Basel, Switzerland.This article is an open access articledistributed under the terms andconditions of the Creative CommonsAttribution (CC BY) license(https://creativecommons.org/licenses/by/4.0/).ArticleStatistically Informed Multimodal (Domain Adaptation byTransfer) Learning Framework: A Domain Adaptation Use-Casefor Industrial Human–Robot CommunicationDebasmita Mukherjee 1,2,3,† and Homayoun Najjaran 2,*1 Department of Electrical Computer Engineering and J. Herbert Smith Centre for Technology ManagementEntrepreneurship, University of New Brunswick, Fredericton, NB E3B 5A3, Canada;debasmita.mukherjee@unb.ca2 Department of Electrical and Computer Engineering, University of Victoria, Victoria, BC V8P 5C2, Canada3 Department of Mechanical Engineering, The University of British Columbia, Kelowna, BC V1V 1V7, Canada* Correspondence: najjaran@uvic.ca† Current address: University of New Brunswick, 3 Bailey Dr, Fredericton, NB E3B 5A3, Canada.Abstract: Cohesive human–robot collaboration can be achieved through seamless communi-cation between human and robot partners. We posit that the design aspects of human–robotcommunication (HRCom) can take inspiration from human communication to create moreintuitive systems. A key component of HRCom systems is perception models developedusing machine learning. Being data-driven, these models suffer from the dearth of compre-hensive, labelled datasets while models trained on standard, publicly available datasets donot generalize well to application-specific scenarios. Complex interactions and real-worldvariability lead to shifts in data that require domain adaptation by the models. Existingdomain adaptation techniques do not account for incommensurable modes of communica-tion between humans and robot perception systems. Taking into account these challenges,a novel framework is presented that leverages existing domain adaptation techniquesoff-the-shelf and uses statistical measures to start and stop the training of models whenthey encounter domain-shifted data. Statistically informed multimodal (domain adapta-tion by transfer) learning (SIMLea) takes inspiration from human communication to usehuman feedback to auto-label for iterative domain adaptation. The framework can handleincommensurable multimodal inputs, is mode and model agnostic, and allows statisticallyinformed extension of datasets, leading to more intuitive and naturalistic HRCom systems.Keywords: human–robot communication; iterative domain adaptation; personalizedmachine learning; feedback enabled labelling; machine perception; model personalization1. IntroductionWhile automation led to the development of robots (and other embodied computersystems) that perform repetitive, non-ergonomic, and dangerous jobs in the industrialsector [1], human operators are still unparalleled in carrying out precise tasks that requiredexterity and flexibility. The complementary nature of human and robot agents’ capabil-ities forms the basis for the design of a shared workspace for the two types of agents tocollaborate. The endgame of human–robot communication (HRCom) is to achieve theseamless team dynamics and communication fluency of an all-human team captured inthe eponymously named sub-field of human–robot collaboration (HRC) [2,3]. Our previ-ous work in the critique of HRCom compared to natural human communication throughthe lens of complexity theory [4] argued that the former has a deficit of two importantElectronics 2025, 14, 1419 https://doi.org/10.3390/electronics14071419Electronics 2025, 14, 1419 2 of 26characteristics: feedback and multiplicity. Multiplicity refers to the variety in cultures,languages, and traditions that lead to large variability in the communication behaviourof humans, while feedback is the circularity of actions in communication that are used toregulate the actions of the agents. Robot perception systems developed using supervisedlearning (SL) and deep convolutional neural networks (CNNs) must be able to handle suchvariabilities in human communication data but fail to do so due to shortcomings in thetraining and development of SL-based models. This paper aims to mitigate some of thosechallenges through the design of a framework that is designed to take inspiration fromhuman communication characteristics like multiplicity and feedback.Challenges to SL in the context of Human–Robot Communication: Supervisedlearning models are predominantly trained under the assumption that they will be op-erating in a closed-world scenario, i.e., the test data it encounters will be drawn fromthe same distribution as the data on which it has been trained [3]. This is referred to asthe data being in-distribution (ID), but real-world applications with the dynamics anduncertainties attached to the continually changing environment seldom generate ID data.Instead, test data are often sourced from a different distribution than that of the source,i.e., from a domain-shifted distribution or out-of-distribution (OOD). This open-worldapproach [5] typically leads to poor performance of the model, even though the latter maybe well-trained on ID data. The distributional shifts may be caused due to differences inclasses. In other words, the perception system encounters an instance or an object thatit has not been trained for (semantic shift) or of instances of the same classes as trainingbut in a change in the distribution (covariate shift). This work only deals with covariateshift, hereon referred to as ‘shift’. Covariate-shifted data lead to degraded predictionaccuracies. Updating models with shifted data is faced with the additional challenge that,as data-driven methods, deep CNNs require large amounts of labelled data. Thus, labellingnew shifted data every time there is any change in the perception scene is a very time- andresource-consuming endeavour.In order to alleviate the challenge posed by shifted data and the high cost of gatheringand labelling data for re-training for robot perception models, the aim of this work isto develop a novel framework leveraging existing machine-learning domain adaptationtechniques for ensuring that the robot perception model performs as well in the real-worldas it does in the hermetic lab conditions in which it has been developed. Particularly,the framework tries to enable higher specificity of the model towards data from the hu-man(s) interacting with it within that application. It takes inspiration from natural humancommunication and uses real-time human feedback to auto-label shifted data in orderto use computational resources only when needed. The setting in the presented work isprimarily designed for “industrial” systems since, in industrial settings, especially in thecontext of communication, short commands would be common and nuances of emotionsin communication signals would not have to be dealt with.The paper is organised as follows: Section 2 contains a discussion on related works,followed by the methodology in Section 3, which presents the contributions, assumptions,design considerations of the framework, and then the framework. Section 4 presents theapplication of the framework to a communication task (hand gestures), Section 5 containsseveral studies carried out in the context of this task, and Section 6 ends the paper withconclusions and discussions on future works.2. Background and Related WorksThe dearth of labelled datasets for training has been effectively tackled by usingtransfer learning (TL), wherein a pre-trained model is used as a starting point to train onthe much smaller labelled dataset. This is an extremely common step in deep learning,Electronics 2025, 14, 1419 3 of 26wherein the idea is to re-use the model that has already learned general features like edges,shapes, and colours in data for improved computational efficiency. ImageNet [6], with1000 object classes, 1,281,167 training images, 50,000 validation images, and 100,000 testimages, is a typical dataset used for this purpose.Currently, multimodal inputs are detected and classified by machine-learning (ML)models by learning a generic feature representation using a large dataset such as Imagenetor MS COCO [7] with over a million labelled images and then fine-tuning based on therequired specific application. This pipeline is followed for almost all aspects of SL, such assemantic segmentation, object detection, and pose estimation. Neural network architectureshave been developed to leverage these features and achieve high accuracies on ID data.Domain adaptation (DA) is a class of techniques that aims to align features betweenthe labelled source domain and the unlabelled target domains so as to increase the per-formance of the model in both domains. Essentially, DA is a special case of TL. In caseswhere the target domain is fully annotated, it is called supervised DA. In [8], researchersintroduced classification and contrastive semantic alignment loss to guide samples fromdifferent domains, but the same classes to map to the feature space created from inputdata. In unsupervised DA (UDA), labels from the target domain are completely missing.Ref. [9] developed a methodology based on paired RGB images in both source as well astarget domains. In another study, the authors proposed a novel residual transfer networkto simultaneously learn adaptive classifiers and transferrable features [10]. The assumptionwas that the source and target classifiers differed by a small residual function.One class of DA technique exists, called iterative algorithms, that aim to “auto-label”target domains iteratively. Co-training for domain adaptation (CODA) was presented in [11].The methodology involved slowly adapting the training set from the source to the targetdomain by the inclusion of target instances with the highest confidence scores. Co-training [12],as an example of unsupervised DA, assumes the existence of two separate views of a singleobject and two trained classifiers, respectively. At each iteration, only inputs that possessconfidence scores above the threshold from only one of the two classifiers are moved to thetraining set; thus, one classifier provides labels while the other provides the data. CODAperformed the same by training two classifiers on different sets of features of the same view.From the discussion above, it is to be noted that if “paired” sources of data existor multiple views of the same object or scene can be utilized, then off-the-shelf domainadaptation methodologies may be used for enriching the model with shifted data. Thereinlies the challenge since, for HRCom, such extensive and synchronized labelled data is notavailable. Additionally, multimodal data can also often be incommensurable, such as inthe case of facial expressions and voice commands or hand gestures and voice commands,and thus are challenging to use as synchronised datasets for model training. In such cases,an off-the-shelf application of domain adaptation techniques may not be feasible. Thiswork focuses on the cases where incommensurable modes of communication are presentand standard DA algorithms cannot be used for updating the model.3. MethodologyIn order to carry out domain adaptation, the framework must detect domain-shifteddata. This shifted or out-of-distribution (OOD) data are unlabelled. SoftMax probabilitiesare used to evaluate model performance where labelled test data are available. In the caseof covariate-shifted data, the literature [13] indicates that the use of the SoftMax function inneural network classifiers leads to the classification of OODs with high confidence. Therefore,statistical measures for OOD detection in conjunction with heuristic decisions must be usedto indicate OOD [14]. This analysis has been used in our framework to answer the designrequirements of when to start and stop training the model on OOD data. A similarity (S)Electronics 2025, 14, 1419 4 of 26metric has been used as a check for calculating and visualizing the covariate domain shift thatoccurs when the model encounters new (user) data. This is elaborated upon in Section 3.4.The shortcomings of supervised learning (SL) for HRCom include, inter alia, the lackof large, labelled datasets, and if labelled datasets are available, for domain shifts, pairedor otherwise multi-view and labelled datasets are challenging to obtain. Thus, unlabelleddata of the new domain must be used in conjunction with labelled ones to adapt to thenew domain. As the human or the work-cell environment changes, the composition ofthe scene changes, leading to a shift in domain. In order to prevent data drift and in turncalibration drift from the dynamic environment, the ML model must learn continuously [15],thus requiring the framework to be iterative in nature. The cost of labelling and trainingprohibits a constant state of online learning, something that generalization and adaptationtechniques do not take into consideration. In technical considerations, practitioners muststop the model from learning a dataset after it reaches an appropriate minimum, else theerror begins to rise again. Thus, the adaptation framework must track when to start andstop training the model. Our previous work [16] presented iterative transfer learning andfeedback generated labels using PF-HRCom, but verification of learning was conductedin a supervised manner using labelled test data. That is not always available, and thus anunsupervised formulation must be used to verify that the model has learnt the shifted data.These were the design considerations that went towards the development of the novelframework presented in this paper.3.1. ContributionsThe literature has described human–robot communication (HRCom) that can beachieved through verbal speech, signals through gesture, haptic communication, andthrough screen-based interfaces [2,17,18]. According to a scoping review conducted by [17],operators in industry establish workplace routines for communication, which need to beupdated with the introduction of robots. This establishes routines for HRCom through theuse of robot perception systems. In a setup for HRCom, a covariate shift in data distributionmay arise due to a change in any component of the scene (human or environment) or changein the sensor capturing the information, leading to a statistical shift in the distribution.Instead of rejecting such shifted observations, the communication model must be updatedwith the typical behaviours of the human(s) it encounters. In other words, much like theadjustment behaviours in an all-human team, the models must be adapted or “personalized”to the behaviour of the human partner for more humanistic design of industrial systems.With these considerations in mind, and also the shortcomings of current DA algo-rithms, the paper proposes a novel framework for human–robot communication, whichis a use-case of domain adaptation: SIMLea—statistically informed multimodal (domainadaptation by transfer) learning. The following are the contributions of the same.1. SIMLea leverages publicly available, large datasets of multimodal data and high-accuracy models trained on them to provide feedback-driven labelling of shifted data.This feature leverages real user feedback to auto-label user data during operation,mimicking a key component of natural human communication, wherein feedbackis used to continuously update the knowledge associated with the shared task andits status. This would reduce mislabelling. This allows the extension of the model’slearnt feature representations with those of new, covariate domain-shifted data andpersonalization of the model to the collaborating human, making communicationmore natural for the human.2. SIMLea is designed to allow feedback to be used from incommensurate modesof communication.Electronics 2025, 14, 1419 5 of 263. Statistical measure of similarity is used to drive the adaptation framework to learnonly relevant data and start and stop the computationally heavy learning processwhen required. Implementation of this aspect is presented in Section 5.2.4. SIMLea allows the amalgamation of empirical thresholds and standards and application-specific heuristics. It is an add-on to the ML model and does not require change inits architecture.This work distinguishes itself, not by introducing a new domain adaptation (DA)technique but by presenting a novel framework that strategically assembles existing DAand similarity measurement methods. This framework enables the effective application ofthese techniques to human–robot collaboration (HRC) scenarios, which inherently presentcomplexities beyond traditional DA tasks. By leveraging a feedback-driven auto-labellingapproach, demonstrated through hand gesture recognition, the framework unlocks thepotential for adaptation across diverse human communication datasets.3.2. Assumptions of SIMLeaThe assumptions taken under which the framework has been developed are as follows.These have been adapted from assumptions in [19] for HRC scenarios.1. As mentioned previously, the system only encounters covariate-shifted data. Thismeans that all commands from the human are “correct” in the lexical sense and willfall into one of the pre-trained classes.2. The deployed base model (model trained on standard, publicly available dataset) iswell-trained on the standard dataset, i.e., it performs well on the test set of that dataset.This assumption is extrapolated to the iteratively generated models as well since theshifted data are from the same application and are similar to the original data, ipsofacto, the training parameters could be re-used for TL without change. By well-trained,one refers to the models possessing the required trade-off between bias and variance,determined based on tribal knowledge of the application.3. The ML classifier that understands (recognizes) feedback from the human has highvariance and high accuracy.Table 1 presents a qualitative comparison between some existing domain adaptationtechniques and the proposed framework, wherein the proposed novel framework canhandle incommensurate modes of data for labelling unlabelled data and use unsupervisedstatistical measures to start and stop training. The assumptions of the techniques refer toassumptions in their sources of data, models, and existence of labels for training.Table 1. Qualitative comparison of domain adaptation techniques with proposed domain adapta-tion framework.Domain Adaptation Technique Methodology AssumptionUnsupervised Domain Adaptation (UDA) [9] Based on paired RGB images in both source aswell as target domainsAssumes existence of paired data in same for-mat in both source and target domainsResidual Transfer Network [10] Based on simultaneously learning adaptiveclassifiers and transferrable featuresAssumes that the source and target classifiersonly differ by a small residual functionCo-Training for Domain Adaptation [11,12]Based on slowly adapting the trainingset from the source to the target domainby inclusion of target instances with thehighest confidence scoresAssumes the existence of two separateviews of a single object and two trainedclassifiers respectivelyPersonalization through feedback-enabled hu-man–robot communication (PF-HRCom) [16]Based on iterative transfer learning andfeedback generated labels, verification oflearning conducted in a supervised mannerusing labelled test data Can be used forincommensurables modesAssumes existence of labelled test data andtrained model for generating feedbackElectronics 2025, 14, 1419 6 of 26Table 1. Cont.Domain Adaptation Technique Methodology AssumptionProposed work: Statistically Informed Multi-modal (Domain Adaptation by Transfer) Learn-ing (SIMLea)Based on iterative transfer learning, feed-back enabled labelling of unlabelled data, sta-tistical similarity measure to start and stoplearning processAssumes trained model for generating feed-back3.3. Metric for an Informed Approach to Domain AdaptationIn the absence of labelled datasets for testing, unsupervised formulations for quan-tifying domain shift may be used for an informed approach to DA [20–22]. CNNs learndata through the creation of feature representation spaces specific to the datasets. Thus,using statistical distance in the analysis of this space instead of the data space may bemore responsive to better understanding the shift in the distribution. This distance canbe visualized as a similarity measure between the datasets. Small visual domain shiftsbecome more pronounced as the data passes through the trained model. This is becausedeep-learning architectures are highly parameterized and can learn the finer details of thedata, thereby losing variance in favour of being biased towards the training data. As thedata travels deeper into the model, the representation space becomes more and more ab-stract. Finally, the fully connected layers use these representations to assign a class duringa classification task. The vast collection of dependent weights and biases renders the modelhighly sensitive to small variations in the visual data. Thus, perceptually small (to thehuman) perturbations in data may lead to large statistical domain shifts in the model’sunderstanding. SIMLea leverages this behaviour of CNNs to use the similarity metricbetween original and shifted data to start and stop training.3.4. Covariate Domain Shift as a Measure of SimilarityA statistical distance metric between the original training set (or in-distribution (ID))or “source” and the new data or “target” can be used to determine the “similarity” (S)between the source and target domains. A universal definition of similarity to a givendataset may be a non-sequitur because the representation spaces used to compare datasetsare highly dependent on the model’s architecture, training, and hyperparameter values.Thus, “similarity” must be empirically defined as a consequence of the application andthe requirements. The reader is directed towards some literature on the various types ofmetrics developed: [3,13,23–30].Conservative approaches must have a lower range of features it considers similar,while less conservative ones may contain higher variance. The application-specific thresh-olds have been defined in Section 4, where validation tests on HG data are discussed.The contributions of the paper lie in the novel framework for domain adaptation and notin the formulation of the specific metric for determining S.3.5. Factor for Cessation of Domain AdaptationThe larger the value of S, the higher the similarity between the datasets. This measureis used to determine if the adapted and newly generated model after an iteration of SIMLeahas learnt the features of the new dataset. If it is found during an iteration that the new S isstill small; new data would be collected to proceed towards another iteration of training. If Sis increased to now lie within the threshold that one can confidently say that the model haslearnt the new features, then the iterative training can be ceased, and the communicationsystem would begin using the newly generated model for operation.Use of an unsupervised statistical distance between the representation spaces ofthe source and target distributions to determine S would allow the framework to beElectronics 2025, 14, 1419 7 of 26model agnostic. This is because it would be an add-on component to the model withoutrequiring any change in the architecture or hyperparameters. This S can then be used formonitoring the execution of the framework and the performance of the model. The decisionto employ this monitoring analysis continually or after certain intervals is at the discretionof the engineer.3.6. Criterion for Feasibility of Transfer LearningThe criterion for feasibility of transfer learning (TL) will depend on the applicationparameters. The threshold could be determined using the size of dataset used, numberof training epochs required, or time of training in order to reach a particular performancemetric. These are typically empirically determined. For the purpose of this paper, wehave used number of epochs to determine the feasibility of transfer learning. Given that,for comparison, we kept dataset sizes the same for re-training, but the sizes of imagesand resolution of the datasets could be different, we elected to use number of epochsfor the same increase in accuracy as comparison. How many epochs and how muchincrease in accuracy are heuristic measures that must be determined by the engineer basedon application requirements and constraints? Section 4 demonstrates on a hand gesturerecognition task.3.7. SIMLea FrameworkThe proposed novel framework for personalization through domain adaptation inHRCom consists of the following steps.1. Train an ML model with a standard, publicly-available dataset such that it performswell on ID or source data: base model (model0).2. Collect the new user data and store in memory. These data are unlabelled sincethey are generated in real-time by the user interacting with the robot. Calculate thesimilarity score (S) between the trained (in-distribution) data and the incoming user(covariate-shifted) data.3. If the similarity score is higher than a pre-set threshold (that is application and modelspecific), then the model can be effectively deployed and used for inferencing with thenew data. If, on the other hand, the data are dissimilar, then transfer learning may beused if deemed feasible.4. The next step is a checkpoint to test if transfer learning would be effective for domainadaptation. The similarity score obtained in step 2 is used again. If the covariate-shifted dataset is similar to the in-distribution data, then iterative re-training can beused. If the datasets are very dissimilar, then instead of transfer learning the modelshould be trained from scratch and the new model deployed. Such a case may indicatea significant shift in the domain data, and either the new data needs to be rejected orthe model needs to be changed.5. In the case of re-training, endeavouring to keep dataset size low, a small numberof empirically determined instances per class of the covariate-shifted data are thenauto-labelled by the robot asking for multimodal (in this case, verbal) feedback fromthe user, as per the work in [16].6. The model is iteratively re-trained with the new batch of covariate-shifted user imagesmixed with a larger batch of the source dataset or in-distribution data, as per the workin [16].7. Once the new model is generated, the representation space of this model is used togauge the similarity of the two datasets, and the loop is re-traced. This ensures thata minimal amount of user data is required for effective learning of the model and isauto-labelled on a need-only basis.Electronics 2025, 14, 1419 8 of 26The framework ensures that the model stops learning iteratively and consumingresources once it has:• Case 1: either learnt the features of the target dataset well.• Case 2: or it can no longer benefit from transfer learning because the datasets aretoo dissimilar.Larger batches of data may also be used and would be beneficial for faster DA, and anysimilarity index may be used, thus making the framework quite flexible for deployment.In order to save resources, and potentially down-time, while the robot perception systemanalyses new data, the option for similarity analysis can be triggered manually whena new user is added or the work-cell changes or the sensor changes instead of it beingautomatic for all incoming data. As the model is deployed for long periods of time andincreases its variance (with relevant data), the engineer can choose to trigger the analysisonly periodically, electing to let some changes not lead to new re-trainings.Furthermore, while human-generated datasets are the primary focus of this work,the framework can be used in continuous learning in other applications as long as afeedback-giving agent exists that can be used for auto-labelling. Validation tests have beencarried out using hand gesture datasets and are examined in the next section. Figure 1presents the consolidated framework of SIMLea.Figure 1. The iterative framework for SIMLea: Statistically informed multimodal (domain adaptationby transfer) learning involving the collection of user data, finding similarity with trained data,determining the effectiveness of transfer learning for domain adaptation, and training the modelbased on the decisions of the framework.Electronics 2025, 14, 1419 9 of 264. Validation of SIMLea with Hand Gesture Recognition DataFollowing the literature reviews cited [2,17], hand gestures (HGs) are used as a meansof communication between humans and robots since hand gestures leave human workersfree to go about their tasks without any cumbersome headsets or wires and hand gesturesare an intuitive means of communication.4.1. Variance in HG DatasetsFor the presented work, an HG classifier using VGG16 [31] was trained witha publicly available hand gesture dataset—Image database for tiny hand gesturerecognition [32]—hereon referred to as TinyHand or TH. The dataset used is composedof seven hand gestures carried out by 40 participants (male and female) with their righthands: fist, l, ok, palm, pointer, thumb down, and thumb up. RGB videos were collected, with1400 colour frames for each gesture. These hand gestures may be considered quite typicalfor communicating with a robot or other intelligent system. Variability has been addedby the creators of the dataset by incorporating gestures performed in different locationsin the images as well as by having the participants positioned with a number of differentbackgrounds, both cluttered as well as clean, solid coloured, and with variations in illu-mination. The human face and body occupy the majority of the image, while the handoccupies only about 10% of the pixels of the image. In a dynamic industrial environment,the human may not always be positioned in a central location or have the hands close tothe camera; thus, the dataset was selected to mimic these inconsistencies in sensor data.Out of the 40 individuals, 25 had a clean, uncluttered background and the humans in aseated position. The TinyHand Uncluttered Dataset or THUC, is used to refer to this subsetof TH. The remaining had different cluttered backgrounds and the humans standing, herereferred to as THC (TinyHand Cluttered). For the tests carried out for SIMLea, THUC wasused with gestures from 23 participants split into training and validation sets in the ratioof (70:30)%, and the remaining two were used as test sets. The training and validation setfinally contained 62,035 images and the test set 4719.The training parameters used were 0.01 learning rate, 16 batch size, and stochasticgradient descent optimizer run till 13 epochs. The data augmentations carried out onthe THUC training set were lateral shifts so as to not overfit the model to expecting thehuman to be at the centre of the scene, shears so that the fish-eye effect of certain camerascould be trained for, zoom to mimic the varying distance of the human from the camerain a dynamic environment, and small rotation and brightness shifts to allow for dynamiclighting conditions. The trained VGG16 model (model0) was then tested on the THUC testset and had an accuracy of 0.96.In order to simulate covariate domain shifts, three user datasets were created. Theseconsisted of the author carrying out the seven gestures. In the first instance, the human(the author) of the scene is changed without any change in the environment. To clarify,the environment was uncluttered and a solid shade of gray similar to that of the sourcetraining set. In this dataset, the user gestured in a seated position and gestured with theirright hand. This dataset is denoted by User Dataset 1 or UDS1. To simulate a change inboth components of the scene, the User Dataset 2 (UDS2) consisted of the user (the author)seated in front of a cluttered background that is different from the background of THUCand also THC. Finally, to add more variations, User Dataset 3 (UDS3) had the user in astanding position in front of a cluttered background. Finally, to demonstrate a situationwhere the application of TL is not useful, a VEnv dataset (VDS) was also created using askin-tone-coloured humanoid carrying out all the gestures in the above-mentioned classes,except pointer, since the “pointing” gesture is usually an in-built option in VR environments,Electronics 2025, 14, 1419 10 of 26and for the VR application, the default one was used. Figure 2 provides a sample of imagesfrom all the datasets grouped by image classes.Apart from the difference in the human or humanoid, the environment of the scene,and the variations in the “correct” gestures themselves (evident through the variety ofdeployment of the joints of the arm, their angles with respect to the body and the camera),the difference in the sensor also leads to a covariate domain shift. Notice that the resolutionsof THUC, UDS1 and 2, and UDS3 are different, although when passed through the modelthey would all be scaled to fit (224, 224, 3), where 3 indicates the three RGB channels.The VDS images are captured through a virtual sensor and thus are more agreeable tocustomized specifications that could be made closer to the original or source dataset.Figure 2. Sample images and classes of hand gesture classification datasets: TinyHand UnclutteredDataset (THUC), User Seated, Uncluttered (UDS1), User Seated, Cluttered (UDS2), User Standing,Cluttered (UDS3), and Virtual (VDS), showing visual differences in the datasets: (a) fist, (b) l, (c) ok,(d) palm, (e) pointer, (f) thumb up, (g) thumb down.The rationale for not including left-handed gestures is that some gestures, such as ok,fist, palm, remain semantically invariant, even when carried out by the left hand. Others,such as l, carried out in the same way as the right-handed one (i.e., index fingers pointingupwards, thumbs pointing towards the body, and the remaining fingers curled downwardsand facing outwards) do not semantically mean l any more. Under natural circumstances,the human would not use the same way to indicate such laterally variant gestures. Thus,in order to retain focus solely on covariate shift for this study, only right-handed gestureswere considered. Even when lateral inversion is used as a data augmentation tool inSection 5.3, this behaviour of the l class of gestures is ignored and semantic shifting is leftto be delved into in future work.4.2. Covariate Domain Shifts in User DataThe model naming conventions are as follows:Electronics 2025, 14, 1419 11 of 26• model0 or base model is trained using standard dataset THUC in all tests, except inSection 5.3, where both THUC and THC are used.• Iterative re-training of model0 on user-labelled data mixed with a portion of standarddata generates modeln, where n refers to the number of iterations.model0 is used to determine the domain shifts in user data. Intuitively through (human)visual inspection, the degree of similarity to THUC would be highest for UDS1 followed byUDS2, UDS3, and finally VDS. This hypothesis should also be validated through the use ofthe covariate shift metric.Shift Analysis with THUC Test Set as SourceStacke et al. proposed an unsupervised formulation called representation shift (RShift)to measure the statistical distance between the feature representation spaces of source andtarget domains [19]. The shift metric (R) is calculated using KL divergence between theoutputs of max pooling layers averaged over all the filters of that particular layer for eachdataset. The outputs of convolution layers are not used due to the high dimensionalityof the data, which would make analyses extremely computationally heavy and wouldin turn render the model impractical for operation. For zero values in the distributions,a small constant (0.001) was added for numerical stability. A caveat to be remembered isthat the discrepancy as calculated through RShift is the model’s understanding and not theobjective or the human’s perspective of the datasets.The formulation for RShift, given as follows, has been reproduced from [19]. Assumethe layers of a CNN are denoted by{l1, . . . lL}Let the layer activations of a trained model for layer l and filter k produced by an inputx be denoted by:Φl(x) = {Φl1(x), . . . ,Φlk(x)},Φlk(x) ∈ {Rh×w}The mean value of each activation per layer for a given filter and data is given byclk(x) =1h1wh,w∑i,jϕlk(x)i,jwhere h and w denote the height and weight of the layer activation.Next, assuming that the activations over all input data can be fitted into anormal distribution, pSclk denotes the continuous distribution of clk(x) andXS = {xS1 , . . . , xSn} is the input data, where n is the sample size. A similar formulation canbe done for a second dataset:XT = {xT1 , . . . , xTm} generating pTclk , with m denoting the new sample size.The representation shift, R, is defined as the mean discrepancy, D, between the distribu-tions over all filters, k, in a given layer, lRl(pS, pT) =1kk∑i=1D(pSclk , pTclk )where D is the discrepancy or distance metric, which is calculated using Kullback–Leibler(KL) divergence.KL divergence is defined between two discrete probability distributions, P and Q, as:KL(P||Q) = ∑x∈XP(x)log(P(x)Q(x))Electronics 2025, 14, 1419 12 of 26VGG16 [31] has been used for the classification of hand gestures, and SIMLea hasbeen applied on a trained VGG16 model. The architecture contains five convolutionalblocks ending with a max pooling (MaxPool) layer (Figure 3). In the figure (which isreproduced from the paper [31]), “conv3” refers to 3D convolution and “FC” to fullyconnected layers. Recall that R is larger for more dissimilar or more shifted datasets, andvice versa. The following steps are taken for the analysis of each layer. Here, “layer” refersto MaxPool.1. Mean and standard deviations of the activations of each filter are obtained for allimages in the dataset. These statistics will represent the central tendency and spreadof the activations for each filter. Then, a normal distribution is fitted over the valuesusing the mean and standard deviation values. These images are passed through themodel unlabelled, and the outputs are obtained through the inference operation ofmodel0. This is done for the source dataset (THUC) and the target datasets (UDS1, 2,and 3, and VDS).2. KL divergence is calculated between pairs of source and target datasets for each filterof a layer.3. The mean of KL divergence values over all filters per layer is obtained, therebygenerating one value per source-target pair per layer.Figure 3. Architecture of VGG16 and VGG19 adapted from [31]. Convolution layers are followed byReLU activations, which are not shown here. n refers to number of classesKL divergence is not a symmetrical distance measure; thus, it is important to keep thesource unchanged in order to obtain the correct estimates of changes. Since the analysisis in service of DA of the model, the training sets for iterative re-trainings change at eachstep. Therefore, the THUC test set was kept as the source for calculations. For THUC RElectronics 2025, 14, 1419 13 of 26analysis, the target dataset is the training set of the same. In order to ensure that the generalstatistical behaviour is independent of the test and training set of TH, the above analysiswas also carried out with the THUC training set as the source (Appendix A). Of course,the succeeding tests could not be re-done with the THUC training set as source since thetraining set of each model during re-training is different from model0.model0 was used for inferencing with the shifted datasets with a small batch of labelleddata to examine the relation of R with accuracy scores. The R and accuracy values for allthe datasets are presented in Table 2 As is evident, our intuition about the similarity ofdatasets is reflected most clearly in the R values for layers 2 and 3. The test set of THUChas the smallest shift from the THUC training set, which is correlated with the highestaccuracy (0.96). This is followed by higher discrepancy values and corresponding loweraccuracy values from layer 2 for UDS1 (87.26, 0.58), UDS2 (486.05, 0.48), UDS3 (943.40,0.16), and VDS (10,571.15, 0.09). This trend is reflected in all layers, with some deviations.Some deviation is to be expected as one goes deeper into the model, since a large amountof information is lost as the model approximates the input to classify to one of the classes.This is intentionally done so as reduce the dimensionality of data to be handled and toneglect insignificant changes in input data using such layers as Max Pooling and BatchNormalization. Apart from that, the features in the deeper layers are more abstract, and itis challenging to use them to interpret the behaviour of the model. Conversely, the initialconvolution layers handle very macro-level features, and they too should not be used forinterpretation of domain shifts.Table 2. Covariate shift (R) and accuracy values for TinyHand Uncluttered Dataset (THUC), UserSeated, Uncluttered (UDS1), User Seated, Cluttered (UDS2), User Standing, Cluttered (UDS3), andVirtual Datasets (VDS) using the THUC test set as source.Tested On THUC UDS1 UDS2 UDS3 VDSAccuracy 0.96 0.58 0.48 0.16 0.09R, Layer 1 0.87 125.04 529.88 407.37 662.69R, Layer 2 0.77 87.26 486.05 943.40 10,571.15R, Layer 3 0.38 33.86 70.78 67.34 116.28R, Layer 4 0.18 8.58 9.78 26.56 6.26R, Layer 5 0.12 2.08 5.08 6.54 6.12Layer refers to MaxPool layer.For this work, the second and third MaxPool layers provided intuitive results (as canbe attested by Table 2 and in the succeeding study in the later part of this chapter), and arethus considered for analysis. The first layer follows the intuition, except for the discrepancyexpected of UDS2 and UDS3. The values for the fouth layer indicate higher similarity forVDS, which is certainly not backed by the accuracy score for VDS. The fifth layer reflectsthe desired trend, except for UDS3 and VDS. Notice that the R values decrease for all casesdeeper into the model. This is a consequence of the loss of information in the attempt toapproximate towards pre-learnt classes.Let us try to visually examine the normal distributions of the datasets for each layerto gain an insight regarding the statistical shifts between the THUC and user datasets.The normal distributions of the activations of the second and third MaxPool layers withrespect to each dataset have been plotted in Figure 4. In Figure 4a, for the second MaxPoollayer, the standard deviation of the THUC training set is the largest since it contains thelargest variance in data. Comparatively, the spread of the THUC test set is smaller butstill larger than the shifted datasets. This is showcased in the other layer as well. UDS3is the most dissimilar among the user datasets, and this is evident from the shape ofthe distribution (more variation, hence flatter and shifted mean value). Thus, as morevisual variations are added to the scene, the more statistically the dataset distribution getsshifted leading to lower accuracies. Layer 3 (Figure 4b) provides the clearest indicationElectronics 2025, 14, 1419 14 of 26of a covariate shift that satisfies our visual inspection hypothesis. The lower standarddeviations in the user sets are also a consequence of the dataset being generated using shortRGB videos and the presence of only one user. Note the different x-axis scales that havebeen used for better visualization. VDS in (a) and (b) appear distinct from the user datasets.Its mean value is shifted from the TH training set, and it contains more variance than theuser datasets.Figure 4. Normal distributions representing activations of each MaxPooling layer for source andtarget datasets. (a) denotes MaxPool layer 2 and (b) layer 3. In (a), the range of horizontal axis is 30 to90 and 0 to 400 in (b). The vertical axis in (a) is 0.00 to 0.25, while in (b) it is 0.00 to 0.09.Batches of around 200 unlabelled images (from all the classes) were used for thestatistical analyses. This would require only about five to seven seconds for each gesturevideo. Smaller batches of images ensure that an acceptable amount of computationalresources are dedicated for the statistical analysis during online learning of the model.Section 5.4 explores the issue of batch size. This shift analysis constitutes step 2 of theSIMLea framework. The similarity metric is used to decide if the model has learnt the newdataset and also if TL for DA is a feasible course of action.5. ResultsSIMLea was applied to a hand gesture recognition task that consisted of the MLmodel constituting the robot perception system encountering new users, changes in theElectronics 2025, 14, 1419 15 of 26collaborative work-cell, and in the sensors. The capability to detect the changes thatmanifested as covariate shifts in the incoming data distribution and to decide the meansto incorporate the changes into the model were enabled by the application of SIMLea.The validation tests undertaken in this work are indicative of the expected trends whenworking with human-generated communication data in the form of RGB images andhandled using deep CNNs. However, while the steps of the framework are generic andapplicable for a wide variety of HRCom scenarios, the architecture and type of classifierused will dictate the parameters of transfer learning (TL), the similarity metric used, and theshift values encountered. Table 3 contains the gist of the section and the studies carriedout. The datasets created based on the publicly available TH and THC datasets and thesubsequent studies carried out are to mimic changes to the work-cell of the human–robotcollaboration scenario: addition of new human workers and changes in the environment ofthe work-cell. This is achieved by the cluttering of the background of the gesturing human,changes in the humans in the dataset, and by the humans standing up or sitting down toadd more variability. Finally, for more top test extreme cases of covariate domain shift,virtual reality datasets are used.Table 3. Studies carried out with SIMLea and their corresponding subsectionsSubsection Datasets Used Aim5.1. Applicationof SIMLeato User DatasetsTinyHand Uncluttered Dataset (THUC),User Seated, Uncluttered (UDS1),User Seated, Cluttered (UDS2),User Standing, Cluttered (UDS3)Application of SIMLea and tracking of co-variate shifts and accuracy of models gen-erated through iterative training5.2. Applicationof SIMLeato Virtual RealityDatasetTinyHand Uncluttered Dataset (THUC),Virtual Dataset (VDS)Application of SIMLea and tracking ofcovariate shifts and accuracy of mod-els generated through iterative trainingQualitative representation of covariate do-main shifts encountered in hand gestureclassification showing the relative shiftsand corresponding similarity in the stan-dard and used shifted datasets5.3. Applicationof SIMLea to asimulated Long-Serving ModelTinyHand Uncluttered Dataset (THUC),TinyHand Cluttered Dataset (THC),User Seated, Uncluttered (UDS1),User Seated, Cluttered (UDS2),User Standing, Cluttered (UDS3)Virtual Dataset (VDS)Study of application of SIMLea to a con-tinuously adaption scenario of a model de-ployed for a simulated long time usage5.4. Examinationof Batch Size forShift CalculationTinyHand Uncluttered Dataset (THUC),TinyHand Cluttered Dataset (THC),Uncluttered Datasets with Left-HandedGestures (UDS1_L)Study of application of SIMLea on largeand small sized covariate shifted datasets5.1. Application of SIMLea to User DatasetsIn this work, voice command (VC) has been used to generate feedback. The thirdassumption is true for voice command classifiers trained on large publicly available datasetsand on simple commands. As an example, VC classifiers can be trained using the GoogleSpeech Commands dataset [33], which contains 65,000 s of recorded and annotated data bythousands of participants. In comparison, other modes of communication, such as handgestures (HG) for specific applications such as the TinyHand dataset, contain data fromonly 40 individuals [32]. The disparity in data leads to disparity in model performance,which can be tackled using SIMLea.Performance Metrics: In the human-generated (HG) datasets used in this paper,the number of epochs required for 20%+ improvement of prediction accuracy scores havebeen in the range of 2 to 3. As more dissimilar datasets need to be adapted, it is logical thatthe more time the model needs to train on, and hence the number of epochs, would increase.Electronics 2025, 14, 1419 16 of 26So, for the purposes of illustration of SIMLea for HG, the number of epochs required for afeasible application of TL for DA is about 3.In order for TL to work effectively, data from all classes must be used at every iterationand for human communication data similar to the training data, small batches of shifted dataare sufficient for the model to learn. Furthermore, the batches for TL must be composed of ahigher fraction of data from the source dataset. SIMLea was implemented and validated onUDS1 and UDS2. Empirically determined batches composed of 20 user images +100 THUCtraining set images for each class and auto-labelled through feedback were used for iterativere-training, as per the results from the work on personalization in HRCom [16].On deploying model0 and it encountering UDS1, R was calculated for all layers, alongwith accuracy, as shown in Table 4. Since the R value indicated domain shift as compared toTHUC training set shift score, DA was required to counter the domain shift. On an ad hocbasis, TL was used as the first study. VC classifier-enabled feedback can be expanded uponand used for the seven classes of HG as well since the names of the classes are commonlyused words. A simulation of feedback generated using voice commands from the humanwas used to label the data, which consisted of 20 images per class of gestures. Conversely,simpler visual or haptic interfaces like touchscreens may also be used instead of using anML model for feedback.TL was used after labelling, which consisted of freezing the layers of model0, exceptfor the fully connected and classification layers. The training was done for two epochs,thus proving that TL was effective, generating model1. The shift calculations were carriedout with model1 with another batch of the same user data, UDS1, the results of whichare presented in Table 4. It was observed that, for all layers, the R value decreased, thusindicating a decrease in the dissimilarity of the datasets with respect to the new model andan increase in the variance of said model. For the purpose of validating, labelling was donefor a new batch of UDS1 and tested with model1, giving an accuracy of 0.73, which was a25.8% increase in one iteration and two epochs of training.This implementation gave an empirical idea of the thresholds for the feasibility of TLfor that particular dataset and model and an indication of accuracy from the R value. Itshould also be noted that the accuracy value for the THUC test set remains high (0.97 formodel1 for UDS1 and 0.94 for UDS2). The framework can be deployed further or stopped onthe basis of the threshold set for the application. A similar approach was taken for UDS2,which also demonstrated a lessening of R for all layers (Table 5), with a new accuracy scoreof 0.59 (an increase of 22.9%) with two epochs.Table 4. New covariate shift (R) and accuracy values for User Seated, Uncluttered (UDS1) oniteratively trained model1.TestedOnNewAccuracyNew R on MaxPool LayerLayer1 Layer2 Layer3 Layer4 Layer5UDS1 0.73 119.49 81.06 28.86 9.10 2.04Table 5. New covariate shift (R) and accuracy values for User Seated, Cluttered (UDS2) on iterativelytrained model1.TestedOnNewAccuracyNew R on MaxPool LayerLayer1 Layer2 Layer3 Layer4 Layer5UDS2 0.59 393.80 171.09 53.07 16.99 5.58Figure 5 contains the distributions of activations of the second and third MaxPoollayers juxtaposed for UDS1, UDS2, and THUC test datasets with respect to each layer:(a) and (b) are for model0, while (c) and (d) and (e) and (f) are for model1s for UDS1 andElectronics 2025, 14, 1419 17 of 26UDS2, respectively. Please note that model1 is different for the two datasets, and the term isreused as a model variable name within the iterative framework. The ranges in verticalaxes in (a) to (f) are as follows: 0.00 to 0.25, 0.00 to 0.08, 0.00 to 0.25, 0.00 to 0.08, 0.00 to0.175, and 0.00 to 0.08, respectively. The ranges in horizontal axes in (a) to (f) are as follows:30 to 100, 60 to 180, 30 to 100, 60 to 180, 30 to 100, and 60 to 180, respectively.A decrease in (learnt) variance in THUC can be observed in the standard deviations ofTHUC in (d), (e), and (f) with respect to (b), (a), and (b) again. This stems from the model’sloss of learnt features and not a change in the source dataset itself. This is a consequence ofTL. Based on such indicators, the designer may add a higher number of THUC images whencreating batches for re-training. Be that as it may, in all instances, the distance between themean values of THUC and the user datasets decreased, thus validating a decrease in theperceived shift of the model when handled using SIMLea.Figure 5. Normal distributions of activations of the second and third MaxPool layers juxtaposed forTinyHand Uncluttered (THUC), User Seated, Uncluttered (UDS1), User Seated, Cluttered (UDS2)datasets with respect to each layer before iterative retraining (a,b) and after (c,d): UDS1 and(e,f): UDS2. The ranges in the vertical axes in (a–f) are as follows: 0.00 to 0.25, 0.00 to 0.08, 0.00 to0.25, 0.00 to 0.08, 0.00 to 0.175, and 0.00 to 0.08, respectively. The ranges in horizontal axes in (a–f) areas follows: 30 to 100, 60 to 180, 30 to 100, 60 to 180, 30 to 100, and 60 to 180, respectively.Electronics 2025, 14, 1419 18 of 265.2. Application of SIMLea to Virtual Reality DatasetsIn order to study a case where, although the task is the same (hand gestures withthe same labels) but the data is so covariate shifted that transfer learning would not befeasible and training from scratch will be more effective, a virtual reality dataset of handgestures was developed. Let us consider this case: for an HRC, or more generally, a human–machine interaction scenario, a virtual reality (VR) environment (VEnv) linked to thereal-world environment (REnv) may contain a representation of the human in it. The realHG in REnv may be recreated in the VEnv, which would then be recognized using an MLmodel, much like REnv gestures. Thus, the VEnv may be conceived as being composed oftwo parts—the representation of the human through a humanoid model and the setting of theimmersive scene, which includes the light(s), the virtual sensor(s), other interacting object(s),and artefacts such as walls, floor, ceiling, their colours, and so on. In order to simulatethe dynamic nature of the industrial setting or work-cell, variability in the positioningof the humanoid as well as variability in the positioning of the hand, the degree to whichthe gestures are performed, and the setting of the work-cell itself must be representedin the dataset. This constitutes an HG dataset that is recognizable by humans belongingto the specific classes as the REnv dataset but is so dissimilar from the perspective ofthe representation space of the model that TL would not be feasible for adaptation. Thisis reflected in the S values that are much lower than those for HG datasets in the REnvand is further confirmed on attempting TL wherein ≥20 epochs are required to achievecomparable gains in accuracy. For this study, two datasets were created—one using ageneric silver coloured humanoid with a generic background and the other closer to thefeatures learnt by the model, i.e., a humanoid with a more “natural” skin tone [34].Even after curating VDS in such a way as to make the images as close to the source,training set of THUC as possible, the shift values, as seen from Table 2, are still very high,indicating a significantly large covariate shift (∼76,071% + of the distance between theTHUC training and test sets). TL is not effective for VDS since a 10% gain in accuracyrequired 21 epochs. Therefore, in terms of the feasibility of TL, such a dataset may beconstrued to be at the limit of usefulness. Figure 6 qualitatively represents the domainshift from the source (training set of THUC) to the farthest target datasets (VDS with the“skin-tone” humanoid and farther shifted for the silver coloured humanoid). Most HGRenv datasets will fall somewhere in between the extremes discussed above, and someheuristics in the form of design thresholds will be required to customize the framework tothe application.Electronics 2025, 14, 1419 19 of 26Figure 6. Qualitative representation of covariate domain shifts encountered in hand gesture classification showing the relative shifts and corresponding similarity inthe standard and used shifted datasetsElectronics 2025, 14, 1419 20 of 265.3. Application of SIMLea to a Simulated Long-Serving ModelSIMLea is designed to enable continuous learning of an HRCom model. It is expectedthat a long-serving model, initially trained with a simpler dataset such as THUC, wouldcontinue to learn the features of new users and environments of the work-cell during itsoperation. At the same time, it should still be able to detect changes in the domain (ifrelevant enough to be further handled) and process the OOD data. Therefore, in order tosimulate such a situation, VGG19 [31] was used to train on the training sets of both THUCand THC. The training and validation sets contained 147,807 images split along (70:30)%and the test set containing 63,341 images. The hyperparameters remained the same, exceptfor the number of epochs, which was 20 for this case. A deeper model was chosen for thiscase, containing three extra convolution layers to account for the larger variability in datathe model would have encountered during its operation. Delving into the dataset, THCcontains 15 individuals (different from THUC) and a collection of seven different clutteredbackgrounds. The new user datasets (UDS1, 2, and 3) contain individuals and backgroundsthat are different from all of the combinations mentioned above. So, SIMLea should stillbe required. A cautionary point to be presented is that the R values correlated to accuracyvalues are not model agnostic and so the shifts in the present analysis would be unrelatedto those in the previous analysis because of differences in the architecture and training sets,leading to significantly different models.This VGG19 model was trained with lateral inversion as a data augmentation so asto enable recognition of the gestures performed with the left hand. UDS1, 2, and 3 inthis case also contained left-handed gestures in order to increase the variance of the data.The discrepancy values of these complex, shifted datasets were evaluated. Tables 6 and 7present the values grouped by MaxPool layers.Table 6. Covariate shift (R) and accuracy values for TinyHand Uncluttered Dataset (THUC), UserSeated, Uncluttered (UDS1), User Seated, Cluttered (UDS2), User Standing, Cluttered (UDS3), andVirtual Datasets (VDS) on model0 trained on both THUC and THC (TinyHand Cluttered Dataset) withTHUC as the source distribution for shift calculations.Tested On THUC UDS1_L UDS1_R UDS2 UDS3 VDSAccuracy 0.97 0.84 0.78 0.59 0.52 0.09R, Layer 1 - 201.43 134.27 329.49 610.39 39,025R, Layer 2 - 68.53 53.63 68.74 61.04 386.46R, Layer 3 - 27.13 32.46 53.34 10.73 15.58UDS1_L: UDS1 with left-handed gestures, UDS1_R: UDS1 with right-handed gestures. Layer refers toMaxPool layer.Table 7. Covariate shift (R) and accuracy values for TinyHand Cluttered Dataset (THC), User Seated,Uncluttered (UDS1), User Seated, Cluttered (UDS2), User Standing, Cluttered (UDS3), and VirtualDatasets (VDS) on model0 trained on both THUC (TinyHand Uncluttered Dataset) and THC withTHUC as the source distribution for shift calculations.Tested On THC UDS1_L UDS1_R UDS2 UDS3 VDSAccuracy 0.72 0.84 0.78 0.59 0.52 0.09R, Layer 1 46.29 2207.42 1897.07 178.75 62.79 2439R, Layer 2 30.18 264.05 256.69 157.53 24.67 459.35R, Layer 3 11.02 47.40 62.81 36.60 11.75 79.57UDS1_L: UDS1 with left-handed gestures, UDS1_R: UDS1 with right-handed gestures. Layer refers toMaxPool layer.Similar to VGG16, VGG19 also contains five convolution blocks ending with fiveMaxPool layers. The following trends can be observed from the data in Tables 6 and 7.1. There is a distribution shift between the training sets of the model itself, i.e., betweenTHUC and THC (indicated by R shift values of 46.29, 30.18, 11.02 of layers 1–3,Electronics 2025, 14, 1419 21 of 26respectively). The shift is the smallest among the pairs due to the model being trainedon both data, although in terms of visible similarity, THUC and UDS1 would be themost similar. Apart from that, the image sizes of the two training dataset were alsodifferent (THUC: 640 × 480 pixels, THC 1920 × 10,180 pixels), indicating a change insensors capturing the data.2. The covariate shift R values for UDS1_R, indicating right-handed gestures are lesserthan UDS1_L, indicating left-handed ones for layers 1 and 2 (134.27 versus 201.43 forlayer 1, 53.63 vs. 68.53 for layer 2 for right-handed and left-handed, respectively). Thisis logical since the original training set contains only right-handed gestures and thusmay be biased toward them.3. UDS1 is more similar to the training sets of THUC than UDS2 or UDS3, and thelevel of similarity decreases in that order. This behaviour is attributed to the similar,uncluttered, clean backgrounds of THUC and UDS1. This is inferred from the shiftvalues: for layer 1: UDS1_R is 134.27, for UDS2 329.49, and for UDS3 610.39, and thistrend also holds for layer 2.4. THC with the cluttered backgrounds and standing humans are interpreted to be moresimilar to UDS3 followed by UDS2 and finally UDS1, which shows a large domainshift. This trend is also followed by the deeper layers. This is expected since UDS3contains the user standing in front of a cluttered background and gesturing and isthus structurally most similar to THC. Shift values of 62.79, 178.75, 1897.07 for layer 1for UDS3, UDS2, and UDS1_R, respectively, indicate this for example.5. VDS is the least similar to THC or THUC and reflects results from the previous studies.Shift values for VDS are the largest in both tables for all three layers as compared toother datasets.6. Accuracy scores for the user datasets are higher than in the previous study. This is alsoexpected since the model contains higher variance. With that said, the metric is stillable to detect a shift in distribution of incoming data. UDS2 and 3 may be used for DA,while UDS1 could be allowed to be classified with the current model. For DA, a mixof THUC, THC, and UDS1 can be used as the “old” data and the data from UDS2 orUDS3, respectively, as the “new” data.5.4. Examination of Batch Size for Shift CalculationThe sample size of images in a dataset should be statistically representative of thepopulation under purview. Under the constraints of sparse data and limited allowance ofcomputation time during operation, this is an extremely pertinent design consideration.All the aforementioned tests were carried out with about 200 images per batch for statisticalanalysis. A final study was conducted with 420 images for UDS1 left-handed gesturesin comparison to the smaller batch containing 210 images. Table 8 contains the R valuesindicating domain shifts with respect to the VGG19 model trained with THC and THUC,as mentioned in Section 5.3. The difference between the R values between the smallerand larger datasets for a given layer, and for either THC or THUC as source sets, is amaximum of 21.8% (layer 2, for THUC as source set). Along with that, the discrepancyvalues follow the same trend for either size of dataset. Hence, a lesser number of imageswere deemed safe to use for the analysis. Finally, a third reason for bolstering the rationalefor small batches is that user data generated from a few seconds of only one user gesturingcontain significantly less variance than the large publicly available datasets. Thus, for morecomplex datasets, a larger batch of images is required, and a smaller batch for simpler onesmay be required.Electronics 2025, 14, 1419 22 of 26Table 8. Covariate shift (R) for User Seated, Uncluttered Datasets with Left-Handed Gestures(UDS1_L) on model0 trained on both THUC (TinyHand Uncluttered Dataset) and THC (TinyHand Clut-tered) with THUC and THC as source distributions for examination of batch size of target distribution.Tested On UDS1_L(Large Set)UDS1_L(Small Set)UDS1_L(Large Set)UDS1_L(Small Set)Source THUC THUC THC THCR, Layer1 190.88 201.43 2360.98 2207.42R, Layer2 53.60 68.53 223.89 264.05R, Layer3 23.44 27.13 45.48 47.40Layer refers to MaxPool layer.6. Conclusions and Future WorkA novel domain adaptation framework, SIMLea, is presented that leverages the ad-vantage of publicly available, large datasets of multimodal data and high-accuracy modelstrained on them to provide feedback from the human for auto-labelling during continu-ous operation. In contrast to the current research under domain adaptation and domaingeneralization, SIMLea is designed to allow feedback to be used from incommensuratemulti-modes of communication. This feature allows the framework to be mode agnosticand flexible in its application, in addition to reducing mislabelling.Domain adaptation allows the extension of the model’s learnt feature representationswith those of new, covariate domain-shifted data. For human–robot collaboration, thisallows personalization of the model to the collaborating human. A statistical measureof similarity is used to drive the adaptation framework. Thus, models under SIMLeaare empowered to learn only relevant data and start and stop the computationally heavylearning process when required. Ergo, during continuous operation, the model learningis statistically and heuristically driven. SIMLea allows the amalgamation of empiricalthresholds and standards and application-specific heuristics. It is an add-on to the MLmodel and does not require change in its architecture. Finally, while human-generateddatasets are the primary focus of this dissertation, the framework can be used for DAin other applications as long as a feedback-giving agent exists that can be used for auto-labelling. The results indicate a lowering of covariate shift values between the training andshifted data on iterative re-training using the feedback-enabled labelling of data. For theuser dataset with seated user and uncluttered background, there was a 25.8% increase inone iteration and two epochs of training, and for the user dataset with cluttered background,there was a 22.9% increase in accuracy with two epochs using SIMLea.Several research works in HRC have used establishing the context as a key componentof HRCom setup. Additional information through context understood by the robot could beused to remove some of the reliance on heuristics in future works. As an example, considerthat the robot perception understands through context that the incoming communicationis not relevant to the task (e.g., the human is speaking to another agent about a task unre-lated to the current robotic agent’s purview), then without expending the computationalresources using SIMLea to find if the data is similar enough to be adapted or not, the robotcan simply reject the data.To provide a consolidated command from the multiple modes of input, the multi-faceted inputs must be combined by way of fusion. Thus, translating complementary inputmodes will make the communication more reliable, resolve the complexity associated withthe robot’s understanding of human behaviour, and hence improve the overall robustnessof the HRC system. Usage of fusion techniques and multimodal communication can alsobe useful for datasets that are lower in quality than required, errors arising out of faultysensors, and for robustness in diverse environments. As an example, noisy environmentscan rely on the fusion of speech data, with more weight placed on vision data.Electronics 2025, 14, 1419 23 of 26The loss of variance contained in the large, standard dataset during the continuedapplication of SIMLea is a limitation of the framework. From a safety standpoint, the higherspecificity of the model is an asset, the trade-off being that, as the model becomes morespecific (towards the humans working with it), TL under SIMLea will need to be usedmore and more. This is an inherent limitation of the DA, DG, and TL techniques whendealing with shifted datasets that are much smaller than the source ones. From an industrialperspective, calibration drift is tackled through the obsolescence of legacy informationand the tuning of the model to the new incoming data. Nonetheless, TL under SIMLeawith portions of “old” data ensures the persistence of features from those datasets, whilegradually lessening them over the operation life of the model.Given the changing variance of the models, another checkpoint may be added toSIMLea in future works to stop further learning once the representation space of the currentmodel contains so much less variance as to be highly biased to the training data. This maylead to all incoming data being flagged as too dissimilar, thus leading to needing the modelto be trained from scratch, thus further reducing the variance in learnt features. Of course,the model would encounter such a situation after multiple changes in the incoming datasets.Statistical analysis of the feature space and heuristics may be employed to indicate whenthe model has reached the high bias, low variance state.Future work also includes a study of scalability of the framework through the calcu-lation and tracking of processing times of the framework to test for real-time application.Even though, since the dataset sizes for re-training were kept low and the number ofepochs was also low, the iterative re-trainings did not take significantly large amounts oftime, formal analyses will be carried out in future work with the aim of studying real-timeoperation. While tests were conducted on a simulated long-serving model, more workneeds to be done on this front in conjunction with rising computational costs with theaddition of more data.Author Contributions: D.M., development of methodology presented, formal analysis, datacuration, investigation, software development of implemented framework and model, visu-alization, writing—original draft; H.N., funding acquisition, project administration, supervi-sion, writing—review and editing. All authors have read and agreed to the published versionof the manuscript.Funding: Research supported by UBC Office of the Vice-President, Research and Innovation in theform of seed funding to establish research on digitalization of manufacturing.Institutional Review Board Statement: Not applicable since datasets used are publicly available,created by other researchers, and cited as such in text. User datasets for testing feature the authors ofthe paper.Informed Consent Statement: Not applicable.Data Availability Statement: The new user data generated and used in this study are available onrequest from the corresponding author due to privacy reasons of the user.Conflicts of Interest: The authors declare no conflicts of interest. The funders had no role in the designof the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; orin the decision to publish the results.Appendix A. Alternate Shift Analysis with THUC Training Set as SourceThe domain shift calculations were repeated with the THUC training set as the sourceso as to ascertain the reasoning that the THUC test and training sets are pulled from thesame (or nearly the same) distributions and can be used as a representation of the THUCdataset. Refer to Table A1 for the covariate representation shift values for this analysis. ItElectronics 2025, 14, 1419 24 of 26can be observed that the trends follow the intuition pertaining to the similarity betweenthe datasets, especially for layers 2 and 3, the same as for Table A1.While using either the THUC training or test set proves effective before re-training andfor gauging the similarity between the source and shifted data, it is not so after re-training.As mentioned earlier, the “training set” is no longer just THUC; rather, it is a mixtureof THUC training images and auto-labelled shifted user images. Thus, in order to keepbench-marking parameters invariant, the THUC test set was used for calculations. Apartfrom that, let us inspect the divergence mathematically.The formulation of KL divergence can be understood as follows:DKL(N1, N2) = logσ2σ1+σ21 + (µ1 − µ2)22σ22− 12From Figure 4, it can be noted that the standard deviation of THUC is > 1 by dintof its greater variance in features, while that of user datasets or VDS are < 1. In order todetermine if DKL(N1, N2) is monotonically increasing with σ1, let us find the first orderdifferential of DKL(N1, N2) with respect to σ1, keeping σ2 constant and the means of thetwo source and target distributions equal.DKL(N1, N2) = logσ2 − logσ1 + σ21+(µ1−µ2)22σ22− 12d(DKL(N1, N2)dσ1) = − 1σ1+ 2σ12σ22In order to be monotonically increasing, d(DKL(N1,N2)dσ1 ) must be > 0 for all values.d(DKL(N1, N2)dσ1) > 0=⇒ − 1σ1+ 2σ12σ22> 0=⇒ σ1σ22>1σ1=⇒ σ21 > σ22This inequality is true for the nature of data being handled, as mentioned above.Therefore, for the same σ2,DKL(N1, N2) ∝ f (σ1).Thus, as variance in the source dataset decreases, the divergence decreases if otherfactors are unchanged. This response of KL divergence would lead to indications of lessercovariate shift or R values if the source dataset does not contain high variability. Thisexplains the lower R values for the analysis using the THUC test set versus training.The THUC training set does contain high variability in features, but accessing a repre-sentative set for analysis every time during operation would prove to be time-consuming.In some cases, the model available may be trained without access to the training data.Keeping in mind such cases, the test set was used effectively for SIMLea. Furthermore,with all the factors kept constant, the effect of the standard deviation of the THUC test setis applied to all the steps of the adaptation, thus maintaining consistency in the process.Thus, for this application, it was deemed satisfactory to use the THUC test set as the sourcefor further DA calculations.Electronics 2025, 14, 1419 25 of 26Table A1. Covariate shift (R) and accuracy values for TinyHand Uncluttered Dataset (THUC), UserSeated, Uncluttered (UDS1), User Seated, Cluttered (UDS2), User Standing, Cluttered (UDS3), andVirtual Datasets (VDS) using THUC training set as source.Tested On THUC UDS1 UDS2 UDS3 VDSAccuracy 0.96 0.58 0.48 0.16 0.09R, Layer 1 146.08 417.86 3863.28 4689.43 8211.66R, Layer 2 183.72 241.45 3037.89 8296.30 4,701,616.77R, Layer 3 1.63 45.87 93.03 72.51 409.55R, Layer 4 3.98 10.94 10.65 25.16 12.11R, Layer 5 0.18 1.72 3.29 4.94 5.83Layer refers to MaxPool layer.References1. Ullah, I.; Adhikari, D.; Khan, H.; Anwar, M.S.; Ahmad, S.; Bai, X. Mobile robot localization: Current challenges and futureprospective. Comput. Sci. Rev. 2024, 53, 100651. [CrossRef]2. Mukherjee, D.; Gupta, K.; Chang, L.H.; Najjaran, H. A Survey of Robot Learning Strategies for Human-Robot Collaboration inIndustrial Settings. Robot. Comput. Integr. Manuf. 2022, 73, 102231. [CrossRef]3. Mukherjee, D. Statistically-Informed Multimodal Domain Adaptation in Industrial hUman-Robot Collaboration Environments.Ph.D. Thesis, University of British Columbia, Vancouver, BC, Canada, 2023. [CrossRef]4. Mukherjee, D.; Gupta, K.; Najjaran, H. A Critical Analysis of Industrial Human-Robot Communication and Its Quest forNaturalness Through the Lens of Complexity Theory. Front. Robot. AI 2022, 9, 477. [CrossRef] [PubMed]5. Drummond, N.; Shearer, R. The Open World Assumption; Technical report; University of Manchester: Manchester, UK, 2006.6. Russakovsky, O.; Deng, J.; Su, H.; Krause, J.; Satheesh, S.; Ma, S.; Huang, Z.; Karpathy, A.; Khosla, A.; Bernstein, M.; et al.ImageNet Large Scale Visual Recognition Challenge. Int. J. Comput. Vis. (IJCV) 2015, 115, 211–252. [CrossRef]7. Lin, T.Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; Zitnick, C.L. Microsoft COCO: CommonObjects in Context. In Proceedings of the Computer Vision—ECCV 2014: 13th European Conference, Zurich, Switzerland,6–12 September 2014; pp. 740–755.8. Motiian, S.; Piccirilli, M.; Adjeroh, D.A.; Doretto, G. Unified Deep Supervised Domain Adaptation and Generalization. arXiv2017, arXiv:1709.10190. [CrossRef]9. Kothandaraman, D.; Nambiar, A.; Mittal, A. Domain Adaptive Knowledge Distillation for Driving Scene Semantic Segmentation.arXiv 2020, arXiv:2011.08007. [CrossRef]10. Long, M.; Zhu, H.; Wang, J.; Jordan, M.I. Unsupervised Domain Adaptation with Residual Transfer Networks. arXiv 2016,arXiv:1602.04433. [CrossRef]11. Chen, M.; Weinberger, K.Q.; Blitzer, J. Co-training for domain adaptation. Adv. Neural Inf. Process. Syst. 2011, 24. Avaliableonline: https://papers.nips.cc/paper_files/paper/2011/hash/93fb9d4b16aa750c7475b6d601c35c2c-Abstract.html (accessed on1 March 2025)12. Blum, A.; Mitchell, T. Combining labeled and unlabeled data with co-training. In Proceedings of the Eleventh Annual Conferenceon Computational Learning Theory, Madison, WI, USA, 24–26 July 1998; pp. 92–100.13. Schwaiger, A.; Sinhamahapatra, P.; Gansloser, J.; Roscher, K. Is uncertainty quantification in deep learning sufficient forout-of-distribution detection? AISafety@ IJCAI 2020, 54, 1–8.14. Hendrycks, D.; Gimpel, K. A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. arXiv2016, arXiv:1610.02136. [CrossRef]15. Chi, S.; Tian, Y.; Wang, F.; Zhou, T.; Jin, S.; Li, J. A novel lifelong machine learning-based method to eliminate calibration drift inclinical prediction models. Artif. Intell. Med. 2022, 125, 102256. [CrossRef]16. Mukherjee, D.; Hong, J.; Vats, H.; Bae, S.; Najjaran, H. Personalization of industrial human—Robot communication throughdomain adaptation based on user feedback. User Model. User Adapt. Interact. 2024, 34, 1327–1367. [CrossRef]17. Sotirios Panagou, W.P.N.; Fruggiero, F. A scoping review of human robot interaction research towards Industry 5.0 human-centricworkplaces. Int. J. Prod. Res. 2024, 62, 974–990. [CrossRef]18. Mukherjee, D.; Gupta, K.; Najjaran, H. An AI-powered Hierarchical Communication Framework for Robust Human-RobotCollaboration in Industrial Settings. In Proceedings of the 2022 31st IEEE International Conference on Robot and HumanInteractive Communication (RO-MAN), Naples, Italy, 29 August–2 September 2022; pp. 1321–1326.19. Stacke, K.; Eilertsen, G.; Unger, J.; Lundström, C. Measuring Domain Shift for Deep Learning in Histopathology. IEEE J. Biomed.Health Inform. 2021, 25, 325–336. [CrossRef]20. Stacke, K.; Eilertsen, G.; Unger, J.; Lundström, C. A Closer Look at Domain Shift for Deep Learning in Histopathology. arXiv 2019,arXiv:1909.11575. [CrossRef]Electronics 2025, 14, 1419 26 of 2621. Nisar, Z.; Vasiljevic, J.; Gancarski, P.; Lampert, T. Towards Measuring Domain Shift in Histopathological Stain Translation in anUnsupervised Manner. In Proceedings of the 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), Kolkata,India, 28–31 March 2022. [CrossRef]22. Ketykó, I.; Kovács, F. On the Metrics and Adaptation Methods for Domain Divergences of sEMG-based Gesture Recognition.arXiv 2019, arXiv:1912.08914. [CrossRef]23. Vangipuram, S.K.; Appusamy, R. A Survey on Similarity Measures and Machine Learning Algorithms for Classificationand Prediction. In Proceedings of the International Conference on Data Science, E-Learning and Information Systems 2021,New York, NY, USA, 5–7 April 2021; pp. 198–204. [CrossRef]24. Li, X.; Yang, X.; Ma, Z.; Xue, J.H. Deep metric learning for few-shot image classification: A Review of recent developments.Pattern Recognit. 2023, 138, 109381. [CrossRef]25. Chechik, G.; Sharma, V.; Shalit, U.; Bengio, S. Large scale online learning of image similarity through ranking. J. Mach. Learn. Res.2010, 11, 1–27.26. Ben-David, S.; Blitzer, J.; Crammer, K.; Pereira, F. Analysis of representations for domain adaptation. Adv. Neural Inf. Process. Syst.2006, 19. Available online: https://dl.acm.org/doi/10.5555/2976456.2976474 (accessed on 18 March 2025).27. Harmouch, M. 17 Types of Similarity and Dissimilarity Measures Used in Data Science. 2021. Available online: https://towardsdatascience.com/17-types-of-similarity-and-dissimilarity-measures-used-in-data-science-3eb914d2681/ (accessed on1 March 2025)28. Jiang, L.; Guo, Z.; Wu, W.; Liu, Z.; Liu, Z.; Loy, C.C.; Yang, S.; Xiong, Y.; Xia, W.; Chen, B.; et al. DeeperForensics Challenge 2020on Real-World Face Forgery Detection: Methods and Results. arXiv 2021, arXiv:2102.09471. [CrossRef]29. Perera, P.; Nallapati, R.; Xiang, B. OCGAN: One-class Novelty Detection Using GANs with Constrained Latent Representations.arXiv 2019, arXiv:1903.08550. [CrossRef]30. Aggarwal, C.C.; Yu, P.S. Outlier detection for high dimensional data. In Proceedings of the 2001 ACM SIGMOD InternationalConference on Management of Data, Santa Barbara, CA, USA, 21–24 May 2001; pp. 37–46.31. Simonyan, K.; Zisserman, A. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv 2014, arXiv:1409.1556.[CrossRef]32. Bao, P.; Maqueda, A.I.; del Blanco, C.R.; García, N. Tiny hand gesture recognition without localization via a deep convolutionalnetwork. IEEE Trans. Consum. Electron. 2017, 63, 251–257. [CrossRef]33. Warden, P. Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition. arXiv 2018, arXiv:1804.03209. [CrossRef]34. Mukherjee, D.; Singhai, R.; Najjaran, H. Systematic Adaptation of Communication-focused Machine Learning Models from Realto Virtual Environments for Human-Robot Collaboration. arXiv 2023, arXiv:2307.11327.Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individualauthor(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury topeople or property resulting from any ideas, methods, instructions or products referred to in the content.
Article
eng
Reviewed
Vancouver : University of British Columbia Library
Multidisciplinary Digital Publishing Institute
CC BY 4.0
Faculty
Researcher
human–robot communication
iterative domain adaptation
personalized machine learning
feedback enabled labelling
machine perception
model personalization
Statistically Informed Multimodal (Domain Adaptation by Transfer) Learning Framework : A Domain Adaptation Use-Case for Industrial Human–Robot Communication
Text