Science, Faculty of
Computer Science, Department of
DSpace
UBCV
Ghimire, Amit
Master of Science - MSc
University of British Columbia
In robot-mediated telepresence, the teleoperator's physical embodiment enhances social interaction with on-site users. Current user representation techniques may result in disparities between the robotic embodiment and the teleoperator's representation, resulting in perceptual dissonance in on-site users which can degrade the interaction quality. To address this dissonance felt by on-site users, we introduce AvatARoid, a novel system that emphasizes the teleoperator's embodiment by superimposing a motion-mapped AR overlay of the teleoperator’s avatar on a humanoid robot. To assess the conceptual validity of AvatARoid, we implemented our system with a virtual robot in a realistic AR simulation carefully designed to maximize the sense of co-presence with the virtual robot. Since our evaluation focused on the different visual representations of the teleoperator, we designed the appearance and behavior of the virtual robot in our simulation to appear visually plausible and realistic within the experimental environment when augmented with different representations of the teleoperator. We conducted a mixed-method between-participants study (n=48) where participants engaged in a "get-to-know-you'' conversation followed by a word guessing game with a confederate teleoperator, represented as a virtual humanoid robot with (a) only the voice of the teleoperator (b) voice and video, or (c) AvatARoid. Our findings suggest that the AvatARoid system significantly improved the on-site user's perceived embodiment of the teleoperator compared to other conditions, particularly enhancing the sense that the teleoperator was situated in the same location as the robot and adding confidence that the teleoperator was in control of the robot. Additionally, AvatARoid allowed for more expressive non-verbal gestures, leading to engaging interactions. In contrast, the video and baseline conditions often resulted in on-site users feeling a pronounced disconnect between the teleoperator and the robot. Our study offers new insights into conceptualizing and evaluating novel user representation techniques to promote social interaction in robot-mediated telepresence systems.
AvatARoid: Using a Motion-Mapped AR Overlay toBridge the Embodiment Gap Between Robot andTeleoperator in Robot-Mediated TelepresencebyAmit GhimireA THESIS SUBMITTED IN PARTIAL FULFILLMENTOF THE REQUIREMENTS FOR THE DEGREE OFMaster of ScienceinTHE FACULTY OF GRADUATE AND POSTDOCTORALSTUDIES(Computer Science)The University of British Columbia(Vancouver)December 2023© Amit Ghimire, 2023The following individuals certify that they have read, and recommend to the Fac-ulty of Graduate and Postdoctoral Studies for acceptance, the thesis entitled:AvatARoid: Using a Motion-Mapped AR Overlay to Bridge the Em-bodiment Gap Between Robot and Teleoperator in Robot-MediatedTelepresencesubmitted by Amit Ghimire in partial fulfillment of the requirements for the de-gree of Master of Science in Computer Science.Examining Committee:Dongwook Yoon, Associate Professor, Computer Science, UBCSupervisorRobert Xiao, Assistant Professor, Computer Science, UBCSupervisory Committee MemberiiAbstractIn robot-mediated telepresence, the teleoperator’s physical embodiment enhancessocial interaction with on-site users. Current user representation techniques mayresult in disparities between the robotic embodiment and the teleoperator’s repre-sentation, resulting in perceptual dissonance in on-site users which can degradethe interaction quality. To address this dissonance felt by on-site users, we intro-duce AvatARoid, a novel system that emphasizes the teleoperator’s embodimentby superimposing a motion-mapped AR overlay of the teleoperator’s avatar on ahumanoid robot. To assess the conceptual validity of AvatARoid, we implementedour system with a virtual robot in a realistic AR simulation carefully designed tomaximize the sense of co-presence with the virtual robot. Since our evaluationfocused on the different visual representations of the teleoperator, we designedthe appearance and behavior of the virtual robot in our simulation to appear visu-ally plausible and realistic within the experimental environment when augmentedwith different representations of the teleoperator. We conducted a mixed-methodbetween-participants study (n=48) where participants engaged in a “get-to-know-you” conversation followed by a word guessing game with a confederate tele-operator, represented as a virtual humanoid robot with (a) only the voice of theteleoperator (b) voice and video, or (c) AvatARoid. Our findings suggest that theAvatARoid system significantly improved the on-site user’s perceived embodimentof the teleoperator compared to other conditions, particularly enhancing the sensethat the teleoperator was situated in the same location as the robot and adding con-fidence that the teleoperator was in control of the robot. Additionally, AvatARoidallowed for more expressive non-verbal gestures, leading to engaging interactions.In contrast, the video and baseline conditions often resulted in on-site users feel-iiiing a pronounced disconnect between the teleoperator and the robot. Our studyoffers new insights into conceptualizing and evaluating novel user representationtechniques to promote social interaction in robot-mediated telepresence systems.ivLay SummaryRobotic telepresence enables social interactions by bringing teleoperators into anon-site user’s environment using robots. The robot should accurately represent theteleoperator to make on-site users feel they are interacting with a person and nota machine. However, traditional methods struggle to authentically represent theteleoperator, creating a disconnect for on-site users which can negatively impactinteraction. To address this, we developed AvatARoid, which overlays a realis-tic AR avatar onto a humanoid robot and synchronizes their motions. Our studyused a virtual robot simulated in AR to compare 3 teleoperator representations:robot-only, video, and AvatARoid. We discovered that AvatARoid substantiallyenhanced participants’ perception of the teleoperator’s presence and control wheninteracting with the robot. Compared to other conditions, participants rated inter-actions using AvatARoid as more engaging. This marks a crucial step forward incombining AR and robotics to offer new possibilities for enriching remote socialinteractions through robotic telepresence.vPrefaceThis thesis is an original intellectual product of the author, Amit Ghimire. Thisthesis is an extension of a conference paper manuscript currently under review forpublication.Dr. Ig Jae Kim at KIST conceptualized the AvatARoid system. The softwaredevelopment described in Chapter 3 and 4 was done by Amit Ghimire. Anova Hou(undergraduate research assistant) assisted in developing the simulation describedin Chapter 4.The study reported in Chapter 5 was conducted with the approval of the UBCBehavioral Research Ethics Board (certificate number H22-01390). Amit Ghimiredesigned the study methods described under the supervision of Prof. DongwookYoon. The study and data analysis were led by Amit Ghimire and Anova Houassisted in conducting the study and qualitative interview analysis.viTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiList of Abbreviations . . . . . . . . . . . . . . . . . . . . . . . . . . . . xivAcknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.1 Different User Representation Techniques in Robotic TelepresenceSystems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.1.1 Challenges in Existing Teleoperator Representations . . . 52.1.2 Augmented Reality (AR) Augmentation of TelepresenceRobots . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.2 Evaluation of AvatARoid: Perceived Embodiment, Social Pres-ence, and Impression Formation in Social Interaction . . . . . . . 72.3 Simulation in Human Robot Interaction (HRI) Studies . . . . . . . 8vii3 Designing AvatARoid: Bridging Embodiment in Robotic Telepresence 103.1 Design Goals of AvatARoid . . . . . . . . . . . . . . . . . . . . 103.2 Design Elements . . . . . . . . . . . . . . . . . . . . . . . . . . 143.2.1 Motion Mapping and Synchronization . . . . . . . . . . . 143.2.2 Overlay Avatar Rendering . . . . . . . . . . . . . . . . . 164 AR Simulation for AvatARoid Evaluation . . . . . . . . . . . . . . . 184.1 Environmental Setup . . . . . . . . . . . . . . . . . . . . . . . . 204.2 Mechanisms to Enhance Co-presence with Virtual Robot in ARSimulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204.3 System Architecture of Simulated Implementation of AvatARoid . 254.4 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . 275 Evaluating AvatARoid . . . . . . . . . . . . . . . . . . . . . . . . . . 295.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305.2 Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305.3 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315.4 Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335.4.1 Subjective Measures . . . . . . . . . . . . . . . . . . . . 335.4.2 Behavorial Measures . . . . . . . . . . . . . . . . . . . . 355.5 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356 Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376.1 AvatARoid Amplifies the Embodiment of the Teleoperator for On-site Users. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386.1.1 The Colocation, Control, and Agency Perceptions in AvatARoidContribute to a Stronger Sense of Embodiment. . . . . . . 386.1.2 The Dissonance of Perceiving the Robot and Teleoperatoras Distinct Agents is Pronounced in the Baseline and VideoConditions, Potentially Diminishing the Sense of Embodi-ment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 416.2 AvatARoid Shows Potential to Increase Feelings of Social Presence 436.2.1 Participants Found AvatARoid to be Personable and Intimate. 43viii6.2.2 AvatARoid Provides Richer Non-verbal Reactions and Ges-tures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 446.2.3 Video and Baseline Conditions Lack the Emotional and In-teractive Depth Achieved with AvatARoid . . . . . . . . . 446.3 No Difference in Subjective and Behavioral Measures of the Im-pression of the Teleoperator . . . . . . . . . . . . . . . . . . . . . 456.4 AvatARoid: Benefits and Limitations . . . . . . . . . . . . . . . . 466.4.1 Overlay and Motion Mapping Enhanced the Sense of Co-location. . . . . . . . . . . . . . . . . . . . . . . . . . . . 466.4.2 Coordinated Motion between the Overlay and the RobotFacilitated the Perception of Agency and Control. . . . . . 476.4.3 Real-time Facial Expressions and Upper-body Gestural Feed-back of AvatARoid are Crucial for Making Emotional Con-nections. . . . . . . . . . . . . . . . . . . . . . . . . . . 476.4.4 Areas of Improvement . . . . . . . . . . . . . . . . . . . 487 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 517.1 Bridging the Embodiment Gap: Unmixing Metaphors? . . . . . . 517.2 AR Simulation Could be a Viable Tool to Evaluate Robot-mediatedTelepresence Systems . . . . . . . . . . . . . . . . . . . . . . . . 537.3 AvatARoid: Implication of Design Implementations and Alterna-tive Implementations . . . . . . . . . . . . . . . . . . . . . . . . 557.3.1 Full Embodiment and Fine Gestures . . . . . . . . . . . . 557.3.2 Enhanced Motion Mapping . . . . . . . . . . . . . . . . . 557.3.3 The Robotic System . . . . . . . . . . . . . . . . . . . . 567.4 Limitations and Future Works . . . . . . . . . . . . . . . . . . . 578 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60A Supporting Materials . . . . . . . . . . . . . . . . . . . . . . . . . . 81A.1 Recruitment Post in Paid Studies Listing . . . . . . . . . . . . . . 82A.2 Consent Form . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83ixA.3 Study Protocol for the Experimenter . . . . . . . . . . . . . . . . 88A.4 Study Protocol for the Teleoperator Confederate . . . . . . . . . . 95A.5 Demographics Survey . . . . . . . . . . . . . . . . . . . . . . . . 100A.6 Post Study Questionnaire . . . . . . . . . . . . . . . . . . . . . . 101A.6.1 Perceived Embodiment of the Teleoperator . . . . . . . . 101A.6.2 Social Presence of the Teleoperator . . . . . . . . . . . . 102A.6.3 Impression of the Teleoperator . . . . . . . . . . . . . . . 102A.6.4 Co-presence with the Robot . . . . . . . . . . . . . . . . 105A.7 Post Study Interview Questions . . . . . . . . . . . . . . . . . . . 105xList of FiguresFigure 3.1 The AvatARoid System: (a) The teleoperator’s motion-mappedavatar is overlaid on a humanoid robot; (b.1) The teleoperatorinteracting from their site; (b.2) The on-site user interactingwith teleoperator in a traditional system; (b.3) The on-site userinteracting with teleoperator in AvatARoid . . . . . . . . . . 11Figure 4.1 Our study setup for the on-site user a) participant setup to beseated in our lab wearing headset, b) the participant’s point ofview of the teleoperator in AvatARoid condition as seen fromthe headset, c) the teleoperator setup in a different room, andd) the teleoperator’s point of view of the participant as seen intheir screen. . . . . . . . . . . . . . . . . . . . . . . . . . . . 21Figure 4.2 Virtual robot waving as seen from the headset . . . . . . . . . 23Figure 4.3 Virtual robot entering and leaving through the real door . . . . 23Figure 4.4 Virtual robot being occluded by real wall . . . . . . . . . . . 24Figure 4.5 Virtual robot casting a shadow on real floor . . . . . . . . . . 24xiFigure 4.6 System architecture of the AvatARoid system, software mod-ules shown in a rectangle: On the left, the teleoperator systemconsists of modules to capture and stream sensor data, enablevoice communication, and display the on-site user video. Inthe middle, the network enables data communication betweenthe two users. On the right, the on-site user system consists ofmodules to create and display the point cloud avatar, a robotcontrol system to send motion commands to the robot, and avoice and video capture module for audio-visual communica-tion from the on-site user. . . . . . . . . . . . . . . . . . . . 25Figure 4.7 Information flow in the AvatARoid system: On the left, theteleoperator system captures the teleoperator’s motion and RGBDimages using a depth cam. In the middle, the joint and pointcloud data are transmitted to the on-site user system. On theright, the on-site user system utilizes the joint data to con-trol the humanoid robot, while the AR headset displays a pointcloud avatar overlaid on the robot. . . . . . . . . . . . . . . . 26Figure 5.1 Three user representations used in our evaluation: (a) Base-line: a humanoid robot, (b) Video: a humanoid robot with ateleoperator video, and (c) AvatARoid . . . . . . . . . . . . 30Figure 5.2 Overview of the study procedure session . . . . . . . . . . . . 32Figure 6.1 Mean embodiment score across three user representations; (*)indicates a significant difference (p <.05). . . . . . . . . . . . 37Figure 6.2 Item-wise rating scores of the embodiment questionnaire; Anasterisk (*) indicates a significant difference based on ordinallogistic regression (p <.05). . . . . . . . . . . . . . . . . . . 38Figure 6.3 Mean gaze duration, expressed as a percentage of the totaltime, reflects the time participants spent looking at the robot’shead across all three conditions or at the teleoperator video inthe Video condition. . . . . . . . . . . . . . . . . . . . . . . 42xiiFigure 6.4 A snapshot from our study session: Participant and Teleopera-tor mirror celebratory gesture after completing word guessinggame a) The teleoperator as seen in participant headset pointof view, b) The participant as seen in teleoperator screen . . . 47Figure 7.1 Histogram of participants frequency distribution for differentco-presence scores . . . . . . . . . . . . . . . . . . . . . . . 53xiiiList of AbbreviationsANOVA Analysis of Variance, a set of statistical techniques to identify sources ofvariability between groupsAR Augmented RealityVR Virtual RealityHMD Head Mounted DisplayHRI Human Robot InteractionIK Inverse KinematicsxivAcknowledgmentsFirst and foremost, I wish to extend my deepest gratitude to my supervisor, Prof.Dongwook Yoon. His support, guidance, and kind gestures have been invaluablethroughout my Master’s degree.A heartfelt thank you goes to everyone in the D-Lab and MUX lab. Being apart of such vibrant and supportive communities has enriched my academic journeyimmeasurably and helped me shape this thesis. Special thanks to Oloff Biermann,Preeti Vyas, Bereket Guta, Rubia Guerra, and Laura Cang for your encouragingwords. I would also like to thank Anova Hou for her invaluable contributions andinsights sincerely. I would also like to thank Prof. Robert Xiao for helping meshape the thesis with his valuable feedback.To my cherished family – my parents (Kishan and Shanti Ghimire), sisters(Sushmita, Ashmita, and Shrasta), brothers (Bishal and Yagya), and nephew (Subidh)– your unwavering love, encouragement, and faith in my abilities have been mybedrock. A special note of appreciation goes to my partner Shraddha and her par-ents (Dr. Bhaveshwar and Nima Pangeni), whose belief in me has been a constantsource of strength and motivation. I am ever grateful for your endless support.Lastly, this research was made possible by the sponsorship from the KIST In-stitutional Program. Additional support came from the NSERC Discovery Grantand CREATE programs and a generous gift from Adobe Research. I am deeplythankful for their support in facilitating this work.xvChapter 1IntroductionRobotic Telepresence Systems enable a teleoperator to control a physical robot inan on-site user’s space, facilitating interactions as if they were physically presenttogether. These systems are gaining traction in sectors like healthcare, education,and the workplace [61], serving to foster engagement between clinicians, long-termpatients, and family members [42, 60, 105, 116, 117], boosting student collabora-tion and motivation in distant learning [41, 104, 114], and enabling in-person likeinteractions in hybrid work scenarios [66, 119]. The consistent theme across thedifferent domains is their potential to enhance social interaction between teleop-erators and on-site users [61]. Social interaction in robot-mediated telepresencehinges on the on-site users viewing the robot as the teleoperator’s embodimentand feeling the teleoperator to be socially present through the robot, both vital forbuilding rapport and a positive impression towards the teleoperator [32, 102].The visual representation of the teleoperator in the robotic telepresence systemplays a crucial role in shaping on-site users’ perception of them. Embodied so-cial presence theory posits that we assign meaning to our interactants’ actions byunderstanding our and their embodiment; thus, understanding others’ embodimentaffects cognitive engagement and perception during collaborative activities [79].This makes our body a conduit for verbal and non-verbal social cues (e.g., identity,body language, and gestures), and meaningful interpretation of these cues bolstersthe sense of social presence and enhances emotional connection [79, 86]. Thus,effective representation increases the sense of others’ perceived embodiment and1makes it easier to understand others’ social cues, making the on-site users viewteleoperators as another social entity [64], giving rise to emotionally resonant in-teraction and a positive impression of the teleoperator.Two distinct embodiments exist in robotic telepresence: the teleoperator’s ownbody and the robot itself. Current user representation methods fail to bridge thegap between these two. The widely used approach of displaying the teleopera-tor’s video attached to the robot [61] does not establish a clear relationship be-tween these two embodiments. While the robot facilitates physical presence, themachine-like form fails to represent the teleoperator’s actions and identity authen-tically causing on-site users to view the system as a machine or object [66, 113]and not the teleoperator’s embodiment, leading to issues like bullying [92] or in-vasion of personal space [113]. Employing a full 3D avatar representation usingAugmented Reality (AR) elevated expectations of the teleoperator’s ability, but therobot’s limitation, failing to match the expectations, landed the experience in the“uncanny valley of telepresence” [48]. Humanoid robots, while enhancing capabil-ities of the robotic system, distorts on-site user perception, assigning robotic traitsto teleoperators [24, 62]. Thus, we must address these existing problems in userrepresentation to bridge the gap between the robotic and teleoperator’s embodi-ment.To this end, we introduce AvatARoid to address our research question Howdoes AvatARoid influence the on-site user’s sense of the teleoperator’s embodi-ment, social presence, and impression compared to robot-only and video represen-tation? AvatARoid is a novel telepresence system that overlays the teleoperator’smotion-mapped photorealistic AR avatar onto a humanoid robot. The humanoidrobot provides physical agency and manipulation capabilities to the teleoperator,letting them “be in on the action” [21], and synchronizing motion between theavatar and robot can reinforce the teleoperator’s embodiment by establishing aone-to-one relation between the robot’s and teleoperator’s actions. This heightenedembodiment, combined with the avatar’s expressivity [48], can reduce the on-siteuser’s dissonance, enhancing social interaction and the teleoperator’s impression.With a simulated implementation of our system using a virtual robot in AR sim-ulation, we compare AvatARoid with existing user representation in telepresencerobots.2To evaluate AvatARoid, we utilized an immersive AR simulation by augment-ing a visually realistic virtual robot with a point-cloud representation of a teleop-erator. Our evaluation focused on how AvatARoid would compare against existinguser representation techniques to display teleoperators in robotic telepresence sys-tems, and we wanted to perform an initial conceptual evaluation with a prototypicalimplementation. We believe having a visually realistic virtual robot in an immer-sive simulation would help us identify differences between different teleoperatorrepresentation techniques. AR simulation maintains the vital connection to thephysical environment, enabling both the teleoperator and the on-site user to interactwithin the actual space of the on-site user. While offering a cost-effective solution,this approach also provides experiential control [14] for evaluating the AvatARoidsystem compared with existing user representation techniques. Furthermore, eval-uating the AvatARoid system presents technical and financial challenges as theunderlying technologies involved, such as robot teleoperation and humanoid con-trol, can be resource and cost-intensive to bring together for early evaluation, withpossibilities of technical issues potentially confounding performance assessmentsduring evaluation [118]. To address these challenges and proceed with a concep-tual evaluation of AvatARoid, we propose using AR simulation as an alternativefor system evaluation. Since the efficacy of simulation depends on its ability toprovide a sense of co-presence between the user and a virtual robot, we present asimulation framework using techniques identified in previous studies to increaseco-presence with virtual entities in AR.Using this simulation, we evaluated how AvatARoid influences the on-siteuser’s sense of the teleoperator’s embodiment, social presence, and impression. Weemployed a mixed-method between-participants study comparing AvatARoid tovideo and robot-only representations. Our results indicate that AvatARoid signifi-cantly enhances on-site users’ sense of the teleoperator’s embodiment, particularlyby increasing the sense of colocation between the robot and the teleoperator whilereinforcing the teleoperator’s control over the robot’s actions. Interview responsesshow AvatARoid’s potential for rich social and emotional engagement. The on-siteuser’s impression (trustworthiness, likeability, similarity, and infrahumanization)of the teleoperator remained the same across representation conditions, hintingthat we need to consider other factors combined with user representation in fu-3ture evaluations. Participants’ high levels of co-presence felt with the virtual robotbolstered our confidence in the evaluation results.Our study makes the following contribution to the field of telepresence.• Introduction of AvatARoid: We introduce a novel robotic telepresence sys-tem that integrates the teleoperator’s motion-mapped photorealistic AR avatarwith a humanoid robot, aiming to bridge the gap between the robot’s physicaland the teleoperator’s actual embodiment.• Empirical Evaluation: We provide empirical evidence from a mixed-methodstudy comparing AvatARoid with other telepresence user representation meth-ods, establishing its potential to enhance the on-site user’s perception of theteleoperator’s embodiment and social presence.• AR Simulation for evaluation: We propose and present an AR simulation ofvirtual humanoid robots as an effective preliminary evaluation method fornovel robotic telepresence systems.These contributions lay a foundation for conceptualizing, realizing, and evaluatingnovel user representation techniques like AvatARoid for augmented telepresencesystems to foster social interactions.The rest of this thesis is structured in the following manner: In Chapter 2, wecontextualize our research within the broader spectrum of scholarly work, focusingon the user representations in robotic telepresence systems, their evaluation and theapplication of simulations in Human Robot Interaction (HRI) research. Chapter 3delves into the core design principles of AvatARoid. In Chapter 4, we outline thesetup, design, and implementation of our AR simulated AvatARoid system. Weprovide methodological details of our evaluation study in Chapter 5. We presentthe findings of our empirical evaluation in Chapter 6. We discuss the implicationsof our findings in Chapter 7 and finally conclude in Chapter 8.4Chapter 2Related Work2.1 Different User Representation Techniques in RoboticTelepresence Systems2.1.1 Challenges in Existing Teleoperator RepresentationsCurrent widely deployed Robotic Telepresence Systems are predominantly de-signed as Mobile Robotic Telepresence [61], best described as “tablet on wheels.”This approach provides mobility for teleoperators but limits their expressivenessand manipulation [21, 98]. With teleoperators represented through a video, lackof effective identity and gestural cues consequently make on-site users often formmental models of the teleoperator as an impersonal object or machine [48, 66, 113],compromising interaction quality and leading to breaches in social and proxemicnorms [66, 113]. AR technologies offer solutions by creating 3D avatars or holo-graphic representations [84, 89] and embodying the remote operator’s gestures andmovements. While they enable expressive gesturing, they lack physical presence,mobility, or manipulative abilities in the on-site user’s setting. Recent studies[48, 54] have combined mobile robots with AR avatar systems to represent theteleoperator. These systems pair the mobility of robots with the expressiveness ofAR avatars. However, they present a dilemma: the robot’s limited physical abilitiesdo not align with expectations set by detailed visual representation of the avatars,leading to uncanny telepresence experiences [48], potentially disorienting users.5Humanoid robots with their mobility, expressiveness, and manipulation capabili-ties [6, 8, 17], enable tangible physical interaction like handshakes and fist bumps,improving social connection and cooperation between teleoperators and on-siteusers [6, 17]. While being effective in inducing socially enriching interactions, likereciprocal gestures [49], they risk distorting the on-site user’s perception of the tele-operator’s personality [23, 24, 62, 101], and negatively impacting the teleoperator’ssocial presence due to strong identity of the robot itself [27]. AvatARoid systemaims to resolve these operational and representational discrepancies by combininghumanoid robots with AR avatars.2.1.2 AR Augmentation of Telepresence RobotsIn the field of AR and robotics, social interaction, and telepresence collaborationhave emerged as significant application domains, as highlighted by a comprehen-sive survey [112]. Integrating AR contents with robotics has been explored toenhance remote interactions, exemplified by systems like RobotAR [120]. Thistoolkit facilitates remote consulting, allowing instructors to overlay AR contentinto a student’s workspace for instructional support during problem-solving tasks.More specifically, exploring user representation via AR in telepresence robotshas yielded various insights. For instance, Zhang et al. [127] demonstrated that anAR-rendered generic avatar head on a robot can assist on-site users in accuratelydetermining the gaze of a teleoperator compared to a conventional tablet display.Hwang et al. [45] introduced a tele-meeting system projecting a 3D video of ateleoperator onto a mobile robot to strengthen the sense of co-presence betweenremote and local participants. More recent innovations have seen the combinationof Virtual Reality (VR) interfaces for teleoperators with Head Mounted Display(HMD) [48] or projection-based AR [54] displays for on-site users, augmenting ateleoperator’s avatar onto wheeled non-humanoid robots.AvatARoid specifically builds on the foundation from the HMD-based AR over-lay system employed by VROOM [48], which offered a promising method to en-rich remote collaboration by enhancing the teleoperator’s expressiveness throughbetter gesturing and non-verbal communication channels. However, despite itspromising potential, the expressive visual representation’s increased expectations6outpaced the capabilities of the non-humanoid robots used, an issue AvatARoid ad-dresses by incorporating a humanoid robot. VROOM’s reliance on a pre-rendered3D avatar created using a static teleoperator image, limited the fidelity of facial ex-pressions. In contrast, AvatARoid utilizes a real-time AR avatar that captures andconveys the teleoperator’s facial expressions accurately. Additionally, VROOM’suse of a HoloLens as the AR HMD encountered challenges due to its restrictedfield of view, which limited gesture communication. To overcome this, AvatARoidadopts a passthrough camera-enabled HMD with a wider field of view, facilitatinga more comprehensive and immersive user experience for the on-site participants.2.2 Evaluation of AvatARoid: Perceived Embodiment,Social Presence, and Impression Formation in SocialInteractionWe evaluate robotic telepresence systems using the Embodied Social Presence The-ory, highlighting embodiment as a pivotal focus during social interaction, influenc-ing users’ cognitive engagement and perception [79]. According to this theory, ourembodied actions provide social cues- from audio-visual to verbal and non-verbal(like body language, movements, and gestures). Interactants assign meaning tothese actions from others in relation to their own embodiment, to generate mean-ing during the interaction. A mutual understanding gives a sense of social presence,the feeling of being genuinely connected to another, understanding their actions,thoughts, and feelings [19, 79, 106]. Thus, the on-site user’s sense of the teleopera-tor’s embodiment and social presence collectively shape the quality of social inter-action and impression of the teleoperator. Findings from previous studies, [32, 102]emphasize the interconnectedness of these elements, suggesting that socially en-gaging interaction is essential for maintaining rapport and physical embodiment inrobotic telepresence by enhancing social presence, subsequently improves both thequality of interaction and impression of the communicating partner. Thus, how theteleoperator’s actual embodiment is represented within the robotic system affectshow the on-site users make sense of the robotic embodiment of the teleoperator.User representation also affects social presence [52, 83], providing various socialcues in combination with the cues afforded by the robot. Previous studies have also7noted that cues like identity [36, 69] and non-verbal gestures provided by differentrepresentations influence the social presence and impression of the communicatingpartner [20, 81]. In robotic telepresence, beyond the physical embodiment improv-ing presence [114], the synergy between user representation cues and the robot’sactions enhances understanding of teleoperators’ gestures and improves interactionand personal impression. Instances, where teleoperators engaged in mutual gazeduring hallway encounters have amplified observers’ impressions and the qualityof social interactions [31]. Notably, achieving synchrony between robotic gesturesand the on-screen teleoperator’s gestures markedly boosts observers’ understand-ing of actions and their impression of the teleoperator [107]. These findings un-derline that providing a detailed representation with AR avatars and aligning robotmotion with the avatar, two foundational aspects of AvatARoid, can further refinethe embodiment, social presence, and impression of the teleoperator.2.3 Simulation in HRI StudiesInteractive 3D simulations offer alternatives to traditional text- and video-basedapproaches for large-scale HRI studies [67] and have recently been used to assesshow co-located robots affect human motivation [58]. Moreover, advancements inVR offer increasingly immersive and realistic simulations across various HRI do-mains, including control of prosthetics arm in assistive robotics [25], collaborationwith robots and manipulators in manufacturing [34, 35, 63, 73, 74, 76, 77, 122],and high-stakes scenarios like search and rescue [13]. While validated as a toolfor assessing social interactions between humans and various types of robots [14,15, 99, 115], limitations of immersive simulations must also be acknowledged.Some studies indicate comparable social experiences and security levels felt be-tween physical and virtual robots [46, 103], others note differences in proxemicspreferences and comfort levels [68] due to VR’s distance compression [47] and re-alism of robots [68]. Despite its limitations, studies have shown VR simulationsto be an ecologically valid and valuable preliminary assessment tool [122] and asuitable substitute for real humanoid robots for studies including verbal interac-tion [75]. Acknowledging these limitations, we believe immersive technologiescan provide rich, interactive, and cost-effective platforms for evaluating novel con-8cepts like AvatARoid, provided factors like distance perception, robot realism, andparticipant’s sense of presence with the robot are well-considered. Building onthe current trend of immersive simulation of HRI studies, we propose AR approachfor simulating telepresence robots, highlighting AR’s ability to integrate virtual en-tities into real-world environments, offering an advantageous context for robotictelepresence interactions.9Chapter 3Designing AvatARoid: BridgingEmbodiment in RoboticTelepresenceAvatARoid superimposes a motion-mapped AR overlay of the teleoperator’s avataron a humanoid robot, providing a direct and enhanced relationship between theteleoperator’s actual and the robotic embodiment in robot-mediated telepresence.At its core, AvatARoid aims to integrate a humanoid robot’s physical manipulationand anthropomorphic appearance with the dynamic expressiveness of an AR avatar.The AR avatar provides rich visual feedback of the teleoperator’s expression andgestures. The synchronization of the avatar and robot offers a streamlined under-standing of the robot’s and avatar’s actions, fostering a genuine and emotionallyfulfilling shared experience with the on-site user. Figure 3.1 depicts an overviewof telepresence interaction using the AvatARoid system. In the following section,we present the design goals of the AvatARoid and the design elements to supportthese goals.3.1 Design Goals of AvatARoidAvatARoid proposes a novel user representation system within the landscape ofrobotic telepresence created to address the limitations of current robotic telepres-10(a) Representation of the Teleoperator in AvatARoid(b.1) The Teleoperator’s View(b) A Comprehensive Overview of the AvatARoid Telepresence System(b.2) On-site User’s View in a Traditional System(b.3) On-site User’s View with AvatARoidFigure 3.1: The AvatARoid System: (a) The teleoperator’s motion-mappedavatar is overlaid on a humanoid robot; (b.1) The teleoperator interact-ing from their site; (b.2) The on-site user interacting with teleoperator ina traditional system; (b.3) The on-site user interacting with teleoperatorin AvatARoidence systems. The overarching design goal of the AvatARoid system is to improvesocial interaction by providing a stronger sense of teleoperator embodiment andsocial presence to the on-site users. Considering this, we present our three designgoals.Enhance social interaction by providing expressive AR representation to theteleoperator. The AvatARoid system acknowledges the importance of physicalgestures and body language in communication, allowing for a more natural andimpactful exchange of non-verbal cues. AR avatars provide a visual semblanceof the teleoperator and bring an element of ’presence’ to the interaction that iscrucial for a richer, more connected experience [18]. The spatial and awarenesscues provided by AR avatars also elicit a strong sense of social presence and co-location, i.e., sharing the physical space together [84, 85]. In contrast, the widelyadopted ’tablets on wheels’ design paradigm, which typically features a tablet dis-playing the teleoperator’s video atop a mobile robot, falls short of emulating therichness of human presence. Its video-only representation confines the teleopera-tor to the screen while the robot occupies the on-site user’s space. This often leadsto the on-site users perceiving the system as a robotic entity more than the tele-11operator’s embodiment [66, 113]. This system also fails to completely present theteleoperator’s non-verbal intents and bodily expressions as the teleoperator’s rep-resentation is limited to the screen. AvatARoid addresses these shortcomings byimproving a video feed’s flat, static representation and introducing a full-bodied,three-dimensional presence that can actively engage with its surroundings. Us-ing AR avatars with non-humanoid robotic systems like ’tablets on wheels’ hasshown that augmenting such AR avatar increased the expressiveness of the tele-operator and showed potential to improve dynamic, collaborative activities wherenon-verbal expressions, spatial awareness, and mobility are important [48].Provide the robot with physical interaction abilities to match with the avatar’sexpressiveness. AvatARoid attempts to support the avatar’s expressive representa-tion through the humanoid robot’s physical capabilities, enabling the teleoperatorsto interact with the environment meaningfully. Humanoid robots offer mobility, ex-pressiveness, and manipulation capabilities [6, 8, 17], enable physical interaction.Handshakes afforded by humanoid robots have improved cooperation during ne-gotiation tasks mediated using humanoid robots [6]. Similarly, social and physicalinteractions such as fist bumps enabled by attaching an arm to wheeled robots im-proved social connection and cooperation between teleoperators and on-site users[17]. They have also been effective in inducing socially enriching interactions,like reciprocal or mirroring non-verbal gestures during communication [49]. Hu-manoid robots also provide a form that resonates with a human. Moreover, previ-ous attempts of augmenting a fully-embodied AR avatar on a non-humanoid robotformed unrealistic expectations about the robotic embodiment’s ability in the on-site user’s space [48]. When the robotic system’s physical action could not matchthe expectations resulting due to the expressiveness of the avatar, it disrupted thetelepresence experience. Combining humanoid robots with AR representationsprovides the full expressive embodiment capable of tangible interaction, which isessential for a fully-rounded telepresence experience and a true sense of “beingthere.”Maintain the focus on the teleoperator’s identity and agency. AvatARoid’s de-sign philosophy centers on enhancing the teleoperator’s social presence and high-lighting their agency over the robot’s actions. AvatARoid’s expressive and realisticAR avatar superimposed over the robot highlights the teleoperator’s identity and12personality. Humanoid robots possess a strong identity themselves, which has beenshown to sometimes overshadow the teleoperator’s human identity and affect theon-site user’s judgment of the personality of the teleoperator [23, 24, 62, 101]. Forusing humanoid robots to their fullest potential, enabling rich interactions with-out letting the robot’s persona eclipse that of the human operator, it is importantto ensure that the humanoid robot is seen as an avatar of the teleoperator, withthe teleoperator’s personality taking center stage. Moreover, a clear indication ofthe humanoid robot’s action should be provided for the on-site users to understandthem as the teleoperator’s actions and establish the teleoperator’s agency. Carefullyaligning the robot’s and avatar’s motion guarantees that the gestures of the teleop-erator’s avatar are mirrored by the robot, helping to create a unified representationthat maintains a coherent narrative of the teleoperator’s presence and agency. Thisis supported by a previous study that synchronizing the teleoperator’s on-screengestures, such as nodding and turning head in physical space by moving or turn-ing an actuated screen [107] improved interpretation of the teleoperator’s actions.AvatARoid improves on this further by synchronizing the gestures on the 3D spaceof the on-site user. Thus, AvatARoid’s realistic AR avatar overlay commands vi-sual focus towards the teleoperator and synchronization of the robot’s and avatar’smotion helps to establish the teleoperator’s agency over the robot.AvatARoid’s conceptual framework is the synergy between the tangible pres-ence of humanoid robots and the visual and emotional expressiveness affordedby AR avatars while still maintaining the system’s identity as the teleoperator’sembodiment. The design elements of these core concepts aim to ensure that therobotic embodiment is seen and felt as a physical entity representing the teleop-erator that can participate actively and contribute to collaborative efforts, therebysolidifying the teleoperator’s presence in the shared space. The following sectionwill delve into the two core elements of AvatARoid’s design to achieve these over-arching goals, exploring the mechanisms that enable realistic representation withAR avatars and synchronizing this avatar with the humanoid robot.133.2 Design ElementsAvatARoid’s design is anchored on two primary design elements: motion synchro-nization between the robot and the AR avatar and detailed rendering of the avatar.Both elements prioritize telepresence accuracy, aiming to match the teleoperator’sexpressions and actions closely. Emphasis is placed on real-time alignment be-tween the avatar and the robot and clear representations of facial and non-verbalexpressions of the teleoperator. In the following sections, we outline the designchallenges, present the design elements we opted for in our problem context, andrationalize our choices against the alternatives.3.2.1 Motion Mapping and SynchronizationThe primary objective of AvatARoid lies in achieving a realistic and responsivesynchronization between the humanoid robot and the AR avatar, particularly inmatching the body articulation, i.e., joint position and motion. This requires ad-dressing the well-researched correspondence problem [37] in robot teleoperation,which entails mapping the teleoperator’s human-like motion onto the mechanicalconstraints of a robot. In AvatARoid, the teleoperator’s motion should be accu-rately and intuitively reflected in both the robot and avatar, providing an integratedrepresentation of the teleoperator’s motion.In our design, we have adopted upper body retargeting and Inverse Kinemat-ics (IK), prioritizing end-effector targeting with precision within the on-site user’sphysical space [30]. IK is a computational technique used to calculate the joint con-figurations of a robot arm that would enable the end-effector (typically the robot’shand or manipulator) to reach a desired position and orientation in physical space.This is achieved by reversing the kinematic equations of a robot’s IK chain to de-termine the joint poses that provide a desired pose for the end-effector. We aimto ensure the end-effector joint position of the teleoperated robot aligns with theavatar’s corresponding joint, essentially the ‘hands.’ This reflects the teleopera-tor’s hand movements in real-time. By making the robot’s arm IK chain target theavatar’s hand position, which follows the real-time movements of the teleoperator’shand, we can achieve spatial congruency. Occasional misalignments between therobot’s full joint sequence and the avatar’s limbs are inevitable due to the differ-14ing kinematic structures. Although all the joints in the IK chain may not alwaysalign with the hand of the avatar, we believe our approach of placing emphasis onthe motion of end-points of the robot in the on-site user’s space, where interactionwith the environment and on-site user typically occurs, provides a clear under-standing of the correspondence between the two entities and predictability of therobot arm’s trajectory relative to the avatar hands. We believe aligning the end-effector and joint positions accurately facilitates a more intuitive understanding ofshared actions between the avatar and the robot. This might assist the on-site userto perceive the robot and avatar as a single, harmonious entity moving in space,bolstering the on-site user’s sense of teleoperator embodiment.Alternative techniques to motion mapping in robotic telepresence include pre-rendered motions using gesture detection [9]. This method classifies the teleopera-tor’s movements into distinct gestures, with each gesture triggering predeterminedanimations in both the robot and avatar. However, this can limit the teleoperator’sexpressiveness and risk a sense of disconnection, as it may not mirror the teleoper-ator’s movements in real time. We also considered replicating joint angles directlythrough forward kinematics, a fundamental technique in robotics that calculatesthe position and orientation of the robot’s end effector based on predefined jointangles [30, 51]. Forward kinematics involves using the robot’s joint parameters(angles and displacements) to compute the position and orientation of the end-effector (e.g., the robot’s hand) in the robot’s coordinate frame. This is done bysequentially applying rotational and translational transformations from the robot’sbase to the end-effector, following the robot’s joint hierarchy. In the context ofAvatARoid, the challenge with applying forward kinematics is the disparity be-tween the kinematic structure of a human’s arm and the arm of the humanoid robotwe are considering. The robot’s arms do not provide the same degree of freedomor range of motion as human arms. Therefore, the spatial positions of the robot’send effector and the avatar’s end-points might not match, leading to potential dis-crepancies during direct physical interactions between the two users, such as fistbumps, handshakes, or object handovers. Such mismatches could disrupt the per-ceived unity of embodiment between the teleoperator’s avatar and the robot.Our AvatARoid system’s primary goal is not only the replication of movementbut the creation of a seamless and convincing representation of the teleoperator’s15presence within the robot-mediated environment. For our current implementation,we believe having the robot’s hand consistently follow and match the avatar’s handwith precise end effector alignment is critical to establishing and maintaining aclear understanding of the relationship between the two entities and also to at-tribute the agency of motion of the robot’s arm to the teleoperator. As humanoidteleoperation is an actively studied field, we are certain future implementations canexplore the integration of more sophisticated algorithms for even finer motion pre-diction, capture, and replication, ultimately closing the gap between human androbot teleoperation.3.2.2 Overlay Avatar RenderingAvatar-based telepresence systems have explored multiple techniques for gener-ating and displaying the avatar of the remote collaborators [12, 84, 91, 96]. Ourobjective with this design element is to create a realistic and expressive represen-tation of the teleoperator avatar, capable of conveying wide spectrum of emotionalnuances and subtleties of gestures. Multiple studies reviewing the visualization ofAR avatars and their effect on social presence have reported improved ratings ofsocial presence using a realistic avatar capable of rendering nuanced non-verbalcues such as facial expressions and gaze [83, 85, 121].Our system employs an approach that leverages a front-view upper body point-cloud avatar generated in real-time through a single RGBD (Red, Green, Blue,and Depth) sensor. The RGBD sensor captures the teleoperator’s color (RGB) anddepth (D) information simultaneously. This captured information enables us toconstruct a volumetric avatar, a three-dimensional, color-enriched point-cloud rep-resentation of the teleoperator, which we can overlay on the robot. This point cloudavatar is then superimposed onto the physical robot, offering a nuanced portrayalof the teleoperator’s facial expressions and body language—crucial elements foreffective social interaction. Findings of previous studies also inform our designchoice, suggesting point cloud avatars to be a better visualization method for so-cial presence in AR as well as full or upper body realistic avatars being the mostpreferred in collaborative AR settings [121, 126].While we explored the possibility of using a pre-rendered rigged 3D avatar16generated from a static image as suggested in prior work [91, 96], we ultimatelydecided against it. While it is easy to create and promising in terms of consistency( having no rendering artifacts), the key limitation of this approach is its inabilityto capture the dynamism and intricacies of facial expressions accurately. Further-more, such a static facial representation can risk invoking the ’uncanny valley’phenomenon when striving for photorealism, making the avatar appear eerily life-like yet not convincingly human [70]. Therefore, our system focuses on real-timedata capture and processing to generate a more authentic and interactive avatar,fostering a more engaging social experience.We circumvent the pitfalls associated with pre-rendered 3D avatars by optingfor a real-time point-cloud approach. More sophisticated real-time avatar creationtechniques exist [84] with high-fidelity facial expressions; they require a complexcapture setup involving rigs containing multiple cameras and complex technicalsetup, which is out of our scope for an initial conceptual evaluation of AvatARoid.Our current system dynamically captures and processes RGBD data from a singlesensor to create a responsive and engaging avatar.17Chapter 4AR Simulation for AvatARoidEvaluationWe wanted an initial conceptual evaluation for AvatARoid and implemented oursystem with a virtual robot within an immersive AR simulation as an early pro-totype. For our evaluation, since we are primarily concerned with how differentuser representations of the teleoperator affects the on-site user’s perceived embod-iment, social presence, and impression of the teleoperator, we believe simulatingdifferent user representation with a visually realistic virtual robot should provide afair and robust framework for comparing difference engendering in these measuressolely due to difference in user representation of teleoperator. Implementing a fullyfunctioning AvatARoid system also poses significant financial challenges due to itscomplex, multi-faceted architecture. Furthermore, modifications due to technicalissues during evaluation could also skew study results, as shown in a previous studywhere the result was confounded due to changing robots midway through the studyafter one robot broke down. [118].Studies comparing the validity of the use of immersive simulations for HRIevaluation using virtual robots have shown promising results for their acceptabil-ity. After storytelling interactions with its physical and virtual versions, a studyassessed children’s engagement and proxemic preferences with a humanoid robot,Arash [103], and found the social experiences with both physical and virtual formswere comparable. Similarly, a study comparing the effects of 2D, 3D, VR, and18physical interaction modes with a humanoid robot suggests a robot in immersiveVR could be feasibly used to replicate the experience of verbal interaction withan actual robot in HRI studies [75]. VR simulations have also been validated asa robust tool for gauging social interactions between humans and non-humanoidrobots as well [99]. Some of the studies comparing the acceptability of human-robot collaboration with VR and real robotic manipulators found that VR can aid inpreliminary assessment through questionnaires [34, 88, 122]. Another study com-paring participants’ sense of security while collaborating with physical and virtualmobile robotic arms during different interactions found security ratings from VRencounters to mirror those from physical robotic arms closely [46].While showing promising alternatives for large-scale HRI studies, simulationsusing VR environment need to consider the compressed distance perception in VR[47], the realism of the simulation, and the sense of presence of the participants.We designed our simulation to have a visually realistic-looking and behaving robot,placed the virtual robot carefully to avoid making the user uncomfortable, and em-ployed various mechanisms to ensure the participants felt the robot’s presence.Some studies have suggested these factors have potential effects to cause differ-ences in the participants’ experiences of interacting with real and virtual humanoidrobots. A study involving a humanoid robot approaching the participants indicatedthat participants felt more discomfort and desired a greater personal distance wheninteracting with the virtual robot compared to its physical counterpart [68]. Au-thors attributed these differences to compressed distance perception when usingan VR HMD and the low realism quality of the virtual robot model. Moreover,participants’ sense of presence is also important, as another study suggested thatparticipants diminished sense of presence in VR made virtual robots less “social”than real robots [125].AR seamlessly blends virtual and real worlds, offering a middle ground thatmaintains the physical context while introducing virtual elements. This integrationis critical for telepresence robots, where having the teleoperator’s presence withinthe real environment is essential. Unlike VR, AR does not isolate the user fromtheir surroundings and enables the crucial sensation of being in a real environment.Therefore, we propose a novel approach using AR simulations, which we believeis more tailored for telepresence systems.194.1 Environmental SetupThe teleoperator and on-site user setup in our AR simulation is shown in figure 4.1.The on-site user was seated on a sofa, wearing an AR HMD and interacting withthe virtual humanoid robot. The teleoperator controlled the robot’s movements andspeech from a separate room. As the on-site user engaged with the robot, the tele-operator viewed them through a camera placed in the participant’s room, and thefeed was displayed on their monitor. In contrast, the on-site user could see the tele-operator presented with the robot via the headset’s passthrough AR display. Theheadset’s speaker and microphone facilitated communication between the teleoper-ator and the on-site user. Although the virtual robot in the simulation restricted usfrom enabling physical interaction typically enabled by robotic telepresence sys-tems, it helped us simulate a verbal interaction between the teleoperator and theon-site user, which enabled us to capture differences between different teleopera-tor representations in robotic telepresence systems.4.2 Mechanisms to Enhance Co-presence with VirtualRobot in AR SimulationAchieving a sense of presence is vital for validating any simulated HRI study [33].For our study, since we augment the physical environment with a virtual robot, weaim to maximize the sense of co-presence with a virtual robot, where users feel theyshare a physical space with another entity [57]. Thus, we employ five mechanismspreviously identified to augment users’ sense of co-presence with a virtual entity inAR while also addressing potential limitations of immersive simulations discussedearlier to enhance the visual and behavioral plausibility of the virtual robot.Size and position of the robot: We used a realistic 3D model of the commercialsocial robot. The robot model was scaled to an approximate human height of 1.65meters as we envision AvatARoid to ensure a human-like presence for teleopera-tors and on-site users (figure 4.2). We positioned it 3 m, slightly further than theaccepted social distance, to account for the distance compression often reported inMixed Reality studies [47].20(a) The participant setup in our lab (b) The participant’s point of view as seen on their headset(c) The Teleoperator Setup (d) The Teleoperator’s Point of View as seen on their screenFigure 4.1: Our study setup for the on-site user a) participant setup to beseated in our lab wearing headset, b) the participant’s point of view ofthe teleoperator in AvatARoid condition as seen from the headset, c) theteleoperator setup in a different room, and d) the teleoperator’s point ofview of the participant as seen in their screen.Displaying spatial awareness: A virtual entity’s awareness of and interactionwith real-world spaces and objects contributes to the shared presence [29, 56, 57,90]. We designed our virtual robot to display spatial awareness; for instance, thevirtual robot used real doors for entering and exiting rooms (figure 4.3). The ex-perimenter also acknowledges the robot’s presence in the room. The experimenterinstructing the participants during study sessions also avoided the robot while mov-ing towards it and looked at it while addressing the teleoperator. We also carefullydesigned the narrative of our study task, including the word “couch” in the word-21guessing game (see Chapter 5 for details), allowing the teleoperator (and hence thevirtual robot) to point toward an actual couch in the room.Blending with the real world: Studies indicate that visually and physically plau-sible visuals and behaviors—like occlusion or avoiding passing through real-worldobjects—enhance the sense of co-presence with virtual agents [53, 55]. For blend-ing with real-world lighting, we used a virtual directional light in the game engineused for developing the simulation to light the robot, matching the position, color,and intensity of the real-world light in the room. This directional light in the gameengine facilitated the lighting and casting of shadows for the virtual robot to blendin with the lighting of the real environment. We used a noise shader for the virtualrobot’s material in the engine to simulate a grainy effect matching the surroundingenvironment, as seen through the passthrough AR HMD’s display. The headset didnot have sensors to provide depth information of the scene, so we used an APIprovided by the manufacturer to create the room geometry manually and used cus-tom shaders to provide occlusion from the real-world objects (figure 4.4) and castshadow using the virtual robot on the real room floor (figure 4.5).Realistic motion: A humanoid robot’s motion must account for its mechanicallimitations, such as the range and speed of its joint movements [75] for behavioralplausibility of the robot’s simulation. We used manufacturer-provided joint speci-fications to simulate the virtual robot’s motion realistically. We confined the virtualrobot’s arm and head movements to realistic ranges by modeling its joint and speedlimitations using the IK solver package. The robot’s locomotion was animated andslowed down to make the robot appear slow while entering or exiting.Realistic auditory cues: As auditory cues (such as spatialized audio) have beenwell established as a crucial factor affecting presence in virtual environments [40,59, 72], a generic servo audio effect simulated a robotic sound when the robot orits arm moved. The game engine’s built-in audio spatialization feature was used tospatialize teleoperator voice and robot-generated sounds.We implemented these elements in an immersive AR simulation as a cost-22Virtual robot wavingFigure 4.2: Virtual robot waving as seen from the headseta) Virtual robot entering the room b) Virtual robot leaving the roomFigure 4.3: Virtual robot entering and leaving through the real dooreffective and rapid alternative to real-world evaluation. The AR simulation ef-fectively facilitated the evaluation of AvatARoid, retaining the connection to thephysical environment, which is essential for simulating telepresence scenarios. Wediscuss the viability and effectiveness of this simulation platform in chapter 7 (sec-tion 7.2).23a) Virtual robot occluded by real wall while enteringb) Virtual robot occluded by real wall while leavingFigure 4.4: Virtual robot being occluded by real wallVirtual robot casting shadow on the room floorFigure 4.5: Virtual robot casting a shadow on real floor24RGBD DataJoint DataAudio Avatar DisplayRobot ControlVoice CommunicationOn-site User in AR HeadsetRGBD SensorVoice CommunicationRGBD +Joint DataNetworkAudio Teleoperator Display Interface On-site User Video CaptureSensor Data CaptureOn-site User VideoRGB CameraOn-site User VideoTeleoperatorHumanoid RobotTeleoperator SystemOn-site User Vide DisplayOn-site User SystemFigure 4.6: System architecture of the AvatARoid system, software modulesshown in a rectangle: On the left, the teleoperator system consists ofmodules to capture and stream sensor data, enable voice communica-tion, and display the on-site user video. In the middle, the networkenables data communication between the two users. On the right, theon-site user system consists of modules to create and display the pointcloud avatar, a robot control system to send motion commands to therobot, and a voice and video capture module for audio-visual communi-cation from the on-site user.4.3 System Architecture of Simulated Implementation ofAvatARoidWe illustrated the architectural design of our simulated AvatARoid system in Fig-ure 4.6. The teleoperator system consists of an Azure Kinect Sensor and moduleto capture and stream the RGBD and joint data of the teleoperator, which facili-tates the creation of a dynamic point-cloud avatar and precise skeletal joint trackingfor motion synchronization. The teleoperator system also contains modules ded-icated to receiving and processing audio and video streams from the on-site userfor audio-visual communication. A screen display unit allows the teleoperator toengage with the on-site user and their environment visually.The on-site user’s system is designed to receive the RGBD and joint data fromthe teleoperator system over a network connection. The robot control module usesan IK algorithm to transform the teleoperator’s hand joint position data into action-able motion commands for the robot arms, thus enabling the humanoid robot arms25Joint dataPoint cloud dataDepth camSkeletal trackingRGB-D framesTeleoperatorPoint-cloud avatar forAR displayJoint data forrobot controlOn-site userAvatARoidFigure 4.7: Information flow in the AvatARoid system: On the left, the tele-operator system captures the teleoperator’s motion and RGBD imagesusing a depth cam. In the middle, the joint and point cloud data aretransmitted to the on-site user system. On the right, the on-site user sys-tem utilizes the joint data to control the humanoid robot, while the ARheadset displays a point cloud avatar overlaid on the robot.to mirror the teleoperator’s gestures. We only control arm chains of the humanoidrobot for our implementation and treat each arm as a separate IK chain. We employthe widely used Cyclic Coordinate Descent algorithm [50] with each hand positionas the target position to the respective robot arms to solve for the orientations of alljoints in each arm’s IK chain. Simultaneously, the AR avatar display module utilizesthe RGBD frames to create the teleoperator’s point cloud avatar and superimposeit onto the humanoid robot in real-time, creating a unified visual experience foron-site users between the avatar and the humanoid robot. We employ a simplisticpipeline explained in a previous study [97] consisting of the acquisition of RGBDdata as a point cloud view of the sensor’s field of view, segmentation to filter outthe teleoperator and finally rendering after remapping to the headset’s coordinatesystem. To ensure the teleoperator’s avatar aligns with the humanoid robot’s pro-portions, we employ a pre-calibration process to match the scale of the capturedRGBD and motion data with the robot’s and synchronize the skeletal and RGBDframes. These frames, RGBD to create the point cloud avatar and skeletal to sendmotion commands to the robot, are timestamped and buffered, guaranteeing thatthe avatar’s movements are in sync with the robot’s actions. As we simulated therobot within a 3D game engine, we used the game engine’s transform hierarchy to26match the AR overlay and the robot’s pose in 3D space. When using a real robot,the robot’s pose can be tracked using image/marker systems or external trackers inreal-world scenarios to superimpose the avatar. The system’s voice communicationmodule captures and streams audio for clear verbal interaction with the teleoper-ator. Similarly, the video capture module is responsible for streaming live visualsof the on-site user and their immediate environment back to the teleoperator. Fig-ure 4.7 shows the information flow within the AvatARoid system, depicting ourkey design elements for motion synchronization with avatar overlay described inchapter 3.4.4 ImplementationWe used a 3D model of the commercial social robot Pepper, developed by Soft-bank Robotics [3] as our robotic platform. We utilized the Unity3D game engine(ver.2021.3f) as a software platform to implement our simulation system. Theteleoperator system ran on the Unity Editor running on Windows 11 OS. Integra-tion with the Azure Kinect device was facilitated through the Azure Kinect SDKver.1.4.1 and the AzureKinectExamplesForUnity package [5], enabling the captureand transmission of RGBD sensor frames and skeletal tracking data to the on-siteuser system. We connected a camera in the on-site user’s room to the teleoperatorsystem with a 10m USB cable, providing a view of the on-site user’s environment,including the robot and the user on a screen. Vuforia [4] was used as a AR plugin todisplay the camera feed and periodically calibrate the camera with image-trackingto align the virtual robot displayed to the teleoperator with its counterpart in theon-site user’s simulation. The teleoperator was in a standing position and used thekeyboard as a control interface to trigger animations for the robot’s locomotion andevents within the experiment.We used Oculus Integration API to develop the on-site user system on a MetaQuest Pro passthrough AR headset. The Scene Understanding API was used tocreate the room geometry for the experiment room where the simulated telepres-ence interaction occurred. Participants were seated and wore the headset to displaythe simulated robot and interacted with the teleoperator. The robot’s arm move-ments were controlled using the Cyclic Coordinate Descent IK component of the27FinalIK package [1], driven by the teleoperator’s hand poses after transformationto match the robot scale and position. Pepper robot’s kinematics data [2] was usedto limit its motion. Using the Rotation Limit component of the package, we con-fined Pepper’s arm and head movements to realistic ranges. The robot arm’s speedwas controlled by limiting the angular speed of the joints. The RGBD data fromKinect was either used to display a video of the teleoperator’s face onto the robot’storso tablet or to project the teleoperator’s point-cloud avatar on the robot. We usedthe transform hierarchy of the unity to transform the point cloud and align it to therobot’s position and scale.We used Photon Unity Networking for voice communication. The Azure Kinectsensor data was streamed over a Local Area Network (LAN) using componentsprovided by the Kinect Unity package.28Chapter 5Evaluating AvatARoidWe conducted a between-subject mixed-method lab study to evaluate the efficacyof the AvatARoid system in comparison to two other existing representation tech-niques. Our design was a one-way factorial, comprising three conditions, as illus-trated in Fig 5.1: (1) Baseline (BL), where only the humanoid robot was presented;(2) Video (VD), where a tablet was attached to the robot’s torso, displaying a livevideo feed of the teleoperator’s face, and (3) AvatARoid (AR), our system as de-tailed in Chapter 3. Many existing robotic telepresence systems use video [61] orjust humanoid robot [6, 17] as teleoperator representation, making them good alter-natives to compare AvatARoid with. Participants were randomly assigned to one ofthese three conditions. The study protocol received approval from the University’sInstitutional Review Board.The experimental system was designed to simulate a representative robot-mediatedtelepresence setup. Our participants played the role of the on-site user. A memberof the research team, acting as a confederate, took on the role of the teleoperator.To ensure enhanced experimental control and standardization, we employed a con-federate alongside a deception protocol, a common approach in HRI studies [93].The confederate teleoperator was unaware of participants’ conditions to avoid biastoward the three conditions. Following the experiment, participants were debriefedabout the role of the confederate and the deception protocol.29(a) Baseline (b) Video (c) AvatARoidFigure 5.1: Three user representations used in our evaluation: (a) Baseline: ahumanoid robot, (b) Video: a humanoid robot with a teleoperator video,and (c) AvatARoid5.1 ParticipantsWe recruited 48 participants (16 per condition) through the university’s onlinestudy advertisement board (See appendix A.1 for the recruitment advertisement).The gender demographics comprised 33 women, 14 men, and one non-binary indi-vidual. The age distribution was diverse, albeit with a slight skew towards youngerdemographics: 31 participants were aged 19-24, 12 were between 25-34, 2 werebetween 35-44, and 3 were between 45-54. Each participant received a $50 hon-orarium upon completing the study. 32 participants had no prior experience withrobots, 10 had minimal prior interaction, mostly with service robots, and 6 had ex-tensive interaction with robots, working or having worked previously with them insome capacity. 26 participants reported no prior usage of AR/VR devices, 17 hadminimal familiarity, and 5 had ample experience either owning a headset or usingthem regularly.5.2 TasksOur participants performed three tasks.Investment game. We used the investment game, based on the trust game [10],to measure participants’ trust levels. Participants started with 5 dollars and couldinvest between 0 and 5 in increments of 1 dollar. They were told the investedamount would be doubled and passed to the teleoperators, who then chose howmuch to return. This was played pre and post-interaction with the teleoperators. To30incentivize the participants, they were initially told that their participation wouldyield a 30-dollar compensation, to which earnings in two rounds of this task wouldbe added. However, at the end of the study, everyone received an additional 20dollars regardless of their game decisions.Get-to-know-you interview. We had a casual structured conversation in whichparticipants answered the questions from a predetermined set and asked them backto the confederate. The confederate’s responses were loosely scripted to not biasthe participants’ impression towards them. We followed a turn-taking approachfor an entire set to avoid participants mirroring the confederate’s scripted disclo-sure of the questions, as disclosure in this conversation was one of our behavioralmeasures. The questions were displayed on a monitor screen for the teleopera-tor confederate and in a virtual text panel on the room wall for the participants.Participants used the headset’s controller to navigate the questions.Collaborative word guessing game. Participants teamed up with the confed-erate for a word-guessing game. The confederate had a list of words to describeshown to them on a screen, and participants attempted to guess as many words aspossible within 3 minutes. This task modeled collaborative activities common intelepresence settings.5.3 ProcedureThe participant first signed the consent form (see appendix A.2) and then proceededto complete a demographic survey before the study (see appendix A.5). Uponarriving at the venue, the experimenter escorted the participant to the study room.Here, they were oriented about the study’s objectives and procedure. The overviewof the procedure is shown in figure 5.2 (See A.3 and A.4 for procedure details forthe experimenter and teleoperator confederate of each session). Each session tookabout an hour. Our procedure encompassed the following four steps:Playing the 1st Investment Game and Entering the Simulation. The exper-imenter explained the rules of the investment game. The participant was thenhanded the money and two envelopes, each corresponding to a particular decision.After their decision, the participant was briefed about the headset, its adjustmentfeatures, and necessary safety precautions. The experimenter remotely launched31Meta-Humanoid Study: Task Flow SheetGreetings & Study BriefingStep 1: Investment GameStep 2: InterviewStep 3: Word Guessing GameStep 4: Investment GamePost-Study Survey & InterviewPut on ARHeadsetTake off ARHeadsetFigure 5.2: Overview of the study procedure session32the simulation application after the participants were ready and wearing the head-set. The teleoperator confederate initiated the robot’s entry animation once theirsystem signaled that the participant system was ready.Conducting the “Get to Know You” and Word Guessing Game Tasks. Next,the experimenter provided instructions for the “get to know you” task, demonstrat-ing how the participant could navigate through the questions using the thumbstickson the headset’s controller. This briefing concluded with the experimenter leavingthe room and the confederate initiating the task. The experimenter monitored thetask’s progress on a connected mobile device and re-entered the room upon com-pletion. Subsequently, the participant engaged in a word-guessing game with theteleoperator for 3 minutes.Exiting the Simulation and Playing the 2nd Investment Game. Following theirfinal greetings, the confederate activated the robot’s exit animation sequence. Par-ticipants were then advised to remove the AR headset and were invited to play inthe investment game once more, thus concluding the study tasks.Post-task Survey and Interview. Then, the participants completed a survey andinterview. Finally, participants were debriefed on the study’s deception protocol,including the role of the confederate and the fictitious nature of the trust game’sdecision on payment implications. The session culminated with participants re-ceiving their due compensation.5.4 MeasuresWe have determined the following measures to assess the effectiveness of the userrepresentation in AvatARoid. This evaluation focuses on the on-site users’ percep-tion of the teleoperator’s embodiment, social presence, and impression.5.4.1 Subjective MeasuresWe used questionnaires for subjective measures of on-site users’ sense of the tele-operator’s embodiment, social presence, and overall impression. We also includeda questionnaire to measure the sense of co-presence with a virtual robot. OurQualtrics survey contained seven questionnaires.Embodiment. We used the perceived embodiment of the teleoperator using336 Likert-scale statements (Cronbach’s α = 0.7), each paired with a seven-pointranking scale to indicate the level of agreement ranging from 1(strongly disagree)to 7(strongly agree). The statements were adopted from a proposed self-avatarembodiment questionnaire in VR [39]. The questionnaire contained two questionsfor the perceived agency (I felt like I talked to a real person/robot), 2 for perceivedcontrol (I felt like the teleoperator/robot was controlling the robot’s movements),and one each for ownership (I felt like the robot’s body was teleoperator’s body)and co-location statements (I felt like the teleoperator was located where the robotwas), as these measures are key aspects of embodiment [39]. The wording waschanged to reflect the perceived embodiment of the teleoperator by the on-site users(see A.6.1 for the perceived embodiment questionnaire).Social Presence. We assessed social presence using six semantic differentialitems (Cronbach’s α = 0.82), adopting the measure used in [82]. This question-naire measures a user’s perception of the medium’s capacity to have authentic so-cial interaction closer to face-to-face interaction, with a higher score meaning themedium’s ability to present the other as a social entity (see A.6.2 for social pres-ence questionnaire).Impression of the Teleoperator. For the impression of the teleoperator, weused four questionnaires to measure on-site users’ impressions regarding trust-worthiness, perceived similarity, likeability, and infrahumanization towards theteleoperator. For trustworthiness, we used 14 trust-related bipolar adjective pairs(Cronbach’s α = 0.91) from [123]. Perceived similarity consisted of nine adjec-tive pairs (Cronbach’s α = 0.83) (for appearance and behavior-related similarity)in a semantic differential scale adopted from [78]. For likeability, we used [95],which contains 11 Likert-scale statements (Cronbach’s α = 0.88) about how warm,friendly, likable, etc. the other person seemed. All of these questionnaire itemswere scaled from 1-7. We averaged the responses to get a final score between 1and 7, with higher scores denoting a higher positive impression of the teleoperator.For infrahumanization, we had participants select 8-10 emotions from 16 emotionschosen from prior work [93], which categorized and validated them as either non-uniquely or uniquely human emotions. The count of non-uniquely human emotionsgives us the measure for infrahumanization. The more non-uniquely human emo-tions they selected, the more they felt the teleoperator was less capable of uniquely34human emotions (see A.6.3 for all impression questionnaires).Co-presence with the Virtual Robot. To verify that the participants felt presentwith the simulated virtual robot, we employed a co-presence questionnaire to mea-sure the extent to which the participants thought they were in the same space asthe virtual robot. We used the questionnaire (Cronbach’s α = 0.77) designed inprevious study [57], which contains 3 Likert-scale statements with a 7-point rank-ing scale and four items on a semantic differential scale. The final two items aresubtracted and their differential taken as a single measure (see A.6.4 for full ques-tionnaire) similar in [57].5.4.2 Behavorial MeasuresWe also implement the following measures based on participant behaviors.Gaze duration. We used gaze duration as a measure of visual attention [7,124] allocated to the head of the robot and the teleoperator representation. Weutilized the headset’s eye-tracking hardware and Unity’s collider to record the timeparticipants fixated their gaze on different robot parts and the representation.Word Count. Similar to previous studies [93], we employed length of response(in terms of word counts) in the “get-to-know-you” task as behavioral measuresto test trust [123] and likeability [28]. We modeled our first task with questionsgradually increasing in scale of personal information similar to previous disclosure-related tasks [93] and measured word count as our disclosure measure.Gain from the Investment Game. We treat the money invested in the first roundas trust towards a stranger and money invested in the second round as trust towardsthe interaction partner, using the difference of the two rounds as trust gained dueto the interaction with the partner, similar to prior studies [94].5.5 AnalysisFor a holistic view of our measures of interest, we integrate the quantitative datafrom questionnaire responses, behavioral measures, and qualitative data from in-terview responses to identify our findings. We present our findings based on theanalysis and triangulation of the multimodal data we collected during our study,including the participant’s behaviors and languages where the patterns were con-35sistently observed.Questionnaire Response. We used Cronbach’s alpha to measure the reliability.We used Analysis of Variance (ANOVA) to compare the effects of user representa-tions on the on-site user’s sense of embodiment, social presence, and impression ofthe teleoperator. We use TukeyHSD post-hoc pairwise comparisons if the ANOVAresults are significant. If significant, we also use ordinal logistic regression toanalyze different items in the questionnaire to identify the effect of different repre-sentations on specific items within the measure. To maintain consistency betweenour statistical analysis and data visualization, we use bar plots to illustrate ANOVAresults when comparing means. We use box plots to showcase item-level variationsin measures when relevant, emphasizing the frequency or likelihood of participantresponses in specific items. For co-presence, we conduct descriptive analysis andvisualize all participants’ frequency distribution of co-presence scores.Behavioral Measures. We use ANOVA followed by Tukey HSD post-hoc com-parisons if significant for our behavioral measures of word count and gain in invest-ment. We combine descriptive analysis of gaze data to supplement our interviewand survey data results.Interview Response. We transcribed the recorded interview audio, then ana-lyzed and coded the responses to discern any patterns, following Braun and Clarke’sthematic analysis approach [22]. The qualitative data were analyzed until satura-tion, and no new data observations contradicted our findings. Our findings fromthe quantitative analysis were also triangulated using these findings from the qual-itative analysis. This complements the statistical results and offers insights andexplanations when the quantitative data is inconclusive.36Chapter 6FindingsWe found strong empirical evidence supporting the credibility of our simulationapproach. Most participants (38 out of 48) rated the co-presence of the robot atfour or higher on a seven-point Likert scale (M = 4.86). In post-study interviews,participants highlighted the act of a “robot coming in the room,” and the robotbeing “visually embedded in the room” made them feel that the “interaction washappening in this room.” These results pave the way for assessing other measureswe discuss below.*0246Baseline Video AvatARoidMean ValuePerceived EmbodimentFigure 6.1: Mean embodiment score across three user representations; (*)indicates a significant difference (p <.05).37*Control by Teleoperator Robot Body as Teleoperator Teleoperator Robot ColocatedAgency of Robot Agency of Teleoperator Control by RobotBaseline Video AvatARoid Baseline Video AvatARoid Baseline Video AvatARoidBaseline Video AvatARoid Baseline Video AvatARoid Baseline Video AvatARoid24624634567246246246Item Value (1−7)Ratings for individual items in Embodiment questionnaireFigure 6.2: Item-wise rating scores of the embodiment questionnaire; An as-terisk (*) indicates a significant difference based on ordinal logistic re-gression (p <.05).6.1 AvatARoid Amplifies the Embodiment of theTeleoperator for On-site Users.A one-way ANOVA revealed a significant difference in perceived embodiment lev-els among the three conditions, F(2, 45) = 4.029, p = .025 (See Fig 6.1). Post hoccomparisons using Tukey’s HSD indicated that the mean score for the AvatARoidcondition (M = 5.07, SD = 0.68) was significantly higher than that of the Baselinecondition (M = 4.12, SD = 1.00), p = .02. However, the Video condition (M = 4.45,SD = 1.16) did not significantly differ from the Baseline condition, p = .174. Anitem-wise questionnaire analysis determined specific aspects of the embodimentwhere the AvatARoid condition excelled. Fig 6.2 presents a box plot of responsesfor each item in the embodiment questionnaire across the three conditions.6.1.1 The Colocation, Control, and Agency Perceptions inAvatARoid Contribute to a Stronger Sense of Embodiment.AvatARoid enhanced participants’ perceptions of co-location and control over therobot, significantly compared to the Baseline condition, resulting to a strongersense of the teleoperator’s embodiment than both the Video and Baseline.38Co-location Participants were more likely to feel that the teleoperator and robotwere co-located in the AvatARoid condition, contributing significantly to their in-creased sense of the teleoperator’s embodiment. Ordinal logistic regression re-vealed a significant enhancement in co-location between the teleoperator and therobot for the AvatARoid condition compared to the Baseline, b = 1.62, SE = 0.64,z = 2.52, p = .012. The Video condition showed a non-significant increase, b =1.06, SE = 0.66, z = 1.60, p = .111 compared to Baseline.Participants consistently indicated in their interview responses that the AvatARoidsystem enhanced their sense of co-location of the robot and the teleoperator. P22ARfelt like the teleoperator was “conversing face-to-face in the same room,” a senti-ment nine other participants in the AvatARoid condition shared. This heightenedsense of spatial presence starkly contrasted experiences in the Video and Baselineconditions. Confusion prevailed in the latter, as participants expressed wondering,“Where is the actual person?” during the interaction. While participants in theVideo condition acknowledged a slightly enhanced presence compared to standardvideo conferencing, many still felt disconnected. Specifically, most participants(P3, P16, P21, P24, P28, P31, P39, P45, P47) in video condition expressed thesentiment of speaking to a screen rather than a present individual. They notedseeing them confined in the screen made the teleoperator feel distant.“It felt like she was in the same room as me. And we were talkingface to face.”(P29AR)“I was talking to the person on the screen, but the robot was theactual thing that was in the room, not the person.” (P39VD)“I just kept questioning myself...where’s the real person?” (P22BL)Control AvatARoid seems to enhance participants’ perception of the teleopera-tor’s control over the robot’s movement while diminishing the robot’s perceived au-tonomy. Ordinal logistic regression analysis suggested that, compared to Baseline,the AvatARoid condition approached significance in predicting both a heightenedperception of control by the teleoperator over the robot’s movements, b = 1.15, SE39= 1.81, z = 0.36, p = .071 and reduced perceived robot autonomy, b = -1.16, SE= 0.64, z = -1.83, p = .067. For the Video condition, no trend toward significanteffects was found on either perceived robot autonomy, b = -0.5005, SE = 0.649, z= -0.770, p = .441 or teleoperator control, b = 0.232, SE = 0.638, z = 0.365, p =.716.Most participants (14) in the AvatARoid condition felt the robot was beingcontrolled by or “mimicking” the teleoperator. P4AR stated, ”Watching the robotmove its arm to follow the teleoperator arm made the two feel connected.” Incontrast, 8 participants in Video and 9 in Baseline Conditions reported ambiguity,with P16VD mentioning that the robot could have been following the teleoperatoror programmed to move its arm automatically and P46BL feeling someone else (theexperimenter) could have controlled it.“Because I saw she was moving (her arms) and the robot was justfollowing, so I didn’t think the robot was the one controlling.” (P22AR)“I thought it was (moving by itself), or might have been a combi-nation of two (automatic and teleoperator controlling).” (P16VD)“I did not think it was the other participant (confederate teleoper-ator) controlling the robot, maybe another researcher, or maybe you(the experimenter) were.” (P46BL)Human Agency Ordinal logistic regression revealed that the type of representa-tion did not significantly influence participants’ perceptions of human agency, nei-ther in terms of feeling they interacted with a robot or with a real person. For theperception of human agency, both the video condition, b = -0.4637, SE = 0.7125,z = -0.651, p = 0.515, and the AvatARoid condition, b = b = -0.1333, SE = 0.6806,z = -0.196, p=0.845, had no significant difference. Test for robot agency also didnot show any significant differences for AvatARoid, b = -0.70454, SE = 0.63827,z = -1.104, p = 0.270 or Video, b = -0.09897, SE = 0.64669, z = -0.153, p = 0.878,when compared to Baseline.Although not supported by their survey response, interview responses of par-ticipants in the Video and Baseline conditions revealed their challenges in dis-40cerning the human agency, adding a layer of confusion that was notably absent inthe AvatARoid condition. P14AR mentioned they were suspicious initially that itmight have been just a robot; the overlay immediately cleared their doubt about thepresence of the teleoperator person. Notably, 7 participants in the Baseline and 5participants in the Video condition reported initial confusion about the agency oftheir communication partner and either employed strategies by “throwing a bait toconfirm its human” or remained confused throughout the study. As this tendencywas more common in Baseline and Video conditions, this confusion in perceivedagency might be another factor in lowering the sense of the teleoperator’s embodi-ment.“First I thought that a robot was coming without a person and Iwill be interacting with the robot, but when I saw (the teleoperator)was on top of the robot, I felt I was going to interact with a real personand not a robot.” (P14AR)“I felt she was an AI agent ... Maybe 70 percent of the time I wasthinking it was an AI agent and 30 that maybe she’s a real person.”(P47VD)“I did (think I was talking to a robot). And what’s interesting isthe voice didn’t really match the appearance.... if I had just heard thevoice, I would probably think it’s a human. But it was hard for me toget past the fact that she looked like a robot. So I felt like I was stilltalking to a robot.” (P1BL)6.1.2 The Dissonance of Perceiving the Robot and Teleoperator asDistinct Agents is Pronounced in the Baseline and VideoConditions, Potentially Diminishing the Sense of Embodiment.In examining participants’ visual attention allocation towards the teleoperator em-bodiment, our descriptive analysis of gaze duration on the robot head—expressedas a percentage of the total interaction time—revealed the following pattern: par-ticipants predominantly fixated on the robot head during the AvatARoid condition(M = 37.76, SD = 14.90), followed by Video (M = 26.13, SD = 21.46), and least41010203040Baseline Video AvatARoidPercentage of Total timeGaze Duration on Robot Head and VideoFigure 6.3: Mean gaze duration, expressed as a percentage of the total time,reflects the time participants spent looking at the robot’s head across allthree conditions or at the teleoperator video in the Video condition.on Baseline (M = 17.83, SD = 13.50). This was interesting as the participant’s facewas not displayed on the robot’s head but on the tablet attached to its torso in thevideo condition. Further examination of the gaze data for the Video condition cap-tures this split: participants spent 15.80 percent (SD = 16.18) of their gaze durationon the teleoperator video and 26.13 percent (SD = 21.40) on the robot head. Thisshowed that the participants perceived the robot and teleoperator as two distinctentities.In post-study interviews, many participants in the Video condition (11) re-ported viewing the robot as merely an inanimate object in the room, likening itto “just another piece of furniture” or “a wall,” serving only as “a support forthe screen displaying the teleoperator.” This highlighted their dual perception ofthe robot and teleoperator as separate entities. Similarly, 8 Baseline participantslikened the robot to “icons” or “webcams” during video calls, and P11BL statedthe inexpressive robot made the disconnect with the teleoperator apparent. Onlyone participant from the AvatARoid condition noted confusion due to the robot’sface, wondering if the robot’s face mirrored Andy’s (the Confederate teleoperator)expressions. However, most others using AvatARoid stated they were primarilyfocused on the teleoperator, mainly disregarding the robot.“And I didn’t exactly think the face and the robot were the samething.... because they are inside a screen in that compartment, it feltlike they were not here.” (P21VD)42“Sometimes Andy will be very lively, and the robot will be justlike (poker face), that’s two different things going on here.” (P11BL)6.2 AvatARoid Shows Potential to Increase Feelings ofSocial PresenceANOVA test to compare social presence for AvatARoid (M = 5.06, SD = 0.91),Video (M = 4.94, SD = 1.14), and Baseline (M = 4.44, SD = 1.22) did not showany significant effect, F(2,45) = 1.447, p = 0.246. We delved deeper into the inter-view reports to investigate the impact of different visual representations on socialpresence. We uncovered insights into AvatARoid’s potential to enhance socialpresence.6.2.1 Participants Found AvatARoid to be Personable and Intimate.Participants perceived AvatARoid as an interactive and emotionally resonant plat-form, enhancing the feeling of presence and connection. Seven participants envi-sioned its applications in personal events like family gatherings and intimate con-versations, especially across international borders. P27AR and P35AR highlightedits potential for close family interactions, while P48AR saw its value in scenariosdemanding a more profound sense of presence, such as long-distance relationships.P25AR mentioned it could be suitable for professional or even therapeutic scenariosas “you’re in the same room with everyone else.” P4AR further expressed contem-plating physical interaction such as hugs due to the combination of the avatar andthe robot. Overall, participants recognized AvatARoid as a tool transcending tradi-tional digital communication, offering a more intimate and embodied experience.“I felt it was much more interactive. For example, I felt like I couldstand up and maybe hug her; I may have been able to interact with therobot.” (P4AR)436.2.2 AvatARoid Provides Richer Non-verbal Reactions andGestures.Participants mentioned that AvatARoid offers a more expressive telepresence expe-rience than conventional video conferencing, enriching non-verbal communication.Participants valued its ability to convey nuanced reactions and body language, suchas hand gestures during greetings. This gestural communication added depth to in-teractions; many felt it was nearly akin to “in-person conversations.” Participantsstated discerning facial cues allowed them to “interpret feelings and reactions” inreal-time while promoting a more genuine engagement by being “able to read”the teleoperators more. P29AR’s statement reinforced this sentiment as they said itfelt like in-person interaction by mimicking genuine eye-to-eye connections. Be-yond gesturing ability, P27AR suggested that the combination of the avatar and therobot’s motions in AvatARoid added “more weight” to the interaction, enhancingthe “sense of presence” and “impact of responses,” making it the “closest thing youcan get other than being actually in front of them.” These observations underscoreAvatARoid’s prowess in delivering a comprehensive telepresence experience, com-bining the robot’s physical presence with facial expressions and body language toemulate authentic, emotionally fulfilling interactions.“I felt like there was more weight to the interaction than if I wasjust doing a Zoom chat... My sense of presence. Yeah, the impact oftheir responses to me, my voice, and it’s the closest thing I can get,other than being right in front of her.” (P27AR)6.2.3 Video and Baseline Conditions Lack the Emotional andInteractive Depth Achieved with AvatARoidContrary to the heightened connection and richer non-verbal engagement observedin the AvatARoid system, most participants in the video and Baseline conditionscompared their experience to the existing video/audio conferencing setup. Someeven reported feeling notably detached and limited in their interactive capabilities.Participants in Baseline conditions expressed that the absence of visual indicatorsfor emotions or facial expressions rendered the robot a mechanical stand-in ratherthan a meaningful extension of the remote participant. Participants in the video44considered the experience as “more fun” and “more present”; however, “not nec-essarily more interactive,” comparing it to a typical video call with the addition ofa robot. Some mentioned they felt the teleoperator was more present and expres-sive, with the arms adding non-verbal gestures to a regular Zoom call. Still, thissentiment was mixed as some said the robot’s limited movements did not provide areflective or emotionally responsive conduit for the teleoperator’s body language.“I feel like a big part of what made me feel like it was just a robotstanding rather than a good representation of the other participant wasthe fact that the robot had no like, no visual indication of the emotionsthe other participant is expressing.” (P11BL)“I don’t think I could gauge her body language from the robot’smovement. So maybe that could be where I felt like I was talking to arobot.” (P45VD)6.3 No Difference in Subjective and Behavioral Measuresof the Impression of the TeleoperatorANOVA tests revealed that the user representations did not significantly affect par-ticipants’ affective perceptions of the teleoperator. Specifically, the test results fordifferent impression measures were for: Trustworthiness, F(2, 45) = 0.217, p =0.806; Likeability, F(2, 45) = 0.189, p = 0.828; Homophily, F(2, 45) = 0.39, p =0.68; Infrahumanization, F(2, 45) = 0.694, p = 0.505; Gain in Investment Game,F(2, 45) = 0.755, p = 0.476; and Word Count in Disclosure Interview, F(2, 45) =0.145, p = 0.865.The ratings for Trustworthiness (for AvatARoid M = 6.01 , SD = 0.47, forVideo M = 5.87, SD = 0.76, for Baseline M = 5.92, SD = 0.68) and Likeability (forAvatARoid M = 5.69 , SD = 0.38, for Video M = 5.61, SD = 0.89, for BaselineM = 5.55, SD = 0.68) were high for all three visual representations. Almost allparticipants reported that their rating was based on the verbal interaction with theteleoperator and the teleoperator’s demeanor and stated they “seemed helpful inthe word-guessing game.” For perceived similarity (for AvatARoid M = 4.29, SD45= 0.84, for Video M = 4.08, SD = 0.82, for Baseline M = 4.32, SD = 0.82), partici-pants reported mostly rating based on the conversation and only weakly guided byappearance when visual cues were present. For infrahumanization( Avataroid M =3.5, SD = 0.73, Video M = 3.44, SD = 0.63, and Baseline M = 3.18, SD = 0.99),participants showed a tendency to assign positively valenced emotions, regardlessof the emotion being uniquely human or not.6.4 AvatARoid: Benefits and LimitationsThe overarching findings from our study suggest that integrating a dynamic avataroverlay and precise motion mapping in AvatARoid drives improvements in co-location, agency perception, and emotional connection with teleoperators.6.4.1 Overlay and Motion Mapping Enhanced the Sense ofCo-location.Our participants consistently reported that the overlay and synchronized motion ofAvatARoid enhanced their sense of co-location between the teleoperator and therobot. They recognized the overlay not as a detached hologram but as seamlesslyintegrated with the humanoid robot. The robot’s consistent presence mitigatedslight avatar imperfections, P27AR mentioning the robot’s hands “complementing”artifacts such as “clipping arm” due to rendering issues in AR avatar. Rather thanperceiving the robot as an isolated machine, participants described it as an integral“accessory” to the teleoperator, acting in perfect harmony. This sentiment of unitywas emphasized by participants like P14AR, who felt the avatar and the robot’scoordinated movements created a unified entity. This harmony between the tele-operator avatar and the robot effectively cultivates a sense of sharing a space witha complete, integrated entity, showcasing the AvatARoid’s capability to bridge thespatial divide in telepresence interactions.“Yes, because it (the robot) just felt a part of Andy. They felt bothas a part because whenever Andy did a movement, the robot also didthe same; they just felt like one entity.” (P14AR)46(a) The On-site User’s View of the Teleoperator(b) The Teleoperator’s View of the On-site UserFigure 6.4: A snapshot from our study session: Participant and Teleoperatormirror celebratory gesture after completing word guessing game a) Theteleoperator as seen in participant headset point of view, b) The partici-pant as seen in teleoperator screen6.4.2 Coordinated Motion between the Overlay and the RobotFacilitated the Perception of Agency and Control.The motion mapping between the overlay and the robot was pivotal in facilitatingthe perception of agency and control for on-site users. Synchronous movementsbetween the avatar and the robot provided a high level of connection between theteleoperator and the robot, making the participants feel the robot was “replicatingwhat the teleoperator was doing.” Participants’ post-study reflections highlightedthe critical importance of synchronized motion cues in the AvatARoid system inreinforcing the teleoperator’s gestures to the robot’s motions. The coordinatedmovements between the overlay and the robot consolidate the notion of agencyand control, making it clear that the teleoperator directs the actions.“I felt like it was a good one-to-one match. I don’t think the robotwas doing something that Andy wasn’t. So it was just giving me areinforcement of what Andy was gesturing to.” (P27AR)6.4.3 Real-time Facial Expressions and Upper-body GesturalFeedback of AvatARoid are Crucial for Making EmotionalConnections.Incorporating real-time facial expressions and upper-body gestural feedback in theAvatARoid system emerged as a crucial factor in forging an emotional connection47between participants by providing affordances such as reciprocating actions andmirroring gestures, for instance, celebrating after completing the game (Fig 6.4).Participants also highlighted the nuances of facial expressions and arm movementsin conveying emotions, explaining that the ability to observe the teleoperator’s fa-cial expressions and body movements through the avatar served as a groundingmechanism to gauge reactions. These observations underscore the vital role theAvatARoid’s real-time facial and gestural feedback played in enriching the emo-tional texture of the telepresence experience.“When I did this (raise hands) as, you know, hurray, we finishedthe game. She also did this (raising her hands). Oh, yay, we are cele-brating together.” (P4AR)6.4.4 Areas of ImprovementWe also identified several rooms for improvements, including refining the avatar forfull-body representation and finer details, enhancing body language cues throughbetter motion mapping, and addressing technical limitations such as latency anddevice resolution.Refining Visual Overlay Rendering: Full Body Representation, Finer Detailsof the Teleoperator, and Avoiding Rendering Artifacts to Maintain a Sense ofPresence.The feedback from participants suggests distinct areas for refinement in the visualrendering of avatars to maintain a sense of social presence. The issue of full-bodyrepresentation stood out prominently. Participants pointed out that the upper-torso-only avatar restricted the ability to gauge more nuanced body language and explic-itly stated the desire for a “full profile of the person.” They found the lack of an en-tire body disconcerting and created a “weird mix” of partial body parts of the avatarand the robot. Including finer details was also called for, especially concerning ar-ticulating smaller body parts like fingers. They also pointed out that awareness ofmissing body parts created a less realistic, face-to-face experience. Other visualdetails, such as “clipping” or “transparency” of the avatar, were also reported bysome participants to create difficulty engaging in the conversation. These findings48indicate that providing a complete and articulated visual representation may benefita more socially engaging telepresence experience using AvatARoid.“I was aware that she was there virtually mostly because the bot-tom half of her was not here, so I was consciously aware of that, whichmade it less like a face-to-face conversation.” (P48AR)Improvements in Motion Mapping: Need of More Natural Motion forImproved Gestural Feedbacks.The feedback from participants also highlighted concerns about the synchronicitywith speech and the naturalness of the gestures, affecting their ability to read theteleoperator’s body language effectively. Participants highlighted the teleoperator’sspoken words and the robot’s bodily movements not being in sync, which disruptedthe natural flow of communication. The lack of synchronized gestures with thespeech made it difficult to feel they were engaging with an “actual person face-to-face whose lips and body move together,” as P10AR mentioned. Secondly, partic-ipants noted the stiffness and restricted mobility in the robotic system’s physicalarticulations made the experience feel more “virtual” and “immobile,” contrastingwith the more fluid body language they would expect in face-to-face interaction.They pointed out the stiffness in the hands and lack of torso movement, which felt“unnatural.” These findings suggest that for users to effectively interpret body lan-guage and thus feel a higher social presence, the telepresence system needs to offernatural and dynamic movements.“I guess not too much (gauge her body language) if I have to behonest because mostly it is just moving its arms. But other than that,it doesn’t move the torso.” (P19AR)Technical Barriers to Social Presence: Latency and Device LimitationsImpede Sense of Presence.The technical issues, particularly latency and device resolution, emerged as signif-icant factors that negatively impacted users’ sense of immediacy and social pres-ence. Participants frequently cited latency as a barrier to achieving a natural flow49of interaction, noting that the lag in speech “definitely added to the aspect of theinteraction being virtual.” Similarly, they mentioned that the delay made it hardto gauge the other person’s reactions in real-time, requiring a “conscious effortto piece together cues.” Some found the slight speech delay disrupted the flowof conversation, often leading to awkward pauses where both parties stopped tonegotiate who should speak next. The limitations of the AR device, specificallyregarding its resolution, were also flagged as issues that broke the sense of im-mersion. Participants mentioned the grainy passthrough AR effect, poor resolution,and the headset’s form factor constantly making them aware of the “virtuality ofinteraction” and preventing the interaction from feeling like face-to-face. Thesefindings highlight the need for addressing technical issues like latency and devicelimitations to enhance the sense of immediacy and presence in AvatARoid.50Chapter 7Discussion7.1 Bridging the Embodiment Gap: UnmixingMetaphors?AvatARoid, by enhancing the on-site user’s perception of teleoperator embodi-ment, especially feeling the teleoperators being co-located with the robot and view-ing them as the controlling agency within the robotic system, shows the potentialto address the “mixing metaphor” problem in robotic telepresence [113]. As dis-cussed earlier, there’s been a pronounced disparity between the robot’s mechanicalform and the genuine portrayal of the teleoperator’s identity [48]. Such a discon-nect paves the way for ‘robomorphism’ [101], where on-site users fail to recog-nize the robotic system as the embodiment of the teleoperator, instead perceivethe robots merely as inanimate machines or objects [21, 66, 113] culminating indetrimental social behaviors like bullying and neglect of personal boundaries [80].AvatARoid, using its avatar overlay, seemingly amplified the teleoperator’s spatial“presence,” sense of the teleoperator being located where the robot is, and poten-tially mitigating issues surrounding proxemic violations [113], and facilitates thegraceful establishment of social norms [21].Compared to the evaluation of existing AR augmentation in VROOM [48],the co-location aspect of embodiment seems to be the most positively impactedby augmenting the robot with an avatar. Participants in both studies expressedthat interaction with an augmented robot was more similar to engaging with an-51other human sharing the same space. AvatARoid’s use of a real-time point cloudfor the teleoperator’s avatar offers a notable improvement in expressiveness overVROOM’s pre-rendered avatars, contributing to a more natural and engaging userexperience. Participants in our study appreciated the enhanced visual cues, espe-cially facial expressions, which allowed them to better interpret the teleoperator’semotional state. This contrasts with VROOM, where the absence of dynamic fa-cial expressions made users feel the avatar’s appearance was unnatural, resultingin them relying more on vocal cues. Findings for the VROOM system’s ability tomake teleoperator’s deictic, lexical, and collaborative gestures more understand-able to the on-site user are corroborated in our study as well, with participantsmentioning the real-time avatar helping them understand the body language andgestures of the teleoperators. However, participants in both studies echoed a com-mon desire for finer gestures, especially for the teleoperator’s hands. In our study,rendering artifacts clipped the hands, and the robot’s hand, which complementedthe clipped hand, was not articulated. The 3D avatar’s hand was not articulatedin VROOM, limiting gesturing using figures. The use of a wider field of viewpassthrough headset has potentially improved the visibility aspect of the avatar asparticipants in the VROOM study mentioned the HoloLens’ limited field of viewrestricted them to fully viewing the teleoperator’s head and arms at the same timeduring some gestures. However, the passthrough AR HMD’s limitation was still feltas our participants mentioned the poor resolution to remind the virtual nature ofthe interaction and impacting their sense of presence.Using a humanoid robot also added “tangible” dimensions to the gestures ofa “floating” avatar, making the teleoperator’s actions feel more “impactful,” asmentioned by our participants. By synchronizing the motions of the avatar andthe humanoid robot, AvatARoid also emphasizes the teleoperator’s agency overthe robot, making it easier to understand the actions of the teleoperators. Thisis in line with a previous study in telepresence robots [107], where synchroniz-ing on-screen gestures with physical gestures of the robotic system enhanced theobservers’ understanding of the teleoperator’s intentions and, consequently, theirsocial impression. Using AR to bring the avatar to the physical space of the on-siteuser, our study extends this beyond synchronization in two distinct spaces - screenand physical. These developments are pivotal in addressing the longstanding gap in52robotic and teleoperator embodiment in telepresence scenarios. When aligned withresults from previous AR-based robotic augmentation [48], that AR representationincreases the expressiveness in robotic telepresence interactions, and humanoidrobots affording physical interaction [17], AvatARoid’s motion synchronizationapproach fosters a seamless integration of the teleoperator’s intentions with therobot’s actions, establishing more harmonious interaction. Moreover, by reinforc-ing the sentiment that the “teleoperator is right here where the robot is” and “iscontrolling” the robot, AvatARoid holds promise in rectifying deficits and othersocial challenges inherent to the remotely located teleoperator in hybrid scenarios[48, 94, 108].7.2 AR Simulation Could be a Viable Tool to EvaluateRobot-mediated Telepresence Systems04812162 4 6Co−Presence Rating (1−7)Number of ParticipantsDistribution of Co−Presence RatingsFigure 7.1: Histogram of participants frequency distribution for different co-presence scoresIn both the questionnaire and interviews, most participants reported a height-ened sense of co-presence with the virtual robot, highlighting the utility of ARsimulations in robotic telepresence and HRI studies. The visual integration of therobot into the participants’ actual environment was consistently noted as a crucialelement for this enhanced co-presence. These findings validate AR as an effectivetool for HRI and telepresence scenarios that require seamless spatial integration.Participants in our study overwhelmingly cited spatial awareness and visual53blending as key elements enhancing their sense of co-presence with the AR robot.The robot’s realistic entry and exit through doors, while being occluded by the wall,effectively conveyed a sense of physical presence in the shared environment. Theexperimenter showing spatial awareness towards the robot reinforced this feelingof sharing the space.“I felt like the robot had a physical presence, when it was movingin... the wall worked, the robot appeared behind the wall... and whenyou walked back into the room, I don’t know if you did this on pur-pose, but you walked behind the robot and around it. And that alsomade me feel like it was it was more real.” (P38BL)The realism of the robot’s motion and appropriate scaling also emerged as sig-nificant factors in maintaining a sense of co-presence. Participants commented that“it moved like a real robot” and highlighted the importance of the robot’s realisticsize, stating the scale as “realistic” and not “uncanny.”Audio cues provided interesting mixed results, while some participants com-mented audio cues helped in fostering co-presence, stating robot movements, soundand sound localization made the interaction feel like it was “happening in thisroom.” Conversely, some participants did not notice the audio spatialization, stat-ing the sound emanated from the headset rather than the robot. This might bebecause the robot was primarily static, except when it entered or exited, failing toprovide enough spatial cues to localize sound.Despite the generally high sense of co-presence with the robot, participantsnoted that the AR headset’s form factor and visual quality occasionally disruptedthe sense of presence. The limitations in resolution, “the grainy” passthrough dis-play, and “being aware of the headset on one’s head” were cited as factors remind-ing them that their experience is virtual. Improvements in AR hardware design andfidelity might address these to achieve a seamless sense of presence.Expanding on our current framework, future simulations could incorporate tan-gible elements to enhance the realism and co-presence of virtual robots. Ther-mochromic ink could allow for dynamic color changes [100] in physical materi-als. Mechanical actuators—like those in actuated tabletop games [65] or wheeledrobots [111] could provide real-world movements that reflect the robot’s virtual54actions. Human actuators, as demonstrated in systems like Turkdeck [26], couldalso be used as conduits for the virtual robot’s physical influence within the actualspace. These additions might bridge the gap between virtual and physical realms,further optimizing the efficacy of AR simulation in testing telepresence and HRIapplications.We advocate that immersive simulation tools are invaluable for initial assess-ments and exploratory research in telepresence and HRI. They enable testing con-cepts or scenarios—such as search and rescue operations, robot malfunctions, andeven some technical issues, such as latency — that may be impractical or hazardousto execute in real-world settings. However, to ensure comprehensive validation,these preliminary findings should be subsequently tested and corroborated throughstudies involving actual robots.7.3 AvatARoid: Implication of Design Implementationsand Alternative ImplementationsOur AvatARoid implementation, despite mitigating spatial disconnection and agencyambiguity, confronts limitations that have implications for social engagement andprompt areas of improvement for future development.7.3.1 Full Embodiment and Fine GesturesIncorporating a full-body avatar would improve the user’s ability to gauge bodylanguage, thus enriching social interaction as previous studies in avatar-mediatedcommunication have shown [12, 87]. Adding granular gestures, like articulatingfingers, could allow for nuanced non-verbal communication, significantly height-ening the sense of social presence [83]. Our current implementation lacked theseabilities. Enhancements in creating user avatars capable of showing whole bodyand fine gesture replication, such as Holoportation [84], or high fidelity avatars[71] could be viable future implementations.7.3.2 Enhanced Motion MappingImproving the system’s motion mapping via advanced teleoperation could addressissues of movement asynchrony. Accurate, dynamic, and synchronized movements55make the teleoperator feel more like an “actual person,” amplifying social presence.Our current IK implementation tracks only joint positions of hands and drive armIK chains. Alternative humanoid teleoperation techniques [11, 30, 38] should beexplored to achieve enhanced motion control.7.3.3 The Robotic SystemThe underlying robotic system the teleoperator embodies is the foundation ourAvatARoid system builds on. Hence, we cannot ignore the design of the roboticsystem itself in future implementations. Two critical aspects of the robotic systemshould be considered:Robot IdentityWe utilize a social robot with a high identity in our current AvatARoid system. Ourfinding hints that this strong identity, such as a distinct facial feature of the pepperrobot itself, can create feelings in on-site users that they are interacting with twoseparate entities. Minimizing the robot’s independent identity could direct focusonto the teleoperator, making the interaction more about human-human engage-ment rather than human-robot. This shift could elevate the teleoperator’s socialpresence within the interaction [27]. Extending on the concept of Minimal HumanDesign [62, 110]-designing telepresence robots with minimal identity cues and hu-man appearance, we propose exploration of “Minimal Functional Human Design,”a structural humanoid robotic system with minimal identity which AvatARoid pro-vides a “skin” to, and evaluating robots with different anthropomorphic levels toidentify an acceptable match.Fidelity of Action ReplicationEnhancing the robot’s ability to replicate nuanced human movements makes in-terpreting the teleoperator’s actions easier. This accurate mirroring would bolsterthe on-site user’s sense of engaging with a coherent social entity. Ensuring thatthe robot’s abilities closely mirror the teleoperator representation could harmonizeexpectations and reality, avoiding the uncanny valley of telepresence [48] and thusenhancing the sense of social presence. Advancements in humanoid teleoperation56and the design of humanoid robots should eventually provide us with better tech-niques to represent the teleoperator’s actions accurately in a humanoid robot.7.4 Limitations and Future WorksWhile our study takes an essential step in evaluating AvatARoid’s potential fortelepresence, it is crucial to acknowledge certain limitations. First, our evaluationfocuses primarily on the on-site user’s experience, sidelining the teleoperator’s per-spective. A more holistic approach incorporating both sides is warranted to under-stand the system’s utility comprehensively. Moreover, the study’s scope did notinclude a teleoperator interface while evaluating AvatARoid. Various asymmetrictelepresence systems are emerging [48], and future work should consider a broadercomparison to evaluate AvatARoids in combination with teleoperator systems.Importantly, although user representation emerged as potentially a significantfactor influencing embodiment, it is imperative to recognize collaboration out-comes are not solely determined by it, as shown by our evaluation results withimpression and social presence measures. Many other variables, such as the na-ture of tasks [7, 44, 109], asymmetries in relational [16] or information dynamics[108], and the specific application scenarios, play a critical role. Future studiesshould, therefore, adopt a multi-dimensional approach encompassing these vari-ables to provide a nuanced understanding of social interaction and impression for-mation with the AvatARoid system.Finally, although we propose that simulation tools are invaluable for initialexploratory research for telepresence systems, we acknowledge that they do notcapture the entirety and nuances of interaction with a physical robot. We accountedfor the limited speed and range of the robot’s motion. However, our simulation tooldid not capture all the details of the robot’s real-world performance, such as thedexterity of the robot’s motion and latency in motion transfer from the teleoperatorto the robot. Latency in motion transfer might make the avatar display deviate fromthe robot’s physical motion. Similarly, the limited dexterity of the robot’s motionmight limit the robot’s manipulation ability, exacerbating the experience deeperinto the uncanny valley of telepresence. Our simulation tool helped us show thepotential of AvatARoid to address the perceptual disparity of user representation57in robot-mediated telepresence. To ensure comprehensive validation incorporatingall these factors, these preliminary findings should be subsequently confirmed oradjusted through studies involving real robots.58Chapter 8ConclusionIn this thesis, we introduced AvatARoid, a novel telepresence system that overlaysthe teleoperator’s avatar on a robotic embodiment and synchronizes their move-ments. We evaluated the system with a lab study focused on AvatARoid’s coremechanisms, AR avatar superimposition and motion alignment, and their impacton on-site users’ perception of the teleoperator’s embodiment, social presence, andoverall impression—factors essential for high-quality social interaction. AvatARoidshows promise in effectively enhancing the on-site users’ perception of unity be-tween the teleoperator representation and the robotic embodiment, making themfeel as if the teleoperator was indeed “located where the robot was” and “in controlof the robot.” Participants also felt that AvatARoid could serve as an emotionallyrich and engaging medium for communication. However, they recommended re-finements like displaying the teleoperator’s entire body and nuanced gestures for amore authentic sense of presence and body language. Future studies must examineAvatARoid’s performance across diverse tasks and application scenarios, considerits impact when combined with the constraints of an actual robot, and expand theevaluation to include both teleoperators and on-site users. As technological ad-vancements continue in the design of humanoid robots, their teleoperation, andmixed reality, AvatARoid offers a foundational step to explore how, through a syn-ergy of these technologies, we could enable telepresence systems with possibilitiesbeyond being there [43], with an ultimate goal of enriching human connections.59Bibliography[1] Final IK | Animation Tools | Unity Asset Store. URLhttps://assetstore.unity.com/packages/tools/animation/final-ik-14290. →page 28[2] Joints — Aldebaran 2.5.11.14a documentation, . URLhttp://doc.aldebaran.com/2-5/family/pepper technical/joints pep.html. →page 28[3] Feature Update: Telepresence Enabled by Pepper | SoftBank Robotics, .URL https://www.softbankrobotics.com/emea/en/blog/news-trends/feature-update-telepresence-enabled-pepper. → page 27[4] Home | Engine Developer Portal. URL https://developer.vuforia.com/. →page 27[5] Azure Kinect Examples for Unity, July 2019. URLhttps://rfilkov.com/2019/07/24/azure-kinect-examples-for-unity/. → page27[6] S. B. Abdul Jalil, B. Osburn, J. Huang, M. Barley, M. Markovich, andR. Amor. Avatars at a meeting. In Proceedings of the 13th InternationalConference of the NZ Chapter of the ACM’s Special Interest Group onHuman-Computer Interaction, CHINZ ’12, pages 84–87, New York, NY,USA, July 2012. Association for Computing Machinery. ISBN978-1-4503-1474-9. doi:10.1145/2379256.2379270. URLhttps://doi.org/10.1145/2379256.2379270. → pages 6, 12, 29[7] A. Abdullah, J. Kolkmeier, V. Lo, and M. Neff. Videoconference andEmbodied VR: Communication Patterns Across Task and Medium.Proceedings of the ACM on Human-Computer Interaction, 5(CSCW2):1–29, Oct. 2021. ISSN 2573-0142. doi:10.1145/3479597. URLhttps://dl.acm.org/doi/10.1145/3479597. → pages 35, 5760[8] S. O. Adalgeirsson and C. Breazeal. MeBot: A robotic platform forsocially embodied telepresence. In 2010 5th ACM/IEEE InternationalConference on Human-Robot Interaction (HRI), pages 15–22, Mar. 2010.doi:10.1109/HRI.2010.5453272. ISSN: 2167-2148. → pages 6, 12[9] I. Ajili, M. Mallem, and J.-Y. Didier. Gesture recognition for humanoidrobot teleoperation. In 2017 26th IEEE International Symposium on Robotand Human Interactive Communication (RO-MAN), pages 1115–1120,Aug. 2017. doi:10.1109/ROMAN.2017.8172443. ISSN: 1944-9437. →page 15[10] C. Alo´s-Ferrer and F. Farolfi. Trust Games and Beyond. Frontiers inNeuroscience, 13, 2019. ISSN 1662-453X. URLhttps://www.frontiersin.org/articles/10.3389/fnins.2019.00887. → page 30[11] M. Arduengo, A. Arduengo, A. Colome´, J. Lobo-Prat, and C. Torras.Human to Robot Whole-Body Motion Transfer. In 2020 IEEE-RAS 20thInternational Conference on Humanoid Robots (Humanoids), pages299–305, July 2021. doi:10.1109/HUMANOIDS47582.2021.9555769.ISSN: 2164-0580. → page 56[12] S. Aseeri and V. Interrante. The Influence of Avatar Representation onInterpersonal Communication in Virtual Social Environments. IEEETransactions on Visualization and Computer Graphics, 27(5):2608–2617,May 2021. ISSN 1941-0506. doi:10.1109/TVCG.2021.3067783.Conference Name: IEEE Transactions on Visualization and ComputerGraphics. → pages 16, 55[13] D. J. Atkinson and M. H. Clark. Methodology for study of human-robotsocial interaction in dangerous situations. In Proceedings of the secondinternational conference on Human-agent interaction, pages 371–376,Tsukuba Japan, Oct. 2014. ACM. ISBN 978-1-4503-3035-0.doi:10.1145/2658861.2658871. URLhttps://dl.acm.org/doi/10.1145/2658861.2658871. → page 8[14] F. Babel, A. Vogt, P. Hock, J. Kraus, F. Angerer, T. Seufert, andM. Baumann. Step Aside! VR-Based Evaluation of Adaptive RobotConflict Resolution Strategies for Domestic Service Robots. InternationalJournal of Social Robotics, 14(5):1239–1260, July 2022. ISSN 1875-4805.doi:10.1007/s12369-021-00858-7. URLhttps://doi.org/10.1007/s12369-021-00858-7. → pages 3, 861[15] F. Bazzano, F. Gentilini, F. Lamberti, A. Sanna, G. Paravati, V. Gatteschi,and M. Gaspardone. Immersive Virtual Reality-Based Simulation toSupport the Design of Natural Human-Robot Interfaces for Service RoboticApplications. In L. T. De Paolis and A. Mongelli, editors, AugmentedReality, Virtual Reality, and Computer Graphics, Lecture Notes inComputer Science, pages 33–51, Cham, 2016. Springer InternationalPublishing. ISBN 978-3-319-40621-3. doi:10.1007/978-3-319-40621-3 3.→ page 8[16] A. J. Beattie, C. Edwards, and M. J. Williams. Telepresence Group LeadersReceive Higher Ratings of Social Attractiveness and Leadership Quality.In Companion of the 2018 ACM/IEEE International Conference onHuman-Robot Interaction, HRI ’18, pages 61–62, New York, NY, USA,Mar. 2018. Association for Computing Machinery. ISBN978-1-4503-5615-2. doi:10.1145/3173386.3176952. URLhttps://dl.acm.org/doi/10.1145/3173386.3176952. → page 57[17] W. N. Benson, Z. Anderson, E. Capelle, M. F. Dunlap, B. Dorris, J. L.Gorlewicz, M. Shimizu, and J. B. Weinberg. Embodied ExpressiveGestures in Telerobots: A Tale of Two Users. ACM Transactions onHuman-Robot Interaction, 12(2):24:1–24:20, Mar. 2023.doi:10.1145/3570908. URL https://dl.acm.org/doi/10.1145/3570908. →pages 6, 12, 29, 53[18] M. Billinghurst and H. Kato. Collaborative augmented reality.Communications of the ACM, 45(7):64–70, July 2002. ISSN 0001-0782.doi:10.1145/514236.514265. URLhttps://dl.acm.org/doi/10.1145/514236.514265. → page 11[19] F. Biocca. The Cyborg’s Dilemma: Progressive Embodiment in VirtualEnvironments [1]. Journal of Computer-Mediated Communication, 3(2):JCMC324, Sept. 1997. ISSN 1083-6101.doi:10.1111/j.1083-6101.1997.tb00070.x. URLhttps://doi.org/10.1111/j.1083-6101.1997.tb00070.x. → page 7[20] N. Bos, J. Olson, D. Gergle, G. Olson, and Z. Wright. Effects of fourcomputer-mediated communications channels on trust development. InProceedings of the SIGCHI Conference on Human Factors in ComputingSystems, CHI ’02, pages 135–140, New York, NY, USA, Apr. 2002.Association for Computing Machinery. ISBN 978-1-58113-453-7.doi:10.1145/503376.503401. URLhttps://dl.acm.org/doi/10.1145/503376.503401. → page 862[21] A. Boudouraki, J. E. Fischer, S. Reeves, and S. Rintel. ”Being in on theAction” in Mobile Robotic Telepresence: Rethinking Presence in HybridParticipation. In Proceedings of the 2023 ACM/IEEE InternationalConference on Human-Robot Interaction, HRI ’23, pages 63–71, NewYork, NY, USA, Mar. 2023. Association for Computing Machinery. ISBN978-1-4503-9964-7. doi:10.1145/3568162.3576961. URLhttps://dl.acm.org/doi/10.1145/3568162.3576961. → pages 2, 5, 51[22] V. Braun and V. Clarke. Using thematic analysis in psychology. Qualitativeresearch in psychology, 3(2):77–101, 2006. Publisher: Taylor & Francis.→ page 36[23] P. Bremner, O. Celiktutan, and H. Gunes. Personality Perception of RobotAvatar Tele-operators. In The Eleventh ACM/IEEE InternationalConference on Human Robot Interaction, HRI ’16, pages 141–148,Christchurch, New Zealand, Mar. 2016. IEEE Press. ISBN978-1-4673-8370-7. → pages 6, 13[24] P. A. Bremner, O. Celiktutan, and H. Gunes. Personality Perception ofRobot Avatar Teleoperators in Solo and Dyadic Tasks. Frontiers inRobotics and AI, 4, 2017. ISSN 2296-9144. URLhttps://www.frontiersin.org/articles/10.3389/frobt.2017.00016. → pages2, 6, 13[25] S. Bustamante, J. Peters, B. Scho¨lkopf, M. Grosse-Wentrup, andV. Jayaram. ArmSym: A Virtual Human–Robot Interaction Laboratory forAssistive Robotics. IEEE Transactions on Human-Machine Systems, 51(6):568–577, Dec. 2021. ISSN 2168-2305. doi:10.1109/THMS.2021.3106865.Conference Name: IEEE Transactions on Human-Machine Systems. →page 8[26] L.-P. Cheng, T. Roumen, H. Rantzsch, S. Ko¨hler, P. Schmidt, R. Kovacs,J. Jasper, J. Kemper, and P. Baudisch. TurkDeck: Physical Virtual RealityBased on People. In Proceedings of the 28th Annual ACM Symposium onUser Interface Software & Technology, UIST ’15, pages 417–426, NewYork, NY, USA, Nov. 2015. Association for Computing Machinery. ISBN978-1-4503-3779-3. doi:10.1145/2807442.2807463. URLhttps://dl.acm.org/doi/10.1145/2807442.2807463. → page 55[27] J. J. Choi and S. S. Kwak. Who is this?: Identity and presence inrobot-mediated communication. Cognitive Systems Research, 43:174–189,June 2017. ISSN 1389-0417. doi:10.1016/j.cogsys.2016.07.006. URL63https://www.sciencedirect.com/science/article/pii/S138904171630047X.→ pages 6, 56[28] N. L. Collins and L. C. Miller. Self-disclosure and liking: a meta-analyticreview. Psychological Bulletin, 116(3):457–475, Nov. 1994. ISSN0033-2909. doi:10.1037/0033-2909.116.3.457. → page 35[29] S. Daher, K. Kim, M. Lee, A. Raij, R. Schubert, J. Bailenson, andG. Welch. Exploring social presence transfer in real-virtual humaninteraction. In 2016 IEEE Virtual Reality (VR), pages 165–166, Greenville,SC, USA, Mar. 2016. IEEE. ISBN 978-1-5090-0836-0.doi:10.1109/VR.2016.7504705. URLhttp://ieeexplore.ieee.org/document/7504705/. → page 21[30] K. Darvish, L. Penco, J. Ramos, R. Cisneros, J. Pratt, E. Yoshida, S. Ivaldi,and D. Pucci. Teleoperation of Humanoid Robots: A Survey. IEEETransactions on Robotics, 39(3):1706–1727, June 2023. ISSN 1941-0468.doi:10.1109/TRO.2023.3236952. Conference Name: IEEE Transactionson Robotics. → pages 14, 15, 56[31] E. de Wolf and J. Li. Double Trouble: The Effect of Eye Gaze on theSocial Impression of Mobile Robotic Telepresence Operators. In A. R.Wagner, D. Feil-Seifer, K. S. Haring, S. Rossi, T. Williams, H. He, andS. Sam Ge, editors, Social Robotics, Lecture Notes in Computer Science,pages 357–368, Cham, 2020. Springer International Publishing. ISBN978-3-030-62056-1. doi:10.1007/978-3-030-62056-1 30. → page 8[32] E. Deng, B. Mutlu, and M. J. Mataric. Embodiment in Socially InteractiveRobots. Foundations and Trends® in Robotics, 7(4):251–356, Jan. 2019.ISSN 1935-8253, 1935-8261. doi:10.1561/2300000056. URLhttps://www.nowpublishers.com/article/Details/ROB-056. Publisher: NowPublishers, Inc. → pages 1, 7[33] L. Dole and W. Ju. Face and Ecological Validity in Simulations: Lessonsfrom Search-and-Rescue HRI. In Proceedings of the 2019 CHI Conferenceon Human Factors in Computing Systems, CHI ’19, pages 1–8, New York,NY, USA, May 2019. Association for Computing Machinery. ISBN978-1-4503-5970-2. doi:10.1145/3290605.3300681. URLhttps://dl.acm.org/doi/10.1145/3290605.3300681. → page 20[34] M. Duguleana, F. G. Barbuceanu, and G. Mogan. Evaluating Human-RobotInteraction during a Manipulation Experiment Conducted in Immersive64Virtual Reality. In R. Shumaker, editor, Virtual and Mixed Reality - NewTrends, Lecture Notes in Computer Science, pages 164–173, Berlin,Heidelberg, 2011. Springer. ISBN 978-3-642-22021-0.doi:10.1007/978-3-642-22021-0 19. → pages 8, 19[35] R. Etzi, S. Huang, G. Scurati, S. Lyu, F. Ferrise, A. Gallace, A. Gaggioli,A. Chirico, M. Carulli, and M. Bordegoni. Using Virtual Reality To TestHuman-Robot Interaction During a Collaborative Task. Aug. 2019.doi:10.1115/DETC2019-97415. → page 8[36] B. Feng, S. Li, and N. Li. Is a Profile Worth a Thousand Words? HowOnline Support-Seeker’s Profile Features May Influence the Quality ofReceived Support Messages. Communication Research, 43(2):253–276,Mar. 2016. ISSN 0093-6502. doi:10.1177/0093650213510942. URLhttps://doi.org/10.1177/0093650213510942. Publisher: SAGEPublications Inc. → page 8[37] S. Gaurav, Z. Al-Qurashi, A. Barapatre, G. Maratos, T. Sarma, and B. D.Ziebart. Deep Correspondence Learning for Effective RoboticTeleoperation using Virtual Reality. In 2019 IEEE-RAS 19th InternationalConference on Humanoid Robots (Humanoids), pages 477–483, Toronto,ON, Canada, Oct. 2019. IEEE. ISBN 978-1-5386-7630-1.doi:10.1109/Humanoids43949.2019.9035031. URLhttps://ieeexplore.ieee.org/document/9035031/. → page 14[38] C. Girard, D. Calderon, A. Lemus, V. Ferman, and J. Fajardo. A MotionMapping System for Humanoids that Provides Immersive TelepresenceExperiences. In 2020 6th International Conference on Mechatronics andRobotics Engineering (ICMRE), pages 165–170, Feb. 2020.doi:10.1109/ICMRE49073.2020.9065155. → page 56[39] M. Gonzalez-Franco and T. C. Peck. Avatar Embodiment. Towards aStandardized Questionnaire. Frontiers in Robotics and AI, 5, 2018. ISSN2296-9144. URLhttps://www.frontiersin.org/articles/10.3389/frobt.2018.00074. → page 34[40] C. Hendrix and W. Barfield. The sense of presence within auditory virtualenvironments. Presence: Teleoperators and Virtual Environments, 5(3):290–301, Jan. 1996. ISSN 1054-7460. doi:10.1162/pres.1996.5.3.290.URL https://doi.org/10.1162/pres.1996.5.3.290. → page 22[41] K. G. Hickenbottom. Almost Like Being There: Embodiment, SocialPresence, and Engagement Using Telepresence Robots in Blended Courses.65Ed.D., Seattle Pacific University, United States – Washington, 2022. URLhttps://www.proquest.com/docview/2656207725/abstract/E0857EFE92E74636PQ/1. ISBN: 9798426815629. → page 1[42] D. M. Hilty, K. Randhawa, M. M. Maheu, A. J. S. McKean, R. Pantera,M. C. Mishkind, and A. izzo. A Review of Telepresence, Virtual Reality,and Augmented Reality Applied to Clinical Care. Journal of Technology inBehavioral Science, 5(2):178–205, June 2020. ISSN 2366-5963.doi:10.1007/s41347-020-00126-x. URLhttps://doi.org/10.1007/s41347-020-00126-x. → page 1[43] J. Hollan and S. Stornetta. Beyond being there. In Proceedings of theSIGCHI Conference on Human Factors in Computing Systems, CHI ’92,pages 119–125, New York, NY, USA, June 1992. Association forComputing Machinery. ISBN 978-0-89791-513-7.doi:10.1145/142750.142769. URLhttps://dl.acm.org/doi/10.1145/142750.142769. → page 59[44] A. B. Hollingshead, J. E. Mcgrath, and K. M. O’Connor. Group TaskPerformance and Communication Technology: A Longitudinal Study ofComputer-Mediated Versus Face-to-Face Work Groups. Small GroupResearch, 24(3):307–333, Aug. 1993. ISSN 1046-4964.doi:10.1177/1046496493243003. URLhttps://doi.org/10.1177/1046496493243003. Publisher: SAGEPublications Inc. → page 57[45] J. Hwang, S. Lee, S. C. Ahn, and H.-g. Kim. Augmented robot agent:Enhancing co-presence of the remote participant. In 2008 7th IEEE/ACMInternational Symposium on Mixed and Augmented Reality, pages161–162, Sept. 2008. doi:10.1109/ISMAR.2008.4637346. → page 6[46] K. Inoue, S. Nonaka, Y. Ujiie, T. Takubo, and T. Arai. Comparison ofhuman psychology for real and virtual mobile manipulators. In ROMAN2005. IEEE International Workshop on Robot and Human InteractiveCommunication, 2005., pages 73–78, Aug. 2005.doi:10.1109/ROMAN.2005.1513759. ISSN: 1944-9437. → pages 8, 19[47] F. E. Jamiy and R. Marsh. Distance Estimation In Virtual Reality AndAugmented Reality: A Survey. In 2019 IEEE International Conference onElectro Information Technology (EIT), pages 063–068, May 2019.doi:10.1109/EIT.2019.8834182. ISSN: 2154-0373. → pages 8, 19, 2066[48] B. Jones, Y. Zhang, P. N. Y. Wong, and S. Rintel. Belonging There:VROOM-ing into the Uncanny Valley of XR Telepresence. Proceedings ofthe ACM on Human-Computer Interaction, 5(CSCW1):59:1–59:31, Apr.2021. doi:10.1145/3449133. URL https://doi.org/10.1145/3449133. →pages 2, 5, 6, 12, 51, 53, 56, 57[49] M. Jung, J. Kim, K. Han, and K. K. Kim. Social TelecommunicationExperience with Full-Body Ownership Humanoid Robot. InternationalJournal of Social Robotics, 14(9):1951–1964, Nov. 2022. ISSN1875-4805. doi:10.1007/s12369-022-00922-w. URLhttps://doi.org/10.1007/s12369-022-00922-w. → pages 6, 12[50] B. Kenwright. Inverse Kinematics – Cyclic Coordinate Descent (CCD).Journal of Graphics Tools, 16(4):177–217, Oct. 2012. ISSN 2165-347X.doi:10.1080/2165347X.2013.823362. URLhttps://doi.org/10.1080/2165347X.2013.823362. Publisher: Taylor &Francis eprint: https://doi.org/10.1080/2165347X.2013.823362. → page26[51] H. Khalil, E. Coronado, and G. Venture. Human Motion Retargeting toPepper Humanoid Robot from Uncalibrated Videos Using Human PoseEstimation. In 2021 30th IEEE International Conference on Robot &Human Interactive Communication (RO-MAN), pages 1145–1152, Aug.2021. doi:10.1109/RO-MAN50785.2021.9515495. ISSN: 1944-9437. →page 15[52] H. Kim, K.-S. Suh, and U.-K. Lee. Effects of collaborative online shoppingon shopping experience through social and relational perspectives.Information & Management, 50(4):169–180, June 2013. ISSN 0378-7206.doi:10.1016/j.im.2013.02.003. URLhttps://www.sciencedirect.com/science/article/pii/S0378720613000128.→ page 7[53] H. Kim, M. Lee, G. J. Kim, and J.-I. Hwang. The Impacts of Visual Effectson User Perception With a Virtual Human in Augmented Reality ConflictSituations. IEEE Access, 9:35300–35312, 2021. ISSN 2169-3536.doi:10.1109/ACCESS.2021.3062037. Conference Name: IEEE Access. →page 22[54] J. Kim, D. Kim, B. Kim, H. Kim, and J. Lee. Holobot: Hologram basedExtended Reality Telepresence Robot. In Companion of the 2023ACM/IEEE International Conference on Human-Robot Interaction, HRI67’23, pages 60–64, New York, NY, USA, Mar. 2023. Association forComputing Machinery. ISBN 978-1-4503-9970-8.doi:10.1145/3568294.3580043. URLhttps://dl.acm.org/doi/10.1145/3568294.3580043. → pages 5, 6[55] K. Kim, G. Bruder, and G. Welch. Exploring the effects of observedphysicality conflicts on real-virtual human interaction in augmented reality.In Proceedings of the 23rd ACM Symposium on Virtual Reality Softwareand Technology, VRST ’17, pages 1–7, New York, NY, USA, Nov. 2017.Association for Computing Machinery. ISBN 978-1-4503-5548-3.doi:10.1145/3139131.3139151. URLhttps://doi.org/10.1145/3139131.3139151. → page 22[56] K. Kim, D. Maloney, G. Bruder, J. N. Bailenson, and G. F. Welch. Theeffects of virtual human’s spatial and behavioral coherence with physicalobjects on social presence in AR. Computer Animation and Virtual Worlds,28(3-4):e1771, 2017. ISSN 1546-427X. doi:10.1002/cav.1771. URLhttps://onlinelibrary.wiley.com/doi/abs/10.1002/cav.1771. eprint:https://onlinelibrary.wiley.com/doi/pdf/10.1002/cav.1771. → page 21[57] K. Kim, R. Schubert, J. Hochreiter, G. Bruder, and G. Welch. Blowing inthe wind: Increasing social presence with a virtual human viaenvironmental airflow interaction in mixed reality. Computers & Graphics,83:23–32, Oct. 2019. ISSN 0097-8493. doi:10.1016/j.cag.2019.06.006.URLhttps://www.sciencedirect.com/science/article/pii/S0097849319301001.→ pages 20, 21, 35[58] L. H. Kim, V. Domova, Y. Yao, and P. E. Paredes. Effects of a Co-LocatedRobot and Anthropomorphism on Human Motivation and Emotion acrossPersonality and Gender. In 2022 31st IEEE International Conference onRobot and Human Interactive Communication (RO-MAN), pages 155–162,Aug. 2022. doi:10.1109/RO-MAN53752.2022.9900694. ISSN: 1944-9437.→ page 8[59] M. Kobayashi, K. Ueno, and S. Ise. The effects of spatialized sounds on thesense of presence in auditory virtual environments: A psychological andphysiological study. Presence: Teleoperators and Virtual Environments, 24(2):163–174, May 2015. ISSN 1054-7460. doi:10.1162/PRES a 00226.URL http://www.scopus.com/inward/record.url?scp=84947220025&partnerID=8YFLogxK. → page 2268[60] N. Koceska, S. Koceski, P. Beomonte Zobel, V. Trajkovik, and N. Garcia.A Telemedicine Robot System for Assisted and Independent Living.Sensors, 19(4):834, Jan. 2019. ISSN 1424-8220. doi:10.3390/s19040834.URL https://www.mdpi.com/1424-8220/19/4/834. Number: 4 Publisher:Multidisciplinary Digital Publishing Institute. → page 1[61] A. Kristoffersson, S. Coradeschi, and A. Loutfi. A Review of MobileRobotic Telepresence. Advances in Human-Computer Interaction, 2013:e902316, Apr. 2013. ISSN 1687-5893. doi:10.1155/2013/902316. URLhttps://www.hindawi.com/journals/ahci/2013/902316/. Publisher: Hindawi.→ pages 1, 2, 5, 29[62] K. Kuwamura, T. Minato, S. Nishio, and H. Ishiguro. Personality distortionin communication through teleoperated robots. In 2012 IEEE RO-MAN:The 21st IEEE International Symposium on Robot and Human InteractiveCommunication, pages 49–54, Sept. 2012.doi:10.1109/ROMAN.2012.6343730. ISSN: 1944-9437. → pages2, 6, 13, 56[63] D. Lanzoni, A. Cattaneo, A. Vitali, D. Regazzoni, and C. Rizzi. MarkerlessMotion Capture and Virtual Reality for Real-Time Ergonomic Analysis ofOperators in Workstations with Collaborative Robots: a preliminary study.In S. Gerbino, A. Lanzotti, M. Martorelli, R. Mira´lbes Buil, C. Rizzi, andL. Roucoules, editors, Advances on Mechanics, Design Engineering andManufacturing IV, Lecture Notes in Mechanical Engineering, pages1183–1194, Cham, 2023. Springer International Publishing. ISBN978-3-031-15928-2. doi:10.1007/978-3-031-15928-2 103. → page 8[64] K. M. Lee, Y. Jung, J. Kim, and S. R. Kim. Are physically embodied socialagents better than disembodied social agents?: The effects of physicalembodiment, tactile interaction, and people’s loneliness in human–robotinteraction. International Journal of Human-Computer Studies, 64(10):962–973, Oct. 2006. ISSN 1071-5819. doi:10.1016/j.ijhcs.2006.05.002.URLhttps://www.sciencedirect.com/science/article/pii/S1071581906000784.→ page 2[65] M. Lee, N. Norouzi, G. Bruder, P. J. Wisniewski, and G. F. Welch. MixedReality Tabletop Gameplay: Social Interaction With a Virtual HumanCapable of Physical Influence. IEEE Transactions on Visualization andComputer Graphics, 27(8):3534–3545, Aug. 2021. ISSN 1941-0506.69doi:10.1109/TVCG.2019.2959575. Conference Name: IEEE Transactionson Visualization and Computer Graphics. → page 54[66] M. K. Lee and L. Takayama. ”Now, i have a body”: uses and social normsfor mobile remote presence in the workplace. In Proceedings of theSIGCHI Conference on Human Factors in Computing Systems, CHI ’11,pages 33–42, New York, NY, USA, May 2011. Association for ComputingMachinery. ISBN 978-1-4503-0228-9. doi:10.1145/1978942.1978950.URL https://dl.acm.org/doi/10.1145/1978942.1978950. → pages1, 2, 5, 12, 51[67] W.-Y. Lee, M. Sakashita, E. Ricci, H. Claure, F. Guimbretie`re, andM. Jung. Interactive Vignettes: Enabling Large-Scale Interactive HRIResearch. In 2021 30th IEEE International Conference on Robot & HumanInteractive Communication (RO-MAN), pages 1289–1296, Aug. 2021.doi:10.1109/RO-MAN50785.2021.9515376. ISSN: 1944-9437. → page 8[68] R. Li, M. van Almkerk, S. van Waveren, E. Carter, and I. Leite. ComparingHuman-Robot Proxemics Between Virtual Reality and the Real World. In2019 14th ACM/IEEE International Conference on Human-RobotInteraction (HRI), pages 431–439, Mar. 2019.doi:10.1109/HRI.2019.8673116. ISSN: 2167-2148. → pages 8, 19[69] S. Li, B. Feng, N. Li, and X. Tan. How Social Context Cues in OnlineSupport-Seeking Influence Self-Disclosure in Support Provision.Communication Quarterly, 63(5):586–602, Oct. 2015. ISSN 0146-3373.doi:10.1080/01463373.2015.1078389. URLhttps://doi.org/10.1080/01463373.2015.1078389. Publisher: Routledgeeprint: https://doi.org/10.1080/01463373.2015.1078389. → page 8[70] J.-L. Lugrin, M. Wiedemann, D. Bieberstein, and M. E. Latoschik.Influence of avatar realism on stressful situation in VR. In 2015 IEEEVirtual Reality (VR), pages 227–228, Mar. 2015.doi:10.1109/VR.2015.7223378. ISSN: 2375-5334. → page 17[71] S. Ma, T. Simon, J. Saragih, D. Wang, Y. Li, F. De La Torre, and Y. Sheikh.Pixel Codec Avatars, Apr. 2021. URL http://arxiv.org/abs/2104.04638.arXiv:2104.04638 [cs]. → page 55[72] D. Madole and D. Begault. 3-D Sound for Virtual Reality and Multimedia.Computer Music Journal, 19(4):99, 1995. ISSN 01489267.doi:10.2307/3680997. URLhttps://www.jstor.org/stable/3680997?origin=crossref. → page 2270[73] K. Mahadevan, M. Sousa, A. Tang, and T. Grossman. “Grip-that-there”:An Investigation of Explicit and Implicit Task Allocation Techniques forHuman-Robot Collaboration. In Proceedings of the 2021 CHI Conferenceon Human Factors in Computing Systems, CHI ’21, pages 1–14, New York,NY, USA, May 2021. Association for Computing Machinery. ISBN978-1-4503-8096-6. doi:10.1145/3411764.3445355. URLhttps://dl.acm.org/doi/10.1145/3411764.3445355. → page 8[74] A. A. Malik, T. Masood, and A. Bilberg. Virtual reality in manufacturing:immersive and collaborative artificial-reality in design of human-robotworkspace. International Journal of Computer Integrated Manufacturing,33(1):22–37, Jan. 2020. ISSN 0951-192X.doi:10.1080/0951192X.2019.1690685. URLhttps://doi.org/10.1080/0951192X.2019.1690685. Publisher: Taylor &Francis eprint: https://doi.org/10.1080/0951192X.2019.1690685. → page8[75] M. Mara, J.-P. Stein, M. E. Latoschik, B. Lugrin, C. Schreiner,R. Hostettler, and M. Appel. User Responses to a Humanoid RobotObserved in Real Life, Virtual Reality, 3D and 2D. Frontiers inPsychology, 12, 2021. ISSN 1664-1078. URLhttps://www.frontiersin.org/articles/10.3389/fpsyg.2021.633178. → pages8, 19, 22[76] E. Matsas and G.-C. Vosniakos. Design of a virtual reality training systemfor human–robot collaboration in manufacturing tasks. InternationalJournal on Interactive Design and Manufacturing (IJIDeM), 11(2):139–153, May 2017. ISSN 1955-2505. doi:10.1007/s12008-015-0259-2.URL https://doi.org/10.1007/s12008-015-0259-2. → page 8[77] E. Matsas, G.-C. Vosniakos, and D. Batras. Effectiveness and acceptabilityof a virtual environment for assessing human–robot collaboration inmanufacturing. The International Journal of Advanced ManufacturingTechnology, 92(9):3903–3917, Oct. 2017. ISSN 1433-3015.doi:10.1007/s00170-017-0428-5. URLhttps://doi.org/10.1007/s00170-017-0428-5. → page 8[78] J. Mccroskey, V. Richmond, and J. Daly. The Development of a Measure ofPerceived Homophily. Human Communication Interaction, 1:323–332,June 1975. doi:10.1111/j.1468-2958.1975.tb00281.x. → page 3471[79] B. E. Mennecke, J. L. Triplett, L. M. Hassall, and Z. J. Conde. EmbodiedSocial Presence Theory. In 2010 43rd Hawaii International Conference onSystem Sciences, pages 1–10, Jan. 2010. doi:10.1109/HICSS.2010.179.ISSN: 1530-1605. → pages 1, 7[80] M. Muratbekova-Touron and E. Leon. “Is there anybody out there?” Usinga telepresence robot to engage in face time at the office. InformationTechnology & People, ahead-of-print(ahead-of-print), Jan. 2021. ISSN0959-3845. doi:10.1108/ITP-01-2021-0080. URLhttps://doi.org/10.1108/ITP-01-2021-0080. → page 51[81] D. T. Nguyen and J. Canny. More than face-to-face: empathy effects ofvideo framing. In Proceedings of the SIGCHI Conference on HumanFactors in Computing Systems, CHI ’09, pages 423–432, New York, NY,USA, Apr. 2009. Association for Computing Machinery. ISBN978-1-60558-246-7. doi:10.1145/1518701.1518770. URLhttps://dl.acm.org/doi/10.1145/1518701.1518770. → page 8[82] K. Nowak and F. Biocca. The Effect of the Agency and Anthropomorphismon Users’ Sense of Telepresence, Copresence, and Social Presence inVirtual Environments. Presence Teleoperators & Virtual Environments, 12:481–494, Oct. 2003. doi:10.1162/105474603322761289. → page 34[83] C. S. Oh, J. N. Bailenson, and G. F. Welch. A Systematic Review of SocialPresence: Definition, Antecedents, and Implications. Frontiers in Roboticsand AI, 5, 2018. ISSN 2296-9144. URLhttps://www.frontiersin.org/article/10.3389/frobt.2018.00114. → pages7, 16, 55[84] S. Orts-Escolano, C. Rhemann, S. Fanello, W. Chang, A. Kowdle,Y. Degtyarev, D. Kim, P. L. Davidson, S. Khamis, M. Dou, V. Tankovich,C. Loop, Q. Cai, P. A. Chou, S. Mennicken, J. Valentin, V. Pradeep,S. Wang, S. B. Kang, P. Kohli, Y. Lutchyn, C. Keskin, and S. Izadi.Holoportation: Virtual 3D Teleportation in Real-time. In Proceedings ofthe 29th Annual Symposium on User Interface Software and Technology,UIST ’16, pages 741–754, New York, NY, USA, Oct. 2016. Association forComputing Machinery. ISBN 978-1-4503-4189-9.doi:10.1145/2984511.2984517. URLhttps://dl.acm.org/doi/10.1145/2984511.2984517. → pages5, 11, 16, 17, 5572[85] N. Osmers, M. Prilla, O. Blunk, G. George Brown, M. Janßen, andN. Kahrl. The Role of Social Presence for Cooperation in AugmentedReality on Head Mounted Devices: A Literature Review. In Proceedings ofthe 2021 CHI Conference on Human Factors in Computing Systems, CHI’21, pages 1–17, New York, NY, USA, May 2021. Association forComputing Machinery. ISBN 978-1-4503-8096-6.doi:10.1145/3411764.3445633. URLhttps://dl.acm.org/doi/10.1145/3411764.3445633. → pages 11, 16[86] A. K. Ostrowski, V. Zygouras, H. W. Park, and C. Breazeal. Small GroupInteractions with Voice-User Interfaces: Exploring Social Embodiment,Rapport, and Engagement. In Proceedings of the 2021 ACM/IEEEInternational Conference on Human-Robot Interaction, pages 322–331,Boulder CO USA, Mar. 2021. ACM. ISBN 978-1-4503-8289-2.doi:10.1145/3434073.3444655. URLhttps://dl.acm.org/doi/10.1145/3434073.3444655. → page 1[87] M. Pakanen, P. Alavesa, N. van Berkel, T. Koskela, and T. Ojala. “Nice tosee you virtually”: Thoughtful design and evaluation of virtual avatar of theother user in AR and VR based telexistence systems. EntertainmentComputing, 40:100457, Jan. 2022. ISSN 1875-9521.doi:10.1016/j.entcom.2021.100457. URLhttps://www.sciencedirect.com/science/article/pii/S1875952121000549.→ page 55[88] A. Paljic. Ecological Validity of Virtual Reality: Three Use Cases. InS. Battiato, G. M. Farinella, M. Leo, and G. Gallo, editors, New Trends inImage Analysis and Processing – ICIAP 2017, Lecture Notes in ComputerScience, pages 301–310, Cham, 2017. Springer International Publishing.ISBN 978-3-319-70742-6. doi:10.1007/978-3-319-70742-6 28. → page19[89] T. Pejsa, J. Kantor, H. Benko, E. Ofek, and A. D. Wilson. Room2Room:Enabling Life-Size Telepresence in a Projected Augmented RealityEnvironment. pages 1716–1725, Feb. 2016. URLhttps://www.microsoft.com/en-us/research/publication/room2room-enabling-life-size-telepresence-in-a-projected-augmented-reality-environment/.→ page 5[90] D. Pimentel and C. Vinkers. Copresence With Virtual Humans in MixedReality: The Impact of Contextual Responsiveness on Social Perceptions.73Frontiers in Robotics and AI, 8, 2021. ISSN 2296-9144. URLhttps://www.frontiersin.org/articles/10.3389/frobt.2021.634520. → page 21[91] T. Piumsomboon, G. A. Lee, J. D. Hart, B. Ens, R. W. Lindeman, B. H.Thomas, and M. Billinghurst. Mini-Me: An Adaptive Avatar for MixedReality Remote Collaboration. In Proceedings of the 2018 CHI Conferenceon Human Factors in Computing Systems, CHI ’18, pages 1–13, New York,NY, USA, Apr. 2018. Association for Computing Machinery. ISBN978-1-4503-5620-6. doi:10.1145/3173574.3173620. URLhttps://doi.org/10.1145/3173574.3173620. → pages 16, 17[92] I. Rae and C. Neustaedter. Robotic Telepresence at Scale. In Proceedingsof the 2017 CHI Conference on Human Factors in Computing Systems,CHI ’17, pages 313–324, New York, NY, USA, May 2017. Association forComputing Machinery. ISBN 978-1-4503-4655-9.doi:10.1145/3025453.3025855. URLhttps://dl.acm.org/doi/10.1145/3025453.3025855. → page 2[93] I. Rae, L. Takayama, and B. Mutlu. One of the gang: supporting in-groupbehavior for embodied mediated communication. In Proceedings of theSIGCHI Conference on Human Factors in Computing Systems, CHI ’12,pages 3091–3100, New York, NY, USA, May 2012. Association forComputing Machinery. ISBN 978-1-4503-1015-4.doi:10.1145/2207676.2208723. URLhttps://dl.acm.org/doi/10.1145/2207676.2208723. → pages 29, 34, 35[94] I. Rae, L. Takayama, and B. Mutlu. In-body experiences: embodiment,control, and trust in robot-mediated communication. In Proceedings of theSIGCHI Conference on Human Factors in Computing Systems, pages1921–1930, 2013. → pages 35, 53[95] S. Reysen. Construction of a New Scale: The Reysen Likability Scale.Social Behavior and Personality: An International Journal, 33(2):201–208,2005. ISSN 1179-6391. doi:10.2224/sbp.2005.33.2.201. Place: NewZealand Publisher: Society for Personality Research. → page 34[96] T. Rhee, S. Thompson, D. Medeiros, R. dos Anjos, and A. Chalmers.Augmented Virtual Teleportation for High-Fidelity Telecollaboration.IEEE Transactions on Visualization and Computer Graphics, 26(5):1923–1933, May 2020. ISSN 1941-0506.doi:10.1109/TVCG.2020.2973065. Conference Name: IEEE Transactionson Visualization and Computer Graphics. → pages 16, 1774[97] H. Ridha-Mahfoudhi and N.-T. Dang. Real Time Point Cloud Self-AvatarWith A Single RGB-D Camera. In Proceedings of the 25th ACMSymposium on Virtual Reality Software and Technology, VRST ’19, pages1–2, New York, NY, USA, Nov. 2019. Association for ComputingMachinery. ISBN 978-1-4503-7001-1. doi:10.1145/3359996.3365041.URL https://dl.acm.org/doi/10.1145/3359996.3365041. → page 26[98] J. A. Rode. On Becoming a Cyborg: A Reflection on Articulation Work,Embodiment, Agency and Ableism. In P. Langdon, J. Lazar, A. Heylighen,and H. Dong, editors, Breaking Down Barriers, pages 239–249, Cham,2018. Springer International Publishing. ISBN 978-3-319-75028-6.doi:10.1007/978-3-319-75028-6 21. → page 5[99] O. Sadka, J. Giron, D. Friedman, O. Zuckerman, and H. Erel.Virtual-reality as a Simulation Tool for Non-humanoid Social Robots. InExtended Abstracts of the 2020 CHI Conference on Human Factors inComputing Systems, pages 1–9, Honolulu HI USA, Apr. 2020. ACM.ISBN 978-1-4503-6819-3. doi:10.1145/3334480.3382893. URLhttps://dl.acm.org/doi/10.1145/3334480.3382893. → pages 8, 19[100] S. Schmidt, O. J. A. Nunez, and F. Steinicke. Blended Agents:Manipulation of Physical Objects within Mixed Reality Environments andBeyond. In Symposium on Spatial User Interaction, SUI ’19, pages 1–10,New York, NY, USA, Oct. 2019. Association for Computing Machinery.ISBN 978-1-4503-6975-6. doi:10.1145/3357251.3357591. URLhttps://doi.org/10.1145/3357251.3357591. → page 54[101] A. P. Schouten, T. C. Portegies, I. Withuis, L. M. Willemsen, andK. Mazerant-Dubois. Robomorphism: Examining the effects oftelepresence robots on between-student cooperation. Computers in HumanBehavior, 126:106980, Jan. 2022. ISSN 0747-5632.doi:10.1016/j.chb.2021.106980. URLhttps://www.sciencedirect.com/science/article/pii/S0747563221003034.→ pages 6, 13, 51[102] E. M. Segura, M. Kriegel, R. Aylett, A. Deshmukh, and H. Cramer. HowDo You Like Me in This: User Embodiment Preferences for CompanionAgents. In Y. Nakano, M. Neff, A. Paiva, and M. Walker, editors,Intelligent Virtual Agents, Lecture Notes in Computer Science, pages112–125, Berlin, Heidelberg, 2012. Springer. ISBN 978-3-642-33197-8.doi:10.1007/978-3-642-33197-8 12. → pages 1, 775[103] A. Shariati, M. Shahab, A. Meghdari, A. Amoozandeh Nobaveh,R. Rafatnejad, and B. Mozafari. Virtual Reality Social Robot Platform: ACase Study on Arash Social Robot. In S. S. Ge, J.-J. Cabibihan, M. A.Salichs, E. Broadbent, H. He, A. R. Wagner, and Castro-Gonza´lez, editors,Social Robotics, Lecture Notes in Computer Science, pages 551–560,Cham, 2018. Springer International Publishing. ISBN 978-3-030-05204-1.doi:10.1007/978-3-030-05204-1 54. → pages 8, 18[104] K. W. C. Shin and J. Han. Children’s Perceptions of and Interactions with aTelepresence Robot. In The Eleventh ACM/IEEE International Conferenceon Human Robot Interaction, HRI ’16, pages 521–522, Christchurch, NewZealand, Mar. 2016. IEEE Press. ISBN 978-1-4673-8370-7. → page 1[105] S. Shin, D. Kang, and S. S. Kwak. Telepresence Robot for Isolated Patientsin the COVID-19 Pandemic: Effects of Socio-relationship andTelecommunication Device Types on Patients’ Acceptance of Robots. InF. Cavallo, J.-J. Cabibihan, L. Fiorini, A. Sorrentino, H. He, X. Liu,Y. Matsumoto, and S. S. Ge, editors, Social Robotics, Lecture Notes inComputer Science, pages 263–276, Cham, 2022. Springer NatureSwitzerland. ISBN 978-3-031-24670-8.doi:10.1007/978-3-031-24670-8 24. → page 1[106] J. Short. The social psychology of telecommunications /, 1976. Place:London : Publisher: Wiley,. → page 7[107] D. Sirkin and W. Ju. Consistency in physical and on-screen actionimproves perceptions of telepresence robots. In Proceedings of the seventhannual ACM/IEEE international conference on Human-Robot Interaction,HRI ’12, pages 57–64, New York, NY, USA, Mar. 2012. Association forComputing Machinery. ISBN 978-1-4503-1063-5.doi:10.1145/2157689.2157699. URLhttps://doi.org/10.1145/2157689.2157699. → pages 8, 13, 52[108] B. Stoll, S. Reig, L. He, I. Kaplan, M. F. Jung, and S. R. Fussell. “Wait,Can You Move the Robot?”: Examining Telepresence Robot Use inCollaborative Teams. In 2018 13th ACM/IEEE International Conferenceon Human-Robot Interaction (HRI), pages 14–22, Mar. 2018. ISSN:2167-2148. → pages 53, 57[109] S. Straus and J. McGrath. Does the Medium Matter? The Interaction ofTask Type and Technology on Group Performance and Member Reactions.76The Journal of applied psychology, 79:87–97, Mar. 1994.doi:10.1037//0021-9010.79.1.87. → page 57[110] H. Sumioka, S. Nishio, and H. Ishiguro. Teleoperated android for mediatedcommunication : body ownership, personality distortion, and minimalhuman design. pages 32–39, Sept. 2012. → page 56[111] R. Suzuki, H. Hedayati, C. Zheng, J. L. Bohn, D. Szafir, E. Y.-L. Do, M. D.Gross, and D. Leithinger. RoomShift: Room-scale Dynamic Haptics forVR with Furniture-moving Swarm Robots. In Proceedings of the 2020 CHIConference on Human Factors in Computing Systems, CHI ’20, pages1–11, New York, NY, USA, Apr. 2020. Association for ComputingMachinery. ISBN 978-1-4503-6708-0. doi:10.1145/3313831.3376523.URL https://doi.org/10.1145/3313831.3376523. → page 54[112] R. Suzuki, A. Karim, T. Xia, H. Hedayati, and N. Marquardt. AugmentedReality and Robotics: A Survey and Taxonomy for AR-enhancedHuman-Robot Interaction and Robotic Interfaces. In CHI Conference onHuman Factors in Computing Systems, pages 1–33, New Orleans LA USA,Apr. 2022. ACM. ISBN 978-1-4503-9157-3.doi:10.1145/3491102.3517719. URLhttps://dl.acm.org/doi/10.1145/3491102.3517719. → page 6[113] L. Takayama and J. Go. Mixing metaphors in mobile remote presence. InProceedings of the ACM 2012 conference on Computer SupportedCooperative Work, CSCW ’12, pages 495–504, New York, NY, USA, Feb.2012. Association for Computing Machinery. ISBN 978-1-4503-1086-4.doi:10.1145/2145204.2145281. URLhttps://doi.org/10.1145/2145204.2145281. → pages 2, 5, 12, 51[114] F. Tanaka, T. Takahashi, S. Matsuzoe, N. Tazawa, and M. Morita.Telepresence robot helps children in communicating with teachers whospeak a different language. In Proceedings of the 2014 ACM/IEEEinternational conference on Human-robot interaction, HRI ’14, pages399–406, New York, NY, USA, Mar. 2014. Association for ComputingMachinery. ISBN 978-1-4503-2658-2. doi:10.1145/2559636.2559654.URL https://doi.org/10.1145/2559636.2559654. → pages 1, 8[115] I. Torre, A. Linard, A. Steen, J. Tumova´, and I. Leite. Should RobotsChicken? How Anthropomorphism and Perceived Autonomy InfluenceTrajectories in a Game-theoretic Problem. In Proceedings of the 2021ACM/IEEE International Conference on Human-Robot Interaction, HRI77’21, pages 370–379, New York, NY, USA, Mar. 2021. Association forComputing Machinery. ISBN 978-1-4503-8289-2.doi:10.1145/3434073.3444687. URLhttps://doi.org/10.1145/3434073.3444687. → page 8[116] T.-C. Tsai, Y.-L. Hsu, A.-I. Ma, T. King, and C.-H. Wu. Developing aTelepresence Robot for Interpersonal Communication with the Elderly in aHome Environment. Telemedicine and e-Health, 13(4):407–424, Aug.2007. ISSN 1530-5627. doi:10.1089/tmj.2006.0068. URLhttps://www.liebertpub.com/doi/10.1089/tmj.2006.0068. Publisher: MaryAnn Liebert, Inc., publishers. → page 1[117] K. M. Tsui, J. M. Dalphond, D. J. Brooks, M. S. Medvedev, E. McCann,J. Allspaw, D. Kontak, and H. A. Yanco. Accessible Human-RobotInteraction for Telepresence Robots: A Case Study. Paladyn, Journal ofBehavioral Robotics, 6(1), Jan. 2015. ISSN 2081-4836.doi:10.1515/pjbr-2015-0001. URLhttps://www.degruyter.com/document/doi/10.1515/pjbr-2015-0001/html.Publisher: De Gruyter Open Access. → page 1[118] J. van Houwelingen-Snippe, J. Vroon, G. Englebienne, and P. Haselager.Blame my telepresence robot joint effect of proxemics and attribution oninterpersonal attraction. In 2017 26th IEEE International Symposium onRobot and Human Interactive Communication (RO-MAN), pages 162–168,Aug. 2017. doi:10.1109/ROMAN.2017.8172296. ISSN: 1944-9437. →pages 3, 18[119] G. Venolia, J. Tang, R. Cervantes, S. Bly, G. Robertson, B. Lee, andK. Inkpen. Embodied social proxy: mediating interpersonal connection inhub-and-satellite teams. In Proceedings of the SIGCHI Conference onHuman Factors in Computing Systems, CHI ’10, pages 1049–1058, NewYork, NY, USA, Apr. 2010. Association for Computing Machinery. ISBN978-1-60558-929-9. doi:10.1145/1753326.1753482. URLhttps://doi.org/10.1145/1753326.1753482. → page 1[120] A. M. Villanueva, Z. Liu, Z. Zhu, X. Du, J. Huang, K. A. Peppler, andK. Ramani. RobotAR: An Augmented Reality Compatible TeleconsultingRobotics Toolkit for Augmented Makerspace Experiences. In Proceedingsof the 2021 CHI Conference on Human Factors in Computing Systems,CHI ’21, pages 1–13, New York, NY, USA, May 2021. Association forComputing Machinery. ISBN 978-1-4503-8096-6.78doi:10.1145/3411764.3445726. URLhttps://doi.org/10.1145/3411764.3445726. → page 6[121] F. Weidner, G. Boettcher, S. A. Arboleda, C. Diao, L. Sinani, C. Kunert,C. Gerhardt, W. Broll, and A. Raake. A Systematic Review on theVisualization of Avatars and Agents in AR & VR displayed usingHead-Mounted Displays. IEEE Transactions on Visualization andComputer Graphics, 29(5):2596–2606, May 2023. ISSN 1941-0506.doi:10.1109/TVCG.2023.3247072. URLhttps://ieeexplore.ieee.org/abstract/document/10049669. ConferenceName: IEEE Transactions on Visualization and Computer Graphics. →page 16[122] V. Weistroffer, A. Paljic, P. Fuchs, O. Hugues, J.-P. Chodacki, P. Ligot, andA. Morais. Assessing the acceptability of human-robot co-presence onassembly lines: A comparison between actual situations and their virtualreality counterparts. In The 23rd IEEE International Symposium on Robotand Human Interactive Communication, pages 377–384, Edinburgh, UK,Aug. 2014. IEEE. ISBN 978-1-4799-6765-0 978-1-4799-6763-6.doi:10.1109/ROMAN.2014.6926282. URLhttp://ieeexplore.ieee.org/document/6926282/. → pages 8, 19[123] L. R. Wheeless and J. Grotz. The Measurement of Trust and ItsRelationship to Self-Disclosure. Human Communication Research, 3(3):250–257, 1977. ISSN 1468-2958.doi:10.1111/j.1468-2958.1977.tb00523.x. URL https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1468-2958.1977.tb00523.x.eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1468-2958.1977.tb00523.x. → pages 34, 35[124] S. Whittaker. Theories and Methods in Mediated Communication. TheHandbook of Discourse Processes, Aug. 2002. → page 35[125] L. Wijnen, P. Bremner, S. Lemaignan, and M. Giuliani. PerformingHuman-Robot Interaction User Studies in Virtual Reality. In 2020 29thIEEE International Conference on Robot and Human InteractiveCommunication (RO-MAN), pages 794–794, Aug. 2020.doi:10.1109/RO-MAN47096.2020.9223521. ISSN: 1944-9437. → page 19[126] B. Yoon, H.-i. Kim, G. A. Lee, M. Billinghurst, and W. Woo. The Effect ofAvatar Appearance on Social Presence in an Augmented Reality RemoteCollaboration. In 2019 IEEE Conference on Virtual Reality and 3D User79Interfaces (VR), pages 547–556, Mar. 2019.doi:10.1109/VR.2019.8797719. URLhttps://ieeexplore.ieee.org/abstract/document/8797719. ISSN: 2642-5254.→ page 16[127] J. Zhang, N. Katzakis, F. Mostajeran, and F. Steinicke. LocalizingTeleoperator Gaze in 360° Hosted Telepresence. In 2019 IEEE Conferenceon Virtual Reality and 3D User Interfaces (VR), pages 1265–1266, Osaka,Japan, Mar. 2019. IEEE. ISBN 978-1-72811-377-7.doi:10.1109/VR.2019.8797819. URLhttps://ieeexplore.ieee.org/document/8797819/. → page 680Appendix ASupporting MaterialsThe supporting materials shared in the following appendix items for the studyrefers to the ”AvatARoid” system as ”Meta-Humaonid” and ”Teleoperator” asa ”Remote User”. The changes were made post-data collection during report doc-umentation for creative purposes.81Study Title : “Meta-Humanoid : Robot-Mediated Telepresence using Visual Overlays ofRemote User (In-person Study)”Researchers:Principal Investigator:Dongwook Yoon, Computer Science Department, UBCEmail: redactedCo-Investigator:Amit Ghimire, Computer Science Department, UBCEmail: redactedDescription: This study aims to learn about how people interact in a robot-mediatedtelepresence system. Participants will use a Mixed Reality (MR) simulation tool to interactwith a virtual robot controlled by another participant and perform two tasks together: 1) Q&Ato get to know each other and 2) Play a word-guessing game. No prior MR experience isneeded.Participants will provide feedback via a survey and an interview following the tasks.This study will take approximately 1 hour, and participants will be compensated between$30-$50 for their time.Eligibility:Participants must meet all of the following criteria:● Be at least 19 years of age.● Have normal or corrected to normal vision.● Have no major motor impairments.● Understand and speak English.Location: UBC Computer Science Building (ICICS/CS) - Room X521. Address: 2366 MainMall, Vancouver, BC V6T 1Z4 (Maps link https://goo.gl/maps/czxoPVkbPxbatLpKA )Contact Information: If you are interested in participating and meet the eligibility criteria,please contact Amit Ghimire via an email(redacted) or Text (redacted). Please include“Meta-Humanoid Study Participation” as your subject or in the text. We will be in touch withfurther study information and scheduling.Reimbursement/Time: Participants will be compensated between $30-$50 for 1 hour.A.1 Recruitment Post in Paid Studies Listing82Consent Form for‘Meta-humanoid : Improving robot mediatedtelepresence using visual overlay of remoteuser‘Department of ComputerScience2366 Main MallVancouver, B.C., V6T 1Z4Principal Investigator:Dongwook Yoon, Computer Science Department, University of British Columbia.Ph: redacted; Email: redacted.Co-Investigator(s):Amit Ghimire, Computer Science Department, University of British Columbia.Ph: redacted; Email: redacted.This is a graduate student project of Mr. Amit Ghimire as a part of the requirementsfor completion of a masters degree.Sponsor:This project is funded by the Natural Sciences and Engineering Research Council(NSERC) and Korea Institute of Science and Technology (KIST)Purpose:The primary purpose of this research is to develop and evaluate methods to improvetelepresence humanoid robot systems with Augmented Reality (AR) technology. Asa secondary objective, we also want to develop and evaluate a Mixed Reality (MR)simulation tool to conduct Human Robot Interaction experiments.Study Procedure: After you have read this document, Amit Ghimire will respond toany questions or concerns that you may have.Once you have signed this consent form, you will be asked to:● fill out a survey for basic demographic data (i.e., gender, age, occupation)● get a brief introduction to our simulation tool along with instructions regardingthe safe usage of MR experience.● participate in a task-based experiment where you will use our tool to interactwith a simulated robot in MR controlled by a remote user● fill out a survey where you will rate our tool, your interactions with the remoteuser and your experience with the telepresence system.● answer interview questions related to your experience of using thetelepresence systemEthics ID: H22-01390 Version 4.0 / June 12, 2023A.2 Consent Form83The demographics survey should take approximately 5 minutes to fill out, and thetasks and interviews should take approximately 55 minutes. The entire study will becompleted in a single session.The sessions may be video and/or audio recorded. Any identifiable element in theimages or audios will be blurred or obscured to prevent identification. You have theoption not to be video/audio recorded.Participants are free to withdraw without giving a reason. If you withdraw from thestudy, all of the data obtained from you will be permanently discarded and will not beused for this study. All of the electronic files will be permanently deleted and all of thephysical copies of documents (e.g., handwritten notes, paper transcriptions and theconsent form) obtained from you will be shredded. Your name will also be removedfrom the code assignment file.Notice of COVID-Related RisksThis study requires one-one interaction between participants and the researcher. Inthe light of Covid-19 pandemic, we require participants to wear a non-surgical facialmask during the entire duration of the study. The sessions will take place followingthe physical distancing and COVID-19 protocols in a well ventilated room. No morethan one participant will be allowed in the session at a time and necessaryequipment (e.g, MR headsets and controllers) and surfaces will be sanitized beforeand after each use.Please follow this link for further information about Notice of COVID-Related risks.https://ethics.research.ubc.ca/sites/ore.ubc.ca/files/documents/Notice%20of%20COVID-related%20risks.pdfProject Outcomes:Although the project outcomes will be determined by the research findings, possibleresearch products will include: journal articles, a report, and plain languagesummaries, and videos.Potential Benefits:There are no explicit benefits to participants. However, the purpose of the study is toimprove research that can help us create better MR simulation tools for testingHuman Robot Interaction scenarios.Potential Risks:This study does not require any physical interaction while using the MR headsetsand the participants will be seated throughout the experiment while using the MREthics ID: H22-01390 Version 4.0 / June 12, 202384headset. Hence there are not any major risks associated with our study. However, Itis well known that the use of a head mounted display for MR applications, especiallyfor long durations, may cause dizziness, nausea, or fatigue in some users known assimulator sickness. To mitigate any risks associated with simulator sickness, wefollow current best practices (e.g. not having any locomotion/excessivemotion/rotation, not using flashing lights/colors in the scene, etc), while developingour simulation. In addition, time during the experiment will explicitly be set aside forbreaks. You are encouraged to report any feelings of dizziness, nausea, or fatigueimmediately, and are free to take a break at any point. You may also discontinue thestudy session without any penalty if you desire. A researcher will be present with theparticipant throughout the entire session to monitor the situation and provideassistance and answers to questions if necessary.Confidentiality:The audio recordings will be transcribed and all identifiable data such as theparticipant's name will be removed from the transcription and all identifiable elementsin the images, or audios will be blurred or obscured. All the hard copies ofdocuments as well as electronic copies will be identified only by code numbers thatAmit Ghimire will assign to each participant and therefore will not contain anyidentifiable data. This means that you will not be identified by name in either thescreen recording, survey data or interview transcript. The link between the codeassociated and the actual names will be stored in the “code assignment file”. Theonly documents containing your real name will be the “code assignment file” and thisconsent form. The code assignment file will be password protected, encrypted, andstored separately from the data.During the study any handwritten notes and paper transcripts will be kept in a lockedcabinet in the researchers' laboratory (Room X660 at ICICS) with controlled accessat UBC, and all electronic files including audio files and “code assignment file” will betemporarily stored on an encrypted hard drive of a password protected andencrypted laptop owned by Amit Ghimire, and there will not be any other copy storedthan the one in Amit’s hard drive.Currently, we do not plan to destroy the data. The data will be stored for a minimumof 5 years following publication at UBC. The principal investigator, Dongwook Yoon,will be responsible for the data.No identifiable data will be published in any of the publishing contents (i.e., anyreports, research papers, thesis documents, and presentations). Participants will benamed by the code Amit Ghimire will assign, instead of their real names. The audiofiles will not be used in any publishing content and the transcription of the audio fileswill be modified to remove all identifying details of the participants. The pictures ofEthics ID: H22-01390 Version 4.0 / June 12, 202385the experiment may be used for the publication, however, any identifiable elementwill be blurred to prevent identification.Open AccessMaking our research publication open access means the research data and resultswill be made available for access to anyone on the internet without any cost or otherrestrictions. The data will be made available to publishing journals or institutions whowill make the research available via their website. If our publication is made openaccess, we will make sure all your identifiable data will be removed before the data isprovided. You can opt to withdraw your participation and data anytime before thepublication is made open access but once it is made open access, you will not beable to withdraw your data. List types of data : interview response, survey response,images - all deidentifiedRemuneration/Compensation:In order to acknowledge the time you have taken to be involved in this project, eachparticipant will receive $30 for one session (45-60 minutes) of experiment. Inaddition, participants will receive up to $20 based on their decision in an investmentgame in our study, making the total ranging from a minimum of $30 to a maximum of$50.Contact for information about the study:If you have any questions or desire further information with respect to this study, youmay contact Amit Ghimire (redacted).Contact for concerns or complaints about the study:If you have any concerns or complaints about your rights as a research participantand/or your experiences while participating in this study, contact the ResearchParticipant Complaint Line in the UBC Office of Research Ethics at 604-822-8598 orif long distance e-mail RSIL@ors.ubc.ca or call toll free 1-877-822-8598.Consent:Your participation in this study is entirely voluntary and you may refuse to participateor withdraw from the study at any time. Your signature below indicates that you havereceived a copy of this consent form for your own records. Your signature indicatesthat you consent to participate in this study.Please Check off the YES box if YOU AGREE and NO box if you DO NOTAGREE with our condition:YES[ ] NO[ ] For the interview, I agree that a video recording can be made of whatI have to say.Ethics ID: H22-01390 Version 4.0 / June 12, 202386YES[ ] NO[ ] For the interview, I agree that an audio recording can be made ofwhat I have to say.YES[ ] NO[ ]I agree that video recordings can be made of me performing theassigned tasks during the studyYES[ ] NO[ ]I agree that behavioral data(no. of words spoken, gaze) can berecorded of me while performing the assigned tasks during the studyYES[ ] NO[ ]I agree that I have read and understood the “Potential Risk” and“Notice of COVID-Related Risks during Research”YES[ ] NO[ ]I agree to complete Covid Health Check prior to taking part inin-person research activityYES[ ] NO[ ]I agree for the research team to retain my contact information forfollow up if needed.Statement of Consent: I, ________________________________, have read theabove information, and have received answers to any questions I asked. I consent totake part in the study. I give permission for assignment submissions and mycomments on the submissions that may possibly be used for future publications inthe form of conference proceedings, research journal, research report, videos andacademic thesis.___________________________________________________________________Participant Signature Date______________________________Printed NamePS: You can also reply to the email through which you received this consent formwith a statement “I agree to the terms of the study” as your consent.Ethics ID: H22-01390 Version 4.0 / June 12, 202387Experimenter Study ProtocolPre-Study ChecklistTest out network connection of quest pro and RU Laptop.Check Ip Address of the RU laptopNew Build if it changed from 192.168.50.108Fix the layout of the room.Setup Guardian + Floor LevelSetup Scene ManagementDeclutter the roomCheck blinds in the window (block out the window light)Sanitize Headset and ControllersCheck Hand Sanitizer in placeHave the following items with youTask Flow diagram sheet10 1-dollar bills2 Envelopes for the investment gameLaptop, MouseMobile device for castingIpad and recorder for audio recordingCashCompensation SheetConnect Quest with ADB via IP in the experimenter laptopConnect Mic and Test Recording in the experimenter laptopGet the survey link ready.https://ubc.ca1.qualtrics.com/jfe/form/SV_8kRGUSMZ4guJfNQEnter Participant ID and condition in the qualtrics survey.Have phone ready and connected to headsetMake sure we can castA.3 Study Protocol for the Experimenter88Study Script: Experimenter● Introduction: [3 min]Remind to sanitize handsStart RecordingHello, my name is Amit. I’m working with Professor Dongwook Yoon (at UBC ComputerScience Department). Thank you for your participation in our study.Please confirm that this is the consent form you signed for this study that you have filledout. (Verify electronic consent form)Please fill out the demographic form (give form to participant)Confirm and verify the electronic consent form.Complete demographics questionnaire.Thank you. This study will take approximately 1 hour, and you will be compensatedbetween $30 to $50. Now, I’ll brief you on the task flow of our study.● Task Flow Briefing: [3 min]Our goal for this study is to learn about how people work together when a robot is usedas a stand-in. We want to learn how someone interacts with a robot that is controlled byanother person, like a coworker or teammate, who is in a different physical location.You’ll work with a fellow participant in this study named Andy. They’re located in adifferent room in this building and are also being briefed right now.Give participants the task flow sheet with instructions.There are 4 steps in this study. After the 4 steps, there will be a post study survey andinterview.89● Step 1 : Investment Game [6 min]Briefing [3 min]You will now play an investment game with Andy.I will explain the rules to you,You will be given an initial sum of 5 dollars. Here are your 5 dollars.Hand Participants 5 1-dollar billsYou now have 2 moves: to invest OR not invest.You can either keep the 5 dollars or invest any amount of this 5 dollars into the game.If you do not invest or decide to keep all of the $5, we will give you the $5. In this caseAndy receives nothing and the game ends.If you decide to invest, Andy will receive double the amount you invest. For example, ifyou invest the entire $5 now, we will give Andy $10. Then, Andy will be given the twochoices: to keep the total amount of money they received, OR invest any of this amountback into the game.We will give you back the amount they invest back in the game. For example, if youdecide to invest $5 and Andy decides to invest all of the $10 they received, you willreceive $10. If they decide to keep the money given to them, you will not receive anymoney back.You will be informed about the result of this game at the end of the experiment. We arenot disclosing the result of this game since it might inform your behavior in the next stepsof the study.Please put the amount you want to keep in this keep box and the amount you want toinvest in this invest box.Hand Participants the two keep/invest boxes.Source: https://en.wikibooks.org/wiki/Bestiary_of_Behavioral_Economics/Trust_Game90● Briefing and Headset Adjustment [5 min]For Step 2 and 3, you will be wearing this headset. I will help you wear and adjust theheadset. You can remain seated here throughout the experiment and don’t need to movefrom your spot.Show them the controllers and explain the thumbstick mechanism.Hand them the headset and explain how to wear it.Your forehead should rest on this part [point to the forehead rest] and your back ofhead should rest on this part [point to the back of the headset]. Rotate this dial tomake sure the headset fits comfortably. [show the dial]Once the headset fits comfortably, if you cannot read the text when I start theapplication, adjust the lens like this [show IPD adjustment]These are the controllers. You will only need to move these thumbsticks.If there are technical difficulties or if you feel discomfort at any point in the study, let usknow and we can pause the study and fix any issues. If any issues continue, we canstop the study.Help participants become familiar with using the VR Setup.Make sure they can see clearly in VR - that the headset is comfortablysnug on their head so that what they see is focused and not blurry.Make sure they are familiar with the Chaperone system - the boundarythat shows up on the screen to warn them of physical obstacles.Adjust Interpupillary Distance, headset depthConnect Phone to HeadsetStart CastingStart Capture from Quest ProStop voice for castingStart Audio CaptureStart application from adb {Try from MQDH, if fails try this}Adb devicesAdb shell ip routeAdb tcpip 5555Adb connect ip:5555Based on conditionAdb shell am start -ncom.dlab.metahumanoid/com.unity3d.player.UnityPlayerActivity91Adb shell am start -ncom.dlab.video/com.unity3d.player.UnityPlayerActivityAdb shell am start -ncom.dlab.baseline/com.unity3d.player.UnityPlayerActivityIf need to stopAdb shell am force-stop com.DLab.{appname}Recenter View from phone[Point to Robot]This is Andy, your teammate, who is operating the robot.Step 2 : Q and A session [~ 15 min]We will now move into Step 2: Answering questions in an interview-like task.First, Andy will ask you some questions. Feel free to answer them however you’d like. Ifyou feel uncomfortable answering any of the questions Andy asks, you can “skip” andmove on to the next question.After you answer all of the questions from Andy, there will be questions that pop up inyour headset view. You will ask these questions to Andy, and can navigate the questionset using thumbsticks in your controllers. You can move the thumbsticks to the right togo to the next questions, and to the left to go to the previous one. The pop-up will displayEND when the questions end.Step 3 : Word Guessing Game [5 min : 2 min briefing + 3 min game]Now, you and Andy will play a word guessing game.This is a game that involves a describer and a guesser. You will be the guesser. Andywill be the describer, who will be shown a word to describe.They can provide hints in any form, they just can’t say the word. Your role is to guess thewords correctly. You and Andy are working together, and your shared goal is to guess asmany words as possible within the time limit of 3 minutes. You are allowed to say skip.● Step 4 : Investment Game [3 min]92Stop Quest CastingStop Quest CaptureInstruct participants to take off their headset.Have 5 1-dollar bills and boxes for the investment game.You can take off the headset now. Now, you will play the investment game again withAndy. This game follows the identical rules as the first time you played. As a reminder,this is a new round and is completely independent from the first time you played.Hand Participants 5 1-dollar bills and the boxes● Post-Study Survey and Interview [15 min]Please complete the survey.Complete Likeability, Trustworthiness, Copresence with robot,Infrahumanization, Social Presence and Homophily Questionnaires.Observe survey results to help drive interviewSetup the mic to capture audio interviews.Complete post-task semi-structured interview.Questions● Wrap-Up [<1 min]This concludes our study.For this study, it was important that I withheld some information from you about someaspects of the study. Now that your participation is complete, I’ll describe whatinformation was withheld.1. Your teammate was a confederate member of our research team and not anotherparticipant. The purpose of withholding this information was to collect data abouthow you interacted with a peer/teammate.2. You will be compensated $50 for the study.This includes $20, the maximum amount you could have earned from the 2rounds of the Investment Game, AND your $30 fee for participation. Regardless93of how much you chose to invest and the final results of the game, your totalcompensation is $50.The purpose of withholding this information was to collect data about yourbehavior in playing the Investment Game.Thank you for your valuable input and time! Do you have any questions?Otherwise, here is the compensation for your participation in this study. Pleasealso sign this summary sheet indicating that you have received the money.Have participants sign a summary sheet regarding compensation and givethem the money.1. Post Study ChecklistMark participant name and ID together on a separate data sheet stored in asecure physical location.Stop and save interview audio recordings.Get Gaze Data Log from Quest ProCopyDelete2. Data ChecklistEthics Consent Form (signed,electronic copy)Initial demographics questionnaire (electronic)Embodiment questionnaireSocial Presence with RU(electronic)Likeability Questionnaire(electronic)Trustworthiness (electronic)Homophily (electronic)Infrahumanization (electronic)Co-presence with robot(electronic)RU POV (Video Recording)User POV (video recording)User Audio RecordingUser Gaze-Data LogsUser Interview AudioCoded notes of post-study interview94Remote User ProtocolRemote User ChecklistCheck audio input and output2-USB Microphone2-USB SpeakerCheck the state of the Unity conditionWhichever the state of the participant, do not have that stateenabled in Unity-Start video capture (remote user screen).Windows key + GRecordMake sure Unity screen is being recordedTurn microphone on (in game bar)Press play in unity before participants wear headsetOnce there are 2 PhotonNetworkPlayers you can start animationPress 0 to enter the room (start animation).After reaching target, Press 1 to take control of robot (away fromanimation)End, Press 7 to leaveStop recording and saveThumbs up from Amit - play 0Say Hi, if say hiWait for Amit introduction of the interviewStep 2: Interview[After Introduction from Amit] (he will leave room)Hi, I’m Andy, what’s your name? (response from participant) Nice to meet you.Okay, I’ll start the formal interview now. I’ll ask the questions provided to me, andyou’ll ask them back to me.A.4 Study Protocol for the Teleoperator Confederate951. How are you?2. What do you do? Are you a student or are you working? Or neither.3. What initially attracted you to this field of work or study?4. What is something that you enjoy about what you do?5. Are there any hobbies or projects you are passionate aboutcurrently?6. What are some of your long-term goals or aspirations, eitherprofessionally and personally?7. What is the best piece of advice you’ve ever received?Thanks for answering my questions! I think it’s your turn to ask questions back tome now. Go ahead.After remote user interview ends, press 3Press 3 on the keypad[Scripted responses]31. I’m pretty good, thanks2. I’m an undergrad studying Marketing3. I’ve been interested in business and communication, and I think it’s auseful skill to have in any industry4. I enjoy working with others and brainstorming creative ideas5. Yes, I’m learning the guitar outside of school6. A long term goal for me would be to travel the world7. The piece of advice I’ve ever received is to never stop learning or growingas a person, and to treat every opportunity as your firstAfter participant interview ends, press 5Press 5 on the keypadStep 3: Word Guessing Game [3 mins](After briefing. Amit leaves the room, then start)96Are you ready? I’ll start with the first word for the guessing game.DO NOT: step out of the designated spot/move in and out of frame[Read out hints for words]Okay, time’s up. Good job! And thanks for working with me!👏Amit comes back into room, gesture “will leave/wave goodbye”It was nice meeting you. See you later, bye!At the end of Step 3Press 7 to leave and save dataStop the video capture and save the videoWindows + GHintsLightningHint 1: It’s something that happens in nature.Hint 2: It occurs when there is a sudden release of electrical energy.Hint 3: It is often accompanied by thunder.Hint 4: It can be dangerous during a storm.HieroglyphicsHint 1: It is a form of ancient writing.Hint 2: It uses pictorial symbols or characters.Hint 3: It was used by ancient Egyptians.0BookHint 1: It contains written or printed pages.Hint 2: It can be fiction or non-fiction.Hint 3: It is often read for entertainment or education.97ClockHint 1: It tells time.Hint 2: It has hands and numbers.Hint 3: It can be analog or digital.CloudHint 1: It is something white or grey that appears in the sky.Hint 2: It can be fluffy or wispy.Hint 3: It can bring rain or shade.ChameleonHint 1: It is a reptile/animal that can camouflage.Hint 2: It has the ability to change color.AlgorithmHint 1: It is a step-by-step procedure or set of rules.Hint 2: It is used in mathematics, computer science, and problem-solving.Hint 3: It can be represented by flowcharts or pseudocode.ChairHint 1: It is a piece of furniture used for sitting.Hint 2: It typically has a back and four legs.Hint 3: It can be found in various sizes and styles.TreeHint 1: It is a tall, woody perennial plant.Hint 2: It has branches, leaves, and roots.Hint 3: It provides shade and produces oxygen.CarHint 1: It is a mode of transportation.Hint 2: It has wheels and is usually powered by an engine.Hint 3: It is commonly used for personal and public transportation.GuitarHint 1: It is a musical instrument.Hint 2: It typically has six strings.98Hint 3: It is played by strumming or plucking the strings.CoffeeHint 1: It is a popular beverage.Hint 2: It is made from roasted coffee beans.Hint 3: It can be enjoyed hot or cold.CameraHint 1: It is a device used for taking photographs or recording videos.Hint 2: It captures and preserves visual memories.Hint 3: It comes in various types, such as digital or film cameras.SunHint 1: It is a star at the center of our solar system.Hint 2: It provides light and heat to Earth.Hint 3: It rises in the east and sets in the west.DogHint 1: It is a common household pet.Hint 2: It barks and wags its tail.99A.5 Demographics Survey1. What is your name?2. What is your email address?3. With which gender do you most identify?• Man• Woman• Transgender Man / Trans Man• Transgender Woman / Trans Woman• Non-Binary• Two-spirit• Not Listed• Prefer not to answer4. If you identify with gender not listed above, please add in this field (leaveblank if prefer not to answer)5. What is your educational qualification? (Ex. High School, Bachelor’s, Mas-ter’s, etc, leave blank if not applicable)6. What subject are/were your studies in? (Leave blank if not applicable)7. What is your occupation?8. What is your age?• 19-24• 25-34• 35-44• 45-54• 55-64• 65 or older100• Prefer not to answer9. What is your cultural background? Select all that apply.• African• European• East Asian• South Asian• Southeast Asian• Hispanic or Latinx• Middle Eastern• First Nations or Indigenous (Please specify below)• Other (Please specify below)• Prefer not to answer10. Please specify your cultural background here (Leave blank if prefer not toanswer).A.6 Post Study QuestionnaireA.6.1 Perceived Embodiment of the Teleoperator7 point agreement rating(from very strongly disagree to very strongly agree) forthe following questions.1. I felt like the other participant was a real person.2. I felt like I was communicating with a robot.3. I felt like the robot’s body was the other participant’s body.4. I felt like the robot’s movements were caused by the other participant’smovements.5. I felt like the robot was moving by itself.6. I felt like the other participant was located where I saw the robot.101A.6.2 Social Presence of the Teleoperator7 scale rating along semantic differential items for following questions1. To what extent did you feel able to assess your partner’s reactions to whatyou said? [Able to assess reactions —- not able to assess reactions]2. To what extent was this like a face-to-face meeting? [A lot like face to face—- not like face to face at all]3. To what extent was this like you were in the same room with your partner?[A lot like being in the same room —- not like being in the same room atall]4. To what extent did your partner seem “real”? [Very real —- not real at all]5. How likely is it that you would choose to use this system of interaction for ameeting in which you wanted to persuade others of something? [Very likely—- not likely at all]6. To what extent did you feel you could get to know someone that you metonly through this system? [Very well —- not at all]A.6.3 Impression of the TeleoperatorTrustworthiness7 scale rating along following semantic Differential items1. Trustworthy - Untrustworthy2. Confidential - Divulging3. Benevolent - Exploitative4. Safe - Dangerous5. Candid - Deceptive6. Not Deceitful - Deceitful1027. Straightforward - tricky8. Respectful - Disrespectful9. Considerate - Inconsiderate10. Honest - Dishonest11. Reliable - Unreliable12. Faithful - Unfaithful13. Sincere - Insincere14. Careful - CarelessLikeability7 point agreement rating(from very strongly disagree to very strongly agree) forthe following questions.1. This person is friendly.2. This person is likable.3. This person is warm.4. This person is approachable.5. I would ask this person for advice.6. I would like this person as a coworker.7. I would like this person as a roommate.8. I would like to be friends with this person.9. This person is physically attractive.10. This person is similar to me.11. This person is knowledgeable.103Homophily7 scale rating along following semantic Differential items1. Doesn’t think like me: Thinks like me2. Behaves like me: Doesn’t behave like me3. Similar to me: Different from me4. Unlike me: Like me5. Treats people like Ido: Doesn’t treat people like I do6. Looks similar to me: Looks different from me7. Different size than I am: Same size I am8. Appearance like mine: Appearance unlike mine9. Doesn’t resemble me: Resembles meInfrahumanizationSelect 8-10 words from below that you think they may experience in a given day1. Attraction2. Desire3. Excitement4. Pleasure5. Agitation6. Anger7. Fear8. Rage9. Regret10410. Disappointment11. Compassion12. Love13. Hope14. Admiration15. Bitterness16. EnthusiasmA.6.4 Co-presence with the RobotCombination of 7 scale likert scale and differential scale items1. I perceived that I was in the presence of the robot in the room with me2. I felt the robot was watching me and was aware of my presence.3. I would feel startled if the robot came closer to me.4. To what extent did you have a sense of being with the robot? [Not at all —-Very Much]5. To what extent was this like you were in the same room with the robot? [Notat all —- Very Much]6. I felt I was in the ......... space. [Virtual —- Physical]7. I felt the robot was in .......... space. [Virtual —- Physical]A.7 Post Study Interview Questions1. In the questionnaire, you rated the [specific aspect] as [rating]. Can youshare more about what led to that rating? What was going through yourmind when you answered [specific question]?1052. What were your thoughts on the visual representation of Andy? Did youhave difficulties identifying the robot as Andy? Did you feel like Andy wascontrolling the robot? Could you recognize Andy’s gestures or movementsor body languages through the robot? Could you elaborate?3. How aware were you of the robot versus Andy? Can you share momentswhen Andy felt really ’human’ or more robot-like? If so, can you describethese moments?4. If you could change anything about this setup, what would that be?5. Did you feel like you were in the same room as the robot? Were there mo-ments when you remembered you were interacting with a virtual robot orwhen it felt like a real robot?6. How did you feel about the AR experience? How much did the visual qualityinterfere or distract you? How much did the control devices interfere withthe performance? How much did the sounds of the environment involveyou? Cou1ld you identify the sound source and where the sound was comingfrom?7. Can you tell us a bit about your past experience/familiarity with Robots?8. Can you tell us a bit about your past experience/familiarity with AR/VRdevices?9. On a scale of 1 to 4, 1 being completely AI and 4 being completely Human,how much would you rate Andy to be an AI or a human? (find out if itaffected their rating)10. On a scale of 1 to 4, 1 being a member of the research confederate and 4 be-ing another participant, how much would you rate Andy to be a confederateor another participant? (find out if it affected their ratings)106
Thesis/Dissertation
eng
Computer Science
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Yoon, Dongwook
AvatARoid : using a motion-mapped AR overlay to bridge the embodiment gap between robot and teleoperator in robot-mediated telepresence
Text