UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Towards enabling human-robot handovers : exploring nonverbal cues for fluent human-robot handovers Pan, Matthew Keith Xi-Jie 2018

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


24-ubc_2018_may_pan_matthew.pdf [ 19.29MB ]
JSON: 24-1.0364493.json
JSON-LD: 24-1.0364493-ld.json
RDF/XML (Pretty): 24-1.0364493-rdf.xml
RDF/JSON: 24-1.0364493-rdf.json
Turtle: 24-1.0364493-turtle.txt
N-Triples: 24-1.0364493-rdf-ntriples.txt
Original Record: 24-1.0364493-source.json
Full Text

Full Text

Towards Enabling Human-Robot Handovers: ExploringNonverbal Cues for Fluent Human-Robot HandoversbyMatthew Keith Xi-Jie PanB.A.Sc. , The University of Waterloo, 2009M.A.Sc. , The University of British Columbia, 2012A THESIS SUBMITTED IN PARTIAL FULFILLMENTOF THE REQUIREMENTS FOR THE DEGREE OFDoctor of PhilosophyinTHE FACULTY OF GRADUATE AND POSTDOCTORALSTUDIES(Mechanical Engineering)The University of British Columbia(Vancouver)March 2018c©Matthew Keith Xi-Jie Pan, 2018AbstractFundamental human to human interactions - sharing spaces, tools, handingover objects, carrying objects together - are part of the everyday experience; formost people, the task of handing over an object to another person is a natural andseemingly effortless task. However, in the context of human-robot interaction,smooth and seamless interaction is an open problem of fundamental interest forrobotics designers, integrators and users alike. This thesis explores how nonver-bal cues exhibited during robot giving and receiving behaviours change how usersperceive the robot, and affect the handover task. Additionally, the work also inves-tigates how robots can recognize and interpret expressions conveyed by a humangiver to infer handover intent.Over the course of several user studies examining human-human and human-robot handovers, the role of nonverbal cues such as gaze and object orientationand how they may play a part in establishing fluency and efficiency of robot-to-human handovers are investigated. These studies provide insights into how robotscan be trained through observation of human-to-human handovers. Furthermore,this thesis examines the role of nonverbal cues in the less-studied human-to-robothandover interaction. In this exploration, kinematic features from motion-capturedskeleton models of a giver are used to establish the intent to handover, therebyenabling a robot to appropriately react and receive the object. Additionally, chang-ing user perceptions, geometry and dynamics of human-to-robot handovers areexplored through variation of initial pose, grasp type during, and retraction speedafter handover of the robot receiver.Findings from this thesis demonstrate that nonverbal cues such as gaze and ob-ject orientation in the case of robot-to-human handovers, and kinodynamics duringiihuman-to-robot handovers, can significantly affect multiple aspects of the interac-tion including user perception, fluency, legibility, efficiency, geometry and fluidityof the handover. Using a machine learning approach, recognizing handover intentfrom nonverbal kinematics of the giver’s pose could also be performed effectively.Thus, the work presented in this thesis indicates that nonverbal cues can serve asa powerful medium by which details of a handover can be subtly communicatedto the human partner, resulting in a more natural experience in this ubiquitous,collaborative activity.iiiLay SummaryHandovers of objects play a major role in successful physical collaborationbetween people. The same is true for collaboration between humans and robots.The objective of this thesis is to explore how robots can both recognize and dis-play nonverbal behaviours to facilitate the receiving and giving of objects duringhandovers.In this work, nonverbal cues used by a robot such as human-inspired gaze cues,object orientation, initial pose of the robot arm, how the robot grasps the object,and the robot’s movement speed following handover were found to significantlyaffect user perception, fluency, legibility and efficiency of handovers. Addition-ally, movements of people handing over objects to other people were recorded andanalyzed to allow recognition of when a person is intending to handover an objectto a robot.Based on this work, future human-robot handovers can be designed to be moreefficient and fluent for both robots and humans.ivPrefaceThe work described in Chapters 3, 4 and 5 was conducted in the CollaborativeAdvanced Robotics and Intelligent Systems Laboratory at the University of BritishColumbia, Point Grey campus. These projects and associated methods were ap-proved by the University of British Columbia’s Research Ethics Board [certificate#H10-00503]. Projects in Chapters 6 and 7 use data collected at Disney ResearchLos Angeles. These projects have been approved by the Disney Research InternalReview Board [IRB number DR-IRB-Pan-2016-02] and have also been approvedby the University of British Columbia’s Research Ethics Board [certificate #H10-00503].A version of Chapter 3 has been published in the Proceedings of the Associa-tion for Computing Machinery (ACM)/Institute of Electrical and Electronics Engi-neers (IEEE) International Conference on Human-Robot Interaction [A Moon, DMTroniak, B Gleeson, et al. (2014) Meet Me Where I’m Gazing: How Shared Atten-tion Gaze Affects Human-Robot Handover Timing. In: Proceedings of the 2014ACM/IEEE International Conference on Human-Robot Interaction - Human-RobotInteraction (HRI) 2014, Bielefeld, Germany: ACM Press, pp. 334-341.]. B Gleeson,DM Troniak, A Moon and myself were the lead investigators responsible for theconcept formulation, planning and directing of the project. I was responsible forthe software development used in the experimental setup, as well as conducting theexperiment and performing the results analysis. The manuscript was collectivelywritten and edited by all authors.Parts of Chapter 4 have been published in the Proceedings of the IEEE/RoboticsSociety of Japan (RSJ) International Conference on Intelligent Robots and Sys-tems (IROS) [WP Chan, MKXJ Pan, EA Croft, et al. (2015) Characterization ofvhandover orientations used by humans for efficient robot to human handovers. In:2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS),IEEE, pp. 1-6.] WP Chan and I were jointly involved in the concept development,design, execution, and analysis of this experiment.A version of Chapter 5 has been published in the The International Journal ofRobotics Research (IJRR) [MK Pan, V Skervøy, WP Chan, et al. (2017) Auto-mated detection of handovers using kinematic features. The International Journalof Robotics Research, SAGE Publications, UK: London, England 36(5-7): 721-738.]. I was the lead investigator in this research, responsible for all major areasof concept formation, data collection and analysis, as well as manuscript compo-sition. WP Chan and I were jointly involved in the design and execution of thisexperiment. Data processing was performed with the assistance of V Skjervøy.Portions of Chapter 6 have appeared in the Human-Robot Interaction in Collab-orative Manufacturing Environments Workshop at the IEEE/RSJ International Con-ference on Intelligent Robots and Systems [MKXJ Pan, EA Croft, and G Niemeyer.(2017) Validation of the Robot Social Attributes Scale (RoSAS) for Human-RobotInteraction through a Robot-to-Human Handover Use Case. In: 2017 IEEE/RSJInternational Conference on Intelligent Robots and Systems (IROS) - Workshops,IEEE]. Additionally, a manuscript closely resembling the contents of Chapter 6,has also been presented at the 13th Annual ACM/IEEE International Conferenceon Human-Robot Interaction (HRI 2018) held in Chicago, IL, USA from March5-8, 2018 and appears in the conference proceedings. I was the lead investi-gator, responsible for all major areas of concept formation, data collection andanalysis, as well as the majority of manuscript composition. This experimentalstudy was conducted at Disney Research Los Angeles (DRLA) under the supervi-sion of G Niemeyer. Analysis of the data was performed at University of BritishColumbia (UBC) under the supervision of both EA Croft and G Niemeyer (re-motely).A version of Chapter 7 has been included in the proceedings of the IEEE Hap-tics Symposium (HAPTICS) 2018 held in San Francisco, California, USA fromMarch 25-28, 2018. I was the lead investigator, responsible for all major areasof concept formation, data collection and analysis, as well as the majority of themanuscript’s composition. This chapter’s contents were derived from the samevistudy found in Chapter 6; the experimental study was conducted at DRLA underthe supervision of G Niemeyer. Analysis of the data was performed at UBC underthe supervision of both EA Croft and G Niemeyer (remotely).The composition of this thesis conforms to the guidelines for structure andformat of UBC theses and dissertations laid out by the UBC Faculty of Graduateand Postdoctoral Studies which can be found in [117].viiTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiiList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xivList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviGlossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xixAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Exploring Handovers . . . . . . . . . . . . . . . . . . . . . . . . 51.2 Research Objective . . . . . . . . . . . . . . . . . . . . . . . . . 61.3 Thesis Organization . . . . . . . . . . . . . . . . . . . . . . . . . 81.3.1 Robot-to-Human Handovers . . . . . . . . . . . . . . . . 91.3.2 Human-to-Robot Handovers . . . . . . . . . . . . . . . . 101.4 Thesis Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121.5 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14viii2.1 Proxemics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152.2 Kinematics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162.3 Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182.4 Timing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 The Effect of Robot Gaze in Robot-to-Human Handovers . . . . . . 203.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223.2 Background - Gaze in Handovers . . . . . . . . . . . . . . . . . . 233.3 Study I: Observing Gaze Patterns in Human-to-Human Handovers 253.3.1 Experimental Procedure . . . . . . . . . . . . . . . . . . 253.3.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 263.3.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 283.4 Study II: Impact of Human-Inspired Gaze Cues on First-Time Robot-to-Human Handovers . . . . . . . . . . . . . . . . . . . . . . . . 303.4.1 Physical Handover Cues . . . . . . . . . . . . . . . . . . 303.4.2 Experimental Gaze Cues . . . . . . . . . . . . . . . . . . 333.4.3 Experimental Procedure . . . . . . . . . . . . . . . . . . 353.5 Technical Implementation . . . . . . . . . . . . . . . . . . . . . 363.6 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373.6.1 Handover Timing . . . . . . . . . . . . . . . . . . . . . . 373.6.2 Subjective Experience . . . . . . . . . . . . . . . . . . . 403.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423.8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 453.8.1 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . 463.8.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . 464 Characterization of Handover Object Orientations Informing Effi-cient Robot-to-Human Handovers . . . . . . . . . . . . . . . . . . . 484.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 514.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 514.3 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 524.4 Object Handover User Study . . . . . . . . . . . . . . . . . . . . 52ix4.4.1 Experiment Design . . . . . . . . . . . . . . . . . . . . . 524.4.2 Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . 534.4.3 Participants . . . . . . . . . . . . . . . . . . . . . . . . . 554.4.4 Participant Task . . . . . . . . . . . . . . . . . . . . . . . 554.4.5 Motion-Capture System . . . . . . . . . . . . . . . . . . 574.4.6 Hypotheses . . . . . . . . . . . . . . . . . . . . . . . . . 594.5 Data Analyses . . . . . . . . . . . . . . . . . . . . . . . . . . . . 594.5.1 Handover Orientation Extraction . . . . . . . . . . . . . . 594.5.2 Affordance Axes . . . . . . . . . . . . . . . . . . . . . . 624.5.3 Patterns in Handover Orientations . . . . . . . . . . . . . 624.5.4 Comparison of Handover Orientations Across Conditions . 634.6 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 644.6.1 Handover Orientation and Affordance Axis . . . . . . . . 644.6.2 Patterns in Handover Orientations . . . . . . . . . . . . . 674.6.3 Comparison of Handover Orientations Across Conditions . 694.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 714.7.1 Patterns in Handover Orientations . . . . . . . . . . . . . 714.7.2 Comparison of Handover Orientations Across Conditions . 714.7.3 Implications Towards Building Intelligent Robots . . . . . 734.8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 745 Automated Detection of Handovers using Kinematic Features . . . . 765.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 785.2 Prior Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 795.2.1 Non-Verbal Cues, Proxemics and Kinematics in Handovers 795.2.2 Machine Learning for Handovers . . . . . . . . . . . . . 805.3 Objectives and Approach . . . . . . . . . . . . . . . . . . . . . . 825.4 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . 835.5 Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 835.5.1 Data Post-Processing . . . . . . . . . . . . . . . . . . . . 835.5.2 Feature Extraction . . . . . . . . . . . . . . . . . . . . . 845.5.3 Labelling . . . . . . . . . . . . . . . . . . . . . . . . . . 865.5.4 Test Set Generation . . . . . . . . . . . . . . . . . . . . . 87x5.5.5 Predictor Feature Selection . . . . . . . . . . . . . . . . . 875.5.6 Hyperparameter Optimization . . . . . . . . . . . . . . . 905.6 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 925.6.1 Training and Holdout Data . . . . . . . . . . . . . . . . . 925.6.2 Predictor Feature Selection . . . . . . . . . . . . . . . . . 925.6.3 Hyperparameter Optimization . . . . . . . . . . . . . . . 985.6.4 Confusion Matrix . . . . . . . . . . . . . . . . . . . . . . 1005.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1035.7.1 Feature Selection . . . . . . . . . . . . . . . . . . . . . . 1035.7.2 Model Verification . . . . . . . . . . . . . . . . . . . . . 1055.7.3 Extendibility of Method to Other Trackers . . . . . . . . . 1065.8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1135.8.1 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . 1145.8.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . 1146 Evaluating Social Perception of Human-to-Robot Handovers . . . . 1176.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1186.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1196.2.1 Handovers . . . . . . . . . . . . . . . . . . . . . . . . . 1196.2.2 Robotic Social Attributes Scale (ROSAS) . . . . . . . . . 1206.3 Experimental Design . . . . . . . . . . . . . . . . . . . . . . . . 1216.3.1 Initial Position of the Arm Before Handover (Down and Up) 1246.3.2 Grasp Type During Handover (Quick and Mating) . . . . . 1246.3.3 Retraction Speed Following Handover (Slow and Fast) . . 1276.3.4 Conditions . . . . . . . . . . . . . . . . . . . . . . . . . 1276.4 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . 1286.4.1 System . . . . . . . . . . . . . . . . . . . . . . . . . . . 1286.4.2 Participants . . . . . . . . . . . . . . . . . . . . . . . . . 1296.4.3 Participant Task . . . . . . . . . . . . . . . . . . . . . . . 1296.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1306.5.1 ROSAS Internal Consistency and Dimensionality . . . . . 1306.5.2 Effect of Conditions . . . . . . . . . . . . . . . . . . . . 1326.5.3 Effect of Repeated Interaction over Time . . . . . . . . . 132xi6.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1346.6.1 ROSAS Internal Consistency and Dimensionality . . . . . 1346.6.2 Effect of Conditions . . . . . . . . . . . . . . . . . . . . 1356.6.3 Effect of Repeated Interaction Over Time . . . . . . . . . 1376.6.4 Implications for HRI . . . . . . . . . . . . . . . . . . . . 1376.7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1386.7.1 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . 1396.7.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . 1397 Exploration of Geometry and Forces Occurring Within Human-to-Robot Handovers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1417.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1427.2 Experimental Design . . . . . . . . . . . . . . . . . . . . . . . . 1437.3 Geometry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1437.3.1 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 1447.3.2 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 1457.4 Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1477.4.1 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 1487.4.2 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 1497.5 Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1517.5.1 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 1537.5.2 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 1537.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1547.6.1 Future Work . . . . . . . . . . . . . . . . . . . . . . . . 1558 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1568.1 Conveying Non-Verbal Cues During Handover . . . . . . . . . . 1578.1.1 Summary of Findings . . . . . . . . . . . . . . . . . . . . 1578.1.2 Implications . . . . . . . . . . . . . . . . . . . . . . . . 1618.1.3 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . 1618.1.4 Future Work . . . . . . . . . . . . . . . . . . . . . . . . 1628.2 Recognizing Non-Verbal Cues from Humans . . . . . . . . . . . 1638.2.1 Summary of Findings . . . . . . . . . . . . . . . . . . . . 163xii8.2.2 Implications . . . . . . . . . . . . . . . . . . . . . . . . 1658.2.3 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . 1658.2.4 Future work . . . . . . . . . . . . . . . . . . . . . . . . . 166Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167A Supporting Materials for Investigating Robot Gaze in Robot-to-HumanHandovers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184A.1 Study Advertisements . . . . . . . . . . . . . . . . . . . . . . . . 184A.1.1 Poster . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185A.1.2 E-Mail . . . . . . . . . . . . . . . . . . . . . . . . . . . 186A.2 Consent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187A.2.1 Written Consent . . . . . . . . . . . . . . . . . . . . . . 187A.2.2 Verbal Consent . . . . . . . . . . . . . . . . . . . . . . . 189A.3 Surveys . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190A.3.1 Single Condition Survey . . . . . . . . . . . . . . . . . . 190A.3.2 Condition Comparison Survey . . . . . . . . . . . . . . . 191B Supporting Materials for Investigation of Handover Object Orien-tations and Automated Detection of Handovers . . . . . . . . . . . . 192B.1 Study Advertisements . . . . . . . . . . . . . . . . . . . . . . . . 192B.1.1 Poster . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193B.1.2 E-Mail . . . . . . . . . . . . . . . . . . . . . . . . . . . 194B.2 Consent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195C Supporting Materials for Investigating Human-to-Robot Handovers 197C.1 Consent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197C.2 Survey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203xiiiList of TablesTable 1.1 Non-exhaustive table of generalized comparisons of strengthsand weaknesses between robots and humans. . . . . . . . . . . 4Table 3.1 Table of questionnaire results in the robot-to-human handovergaze study. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41Table 4.1 Results of comparison of handover orientations between condi-tions in the handover object orientation study. . . . . . . . . . 70Table 5.1 Features extracted from kinematic data obtained from the mo-tion capture system during the study investigating the automateddetection of handovers. . . . . . . . . . . . . . . . . . . . . . 86Table 5.2 Training and testing data specifications for the automated de-tection of handovers. . . . . . . . . . . . . . . . . . . . . . . . 92Table 5.3 The 22 features selected for use as the Support Vector Machine(SVM) input space to automatically detect handovers and theirpredictive ability rankings. . . . . . . . . . . . . . . . . . . . 94Table 5.4 Optimized hyperparameter values and corresponding cross-validationerror for the SVMs using various kernels. . . . . . . . . . . . . 100Table 5.5 Confusion matrix for handover detection using a SVM with var-ious kernels on motion capture data. . . . . . . . . . . . . . . 102Table 5.6 Pilot study confusion matrix for handover detection using a SVMwith Radial Basis Function (RBF) kernel applied to featuresgenerated via a Kinect version 2 sensor. . . . . . . . . . . . . . 110xivTable 6.1 Table of ROSAS items categorized by the factors competence,warmth and discomfort. . . . . . . . . . . . . . . . . . . . . . 121Table 6.2 Table of experimental conditions for the human-to-robot han-dover study. . . . . . . . . . . . . . . . . . . . . . . . . . . . 128Table 7.1 Comparison of Cartesian positions of handover objects betweenhuman-robot and human-human handovers. . . . . . . . . . . 146Table 8.1 Summary table of robotic nonverbal cues studied in this thesisand their observed impacts. . . . . . . . . . . . . . . . . . . . 158xvList of FiguresFigure 3.1 Timeline of giver’s gaze patterns observed from human-humanhandovers. . . . . . . . . . . . . . . . . . . . . . . . . . . . 27Figure 3.2 Demonstration of two frequently observed gaze behaviours fromthe human-human handover study. . . . . . . . . . . . . . . . 29Figure 3.3 Illustration demonstrating experimental set-up and gaze for eachcondition in the robot-to-human handover gaze study. . . . . . 32Figure 3.4 Diagram depicting the gaze cues for the No Gaze, Shared At-tention, and Turn-Takinggazes. . . . . . . . . . . . . . . . . . 34Figure 3.5 System flow diagram describing algorithm used for PR2 robotgaze during handovers. . . . . . . . . . . . . . . . . . . . . . 38Figure 3.6 Chart of handover timing results for various gaze conditions inthe robot-to-human handover study. . . . . . . . . . . . . . . 39Figure 4.1 Illustration depicting the importance of considering object ori-entation during robot-to-human handovers. . . . . . . . . . . 49Figure 4.2 Common everyday objects affixed with retro-reflective mark-ers that were used in the handover object orientation user study. 54Figure 4.3 Experimental setup for examining object orientations duringhandovers. . . . . . . . . . . . . . . . . . . . . . . . . . . . 56Figure 4.4 Diagram of retroreflective marker placement and labels on par-ticipants during the study investigating the automated detec-tion of handovers. . . . . . . . . . . . . . . . . . . . . . . . . 58Figure 4.5 Charts representing typical hand trajectories of giver and receiver. 60xviFigure 4.6 Base coordinate frame used during examination of object ori-entations during handovers as defined by giver and receiverlocations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61Figure 4.7 Monte Carlo simulation results showing histograms of the an-gle between the mean affordance axis for each object and theaffordance axis of the object rotated by the orientation of thatobject used by each participant. . . . . . . . . . . . . . . . . 63Figure 4.8 Figure showing orientation of a teapot for one and multipletrials as it is being handed over from giver to receiver. . . . . . 65Figure 4.9 Handover orientations employed during human-human han-dover for all objects in all conditions. . . . . . . . . . . . . . 66Figure 4.10 Histograms of the angles between R¯φˆAff and RiφˆAff (i.e., θi) foreach object in condition C during the study examining han-dover orientations. . . . . . . . . . . . . . . . . . . . . . . . 68Figure 5.1 Diagram showing a giver passing a book to the receiver, andthe corresponding motion-captured scene. . . . . . . . . . . . 85Figure 5.2 Charts depicting resubstitution loss, generalization error andout-of-bag error for Gini and entropy/information gain splitcriterion as a function of bootstrap sampling fraction. . . . . . 89Figure 5.3 Charts depicting resubstitution loss, generalization error andout-of-bag error for Gini and entropy/information gain splitcriterion as a function of number of learners. . . . . . . . . . 89Figure 5.4 Plot of the SVM feature space arranged by importance scoresgenerated from the bagged random forest. . . . . . . . . . . . 96Figure 5.5 Parallel coordinates plot of the input space of the handoverclassifier determined by using a bagged random forest. . . . . 97Figure 5.6 Kernel search spaces for optimization of the regularization pa-rameter (C) for the handover classifying SVM . . . . . . . . . 99Figure 5.7 Diagram of joints and labels as automatically generated by theMicrosoft Kinect version 2 sensor. . . . . . . . . . . . . . . . 108Figure 6.1 Experimental setup for human-to-robot handover experiment. 123xviiFigure 6.2 Flowchart of quick (top row) and mating grasp types used dur-ing the human-to-robot handover experiment. . . . . . . . . . 126Figure 6.3 Scree plots for ROSAS factors to examine dimensionality ofitems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131Figure 6.4 Participants ratings of the robot’s competence, warmth and dis-comfort over conditions as reported during the human-to-robothandover study. . . . . . . . . . . . . . . . . . . . . . . . . . 133Figure 6.5 Interaction plot showing participants ratings of the robot’s dis-comfort based on grasp type at levels of retraction speed. . . . 134Figure 6.6 Participants ratings of the robot’s competence, warmth and dis-comfort over time as recorded during the human-to-robot han-dover study. . . . . . . . . . . . . . . . . . . . . . . . . . . . 135Figure 7.1 Scatterplots of initial object positions (from the overhead per-spective) for both the down and up initial arm pose conditionspresented during the human-to-robot handover study. . . . . . 144Figure 7.2 Scatterplots of initial object azimuth and elevation angles forboth the down and up initial arm pose conditions presentedduring the human-to-robot handover study . . . . . . . . . . . 145Figure 7.3 Sample end effector axial force vs. time plot for a participantdepicting features used in the dynamics analysis of the human-to-robot handover study. Negative forces indicate pushing (com-pressive) forces exerted against the end effector whereas posi-tive forces indicate pulling (tension) forces. . . . . . . . . . . 148Figure 7.4 Maximum absolute and relative retraction forces as experi-enced by the giver in the human-to-robot handover study. . . . 149Figure 7.5 Maximal forces during handover as experienced or applied bygivers during the human-to-robot handover study over trialsordered chronologically. . . . . . . . . . . . . . . . . . . . . 152xviiiGlossaryANOVA Analysis of Variance, a set of statistical techniques to identify sources ofvariability between groupsCI Confidence IntervalDMP Dynamic Movement PrimativeGSS Golden Section SearchHCI Human-Computer InteractionHHI Human-Human InteractionHRI Human-Robot InteractionIIWA Intelligent Industrial Work Assistantused to describe a series of robotarms developed by KUKA Robotics designed for human-robotcollaborative tasksLBR Leichtbauroboter, German for lightweight robot - used to describe aseries of robot arms developed by KUKA Robotics designed forhuman-robot collaborative tasksMANOVA Multivariate Analysis of Variance, an ANOVA with several dependentvariablesPR2 Personal Robot 2, a robot platform developed by Willow GarageROS Robot Operating SystemxixRBF Radial Basis FunctionROSAS Robotic Social Attributes ScaleSPI Successive Parabolic InterpolationSVM Support Vector MachinexxAcknowledgementsI offer my enduring gratitude to several people and organizations who havemade significant contributions not only to this thesis, but to my knowledge of thescientific world:• My supervisor Prof. Elizabeth A. Croft, who has graciously accepted me asgraduate student and have provided me with support and guidance through-out my time as a masters and doctoral student,• Dr. Gu¨nter D. Niemeyer and Lanny Smoot who have encouraged and sup-ported me in my forays into mixing virtual reality and human-robot interac-tion during my internship at DRLA,• My colleagues and friends at the Collaborative Advanced Robotics and Intel-ligent Systems (CARIS) Laboratory for always being there to exchange ideasand over lunches, dinners and spontaneous meetings,• The National Sciences and Engineering Research Council of Canada (NSERC)for providing me with funding that was critical in carrying out this research,• My parents Danny and Tina Pan, the people who never stopped believing inmy abilities as a scientist; and lastly,• My wife and best friend, Bernice Li, for encouraging me to reach farther anddream bigger throughout my graduate career.I would like to express additional thanks to Karon MacLean, Wesley Chan,Vinay Chawda, Winston Yao, Vidar Skervøy, AJung Moon, Brian Gleeson, MinhuaxxiZheng, Benjamin Blumer, Jonathan Benitez and Alisa Wyman for their support onvarious parts of the work presented in this thesis.xxiiDedicated to my loving wife Bernice and daughter Maven.xxiiiChapter 1IntroductionThe worldwide demand for robotics technologies has been steadily growingover the last two decades to meet the needs of industrial and service industries[1, 57]. There is little doubt that this trend will persist, at the very least, as roboticstechnologies become sufficiently advanced to enter into new areas of service. TheInternational Federation of Robotics (IFR) predicts that the demand for domesticand service robots, for example, is expected to increase six-fold within the next twoyears alone [57] as the abilities of robots mature to address growing opportunitiesand challenges in various markets and sectors. They predict that future technologydevelopment will allow robots to:• deploy in growing consumer markets and decreased product life cycles whichrequire expansion of output capabilities for competitiveness• provide a solution for a significant workforce gap created by an aging andretiring population• address the health care needs of this aging population• perform work considered tedious, hazardous and/or unsanitary to humanworkersWith advancing technological capabilities and increasing evidence that roboticswill be able to serve in such capacities, the use of robots in the field has expanded1greatly - from initially being used in heavy industries (such as automotive man-ufacturing) in the 1960s, they are now seen performing tasks such as warehouselogistics [e.g., 7], home vacuum cleaning [e.g., 59], customer service [e.g., 108]and laparoscopic surgery [e.g., 58]. The increasing pervasiveness of robotics inthese environments capitalizes on rapidly developing technologies to deliver con-venience, cost-savings, and/or higher efficiencies. Many organizations adoptingrobotic technologies see this as a way to lay the foundations for innovative and sus-tainable development, and to improve consistency and efficiency of how productsare produced and services rendered. In many of these applications, the predomi-nant model of how robots are deployed is to have them completely replace humanworkers. For many dangerous, repetitive, precise, and simple tasks which may ad-versely affect human workers (either mentally or physically), the justification forusing this model is sound: robots are undeniably useful for rote operations whichmay involve handling heavy/dangerous materials or require precise positioning.Such tasks involve operations that humans are neither good at or designed for.However, there are abilities that robots continue to lack or under-perform inwhen compared to humans. A recent article published by the Institute of Electricaland Electronics Engineers (IEEE) Spectrum Magazine entitled “Shockingly, RobotsAre Really Bad at Waiting Tables” highlights some of the problems when currentrobot technologies attempt to replace humans though do not have the necessaryskill-sets [2]. The article describes two restaurants in China which gained notorietyfor using robotic waiters to deliver food to people, a scheme which was scrappedshortly after the robots’ introduction. One restaurant employee was quoted as say-ing that “their skills are somewhat limited...they can’t take orders or pour hot wa-ter for customers.” Indeed, the robots used in the restaurants were only able totravel on pre-planned paths and carry trays of food to the vicinity of tables. Acker-man writes: “those are just two of the many, many more skills that human servershave, because it’s necessary to have many, many more skills than this to be a goodserver”. Compared to humans, robots currently under-perform in other areas in-cluding object recognition, flexibility in decision-making, understanding natural-language queries, and dexterity. If a model of robots being used to replace humansis blindly adopted without considering the strengths and weaknesses of the robot,situations such as those experienced with the robot waiters will occur.2Given that robotics is still a developing technology, many sectors are investingin another model of robotics which mitigate the aforementioned problems: insteadof having robots replace humans, have robots work alongside humans to have acompliment of capabilities which negate the weaknesses of both. Table 1.1 showssome comparisons of select strengths and weaknesses of both, and aims to high-light how collaboration within human-robot teams may significantly increase taskproficiency over the robot or human individually. In this collaboration model, hu-mans and robots actively collaborate in close proximity to each other to completetasks. For example, in a manufacturing scenario, a robot could assist a humanworker in holding up a heavy part while the worker secures the part to an assemblyusing fasteners. Also, a robot could handover tools to a worker at the right timeto enable the worker to perform certain tasks, much like how a nurse hands oversurgical instruments to a surgeon during an operation. In these situations, the pur-pose of the robot is to enable, support, and enhance the capabilities of the humanworker through Human-Robot Interaction (HRI). Alternatively, the reverse will betrue equally often - for example, the human may be instructed by the robot to pro-vide dexterity in a job that the robot can otherwise handle. In this case where therobot is managing the overall task, it can be argued that the human is supportingand augmenting the capabilities of the robot. In both these scenarios, however, it isthe co-operation and collaboration of both human and robotic agents which leadsto a more versatile work unit.To aid in deploying human-robot collaborative systems, robotic manufactur-ers are beginning to provide platforms designed to be used for HRI; for exam-ple, KUKA Robotics Corporation (Augsburg, Germany) have recently releasedtheir LBR IIWA lightweight robot which they describe as the “world’s first series-produced sensitive, and therefore human-robot collaborative compatible, robot”[70]. Other robotics-centered organizations such as Rethink (Boston, MA), ABB(Zu¨rich, Switzerland), Franka Emika (Munich, Germany) are also offering robotsthat specialize in HRI. Users of robotics technologies are also investing in thehuman-robot collaborative model: General Motors has partnered with several aca-demic institutions in a research project called Collaborative, Human-focused, As-sistive Robotics for Manufacturing (CHARM) which explored ways of deploying3Table 1.1: Non-exhaustive table of generalized comparisons of strengths andweaknesses between robots and humans.Robots HumansNarrow impedance range Wide impedance rangeLarge payload capacity Small payload capacityPrecise, repeatable actuation Non-precise, semi-repeatable actua-tion (can be affected by factors such asdisease, muscle weakness, etc.)Limited flexibility (added flexibility iscostly)Very flexibleLimited dexterity (added dexterity iscostly)Very dexterousAble to accurately plan and keep trackof place in multi-step proceduresProne to mistakes in executing orderedprocedures and/or inefficient planningof tasksSensor specialization and access todatabase information allows for accu-rate and timely quality assessmentQuality assessment methods generallylimited by trade-offs (e.g., crude andfast or comprehensive and slow)Limited ability to self-diagnose issuesaffecting task performanceAble to investigate and establishcausality to rectify issues affectingtask performancerobots in manufacturing environments to collaborate with and enhance workers’capabilities while making production lines more flexible [40].As service robots are deployed in environments that permit close proximity tohuman users for human-robot collaboration, it is essential that critical capabilitiesin HRI are developed. In particular, in order for robots to interact effectively andnaturally with human counterparts, their behaviours must be carefully designedand informed by the recognition of cues provided by the human and environmentalelements. Likewise, dexterous manipulations performed by the robot should beperceived as safe, reliable and predictable in context of the situation and/or envi-ronment. In other words, cooperative interaction between robots and people must4consider how the robots perceive the people and environments they are interactingwith, and how the robots themselves are perceived.This thesis explores these concepts by examining one type of fundamentallynecessary interaction between robots and humans during physical cooperation: ob-ject handovers. The ability of robots to safely and effectively handover objectsis a crucial capability for collaborative human-robot interaction. It is a necessaryskill for robots if they are to be able to fully assist and cooperate with human userswithin shared task environments. Revisiting the example of waiter robots deployedin restaurants, if the robot were able to appropriately handover dishes and glasses,the robot would arguably be a much more effective waiter (granted, the robot wouldalso need additional skills to be able to completely replace human waiters). Giv-ing or receiving an object is a skill that will allow robots to be increasingly usefulin contexts such as manufacturing and assistive care. For example, a robot canhandover or retrieve a tool to and from a line worker in a manufacturing plantdepending on which tool the worker needs at the time, increasing worker produc-tivity. In an assisted living situation, a robotic assistant can help fetch a TV remoteor drink for a mobility-limited user. In these and other use cases, the ability for therobot to effectively participate in a handover adds immense value for interactiverobots that will increase their ubiquity within the home and workplace.1.1 Exploring HandoversThe act of handing over objects is perhaps the quintessential representationof cooperation between people. It is a necessary ability that is conducted habit-ually and naturally everyday; it is overlooked by most as a rudimentary skill thatoccurs almost reflexively. However, when carefully examining the coordinationinvolved in handovers, it is apparent that they are intricate interactions involvingmultiple subtle, nonverbal cues generated and interpreted by both agents. Objecthandovers require kinodynamic (kinematic + dynamic) negotiation, gesture andmotion prediction, coordinated hand-eye motions, and tactile sensing on the partof both participants - all within a matter of seconds.Examining and considering these nonverbal cues is especially important fora developing a robot assistant that can give and receive objects to a person: to5have a robot smoothly and seamlessly give or receive an object is a technical and,arguably, an artistic) challenge of being able to generate, interpret and react tononverbal cues. This presents an open problem of designing, implementing and in-tegrating several sub-systems which allow a robot to sense and appropriately reactbefore, during and after object handovers. Specifically, these sub-systems shouldprovide the robot with the ability to recognize and understand the underlying kine-matic and dynamic interactions between the giver and receiver, with the intentionof and participating in this negotiation in a fluent and efficient manner. In the ex-ample of a robot giver passing an object to a human, this includes having the robotprovide clear and understandable cues to indicate that it wants to hand over an ob-ject, fluently negotiate where and when the handover is to occur, recognize whenthe human user has a stable grasp of the object, and then appropriately release its’own grasp on the object before safely retracting its manipulator. For the inversesituation where a robot receives an object from a human, the robot must interpretthe human’s nonverbal cues that he/she wants to hand over an object, fluently nego-tiate where and when the handover is to occur, obtain and maintain a stable graspthe object, and provide cues to effectively indicate to the human that it achieveda stable grasp. Both directions of handover (human-to-robot and robot-to-human)are examined in this thesis.Considering these challenges, it is apparent that handovers present an exemplarand useful context in which to investigate robot behaviours that support HRI. Theyare an interaction which can be used as a model to study other HRIS. From thisperspective, the broad aim of this research is to contribute to the understanding ofHRI design with respect to implementation of simple, everyday interaction tasks.1.2 Research ObjectiveThis thesis explores the roles that a handful of nonverbal cues play in human-robot handovers and the effects of manipulating these cues on a robot that acts asboth giver and receiver of objects. This work does not attempt to map the bound-aries of the design space that is available to robot designers, integrators and usersfor developing handover gestures, nor does it limit exploration to human-like meth-ods of conducting handovers. Rather, it explores how some nonverbal cues exhib-6ited by a human can be recognized and leveraged during the handover negotiationand how robots can use nonverbal cues to affect the interaction and how peoples’perceptions of them. The goal of this body of work is to provide some indicationon how fluent and efficient human-robot handovers can be achieved through the useof nonverbal cuing and interaction. In particular, the nonverbal cues that are exam-ined include: robot gaze, object alignment/orientation, proxemics, robot grasping,kinematics, and dynamics. The primary aim of this thesis is to convey that justlike human-human handovers, the negotiation that occurs in achieving fluent andefficient human-robot handovers can be deceptively complex and relies on manynonverbal cues.The corpus of work contained in this thesis primarily centers around two re-search questions:Q1 How do nonverbal cues exhibited by the robot during robot giving and re-ceiving behaviours change how users perceive the robot, and affect the han-dover negotiation?Q2 How can a robot adequately recognize and interpret nonverbal cues conveyedby a human to infer object attributes as well as handover intent?To address the first question, the effects that gaze cues generated by a robotgiver (with an articulated head) have on handover interactions are considered (Chap-ter 3). In the case where a robot arm acts as a receiver in the handover negotiation,this thesis examines the outcomes of manipulating seemingly innocuous factorssuch as initial arm pose prior to, grasp method during, and object retraction follow-ing object handover changes people social perceptions of the robot (Chapter 6), aswell as the kinodynamics of the interaction (Chapter 7).To examine the premise of the second research question, a portion of this workhas been dedicated to explore how robots can be trained to pass objects to humanreceivers while considering the object’s affordances, such that an object can bepassed to a human receiver in a manner which is most comfortable to the receiver(Chapter 4). In this study, the viability of using human-human handover data setsfor this task is examined. In a separate study, how user poses and kinematics canbe leveraged through motion tracking systems to detect a human giver’s intent tohandover through machine learning are considered (Chapter 5).7A more detailed summary of each of these works appears in the followingsection.1.3 Thesis OrganizationThe investigations on human-robot handovers contained within this thesis canbe considered to span two different sub-domains or tasks: robot-to-human han-dovers and human-to-robot handovers. At first glance, it may appear that thereis very little distinction between these two categories since both have robot andhuman agents participating in a single type of activity. However, from the perspec-tive of the robot, these tasks are very different: robot-to-human handovers requirethe robot to act as an object giver, whereas human-to-robot handovers employ therobot as an object receiver - these activities encompass very different technicalchallenges.For robot-to-human handovers, where the robot acts as an object giver (assum-ing that the giver initiates and receiver is ready to participate in the handover), therobot is required to:1. grasp the handover object safely and securely prior to handover;2. effectively communicate to a human counterpart that it wants to handover anobject;3. consider how convenient it is for the human receiver to physically grasp theobject, and orient objects such that their affordances are directed towards thereceiver;4. dynamically negotiate with the human receiver to perform the handover inan expeditious and safe (in terms of the object) manner; and5. recognize when the receiver has a stable grasp of the object before tactfullyreleasing its own grasp to complete the exchange.On the other hand, in human-to-robot handovers, the robot receiver must:1. recognize that a human is initiating a handover;82. detect when and where the handover will take place;3. consider how to grasp the object; and4. communicate to the human giver that it ’has’ the object.Due to these very distinct sets of abilities required by the robot givers and re-ceivers, this thesis has been organized into two parts: the first part (Chapters 3 and4) contains work on robot-to-human handover, whereas the second part (Chapters5 to 7) examines human-to-robot handovers.1.3.1 Robot-to-Human HandoversChapter 3 provides empirical evidence that using human-like gaze cues duringhuman-robot handovers can improve the timing and perceived quality of the han-dover event. Fluent, legible handover interactions require appropriate nonverbalcues to signal handover intent, location and timing. Based upon observations ofhuman-human handovers, gaze behaviours were implemented on a PR2 humanoidrobot. The robot handed over water bottles to a total of 102 inexperienced subjectswhile varying its gaze behaviour: no gaze, gaze designed to elicit shared atten-tion at the handover location, and the shared attention gaze complemented witha turn-taking cue. Subject perception of and reaction time to the robot-initiatedhandovers were compared across the three gaze conditions. Results indicate thatsubjects reach for the offered object significantly earlier when a robot provides ashared attention gaze cue during a handover. A statistical trend of subjects pre-ferring handovers with turn-taking gaze cues over the other conditions was alsoobserved. The work presented in Chapter 3 demonstrates that gaze can play akey role in improving user experience of human-robot handovers, and help makehandovers fast and fluent. Within the context of this thesis, the inclusion of thismanuscript assists in highlighting that subtle nonverbal cues profoundly impact the9way that handover tasks are carried out and can affect user experience of the task.1In Chapter 4, steps are taken towards enabling robot givers to naturally passobjects to human receivers comfortably by considering the object’s affordances.In line with the first research question investigated in this thesis and towards en-abling robots to learn proper handover object orientations during handover, thiswork seeks to determine if the nonverbal cue of natural object orientation observedduring human-to-human handovers can be used to train robots to consider handoverobject affordances. By observing human-human handovers, natural handover ob-ject orientations are compared with giver-centered and receiver-centered handoveralignments for twenty common objects. A distance minimization approach wasused to compute mean handover orientations. It is posited that computed meansof receiver-centered orientations could be used by robot givers to achieve more ef-ficient and socially acceptable handovers. Furthermore, the notion of affordanceaxes for comparing handover alignments is introduced in this work, and offers adefinition for computing them. As a result of this study, observable patterns werefound in receiver-centered handover orientations. Comparisons show that depend-ing on the object, natural handover orientations may not be receiver-centered; thus,robots may need to distinguish between good and bad handover orientations whenlearning from natural handovers. 21.3.2 Human-to-Robot HandoversAs opposed to the wide body of work conducted in robot-to-human handovers,human-to-robot handovers is a domain that, at the time of writing, remains largely1The work presented in Chapter 3 was the result of a collaboration between several research or-ganizations and individuals, including myself. As mentioned in the preface to this thesis, B Gleeson,DM Troniak, A Moon and myself were the lead investigators responsible for the concept formula-tion, planning and directing of the project. I was responsible for the software development used in theexperimental setup, as well as conducting the experiment and performing the results analysis. Themanuscript was collectively written and edited by all authors. In addition to this work’s inclusion inthis thesis on nonverbal cues during handovers, it also appears in A Moon’s thesis where she focuseson the interweaving of cues and plans between humans and robots [87].2The work presented in Chapter 4 was the result of a collaboration between WP Chan and myself.WP Chan and I were jointly involved in the concept development, design, execution, and analysis ofthis experiment.10explored. Thus, the work presented in this thesis within this domain focuses ondeveloping building blocks for robot receiving.As one of these building blocks, Chapter 5 details work conducted to have arobot recognize that a human giver is initiating an object handover. In this project,the use of kinematic motions recognized by Support Vector Machines (SVMs) forthe automatic detection of object handovers from the perspective of an object re-ceiver. The classifier uses the giver’s kinematic behaviours (e.g., joint angles, dis-tances of joints from each other and with respect to the receiver) to determine agiver’s intent to handover an object. A bagged random forest was used to deter-mine how informative features were in predicting the occurrence of handovers, andto assist in selecting a core set of features to be used by the classifier. Altogether,22 kinematic features were chosen for developing handover detection classifica-tion models. Test results indicated an overall maximum accuracy of 97.5% by theSVM in its capacity to distinguish between handover and non-handover motions.The classification ability of the SVM was found to be unaffected across four kernelfunctions (linear, quadratic, cubic and radial basis). These results directly addressthe 2nd research posed by this thesis and demonstrate considerable potential fordetection of handovers and other gestures for human-robot interaction using kine-matic features. 3Following this work, a robot receiving system was developed using a KUKALBR IIWA 7 R800 robot arm (KUKA, Augsburg, Germany). Chapters 6 and 7,leverage this system to examine how users behave and perceive robots in human-to-robot handovers. In Chapter 6, users’ social perceptions of robots within thedomain of human-to-robot handovers are explored. Using the Robotic Social At-tributes Scale (ROSAS), developed by Carpinella et al. in [25], the work exploreshow users socially judge robot receivers as three kinodynamic factors are varied:initial position of the robot arm prior to handover (up vs. down), grasp method em-ployed by the robot when receiving a handover object trading off perceived objectsafety for time efficiency or vice versa, and retraction speed of the arm following3I was the lead investigator in the research presented in Chapter 5, responsible for all major areasof concept formation, data collection and analysis, as well as manuscript composition. WP Chanand I were jointly involved in the design and execution of this experiment. Data processing wasperformed with the assistance of V Skjervøy.11handover (slow vs. fast). The results show that over multiple handover interactionswith the robot, users gradually perceive the robot receiver as being less discom-forting and having more emotional warmth. Additionally, we have found that byvarying grasp method and retraction speed, users may hold significantly differ-ent judgments of robot competence and discomfort. These results provide empiri-cal evidence that users are able to develop social perceptions of robots which canchange through modification of robot nonverbal receiving behaviours and throughrepeated interaction with the robot, addressing the first question posed in this the-sis.In Chapter 7, kinematic and dynamic data recorded from human-to-robot han-dover trials are examined. Analysis of the kinodonamic data shows that the robot’sinitial pose can inform the giver about the upcoming handover geometry and im-pact fluency and efficiency. Also, variations in grasp method and retraction speedappear induce significantly different interaction forces. This effect may occur bychanging the giver’s perception of object safety and hence their release timing.Alternatively, it may stem from unnatural or mismatched robot movements. Ad-ditionally, the findings indicate that making the robot predictable is important: alearning effect within the forces applied to the handover object linearly declineover repeated trials. Simultaneously, as shown in Chapter 6, the participants’ self-reported discomfort with the robot decreases and perception of emotional warmthincreases. Thus, it is posited that users are learning to predict the robot, becom-ing more familiar with its behaviours, and perhaps becoming more trusting of therobot’s ability to safely receive the object. As a result, these findings suggest that arobot can become a trusted partner in collaborative tasks.1.4 Thesis NotesIn this thesis, consistent terminology will be used when referring to participantsinvolved in an object handover, regardless if whether the participant is a human orrobot. The participant that initially holds the object and transfers the object to theother participant through the handover interaction will be referred to as the giver.The participant that the giver transfers the object to during the handover interactionwill be referred to as the receiver.12Additionally, to narrow the scope of this thesis, the work contained within thisthesis adopts a model of object handover that is most commonly found and usedin literature (presented in Chapter 2). In this model, the following is assumed orenforced:• Handovers are conducted nonverbally between participants.• The giver does not consider readiness of the receiver to participate in thehandover.• The giver always initiates the handover interaction and waits for the receiverto take the object.1.5 ContributionsTo achieve the objectives of the research described in this thesis, the followingcontributions were made:• Experimental results of how nonverbal gaze cues can affect fluency and effi-ciency of handover interactions between robots and humans were obtained.• An exploration of how objects can be characterized by their affordances thatmakes it more comfortable for a human to receive objects being handed overhas been conducted.• A robot receiving system for automatically receiving an object from a humangiver was developed.• Experimental qualitative and quantitative results relating to changing param-eters of robot receiving behaviours were obtained.13Chapter 2BackgroundThis chapter provide a brief overview on the current state-of-the-art work inresearch areas related to this thesis. In particular, it examine prior work directedtowards enabling robots to participate in handovers through the use and recognitionof nonverbal cues. Chapters 3 to 7, containing the main body of work within thisthesis, will make reference to literature presented here. However, each of thesechapters will contain additional background information that is more-specific tothe topics covered by the chapter (e.g., machine learning, gaze cues).Prior work has demonstrated that nonverbal cues play a significant role in co-ordinating handovers between human participants [43]. In particular, nonverbalcues provide an efficient and often unmindful modality of communication duringhuman-human handovers that can help both giver and receiver fluently negotiatethe transfer of an object in a matter of seconds. To obtain the same efficiencyin human-robot handovers, researchers have examined human behaviours and cuegeneration that can lead to better robot interaction design. Studying nonverbal cuesbetween humans have allowed some robot to mimic what people do when givingor receiving objects, leading to more time-efficient and desirable handovers [e.g.,22]. However, these observations are also particularly important for the develop-ment of a handover controller as robots may be able to anticipate rather than reactto handovers, leading to more fluent interactions [50]. That is, if nonverbal cuesindicating an object is being passed can be observed early enough, a robot may be14able to act quickly enough to allow the handover to proceed seamlessly and fluentlyfrom the perspective of a human giver [e.g., 106].Previously, researchers have studied how some nonverbal cues in the spaces ofproxemics, kinematics, dynamics and timing have affected the handover interactionin contexts of both human-human and human-robot dyads. This prior work isorganized and presented with regards to these spaces and domains of nonverbalcues.2.1 ProxemicsMuch of the early work in robot-to-human handovers relied on proxemics stud-ies by Hall, whose work suggested that distances between persons interacting so-cially are influenced by culture, attitudes, social standing, and relationships to oneanother [45]. In particular to human-human handovers, Basili et al. found that han-dovers occur approximately at half the distance between the giver and receiver andslightly to the right with respect to the receiver[13]. The interpersonal distancesbetween givers and receivers varied considerably with the average distance being1.16 m away roughly the distance where both giver and receiver must have armsoutstretched during handover [13]. This spatial positioning is also corroborated byHuber et al. [53]. These studies did not examine factors leading to variances ofinterpersonal distances during handovers.To compare this with proxemics between robots and humans, Walters et al.found that most of their participants allowed a robot to approach to within a per-sonal distance (0.6 to 1.25 meters), suggesting that humans may treat robots similarto humans socially [121]. However, 40% of their participants allowed a robot to ap-proach to within an intimate distance of 0.5 meters, implying that humans are moretolerant of robotic close encounters, which would be perceived as over-familiar orthreatening in a similar human-human context. Similarly Koay et al. also foundthat participants would prefer a robot to approach to within approximately 0.6 -0.75 meters in their study [67].152.2 KinematicsThere have been numerous studies trying to better understand the kinematicsinvolved in human-to-human handovers [e.g., 13, 50, 53, 60, 81, 104]. Several stud-ies have looked at where handovers occur in the spatial domain [13, 54], and thejoint/limb kinematics of both the giver and receiver during handover [41, 60, 106].Kajikawa et al. and Koay et al. have both investigated motion planning of robotsconducting handovers in close proximity to human counterparts, identifying typicalkinematic characteristics in human-human handovers and developing appropriatehandover trajectories based on such findings [60, 67]. Kajikawa et al. have deter-mined that handovers between humans share several common kinematic character-istics, e.g., rapid increase in the giver’s arm velocity at the start of the handover,[60].Other researchers have investigated robot handover trajectory and pose, report-ing guidelines for how a robot arm should be positioned for handover and howthat position should be achieved. For example, Agah and Tanie presented a han-dover motion controller that was able to compensate for unexpected movements ofa human to achieve a safe interaction between human and robot [4]. Koay et al.identified human preferences for coordinated arm-base movement in the handoverapproach, observing that the majority of people preferred robots to approach ahandover interaction from the front [67]. Pandey et al. and Mainprice et al. inves-tigated the selection and recognition of handover locations based on the amount ofhuman motion required to complete the handover [78, 97]. As a result of this work,Mainprice et al. designed an approach planner that considers both the mobility ofthe human receiver and robot giver in a cluttered environment. In this system, amobile robot giver is able to plan the handover location and the approach to that lo-cation to allow for a handover that is safe, comfortable, and tailored to how mobilethe human is. In a situation where the human receiver has low mobility (e.g., in aseated position, or is known to be infirmed), the robot plans a long path which getsit close to a human receiver to perform the handover, requiring little by the human.For a human receiver with high mobility (e.g., where the receiver is standing and/orknown to be in good health), the robot plans a shorter path and chooses a handoverlocation requiring shared effort by both the robot and human to reach that location16[78]. In a related stream of work, Sisbot and Alami used kinematic features, alongwith preferences and gaze of the human receiver, to help a robot giver plan trajecto-ries to navigate to a handover location safely and in a socially comfortable manner[106]. In examining human-to-robot handovers, Edsinger and Kemp demonstratedstudy that humans inexperienced with robots were able to hand over and receiveobjects from a robot without explicit instructions [36]. They also found that hu-mans tended to control object position and orientation to match the configurationof the robot’s hand in order to make the robot’s task of grasping the object simpler.Several works have examined how handover trajectories and final handoverposes can best signal the intent to initiate a handover [21, 22, 41, 53, 113]. Huberet al. used minimum jerk profiles in robot handover tasks to emulate human trajec-tories and behaviors [53]. In their work, they found that participants in their ex-periment demonstrated a significantly shorter response time for minimum jerk pro-files compared to trapezoidal joint trajectories, leading to a recommendation thatminimum-jerk trajectory profiles be used for human-robot handovers. Glasaueret al. investigated how a robot can use human-like timing and spatial coordinationof reaching gestures as an efficient and seamless method to convey the intent tohandover an object and signal readiness [41]. They note that their work, “showsthat both the position and the kinematics of the [robot] partner’s movement areused to increase the confidence [of the human receiver] in predicting hand-over intime and space.” In building upon this work, Cakmak et al. and Strabala et al. foundthat the final handover pose should feature a nearly fully extended arm in a natu-ral (human achievable) pose with the elbow, wrist, and distal point on the objectpositioned, respectively, from closest to furthest away from the body in all threedimensions. The object should be held in its default orientation and positioned toallow easy grasping by the human [22, 113]. Cakmak et al. defines default orienta-tion as one in which an object is viewed most frequently in everyday environments.They note that “these orientations are often the most stable orientation for the ob-ject. Rotations around the vertical axis often do not effect the stability of the object,however for non-symmetric objects it can result in different functional properties.”In their study of five objects (plate, notebook, spice shaker, bottle, and mug), thedefault orientations were the upright orientation for the bottle, plate and shaker;upright positions with the handle on the right side for the mug; and the lying in a17readable orientation for the notebook. A related study emphasized the importanceof the physical cues in human-robot handovers, showing that poorly designed han-dover poses and trajectories were often unsuccessful in communicating intent andultimately resulted in handover failure [22]. In this study, the authors developedgestures through temporal (e.g., spatial variation of poses when transitioning intoa handover) and spatial (e.g., difference in pose configurations of the robot armamongst tasks) contrasts to provide fluidity and fluency to the handover interactionand in cueing when a user can successfully execute a handover [22]. They foundthat intent is best communicated by having high contrast between the pose used forholding the object and the pose used for handing over the object, which deterredusers in reaching too early.Recent studies investigating nonverbal gestures involved in handovers haveexamined more subtle kinematic cues such as gaze and eye contact. This workdemonstrated that where participants look during handover plays a significant partin the coordination of handovers [83, 112, 113, 123]. A more comprehensive re-view of prior literature found on gaze cues can be found in Chapter 3, in Sec-tion DynamicsIn studies of grip forces during human-human object handovers, Mason andMackenzie studied force profiles during transport and transfer of the object, findingthat both giver and receiver use somatosensory feedforward control to synchronizetransfer rate during handover [81]. Grip and load forces have also been shown byChan et al. and Kim and Inooka to play a significant role in the coordination of han-dovers. Chan et al. showed that both the giver and receiver utilize similar strategiesfor controlling grip forces on the transferred object in response to changes in loadforces. Through analysis of force loading on the transferred object, they found thatthe giver is primarily responsible for ensuring object safety in the handover and thereceiver is responsible for maintaining the efficiency of the handover [26].182.4 TimingKajikawa et al. observed the occurrence of a delay in handover reaches, as thereceiver begins their reach for the object only after the giver achieves a maximumapproach velocity [60]. Basili et al. quantified this delay by observing that non-verbal indicators suggestive of a handover interaction happened approximately 1.2seconds prior to the occurrence of the actual handover [13]. In another stream ofwork by Admoni et al., it was found that delays in handover increased the amountof attention participants paid to the head, which increased human receivers aware-ness of nonverbal gaze cues [3]. The handover delay also increased compliancewith those suggestions.2.5 SummaryThis chapter presented an overview of the literature on human-robot handoversthat is relevant in the framing of this thesis. Previous studies demonstrate thatnonverbal cues are a very necessary part of handover interactions, both betweenhuman and human/robot dyads. From this work, it is apparent that for human-robot handovers, robots are required to both emote and recognize nonverbal cuesin a variety of different spaces in order to participate in fluent, efficient handoverinteractions. This thesis aims to build upon this previous work by continuing toexplore how cues such as gaze and object orientation may affect handover inter-actions, both quantitatively and qualitatively. Also, in the latter half of this thesis,nonverbal cues in human-to-robot handovers are considered - a topic that has notbeen studied extensively in prior work.19Chapter 3The Effect of Robot Gaze inRobot-to-Human HandoversAside from the shard recognition that the object is to transfer an object from therobot’s grasp into the human’s grasp without dropping the object during robot-to-human handovers, both robot givers and human receivers are additionally requiredto spatiotemporally coordinate their motions to achieve this objective. In human-human interactions, this coordination is done without verbal communication, butrather with subtle nonverbal cues. As seen in Chapter 2, prior work has shownthat there are a great variety of subtle signals that may mediate human-human tocreate fluent and efficient interactions while ensuring the safety of the handoverobject (i.e., that it is not dropped). To investigate whether these subtle and naturalsignals occurring between human agents can improve robot-to-human handovers,these cues can be adapted for, and implemented on robot givers.Using this exploratory framework, this chapter examines the first of the two re-search questions explored in this thesis: “How do nonverbal cues exhibited duringrobot giving and receiving behaviours change how users perceive the robot and af-fect the handover negotiation?” The studies presented in this chapter helps motivatethe examination of nonverbal cues during handovers and provides an exemplar casestudy which focuses on the extent to which a robot’s use of nonverbal cues can pro-foundly impact the way that handovers are carried out between humans and robotsas well as affect user experience of the task. More specifically, this work examines20how human-inspired robot gaze cues can affect spatiotemporal characteristics andsubjective experience of a robot-to-human handover task.In human-human social behaviour, gaze has shown to have a profound effecton communication as a signal for interpersonal attitudes and emotions. It is a cuemedium that has shown to have vast bandwidth, being able to convey a wide varietyof cues including attention, mood, attraction, intention, and specific conversationalactions [8, 10, 61]. Because of its versatility, gaze has been studied within HRISboth as a method for a robot to better understand human agents [11, 111], and howanthropomorphic robots having a head can leverage gaze cues to establish moreefficient communication with a human user [55, 90, 107]. In line with the latterstream of work, this chapter examines how certain gaze cues, inspired by observedgaze profiles used by humans during object handovers, play a role in establishingfluency and quality of a robot-to-human handover.Studies presented in this chapter address whether gaze can be used to augmenta robot-to-human handover by subtly communicating handover location and tim-ing, while also providing an intuitive social interaction modality to help inform thehuman receiver’s behavioural decision on when and where to reach for the han-dover object. Gaze cues, in either human-human or human-robot interaction, haveproven to be effective for communicating attention [66, 90]. During a handover,givers use verbal or nonverbal cues to direct the receiver’s attention to an object.Successful handovers typically take place when the two parties achieve shared at-tention on the same object. Previous studies [74, 112, 113] indicate that gaze canbe used by robots to signal handover intent to users before the handover event.However, unlike these studies, the work presented in this chapter investigates theeffect of robot gaze during the handover on the timing of the handover event. Here,gaze behaviours inspired via observations of human-human handovers were im-plemented on a PR2 humanoid robot. Through observation of user preferencesand timing of the interaction, this research demonstrates that gaze can play a keyrole in improving the user experience of human-robot handovers, and help makehandovers fast and fluent.The contents of this chapter are largely transcribed from a manuscript of whichI was one of the authors, entitled, “Meet me where I’m gazing: how shared atten-tion gaze affects human-robot handover timing” [88]. This manuscript was sub-21mitted to the 9th Association for Computing Machinery (ACM)/IEEE InternationalConference on Human-Robot Interaction held from March 3-6, 2014 at BielefeldUniversity in Bielefeld, Germany. This work was awarded “Best Paper” at the con-ference. The version of the manuscript that appears in this chapter contains severalminor modifications to the original conference paper. These modifications include:• Figures have been updated for enhanced clarity.• The introduction contained in Section 3.1 has been abridged.• Background information originally found in Section 3.2 that reiterates theliterature review contained in Chapter 2 has been removed.• Terminology has been updated to match terms used in this thesis.• Experimental conditions have been renamed for clarity.• Discussion (Section 3.7) and conclusions (Section 3.8) of this work havebeen expanded to address impact and limitations of the work presented inthe chapter, as well as directions for future work.Supporting materials for this work including advertisements, consent forms andsurveys can be found in Appendix A.3.1 IntroductionThis work seeks to improve human-robot handovers by investigating how gazecan be used to augment a handover event, subtly communicating handover location,handover timing, and providing acceptable social interaction signals. Gaze cues,in either Human-Human Interaction (HHI) or in HRI, have proven to be efficientfor communicating attention [66, 90]. During a handover, givers use verbal ornonverbal cues to direct the receiver’s attention to an object. Successful handoverstypically take place when the two parties achieve shared attention on the sameobject. Previous studies indicate that gaze can be used by robots to signal handoverintent to users prior to the handover event [74, 112, 113]. However, these studiesdid not explore the effect of robot gaze during the handover on the timing of thehandover event.22It is hypothesized that the use of gaze cues during human-robot handover caninfluence handover timing and the subjective experience of the handover by im-plicitly increasing communication transparency and perception of naturalness forthe interaction.In this work, two studies are conducted: a human-human study (Section 3.3)followed by a human-robot study (Section 3.4). The human-human study was con-ducted to observe gaze patterns used during human-human handovers. The resultsof this first study was used to inform the second study on robot gaze during robot-to-human handovers: in this latter human-robot study, the two most frequentlyobserved gaze patterns from the first human-human study were implemented alongand a ’No Gaze’ condition on a PR2 humanoid robot platform (Willow Garage Inc.,Menlo Park, California, USA) in the giver role. The following two questions areaddressed through these studies:1. Can gaze improve the subjective experience of handovers?2. Can gaze be used to produce faster, more fluent handovers?The key results from this work are as follows:• Subjects reach for the object significantly faster when the robot directs itsgaze towards the intended handover location (Shared Attention gaze) thanwhen no gaze cues are used.• Subjects tend to perceive a handover as more natural, communicative of tim-ing, and preferable when the robot provides Turn-Taking gaze in addition toShared Attention gaze.3.2 Background - Gaze in HandoversGaze is an important and useful cue in HHI. People repeatedly look each otherin the eye during social interaction and people do not feel that they are fully en-gaged in communication without eye contact [9]. Studies in psychology haveshown various functions of gaze in social interaction, such as seeking and pro-viding information, regulating interaction, expressing intimacy, exercising social23control, etc. [9, 66, 98]. Gaze can be named differently in different social situa-tions [90]; for example, mutual gaze or eye contact is defined as two people lookinginto each other’s face or eye region [119], while deictic gaze or shared visual at-tention is defined as one person following the other’s direction of attention to lookat a fixed point in space [20].Previous work has shown the importance of gaze in HRI. For example, Staudteand Crocker [111] demonstrated that humans react to robot gaze in a manner typi-cal of HHI. Since gaze behaviour is closely linked with speech [8], much work hasbeen done on the conversational functions of gaze in HRI [72, 75, 83, 89, 91, 118].Gaze is particularly effective in regulating turn-taking during human-robot conver-sation. Kuno et al. [72] developed gaze cues for a museum guide robot to coordi-nate conversational turn-taking. Matsusaka et al. [83] used gaze cues to mediateturn-taking between participants in a group conversation.Another large body of literature focus on using gaze to direct people’s atten-tion in HRI [16, 47, 56, 100, 105]. Gaze was combined with pointing gestures in[16, 47, 56] to direct people’s attention, which the authors believed would makethe interaction more human-like [16] while minimizing misunderstanding [47]. InRich et al.’s work, four types of connection events were identified from HHI videos,namely directed gaze, mutual facial gaze, adjacency pairs and back-channels. Im-plementing them in an HRI game showed a high success rate in forming human-robot connection or joint attention [100]. In work done by Sidner et al., peopledirected their attention to the robot more often in interactions where gaze waspresent, and people found interactions more appropriate when gaze was present[105].Introducing gaze cues can also benefit HRI in other ways. Mutlu et al. andSkantze et al. showed that gaze increased human performance in certain human-robot tasks [89, 107]. Other studies have shown that gaze heightened human-robotengagement and contributed to the perceived naturalness of a communicating robot[72, 75, 105].In the study of human-robot handovers, other researchers have shown that gazecan be useful in communicating the intent to initiate a handover. Lee et al. [74]studied human motion and gaze cues as people approached each other for han-dovers. They found that people looked at the object or at the receiver as they ap-24proached the receiver. Strabala et al. [112] examined the signals that humans useto communicate handover intent before a handover takes place. They initially ac-knowledged gaze as one of the important features that mark the difference betweendifferent phases in handover, but they did not find gaze to be an effective predictorof handover intent. In contrast, Kirchner et al. [65] demonstrated how robot gazecan be effective in targeting an individual recipient out of a group of people for arobot initiated handover. Atienza and Zelinsky [11] augmented handover interac-tions with gaze, demonstrating a system that allowed a human to request an objectfor handover by looking at it.While the above studies addressed gaze in pre-handover cuing and communica-tion of intent to handover, this work examines the use of gaze during the handoverevent. Although the effectiveness of gaze in regulating handover intent remainsan open question, gaze may have a positive effect when used during the handoverevent, since it helps establish shared attention and has been shown to improvehuman-robot tasks. Hence, it is hypothesized that gaze may be useful in improvingthe handover itself by establishing shared attention and signalling the robot’s endof turn.3.3 Study I: Observing Gaze Patterns inHuman-to-Human HandoversTo inform the human-robot handover study design, a study on gaze behavioursof human-to-human handovers was conducted. In this study, a bottle of waterwas handed over between participants acting as giver and receiver multiple times.The gaze behaviour of the giver was recorded and observed during the handover.While other researchers have observed gaze in human-human handovers beforethe handover event, [e.g., 112], the work described here aims to augment theseprevious results with a study focusing on gaze during the handover event.3.3.1 Experimental ProcedureTwelve volunteers (10 male, 2 female) participated in this study in pairs. Thegiver was asked to handover ten bottles from a side table to the receiver one at atime. The receiver was asked to bring the bottles to a collection box about two25meters behind them one at a time, requiring him/her to walk away from the giverbetween each handover. This process repeated until all ten handovers were com-pleted. Each participant performed the role of the giver, then was paired withanother participant and performed the role of the receiver, resulting in a total oftwelve giver-receiver pairs (120 handover events in total).In order to collect human gaze patterns that can inform the design of human-robot handovers, the giver and receiver were instructed not to talk during this pro-cess. The giver was also instructed to pick up the bottles from the side table onlyafter the receiver returned from the collection box and had put his/her hands on thetable. By requiring the receiver to turn and walk away, the common attention be-tween the giver and receiver was interrupted after each handover and participantsneeded to re-connect for the next handover.3.3.2 ResultsAnnotation of a frame-by-frame video analysis of the givers’ gaze patternsindicates that the giver’s gaze during a handover can shift between three positions:the object being transferred, the expected handover position, or the receiver’s face.Through a frame-by-frame analysis, givers’ gaze patterns from video recordings ofthe 120 handovers were annotated. From this process, the following gaze patternswere found (Figure 3.1):26Figure 3.1: Giver’s gaze patterns observed from human-human handovers. ’Shared Attention’: continual shared atten-tion gaze; ’Face’: continual face gaze; ’Turn-Taking’: long Shared Attention gaze followed by a short Face gaze;’Short Face Shared Attention’: short Face gaze followed by a long Shared Attention gaze; ’Long Face SharedAttention’: long Face gaze followed by a short Shared Attention gaze.27• Shared Attention Gaze (Figure 3.2 Top): The most frequent gaze pattern(68% of all handovers observed) consists of the giver gazing at a projectedhandover location as s/he reaches out to execute the handover. After pickingup the bottle, the giver turns to face the receiver, looks down at a midpointbetween the giver and the receiver, and keeps the gaze there until the receivertakes control of the bottle. This midpoint is approximately where the han-dover takes place. There is no eye contact between the giver and the receiverthroughout this handover gaze pattern.• Face Gaze: In some other (10%) handovers, the giver gazes at the receiver’sface, perhaps to establish an eye contact, throughout the handover. This gazebehaviour towards the receiver’s face is labeled ’Face Gaze’.• Turn-Taking Gaze (Figure 3.2 Bottom): In 9% of handovers, a slight varia-tion from the Shared Attention Gaze was observed. In addition to gazing ata projected handover location while reaching out, the giver also looked up tomake eye contact with the receiver near the end of the handover motion, atapproximately the time that the receiver made contact with the bottle.• Short Face Shared Attention Gaze: In 8% of the handovers, the giver lookedat the receiver’s face and quickly glanced at the bottle when the receiver isabout to touch the bottle.• Long Face Shared Attention Gaze: In 5% of the cases, the giver glancedat the receiver before but not during handover, and shifted the gaze to thehandover location when the receiver is about to touch the bottle.3.3.3 DiscussionThe results of this study suggest that humans use a variety of gaze patternswhile handing over an object to another person. In general, the giver tends to shifthis/her gaze from the object being handed over to the receiver’s face (Face gaze),the projected location at which the handover should take place (shared attentiongaze), or a combination of the two. The shared attention gaze in the Shared At-tention, Turn-Taking, Short Face Shared Attention and Long Face Shared Attention28Figure 3.2: Demonstration of two frequently observed gaze behaviours fromthe human-human handover study. Top: Shared Attention gaze - thegiver looks at the location where the handover will occur. Bottom: Turn-Taking gaze - the giver looks up at the receiver after the shared attentiongaze. 29patterns can be interpreted as serving the function of communicating where thephysical transfer of the object should happen. The long Face gaze in the Face andLong Face Shared Attention gaze patterns serves a similar function as the face gazein verbal conversation of regulating a turn [65]; in verbal conversations, the speakertypically ends his/her utterance with a sustained gaze at the listener, signaling will-ingness to hand over the speaker role, while in this case, to hand over the object.The short Face gaze in the Turn-Taking and Short Face Shared Attention gaze pat-terns serves a monitoring function [65], appearing to observe whether the receiveris paying attention to, or is ready for, the transfer of the object. These results inspirethe hypothesis that an implementation of analogous gaze cues for a human-robothandover could serve similar functions and help the human-robot dyad performmore fluently. Hence, the gaze patterns observed in this study informed the designof experimental conditions in a second study. Section 3.7 provides more discus-sions of the results from the two studies contrasting human-human and human-robot handovers.3.4 Study II: Impact of Human-Inspired Gaze Cues onFirst-Time Robot-to-Human HandoversTo examine the impact of robot gaze on human receiver behaviour, a PR2humanoid mobile robotic platform (Willow Garage Inc., Menlo Park, California,USA) was used with a pan-tilt head and two 7-DOF arms, each with a two-fingered,1-DOF gripper. In the following section, the physical handover cues used by thePR2 are outlined (Section 3.4.1), describe the three experimental gaze conditionsinspired from the human-human handover study (Section 3.4.2), and outline theexperiment design and technical implementation (Sections 3.4.3 and 3.5).3.4.1 Physical Handover CuesIn designing the robot’s motions performed during handover, we looked to-wards work done by Cakmak et al. and Strabala et al. which emphasized the im-portance of the physical cues in human-robot handovers, showing that poorly de-signed handover poses and trajectories were often unsuccessful in communicatingintent and ultimately resulted in handover failure [22, 113]. They found that intent30is best communicated by having high contrast between the pose used for holdingthe object and the pose used for handing over the object. We have followed theseguidelines in the design of our handover poses and trajectories.In the experiment, based on the findings of Basili et al. [13] and Koay et al.[67], the robot was positioned such that it was facing the participant approximately1 meter away. The robot executed the handover with its right gripper, as recom-mended by Koay et al.. In the beginning of each handover, the robot starts itsmotion at the Grasp Position with its end-effector prepared to grasp a bottle sit-ting on a table at the robot’s right side. When subject is ready, the end-effectorgrabs the bottle (marking a start time, t = 0 of the interaction), then moves thebottle horizontally to a position in front of the robot’s center-line (Ready Position).Then the robot moves from the Ready Position forward to the handover location.The joint-angle goals of the Grasp Position, Ready Position, and Handover Loca-tion were predefined such that when the robot’s end-effector is extended, the arm ispositioned in a natural pose: the elbow located below the shoulder, and the gripperlocated below the distal point on the bottle, as shown in Figure 3.3. The HandoverLocation was selected in accordance with the recommendations of previous work[65, 74]. The three locations are constant for all three gaze conditions. While otherresearchers have proposed handover controllers that adapt to the position of thehuman’s hand,[e.g., 37], a constant Handover Location was maintained through-out the experiment and only vary gaze cues used during handovers.When the robot’s arm reaches the Handover Location, the robot waits for aparticipant to grasp and pull up on the object. The force the gripper exerts on thebottle is a linear function of the downward force exerted by the bottle as describedby Chan et al. [27]. Thus, as the subject takes the weight of the bottle, the robotreleases its grip (marked as the release time). The PR2’s fingertip pressure sensorarrays were used to realize Chan et al.’s handover controller. Finally, after releasingthe object, the robot returns to the Grasp Position, ready to grasp and lift the nextobject.31Figure 3.3: Illustration demonstrating the experimental set-up and the threeconditions at the Handover Location: a) No Gaze; b) Shared Attention;and c) Turn-Taking. An array of infrared sensors was located at theedge of the table. The green dotted lines represent the location wheresubject’s reach motion is detected. Participants stood at a specified lo-cation marked on the floor.323.4.2 Experimental Gaze CuesIn this study, the PR2 robot expressed gaze through head orientation. Imaiet al. [55] showed that robot head orientation can be an effective substitute forhuman-like gaze and that head orientation is interpreted as gaze direction. In orderto minimize any possible confusion regarding the robot’s gaze direction, a singleobject for the handovers was used.Three different gaze patterns in human-robot handovers were tested, as shownin Figure 3.4. In all conditions, the robot’s gaze tracks its end-effector from theGrasp Position to the Ready Position as though the robot is attending to the acqui-sition of the bottle. When the end-effector arrives at the Ready Position, the robot’shead is tilted downwards towards the end-effector. Only when the robot arm tran-sitions between the Ready Position to Handover Location does the robot transferits gaze according to the following gaze patterns.The No Gaze gaze condition acts as the baseline for this study. The robot headremains looking down towards the ground while the end-effector extends forwardfor the handover.The Shared Attention gaze condition models the most frequently observed gazepattern from the human-human handover study. When the robot starts to movefrom the Ready Position to the Handover Location, it smoothly transitions its gaze(head orientation) from the bottle to the location in space where the handover willoccur, as an implicit cue intended to direct the human’s attention towards the pro-jected Handover Location. With this condition, the hypothesis that shared attentioncan be established through gaze during handovers is testable, and that doing so ben-efits the handover interaction. Establishing shared gaze at an object or location canserve to direct shared attention ([e.g., 56]) and can aid in the successful executionof human-robot cooperative tasks ([e.g., 107]).The Turn-Taking gaze condition is also derived from the human-human han-dover study, and is analogous to the third most frequently observed gaze pattern.This gaze condition was implemented on the robot over the second most frequentlyobserved gaze pattern - the Face gaze - as it appeared to offer more information tothe user i.e., a turn-taking timing cue, and would pose as a more interesting ex-perimental condition. When the handover trajectory begins, the robot smoothly33Figure 3.4: Diagram depicting the gaze cues for the No Gaze (bottom),Shared Attention (middle), and Turn-Taking (top) gazes. In the Turn-Taking condition, the robot shifts its gaze from the Handover Loca-tion to the human’s face midway through the handover motion.34transfers its gaze to the Handover Location, as in the Shared Attention condition,but then shifts its gaze up to the human’s face in a quick motion, reaching the finalgaze position at approximately the same time that the handover motion completes.Here, two hypotheses are tested: that (a) this gaze shift can cue handover timing,and (b) looking at the face can improve the subjective experience of the handover.This type of gaze shift has been shown to be a meaningful human-robot turn-takingcue [23] and mutual gaze can increase the sense of engagement and naturalness inhuman-robot interactions [75, 105].3.4.3 Experimental ProcedureA paired-comparison handover study was conducted in a controlled room. Thestudy took place on the day of a university orientation event such that many anddiverse naive participants could be rapidly recruited during the public event. Sec-tion A.1 shows the advertisements used for recruitment. The experiment was struc-tured as a balanced incomplete block design (v= 3,b= 96,r = 64,k= 2,λ = 32) 1to both support rapid trials (maximum 5 minutes) and include only naive reactions:each participant evaluated one of the three condition pairings. Condition order wasrandomized and presentation order counterbalanced among trials.Participants read a consent document, shown in Section A.2.1, outlining theexperimental procedure and risks. Each participant provided verbal informed con-sent (Section A.2.2), then entered the room where instructions were given. Theywere told to stand at a marked area facing the robot, and informed they would par-ticipate in a handover interaction. Participants were also told that the robot wouldpick up the water bottle placed beside it and hand it to them. They were asked totake the bottle from the robot whenever they felt it was the right time to do so. Toavoid unintended cuing, during handovers the experimenters sat out of the field ofview of participants.After receiving the first bottle, participants placed the bottle in a box approx-imately 3 meters behind him/her. This served as a washout between handovers,1These variables indicate the structure of a balanced incomplete block design: v = number oftreatments (i.e., three conditions were tested: No Gaze, Shared Attention, and Turn-Taking gazeconditions); b = number of blocks (i.e., observations from a total of 96 participants was analyzed);r = number of replicates (i.e., each condition was tested on a total of 64 participants); k = block size(i.e., each participant saw two conditions); and λ = r(k−1)/(v−1).35breaking the participant’s focus on the robot and the handover, as was done pre-viously by Cakmak et al. [22]. Participants then filled out a short questionnaireregarding the handover interaction they just conducted (shown in Section A.3.1),returned to the same marked area front of the robot and participated in a secondhandover. Participants were permitted to keep the last bottle given to them by therobot.During each handover, the following events were timestamped: start of robotmotion (start time), end of robot motion (end of motion time), start of release of therobot’s gripper (release time), and the participant’s first reach for the object (reachtime) as measured by the motion sensor array described in Section 3.5.After the second handover, participants left the room and completed a shortquestionnaire (shown in Section A.3.2) comparing the two handovers on three sub-jective metrics: overall preference, naturalness, and timing communication. Foreach of the following three questions, participants were asked to select either thefirst or second handover:1. Which handover did you like better, overall?2. Which handover seemed more natural?3. We think that timing is important in human-robot handovers. Which han-dover made it easier to tell when, exactly, the robot wanted you to take theobject?Participants could also provide additional comments.3.5 Technical ImplementationTo control the PR2, the Robot Operating System (ROS) (Willow Garage Inc.,Menlo Park, California, USA) used by the PR2 was extended with a series of soft-ware modules coordinated via the Blackboard architectural design pattern [48] asshown in Figure 3.5. One module controlled the robot’s arm and another, its head.The head-control module provided object tracking functionality for bringing theobject to the Ready Position, and a smooth, fast gaze transition (average 90 de-grees/second) functionality to enable the Shared Attention and Turn-Taking gaze36conditions during the handover motion. An independent module logged quantita-tive measurements of robot’s start time, end of motion time, and release time.An array of three passive SEN-08630 infrared motion sensors (SparkFun Elec-tronics, Boulder, Colorado, USA) configured as a light curtain was placed at theedge of the table (as shown in Figure 3.4), and was used to detect the start of theparticipant’s reach (reach time) triggered by the participant’s hand crossing the ta-ble edge. An Arduino UNO microprocessor (Arduino LLC, Turin, Italy) relayedthe sensor reading to the computer controlling the robot. Sensor readings werelogged and time-synchronized with the robot.3.6 ResultsA total of 102 volunteers participated in the experiment. Six records were re-jected as instructions outlined in Section 3.4.3 were not followed. Subsequently,data from 96 participants (33 females, 63 males), aged 18-61 years [M=23, SD=5.59]was analyzed. Due to technical error, reach time was not logged in the second han-dover for five of the participants. This did not affect the analysis of handovertiming, since the focus of this study the focus of this study on first-time responsesrequires reach time measures from only the first handovers. No other technicalfailures occurred and all handovers were successful, i.e., no bottles were dropped.3.6.1 Handover TimingFigure 3.6 shows the distribution of three key times: the robot’s end of motiontime, participant’s reach time, and robot’s gripper release time. All times are mea-sured relative to the start time of the interaction - i.e., the robot’s start of motion.A comparison using unpaired t tests of these times to human-human handoversobserved by Basili et al. show that duration between handover initiation to thegiver’s release of the object is significantly slower (by approximately two seconds)for human-robot handovers compared to the human-human handovers (p < .0001)[13].37Figure 3.5: System flow diagram describing algorithm used for PR2 robotgaze during handovers.38Figure 3.6: Chart of handover timing results for various gaze conditions in the robot-to-human handover study. Alltimes are measured with respect to the robot’s start of motion at t = 0. The dashed line at 2 seconds indicates theend of robot motion at the Handover Location. Reach time indicates the participant’s reach toward the profferedobject crossing the infrared sensors. Note that in the case of the Shared Attention condition, participants start toreach before the robot has reached Handover Location. The mean reach time for the Shared Attention conditionis significantly earlier than that of the No Gaze condition. Error bars indicate 95% confidence intervals.39A one-way Analysis of Variance (ANOVA) was conducted on participants’ reachtime across the three conditions. A significant learning effect between the first andsecond handover trials is observed [t(90)=4.21, p<.001, d=0.43]. In this learningeffect, reach time is significantly earlier for the second set of handovers for partici-pants. As the goal of this work is to understand first time, inexperienced behaviour,only the reach time collected during the first of the two handovers performed byeach participant was used. The entire robot motion from the Grasp Position to theHandover Location consistently took 2.02 seconds (SD = 0.01).Participants’ reach time varied across the three gaze conditions [F(2,93)=6.49,p<.005] as plotted in Figure 3.6; post-hoc analyses used a Bonferroni correc-tion. Participants reached for the object significantly earlier with Shared Atten-tion [M=1.91, SD=0.52] than with No Gaze [M=2.54, SD=0.76] (p < .005). Notethat the mean reach time for Shared Attention occurs before the robot has stoppedmoving at the Handover Location (reach time <end of motion time). No signifi-cant differences were found between Shared Attention and Turn-Taking [M=2.26,SD=0.79], or between Turn-Taking and No Gaze.3.6.2 Subjective ExperienceDurbin’s test [35] was employed to contrast overall preference, perceived natu-ralness, and timing communication across the three gaze patterns during handoverson the questionnaire data. This test is analogous to a Friedman test for rank data,but adapted to balanced incomplete block designs.Possible gender effects were checked using Mann-Whitney U tests. No signif-icant effects of gender were found (overall preference: [U=935.0, p=.23, r=0.12];naturalness: [U=918.5, p=.18, r=0.14]; timing communication: [U=935.5, p=.22,r=0.12]). One-sample Wilcoxon signed rank tests allowed for observation of po-tential bias in selecting the first or second handover experience in the questionnaire.From these tests, significant biases towards selecting the second handover on thetiming communication metric [Z=2.22, p<.05] and a weak trend to select the sec-ond handover on both overall preference [Z=1.62, p= 0.11] and naturalness metrics[Z=1.41, p= 0.16] were found. The rank data collected using the questionnaire isinsufficient to correct for this bias statistically.40Given this general bias to select the second handover, finding statistical signif-icance to α = .10 in questionnaire results is noteworthy [116]. Hence, observationof trends having p < .10 is reported in Table 3.1.Table 3.1: Ranking of questionnaire results. Each cell represents the numberof people who chose the row condition over the column condition. *indicate pairwise comparisons that are significant to p < .10 (none weresignificant to p < .05). Note that participants’ bias to select the secondhandover experience regardless of experiment condition was observed tobe significant.Overall PreferenceCondition Turn-Taking Shared Attention No GazeTurn-Taking - 21* 19*Shared Attention 11 - 17No Gaze 13 15 -NaturalnessCondition Turn-Taking Shared Attention No GazeTurn-Taking - 20* 19Shared Attention 12 - 19No Gaze 13 13 -Timing CommunicationCondition Turn-Taking Shared Attention No GazeTurn-Taking - 21* 18Shared Attention 11 - 19No Gaze 14 13 -Overall Preference: No significant difference in user preference across thethree gaze conditions was found [T 2 = 2.04, p = .14]. However, one-tailed pair-wise comparisons demonstrate a trend for preference toward Turn-Taking over NoGaze (p < 0.10) and Shared Attention (p < .10) conditions.Naturalness: No significant difference in perceived naturalness of the han-dovers across the three gaze conditions were found [T 2 = 1.82, p = .17]. How-41ever, participants tended to choose Turn-Taking as more natural than Shared Atten-tion (p < .10) but not over the No Gaze condition.Timing Communication: No significant differences were found in the per-ceived communication of timing across the gaze conditions [T 2 = 1.65, p = .2].However, participants tended to choose Turn-Taking over Shared Attention (p <0.10), but not over No Gaze, as easiest to communicate handover timing.In total, 59% of all participants provided additional comments (optional) onthe questionnaire. Twelve subjects who experienced the Turn-Taking conditionexplicitly used words such as “head motion”, “eye contact” or “looking at me”and expressed the condition in a positive light (e.g., Participant 90 compared NoGaze with Turn-Taking: “During second handover [Turn-Taking], robot made eyecontact, which made it easier to tell when the bottle should be taken.”; Participant10 compared Shared Attention and Turn-Taking: “I liked it when robot looked atme. That confirms it’s good to take.”). However, another twelve subjects expressedthat they did not notice any difference between the conditions.3.7 DiscussionBuilding on previous work that studied communication of intent to handoverusing gaze, these two studies explored how gaze during handovers (i.e., after theintent to handover has already been communicated and while the handover is takingplace).A handover interaction typically involves the well-defined role assignments ofa giver and receiver, and a clear sequence of actions that must take place - i.e., thegiver grabs and transports the object, and then the receiver receives the object fromthe giver. However, even in such a well-defined interaction between two agents,the details of when and where must be dynamically negotiated for the interactionto be successful.As an effort to understand what nonverbal cues may help in this process, thefirst study (Study I) was conducted to identify gaze patterns human givers use whenhanding over an object to another person. Results of this study not only providefive different gaze patterns human givers use in an human-human handover, but it42also suggests that the observed patterns more often than not involve gazing at theprojected handover location.Following Study I, Study II explored the impact of robot gaze on human-robothandover timing and user perception of the handover experience. Results show thatparticipants reached for the proffered object significantly earlier when the robotperformed a Shared Attention gaze at the projected Handover Location. In fact,participants reached for the object even before the robot had arrived and stoppedat the Handover Location (a mean of 0.11 seconds before the end of motion time).This is in contrast to the No Gaze condition where the mean reach time is 0.52seconds after the robot’s end of motion. In this study, participants were explicitlytold that the robot would be handing over objects to them and that they were to takethe object from the robot. In addition to this foreknowledge, the robot used highlycontrasting poses between the Ready Position and the Handover Location which,according to Cakmak et al. [22], makes the robot’s intent to hand over the bottlevery clear. Hence, it is unlikely that the observed difference in timing betweenthe gaze conditions is due to uncertainties in understanding the robot’s handoverintent. Rather, the results suggest that the robot’s gaze at the projected HandoverLocation supplements the communicated intent with implicit information on wherethe handover should take place. This may be helping to establish shared attentionon the Handover Location even before the robot arrives there, naturally allowingparticipants to respond and meet the robot at the location earlier than when such acue is absent. Thus, the result best supports an increase of fluidity in the executionof the handover as it takes place.However, the role of mutual gaze used in the Turn-Taking condition requiredfurther investigation. At the beginning of the robot’s handover motion, the robotexpresses the same locational, shared attention gaze in both the Shared Atten-tion and Turn-Taking conditions. Hence, it was surprising to find that the reachtime of the Turn-Taking condition is not significantly earlier than that of the NoGaze condition.The Turn-Taking condition was intended to test two hypotheses: that the Turn-Taking gaze would cue handover timing, and that looking at the participant’s facewould improve the subjective experience of handover. While a trend was observedsuggesting that the robot’s gaze directed at the face improves the subjective ex-43perience of handover, it appears that the shared attention gaze cues the handovertiming instead of the mutual, turn-taking gaze. It may be that the mutual gaze im-plemented for this study served the function of acknowledgement rather than theintended function of turn-taking. Qualitative differences in participants’ reactionmay exist between the Shared Attention and Turn-Taking conditions. For exam-ple, participants may have started to respond to the robot’s shared attention gazein both conditions but, prior to being detected by the reach sensor, saw the robot’sturn-taking gaze and stopped to make eye contact before continuing to reach acrossthe table.This raises unexplored questions about how participant’s reach time is affectedby the timing of the robot’s gaze. How much would varying the robot’s gaze tim-ing affect human reach time? Is the timing of the robot’s gaze a more dominantcue than the location the robot is gazing at? That is, would a robot that shifts itsgaze from the object directly to the person’s face during handovers have the sameeffect as the Shared Attention condition? Would we see changes in participants’reach direction if the robot gazed at a different location? Without a thorough qual-itative analysis, it is difficult to tell, with accuracy, if and when shared attention isestablished with the participant. A separate experiment with a gaze tracking devicewould help answer these questions, and is left as future work.It is important to note that the results may be representative of first-time, in-experienced participants only, where novelty effects may have motivated themto observe the robot more carefully than they would if they were more familiarwith the robot. Unsurprisingly, a significant training effect was observed in thereach time data, as well as a bias toward describing the second handover experi-ence more favourably in the questionnaire regardless of the condition experienced.Some of the participants’ comments suggest that in certain cases, people did notpay attention to the head of the robot at all. Indeed, it is suspected that in manyhuman-human handover scenarios, especially those that are repetitive or trained(e.g., passing a baton in a sprint relay race), people do not use gaze cues at all andyet succeed in object handover. Thus, it is hypothesized that robot gaze cues maynot have the same effect on trained or familiarized users.Although the earlier reach time of participants in a handover may seem moresimilar to natural, unscripted human-human handovers, this may not necessarily44be desired in some human-robot handover situations. Depending on the handovercontroller implemented on a robot, handover timing may need to be controlledsuch that people naturally grab the object only when it is safe to do so. Many ofthe handover controllers that modulate the release time of the object are built forcases where the robot’s gripper is already at the Handover Location before peoplegrab the object. A situation where the object is grabbed before the robot is ready torelease the object could lead people to pull hard on the object, possibly damagingor dropping the object, or resulting in a negative perception of the robot.In the context of this thesis, this work motivates the examination of nonverbalcues within human-robot handovers and shows that the use and adjustment of robotgaze profiles can significantly alter the timing and user experience of the interac-tion. Although we find that handovers between robots and humans observed in thisstudy are still significantly slower than handovers between humans, the significantdifference in interaction duration between the Shared Attention and No Gaze gazeconditions supports the notion that the appropriate design of robot gaze profilesmay lead to a more streamlined encounter. Beyond gaze, the finding of these stud-ies presented here suggests that carefully designing nonverbal cues for robots, per-haps through observation of human behaviours, can indeed improve fluency andefficiency of human-robot handovers.3.8 ConclusionsPrevious work on gaze in human-robot interactions has offered evidence thatgaze can affect human behaviours. The studies presented in this chapter examinethe effect of robot-generated gaze cues can effectively influence details of humanaction for a successful, fluent robot-to-human handover. Results from a human-human handover study identified five types of gaze patterns that humans tend touse during handovers. In a second study observing robot-to-human handovers, arobot mimicked two gaze cues identified from the first study (Shared Attention andTurn-Taking), showing how a robot’s use of human-inspired gaze expressions canaffect the timing of a robot-to-human handover in first-time participant responses.The study provides empirical evidence that a human-inspired gaze pattern (SharedAttention) implemented on a robot can elicit a human receiver to reach for and45retrieve the proffered object earlier compared to a condition where no gaze cues(No Gaze) are offered.Addressing the first research question of this thesis, this work has demonstratedhow a robot’s human-like use of gaze expressions in human-robot handover can af-fect the timing of the handover event. Once the intention to handover is established,providing a gaze cue to the projected Handover Location seems to offer rich gestu-ral information about the handover event, and allows users to reach for the objectsooner, even before the robot arrives at the location.3.8.1 LimitationsCross-culturally, gaze may signal differently to people of various ethnic ori-gins, and thus the results of this study may not be reproducible outside of NorthAmerica. As the results were obtained from a majority sample of non-expert usershaving little to no experience with robotics that were interacting with the robot ina controlled setting, the findings may not applicable to users who will have had ex-tensive opportunity to interact with robots in the field. As the results of the human-robot study show the occurrence of a learning effect happening with participants,we would expect findings will differ for expert users.3.8.2 Future WorkAs a result of this work, there are several directions for future inquiry that mayexamine research questions related to gaze in handovers. One avenue of possiblework examines direction of gaze: if the robot’s gaze was intentionally directed to adifferent location rather than the predicted handover location or the receiver’s face(perhaps in an attempt to intentionally mislead the user) would there be changes tothe handover timing or participants’ reach direction? If so, it would offer additionalevidence to the importance of gaze as a nonverbal cue that may be important forother HRIS. Another research question that would be interesting to explore relatesto if, when and under what conditions shared attention mutual gaze is establishedbetween robot and human during a handover event. Furthermore, do users prefereye-contact gaze in addition to the locational, shared attention gaze? An experi-46mental setup with a gaze tracking device would help answer these questions in afuture study.47Chapter 4Characterization of HandoverObject Orientations InformingEfficient Robot-to-HumanHandoversThe work presented in the previous chapter showed that subtle gaze cues gen-erated by a robot in a robot-to-human handover can have a significant impact on theefficiency and fluency of the handover. In reference to the first research questionposed in Section 1.2, observing that gaze cues can affect these aspects of an inter-action suggests that other nonverbal cues may have significant, and perhaps unique,consequences within the scope of HRIS. This chapter continues to investigate cuesfor robot-to-human handovers, though begins to address the second research ques-tion of this thesis: how can a robot adequately recognize and interpret nonverbalcues conveyed by a human giver to infer object attributes? In particular, focus isgiven to a more implicit cue: the orientation of objects as they are being passedfrom giver to receiver. From observation of these orientations, the aim of this workis to determine if and how a robot may be able to infer an object’s affordances.Objects’ affordances and how they are oriented during handovers have a sig-nificant impact on the handover experience, and in particular, for the receiver of48a handover object [5]. Good handover orientations - those that direct object af-fordances such as a handle towards the receiver - assist in maintaining handoverefficiency and fluency. On the other hand, bad handover orientations which do notconsider object affordances may not only handicap efficiency and fluency, but mayalso cause awkwardness, mistaken intentions and/or injurious results. Figure 4.1illustrates examples of both good and bad handover orientations. To ensure thatrobots act as well-behaved givers, they must be trained to consider appropriateobject orientations during handovers. Specifically, robots must always direct theobject’s affordances towards the receiver for a more fluent handover. This chapterwill examine natural handover orientations used by people as a step towards en-abling robots to learn how to appropriately orient objects during robot-to-humanhandovers.Figure 4.1: Illustration depicting the importance of considering object orien-tation during robot-to-human handovers. On the left, the orientation ofthe knife used by the robot during the handover allows the user to graspthe knife’s affordance, the handle, easily. On the right, the robot doesnot consider the object’s affordance when attempting to perform a han-dover; thus, the intent of the robot is not fluently communicated, theinteraction is awkward/dangerous, and the human no longer wishes toparticipate in future HRI experiments.In this work, a user study is performed to examine orientations of 20 common,household objects during handovers between people. This study compares how un-prompted humans naturally orient handover objects to orientations that are giver-49and receiver-centered in terms of the objects. The intent of this comparison is toexamine if observations of natural handovers between humans can be used as tem-plates by robot givers to achieve more efficient and socially acceptable handovers;that is, can robots assume that givers in natural handovers use object orientationsthat are most considerate to the receiver? Through this exploration, this work alsointroduces the notion of affordance axes for certain objects as a method of compar-ing handover orientations and quantifying object affordances. This research showsthat depending on the object, natural handover orientations may not be receiver-centered; thus, robots may need to examine the quality of natural handovers beforeusing them as a template for selecting appropriate object orientations.The remainder of this chapter is an adaptation of a manuscript that I co-authored,entitled, “Characterization of Handover Orientations used by Humans for EfficientRobot to Human Handovers” [29]. This manuscript is published in the proceed-ings of the 2015 IEEE/Robotics Society of Japan (RSJ) International Conference onIntelligent Robots and Systems (IROS) held from September 28 to October 2, 2015in Hamburg, Germany. The version of the manuscript that appears in this chaptercontains several minor modifications compared to the submitted conference paper.These modifications are listed below.• Figures have been replaced with high-resolution reproductions for enhancedclarity.• The introduction contained in Section 4.1 has been abridged.• Experimental conditions have been renamed for improved comprehensibil-ity.• Description of the experimental design in Section 4.4 has been expanded toinclude additional details.• Conclusions Section 4.8 have been expanded to address how the work fitsinto the themes of the thesis.Supplementary materials for this work can be found in Appendix B.504.1 IntroductionConsideration of an object’s affordances and how it is handed over betweengiver and receiver has previously been revealed to significantly impact the han-dover interaction experience between people and robots. This work examines thehandover orientation used by people - that is, how the giver of a handover inter-action orients the object when handing it over. Studies have shown that by usingan appropriate handover orientation, a robot can more clearly convey its handoverintent, reduce task completion time, and make the human receiver feel safer [5, 22].Building on these findings of the importance of handover orientation, the work pre-sented in this chapter focuses on methods for determining proper handover orien-tations. Additionally, it assesses the feasibility of training robots to use appropriatehandover orientations based on observations of natural human-human handovers.4.2 BackgroundDetermining appropriate handover orientations is a challenging task, as theproper handover orientation of an object not only depends on the object’s physicalproperties, but also factors such as receiver state [120], object function, [21] andobject affordances. Furthermore, the proper orientation of an object may be non-unique due to multiple possible usages of the object [17]. In some previous studies,the handover orientation is provided to the robot a priori, [e.g., 5], while in others,a few methods for computing handover orientations have been developed. Kim etal. proposed a dual arm planner that uses user-provided object-specific affordanceinformation to compute handover orientations [64]. Cakmak et al. performed asurvey and determined orientations for a cylindrical object that better convey therobot’s handover intent [22]. Later, Cakmak et al. proposed two methods fordetermining handover orientations [21]: one uses user-provided examples of goodand bad handover orientations, and the other plans handover orientations using ahuman kinematic model, while ignoring object affordances.Existing methods often rely on user-provided object-specific information tocapture the object’s affordances. To avoid the need for explicit user training andallow robots to automatically determine appropriate handover orientations, Chanet al. proposed an approach based on observing natural human handovers and51learning the handover orientations used [28]. This approach assumes that naturalhandover orientations used by humans are indeed appropriate. However, the natureof handover orientations used in human handovers is still not very well understoodin a way that they can be applied to robot handovers. Thus, this study investi-gates the nature of the handover orientations used by humans towards developinga relevant mapping to selection of appropriate robotic handover.4.3 ObjectivesThe objectives of this study are to collect a set of handover orientations used byhumans for a set of common objects in different conditions, and determine featuresthat describe handover orientations used in natural handovers, for guiding robot-human handover design. The research questions for this work are:1. What are the handover orientations used by people for handing over differentcommon objects?2. Are there observable patterns in the handover orientations used by people?3. Do people naturally use a handover orientation that is considerate of thereceiver?It is posited that in collaborative scenarios, objects are handed over with theintention of letting the receiver use the object to complete the task at hand. Thus,a giver should use a handover orientation that is considerate of the receiver andallows them to readily take and use the object to ensure efficient collaboration.4.4 Object Handover User Study4.4.1 Experiment DesignA user study was conducted where participants, consisting of giver and receiverdyads, handed over various objects while the motions and orientations of both theparticipants and objects were tracked through a motion capture setup (describedlater in Section 4.4.5). This study aims to uncover what orientations people nat-urally use for handing over the objects, and if the orientations used would differ52depending on the giver’s focus. Thus, three conditions are tested within participantpairs:• Condition A: (Natural handovers) Participants are not given any explicit in-structions on how to handover the objects.• Condition B: (Giver-centered handovers) Focus is placed on the giver byasking the giver to hand the object over in a way that is the easiest and mostconvenient to him/herself.• Condition C: (Receiver-centered handovers) Focus is placed on the receiverby asking the giver to hand the object over in a way that is the most comfort-able and convenient to the receiver, giving consideration to the usage of theobject and function of the different parts.4.4.2 ObjectsTwenty different, common objects that people were expected to hand over on adaily basis were selected to be handed over during the experiment (Figure 4.2). Theobjects that were chosen varied in size, weight, value, affordances, fragility andperceived danger if the object were mishandled (i.e., there would be consequencesif the object was dropped or handed over inappropriately) to represent the varietyof objects that assistive robots may be expected to receive and handle in a home,work or industrial setting. Selecting a range of objects ensures that the set of kine-matic features detected are not specific to a particular object’s specific shape, size,weight, use, value, fragility or other intrinsic features. All of the chosen objectscould be single-handedly transferred from a giver to receiver, and are consideredto be within the payload capabilities of most current robots used for human-robotinteraction - e.g., KUKA LBR IIWA (KUKA Robotics, Augsburg, Bavaria, Ger-many) [70], Willow Garage PR2 (Willow Garage, California, USA), Barrett WAM(Barrett Technology, LLC, Massachusetts, USA) - to make the results of the workimmediately applicable.53Figure 4.2: Common everyday objects that were used in the handover objectorientation user study. Approximate locations of retro-reflective mark-ers on the objects are shown as blue dots. Arbitrary coordinate framesassigned to the objects are also displayed.544.4.3 ParticipantsRecruitment for this experiment was advertised through the laboratory webpage, mailing lists (Section B.1.2), campus bulletin board postings (Section B.1.1),and through word-of-mouth. Twenty participants (9 females, 11 males), aged 19-61 years [M=28.15, SD=11.55] were recruited in total. All participants providedtheir informed consent prior to the experiment using the form shown in Section B.2;they were notified that their participation was voluntary, and they were allowed towithdraw from the experiment at any time. Additionally, permission was obtainedfrom all participants to record both video and motion-capture data from the ex-periment. Participants were rewarded for their time with a candy bar followingcompletion of the experiment.4.4.4 Participant TaskFor this experiment, participants worked in pairs. At the start of a set of trialsone participant was arbitrarily designated the object giver, and the other the objectreceiver in the handover task. Prior to the start of a trial, each participant stoodfacing the other approximately 1.5 m apart. The experimenter placed one of thetwenty objects in a stable position on one of three tables to the rear and sides of thegiver (Figure 4.3); both the placement and orientation of the object was random-ized. At the sound of an experimenter-given verbal ’go’ signal, the giver picked upthe object and handed it to the receiver. For consistency, all participants were askedto use their right hand, regardless of their handedness. Following the handover, theexperimenter retrieved the object from the receiver and asked the participants toreturn to their starting positions. The experimenter, again, placed another objecton one of the three tables and the task was repeated. After performing handoversof all twenty objects, the participants were given an opportunity to take a breakbefore new instructions for the following set of trials were given. The participantsthen swapped giver/receiver roles and performed another set of twenty handovers.On average, the time taken for a pair of participants to complete the tasks was 1.5hours.To measure natural handovers without carryover effects for each pair, the firsttwo sets of twenty handovers were condition A. The remaining four sets varied be-55Figure 4.3: Experimental setup for examining object orientations during han-dovers. Motion capture equipment and garments are not depicted.56tween conditions B and C in counter-balanced order. There were short pauses aftereach set of twenty handovers for explanation of the next condition, and optionalbreaks were given to prevent fatigue; however, none required breaks. A total ofsix sets of twenty handovers were performed per pair of participants, with giverand receiver switching roles after every 20 handovers to cover all three conditions.Thus, 1200 handovers were captured for the data set used in this study.4.4.5 Motion-Capture SystemData for this experiment was collected using a Vicon motion-capture systemMX T-160, (Vicon Motion Systems, Oxford, UK) with eight cameras. Infraredretroreflective markers were placed on the objects (Figure 4.2) and the participants(Figure 4.4) for tracking of key features with respect to an arbitrary global coor-dinate frame. Time-series kinematic data of these markers were captured at a rateof 300 Hz. The Vicon Nexus software (Vicon Motion Systems, Oxford, UK) wasused for computing object orientation and joint angles in post processing.57Figure 4.4: Diagram of retroreflective marker placement and labels on participants. Both markers that were elected foruse in this study and those that were not used are shown in different colours.584.4.6 HypothesesAs most objects have specific affordances (for cutting, pressing, grasping, etc.),it is plausible that if givers present objects in a receiver-centered fashion, theywould orient the object to present the grasping part to the receiver; thus, therewould be observable patterns in the handover orientations used in condition C. Ifgivers present objects in a giver-centered fashion, they may simply use any arbi-trary orientation that is convenient at the moment, or they may grasp an affordancethat is convenient to them, such as the handle. Thus, it is expected that differenceswill arise between the handover orientations used in conditions B and C. Further-more, in natural handovers (condition C), it is predicted that the giver will eitheruse a handover orientation that is convenient to himself/herself or one that is con-venient to the receiver. Stated formally, the hypotheses are as follows:H1 There are observable patterns in the handover orientations used among par-ticipants in receiver-centered handovers (condition C).H2 Handover orientations used in giver-centered handovers (condition B) andreceiver-centered handovers (condition C) are significantly different.H3 Handover orientations used in natural handovers (condition A) are either sim-ilar to those used in giver-centered (condition B) or receiver-centered (con-dition C) handovers.4.5 Data Analyses4.5.1 Handover Orientation ExtractionTo extract the handover orientation, the hand trajectories from the motion cap-ture data were examined. Figure 4.5 shows the typical trajectory of the giver’sand receiver’s hands in a handover trial. In the bottom plot of this figure depictingthe distance between the giver’s and receiver’s hands, a distinct trough is observedwhich is assumed to represent the instant that the object is transferred from giverto receiver. The instant of object transfer was found by determining when this59distance between hands is minimized and the object’s orientation at this time wasdefined as the handover orientation.Figure 4.5: Charts representing typical hand trajectories of giver and receiverin Cartesian coordinates. The instant of object transfer is found by lo-cating the minimum distance between giver’s and receiver’s hands.An arbitrary coordinate frame was assigned for each object (shown in Fig-ure 4.2). To be able to translate the results to different locations in space, thehandover orientations were computed in a base frame defined by the giver and re-ceiver locations. The ground surface normal is treated as the base frame z-axis andthe vector pointing from the receiver’s to the giver’s torso was chosen to be the x-axis; y-axis assignment was automatically completed by enforcing a right-handedcoordinate frame (Figure 4.6).After extracting the handover orientations from all participants, the mean han-dover orientation R¯ was calculated for each object using Equation 4.1:R¯ = argminR∑id(R,Ri) (4.1)where Ri is the handover orientation extracted from participant i for the object,and d(R,Ri) to be the positive value of the angle of rotation between R and Ri asmathematically described in Equation 4.2:60Figure 4.6: Base coordinate frame used during examination of object orien-tations during handovers as defined by giver and receiver locations.d(R,Ri) = arccos( trace(R−1Ri)−12)(4.2)where trace(R−1Ri) is the trace of the matrix R−1Ri. Equation 4.2 essentially con-verts the rotation matrix R−1Ri into the angle component of the axis-angle repre-sentation of the rotation. Thus, Equation 4.1 finds a mean rotation R¯ that minimizesthe sum of rotation angles between R¯ and each measured Ri. The MATLAB (Math-Works, Natick, Massachusetts, USA) built-in fminsearch utility was used tooptimize Equation 4.1.614.5.2 Affordance AxesFrom preliminary inspection of the data, participants tended to display patternsin handover orientation for most objects. For example when handing over the mug,participants tended to keep the mug upright, and when handing over the hammerin condition C, most participants orientated the handle towards the receiver. It ap-peared that participants tended to align a specific axis associated with the objectin the same general direction when handing it over. To help classify this phe-nomenon, the notion of an affordance axis was created. It is hypothesized that formost objects, there is an affordance axis φAff associated to the object, such thatwhen people hand over the object while giving consideration to the object’s affor-dances, they orient the object to align φAff in a certain direction. Intuitively, thiswould be an axis along the handle for the hammer, and an axis oriented perpendic-ular to the bottom surface for the mug. Mathematically, φAff is defined as follows:let φˆAff be a unit vector pointing along φAff. φˆAff is computed using Equation 4.3:φˆAff = argminφˆ∑iarccos(R¯φˆ •Riφˆ) (4.3)In Equation 4.3, R¯φˆ and Riφˆ rotates a unit vector φˆ in the object frame by R¯ andRi respectively to obtain the vector in world frame. If φˆ indeed gives the affordanceaxis of the object, then the vectors R¯φˆ and Riφˆ should align. Thus, φˆAff is foundby finding the φˆ that minimizes the sum of angles between R¯φˆ and each Riφˆ . Theaffordance axes were from the receiver-centered handover orientations extractedfrom condition C, and the MATLAB fminsearch function was used to performthe optimization.4.5.3 Patterns in Handover OrientationsIn Section 4.4.6, it was hypothesized that there are observable patterns in thehandover orientations used in condition C (H1). To find support for this hypothesis,the angles between R¯φˆAff and RiφˆAff for each object, labelled θi, were computedand plotted in histograms. If H1 were false, meaning that there are no observ-able patterns among the handover orientations, and Ri were randomly distributedin rotation space, then RiφˆAff would not be in general alignment with R¯φˆAff, and62the histogram of θi would be distributed across all angles. A Monte Carlo sim-ulation of twenty random handover orientations indeed confirms this expectation(Figure 4.7). Kuiper’s test was used to determine if the distribution of θi for eachobject is different from a uniform distribution.Figure 4.7: Four repetitions of Monte Carlo simulation results showing his-tograms of θi, the angle between R¯φˆAff and RiφˆAff. 20 random handoverorientations were generated for each simulation repetition. A spread ofθi among all angles can be seen.4.5.4 Comparison of Handover Orientations Across ConditionsTo compare handover orientations used across conditions and test hypothesesH2 and H3, differences between handover orientations used were examined foreach condition. Let RUi and RVi be the handover orientations used by participant ifor conditions U and V respectively. The measurement of difference, θUVi , betweenthese two orientations is computed as the angle between the affordance axes inthese orientations as described in Equation 4.4:θUVi = arccos(RUi φˆAff •RVi φˆAff) (4.4)If the handover orientations used in conditions U and V are similar, then themean angular difference, θ¯UV , should be small and less than some δ . Note that δis not equal to zero, due to natural variations in human motion. To estimate δ , theaverage spread in the angle between the affordance axes measured in condition Cwas computed using Equation 4.5:63δ = ∑iarccos(R¯CφˆAff •RCi φˆAff)n(4.5)where n is the total number of participants. To determine if the handover orien-tations between conditions were different, t-tests were performed to determine ifθ¯UV is different from δ . Significant results are reported at an α level of 0.05 withBonferroni correction.4.6 Results4.6.1 Handover Orientation and Affordance AxisTo examine the data, the extracted handover orientations were plotted in 3Dplots. The left chart in Figure 4.8 shows an example from a teapot handover trialin condition C. Referring to the teapot’s assigned coordinate frame shown in Fig-ure 4.2, in this trial, the teapot is handed over upright, with the handle pointedtowards the receiver, slightly to their right. The right chart in Figure 4.8 shows thehandover orientation from all trials. The computed mean orientation R¯ is shown asthe bold, dotted coordinate frame, and the long thin line shows the computed affor-dance axis θˆAff in the mean orientation frame. For visualization and comparisonacross conditions, Figure 4.9 shows the handover orientations from all trials, themean orientation, and affordance axis, for all objects in all conditions.64Figure 4.8: Figure showing orientation of a teapot for one and multiple trialsas it is being handed over from giver to receiver. The left image showsan example teapot handover orientation from condition C for a singletrial. Red, green, and blue lines represent X, Y, and Z axes respectively.In the right image, teapot handover orientations for all trials from condi-tion C. The thick, dotted red, green, and blue lines denotes the computedmean R¯, and the long thin line represents the computed affordance axisφˆAff in mean handover orientation frame R¯.65Figure 4.9: Handover orientations employed during human-human handover for all objects in all conditions. Bold,dotted coordinate frames represent mean orientations and long thin lines represent computed affordance axes.664.6.2 Patterns in Handover OrientationsFor a few objects, it is quite easy to see from Figure 4.9 that there is a clearpattern in the handover orientations. For example, for the mug and teapot, it canbe observed that most trials have a handover orientation where the z-axis of theseobjects are closely aligned with each other. Indeed the computed affordance axesof these objects lie roughly along the z-axis. To facilitate the recognition of suchpatterns in other objects as well, histograms were plotted of θi for each object incondition C in Figure 4.10. Kuiper’s test shows that for all objects, the distributionof θi is significantly different from a uniform distribution (knife: p = 0.008, pen:p = 0.009, all other objects: p < 0.005).67Figure 4.10: Histograms of the angles between R¯φˆAff and RiφˆAff (i.e., θi) for each object in condition C during the studyexamining handover orientations.684.6.3 Comparison of Handover Orientations Across ConditionsTable 4.1 shows the computed means and standard deviations for θ¯AB, θ¯AC, andθ¯BC, as well as the t-test results, with statistically significant results highlighted.The results varied by object. For the book, mug, and plate, comparisons did notyield significant differences in handover orientations between all conditions. Forthe bottle, camera, fork, hammer, pen, remote, scissors, screwdriver, wineglass,and wrench, comparison across all conditions were significant suggesting that adifferent handover orientation was used for each condition. For the remainder ofthe objects, comparisons between condition pairs yielded mixed results.69Table 4.1: Results of comparison of handover orientations between conditions. Mean difference (M), standard devia-tion (SD), p-values (p), and t-statistics (t(19)) are presented. Significant results from t-tests are highlighted.BookBottleCameraCerealBoxUmbrellaFlowersForkHammerKnifeMugPenPlateRemoteScissorsScrewdriverStaplerTeapotTomatoWineglassWrenchδ 22.7 18.3 28.3 35.7 42.4 26.0 27.9 36.7 42.3 11.1 50.1 6.5 18.6 29.8 38.3 47.5 7.9 13.5 11.3 27.4θ¯ABM 34.4 32.6 53.7 65.8 81.9 39.5 52.7 99.1 72.2 19.9 82 13.8 31.6 74.4 78 68.8 11.3 31.2 33.8 88.4SD 41.8 24.5 39.5 42.5 53.4 28.3 39.2 58.5 45.1 28.3 51.2 23 18.2 42.3 60.9 53.0 8.7 22.7 34.8 44.9p .112 .008 .005 .003 .002 .023 .005 .000 .004 .09 .006 .084 .002 .000 .004 .045 .047 .001 .005 .000t(19) 1.25 2.62 2.87 3.17 3.31 2.13 2.83 4.76 2.96 1.39 2.79 1.43 3.21 4.72 2.91 1.79 1.76 3.48 2.88 6.08θ¯ACM 28.1 30.8 46.9 40.4 71.7 42.3 47.5 67.8 54.4 12.6 94.8 8.4 40.8 56.0 58.6 96.7 10.8 20.3 20.9 65.9SD 41.0 21.2 26.4 26.6 52.7 22.5 25.7 57.2 50.3 8.2 50.8 5.5 39.7 42.0 37.8 56.4 7.2 14.0 16.9 54.2p .283 .008 .003 .217 .011 .002 .001 .013 .148 .218 .000 .071 .011 .006 .013 .000 .043 .021 .010 .002t(19) 0.59 2.62 3.15 0.8 2.49 3.23 3.41 2.43 1.07 0.79 3.93 1.53 2.51 2.79 2.4 3.9 1.81 2.18 2.52 3.17θ¯BCM 46.4 35.8 46.3 62.2 65.2 53.0 50.8 95.2 88.3 19.2 90.8 12.5 35.1 74.8 79 71.8 10.7 26.3 32.6 97.8SD 53.1 28.6 33.8 39.5 46.1 31.8 39.8 58.1 61.8 23.8 55.9 21.0 32.1 44.9 58.2 59.6 3.9 20.6 31.6 59.3p .030 .007 .014 .004 .020 .001 .009 .000 .002 .072 .002 .107 .016 .000 .003 .042 .002 .006 .004 .000t(19) 2.0 2.73 2.38 3.0 2.22 3.79 2.57 4.5 3.33 1.52 3.25 1.29 2.31 4.48 3.13 1.82 3.2 2.78 3.02 5.32704.7 Discussion4.7.1 Patterns in Handover OrientationsComparing Figure 4.7 and Figure 4.10, it can be observed that for most objects,the distribution resulting from condition C clearly differs from those generated byrandom orientations given in Section 4.5.3. In particular, the book, bottle, camera,mug, plate, remote, teapot, tomato, and wineglass exhibit a pronounced peak near0◦, with equal to or more than 50% count falling in the smallest bin of θi < 10◦.For the rest of the objects, except the pen, although less pronounced, a skeweddistribution towards 0◦ can still be seen. This suggests that participants do havea preference on aligning the affordance axis of the objects in a certain direction,and thus support is found for the claim in H1. Although as a first step, θi has beentested using the Kuiper’s test to compare with a simple uniform distribution andobtained significant results for all objects, additional investigation is required todetermine whether a uniform distribution indeed represent the distribution of θi inthe random case. Furthermore, rotational symmetry in some of the objects mayalso affect how the random case distribution looks like. Further work is needed todraw stronger conclusions.4.7.2 Comparison of Handover Orientations Across ConditionsT-test results in Table 4.1 for θ¯BC comparing handover orientations betweenconditions B and C show significance for a majority of the objects, including thebottle, camera, cereal box, flowers, fork, hammer, knife, remote, scissors, screw-driver, teapot, tomato, wineglass, and wrench. This suggests that for these objects,the givers use a different handover orientation depending on whether they are fo-cusing on their own comfort (condition B) or on the receiver’s comfort (conditionC). Indeed, with a closer inspection of the data and experiment videos, there areobservable difference in the handover orientations used. For example, with thebottle, participants orient it upright, but with a wide variation in the tilt directionin condition B. In condition C, there is less variation and participants tend to tiltthe bottom towards the receiver. Similarly, with the hammer and the wrench, mostparticipants oriented the hammer head and wrench head towards the receiver in71condition B, but presented the handle in condition C. At this point, it is also worthnoting that the computed mean orientations captures such characteristics quite wellfor these objects as well as others. Thus, it is possible to teach the handover ori-entations extracted from condition C to robot givers, allowing them to performreceiver-centered handovers. Based on these results, support has been for hypoth-esis H2 for the above objects. The handover orientation used by the giver changesdepending on their focus during the handover.Although the statistical tests for the mug were not significant, it was observedthat in more than 50% of the trials within condition B, the giver either grabs ontothe handle or the rim without presenting the handle to the receiver; whereas incondition C, the handle was presented to the receiver in over 75% of the trials.This difference in the handover configuration between conditions B and C is indeedcaptured by the computed mean orientations. The t-test result indicates that theaffordance axis of the mug (computed to be along the z-axis) aligns in the samedirection, meaning that even though the givers were asked to focus on their owncomfort in condition B, they still tended to keep the mug upright. This is likelydue to the associated potential consequence of spilling, if the cup were filled withliquid.Comparing the handover orientations used in condition A with those in condi-tions B and C (Table 4.1 θ¯AB and θ¯AC), results revealed no significant differencebetween the handover orientations used between conditions A and B, and betweenA and C for the book, mug, plate, and teapot. Thus, support for H3 was found,showing that for these objects the handover orientation in condition A is similar tothose in conditions B and C. The mug, plate, and teapot all have the associated riskof spilling if they had contents. Hence, in all conditions, ≥ 85% of the time thegiver kept the object upright. As for the book, people normally read a book fromthe beginning. Thus, the giver seems to naturally orient the front cover upwards inall conditions.For the cereal box, knife, and tomato, t-test results show that handover orien-tations used in condition A were similar to that of condition C, but different fromcondition B. When handing over a knife, there is a possibility of injuring the re-ceiver if an inappropriate orientation is used. Thus, in condition C givers tend toorient the knife tip away from the receiver, and naturally, in condition A, they use a72similar orientation. In condition B, however, when the givers were explicitly askedto focus on their own comfort, they seem to then change the way they orient theknife. A change in handover orientation for the mug and plate was not seen - it ispredicted that this is due to the giver knowing that if the mug or plate were not heldupright, any contents would be sure to spill. However, in the case of handing overthe knife, as long as the receiver exercises extra caution, injury can still be avoided.The results of this study shows that for the bottle, camera, umbrella, fork, ham-mer, remote, scissors, screwdriver, wineglass, and wrench, handover orientationsin condition A were different from those in conditions B and C. For some of theseobjects, namely the hammer and wrench, it appears that this is because the orienta-tions used in conditions A and B are quite widely distributed. This perhaps suggeststhat these objects do not have a strong enough affordance characteristic that wouldprompt givers to hand them over in any particular orientation unless asked explic-itly to consider the object’s function. For some other objects such as the remote, itappears that in condition A, there is a mix of handover orientations from conditionsB and C, suggesting that depending on the individual, some givers naturally use ahandover orientation that is giver-centered, and some, receiver-centered.4.7.3 Implications Towards Building Intelligent RobotsThe end goal of this work is to allow robots to learn receiver-centered han-dover orientations from observing natural handovers. The work presented in thischapter has contributed towards this goal by introducing the idea of the object af-fordance axis, and providing a method for computing mean handover orientations.The study presented here has shown that handover orientations used in naturalhandovers may not all be receiver-centered. Thus robots may need a method fordistinguishing them. In the cases where natural handover orientations differ fromreceiver-centered orientations, there seems to be a wide variation across the han-dover orientations used. Therefore, by identifying the affordance axis and com-puting the mean or variance of θi, the robot may be able to determine whether theobserved natural handover orientations are receiver-centered.This work provides an optimization based method for calculating mean han-dover orientations. This method can be used by robots to compute the proper han-73dover orientation from a set of observed handover orientations. Using this method,computation of handover orientations for a set of common objects in differentconditions was performed. The mean orientations calculated from the receiver-centered handovers could potentially be used by robots when handing over theobjects to people to facilitate more efficient and socially acceptable interaction.4.8 ConclusionsThe investigations presented within this chapter provides building blocks to-wards allowing robots to interpret human nonverbal cues to infer object attributes,i.e., part of the second research question of this thesis. Specifically, the aim isto have robots be able to orient handover objects appropriately (in terms of theobjects’ affordances) to human receivers. This work adopts a scheme of havingrobots trained on the appropriateness of handover orientations via observations ofhuman-to-human handovers. To this end, a study was conducted with the intentionof determining the suitability of using human-to-human handover examples to trainrobot givers on how to appropriately orient handover objects based on the objects’affordances, while simultaneously developing the necessary tools to enable robotsto learn how to appropriately orient objects during robot-to-human handovers usingthese observations.Towards enabling robots to learn proper handover orientations from observa-tions, this work introduces the idea of object affordance axes, and an optimizationbased method from computing mean handover orientations. Through a user study,handover orientations used by humans were surveyed for a set of objects in dif-ferent conditions. Mean orientations were computed from the data, and extractedmean orientations from receiver-centered handovers could potentially be used toteach robots how to appropriately hand over these objects to people to promotemore effective cooperation. Through comparison of three structured handover con-ditions, natural handover orientations were found to differ from receiver-centeredorientations for some objects. Thus when a robot tries to learn from natural han-dovers, it may need to consider the quality of the observed handover orientations.Measurement of the variation in the observed handover orientations may providesome kind of a quality measurement, as objects that have different natural and74receiver-centered handover orientations appear to exhibit larger variance in the nat-ural handover orientations.In relation to the second research question of this thesis, the results of this worksuggests that the aim of having robots observe, learn, and/or mimic behavioursfrom human nonverbal cues may be confounded by variation in these observations.Factors such as personal preferences, environmental configuration and cultural as-pects will, with no doubt, impact how humans carry out handovers with respect tothe nonverbal cues that are used. Thus, having a robot learn by example may re-quire a constrained or artificial data set - e.g., only learning handover orientationsusing the receiver-centered handover data set. Alternatively, as previously men-tioned, the quality of the dataset may need to be assessed to sort good examplesfrom sub-par ones. Future work includes running user studies to confirm that theextracted receiver-centered mean orientations indeed improve robot-human han-dovers and facilitate better interaction, as well as devising methods for identifyingreceiver-centered handover orientations from natural handovers.Continuing the theme of recognizing and interpreting nonverbal cues conveyedby a human, the following chapter examines how a robot receiver can learn to detecta human giver’s intent to handover an object. In that work, a machine learningapproach is used to both automatically identify important kinematic cues indicatinghandover intent as well as detect when a human giver intends to handover an objectto a receiver. The work serves as a first step or building block in allowing a robotto receive an object from a human counterpart.75Chapter 5Automated Detection ofHandovers using KinematicFeaturesPreviously, Chapters 3 and 4 have examined how nonverbal cues and properhandover object orientation may improve aspects of a robot-to-human handover.These works have examined and discussed how a robot giver could leverage ob-ject affordances and gaze cues when handing over an object to a human receiverto make the experience more fluent and efficient. The rest of the works containedwithin this thesis will study the role of nonverbal cues in handover interactionswhere the roles of the human and robot are opposite to that examined in the pre-vious two chapters; that is, where the human acts as the giver and the robot asreceiver (human-to-robot handover).In establishing what prior work has been done in the area of human-to-robothandovers, there appears to be little on the subject; rather, much of the literatureon handovers between human and robot agents explore robot-to-human handovers.Thus, having somewhat of a clean slate to work with, the research described inthe next few chapters attempts to establish the basics of the interaction while alsoprobing the available design space.This chapter continues the thread picked up in Chapter 4 and focuses on hav-ing robots recognize and interpret nonverbal cues generated by a human during76handovers - i.e., the second research question posed in this thesis. Here, one ofthe first steps in what a robot needs to do when receiving an object from a personis examined, namely, recognizing the intent of that person to handover an object.To achieve automated detection of handovers, a machine learning approach is uti-lized for recognizing handover intention from kinematic motions of the humangiver. Through a structured procedure of selecting 22 kinematic features based onpredictive value for handover intent, machine learning models were able to be de-veloped which could detect handover motions with 97.5% accuracy. Given that amotion capture system was used for collecting kinematic data from human-humandyads - a technique that would most likely not be available for robots in the field- a pilot study conducted using a more typical sensor used in unstructured envi-ronments (the Microsoft Kinect version 2) was also performed. The results areencouraging, demonstrating considerable potential and feasibility of this methodfor detecting handover intent (and perhaps other gestures for human-robot interac-tion) using kinematic features.This work presented here has been published in The International Journal ofRobotics Research (IJRR) volume 36, issue 5-7, a special issue on HRI in 2017. Ico-authored the original manuscript, entitled, “Automated Detection of Handoversusing Kinematic Features” [96]. The data and results which appeared in this workwere derived from the same data used in the previous chapter (Chapter 4). Severalchanges to this manuscript were made for it to be included in this thesis whichinclude:• Figures have been replaced with high-resolution reproductions for enhancedclarity.• The introduction of the manuscript (Section 5.1) has been abridged.• Background information that has already presented in Chapter 2 has beenremoved.• Details of the experimental setup of this work is the same as that whichappears in the previous chapter (see Section 4.4) and has been removed.Supplementary materials for this work can be found in Appendix B.775.1 IntroductionWhile seemingly effortless for most humans, recognizing a handover motionand receiving the object in an appropriate manner from a giver is a challenging taskfor a robot assistant. A key skill enabler is the ability to detect nonverbal cues fromthe giver in order to infer the timing and location of the handover.This chapter presents work which was done to develop building blocks to-wards enabling robots to robustly and naturally receive an object from a humangiver: namely, to detect when a human giver initiates a handover of an objectthrough recognition of nonverbal kinematic cues. This work is predicated on the re-cent availability of inexpensive, off-the-shelf sensors such as the Microsoft Kinect(Microsoft, Redmond, Washington, USA), providing ready access to human bodykinematic data from which cues relating to handovers and other interactions canbe detected. In essence, the aim of this work is to examine the feasibility of usingdata from such devices for this purpose through an exploratory study.This work is a departure from prior work in that it attempts to recognize thegiver’s intent to hand over objects through machine learning of kinematic data ob-tained from the giver rather than performing recognition on image or manuallycoded data. The advantage of this method is that it allows online handover detec-tion using off-the-shelf depth sensors that are able to perform skeleton tracking.With this method of detection, a robot would be able to differentiate between han-dovers aimed towards it as the intended receiver and those directed at other po-tential receivers. Furthermore, a robot would not need to rely on verbal cues orthe observation of an object in the giver’s hand to detect that a handover is takingplace.In the following section, prior research is reviewed in the areas of nonverbalcues, proxemics, kinematics, and the use of machine learning in recognizing han-dovers (Section 5.2). This is followed by a description of the objectives and ap-proach of this study (Section 5.3), experimental methods used to collect the data(Section 5.4) and how the data was analyzed (Section 5.5). An overview of the re-sults is then presented (Section 5.6) followed by a discussion of the impact of theseresults (Section 5.7). The chapter is then concluded by discussing the implications,limitations, and areas for future examination of this work (Section 5.8).785.2 Prior WorkThe work presented in the paper is derived from several fields of research: non-verbal cues, proxemics, and kinematics in handover tasks, and machine learning.An overview of prior work in these areas is presented here.5.2.1 Non-Verbal Cues, Proxemics and Kinematics in HandoversPrior works have demonstrated that nonverbal cues play a significant role incoordinating handovers between human participants [43]. Recent studies investi-gating nonverbal gestures involved in handovers have examined more subtle cuessuch as gaze and eye contact. This work demonstrated that where participantslook during handover plays a significant part in the coordination of handovers[83, 88, 112, 113, 123]. Work on social gaze for robots by Mutlu et al. providessome guidance on how gaze can be used for such coordination [92]. In addition togaze, grip and load forces have been shown by Chan et al. and Kim and Inooka toplay a significant role in the coordination of handovers, and have led to the observa-tion that the giver seems to be responsible for the safety and successful handover ofthe object, while the receiver is responsible for the timing of the handover [26, 63].Much of the early work in robot-to-human handovers relied on proxemics stud-ies by Hall, whose work suggested that distances between persons interacting so-cially are influenced by culture, attitudes, social standing, and relationships to oneanother [45]. A later study showed that interpersonal distance between humangivers and receivers at the moment an object is handed over varied considerably,with the average distance being 1.16 m away - roughly the distance where bothgiver and receiver must have arms outstretched during handover [13]. To comparethis to proxemics between robots and humans, Walters et al. found that most oftheir participants allowed a robot to approach to within a personal distance (0.6 to1.25 m), suggesting that humans may treat robots similar to humans socially [121].However, 40% of their participants allowed a robot to approach to within an inti-mate distance of 0.5 m, implying that humans are more tolerant of robotic closeencounters, which would be perceived as over-familiar or threatening in a similarhuman-human context.79In order to better understand the spatial behaviours of participants within human-human handovers, several studies have considered where handovers occur in thespatial domain [13, 54], and the joint/limb dynamics of both the giver and receiverduring handover [41, 60, 106]. In Sisbot and Alami’s work, the authors use kine-matic features, along with preferences and gaze of the human receiver, to help arobot giver plan trajectories to navigate to a handover location safely and in a so-cially comfortable manner [106]. Kajikawa et al. have determined that handoversbetween humans share several common kinematic characteristics such as a rapidincrease in the giver’s arm velocity at the start of the handover [60]. They also ob-served the occurrence of a delay in handover reaches, as the receiver begins theirreach for the object only after the giver achieves a maximum approach velocity.Basili et al. quantified this delay by observing that nonverbal indicators that a han-dover was to occur happened approximately 1.2 s prior to the occurrence of theactual handover [13].These observations are particularly important for the development of a han-dover controller as robots may be able to anticipate rather than react to handovers,leading to more fluent interactions [50]. That is, if nonverbal cues indicating anobject is being passed can be observed early enough, a robot may be able to actquickly enough to allow the handover to proceed seamlessly and fluently from theperspective of a human giver.5.2.2 Machine Learning for HandoversThe use of machine learning for classifying kinematic data has been a recentarea of interest. Machine-learning approaches have been successful in recognizinggait patterns [14, 46, 73], physical gestures [34, 51, 52] and other day-to-day physi-cal activities [80, 93]. For example, Takano et al. presented a hidden Markov modelbased method for learning and recognizing interactive behaviours from motion-capture data used to generate interactive robot behaviour [114]. More recently,Gaussian mixture models have become more common in human gesture recogni-tion tasks [99, 109, 110], achieving varying degrees of classification accuracy from70% up to 90%.80A few studies have utilized database searching and machine learning to assist inobject handovers between robots and humans. Yamane et al. developed a databaseof human-human handover motions used to synthesize robot receiving motions[122]. Online performance of searching depended on the number of features beingtracked and the length of observation window. In a parallel vein of work, Strabalaet al. explored the use of machine-learning classifiers in recognizing the intent ofa giver to hand over an object [112]. In their work, sequenced features such asorientation of the giver with respect to the receiver, eye gaze location, giver handoccupancy (holding or not holding an object), handover signals (e.g., extension ofthe arm), and inter-subject distances were used to recognize the intent of a humangiver to hand over an object to a human receiver. These features were manuallycoded from video recordings.The approach taken in this work differs from prior work in several ways. Inthis work, machine learning is applied to automatically select features taken fromeach frame of a set of time-series kinematic data. Those features that are expectedto offer the most utility in detecting giver-initiated handovers form the input spacefor a machine learning classifier to make independent predictions at each time step.In contrast with how handover motions are recognized by Yamane et al. in [122],no search of a database or pattern recognition over a sequence of frames is requiredby the method to perform the detection. Additionally, unlike the work of Strabalaet al. [112], the approach of this work is agnostic to whether or not the giver’shand is occupied with an object. This is greatly beneficial as object sensing andidentification is a challenging task: unless the object is known to the vision systemand is well-behaved (i.e., does not deform or change properties such as colourand reflectance), vision systems typically have difficulty in detecting an object,especially if it is partially or fully occluded by the giver’s hand. In Chapter 4, suchissues with object identification and tracking were avoided as a motion capturesystem was used: objects could be robustly and uniquely tracked as long as threeor more retroreflective markers of a constellation for each object were visible to anarray of cameras. However, using motion capture systems to track handover objectsin the field is not feasible, particularly if there are a large number of objects, sinceevery object to be tracked will need to be fitted with a unique pattern of intrusivemarkers.81The approach proposed by this work avoids these issues by observing thegiver’s behaviours rather than the handover object. However, an obvious limitationof this approach is that the classifier may interpret non-handover gestures as han-dovers (e.g., handshakes and fist-bumps). There are multiple strategies to mitigatethis limitation by coupling the system with other sub-systems which can recognizeand recall an object pick-up gesture by a person, or use a vision system to detect theappearance/disappearance of objects to calculate probabilities that a potential giverpicked up an object to be handed over. Investigation of these systems, however, arebeyond the scope of this work.5.3 Objectives and ApproachThe objectives of this study were to collect a set of sample kinematic dataused by givers during the handover of various common everyday objects, and todetermine a set of features that can be used to detect the occurrence/initiation of ahandover. The research questions of this work are:1. What is the set of kinematic features of givers that could be used to detectthe initiation of a handover (from the receiver’s perspective)?2. How well can these features be used by a machine-learning approach to de-tect the occurrence of a handover? In particular, how does the performanceof this approach compare with approaches used in prior work [e.g., 112]?In this context, accuracy, sensitivity, specificity, and precision as defined as follows:accuracy is the proportion of correct binary handover/non-handover motion classi-fications among all observations examined. Sensitivity (also known as true positiverate) is the ability of the classifier to correctly classify data indicative of handoverintent. Specificity (also known as true negative rate) is the ability of the classifier tocorrectly classify data indicating lack of handover intent (non-handover). Finally,precision (also known as positive predictive value) is the proportion of all correctlyclassified data indicative of handover intent among correct and incorrectly classi-fied data indicating handover intent.To achieve learning of handover cues from kinematic data, the use of SVMsis proposed. SVMs are recognized as a robust method in pattern recognition and82classification; they have been applied to numerous classification and regressionproblems with exceptionally good performance [15]. There are several advantagesto using SVMs for classification tasks including the exploitation of the kernel trick(being able to use non-linear functions for classification efficiently) and inclusionof regularization and a large margin for better generalization. Their robustness inbinary classification and previous success in other work classifying kinematic data- gait in particular [14, 39, 73] - makes SVMs a good fit for this work. Hence, SVMsis applied in the study reported here for automated recognition of when a handoveroccurs using human body kinematics.5.4 Experimental SetupThe data used for this study was collected within the same experiment pre-sented in Chapter 4 on the orientations of objects during handovers. Thus, expla-nations and details regarding the experimental setup can be found in Section 4.4and will not be repeated here. For this study, only motion data of the markers lo-cated on the giver in each trial (see Figure 4.4) was analyzed; motions of the objectand receiver were not examined.5.5 Data Analysis5.5.1 Data Post-ProcessingPost-processing of the motion-capture data for participants’ motions duringthe experiment was done using Vicon Nexus software version 1.8.5 (Vicon MotionSystems, Oxford, UK) and MATLAB (MathWorks, Natick, Massachusetts, USA).Markers were automatically identified within the data by performing model match-ing against template skeleton and object marker models. Gaps in marker data whereone or more markers temporarily disappeared from the view of the cameras (usu-ally caused by occlusion of the marker) were semi-manually filled using splines orkinematic information from other markers.As a large amount of data was obtained (˜2 million observations), the kine-matic data recordings were downsampled from 300 Hz to 30 Hz to allow for fasterdata processing and manipulation. This reduction in sampling rate also conve-83niently allows the data capture rate to be roughly the same as with off-the-shelfmotion trackers such as the PrimeSense 3D Sensor (PrimeSense, TelAviv, Israel);Intel RealSense (Intel, Santa Clara, California, USA); Kinect Version 2 (Microsoft,Redmond, Washington, USA). To reduce noise in marker trajectories, the data waspassed through a fourth-order low-pass Butterworth filter with a 6 Hz cut-off fre-quency (the approximate frequency limit of human limb motion).To limit redundancy in the data, a number of markers were ignored. These in-cluded markers on the head except for the forehead (FHead) marker, any markerlocated mid-segment between two joints (e.g., markers located on the collarbone,upper arm and forearm), the Back marker that would be occluded from the perspec-tive of the receiver during a handover, and those markers that were felt to be fairlyinvariant with respect to other included markers (e.g., LChest, RChest, LWrist, andRWrist).Since the end goal of this work is for a robot be able to recognize that it is at thereceiving end of a handover, the data was required to represent motions from theperspective of the receiver. This was achieved by remapping all of the data fromthe arbitrary global coordinate frame of the motion capture system into an estab-lished receiver-based frame. It was determined that the origin of this frame wasto be located halfway between the initial positions of the back and chest markersof the receiver. The X axis vector points towards the receiver’s right, the Y axispoints forward towards the chest marker, and the Z direction points up (Figure 5.1).It should be mentioned that although the data processing performed here is accom-plished offline or semi-manually (e.g., the filling in of missing marker data), all ofthis processing can be performed online as well (as shown in Section 5.7.3).5.5.2 Feature ExtractionOnce the giver’s kinematic data had been extracted, filtered and transformed,MATLAB scripts were developed for the computation of several classes of fea-tures. These features are categorized in Table 5.1. In total, 176 unique featureswere generated for consideration at each time step. All features considered hereconsist of filtered data from the motion-capture system, or are procedurally gener-ated based solely upon this data (i.e., features that are not directly drawn from the84Figure 5.1: Diagram showing giver passing a book to the receiver (top) andcorresponding motion-captured scene (bottom). The coordinate systemseen superimposed on the receiver motion-capture model represents theone used for calculating features.85motion-capture system are generated by performing calculations upon the filteredmotion-capture data). No manual coding of features is performed and all featurescan be generated online, significantly increasing the viability and extendibility ofthe method to the online detection of handovers. Thus, these features can be gen-erated for any skeleton model containing similar markers - such as the shoulder,neck, hands, chest and head - that current sensing packages such as the MicrosoftKinect and Intel RealSense track directly out-of-the-box.Table 5.1: Features extracted from kinematic data obtained from the motioncapture system during the study investigating the automated detection ofhandovers.Measure DescriptionDistance Euclidean and axis component distances betweentwo markersReceiver-based kinematics Kinematics examining how the vector betweena specified marker and the receiver frame originchanges (created to facilitate assignment of direc-tionality to magnitude kinematic measures)Linear Kinematics Magnitude and axis-component kinematic mea-sures (e.g., velocity in the Y-axis and accelerationin the Z-axis)Angular Kinematics Joint angular features considered with respect tothe simple angle between two plane-defining vec-tors that form a joint in both simple angle andin quaternion form, and segment angular featurescomputed between limb segments and basic coor-dinate planes5.5.3 LabellingTo distinguish between handover and non-handover motions, the start of a han-dover was defined as the moment that the giver begins moving his/her hand topresent the handover object to the receiver; the end of a handover was defined asthe moment when the receiver has grasped the handover object. Using these defi-86nitions, observations were manually labelled in each trial between the start and endof a handover as positive (1); and those before or after a handover as negative (0).Thus, independent predictions can be made for each time step.Since an objective of the classifier is to distinguish handovers from all otherhuman motions, it is imperative that the negative (non-handover motion) data en-compass a wide range of behaviours - not just those occurring before and after ahandover. Thus, supplementary negative data was collected to introduce more di-versity in the negative dataset to improve the generalization ability of the classifier.Giver motions in this dataset include more common actions such as walking pastthe receiver in a variety of directions and moving into sitting and standing posi-tions, as well as uncommon actions such as performing jumping jacks and danceroutines.5.5.4 Test Set GenerationFrom the data set collected, the recordings from two participant pairs were re-served as a test set partition to be used to test the generalization of the trained SVMsand some of the supplementary negative data. The test set included approximately25% of the total data. The rest of the data formed a training set used to train theSVMs.5.5.5 Predictor Feature SelectionOne of the disadvantages of using a supervised learning technique, such asthe SVM, is the requirement to determine which features are relevant in producinga good recognition model. Often, in previous work, selection of features is per-formed in an ad hoc fashion [e.g., 112]. In contrast, this process is conducted sys-tematically in the work presented here. To eliminate features that have a low pre-dictive value in classifying positive and negative data (simplifying the model),anensemble learning technique known as Breiman’s random forest algorithm [19]was used. The random forest algorithm is a well-known method for estimatingpredictor importances, that is, how well a single feature is able to discriminate be-tween the target classes, reducing high-dimensional feature spaces into a core set ofexcellent predictors while also eliminating all irrelevant or redundant (correlated)87features [31]. Thus, this algorithm enables the ranking of the predictive ability ofthe 176 obtained features and provides insight into the importance of features fordetecting handovers.Random forests typically perform better in environments where learning mod-els are diverse [71]. Bootstrap aggregating, or bagging, was used to grow the ran-dom forest as a method for increasing diversity between trees; i.e., weak learnerswere bootstrapped through random resampling of the data. Performance of baggedrandom forests relies on several parameters; two of the more influential ones be-ing the size of the bootstrap samples, also known as the sampling fraction, and thenumber of weak learners in the forest. The sampling fraction is a critical parameterfor the injection of diversity into weak learners, and the number of weak learnersin the forest affects both computational efficiency and the robustness of the forest.The effect of both of these parameters can be measured using three error metrics:the generalization error, out-of-bag error and resubstitution loss. The generaliza-tion error is a measure of how well an algorithm performs on a hold-out partitionreserved from the training data. The out-of-bag error is similar to the generalizationerror, but uses samples not seen in training as an independent test set. That is, forevery example vector (xi,yi) in the training data D, select all bootstrapped subsetsD j that do not include (xi,yi). Then, the example (xi,yi) can be passed through alltrees Tj trained on D j for an aggregate prediction of that observation. This is car-ried out for all test observations and compared against the assigned labels. Lastly,resubstitution loss reflects the would-be test performance had the training set beenthe test set.The effect of the sampling fraction on these error metrics was investigated bytraining multiple forests with 100 learners grown using varying sampling ratios.Similarly, the effect of the number of learners on the same measures were examinedby training multiple forests of different sizes. Results using both Gini’s diversityindex and information gain (also known as cross-entropy or maximum deviancereduction) split criterion were obtained. From Figures 5.2 and 5.3, it is apparentthat all of the error metrics generally decrease with increased sampling fraction,and are minimally affected past 100 learners.Based on these results, 500 weak learners were used to provide a sizable safetymargin against any potential variations in the final model that could alter error88Figure 5.2: Charts depicting resubstitution loss, generalization error and out-of-bag error for Gini (left) and entropy/information gain (right) splitcriterion as a function of bootstrap sampling fraction.Figure 5.3: [Charts depicting resubstitution loss, generalization error andout-of-bag error for Gini (left) and entropy/information gain (right) splitcriterion as a function of number of learners.89rates. Each learner/tree was bootstrapped through random resampling of the datawith replacement where the size of each bootstrap dataset was the same as theoriginal, sampled dataset (sampling fraction = 1). A random subspace of featureswas chosen for each tree, the dimensionality of which was set to Breiman’s recom-mendation of the square root of the total feature space:√p =√176 u 11. Gini’sdiversity index and information gain were both used as split criterion having threeensemble models trained for each.Feature importance estimates were calculated by summing weighted decreasesin Gini impurity as a result of splits on each feature and dividing by the numberof branch nodes. Each ensemble produced a set of rankings of all features basedon these importances, and the rankings for each feature was averaged amongst thesix ensembles. The top 20 features were selected to form the input space for SVMtraining, not only to improve computational efficiency both in training and hyper-parameter optimization, but also in online computation of features and predictionof new samples. In addition to these 20 base features, any right-hand positional(non-derivative) features not in the top 20 were included as well. Inclusion of theright-hand position features ensures that the classifier can still identify the han-dover event late if identification fails during the handover motion. In other words,the terminal stages of the handover, in which the giver’s arm is outstretched andidle, must be represented in the training data.5.5.6 Hyperparameter OptimizationFour kernels were used for the SVMs: linear, quadratic, cubic and Radial BasisFunction (RBF). The two tunable hyperparameters for each model are the regu-larization or penalty parameter, C, and the kernel scale, γ . The hyperparameter Coperates as an adjustable tuning to control the degree of underfitting/overfitting ofthe data. For large values of C, the optimization will choose a smaller-margin hy-perplane in an attempt to correctly classify all training points. Conversely, a verysmall value of C will cause the optimizer to look for a larger-margin separatinghyperplane, even if that hyperplane misclassifies more points. The kernel scale hy-perparameter controls the flexibility of the decision boundary. For example, withthe RBF kernel, a larger γ leads to a decision boundary that forms a tight perimeter90around a region where positive training data is clustered. The optimization objec-tive is to find a hyperparameter pair that minimizes the cross-validation error.Searching the parameter space requires the classifier to be trained using a time-intensive procedure at every iteration. Normally, a robust k-fold procedure is usedfor cross-validation optimization of the hyperparameters. However, if k-fold vali-dation were used with, say, 10 folds, the classifier would need to be trained 10 timesper iteration. Given that the training dataset has approximately 200,000 observa-tions, a k-fold approach was deemed infeasible due to a lengthy execution time.Instead, a holdout cross-validation approach was used, while keeping in mind thatthe optimization results would not be as robust as in the case of using a k-foldscheme [18]. A holdout set randomly sampled with 90:10 split from the total train-ing data was reserved prior to each optimization. Due to the size of the trainingdata, a 10% subset was deemed to be sufficiently large to capture the overall di-versity of motions to be classified. Once optimal hyperparameters were obtained,five-fold cross-validation was carried out as a more robust evaluation of whetheror not the optimization had indeed converged on an error minimum. Further re-duction in computation time was achieved through the use of MATLAB’s built-inheuristic for automatically determining the kernel scale, effectively reducing theoptimization problem into a univariate search.For all of the kernels explored, it was assumed that the surface of the holdouterror objective function was likely to be non-smooth and non-convex. Normally,a numerical method that could optimize non-linear problems such as Nelder-Meadwould be used. However, due to the infeasible execution time it took to run the op-timization on the large data set collected in this study, a hybrid of Golden SectionSearch (GSS) and Successive Parabolic Interpolation (SPI) was used instead. Asthis optimization method requires a unimodal objective function, the search wascarried out over bracketed intervals of the search space to reduce the in-bracketfunction variance and capture potential convexity. The regularization parame-ter, C, was initially investigated over an initial range of (0,20]. This range wassplit into five intervals, with a minima search executed within each interval inde-pendently. Using this approach, five local minima were identified overall. Theglobal minimum, or lowest function evaluation found within the search bracket,was taken as the kernels’ optimal hyperparameters. Significant downwards error91trends within the range [0.001,20] might suggest better minima exist and wouldnecessitate searching beyond C = 20.5.6 Results5.6.1 Training and Holdout DataTable 5.2 shows the specifications of the resulting training and testing data(holdout partition). As mentioned in Section 5.5.4, approximately 75% of the totalobservations were used as training data and the remaining 25% were held out fortesting. An approximate 50% split was maintained between positively and nega-tively labelled observations for both training and testing data sets.Table 5.2: Training and testing data specifications for the automated detectionof handovers. The total observations made, how many of those observa-tions were labelled positively (indicating handover intent) and negatively(indicating non-handover), and respective percentages of total data col-lected are shown.Data set Observations ’+’ observations ’+’(%)’-’ observations ’-’(%)Trainingdata205,365 93,407 45.5 111,958 54.5Testingdata50,845 27,722 54.5 23,123 45.5Total 256,210 121,129 47.3 135,081 Predictor Feature SelectionThe set of features chosen by the random forest method to form the input spacefor SVM training is shown in Table 5.3 in order of importance ranking. As ex-pected from indications in prior work, right hand-, chest-, and forehead-relatedfeatures score very highly. Y-axis kinematic measures are strongly represented, asare angular measures projected onto the XZ and XY planes. The receiver-based92kinematics class of features (e.g., velocity of the giver’s chest along the vector be-tween the giver’s chest marker and the receiver frame origin) perform exceptionallywell as predictive features, holding three of the top six places (ranks 1, 4 and 6) inability to predictively classify data.93Table 5.3: The 22 features selected for use as the SVM input space to automatically detect handovers and their predictiveability rankings. The last two features (23 and 29) were not ranked in the top 20, but were included as part of theinput space since they describe position of the right hand (the hand that was used to hand over objects in thisexperiment), such that the classifier can still identify the handover event later if identification fails during thehandover motion.Rank Measure Marker(s) Feature Abbreviation1 Receiver-Based Kinematics RHand Velocity RB RHand V2 Linear Kinematics Chest Y- Component Velocity LK Chest Vy3 Linear Kinematics RHand Y- Component Velocity LK RHand Vy4 Receiver-Based Kinematics Chest Velocity RB Chest V5 Linear Kinematics FHead Y - Component Velocity LK FHead Vy6 Receiver-Based Kinematics FHead Velocity RB FHead V7 Linear Kinematics RHand Z - Component Velocity LK RHand Vz8 Angular Kinematics RHand- RElbow Angle w.r.t. XY Plane AK RHand-RElbow xy9 Linear Kinematics FHead Z - Component Velocity LK FHead Vz10 Angular Kinematics RHand- RElbow Angle w.r.t. XZ Plane AK RHand-RElbow xz11 Distance RHand-RShoulder Z - Component Distance DI RHand-Rshoulder z12 Distance RHand- Neck Z - Component Distance DI RHand-Neck z13 Distance RHand- RShoulder Y - Component Distance DI RHand RShoulder y14 Linear Kinematics Chest Z - Component Velocity LK Chest Vz15 Angular Kinematics RElbow- RShoulder Angle w.r.t. XZ Plane AK RElbow RShoulder xz16 Distance RHand- Neck Y- Component Distance DI RHand Neck y17 Angular Kinematics LShoulder- Neck Angular Velocity w.r.t. XZ Plane AK LShoulder Neck Vxz18 Angular Kinematics RElbow- RShoulder Angle w.r.t. XY Plane AK RElbow-RShoulder xy19 Linear Kinematics RHand Z- Component Position LK RHand Pz20 Angular Kinematics RElbow- RShoulder Angular Velocity w.r.t. XZ Plane AK RElbow Rshoulder Vxz23 Linear Kinematics RHand Position Magnitude LK Rhand Pmag29 Linear Kinematics RHand Y- Component Position LK RHand Vy94Axis component velocities and segment angles also score well. Four measuresof distance between markers appear in the top 20, which relate the distance of theright hand to the neck base and right shoulder markers in the Y-(receiver forward)and Z- (vertical) axes, signifying that arm reach plays an important part in handovermotions.Overall, importance rankings, as seen in Figure 5.4, take a somewhat logarith-mic shape. This suggests that the top end features are an excellent set of predictors,whereas the importance of other predictors rapidly deteriorates down the ranking.It would therefore offer little benefit to train on a larger number of features, es-pecially considering the greater computational expense that would be incurred inso doing. The discriminatory ability of these features can be seen using a parallelcoordinate plot (Figure 5.5). Here, the vertical axis represents the normalized co-ordinate value of observations for each feature listed along the horizontal axis. Thefeatures are listed in rank order from left to right along the horizontal axis. Theplotted heavy lines link the average coordinate values for handover/non-handoverfor each feature, and the thin lines represent the quartile ranges surrounding theaverages. These lines form threads that illustrate the ability of the features to sep-arate positive and negative labelled data into disparate clusters. Using this format,strong clustering can be observed in the first six features where the quartile rangesare entirely separated. Proceeding to the right of the plot, features that are rankedlower demonstrate, as expected, less separation. This can be seen as further con-firmation of the success of the bagged random forest approach for feature rankingfor this problem.95Figure 5.4: Plot of the SVM feature space obtained from the bagged random forest arranged by importance scores inhighest to lowest order. The horizontal black line delineates the top 20 features chosen to be included in the SVMfeature space. Bars of similar colour represent features of the same type of measure.96Figure 5.5: Parallel coordinates plot of the input space determined by using a bagged random forest with 500 learnersand sampling fraction = 1.0. The top 20 features are included, along with the two RHand positional features notfound in the top 20 but which are still included in the SVM input space. The thick blue dotted line represents thenormalized mean value of the observations labelled negative (non-handover) within each feature space. Similarly,the thick green solid line represents the normalized mean value of the observations labelled positive (handover)within each feature space. Quartiles are shaded.975.6.3 Hyperparameter OptimizationFrom Figure 5.6, the algorithm can be seen to converge on approximately con-vex regions of the search space. For the linear kernel optimization, there alsoappears to be instances of SPI degeneracy. In these cases, the algorithm has beenable to proceed anyway, which may be due to the hybridization of the SPI and GSSalgorithms. The search space plots also reveal that, as correctly predicted, all ob-jective functions are non-smooth and, in fact, very noisy. For quadratic and cubickernels, relatively large variations in cross-validation error exist between minute,fractional variations in the regularization parameter, C. This might suggest that theoptimization would have benefited from a larger holdout set. However, the cross-validation error variations are relatively small as, overall, the objective function isextremely invariant (<1% holdout error) over the search space. Furthermore, therange of holdout error is far smaller than expected, suggesting a high accuracy andan exceptional level of separability. One might attribute the accuracy to overfitting;however, it is unlikely that this is the case as function evaluations are performed atC <20 for all models. It is possible that small adjustments in C are leading to mis-classification of only a handful of points, which, at this high level of accuracy, issufficient to cause the apparent noisiness of the objective function. Regardless, theGSS/SPI hybrid optimization method has honed in on minima within each bracket.Due to the invariance of the objective function, searching at C >20 is unneces-sary. While this invariance does decrease with increasing kernel complexity, eventhe RBF kernel’s error does not vary with more than a fraction of a percent. Further-more, the maximum holdout accuracies of all models are all quite low. Satisfactoryresults have been found within the initial range, and the models will be more proneto overfitting at higher values.Table 5.4 shows the values of the optimized hyperparameters for each kernel aswell as the cross-validation errors for both holdout and five-fold procedures. Thelinear kernel’s global minimum is found in the first bracket, (0,4], at C = 1.624.At this low level of regularization, the model will not penalize misclassificationof the training data as strongly. The resultant low model complexity may lenditself to the overall generalisability of the final classifier. The quadratic, cubic andRBF kernels’ global minimum are found in the last bracket, [16,20], at C = 17.528,98Figure 5.6: Kernel search spaces for optimization of the regularization pa-rameter (C) for the handover classifying SVM. (a) Linear kernel; (b)quadratic kernel; (c) cubic kernel; (d) RBF kernel. The objective func-tions all appear to yield results that are localized to small holdout errorand relatively invariant over the search range, invalidating the need tosearch beyond C >20.9917.655, and 19.416 respectively. This means that in comparison to the linear kernel,the remaining kernels will have a significantly higher level of regularization andmodel complexity. Nonetheless, with the knowledge of high holdout performanceat these relatively low C values, it is apparent that there is an intuitive resistanceto overfitting. Hence, although the cubic and RBF kernels demonstrated an inverserelationship between holdout error and regularization, these kernels would incur anunjustifiably higher risk of overfitting if searching at higher values of C.Table 5.4: Optimized hyperparameter values and corresponding cross-validation error for the SVMs using various kernels.Model C γ Holdout error (%) Five-fold error (%)Linear 1.624 2.624 3.57 4.0Quadratic 17.528 2.644 1.48 2.0Cubic 17.655 0.688 0.79 1.3RBF 19.416 2.659 0.48 Confusion MatrixThe confusion matrix shown as Table 5.5 plots the resulting predicted classifi-cations produced by the SVM with the four kernels tested against the actual classi-fications. The definitions for each of the metrics shown in the confusion matrix aredisplayed below in Equations 5.1 through 5.9.Speci f icity =T NT N+FP(5.1)Sensitivity =T PT P+FN(5.2)False positive rate =FPFP+T N(5.3)Positive predictive value =T PT P+FP(5.4)100Negative predictive value =T NT N+FN(5.5)False omission rate =FNT N+FN(5.6)False discovery rate =FPT P+FP(5.7)Negative predictive value =FNT P+FN(5.8)Accuracy =T P+T NT P+FP+T N+FN(5.9)where T N = true negative, FN = false negative, FP = false positive and T P = truepositive.101Table 5.5: Confusion matrix for handover detection using a SVM with various kernels on motion capture data.Total Population: 50845Actual conditionPrevalence: 55.0%0 1PredictedCondition0True Negative False Negative Negative Predictive Value False Omission RateLIN 43.3 LIN 1.9 LIN 95.7 LIN 4.3QUA 43.8 QUA 1.3 QUA 97.2 QUA 2.8CUB 43.9 CUB 4.3 CUB 91.1 CUB 8.9RBF 44.0 RBF 2.4 RBF 94.8 RBF 5.21False Positive True Positive Positive Predictive Value False Discovery RateLIN 1.7 LIN 53.1 LIN 96.9 LIN 3.1QUA 1.2 QUA 53.8 QUA 97.8 QUA 2.2CUB 1.1 CUB 50.8 CUB 98.0 CUB 2RBF 1.0 RBF 52.6 RBF 98.2 RBF 1.8Specificity Sensitivity AccuracyLIN 96.3 LIN 96.5 LIN 96.4QUA 97.4 QUA 97.7 QUA 97.5CUB 97.7 CUB 92.2 CUB 94.7RBF 97.8 RBF 95.6 RBF 96.6False Positive Rate False Negative RateLIN 3.7 LIN 3.5QUA 2.6 QUA 2.3CUB 2.3 CUB 7.8RBF 2.2 RBF 4.41025.7 Discussion5.7.1 Feature SelectionPrior work seems to indicate that being able to choose from a rich selectionof kinematic features rather than simply using marker data (e.g., marker positionsand velocities) improves classification ability [14, 79]. For example Begg andKamruzzaman experienced poor accuracy (62.5%), sensitivity (66.7%), and speci-ficity (58.3%) in classifying gait patterns when only baseline naı¨ve kinematic fea-tures such as position and velocity were used in a RBF-based SVM [14]. However,these performance measures rise drastically (83.3% accuracy, 75.0% sensitivity,and 91.7% specificity) when many other kinematic features were considered aswell, such as ankle and knee angles. Thus, one might expect that the exceptionalresults obtained in this study, as shown in Table 5.5 (94-96% accuracy, 92-97%sensitivity and 96-97% specificity), were obtained as a result of access to a richerkinematic feature set as opposed to using only position and velocity motion-captureinformation as features.The features selected through the random forest algorithm are quite comparableto nonverbal cues investigated in other handover detection systems. More than halfof the features selected relate to the kinematics of the giver’s right arm and hand(the hand that was used to hand over objects). Four of the top six features relate tothe approach of the giver towards the receiver (e.g., chest and forehead positionsand motions) and two features relate to the vertical motion of the giver’s body. Theimportance of these features in handover detection supports prior work that has ei-ther observed [13, 54] or used [36, 85] similar features. Thus, this work providesfurther confirmation that features deemed important in previous studies are indeeduseful for handover detection through a systematic method of kinematic featureselection. However, in previous studies only the location and velocity of the hand,or joint angle relations between elbow, shoulder and hand, were observed/used.The feature set presented here indicates that several more features relating to theCartesian and joint angle features of the giver’s arm involved in the handover couldbe used for the detection of handovers (e.g., component distances between joint lo-cations and joint angle velocities) than those traditionally examined in prior work.103It is interesting that a left-sided feature, namely, the angular velocity of theleft-shoulder to neck segment with respect to the XZ plane - was also found tobe in the top 20 features. Both video footage and motion-capture data from theexperiment were reviewed to investigate why this feature was included. From thisreview, it was found that in the majority of handover motions, the giver twisted theirbody about their center vertical axis such that the right shoulder rotated towards thereceiver, extending the reach of the right hand and causing the left shoulder to rotateaway from the receiver. Although the observance of this rotation seems natural, noreference to it as a characteristic cue in previous handover studies was found. Oneexplanation stems from proxemics and the experimental set up: as most givers andreceivers had no relationship outside the experiment, givers were more likely toapproach only to within a social distance (>1.25m) [45, 121]. Thus, to facilitatethe handover at that distance, the giver rotates their body to extend their reach. Analternative, though equally likely, explanation for this behaviour is that the giverperforms the handover in a way that is believed to be energy-efficient given thesituation, in the same way that a person lying on a couch is more likely to try toreach for a remote control with an outstretched arm rather than walk to retrieve it.Whatever the case may be, the results of this study suggest that this rotation shouldnot be overlooked as a feature in handover detection.None of the features in the input space reflect the proximity of the giver to thereceiver, except for the Y- component position of the right hand, which was lateradded to the feature set. This is unusual as several studies have observed patternsin distances between giver and receivers during handover [13, 84, 115]. However,Strabala et al. also mentions that proxemic features are not used in their classifier[113]. A review of scatter plots of specific marker positions within the trainingset seems to explain this absence: clusters of positive and negative observationsseem to perfectly overlap each other, leading to positional features having poordiscriminatory ability by the random forest algorithm. This may be an unintendedconsequence of collecting data from a laboratory environment where the partici-pants’ initial proximal positions are similar to the positions seen during handover.Strabala et al. offer another explanation: that givers start handover motions inde-pendently of the distance from the receiver. Regardless, it could be argued that theabsence of proxemic features reduces the rate of false positives of handover detec-104tion occurring due to two people standing in close proximity, without handing overobjects. More investigation is required.Given the exceptional performance of the classifier on these selected featuresand that the importance of these features falls off logarithmically (Figure 5.4), andthat the number of features used in the classifier was arbitrary, it may be possibleto further reduce the number of features used by the SVM classifier. Instead ofselecting the top 20 features, it may be possible to select a smaller number offeatures, e.g., the top ten features or a subset of the top 20 that are the most accurateor efficient to measure given current technologies, without significantly degradingthe classifier’s performance. This would not only be beneficial in reducing thecomplexity and execution time of training and running the classifier models, butalso reduce computational complexity if these models were to be run online.Overall, this study demonstrated that the use of the bagged random forestmethod for the automatic ranking and selection of predictive features indicativeof handover motions provides intuitive and useful information regarding commonnonverbal cues that can be used to anticipate object handover motions from theperspective of the receiver. The high accuracy of the SVM models using these fea-tures provides additional evidence for the importance and discriminatory ability ofthese automatically-selected kinematic features, validating the use of the baggedrandom forest approach to feature selection. Furthermore, it is reassuring that anautonomous algorithm was able to select features that were highlighted by previousworks as potentially important nonverbal cues in handovers.5.7.2 Model VerificationTest data performance from all SVM models far exceeds expectations. All mod-els have specificity, sensitivity, precision (positive predictive value), negative pre-dictive value, and overall accuracy in excess of 90%. Linear and quadratic kernelsslightly outperformed the cubic and RBF kernels both in overall accuracy and sen-sitivity, though only marginally. Otherwise, all kernels perform similarly on theholdout test set. This seems to signify that the hyperparameters selected for eachkernel was able to balance the trade-offs between overfitting and underfitting thedata, allowing the models to generalize well to the holdout test set. A review of105the classification results from the time series suggests that approximately half ofthe classification errors occurred near the start and end of handovers at locationswhere labelled classifications transition from nonhandover to handover. In effect,the SVM classifiers tend to detect handovers slightly earlier or later than how thedata is labelled, which is to be expected due to imperfections in the labelling pro-cess. With a 30 Hz sampling frequency, this may represent a few milliseconds timedifference in detection.Although results suggest that using any of the kernels should produce ade-quate handover detection, it may be more robust for robots to use the RBF kernelrather than any of the polynomial-based kernels to limit false positives. To explain,consider the case of using the linear kernel: a linear discriminant function can bethought of as a weighted sum of normalized features and, thus, any abnormallylarge value for any one of the features could cause the model to falsely detect ahandover. For example, running towards a robot equipped with a linear discrimi-nant handover detector will most likely produce a detection due to abnormally highchest and right hand velocities in the Y-axis, thus causing the linear detector to de-clare a positive detection when, in fact, no handover occurred. Similar problemsmay exist for higher order polynomial kernels as well, though a nonlinear kernel(such as RBF) or additional rules for validating data could alleviate these issues.The results from the testing of these classifiers show that classification per-formance exceeds that of the classifiers developed by Strabala et al., which hadan average classification accuracy of 80% (peaking at 89%)[112]. It is evidentfrom this comparison that the application of SVMs on kinematic data is definitely apromising method for the detection of handovers. Our intuition suggests that othermachine-learning approaches, e.g., neural networks and random forests, may alsobe similarly successful in performing classification on kinematic data.5.7.3 Extendibility of Method to Other TrackersTracking human motion with depth cameras such as the Intel RealSense, Prime-Sense, and Microsoft Kinect is a more attractive option for developing human-robot interactive systems than traditional motion-capture setups, since these sen-sors:106• are inexpensive compared to the cost of a motion-capture setup;• can be easily mounted to a mobile robot that is not restricted to travel withina confined space (as opposed to having to stay within a motion-capture track-ing area);• do not require users to wear garments with retroreflective markers; and• require only one camera sensor rather than the handful need by motion-capture systems.The widespread availability of off-the-shelf body tracking sensors such as theIntel RealSense or Microsoft Kinect version 2 makes the implementation of machine-learning techniques based on kinematic data more straightforward than previoustechniques, especially those based upon image processing used in several priorworks relating to gesture detection and handovers [34, 112]. To provide insightinto the feasibility of this method for such sensors, a pilot study was performedusing a Microsoft Kinect version 2 sensor for online detection of handovers. Forthis study, an experimental setup and participant task similar to that described inSection 4.4.4 was used; however, a Kinect was situated slightly above eye-heightin front of the receiver to track the giver’s upper body. As the Kinect’s skeletontracking uses similar key body points (Figure 5.7) compared to the motion-capturemarker placements used in the main study (Figure 4.4), the same selection of pre-dictor features as found in Section 5.5.5 were used - features were computed di-rectly from the Kinect body points that were within the vicinity of the originalmarkers.Because of the slight differences in tracking, the previously obtained SVM mod-els were not reused; instead, a new SVM was trained using an RBF kernel (forreasons presented in Section 5.7.2) with a much smaller dataset consisting of 50handovers from a single giver (15,312 observations, 55% negative, 45% positive).The LibSVM library [30] was used to perform model training. This model wasthen applied to an additional set of 25 handovers collected from the same partici-pant for handover detection. Detection was performed online (as opposed to offlineprocessing performed in the main study) at the Kinect’s frame rate of 30 Hz to de-termine the feasibility of using this handover detection method for human-robot107Figure 5.7: Diagram of joints and labels as automatically generated by theMicrosoft Kinect version 2 sensor. Green-outlined labels representjoints that were selected for use in the pilot study. Labels outlined inblue and shaded represent markers that were not used.handovers. Similar to the post-processing that was performed on marker data inthe motion-capture data for the main study (Section 5.5.1), missing tracked pointswere inferred automatically within each frame by the Kinect’s skeleton tracker andthe data was filtered using the same 4th-order Butterworth filter prior to featuregeneration.The results are shown in Table 5.6. It is important to remember that this is apilot study that uses training and testing data from the same participant: a classifierthat incorporates data from multiple participants may strengthen or weaken theresults as seen here. Thus, only trends as seen from the data can be are discussed. Inthe resulting confusion matrix obtained from the pilot study, it can be seen that thestatistics are similar to the results in Table 5.5: accuracy, sensitivity, and specificity108are all above 80%, though marginally lower than what is observed from the RBFkernel performance in the main study. This slight performance decrease was to beexpected due to the increased noise from the Kinect needing to fit a skeleton modelto time-of-flight depth data.109Table 5.6: Pilot study confusion matrix for handover detection using a SVM with RBF kernel applied to features gener-ated via a Kinect version 2 sensor.Total Population: 7312Actual ConditionPrevalence: 48.9%0 1Predicted Condition0True Negative False Negative Negative Predictive Value False Omission Rate49.1 2.0 96.1 3.91False Positive True Positive Positive Predictive Value False Discovery Rate5.7 43.2 88.3 12.7Specificity Sensitivity Accuracy89.6 95.5 92.3False Positive Rate False Negative Rate10.4 4.5110Although the preliminary results indicate good performance, there are some is-sues that may hamper the method’s applicability to the Kinect and similar sensors.One concern (though not observed from the pilot study) is that objects may occludehands or arms or cause poor fitting of the skeleton model. This, in turn, may leadto imperfect tracking and representation of joint features, causing poor predictionperformance. This issue can be remedied with advanced object detection and seg-mentation methods, such as those employed by Schmidt et al. [103]. Detectionaccuracy and speed may also be of concern with a larger pool of participants due tothe noisy nature of data obtained from such devices. There are also limitations dueto assumptions that were imported in terms of experimental control. One such lim-itation relates to inherent constraints placed on participants due to the experimentalsetup: givers initiated the handover directly in front of the receiver and in view ofthe camera; as opposed to having participants moving fluidly around the room andperforming handovers in less constrained locations and orientations. Needless tosay, handover gestures occurring out of view of the camera would not be detected.Also, if participants are positioned with the giver standing side-by-side or off tothe side of the receiver as opposed to face-to-face (which was the default positionused in the experimental setup), such handovers will likely not be detected as theSVM of this work has not be trained to recognize these handover configurationsand the Kinect (as well as similar body tracking cameras) require an unobstructedview of the upper body to track a person’s limbs. Being able to have the systemrecognize new handover configurations is a matter of retraining SVM with a widerrange of participant positions during handover. However, the latter issue of havingthe kinect ’see’ the receiver and the handover gesture is a shortcoming of using theKinect. Given that the field-of-view of the Kinect is much more limited comparedto the the motion capture setup used in the main experiment, optimal placement ofthe sensor becomes an important question. For a humanoid robot, such as the oneon a PR2 robot (Willow Garage Inc., Menlo Park, California, USA), placementof the sensor on the articulated anthropomorphic head (which happens to be thedefault placement of the sensor on the PR2 as recommended by Willow Garage)may mitigate this issue. Since the handover detection method is receiver-centric inthat the features used by the SVM are measured with respect to a coordinate frame111centered at the receiver, having the Kinect mounted on a robot head makes sensefor many reasons:• Articulation of the head can allow the Kinect to sense a wider field in space.• The receiver coordinate frame used as a frame of reference for feature calcu-lations rotate along with the head. Thus, as long as a human giver approachesand initiates a handover in the direction of the gaze of the robot head, thehandover should be detectable by the SVM.• Users can obtain a sense (via comparison of expectations during human-human handovers) of whether a handover gesture is detectable by the robotbased on the position of the head that is consistent with the sensor’s field-of-view limitations - e.g., if the robot head is turned away from the giver,the giver may recognize that the robot may not be able to see their handovergesture and re-approach the robot from a direction in line with the robot’sgaze.For non-humanoid robots, specifically robot arms, placement may be tricker. Onesolution may be to arbitrarily affix the sensor to a stationary feature, and constrainthe direction of handover interaction. Alternatively, an array of Kinects mountedon the base of the robot to obtain greater coverage of the field.Overall, however, the results of the pilot study suggests that the online appli-cation of the handover detection method on data streamed from the Kinect, andperhaps other skeleton tracking sensors, is viable and can be considered as an op-tion for robots to detect handover gestures. Additionally, there is also evidence thatthe feature set selected for recognizing handover gestures is able to extend readilyto the Kinect version 2, despite slight differences in the skeleton models that aretracked by the motion-capture system and Kinect version 2. This implies that norecoding and/or reselection of features is required for data gathered from the Kinectversion 2 (and possibly similar sensors such as the PrimeSense or Intel RealSense)to adequately classify handover gestures.1125.8 ConclusionsIn this study, multiple handover-detecting SVM models that exceeded the objec-tive of >80% classification accuracy (with a peak of 97.5%) were developed uponhuman kinematic data. Having been successful in using this method for handoverdetection, the following statements appear to be supported by the results:• Kinematic data obtained directly from motion-capture devices are able tooffer information on the intent to hand over.• Using machine learning to both automatically select features that may of-fer high discriminative ability and perform classification on kinematic dataoffers great promise in the area of handover detection, as well as gesturedetection.Although the SVM algorithm and features used are certainly not novel, the sys-tematic method for automatically determining kinematic features - confirming theimportance of some features found in prior work - and the application of thesefeatures to handover detection using the SVM offers a new method for advancingshared object manipulation in human-robot interaction, where prior methods in-clude image analysis and manual coding of time-series data. With this result, it isfeasible to imagine that a robot equipped with an SVM classifier and skeleton track-ing sensor can accurately recognize when a human is intending to pass an objectto it and prepare a trajectory to receive that object. Although the Kinect version2 has a frame rate 10× slower than the Vicon system and is susceptible to noisewhen capturing joint and limb data, the use of down-sampled motion-capture dataand filtering in the analysis has indicated that detection is possible under similarconditions. This hypothesis is further supported by a preliminary pilot study inwhich handover detection is performed online with data from a Kinect version 2sensor. Thus, this work suggests that machine learning on kinematic data offers ahighly promising approach to allow robots to detect handovers, directly addressingthe second research question posed in this thesis.The ability for a robot helper to accurately detect handovers has a variety ofimplications for the degree in which robots will be able to autonomously and ap-propriately respond to users within the home or workspace. Most of these envi-113ronments have tasks that rely on the dexterity of humans and cannot be completelysurrendered to automation. However, having robots that can assist persons by hand-ing over or receiving tools and supplies much like a nurse assists a surgeon duringan operation could allow robots to be more valuable in these environments. Forthese application areas, a handover detection system would be an invaluable addi-tion to a robot’s toolbox of abilities. Thus, the work presented in this paper shouldbe of great interest to robot designers, integrators, and users alike.5.8.1 LimitationsExperiments and data collection for this work was performed in a controlledlaboratory setting where the handover procedure was structured (e.g., handoverswere right-handed, handovers occurred with participants facing each other). Ad-ditionally, the data used for this study was collected under the pretext of anotherstudy. As such, some of the procedures used to collect this data were not optimalfor this work. Both of these factors may have influenced handover behaviours. Thenumber of participants recruited for this study was also limited. Another key limi-tation of this study lies in the classifier: the classifier, by design, is focused solelyon discrimination and may ignore kinematic features that are redundant, commonto both positive and negative training data, and have little or no perceived additivevalue in discriminating between negative and positive handovers. For example, it isquite possible that there is a feature that, when used alone, is very good at discrim-inating between positive and negative examples, but is ultimately eliminated by theclassifier as it produces no additional benefit compared to the mosaic of featuresalready included in the classifier. Hence, there may be features either explored orbeyond the scope of this work that could be beneficial in detecting handovers, buthave been ignored.5.8.2 Future WorkAs the data was processed offline, the application of SVMs or other supervisedmachine-learning methods for online detection of handovers should be explored.Current trends in machine learning lean towards unsupervised learning as a methodfor gesture recognition, and current work in this area has shown some promise [86].114Such data-driven approaches rely only on raw data and do not require any pre-processing or test sets to teach classification, in contrast to what was performedin this work. Although good preliminary results were observed from transplantingthe SVM from a motion-capture system to a Kinect, unsupervised learning offersgreater flexibility in deployment to other sensors, where its algorithms can adapt todifferent data streams to classify gestures. In this sense, using skeleton tracking isnot necessary with unsupervised learning, and raw data from depth cameras couldbe used to drive classification instead. However, because unsupervised learningself-clusters data into classifications rather than forcing data into pre-defined ones,a generic gesture clustering system would most likely be developed rather than ahandover detector. Additionally, the association of a learned gesture with an appro-priate reaction to that gesture must be learned as well, increasing the complexityof the system. Regardless, it would be interesting to see if unsupervised learningmethods might similarly excel at recognizing users’ intent to hand over objects.A more extensive study of using a body tracking system that is more likely tobe used on a robot (e.g., PrimeSense, Intel RealSense, and Kinect sensors) ratherthan using scrubbed data from a Vicon system would be useful. Pilot data on oneparticipant presented in this work indicates that the feature-based SVM approachto detecting handovers is easily transferable to the Kinect version 2 sensor withgood results, though a more expansive user study would definitively confirm thesefindings.The SVM classifiers developed are specific to right-handed handovers alone.There are several methods for remedying this issue, however, which include theuse of a more diverse data set that includes left-handed handovers or a mirroringtrick (e.g., running two classifiers simultaneously with one using giver data thatis mirrored about the YZ plane and taking the maximum of the two hypothesesgenerated).Due to the high accuracy obtained with pure handover prediction alone, thereis potential for classification of more specific characteristics of the handovers. Asignificant diversity of objects was used in the handover experiment, and so it maybe possible to classify object properties. For instance, the weight or fragility ofthe handover object could be ascertained through features such as handover speed,joint configuration, or one- versus two-handed grasps. Here, it might be most115telling to also train on the object pickup time interval (e.g., when picking up morefragile objects, givers may spend more time adjusting grasp before performing ahandover). Successfully classifying these properties would significantly simplifythe regulation of the robot’s receiving behaviour, as appropriate adjustments can bemade towards the expected weight and fragility.116Chapter 6Evaluating Social Perception ofHuman-to-Robot HandoversHaving demonstrated a method that enables robots to react to handover intentwithin human-to-robot handovers in the previous chapter, here, an exploration ofthe design space is initiated for robot receivers in terms of nonverbal gestures andcues. In the same vein of researched presented in Chapter 3 where robot gaze wasexamined as nonverbal cue that can affect factors of robot-to-human handovers,the work presented here and and in Chapter 7 continues to investigate how certainnonverbal cues may affect factors of human-to-robot handovers. Particularly, thischapter examines how robot nonverbal cues such as initial pose, grasp method, andretraction speed of the robot before, during and after the handover, respectively,affects how people qualitatively perceive the robot. Being able to determine hownonverbal cues change how users socially perceive robots may help determine howto improve the handover interaction. To evaluate user perceptions, a recently de-veloped psychometric tool called the ROSAS, developed by Carpinella et al. [25],is used.A portion of this work presented in this chapter was disseminated at the Human-Robot Interaction in Collaborative Manufacturing Environments (HRI-CME) work-shop of the 2015 IEEE/RSJ IROS held on September 24, 2017 in Vancouver, BC.Canada. The title of the submission which I co-authored was “Validation of theRobotic Social Attributes Scale for Human-Robot Interaction through a Human-117to-Robot Handover Use Case”. The contents of this chapter were presented at the13th ACM/IEEE International Conference on Human-Robot Interaction held fromMarch 5-8, 2018 in Chicago, Illinois, USA. A version of the manuscript includedin the proceedings entitled “Evaluating Social Perception of Human-to-Robot Han-dovers using the Robotic Social Attributes Scale (ROSAS)” is reproduced in thisthesis with only minor modifications. Supplementary materials relating to the ex-periment conducted in this work can be found in Appendix C.6.1 IntroductionThe ability of robots to safely and effectively handover objects is a crucialcapability for collaborative human robot interaction. It is a skill that will allowrobots to be increasingly useful in contexts such as manufacturing and assistivecare. In particular, handovers between robot and human agents have previouslybeen well studied. As robots are often tasked with delivering objects, most workplaces the robot in the giver role. In this work, focus is given the reverse interaction,namely, handovers from humans to robots.Previous studies have shown that nonverbal cues such as gaze and kinemat-ics have significantly affected fluency, legibility and efficiency of robot-to-humanhandovers [3, 21, 22, 113]. Given the importance that nonverbal cues play in suchinteractions, it can be argued that similar attention should be given to nonverbalcues displayed by the robot during human-to-robot handovers. Yet, this topic isrelatively unexplored by prior work. Consequently, an exploration of the designspace is necessary to determine how best to facilitate human-to-robot handovers.As a first step, this work investigates user perceptions of the robot during this in-teraction.The goal of this work is to gain a deeper understanding of robots in a receiverrole and how factors influence users’ behaviours and perception. The study pre-sented here examines how human collaborators perceive their robotic counterpartsfrom a social perspective during object handovers. Specifically, this work exploreshow changing conditions of how the robot receives an object may change useropinion of the robot. Such a user-driven approach to exploring the design spaceallows for identification of facets of the interaction which serve are key elements118in an interaction from those that do not, and hence, may inform the selection anddevelopment of robot behaviours which may be better suited for HRIS. Thus, thisexamination of how users socially view and evaluate collaborative robots may leadto a greater insight of what features, characteristics, and/or behaviours of robotscan drive more engaging, efficient and fluent handovers, and more broadly, HRIS.For this exploration, the Robotic Social Attributes Scale (ROSAS) - recently devel-oped by Carpinella et al. [25] - is used as a tool for measuring user perception ofthe robot.6.2 Background6.2.1 HandoversPrior work studying handovers has mainly focused on human-to-human androbot-to-human handovers. Among these studies, there seems to be no consensuson which set of factors are important in determining how handovers are carriedout; rather, a survey of the literature indicates that a multitude of unique factorsaffect how a handover is carried out by participants. Many studies considered howseemingly inconspicuous nonverbal cues can play an important role in coordinatingand directing handovers [3, 13, 22, 26, 53, 77, 88, 112, 123]. For example, multiplestudies have found that gaze and eye contact for both humans and anthropomorphicrobots can affect timing and coordination of handovers [3, 88, 112, 123]. Anotherstream of work has examined how grip and load forces plays an important partin allowing givers and receivers negotiate handovers leading to insight into theroles participants assume within a handover interaction [26, 63]. Other studiedfactors include arm kinematics and movement timing [3, 22], proxemics [13, 68]and handover object orientation [6, 29].Since the number of factors within the design space for human-to-human androbot-to-human handovers appears vast, it can be expected that the design spacefor human-to-robot handovers would be no different, although largely unexplored.The work presented here begins by delving into factors that are expected to af-fect perceptual judgments of the robot receiver, measured using the Robotic SocialAttributes Scale.1196.2.2 Robotic Social Attributes Scale (ROSAS)In prior work, many studies of HRIS have employed the Godspeed scale de-veloped by Bartneck et al. [12]. The Godspeed scale features five dimensionsfor rating robots: anthropomorphism (human-like vs machine-like), animacy (howlife-like the robot appears or behaves), likeability (how friendly a robot seems),perceived intelligence, and perceived safety. However, despite its widespread ap-peal, Ho and MacDorman and Carpinella et al. have found shortcomings to thescale including: lack of empirical work examining its psychometric properties, oc-currences where scale items are confounded with positive and negative affect, sit-uations where items do not correspond to the underlying constructs they are meantto measure, high correlations between constructs, and multidimensionality of someitem pairings [25, 49].Thus, in an effort to provide a more valid scale, Carpinella et al. developedthe Robotic Social Attributes Scale (ROSAS) which attempts to address these is-sues through exploratory factor analyses and empirical validation. The ROSAS isa social psychometric instrument aimed towards measuring social perception andjudgments of robots across multiple contexts and robotic platforms [25]. The de-velopment of the ROSAS is based upon the Godspeed scale and claims to improvecohesiveness, eliminate unnecessary dimensions through factor analysis, and notbe tethered to specific types or models of robots. The scale measures three un-derlying robotic attributes - competence, warmth, and discomfort using 18 itemswhich are shown in Table 6.1. While the scale borrows the competence and warmthfactors from more standard psychometric instruments used in social psychologymeasuring social perception [38], work by [25] shows that evaluations of robotsare somewhat more complex, employing the third, discomfort factor that is addi-tionally measured by the ROSAS. The scale was validated by Carpinella et al. via astudy which had participants evaluate gendered human, robot, and blended human-robot faces shown on a screen. In contrast, this work proposes to use the ROSAS toevaluate a physical HRI.The ROSAS has been chosen for this work as it provides a empirically validatedmethod of measuring how users perceive their robotic counterpart. Additionally, asthe ROSAS shares the competence and warmth dimensions with measures of social120Table 6.1: Table of ROSAS items categorized by the factors competence,warmth and discomfort.Competence Warmth DiscomfortReliable Organic AwkwardCompetent Sociable ScaryKnowledgeable Emotional StrangeInteractive Compassionate AwfulResponsive Happy DangerousCapable Feeling Aggressiveperception of people, it allows for intuitive comparisons and extrapolation of howthe robot may be matched against a human in terms of these dimensions.6.3 Experimental DesignAs this work is mainly an exploration of design space for robot receiving duringhandovers, a small number of variables were selected from a potentially large poolto test to limit the scope of the study.Several criteria were applied for selecting factors to examine. Factors selected:• affect the chronological beginning, middle, and end of the handover (to allowmultiple factors to be tested in a condition);• were previously studied in human-human interactions, the results of whichcould be used as benchmarks and/or for drawing comparisons;• are impartial to participants’ dominant/sub-dominant handedness (to limitcomplexity of implementation of the experimental setup); and• contain levels achievable by the experimental setup, as shown in Figure 6.1.Three factors which emerged as variables for robot receiving during handoversduring a pilot study are systematically tested in this study: initial position of thearm prior to handover, grasp type, and retraction speed following handover. To121constrain the length of time needed to run the experiment for each participant, thenumber of levels per factor were limited to two.122Figure 6.1: Experimental setup for the human-to-robot handover experiment. Diagram shows both the up and downinitial arm positions tested as conditions in the experiment.1236.3.1 Initial Position of the Arm Before Handover (Down and Up)For this factor, the initial arm position of the robot displayed to the giver ismodified prior to handover. Two positions are used which are labelled up anddown. Both initial positions are shown in Figure 6.1. The initial arm position waschosen to be examined as a factor since it is expected that differences in pose of therobot may affect giver behaviour when they are reaching out to indicate where andwhen a handover takes place. For example, the up position could convey the robotis awaiting the handover object, whereas the down position might suggest that therobot has not yet recognized the givers intent. They also present slightly differentinitial spacing between the robot end effector and user, which may affect where thehandover takes place as indicated by Huber et al. in [54] and Basili et al. in [13].Examination of differences in handover locations between both of these levels mayprovide insight into how users behave, in terms of proxemics, to a disembodiedrobot arm verses human/humanoid agents.6.3.2 Grasp Type During Handover (Quick and Mating)Motivated by prior studies on haptic negotiation in Human-Computer Interac-tion (HCI) which suggest that dynamic interactions are able to change how ’per-sonal’ and ’human-like’ an interaction is [44, 94], robot grasping is examined asanother factor in this work. Gripper design and grasping is still an active area ofresearch. Much of this work tries to solve the problem of matching the speed,smoothness, dexterity and conformity of the human grasp. Current state-of-the-artgrasping methods either carefully plan feasible grasps and execute them slowly, orapplies brute force to ’robotically’ grasp without delicacy of human touch. Ratherthan focus on object grasping, a simple co-planar interface (electromagnet) is usedto allow for emulation of both extremes. With this grasp method, speed and bruteforce can both be achieved by turning on the magnet which creates sudden im-pulses due to minute misalignments. Alternatively, misalignments can be slowlyaccommodated for to create a smooth yet slow contact. In the quick grasp, the robotmoves its electromagnetic end effector to within 1 cm distally from the cap of thebaton during a handover. As soon as the 1 cm threshold is met, the electromagnetis activated and draws in the baton. In the mating grasp, the robot deliberately124moves all the way into contact with the baton. Then, based on measurements ofan ATI Mini45 Force/Torque sensor (ATI Industrial Automation, Apex, North Car-olina, USA) located in series with the electromagnet at the robot end effector (seeFigure 6.1), it further adjusts its orientation to achieve flush contact. Only whenthe electromagnet is coplanar with the baton’s cap is it activated. This behaviourallows the robot to ensure stable contact and thus safety of the object during han-dover before retracting. A flowchart of how these grasping behaviours are carriedout can be found in Figure 6.2.125Figure 6.2: Flowchart of quick (top row) and mating grasp types. The quick grasp pulls in the baton magnetically whilethe mating grasp establishes coplanar contact, gently pressing against the baton before activating the magnet.1266.3.3 Retraction Speed Following Handover (Slow and Fast)Retraction speed was selected as a factor for examination as prior work hasshown that a robot’s speed of movement seems to play a significant role in howhuman observers and collaborators subjectively perceive the robot [95, 101, 124].For example, in an experiment conducted by Zoghbi et al., they found that fastrobot motions were correlated with increased user arousal and decreased valenceduring self-reports of affect [124]. Thus, in this work, retraction speed followingobject handover is hypothesized to affect how users perceive the robot in terms ofthe ROSAS measures of warmth and discomfort - e.g., slow retraction speed maybe rated as higher warmth and lower discomfort as opposed to higher speed whichmay lead to less warmth and greater discomfort. The slow setting was set to 10cm/s, whereas the fast setting was set at 20 cm/s. These settings were designed toemulate a gentle tug and a firm yank.6.3.4 ConditionsA 2x2x2 experiment design was used to test these factors, which form 8 con-ditions, as shown in Table 6.2, which were counterbalanced between participantsusing a Latin square design to prevent carry-over effects. These factors were notonly examined to see how they affect user perception of the robot’s attributes, butalso to study how they affected proxemics and kinodynamics of the handover inter-action. For example, it was hypothesized that examining initial arm position couldhelp determine how people approach and direct handover gestures to a disembod-ied robot arm and how these gestures compare to human receivers studied in priorwork [13, 96]; retraction speed and grasp type were selected to research the force/-torque interaction between the giver and receiver and to establish what dynamicnegotiations occur during human-to-robot handovers. However, investigations ofthe impact of these factors on physical characteristics of handovers is beyond thescope of this study and is left as future work.127Table 6.2: Table of experimental conditions for the human-to-robot handoverstudy.Condition Arm Position Grasp Type Retraction SpeedA down quick slowB up quick slowC down quick fastD up quick fastE down mating slowF up mating slowG down mating fastH up mating fast6.4 Experimental Setup6.4.1 SystemA KUKA LBR IIWA 7 R800 robot (KUKA, Augsburg, Germany) was used inthis study to receive objects from participants. The robot was mounted as shownin Figure 6.1, 135 cm above ground level and fitted with a simple electromagneticgripper. When activated, the gripper allowed the robot to securely grasp a handoverbaton via coplanar interfacing with a ferromagnetic cap mounted to the top end ofthe baton.A set of 12 OptiTrack Flex 13 motion capture cameras (NaturalPoint, Cor-vallis, Oregon, USA) were used to track objects within an approximately 3x3mspace. Each tracked object uses a unique constellation of retroreflective markers.The user’s hand, handover object, and robot end effector were tracked. The Flex13 cameras have a frame rate of 120 frames per second with an average latencyof 8.33ms (as reported by OptiTrack’s Motive software). Position and orientationtracking data of each object were transmitted via UDP to a second computer con-trolling the robot’s behaviour.For this system, a handover model which stipulated that the robot receiver re-acts to the giver is used. Thus, in the study, participants initiated the handover by128holding out the baton towards the robot, similar to how handovers have been initi-ated in previous studies [96]. The robot checked to see if the baton is in its reach-able workspace; if so, the robot proceeded to move to grasp the object from itsinitial position. Once certain grasp conditions were met (determined by the graspcondition), the robot activated the electromagnet and began retracting the arm andbaton by 10cm, before moving into the arm down position (see Figure 6.1). If, atany point during the retraction and movement to the arm down position, the sys-tem detected that the baton was not being grasped (e.g., the giver did not releasethe baton and overcame the electromagnet), the robot immediately returned to thebaton to reattempt grasping.6.4.2 ParticipantsThis study was reviewed and approved by the Disney Research Institutional Re-view Board. A priori power analyses were conducted to determine the sample sizerequired for this study. With α = .05, a sample size of 20 was needed to detect amoderate effect size (η2partial = .13) with 90% power (1−β ) [32]. Recruitment wasperformed within Walt Disney Imagineering Advanced Development and DisneyResearch Los Angeles (DRLA). Twenty-two participants (11 females, 11 males),aged 22-52 years [M=30.32, SD=8.12] were recruited in total. All participantsprovided their informed consent prior to the experiment using the form shown inSection C.1; they were notified that their participation was voluntary, and theywere allowed to withdraw from the experiment at any time. Additionally, permis-sion was obtained from all participants to record both video and motion capturedata from the experiment. No reward was given for participation in this study.6.4.3 Participant TaskParticipants were asked to provide consent to participate in the study and ac-knowledge the risks of participating using the form shown in Section C.1. Atthe start of each experiment session, participants were led into the motion capturespace and asked to wear a motion-tracked glove on their dominant hand. They wereasked to stand behind a table, as shown in Figure 6.1, to reduce any likelihood of129injury to participants by restricting their body (except the hand holding the baton)from entering the robot’s reachable workspace.For each trial, participants picked up the baton off the table and initiated ahandover to the robot after hearing the experimenter say ’go’. Upon detecting thebaton in its workspace, the robot would move to retrieve the baton in a way that wasconsistent with the condition being tested. Three trials were performed for eachcondition (3 trials * 8 conditions = 24 trials in total per participant). Following eachset of three handover trials for a condition, participants were asked to complete thefull ROSAS inventory (shown in Section C.2) which asked them to rate how closelyeach of the 18 items associated with the robotic handovers they just performed.Ratings were on a scale from 1 to 7 where 1 was ’not at all’, 4 was ’a moderateamount’, and 7 was ’very much so’. Each experiment session lasted approximately30 minutes.6.5 Results6.5.1 ROSAS Internal Consistency and DimensionalityAs the ROSAS is a relatively new scale that has not yet been applied to human-robot interactions [25], an internal consistency test is conducted to confirm theresults of the exploratory factor analysis performed by Carpinella et al.. Internalconsistency measures how closely the ROSAS inventory items fit within the three at-tributes (competence, warmth and discomfort) using the data in this study. For test-ing, Cronbach’s alpha was used; an αCronbach ≥ .80 is considered to represent highscale reliability. Items for competence [αCronbach = .90], warmth [αCronbach = .94]and discomfort [αCronbach = .81], all scored above this threshold suggesting that theitems have relatively high internal consistency within their respective attributes.In addition to investigating consistency, dimensionality of the items of eachfactor of the ROSAS were considered as well. Unidimensionality indicates that theitems of each factor measures and corresponds to only one dimension of the scale.On the other hand, if it was found that the items of one factor, e.g., competence,was multi-dimensional (i.e., two or more items are needed to explain a majority of130the variance within that factor), this would invalidate ROSAS as the items do notsolely measure competence, but would be measuring something else as well.A factor analysis was performed to ensure that the items for each attribute areunidimensional. Here, eigenvalues which represent how much variation in eachattribute is explained by each item were examined; the larger the eigenvalue, themore variation the item explains. For an attribute to be unidimensional, one wouldexpect to see one item account for a large portion of the variance within the at-tribute, and other items account for much less variation. As shown in Figure 6.3,the results show that the first items in competence, warmth and discomfort at-tributes explains 67.7%, 76.9%, and 53.5% of the variance respectively. Giventhat a majority of the variances are explained by one item within each factor, thesefindings suggests that the items for each attribute are unidimensional.Figure 6.3: Scree plots for ROSAS factors to examine dimensionality of items.1316.5.2 Effect of ConditionsA three-way repeated measures Multivariate Analysis of Variance (MANOVA)was conducted to test the effect of the manipulated variables (initial arm config-uration, speed of retraction, and grasp type) on the ROSAS attributes (Figure 6.4).Effect sizes in terms of partial eta squared (η2partial) are reported; as a rule of thumb,Cohen indicates that partial eta square values of .0099, .0588, and .1379 may serveas benchmarks for small, medium, and large effect sizes [32]. Significant main ef-fects of grasp on reports of competence [F(1,21)=25.660, p<.001, η2partial =.550]and discomfort [F(1,21)=7.485, p=.012, η2partial =.263]] were found. The lat-ter effect is qualified by a significant interaction effect of speed by grasp on re-ports of discomfort [F(1,21)=7.360, p=.013, η2partial =.260]. A post hoc pairwisecomparison indicates that the average competence score for the quick [M=5.225,SD=0.980] grasp type is 1.017 points higher than the mating [M=4.208, SD=1.135]grasp type [p < .001], representing a large effect size [d = 0.835]. No other mainor interaction effects were found to hold statistical significance.The significant retraction speed by grasp interaction effect (Figure 6.5) was fur-ther investigated using paired t-tests at levels of retraction speed (α = .025). A sig-nificant difference in discomfort scores between the quick [M=1.648, SD=0.638]and mating [M=2.580, SD=1.140] grasp types was found at low speed [t(43)=2.621,p<.001, d=1.048]. No significant difference in discomfort scores between quick[M=2.242, SD=1.241] and mating [M=2.326, SD=1.108] grasp types was found athigh speed [t(43)=0.370, p>.05, d=0.072]. There was also a failure to detect a sig-nificant difference between scores at slow and fast retraction speeds for the matinggrasp.6.5.3 Effect of Repeated Interaction over TimeAlthough the presentation order of conditions was counterbalanced across par-ticipants, it was questioned whether participants’ perception changed over the courseof repeated handover interactions with the robot. To examine this effect, partici-pants’ trials were categorized by the order in which they were presented in timerather than by experimental condition as shown in Figure 6.6. Trend analysis, a sta-tistical test based upon the F-statistic that is an alternative to an ANOVA [42], was132Figure 6.4: Participants ratings of the robot’s competence, warmth and dis-comfort over condition as reported during the human-to-robot handoverstudy. Error bars represent 95% CIs.conducted for each factor with appropriate corrections for non-spherical data. Re-sults showed a significant positive linear trend for warmth [F(1,21)=7.375, p=.013,η2partial =.260] and negative linear trend for discomfort [F(1,21)=6.442, p=.019,η2partial =.235]; no significant linear trend was detected for competence. Higherorder trends were non-significant for all factors.133Figure 6.5: Interaction plot showing participants ratings of the robot’s dis-comfort based on grasp type (quick, mating) at levels of retraction speed(slow, fast). Error bars represent 95% CIs.6.6 Discussion6.6.1 ROSAS Internal Consistency and DimensionalityCarpinella et al. claims in [25] that “ROSAS provides a psychometrically vali-dated, standardized measure that can be used to measure robots developed by dif-ferent people in different places for differing purposes and over time.” As this isthe first known usage of the ROSAS for human-robot interaction, it was importantto examine the integrity of the scale as it applies to data collected in this study.Although a full validation of the ROSAS using confirmatory factor analysis was notperformed due to small sample size, examination of the results show that the 18items of the scale conform to the three measures of the scale - competence, warmthand discomfort - with a high degree of consistency. Additional testing showed that134Figure 6.6: Participants ratings of the robot’s competence, warmth and dis-comfort over time as recorded during the human-to-robot handoverstudy (segmented into condition presentation blocks). Significant trendsare shown. Error bars represent 95% CIs.the items of each attribute were highly unidimensional. Thus, the results suggeststhat the application of ROSAS for this work, and perhaps more generally to otherHRIS, appears to be valid - though more work is needed to concretely confirm this.6.6.2 Effect of ConditionsAs shown by the results presented in Figure 6.4, grasp type had a significanteffect on competence scores, with the quick grasp scoring significantly higher thanmating grasping. This find runs contrary to the expectation that having the robotensure the handover object’s safety through stable contact would demonstrate moreintelligent/competent behaviour. One explanation for this finding is that althoughthe mating grasp demonstrates more intelligent algorithms to ensure handover ob-ject safety, users may actually find the method to be a significant departure fromhandovers between human participants compared to the quick grasp; thus, they arenot able to adapt easily to this novel method of handover. For example, in human-human handovers, receivers apply pulling/tugging forces to the object which signal135to the giver to release the object [26]. As opposed to the quick grasp, the robot ini-tially applies pushing forces to the object in the mating grasp, which runs contraryto expectation and leads to confusion. As evidence for this, review of video record-ings show participants complying to the robot pushing against the baton.An alternative but complementary explanation for the phenomenon relates totrade-offs made by each grasp type: the quick grasp trades off object safety forefficiency in terms of time to complete the handover, whereas mating does the op-posite. Having faster, more seamless handovers may factor more into competencescores than ensuring object safety, particularly if the role of maintaining the ob-ject’s safety throughout the handover is the giver’s responsibility rather than thereceiver’s as suggested by Chan et al. [26]. In this case, having both participantsin the handover be responsible for object safety may feel redundant to the user.As seen from the results, a significant retraction speed by grasp interaction ef-fect was detected on discomfort scores. Analysis of this interaction effect suggeststhat for the mating grasp, the discomfort rating was unaffected by retraction speed,whereas the quick grasp increased discomfort to within the same range as the mat-ing grasp in the fast retraction speed condition. This may be due to object safetybeing doubly compromised by both the quick grasp type and fast retraction whichemphasizes speed over safety causing participants to feel that the robot appears too’brash’ (as one participant was quoted) in how the object is handled by the robotduring the handover. It appears that the quick grasp coupled with slow retractionwas rated less discomforting possibly due to increased time for the giver to ensurethat the baton is securely grasped by the robot during the retraction phase of thehandover. It is possible that discomfort decreased only when both grasp and retrac-tion speed matched their expectations. To further explore if this is indeed the case,analysis of dynamic data was performed and presented in Chapter 7.As opposed to retraction speed and grasp type, there was a failure to detect anymain effects of initial arm position on any of the ROSAS measures. This suggeststhat user perception of robot competence, warmth and discomfort may be betterinformed by robot dynamic behaviours rather than static poses. However, it isposited that initial position may still impact on location of the handover, as wellas how the negotiation during handover is accomplished. This is explored in thefollowing chapter (Chapter 7).1366.6.3 Effect of Repeated Interaction Over TimeExamination of participants evaluations of the robot’s competence, warmth,and discomfort over repeated interactions showed a significant linear increase inwarmth and linear decrease in discomfort. Although this may be an effect of re-peated application of the scale itself, i.e., participants may tend to centralize theirresponses in the survey, a linear trend in the force data collected from this studyhas also been found (presented in Chapter 7). The occurrence of linear trends inboth types of data implies that participants are actually changing their perceptionof the robot, and thus the way they interact with the robot over time. This suggeststhat the more people interact with the robot, the more they normalize their attitudestowards the robot. The development of familiarity or affinity towards robots is notat all surprising to see as other studies have shown this phenomenon to occur inother contexts such as in assistive home care [62, 76] or military robotics [24].However, the observation of linear trends in both ratings of warmth and discomfortover time is a notable result. This suggests that changing interaction parametersor attributes of the robot’s receiving gestures could lead to changes in trend ratesfor warmth/discomfort ratings. If so, these parameters may be tuned or optimizedto obtain a fast increase for warmth ratings and decrease for perceived discomfortlevels. In turn, this may provide some benefits to having inexperienced users feelcomfortable interacting with robots that may appear imposing or foreign - e.g.,quickly having factory workers become comfortable working with collaborativeindustrial robotics. Further study is required.Failing to detect any significant trends in ratings of competence over time sug-gests that how competent or able a robot appears to users is not a function of re-peated interaction, but rather simply of behaviours attributed to the robot, as seenby the significant main effect of grasp type on competence.6.6.4 Implications for HRIAlthough this work focuses on handovers, some results observed may havewider implications for other HRIS.• Short of full validation, the ROSAS was shown to be internally consistent andunidimensional across factors. This result is promising in that it suggests that137the scale may be used to similarly evaluate user perceptions of other HRIS,and thus has the potential to serve as a standardized metric for HRI. Addi-tionally, the ROSAS can serve as a valuable tool for designing and evaluatingrobot appearances and behaviours.• The effects that were observed with grasp and retraction speed impactingpeople’s perceptions of the robot’s competence and discomfort may havebroader implications for HRI in terms of the trade-offs they present - e.g.,faster interaction during HRI may be more efficient, but may cause greaterdiscomfort to users, and seemingly more intelligent behaviours by the robotmay not be perceived as such due to decreased efficiency of the interaction.Thus, these observations highlights the importance of considering user per-ceptions when efforts to develop HRIS which add efficiency or capabilitiesare undertaken.• Lastly, the observation of trends over repeated interactions with the robotmay not be isolated to just the handover use case. The finding that the morethat users interact with the robot, the more they increase their ratings ofwarmth, while decreasing ratings of discomfort towards the robot may justas likely occur with other HRIS. Thus, this implies that examining inexperi-enced participants reactions during studies of HRIS may not be as importantas considering longitudinal effects and how fast people’s perceptions changeover repeated interactions.6.7 ConclusionsWithin this work, factor analysis and examination of the dimensionality ofitems relating to factors in the ROSAS indicate that the scale appears to be anacceptable tool for evaluating subjective experiences during physical HRI con-texts. Thus, in this work, the ROSAS has been used to evaluate user perceptionof robotic social behaviours during a human-to-robot handover task. Using theROSAS tool, this work has found that by varying simple parameters such as grasp-ing behaviour and retraction speed of the robot within human-to-robot handoverinteractions, users can hold significantly different views on social qualities of the138robot in terms of competence and discomfort. Ironically, even though the robotdemonstrated a more intelligent grasping strategy in the mating grasp compared tothe quick grasp, participants perceived the robot as being less competent and morediscomforting. Thus, seemingly intelligent robot behaviours doesn’t necessarilyconstitute competent or comfortable behaviours in the eyes of users. It appears,rather, that interaction efficiency and/or similarity to human-human handovers (atleast in terms of force profiles) constitutes a larger part of establishing more posi-tive user affect when working with the robot. Also, the results of this work indicatethat users perceive robots as being less discomforting and having more emotionalwarmth the more exposure they have to handover over objects to the robot - thismay apply to other human-robot interactions as well.The results presented here offers a glimpse into how users ascribe social at-tributes to robots during collaborative tasks and how ROSAS can be used to evaluatethese perceptions. It is anticipated that the results of this study may inform otherhuman-robot interactions which can be similarly evaluated.6.7.1 LimitationsAlthough the sample size of this study was determined to be sufficient for de-tecting a target range of effect sizes in user perception, a larger sample size couldbe useful in shoring up the defensibility of findings. Furthermore, the specifics ofthe experimental setup may limit the generalisability of findings to other human-to-robot handover contexts. In particular, the disparity of user perception betweenlevels of grasp type may be restricted to the use of an electromagnetic end effectorand may not be pertinent to other end effector types. A closer examination of othergrasping mechanisms and techniques may yield different results.6.7.2 Future WorkAs discussed in the analysis, the results of this study have generated more re-search questions and numerous pathways for additional examination which can beexplored. As only a small subset of non-verbal factors have been explored, fu-ture work can be directed towards exploring other factors, e.g., approach speed,laterally-varying initial poses with the robot approaching from the users’ domi-139nant and sub-dominant sides, up/down elbow configuration during handover aimto determine whether similarities or differences exist between human-human andhuman-robot handovers.Future studies can also be directed towards determining what differences existbetween human-human and human-robot handovers in terms of roles and users’approaches to handover. Previously, Chan et al. established that both participantsin a handover implicitly take up roles during the handover negotiation where thegiver is responsible for the safety of the object and the receiver is responsible forthe efficiency and pace of the handover [26]. It could be asked whether these rolesalso exist within the framework of human-to-robot handovers. With regards tothe ROSAS, additional work can be done to expand its use for other HRIS, obtainfurther substantiation of its validity and further explore how social perceptionscould/should shape such interactions.140Chapter 7Exploration of Geometry andForces Occurring WithinHuman-to-Robot HandoversIn Chapter 6, a human-to-robot experimental setup was used to examined howchanges in the robot’s behaviour (i.e., initial pose, grasp type, and retraction speed)influenced participants’ social perception of the handover via the ROSAS. As a re-sult of that study, it was determined that the way in which a robot exhibits nonverbalcues during handovers has significant impacts on how that robot is perceived.This chapter continues to explore the design space for human-to-robot han-dovers by examining how interaction position and force are also affected by therobot’s nonverbal behaviours. Using kinematic and dynamic data gathered fromthe human-to-robot handover study presented in Chapter 6, handover object ge-ometries (i.e., position and orientation) and dynamics (i.e., forces) can be com-pared to results obtained from prior work investigating human-to-human handovers- namely work done by Basili et al. [13] and Chan et al. [26]. Additionally, thework in this chapter examines how handover object geometries and dynamics areaffected by the nonverbal behaviours of initial pose, grasping method and retrac-tion speed of the robot in terms of interaction fluency and giver/receiver roles assuggested by Chan et al. [26]. Finally, this work studies how repeated interactionswith the robot changes force profiles seen during the handover negotiation, draw-141ing ties to similar changes in social perception observed in Chapter 6, suggestingthat trustworthiness of the robot may be linked to the forces imparted on the objectby human givers.A manuscript derived from the work in this chapter was presented at the IEEEHaptics Symposium (HAPTICS 2018) held in San Francisco, California, USAfrom March 25-28, 2018, and is included in the symposium’s proceedings. Somedifferences exist between this chapter’s contents and the conference paper entitled”Exploration of Geometry and Forces Occurring Within Human-to-Robot Han-dovers”; these differences are listed below:• The introduction of this work has been modified to more closely follow thework presented in Chapter 6.• The background information section has been removed to avoid repeatinginformation found in Chapter 2.• References to the the manuscript examining social perceptions derived fromthe subjective results of this work have been redirected to Chapter 6.• As the work in this chapter is derived from the same experiment conductedin Chapter 6, details of the experimental design and setup have been replacedwith references to Section 6.3 and Section 6.4, respectively.Supplementary materials for this work can be found in Appendix C.7.1 IntroductionIn the previous chapter (Chapter 6), a human-to-robot handover user study wasconducted examining the effects of changing robot behaviours/nonverbal cues onparticipants’ social perception of the robot. Specifically, initial arm pose, grasptype, and retraction speed of a robot arm during handover interactions were mod-ified to study how they affected user ratings of the robot’s warmth, competence,and discomfort. The results of the study showed that some robot behaviours, suchas grasp type and retraction speed, affected user ratings of competence and dis-comfort. Additionally, repeated interactions with the robot yielded a bonding/ac-142climatization effect - higher ratings of warmth and lower ratings of discomfort werereported over time.Here, the same user study is re-examined to investigate physical aspects ofthe interaction. In particular, this chapter investigates how human givers presentthe object to the robot (geometry) and what forces they imparted on the object (dy-namics) during the human-to-robot handover. Effects of changing the starting pose,grasp type, and retraction speed of the robot are considered as they relate to poten-tially changing the geometric and kinodynamic negotiation of the object within thehandover. Additionally, this work aims to determine if users adapt to repeated in-teractions with the robot, i.e., to observe learning effects. Through observations ofchanges in the human giver’s behaviour, the aim of this work is to begin character-izing the design space for human-to-robot handovers from the physical interactionperspective, and be able to inform how subtle alterations of the robot may affecthuman users.7.2 Experimental DesignThe experimental design and setup used for this study can be found in Sec-tion 6.3 and Section 6.4, respectively, and will not be repeated here.7.3 GeometryThe position and orientation of the baton as held by the participants to initi-ate handover were recorded with respect to the base coordinate frame of the robot(shown in 6.1). Figure 7.1 shows a scatterplot of positions with X measuring thedistance to the robot, Y the lateral offset, and Z the height above ground (not plot-ted). Figure 7.2 shows the orientation composed of elevation as the pitch angleof the baton relative to horizontal and azimuth as the lateral yaw angle. Rotationalong the baton axis was not considered due to symmetry. As the factors of grasptype and retraction speed are non-causal to how participants initially pose the ba-ton (e.g., these factors chronologically occur after the participants’ initiation of thehandover and should not affect how the baton is initially positioned and orientedby participants), this data was analyzed with respect to the up versus down initialrobot poses.143Only the elevation angle was found to have a significantly differing mean. Par-ticipants pointed the baton more horizontally when the robot started in the up pose.The up pose also led to less variance in both the lateral position and azimuth angle.Figure 7.1: Scatterplots of initial object positions (from the overhead per-spective) for both the down (left) and up (right) initial arm pose con-ditions presented during the human-to-robot handover study.7.3.1 ResultsPaired samples t-tests showed no significant differences in means under downversus up conditions for all position axes and azimuth angle. Only the mean eleva-tion differed for down [M=14.840◦, SD=10.889◦] and up [M=10.664◦, SD=8.209◦]arm pose conditions [t(21)=3.470, p=.002, d=0.433].An equivalence test (TOST procedure) was conducted for each position axisusing a ±25 mm margin of equivalence. This margin was based on statistical re-sults obtained by Basili et al. in [8] examining the handover object to giver distancefor 26 giver/receiver dyads [M=646.68 mm, SD=87.3 mm]. The sample size of 22was calculated to be sufficient (with a two-sided 90% CIs and 80% power) to es-tablish equivalence, even with a 10% participant loss. Results yielded statisticalequivalence between the up and down groups for all axes at p < 0.05.144Figure 7.2: Scatterplots of initial object azimuth and elevation angles for boththe down (left) and up (right) initial arm pose conditions presented dur-ing the human-to-robot handover study.F-tests for comparing variances was performed for each position and angle.The variance in the Y position was significantly different [F(21,21)=2.149, p=.043],with the variance for the up condition being smaller than for the down condition.Similarly, the variance in azimuth angles differed [F(21,21)=2.207, p=.04] betweendown [M=0.661◦, SD=3.621◦] and up [M=0.934◦, SD=2.437◦] conditions, againwith the up condition leading to a smaller variance. Differences in variance for Xand Z positions and elevation angles were not significant.7.3.2 DiscussionComparison to Human-Human HandoversTable 7.1 compares handover object position results obtained from the human-to-robot experiment with human-human handovers, in particular, as studied by145Basili et al. [13]. Relative to human-to-human handovers, it is observed that par-ticipants in the human-to-robot study held the baton approximately 27 cm lower.To explain this discrepancy, it should be considered that the robot apparatus usedin this study is approximately 150 cm tall, as seen in Figure 6.1. Meanwhile theaverage human receiver height appears to be 180 cm in [13], presenting a roughly30 cm difference. This apparent correlation suggests that givers may be influencedby the proportions of the receiver and place the object conveniently for the re-ceiver. If true, this would imply robots should try to receive handovers at a heightproportional to their stature.Table 7.1: Comparison of Cartesian positions of handover objects betweenhuman-robot and human-human handovers.Human-to-RobotHuman-to-Human(Basili et al. [13])Direction Mean(mm)Std.Dev.Mean(mm)Std.Dev.t(46) pX 1117 73.36 506.2 131.4 21.446 <.001Y -7 54.18 36 53.3 2.764 =.008Z 1131 81.988 1407.8 53.7 19.377 <.001Direct comparisons of lateral and distal positioning within the horizontal planeis challenged by differences in experimental procedure. For example, the protocolof the study presented here includes a table that was placed between giver and robotreceiver which was not used in Basili et al.’s experiment. Additionally, right- aswell as left-handed handovers were allowed, whereas only right-handed handoverswere permitted by Basili et al.. Nevertheless, regardless of whether the receiveris human or robot, both studies observe that the handovers occur roughly halfwayand centered between giver and receiver.Effect of Initial Robot Pose on GeometryThe results of the study shows that the initial robot pose significantly affects agiver’s placement and orientation of an object for handover. When the robot starts146in the up pose, thus closer to giver and the eventual handover location, a givermore tightly places and orients the object in the horizontal plane. They also lowerthe elevation angle, more aligned with though still significantly above the robot’send-effector angle of 5.94◦[t(21)=2.698, p=0.014, d=0.575]. It appears that giversgenerally attempt to place and orient the object complimentary to the end-effector,at least as much as is comfortable.This observation agrees with arguments made by Cakmak et al. maintainingthat the spatial configuration may be an important tool for improving handoverinteraction fluency through implicit, nonverbal communication [22]. Thus, this re-sult suggest the robot’s up pose implicitly communicates to users, better informingthem where and in what orientation the robot can reach for the object.Such communication is particularly important in human-to-robot interactions.Where human givers likely have lots of experience handing objects to human re-ceivers, they may have limited a-priori understanding of robot handovers. Espe-cially in the down pose, the amorphous shape of the KUKA LBR IIWA providesfew cues and givers may remain uncertain how to present the object. Possibly poorplacements could then require longer robot trajectories or awkward grasp angles,limiting fluency and efficiency of the interaction. Thus, it is hypothesized that therobot’s up pose, illustrating the preferred handover angles and location, increaseshandover fluency and efficiency.7.4 DynamicsInteraction forces during the handover was captured using the force/torque sen-sor attached to the robot’s end-effector. For the purposes of this study, however,only the forces applied axially with respect to the end-effector were analyzed. Iner-tial and gravitational components were subtracted from the data using the observedkinematics to calculate the isolated interaction forces experienced by the humangiver. Additionally, data was filtered using a fourth-order low-pass Butterworthfilter with a 14 Hz cutoff, similar to [26].For this analysis, the maximum retraction/pull force applied by the robot to thegiver was considered. It has been postulated that this absolute level communicatesthat the receiver is in full control of the object and triggers the giver’s release.147The maximal change in retraction force, relative to the force immediately beforeretraction is also considered as relative changes may provide additional informationand triggers to the giver. These metrics are illustrated in Figure 7.3. Pulling forcesapplied to the end effector are denoted as positive, whereas pushing forces arenegative.Figure 7.3: Sample end effector axial force vs. time plot for a participantdepicting features used in the dynamics analysis of the human-to-robothandover study. Negative forces indicate pushing (compressive) forcesexerted against the end effector whereas positive forces indicate pulling(tension) forces.7.4.1 ResultsThe mean maximal absolute and relative retraction forces are depicted in Fig-ure 7.4. In particular, the overall mean maximal absolute retraction force was5.48 N [SD=7.11N] or 223% (SD=290%) of the baton’s 250 g weight. A three-way repeated measures MANOVA was conducted to test the effect of the manip-ulated variables (initial arm configuration, speed of retraction, and grasp type) onthe mean maximal absolute and relative retraction forces. Effect sizes in terms148of partial eta squared (η2partial) are reported. Results showed significant main ef-fects of grasp type [F(1,21)=9.765, p=.005, η2partial =.317] and retraction speed[F(1,21)=10.322, p=.004, η2partial =.330] on mean maximal absolute retractionforces. For the relative retraction forces, a significant main effect was only ob-served for retraction speed [F(1,21)=10.888, p=.003, η2partial =.341]. No othermain or interaction effects were found to be significant.Figure 7.4: Maximum absolute and relative retraction forces as experiencedby the giver in the human-to-robot handover study. Asterisks representsignificant comparisons at the p < 0.05 level and error bars represent95% CIs.7.4.2 DiscussionComparison to Human-Human HandoversChan et al. report that human givers tend to delay the release of an object evenafter the receiver is fully supporting the object’s weight [26]. They measured amaximum excess receiver load and hence a positive maximum retraction force of2.36% [SD=4.16%] of their baton’s 483-678 g weight. With the receivers pulling149more than the object’s weight, they hypothesized this may be a precautionary be-haviour on the part of the giver to ensure safe object transfer.For a robotic receiver, a nearly 100-fold increase in this metric is observed.Following the above hypothesis that a giver only releases the object when theybelieve safety is guaranteed, this could imply participants were not as confident ortrusting in the robot receiver. Such a lack of confidence would be consistent withinexperience in human-to-robot handovers. But this argument would necessitatethat the interaction forces are only created by voluntary giver actions.An alternative explanation would come from involuntary forces. If the roboticretraction follows a different timing, motion profile, speed, or even impedance thana human retraction, the interaction forces might also differ without any voluntaryconsideration. For example, the retraction forces could be generated before thereceiver has a chance to react. As such, this could suggest an efficient human-to-robot handover will require subtle retraction movements.Effect of Grasp TypeThe grasp type had a significant effect on the maximal absolute retraction force,with mating grasps resulting in approximate half the force of quick grasps. Follow-ing the above logic, this could signify that participants felt more trusting of themating grasp and thus released the object at a lower absolute force threshold.However, recall that during the mating grasp, the robot initially applies a push-ing force in an attempt to obtain flush contact. Meanwhile in the quick grasp, themagnet is already pulling the object. Indeed, the maximal relative retraction forcedoes not show a significant difference between the two conditions. This couldsuggest that givers are relatively indifferent to the grasping type and trigger theirrelease on a relative force change. And as before, any involuntary reaction forcesmay compound the observations.Effect of Retraction SpeedThe slow and fast retraction speeds may shed the most light on the issues ofinvoluntary force buildup. Both the maximal absolute and relative retraction forceswere significantly affected by the retraction speed condition. In particular, twice150the retraction speed resulted in nearly twice the retraction force. Also several par-ticipants noted that in the fast condition, they felt the robot yanking the baton outof their hand.The differences in force profiles may have less to do with voluntary forcethresholds and more with human reaction time, which is on the order of 150 msfor haptic stimuli. In the slow condition, the maximal retraction force occurs 147ms [SD=50 ms] after the start of the retraction. In the fast condition, the timingis much shorter. A fast retraction thus exceeds the giver’s grip forces before theycan react. To avoid any sensation of yanking and generally to allow the giver tovoluntarily control forces, releasing the object as appropriate, the robot receiverwill need to carefully modulate and limit retraction speeds. At least until the robotcan determine that the object has been released.Finally, it is noted that the apparent correlation between forces and retractionspeeds suggests that the givers are presenting repeatable impedances during han-dover. Such findings could also help guide robot behaviours in handover to a hu-man.7.5 LearningAlthough the presentation order of conditions was counterbalanced across par-ticipants, each participant’s perception was expected to change over the course oftheir repeated handover interactions with the robot. To examine this effect, partic-ipants’ trials are shown in chronological order in Figure 7.5.151Figure 7.5: Maximal forces during handover as experienced or applied by givers during the human-to-robot handoverstudy over trials ordered chronologically. Lines of best fit for each are shown. Error bars represent 95% CIs.1527.5.1 ResultsTrend analysis was conducted for each factor with appropriate corrections fornon-spherical data. Results showed significant negative linear trends for maxi-mal absolute [F(1,21)=11.924, p=.002, η2partial =.362] and relative [F(1,21)=7.607,p=.012, η2partial =.266] retraction force. Higher order trends were non-significantfor all measures.7.5.2 DiscussionThe observation of negative linear trends in both force measures over repeatedinteraction with the robot is notable as it indicates that participants are adaptingtheir force behaviour to the robot. If voluntary behaviour is considered, the giversmay be building up trust in the robot to safety receive the object and releasingsooner. Alternatively, if involuntary forces are considered, givers may be learningto predict the robot’s behaviours and moving or relaxing predictively without nec-essarily releasing sooner. Indeed these two aspects may be fundamentally linkedin the human givers, as the ability to predict would seem to go hand in hand withany willingness to trust.Further evidence of learning can be derived from participant’s subjective rat-ings of the robot during the experiment through the ROSAS inventory: ratings of therobot’s warmth linearly increased over repeated interactions, while discomfort si-multaneously decreased (see Sections 6.5.3 and 6.6.3). This suggests that the morepeople interact with the robot, the more they develop positive attitudes towardsthe robot. Both warmth and discomfort are known factors in the determination oftrustworthiness of both humans and robots [33, 82]. Thus, when considering bothsets of trends together - decreasing force and increasing positive social perceptionof the robot - there appears to be strong evidence that forces imparted on the objectby the giver are related to how willing they are to trust the robot with the safety ofthe object. Although, again, it is unclear whether lower forces cause higher ratingsof warmth and decreased discomfort (or vice versa), or whether both are effects ofanother factor at play, e.g., of familiarity or predictability of the robot.1537.6 ConclusionsBeyond providing a demonstration of a simple human-to-robot handover, theuser study was able to elicit some basic lessons on appropriate behaviour. First, theresults of the study showed that the robot’s initial pose affects the handover geom-etry. As a result, it is posited that a pose can communicate appropriate location andorientations for the handover, information that may not be obvious to an inexperi-enced giver. As such, the initial pose influences interaction fluency. Additionally,evidence was found to indicate that givers may cue off the robot’s height, as theywould off a human receiver’s height.The examination of interaction forces suggests that givers release the objectwhen they detect an appropriately large change in retraction force. That is, anincrease in the force by which the robot is pulling would indicate it is securelyholding the object and trigger the release. One theory is that this level depends onthe giver’s general trust in the robot’s ability to grasp the object; however, as thereis no direct measure of this trust, direct correspondence cannot be established atthis time.Results also show a main effect of retraction speed which caused significantlylarger interaction forces for faster retraction speeds.It is posited that a fast retrac-tion preempts the giver’s ability to react to the withdrawal. The robot simply over-comes the grip forces and yanks the object away. Thus, to provide a refined han-dover experience, human reaction time must be considered and retractions must bemodulated carefully.Compared with human-to-human handovers, the interaction forces were gen-erally significantly higher. This difference could be attributed to the inexperienceof participants in handing over to the robot; participants may seek a larger forceto confirm that the robot has securely received the object. Symmetrically, thismight suggest that the robot is not acting exactly like a human receiver and hencepresenting unexpected or unpredicted movements. Over time and repeated inter-actions, however, this effect and the force levels linearly decrease. Separate socialperception evaluations (using the ROSAS) mirror this trend with a significant linearincrease in warmth and linear decrease in discomfort. Together this may indicate154givers are learning to predict the robot and developing trust in the robot to completethe handover successfully.These findings show that slight changes to robot behaviours may significantlyalter interaction dynamics of the negotiation that occurs during handovers: wehave found significant differences to the way users kinodynamically participatedas givers during the handover through varying three robot attributes. The resultsof this work more generally suggests that an exploration of the design space forhuman-to-robot handovers may assist in achieving more fluent and legible, thoughnot necessarily human-like, handovers. Such improvements in handovers (and os-tensibly other human-robot interactions) may be measurable through examinationof interaction geometries and forces, as demonstrated here.7.6.1 Future WorkAs discussed in the analysis, the results of this study have generated more av-enues of work to be explored. Specifically, future studies can more definitively ad-dress how repeated handovers with the robot affect force levels and, more broadly,trust in the robot with respect to the object’s safety. Another related research topicmight be an investigation of how perceived value and/or fragility of the objectmight impact the necessary force levels imparted on the object by the user during ahuman-to-robot handover. Also, what factors can be changed or improved regard-ing the robot’s behaviour that might allow users to trust the robot more quickly?Future studies may also address how robot height and appearance, as well as con-tact impedance and movement fluidity may impact the interaction.155Chapter 8ConclusionsBeing able to achieve seamless object handovers between robots and humansoffers many opportunities for closer human-robot teaming and collaboration. Forexample, in manufacturing, the handover of tools or parts between robot-workerpairs can improve productivity and cost-savings. Alternatively, in assistive carefor the elderly and infirmed, robotic companions could help retrieve and handoverhard to reach objects such as a bottle of pills or television remote, enhancing theautonomy of patients. In these and other cases where physical cooperation betweena robot and human can augment the person’s abilities and efficiency, having a robotbe able to pass or receive an object to a person can be an especially useful ability.Towards the goal of enabling robotic agents to participate seamlessly in han-dover interactions, this thesis explored how robots might both recognize and dis-play nonverbal cues to facilitate object handovers to and from a human. A series ofsix human-subject studies collectively explored the following two research ques-tions:Q1 How do nonverbal cues exhibited during robot giving and receiving be-haviours change how users perceive the robot, and affect the handover nego-tiation?Q2 How can a robot adequately recognize and interpret nonverbal cues conveyedby a human to infer object attributes as well as handover intent?156Results from the six studies support that nonverbal cues offer effective meansto improve human-robot handover interactions in terms of fluency, efficiency, anduser perception. Additionally, in the case of human-to-robot handovers, we alsofind that detecting nonverbal cues from a human giver can enable a robot to rec-ognize handover intent. The remainder of this chapter presents a summary of thefindings, implications, and future work that pertain to each of the questions.8.1 Conveying Non-Verbal Cues During Handover8.1.1 Summary of FindingsNon-verbal cues of a robot have been used in various HRI contexts to establishjoint attention with and communicate a robot’s intent and internal states to a user.A large portion of this thesis focused on the role nonverbal robot behaviours canplay during handovers. A summary of the cues that have been studied can be foundbelow in Table 8.1.GazeThe investigations presented in Chapter 3 examined how gaze cues can sig-nal handover intent, location and timing, thus establishing fluency and legibility ofhandover interactions. From a human-human handover study, givers were foundto use a small set of gaze profiles. Of particular interest were two gaze profilesnamed the Shared-Attention and Turn-Taking gazes. In the Shared Attention gazeprofile, the giver would direct his/her gaze towards the future handover location ashe/she was moving the object to that location. In the Turn-Taking gaze, the giverlooks up and directs his/her gaze towards the receiver’s face following a SharedAttention gaze and completion of trajectory placing the object at the handover lo-cation. From observation of these two gaze profiles being used, it is posited thatthey serve distinct purposes in increasing fluency of the handover interaction - theShared-Attention gaze serves to inform the receiver where the handover is to takeplace, whereas the Turn-Taking gaze informs when it is appropriate for the receiverto take the object.157Table 8.1: Summary table of robotic nonverbal cues studied in this thesis andtheir observed impacts.Non-VerbalCueHandoverTypeImpact(s) on Handover ReferenceGaze R→H Handover timing (Efficiency) Chapter 3Initial ArmPoseH→R Giver’s placement and orientationof an object for handover (Fluencyand Efficiency)Chapter 7GraspStrategyH→R Perceived competence and discom-fort of robot (Fluency)Chapter 6Maximum retraction force im-parted on handover objectChapter 7ObjectRetractionSpeedH→R Perceived discomfort of robot (Flu-ency)Chapter 6Maximum retraction force im-parted on handover objectChapter 7A second study was conducted which had a robot giver imitate both the SharedAttention and Turn-Taking gaze profiles, as well as a control gaze profile where therobot gazed downwards for the entire interaction (No Gaze). This study explicitlyaddressed the impact a robot’s gaze has on human behaviours during a handover,measuring subject perception and reaction time of human receivers. Results ofthese robot-to-human handovers demonstrated that the robot’s use of gaze can im-pact when human receivers decide to move their hands to receive the object fromthe robot. Participants were observed to reach out for the object proffered by therobot significantly earlier when the robot exhibited the Shared Attention gaze thanthe No Gaze condition which did not use a human-inspired gaze pattern. Thissuggests that the implementation of nonverbal cues on a robot, such as gaze, caninfluence people’s behavioural responses to an interaction while the interaction istaking place. In addition, with the Shared Attention gaze, participants reached tothe projected handover location before the robot had fully reached the location. By158cueing the participants to reach out earlier to meet the proffered object before therobot has finished moving, the interaction is not only more efficient but also morefluid.Exploring Non-Verbal Cues in Human-to-Robot HandoversChapters 6 and 7 explore how robot nonverbal cues during human-to-robothandovers may affect user perception of the robot, geometry (i.e., position andorientation) of the handover object, and negotiation forces experienced during theinteraction. Unlike the other studies mentioned in this thesis, this work does notexplicitly draw upon inspiration from human-human interactions. The study pre-sented in both chapters features a disembodied robot arm which received a han-dover baton from human participants during handovers. Initial arm pose prior,grasping behaviour during, and arm retraction speed following handover were var-ied in a 2x2x2 factor experimental design.Through survey of participants’ perception of the robot across different condi-tions using the ROSAS, results showed that participants perception of the robot’scompetence and discomfort were significantly affected by grasp type. It was foundthat a ’brute force’ approach to grasping, where the robot energizes its electromag-netic gripper when it comes to within 1 cm of the handover baton (quick grasp),was rated as more competent and less discomforting (in the case of slower retrac-tion speed) compared to a grasping behaviour which carefully, yet more slowly,obtains flush contact by pushing against the baton before activating the magnet(mating grasp). One possible explanation for this result is that that although themating grasp demonstrates more intelligent behaviour to ensure handover objectsafety, users may actually find the method to be a significant departure (in terms offorce profile) from human-human handovers compared to the quick grasp. Thus,participants may be associating expectations of how handovers should occur tocompetency, and since they are not able to adapt easily to this difference in expec-tation in the case of the mating grasp, they rate the interaction as less competent.Alternatively, participants may also be associating interaction efficiency with com-petency, where the simpler, quick grasp type outperforms the more complicatedmating grasp type because of its speed and no-frills grasping algorithm. A take159away of this result is that robot behaviours deemed intelligent by the designer donot necessarily constitute competent or comfortable behaviours from the perspec-tive of users. Thus, users should be consulted on the design of robot behaviours.In Chapter 7, results showed that initial pose affects the handover geometry.Specifically, participants tended to reduce variation in position and orientation ofthe handover object (baton) when the robot was posed in a way that conveys therobot is awaiting the handover object (up pose) compared to when the robot armwas positioned hanging straight down (down pose). Thus, it appears that initialpose of the robot can intrinsically communicate appropriate geometries for thehandover, leading to improved interaction fluency and efficiency.When examining interaction forces occurring during handovers, it appears thatgivers release the object when they detect a large change in retraction force - i.e.,a relative threshold. In this case, an increase in the force imparted on the object asthe robot is retracting it indicates to the giver that the object is being securely held,and can let go without fear of the object dropping. Although it is hypothesized thatthis threshold depends on trustworthy the human giver feels the robot is with re-spect to object handover, testing this theory is left as future work. When comparingthe magnitude of forces experienced in human-to-human against human-to-robothandovers, the forces were found to be significantly higher in the later case - on theorder of a 100-fold increase. The inexperience of participants in handing over to therobot could explain much of this difference as participants may use a larger forcethreshold to confirm that the robot has securely received the object before releasingit - having the robot behave unexpectedly compared to what human givers may befamiliar with in human-to-human handovers may cause confusion as to how thegiver is to behave in terms of the force negotiation involved in handover over theobject to the robot. Lastly, results also show that faster retraction speeds causedsignificantly larger interaction forces. From examination of force data, it appearsthat fast retraction of the object causes the robot to simply overcome the grip forcesimparted by the user and yanks the object away before the user can react. Accord-ing to participant subjective reports, this leads to a more discomforting interaction.Thus, this suggests that to provide a refined handover experience, human reactiontime must be considered and post-grasp retractions must be modulated carefully.1608.1.2 ImplicationsThese results from these studies supplement the previous work in examininghuman-human and human-robot handovers discussed in Chapter 2. In previousstudies, nonverbal cues were shown to be effective in communicating a robot’starget object and internal states to an observing person. Findings from this thesisadd to prior work in that they demonstrate that nonverbal cues such as gaze andobject orientation in the case of robot-to-human handovers, and initial pose, grasptype, and retraction speed during human-to-robot handovers, can significantly af-fect multiple aspects of the interaction; such aspects include user perception, flu-ency, legibility, efficiency, geometry, kinodynamics and fluidity of the handover.Thus, the work presented in this thesis indicates that nonverbal cues can serve as apowerful medium by which details of a handover can be subtly communicated tothe human receiver and elicit desired kinodynamic behaviours.The implications of this work suggest that those who design behaviours forHRI, must be cognizant of how nonverbal cues can improve interactions betweencollaborating robots and humans. Non-consideration or implementation of poorlydesigned cues may be significantly detrimental to interaction efficiency, fluency,fluidity and legibility; whereas considering where and how cues can be used, per-haps through human-inspired means, can drastically improve how people perceiveand interact with the robot. Results of studies presented in this thesis also indicatethat the design space for nonverbal cues within handovers is potentially quite vastand largely unexplored. Although much of the prior literature uses human non-verbal behaviours as a framework for the design and exploration of cues withinthis space, deviating from this approach (much like how robot grasp type was in-vestigated in Chapters 6 and 7) may allow researchers to investigate unique andunexpected robot cues yielding alternative interaction cues for human-robot han-dovers and other interactions.8.1.3 LimitationsVarious nonverbal cues may signal differently to people of various ethnic ori-gins and backgrounds. Thus, one caveat of this work is that the results may onlybe applicable to populations within North America. Much more extensive studies161involving various contexts and nonverbal cues using other populations will be nec-essary before being able to draw generalized conclusions about the roles a robot’snonverbal cues can have on characteristics of human-robot handovers.These studies mainly relied upon non-expert users having little to no experienceor background with robotics. Thus, the reported results may not be applicable toexpert users having an understanding of the limitations and peculiarities of therobots they are interacting with. Backed by some findings reported in this thesisthat a learning effect has been detected (particularly for work presented in Chapters3, 6 and 7), it would be expected that the obtained results would differ in the casewhere employing expert participants were used for the studies.As a related limitation, all of these studies employed physical barriers (e.g., ta-ble, low wall) to limit non-expert users from approaching too closely to the robotsin an effort to curb unintentional and potentially injurious physical encounters.While being able to limit users’ physical exposure to the robots and also satis-fying safety concerns of behavioural research ethics review boards, the resultingexperimental setups are sub-optimal for exploring human-robot interactions thatmight realistically occur in the field, where humans and collaborating robots areworking in close proximity. As such, future experiments in this and other HRI con-texts could be aimed towards having expert users work with robots without needingsafety barriers.While limited in scope, however, the results of the work presented in this thesisprovides empirical support that even the subtle, and supplementary nonverbal cuesdisplayed by a robot can play a significant role in object handovers.8.1.4 Future WorkIn order to further explore how nonverbal cues may affect handover interac-tions, a method of study which may provide interesting results would be to inten-tionally misdirect or mislead participants using nonverbal cues. For example, interms of the gaze study found in Chapter 3, if the robot intentionally gazed upon alocation other than the handover location, would that adversely affect the durationof the handover? Additionally, with reference to the dynamic interactions duringhuman-to-robot handovers reported in Chapter 7, another study could be conducted162which examined if human givers actually employ a force thresholding technique indetermining when to let go of the object. In such a study, the electromagnetic grip-per would randomly fail to activate prior to the robot arm retracting - if participantsare indeed using force thresholding, the giver would continue to hold onto the ob-ject following retraction, whereas if another methodology is employed, the givermay drop the object altogether.As a result of observing linear trends in user perceptions of competence andwarmth (Section 6.6.3) and interaction forces (Section 7.5) over repeated interac-tions with a robot, another avenue of future work may address how repeated han-dovers with the robot affects forces experienced during the negotiation, and morebroadly, trust in the robot with respect to the object’s safety. Specifically, what fac-tors can be changed/improved regarding the robot’s behaviour that can have usersbecome more trusting of the robot more quickly?8.2 Recognizing Non-Verbal Cues from HumansThe second theme and research question of this thesis dealt with having robotsrecognize and interpret nonverbal cues from huamans. In relation to this theme,one chapter (Chapter 4) examines the quality of nonverbal cue content naturallygenerated by humans (object orientations during handover in particular) as it per-tains to using such observations of human behaviours for training robot behaviours.Another chapter (Chapter 5) investigates how robots can infer handover intent fromhuman kinematic behaviours to enable human-to-robot handovers.8.2.1 Summary of FindingsObject OrientationChapter 4 presents a human-human study where the orientation in which house-hold objects are presented by human givers to human receivers are examined. Thegoal of this work was to determine if human-human handover data could be used toteach robots how to appropriately hand over objects. Givers were asked to handovertwenty household objects under three conditions: without instruction (natural), in afashion that was considerate of him or herself as the giver (giver-centered), and in a163fashion that was considerate of the receiver (receiver-centered). Mean orientationsof the objects as used by givers were calculated using an optimization function forthe three types of handover conditions. Results of this study showed patterns inthe way participants oriented objects during handovers where objects were alignedby a specific axis associated with the object across multiple participant pairs andhandover trials. This pattern was particularly prominent in the receiver-centeredhandover condition. Postulating that these patterns arise from the affordances ofthe objects, the concept of an affordance axis as well as a mathematical definitionwas developed to track these alignments across conditions. From comparison ofthe affordance axes between receiver-centered and giver-centered handovers, sig-nificant differences in orientation were detected. This finding suggests that meanorientations and/or affordance axes of objects may be useful in teaching robots howto appropriately align objects during robot-to-human handovers.Natural handover object orientations were found to be significantly differentfrom orientations observed in the other conditions for a majority of the objectstested, perhaps due to mixing of receiver- and giver-centered handovers. This im-plies that a robot will need to consider the quality of naturally-observed handoverorientations when learning from them. Variation of orientations may provide a met-ric for quality could be as objects that have different natural and receiver-centeredhandover object orientations appear to exhibit larger variance during natural han-dovers.Recognizing Handover IntentThe question of weather nonverbal cues of a human giver be used to infer han-dover intent by a robot is addressed in Chapter 5. In developing a system for rec-ognizing handover intent, machine learning classifiers were used on observation ofkinematic behaviours due to the widespread availability of sensors that could trackhuman motion and kinematics (i.e., the Microsoft Kinect). A human-human han-dover study was conducted where kinematic behaviours of the giver were recordedvia a motion capture setup. Overall, 176 features were obtained from the motioncapture data, however, a systematic, bagged random forest approach was used ona training set to select a smaller set of 22 features deemed to be highly discrimi-164native between handover and non-handover motions. These 22 features were thenused to train several SVM models with different kernels to detect handover intent.Test results on a holdout partition indicated an >80% accuracy for all kernels, anda maximum accuracy of 97.5% by the SVM with an RBF kernel in its capacity todetect handover motions.To further demonstrate feasibility of this method using more commonly foundmotion tracking sensors employed by robots, an SVM was similarly trained on aKinect version 2 and tested in a pilot study. The results of this study showed thathandover detection performance was similar to that seen with the models trainedusing the motion capture data.8.2.2 ImplicationsWhen taken altogether, these results demonstrate considerable potential for thedetection of object properties, handover events and other gestures for HRI usingkinematic features, whereas prior methods used include image analysis and manualcoding of time-series data. The ability for a robot helper to accurately recognize el-ements of handovers provides a solid, first step in allowing robots to appropriatelydetermine how to handover objects efficiently, react to human handover gestures,and receive objects from people. Also, the findings of these studies build on top ofprior work highlighting the feasibility of recognizing human nonverbal cues whichenable autonomous behaviours within the context an HRI task. Having robots rec-ognize what a collaborating human agent is trying to nonverbally communicate,and determining how best to assist this person provides a powerful capability whichcan allow robots to be much more valuable in a wide variety of environments.8.2.3 LimitationsExperiments and data collection for these works was performed in a controlledlaboratory setting where the handover procedure was structured. Thus, the resultsobtained may not be representative of what might occur in the field. Sample sizesof these studies are relatively modest, which may limit the generalisability of thesefindings.1658.2.4 Future workGiven that several types of nonverbal cues are recognizable by robots, onecourse of future work would be to have robots learn, from human demonstration, totrajectorize handover giving and receiving motions such that they may appear flu-ent and possibly human-like to human collaborators. One method which might en-able this is the use of Dynamic Movement Primatives (DMPs) to both learn motionsand adapt to variation in handover parameters (i.e., handover location) in real-time[102]. A similar treatment might also be given to how a robot renders forces on theshared object as it is being passed from giver to receiver in a handover negotiation- such an effort may borrow from prior work in haptics demonstrating observationand mimicry of force profiles [69].166Bibliography[1] World Robotics 2016: Industrial Robotics. Technical report, InternationalFederation of Robotics, Frankfurt, Germany, 2016. → pages 1[2] E. Ackerman. Shockingly, Robots Are Really Bad at Waiting Tables. IEEESpectrum, Apr 2016. URL spectrum.ieee.org/automaton/robotics/home-robots/shockingly-robots-are-really-bad-at-waiting-tables.[Accessed Sep 23, 2017]. → pages 2[3] H. Admoni, A. Dragan, S. S. Srinivasa, and B. Scassellati. Deliberatedelays during robot-to-human handovers improve compliance with gazecommunication. In Proceedings of the ACM/IEEE InternationalConference on Human-Robot Interaction, pages 49–56, Bielefeld,Germany, Mar 2014. ACM Press. ISBN 9781450326582.doi:10.1145/2559636.2559682. → pages 19, 118, 119[4] A. Agah and K. Tanie. Human interaction with a service robot:mobile-manipulator handing over an object to a human. In Proceedings ofthe International Conference on Robotics and Automation, volume 1, pages575–580, Albuquerque, New Mexico, USA, Apr 1997. IEEE. ISBN0-7803-3612-7. doi:10.1109/ROBOT.1997.620098. → pages 16[5] J. Aleotti, V. Micelli, and S. Caselli. Comfortable robot to human objecthand-over. In Proceedings of the 21st IEEE International Symposium onRobot and Human Interactive Communication, pages 771–776, Paris,France, Sep 2012. IEEE. ISBN 978-1-4673-4606-1.doi:10.1109/ROMAN.2012.6343845. → pages 49, 51[6] J. Aleotti, V. Micelli, and S. Caselli. An Affordance Sensitive System forRobot to Human Object Handover. International Journal of SocialRobotics, 6(4):653–666, Nov 2014. ISSN 1875-4791.doi:10.1007/s12369-014-0241-3. → pages 119167[7] Amazon. Amazon Robotics, 2017. URLhttps://www.amazonrobotics.com/{#}/. [Accessed Sep 23, 2017]. → pages2[8] M. Argyle and M. Cook. Gaze and mutual gaze. Cambridge UniversityPress, Oxford, England, 1976. → pages 21, 24[9] M. Argyle and J. Dean. Eye-Contact, Distance and Affiliation. Sociometry,28(3):289, Sep 1965. ISSN 00380431. doi:10.2307/2786027. → pages 23,24[10] M. Argyle, R. Ingham, F. Alkema, and M. McCallin. The DifferentFunctions of Gaze. Semiotica, 7(1):19–32, 1973. ISSN 0037-1998.doi:10.1515/semi.1973.7.1.19. → pages 21[11] R. Atienza and A. Zelinsky. Intuitive Human-Robot Interaction ThroughActive 3D Gaze Tracking. In Robotics Research. The EleventhInternational Symposium, pages 172–181. Springer, Berlin, Heidelberg,2005. doi:10.1007/11008941 19. → pages 21, 25[12] C. Bartneck, D. Kulic´, E. Croft, and S. Zoghbi. Measurement Instrumentsfor the Anthropomorphism, Animacy, Likeability, Perceived Intelligence,and Perceived Safety of Robots. International Journal of Social Robotics, 1(1):71–81, Jan 2009. ISSN 1875-4791. doi:10.1007/s12369-008-0001-3.→ pages 120[13] P. Basili, M. Huber, T. Brandt, S. Hirche, and S. Glasauer. InvestigatingHuman-Human Approach and Hand-Over. In H. Ritter, G. Sagerer,R. Dillmann, and M. Buss, editors, Human Centered Robot Systems,volume 6 of Cognitive Systems Monographs, pages 151–160. SpringerBerlin Heidelberg, Berlin, Heidelberg, 2009. ISBN 978-3-642-10402-2.doi:10.1007/978-3-642-10403-9. → pages 15, 16, 19, 31, 37, 79, 80, 103,104, 119, 124, 127, 141, 146[14] R. Begg and J. Kamruzzaman. A machine learning approach for automatedrecognition of movement patterns using basic, kinetic and kinematic gaitdata. Journal of Biomechanics, 38(3):401–408, Mar 2005. ISSN0021-9290. doi:10.1016/j.jbiomech.2004.05.002. → pages 80, 83, 103[15] A. Ben-Hur and J. Weston. A user’s guide to support vector machines.Methods in Molecular Biology, 609:223–239, Jan 2010. ISSN 1940-6029.doi:10.1007/978-1-60327-241-4 13. → pages 83168[16] M. Bennewitz, F. Faber, D. Joho, M. Schreiber, and S. Behnke. Towards ahumanoid museum guide robot that interacts with multiple persons. InProceedings of the 5th IEEE-RAS International Conference on HumanoidRobots, pages 418–423, Tsukuba, Japan, Dec 2005. IEEE. ISBN0-7803-9320-1. doi:10.1109/ICHR.2005.1573603. → pages 24[17] E. Bicici and R. St. Amant. Reasoning About the Functionality of Toolsand Physical Artifacts. Technical Report TR-2003-22, Department ofComputer Science, North Carolina State University, 2003. → pages 51[18] A. Blum, A. Kalai, and J. Langford. Beating the hold-out. In Proceedingsof the Twelfth Annual Conference on Computational Learning Theory,pages 203–208, Santa Cruz, California, USA, Jul 1999. ACM Press. ISBN1581131674. doi:10.1145/307400.307439. → pages 91[19] L. Breiman. Random Forests. Machine Learning, 45(1):5–32, 2001. ISSN1573-0565. doi:10.1023/A:1010933404324. → pages 87[20] G. Butterworth. The ontogeny and phylogeny of joint visual attention. InA. Whiten, editor, Natural theories of mind: Evolution, development andsimulation of everyday mindreading, pages 223–232. Basil Blackwell,Cambridge, MA, 1991. → pages 24[21] M. Cakmak, S. S. Srinivasa, J. Forlizzi, and S. Kiesler. Human preferencesfor robot-human hand-over configurations. Proceedings of the IEEE/RSJInternational Conference on Intelligent Robots and Systems, pages1986–1993, Sep 2011. doi:10.1109/IROS.2011.6094735. → pages 17, 51,118[22] M. Cakmak, S. S. Srinivasa, M. K. Lee, S. Kiesler, and J. Forlizzi. Usingspatial and temporal contrast for fluent robot-human hand-overs. InProceedings of the 6th ACM/IEEE International Conference onHuman-Robot Interaction, page 489, Lausanne, Switzerland, Mar 2011.ACM Press. ISBN 9781450305617. doi:10.1145/1957656.1957823. →pages 14, 17, 18, 30, 36, 43, 51, 118, 119, 147[23] E. Calisgan, A. Haddadi, H. Van der Loos, J. A. Alcazar, and E. Croft.Identifying nonverbal cues for automated human-robot turn-taking. InProceedings of the 21st IEEE International Symposium on Robot andHuman Interactive Communication, pages 418–423, Paris, France, Sep2012. IEEE. ISBN 978-1-4673-4606-1.doi:10.1109/ROMAN.2012.6343788. → pages 35169[24] J. Carpenter. Culture and human-robot interaction in militarized spaces : awar story. Ashgate Publishing, 2016. ISBN 147244311X. → pages 137[25] C. M. Carpinella, A. B. Wyman, M. A. Perez, and S. J. Stroessner. TheRobotic Social Attributes Scale (RoSAS). In Proceedings of theACM/IEEE International Conference on Human-Robot Interaction, pages254–262, Vienna, Austria, Mar 2017. ACM Press. ISBN 9781450343367.doi:10.1145/2909824.3020208. → pages 11, 117, 119, 120, 130, 134[26] W. P. Chan, C. A. C. Parker, H. F. M. van der Loos, and E. A. Croft. GripForces and Load Forces in Handovers : Implications for DesigningHuman-Robot Handover Controllers. In Proceedings of the ACM/IEEEInternational Conference on Human-Robot Interaction, pages 9–16,Boston, Massachusetts, USA, Mar 2012. ISBN 9781450310635.doi:10.1145/2157689.2157692. → pages 18, 79, 119, 136, 140, 141, 147,149[27] W. P. Chan, C. A. C. Parker, H. F. M. Van der Loos, and E. A. Croft. AHuman-Inspired Object Handover Controller. International Journal ofRobotics Research, 32(8):971–983, 2013.doi:10.1177/0278364913488806. → pages 31[28] W. P. Chan, Y. Kakiuchi, K. Okada, and M. Inaba. Determining propergrasp configurations for handovers through observation of object movementpatterns and inter-object interactions during usage. In Proceedings of theIEEE/RSJ International Conference on Intelligent Robots and Systems,pages 1355–1360, Chicago, Illinois, USA, Sep 2014. IEEE. ISBN978-1-4799-6934-0. doi:10.1109/IROS.2014.6942733. → pages 52[29] W. P. Chan, M. K. Pan, E. A. Croft, and M. Inaba. Characterization ofhandover orientations used by humans for efficient robot to humanhandovers. In Proceedings of the IEEE/RSJ International Conference onIntelligent Robots and Systems, pages 1–6, Hamburg, Germany, Sep 2015.IEEE. ISBN 978-1-4799-9994-1. doi:10.1109/IROS.2015.7353106. →pages 50, 119[30] C.-C. Chang and C.-J. Lin. LIBSVM: A library for support vectormachines. ACM Transactions on Intelligent Systems and Technology, 2(3):27:1—-27:27, 2011. → pages 107[31] Y.-W. Chen and C.-J. Lin. Combining SVMs with Various FeatureSelection Strategies. In I. Guyon, M. Nikravesh, S. Gunn, and L. A. Zadeh,170editors, Feature Extraction, volume 207 of Studies in Fuzziness and SoftComputing, pages 315–324. Springer Berlin Heidelberg, Berlin,Heidelberg, 2006. ISBN 978-3-540-35487-1.doi:10.1007/978-3-540-35488-8. → pages 88[32] J. Cohen. Statistical Power Analysis for the Behavioral Sciences.Routledge, 1988. ISBN 0805802835. → pages 129, 132[33] A. J. Cuddy, P. Glick, and A. Beninger. The Dynamics of Warmth andCompetence Judgments, and Their Outcomes in Organizations. Researchin Organizational Behavior, 31:73–98, 2011. → pages 153[34] N. H. Dardas and N. D. Georganas. Real-Time Hand Gesture Detectionand Recognition Using Bag-of-Features and Support Vector MachineTechniques. IEEE Transactions on Instrumentation and Measurement, 60(11):3592–3607, Nov 2011. ISSN 0018-9456.doi:10.1109/TIM.2011.2161140. → pages 80, 107[35] J. Durbin. Incomplete blocks in ranking experiments. British Journal ofStatistical Psychology, 4(2):85–90, Jun 1951. ISSN 0950561X.doi:10.1111/j.2044-8317.1951.tb00310.x. → pages 40[36] A. Edsinger and C. C. Kemp. Human-Robot Interaction for CooperativeManipulation: Handing Objects to One Another. In Proceedings of the16th IEEE International Symposium on Robot and Human InteractiveCommunication, pages 1167–1172, Jeju, South Korea, Aug 2007. IEEE.ISBN 978-1-4244-1634-9. doi:10.1109/ROMAN.2007.4415256. → pages17, 103[37] M. S. Erden, K. Leblebiciolu, and U. Halici. Multi-Agent System-BasedFuzzy Controller Design with Genetic Tuning for a Mobile ManipulatorRobot in the Hand Over Task. Journal of Intelligent and Robotic Systems,39(3):287–306, Mar 2004. ISSN 0921-0296.doi:10.1023/B:JINT.0000021039.56110.c8. → pages 31[38] S. T. Fiske, A. J. Cuddy, and P. Glick. Universal dimensions of socialcognition: warmth and competence. Trends in Cognitive Sciences, 11(2):77–83, Feb 2007. ISSN 13646613. doi:10.1016/j.tics.2006.11.005. →pages 120[39] R. K. Fukuchi, B. M. Eskofier, M. Duarte, and R. Ferber. Support vectormachines for detecting age-related changes in running kinematics. Journal171of Biomechanics, 44(3):540–2, Feb 2011. ISSN 1873-2380.doi:10.1016/j.jbiomech.2010.09.031. → pages 83[40] General Motors, University of British Columbia, Universite´ Laval, andMcGill University. CHARM: Collaborative, Human-focused, AssistiveRobotics for Manufacturing, 2017. URL http://charm.sites.olt.ubc.ca/.[Accessed Sep 23, 2017]. → pages 4[41] S. Glasauer, M. Huber, P. Basili, A. Knoll, and T. Brandt. Interacting intime and space: Investigating human-human and human-robot joint action.In Proceedings of the 19th International Symposium in Robot and HumanInteractive Communication, pages 252–257, Viareggio, Italy, Sep 2010.IEEE. ISBN 978-1-4244-7991-7. doi:10.1109/ROMAN.2010.5598638. →pages 16, 17, 80[42] G. V. Glass and K. D. Hopkins. Multiple Comparisons and Trend Analysis.In Statistical Methods in Education and Psychology, chapter 17. Pearson,3rd ed. edition, 1996. ISBN 0205673538. → pages 132[43] E. C. Grigore, K. Eder, A. G. Pipe, C. Melhuish, and U. Leonards. Jointaction understanding improves robot-to-human object handover. InProceedings of the IEEE/RSJ International Conference on IntelligentRobots and Systems, pages 4622–4629, Tokyo, Japan, Nov 2013. IEEE.ISBN 978-1-4673-6358-7. doi:10.1109/IROS.2013.6697021. → pages 14,79[44] R. Groten, D. Feth, H. Goshy, A. Peer, D. A. Kenny, and M. Buss.Experimental analysis of dominance in haptic collaboration. InProceedings of the 18th IEEE International Symposium on Robot andHuman Interactive Communication, pages 723–729, Toyama, Japan, Sep2009. IEEE. ISBN 978-1-4244-5081-7.doi:10.1109/ROMAN.2009.5326315. → pages 124[45] E. T. Hall. The Hidden Dimension. Anchor Books, Garden City, New York,1990. → pages 15, 79, 104[46] Han Su and Feng-Gang Huang. Human gait recognition based on motionanalysis. In Proceedings of the International Conference on MachineLearning and Cybernetics, volume 7, pages 4464–4468, Guangzhou,China, Aug 2005. IEEE. ISBN 0-7803-9091-1.doi:10.1109/ICMLC.2005.1527725. → pages 80172[47] M. Ha¨ring, J. Eichberg, and E. Andre´. Studies on Grounding with Gazeand Pointing Gestures in Human-Robot-Interaction. In Proceedings of theInternational Conference on Social Robotics, pages 378–387. Springer,Berlin, Heidelberg, Oct 2012. doi:10.1007/978-3-642-34103-8 38. →pages 24[48] B. Hayes-Roth. A blackboard architecture for control. ArtificialIntelligence, 26(3):251–321, Jul 1985. ISSN 00043702.doi:10.1016/0004-3702(85)90063-3. → pages 36[49] C. Ho and K. MacDorman. Revisiting the uncanny valley theory:Developing and validating an alternative to the Godspeed indices.Computers in Human Behavior, 26(6):1508–1518, Nov 2010. ISSN0747-5632. doi:10.1016/J.CHB.2010.05.015. → pages 120[50] G. Hoffman and C. Breazeal. Cost-Based Anticipatory Action Selection forHumanRobot Fluency. IEEE Transactions on Robotics, 23(5):952–961,Oct 2007. ISSN 1552-3098. doi:10.1109/TRO.2007.907483. → pages 14,16, 80[51] J.-T. Hu, C.-X. Fan, and Y. Ming. Trajectory image based dynamic gesturerecognition with convolutional neural networks. In Proceedings of the 15thInternational Conference on Control, Automation and Systems, pages1885–1889, Busan, South Korea, Oct 2015. IEEE. ISBN978-8-9932-1508-3. doi:10.1109/ICCAS.2015.7364671. → pages 80[52] D.-Y. Huang, W.-C. Hu, and S.-H. Chang. Vision-Based Hand GestureRecognition Using PCA+Gabor Filters and SVM. In Proceedings of theFifth International Conference on Intelligent Information Hiding andMultimedia Signal Processing, pages 1–4, Kyoto, Japan, Sep 2009. IEEE.ISBN 978-1-4244-4717-6. doi:10.1109/IIH-MSP.2009.96. → pages 80[53] M. Huber, M. Rickert, A. Knoll, T. Brandt, and S. Glasauer. Human-robotinteraction in handing-over tasks. In Proceedings of the 17th IEEEInternational Symposium on Robot and Human InteractiveCommunication, pages 107–112, Munich, Germany, Aug 2008. IEEE.ISBN 978-1-4244-2212-8. doi:10.1109/ROMAN.2008.4600651. → pages15, 16, 17, 119[54] M. Huber, A. Knoll, T. Brandt, and S. Glasauer. Handing over a cube:spatial features of physical joint-action. Annals of the New York Academyof Sciences, 1164:380–382, May 2009. ISSN 1749-6632.doi:10.1111/j.1749-6632.2008.03743.x. → pages 16, 80, 103, 124173[55] M. Imai, T. Kanda, H. Ishiguro, and K. Mase. Robot mediated round table:Analysis of the effect of robot’s gaze. In Proceedings of the IEEEInternational Symposium on Robot and Human InteractiveCommunication, pages 411–416. IEEE, Sep 2002. ISBN 0-7803-7545-9.doi:10.1109/ROMAN.2002.1045657. → pages 21, 33[56] M. Imai, T. Ono, and H. Ishiguro. Physical relation and expression: jointattention for human˜robot interaction. IEEE Transactions on IndustrialElectronics, 50(4):636–643, Aug 2003. ISSN 0278-0046.doi:10.1109/TIE.2003.814769. → pages 24, 33[57] International Federation of Robotics. World Robotics 2016: ServiceRobotics. Technical report, International Federation of Robotics, Frankfurt,Germany, 2016. → pages 1[58] I. Intuitive Surgical. da Vinci Surgical System, 2017. URLhttps://www.intuitivesurgical.com/. [Accessed Sep 23, 2017]. → pages 2[59] iRobot Corporation. iRobot Roomba, 2017. URL http://www.irobot.com/.[Accessed Sep 23, 2017]. → pages 2[60] S. Kajikawa, T. Okino, K. Ohba, and H. Inooka. Motion planning forhand-over between human and robot. In Proceedings of the IEEE/RSJInternational Conference on Intelligent Robots and Systems. Human RobotInteraction and Cooperative Robots, volume 1, pages 193–199, Pittsburgh,Pennsylvania, USA, Aug 1995. IEEE Computer Society Press. ISBN0-8186-7108-4. doi:10.1109/IROS.1995.525796. → pages 16, 19, 80[61] A. Kendon. Some functions of gaze-direction in social interaction. ActaPsychologica, 26:22–63, Jan 1967. ISSN 0001-6918.doi:10.1016/0001-6918(67)90005-4. → pages 21[62] C. Kidd, W. Taggart, and S. Turkle. A sociable robot to encourage socialinteraction among the elderly. In Proceedings of the IEEE InternationalConference on Robotics and Automation, pages 3972–3976, Orlando,Florida, USA, May 2006. IEEE. ISBN 0-7803-9505-0.doi:10.1109/ROBOT.2006.1642311. → pages 137[63] I. Kim and H. Inooka. Hand-over of an object between human and robot.Proceedings of the IEEE International Workshop on Robot and HumanCommunication, pages 199–203, 1992.doi:10.1109/ROMAN.1992.253888. → pages 18, 79, 119174[64] J. Kim, J. Park, Y. K. Hwang, and M. Lee. Advanced Grasp Planning forHandover Operation Between Human and Robot: Three HandoverMethods in Esteem Etiquettes Using Dual Arms and Hands ofHome-Service Robot Grasp Planners for Handover Operations. InProceedings of the 2nd International Conference on Autonomous Robotsand Agents, pages 34–39, 2004. → pages 51[65] N. Kirchner, A. Alempijevic, and G. Dissanayake. Nonverbal robot-groupinteraction using an imitated gaze cue. In Proceedings of the 6thACM/IEEE International Conference on Human-Robot Interaction, page497, Lausanne, Switzerland, 2011. ACM Press. ISBN 9781450305617.doi:10.1145/1957656.1957824. → pages 25, 30, 31[66] C. L. Kleinke. Gaze and eye contact: A research review. PsychologicalBulletin, 100(1):78–100, 1986. ISSN 1939-1455.doi:10.1037/0033-2909.100.1.78. → pages 21, 22, 24[67] K. Koay, E. Sisbot, D. Syrdal, M. Walters, K. Dautenhahn, and R. Alami.Exploratory study of a robot approaching a person in the context ofhanding over an object. In Proceedings of the AAAI Spring Symposium onMulti-disciplinary Collaboration for Socially Assistive Robotics, pages18–24, Palo Alto, California, USA, Mar 2007. → pages 15, 16, 31[68] F. Kobayashi, K. Okamoto, and F. Kojima. Robot-human handover basedon position and posture of human hand. In Proceedings of the Joint 7thInternational Conference on Soft Computing and Intelligent Systems (SCIS)and 15th International Symposium on Advanced Intelligent Systems (ISIS),pages 918–921, Kitakyushu, Japan, Dec 2014. IEEE. ISBN978-1-4799-5955-6. doi:10.1109/SCIS-ISIS.2014.7044735. → pages 119[69] K. Kuchenbecker, J. Fiene, and G. Niemeyer. Event-Based Haptics andAcceleration Matching: Portraying and Assessing the Realism of Contact.In Proceedings of the First Joint Eurohaptics Conference and Symposiumon Haptic Interfaces for Virtual Environment and Teleoperator Systems,pages 381–387, Pisa, Italy, Mar 2005. IEEE. ISBN 0-7695-2310-2.doi:10.1109/WHC.2005.52. → pages 166[70] KUKA Robotics. KUKA LBR iiwa, 2017. URLwww.kuka.com/en-ca/products/robotic-systems/industrial-robots/lbr-iiwa.[Accessed Jul 17, 2017]. → pages 3, 53[71] L. I. Kuncheva and C. J. Whitaker. Measures of Diversity in ClassifierEnsembles and Their Relationship with the Ensemble Accuracy. Machine175Learning, 51(2):181–207, 2003. ISSN 1573-0565.doi:10.1023/A:1022859003006. → pages 88[72] Y. Kuno, K. Sadazuka, M. Kawashima, K. Yamazaki, A. Yamazaki, andH. Kuzuoka. Museum guide robot based on sociological interactionanalysis. In Proceedings of the SIGCHI conference on Human factors incomputing systems, page 1191, San Jose, California, USA, Apr 2007. ACMPress. ISBN 9781595935939. doi:10.1145/1240624.1240804. → pages 24[73] L. Lee and W. Grimson. Gait analysis for recognition and classification. InProceedings of the 5th IEEE International Conference on Automatic FaceGesture Recognition, pages 155–162, Washington, DC, USA, May 2002.IEEE. ISBN 0-7695-1602-5. doi:10.1109/AFGR.2002.1004148. → pages80, 83[74] M. K. Lee, J. Forlizzi, S. Kiesler, M. Cakmak, and S. Srinivasa.Predictability or adaptivity?: designing robot handoffs modeled fromtrained dogs and people. In Proceedings of the 6th ACM/IEEEInternational Conference on Human-Robot Interaction, page 179,Lausanne, Switzerland, Mar 2011. ACM Press. ISBN 9781450305617.doi:10.1145/1957656.1957720. → pages 21, 22, 24, 31[75] C. Liu, C. T. Ishi, H. Ishiguro, and N. Hagita. Generation of nodding, headtilting and eye gazing for human-robot dialogue interaction. InProceedings of the 7th ACM/IEEE International Conference onHuman-Robot Interaction, pages 285–292, Boston, Massachusetts, USA,Mar 2012. ACM/IEEE. ISBN 9781450310635. → pages 24, 35[76] W.-Y. G. Louie, D. McColl, and G. Nejat. Acceptance and AttitudesToward a Human-like Socially Assistive Robot by Older Adults. AssistiveTechnology, 26(3):140–150, Jul 2014. ISSN 1040-0435.doi:10.1080/10400435.2013.869703. → pages 137[77] J. Mainprice, E. A. Sisbot, and T. Sim. Planning Safe and LegibleHand-over Motions for Human-Robot Interaction. IARP Workshop onTechnical Challenges for Dependable Robots in Human Environments, 2(6):7, 2010. → pages 119[78] J. Mainprice, M. Gharbi, T. Simeon, and R. Alami. Sharing effort inplanning human-robot handover tasks. In Proceedings of the 21st IEEEInternational Symposium on Robot and Human InteractiveCommunication, pages 764–770, Paris, France, Sep 2012. IEEE. ISBN978-1-4673-4606-1. doi:10.1109/ROMAN.2012.6343844. → pages 16, 17176[79] H. H. Manap, N. M. Tahir, and R. Abdullah. Performance comparison ofparkinsonian gait based on Principal Component Analysis. In Proceedingsof the IEEE Symposium on Industrial Electronics & Applications, pages216–221, Kuching, Malaysia, Sep 2013. IEEE. ISBN 978-1-4799-1125-7.doi:10.1109/ISIEA.2013.6738997. → pages 103[80] A. Mannini and A. M. Sabatini. Machine learning methods for classifyinghuman physical activity from on-body accelerometers. Sensors (Basel,Switzerland), 10(2):1154–1175, Jan 2010. ISSN 1424-8220.doi:10.3390/s100201154. → pages 80[81] A. H. Mason and C. L. Mackenzie. Grip forces when passing an object to apartner. Experimental brain research. Experimentelle Hirnforschung.Expe´rimentation ce´re´brale, 163(2):173–187, May 2005. ISSN 0014-4819.doi:10.1007/s00221-004-2157-x. → pages 16, 18[82] M. B. Mathur and D. B. Reichling. An uncanny game of trust: socialtrustworthiness of robots inferred from subtle anthropomorphic facial cues.In Proc. ACM/IEEE Int. Conf. Human-Robot Interaction, page 313, LaJolla, California, USA, 2009. ACM Press. ISBN 9781605584041.doi:10.1145/1514095.1514192. → pages 153[83] Y. Matsusaka, S. Fujie, and T. Kobayashi. Modeling of ConversationalStrategy for the Robot Participating in the Group Conversation. InProceedings of the 7th European Conference on Speech Communicationand Technology, pages 2173–2176, Aalborg, Denmark, Sep 2001. → pages18, 24, 79[84] R. Mead, A. Atrash, and M. J. Mataric. Recognition of spatial dynamicsfor predicting social interaction. In Proceedings of the 6th ACM/IEEEInternational Conference on Human-Robot Interaction, pages 201–202,Lausanne, Switzerland, Mar 2011. ACM Press. ISBN 9781450305617.doi:10.1145/1957656.1957731. → pages 104[85] V. Micelli, K. Strabala, and S. Srinivasa. Perception and ControlChallenges for Effective Human-Robot Handoffs. In Research Showcase@ CMU, Pittsburgh, Pennsylvania, USA, 2011. → pages 103[86] Y. Mohammad, T. Nishida, and S. Okada. Unsupervised simultaneouslearning of gestures, actions and their associations for Human-RobotInteraction. In Proceedings of the IEEE/RSJ International Conference onIntelligent Robots and Systems, pages 2537–2544, St. Louis, Missouri,177USA, Oct 2009. IEEE. ISBN 978-1-4244-3803-7.doi:10.1109/IROS.2009.5353987. → pages 114[87] A. Moon. Negotiating with robots : meshing plans and resolving conflictsin human-robot collaboration. PhD thesis, University of British Columbia,2017. → pages 10[88] A. Moon, D. M. Troniak, B. Gleeson, M. K. Pan, M. Zeng, B. A. Blumer,K. MacLean, and E. A. Croft. Meet me where I’m gazing: how sharedattention gaze affects human-robot handover timing. In Proceedings of theACM/IEEE International Conference on Human-Robot Interaction, pages334–341, Bielefeld, Germany, Mar 2014. ACM Press. ISBN9781450326582. doi:10.1145/2559636.2559656. → pages 21, 79, 119[89] B. Mutlu, J. Forlizzi, and J. Hodgins. A Storytelling Robot: Modeling andEvaluation of Human-like Gaze Behavior. In Proceedings of the 6thIEEE-RAS International Conference on Humanoid Robots, pages 518–523,Genova, Italy, Dec 2006. IEEE. ISBN 1-4244-0199-2.doi:10.1109/ICHR.2006.321322. → pages 24[90] B. Mutlu, J. Hodgins, S. Kiesler, and J. Cassell. Designing Gaze Behaviorfor Humanlike Robots. PhD thesis, Carnegie Mellon University, 2009. →pages 21, 22, 24[91] B. Mutlu, T. Shiwa, T. Kanda, H. Ishiguro, and N. Hagita. Footing inhuman-robot conversations: How robots might shape participant rolesusing gaze cues. In Proceedings of the 4th ACM/IEEE InternationalConference on Human-Robot Interaction, page 61, La Jolla, California,USA, Mar 2009. ACM Press. ISBN 9781605584041.doi:10.1145/1514095.1514109. → pages 24[92] B. Mutlu, F. Yamaoka, T. Kanda, H. Ishiguro, and N. Hagita. Nonverballeakage in robots: Communication of intentions through seeminglyunintentional behavior. In Proceedings of the ACM/IEEE InternationalConference on Human-Robot Interaction, pages 69–76, La Jolla,California, USA, Mar 2009. → pages 79[93] N. Nguyen, D. Phung, S. Venkatesh, and H. Bui. Learning and DetectingActivities from Movement Trajectories Using the Hierarchical HiddenMarkov Model. In Proceedings of the IEEE Computer Society Conferenceon Computer Vision and Pattern Recognition, volume 2, pages 955–960,San Diego, California, USA, Jun 2005. IEEE. ISBN 0-7695-2372-2.doi:10.1109/CVPR.2005.203. → pages 80178[94] S. O. Oguz, A. Kucukyilmaz, T. M. Sezgin, and C. Basdogan. Hapticnegotiation and role exchange for collaboration in virtual environments. InProceedings of the IEEE Haptics Symposium, pages 371–378, Waltham,Massachusetts, USA, Mar 2010. IEEE. ISBN 978-1-4244-6821-8.doi:10.1109/HAPTIC.2010.5444628. → pages 124[95] E. Pacchierotti, H. Christensen, and P. Jensfelt. Human-robot embodiedinteraction in hallway settings: a pilot user study. In Proceedings of theIEEE International Workshop on Robot and Human InteractiveCommunication, pages 164–171, Nashville, Tennessee, USA, Aug 2005.IEEE. ISBN 0-7803-9274-4. doi:10.1109/ROMAN.2005.1513774. →pages 127[96] M. K. Pan, V. Skjervøy, W. P. Chan, M. Inaba, and E. A. Croft. Automateddetection of handovers using kinematic features. The International Journalof Robotics Research, 36(5-7):721–738, Jun 2017. ISSN 0278-3649.doi:10.1177/0278364917692865. → pages 77, 127, 129[97] A. K. Pandey, M. Ali, M. Warnier, and R. Alami. Towards multi-statevisuo-spatial reasoning based proactive human-robot interaction. InProceedings of the 15th International Conference on Advanced Robotics,pages 143–149, Tallinn, Estonia, Jun 2011. IEEE. ISBN978-1-4577-1159-6. doi:10.1109/ICAR.2011.6088642. → pages 16[98] M. L. Patterson. A sequential functional model of nonverbal exchange.Psychological Review, 89(3):231–249, 1982.doi:10.1037/0033-295X.89.3.231. → pages 24[99] L. Piyathilaka and S. Kodagoda. Gaussian mixture based HMM for humandaily activity recognition using 3D skeleton features. In Proceedings of the8th IEEE Conference on Industrial Electronics and Applications, pages567–572, Melbourne, VIC, Australia, Jun 2013. IEEE. ISBN978-1-4673-6322-8. doi:10.1109/ICIEA.2013.6566433. → pages 80[100] C. Rich, B. Ponsler, A. Holroyd, and C. L. Sidner. Recognizingengagement in human-robot interaction. In Proceedings of the 5thACM/IEEE International Conference on Human-Robot Interaction, pages375–382, Osaka, Japan, Mar 2010. IEEE. ISBN 978-1-4244-4892-0.doi:10.1109/HRI.2010.5453163. → pages 24[101] M. Saerbeck and C. Bartneck. Perception of affect elicited by robotmotion. In Proceedings of the ACM/IEEE International Conference on179Human-Robot Interaction, pages 53–60, Osaka, Japan, Mar 2010. IEEE.ISBN 978-1-4244-4892-0. doi:10.1109/HRI.2010.5453269. → pages 127[102] S. Schaal. Dynamic Movement PrimitivesA Framework for Motor Controlin Humans and Humanoid Robotics. In H. Kimura, K. Tsuchiya,A. Ishiguro, and H. Witte, editors, Adaptive Motion of Animals andMachines, pages 261–280. Springer, Tokyo, Japan, 2006. ISBN978-4-431-24164-5. doi:10.1007/4-431-31381-8 23. → pages 166[103] T. Schmidt, K. Hertkorn, R. Newcombe, Z. Marton, M. Suppa, and D. Fox.Depth-based tracking with physical constraints for robot manipulation. InProceedings of the IEEE International Conference on Robotics andAutomation, pages 119–126, Seattle, Washington, USA, May 2015. IEEE.ISBN 978-1-4799-6923-4. doi:10.1109/ICRA.2015.7138989. → pages 111[104] S. Shibata, K. Tanaka, and A. Shimizu. Experimental analysis of handingover. In Proceedings 4th IEEE International Workshop on Robot andHuman Communication, pages 53–58, Tokyo, Japan, Jul 1995. IEEE.ISBN 0-7803-2904-X. doi:10.1109/ROMAN.1995.531934. → pages 16[105] C. L. Sidner, C. Lee, C. D. Kidd, N. Lesh, and C. Rich. Explorations inengagement for humans and robots. Artificial Intelligence, 166(1-2):140–164, Aug 2005. ISSN 00043702. doi:10.1016/j.artint.2005.03.005. →pages 24, 35[106] E. A. Sisbot and R. Alami. A Human-Aware Manipulation Planner. IEEETransactions on Robotics, 28(5):1045–1057, Oct 2012. ISSN 1552-3098.doi:10.1109/TRO.2012.2196303. → pages 15, 16, 17, 80[107] G. Skantze, A. Hjalmarsson, and C. Oertel. Exploring the effects of gazeand pauses in situated human-robot interaction. In Proceedings of theSIGdial Meeting on Discourse and Dialogue, pages 163–172, Metz,France, Aug 2013. → pages 21, 24, 33[108] SoftBank. SoftBank Robotics, 2017. URLhttps://www.softbank.jp/en/robot/. [Accessed Sep 23, 2017]. → pages 2[109] D. Song, N. Kyriazis, I. Oikonomidis, C. Papazov, A. Argyros,D. Burschka, and D. Kragic. Predicting human intention in visualobservations of hand/object interactions. In Proceedings of the IEEEInternational Conference on Robotics and Automation, pages 1608–1615,Karlsruhe, Germany, May 2013. IEEE. ISBN 978-1-4673-5643-5.doi:10.1109/ICRA.2013.6630785. → pages 80180[110] Y. Song, Y. Gu, P. Wang, Y. Liu, and A. Li. A Kinect based gesturerecognition algorithm using GMM and HMM. In Proceedings of the 6thInternational Conference on Biomedical Engineering and Informatics,pages 750–754, Hangzhou, China, Dec 2013. IEEE. ISBN978-1-4799-2761-6. doi:10.1109/BMEI.2013.6747040. → pages 80[111] M. Staudte and M. Crocker. The utility of gaze in spoken human-robotinteraction. In Proceedings of Metrics for Human-Robot Interaction, aWorkshop at the ACM/IEEE International Conference on Human-RobotInteraction, pages 53–59, Pittsburgh, Pennsylvania, USA, Mar 2008. →pages 21, 24[112] K. Strabala, M. K. Lee, A. Dragan, J. Forlizzi, and S. S. Srinivasa.Learning the Communication of Intent Prior to Physical Collaboration. InProceedings IEEE International Symposium on Robot and HumanInteractive Communicaton, pages 968 – 973, Paris, France, Sep 2012.ISBN 978-1-4673-4604-7. doi:10.1109/ROMAN.2012.6343875. → pages18, 21, 22, 25, 79, 81, 82, 87, 106, 107, 119[113] K. Strabala, M. K. Lee, A. Dragan, J. Forlizzi, S. S. Srinivasa, andM. Cakmak. Towards Seamless Human-Robot Handovers. Journal ofHuman-Robot Interaction, 1(1):1–23, 2013. → pages 17, 18, 21, 22, 30,79, 104, 118[114] W. Takano, K. Yamane, T. Sugihara, and Y. Nakamura. Primitivecommunication based on motion recognition and generation withhierarchical mimesis model. In Proceedings of the IEEE InternationalConference on Robotics and Automation, pages 3602–3609, Orlando, FL,USA, May 2006. IEEE. ISBN 0-7803-9505-0.doi:10.1109/ROBOT.2006.1642252. → pages 80[115] T. Tasaki, S. Matsumoto, H. Ohba, M. Toda, K. Komatani, T. Ogata, andH. G. Okuno. Distance-Based Dynamic Interaction of Humanoid Robotwith Multiple People. In Innovations in Applied Artificial Intelligence,pages 111–120. Springer Berlin Heidelberg, 2005. ISBN978-3-540-31893-4. doi:10.1007/11504894 18. → pages 104[116] M. S. Thiese, B. Ronna, and U. Ott. P value interpretations andconsiderations. Journal of Thoracic Disease, 8(9):E928–E931, Sep 2016.ISSN 2072-1439. doi:10.21037/jtd.2016.08.16. URLhttp://www.ncbi.nlm.nih.gov/pubmed/27747028http:181//www.pubmedcentral.nih.gov/articlerender.fcgi?artid=PMC5059270. →pages 41[117] UBC Faculty of Graduate and Postdoctoral Studies. Structure of UBCTheses and Dissertations, 2017. URLhttps://www.grad.ubc.ca/current-students/dissertation-thesis-preparation/structure-ubc-theses-dissertations. [Accessed Dec 20, 2017]. → pages vii[118] R. Vertegaal and Y. Ding. Explaining effects of eye gaze on mediatedgroup conversations: amount or synchronization? In Proceedings of theACM Conference on Computer Supported Cooperative Work, page 41, NewOrleans, Louisiana, USA, Nov 2002. ACM Press. ISBN 1581135602.doi:10.1145/587078.587085. → pages 24[119] M. von Cranach, J. H. Ellgring, and European Association of ExperimentalSocial Psychology. The perception of looking behaviour. In M. vonCranach and I. Vine, editors, Social Communication and Movement:Studies of Interaction and Expression in Man and Chimpanzee. AcademicPress, London, 1973. → pages 24[120] M. Walters, K. Dautenhahn, S. Woods, and K. Koay. Robotic etiquette:Results from user studies involving a fetch and carry task. In Proceedingsof the 2nd ACM/IEEE International Conference on Human-RobotInteraction, pages 317–324, Arlington, Virginia, USA, Mar 2007. IEEE.ISBN 978-1-59593-617-2. doi:10.1145/1228716.1228759. → pages 51[121] M. L. Walters, K. Dautenhahn, S. N. Woods, K. L. Koay, R. Te Boekhorst,and D. Lee. Exploratory studies on social spaces between humans and amechanical-looking robot. Connection Science, 18(4):429–439, Dec 2006.ISSN 0954-0091. doi:10.1080/09540090600879513. → pages 15, 79, 104[122] K. Yamane, M. Revfi, and T. Asfour. Synthesizing object receivingmotions of humanoid robots with human motion database. In Proceedingsof the IEEE International Conference on Robotics and Automation, pages1629–1636, Karlsruhe, Germany, May 2013. IEEE. ISBN978-1-4673-5643-5. doi:10.1109/ICRA.2013.6630788. → pages 81[123] M. Zheng, A. Moon, E. A. Croft, and M. Q.-H. Meng. Impacts of RobotHead Gaze on Robot-to-Human Handovers. International Journal of SocialRobotics, 7(5):783–798, Jun 2015. ISSN 1875-4791.doi:10.1007/s12369-015-0305-z. → pages 18, 79, 119182[124] S. Zoghbi, E. Croft, D. Kulic, and M. Van der Loos. Evaluation of affectivestate estimations using an on-line reporting device during human-robotinteractions. In Proceedings of the IEEE/RSJ International Conference onIntelligent Robots and Systems, pages 3742–3749, St. Louis, Missouri,USA, Oct 2009. IEEE. ISBN 978-1-4244-3803-7.doi:10.1109/IROS.2009.5354177. → pages 127183Appendix ASupporting Materials forInvestigating Robot Gaze inRobot-to-Human HandoversA.1 Study Advertisements184 A.1.1 Poster185A.1.2 E-MailName: UBC PR2 Hackathon Team, Collaborative Advanced Robotics and Intelli-gent Systems LaboratoryDepartment: Mechanical Engineering and Computer ScienceE-mail Address:Event Name: Get a drink from our robotEvent Date: September 3, 2013 (Imagine Day)Time: 10am 2pmSchedule of Events: Drop by any time between 10am 2pmLocation (Include building name & room number): ICICS building, Room 146Event Description: Thirsty? Want to see a robot? Come find out what it is liketo get a drink (bottled non-alcoholic beverage) from a robot. We are conducting avery short (5 minute) experiment to study how robots should handover objects topeople. All you need to do is to drop by to get a free drink from our robot and fillout our questionnaire to tell us what you think about it. We will not pay you forparticipating in this study.We need lots of people to come participate. So bring your friends! Any Uni-versity of British Columbia (UBC) student of 17 years of age or older, and anyone(even non-students) of 19 years of age or older can participate.Website (if Available): http://bit.ly/caris studyRegistration Required (Yes/No): NoWho to Contact for More Information: AJung MoonThis event is open to: All UBC students over 17 years of age, and all faculty, staffand visitors of 19 years of age or older.Last Revised: August 12, 2013 Rev 1186A.2 ConsentA.2.1 Written ConsentThe University of British ColumbiaCollaborative Advanced Robotics and Intelligent Systems (CARIS) LaboratoryDepartment of Mechanical Engineering, UBC6250 Applied Science Lane, Vancouver, BC V6T 1Z4Tel: (604) 822-3147 Fax: (604) 822-2403Website: http://caris.mech.ubc.caHRI-Cues: Human-Robot Handover Study Consent FormProject Title: HRI-Cues: Human-Robot Handover StudyPrincipal Investigator: Dr. Elizabeth Croft,Contact: AJung Moon,Funding: This research is funded by the National Sciences and Engineering Re-search Council of Canada (NSERC).Purpose: The purpose of this project is to investigate how robots should hand overobjects to people. Results from this study will help us develop robots that can in-teract with people better.Procedures: The entire experiment will take no longer than five minutes. In orderto participate in this study, you must be 19 or older, or a UBC student of 17 or older.You will be asked to stand at a designated spot in front of a robot. The robot willhand over one or more bottled non-alcoholic beverages for you to take. After eachtime the robot gives you a beverage, as well as at the end of the experiment, youmay be asked to fill out a questionnaire about the handover.You may keep one of the beverages the robot will give you, but we will not pay you187for participating in this study. You may refuse to participate in this study and youmay withdraw at any time by exiting the room without interacting with the robot.Potential Risks: You may physically come into contact with the robot. This robothas been designed for safe human interaction and robot speeds and forces will bekept to safe levels. The handover has been tested and found to be safe in earlierstudies.Confidentiality: No identifying information will be collected or stored with yourdata, in order to ensure your privacy. This study will be video recorded, includingyour face, for analysis purposes. Video recordings from the experiment may bepresented at scientific conferences or published in reports/journals, but identifyingfeatures (including your face) will be blurred using digital blurring tools. Onlythe researchers involved with this study will be able to view the unblurred videorecordings. Data collected during the experiment will be stored on a password pro-tected computer in the CARIS Lab, which has restricted secure access and is lockedat all times.If you have any concerns about your treatment or rights as a research subject, youmay telephone the Research Subject Information Line in the UBC Office of Re-search Services at the University of British Columbia, at (604) 822-8598.Last Revised: August 12, 2013 Rev 1188A.2.2 Verbal ConsentConsent script to be read by co-investigators and/or additional study teammembers.This is an experiment being conducted by the CARIS lab to investigate human-robothandovers.You will be asked to stand in a designated location and a robot will hand you oneor more bottled non-alcoholic beverages. After each handover, as well as at the endof the experiment, you might be asked to answer a few short questions.The study will be video recorded, but your face will be blurred before the video isshown outside of our research group.You may withdraw from the study at any time by exiting the room.You may keep one of the beverages the robot gives you, but you will not be paid.Here is detailed information about the study, including contact information if youhave any questions or concerns.[Hand participant the printed consent form (refer to Section A.2.1)]Do you agree to participate in this study?[Obtain verbal consent]Are you 19 or older, or a UBC student of age 17 or 18?[Obtain verbal agreement]Do you agree to be video recorded for this study?[Obtain verbal consent]Do you have any questions?[Obtain any questions]Last Revised: August 12, 2013 Rev 1189A.3 SurveysA.3.1 Single Condition SurveySubject Number:Handover:What is your age?What is your gender (circle one)? M FFor each question, circle the number that represents how you felt about the han-dovers.1. I liked this handover.1 2 3 4 5stronglydisagreestronglyagree2. The handover seemed natural.1 2 3 4 5stronglydisagreestronglyagreeWe think that timing is important in human-robot handovers. Considering the tim-ing in the handover, please answer the following:3. It was easy to tell when, exactly, the robot wanted me to take the object.1 2 3 4 5stronglydisagreestronglyagreeAny additional comments?Last Revised: August 7, 2013 Rev 1190A.3.2 Condition Comparison SurveySubject Number:Handover 1:Handover 2:What is your age?What is your gender (circle one)? M F1. Which handover did you like better, overall (circle one)?First Handover Second Handover2. Which handover seemed more natural (circle one)?First Handover Second Handover3. We think that timing is important in human-robot handovers. Which handovermade it easier to tell when, exactly, the robot wanted you to take the object (circleone)?First Handover Second HandoverAny additional comments?Last Revised: August 7, 2013 Rev 1191Appendix BSupporting Materials forInvestigation of Handover ObjectOrientations and AutomatedDetection of HandoversB.1 Study Advertisements192Collaborative Advanced Robotics and Intelligent System Laboratory ICICS X015, 2366 Main Mall, UBC Campus Tel: (604) 822-3147 Website: http://caris.mech.ubc.ca  Volunteers Wanted for Human-Robot Interaction Study Project Title: HRI-Cues: Human-Human Handover Study II Principal Investigator: Elizabeth Croft Co-Investigators and Contact Persons: Matthew Pan, Wesley Chan  The CARIS Laboratory in the UBC Department of Mechanical Engineering is seeking healthy adult volunteers to participate in a study investigating how humans hand objects to each other. The knowledge gained from this study will be used in our ongoing work to develop intelligent robotic assistants.  The study will consist of two participants handing objects back and forth. We encourage participants to sign up in pairs, but all volunteers are welcome. Please include “HRI-subject” somewhere in the subject line of your email, and thank you for your interest.  This study will run throughout November 2014. For information regarding this study or to volunteer as a participant, please contact:   Matthew Pan ICICS X015, 2366 Main Mall  mpan9@interchange.ubc.ca (604)822-3147 Please place “HRI” somewhere in the subject line. Thank you for your interest in this work. Matthew Pan, PhD Student, Mechanical Engineering, UBC, mpan9@interchange.ubc.ca Wesley Chan, PhD Student, Information Science & Technology, University of Tokyo, wesleyc@jsk.imi.i.u-tokyo.ac.jp  Elizabeth Croft, Professor, Mechanical Engineering, UBC, elizabeth.croft@ubc.ca  Human-Robot Interaction Study http://tinyurl.com/robothandover mpan9@interchange.ubc.ca  Human-Robot Interaction Study http://tinyurl.com/robothandover mpan9@interchange.ubc.ca  Human-Robot Interaction Study http://tinyurl.com/robothandover mpan9@interchange.ubc.ca  Human-Robot Interaction Study http://tinyurl.com/robothandover mpan9@interchange.ubc.ca  Human-Robot Interaction Study http://tinyurl.com/robothandover mpan9@interchange.ubc.ca  Human-Robot Interaction Study http://tinyurl.com/robothandover mpan9@interchange.ubc.ca  Human-Robot Interaction Study http://tinyurl.com/robothandover mpan9@interchange.ubc.ca  Human-Robot Interaction Study http://tinyurl.com/robothandover mpan9@interchange.ubc.ca  Human-Robot Interaction Study http://tinyurl.com/robothandover mpan9@interchange.ubc.ca  Human-Robot Interaction Study http://tinyurl.com/robothandover mpan9@interchange.ubc.ca  Human-Robot Interaction Study http://tinyurl.com/robothandover mpan9@interchange.ubc.ca  Human-Robot Interaction Study http://tinyurl.com/robothandover mpan9@interchange.ubc.ca  Human-Robot Interaction Study http://tinyurl.com/robothandover mpan9@interchange.ubc.ca  B.1.1 Poster193B.1.2 E-MailRe: Call for Volunteers for a Human-Robot Interaction StudyThe Collaborative Advanced Robotics and Intelligent Systems (CARIS) Laboratoryis conducting a study in human-robot interaction. We are seeking healthy adult vol-unteers to help us better understand how humans hand objects to each other. Ouraim in this study is to characterize this behaviour and identify cues that humans useto coordinate object transfer. The knowledge gained from this study will be usedto develop robots capable of handing objects to humans and vice versa safely andintuitively.The study will be conducted in the Human Measurement Laboratory (roomX527) in the ICICS building, and will require approximately 1 hour of your time.Participants will be paired up and asked to hand a several household objects andforth while wearing motion capture markers. You will be asked to repeat the han-dover task with different objects each time. Non-identifying motion capture datawill be collected. The experiment will also be videotaped if both participants con-sent. All collected data will remain anonymous. You may sign up for this studywith another person as a pair or by yourself.The study will take place throughout the month of November. For more infor-mation regarding this study or to volunteer as a participant, please contact:Matthew Pan,orWesley Chan,Thank you for your interest.Matthew Pan, Ph.D. Student, UBC Mechanical EngineeringWesley Chan, Ph.D. Student, University of TokyoElizabeth Croft, Professor, UBC Mechanical Engineering194Last Revised: November 3, 2014 Rev 1B.2 ConsentThe University of British ColumbiaCollaborative Advanced Robotics and Intelligent Systems (CARIS) LaboratoryDepartment of Mechanical Engineering, UBC6250 Applied Science Lane, Vancouver, BC V6T 1Z4Tel: (604) 822-3147 Fax: (604) 822-2403Website: http://caris.mech.ubc.caHRI-Cues: Human-Robot Handover Study Consent FormProject Title: HRI-Cues: Human-Human Handover Study IIPrincipal Investigator: Dr. Elizabeth Croft,Co-Investigators and Contact Persons: Wesley Chanand Matthew PanFunding: This research is funded by the National Sciences and Engineering Re-search Council of Canada (NSERC).Purpose: The purpose of this study is to observe and kinematic behaviours inhuman-human handovers, and to explore how a human giver and receiver negoti-ate handovers of several objects. Results from this study will be used in subsequentresearch to improve the ability of robotic assistants to interact with non-expert hu-man users.Procedures: Before the actual handover experiment, you will be asked to fill out apreliminary questionnaire which asks you for some demographic information (e.g.,age, gender, etc). For the experiment, you will be paired with another participantto perform a series of handovers using several everyday objects while wearing a195jacket and a cap with motion capture markers. One participant will play the roleof the giver, while the other will be the receiver. During the experiment, both thegiver and the receiver will be standing. Both the giver and receiver will start withtheir hands placed at their side. On a verbal Go signal, the giver will retrieve anobject, and hand it over to the giver. After object transfer, both the giver and thereceiver will return to their start positions.The experiment will last approximately an hour. You may refuse to participatein this experiment and you may withdraw at any time. We will be recording motioncapture and video data, although the latter is not required for your participation.Potential Risks: Slight temporary fatigue from passing various objects back andforth.Confidentiality: No identifying information will be collected or stored with yourdata. Data collected from the survey will be stored on a password protected com-puter or a locked cabinet in the CARIS Lab, which has restricted secure access andis locked at all times. If you have any questions or concerns about what we are ask-ing of you, please contact the study leader or one of the study staff. The names andtelephone numbers are listed at the top of the first page of this form. If you haveany concerns about your rights as a research subject and/or your experiences whileparticipating in this study, you may contact the Research Subject Information Linein the UBC Office of Research Services at 604-822-8598 or if long distance e-mailRSIL@ors.ubc.ca or call toll free 1-877-822-8598.Consent: By signing this form, you consent to participate in this study, and ac-knowledge you have received a copy of this consent form.I agree to allow myself videotaped during this experiment (please circle one):YES NOName (print): Date:Signature:Last Revised: November 3, 2014 Rev 1196Appendix CSupporting Materials forInvestigating Human-to-RobotHandoversC.1 ConsentConsent for Participation in ResearchThis is an informed consent (”Consent”) provided by the individual identified be-low who has agreed to participate in a research study (”Study”) conducted byDisney Research.Study Title: A study of proprioceptive and kinodynamic behaviors duringhuman-robot interaction with visual and haptic feedbackPrincipal Investigator: Matthew Pan, Lab Associate197Address/Contact Info: Disney Research Los Angeles, 521 Circle Seven DriveLos Angeles CA 91201E-mail:Phone:Senior Advisors: Gu¨nter Niemeyer, Senior Research Scientist; Lanny Smoot, Dis-ney Research FellowOther Investigator(s): Vinay Chawda, Postdoctoral FellowPurpose of this StudyThe purpose of this Study is to observe user kinematic, proprioceptive and hap-tic behaviors in human-robot and human-human handovers of objects with andwithout the user simultaneously immersed in a virtual reality environment via ahead-mounted display. Results from this study will be used in subsequent researchto improve the ability of robotic assistants to interact with non-expert human usersduring co-operative object manipulation and enhancing virtual reality experienceswith robotics-based haptic interactions.Procedures for Study:Before the Study, you will be asked to fill out a preliminary questionnaire whichasks you for some demographic information (e.g., age, gender, etc.). You will per-form a series of object manipulation tasks while seated including, but not limitedto:• Picking up and putting down objects• Handovers and/or toss-catch exercises with a robot or an experimenter• Juggling, or attempting to juggle ballsDepending on the experimental condition, you may or may not be immersed in avirtual reality environment which will be rendering the physical objects that are tobe handed over or tossed by an experimenter or robot. In each trial, you will beasked to evaluate the interaction. For the purposes of tracking your motions within198the virtual environment, you may be asked to wear a jacket, cap and/or pair ofgloves with motion markers. Following the Study, there will be a debriefing surveyto collect feedback on your experience during the Study.Duration and Location of Study: 1 Hour, Disney Research Los AngelesParticipant Requirements:To participate in the Study, you must:• Be over 18• Never have had a seizure or blackout before and have no history of seizuresor epilepsyRisksDuring the Study, you will be donning a virtual reality head-mounted display thatwill fully occlude your vision of the physical environment. You may also be per-forming interaction tasks (e.g., object handovers) with a robot arm. To minimizethe risks of tripping/falling/collisions, you will be seated at all times when inter-acting with the robot and wearing the head mounted display. Additionally, objectsused will be lightweight. Other risks of participation include: motion sickness/-nausea, eye-strain, fatigue and repetitive stress injuries of muscles, joints and skin,tripping and dizziness. To reduce these risks, you will be able to take a break ordiscontinue participation in the Study at any time.BenefitsThere may be no personal benefit for you from your participation in the Study butthe knowledge received may be helpful to this research and may be of value tohumanity.Compensation & CostsThere is no compensation provided for this study.Privacy Policy/Ownership199The security, integrity and confidentiality of your information are extremely im-portant to us. We have implemented technical, administrative and physical securitymeasures that are designed to protect guest information from unauthorized access,disclosure, use and modification. From time to time, we review our security proce-dures to consider appropriate new technology and methods. Please be aware that,despite our best efforts, no security measures are perfect or impenetrable.The Study and its data, information and results as well as all ideas and sugges-tions communicated by you during your participation in the Study, will be ownedexclusively by Disney Research, and may be used, published and/or disclosed byDisney Research to others outside of Disney Research. However, your name, ad-dress, contact information and other direct personal identifiers will not be men-tioned in any such publication or dissemination of the research data and/or resultsby Disney Research. You will not receive credit in any materials in connectionwith the Study.Permission to Use Recorded Audio Visual DataDisney Research may want to record and use a portion of any video, audio or otherrecording for illustrative reasons in presentations of this Study for scientific or edu-cational purposes. Please initial below if you wish to consent to the following usesof your recorded data:I give my permission to the following uses of my recorded data:1. Use of my recorded data in research venues, including publications & publicpresentations, for illustrative and educational purposes:Please initial here: YES NO2. Use of my recorded data without additional identifying information (your name,address, etc) by other research groups or for other research purposes than thosedescribed herein in the future:Please initial here: YES NO200If you give your permission to any or all of the uses above, this means that youhave granted Disney Research the right to use and modify your recorded data asrequired by Disney Research for any reason related to the permitted uses of thedata.RightsYour participation in the Study is voluntary. You are free to stop your participationin the Study at any point. Refusal to participate or withdrawal of your consentor discontinued participation in the Study will not result in any penalty or loss ofbenefits or rights to which you might otherwise be entitled. The Principal Investi-gator may at his/her discretion remove you from the Study for any of a number ofreasons. In such an event, you will not suffer any penalty or lose any benefits towhich you might otherwise be entitled.This Consent form applies only to the Study described above and does notmodify or replace any other agreements you may have with Disney Research orThe Walt Disney Company.Right to Ask Questions & Contact InformationIf you have any questions about this Study, you should feel free to ask them now.If you have questions later, desire additional information, or wish to withdraw yourparticipation please contact the Principal Investigator by mail, phone or e-mail inaccordance with the contact information listed on the first page of this Consent.If you have questions pertaining to your rights as a research participant; orto report concerns regarding this Study, you should contact the Chair of the IRBCommittee, Steve Stroessner.Email:Phone: .Voluntary ConsentADULT PARTICIPANTBy signing below, I agree that the above information has been explained to me andall my current questions have been answered. I have been encouraged to ask ques-201tions about any aspect of this research Study during the course of the Study and inthe future. By signing this form, I acknowledge that I have read and I understandthis Consent, this Study has been explained to me and I agree to participate in thisresearch Study.PARTICIPANT SIGNATURE DATEPrint Name TelephoneStreet Address City, Sate, ZipEmail AddressI certify that I have explained the nature and purpose of this research Study tothe above individual and I have discussed the potential benefits and possible risksof participation in the Study. Any questions the individual has about this Studyhave been answered and any future questions will be answered as they arise.SIGNATURE OF PERSON OBTAINING CONSENT DATELast Revised: November 14, 2016 Rev 1202C.2 SurveyParticipant:Condition (circle one): 1 2 3 4 5 6 7 8Using the scale provided, how closely are the following attributes associated withthe robotic handover you just performed (Place a checkmark in the circle)?Not at all A moderate amount Very much so1 2 3 4 5 6 7Reliable o o o o o o oDexterous o o o o o o oAwkward o o o o o o oOrganic o o o o o o oCompetent o o o o o o oSociable o o o o o o oScary o o o o o o oKnowledgeable o o o o o o oEmotional o o o o o o oInteractive o o o o o o oClumsy o o o o o o oStrange o o o o o o oResponsive o o o o o o oAwful o o o o o o oCompassionate o o o o o o oDangerous o o o o o o oCapable o o o o o o oHappy o o o o o o oAggressive o o o o o o oFeeling o o o o o o oTrustworthywith fragileobjectso o o o o o o1 2 3 4 5 6 7Not at all A moderate amount Very much soLast Revised: April 2, 2017 Rev 2203


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items