UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Online experimenter : an evaluation of experiments conducted under local and remote conditions Zhang, Ying 2008

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Notice for Google Chrome users:
If you are having trouble viewing or searching the PDF with Google Chrome, please download it here instead.

Item Metadata

Download

Media
24-ubc_2008_fall_zhang_ying.pdf [ 3.32MB ]
Metadata
JSON: 24-1.0051242.json
JSON-LD: 24-1.0051242-ld.json
RDF/XML (Pretty): 24-1.0051242-rdf.xml
RDF/JSON: 24-1.0051242-rdf.json
Turtle: 24-1.0051242-turtle.txt
N-Triples: 24-1.0051242-rdf-ntriples.txt
Original Record: 24-1.0051242-source.json
Full Text
24-1.0051242-fulltext.txt
Citation
24-1.0051242.ris

Full Text

ONLINE EXPERIMENTER AN EVALUATION OF EXPEPJMENTS CONDUCTED UNDER LOCAL AND REMOTE CONDITIONS by YING ZHANG B.Sc., The University of British Columbia, 2005 A THESIS SUBMITTED IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE in THE FACULTY OF GRADUATE STUDIES (Computer Science) THE UNIVERSITY OF BRITISH COLUMBIA (Vancouver) September 2008 ©Ying Zhang, 2008 Abstract When conducting experiments, researchers traditionally work in a lab setting where they have access to specialized equipment and can meet their subjects face- to-face. This is definitely the way that traditional experiments in behavioral research are carried out. However, with the growing need for recruiting experiment participants from different social and cultural backgrounds, it may be difficult to design a traditional lab experiment for the research. Moreover, highly specialized lab equipment cannot always be easily accessed, and the costs of commuting present financial constraints. We explored the necessity of co presence in experimentation by evaluating a facility to determine whether experiments conducted in a non co-present context would achieve the same scientific outcomes as the ones conducted in a traditional lab context. The facility that we used was designed to allow researchers’ access to a sophisticated lab for the purpose of experimentation in educational research. There are two research goals explored in this thesis: (1) to evaluate the relative efficacies of running an experiment under co-present and non co-present conditions, and (2) to assess whether the participants’ experience differs in the experimental process between the co-present and non co-present condition. 11 Table of Contents Abstract .11 Table of contents iii List of tables xi List of figures ix Acknowledgements xi Dedication xiii 1. Introduction 1 1.1 What are the elements of traditional scientific lab experiments9 3 1.2 “Why is the co-presence of experimenters and subjects desirable9 5 1.2.1 Human to human communication 7 1.2.2 Quantitative and qualitative measures 7 1.2.3 Ethical concerns 8 1.2.4 Alternatives to face-to-face experiments 9 1.3 Goals and objectives of the research study 14 1.3.1 Assessment of experimental data 15 1.3.2 Assessment of experimental process 16 1.4 Summary of the research problem 17 1.5 Summary of research partners 20 1.6 Summary of terminology 21 1.7 Summary of our study 23 1.8 Summary of results 25 1.9 Summary of our contribution 26 111 1.10 Summary of the rest of the thesis .27 2. Related Work and Research Infrastructure 28 2.1 Related work and literature 28 2.1.1 Communication technology as research infrastructure 28 2.1.1.1 Emulating realistic face-to-face communication 29 2.1.1.2 Improving spatial realism 30 2.1.1.3 Additional modes of communication 31 2.1.2 Communication technology supporting education 32 2.1.2.1 Video-conferencing in online education 33 2.1.2.2 Tele-instrumentation or Tele-operation as educational technology 34 2.1.2.3 Supporting Tele-instrumentation in remote collaboration 36 2.1.3 Summary 37 2.2. Research infrastructure 37 2.2.1 Backbone of UQAM infrastructure-The BEST project 37 2.2.2 QUAM infrastructure — LORIT 38 2.2.3 UBC infrastructure — a video-conferencing lab 39 2.3 Tools 41 2.3.1 Click-To-Meet 41 2.3.2 NetMeeting 43 2.3.3 Access Grid with virtual network computing (VNC) 45 2.3.3.1 Access Grid 45 2.3.3.2VNC 46 3. Fitts’s Law Studies of Directional Movement between Two Target Positions 49 iv 3.1 Index of difficulty in Fitts’s formulation.52 3.2 Characteristics of Fitts’s Law formulation 53 3.3 Fitts’s Law in interaction studies 55 3.4 An example of a Fitts’s Law Experiment 56 3.4.1 Practice sessions 58 3.4.2 Experiment sessions 59 3.4.3 Experiment software 61 4. Study Design 62 4.1 Experimental Design 64 4.1.1 Hypotheses 65 4.1.2 Conditions 67 4.1.3 Tasks 68 4.2 Participants 70 4.3 Apparatus 71 4.3.1 UBC Apparatus 72 4.3.1.1 Remote condition 72 4.3.1.2 Local Condition 72 4.3.2 UQAM Apparatus 77 4.3.2.1 Workstation hardware 78 4.3.2.2 Workstation software 79 4.3.3 Experiment Software 79 4.4 Setup 79 4.4.1 Remote setup 79 4.4.1.1 Video-conferencing software setup 80 V 4.4.1.2 Video camera setup.81 4.4.1.3 Desktop sharing setup 81 4.4.2 Local setup 82 4.5 Data Collection 82 4.5.1 Logging 82 4.5.2 Questionnaire 83 4.5.3 Interview 84 4.6 Procedure 85 4.7 Summary 94 5. Results 95 5.1 Fitts’s Law Performance 96 5.2 Analysis of Slopes and Intercepts 100 5.2.1 Intercept coefficient Ai 101 5.2.2 Slope coefficient Bi 102 5.2.3 Performance time 103 5.3 Errors 103 5.4 Analysis of time to run a Fitts’s Law Experiment 104 5.4.1 Total time spent at each task block at each index of difficulty 105 5.4.1.1 Within-subjects effect 107 5.4.1.2 Between-subjects effect 108 5.4.1.3 Interaction 108 5.4.2 Total time to complete the Experiment in the remote and local conditions 110 5.5 Questionnaire analysis ill vi 5.5.1 Interaction Survey.111 5.5.1.1 Measures between conditions based on roles 112 5.5.1.2 Participants’ experience within each condition 114 5.6 Interview analysis 115 5.7 Summary 118 6. Discussion 120 7. Conclusions and Future Work 124 7.1 Conclusions 124 7.2 Lessons learned from our Study 125 7.3 Contribution 127 7.4 Challenges 128 7.5 Future work 128 7.5.1 Study design 128 7.5.2 Use of Chat, Video, and Audio 129 7.5.3 Configuration of Equipment 129 7.5.4 Possible modification of remote lab web interface 130 Bibliography 132 Appendices 137 Appendix A 137 Appendix B 158 Appendix C 160 Appendix D 166 Appendix E 170 vii List of Tables Table 3.1 Index of difficulty and movement time.53 Table 3.2 Design of our Fitts’s Law Experiment 61 Table 4.1 Statistical interpretations of outcomes 66 Table 4.2 Sections in the questionnaire 84 Table 4.3 Semi-structured interview schedule 85 Table 4.4 Our Study design 94 Table 5.1 r and r2 values obtained from remote and local conditions 98 Table 5.2 Descriptive statistics of r and r2 98 Table 5.3 Intercept Ai and slope Bi (one anomalous value is highlighted) 101 Table 5.4 Descriptive statistics for slope and intercepts 101 Table 5.5 Descriptive statistics of error rate in remote and local conditions 104 Table 5.6 Mixed between and within Study: condition by index of difficulty 106 Table 5.7 Descriptive statistics of performance time between four different indices of difficulty and two Study conditions 107 Table 5.8 Descriptive statistics of total time spent in each task block corresponding to each index of difficulty 107 Table 5.9 Descriptive statistics of total time spent to complete the all of the twelve tasks 108 Table 5.10 Descriptive statistics of total time spent in remote and local Experiments 110 Table 5.11 Summary of results 119 Table 6.1 Patterns of interaction 120 viii List of Figures Figure 2.1 LORIT lab and equipment of UQAM 39 Figure 2.2 UBC video-conferencing lab and equipment 40 Figure 2.3 Click-To-Meet window 42 Figure 2.4 Mosaic view 43 Figure 2.5 Screenshot of NetMeeting 45 Figure 2.6 Networks connection for videoconference experiments 47 Figure 3.1 Fitts’s Law task 50 Figure 3.2 a Schematic drawing of our Fitts’s Law task --- Target at its starting position 58 Figure 3.2 b Schematic drawing of our Fitts’s Law task --- Target at a surrounding position 58 Figure 3.3 Four block windows with four different target sizes 60 Figure 4.1 Configuration of the Study in the local condition 67 Figure 4.2 Configuration of the Study in the remote condition 68 Figure 4.3 UBC workstations in remote condition 72 Figure 4.4 LORIT interface 75 Figure 4.5 UBC workstations in local condition 76 Figure 4.6 LORIT Lab 77 Figure 4.7 Video windows 81 Figure 4.8 Procedure of local Experimenter 87 Figure 4.9 Procedure of local Subject 89 Figure 4.10 Procedure of remote Experimenter 90 ix Figure 4.11 Procedure of remote Subject 92 Figure 5.1 Scatter plot for remote and local conditions 99 Figure 5.2 Total time spent to complete the all of the twelve tasks 109 x Acknowledgements I would like to express my deepest and most sincere gratitude to my supervisors Dr. Kellogg Booth and Dr. Brian Fisher for providing me with the motivation and the ideas, and for their important support and encouragement to pursue my research. I would like to extend my gratitude to the research team at the L’Universite du Quebec a Montreal, including Gregory Petit, Mario Archambault, and Racha Ben Au, for providing me access to the LORIT facility and for the technical support that made my research succeed in such a unique environment. Special thanks to Professor Auder. Dufresne from University of Montreal. As the head of the LORIT@d (Econtrol) development, she designed the interface that directly supported the research, and organized the participation from Montreal in my experiments. I would like to thank my second reader Dr. Jacqueline Bourdeau, for providing invaluable comments and suggestions on my thesis. I am also grateful to my lab manager Ron Fussell for his kindly support on everything during the project. I would like to express my warm and sincere thanks to Dr. Barry Po, who noticed my potential and encouraged me to pursue post-graduate studies in the field of Human Computer Interaction. xi I would like to acknowledge my fellow students and friends for their support and feedback. This research could not have been completed successfully without the help of Stiliman Jaguar, Zhangbo Lu, Man Hon Chan, Jian Xu, Gang Peng, Hao Shuang, Garth Shoemaker, Clarence Chan, Miranda Poon, Bernard Ng, Albert Huang, and Kevin Ming. I also wish to thank Ms. Sundari Bala, Ms. Emily Chan, and Mr. Geoffrey Foster for revising the English of my thesis. My warm thanks are also due to Ian, Wendy, and Lily. They made my years at the University of British Columbia an enjoyable and memorable experience. Last, but not least, my thanks go to my parents for their tireless encouragement, endless support, and unconditional love. Without their support and understanding it would have been impossible for me to finish this thesis. Special gratitude is due to my cousins and their families for their loving support. The research reported in this thesis was financially supported by The University of British Columbia through the Media and Graphics Interdisciplinary Centre (MAGIC), the Canadian Foundation for Innovation (CFI) through the SCORE project, the Canadian Network for the Advancement of Research, Industry, and Education (CANARIE) through the BEST project, and the strategic network program of the Natural Science and Engineering Research Council of Canada (NSERC) under the Network for Effective Collaboration Technologies through Advanced Research (NECTAR). xii Dedication To my parents and my grandmother xiii Chapterl Introduction Many scientific research fields rely on experimental studies, for recording and modeling species behavior, observing and gaining insight into scientific phenomena, or determining and evaluating the effects of innovations (Preece, Rogers, & Sharp, 2002). This is particularly true for the field of human-computer interaction (HCI) in computer science. In this thesis we describe a system for conducting experiments with human subjects to evaluate the benefits of educational technology; specifically we investigate the feasibility of using advanced communication technology to support experiments by doctoral students conducting research on educational technology in a distance education program. For such experiments, doctoral students would normally work in a research lab with access to specialized educational technology (computers, software, and trained laboratory technicians), where they could work face-to-face with the subjects of their experiments. This is the traditional way that behavioral research is conducted when evaluating new educational technology; the problem this poses is the necessity for co-presence, whereas most of the doctoral students’ other activities in distance education do not depend on co-presence. The costs (travel, time, inconvenience) associated 1 with face-to-face experiments are those that we would like to avoid. For this reason, the Laboratorie observatoire de Recherche sur le Télé-apprentissage (LORIT), was developed at the University of Montreal! UUniversité du Québec a Montréal (UQAM) with advanced communication technology that allows doctoral students remote access to the laboratory to conduct their research experiments. Our contribution is an evaluation of this facility, to determine whether remotely conducted experiments can achieve the same scientific outcomes as do those conducted face-to-face. This first chapter provides background information about the ways that experiments are conducted, especially those with human subjects using technology, as it is common in HCI and in education. We focus particularly on the trade-offs between face-to-face experiments and those in which the experimenter is working remotely from the laboratory and from the subjects. This is precisely the situation for LORIT; doctoral students use the facility to learn how to conduct experiments, and once they have learned, they run experimental investigations for their dissertation research. The goal is to be able to do this remotely, taking advantage of the communications technology provided by LORIT. After describing the problem, we present the research hypotheses and then provide an overview of the study methodology and a summary of its results, each of which is explained more fully in subsequent chapters. 2 1.1 What are the elements of traditional scientific lab experiments? There are several elements needed for a traditional laboratory experiment. Lab equipment in the facility is used to perform scientific analysis, measurement, or observation; lab technicians provide support and instruction, reinforce lab safety protocols, and monitor the safe and standard operation of lab equipment, while the experimenters’ role is to investigate the objectives of their research. These essential elements are typically seen in physics and chemistry labs, as well as in many other laboratory settings. However, in behavioral research, such as in biology or psychology, the subject is a further necessary element. Subjects can be “any organism (animal or plant) that is observed for purposes of research” . As the term HCI (human computer interaction) suggests, human subjects are essential for HCI experiments. Technology has been increasingly used as a strategic tool for lab experiments. In HCI, researchers experiment with human subjects with various objectives, using advanced technology, including specialized computer software and hardware, computerized lab equipment or computer-controlled devices to support their The Language of Psychology - Dictionary and Research Guide. http:I/www. 1 23exp-health.com/psychology/ 3 experiments. Examples are evaluating a particular vehicle navigation technology in a computer-simulated environment, investigating defects and deficiencies of a piloting interface in a virtual piloting environment, or using an artificial speech system to convey pre-recorded experimental instructions to human subjects. All of these involve human subjects and their interaction with computer technology, often in formal HCI experiments. In the field of education, technology has been integrated extensively into instructional resources. As researchers in education have discovered, proper use of computer technology in education “can increase opportunities for students” (Svedkauskaite & Reza-Hernandez, 2008, p. 1). Experimental investigation of the interrelations between learning technology and students has evolved rapidly in a number of educational disciplines. Doctoral students in education have used technology to conduct human-subject-oriented experiments for various purposes, such as measuring the effectiveness of educational software being developed for targeted student groups, or evaluating learner-to-learner or educator-to-learner communication strategies based on computer-mediated technology. In educational research, we see more researchers conducting experiments, recruiting subjects and instructing them how to interact with computer technology, observing how they can benefit from it and how their learning experiences can be improved by using technology in education. 4 1.2 Why is the co-presence of experimenters and subjects desirable? Co-presence in the laboratory of all of the participants in an experiment is desirable for a number of reasons. Laboratory safety provisions traditionally call for chemistry and physics experiments to be conducted in a controlled environment, with lab technicians ensuring safety; safety equipment in a lab protects experimenters and subjects alike from potential dangers or hazards, while also reducing any potential harmful effects to the public at large and to the environment. In HCI, co-presence is favoured by researchers most of all for its convenience. Experiments are constrained by practical issues, including tight schedules, low budgets, finding suitable poois of subjects, and getting necessary evaluation resources. Furthermore, these constraints are sometimes interrelated; for example, in user study scheduling, a typical user study session can take from minutes to hours. Some user studies require subjects to participate in more than one study session, for reasons of follow-up or resource availability. With co-presence, some constraints on scheduling and coordination of subject presentation can be removed. Typically, co-presence is convenient when booking evaluation resources; with co-presence, desired equipment or lab facilities, lab personnel or technical support staff, study subjects, and researchers are all geographically located at the same place and in the same time zone, reducing the costs of 5 commuting, retrieving experimental resources, hiring, screening, and scheduling of potential subject candidates and increasing the convenience of equipment access, setup, trouble-shooting, and the tangibility of technical support. Thus, co presence makes conducting experiments more cost-effective, since it offers direct physical access to available resources. The most advantageous factor favouring onsite user experiments is the ability to restrain certain aspects (Moløkken-Ostvold, 2005), ensuring that experimental results are reproducible under the same conditions and with access to similar equipment. An example would be keeping temperature at a certain level to maintain the stability of a chemical reaction in order to produce an expected product. Particularly in HCI, a popular evaluation method is the controlled study, in which researchers and developers make observations while subjects attending a lab session are asked to perform a set of predetermined tasks to evaluate the usability of a piece of software or hardware designed for an intended user population. (Preece et al., 2002). Using this approach, researchers can control confounding factors and possibly improve the usefulness of the results of their evaluation in terms of the desired outcomes. The various forms of human computer interaction incorporate the following additional factors that make researchers inclined to prefer co-presence. 6 1.2.1 Human to human communication In user evaluation, it is easier to observe and communicate with subjects if the researcher can interact with them face-to-face. From the researchers’ viewpoint, communication seems more effective when it is done face-to-face, since they can easily see a smile, a frown, or other facial expression that might indicate a subject’s thoughts, feelings, opinions, and understanding, at the same time clearly hearing the tone of voice or pauses in speech. Based on these rich forms of feedback, researchers are able to extract potentially interesting phenomena and take them into account as their designs evolve. With co-presence, the communication flow between subjects and experimenters is not subject to any physical barrier; issues or concerns can be directly and instantly delivered via oral expression or eye contact; any potential subject actions that could influence the study can be noticed and dealt with right away. 1.2.2 Quantitative and qualitative measures Even though “field studies and heuristic evaluation have grown in prominence” (Preece et al, 2002, p. 341), usability testing is still regarded as an important evaluation technique in HCI. This kind of evaluation measures the response of a subject to an interface or a study design in both quantitative and qualitative terms. Quantitative measures evaluate a subject’s performance by, for instance, recording the number of errors and the amount of time needed to complete a set of carefully designed tasks. Qualitative measurements are elicited via questionnaires 7 or interviews. “The defining characteristic of usability testing is that it is strongly controlled by the evaluator (Mayhew, 1999) ... typically, tests take place in laboratory-like conditions that are controlled” (Preece et a!, 2002, p. 341). The researcher is able to be in charge when running usability testing in a lab with co presence. When all factors are accessible, researchers can decide beforehand what objective measurements and subjective measurements to collect, while also being free to focus on any additional objectives or subjective measurements that seem interesting and worthy of further evaluation. This enables them to have a better understanding of their subjects’ experimental experience, at the same time identifying further potential research topics. Objective data and subjective data collection is made easy, because user performance and user feedback can be directly observed and measured. 1.2.3 Ethical concerns There are a number of factors that need to be taken into consideration before deciding whether a traditionally co-located lab experiment can be performed remotely. Since HCI evaluation involves human subjects, it inevitably leads to ethical issues. In typical usability testing, subjects are asked to provide their personal data for demographic purposes; during the experiment, they may be monitored, videotaped, and voice-recorded. The personal data of subjects along with recordings of their images and sounds are used for research purposes; these can 8 act as potentially powerful evidence, but may impact on the privacy of the subjects. Thus, researchers should not only aim to have their usability or evaluation expectations fulfilled in the study, but they must also obey ethical guidelines and avoid any psychological or physical harm to their subjects. In a face-to-face setting, subject frustration or fatigue can often be observed. In onsite or co-present contexts, ethics obligations can be easily complied with, since the personal images of participants are exposed to only a limited number of personnel physically attending a lab session and only researchers have access to participants’ personal data. Any personal data collected is permitted by ethics guidelines to be used, analyzed, and processed for stated research purposes only. In addition, data storage is usually centralized in a safe and secure venue on site. 1.2.4 Alternatives to face-to-face experiments Because of these concerns, and for other reasons as well, co-presence has so far been the dominant method of conducting experiments, since it can ensure that throughout an entire experiment the above concerns are not compromised. Nevertheless, in some fields, such as engineering education, the frequency of co presence in experiments is declining. Instead, the use of simulation has increased; research suggests “physical experiments are costly both to build and maintain” (Foss, Eikaas & Hovd, 2000, p.2944), pointing out the disadvantages of persisting with co-presence in experiments. Experiments involving human subjects and 9 premium equipment are tightly coupled with schedules; in the case of a remotely located lab, a researcher has to physically travel to the lab to run experiments. If the researcher discovers some defects with his study in the first few experiment sessions, he has to cancel the rest of the subject schedule, come back to his original location and modify his experiments, re-schedule all the experiments, and then return to the remote venue to continue his study. This kind of situation demonstrates how costly and difficult it is when co-presence has to be maintained in an experiment. Sometimes, not all elements need to be co-located. Examples would be a highly instrumented lab facility located far away from both experimenters and subjects, or a knowledgeable technician who is in the middle of a vacation trip but is required to trouble-shoot a piece of lab equipment located somewhere far away. Other examples would be separation of experimenters and subjects by geographical distance. Furthermore, some research activities may need to recruit participants from different social and cultural backgrounds, and these kinds of candidates might not be co-located. For example, when gathering survey information of people in an exotic environment, the subject-hiring process could be challenging if the potential subject pool is restricted by geographical boundaries. If any of these situations emerge during preparation for a user study, the presumption of it being run as a traditional face-to-face study no longer holds. 10 Can an experiment be run without the co-presence of every element? Researchers in human computer interaction have been exploring the possibility of conducting experiments without co-locating all their elements. With the availability of computer networks, personal computers, desktop workstations and web cameras, powered by commercial software products and Internet or web services — such as LogMeln, Click-to-Meet, and Net-meeting — a number of communication technologies have been developed and used to support remote lab access for experimental research and study. Researchers have used web cameras, wireless technology, and web services to create applications that allow users to control and configure equipment remotely in an automated lab (Zhang, Ball, C. Extine, & W. Extine, 2007). This kind of technology gives users the ability to conduct a live lab facility operation painlessly and be able to “experiment and monitor experiment results in real time” (Zhang et al., 2007, p. 231). Other researchers have created virtual labs that “enbance students’ learning experiences through a simulation environment” (Zhang et al., 2007, p. 231), such as the “LabVIEW” virtual instrument developed by Zaimovice-Uzunovice, Lemes, and Petkovice (2001) and Gu, Zhu, Dong, Shi, and Wang’s virtual circuit laboratory (2005). By combining research and commercial technologies, computer systems can now support a user with access to a remote lab, regardless of the users’ location, as long as an Internet connection can be established. 11 The advantages of running experiments in a non-co-present context include having equal access to a larger subject pooi than under co-presence. Because researchers are not physically present in a post-secondary distance education institution, subjects are often recruited through a designated website of a supporting or sponsoring university. This opens a larger subject population pooi to researchers, as though they were physically present in the institutions. As resource sharing is encouraged in the academic and research field, running experiments in a distributed way can also enable researchers to easily access a premier lab facility through its web-based interface that is supported by their university or other participating institution. In addition, remote experimentation frees researchers and technical support personnel from commuting, traveling costs and other constraints. Perhaps the chief advantage is that researchers can prototype and pilot their experiments with their remote co-workers and subjects, by means of distributed experimentation. However, being non co-present also poses some disadvantages. With experiments conducted in a distributed way, data collection is no longer centralized. For instance, visual and audio information might be collected and stored separately at each location. Any uncoordinated organization or destruction of the information “increases the chance of improper use or disclosure of subject data” (Patrick, 2007, p. 7). 12 Before web technology was introduced to support distance education or remote interaction, information collected from an experiment was kept locally. With the introduction of web technology, visual and audio information can be broadcast or transmitted on line. This brings in another potential ethical risk, since the information can be intercepted, copied or stolen in real time. Furthermore, to ensure that a distributed lab experiment is done properly, the network connection is the most important element. A network outage could ruin an entire study session, as manipulation of instruments and objective communications, such as visual and audio data, will be lost immediately. As researchers have suggested, technology support is one aspect of remote experimentation, but there is another important aspect to be considered. Without physical presence at a lab, how can the actual feeling of using the lab be recreated? (Rohrig & Jochheimm, n.d.). In a situation where remotely accessing a lab facility is being done via virtual reality or by remote control of a lab experiment (Schmid, Eikaas, Foss, & Gillet, 2001, p.1), distributed teams collaborating on a remote experiment “required a thorough preparation in terms of research protocol as well as of technical support” (Bourdeau, Dufresne, Tchetagni, & Ben Au, 2006, p. 8). As Bourdeau et al. have pointed out, “a high quality of audio and video is essential for participants to feel totally immersed or even to ideally forget the distance and the technology” (2006, p.8). This brings out another key fact: high-quality network performance is required to achieve 13 smooth remote experimentation. As distance teaching and distance learning gain a stronger presence in post-secondary education, the need to have access to a remote or distributed sophisticated lab equipment facility increases, and researchers are no longer willing to be constrained by geographical separation. These and other innovations in experimentation raise an important question: With the level of advanced technology available currently, can a remote experimentation facility support any traditional experiment or study in HCI as effectively as it can be done locally? This is the question that we are trying to answer in this thesis. 1.3 Goals and objectives of the research study The initial concern that motivated this research was the necessity for co-presence. Research has verified that co-present condition is still the “gold standard” preference (Hauber, Regenbrecht, Billinghurst, & Cockburn, 2006, p. 413). There have been ongoing desires and attempts to overcome geographical distance. Researchers still hope to run their research activity with remote teams while achieving the same results as though the same activity were being performed with co-located teams. We can assess the results of an experiment on two levels. The first level is fairly literal. We can look at the data derived from a remote experiment and compare it 14 to that from a face-to-face or co-located experiment to see whether there are differences attributable to the experimental environment. At a higher level, we can ask whether the process of conducting the experiment differs, either in tenns of objective measurements of what happened or in terms of subjective measurements of how the participants felt about the process. Both levels of analysis are of interest, especially for researchers in HCI where “user experience” is always a relevant component of any system. We had the following expectations for what we might find. These are stated informally here, but will be re-stated in later chapters, as formal research hypotheses. 1.3.1 Assessment of experimental data As the “face-to-face” condition is regarded as the most effortless experimental condition (Hauber et al., 2006), we expected that in an experimental context, local participants (where “local” here means co-present or face-to-face) might outperform those who are remote (where “remote” here means distributed or non co-present). The types of differences that might be seen would be faster task completion times by local subjects, higher completion rates, or more accurate performance (measured either by a performance scale or by error rates). There might also be differences in terms of the statistical properties of the data, such as less variation 15 within or between subjects, or, perhaps, whether or not statistical significance was obtained. An example of these can be illustrated by a Fitts’s Law experiment (see Chapter 3 for a full description of Fitts’s Law and the types of experiments that are conducted to verify it). If the co-located and remote conditions differ in terms of data, we might see different coefficients arising from regression testing of the data that could be the result of faster performance by subjects in one of the conditions, or a higher rate of errors, or, in extreme cases, simply a failure of statistical confirmation of the hypotheses. 1.3.2 Assessment of experimental process Even if there were no differences in the data obtained from co-located and remote experiments, the processes might differ in a number of meaningful ways. We expected that local participants might experience richer human-to-human interaction than remote participants. This could be measured in terms of awareness of presence, ease of speaking to or hearing, or overall satisfaction of the experiment during conversations among participants. We suspected that there might be a less attention shift on local participants, and a higher load on remote participants. This could lead to differences in data (in which case the process differences might be what were causing the differences in data) or it might mean that either experimenters or subjects were not satisfied with 16 the experience (subjects might withdraw from a study, experimenters might form an unduly negative opinion about the benefits of conducting a study weighed against the costs of conducting it). We also expected that the support of a lab assistant might be a necessity under the remote condition but optional in co-presence, for dealing with both technology- oriented and interpersonal issues. 1.4 Summary of the research problem The goal of our research project is to determine the efficacy of conducting experimental studies of subjects where the subject and the experimenter are not co-located, in contrast to the condition where they are co-located. The particular situation of interest is that of doctoral students researching in the field of educational technology who are assessing the benefits of an innovative technology developed to support the learning of a specific subject. These students need to conduct experimental evaluations of subjects who use the technology in learning tasks, in comparison to a control group that does not use the new technology. When doctoral students are enrolled in distance education programs that do not provide them easy access to the necessary laboratory facilities, it may be difficult for them to conduct their research. Examples might be software developed by a 17 doctoral student to support acquisition by students of certain mathematics skills. The doctoral student would normally conduct laboratory experiments with subjects using facilities in the department in which the doctoral student is enrolled. In a distance education setting, where the doctoral student might not have access to such a laboratory, because travel time and expense would be prohibitive, there is a need to consider alternative mechanisms for conducting the experiments. Our research partners in LORIT at the University of Montreal / L’Université du Québec a Montréal (UQAM) have developed a virtual laboratory facility that allows an experimenter (a doctoral student) to conduct experiments with subjects (drawn from the usual subject pools at the university) with the experimenter operating remotely through a networked connection, while subjects are located in a facility at the university that is staffed by a research technician who works with the experimenter. The virtual lab facility originated from the BEST project (Banc d’Essai de Services de liaisons optiques pour un environnement de Téléformation doctorale) at the University of Montreal as a component for a virtual doctoral training environment. Cooperating with our partners at UQAM, we studied and evaluated the virtual lab facility, which is an interface supporting remote operation of lab equipment, data collection, and multi-media recording. This interface belongs to an experimental research lab called Laboratorie observatoire de Recherche sur le Télé-apprentissage (LORIT); it is backed by a fiber optical 18 network provided by the Canadian Network for the Advancement of Research, Industry and Education (CANARIE). Because one of the intentions in creating this online virtual doctoral training environment was to utilize the LORIT lab to support doctoral student immersion and access to instrumentation for experimentation, this lab and its virtual facility provided the technology base for our study. We collaborated with the LORIT team and utilized the LORIT lab and its online web interface to run our study. The study we are conducting for this thesis is designed to answer the question: Does the virtual laboratory provide an adequate environment to conduct experiments of the type just described? In our current work we deem an environment to be adequate if the outcome of the experiment (results) are similar to what would be achieved in a traditional face-to-face experiment using the same procedures, and if doctoral students obtain the same degree of experience from conducting the experiment. The second requirement recognizes the training aspect of a doctoral student’s experimental work. The results of an experiment are of course a primary outcome of the experiment, but an equally important outcome is the experience gained by the doctoral student in conducting research with human subjects. Our study aims to address the first requirement: Are the results comparable when the experimenter and subject are face-to-face and when the experimenter is located remotely? In making this comparison, we consider both 19 differences in data and in process. Once we are confident the first requirement has been met, subsequent studies can address the second requirement. 1.5 Summary of research partners This highly-interdisciplinary research was conducted with a number of partners, many of whose contributions have been noted above. We summarize here the various organizations that participated and briefly describe their role. The individuals who were involved are listed in the Acknowledgment section. LORIT --- LORIT is a distance learning engineering research lab at LICEF, a research centre for learning environments at Télé-université and the University of Montreal (TELUQ-QUAM). The LICEF research centre supported our study by granting us physical access to the LORIT facility, online access through their e control software, and excellent research and technical support. SCORE ---The SCORE program is led by Dr. Jacqueline Bourdeau at TELUQ QUAM. The program has a partnership with the University of Montreal (UQAM) and the University of British Columbia (UBC). SCORE is supported by the Canada Foundation for Innovation, which, together with Réseau d’informations scientifiques du Québec and BCNet, provided us with a dedicated optical network “lightpath” connection. 20 BEST --- BEST is a project under the SCORE research program. The project is led by Dr. Jacqueline Bourdeau at TELUQ-QUAM. The project is a joint project with QUAM and UBC. MAGIC --- The Media And Graphics Interdisciplinary Centre conducts interdisciplinary research on computer-based and computer-associated media at The University of British Columbia. MAGIC worked closely with BEST and SCORE to establish the research partnership with LICEF at TELUQ-QUAM. It also worked closely with CANARIE for partially funding our research. NECTAR --- The Network for Effective Collaboration Technologies through Advanced Research is an NSERC Research Network focused on collaboration technologies. It provided us with the major source of funding for our research. 1.6 Summary of terminology Our study is an experiment about conducting experiments; therefore, there is a lot of potential for confusion in the terminology describing it and the experiment embedded within the study, which is the activity (task) that participants in our study will engage in. Part of the terminology problem is that some participants in our study will play the roles of experimenter (i.e., they will carry out the tasks that a hypothetical doctoral student might do in a real experiment) while other participants will play 21 the roles of subjects. This could lead difficult differentiating between the terms “participant” (someone in our study) and “subject” (also someone in our study, but playing a specific role, that of subject in an experiment within an experimental study). To avoid a “Who’s on first?” scenario (Abbot & Costello, 1939), we will adopt specific terminology throughout the remainder of this thesis, as follows: STUDY — The research we are engaged in, which is an evaluation of a classical human computer interaction theory modeling the act of pointing (Fitts’s Law), will be used as a vehicle to test our hypotheses about the possible differences between co-present and remote environments for conducting experiments. RESEARCHER — The thesis author, any member of the supervisory committee, or the LORIT technical staff involved in conducting and analyzing the results of the STuDY. PARTICIPANT — Any person who is recruited to take part in the STUDY. EXPERIMENT — The task that PARTICIPANTS carry out in the STUDY (a simple Fitts’ s Law experiment that gathers data about the time required to point to a target and return back to a starting position using a computer mouse). SUBJECT — Any PARTICIPANT whose role is to use the experimental apparatus (a computer mouse) to execute Fitts’s Law trials. 22 EXPERIMENTER — Any PARTIcIPANT whose role is to direct a second PARTICIPANT who is playing the role of SuBJECT. Whenever any of these words are in SMALL CAPS they are intended to stand for precisely the above terms. If we think of our research study as being at the meta level, and the activity we study as being at the domain of study, the intent is that STUDY, RESEARCHER, and PARTICIPANT all pertain to the meta-level, whereas ExPERIMENT, SuBJECT, and EXPERIMENTER all pertain to the domain. Hopefully keeping these distinctions in mind will help readers avoid confusion. 1.7 Summary of our study In order to draw comparisons between the outcomes of experiments conducted in local and distributed settings, we chose an experiment for which we were fairly confident that we would be able to achieve clear results in both local and distributed settings. We assumed that Firts’s Law is a very well-known motor control task that is easily replicated in a laboratory if only a modest level of care is taken to follow the experimental procedure for trials. As such, it serves as a good initial measure of the degree to which a virtual laboratory provides the level of support necessary for doctoral students to carry out the type of research described earlier. 23 The primary dependent measurement in our STUDY will be the r-squared correlation parameters obtained from the data generated by each ExPERIMENTER- SuBJEcT pair of PARTICIPANTS. The independent variable is whether the ExPERIMENTER and the SuBJEcT are co-located during their ExPERIMENT. We will also look at secondary dependent measures, the regression coefficients a and b in the standard formulation of Fitts’s Law T=a+blog(A/W+ 1/2) where A is target amplitude (amount of movement required to acquire a target) and W is target width. The goal of our STUDY was not to test Fitts’s law, which is well established; however, by analyzing the r-squared values we hope to determine if there are quantitatively measurable differences between the control condition (co-located EXPERIMENTER and SUBJECT) in our STuDY and the experimental condition (non- co-located ExPERIMENTER and SuBJECT). The data from a Fitts’s Law experiment is usually subjected to a linear regression test to fit the equation above. A large r squared is suggestive of a successful EXPERIMENT, a small r-squared is suggestive of a poorly conducted experiment; it is a measure of the statistical validity of the data, and thus a primary metric for the “quality” of the experimental data. The coefficients a and b are measures of the data itself. Looking at these as well as at r-squared may alert us to the possibility of equal statistical validity but different conclusions being drawn from the two sets of data. 24 We also looked at the experimental process in both conditions using a variety of objective and subjective measures. We explain these in more detail in later chapters. We evaluated the efficacy of conducting the EXPERIMENT under distributed conditions and compared it to conducting an experiment under co-located conditions. We used Fitts’s Law to study and compare the performance of SuBJECTS under different circumstances: with and without the co-presence of ExPERIMENTERs and SUBJEcTs. In each study condition, one SUBJECT was paired with one ExPERIMENTER. Under the co-present condition, SuBJEcTs performed the EXPERIMENT under the instructions of their ExPERIMENTERs face-to-face. Under the distributed condition, we used a video-conferencing infrastructure via a high- speed network facility. In this context, we conducted the EXPERIMENT with SUBJECTS present at a research lab in a university located in Eastern Canada, and at the same time with EXPERIMENTERs present at a video conferencing lab at the University of British Columbia. 1.8 Summary of results Quantitative measurements arising from our experiments fell within our prediction; we obtained statistically similar performance between the traditional face-to-face lab condition and the distributed lab condition, both in terms of the 25 data (the various parameters derived for Fitt’s Law and the statistical validity of those parameters). In addition to quantitative results, we looked at qualitative measurements. Surprisingly, our results suggest that EXPERIMENTERS felt more comfortable interacting with SuBJEcTS in the remote method than they did in the co-present condition. This indicates that EXPERIMENTERS did experience certain benefits under the distributed condition, although the distributed condition needs to be undertaken with caution as it may encourage in the ExPERIMENTER a lack of engagement in the task. Evidence for this was observed and reflected in qualitative analysis of data collected from the distributed condition. Our analysis suggests that the possible cause of these problems is mainly induced by ExPERIMENTERS’ over-concentration on certain procedural details in the remote condition. 1.9 Summary of our contribution To compare the performance of SuBJECT-ExPERIMENTER pairs under the two conditions, we carried out analysis in several ways to evaluate their cognitive load, interactivity, and task measurements separately and in combination. The results of the study will inform the next round of remote experiment study design. We believe that our results and some of our recommendations will positively affect collaborative experiments in a remote context. 26 A secondary benefit is that by utilizing an experimental research lab, LORIT, to conduct our user study under the remote condition, we implicitly evaluated the usability of LORIT’s web interface (LORIT(äD or Econtrol) and provided our research partners, the LORIT team, with valuable feedback about the interface. 1.10 Summary of the rest of the thesis In Chapter 2, we describe related work regarding the use of communications technology as research infrastructure. We discuss the selection of an appropriate video-conferencing system for LORIT. Chapter 3 gives a brief introduction to the classical theory of Fitts’s Law, which we used as the basis for simulated experiments in our study to evaluate the effectiveness of LORIT. The complete study design is described in Chapter 4. Chapter 5 presents the results, and Chapter 6 discusses some interesting observations drawn from the study. Chapter 7 presents the lessons learned from our experience to date and possible directions for future work. The appendices contain samples of the materials used in the study. 27 Chapter 2 Related Work and Research Infrastructure 2.1 Related work and literature With recent advances, communication technology has become more important and popular in the academic field. In particular, these innovations have revolutionized methods of research and education; by using these technologies, researchers can now conduct their field studies without leaving their research lab. Students can now have access to school lab equipment and perform their experiments without having to commute physically to their institution. In this context, we will first consider related studies into the use of communication technology in general support of research infrastructure, and then narrow down our focus to its support of online experimentation. 2.1.1 Communication technology as research infrastructure With advanced high quality communication systems as research infrastructure, “there have been increasing opportunities for researchers in different fields to collaborate with each other” (Matsukara, Koyamada, Tan, Karube, & Morlya, 2004, p. 205). Treloar points out that use of technologies increases “volumes of 28 information generated through research, and a recognition of the need to work across traditional disciplinary, institutional and national borders” (n.d., Abstract section, para. 2). Video-conferencing is one communication technology that has been widely used by researchers to support various research activities, such as holding web-based research seminars or broadcasting presentations. Related research falls into three categories: emulating realistic face-to-face communication, improving spatial realism and introducing additional modes of communication. 2.1.1.1 Emulating realistic face-to-face communication To provide researchers with a better infrastructure, there has been extensive research into creating an ideal illusory face-to-face condition with video conferencing techniques. This kind of design is referred to as shared virtual table environment (SVTE). One of the earliest and most well known implementations of realistic face-to-face remote communication is Tele-cubic by Chen, Towles, Nyland, Welch, and Fuchs (2000). Later, research by Schreer, Brandenburg, Askar, and Kauff extended the same research principle in an endeavor to develop a video-conferencing system that “enable the participants to make use of rich communication modalities as similar as possible to those used in a face to face meeting” (2001, Introduction section, para. 1). Kauff and Schreer suggest that, “A common theme of all these efforts is to exploit the benefits of tele-immersion, often in a way that the participants will have the impression of being present in a 29 suggesting spatial proximity, enabling a higher degree of natural interaction and effective collaboration” (2001, p. 106). SVTE-based systems have up to now provided meeting attendees with a scenario closer to real co-location, assuring perfect eye contact and use of gesture. To some degree, however, these kinds of effort have concentrated overmuch on superficial aspects of remote interaction, while unintentionally neglecting its essential novelty. Hollan and Stornetta (1992) suggest that more attention needs to be paid to discovering the technological advantages of remote participation. Similarity, Birnholtz and Horn indicates that remote interaction design should “exploit the unique attributes of the technologies being used” (2007, Beyond being there section, para. 1). With the cost of large laboratory instruments increasing, and growing pressure to share superior lab equipment using existing network connections, research infrastructures based on video-conferencing could expand the way that researchers interact with one another. 2.1.1.2 Improving spatial realism Although research shows that face-to-face communication is still the friendliest and richest modality (Hauber et al., 2006, p. 413), some researchers have looked into ways of preserving proper spatial faithfulness that will give users a face-to face feeling. Instead of striving to achieve such an illusory face-to-face scenario, these researchers focus more on effectively supporting online interaction, or on accommodating multiple distributed users in a video-conferencing software 30 application. Hauber et al. explored “ways to combine the video of a remote person with a shared tabletop display to best emulate face-to-face collaboration” (2006, p. 413). To preserve spatial faithfulness and make interaction effective, some research efforts have looked into maintaining eye contact during a video- conference. One example is an investigation into gaze conditions in video conferencing; to maintain perfect eye-to-eye contact; Vertegaal, Sohn, and Cheung conducted a study using hidden video tunnels, using an array of cameras at the same height as a participant’s image screen (2003). By “selecting a single camera closet to where the user is looking for broadcast,” Vertegaal et al.’s design managed to preserve good eye contact among video-conferencing attendees (2006, p. 521). However, these studies were restricted to cases with rich interaction. Although they pointed out “if communication channels were missing, users automatically compensated for this problem with other extensive media,” Hauber et a!. did not evaluate this aspect (2006, p. 414). 2.1.1.3 Additional modes of communication As video-conferencing systems become used more frequently in the research infrastructure, other researchers have focused on evaluating and comparing the different media components of a video-conferencing system in terms of how they support research activity. Scholl, MaCarthy, and Harr did a user study comparing text-based chat and audio in a multimedia conferencing environment (2006). The 31 researchers found out that chat is an essential communication tool in video conferencing (Scholl et al., 2006). Sun and Razdan also chose to add a chat plug- in to their system to enable text messaging between operators and observers (n.d). Our study used Econtrol, an online web interface enabling remote operation and manipulation of video-conferencing equipment in an experimental research lab. Econtrol is hooked up to NetMeeting, which is a video-conferencing application with text-messaging. In the pilot study, we found this function to be very beneficial to communication, since the text-messaging system could maintain bi directional communication when audio and visual transmission went down. 2.1.2 Communication technology supporting education Recent technological innovation in communication has not only transformed the way that people do research but has also broken down the wall of traditional learning and teaching methodology. With the availability of the Internet, wireless networks, fiber optical networks, streaming video and audio, ISDN, and satellite communication technology, education is no longer restricted by geographical distance or institutional boundaries. Among all the communication technologies that support education, video-conferencing is considered as one of the key tools for delivering quality education. 32 2.1.2.1 Video-conferencing in online education Recently, video-conferencing has reached “a level of stability, usability and affordability which permits its use in real teaching scenarios rather than research projects” (Coventry, n.d, About this Report section, para. 2); so it is gradually being integrated with educational technology. Educational applications empowered by video-conferencing are now capable of delivering web-based, real time teaching, learning, and even instrumentation. However, there has been very little research evaluating its effectiveness in education; a preliminary report (Heeler & Hardy, 2005) tested the use of video, audio, and text messaging in online education. The remainder of this paragraph describes Heeler and Hardy’s study. Heeler and Hardy conducted two user studies to measure the effectiveness of using online media to teach graduate courses remotely, and their study results reflected the fact that the virtual space — the combination of video, audio, and text messaging — certainly benefited users and provided them with a rich interaction aid. However, their focus was on familiarizing students and professors with the use of video-conferencing equipment for holding courses online, and finding ways to effectively employ video technology to deliver lecture instructions and materials. Heeler and Hardy claimed that they used the video technology to enable a successful teaching style suited to the online teaching environment rather than to the traditional face-to-face lecture. As the online medium is taking on a more important role in the future of education, Heeler and Hardy make several suggestions to potentially extend their research; one of these being to extend the 33 research into “monitoring students’ progress in lab activities” (2005, p. 132). This matched the initial motivation for our user study. 2.1.2.2 Tele-instrumentation or Tele-operation as educational technology So far, not much research effort in HCI has been spent on investigating remotely operating lab equipment via a video-conferencing system. However, research on remote laboratories has been active in physics and engineering education (Li, Esche, & Chassapis, 2007). Taylor et al. have developed a “Nanomanipulator” that is a virtual-reality interface to a scanning tunneling electron microscope, based on existing network infrastructure, designed to allow scanning tunneling microscope users “to see, feel, and manipulate matter at the nanometer scale” (1993, p. 128). Taylor et a!. claim that, before the development of their application, users using scanning tunnel microscopes had to learn how to use a computer-based application to operate the microscope; with their invention, users can avoid that bother and concentrate on actual manipulation of materials (1993). Sharing design goals comparable with the “Nanomanipulator” (Taylor et al., 1993), Sun and Razdan have attempted a similar approach (n.d.). The remainder of the paragraph summarizes Sun and Razdan’s study. Sun and Razdan’s goal is to transmit live instrumental data to a remote place via a video-conferencing system, wrapped up with a graphical user interface to remotely visualize data captured, with the instrument being remotely controlled. As in our study, their 34 research uses the Internet to enable a two-way interaction; their intention is to enable novice users to operate a complex piece of equipment without any training, aiming to use the online media to operate a scanning probe microscope and make observations with it in a sharable fashion. At the same time, they want to improve levels of collaboration with the remotely accessed device. However, not much user evaluation has been done on these tele-instrumentation systems in terms of their effectiveness and efficiency for supporting user interaction. Birnholtz and Horn have looked into earthquake simulation in civil engineering (2007). The remainder of the paragraph summarizes Birnholtz and Horn’s research. Birnholtz and Horn’s research aims at the potential for conducting experiments involving large-scale equipment in a remote context. Concerned about how an earthquake test is conducted, and about the objectives, expectations and presumed accomplishments of this kind of experiment, Birnholtz and Horn’s approach to the problem is through interviews and observations. Birnholtz and Horn reflect a traditional comment that “physical presence at a test provided richer access to the ‘details of the test’ that could not be replicated” (2007, Concerns about Remote Manipulation section, para. 2), but participants in their survey did suggest there was great potential in using remote observation rather than manipulation. Some other advantages of remote participation were also identified: it eases lab safety issues, avoiding subjects being exposed to harmful and dangerous lab materials or experimental situations. Birnholtz and Horn state 35 “there is value in thinking carefully about the goals of e-science collaborators at all phases of the research process.” (2007, Explaining the Findings section, para. 2) 2.1.2.3 Supporting Tele-instrumentation in remote collaboration Instead of one-way tele-instrumentation, some researchers have looked into collaborative experimentation supported by video-conferencing, both synchronous and asynchronous collaborative experimentation under remote conditions. The closest study to this was done by Ranjan, Bimholtz, and Balakrishnan., who looked at using collaboration technology for distributed teams to work together to accomplish tasks, with the purpose of exploring the relationship between hand motion and video information (2006). The remainder of the paragraph summarizes Ranjan et al. ‘s research. Ranj an’s team conducted a non-co-located video- and audio-mediated user study under three conditions of camera monitoring. As well as a fixed-position camera to capture video, their study used a hand-controlled camera and an automated camera to improve collaboration in a remote setting. Coincidently to our study, there were two participation roles in their study as well; one called “the helper” and the other called “the worker” (Ranjan et al., p. 403). Ranjan et al. found that there is a mathematical relationship between things should be seen by a helper and things can be seen by a helper in a camera view. In terms of providing an optimal camera view, Ranjan advised that there is no perfect formula, since workers 36 modify their behavior actively according to the visual feedback cues that workers consider their helper can or cannot see. Moreover, although their video analysis favors automated camera control, Ranjan et al. discovered some positive aspects of using a fixed-position camera, which does not need to be moved every time a hand-shift happens; on the contrary, helpers can independently adjust the camera to acquire a different view, such as an overview, to bring in flexible visual information. Ranjan et al’s user study also indicates a subjective monitoring style; helpers did not constantly monitor their workers every minute and in between (2007). 2.1.3 Summary The literature review in this section of the chapter discussed prior research in collaboration technology and techniques for conducting experiments in a distributed environment. It also provided additional background on the research issues that are being addressed in this thesis. 2.2. Research infrastructure 2.2.1 Backbone of UQAM infrastructure-The BEST project The BEST project (Bourdeau et a!., 2006) is another research project that aims to improve tele-learning experiences. The rest of the paragraph summarizes the BEST project. The goal of the project was to introduce an online interactive 37 system that assisted prospective doctoral students in planning a specific academic program and helped existing professional scholars with their progress in a doctoral program. With the cooperation of Kaleidoscope, which is a joint European research network aiming to explore learning technology via digital media, the BEST team designed an online doctoral training system (DTE). The project fulfilled two key factors that Bourdeau identified when researching the requirements of doctoral training: “co-experimentation” and “access to top-level instrumentation” (Bourdeau et al., 2006, p. 1). To achieve the second feature, one research team of the BEST project implemented an online lab interface for accessing and controlling the equipment of an existing experimental research lab, LORIT, which is equipped with apparatus that “allows observation and capture of multimedia data from multiple pre-synchronized sources” (Bourdeau et al., 2006, p. 7). By introducing this virtual environment, the BEST project successfully enabled distributed research teams to have access to remotely located equipment and facilities. As our research used LORIT to help with the user study, we will briefly introduce LORIT in the next section. 2.2.2 QUAM infrastructure — LORIT LORIT is owned by LICEF, a research center at the University of Montreal. It is a distance engineering research lab supporting research with a multi-media approach, designed to support video-conferencing, research and development in training, and multimedia usability testing (Bourdeau, Dufresne, Prom Tep, & 38 Turcotte, 2005). Its infrastructure is composed of sophisticated visual and audio facilities (Figure 2.1), and is designed to support collaborative work and to conduct experiments/experimentation through a rich media that can be conveniently accessed (Bourdeau, Dufresne, Tep, & Leclercq, n.d.). The details of LORIT will be described in Chapter 4. Figure 2.1: LORIT lab and equipment of UQAM (picture taken from LORIT’s website) 2.2.3 UBC infrastructure — a video-conferencing lab The Department of Computer Science at UBC has a video-conferencing lab equipped with premiere visual and audio equipment. It has been used for broadcasting research and scientific seminars, such as the WestGrid scientific seminars, during the school year; scientists have also used this lab as a research and discussion venue (Figure 2.2). During recent video-conferencing testing with LORIT, it was proved to be a well-equipped research lab that is suitable for remote experimentation. The details of UBC lab will be described in Chapter 4. 39 Figure 2.2: UBC video-conferencing lab and equipment Access Grid Shelf 40 2.3 Tools We went through a selection process to find the most suitable video-conferencing tool for our study. There were three candidate applications: Click-To-Meet, Access Grid with VNC, and Net-Meeting. 2.3.1 Click-To-Meet Click-To-Meet was developed by Radvision. It is a scalable desktop video conferencing tool that enables real-time interactive communication in the forms of bi-directional audio, video, and data. It offers an infrastructure that is capable of bringing distributed teams to meet in online venues for different kinds of interaction; its main feature is video-conferencing (Figure 2.3). 41 Whiteboard We ran a few trial study sessions using Click-To-Meet. It did provide a user- friendly interface for video-conferencing meetings, and it enabled desktop application sharing easily; however, its latency over the data communication network severely affected participants’ performance in the trials. There were several cases in which a study session had to be re-run due to a system deadlock. Figure 2.3: Click-To-Meet window Current Speaker Screen Layout 42 When sharing the application, if a user actively accessed an application on the workstation of another user, the shared application would be blocked from view by a mosaic. To make the application visible, the accessing user had to right-click the mouse to cancel the mosaic; unless this was done, the mosaic would remain (Figure 2.4). Figure 2.4: Mosaic view 2.3.2 NetMeeting Net-Meeting is a “conferencing client developed by Microsoft that allows users to have real-time interaction over the Internet” (Netmeeting for Beginners, 2007, 43 1 http:/ wrad si,n.c,m - kADVISION - MoziII,Fjre(ax - ID -J’;-- •-. ic’-.... ;..- ..— -. --tc’- — - -. L— 1 - .- —. — - i-. - 1-. 1-.. ‘-z’-. -c’c”c’’-’ .—.- -.‘-.. -. - ‘ - .‘ .. — ‘ ‘ .. — ... ... c’ - 1’ -c’ Ic’ •- I... - Ic’ -c:--tc’ I- -. — .. 2 c’-c’c’rc’ — — — - z’ ‘;;- ;- - —‘ c’ ic’ .. — .. ‘ c’ ‘‘ —‘ [I I i ___ I L;;r.tc’ .2 - r--’.i c”1 1 1 1- - —---- —— -‘ .- -. — t._ What is NetMeeting section, para. 1). NetMeeting is hooked up to LORIT@D, and it is automatically invoked when LORlT(aD is loaded. During our pilot study with Click-To-Meet, we encountered audio and video transmission problems several times. Each time we made the decision to use Net-Meeting temporarily. With the help of Net-Meeting, we resumed the audio and video transmission right away, had Click-To-Meet trouble-shot by the remote technical support staff, and afterwards returned to Click-To-Meet to continue our study session. From this experience of Net-Meeting, we found that although Net-Meeting provides users with only minimal features for video-conferencing, compared to Click-To-Meet, its lower network bandwidth requirement and fast and robust connection over the Internet made it an attractive replacement for Click-To-Meet. However, when testing it with our experimental software, we found the real estate of its video screen was exceptionally small; any aggressive mouse movement could easily escape from the video screen. Since our study required spacious screen real estate to enable sufficient mouse movement, Net-Meeting did not offer an ideal selection. 44 Figure 2.5: Screenshot of Net-Meeting NetMeeting - Not in a ... loolsCaH View 206.167. _____ jeip ?ideo Audio Tuning Wizard... Sharing Chat Whiteboard File Transfer Whiteboard (1.0- 2.x) Remote Desktop Sharing,.. Options... ‘I 1 I CtrI+5 CtrI+T CtrI+W Ctrl+F Name r,i rii ‘i i41 CO ains commands For conferencing. 2.3.3 Access Grid with virtual network computing (VNC) 2.3.3.1 Access Grid Like the previous two video-conferencing tools, AccessGrid (Version 3.0) is multi-media video conferencing software. It has the scalability of accommodating 45 physically distributed teams to hold online meetings, and provides users with the features of video, audio and text messaging. 2.3.3.2 VNC To enable application sharing, AccessGrid needs to rely on a third party component; although it does supply some default collaboration tools, other third party application-sharing tools have been developed for it, VNC being one of the most popular (Hasan, Lewis, Alexandrov, Dove, & Tucker, 2007). \TNC stands for Virtual Network Computing; it shares an entire view of a person’s desktop with another party. Unlike Click-To-Meet and Net-Meeting, the application sharing feature provided by VNC is modifiable, and this allows some desktop applications containing sensitive information to be hidden (Berry, 2005). VNC consists of a viewer and a server; the function of the VNC viewer is to view a shared desktop, while the VNC server enables a personal desktop computer to be accessed, manipulated, and shared with other participants in an online group meeting. In our case, this feature allows the remote party to access our workstation for technical support when audio or video transmission goes down. Furthermore, VNC provides meeting participants with a point-to-point communication mechanism. Using AccessGrid, we were also able to separate the video-conferencing and file-sharing processes from each other, so as to reduce the transmission burden on the network, reducing the system network load and improving the performance of the experiment software. The only problem with 46 AccessGrid was that it did not provide users with a decently designed conference window layout; with its separate audio, video, and chat windows scattered around a desktop, users had to re-arrange the windows manually and voluntarily. After testing Click-To-Meet, NetMeeting, and AccessGrid (with VNC) with LORIT, AccessGrid eventually won out because of its system capability, easy technical support, and separation of desktop sharing. Thus, we decided to use AccessGrid along with VNC as the video-conferencing tool for our study. Figure 2.6: Networks connection for videoconference experiments Apart from the tool selection process described above, we went through two phases of study design; there were problems discovered in each specific phase. In the first phase, we used Click-To-Meet to pilot our study under local and remote conditions. In the second phase, we screened through the three video conferencing applications mentioned before choosing AccessGrid. Finally, we used Lightpath with Access Grid and VNC to run our study. As well, although World Wide WE 47 not selected as a tool for our study, Net-Meeting was used as a supporting tool for troubleshooting network connection when technical difficulties were raised with video or audio in phase I. 48 Chapter 3 Fitts’s Law Studies of Directional Movement between Two Target Positions This chapter discusses Fitts’s Law, the theoretical basis for the EXPERIMENT that will be the subject of our STUDY. Fitts’s Law was originally proposed by psychologist Paul M. Fitts to model human aiming performance in motor (movement) tasks (Fitts, 1954). All the description about Fitts’s Law here is based on the research of Fitts (1954). In his paper, Fitts predicted that although it was known that movement time and target hitting accuracy change as the distance between two targets was adjusted, there must exist a relation between speed of movement, target distance, and target hitting accuracy (Fitts, 1954). Based on this prediction (Fitts, 1954), a hypothesis was proposed regarding the channel capacity of the human motor system, and it was estimated that the capacity remains relatively constant as movement amplitude varies. To verify this hypothesis, Fifts designed a study with several sets of tasks that required subjects to make rapid, reciprocal, and uniform movements between two target locations (Figure 3.1). 49 Figure 3.1: Fitts’s Law task A ____ In the study design, Fitts manipulated two sets of variables: width of a target (W) and amplitude (A) between two target plates (1954). The results of the study showed that subject performance changed uniformly as the variables were manipulated. To calculate the desired rate of subject performance, Fitts (1954) proposed an index of performance (Ip) which is formulated as: Ip = l/t * Log2(2A/W) After re-arrangement, the formulation became t(time) = Log2(2A!W). From the nature of the study design, the formulation indicates the relationship between time of movement and variation of target dimensions. The formulation suggests that the time it takes to acquire a target decreases monotonically as the size of the target increases, and increases monotonically as the distance to the target increases. In other words, Fitts’s Law predicts the amount of time that it takes to select a target, based the location and size of the target. Thus, scientists in human computer interaction have adopted this model in a variety of interface research 50 domains and interface design tasks (Schedlbauer, Pastel, & Heines, 1995). The formulation has also been generalized to following: MT a + b Log2(2A/W) where • MT is movement time, the dependent measure. The two independent measures are • A is the distance (or amplitude) of movement from a starting position to the target center • Wis the width of the target, which corresponds to ‘accuracy’ (Zhao, 2002). According to Fitts , W is considered the allowed error tolerance in the final position of a tapping movement, since the final stopping spot must fall within +1- W/2 of the target’s center (1954) • a and b are empirically determined constants derived using regression analysis; they are device dependent. Since the Fitts’s Law formulation was initially introduced to HCI, MacKenzie points out that other researchers in the field have proposed several variations of the formulation (1992). Two of the most well-known are the variations introduced by MacKenzie and Welford (MacKenzie, 1992). These two modified versions of Fitts’s model differ from the original only in terms of the Logarithmic function 51 used to provide a better fit with observation (MacKenzie & Buxton, 1992). The Welford formulation takes the following form (MacKenzie & Buxton, 1992): MT= a + b Log2(A / W + 1/2) And MacKensize formulation (MacKenzie & Buxton, 1992): MT= a + b Log2(A / W + 1) 3.1 Index of difficulty in Fitts’s formulation To analyze subject performance in the tapping tasks, Fitts adopted a binary index of difficulty (1954). The original difficulty index (ID) derived by Fitts takes the form: ID = Log2(A/W) In Welford’s version, the index of difficulty is: Log2(A/W+1/2) and MacKenzie’s version is: Log2(A/W +1) Fitts derived the index of difficulty based on Information Theory (Fitts, 1954). According to the rationale provided by Fitts, the minimum amount of information 52 required by a human subject to make a movement is proportional to the logarithm of the ratio formed by the amplitude divided by the target width (Fitts, 1954). Given that the original study design of Fitts’s Law decreased the width of the targets by a factor of 2 (2, 1, Y2, ¼ inch) and increased the amplitude between two targets by a factor of 2 (2, 4, 8, and 16 inch), the base value of the logarithm is chosen to be 2. As the key measurement of Fitts’s model, from Table 1, it can be observed that as the value of the ID increases, the movement time increases. Table 3.1 Index of difficulty and movement time ID MT 6 0.456 7 0.798 8 0.872 9 0.927 10 0.978 3.2 Characteristics of Fitts’s Law formulation The index of difficulty (ID) of Fitts’s formulation takes the logarithm of the target distance over the target width; thus, ID can be any value. Movement time (MT) is a calculated from the index of difficulty. Since the index of difficulty is an independent variable, and the movement time is a dependent variable, the Fitts ‘s formulation can be transformed to the form of a linear function: 53 MT=a+b*ID Thus, Fitts’s Law model indicates “movement time (MT) is a linear function of ID” (MacKenizie, 1992, A Brief Tour of Fitts’ Law section, para. 2). Consequently, the values of MT and ID are fit to the line of the equation by linear regression analysis. Since the bivariate regression is one of the most frequently applied regression models used to illustrate the relationship between two variables (Huck, 2004), the Fitts’s Law model can, indeed, can be tested by using bivariate linear regression to fit user test data to the predictions of the model. There are two types of results obtained from a linear regression analysis of the Fitts’s Law model. The first type of data is the equation constants which are a and b. b is the slope of a straight line fitted to a linear regression equation. It indicates the trend of the data (MT versus ID) on the plotted line, a is the intercept of the linear regression line on the axis of the dependent variable, MT. In Fitts’s Law fonnulation, a and b are device-dependent constants, and they are used to determine characteristics of human performance for a particular setting or to compare across settings. A secondary analysis can be conducted on the resulting parameter values produced by the bivariate regression. This includes the correlation coefficient and the coefficient of determination. A correlation coefficient (r) indicates “the correlation between the independent and dependent variables” (Huck, 2004, p. 425). Its value ranges from -1 to 1, with 1 suggesting a perfect positive 54 correlation, and —l suggesting a perfect negative correlation between the dependent and independent variables. If the value of r is 0, this means that there is no linear relationship between the variables (Huck, 2004). For Fitts’s Law, r quantifies the deviations between the predicted movement time and the actual movement time. The coefficient of determination (r2) “indicates the proportion of variability in one variable that is associated with (or explained by) variability in the other variable. The value of r2 will lie somewhere between 0 and +1.00” (Huck, 2004, p. 68). In Fitts’s Law model, this suggests the percentage of the variation of movement time that can be attributed to changes in the index of difficulty. We look at both r and r2, but we focus on r2 to determine whether each of our experimental conditions yields valid results. 3.3 Fitts’s Law in interaction studies As mentioned previously, Fitts’s Law has been applied extensively to interface research in HCI. Classical Fitts’s Law research applied the Fitts’s Law model to measure human performance on single variable tasks, which was “making horizontal moves toward a target” (MacKenzie and Buxton, 1992, p. 220). Later on, researchers extended the Fitt’s Law model to two-dimensional and three dimensional pointing tasks (Murate & Iwase, 2001). “Due to its accuracy and robustness, Fitts’s Law has been a popular research topic. Numerous studies have 55 been conducted to explain ... extend ... and apply Fitts’s Law to various domains” (Accot & Zhai, 1997, p. 295). Some research extends Fitts’s Law to other domains and tasks. An example is the research of Schedlbauer, Pastel, and Heines, which applies “Fitts’ Law to pointing with a trackball with subjects standing, and compares the trackball pointing performances while sitting versus standing” (1995, Movement Time Models section, para. 1). The application of Fitts’s Law also assists user interface design, such as “[invention of pop-up menus] and in interface evaluation [e.g., “embedded model” in [13]]” (Zhao, 2002, Applicability to HCI Discussions section, para. 3). Fitts’s Law is also used for comparing performances of different input devices (Zhao, 2002), such as user interaction with joystick and mouse, in terms of their behavior of pointing and clicking on graphical interface components such as buttons or images. 3.4 An example of a Fitts’s Law EXPERIMENT A typical Fitts’s Law study is “undertaken in a controlled experiment using a group of subjects and one or more input devices and task conditions” (MacKenzie, 1992, Building a Fitts’ Law Model section, para. 1). In such an experiment, researchers manipulate the two variables, A and W, to achieve different levels of index of difficulty. Subjects are asked to acquire targets as quickly as possible. Subject performance is measured based on the movement time of acquiring targets. 56 There have been many Fitts’s Law studies designed by researchers in HCI. Following, we give a concrete example of a Fitts’s Law EXPERIMENT, which we designed for our research: reciprocal tapping between two positions. Using the Welford’s formulation, we designed the Fitts’s Law tasks to change only one of the variables, which is the width of a target. The target is designed to take the shape of a circle. We set the target width (W) to be 8, 16, 32, and 64 pixels to form four levels of difficulty. The circle radiuses establish the tolerance range within which movements had to be terminated. The amplitude/distance (A) between two target positions is kept constant, and the value of A is set at 168 pixels, which is equal to 52 mm. Our Fitts’s ExPERIMENT contains four blocks; in each block, there are three Fitts’s tasks. In total, there are twelve Fitts’s Law tasks in our ExPERIMENT. In each task, a set of nine dots is presented in a circular arrangement with the current target position colored in red. The PARTICIPANTs are asked to use the mouse to point and click the red dot that appears on the screen. Once the current target position is clicked, the red target shifts to another position. The PARTICIPANTs are required to alternate between clicking the centre position and a position in the surrounding periphery (Figure 3.2 a and Figure 3.2 b; the color of the actual target is red, but for illustration purpose, it is presented in white in the figure). Two different surrounding target positions are randomly selected to remove any prediction or learning effect. For each target position, a PARTICIPANT uses the 57 mouse to point and click the red target back and forth between the target positions twenty times. The estimated average time to complete each Fitts’s task is two minutes. PARTICIPANTs are expected to complete the entire Fitts’s EXPERIMENT within twenty-four minutes. The positions of the red target are pre-decided by randomization in each task. When presented to the PARTICIPANTS, the order of showing the positions and the sizes of the targets is counter-balanced based on two pre-designed data files. • • • Figure 3.2 a: Schematic drawing of our Fitts’s Law task --- Target at its starting position 3.4.1 Practice sessions Figure 3.2 b: Schematic drawing of our Fitts’s Law task --- Target at a surrounding position Before moving on to the actual Fitts’s tasks, a PARTICIPANT is encouraged to perform a practice session. The PARTICIPANT is allowed to run the trial Session as many times as he/she wants until perceiving comfort and familiarity with the F 58 session. Once a PARTICIPANT is comfortable with the trial session, the PARTIcIPANT is instructed to move on to doing the actual tasks. 3.4.2 ExPERIMENT sessions As mentioned above, the ExPERIMENT contains four blocks of tasks. Each PARTICIPANT is asked to complete one block of tasks at a time, and then to take a five minute break before moving on to the next block. In each block of tasks, the distance between the central target position and all the surrounding target positions remains constant, but a different size of targets is presented. From left to right and from top to bottom, Figure 3.5 shows four block windows of the four different target sizes. 59 The data recorded for the Fitts’s Law EXPERIMENT are the actual amplitude distance, time to complete each Fitts’s task, time to complete an entire Fitts’s Law ExPERIMENT, and errors made through each task. Table 3.2 summarizes the design of our Fitts’s Law EXPEJUMENT. 4 Figure 3.3: Four block windows with four different target sizes r . L 0 . . . . p . . . 60 Table 3.2: Design of our Fitts’s Law EXPERIMENT Independent Variables 1. N levels of Target Width 2. M levels of Amplitude Dependent Variables 1. Movement Time (MT) 2. Number of Errors 3.4.3 ExPE1UMENT software The software that runs the Fitts’s Law ExPERIMENT was implemented using the Java virtual machine version 5 and the Eclipse platform version 3.2. It runs on the Microsoft Windows XP platform. The software is designed to exactly implement the design of our Fitts’s Law ExPERIMENT. The software is embedded with a logging routine, which records the measures summarized in Table 3.2. 61 Chapter 4 Study Design This chapter describes a STUDY whose purpose was to evaluate the efficacy of conducting experiments when the SuBJEcT and the ExPERIMENTER are not co present, compared to the usual condition when the SUBJECT is co-present with the ExPERIMENTER. The terminology introduced earlier will be employed throughout this chapter. The word “STUDY” refers to the experimental investigation we are reporting, and the word “PARTICIPANT” refers to a person who participated in the STUDY as an experimental SUBJECT. The term “EXPERIMENT” refers to the activity within that STUDY in which PARTICIPANTS carried out a Fitts’s Law task. We reserve the word “SUBJECT” to refer to a PARTICIPANT playing the role of a SUBJECT, and the word “EXPERIMENTER” refers to a PARTICIPANT playing the role of an EXPERIMENTER. The questions that we seek to answer are motivated by our desire to run experiments via a video-conferencing system in a “virtual lab space.” We want to know if the experimental results will be similar to the ones that we obtain in a conventional lab setting, following the same procedure, with SUBJECTS and EXPERIMENTERS working together face-to-face. We also want to know if an EXPERIMENTER acquires the same experience conducting an EXPERIMENT in a 62 virtual lab space. Our STUDY was designed to address these and related issues, all related to the larger question of whether the results obtained through this kind of experimental approach can demonstrate a valid research methodology. We decided to focus on the first question, whether the results obtained using a virtual lab would be similar to those obtained in a conventional lab. One way to examine this question would be to simulate the non-co-present condition through controlled laboratory studies that ran EXPERIMENTs in a faked remote condition in which the EXPERIMENTER and the SuBJECT were located in two separate rooms, side by side. We initially considered this approach, and in fact, ran some preliminary EXPERIMENTS to test this methodology. However, we realized there were a number of potential problems with the approach, so we decided that we would go the extra effort to test the methodology in a more realistic, ecologically valid setting. We worried that PARTIcIPANTs might not suspend their disbelief if they knew that they were in fact not engaged in a remote collaboration, and that it would be difficult to fool them into thinking that they were. For these and other reasons, we chose to conduct a STUDY with a genuine remote condition, where an EXPERIMENTER is on one side of a country and a SUBJECT is on the other side of the country. This would allow us to assess the subjective and objective efficacy of remote experimentation as it is experienced by both an ExPERIMENTER and a SuBJEcT. Another reason we did this was that the remote lab facility and the virtual environment we utilized to run our STuDY, LORIT, was 63 designed and built to support doctoral students in the field of education by allowing remote access to the lab for the purpose of running experiments during their dissertation research. We were therefore fairly confident in assuming that LORIT was a natural candidate for supporting remote experimentation and that it afforded a good test bed to answer the questions we were posing. The first objective of our STUDY was to determine whether remote and co-present experimentation would yield statistically similar SuBJEcT performance. We chose a Fitts’s Law task for this, because we knew it had been well studied and was quite robust in outcomes, even in the face of various confounding factors. Furthermore, it could be easily conducted by naïve PARTICIPANTs in our lab setting. To address this objective, we looked both at the results obtained in the two conditions and at the statistical properties of those results. As a second objective, our STUDY also examined the communication and collaboration experience between EXPERIMENTERs and SUBJECTs as they occurred in the two conditions. As we will explain later in Chapter 5, some aspects of the ExPERIMENTER-SUBJECT interactions did turn out to be quite interesting. 4.1 Experimental Design We tested two collaboration conditions suitable for conducting an EXPERIMENT about Fitts’s Law tasks, in which PARTICIPANTS were required to see and talk to each other. We used a 2x4 mixed model design with co-presence as a between 64 subject factor (with two levels — co-presence and non co-presence) and target width as a within-subject factor, using the four levels described in the end of the previous chapter. 4.1.1 Hypotheses The hypotheses we were trying to test in the STUDY were: Hypothesis #1— Co-presence of EXPERIMENTERs will affect SUBJECTs’ performance in an ExPERIMENT. Hypothesis #2— The interpersonal experience between EXPERIMENTERs and SUBJECTS will depend on whether the EXPERIMENTER and SUBJECT are collocated or not. Hypothesis #3 -- The amount of cognitive load experienced by a PARTICIPANT will depend on whether the EXPERIMENTER and SUBJECT are collocated or not. These hypotheses led to the following testable operational hypotheses: Hal.1: Statistically, the validity of the Fitts’s Law EXPERIMENT data obtained in the co-present condition is better than those obtained in the non co-present condition H21.2: Performance of local SuBJECTs is better (faster and less error-prone) than that of remote SUBJECTs in the Fitts’s Law EXPERIMENT. 65 Hal.3 Local SUBJECTS spend less time to conduct all the tasks in each task block and overall they spend less time to complete all the Fitts’s Law tasks than that of remote SUBJECTs in the Fitts’s Law ExPERIMENT. Ha2: Interaction experience is richer in the co-present condition than in the non co-present condition. Ha3: Local EXPERIMENTERs would experience a less attention shift than the remote EXPERIMENTERS. Based on our hypotheses 1.1 and 1.2, Table 4.1 lists the possible interpretation of the statistical interpretations of outcomes of Fitts’s Law EXPERIMENTS in local and remote conditions: Table 4.1 Statistical interpretations of outcomes Quality of the correlation as measured by r2 same different same The local and remote One experimental conditions are identical condition is better, but both in terms of human performance is experimental validity probably the sameHuman and measured humanperformance performance as measured different The EXPERIMENTS are Only one conditionby the Fitts’s Law equally valid in the local demonstrates true and remote conditions, human performancecoefficients a but human performance (the one with better&b is different r2) and the other yields a false measure of human performance 66 4.1.2 Conditions Figure 4.1 Configuration of the STUDY in the local condition N N N Meta-observation N N N N N N ryWorkstafl_J SUBJECT Condition 1. “Co-presence/Local/Co-located/Face-To-Face”: In the local condition, one PARTIcIPANT playing the role of ExPERIMENTER sat right beside another PARTIcIPANT playing the role of a SUBJEcT in a laboratory at UBC in Vancouver (Figure 4.1). Both parties collaborated and communicated with each other verbally and interacted with each other through hand gestures and body movement. N N N N is t / // Researcher N N Meta-observation/’ 0fr’ A ExPERIMENTER\ Observation Prim . Interaction V;1 Configuration UBC of the study in local condition 67 Figure 4.2 Configuration of the STUDY in the remote condition Web Condition 2. “Non co-presence/Remote/Non co-located/Non Face-To-Face”: Under this condition, one PARTICIPANT playing the role of ExPEIUMENTER sat in a lab located in UBC, and another PARTICIPANT playing the role of SuBJECT sat in LORIT at UQAM in Montreal (Figure 4.2). They collaborated and communicated with each other through video conferencing. 4.1.3 Tasks In each condition, PARTICIPANTS were assigned to play the role of either being a SUBJECT or an ExPERIMENTER. The task of a SUBJECT was to do a Fitts’s Law ExPERIMENT, to be accomplished by following the instructions of an ExPERIMENTER (Chapter 3 gives the detail of Configuration of the study in remote condition 68 the Fitts’s Law ExPERiMENT). Each SUBJECT was allowed as much time as needed to complete the EXPERIMENT, but the SuBJECT was encouraged to take as little time as possible. In case of instructional confusion, the SuBJECT was required to communicate with the EXPERIMENTER for clarification. The task of an EXPERIMENTER was first to go through a training session with a lab assistant, to familiarize the ExPERIMENTER with the Fitts’s Law ExPERIMENT. Then the EXPERIMENTER was required to instruct the SuBJECT on how to accomplish the Fitts’s Law ExPERIMENT. The EXPERIMENTER was also required to observe the SUBJECT’s behaviour and to make notes of any noticeable and interesting phenomena. If an issue arose, the EXPERIMENTER was required to make necessary interventions in a proper form to ensure the SUBJECT’s continuation with the Fitts’s Law task. The EXPERIMENTER was also responsible for answering any questions raised by the SUBJECT. Another duty of the EXPERIMENTER was to collaborate with a co-located lab assistant to resolve any technical issues arising in the EXPERIMENT. In the remote condition only, after training with a lab assistant and before instructing a SUBJECT, the EXPERIMENTER in Vancouver was required to set up (configure) some of the equipment in the remote lab in Montreal. 69 4.2 Participants Forty-two PARTICIPANTS (28 female and 14 male) took part in the ExPERIMENT. They formed 21 pairs, with each pair assigned to one STuDY session. Pairs of PARTICIPANTS each conducted 12 trials in a single session of the STUDY, for a total of 252 trials. Due to technical problems, only 20 out of 21 sessions could be used, which meant there were two extra PARTICIPANTS whose data was not analyzed. The age of PARTICIPANTs ranged from 18 to over 40 years old. PARTICIPANTS were college students with experience in using a mouse and a personal computer. They were comfortable using a mouse to interact with a software application running on desktop personal computers. Additionally, all PARTICIPANTS were right-handed and had no color-blind deficiencies. PARTICIPANTs were required not to have conducted a significant amount of research in video-conferencing or remote interaction. Out of the 40 PARTICIPANTs who completed the STUDY (27 female and 13 male) 4 were in business programs, 24 were in engineering or science programs, and 12 were in arts programs. All PARTICIPANTs had normal or corrected-to-normal vision. PARTICIPANTS were paid $20 for their participation in the STUDY. PARTICIPANTs were recruited from university students in different departments at the UBC and UQAM. PARTICIPANTS had no prior knowledge of the ExPERIMENT. PARTICIPANTS were also screened either by phone or by email to ensure their eligibility. 70 In the local condition, the 20 PARTIcIPANTs were all from UBC. Half were randomly assigned to act as a SUBJECT and the other half to act as an ExPERIMENTER. In the remote condition, all 10 UBC PARTICIPANTs were designated to be ExPERIMENTERs, and all 10 UQAM PARTICIPANTs were designated as SUBJECTs. A more complete randomization of roles was not possible due to geographical constraints imposed by the lack of a specialized facility similar to LORIT in Vancouver. 4.3 Apparatus Hardware and software was used in the two labs, chosen to be as similar as possible for the workstations that were used by PARTICIPANTS, but some differences existed. The remote condition required additional communications hardware and software. In both remote and local conditions, the workstation used by a SuBJECT to conduct the Fitts’s Law ExPERIMENT was defined to be the primary workstation. In the remote condition, the primary workstation was one of the user stations in LORIT. In the local condition, the primary workstation was the only workstation in the UBC lab. Because the remote condition required video-conferencing to enable the communication and collaboration between UBC EXPERIMENTERs and SUBJECTs, in addition to the primary workstation, there were a few additional workstations used in the remote condition. We describe the workstation details in the following sections. 71 4.3.1 UBC Apparatus 4.3.1.1 Remote condition In the remote condition, there were three desktop workstations in the UBC lab (shown in Figure 4.3): EXPERIMENTER’s dominant workstation, EXPERIMENTER’s supplementary workstation, and the workstation of ExPERIMENTER’s lab assistant. EXPERIMENTER’s dominant workstation was used most frequently by an EXPERIMENTER and was considered the secondary workstation. EXPERIMENTER’s supplementary workstation was used less frequently by an EXPERIMENTER, and it was used to control the instrumentation that monitored the SUBJECT in LORIT. The lab assistant’s workstation was used by the UBC lab assistant to communicate with the lab assistant in the remote venue in LORIT. Figure 4.3 UBC workstations in remote condition / EXPERIMENTER’S / ExPERIMENTER’s 1 ,, Workstation of/ supplementary / dominant workstation / EXPERIMENTER’S / wr1cctitinn / / lab assistant (r r 72 Workstation hardware ExPERIMENTER’s dominant workstation was a dual monitored personal computer equipped with Intel Pentium III CPU with 1 GB RAM, a 17” WACOM Video monitor, and a second 14” ViewSonic video monitor. As well, a Logitech wireless keyboard and a Microsoft wireless 1001 mouse were attached to the workstation. They were used by UBC PARTICIPANTS to input information and interact with the workstation. The ExPERIMENTER’s dominant workstation was dedicated to video-conferencing and thus it was connected to a fiber optical network (Lightpath) for transmitting real-time audio and video from LORIT to UBC. On the 17” monitor, a UBC PARTICIPANT could see the desktop of the primary workstation in LORIT and the images of themselves and the remote party; on the 14” monitor, a UBC PARTICIPANT could see the control panel of the video-conferencing software. To capture the image and voice information of a UBC PARTICIPANT, a Neutrik AKG C1000S microphone with echo-cancellation and a Logitech QuickCam were mounted on the workstation. There was also a pair of stereo loudspeakers attached to the workstation; it transmitted the voice of a remote SUBJECT from LORIT to UBC. The EXPERIMENTER’s supplementary workstation was a personal computer equipped with Intel Pentium III CPU with 1 GB RAM and a 15” ViewSonic video monitor and a regular Internet connection independent of the Lightpath 73 connection used by EXPERIMENTER’S dominant workstation. The purpose of adding EXPERIMENTER’s supplementary workstation to the remote condition was to load up the LORIT web interface — LORIT@D. As a fiber-optical network, the Lightpath connection was assigned to specific network address that is different from the network address of general Internet. As such, Lightpath cannot act as general Internet connection. Because the LORIT web interface only loads on Internet Explorer, and Internet Explorer can only be loaded via general Internet, we had to look for ways to add a second Internet connection. Although we could have added an extra network card onto the secondary workstation for regular Internet connection, due to time constraints and technical support reasons, we did not do that. Instead we added this workstation right beside EXPERIMENTER’s dominant workstation and dedicated it to connecting to the LORIT@D web interface. The workstation of EXPERIMENTER’s lab assistant was a personal laptop computer with 14” display connected to regular Internet. The UBC lab assistant used this workstation as a separate channel to communicate with the remote lab assistant in LOR1T. Workstation software The EXPERIMENTER’S dominant workstation was equipped with the Windows XP Pro SP1.O operating system. To enable video-conferencing in the remote 74 condition, the workstation was installed with Access Grid version 3.0 (video conferencing software), and VNC (virtual network computing). The EXPERIMENTER’s supplementary workstation was installed with the Windows XP Pro SP 1.0 operating system and Internet Explorer with windows online messenger (msn). When loading up the LORIT interface (Figure 4.4), Microsoft ActiveX component was installed on this workstation as well. Figure 4.4: LORIT interface I 0 — ..... —ImUcn ? I -. , Aisistaic. TliconMr.nc. I Prs..,t.ton _____________ _____________ • MS —-.s- 9/ The workstation of EXPERIMENTER’s lab assistant was installed with the Windows XP Pro SP 1.0 operating system and Internet Explorer with windows online messenger (msn). 75 4.3.1.2 Local Condition In the local condition, there was only one workstation used in the UBC lab, and it was the primary workstation (shown in Figure 4.5). It was strictly used by local SUBJECTS. Figure 4.5: UBC workstations in local condition Primary workstation Workstation hardware The primary workstation was a single monitored personal computer equipped with Intel Pentium III CPU with 1 GB RAM and a 17” WACOM Video monitor. As well, a Logitech wireless keyboard and a Microsoft wireless 1001 mouse were attached to the workstation. They were used by UBC PARTICIPANTs to input information and interact with the workstation. Workstation software The Primary workstation was installed with the Windows XP Pro SP 1.0 operating system, and Fitts’s Law ExPERIMENT software was installed on this workstation. 76 In both the remote and local condition, besides the workstations, a Sony mini-DV 3.0 camera mounted onto a Velbon tn-pod was set up in the UBC lab for video taping each STUDY session. 4.3.2 UQAM Apparatus Figure 4.6: LORJT Lab The remote lab, LORIT (Figure 4.6), is a multi-media video-conferencing lab located at the UQAM. From now on, LORIT refers to the remote lab. 77 4.3.2.1 Workstation hardware There are six PC workstations in LORIT, each of them equipped with Intel Pentium III CPU with 1 GB RAM, P/S2 Optical Mouse, ViewSonic Graphics Series G5773 19” display, QWERTY Keyboard, and Sony EVI D30 short range microphones. In the remote condition, we only utilized two of the workstations, Workstation No. 6 and No. 8. The SUBJECTs at LORIT only used Workstation No.6, the primary workstation in the remote condition, to do the Fitts’s Law EXPERIMENT. Regularly, all the workstations at LORIT are connected to regular Internet; however, to support our remote STUDY condition, the network connection of this station was temporarily switched to Lightpath and communicated directly with EXPERIMENTER’s dominant workstation at UBC.. Workstation No. 8 was used by the LORIT lab assistant as a separate channel to communicate with the UBC lab assistant, and this workstation communicated directly with the workstation of EXPERIMENTER’s lab assistant at UBC. LORIT is also equipped with three cameras controlled by an auto-controlling system. Our STUDY only used one of the cameras, Camera No. 2, to capture the image of the PARTICIPANTs. The audio of the PARTICIPANTs was picked up by the short-range microphones on the table of the primary workstation. There is a console right beside the workstation of the PARTIcIPANTs. It was used by the remote lab assistant to control all devices and equipment within the lab. 78 4.3.2.2 Workstation software To enable video-conferencing, each workstation was installed with Windows XP Pro SP2, Access Grid version 3.0, VNC (virtual network connection), and Windows Media Encoder. The Fitts’s Law software was installed on workstation No. 6. Workstation No. 8 was temporarily installed with Microsoft windows messenger (msn). 4.3.3 Experiment Software We implemented the Fitts’s Law EXPERIMENT software (mentioned in Chapter 3) as an application for conducting Fitts’s Law trials. In the remote condition, the Fitts’s Law ExPERIMENT software was pre-installed on the primary workstation, which was Workstation No. 6 in LORIT; in the local condition, the same software was still pre-installed on the primary workstation, which was the only workstation at UBC (Figure 4.5). 4.4 Setup 4.4.1 Remote setup The remote condition required communication and camera equipment to be set up, since an EXPERIMENTER and a SUBJECT were located in separate geographical locations. In this condition, both parties had to observe and communicate with each other via the video and audio channels provided by a video-conferencing 79 infrastructure over Lightpath (fiber-optical network). The following setups were performed: 4.4.1.1 Video-conferencing software setup Lab assistants on both sides started running video-conferencing software and tried entering the same web-conferencing venue. Once both parties had successfully entered the conference venue, both lab assistants tested the video and audio channels to ensure that video and audio transmissions were running properly. On the right side of an EXPERiMENTER’s screen, two video windows were arranged in a format shown in Figure 4.7. The lower window showed the image of the EXPERIMENTER. The upper video window showed the image of a remote SUBJECT. On a SUBJECT’s screen, there were two video windows arranged as shown in Figure 4.7. The top window showed the SUBJECT at the remote lab, and the bottom window showed the ExPERIMENTER at UBC. The window arrangement was specifically designed to show a remote PARTICIPANT on the upper window and the local PARTICIPANT on the lower window. We assumed that a PARTICIPANT would naturally pay more attention to the remote party became positioning a window in a higher location on the screen draws more importance. 80 Figure 4.7 Video windows 4.4.1.2 Video camera setup EXPERIMENTERS at UBC were required to adjust one of the cameras in LORIT to have a clear view of their SUBJECTS. With a lab assistant’s help, a UBC EXPERIMENTER opened a web browser on EXPERIMENTER’S supplementary workstation and loaded the web interface of LORIT lab to configure a designated video camera, until a satisfactory view of a SUBJECT at LORIT was achieved. If a SUBJECT in LORIT could not observe an ExPERIMENTER properly, the SUBJECT had to ask the ExPEFUMENTER to adjust the UBC camera manually to achieve a proper view, because the UBC camera was not attached to any camera-control system. 4.4.1.3 Desktop sharing setup Since a SuBJECT only used the primary workstation, the Fitts’s Law ExPERIMENT software was run on the desktop of the workstation. To view the desktop of the 81 SUBJECT’S workstation in LORIT, a UBC ExPERIMENTER needed to activate a desktop sharing application, VNC, and establish a connection with the SUBJECT’S workstation. Once the VNC connection was made, a shared view of the SUBJECT’s desktop with the view of the ExPEIUMENT application window was positioned in the center of the screen of the ExPERIMENTER’s dominant workstation. Both PARTICIPANTs could interact with the application in the VNC window. 4.4.2 Local setup The setup of the local condition was much simpler. It only required activating and running the Fitts’s Law ExPERIMENT software on the primary workstation, the SUBJECT’S workstation at UBC. 4.5 Data Collection 4.5.1 Logging To monitor PARTICIPANTS’ performance, a logging widget was embedded into the ExPERIMENT software. It was also written in Java. We used the logging widget to extract several pieces of information, including PARTICIPANTs’ full name, coordinates of target positions, time taken to click a starting target position to an ending position, target number, and the labeling number of each trial. 82 4.5.2 Questionnaire At the end of each ExPEIUMENT session, PARTICIPANTS were asked to complete a short questionnaire (Appendix A). The questionnaire contained two parts. The first part was a demographic information survey, and the second part was the actual EXPERIMENT survey. Questions 3, 6, and 7 were adopted from Hauber et a!. (2006). PARTICIPANTs were given a specific questionnaire according to the roles that they played in the STUDY. Table 4.2 gives the detailed questionnaire design. The actual EXPERIMENT survey part of the questionnaire consisted of five parts: • Interaction: PARTICIPANTS were asked to rank their interaction experience. • Training: one statement about their level of confidence after going through a training session. • Videoconferencing: six statements asking about PARTICIPANTs’ experience using the video-conferencing system to conduct the EXPERIMENT in the remote condition • Setup: five statements asking about PARTICIPANTs’ interaction expenence with the LORIT interface. • Free form comments 83 Table 4.2: Sections in the questionnaire Condition Remote Local PARTICIPANT EXPERIMENTER SUBJECT EXPERIMENTER SUBJECT Interaction (Qi— Qi 1) Applied Applied Applied Applied Training (Q12) Applied Applied Applied Applied Video- Applied Applied N/A N/A conferencing (Q13 - Q18) Set-up (Q19-Q23) Applied N/A N/A N/A Free form Applied Applied Applied Applied comments There were twenty-three questions in total, divided into four sections, with four versions of the questionnaire depending on the role a PARTICIPANT played and whether the PARTICIPANT was in the local or remote condition. Only PARTICIPANTs in the remote condition who played the role of EXPERIMENTER were asked to answer all twenty-three questions. Those who played the role of SuBJECTs rn the remote condition were asked about video conferencing, but not about the set up remote procedure. PARTICIPANTS in the local condition were asked neither of these sets of questions. 4.5.3 Interview After the questionnaires were completed, all EXPERIMENTERS were required to participate in a post-EXPERIMENT interview (Appendix B). We used this interview 84 to capture any interesting issues that were missed in the questionnaire. Table 4.3 gives the detailed interview design. The interview consisted of two parts: • Role-playing issues: EXPERIMENTERS were asked to comment on their experience of playing the EXPERIMENTER’s role and the problems they encountered when running the EXPERIMENT. • Video-conferencing issues: EXPERIMENTERS were asked to describe the problems or concerns that they experienced while using the video conferencing software and in setting up a LORIT’s camera. As shown in Table 4.3, remote EXPERIMENTERs were asked to give free form feedback on all four questions, and local EXPERIMENTERs were asked to only give free form feedback on the first two questions. Table 4.3: Semi-structured interview schedule Condition Remote Local Qi -2(role-playing) Applied Applied Q3 -4(video-conferencing) Applied N/A 4.6 Procedure One pair of PARTICIPANTs was present for every 1.5-hour session. Upon their arrival, PARTICIPANTs were given a consent form, which outlined: 1) the goal of 85 the EXPERIMENT, 2) the general procedure, 3) the anonymity of the ExPERIMENT, and 4) a PARTIcIPANT consent text, which required a signature. According to the role that a PARTICIPANT was assigned to play, specific oral instructions that described the tasks of the STuDY were delivered. After any questions with respect to the tasks were answered, each PARTICIPANT started preparing for the ExPERIMENT separately. In the local interaction condition, an ExPERIMENTER was first trained face-to-face by a local lab assistant to go through the whole experimental procedure (Figure 4.8). The purpose of this was to make an ExPEJUMENTER understand the objectives and gain the experience of being a SUBJECT. After the training, an EXPERIMENTER met with a SuBJECT and started delivering instructions to the SuBJEcT and running the EXPERIMENT. 86 Figure 4.8: Procedure of local EXPERIMENTER (EXPERIMENTER procedure __j) Given a consent form, which outlines 1) the goal of the EXPERIMENT, 2) the general procedure, 3) the anonymity of the EXPERIMENT, 4) a PARTICIPANT consent text, which is to be signed by them. Specific oral instructions are delivered, which describe the tasks of the STUDY After potential questions with respect to the tasks are answered, PARTICIPANT starts preparing for the EXPERIMENT First trained face-to-face by a local lab assistant to go through the whole EXPERIMENT procedure No Running EXPERIMENTs Blocki BIock2 Block3 Block4 144 EXPERIMENTER filled out questionnair form JEnd D 87 Still under the local condition, when an EXPERIMENTER was in training, a local SuBJEcT was given some information about Fitts’s Law. Once the training was done, that SuBJEcT would rejoin an ExPERIMENTER. Following the EXPERIMENTER’s guidance, the SuBJEcT then started doing a trial session of the STuDY as a warm-up. After the trial session, the SUBJECT was then asked to move onto the actual EXPERIMENT (Figure 4.9). 88 Figure 4.9: Procedure of local SUBJECT In the remote condition, there were some additional steps. In this condition, an EXPERIMENTER was first trained by the lab assistant in LORIT via a video conferencing system to go through the whole ExPERIMENT (Figure 4.10). If any Following the EXPERIMENTER’s guidance, the SUBJECT then starts doing a trial session of the STUDY as a warm-up No SUBJECT filled out questionnair form 89 problems came up, the EXPERIMENTER would ask for help from the co-located lab assistant. Once the training was complete, the ExPERIMENTER was required to perform the video camera setup mentioned in Section 4.4.2. After the cameras were set up properly on both sides, then the ExPERiMENTER was required to perform the desktop sharing setup and ran the Fitts’s Law EXPERIMENT just as with the local procedure. If problems occurred, the EXPERIMENTER would either intervene directly via VNC or instruct the remote lab assistant to physically perform the intervention. Figure 4.10: Procedure of remote EXPERIMENTER 90 (Remote EXPER IMENTER ‘s procedure 4 EXPERIMENTER training: 1.The EXPERIMENTER sat in front of a running video- ccnferencing system — 2.The EXPERIMENTER was trained by a remote lab assistant via the video conterencing infrastructure to walk through whole FiUs’s law EXPERIMENT Request help from lab assistant Camera Setup: 1.Load up the LORITID (remote lab interface) on the secondary workstation 2.On this interface, the EXPERIMENTER need to find the correctcamera icon i adjust one of the remote cameras to acquire a clear view of the remote subject i II the subject that the EXPER IMEN TER is setting up the camera i check the webcam of the EXPERIMENTER to ensure the subject also see a clear vias of the EXPERIMENTER once the both parties agree on the camera setup, then introduced to each other 4 Desktop setup: i ask theSUBJECT to activate the EXPERIMENTsoftware i the EXPERIMENT accesses the SUBJECTs workstation via VNC (virtual network connection). Request help from lab assistant 1 Run the Fitts’s Ia vu EXPERIMEN Tjust as the local procedure. EXPERIMENTER spoke to the subject via a microphone, listened to the SUBJECT via a louderspeaker observed the SUBJECT via the video window in the desktop. If problems, concerns, or mistakes come, up either intervene with the SUBJECT directly via VNC or communicate with the remote lab assistant and let him physically intervene with the SUBJECT , 4 BlocIci BIcck2 Block3 BIock4 I 4 t4a 4 EXPERIMENTER filled cut questionnaire form 4 I Interview 4 ( End ) 91 Still in the remote condition, before the EXPERIMENT could be started, a SuBJEcT was required to cooperate with his/her EXPERIMENTER to accomplish the desktop sharing necessary to ensure that both parties shared the same view of the SuBJEcT’S desktop. Then, the remote SUBJECTs followed the same procedure as the SUBJEcTs in the local condition (Figure 4.11). Under both conditions, after completing the Fitts’s Law EXPERIMENT, SUBJECTS and EXPERIMENTERS were both required to fill out a questionnaire addressing different communication and interaction issues. Figure 4.11: Procedure of remote SUBJECT 92 C Remote SJ8JECT procedure) Desktop sharing setup: ask the SuDJEcT to activate the experiment softre Te EXPERIMENT accessthe SuBJECT’s orkstation via ‘iNC (‘Artual net’nrkccnnedion). “p 11 Request help from EXPERIMENTER or lab assistant I lntervie I 4 End 3 Gi’ve some information of Fitts’s law Folio Wng the instructions of EXPERIMENTER Dothe EXPERIMENT accordingtothe instruction of EXPERIMENTER - speak to the EXPE RIME NTER via a microphone, - listen to the EXP ERIMENTER Aa a louder speaker - observe the EXP ERIMENTER ‘yia the video v’indowin the desktop. If problems happened either intenened ‘Aa ‘iNC or physically corrected by collocated lab assistant 1 SuBJE CT filled out questionnair form 93 In addition, PARTICIPANTS in both conditions who played the role of an ExPERIMENTER were also interviewed after each ExPERIMENT session. Depending on the conditions, the PARTICIPANTs were asked either two or four questions to evaluate their experience of being an ExPERIMENTER. At the end of a STuDY session, the PARTICIPANTs were thanked and financially compensated. 4.7 Summary In Chapter 3, we have listed the design of our Fitts’s Law EXPERIMENT; here we give a complete summary our STUDY design in terms of variables and measures (Table 4.4). Table 4.4: Our STUDY design Independent Variables Dependent Variables 1. Collaboration Mode r2, r, intercept, slope, time spent (between-subject) on each Fitts’s Law task, and --- Co-present and time spent on a Fitts’s Law Non co-present ExPERIMENT, subjective measures 2. Target Width (within- Movement time, number of subject) errors --- 8, 16, 32, and 64 pixels) 94 Chapter 5 Results Our hypotheses were (1) that SUBJECTS’ performance in an ExPERIMENT would be affected by the co-presence of their ExPERIMENTER and also that (2) the interaction experience between ExPERIMENTERs and SUBJECTS would depend on whether or not the ExPERIMENTER and SUBJECT were co-present. In other words, we expected that the face-to-face interaction condition would be a richer mode of collaboration, and that the time it took to complete a block of tasks would be significantly less with co-presence (Hauber et al., 2006). Furthermore, we predicted that the amount of attention shift experienced by a PARTICIPANT would depend on whether the EXPERIMENTER and SuBJECT are collocated or not. There were 21 pairs of PARTICIPANTS in the STUDY; each pair undertook 12 Fitts’s Law tasks for a potential total of 252 Fitts’s Law tasks. One pair of PARTICIPANTs (#4 in the remote condition) was excluded from analysis because a network outage that occurred during set up, leaving 20 valid PARTICIPANT pairs with 240 tasks to be analyzed. Each task had 40 movement trials, all with the same target size and thus the same index of difficulty. Trials for which target acquisition took more than 3000 95 milliseconds (3 seconds) were excluded from the analysis (25 of the 9600 trials were excluded). The average movement time was computed for all remaining trials having the same index of difficulty, yielding four data points for each pair of PARTICiPANTs. These were used in the analysis. To test our hypotheses, we looked at both quantitative and qualitative data. Quantitative data was used to compare SUBJECT’S performance in the two conditions and subjective measures of comfort provided by both the SUBJECTs and the EXPERIMENTERs. Qualitative data obtained from interviews and observations made during the experimental sessions were used to assess effectiveness. Both quantitative data and subjective measures were analyzed using the SPSS statistical package (version 15.0). Each analysis is discussed in turn. 5.1 Fitts’s Law Performance Our primary concern was that the remote condition might fail to adequately support the experimental process, perhaps due to increased PARTICIPANT frustration, poorer data cohesion, lengthier STUDY sessions, or more problems in communication. We tested for data congruence between co-present and non co present conditions using two quantitative measures obtained from the Fitts’s Law data collected in the Fitts’s Law tasks: the correlation coefficient, i and the coefficient of determination, r2. We used these as measures for the “quality” of the experimental process because they are direct indicators of statistical 96 significance. Moreover, we chose these two measures because Fitts’s Law is a well-studied phenomenon that typically yields high r2 values for its various formulations, including Welford’s. Preliminary analyses were conducted to discover any violation of the assumptions of normality, linearity, and homoscedasticity of the values of r and r2. None were found. Following MacKenzie’s test of correlation and linear regression analysis (MacKenzie, 1991), we computed r and r2 for each SUBJECT-EXPERIMENTER pair. The operational hypothesis that sought to reject (as stated in Chapter 4) was the following. Hal.1: Statistically, the validity of the Fitts’s Law ExPERIMENT data obtained in the co-present condition is better than those obtained in the non co-present condition As we predicted, the values of r and r2 between the two variables, index of difficulty (ID) and movement time (MT), were consistently high for both STUDY conditions (Table 5.1). Table 5.2 shows the means and standard deviations of r and r2. In all of the 20 pairs, r was at least +0.780, indicating a strong correlation. 97 Table 5.1: r and r2 values obtained from remote and local conditions Remote Local 2 2 r r r r 0.808 0.653 0.899 0.808 0.991 0.982 0.843 0.711 0.941 0.885 0.780 0.608 0.992 0.984 0.999 0.998 0.972 0.945 0.998 0.996 0.954 0.910 0.921 0.848 0.9 13 0.834 0.996 0.992 0.934 0.872 0.989 0.978 0.989 0.978 0.992 0.984 0.996 0.992 0.985 0.970 Table 5.2 Descriptive statistics of r and r2 Condition Remote Local 2 2Measure r r r r Mean 0.949 0.904 0.940 0.889 SD 0.057 0.104 0.776 0.139 Across the conditions, the mean values for r were 0.949 and 0.940; the mean values for r2 were 0.904 and 0.889 in the remote and local conditions respectively. 98 Figures 5.1 presents a scatter plot of the relation between the ID and MT for all the SUBJECTs’ performance in both remote and local conditions. Individual scatter plots (remote and local conditions) which indicate the relation between the ID and MT are listed in Appendix D. Figure 5.1: Scatter plot for remote and local conditions E I- Remote Subjects ID a 1.010 1.010 Local 0 2.010 2.010 a 1.020 ‘—. 1.020 Subjects 0 —..... 1.030 1.030 2.040 2.040 0 1.050 —.1.050 Q 2.050 —.2O5O 1.060 1.060 2.060 2.060 0 1.070 -1.070 0 Q 1 .080 1 .080 2.090 2.090 1.090 1.090 C) 2.100 . 2.100 1.110 1.110 C) 1.120 99 As MacKenzie did to compare the values of r produced by different Fitts’s Law models (1991) we then conducted an independent samples t-test to determine whether the local condition achieved better (stronger) correlations r and coefficients of determination r2 than the remote condition. There was no significant difference for the values of r computed for the remote or the local condition (t(18) = .289, p = .776, effect size 112 = .005), nor was there any significant difference for the values of r2 between the two conditions (t(18) =.257, p =.800, effect size r12 = .004). Non-significant trends were not supportive of our hypothesis that remote experimentation might negatively affect SUBJECT performance. The high p-values for all of these indicate that we can reject the operational hypothesis Hal .1 because the local condition achieved no better correlations and coefficients of determination than did the remote condition. 5.2 Analysis of Slopes and Intercepts Given the values of correlations, r, and coefficients of determination, r2 obtained from both conditions, as measured by the statistical significance of the outcomes obtained, did not significantly differ between whether ExPERIMENTER was local or remote, we then analyzed the intercepts, A, and slopes, B1, of the correlation equations to see whether there were differences in SUBJECTs’ performance between the two conditions. The values of A1 and B1 are listed in Table 5.3. Means and standard deviations for slope and intercept are listed in Table 5.4. 100 Table 5.3 Intercept A and slope B (one anomalous value is highlighted) Remote Local A, B, A B 205.300 583.680 126.880 788.290 232.260 308.350 177.710 835.580 173.790 615.670 23.360 1211.730 174.750 501.290 281.640 472.810 233.870 519.990 226.380 452.440 131.170 530.290 163.260 890.060 220.580 635.750 234.130 442.190 207.100 525.960 264.880 503.780 233.910 577.890 142.460 440.710 174.480 609.690 179.560 654.810 Table 5.4 Descriptive statistics for slope and intercepts Condition Remote Local Measure A, B, A B Mean 182.025 540.858 198.719 669.237 SD 75.740 93.653 34.217 258.589 5.2.1 Intercept coefficient A• The A, intercept values obtained from the remote and the local conditions were all close to MacKenzie’s reported value of 230 (MacKenzie, 1992) except for one 101 pair in the local condition (the third pair in Table 5.3). The mean A1 for the remote condition was 182.025, and the mean A, for the local condition was 198.7 19. Preliminary analyses were conducted to discover any violation of the assumptions of normality and equal variance of the A, values. None were found. A parametric independent t-test was run to compare the A, values obtained from the remote and local conditions. There was no significant difference in A values between local studies and remote studies (t(18) = .635, p = .533). The magnitude of the difference of the A, means was small (2 = .020). 5.2.2 Slope coefficient B The B, values obtained from the remote and the local conditions were larger than MacKenzie’s reported value of 166 (MacKenzie, 1991). It is possible that this was due to the shape of the targets used in the Fitts’s task as MacKenzie used rectangular targets and our STUDY used circular targets. The mean B for the remote condition was 540.858, and for the local condition it was 669.237. Preliminary analyses were conducted to discover any violation of the assumptions of normality and equal variance of the A- values. None were found. A parametric independent t-test was conducted. There was no significant difference in B values for local studies and remote studies (t(11.321) = —1.476, p .167). The magnitude of the difference in the mean was moderate (ri2 = .100). 102 5.2.3 Performance time A similar analysis was conducted for performance time, which is modeled by the intercept and slope coefficients. As would be expected, no significant differences were found between the local and remote conditions, and the mean trial times were similar to what Mackenzie reported (1991). This suggests that even though the apparatus used at UBC and UQAM were not the same, this did not significantly affect the performance time of SUBJECTS or the device (mouse) constants, despite the moderate effect size for the larger slopes in the local condition. It may be that the Fitts’s Law STUDY is not very sensitive to minor apparatus differences; and thus any differences between local and remote apparatus would not significantly affect the results. Based on the results of sections 5.2.1 and 5.2.2, regardless of co-presence or non co-presence, the data obtained in the co-present condition is similar to than that obtained in the non co present condition. Thus, we can thus reject the first two of the operational hypotheses, Hal.1 and Hal.2. 5.3 Errors To determine whether the second part of the operational hypotheses Hal.1, that performance would be more error-prone, we need to look at errors. An error in a trial was defined as pointing and clicking outside the target. As known in Table 5.5, the error rates for both local and remote conditions were more than double 103 than the recommended rating of 4% (MacKenzie & Buxton., 1992), with means of 10.208 % and 9.542 % respectively. The error rate was analyzed using an independent samples t-test. The statistical difference between the remote and local error rate was not significant (t (18) = .226, p = .824, effect size 2 = .003 small effect). This suggests that SUBJECTS in the remote conditions produced no more errors than those in the local conditions. Table 5.5 Descriptive statistics of error rate in remote and local conditions Remote Local Mean 10.208 % 9.542 % SD 6.52 1 % 6.668 % Based on the results of section 5.2 and this section, we can fully reject the operational hypothesis (Hal .2) that performance of local SUBJEcTs would be better (faster and less error-prone) than that of remote SUBJEcTS in the Fitts’s Law ExPERIMENT is rejected. Furthermore, the results suggest that the co-presence of EXPERIMENTERs does not necessarily improve SuBJECTs’ performance in a Fitts’s Law ExPERIMENT. 5.4 Analysis of time to run a Fitts’s Law ExPERIMENT Although the outcome indicated there was no difference in the quality of the experimental data, as measured by the degree of correlation found in the data, or in the physical performance of SUBJECTs, as measured by the intercepts and slopes of the correlation equations between the local and remote conditions under 104 a Fitts’s Law STUDY, we realized that there might still be a difference in how long it took to run the EXPERIMENT. To determine whether there were such time differences between local and remote conditions, we first examined the total time it took to conduct each of the four blocks of tasks at the four levels of difficulty, taking into account that higher indices of difficulty required longer trial times. We then examined the total amount of time (session time) taken to complete the ExPERIMENT in the remote and local conditions. 5.4.1 Total time spent at each task block at each index of difficulty As mentioned above, we first look at SUBJECTs’ total performance time on completing each of the four task blocks at each index of difficulty in the remote and local conditions. Table 5.6 summarizes the performance time of each task block for both remote and local SUBJECTs. The descriptive statistics of the data are listed in Table 5.7. 105 Table 5.6: Mixed between and within STUDY: condition by index of difficulty . . ID1 1D2 1D3 1D4Condition (seconds) (seconds) (seconds) (seconds) 100.750 114.578 173.734 170.906 70.562 98.172 122.141 158.406 106.797 115.547 140.515 196.672 89.000 200.703 157.609 167.000 99.953 137.140 205.828 197.984 Remote 76.234 104.781 73.641 91.907 116.922 133.718 145.828 200.828 72.156 101.688 131.094 189.875 113.906 262.235 149.422 200.657 119.031 155.625 146.719 175.375 113.187 133.797 141.297 189.375 135.094 151.062 156.140 238.140 158.391 205.141 161.609 184.375 100.437 133.031 177.547 225.610 95.063 114.797 146.406 190.234 Local 127.328 164.953 165.110 230.516 185.704 123.219 296.313 282.047 105.125 143.468 161.703 223.922 84.468 118.718 114.781 143.969 80.563 85.453 113.188 126.250 106 Table 5.7 Descriptive statistics of performance time between four different indices of difficulty and two STUDY conditions Condition Remote Measure ID1(seconds) 1D2(seconds) 1D3(seconds) 1D4(seconds) Mean 96.531 142.418 144.653 174.961 SD 18.547 52.240 34.213 33.007 Condition Local Measure ID1(seconds) 1D2(seconds) 1D3(seconds) 1D4(seconds) Mean 118.536 137.363 163.409 203.443 SD 33.567 32.254 51.196 20.343 5.4.1.1 Within-subjects effect The means and standard deviations of performance time for each index of difficulty are presented in Table 5.8. There was a significant main effect for total point-and-click time among four different indices of difficulty: ID1, 1D2, 1D3, 1D4 (F (3,54) = 48.249, p < .001, r = .900). This confirmed the Fitts’s Law hypothesis that as target sizes increase, the time taken to acquire a target decreases. Table 5.8 Descriptive statistics of total time spent in each task block corresponding to each index of difficulty. ID1(seconds) 1D2(seconds) 1D3(seconds) 1D4(seconds) Mean 107.533 139.891 154.031 189.202 SD 28.707 42.334 43.458 41.750 107 5.4.1.2 Between-subjects effect The means and standard deviations for each STUDY condition are presented in Table 5.9. There was no significant main effect in the target acquisition time between remote and local conditions (F (1,18) = 1.382 p = .255, ii2 = .071). The magnitude of the effect size was moderate. This indicates that although data did not demonstrate any statistical significant difference, in practical terms, it took SuBJECTs moderately more time on average to complete all twelve tasks in the local condition than the same tasks in the remote condition, as shown in Figure 5.2 Table 5.9 Descriptive statistics of total time spent to complete the all of the twelve tasks Remote Local (seconds) (seconds) Mean 139.641 155.683 SD 75.863 58.693 5.4.1.3 Interaction According to the operational hypothesis Hal .3, we predicted that there might be an interaction between how long it took to run the block of point-and-click tasks for each index of difficulty and for each STUDY condition. A mixed between within subjects ANOVA was conducted to analyze the total amount of time SUBJECTS spent in each of the four blocks between remote and local conditions. 108 Preliminary analysis for within-subject conditions (four indices of difficulty) and between-subject conditions (remote and local STUDY condition) was done to determine if there was any violation of the normality, equal variance, and sphericity assumption. Sphericity was not satisfied. We therefore report the multivariate test results rather than an ANOVA as recommended by statistical textbooks. There was no significant interaction between the indices of difficulty and the STUDY conditions (F (3,54) = 1.193, p = .034, Wilks’ Lambda = .8 17). Figure 5.2 Total time spent to complete the all of the twelve tasks Estimated Marginal Means of MEASURE_I 220000.0000 condition 200000.0000- C 180000.0000- . 160000.0000- I 140000.0000- 120000.0000- 1 00000.0000- 80000.0000- ID Remote Local 109 5.4.2 Total time to complete the ExPERIMENT in the remote and local conditions An independent samples t-test was conducted to compare the total amount of time taken to conduct the twelve point-and-click tasks in remote and local conditions. Descriptive statistics of the data are shown in Table 5.10. Preliminary analysis was performaned to check normality and equal variance. None was found. There was no statistically significant difference in total task completion time between the remote condition (mean = 640.261, SD = 150.51) and the local condition (mean =747.741, SD = 154.010, F(18) = —1.494, p = .153). The magnitude of the effect is moderate (r2 = .120). This suggests that local SUBJECTS did take more time to complete the ExPERiMENT than remote SUBJECTS did. Table 5.10 Descriptive statistics of total time spent in remote and local EXPERIMENTS Remote Local (seconds) (seconds) Mean 640.261 747.741 SD 150.51 154.010 110 5.5 Questionnaire analysis 5.5.1 Interaction Survey A quantitative analysis was made of the subjective measures collected by questionnaire (Appendix A) of PARTICIPANTS’ experience of the two roles in the remote and local conditions. For each question, PARTICIPANTS are asked to rank their experience from 1 to 5, with 1 means strongly agree and 5 means strongly disagree. The survey results were analyzed using SPSS (version 15.0). Each of the questions measured a single aspect. The descriptive statistics were calculated for each question separately, as shown in E.l in Appendix E. In this section, we discuss only questions that presented a statistical or practical difference. Differences were looked for between SUBJECTs and ExPERIMENTERs within a condition, as well as across conditions. Each STUDY session was designed to accommodate one EXPERIMENTER and one SUBJECT as they were interacting with one another. It was predicted that they would share similar experiences of a STUDY and thus form a comparable pair. The pair-wise responses from the PARTICIPANTs can be found in Table E.2 Appendix E. The EXPERIMENTERs in the local and remote conditions formed two independent groups, as did the SUBJECTs. A mixed between-within subjects ANOVA should have been applied to the questions answered by EXPERIMENTERS and SUBJECTS in both conditions; however, none of the results of these questions satisfied the assumptions of normality or equal variance. Because of this, instead of 111 conducting an ANOVA, a paired-sample t-test was conducted for each of the twelve questions asked of both SuBJEcTs and EXPERIMENTERS. PARTiCIPANTs playing the same role in the remote and the local conditions had their survey results compared by two-sample independent t-tests, listed in E.3 in Appendix E. For remote EXPERIMENTER-SUBJECT pairs, the survey results were analyzed by paired sample t-tests for matched-up questions. The same applies to local STUDY pairs. The results are shown in table E.3.l and E.3.2 Appendix E. 5.5.1.1 Measures between conditions based on roles Only Questions 3 and 8 showed any significant differences related to our hypotheses. Further analysis shows that there is a large practical difference between the responses of ExPERIMENTERs in Question 3 and 8. Question 3: “It was easy to talk to and hear the SuBJEcT/ExPERIMENTER “. PARTICIPANTS answered on a scale of 1-5, with 1 being a high (good) ranking, and 5 being a low (bad) ranking. This question was ranked low by remote EXPERIMENTERS, with a mean of 4.6, while local EXPERIMENTERS ranked the same question high, with a mean of 1.6. There is a significant and practical difference between the remote and local EXPERIMENTERS (Z (18) = -2.006, p = .045, 92 = .180 (large effect). 112 Question 3 ranks the ease of talking to and hearing the other party in the EXPERIMENT. As expected, the results show that there is a significant difference on the EXPERIMENTERS’ side. In the local condition, PARTICIPANTs communicated with each other face-to-face, as oral communication was not obstructed by any factor. In the remote condition, PARTICIPANTS communicated with each other via microphones. Although the audio equipment was inspected beforehand to ensure that both parties could hear each other clearly, the results of Question 3 revealed that UBC PARTICIPANTs still experienced difficulty when listening to their remote party. As an observation, one factor that may have contributed to this is that some UQAM SUBJECTS spoke English with a French accent, which might have posed some communication barriers to the UBC PARTICIPANTS. While this might Suggest that co-presence is a richer interaction condition than the non co-presence, the confound of some remote SuBJECTs speaking with an accent means we cannot draw any definitive conclusions from these results. Question 8: “Overall, from my own point of view, how well did the EXPERiMENT go?” PART1CIPANT5 again answered on a scale of 1-5, with 1 being a high (good) ranking, and 5 being a low (bad) ranking. The rankings for this question by both remote and local EXPERIMENTERS was high, with mean of 1.9 for remote EXPERIMENTERS and 1.5 for local EXPERIMENTERs. There is a larger practical 113 difference for remote and local EXPERIMENTERS (Z (18) = -1.523, p = .128, and effect size = .130 (large effect). Question 8 inquires about the overall satisfaction of the ExPERIMENT. Local EXPERIMENTERs ranked this question with a higher score than the remote ones. This suggests that EXPERIMENTERs experienced more satisfaction in the local condition. This suggests that the richness of the face-to-face condition enables more interaction and that it leads to higher participation satisfaction than the remote condition. 5.5.1.2 PARTICIPANTS’ experience within each condition Among the pair-wise measurements respectively in the remote and the local conditions, only Question 6 showed any significant differences related to our hypotheses. The analysis of Question 6 shows a large practical difference (though not a statistically significant difference). Question 6: “I was aware ofmy EXPERIMENTER /SUBJECT’s presence all the time.” In the remote condition, the ranking of this question by EXPERIMENTERs was equal to that of SUBJECTs, with a mean of 2.1 for both EXPERIMENTERs and SuBJECTs. In the local condition, EXPERIMENTERs ranked this question higher than their SUBJECTS, with a mean of 1.4 for EXPERIMENTERs and 2.6 for SUBJECTS. There was a larger practical difference between the local EXPERIMENTERs and 114 SUBJECTS (Z (9) = -1.897, p = .058, effect Size = .280 (large effect), which was almost statistically significant at the pc05 level. Question 6 measures the awareness of the presence of the other party. Local EXPERIMENTERS indicated that they were more aware of their SUBJECTs’ presence than were remote EXPERIMENTERS. This suggests that the physical presence of SUBJECTS did lead to EXPERIMENTERS being more aware of them. In summary, the results of these questions confirm one of our initial hypotheses (Ha2): that the PARTICIPANTS did experience richer human-to-human interaction in the co-present or face-to-face condition than in the remote condition. 5.6 Interview analysis Qualitative analysis was also performed on the feedback collected from post- STUDY interview about PARTICIPANTs’ experience in the remote and local conditions. Not only did we discover important facts that test our hypotheses, but also other interesting factors about the multimedia aids used to support our STUDY. Question 1: “How was your experience ofbeing an ExPERIMENTER”? Question 2: “What kind ofproblems did you experience when interacting with your SUBJECT?” 115 Interview question 1 and 2 ask about EXPERIMENTERs’ rnteraction experience. From these questions, we typically discovered evidence that refutes our third hypothesis (Ha3). EXPERIMENTERs’ responses to this question varied. Regardless of the video, audio, and VNC issues, our pre-STuDY assumption was that a local EXPERIMENTER would experience less attention shift than a remote ExPERIMENTER. However, when interviewing the local EXPERIMENTERS, we found that 6 out of 10 EXPERIMENTERs commented that being a local EXPERIMENTER was very stressful. When interviewing the remote EXPERIMENTERS, only 1 out of 10 EXPERIMENTERs made the same comment. During the interviewS, local EXPERIMENTERS explained the stressful aspects of their experience. The EXPERIMENTERS complained that it was hard for them to read the instructions and look at their SUBJECTS at the same time. They often commented that when delivering instructions, they had to interpret the facial expression of the SUBJECTs as it provided them with the feedback of instruction comprehension. However, after checking their SUBJECT’S facial expression, it was hard for them to go back to their instruction sheet and continue from where they left off. This suggests that local EXPERIMENTERs have to split their attention between two tasks at the same time: one was delivering the instructions by paying attention to the instruction sheet, and the other was interpreting their SUBJECTs’ facial expression. Somehow, the remote condition allowed a virtual distance between PARTICIPANTS and reduced the amount of attention shift between 116 EXPERIMENTERS and SUBJECTs — one remote EXPERIMENTER commented that it was easier to play the role of an ExPERIMENTER. Question 3: “Between audio and video, what media served a more important role in communication during the ExPERiMENT?” (For remote EXPERIMENTERs) Question 3 asks about the importance of audio versus video. In contrast to pre STUDY assumptions, most PARTICIPANTs treated audio as a much more important medium than video. Post- STuDY interviews showed that 9 out of 10 remote PARTICIPANTs playing the EXPERIMENTER’s role found the audio channel more expressive than the video, and that information about interaction flowed in a more understandable way via audio. Question 4: “What was the most challenging part of setup?” (For remote EXPERIMENTERS) PARTICIPANTs came up with the following responses: Loading the interface PARTICIPANTs complained about their frustration of loading up LORIT’s interface on UBC’s workstations. PARTICIPANTs explained that they had to go through three extra steps before getting access to the website. After entering the website address in a web browser, besides providing a user name and a password for the login dialog box, PARTICIPANTs had to install the ActiveX component, cancel a 117 Microsoft NetMeeting invitation, and click twice on a green button before the interface showed up on screen. In addition, the web interface was only supported by Microsoft Internet Explorer version 6.0 or higher, which mean the interface cannot be loaded by Mozilla Firefox. There were several cases when PARTICIPANTS attempted to load the web interface on Mozilla Firefox and failed. 5.7 Summary The results of our STuDY provide useful information about the use of remote experimentation to support research on tasks similar to Fitts’s Law. Table 5.11 summaries our results by hypotheses. 118 Table 5.11: Summary of results Hypothesis Expected Results Actual Results Hal .1 Average r2 measured in the Average r2 measured in the local condition is better than local condition is NO better than Average r2 measured in the Average r2 measured in the remote remote Hal.2 Performance of local SUBJECTS Performance of local SUBJECTs is better (faster and less error- is NO better (faster and less prone) than that of remote error-prone) than that of remote SUBJECTs in the Fitts’s Law SUBJECTs in the Fitts’s Law ExPERIMENT EXPERIMENT Hal.3 Local SUBJEcTs spend less Local and remote SUBJECTs time to conduct all the tasks in spent equal amount of time to each task block and overall conduct all the tasks in each they spend less time to task block and to complete all complete all the Fitts’s Law the Fitts’s Law tasks tasks than that of remote SUBJECTS in the Fitts’s Law EXPERiMENT. Ha2 Interaction experience is richer Interaction experience was in the co-present condition than richer in the co-present the non co-present condition. condition than in the non co present condition. Ha3 Local EXPERIMENTERS would Local EXPERIMENTERs experience a less attention shift experienced more attention shift than the remote than the remote EXPERIMENTERs EXPERIMENTERS We believe that many of the experiments that are conducted by doctoral students by to evaluate educational technology fall into this category. The next chapter discusses some interesting interaction patterns observed from our STuDY and in Chapter 7 we draw a number of conclusions and recommendations that arise from our research. 119 Chapter 6 Discussion In this chapter, we discuss some interesting interaction patterns drawn from the real STUDY. Patterns of Interaction There are several patterns of interaction observed in remote and local conditions, and they are listed in Table 6.1. Table 6.1 Patterns of interaction Patterns of Interaction Local Remote SUBJECT <-> ExPERIMENTER observed observed EXPERIMENTER <-> lab ExPERIMENTER <-> EXPERIMENTER <-> assistant UBC lab assistant UQAM lab assistant EXPERIMENTER <-> UBC lab assistant SUBJECT <-> lab assistant SUBJEcT <-> UBC lab SuBJECT <-> UQAM assistant lab assistant UBC Lab assistant <-> N/A UBC lab assistant <-> UQAM lab assistant UQAM lab assistant 1). Interaction between ExPERiMENTER and lab assistants 120 As predicted, remote STUDY sessions suffered from large overheads for videoconference set-up and media sharing over the network. Thus, more interaction was observed in remote sessions than local sessions. In the training phase of a remote STUDY session, the UQAM lab assistant walked through the Fitts’s EXPERIMENT with a UBC PARTICIPANT. In the camera-setting phase, the UBC lab assistant assisted a PARTICIPANT in loading and opening the remote lab web interface and responding to any related questions. These questions ranged from lab equipment configuration, to activation status of equipment, and identifying the widgets that could be manipulated. When moving on to the actual EXPERIMENT phase, both UQAM and UBC lab assistants intervened or interacted with EXPERIMENTERs based on the context and occurrence of problems. Compared to the remote condition, local sessions required neither videoconference setup nor camera setting. As in a remote session, a local ExPERIMENTER needed to go through a training phase with a lab assistant. Observation showed that local EXPERIMENTERs only communicated with the UBC lab assistant for EXPERIMENT related concerns in a minor way. In the actual ExPERIMENT phase, a local ExPERIMENTER only interacted with the local lab assistant. 2). Interaction between SUBJECTS and ExPERIMENTERs In terms of interaction between SuBJECTs and EXPER1MENTER5, remote sessions resulted in more pair-wise interaction then the local. As mentioned previously, 121 audio served a more important role than video. Thus, poor audio directly encouraged EXPERIMENTERS and SUBJECTs to proactively communicate with each other to ensure a correct mutual understanding. Communication forms included asking a remote party to repeat an instruction, multiple checking of certain progress checkpoints, and reminding the obedience of the interaction protocol. Unlike remote sessions, local sessions did not require a dedicated audio channel. Thus, participatory pairs did not need as much interaction as the remote pairs, except for inquiring about what the SUBJECTs thought about the ExPERIMENT. 3). Interaction between SUBJECTS and lab assistants SUBJECTs were co-located with their lab assistants in both conditions. In the remote STUDY, when problems or errors occurred, instead of directly asking their EXPERIMENTERs, SUBJECTS preferred to approach their co-located lab assistant for responses. Co-location did provide PARTICIPANTs with the assurance of being able to receive instant feedback from a knowledgeable individual. This indicates that the presence of another person encouraged interpersonal communication. In the local STuDY, this interaction pattern did not manifest frequently. When issues arose, SUBJECTs waited for their EXPERIMENTERs to tell them how to deal with them. However, if SUBJECTs sensed their EXPERIMENTERs were not knowledgeable about the answer to a question or not aware of certain mistakes and a lab assistant had to interfere in some urgent context, then the interaction pattern showed up. If this situation occurred, EXPERIMENTERs would lose their 122 credibility of being a knowledgeable partner. Consequently, SUBJECTS preferred to turn to the lab assistant for any concerns or questions thereafter. We call this situation a role shift. 4). Interaction between two lab assistants This pattern only applied to the remote condition, in which lab assistants had to communicate with each other to check videoconference setup, including video and audio channel configuration. They also needed to communicate with one another to ensure that the PARTIcIPANTs were making progressing correctly in a STuDY session. 123 Chapter 7 Conclusions and Future Work 7.1 Conclusions We conducted an EXPERIMENT under both co-present and distributed conditions. Based on the results of our STUDY, we conclude that in terms of experimental performance, co-presence of ExPERIMENTERs does not necessarily affect SUBJECTs’ performance. In fact, SuBJEcTs’ performance did not show any statistically significant differences under either co-present or non co-present conditions. However, local SUBJECTs practically spent more time doing the EXPERIMENT than remote SUBJECTs, confirming Hauber et al.’s finding that “adding spatiality is capable of creating a collaborative context that is closer to face-to-face, but at the same time loses the efficiency of a task-focused two- dimensional interface.” (2006, p. 420). In terms of interaction, the interpersonal experience differed between ExPERiMENTERS and SUBJEcTs only in certain aspects, indicating that co-presence is richer but not necessarily more pleasant than non co-presence. In fact, those PARTICIPANTS acting as EXPERIMENTERs in the face-to-face condition experienced higher pressure. 124 When building our STUDY, we looked for ways to bring together commercial off-the-shelf software and available hardware and combine them to form a research infrastructure. Working with our research partners at UQAM, we lifted that environment to the point of supporting remote collaboration. Our STuDY design did not strive to provide our remote PARTICIPANTs with an illusory face-to- face experience, as such a focus would have missed the substantive point of promoting inventive forms of remote experimentation (Birnholtz & Horn, 2007). Instead, we designed our STuDY to make our PARTICIPANTs conscious of the remote experience while still fulfilling the goal of our STuDY. Given that we have successfully conducted our STUDY in a distributed condition, this confirms that traditional research experiments can indeed be conducted in a remote way when researchers are not co-located, or when equipment access is restricted due to geographical separation or facility shortcomings. 7.2 Lessons learned from our STUDY Our STUDY revealed that the PARTICIPANTS playing the EXPERIMENTER’s role had not acquired adequate knowledge of their tasks and of the procedure of the Fitts’s Law EXPERIMENT; PARTICIPANTs’ feedback revealing the importance of comprehensive familiarity with the experimental materials. More thorough training is essential; this includes practicing the experimental procedure many times until one clearly understands the expectations for procedure and progress. Furthermore, it would be worthwhile to run through the experimental instructions 125 several times and then encourage the ExPERIMENTERs to deliver them in their own words — in other words, more time should have been allocated to training. We predicted initially that capturing a clear and proper view of PARTICIPANTS ifl the remote STUDY was most important, and that audio would serve only as an auxiliary channel of communication. Our qualitative evaluation provided us with the completely opposite result; it was evident that audio was a more expressive and important medium than video, and that a high-quality audio channel was needed far more than video. Although the camera view of STUDY partners did not affect the results of our Fitts’s Law ExPERIMENT, we consider that it is still worthwhile to provide PARTICIPANTs with a clear view of their remote party. Since there are three auto- controlled cameras in LORIT, we believe that letting PARTICIPANTs choose their preferred camera, could affect the remote PARTICIPANTS’ interaction experience. In addition, during the STUDY, a screen-saver repeatedly appeared on the secondary workstation at UBC, introducing a degree of interference. Each time we woke up the ExPERIMENT Station from the screen-saver, video windows were frozen and had to be re-activated and re-sized to ensure a proper view of the remote party. Thus, turning off the screen saver is an essential step in the remote condition. 126 7.3 Contribution This work has contributed to the emerging field of online learning and research in the following ways: • We formally evaluated the relative efficacy of conducting an EXPERIMENT in both co-present and non-co-present conditions. Although PARTICIPANTs remained conscious of the geographic distance, the results are still encouraging. Our experimental outcome shows that a non-interactive ExPERIMENT can still be conducted and achieve expected results, even in the absence of an ExPERIMENTER. However, this requires appropriate cooperation and communication between the human parties and proper support from high-quality software and hardware. • We used an experimental research lab, LORIT, and its web interface, Econtrol, to conduct our STuDY in the non-co-present condition. By making use of LORIT, we evaluated its usability in a genuinely distributed experimental context as LORIT is a technological module designed to support doctoral students to conduct educational research in an e-learning environment. • In the remote condition, we used the lightpath network connection service to support the video-conferencing software, and we evaluated the performance of Lightpath by running multi-media applications (Virtual network computing and Access Grid 3.0) and our own experimental 127 software (developed in Java) in parallel. With Lightpath’s high performance, we were able to conduct our remote EXPERIMENT in real time. Thus, the lightpath network’s dedicated, secure, and confidential service was critical to the success of our STuDY. 7.4 Challenges We encountered some challenges when preparing our remote STUDY. The most challenging factor is that the two cities are in different time zones; Vancouver is three hours behind Montreal. This temporal difference imposed some complexity in scheduling PARTICIPANTS and booking lab facilities at both UBC and UQAM. Moreover, the availability of LORIT also changes from day to day. Due to these obstacles, we could only arrange a maximum of two remote sTuDies per weekday, since the setup for remote studies takes longer than that for local studies. 7.5 Future work The problems in our STUDY leave room for future research, while the results of our STUDY indicate a number of areas that we need to improve. 7.5.1 STUDY design We chose a simple and robust classical EXPERIMENT, the Fitts’s Law STUDY, to verify our research hypothesis. The next version of the STuDY could undertake a more complex and sophisticated research experiment. Also, on this occasion, we 128 only accommodated one SUBJECT in each STuDY session. In reality, some research experiments are designed to accommodate multiple SUBJECTs and observe the behavior among them. Therefore, further research could be performed using that perspective. 7.5.2 Use of Chat, Video, and Audio Although text messaging or chat was completely ignored in our STUDY, there is potential in extending the research into the text-messaging medium, as some research has suggested that chat is a non-obtrusive channel compared to audio, and that it protects people’s privacy under open-context conditions (Scholl, McCarthy, & Harr, 2006). In our case, if the population of each SUBJECT group increases, some other communication issues may emerge, which might require an additional channel, such as chat, to broaden the range of communication among several PARTICIPANTs. 7.5.3 Configuration of Equipment In our STUDY, the UBC venue did not have an equipment configuration symmetrical to that of LORIT. Consequently, the SUBJECTs at LORIT could not adjust the UBC camera in the same way that the UBC ExPERIMENTERs could adjust the LORIT camera. If our STUDY were to be extended, it would beneficial to equip the UBC lab with a fixed and auto-controllable camera, so that UQAM 129 PARTICIPANTS can have freedom of camera adjustment equal to that possible from UBC to LORIT. Our remote STuDY revealed that audio quality was satisfactory in only one direction. The audio quality from LORIT to UBC was barely average. Added to the issue of the remote PARTICIPANTs’ accents, audio posed some challenge to UBC PARTICIPANTs. The next version of the STuDY preparation should pay more attention to tweaking the audio quality for both parties. It would be beneficial to use a microphone directly attached to a PARTICIPANT, or even a microphone station allowing flexible position configuration. 7.5.4 Possible modification of remote lab web interface Our STUDY took advantage of the remote lab, LORIT, and its web interface for remotely controlling its devices and equipment. However, several unexpected problems were also caused by the interface. One problem was a mismatch of people’s mental model of the icon symbolizing the exit from LORIT. One PARTICIPANT accidentally shut down the entire remote lab by clicking this icon. Modifications should certainly be made to this interface component — Another problem was related to French labels on the interface. Although our UBC PARTICIPANTs were able to manage to use the Econtrol interface despite its French labels, it might still have posed a certain amount of difficulty to PARTICIPANTs 130 who could not understand French. One suggestion would be to have an alternative Econtrol interface version with English labels. 131 Bibliography — (2007). Netmeeting for Beginners, Retrieved November 4, 2007, from http://www .ncsu.edu/it/multimedialnetmeeting!netmeeting.html Accot, J. & Zhai, S. (1997). Beyond Fitts’ Law: Models for trajectory-based HCI tasks. In Proceedings ofthe SJGCHJ Conference on Human Factors in Computing Systems. (CHI ‘97), (pp. 295 — 302). New York, NY: ACM Press. Abbot, Costello, (1939), Who is on first?. Retrieved February 22, 2008, from www.youtube.comlwatch?v=sShMA85pv8M Berry, L. (2005). Role-Based Control of Shared Application Views. (Master dissertation, the University of British Columbia, 2005). Retrieve from http ://www.cs.ubc.caJgrads/resources/thesis/NovO5/ Birnholtz, J.P., & Horn, D.B. (2007). Shake, rattle and roles: Lessons from experimental earthquake engineering for incorporating remote users in large-scale e-science experiments., Journal ofComputer-Mediated Communication, 12(2), article 17. Retrieved February 23, 2008, from http://icmc.indiana.edu/voli2/issue2/birnholtz.html Bourdeau, J., Henri, F., Dufresne, A., Tchetagni, J., & Ben Ali, R. (2006). Design of a doctoral training environment: five challenges. Retrieved February 2, 2008, from http://www.lornet.org/Portals/i O/12L0R06/22_Design%20oJ%20a%2ODoctoral %2OTraining-%2OFive%2OCahallenges.pdf Bourdeau, J., Dufresne, A., Prom Tep, S., & Turcotte, S. (2005). Test et validation des environnements de Téléapprentissage, in Pierre, S. (Ed.), Développement, integration et evaluation des technologies de formation et d’apprentissage, chap. 16. Montréal : Les Presses internationales de Polytechnique, 479-499. English version: A Scientific Instrument for the Engineering of Telelearning Environments. Chen, W., Towles, H., Nyland, L., Welch, G., & Fuchs, H. (2000). Toward a compelling sensation of telepresence: Demonstrating a portal to a distant (Static) office. In Proceedings ofthe 11th IEEE Visualization 2000 Conference (VIS 2000) (October 08 - 13, 2000). VISUALIZATION. IEEE Computer Society, Washington, DC 132 Coventry, L. (n.d.) Video conferencing in higher education. Retrieve February 20, 2008, from http ://www.agocg.ac.uk/reports/mmedia/video3/video3.pdf Fitts, M. P. (1954). The information capacity of the human motor system in controlling the amplitude of movement. Journal ofExperimental Psychology (pp. 262-269). (Reprinted from Journal of Experimental Psychology, 47, 381-391, 1954). Retrieved February 20, 2008, from http://www.cs.princeton.edu/courses/archive/fallO7/cos436/FittsJEP 1 954.pdf Foss, B. A., Eikaas, T. I., & Hovd, M. (2000). Merging physical experiments back into the learning arena. Proceedings ofthe American Control Conference (pp. 2944-2948). Chicago: AACC Press. Gu, R., Zhu, M. L., Dong, Y. B., Shi, D.D., & Wang, Y. G., (2005, July). A case study ofvirtual circuit laboratoryfor undergraduate student courses. Paper presented at ITHET 6th Annual International Conference, Juan Dolio, Dominican Republic, (T2B-21-T2B-24). Retrieved February 20, 2007, from IEEE Explore digital library. Hasan, S. M., Lewis, G.J., Alexandrov, V. N., Dove, M.T., & Tucker, M.G. (2007). Multicast application sharing tool for the Access Grid toolkit., Retrieved November 4, 2007, from http://www.allhands.org.uk/2005/proceedings/papers/493.pdf Hauber, J., Regenbrecht, H., Billinghurst, M., & Cockburn, A. (2006). Spatiality in videoconferencing: Trade-offs between efficiency and social presence. Proceedings ofthe 2004 ACM conference on Computer Supported Collaborative Work, (pp. 4 13-422). New York, NY: ACM Press. Heeler, P., & Hardy, C. (2005). A preliminary report on the use of video technology in online courses. Journal ofComputing Sciences in Colleges, 20(4), 127-133. Huck, S. W. (2004). Reading statistics and research. Boston: Pearson Education Inc. Hollan, J., & Stometta, S. (1992). Beyond being there. In Proceedings of the ACM Conference on Human Factors in Computing Systems. (pp. 119-125). New York, NY, USA: ACM Press. Kauff, P. & Schreer, 0. (2002). An immersive 3D video-conferencing system using shared virtual team user environments. In Proceedings of the 4th international conference on Collaborative virtual environments (CVE 2002) (pp. 105-116). New York, NY, USA: ACM Press. 133 Li, Y. Y., Esche, S., & Chassapis, C. (2007). AC 2007-2564: An architecture for real-time remote laboratories. Retrieved February 20, 2008, from http ://www. icee.usm.edu/icee/conferences/asee2007/papers/2564ANARCHITE CTURE_FOR_REAL_TIME_REMOTE_LAB.pdf MacKenzie, I. S. (1991). Fitts’ Law as a Performance Model in Human-Computer Interaction. (Doctoral applied dissertation, the University of York, 1991). Retrieve from http://www.yorku. ca/mack/phd MacKenzie, I. S. (1992) Movement time prediction in human-computer interfaces. Readings in human-computer interaction (211 ed.) (pp. 483-493). Retrieved February 20, 2008, from http ://www.yorku.ca/mack/G192 .html MacKenzie, I. S., & Buxton, W. (1992). Extending Fitts’s law to two-dimensional tasks. Proceedings of the ACM Conference on Human Factors in Computing Systems - CHI ‘92, (pp. 2 19-226). New York, NY, USA: ACM Press. Matsukura, R., Koyamada, K., Tan, Y., Karube, Y., & Morlya, M. (2004). VizGrid: Collaborative visualization grid environment for natural interaction between remote researchers. The Fujitsu Scientific and Technical Journal, 40(2), 205-2 16. Retrieved September 8, 2007, from http://wwwfujitsu.com/downloads/MA G/vol4O-2/paper04.pdf Moløkken-østvold, K. (2005). Ethical concerns when increasing realism in controlled experiments with industrial participants. Retrieved from IEEE Explore digital library Murata, A. & Iwase, H. (2001). Extending Fitts’ law to a three-dimensional pointing task. Retrieved February 4, 2008, from Science Direct website: http://www.sciencedirect. com/science?_ob =Article URL&_udi=B6V8T- 445GW33- 2&_userl022551&_rdoc=1 &Jmt=&_origsearch&_sort=d&viewc&_acct= C000050484& version = 1 & un Version 0& usenid=1 022551 &md5 = 70c35f5J94 bad9d48l4fc2el8fcb83] 0 Nguyen, D., & Canny, J. (2005). MultiView: Spatially Faithful Group Video Conferencing. Proceedings of the 2005 Conference on Human Factors in Computing Systems (CHI’05) 512-521. Portland, Oregon. Patrick, A. (2007, March 9). Privacy practicesfor HCI research. Retrieved Dec 15, 2007, from http ://www.andrewpatrick.ca/essays/privacy-practices-for-hci research! 134 Preece, J., Rogers, Y., & Sharp, H. (2002). Interaction design. New York: John Wiley & Sons, Inc. Ranjan, A., Birnholtz, J.P., & Balakrishnan, R. (2006). An exploratory analysis of partner action and camera control in a video-mediated collaborative task. Proceedings ofthe 2006 20th anniversary conference on Computer Supported Cooperative Work, (pp. 403-422). New York, NY, USA: ACM Press. Rohrig, C., & Jochheim, A. (n.d.). Java-basedframeworkfor remote access to laboratory experiments. Retrieved Dec 7, 2007, from University of Hagen, Department of Electrical Engineering website: http://prtfernuni hagen. de/virtlab/ace2k/ace2k.pdf Schedlbauer, M., Pastel, R., Heines, J. (1995). An extensible and interactive research platform for exploring Fitts’ Law. Retrieved February 3, 2008, from http://teaching. cs. uml. edu/’-heines/techrpts/Papers/Schedlbauer-TR-ECIS MTE.pdf Schmid, C., Eikaas, T.I., Foss, B.A., & Gillet D. (2001). A remote laboratory experimentation network. Retrieved February 20, 2008 from Scientific Commons database Scholl, J., McCarthy, J., & Harr, R. (2006). A comparison of chat and audio in media rich environments. Proceedings of the Computer Supported Collaborative Work (CSCW 2006), (pp. 323-332). New York, NY, USA: ACM Press. Schreer, 0., Brandenburg, N., Askar, S., & Kauff P. (2001). Hybrid recursive matching and segmentation-basedpostprocessing in real-time immersive video conferencing. Paper presented at the conference on VMV ‘01, Stuttgart, Germany. Retrieved November 20, 2007, from CiteSeerX digital library. Sun, J. & Razdan, A. (n.d.). Remote control & visualization of scanning probe microscopes via web. Retrieved February 20, 2008, from IEEE Explore digital library Svedkauskaite, A., Reza-Hernandez, L., & Mary Clifford (2003, June 24). Critical Issue: Using technology to support Limited- English-Proficient (LEP) students’ learning experiences. Retrieved February 23, 2008, from Learning Point Associates, North Central Regional Educational Laboratory website: http://www. ncrel. org/sdrs/areas/issues/methods/technlgy/te900. htm Taylor, R.M., Robinett, W., Chi, V.L., Brooks, F.P., Wright, W.V., Williams R.S., & Snyder, E.J. (1993). The nanomanipulator: A virtual-reality interface to a scanning tunneling microscope. Proceedings ofthe 20th annual conference on 135 Computer graphics and interactive techniques, (pp. 127 -134). New York, NY, USA: ACM Press. Treloar, A. (n.d.). DART: Building the new collaborative e-research infrastructure. Retrieved March 13, 2008, from The Council of Australian University Directors of Information Technology website: http://www.caudit.edu. au/educauseaustralasiaO7/authors_papers/Treloar-1 83.pdf Vertegaal, R., Weevers, I., Sohn, C., & Cheung, C. (2003). GAZE-2: Conveying eye contact in group video conferencing using eye-controlled camera direction. Proceedings ofthe SIGCHI conference on Humanfactors in computing systems (CH12003), (pp. 521-528). New York, NY, USA: ACM Press. Wei Chao, C., Towles, H., Nyland, L., Welch, G., & Fuchs. H. (2000). Toward a compelling sensation of telepresence: demonstrating a portal to a distant (static) office. Proceedings ofthe conference on Visualization ‘00 (IEEE visualization 2000), (pp. 327- 333). Los Alamitos, CA, USA: IEEE Computer Society Press. Wei-Chao, C., Towles, H., Nyland, L., Welch, G. & Fuchs, H. (2000). Toward a Compelling Sensation of Telepresence: Demonstrating a portal to a distant (static) office. IEEE Visualization, Proceedings ofthe conference on Visualization ‘00, Salt Lake City, Utah, United States, Pages: 327 — 333. http://delivery.acm.org/10.1i45/3800OO/375263/p327- chen.pdJ?key] =375263&key2=2138735021&collGUIDE&dlGUIDE&CFID= 59095509& CFTOKEN=342991 21 Zaimovice-Uzunovice, N., Leme, S., & Petkovice, D. (2001, August). Virtual instruments — a chance to teach engineering at a distance. Paper presented at International Conference on Engineering Education, Oslo, Norway, (7D1-24- 7D1-26). Retrieve February 20, 2008, from http://www.ineer.org/Events/ICEE200 1/Proceedings/authors!. ./papers/43 7.pdf Zhang , J. Z., Ball, A. K., Extine, M. C., & Extine, W. (2007). Design ofa real time remote-access engineering laboratory using integrated web service and wireless technologyfor distance learners. Retrieved February 23, 2008, from http://www.eng.monash.edu.au/uicee/worldtransactions/WordTransA bstracts Vo14 No2/1 5-Zhang23.pdf Zhao, H. (2002, October) Fitts’ Law: Modeling movement time in HCI. Retrieved November 12, 2007, from the University of Maryland, Department of Computer Science web site: http://www.cs.umd.edu/class/fall2002/cmsc838s/tichi/fitts. html 136 Appendices Appendix A Questionnaire The following questionnaire was administered to all PARTICIPANTS in our STuDY. The results were analyzed and are reported in Chapter 5 of this thesis. The STUDY was conducted under the ethics certificate 1107-00128 issued by the Behavioral Ethics Research Board of the University of British Columbia. 137 Remote Experimenter Questionnaire: Personal information: 1. What age group do you fall into: 15—19 I 20—24 125—29 I 30—34 I 35—39 I 40 and higher 2. Are you familiar with the idea of Fitts Law? 3. Have you done any point-and-click experiment before? Interaction: 1. As an experimenter, my overall feeling about interacting with the subject was pleasant and comfortable. Rank from a scale of 1 to 5 (1 strongly agree, 5 strongly disagree) 2. Did I feel that I have delivered my instruction in a way that was clear and comprehensible to a subject? Select from a scale of 1 to 5: 138 a. My subject understood all instructions clearly when they were stated only once. b. My subject understood some instructions clearly if they were stated once. c. I had to repeat the instructions multiple times before my subjects could understand them d. My instructions were extremely poorly delivered and not understood by subjects at all so I had to repeat multiple times e. others: please specify_____________________________________ 3. It was easy to talk to and hear the subject. Rank from a scale of 1 to 5 (1 strongly agree, 5 strongly disagree) 4. When delivering instructions and explanations, I frequently used hand gesture to interact with a subject. Rank from a scale of 1 to 5 (1 strongly agree, 5 strongly disagree) 5. I looked at my subject frequently to make sure he / she was making correct progress on the experiment. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree) 139 6. I was aware of my subject’s presence all the time. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree) 7. Rank from a scale of 1 to 5 (1 means agree, 5 means disagree) a. I confused my subject very often b. My subject confused me very often 8. Overall, from my own point of view, how well did the experiment go? Please rank based on a scale of 1 to 5 (1 means very well and smoothly, 5 means very difficult and badly). 9. How much sense of control did I feel over the experiment? Please rank on a scale of 1 to 5 (1 means totally in control over the experiment, 5 means not at all in control over the experiment) 10. I DID NOT need much interaction with my subject when doing the tasks since all of them were non-interactive. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree) 11. I was making eye contact with my subject all the time. 140 Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree) Training: 12. After being trained by my assistant, how confident did I feel about playing the role of an experimenter and running the study with a subject? Rank from a scale of 1 to 5 (1 means most confident, 5 not confident at all) Video-conferencing: 13. The frequent interruption from a research lab assistant was pleasant and helpful. Rank from a scale of 1 to 5 (1 means strongly agree, and 5 means strongly disagree) 14. Using VNC (virtual network connection), I could control my subject’s screen and interfere with the subject’s interaction, how did I feel about this kind of control? Rank from on a scale of 1 to 5: (1 means most pleasant and not disturbed or annoyed at all, 5 means very unpleasant and outrageously disturbed and annoyed). 141 15. At the time I got access to my subject’s PC via VNC, I immediately knew where to look for on subject’s screen. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree) 16. If my subject made a mistake, I used oral instructions and VNC to intervene the subject. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree) 17. I felt very comfortable seeing my own face on screen. Rank from a scale of 1 to 5 (1 means strongly agree, and 5 means strongly disagree) 18. What kind of screen presentation of videos made I feel that I am in control of an experiment (gave me the sense of observing my subject)? El Seeing both subject’s and my own face on the pc screen El Seeing just the subject’s face on the screen Set-up: 19. Were the camera icons intuitive enough to be recognized? Please rank from a scale of 1 to 5 (1 means “recognized it right away”, 5 means “did 142 not know it represented a camera”). 20. By looking at the layout of the remote lab, could I somehow get the idea where my subject was located in the remote lab? Please rank from a scale of 1 to 5 (1 means “I knew where my subject was in the lab”, 5 means “I had no idea where my subject was in the lab”). 21. How did I feel about the response time of camera adjustment? Please rank from 1 to 5 (1 means very quickly and instantly, 5 means very slowly and long delayed). 22. Overall, how was my experience about using the econtrol interface to adjust the cameras in remote? Please rank from a scale of 1 to 5 (1 means very easy to use, 5 means very difficult to use). 23. Please select all the media that were used to help with the camera set-up: Video, Audio, VNC, Chat Other Comments: 143 Remote Subject Questionnaire: Personal information: 1. What age group do you fall into: [1 15—19 [1 20 —24 [125—29 [130—34 [135—39 [140 and higher 2. Are you familiar with the idea of Fitts Law? 3. Have you done any point-and-click experiment before? Interaction: 1. As a subject, my overall feeling about interacting with the experimenter was pleasant and comfortable. Rank from a scale of 1 to 5 (1 strongly agree, 5 strongly disagree) 2. Did I feel that I have received my experimenter’s instruction in a way that was clear and comprehensible? Select from a scale of 1 to 5: 144 b. I understood all instructions clearly when they were stated only once. c. I understood some instructions clearly if they were stated once. d. I had to ask my experimenter to repeat the instructions multiple times before I could understand them. e. My experimenter delivered the instructions extremely poorly and could not be understood at all so I had to ask my experimenter to repeat multiple times f. others: please specif’______________________________________ 3. It was easy to talk to and hear the experimenter. Rank from a scale of 1 to 5 (1 strongly agree, 5 strongly disagree) 4. When listening to instructions and explanations, I frequently used hand gesture to interact with the experimenter. Rank from a scale of 1 to 5 (1 strongly agree, 5 strongly disagree) 5. I looked at my experimenter frequently to ensure that I was making correct progress on the experiment. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree) 145 6. I was aware of my experimenter’s presence all the time. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree) 7. Rank from a scale of 1 to 5 (1 means agree, 5 means disagree) a. I confused my experimenter very often b. My experimenter confused me very often 8. Overall, from my own point of view, how well did the experiment go? Please rank based on a scale of 1 to 5 (1 means very well and smoothly, 5 means very difficult and badly). 9. How much sense of BEING controlled did I feel over the experiment? Please rank on a scale of 1 to 5 (1 means totally being controlled over the experiment, 5 means not at all being controlled over the experiment) 10. 1 DID NOT need much interaction with my experimenter when doing the tasks since all of them were non-interactive. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree) 11. I was making eye contact with my experimenter all the time. 146 Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree) Training: 12. After being trained by a research lab assistant, how much confidence did I sense about the other participant playing the role of an experimenter and running the study with me? Rank from a scale of 1 to 5 (1 means most confident, 5 not confident at all) Video-conferencing: 13. The frequent interruption from a research lab assistant was pleasant and helpful. Rank from a scale of 1 to 5 (1 means strongly agree, and 5 means strongly disagree) 14. Using VNC (virtual network connection), my experimenter could control my screen and interfere with my interaction, how did I feel about this kind of control? Rank from on a scale of 1 to 5: (1 means most pleasant and not disturbed or annoyed at all, 5 means very unpleasant and outrageously disturbed and annoyed). 147 15. At the time my experimenter got access to my PC via VNC, he/she immediately knew where to look for on my screen. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree) 16. If I made a mistake, my experimenter used oral instructions and VNC to intervene me. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree) 17. I felt very comfortable seeing my own face on screen. Rank from a scale of 1 to 5 (1 means strongly agree, and 5 means strongly disagree) 18. What kind of screen presentation of videos made me feel that the study was controlled by my experiment (gave me the sense that my experimenter was observing me)? L Seeing both the experimenter’s face and my own face on the pc screen LI Seeing just the experimenter’s face on the screen 148 Other Comments: Local Experimenter Questionnaire: Personal information: 1. What age group do you fall into: 15 — 19 D 20 —24 25—29 30—34 D 35—39 D 40 and higher 2. Are you familiar with the idea of Fitts Law? 3. Have you done any point-and-click experiment before? Interaction: 1. As an experimenter, my overall feeling about interacting with the subject was pleasant and comfortable. 149 Rank from a scale of 1 to 5 (1 strongly agree, 5 strongly disagree) 2. Did I feel that I have delivered my instruction in a way that was clear and comprehensible to a subject? Select from a scale of 1 to 5: a. My subject understood all instructions clearly when they were stated only once. b. My subject understood some instructions clearly if they were stated once. c. I had to repeat the instructions multiple times before my subjects could understand them d. My instructions were extremely poorly delivered and not understood by subjects at all so I had to repeat multiple times e. others: please specify___________________________________ 3. It was easy to talk to and hear the subject. Rank from a scale of 1 to 5 (1 strongly agree, 5 strongly disagree) 4. When delivering instructions and explanations, I frequently used hand gesture to interact with a subject. Rank from a scale of 1 to 5 (1 strongly agree, 5 strongly disagree) 150 5. I looked at my subject frequently to make sure he / she was making correct progress on the experiment. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree) 6. I was aware of my subject’s presence all the time. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree) 7. Rank from a scale of 1 to 5 (1 means agree, 5 means disagree) a. I confused my subject very often b. My subject confused me very often 8. Overall, from my own point of view, how well did the experiment go? Please rank based on a scale of 1 to 5 (1 means very well and smoothly, 5 means very difficult and badly). 9. How much sense of control did I feel over the experiment? Please rank on a scale of 1 to 5 (1 means totally in control over the experiment, 5 means not at all in control over the experiment) 151 10. I DID NOT need much interaction with my subject when doing the tasks since all of them were non-interactive. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree) 11. 1 was making eye contact with my subject all the time. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree) Training: 12. After being trained by my assistant, how confident did I feel about playing the role of an experimenter and running the study with a subject? Rank from a scale of 1 to 5 (1 means most confident, 5 not confident at all) Other comments: 152 Remote Subject Questionnaire: Personal information: 1. What age group do you fall into: LI 15—19 LI 20 —24 LI 25—29 LI 30—34 LI 35—39 LI 40 and higher 2. Are you familiar with the idea of Fitts Law? 3. Have you done any point-and-click experiment before? Interaction: 1. As a subject, my overall feeling about interacting with the experimenter was pleasant and comfortable. Rank from a scale of 1 to 5 (1 strongly agree, 5 strongly disagree) 2. Did I feel that I have received my experimenter’s instruction in a way that was clear and comprehensible? Select from a scale of 1 to 5: 153 a. I understood all instructions clearly when they were stated only once. b. I understood some instructions clearly if they were stated once. c. I had to ask my experimenter to repeat the instructions multiple times before I could understand them. d. My experimenter delivered the instructions extremely poorly and could not be understood at all so I had to ask my experimenter to repeat multiple times e. others: please specify______________________________________ 3. It was easy to talk to and hear the experimenter. Rank from a scale of 1 to 5 (1 strongly agree, 5 strongly disagree) 4. When listening to instructions and explanations, I frequently used hand gesture to interact with the experimenter. Rank from a scale of 1 to 5 (1 strongly agree, 5 strongly disagree) 5. I looked at my experimenter frequently to ensure that I was making correct progress on the experiment. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree) 154 6. I was aware of my experimenter’s presence all the time. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree) 7. Rank from a scale of 1 to 5 (1 means agree, 5 means disagree) a. I confused my experimenter very often b. My experimenter confused me very often 8. Overall, from my own point of view, how well did the experiment go? Please rank based on a scale of 1 to 5 (1 means very well and smoothly, 5 means very difficult and badly). 9. How much sense of BEING controlled did I feel over the experiment? Please rank on a scale of 1 to 5 (1 means totally being controlled over the experiment, 5 means not at all being controlled over the experiment) 10. I DID NOT need much interaction with my experimenter when doing the tasks since all of them were non-interactive. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree) 11. I was making eye contact with my experimenter all the time. 155 Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree) Training: 12. After being trained by a research lab assistant, how much confidence did I sense about the other participant playing the role of an experimenter and running the study with me? Rank from a scale of 1 to 5 (1 means most confident, 5 not confident at all) Video-conferencing: 13. The frequent interruption from a research lab assistant was pleasant and helpful. Rank from a scale of 1 to 5 (1 means strongly agree, and 5 means strongly disagree) 14. Using VNC (virtual network connection), my experimenter could control my screen and interfere with my interaction, how did I feel about this kind of control? Rank from on a scale of 1 to 5: (1 means most pleasant and not disturbed or annoyed at all, 5 means very unpleasant and outrageously disturbed and annoyed). 156 15. At the time my experimenter got access to my PC via VNC, he/she immediately knew where to look for on my screen. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree) 16. If I made a mistake, my experimenter used oral instructions and VNC to intervene me. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree) 17. I felt very comfortable seeing my own face on screen. Rank from a scale of 1 to 5 (1 means strongly agree, and 5 means strongly disagree) 18. What kind of screen presentation of videos made me feel that the study was controlled by my experiment (gave me the sense that my experimenter was observing me)? L Seeing both the experimenter’s face and my own face on the pc screen Li Seeing just the experimenter’s face on the screen Other Comments: 157 Appendix B Interview The following interview questions were administered to EXPERIMENTERS lfl our STUDY. The results were analyzed and are reported in Chapter 5 of this thesis. The STUDY was conducted under the ethics certificate H07-00128 issued by the Behavioral Ethics Research Board of the University of British Columbia. 158 Interview Script: 1. How was your experience of being an experimenter? 2. What kind of problems did you experience when interacting with your subject? For experimenters under the remote condition only: 3. Between audio and video, what media served as a more important role in communication during the experiment? 4. What was the most challenging part of setup? 159 Appendix C Ethics Certificates The following are the ethics certificates issued by the Behavioral Ethics Research Board of the University of British Columbia and the Comité d’éthique de la recherche de la Faculté des arts et des sciences of L’Université du Québec a Montréal. 160 CERTIFICATE OF APPROVAL- MINIMAL RISK RENEWAL PRINCIPAL INVESTIGATOR: DEPARTMENT: USC BREB NUMBER: Kellogg S. Booth UBCIScience/ComputerScience :HO7OO1 28 INSTITUTION(S) WHERE RESEARCH WILL BE CARRIED OUT: Institu ion Site UBC Vancouver (excludes UBC Hospital) Other locations where the research will be conducted: A laboratory at the Unwersity of Montreal under the direction of Dr Aude Dutresne LORIT Laboratopre Observatoire de Recherohe air le Télé-Apprentissage en collaboration avoc Is LICEF/TELUO-UQAM. CO-INVESTIGATOR(S): Ying Zhang SPONSORING AGENCiES: Natural Sciences and Engineering Research Council of Canada (NSERC) ‘Network for effective collaboration technologies through advanced research (NECTAR) - researcfi Tele-universite- ‘Network Agreement: Canarie Advanced Applications Program: BEST Projecr PROJECT TITLE: Conducting experiments with human subjects in local and remote laboratories EXPIRY DATE OF THISAPPROVAL: March31, 2009 APPROVAL DATE: March 31, 2008 The Annual Renewal for Study have been reviewed and the procedures were found to be acceptable on ethical grounds for research involving human subjects. Approval is Issued on behalf of the Behavioural Research Ethics Board Dr. M. Judith Lynani, Chair Dr. Ken Craig, Chair Dr. Jim Rupert. Associate Chair Dr Laurie Ford Associate Chair Dr Daniel Salhani Associate Chair Dr Anita Ho Associate Chair 161 Université de Montréal Comité d’ethique de Ia recherche de Ia Facuttê des arts et des sciences (CERFAS) DIRECTIVES ET QUESTiONNAIRE EVALUATION ETHIQUE DES PROJETS DE RECHERCHE DIRECTIVES 1) Veuillez rerriptir le questionnaire cl-joint. Vous pouvez Ic dactylographier ou le remplir directementà lécran sur Ic site Web http:/iwwwfasumentrealcaifasinfoiforrnulaire_ethiguehtm puis limprirner; n’oubliez pas de Ic signer. Vous devez en foumir 4 exemplaires (Poriginal et trois copies). 2) Veuiflez joindre aux 4 exemplaires du questionnaire 4 copies de Ta description du projet. S’iI sagit d’une demande de subvention: partie de Ta dernande de subvention décrivant Ic projet et Ta methodologie; ne pas inclure a bibliographie, les sections budgétaires, Ic curriculum vitae. Vous pouvez ajouter tout autre renseignement susceptible de faciliter le travail des membres du Comité dethique. 3) Veuillez egalement joindre 4 copies du(des) formulaire(s) de consentement (ou du synopsis des informations données aux participants afin d’obtenir leur consentement verbal). Le formulaire de consernement (ou Ic synopsis le cas échéant) est trôs important et dolt être fait scion les riormes. Avant de Ic produire, veuiUez consulter le Document d’in formation sur les formulaires de consentement disponible sur Ic site Web: htto:Jfwww.fasumontreal ca’fasinfo/formuiaire_ethiue.htrn Vous peuvez ajouter tout autre renseignement susceptible de facitter Ic travail des membres du Comité dethique (questionnaires, schema d’entrevue etc.). 4) Veuillez retourner Ic questionnaire et les documents en quatre exemptaires a: Si vous êtes tudiant-e: TechnicIenne en gestion des dossiers étudiants au secretariat de votre departernent ou école Si vous êtes professeur-e ou chercheur-e: Vice-décanat a Ia recherche FacultC des arts et des sciences Pavilion Lione!-Groulx - Bureau C-9109 UniversitC de Montréal 162 Unlversftë de Montréal QUESTIONNAIRE EVALUATION ETHIQUE DES PROJETS DE RECHERCHE Comitê d’éthique de Ia recherche de Ia Faculté des arts et des sciences (CERFAS) Dact1ogFapher ou remplir clirectement stw Ic site http/www.fas.umontreaLca/fasinfo/formulaire ethiguehtm NOM : DUFRESNE TITRE Professeure titulaire PRENOM: Aude DEPARTEMENT: Communication SECTiON A 1. Les participants(ou une partie d’entre eux) sont-ils ages de moms de 18 ans? 2. Les participants (ou une partie d’entre eux) sont-ils inaptes (ineapacitC a donner un consentement Cclairé par exemple en raison de curs capacitCs mentales)? Si OUt a a Question 1 su 2 Etuchant-e-s: DIPLÔME POSTULE: DIRECTEUR-TRICE DE RECHERCHE: ADRESSE COURRIEL DE L’ETUDIANT-E ETNUMERODETELEPHONE SOURCE DE FINANCEMENT (Ic cas échéant): (apparaitra sur le certificat déthique) COCHERCHEUR(S) (le cas óchóant): PETIT Gregory, REISS Stéphane, ZHANG Ying TITRE EXACT DU PROJET (sera inscrit tel quel sur Je certificat déthique): Human-Computer Interaction Research Project (CPSC 549A) : Remote Interaction DATE DU DEBUT DE LA COLLECTE DES DONNEES 30 avrtl 2007 Le consentement dun parent ou d’un tuteur sera-t-il obtenu par écrit? 3.. Les participants (ou une partie dentre eiix) proviennent-ils dune population <<captive ou dCpendante (ex personnes en miteu carcéral ou de protection)? OUI NON C 163 OUI NON 4. Le protocole de recherche prévoit-il que des groupes de participants doivent ètre 0 exclus de Ia recherche en raison de risques pour leur sante (par exemple en raison de grossesse, de troubles cardio-vasculaires ou respiratoires, etc.) 7 5. Les participants (ou une partie d’entre eux) sont-ils soumis urie procedure Q d’ordre medical dans le cadre de Ia recherche (par exemple prise de sang, utilisation de médicaments. etc.)? 6. Les participants peuverit-ils faire lobjet dun signalernent obligatoire en vertu U de Ia Loi sur Jo protection de Ia jeunesse (violence envers les enfants>? 7. Au-deI dun signalement obligatoire, les inforrnations de nature nominative ou confidentielle données par les participants aux chercheurs peuvent-elles ètre transmises a d’autres personnes ou organismes? 8. Le protocole de recherche nécessite-t-iI de ne divulguer aux participants que des renseignements partiels sur létude ou de les induire temporairement en erreur? 9. La recherche se déroulera-t-elle dons un autre pays que le Canada? U Si oui, le(s)quel(s): ________________________________________________ 10. La participation a a recherche peut-elle entrainer des risques sérieux pour Ia U sante mentale ou physique des participants (ou pour une partie d’entre eux)? 11. La participation a a recherche peut-elle entrainer involontairement des prejudices aux participants dans le cadre de leur milieu familial (relations conjugales paren tales, etc.) ou tie leur milieu de travail (par exemp[e face a lemployeur, au syndicat ou aux collCgues)? SECTION B Vous pouvez répondre sur ces pages ou utiliser des pages distinctes pour répondre a cette section, en reproduisant Ic numéro et 1€ titre de La question; ne pas oublier de signer le document. 1. Résumé du projet (DCcrire en 10 lignes es objectifs du projet) The purpose of this experiment is to test the hypothesis that with correct equipment settings, experiments conducted remotely should not aflect performance in a simple 1-itt’s law (1 here is a relationsnip between me wiatn of a target and me time requirea to acquire the targer The participant will be asked to use a visioconterence to perform tasks. He/she will be selected to play the roie as either a subject or an experiment in one session. As a subject, he/she wili be asked to perform a point- and-click experiment; as an experimenter, he/she will supervise me subject and interview him. I he Interaction will be recoraed and analyzed (quantitative and qualitative). I he participant will be asked to participate in only one session, each lasting no more than 1 hour. Recordings will only be used for analysis and with the subject consent it may also be used br class project and other research presentations at the University of British Columbia or University of Montreal. 2. Quelles soot les caractéristiques des participants qui prendront part a Ia recherche (ige, sexe, milieu, etc.)? Préciser le nombre de participants. Préciser Cgalement s’il y a des critéres d’exclu sion (caractéristiques empécharit des personnes tie participer a la recherche) Les participants seront des étudiants de Montréal et des étudiants de Vancouver. 164 3. De queue facon es participants seront-ils recrutés? Une annonce sera passée au sein des ditférentes universités de Montréal pour recruter des participants parlant correctement anglais. 4. A quelleCs) actMtés(s) participeront les sujets (entrevues, questionnaires, tests, etc.)? Les participants interviendront dans lexpérimentation, soit en tant que sujet, soit en tant qu’expérimentateur, et devront repondre a un questionnaire a La fin de ‘experimentation. Les participants seront flumes pendant rexpérimentation mais pourront refuser l’enreistrement s’ils Ic souhaitent. 5. Les participants encourent-ils des risques ou inconvénients? Queues mesures seront prises pour contrer ces risques ou inconvénients? Aucun risque ou inconvenient ne sera encouru. 6. Queue forme de consentement le chercheur demande-t-il aux participants? Avant de remplir cette section, veuillez consulter Ic document Informations relatives aux formulaires de consentement Consentement ècrit - Joindre 4 copies du(des) formulaire(s) de consentement Consentement verbal — Joindre 4 copies d’un synopsis des informations données verbalement aux participants aim d’obtenir teur consentement verbal. Le recoLirs au seul consentement verbal constitue une exception. Veuillez justifier ce choix ci-dessous. Signalure: _______________________________________ Date: 17 avril 2007 165 Appendix D D.1: Scatter plots for remote conditions 120000 /e30:2322/ .00 2.00 300 4 IOU 2.00 3.00 4.00 Subject I Subject 2 0 1200.00 — ,J3;6166?1?3?9I 6O%7/ IOU 2.00 300 4.00 1.00 200 3.00 4.00 Su1ject 3 Subject 5. 166 1660.004 1100.00 Time_C = 519.99 .233.97 * I U0O/ i hI 100 200 300 400 .00 2.00 300 400 Subject 6 Subjeàt7 1600.00 Time_9 = 525.96 + 281.10 * / R-Square = 0.07 ,,J91s+22osaI / 100 200 300 400 160 200 300 400 SubjectS Subject 9 1300.00 / / Time_12 60969 + 174.49 IDTime Ii 577.09 + 233.91 * P-Square = 0.99 ::: / 1.00 2.00 3.00 4.00 100 2.00 3.00 4.00 Subject 12 Subject 11 167 D.2: Scatter plots for local conditions 1600.00 1300.00 re7f,126Y/ :: 1000.00 I I 1.00 2.00 3M 4.00 ISO 2.00 3.00 4.00 Subject 2Subject I 1320.OO.J 9 9I Time 4 = 412.81 + 281.64 * Iii 1000.00’ P-Square = 1.88 I300.00 .J Time 3 1211.13 +23.38* IIY’ __ / .00 2.00 3.00 4.00 1.00 2.00 3.00 4.00 Subject 3 Subject 4 //t816320h1 1.0 2.00 3.00 4.00 1.0 2.00 350 4.00 Subject 6Subject 5 168 140000 0 __________________ i.::: __________________ 1 00 200 3.00 4.D0 IOU 2.00 3.00 4.00 Subject7 Subject S 100000 ______ ___ E:0H/2bb956nu1 2.00 3.00 4.00 1.00 2.00 3.00 4.00 Subject 9 Subject 10 169 Appendix E E.1: Mean ranks and standard deviations of each question in the questionnaires on a 5-point, agreeable scale ranging from 1(strongly agree) to 5(strongly disagree). Question Local Local subject Remote Remote Number experimenter experimenter subject Interaction 1 Mean = 1.400, Mean = 2.300, Mean 2.400, Mean = 2.100, SD = .699 SD 1.418 SD = 1.075 SD = 1.433 2 Mean =2.050, Mean= 1.30, Mean 1.600, Mean= 1.600, SD1.212 SD.675 SD=.516 sd=.844 3 Mean = 1.600, Mean = 1.900, Mean 4.600, Mean = 1.900, SD = .843 SD 1.449 SD = .699 SD = .316 4 Mean = 3.400, Mean = 4.000, Mean 4.600, Mean = 4.900, SD = 1.647 SD 1.414 SD = .699 SD = 3.162 5 Mean = 2.300, Mean = 3.200, Mean = 2.500, Mean = 3.900, SD = 1.160 SD = 1.476 SD = 1.501 SD = 1.287 6 Mean = 1.400, Mean = 2.600, Mean = 2.100, Mean = 2.100, SD = .966 SD 1.647 SD = 1.370 SD = 1.449 7a Mean = 3.900, Mean = 4.100, Mean 4.000, Mean = 4.500, SD = 1.449 SD 1.287 SD = .943 SD = .707 7b Mean = 4.300, Mean = 4.500, Mean = 4.600, Mean = 4.100, SD = 1.059 SD .707 SD = .699 SD = 1.101 8 Mean = 1.500, Mean = 1.850, Mean 1.900, Mean = 1.800, SD=.850 SD=.818 SD=.568 SD=.918 9 Mean = 1.950, Mean = 2.700, Mean = 2.200, Mean = 2.400, SD = .599 SD 1.494 SD = .632 SD = .966 10 Mean = 2.600, Mean = 2.300, Mean 2.900, Mean = 1.800, SD = 1.350 SD = 1.059 SD = 1.197 SD = .919 1 1 Mean = 3.300, Mean = 3.850, Mean 3.900, Mean = 3.800, SD = 1.337 SD .747 SD = 1.287 SD = 1.229 170 Question Local Local subject Remote Remote Number experimenter experimenter subject Training 12 Mean=2.000, Mean= 1.800, Mean=2.500, Mean2.700, SD = .943 SD = .919 SD = 1.080 SD = 1.337 Video conferencing 13 N/A N/A Mean= 1.800, Mean= 1.900, SD=1.033 SD=1.101 14 N/A N/A Mean= 1.900, Mean=2.300, SD = .738 SD = 1.337 15 N/A N/A Mean = 2.400, Mean = 2.600, SD = 1.265 SD = 1.173 16 N/A N/A Mean = 2.300, Mean = 3.00, SD=1.418 SD=1.944 17 N/A N/A Mean = 1.500, Mean = 2.200, SD = 1.080 SD = .790 18 N/A N/A Mean = 1.300, Mean = 1.6, SD = .480 SD = .520 Set up 1 N/A N/A Mean=1.820, N/A SD= 1.080 2 N/A N/A Mean=2.360, N/A SD= 1.120 3 N/A N/A Mean =2.180, N/A SD=.870 4 N/A N/A Mean = 1.720, N/A SD = .900 5 N/A N/A N/A N/A 171 E.2 Pair-wise experimenter vs. subject (condition ignored) Question Experimenter Subject difference Independent number samples T-test results Interaction 1 Mean = 1.400, Mean = Mean = T (df= 19) = -3.018, SD = .681 2.300, 1.100, SD p = .007, effect size SD= =1.165 =.324 1.380 2 Mean=1.825, Mean= Mean= T(df19)=1.325,p SD =.935 1.450, .875, SD = .2 10, effect size = SD =.971 .084 =.759 3 Mean=3.100, Mean= Mean= T(df= l9)-.922,p SD =1.7 13 3.400, .900 , SD = .368, effect size = SD= =1.165 .043 1.846 4 Mean = 4.000, Mean = Mean = T(df 19) = -1.443, SD =1.376 4.450, .950, SD p = 1.652, effect size SD = = 1.099 = .098 1.099 5 Mean = 2.400, Mean = Mean = T(df= 19) = -2.489, SD = 1.3 13 3.550, 2.050, SD p = .022, effect size SD =1.099 = .246 =1.394 6 Mean=1.750, Mean= Mean= T(df=19)=-1.318, SD = 1.208 2.350, 1.500, SD p = .203, effect size SD =1.468 = .084 =1.531 7a Mean = 3.950, Mean = Mean = T(df= 19) = -1.046, SD = 1.190 4.300, 1.050, SD p = .309, effect size SD =1.099 =.054 =1.031 7b Mean = 4.450, Mean = Mean = T(df= 19) = .497, p SD = .887 4.300, .950, SD = .625, effect size = SD =.944 .012 =.923 8 Mean= 1.700, Mean= Mean= T(df= l9)-.488,p SD = .732 1.925, .825, SD = .63 1, effect size = SD =.654 .012 =.847 172 Question Experimenter Subject difference Independent number samples T-test results 9 Mean =2.075, Mean= Mean= T(df= 19)=-1.758, SD = .612 2.550, .875, SD p = .095, effect size SD =.944 = .139 = 1.234 10 Mean=2.750, Mean= Mean= T(df= l9)2.OO8,p SD = 1.251 1.050, 1.300, SD = .059, effect size = SD =1.080 .175 =.998 11 Mean=3.600, Mean= Mean= T(df=19)=-.659,p SD = 1.313 3.825, 1.0750, = .518, effect size = SD SD =1.079 .022 =.990 12 Mean = 2.250, Mean = Mean = T(df= 19) = .00 1, p SD = 1.019 2.250, 1.100 , SD = .900, effect size SD =.967 =.001 = 1.208 173 E.3 :Independent t-test results: Experimenter vs. experimenter; subject vs. subject Question Remote experimenter vs. Remote subject vs. local subject Number local experimenter 1(significant) Z(l8) = -2.471, p .013, i12 = T(18) = -.209, p = .737, r2 .002, .250, large effect small effect 2 Z(18)= -i362,p= .508, r12 = Z(18)=-.936,p= .349, r2= .046, .024, small effect moderate effect 3( significant) Z(18) = -2.006, p = .045, i2 Z(18) = -.612, p = .541, M2 = .020, = .180, large effect small effect 4 Z(18) = -1.745, p = .081, ri2 = Z(18) = -1.693, p = .090, ri2 = .140, .140, large effect large effect 5 T(18) = .332, p = .743, i2 = Z(18) = -1.107, p = .268, ri2 = .060, .006, small effect small effect 6 Z(18) = -1.378, p = .168, r2 = Z(18) = -.677, p = .498, ri2 .020, .006, small effect small effect 7a Z(18)=-.32l,p=.748,r2= Z(18)-.589,p=.556,2=.019, .006, small effect small effect 7b Z(18)=-.580,p=.562,2= Z(18)=-.753,p=.452,2=.031, .0 18, small effect small effect 8 Z(18) = -1.523, p = .128, ii2 Z(18) = -.161, p = .872, M2 = .001, = .130, moderate effect small effect 9 Z(18)= -.8l6,p = .414, ri2 = Z(18)= .196, p = .845,i2 = .002, .042, small effect small effect 10 Z(18) = -.798, p = .425, r2 = Z(18) = -1.071, p = .284, ri2 = .056, .031, small effect small effect 11 Z(18)-l.331,p.183,ri2= T(l8)=-.l10,p=.914,r2=.001, .002, small effect small effect 12 Z(l8)-l.109,p=.268,ri2= Z(18)-1.461,p=.l44,Ti2=.llO, .064, moderate effect moderate effect 174 E.3.1: Paired-sample t-test for remote experimenter vs. subject, p = .025 Question Experimenter Subject Difference Paired-sample T-test Number 1 Mean = 2.400, Mean = 2.100, Mean = 1.300, Z(9) = -.3 12, p .755, SD = 1.075 SD 1.433 SD = 1.060 i2 =.010 small effect 2 Mean = 1.600, Mean = 1.600, Mean = .600, Z(9) = .001, p = .90, SD = .516 SD = .844 SD .516 ii2 < .001, small effect 3 Mean = 4.600, Mean = 4.900, Mean = .500, Z(9) = -1.134, p = SD=.699 SD=.316 SD=.707 .257,r2=.l20, moderate 4 Mean = 4.600, Mean = 4.900, Mean = .500, Z(9) = -1.134, p = SD = .699 SD = 3.162 SD = .707 .257, i2 = .120 moderate effect 5 Mean=2.500, Mean=3.900, Mean=2.000, Z(9)-1.679,p SD 1.501 SD = 1.287 SD = 1.333 .090,112 = .240, large effect 6 Mean=2.100, Mean=2.l00, Mean= 1.600, Z(9)=.085,p=.932, SD = 1.370 SD = 1.450 SD = 1.506 r2 = .001 small effect 7a Mean = 4.0 10, Mean = 4.500, Mean = .900, Z(9) = -1.089, p = SD=.943 SD=.707 SD=1.10l .276,i2=.116 (moderate effect) 7b Mean = 4.600, Mean = 4.100, Mean = .900, Z(9) = -1.186, p = SD = .699 SD = 1.101 SD = .994 .236, ri2 = .130 (moderate effect) 8 Mean = 1.900, Mean = 1.800, Mean = .100, Z(9) -.378, p = .705, SD = .568 SD = .918 SD = .483 ii2 = .015 small effect 9 Mean = 2.200, Mean = 2.400, Mean = .600, Z(9) = -.707, p .480, SD = .632 SD .966 SD = .699 ri2 = .001 small effect 10 Mean=2.9, SD = Mean= 1.8, SD Mean= 1.1, SD Z(9)=-2.232,p= 1.197 =.919 =1.197 .026,i2=.356large effect 11 Mean=3.900, Mean3.800, Mean= 1.100, Z(9)-.l73,p=.862, SD = 1.287 SD = 1.229 SD = .876 i2 = .003, small effect 12 Mean=2.500, Mean=2.700, Mean= 1.400, F(9)=- .36l,p= SD = 1.080 SD = 1.337 SD = .966 .726, i2 = .014 , small effect 13 Mean = 1.800, Mean = 1.900, Mean = 900, SD Z(9) = -.141, p = SD = 1.033 SD = 1.101 = .994 .888,112 = .003 175 Question Experimenter Subject Difference Paired-sample T-test Number 14 Mean = 1.900, Mean = 2.300, Mean = .800, Z(9) = -.966, p = .334, SD = .738 SD = 1.337 SD = .844 1i2 = .090 small effect 15 Mean 2.400, sd Mean = 2.600, Mean = .800, Z(9) = -.632, p = .527 1.265 SD = 1.173 SD = .632 , r2 = .040 small effect 16 Mean2.300, Mean=3.000, Mean= 1.500, F(9)=-1.076,p= SD=l.418 SD=1.944 SD=1.509 .310,i2=.llO moderate effect’ 17 Mean = 1.500, sd Mean = 2.200, Mean = 1.300, Z(9) = -1.425, p = = 1.080 SD = .790 SD = .950 .154, ii2 = .180 large effect 18 Mean = 1.300, Mean = 1.600, Mean = .300 SD Z(9) = -1.732, p = SD = .480 SD = .520 = .480 .083, i2 = .250, large effect 176 E.3.2: Paired-sample t-test for local experimenter vs. subject: Question Local Local subject Difference Local experimenter Number experimenter vs. subject 1 Mean 1.400, Mean=2.300, Mean= 1.100, Z(9)=-1.930,p= SD .699 SD = 1.418 SD = 1.200 .054, i2 = .290 large effect 2 Mean = 2.050, Mean = 1.300, Mean = 1.150, Z(9) = -1.450, p = SD = 1.2 12 SD = .675 SD = 1.250 .147,112 = .190, large effect 3 Mean = 1.600, Mean = 1.900, Mean = 1.300, Z(9) = -.422, p SD= .843 SD= 1.449 SD= 1.420 .673,ri2=.020, small effect 4 Mean = 3.400, Mean = 4.000, Mean = 1.400, Z(9) = -.862, p = SD = 1.647 SD 1.414 SD = 1.260 .389,112 = .070, moderate effect 5 Mean = 2.300, Mean = 3.200, Mean = 2.100, T(9) = -1.034, p = SD= 1.159 SD= 1.475 SD=.880 12S,i2.=.llO moderate effect 6 Mean = 1.400, Mean = 2.600, Mean = 1.400, Z(9) = -1.897, p = SD = .966 SD = 1.647 SD = 1.520 .058, q2 = .280 (large effect) 7a Mean = 3.900, Mean = 4.100, Mean = 1.200, Z(9) = -.259, p = SD = 1.449 SD = 1.287 SD = 1.400 .796, 112 = .007,moderate effect 7b Mean = 4.300, Mean = 4.500, Mean = 1.000, Z(9) = -.35 1, p = SD = 1.059 SD = .707 SD = .940 .726, i2 = .010, small effect 8 Mean= 1.500, Mean= 1.850, Mean= 1.150, Z(9)=-.855,p= SD = .850 SD = .818 SD = .750 .393,112 = .070, moderate effect 9 Mean= 1.950, Mean=2.700, Mean= 1.150, Z(9)-1.552,p SD=.597 SD=l.494 SD=1.100 .12l,112=.210, large effect 10 Mean = 2.600, Mean = 2.300, Mean = 1.500, Z(9) = -.423, p = SD = 1.350 SD = 1.059 SD = .970 .672,112 = .019, small effect 177 Question Local Local subject Difference Local experimenter Number experimenter vs. subject 11 Mean = 3.300, Mean = 3.85, Mean = 1.050, Z(9) = -1.089, p = SD= 1.337 SD=.747 SD= 1.300 .276,ri2=.116, moderate effect 12 Mean=2.000, Mean= 1.800, Mean=.800, Z(9)=-.333,p= SD = .943 SD = .919 SD = .920 .739, i2 = .010, small effect 178

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            data-media="{[{embed.selectedMedia}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
https://iiif.library.ubc.ca/presentation/dsp.24.1-0051242/manifest

Comment

Related Items