UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Online experimenter : an evaluation of experiments conducted under local and remote conditions Zhang, Ying 2008

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata

Download

Media
24-ubc_2008_fall_zhang_ying.pdf [ 3.32MB ]
Metadata
JSON: 24-1.0051242.json
JSON-LD: 24-1.0051242-ld.json
RDF/XML (Pretty): 24-1.0051242-rdf.xml
RDF/JSON: 24-1.0051242-rdf.json
Turtle: 24-1.0051242-turtle.txt
N-Triples: 24-1.0051242-rdf-ntriples.txt
Original Record: 24-1.0051242-source.json
Full Text
24-1.0051242-fulltext.txt
Citation
24-1.0051242.ris

Full Text

ONLINE EXPERIMENTER AN EVALUATION OF EXPEPJMENTS CONDUCTED UNDER LOCAL AND REMOTE CONDITIONS  by YING ZHANG B.Sc., The University of British Columbia, 2005  A THESIS SUBMITTED IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE  in  THE FACULTY OF GRADUATE STUDIES (Computer Science)  THE UNIVERSITY OF BRITISH COLUMBIA (Vancouver)  September 2008  ©Ying Zhang, 2008  Abstract When conducting experiments, researchers traditionally work in a lab setting where they have access to specialized equipment and can meet their subjects faceto-face. This is definitely the way that traditional experiments in behavioral research are carried out. However, with the growing need for recruiting experiment participants from different social and cultural backgrounds, it may be difficult to design a traditional lab experiment for the research. Moreover, highly specialized lab equipment cannot always be easily accessed, and the costs of commuting present financial constraints.  We explored the necessity of co  presence in experimentation by evaluating a facility to determine whether experiments conducted in a non co-present context would achieve the same scientific outcomes as the ones conducted in a traditional lab context. The facility that we used was designed to allow researchers’ access to a sophisticated lab for the purpose of experimentation in educational research. There are two research goals explored in this thesis: (1) to evaluate the relative efficacies of running an experiment under co-present and non co-present conditions, and (2) to assess whether the participants’ experience differs in the experimental process between the co-present and non co-present condition.  11  Table of Contents Abstract  .11  Table of contents  iii  List of tables  xi  List of figures  ix  Acknowledgements  xi  Dedication 1. Introduction  xiii 1  1.1 What are the elements of traditional scientific lab experiments 9  3  1.2 “Why is the co-presence of experimenters and subjects desirable 9  5  1.2.1 Human to human communication  7  1.2.2 Quantitative and qualitative measures  7  1.2.3 Ethical concerns  8  1.2.4 Alternatives to face-to-face experiments  9  1.3 Goals and objectives of the research study  14  1.3.1 Assessment of experimental data  15  1.3.2 Assessment of experimental process  16  1.4 Summary of the research problem  17  1.5 Summary of research partners  20  1.6 Summary of terminology  21  1.7 Summary of our study  23  1.8 Summary of results  25  1.9 Summary of our contribution  26  111  1.10 Summary of the rest of the thesis 2. Related Work and Research Infrastructure  2.1 Related work and literature  .27  28 28  2.1.1 Communication technology as research infrastructure  28  2.1.1.1 Emulating realistic face-to-face communication  29  2.1.1.2 Improving spatial realism  30  2.1.1.3 Additional modes of communication  31  2.1.2 Communication technology supporting education  32  2.1.2.1 Video-conferencing in online education  33  2.1.2.2 Tele-instrumentation or Tele-operation as educational technology  34  2.1.2.3 Supporting Tele-instrumentation in remote collaboration  36  2.1.3 Summary  37  2.2. Research infrastructure  37  2.2.1 Backbone of UQAM infrastructure-The BEST project  37  2.2.2 QUAM infrastructure  38  2.2.3 UBC infrastructure  —  —  LORIT  a video-conferencing lab  2.3 Tools  39 41  2.3.1 Click-To-Meet  41  2.3.2 NetMeeting  43  2.3.3 Access Grid with virtual network computing (VNC)  45  2.3.3.1 Access Grid  45  2.3.3.2VNC  46  3. Fitts’s Law Studies of Directional Movement between Two Target Positions  49  iv  3.1 Index of difficulty in Fitts’s formulation  .  52  3.2 Characteristics of Fitts’s Law formulation  53  3.3 Fitts’s Law in interaction studies  55  3.4 An example of a Fitts’s Law Experiment  56  3.4.1 Practice sessions  58  3.4.2 Experiment sessions  59  3.4.3 Experiment software  61  4. Study Design 4.1 Experimental Design  62 64  4.1.1 Hypotheses  65  4.1.2 Conditions  67  4.1.3 Tasks  68  4.2 Participants  70  4.3 Apparatus  71  4.3.1 UBC Apparatus  72  4.3.1.1 Remote condition  72  4.3.1.2 Local Condition  72  4.3.2 UQAM Apparatus  77  4.3.2.1 Workstation hardware  78  4.3.2.2 Workstation software  79  4.3.3 Experiment Software 4.4 Setup 4.4.1 Remote setup 4.4.1.1 Video-conferencing software setup  79 79 79 80  V  4.4.1.2 Video camera setup  .  4.4.1.3 Desktop sharing setup  81 81  4.4.2 Local setup  82  4.5 Data Collection  82  4.5.1 Logging  82  4.5.2 Questionnaire  83  4.5.3 Interview  84  4.6 Procedure  85  4.7 Summary  94  5. Results 5.1 Fitts’s Law Performance 5.2 Analysis of Slopes and Intercepts  95 96 100  5.2.1 Intercept coefficient Ai  101  5.2.2 Slope coefficient Bi  102  5.2.3 Performance time  103  5.3 Errors  103  5.4 Analysis of time to run a Fitts’s Law Experiment  104  5.4.1 Total time spent at each task block at each index of difficulty  105  5.4.1.1 Within-subjects effect  107  5.4.1.2 Between-subjects effect  108  5.4.1.3 Interaction  108  5.4.2 Total time to complete the Experiment in the remote and local conditions 5.5 Questionnaire analysis  110 ill  vi  5.5.1 Interaction Survey  .  111  5.5.1.1 Measures between conditions based on roles  112  5.5.1.2 Participants’ experience within each condition  114  5.6 Interview analysis  115  5.7 Summary  118  6. Discussion  120  7. Conclusions and Future Work  124  7.1 Conclusions  124  7.2 Lessons learned from our Study  125  7.3 Contribution  127  7.4 Challenges  128  7.5 Future work  128  7.5.1 Study design  128  7.5.2 Use of Chat, Video, and Audio  129  7.5.3 Configuration of Equipment  129  7.5.4 Possible modification of remote lab web interface  130  Bibliography  132  Appendices  137  Appendix A  137  Appendix B  158  Appendix C  160  Appendix D  166  Appendix E  170  vii  List of Tables Table 3.1  Index of difficulty and movement time  Table 3.2  Design of our Fitts’s Law Experiment  61  Table 4.1  Statistical interpretations of outcomes  66  Table 4.2  Sections in the questionnaire  84  Table 4.3  Semi-structured interview schedule  85  Table 4.4  Our Study design  94  Table 5.1  r and r2 values obtained from remote and local conditions  98  Table 5.2  Descriptive statistics of r and r2  98  Table 5.3  Intercept Ai and slope Bi (one anomalous value is highlighted) 101  Table 5.4  Descriptive statistics for slope and intercepts  Table 5.5  Descriptive statistics of error rate in remote and local conditions 104  Table 5.6  Mixed between and within Study: condition by index of difficulty 106  Table 5.7  Descriptive statistics of performance time between four different indices of difficulty and two Study conditions 107  Table 5.8  Descriptive statistics of total time spent in each task block corresponding to each index of difficulty  Table 5.9  .  53  101  107  Descriptive statistics of total time spent to complete the all of the twelve tasks 108  Table 5.10 Descriptive statistics of total time spent in remote and local Experiments  110  Table 5.11  Summary of results  119  Table 6.1  Patterns of interaction  120  viii  List of Figures Figure 2.1  LORIT lab and equipment of UQAM  39  Figure 2.2  UBC video-conferencing lab and equipment  40  Figure 2.3  Click-To-Meet window  42  Figure 2.4  Mosaic view  43  Figure 2.5  Screenshot of NetMeeting  45  Figure 2.6  Networks connection for videoconference experiments  47  Figure 3.1  Fitts’s Law task  50  Figure 3.2 a Schematic drawing of our Fitts’s Law task position Figure 3.2 b Schematic drawing of our Fitts’s Law task surrounding position  ---  Target at its starting 58  ---  Target at a 58  Figure 3.3  Four block windows with four different target sizes  60  Figure 4.1  Configuration of the Study in the local condition  67  Figure 4.2  Configuration of the Study in the remote condition  68  Figure 4.3  UBC workstations in remote condition  72  Figure 4.4  LORIT interface  75  Figure 4.5  UBC workstations in local condition  76  Figure 4.6  LORIT Lab  77  Figure 4.7  Video windows  81  Figure 4.8  Procedure of local Experimenter  87  Figure 4.9  Procedure of local Subject  89  Figure 4.10 Procedure of remote Experimenter  90  ix  Figure 4.11 Procedure of remote Subject  92  Figure 5.1  Scatter plot for remote and local conditions  99  Figure 5.2  Total time spent to complete the all of the twelve tasks  109  x  Acknowledgements I would like to express my deepest and most sincere gratitude to my supervisors Dr. Kellogg Booth and Dr. Brian Fisher for providing me with the motivation and the ideas, and for their important support and encouragement to pursue my research.  I would like to extend my gratitude to the research team at the  L’Universite du Quebec a Montreal, including Gregory Petit, Mario Archambault, and Racha Ben Au, for providing me access to the LORIT facility and for the technical support that made my research succeed in such a unique environment. Special thanks to Professor Auder. Dufresne from University of Montreal. As the head of the LORIT@d (Econtrol) development, she designed the interface that directly supported the research, and organized the participation from Montreal in my experiments. I would like to thank my second reader Dr. Jacqueline Bourdeau, for providing invaluable comments and suggestions on my thesis. I am also grateful to my lab manager Ron Fussell for his kindly support on everything during the project. I would like to express my warm and sincere thanks to Dr. Barry Po, who noticed my potential and encouraged me to pursue post-graduate studies in the field of Human Computer Interaction.  xi  I would like to acknowledge my fellow students and friends for their support and feedback. This research could not have been completed successfully without the help of Stiliman Jaguar, Zhangbo Lu, Man Hon Chan, Jian Xu, Gang Peng, Hao Shuang, Garth Shoemaker, Clarence Chan, Miranda Poon, Bernard Ng, Albert Huang, and Kevin Ming. I also wish to thank Ms. Sundari Bala, Ms. Emily Chan, and Mr. Geoffrey Foster for revising the English of my thesis. My warm thanks are also due to Ian, Wendy, and Lily. They made my years at the University of British Columbia an enjoyable and memorable experience. Last, but not least, my thanks go to my parents for their tireless encouragement, endless support, and unconditional love. Without their support and understanding it would have been impossible for me to finish this thesis. Special gratitude is due to my cousins and their families for their loving support. The research reported in this thesis was financially supported by The University of British Columbia through the Media and Graphics Interdisciplinary Centre (MAGIC), the Canadian Foundation for Innovation (CFI) through the SCORE project, the Canadian Network for the Advancement of Research, Industry, and Education (CANARIE) through the BEST project, and the strategic network program of the Natural Science and Engineering Research Council of Canada (NSERC) under the Network for Effective Collaboration Technologies through Advanced Research (NECTAR).  xii  Dedication To my parents and my grandmother  xiii  Chapterl  Introduction Many scientific research fields rely on experimental studies, for recording and modeling species behavior, observing and gaining insight into scientific phenomena, or determining and evaluating the effects of innovations (Preece, Rogers, & Sharp, 2002). This is particularly true for the field of human-computer interaction (HCI) in computer science.  In this thesis we describe a system for conducting experiments with human subjects to evaluate the benefits of educational technology; specifically we investigate the feasibility of using advanced communication technology to support experiments by doctoral students conducting research on educational technology in a distance education program. For such experiments, doctoral students would normally work in a research lab with access to specialized educational technology (computers, software, and trained laboratory technicians), where they could work face-to-face with the subjects of their experiments. This is the traditional way that behavioral research is conducted when evaluating new educational technology; the problem this poses is the necessity for co-presence, whereas most of the doctoral students’ other activities in distance education do not depend on co-presence. The costs (travel, time, inconvenience) associated  1  with face-to-face experiments are those that we would like to avoid. For this reason, the Laboratorie observatoire de Recherche sur le Télé-apprentissage (LORIT), was developed at the University of Montreal! UUniversité du Québec  a  Montréal (UQAM) with advanced communication technology that allows doctoral students remote access to the laboratory to conduct their research experiments. Our contribution is an evaluation of this facility, to determine whether remotely conducted experiments can achieve the same scientific outcomes as do those conducted face-to-face.  This first chapter provides background information about the ways that experiments are conducted, especially those with human subjects using technology, as it is common in HCI and in education. We focus particularly on the trade-offs between face-to-face experiments and those in which the experimenter is working remotely from the laboratory and from the subjects. This is precisely the situation for LORIT; doctoral students use the facility to learn how to conduct experiments, and once they have learned, they run experimental investigations for their dissertation research. The goal is to be able to do this remotely, taking advantage of the communications technology provided by LORIT.  After describing the problem, we present the research hypotheses and then provide an overview of the study methodology and a summary of its results, each of which is explained more fully in subsequent chapters.  2  1.1 What are the elements of traditional scientific lab experiments? There are several elements needed for a traditional laboratory experiment. Lab equipment in the facility is used to perform scientific analysis, measurement, or observation; lab technicians provide support and instruction, reinforce lab safety protocols, and monitor the safe and standard operation of lab equipment, while the experimenters’ role is to investigate the objectives of their research. These essential elements are typically seen in physics and chemistry labs, as well as in many other laboratory settings.  However, in behavioral research, such as in biology or psychology, the subject is a further necessary element. Subjects can be “any organism (animal or plant) that is observed for purposes of research”  .  As the term HCI (human computer  interaction) suggests, human subjects are essential for HCI experiments.  Technology has been increasingly used as a strategic tool for lab experiments. In HCI, researchers experiment with human subjects with various objectives, using advanced technology, including specialized computer software and hardware, computerized lab equipment or computer-controlled devices to support their The Language of Psychology Dictionary and Research Guide. -  http:I/www. 1 23exp-health.com/psychology/  3  experiments. Examples are evaluating a particular vehicle navigation technology in a computer-simulated environment, investigating defects and deficiencies of a piloting interface in a virtual piloting environment, or using an artificial speech system to convey pre-recorded experimental instructions to human subjects. All of these involve human subjects and their interaction with computer technology, often in formal HCI experiments.  In the field of education, technology has been integrated extensively into instructional resources. As researchers in education have discovered, proper use of computer technology in education “can increase opportunities for students” (Svedkauskaite & Reza-Hernandez, 2008, p. 1). Experimental investigation of the interrelations between learning technology and students has evolved rapidly in a number of educational disciplines.  Doctoral students in education have used  technology to conduct human-subject-oriented experiments for various purposes, such as measuring the effectiveness of educational software being developed for targeted student groups, or evaluating learner-to-learner or educator-to-learner communication strategies based on computer-mediated technology. In educational research, we see more researchers conducting experiments, recruiting subjects and instructing them how to interact with computer technology, observing how they can benefit from it and how their learning experiences can be improved by using technology in education.  4  1.2 Why is the co-presence of experimenters and subjects desirable? Co-presence in the laboratory of all of the participants in an experiment is desirable for a number of reasons. Laboratory safety provisions traditionally call for chemistry and physics experiments to be conducted in a controlled environment, with lab technicians ensuring safety; safety equipment in a lab protects experimenters and subjects alike from potential dangers or hazards, while also reducing any potential harmful effects to the public at large and to the environment.  In HCI, co-presence is favoured by researchers most of all for its convenience. Experiments are constrained by practical issues, including tight schedules, low budgets, finding suitable poois of subjects, and getting necessary evaluation resources. Furthermore, these constraints are sometimes interrelated; for example, in user study scheduling, a typical user study session can take from minutes to hours. Some user studies require subjects to participate in more than one study session, for reasons of follow-up or resource availability. With co-presence, some constraints on scheduling and coordination of subject presentation can be removed. Typically, co-presence is convenient when booking evaluation resources; with co-presence, desired equipment or lab facilities, lab personnel or technical support staff, study subjects, and researchers are all geographically located at the same place and in the same time zone, reducing the costs of  5  commuting, retrieving experimental resources, hiring, screening, and scheduling of potential subject candidates and increasing the convenience of equipment access, setup, trouble-shooting, and the tangibility of technical support. Thus, co presence makes conducting experiments more cost-effective, since it offers direct physical access to available resources.  The most advantageous factor favouring onsite user experiments is the ability to restrain certain aspects (Moløkken-Ostvold, 2005), ensuring that experimental results are reproducible under the same conditions and with access to similar equipment. An example would be keeping temperature at a certain level to maintain the stability of a chemical reaction in order to produce an expected product. Particularly in HCI, a popular evaluation method is the controlled study, in which researchers and developers make observations while subjects attending a lab session are asked to perform a set of predetermined tasks to evaluate the usability of a piece of software or hardware designed for an intended user population. (Preece et al., 2002). Using this approach, researchers can control confounding factors and possibly improve the usefulness of the results of their evaluation in terms of the desired outcomes.  The various forms of human computer interaction incorporate the following additional factors that make researchers inclined to prefer co-presence.  6  1.2.1 Human to human communication In user evaluation, it is easier to observe and communicate with subjects if the researcher can interact with them face-to-face. From the researchers’ viewpoint, communication seems more effective when it is done face-to-face, since they can easily see a smile, a frown, or other facial expression that might indicate a subject’s thoughts, feelings, opinions, and understanding, at the same time clearly hearing the tone of voice or pauses in speech. Based on these rich forms of feedback, researchers are able to extract potentially interesting phenomena and take them into account as their designs evolve. With co-presence, the communication flow between subjects and experimenters is not subject to any physical barrier; issues or concerns can be directly and instantly delivered via oral expression or eye contact; any potential subject actions that could influence the study can be noticed and dealt with right away.  1.2.2 Quantitative and qualitative measures Even though “field studies and heuristic evaluation have grown in prominence” (Preece et al, 2002, p. 341), usability testing is still regarded as an important evaluation technique in HCI. This kind of evaluation measures the response of a subject to an interface or a study design in both quantitative and qualitative terms. Quantitative measures evaluate a subject’s performance by, for instance, recording the number of errors and the amount of time needed to complete a set of carefully designed tasks. Qualitative measurements are elicited via questionnaires  7  or interviews. “The defining characteristic of usability testing is that it is strongly controlled by the evaluator (Mayhew, 1999)  ...  typically, tests take place in  laboratory-like conditions that are controlled” (Preece et a!, 2002, p. 341). The researcher is able to be in charge when running usability testing in a lab with co presence. When all factors are accessible, researchers can decide beforehand what objective measurements and subjective measurements to collect, while also being free to focus on any additional objectives or subjective measurements that seem interesting and worthy of further evaluation. This enables them to have a better understanding of their subjects’ experimental experience, at the same time identifying further potential research topics. Objective data and subjective data collection is made easy, because user performance and user feedback can be directly observed and measured.  1.2.3 Ethical concerns There are a number of factors that need to be taken into consideration before deciding whether a traditionally co-located lab experiment can be performed remotely.  Since HCI evaluation involves human subjects, it inevitably leads to ethical issues. In typical usability testing, subjects are asked to provide their personal data for demographic purposes; during the experiment, they may be monitored, videotaped, and voice-recorded. The personal data of subjects along with recordings of their images and sounds are used for research purposes; these can  8  act as potentially powerful evidence, but may impact on the privacy of the subjects. Thus, researchers should not only aim to have their usability or evaluation expectations fulfilled in the study, but they must also obey ethical guidelines and avoid any psychological or physical harm to their subjects. In a face-to-face setting, subject frustration or fatigue can often be observed. In onsite or co-present contexts, ethics obligations can be easily complied with, since the personal images of participants are exposed to only a limited number of personnel physically attending a lab session and only researchers have access to participants’ personal data. Any personal data collected is permitted by ethics guidelines to be used, analyzed, and processed for stated research purposes only. In addition, data storage is usually centralized in a safe and secure venue on site.  1.2.4 Alternatives to face-to-face experiments Because of these concerns, and for other reasons as well, co-presence has so far been the dominant method of conducting experiments, since it can ensure that throughout an entire experiment the above concerns are not compromised.  Nevertheless, in some fields, such as engineering education, the frequency of co presence in experiments is declining. Instead, the use of simulation has increased; research suggests “physical experiments are costly both to build and maintain” (Foss, Eikaas & Hovd, 2000, p.2944), pointing out the disadvantages of persisting with co-presence in experiments.  Experiments involving human subjects and  9  premium equipment are tightly coupled with schedules; in the case of a remotely located lab, a researcher has to physically travel to the lab to run experiments. If the researcher discovers some defects with his study in the first few experiment sessions, he has to cancel the rest of the subject schedule, come back to his original location and modify his experiments, re-schedule all the experiments, and then return to the remote venue to continue his study. This kind of situation demonstrates how costly and difficult it is when co-presence has to be maintained in an experiment.  Sometimes, not all elements need to be co-located. Examples would be a highly instrumented lab facility located far away from both experimenters and subjects, or a knowledgeable technician who is in the middle of a vacation trip but is required to trouble-shoot a piece of lab equipment located somewhere far away. Other examples would be separation of experimenters and subjects by geographical distance. Furthermore, some research activities may need to recruit participants from different social and cultural backgrounds, and these kinds of candidates might not be co-located. For example, when gathering survey information of people in an exotic environment, the subject-hiring process could be challenging if the potential subject pool is restricted by geographical boundaries. If any of these situations emerge during preparation for a user study, the presumption of it being run as a traditional face-to-face study no longer holds.  10  Can an experiment be run without the co-presence of every element? Researchers in human computer interaction have been exploring the possibility of conducting experiments without co-locating all their elements. With the availability of computer networks, personal computers, desktop workstations and web cameras, powered by commercial software products and Internet or web services LogMeln, Click-to-Meet, and Net-meeting  —  —  such as  a number of communication  technologies have been developed and used to support remote lab access for experimental research and study. Researchers have used web cameras, wireless technology, and web services to create applications that allow users to control and configure equipment remotely in an automated lab (Zhang, Ball, C. Extine, & W. Extine, 2007). This kind of technology gives users the ability to conduct a live lab facility operation painlessly and be able to “experiment and monitor experiment results in real time” (Zhang et al., 2007, p. 231). Other researchers have created virtual labs that “enbance students’ learning experiences through a simulation environment” (Zhang et al., 2007, p. 231), such as the “LabVIEW” virtual instrument developed by Zaimovice-Uzunovice, Lemes, and Petkovice (2001) and Gu, Zhu, Dong, Shi, and Wang’s virtual circuit laboratory (2005). By combining research and commercial technologies, computer systems can now support a user with access to a remote lab, regardless of the users’ location, as long as an Internet connection can be established.  11  The advantages of running experiments in a non-co-present context include having equal access to a larger subject pooi than under co-presence. Because researchers are not physically present in a post-secondary distance education institution, subjects are often recruited through a designated website of a supporting or sponsoring university. This opens a larger subject population pooi to researchers, as though they were physically present in the institutions. As resource sharing is encouraged in the academic and research field, running experiments in a distributed way can also enable researchers to easily access a premier lab facility through its web-based interface that is supported by their university or other participating institution. In addition, remote experimentation frees researchers and technical support personnel from commuting, traveling costs and other constraints. Perhaps the chief advantage is that researchers can prototype and pilot their experiments with their remote co-workers and subjects, by means of distributed experimentation.  However, being non co-present also poses some disadvantages. With experiments conducted in a distributed way, data collection is no longer centralized.  For  instance, visual and audio information might be collected and stored separately at each location. Any uncoordinated organization or destruction of the information “increases the chance of improper use or disclosure of subject data” (Patrick, 2007, p. 7).  12  Before web technology was introduced to support distance education or remote interaction, information collected from an experiment was kept locally. With the introduction of web technology, visual and audio information can be broadcast or transmitted on line. This brings in another potential ethical risk, since the information can be intercepted, copied or stolen in real time.  Furthermore, to ensure that a distributed lab experiment is done properly, the network connection is the most important element. A network outage could ruin an entire study session, as manipulation of instruments and objective communications, such as visual and audio data, will be lost immediately. As researchers have suggested, technology support is one aspect of remote experimentation, but there is another important aspect to be considered. Without physical presence at a lab, how can the actual feeling of using the lab be recreated? (Rohrig & Jochheimm, n.d.). In a situation where remotely accessing a lab facility is being done via virtual reality or by remote control of a lab experiment (Schmid, Eikaas, Foss, & Gillet, 2001, p.1), distributed teams collaborating on a remote experiment “required a thorough preparation in terms of research protocol as well as of technical support” (Bourdeau, Dufresne, Tchetagni, & Ben Au, 2006, p. 8). As Bourdeau et al. have pointed out, “a high quality of audio and video is essential for participants to feel totally immersed or even to ideally forget the distance and the technology” (2006, p.8). This brings out another key fact: high-quality network performance is required to achieve  13  smooth remote experimentation. As distance teaching and distance learning gain a stronger presence in post-secondary education, the need to have access to a remote or distributed sophisticated lab equipment facility increases, and researchers are no longer willing to be constrained by geographical separation.  These and other innovations in experimentation raise an important question: With the  level  of advanced  technology  available  currently,  can  a  remote  experimentation facility support any traditional experiment or study in HCI as effectively as it can be done locally? This is the question that we are trying to answer in this thesis.  1.3 Goals and objectives of the research study The initial concern that motivated this research was the necessity for co-presence. Research has verified that co-present condition is still the “gold standard” preference (Hauber, Regenbrecht, Billinghurst, & Cockburn, 2006, p. 413). There have been ongoing desires and attempts to overcome geographical distance. Researchers still hope to run their research activity with remote teams while achieving the same results as though the same activity were being performed with co-located teams.  We can assess the results of an experiment on two levels. The first level is fairly literal. We can look at the data derived from a remote experiment and compare it  14  to that from a face-to-face or co-located experiment to see whether there are differences attributable to the experimental environment. At a higher level, we can ask whether the process of conducting the experiment differs, either in tenns of objective measurements of what happened or in terms of subjective measurements of how the participants felt about the process. Both levels of analysis are of interest, especially for researchers in HCI where “user experience” is always a relevant component of any system.  We had the following expectations for what we might find. These are stated informally here, but will be re-stated in later chapters, as formal research hypotheses.  1.3.1 Assessment of experimental data As the “face-to-face” condition is regarded as the most effortless experimental condition (Hauber et al., 2006), we expected that in an experimental context, local participants (where “local” here means co-present or face-to-face) might outperform those who are remote (where “remote” here means distributed or non co-present).  The types of differences that might be seen would be faster task completion times by local subjects, higher completion rates, or more accurate performance (measured either by a performance scale or by error rates). There might also be differences in terms of the statistical properties of the data, such as less variation  15  within or between subjects, or, perhaps, whether or not statistical significance was obtained.  An example of these can be illustrated by a Fitts’s Law experiment (see Chapter 3 for a full description of Fitts’s Law and the types of experiments that are conducted to verify it). If the co-located and remote conditions differ in terms of data, we might see different coefficients arising from regression testing of the data that could be the result of faster performance by subjects in one of the conditions, or a higher rate of errors, or, in extreme cases, simply a failure of statistical confirmation of the hypotheses.  1.3.2 Assessment of experimental process Even if there were no differences in the data obtained from co-located and remote experiments, the processes might differ in a number of meaningful ways. We expected that local participants might experience richer human-to-human interaction than remote participants.  This could be measured in terms of  awareness of presence, ease of speaking to or hearing, or overall satisfaction of the experiment during conversations among participants.  We suspected that there might be a less attention shift on local participants, and a higher load on remote participants. This could lead to differences in data (in which case the process differences might be what were causing the differences in data) or it might mean that either experimenters or subjects were not satisfied with  16  the experience (subjects might withdraw from a study, experimenters might form an unduly negative opinion about the benefits of conducting a study weighed against the costs of conducting it).  We also expected that the support of a lab assistant might be a necessity under the remote condition but optional in co-presence, for dealing with both technologyoriented and interpersonal issues.  1.4 Summary of the research problem The goal of our research project is to determine the efficacy of conducting experimental studies of subjects where the subject and the experimenter are not co-located, in contrast to the condition where they are co-located. The particular situation of interest is that of doctoral students researching in the field of educational technology who are assessing the benefits of an innovative technology developed to support the learning of a specific subject. These students need to conduct experimental evaluations of subjects who use the technology in learning tasks, in comparison to a control group that does not use the new technology.  When doctoral students are enrolled in distance education programs that do not provide them easy access to the necessary laboratory facilities, it may be difficult for them to conduct their research. Examples might be software developed by a  17  doctoral student to support acquisition by students of certain mathematics skills. The doctoral student would normally conduct laboratory experiments with subjects using facilities in the department in which the doctoral student is enrolled. In a distance education setting, where the doctoral student might not have access to such a laboratory, because travel time and expense would be prohibitive, there is a need to consider alternative mechanisms for conducting the experiments.  Our research partners in LORIT at the University of Montreal / L’Université du Québec  a  Montréal (UQAM) have developed a virtual laboratory facility that  allows an experimenter (a doctoral student) to conduct experiments with subjects (drawn from the usual subject pools at the university) with the experimenter operating remotely through a networked connection, while subjects are located in a facility at the university that is staffed by a research technician who works with the experimenter. The virtual lab facility originated from the BEST project (Banc d’Essai de Services de liaisons optiques pour un environnement de Téléformation doctorale) at the University of Montreal as a component for a virtual doctoral training environment. Cooperating with our partners at UQAM, we studied and evaluated the virtual lab facility, which is an interface supporting remote operation of lab equipment, data collection, and multi-media recording. This interface belongs to an experimental research lab called Laboratorie observatoire de Recherche sur le Télé-apprentissage (LORIT); it is backed by a fiber optical  18  network provided by the Canadian Network for the Advancement of Research, Industry and Education (CANARIE). Because one of the intentions in creating this online virtual doctoral training environment was to utilize the LORIT lab to support doctoral  student immersion and access to  instrumentation  for  experimentation, this lab and its virtual facility provided the technology base for our study. We collaborated with the LORIT team and utilized the LORIT lab and its online web interface to run our study.  The study we are conducting for this thesis is designed to answer the question: Does the virtual laboratory provide an adequate environment to conduct experiments of the type just described? In our current work we deem an environment to be adequate if the outcome of the experiment (results) are similar to what would be achieved in a traditional face-to-face experiment using the same procedures, and if doctoral students obtain the same degree of experience from conducting the experiment. The second requirement recognizes the training aspect of a doctoral student’s experimental work. The results of an experiment are of course a primary outcome of the experiment, but an equally important outcome is the experience gained by the doctoral student in conducting research with human subjects. Our study aims to address the first requirement: Are the results comparable when the experimenter and subject are face-to-face and when the experimenter is located remotely? In making this comparison, we consider both  19  differences in data and in process. Once we are confident the first requirement has been met, subsequent studies can address the second requirement.  1.5 Summary of research partners This highly-interdisciplinary research was conducted with a number of partners, many of whose contributions have been noted above. We summarize here the various organizations that participated and briefly describe their role. The individuals who were involved are listed in the Acknowledgment section.  LORIT  ---  LORIT is a distance learning engineering research lab at LICEF, a  research centre for learning environments at Télé-université and the University of Montreal (TELUQ-QUAM). The LICEF research centre supported our study by granting us physical access to the LORIT facility, online access through their e control software, and excellent research and technical support.  SCORE ---The SCORE program is led by Dr. Jacqueline Bourdeau at TELUQ QUAM. The program has a partnership with the University of Montreal (UQAM) and the University of British Columbia (UBC).  SCORE is supported by the  Canada Foundation for Innovation, which, together with Réseau d’informations scientifiques du Québec and BCNet, provided us with a dedicated optical network “lightpath” connection.  20  BEST  ---  BEST is a project under the SCORE research program. The project is  led by Dr. Jacqueline Bourdeau at TELUQ-QUAM. The project is a joint project with QUAM and UBC.  MAGIC  ---  The Media And Graphics Interdisciplinary Centre conducts  interdisciplinary research on computer-based and computer-associated media at The University of British Columbia. MAGIC worked closely with BEST and SCORE to establish the research partnership with LICEF at TELUQ-QUAM. It also worked closely with CANARIE for partially funding our research.  NECTAR  ---  The Network for Effective Collaboration Technologies through  Advanced Research is an NSERC Research Network focused on collaboration technologies. It provided us with the major source of funding for our research.  1.6 Summary of terminology Our study is an experiment about conducting experiments; therefore, there is a lot of potential for confusion in the terminology describing it and the experiment embedded within the study, which is the activity (task) that participants in our study will engage in.  Part of the terminology problem is that some participants in our study will play the roles of experimenter (i.e., they will carry out the tasks that a hypothetical doctoral student might do in a real experiment) while other participants will play  21  the roles of subjects. This could lead difficult differentiating between the terms “participant” (someone in our study) and “subject” (also someone in our study, but playing a specific role, that of subject in an experiment within an experimental study). To avoid a “Who’s on first?” scenario (Abbot & Costello, 1939), we will adopt specific terminology throughout the remainder of this thesis, as follows:  STUDY  —  The research we are engaged in, which is an evaluation of a classical  human computer interaction theory modeling the act of pointing (Fitts’s Law), will be used as a vehicle to test our hypotheses about the possible differences between co-present and remote environments for conducting experiments.  RESEARCHER  —  The thesis author, any member of the supervisory committee,  or the LORIT technical staff involved in conducting and analyzing the results of the  PARTICIPANT  EXPERIMENT  —  —  STuDY.  Any person who is recruited to take part in the  The task that  PARTICIPANTS  carry out in the  STUDY.  STUDY  (a simple  Fitts’ s Law experiment that gathers data about the time required to point to a target and return back to a starting position using a computer mouse).  SUBJECT  —  Any  PARTICIPANT  whose role is to use the experimental apparatus  (a computer mouse) to execute Fitts’s Law trials.  22  EXPERIMENTER  Any PARTIcIPANT whose role is to direct a second  —  PARTICIPANT  who is playing the role of SuBJECT.  Whenever any of these words are in  SMALL CAPS  they are intended to stand for  precisely the above terms. If we think of our research study as being at the meta level, and the activity we study as being at the domain of study, the intent is that STUDY, RESEARCHER,  and  ExPERIMENT, SuBJECT,  PARTICIPANT  and  all pertain to the meta-level, whereas  EXPERIMENTER  all pertain to the domain. Hopefully  keeping these distinctions in mind will help readers avoid confusion.  1.7 Summary of our study In order to draw comparisons between the outcomes of experiments conducted in local and distributed settings, we chose an experiment for which we were fairly confident that we would be able to achieve clear results in both local and distributed settings. We assumed that Firts’s Law is a very well-known motor control task that is easily replicated in a laboratory if only a modest level of care is taken to follow the experimental procedure for trials. As such, it serves as a good initial measure of the degree to which a virtual laboratory provides the level of support necessary for doctoral students to carry out the type of research described earlier.  23  The primary dependent measurement in our  STUDY  will be the r-squared  correlation parameters obtained from the data generated by each SuBJEcT pair of ExPERIMENTER  PARTICIPANTS.  and the  SuBJEcT  ExPERIMENTER-  The independent variable is whether the  are co-located during their ExPERIMENT. We will  also look at secondary dependent measures, the regression coefficients a and b in the standard formulation of Fitts’s Law T=a+blog(A/W+ 1/2) where A is target amplitude (amount of movement required to acquire a target) and W is target width.  The goal of our  STUDY  was not to test Fitts’s law, which is well established;  however, by analyzing the r-squared values we hope to determine if there are quantitatively measurable differences between the control condition (co-located EXPERIMENTER  co-located  and  SUBJECT)  ExPERIMENTER  and  in our  STuDY  SuBJECT).  and the experimental condition (non-  The data from a Fitts’s Law experiment  is usually subjected to a linear regression test to fit the equation above. A large r squared is suggestive of a successful  EXPERIMENT,  a small r-squared is suggestive  of a poorly conducted experiment; it is a measure of the statistical validity of the data, and thus a primary metric for the “quality” of the experimental data. The coefficients a and b are measures of the data itself. Looking at these as well as at r-squared may alert us to the possibility of equal statistical validity but different conclusions being drawn from the two sets of data.  24  We also looked at the experimental process in both conditions using a variety of objective and subjective measures. We explain these in more detail in later chapters.  We evaluated the efficacy of conducting the  EXPERIMENT  under distributed  conditions and compared it to conducting an experiment under co-located conditions. We used Fitts’s Law to study and compare the performance of SuBJECTS  under different circumstances: with and without the co-presence of  ExPERIMENTERs  with one  and  SUBJEcTs.  ExPERIMENTER. Under  EXPERIMENT  In each study condition, one the co-present condition,  under the instructions of their  SUBJECT  SuBJEcTs  ExPERIMENTERs  was paired  performed the  face-to-face. Under  the distributed condition, we used a video-conferencing infrastructure via a highspeed network facility. SUBJECTS  In this context, we conducted the  EXPERIMENT  with  present at a research lab in a university located in Eastern Canada, and  at the same time with  EXPERIMENTERs  present at a video conferencing lab at the  University of British Columbia.  1.8 Summary of results Quantitative measurements arising from our experiments fell within our prediction; we obtained statistically similar performance between the traditional face-to-face lab condition and the distributed lab condition, both in terms of the  25  data (the various parameters derived for Fitt’s Law and the statistical validity of those parameters).  In addition to quantitative results, we looked at qualitative measurements. Surprisingly, our results suggest that interacting with  SuBJEcTS  EXPERIMENTERS  felt more comfortable  in the remote method than they did in the co-present  condition. This indicates that  EXPERIMENTERS  did experience certain benefits  under the distributed condition, although the distributed condition needs to be undertaken with caution as it may encourage in the  ExPERIMENTER  a lack of  engagement in the task. Evidence for this was observed and reflected in qualitative analysis of data collected from the distributed condition. Our analysis suggests that the possible cause of these problems is mainly induced by ExPERIMENTERS’  over-concentration on certain procedural details in the remote  condition.  1.9 Summary of our contribution To compare the performance of  SuBJECT-ExPERIMENTER pairs under the two  conditions, we carried out analysis in several ways to evaluate their cognitive load, interactivity, and task measurements separately and in combination. The results of the study will inform the next round of remote experiment study design. We believe that our results and some of our recommendations will positively affect collaborative experiments in a remote context.  26  A secondary benefit is that by utilizing an experimental research lab, LORIT, to conduct our user study under the remote condition, we implicitly evaluated the usability of LORIT’s web interface (LORIT(äD or Econtrol) and provided our research partners, the LORIT team, with valuable feedback about the interface.  1.10 Summary of the rest of the thesis In Chapter 2, we describe related work regarding the use of communications technology as research infrastructure. We discuss the selection of an appropriate video-conferencing system for LORIT. Chapter 3 gives a brief introduction to the classical theory of Fitts’s Law, which we used as the basis for simulated experiments in our study to evaluate the effectiveness of LORIT. The complete study design is described in Chapter 4. Chapter 5 presents the results, and Chapter 6 discusses some interesting observations drawn from the study. Chapter 7 presents the lessons learned from our experience to date and possible directions for future work. The appendices contain samples of the materials used in the study.  27  Chapter 2  Related Work and Research Infrastructure 2.1 Related work and literature With recent advances, communication technology has become more important and popular in the academic field. In particular, these innovations have revolutionized methods of research and education; by using these technologies, researchers can now conduct their field studies without leaving their research lab. Students can now have access to school lab equipment and perform their experiments without having to commute physically to their institution. In this context, we will first consider related studies into the use of communication technology in general support of research infrastructure, and then narrow down our focus to its support of online experimentation.  2.1.1 Communication technology as research infrastructure With advanced high quality communication systems as research infrastructure, “there have been increasing opportunities for researchers in different fields to collaborate with each other” (Matsukara, Koyamada, Tan, Karube, & Morlya, 2004, p. 205). Treloar points out that use of technologies increases “volumes of  28  information generated through research, and a recognition of the need to work across traditional disciplinary, institutional and national borders” (n.d., Abstract section, para. 2). Video-conferencing is one communication technology that has been widely used by researchers to support various research activities, such as holding web-based research seminars or broadcasting presentations. Related research  falls  into  three  categories:  emulating  realistic  face-to-face  communication, improving spatial realism and introducing additional modes of communication.  2.1.1.1 Emulating realistic face-to-face communication  To provide researchers with a better infrastructure, there has been extensive research into creating an ideal illusory face-to-face condition with video conferencing techniques. This kind of design is referred to as shared virtual table environment (SVTE). One of the earliest and most well known implementations of realistic face-to-face remote communication is Tele-cubic by Chen, Towles, Nyland, Welch, and Fuchs (2000). Later, research by Schreer, Brandenburg, Askar, and Kauff extended the same research principle in an endeavor to develop a video-conferencing system that “enable the participants to make use of rich communication modalities as similar as possible to those used in a face to face meeting” (2001, Introduction section, para. 1).  Kauff and Schreer suggest that,  “A common theme of all these efforts is to exploit the benefits of tele-immersion, often in a way that the participants will have the impression of being present in a  29  suggesting spatial proximity, enabling a higher degree of natural interaction and effective collaboration” (2001, p. 106). SVTE-based systems have up to now provided meeting attendees with a scenario closer to real co-location, assuring perfect eye contact and use of gesture.  To some degree, however, these kinds of effort have concentrated overmuch on superficial aspects of remote interaction, while unintentionally neglecting its essential novelty. Hollan and Stornetta (1992) suggest that more attention needs to be paid to discovering the technological advantages of remote participation. Similarity, Birnholtz and Horn indicates that remote interaction design should “exploit the unique attributes of the technologies being used” (2007, Beyond being there section, para. 1).  With the cost of large laboratory instruments  increasing, and growing pressure to share superior lab equipment using existing network connections, research infrastructures based on video-conferencing could expand the way that researchers interact with one another.  2.1.1.2 Improving spatial realism Although research shows that face-to-face communication is still the friendliest and richest modality (Hauber et al., 2006, p. 413), some researchers have looked into ways of preserving proper spatial faithfulness that will give users a face-to face feeling. Instead of striving to achieve such an illusory face-to-face scenario, these researchers focus more on effectively supporting online interaction, or on accommodating multiple distributed users in a video-conferencing software  30  application.  Hauber et al. explored “ways to combine the video of a remote  person with a shared tabletop display to best emulate face-to-face collaboration” (2006, p. 413). To preserve spatial faithfulness and make interaction effective, some research efforts have looked into maintaining eye contact during a videoconference. One example is an investigation into gaze conditions in video conferencing; to maintain perfect eye-to-eye contact; Vertegaal, Sohn, and Cheung conducted a study using hidden video tunnels, using an array of cameras at the same height as a participant’s image screen (2003). By “selecting a single camera closet to where the user is looking for broadcast,” Vertegaal et al.’s design managed to preserve good eye contact among video-conferencing attendees (2006, p. 521).  However, these studies were restricted to cases with rich interaction. Although they pointed out “if communication channels were missing, users automatically compensated for this problem with other extensive media,” Hauber et a!. did not evaluate this aspect (2006, p. 414).  2.1.1.3 Additional modes of communication  As video-conferencing systems become used more frequently in the research infrastructure, other researchers have focused on evaluating and comparing the different media components of a video-conferencing system in terms of how they support research activity. Scholl, MaCarthy, and Harr did a user study comparing text-based chat and audio in a multimedia conferencing environment (2006). The  31  researchers found out that chat is an essential communication tool in video conferencing (Scholl et al., 2006). Sun and Razdan also chose to add a chat plugin to their system to enable text messaging between operators and observers (n.d).  Our study used Econtrol, an online web interface enabling remote operation and manipulation of video-conferencing equipment in an experimental research lab. Econtrol is hooked up to NetMeeting, which is a video-conferencing application with text-messaging. In the pilot study, we found this function to be very beneficial to communication, since the text-messaging system could maintain bi directional communication when audio and visual transmission went down.  2.1.2 Communication technology supporting education Recent technological innovation in communication has not only transformed the way that people do research but has also broken down the wall of traditional learning and teaching methodology. With the availability of the Internet, wireless networks, fiber optical networks, streaming video and audio, ISDN, and satellite communication technology, education is no longer restricted by geographical distance or institutional boundaries. Among all the communication technologies that support education, video-conferencing is considered as one of the key tools for delivering quality education.  32  2.1.2.1 Video-conferencing in online education  Recently, video-conferencing has reached “a level of stability, usability and affordability which permits its use in real teaching scenarios rather than research projects” (Coventry, n.d, About this Report section, para. 2); so it is gradually being integrated with educational technology.  Educational applications  empowered by video-conferencing are now capable of delivering web-based, real time teaching, learning, and even instrumentation. However, there has been very little research evaluating its effectiveness in education; a preliminary report (Heeler & Hardy, 2005) tested the use of video, audio, and text messaging in online education. The remainder of this paragraph describes Heeler and Hardy’s study. Heeler and Hardy conducted two user studies to measure the effectiveness of using online media to teach graduate courses remotely, and their study results reflected the fact that the virtual space messaging  —  —  the combination of video, audio, and text  certainly benefited users and provided them with a rich interaction  aid. However, their focus was on familiarizing students and professors with the use of video-conferencing equipment for holding courses online, and finding ways to effectively employ video technology to deliver lecture instructions and materials. Heeler and Hardy claimed that they used the video technology to enable a successful teaching style suited to the online teaching environment rather than to the traditional face-to-face lecture. As the online medium is taking on a more important role in the future of education, Heeler and Hardy make several suggestions to potentially extend their research; one of these being to extend the  33  research into “monitoring students’ progress in lab activities” (2005, p. 132). This matched the initial motivation for our user study.  2.1.2.2 Tele-instrumentation or Tele-operation as educational technology  So far, not much research effort in HCI has been spent on investigating remotely operating lab equipment via a video-conferencing system. However, research on remote laboratories has been active in physics and engineering education (Li, Esche, & Chassapis, 2007). Taylor et al. have developed a “Nanomanipulator” that is a virtual-reality interface to a scanning tunneling electron microscope, based on existing network infrastructure, designed to allow scanning tunneling microscope users “to see, feel, and manipulate matter at the nanometer scale” (1993, p. 128).  Taylor et a!. claim that, before the development of their  application, users using scanning tunnel microscopes had to learn how to use a computer-based application to operate the microscope; with their invention, users can avoid that bother and concentrate on actual manipulation of materials (1993).  Sharing design goals comparable with the “Nanomanipulator” (Taylor et al., 1993), Sun and Razdan have attempted a similar approach (n.d.). The remainder of the paragraph summarizes Sun and Razdan’s study. Sun and Razdan’s goal is to transmit live instrumental data to a remote place via a video-conferencing system, wrapped up with a graphical user interface to remotely visualize data captured, with the instrument being remotely controlled. As in our study, their  34  research uses the Internet to enable a two-way interaction; their intention is to enable novice users to operate a complex piece of equipment without any training, aiming to use the online media to operate a scanning probe microscope and make observations with it in a sharable fashion. At the same time, they want to improve levels of collaboration with the remotely accessed device. However, not much user evaluation has been done on these tele-instrumentation systems in terms of their effectiveness and efficiency for supporting user interaction.  Birnholtz and Horn have looked into earthquake simulation in civil engineering (2007). The remainder of the paragraph summarizes Birnholtz and Horn’s research. Birnholtz and Horn’s research aims at the potential for conducting experiments involving large-scale equipment in a remote context. Concerned about how an earthquake test is conducted, and about the objectives, expectations and presumed accomplishments of this kind of experiment, Birnholtz and Horn’s approach to the problem is through interviews and observations. Birnholtz and Horn reflect a traditional comment that “physical presence at a test provided richer access to the ‘details of the test’ that could not be replicated” (2007, Concerns about Remote Manipulation section, para. 2), but participants in their survey did suggest there was great potential in using remote observation rather than manipulation. Some other advantages of remote participation were also identified: it eases lab safety issues, avoiding subjects being exposed to harmful and dangerous lab materials or experimental situations. Birnholtz and Horn state  35  “there is value in thinking carefully about the goals of e-science collaborators at all phases of the research process.” (2007, Explaining the Findings section, para. 2)  2.1.2.3 Supporting Tele-instrumentation in remote collaboration Instead of one-way tele-instrumentation, some researchers have looked into collaborative  experimentation  supported  by  video-conferencing,  both  synchronous and asynchronous collaborative experimentation under remote conditions. The closest study to this was done by Ranjan, Bimholtz, and Balakrishnan., who looked at using collaboration technology for distributed teams to work together to accomplish tasks, with the purpose of exploring the relationship between hand motion and video information (2006). The remainder of the paragraph summarizes Ranjan et al. ‘s research. Ranj an’s team conducted a non-co-located video- and audio-mediated user study under three conditions of camera monitoring. As well as a fixed-position camera to capture video, their study used a hand-controlled camera and an automated camera to improve collaboration in a remote setting. Coincidently to our study, there were two participation roles in their study as well; one called “the helper” and the other called “the worker” (Ranjan et al., p. 403). Ranjan et al. found that there is a mathematical relationship between things should be seen by a helper and things can be seen by a helper in a camera view. In terms of providing an optimal camera view, Ranjan advised that there is no perfect formula, since workers  36  modify their behavior actively according to the visual feedback cues that workers consider their helper can or cannot see. Moreover, although their video analysis favors automated camera control, Ranjan et al. discovered some positive aspects of using a fixed-position camera, which does not need to be moved every time a hand-shift happens; on the contrary, helpers can independently adjust the camera to acquire a different view, such as an overview, to bring in flexible visual information.  Ranjan et al’s user study also indicates a subjective monitoring  style; helpers did not constantly monitor their workers every minute and in between (2007).  2.1.3 Summary The literature review in this section of the chapter discussed prior research in collaboration technology and techniques for conducting experiments in a distributed environment. It also provided additional background on the research issues that are being addressed in this thesis.  2.2. Research infrastructure 2.2.1 Backbone of UQAM infrastructure-The BEST project The BEST project (Bourdeau et a!., 2006) is another research project that aims to improve tele-learning experiences. The rest of the paragraph summarizes the BEST project. The goal of the project was to introduce an online interactive  37  system that assisted prospective doctoral students in planning a specific academic program and helped existing professional scholars with their progress in a doctoral program. With the cooperation of Kaleidoscope, which is a joint European research network aiming to explore learning technology via digital media, the BEST team designed an online doctoral training system (DTE). The project fulfilled two key factors that Bourdeau identified when researching the requirements of doctoral training: “co-experimentation” and “access to top-level instrumentation” (Bourdeau et al., 2006, p. 1). To achieve the second feature, one research team of the BEST project implemented an online lab interface for accessing and controlling the equipment of an existing experimental research lab, LORIT, which is equipped with apparatus that “allows observation and capture of multimedia data from multiple pre-synchronized sources” (Bourdeau et al., 2006, p. 7).  By introducing this virtual environment, the BEST project successfully  enabled distributed research teams to have access to remotely located equipment and facilities. As our research used LORIT to help with the user study, we will briefly introduce LORIT in the next section.  2.2.2 QUAM infrastructure  —  LORIT  LORIT is owned by LICEF, a research center at the University of Montreal. It is a distance engineering research lab supporting research with a multi-media approach, designed to support video-conferencing, research and development in training, and multimedia usability testing (Bourdeau, Dufresne, Prom Tep, &  38  Turcotte, 2005). Its infrastructure is composed of sophisticated visual and audio facilities (Figure 2.1), and is designed to support collaborative work and to conduct experiments/experimentation through a rich media that can be conveniently accessed (Bourdeau, Dufresne, Tep, & Leclercq, n.d.). The details of LORIT will be described in Chapter 4. Figure 2.1: LORIT lab and equipment of UQAM (picture taken from LORIT’s website)  2.2.3 UBC infrastructure  —  a video-conferencing lab  The Department of Computer Science at UBC has a video-conferencing lab equipped with premiere visual and audio equipment. It has been used for broadcasting research and scientific seminars, such as the WestGrid scientific seminars, during the school year; scientists have also used this lab as a research and discussion venue (Figure 2.2). During recent video-conferencing testing with LORIT, it was proved to be a well-equipped research lab that is suitable for remote experimentation. The details of UBC lab will be described in Chapter 4.  39  Figure 2.2: UBC video-conferencing lab and equipment  Access Grid Shelf  40  2.3 Tools We went through a selection process to find the most suitable video-conferencing tool for our study. There were three candidate applications: Click-To-Meet, Access Grid with VNC, and Net-Meeting.  2.3.1 Click-To-Meet Click-To-Meet was developed by Radvision. It is a scalable desktop video conferencing tool that enables real-time interactive communication in the forms of bi-directional audio, video, and data. It offers an infrastructure that is capable of bringing distributed teams to meet in online venues for different kinds of interaction; its main feature is video-conferencing (Figure 2.3).  41  Figure 2.3: Click-To-Meet window Current Speaker  Screen Layout  Whiteboard We ran a few trial study sessions using Click-To-Meet. It did provide a userfriendly interface for video-conferencing meetings, and it enabled desktop application sharing easily; however, its latency over the data communication network severely affected participants’ performance in the trials.  There were  several cases in which a study session had to be re-run due to a system deadlock.  42  When sharing the application, if a user actively accessed an application on the workstation of another user, the shared application would be blocked from view by a mosaic. To make the application visible, the accessing user had to right-click the mouse to cancel the mosaic; unless this was done, the mosaic would remain (Figure 2.4).  Figure 2.4: Mosaic view 1 http:/ wrad si,n.c,m  -  kADVISION  -  MoziII,Fjre(ax  ID -J’;--  •-.  -  L—  -.  ‘  -.‘-..  -.  ‘  — ..  -  .-  ;..-  ..—  —.  —  Ic’  z’  i-.  — -  I...  -  .‘  ...  —  •-  1-. 1-..  -  Ic’  — c’  .. ...  -c:--tc’ I-  2 ‘;;-  --tc’-  ‘  -  -.  c’-c’c’rc’ —  -  -.  -c’c”c’’-’  ..  1’ -c’  -  ic’-.... 1  ‘-z’-.  .—.-  -  ;-  -  —‘  c’  ic’ .. —  —  —  ..  [I  I  ‘  c’  -  r--’.i  —‘  ‘‘  i  I  L;;r.tc’  1 c”  .2  1  1  -  —---—  ——  -‘  t._  .-  1-.  2.3.2 NetMeeting Net-Meeting is a “conferencing client developed by Microsoft that allows users to have real-time interaction over the Internet” (Netmeeting for Beginners, 2007,  43  What is NetMeeting section, para. 1). NetMeeting is hooked up to LORIT@D, and it is automatically invoked when LORlT(aD is loaded. During our pilot study with Click-To-Meet, we encountered audio and video transmission problems several times. Each time we made the decision to use Net-Meeting temporarily. With the help of Net-Meeting, we resumed the audio and video transmission right away, had Click-To-Meet trouble-shot by the remote technical support staff, and afterwards returned to Click-To-Meet to continue our study session. From this experience of Net-Meeting, we found that although Net-Meeting provides users with only minimal features for video-conferencing, compared to Click-To-Meet, its lower network bandwidth requirement and fast and robust connection over the Internet made it an attractive replacement for Click-To-Meet. However, when testing it with our experimental software, we found the real estate of its video screen was exceptionally small; any aggressive mouse movement could easily escape from the video screen. Since our study required spacious screen real estate to enable sufficient mouse movement, Net-Meeting did not offer an ideal selection.  44  Figure 2.5: Screenshot of Net-Meeting  NetMeeting Not in a -  CaH  View  206.167.  lools  ...  jeip  ?ideo Audio Tuning Wizard... Sharing Chat Whiteboard File Transfer Whiteboard (1.0- 2.x)  CtrI+5 CtrI+T CtrI+W Ctrl+F  Remote Desktop Sharing,.. Options...  ‘I 1 I Name  r,i rii ‘i i41 CO ains commands For conferencing.  2.3.3 Access Grid with virtual network computing (VNC) 2.3.3.1 Access Grid Like the previous two video-conferencing tools, AccessGrid (Version 3.0) is multi-media video conferencing software. It has the scalability of accommodating  45  physically distributed teams to hold online meetings, and provides users with the features of video, audio and text messaging.  2.3.3.2 VNC To enable application sharing, AccessGrid needs to rely on a third party component; although it does supply some default collaboration tools, other third party application-sharing tools have been developed for it, VNC being one of the most popular (Hasan, Lewis, Alexandrov, Dove, & Tucker, 2007). \TNC stands for Virtual Network Computing; it shares an entire view of a person’s desktop with another party. Unlike Click-To-Meet and Net-Meeting, the application sharing feature provided by VNC is modifiable, and this allows some desktop applications containing sensitive information to be hidden (Berry, 2005). VNC consists of a viewer and a server; the function of the VNC viewer is to view a shared desktop, while the VNC server enables a personal desktop computer to be accessed, manipulated, and shared with other participants in an online group meeting. In our case, this feature allows the remote party to access our workstation for technical support when audio or video transmission goes down. Furthermore,  VNC  provides  meeting participants  with  a point-to-point  communication mechanism. Using AccessGrid, we were also able to separate the video-conferencing and file-sharing processes from each other, so as to reduce the transmission burden on the network, reducing the system network load and improving the performance of the experiment software. The only problem with  46  AccessGrid was that it did not provide users with a decently designed conference window layout; with its separate audio, video, and chat windows scattered around a desktop, users had to re-arrange the windows manually and voluntarily.  After testing Click-To-Meet, NetMeeting, and AccessGrid (with VNC) with LORIT, AccessGrid eventually won out because of its system capability, easy technical support, and separation of desktop sharing. Thus, we decided to use AccessGrid along with VNC as the video-conferencing tool for our study.  Figure 2.6: Networks connection for videoconference experiments  World Wide WE  Apart from the tool selection process described above, we went through two phases of study design; there were problems discovered in each specific phase. In the first phase, we used Click-To-Meet to pilot our study under local and remote conditions. In the second phase, we screened through the three video conferencing applications mentioned before choosing AccessGrid. Finally, we used Lightpath with Access Grid and VNC to run our study. As well, although  47  not selected as a tool for our study, Net-Meeting was used as a supporting tool for troubleshooting network connection when technical difficulties were raised with video or audio in phase I.  48  Chapter 3  Fitts’s Law Studies of Directional Movement between Two Target Positions This chapter discusses Fitts’s Law, the theoretical basis for the  EXPERIMENT  that  will be the subject of our STUDY.  Fitts’s Law was originally proposed by psychologist Paul M. Fitts to model human aiming performance in motor (movement) tasks (Fitts, 1954). All the description about Fitts’s Law here is based on the research of Fitts (1954). In his paper, Fitts predicted that although it was known that movement time and target hitting accuracy change as the distance between two targets was adjusted, there must exist a relation between speed of movement, target distance, and target hitting accuracy (Fitts, 1954). Based on this prediction (Fitts, 1954), a hypothesis was proposed regarding the channel capacity of the human motor system, and it was estimated that the capacity remains relatively constant as movement amplitude varies. To verify this hypothesis, Fifts designed a study with several sets of tasks that required subjects to make rapid, reciprocal, and uniform movements between two target locations (Figure 3.1).  49  Figure 3.1: Fitts’s Law task  A  In the study design, Fitts manipulated two sets of variables: width of a target (W) and amplitude (A) between two target plates (1954). The results of the study showed that subject performance changed uniformly as the variables were manipulated. To calculate the desired rate of subject performance, Fitts (1954) proposed an index of performance (Ip) which is formulated as:  Ip  =  l/t  *  (2A/W) 2 Log  After re-arrangement, the formulation became t(time)  =  (2A!W). From the 2 Log  nature of the study design, the formulation indicates the relationship between time of movement and variation of target dimensions. The formulation suggests that the time it takes to acquire a target decreases monotonically as the size of the target increases, and increases monotonically as the distance to the target increases. In other words, Fitts’s Law predicts the amount of time that it takes to select a target, based the location and size of the target. Thus, scientists in human computer interaction have adopted this model in a variety of interface research  50  domains and interface design tasks (Schedlbauer, Pastel, & Heines, 1995). The formulation has also been generalized to following:  MT  (2A/W) 2 a + b Log  where  •  MT is movement time, the dependent measure.  The two independent measures are •  A is the distance (or amplitude) of movement from a starting position to the target center  •  Wis the width of the target, which corresponds to ‘accuracy’ (Zhao, 2002). According to Fitts W is considered the allowed error tolerance in ,  the final position of a tapping movement, since the final stopping spot must fall within +1- W/2 of the target’s center (1954) •  a and b are empirically determined constants derived using regression analysis; they are device dependent.  Since the Fitts’s Law formulation was initially introduced to HCI, MacKenzie points out that other researchers in the field have proposed several variations of the formulation (1992). Two of the most well-known are the variations introduced by MacKenzie and Welford (MacKenzie, 1992). These two modified versions of Fitts’s model differ from the original only in terms of the Logarithmic function  51  used to provide a better fit with observation (MacKenzie & Buxton, 1992). The Welford formulation takes the following form (MacKenzie & Buxton, 1992): MT= a + b Log (A / W + 1/2) 2  And MacKensize formulation (MacKenzie & Buxton, 1992):  MT= a + b 2 Log ( A/W  +  1)  3.1 Index of difficulty in Fitts’s formulation To analyze subject performance in the tapping tasks, Fitts adopted a binary index of difficulty (1954). The original difficulty index (ID) derived by Fitts takes the form:  ID  =  Log ( 2 A/W)  In Welford’s version, the index of difficulty is:  Log ( 2 A/W+ 1/2) and MacKenzie’s version is: Log2(A/W +1)  Fitts derived the index of difficulty based on Information Theory (Fitts, 1954). According to the rationale provided by Fitts, the minimum amount of information  52  required by a human subject to make a movement is proportional to the logarithm of the ratio formed by the amplitude divided by the target width (Fitts, 1954). Given that the original study design of Fitts’s Law decreased the width of the targets by a factor of 2 (2, 1,  Y2,  ¼ inch) and increased the amplitude between two  targets by a factor of 2 (2, 4, 8, and 16 inch), the base value of the logarithm is chosen to be 2. As the key measurement of Fitts’s model, from Table 1, it can be observed that as the value of the ID increases, the movement time increases. Table 3.1 Index of difficulty and movement time  ID  MT  6  0.456  7  0.798  8  0.872  9  0.927  10  0.978  3.2 Characteristics of Fitts’s Law formulation The index of difficulty (ID) of Fitts’s formulation takes the logarithm of the target distance over the target width; thus, ID can be any value. Movement time (MT) is a calculated from the index of difficulty. Since the index of difficulty is an independent variable, and the movement time is a dependent variable, the Fitts ‘s formulation can be transformed to the form of a linear function:  53  MT=a+b*ID  Thus, Fitts’s Law model indicates “movement time (MT) is a linear function of ID” (MacKenizie, 1992, A Brief Tour of Fitts’ Law section, para. 2). Consequently, the values of MT and ID are fit to the line of the equation by linear regression analysis. Since the bivariate regression is one of the most frequently applied regression models used to illustrate the relationship between two variables (Huck, 2004), the Fitts’s Law model can, indeed, can be tested by using bivariate linear regression to fit user test data to the predictions of the model. There are two types of results obtained from a linear regression analysis of the Fitts’s Law model. The first type of data is the equation constants which are a and b. b is the slope of a straight line fitted to a linear regression equation. It indicates the trend of the data (MT versus ID) on the plotted line, a is the intercept of the linear regression line on the axis of the dependent variable, MT.  In Fitts’s Law  fonnulation, a and b are device-dependent constants, and they are used to determine characteristics of human performance for a particular setting or to compare across settings.  A secondary analysis can be conducted on the resulting parameter values produced by the bivariate regression. This includes the correlation coefficient and the coefficient of determination. A correlation coefficient (r) indicates “the correlation between the independent and dependent variables” (Huck, 2004, p. 425).  Its value ranges from -1 to 1, with 1 suggesting a perfect positive  54  correlation, and —l suggesting a perfect negative correlation between the dependent and independent variables. If the value of r is 0, this means that there is no linear relationship between the variables (Huck, 2004). For Fitts’s Law, r quantifies the deviations between the predicted movement time and the actual movement time.  The coefficient of determination (r ) “indicates the proportion of variability in one 2 variable that is associated with (or explained by) variability in the other variable. The value of r 2 will lie somewhere between 0 and +1.00” (Huck, 2004, p. 68). In Fitts’s Law model, this suggests the percentage of the variation of movement time that can be attributed to changes in the index of difficulty. We look at both r and , but we focus on 2 2 r r to determine whether each of our experimental conditions yields valid results.  3.3 Fitts’s Law in interaction studies As mentioned previously, Fitts’s Law has been applied extensively to interface research in HCI. Classical Fitts’s Law research applied the Fitts’s Law model to measure human performance on single variable tasks, which was “making horizontal moves toward a target” (MacKenzie and Buxton, 1992, p. 220). Later on, researchers extended the Fitt’s Law model to two-dimensional and three dimensional pointing tasks (Murate & Iwase, 2001). “Due to its accuracy and robustness, Fitts’s Law has been a popular research topic. Numerous studies have  55  been conducted to explain  ...  extend  ...  and apply Fitts’s Law to various  domains” (Accot & Zhai, 1997, p. 295). Some research extends Fitts’s Law to other domains and tasks. An example is the research of Schedlbauer, Pastel, and Heines, which applies “Fitts’ Law to pointing with a trackball with subjects standing, and compares the trackball pointing performances while sitting versus standing” (1995, Movement Time Models section, para. 1).  The application of  Fitts’s Law also assists user interface design, such as “[invention of pop-up menus] and in interface evaluation [e.g., “embedded model” in [13]]” (Zhao, 2002, Applicability to HCI Discussions section, para. 3). Fitts’s Law is also used for comparing performances of different input devices (Zhao, 2002), such as user interaction with joystick and mouse, in terms of their behavior of pointing and clicking on graphical interface components such as buttons or images.  3.4 An example of a Fitts’s Law EXPERIMENT A typical Fitts’s Law study is “undertaken in a controlled experiment using a group of subjects and one or more input devices and task conditions” (MacKenzie, 1992, Building a Fitts’ Law Model section, para. 1). In such an experiment, researchers manipulate the two variables, A and W, to achieve different levels of index of difficulty. Subjects are asked to acquire targets as quickly as possible. Subject performance is measured based on the movement time of acquiring targets.  56  There have been many Fitts’s Law studies designed by researchers in HCI. Following, we give a concrete example of a Fitts’s Law  EXPERIMENT,  which we  designed for our research: reciprocal tapping between two positions. Using the Welford’s formulation, we designed the Fitts’s Law tasks to change only one of the variables, which is the width of a target.  The target is designed to take the shape of a circle. We set the target width (W) to be 8, 16, 32, and 64 pixels to form four levels of difficulty. The circle radiuses establish the tolerance range within which movements had to be terminated. The amplitude/distance (A) between two target positions is kept constant, and the value of A is set at 168 pixels, which is equal to 52 mm.  Our Fitts’s  ExPERIMENT  contains four blocks; in each block, there are three Fitts’s  tasks. In total, there are twelve Fitts’s Law tasks in our  In each  ExPERIMENT.  task, a set of nine dots is presented in a circular arrangement with the current target position colored in red. The  PARTICIPANTs  are asked to use the mouse to  point and click the red dot that appears on the screen. Once the current target position is clicked, the red target shifts to another position. The  PARTICIPANTs  are  required to alternate between clicking the centre position and a position in the surrounding periphery (Figure 3.2 a and Figure 3.2 b; the color of the actual target is red, but for illustration purpose, it is presented in white in the figure).  Two  different surrounding target positions are randomly selected to remove any prediction or learning effect. For each target position, a  PARTICIPANT  uses the  57  mouse to point and click the red target back and forth between the target positions twenty times. The estimated average time to complete each Fitts’s task is two minutes.  are expected to complete the entire Fitts’s  PARTICIPANTs  EXPERIMENT  within twenty-four minutes.  The positions of the red target are pre-decided by randomization in each task. When presented to the  PARTICIPANTS,  the order of showing the positions and the  sizes of the targets is counter-balanced based on two pre-designed data files.  •  F  •  •  Figure 3.2 a: Schematic drawing of our Fitts’s Law task Target at its starting position ---  Figure 3.2 b: Schematic drawing of our Fitts’s Law task Target at a surrounding position ---  3.4.1 Practice sessions Before moving on to the actual Fitts’s tasks, a perform a practice session.  The  PARTICIPANT  PARTICIPANT  is encouraged to  is allowed to run the trial Session  as many times as he/she wants until perceiving comfort and familiarity with the  58  session.  Once a  PARTIcIPANT  PARTICIPANT  is comfortable with the trial session, the  is instructed to move on to doing the actual tasks.  3.4.2 ExPERIMENT sessions As mentioned above, the PARTICIPANT  ExPERIMENT  contains four blocks of tasks. Each  is asked to complete one block of tasks at a time, and then to take a  five minute break before moving on to the next block. In each block of tasks, the distance between the central target position and all the surrounding target positions remains constant, but a different size of targets is presented. From left to right and from top to bottom, Figure 3.5 shows four block windows of the four different target sizes.  59  Figure 3.3: Four block windows with four different target sizes r . 4  0  L  .  .  . .  .  .  p  .  The data recorded for the Fitts’s Law EXPERIMENT are the actual amplitude distance, time to complete each Fitts’s task, time to complete an entire Fitts’s Law ExPERIMENT,  and errors made through each task. Table 3.2 summarizes the  design of our Fitts’s Law  EXPEJUMENT.  60  Table 3.2: Design of our Fitts’s Law EXPERIMENT Independent Variables 1. N levels of Target Width  2. M levels of Amplitude Dependent Variables  1. Movement Time (MT) 2. Number of Errors  3.4.3 ExPE1UMENT software The software that runs the Fitts’s Law  ExPERIMENT  was implemented using the  Java virtual machine version 5 and the Eclipse platform version 3.2. It runs on the Microsoft Windows XP platform. The software is designed to exactly implement the design of our Fitts’s Law  ExPERIMENT.  The software is embedded  with a logging routine, which records the measures summarized in Table 3.2.  61  Chapter 4  Study Design This chapter describes a  STUDY  whose purpose was to evaluate the efficacy of  conducting experiments when the  SuBJEcT  and the  present, compared to the usual condition when the ExPERIMENTER.  “STUDY”  reporting, and the word  “PARTICIPANT”  as an experimental  within that  SUBJECT  is co-present with the  STUDY  refers to the experimental investigation we are  SUBJECT.  in which  refers to a person who participated in the  The term  PARTICIPANTS  “EXPERIMENT”  and the word  “EXPERIMENTER”  refers to the activity  carried out a Fitts’s Law task. We  reserve the word “SUBJECT” to refer to a PARTICIPANT SUBJECT,  are not co  The terminology introduced earlier will be employed throughout  this chapter. The word  STUDY  ExPERIMENTER  refers to a  playing the role of a  PARTICIPANT  playing the role  of an EXPERIMENTER.  The questions that we seek to answer are motivated by our desire to run experiments  via a video-conferencing system in a “virtual lab space.” We want to  know if the experimental results will be similar to the ones that we obtain in a conventional lab setting,  following  the same procedure, with  SUBJECTS and  EXPERIMENTERS working together face-to-face. We also want to know if an EXPERIMENTER acquires  the same experience conducting an  EXPERIMENT  in a  62  virtual lab space. Our  STUDY  was designed to address these and related issues, all  related to the larger question of whether the results obtained through this kind of experimental approach can demonstrate a valid research methodology.  We decided to focus on the first question, whether the results obtained using a virtual lab would be similar to those obtained in a conventional lab. One way to examine this question would be to simulate the non-co-present condition through controlled laboratory studies that ran which the  EXPERIMENTER  and the  EXPERIMENTs  SuBJECT  in a faked remote condition in  were located in two separate rooms,  side by side. We initially considered this approach, and in fact, ran some preliminary  EXPERIMENTS  to test this methodology. However, we realized there  were a number of potential problems with the approach, so we decided that we would go the extra effort to test the methodology in a more realistic, ecologically valid setting. We worried that  PARTIcIPANTs  might not suspend their disbelief if  they knew that they were in fact not engaged in a remote collaboration, and that it would be difficult to fool them into thinking that they were.  For these and other reasons, we chose to conduct a condition, where an  EXPERIMENTER  STUDY  with a genuine remote  is on one side of a country and a  SUBJECT  is  on the other side of the country. This would allow us to assess the subjective and objective efficacy of remote experimentation as it is experienced by both an ExPERIMENTER  and a  SuBJEcT.  Another reason we did this was that the remote  lab facility and the virtual environment we utilized to run our  STuDY,  LORIT, was  63  designed and built to support doctoral students in the field of education by allowing remote access to the lab for the purpose of running experiments during their dissertation research. We were therefore fairly confident in assuming that LORIT was a natural candidate for supporting remote experimentation and that it afforded a good test bed to answer the questions we were posing.  The first objective of our  STUDY  was to determine whether remote and co-present  experimentation would yield statistically similar  SuBJEcT  performance. We chose  a Fitts’s Law task for this, because we knew it had been well studied and was quite robust in outcomes, even in the face of various confounding factors. Furthermore, it could be easily conducted by naïve  PARTICIPANTs  in our lab  setting. To address this objective, we looked both at the results obtained in the two conditions and at the statistical properties of those results. As a second objective, our  STUDY  experience between  also examined the communication and collaboration  EXPERIMENTERs  and  SUBJECTs  as they occurred in the two  conditions. As we will explain later in Chapter 5, some aspects of the ExPERIMENTER-SUBJECT  interactions did turn out to be quite interesting.  4.1 Experimental Design We tested two collaboration conditions suitable for conducting an about Fitts’s Law tasks, in which  PARTICIPANTS  EXPERIMENT  were required to see and talk to  each other. We used a 2x4 mixed model design with co-presence as a between  64  subject factor (with two levels  —  co-presence and non co-presence) and target  width as a within-subject factor, using the four levels described in the end of the previous chapter.  4.1.1 Hypotheses The hypotheses we were trying to test in the  STUDY  were:  Hypothesis #1— Co-presence of EXPERIMENTERs will affect performance in an  SUBJECTs’  ExPERIMENT.  Hypothesis #2— The interpersonal experience between EXPERIMENTERs and SUBJECTS  will depend on whether the  EXPERIMENTER  and  SUBJECT  are collocated  or not. Hypothesis #3  --  The amount of cognitive load experienced by a PARTICIPANT  will depend on whether the  EXPERIMENTER  and  SUBJECT  are collocated or not.  These hypotheses led to the following testable operational hypotheses: Hal.1: Statistically, the validity of the Fitts’s Law EXPERIMENT data obtained in  the co-present condition is better than those obtained in the non co-present condition H 1 2 .2: Performance of local that of remote  SUBJECTs in  SuBJECTs is  better (faster and less error-prone) than  the Fitts’s Law EXPERIMENT.  65  Hal.3 Local  SUBJECTS  spend less time to conduct all the tasks in each task block  and overall they spend less time to complete all the Fitts’s Law tasks than that of remote  SUBJECTs  in the Fitts’s Law  ExPERIMENT.  : Interaction experience is richer in the co-present condition than in the non 2 Ha co-present condition. : Local 3 Ha  EXPERIMENTERs  would experience a less attention shift than the  remote EXPERIMENTERS.  Based on our hypotheses 1.1 and 1.2, Table 4.1 lists the possible interpretation of the statistical interpretations of outcomes of Fitts’s Law EXPERIMENTS in local and remote conditions: Table 4.1 Statistical interpretations of outcomes  same  Human performance measured as by the Fitts’s Law coefficients a &b  different  Quality of the correlation as measured by r 2 same different The local and remote One experimental conditions are identical condition is better, but both in terms of human performance is experimental validity probably the same and measured human performance The EXPERIMENTS are Only one condition equally valid in the local demonstrates true and remote conditions, human performance but human performance (the one with better is different ) and the other 2 r yields a false measure of human performance  66  4.1.2 Conditions Figure 4.1 Configuration of the STUDY in the local condition  N N N  //  is t / Researcher  N N N N N N  Meta-observation/’  Meta-observation  N  N N N N N  0fr’  Observation Prim ryWorks fl_J a t A .  Interaction  ExPERIMENTER\  1 V;  UBC  SUBJECT  Configuration of the study in local condition Condition 1. condition, one another  “Co-presence/Local/Co-located/Face- To-Face”: In the local PARTIcIPANT  PARTIcIPANT  playing the role of  ExPERIMENTER  sat right beside  playing the role of a SUBJEcT in a laboratory at UBC in  Vancouver (Figure 4.1). Both parties collaborated and communicated with each other verbally and interacted with each other through hand gestures and body movement.  67  Figure 4.2 Configuration of the STUDY in the remote condition  Web  Configuration of the study in remote condition  Condition 2. “Non co-presence/Remote/Non co-located/Non Face-To-Face”: Under this condition, one PARTICIPANT playing the role of ExPEIUMENTER sat in a lab located in UBC, and another PARTICIPANT playing the role of SuBJECT sat in LORIT at UQAM in Montreal (Figure 4.2). They collaborated and communicated with each other through video conferencing.  4.1.3 Tasks In each condition, PARTICIPANTS were assigned to play the role of either being a SUBJECT or an ExPERIMENTER.  The task of a SUBJECT was to do a Fitts’s Law ExPERIMENT, to be by following the instructions of an ExPERIMENTER (Chapter  accomplished  3 gives the detail of  68  the Fitts’s Law  ExPERiMENT).  to complete the  EXPERIMENT,  Each  SUBJECT  but the  was allowed as much time as needed  SuBJECT  was encouraged to take as little  time as possible. In case of instructional confusion, the communicate with the  The task of an  EXPERIMENTER for  EXPERIMENTER  assistant, to familiarize the Then the  EXPERIMENTER  accomplish the Fitts’s Law to observe the  SUBJECT’s  SuBJECT  was required to  clarification.  was first to go through a training session with a lab  ExPERIMENTER  with the Fitts’s Law  was required to instruct the ExPERIMENT.  The  ExPERIMENT.  SuBJECT  EXPERIMENTER  was also required  behaviour and to make notes of any noticeable and  interesting phenomena. If an issue arose, the  EXPERIMENTER  was required to  make necessary interventions in a proper form to ensure the continuation with the Fitts’s Law task. The  EXPERIMENTER  for answering any questions raised by the EXPERIMENTER  on how to  SUBJECT.  SUBJECT’s  was also responsible  Another duty of the  was to collaborate with a co-located lab assistant to resolve any  technical issues arising in the  EXPERIMENT.  In the remote condition only, after  training with a lab assistant and before instructing a  SUBJECT,  the  EXPERIMENTER  in Vancouver was required to set up (configure) some of the equipment in the remote lab in Montreal.  69  4.2 Participants Forty-two  PARTICIPANTS  (28 female and 14 male) took part in the  They formed 21 pairs, with each pair assigned to one PARTICIPANTS  STuDY  ExPERIMENT.  session. Pairs of  each conducted 12 trials in a single session of the  STUDY,  for a  total of 252 trials. Due to technical problems, only 20 out of 21 sessions could be used, which meant there were two extra analyzed. The age of PARTICIPANTS  PARTICIPANTs  PARTICIPANTS  whose data was not  ranged from 18 to over 40 years old.  were college students with experience in using a mouse and a  personal computer. They were comfortable using a mouse to interact with a software application running on desktop personal computers. Additionally, all PARTICIPANTS PARTICIPANTs  were  right-handed  and  had  no  color-blind  deficiencies.  were required not to have conducted a significant amount of  research in video-conferencing or remote interaction. Out of the 40 who completed the  STUDY  PARTICIPANTs  (27 female and 13 male) 4 were in business programs,  24 were in engineering or science programs, and 12 were in arts programs. All PARTICIPANTs  had normal or corrected-to-normal vision.  paid $20 for their participation in the  PARTICIPANTS were  STUDY.  PARTICIPANTs were recruited from university students in different departments at the UBC and UQAM.  PARTICIPANTS  had no prior knowledge of the  ExPERIMENT.  PARTICIPANTS were also screened either by phone or by email to ensure their eligibility.  70  In the local condition, the 20  PARTIcIPANTs  randomly assigned to act as a ExPERIMENTER.  and the other half to act as an  In the remote condition, all 10 UBC  designated to be designated as  SUBJECT  were all from UBC. Half were  ExPERIMENTERs,  SUBJECTs.  and all 10 UQAM  PARTICIPANTs  were  PARTICIPANTs  were  A more complete randomization of roles was not  possible due to geographical constraints imposed by the lack of a specialized facility similar to LORIT in Vancouver.  4.3 Apparatus Hardware and software was used in the two labs, chosen to be as similar as possible for the workstations that were used by  PARTICIPANTS,  but some  differences existed. The remote condition required additional communications hardware and software. In both remote and local conditions, the workstation used by a  SuBJECT  to conduct the Fitts’s Law  ExPERIMENT  was defined to be the  primary workstation. In the remote condition, the primary workstation was one of the user stations in LORIT. In the local condition, the primary workstation was the only workstation in the UBC lab.  Because the remote condition required  video-conferencing to enable the communication and collaboration between UBC EXPERIMENTERs  and  SUBJECTs,  in addition to the primary workstation, there were  a few additional workstations used in the remote condition. We describe the workstation details in the following sections.  71  4.3.1 UBC Apparatus 4.3.1.1 Remote condition In the remote condition, there were three desktop workstations in the UBC lab (shown in Figure 4.3):  EXPERIMENTER’s  dominant workstation,  EXPERIMENTER’s  supplementary workstation, and the workstation of ExPERIMENTER’s lab assistant. EXPERIMENTER’s EXPERIMENTER  dominant workstation was used most frequently by an  and was considered the secondary workstation.  supplementary workstation was used less frequently by an  EXPERIMENTER’s  EXPERIMENTER,  was used to control the instrumentation that monitored the  SUBJECT  and it  in LORIT.  The lab assistant’s workstation was used by the UBC lab assistant to communicate with the lab assistant in the remote venue in LORIT. Figure 4.3 UBC workstations in remote condition /  /  / (r  EXPERIMENTER’S  supplementary wr1cctitinn  /  /  /  ExPERIMENTER’s  1  dominant workstation  ,,  /  /  Workstation of EXPERIMENTER’S  lab assistant  r  72  Workstation hardware  ExPERIMENTER’s dominant workstation was a dual monitored personal computer equipped with Intel Pentium III CPU with 1 GB RAM, a 17” WACOM Video monitor, and a second 14” ViewSonic video monitor. As well, a Logitech wireless keyboard and a Microsoft wireless 1001 mouse were attached to the workstation. They were used by UBC  PARTICIPANTS  to input information and interact with the  workstation.  The ExPERIMENTER’s dominant workstation was dedicated to video-conferencing and thus it was connected to a fiber optical network (Lightpath) for transmitting real-time audio and video from LORIT to UBC. On the 17” monitor, a UBC PARTICIPANT  could see the desktop of the primary workstation in LORIT and the  images of themselves and the remote party; on the 14” monitor, a UBC PARTICIPANT  could see the control panel of the video-conferencing software. To  capture the image and voice information of a UBC  PARTICIPANT,  a Neutrik AKG  C1000S microphone with echo-cancellation and a Logitech QuickCam were mounted on the workstation. There was also a pair of stereo loudspeakers attached to the workstation; it transmitted the voice of a remote  SUBJECT  from LORIT to  UBC.  The  EXPERIMENTER’s  supplementary workstation was a personal computer  equipped with Intel Pentium III CPU with 1 GB RAM and a 15” ViewSonic video monitor and a regular Internet connection independent of the Lightpath  73  connection used by adding  EXPERIMENTER’S  EXPERIMENTER’s  dominant workstation.  The purpose of  supplementary workstation to the remote condition was  to load up the LORIT web interface  —  LORIT@D. As a fiber-optical network,  the Lightpath connection was assigned to specific network address that is different from the network address of general Internet. As such, Lightpath cannot act as general Internet connection. Because the LORIT web interface only loads on Internet Explorer, and Internet Explorer can only be loaded via general Internet, we had to look for ways to add a second Internet connection. Although we could have added an extra network card onto the secondary workstation for regular Internet connection, due to time constraints and technical support reasons, we did not do that. Instead we added this workstation right beside EXPERIMENTER’s  dominant workstation and dedicated it to connecting to the  LORIT@D web interface.  The workstation of EXPERIMENTER’s lab assistant was a personal laptop computer with 14” display connected to regular Internet. The UBC lab assistant used this workstation as a separate channel to communicate with the remote lab assistant in LOR1T.  Workstation software The EXPERIMENTER’S  dominant workstation  Pro SP1.O operating system.  was equipped with the Windows XP  To enable video-conferencing in the remote  74  condition, the workstation was installed with Access Grid version 3.0 (video conferencing software), and VNC (virtual network computing).  The EXPERIMENTER’s supplementary workstation was installed with the Windows XP Pro SP 1.0 operating system and Internet Explorer with windows online messenger (msn). When loading up the LORIT interface (Figure 4.4), Microsoft ActiveX component was installed on this workstation as well. Figure 4.4: LORIT interface I 0  — ..... — ?  I mUcn  ,  -.  I I  Aisistaic.  TliconMr.nc.  Prs..,t.ton  •  MS  —-.s-  9/  The workstation of EXPERIMENTER’s lab assistant was installed with the Windows XP Pro SP 1.0 operating system and Internet Explorer with windows online messenger (msn).  75  4.3.1.2 Local Condition  In the local condition, there was only one workstation used in the UBC lab, and it was the primary workstation (shown in Figure 4.5). It was strictly used by local SUBJECTS.  Figure 4.5: UBC workstations in local condition  Primary workstation  Workstation hardware  The primary workstation was a single monitored personal computer equipped with Intel Pentium III CPU with 1 GB RAM and a 17” WACOM Video monitor. As well, a Logitech wireless keyboard and a Microsoft wireless 1001 mouse were attached to the workstation. They were used by UBC  PARTICIPANTs  to input  information and interact with the workstation.  Workstation software  The Primary workstation was installed with the Windows XP Pro SP 1.0 operating system, and Fitts’s Law ExPERIMENT software was installed on this workstation.  76  In both the remote and local condition, besides the workstations, a Sony mini-DV 3.0 camera mounted onto a Velbon tn-pod was set up in the UBC lab for video taping each STUDY session.  4.3.2 UQAM Apparatus Figure 4.6: LORJT Lab  The remote lab, LORIT (Figure 4.6), is a multi-media video-conferencing lab located at the UQAM. From now on, LORIT refers to the remote lab.  77  4.3.2.1 Workstation hardware  There are six PC workstations in LORIT, each of them equipped with Intel Pentium III CPU with 1 GB RAM, P/S2 Optical Mouse, ViewSonic Graphics Series G5773 19” display, QWERTY Keyboard, and Sony EVI D30 short range microphones. In the remote condition, we only utilized two of the workstations, Workstation No. 6 and No. 8. The  SUBJECTs  at LORIT only used Workstation  No.6, the primary workstation in the remote condition, to do the Fitts’s Law EXPERIMENT.  Regularly, all the workstations at LORIT are connected to regular  Internet; however, to support  our  remote  STUDY  condition,  the  network  connection of this station was temporarily switched to Lightpath and communicated directly with  EXPERIMENTER’s  dominant workstation at UBC..  Workstation No. 8 was used by the LORIT lab assistant as a separate channel to communicate with the UBC lab assistant, and this workstation communicated directly with the workstation of EXPERIMENTER’s lab assistant at UBC. LORIT is also equipped with three cameras controlled by an auto-controlling system. Our STUDY  only used one of the cameras, Camera No. 2, to capture the image of the  PARTICIPANTs.  The audio of the  PARTICIPANTs  was picked up by the short-range  microphones on the table of the primary workstation. There is a console right beside the workstation of the  PARTIcIPANTs.  It was used by the remote lab  assistant to control all devices and equipment within the lab.  78  4.3.2.2 Workstation software To enable video-conferencing, each workstation was installed with Windows XP Pro SP2, Access Grid version 3.0, VNC (virtual network connection), and Windows Media Encoder. The Fitts’s Law software was installed on workstation No. 6. Workstation No. 8 was temporarily installed with Microsoft windows messenger (msn).  4.3.3 Experiment Software We implemented the Fitts’s Law  EXPERIMENT software  (mentioned in Chapter 3)  as an application for conducting Fitts’s Law trials. In the remote condition, the Fitts’s Law  ExPERIMENT  software was pre-installed on the primary workstation,  which was Workstation No. 6 in LORIT; in the local condition, the same software was still pre-installed on the primary workstation, which was the only workstation at UBC (Figure 4.5).  4.4 Setup 4.4.1 Remote setup The remote condition required communication and camera equipment to be set up, since an  EXPERIMENTER  and a  SUBJECT  were located in separate geographical  locations. In this condition, both parties had to observe and communicate with each other via the video and audio channels provided by a video-conferencing  79  infrastructure over Lightpath (fiber-optical network). The following setups were performed:  4.4.1.1 Video-conferencing software setup Lab assistants on both sides started running video-conferencing software and tried entering the same web-conferencing venue. Once both parties had successfully entered the conference venue, both lab assistants tested the video and audio channels to ensure that video and audio transmissions were running properly. On the right side of an  two video windows were arranged in  EXPERiMENTER’s screen,  a format shown in Figure 4.7. The lower window showed the image of the EXPERIMENTER.  On a  The upper video window showed the image of a remote  SUBJECT’s  SUBJECT.  screen, there were two video windows arranged as shown in  Figure 4.7. The top window showed the bottom window showed the  SUBJECT  ExPERIMENTER  was specifically designed to show a remote and the local PARTICIPANT on the  at the remote lab, and the  at UBC. The window arrangement PARTICIPANT  on the upper window  lower window. We assumed that a  PARTICIPANT  would naturally pay more attention to the remote party became positioning a window in a higher location on the screen draws more importance.  80  Figure 4.7 Video windows  4.4.1.2 Video camera setup EXPERIMENTERS  at UBC were required to adjust one of the cameras in LORIT to  have a clear view of their  SUBJECTS.  With a lab assistant’s help, a UBC  opened a web browser on  EXPERIMENTER  EXPERIMENTER’S  supplementary  workstation and loaded the web interface of LORIT lab to configure a designated video camera, until a satisfactory view of a SUBJECT  in LORIT could not observe an  had to ask the  ExPEFUMENTER  SUBJECT  at LORIT was achieved. If a  ExPERIMENTER  properly, the  SUBJECT  to adjust the UBC camera manually to achieve a  proper view, because the UBC camera was not attached to any camera-control system.  4.4.1.3 Desktop sharing setup Since a  SuBJECT  only used the primary workstation, the Fitts’s Law  ExPERIMENT  software was run on the desktop of the workstation. To view the desktop of the  81  SUBJECT’S  workstation in LORIT, a UBC  ExPERIMENTER  needed to activate a  desktop sharing application, VNC, and establish a connection with the workstation. SUBJECT’s  SUBJECT’S  Once the VNC connection was made, a shared view of the  desktop with the view of the  ExPEIUMENT  positioned in the center of the screen of the workstation. Both  PARTICIPANTs  application window was  ExPERIMENTER’s  dominant  could interact with the application in the VNC  window.  4.4.2 Local setup The setup of the local condition was much simpler. It only required activating and running the Fitts’s Law ExPERIMENT software on the primary workstation, the SUBJECT’S  workstation at UBC.  4.5 Data Collection 4.5.1 Logging To monitor PARTICIPANTS’ performance, a logging ExPERIMENT software.  widget was embedded into the  It was also written in Java. We used the logging widget to  extract several pieces of information, including  PARTICIPANTs’  full name,  coordinates of target positions, time taken to click a starting target position to an ending position, target number, and the labeling number of each trial.  82  4.5.2 Questionnaire At the end of each ExPEIUMENT session, PARTICIPANTS were asked to complete a short questionnaire (Appendix A). The questionnaire contained two parts. The first part was a demographic information survey, and the second part was the actual EXPERIMENT survey. Questions 3, 6, and 7 were adopted from Hauber et a!. (2006). PARTICIPANTs were given a specific questionnaire according to the roles that they played in the STUDY. Table 4.2 gives the detailed questionnaire design. The actual EXPERIMENT survey part of the questionnaire consisted of five parts:  •  Interaction: PARTICIPANTS were asked to rank their interaction experience.  •  Training: one statement about their level of confidence after going through a training session.  •  Videoconferencing: six statements asking about PARTICIPANTs’ experience using the video-conferencing system to conduct the EXPERIMENT in the remote condition  •  Setup: five statements asking about PARTICIPANTs’ interaction expenence with the LORIT interface.  •  Free form comments  83  Table 4.2: Sections in the questionnaire  Condition  Remote  Local  PARTICIPANT  EXPERIMENTER  SUBJECT  EXPERIMENTER  SUBJECT  Interaction (Qi— Qi 1)  Applied  Applied  Applied  Applied  Training (Q12)  Applied  Applied  Applied  Applied  Videoconferencing (Q13 Q18)  Applied  Applied  N/A  N/A  Set-up (Q19-Q23)  Applied  N/A  N/A  N/A  Free form comments  Applied  Applied  Applied  Applied  -  There were twenty-three questions in total, divided into four sections, with four versions of the questionnaire depending on the role a  PARTICIPANT  whether the  PARTICIPANT  PARTICIPANTs  in the remote condition who played the role of  played and  was in the local or remote condition. Only EXPERIMENTER  were asked to answer all twenty-three questions. Those who played the role of SuBJECTs  rn the remote condition were asked about video conferencing, but not  about the set up remote procedure.  PARTICIPANTS  in the local condition were  asked neither of these sets of questions.  4.5.3 Interview After the questionnaires were completed, all  EXPERIMENTERS  were required to  participate in a post-EXPERIMENT interview (Appendix B). We used this interview  84  to capture any interesting issues that were missed in the questionnaire. Table 4.3 gives the detailed interview design. The interview consisted of two parts: •  Role-playing issues:  EXPERIMENTERS  experience of playing the  EXPERIMENTER’s  encountered when running the •  were asked to comment on their  Video-conferencing issues:  role and the problems they  EXPERIMENT.  EXPERIMENTERS  were asked to describe the  problems or concerns that they experienced while using the video conferencing software and in setting up a LORIT’s camera. As shown in Table 4.3, remote  EXPERIMENTERs  feedback on all four questions, and local  were asked to give free form  EXPERIMENTERs  were asked to only give  free form feedback on the first two questions.  Table 4.3: Semi-structured interview schedule  Condition  Remote  Local  Qi -2(role-playing)  Applied  Applied  Q3 -4(video-conferencing)  Applied  N/A  4.6 Procedure One pair of arrival,  PARTICIPANTs  PARTICIPANTs  was present for every 1.5-hour session. Upon their  were given a consent form, which outlined: 1) the goal of  85  the  EXPERIMENT,  and 4) a  2) the general procedure, 3) the anonymity of the  PARTIcIPANT  consent text, which required a signature.  According to the role that a  PARTICIPANT  instructions that described the tasks of the  was assigned to play, specific oral STuDY  were delivered. After any  questions with respect to the tasks were answered, each preparing for the  ExPERIMENT,  ExPERIMENT  PARTICIPANT  started  separately.  In the local interaction condition, an  ExPERIMENTER  was first trained face-to-face  by a local lab assistant to go through the whole experimental procedure (Figure 4.8). The purpose of this was to make an objectives and gain the experience of being a EXPERIMENTER SuBJEcT  met with a  and running the  SuBJECT  ExPEJUMENTER SUBJECT.  understand the  After the training, an  and started delivering instructions to the  EXPERIMENT.  86  Figure 4.8: Procedure of local  EXPERIMENTER  (EXPERIMENTER procedure __j)  Given a consent form, which outlines 1) the goal of the EXPERIMENT, 2) the general procedure, 3) the anonymity of the EXPERIMENT, 4) a PARTICIPANT consent text, which is to be signed by them.  Specific oral instructions are delivered, which describe the tasks of the STUDY  After potential questions with respect to the tasks are answered, PARTICIPANT starts preparing for the EXPERIMENT  First trained face-to-face by a local lab assistant to go through the whole EXPERIMENT procedure  No  Running EXPERIMENTs  Blocki  BIock2  Block3  Block4  144 EXPERIMENTER filled out questionnair form  JEnd  D 87  Still under the local condition, when an SuBJEcT  done,  was in training, a local  was given some information about Fitts’s Law. Once the training was  that  SuBJEcT  EXPERIMENTER’s STuDY  EXPERIMENTER  would  guidance, the  rejoin SuBJEcT  an  ExPERIMENTER.  EXPERIMENT (Figure  the  then started doing a trial session of the  as a warm-up. After the trial session, the  onto the actual  Following  SUBJECT was  then asked to move  4.9).  88  Figure 4.9: Procedure of local SUBJECT  Following the EXPERIMENTER’s guidance, the SUBJECT then starts doing a trial session of the STUDY as a warm-up  No  SUBJECT filled out questionnair form  In the remote condition, there were some additional steps. In this condition, an EXPERIMENTER  was first trained by the lab assistant in LORIT via a video  conferencing system to go through the whole  ExPERIMENT  (Figure 4.10). If any  89  problems came up, the assistant.  EXPERIMENTER  would ask for help from the co-located lab  Once the training was complete, the  ExPERIMENTER  was required to  perform the video camera setup mentioned in Section 4.4.2. After the cameras were set up properly on both sides, then the  ExPERiMENTER  perform the desktop sharing setup and ran the Fitts’s Law with the local procedure. If problems occurred, the  was required to  EXPERIMENT  EXPERIMENTER  just as  would either  intervene directly via VNC or instruct the remote lab assistant to physically perform the intervention. Figure 4.10: Procedure of remote  EXPERIMENTER  90  (Remote EXPER IMENTER procedure ‘s  4 EXPERIMENTER training: 1.The EXPERIMENTER sat in front of a running video- ccnferencing system 2.The EXPERIMENTER was trained by a remote lab assistant via the video conterencing infrastructure to walk through whole FiUs’s law EXPERIMENT  —  Request help from lab assistant  Camera Setup: 1.Load up the LORITID (remote lab interface) on the secondary workstation 2.On this interface, the EXPERIMENTER need to find the correctcamera icon i adjust one of the remote cameras to acquire a clear view of the remote subject i II the subject that the EXPER IMEN TER is setting up the camera i check the webcam of the EXPERIMENTER to ensure the subject also see a clear vias of the EXPERIMENTER once the both parties agree on the camera setup, then introduced to each other  4  Desktop setup: i ask theSUBJECT to activate the EXPERIMENTsoftware i the EXPERIMENT accesses the SUBJECTs workstation via VNC (virtual network connection).  Request help from lab assistant  1 Run the Fitts’s Ia vu EXPERIMEN Tjust as the local procedure. EXPERIMENTER spoke to the subject via a microphone, listened to the SUBJECT via a louderspeaker observed the SUBJECT via the video window in the desktop. If problems, concerns, or mistakes come, up either intervene with the SUBJECT directly via VNC or communicate with the remote lab assistant and let him physically intervene with the SUBJECT  4  ,  BIcck2  BlocIci  Block3  4  I  BIock4  4  t4a  EXPERIMENTER filled cut questionnaire form  4 Interview  I  4  (  End  )  91  Still in the remote condition, before the was required to cooperate with his/her  EXPERIMENT  could be started, a  EXPERIMENTER  SuBJEcT  to accomplish the desktop  sharing necessary to ensure that both parties shared the same view of the SuBJEcT’S  the  desktop. Then, the remote  SUBJEcTs  SUBJECTs  followed the same procedure as  in the local condition (Figure 4.11). Under both conditions, after  completing the Fitts’s Law EXPERIMENT, SUBJECTS  and  EXPERIMENTERS were  both required to fill out a questionnaire addressing different communication and interaction issues.  Figure 4.11: Procedure of remote SUBJECT  92  C  Remote SJ8JECT procedure)  Desktop sharing setup: ask the SuDJEcT to activate the experiment softre Te EXPERIMENT accessthe SuBJECT’s orkstation via ‘iNC (‘Artual net’nrkccnnedion).  “p  Gi’ve some information of Fitts’s law  11  Folio Wng the instructions of EXPERIMENTER  Request help from EXPERIMENTER or lab assistant  Dothe EXPERIMENT accordingtothe instruction of EXPERIMENTER speak to the EXPE RIME NTER via a microphone, listen to the EXP ERIMENTER Aa a louder speaker observe the EXP ERIMENTER ‘yia the video v’indowin the desktop. If problems happened either intenened ‘Aa ‘iNC or physically corrected by collocated lab assistant -  -  -  1 SuBJE CT filled out questionnair form  I  I  lntervie  4  End  3 93  In addition,  PARTICIPANTS  ExPERIMENTER  in both conditions who played the role of an  were also interviewed after each  on the conditions, the  PARTICIPANTs  PARTICIPANTs  session. Depending  were asked either two or four questions to  evaluate their experience of being an session, the  ExPERIMENT  ExPERIMENTER.  At the end of a  STuDY  were thanked and financially compensated.  4.7 Summary In Chapter 3, we have listed the design of our Fitts’s Law give a complete summary our  STUDY  EXPERIMENT;  here we  design in terms of variables and measures  (Table 4.4). Table 4.4: Our STUDY design  Independent Variables 1. Collaboration Mode (between-subject) Co-present and Non co-present ---  2. Target Width (withinsubject) 8, 16, 32, and 64 pixels)  Dependent Variables , r, intercept, slope, time spent 2 r on each Fitts’s Law task, and time spent on a Fitts’s Law ExPERIMENT, subjective measures Movement time, number of errors  ---  94  Chapter 5  Results Our hypotheses were (1) that  performance in an  SUBJECTS’  affected by the co-presence of their interaction experience between whether or not the  ExPERIMENTER  ExPERIMENTERs  ExPERIMENTER  and  SUBJECT  and  ExPERIMENT  would be  and also that (2) the  SUBJECTS  would depend on  were co-present. In other words,  we expected that the face-to-face interaction condition would be a richer mode of collaboration, and that the time it took to complete a block of tasks would be significantly less with co-presence (Hauber et al., 2006). Furthermore, we predicted that the amount of attention shift experienced by a depend on whether the  EXPERIMENTER  and  SuBJECT  There were 21 pairs of PARTICIPANTS in the  STUDY;  PARTICIPANT  would  are collocated or not.  each pair undertook 12 Fitts’s  Law tasks for a potential total of 252 Fitts’s Law tasks. One pair of PARTICIPANTs (#4 in the remote condition) was excluded from analysis because a network outage that occurred during set up, leaving 20 valid  PARTICIPANT  pairs with 240  tasks to be analyzed.  Each task had 40 movement trials, all with the same target size and thus the same index of difficulty. Trials for which target acquisition took more than 3000  95  milliseconds (3 seconds) were excluded from the analysis (25 of the 9600 trials were excluded). The average movement time was computed for all remaining trials having the same index of difficulty, yielding four data points for each pair of PARTICiPANTs.  These were used in the analysis.  To test our hypotheses, we looked at both quantitative and qualitative data. Quantitative data was used to compare  SUBJECT’S  performance in the two  conditions and subjective measures of comfort provided by both the the  EXPERIMENTERs.  SUBJECTs  and  Qualitative data obtained from interviews and observations  made during the experimental sessions were used to assess effectiveness. Both quantitative data and subjective measures were analyzed using the SPSS statistical package (version 15.0). Each analysis is discussed in turn.  5.1 Fitts’s Law Performance Our  primary  concern was that the remote condition might fail to adequately  support the experimental process, perhaps due to increased frustration, poorer data cohesion, lengthier  STUDY  PARTICIPANT  sessions, or more problems in  communication. We tested for data congruence between co-present and non co present conditions using two quantitative measures obtained from the Fitts’s Law data collected in the Fitts’s Law tasks: the correlation coefficient,  i  and the  coefficient of determination, r . We used these as measures for the “quality” of 2 the experimental process because they are direct indicators of statistical  96  significance. Moreover, we chose these two measures because Fitts’s Law is a well-studied phenomenon that typically yields high r 2 values for its various formulations, including Welford’s. Preliminary analyses were conducted to discover any violation of the assumptions of normality, linearity, and homoscedasticity of the values of r and r . None were found. Following 2 MacKenzie’s test of correlation and linear regression analysis (MacKenzie, 1991), we computed r and r 2 for each  SUBJECT-EXPERIMENTER pair.  The operational hypothesis that sought to reject (as stated in Chapter 4) was the following.  Hal.1: Statistically, the validity of the Fitts’s Law ExPERIMENT data obtained in  the co-present condition is better than those obtained in the non co-present condition  As we predicted, the values of r and r 2 between the two variables, index of difficulty (ID) and movement time (MT), were consistently high for both  STUDY  conditions (Table 5.1). Table 5.2 shows the means and standard deviations of r and r . In all of the 20 pairs, r was at least +0.780, indicating a strong correlation. 2  97  Table 5.1: r and r 2 values obtained from remote and local conditions Remote  Local  r  r2  r  r2  0.808  0.653  0.899  0.808  0.991  0.982  0.843  0.711  0.941  0.885  0.780  0.608  0.992  0.984  0.999  0.998  0.972  0.945  0.998  0.996  0.954  0.910  0.921  0.848  0.9 13  0.834  0.996  0.992  0.934  0.872  0.989  0.978  0.989  0.978  0.992  0.984  0.996  0.992  0.985  0.970  Table 5.2 Descriptive statistics of r and r 2 Condition  Remote  Local  r  r2  r  r  Mean  0.949  0.904  0.940  0.889  SD  0.057  0.104  0.776  0.139  Measure  2  Across the conditions, the mean values for r were 0.949 and 0.940; the mean values for r 2  were 0.904 and 0.889 in the remote and local conditions  respectively.  98  Figures 5.1 presents a scatter plot of the relation between the ID and MT for all the SUBJECTs’ performance in both remote and local conditions.  Individual  scatter plots (remote and local conditions) which indicate the relation between the ID and MT are listed in Appendix D. Figure 5.1: Scatter plot for remote and local conditions  E I-  ID Remote Subjects  a a 0 0 Q C)  1.010 1.020  1.010 ‘—.  1.020  Local Subjects  0 0  1.030  1.030  1.050  —.1.050  Q  1.060 1.070  1.060 -1.070  0  1 .080 1.090 1.110  C)  1 .080 1.090 1.110 1.120  2.010  2.010 —.....  2.040 2.050 2.060  2.090 2.100  2.040 —.2O5O 2.060  .  2.090 2.100  99  As MacKenzie did to compare the values of r produced by different Fitts’s Law models (1991) we then conducted an independent samples t-test to determine whether the local condition achieved better (stronger) correlations r and  coefficients of determination r 2 than the remote condition.  There was no  significant difference for the values of r computed for the remote or the local condition (t(18)  =  .289, p  =  .776, effect size  112  =  .005),  nor was there any  significant difference for the values of r 2 between the two conditions (t(18) =.257, p =.800, effect size r1 2  =  .004). Non-significant trends were not supportive of our  hypothesis that remote experimentation might negatively affect SUBJECT performance. The high p-values for all of these indicate that we can reject the operational hypothesis Hal .1 because the local condition achieved no better correlations and coefficients of determination than did the remote condition.  5.2 Analysis of Slopes and Intercepts Given the values of correlations, r, and coefficients of determination, r 2 obtained from both conditions, as measured by the statistical significance of the outcomes obtained, did not significantly differ between whether ExPERIMENTER was local or remote, we then analyzed the intercepts, A, and slopes, B , of the correlation 1 equations to see whether there were differences in SUBJECTs’ performance between the Means  two conditions. The values of A 1 and B 1 are listed in Table 5.3.  and standard deviations for slope and intercept are listed in Table 5.4.  100  Table 5.3 Intercept A and slope B (one anomalous value is highlighted) Remote  Local  A,  B,  A  B  205.300  583.680  126.880  788.290  232.260  308.350  177.710  835.580  173.790  615.670  23.360  1211.730  174.750  501.290  281.640  472.810  233.870  519.990  226.380  452.440  131.170  530.290  163.260  890.060  220.580  635.750  234.130  442.190  207.100  525.960  264.880  503.780  233.910  577.890  142.460  440.710  174.480  609.690  179.560  654.810  Table 5.4 Descriptive statistics for slope and intercepts Condition  Measure  Remote  Local  A,  B,  A  B  Mean  182.025  540.858  198.719  669.237  SD  75.740  93.653  34.217  258.589  5.2.1 Intercept coefficient A• The A, intercept values obtained from the remote and the local conditions were all close to MacKenzie’s reported value of 230 (MacKenzie, 1992) except for one  101  pair in the local condition (the third pair in Table 5.3). The mean A 1 for the remote condition was 182.025, and the mean A, for the local condition was 198.7 19.  Preliminary analyses were conducted to discover any violation of the assumptions of normality and equal variance of the A, values. None were found. A parametric independent t-test was run to compare the A, values obtained from the remote and local conditions. There was no significant difference in A values between local studies and remote studies (t(18) difference of the A, means was small  =  .635, p  (2 =  =  .533). The magnitude of the  .020).  5.2.2 Slope coefficient B The B, values obtained from the remote and the local conditions were larger than MacKenzie’s reported value of 166 (MacKenzie, 1991). It is possible that this was due to the shape of the targets used in the Fitts’s task as MacKenzie used rectangular targets and our STUDY used circular targets. The mean B for the remote condition was 540.858, and for the local condition it was 669.237.  Preliminary analyses were conducted to discover any violation of the assumptions of normality and equal variance of the A- values. None were found. A parametric independent t-test was conducted. There was no significant difference in B values for local studies and remote studies (t(11.321)  =  of the difference in the mean was moderate (ri 2  —1.476, p  =  .167). The magnitude  .100).  102  5.2.3 Performance time A similar analysis was conducted for performance time, which is modeled by the intercept and slope coefficients. As would be expected, no significant differences were found between the local and remote conditions, and the mean trial times were similar to what Mackenzie reported (1991).  This suggests that even though the apparatus used at UBC and UQAM were not the same, this did not significantly affect the performance time of SUBJECTS or the device (mouse) constants, despite the moderate effect size for the larger slopes in the local condition. It may be that the Fitts’s Law STUDY is not very sensitive to minor apparatus differences; and thus any differences between local and remote apparatus would not significantly affect the results.  Based on the results of  sections 5.2.1 and 5.2.2, regardless of co-presence or non co-presence, the data obtained in the co-present condition is similar to than that obtained in the non co present condition.  Thus, we can thus reject the first two of the operational  hypotheses, Hal.1 and Hal.2.  5.3 Errors To determine whether the second part of the operational hypotheses Hal.1, that performance would be more error-prone, we need to look at errors. An error in a trial was defined as pointing and clicking outside the target. As known in Table 5.5, the error rates for both local and remote conditions were more than double  103  than the recommended rating of 4% (MacKenzie & Buxton., 1992), with means of 10.208 % and 9.542 % respectively. The error rate was analyzed using an independent samples t-test.  The statistical difference between the remote and  local error rate was not significant small effect). This suggests that  (t  (18)  =  SUBJECTS  .226, p  =  .824, effect size  2 =  .003  in the remote conditions produced no  more errors than those in the local conditions. Table 5.5 Descriptive statistics of error rate in remote and local conditions  Mean SD  Remote 10.208 % 6.52 1 %  Local 9.542 % 6.668 %  Based on the results of section 5.2 and this section, we can fully reject the operational hypothesis (Hal .2) that performance of local better (faster and less error-prone) than that of remote ExPERIMENT  SUBJEcTs  SUBJEcTS  would be  in the Fitts’s Law  is rejected. Furthermore, the results suggest that the co-presence of  EXPERIMENTERs  does not necessarily improve  SuBJECTs’  performance in a Fitts’s  Law ExPERIMENT.  5.4 Analysis of time to run a Fitts’s Law ExPERIMENT Although the outcome indicated there was no difference in the quality of the experimental data, as measured by the degree of correlation found in the data, or in the physical performance of  SUBJECTs,  as measured by the intercepts and  slopes of the correlation equations between the local and remote conditions under  104  a Fitts’s Law  STUDY,  it took to run the  we realized that there might still be a difference in how long  EXPERIMENT.  To determine whether there were such time  differences between local and remote conditions, we first examined the total time it took to conduct each of the four blocks of tasks at the four levels of difficulty, taking into account that higher indices of difficulty required longer trial times. We then examined the total amount of time (session time) taken to complete the ExPERIMENT  in the remote and local conditions.  5.4.1 Total time spent at each task block at each index of difficulty As mentioned above, we first look at  SUBJECTs’  total performance time on  completing each of the four task blocks at each index of difficulty in the remote and local conditions. Table 5.6 summarizes the performance time of each task block for both remote and local  SUBJECTs.  The descriptive statistics of the data  are listed in Table 5.7.  105  Table 5.6: Mixed between and within STUDY: condition by index of difficulty  .  .  Condition  ID1 (seconds)  1D2 (seconds)  1D3 (seconds)  1D4 (seconds)  100.750  114.578  173.734  170.906  70.562  98.172  122.141  158.406  106.797  115.547  140.515  196.672  89.000  200.703  157.609  167.000  99.953  137.140  205.828  197.984  76.234  104.781  73.641  91.907  116.922  133.718  145.828  200.828  72.156  101.688  131.094  189.875  113.906  262.235  149.422  200.657  119.031  155.625  146.719  175.375  113.187  133.797  141.297  189.375  135.094  151.062  156.140  238.140  158.391  205.141  161.609  184.375  100.437  133.031  177.547  225.610  95.063  114.797  146.406  190.234  127.328  164.953  165.110  230.516  185.704  123.219  296.313  282.047  105.125  143.468  161.703  223.922  84.468  118.718  114.781  143.969  80.563  85.453  113.188  126.250  Remote  Local  106  Table 5.7 Descriptive statistics of performance time between four different indices of difficulty and two STUDY conditions Condition  Remote  Measure  ID1(seconds)  1D2(seconds)  1D3(seconds)  1D4(seconds)  Mean  96.531  142.418  144.653  174.961  SD  18.547  52.240  34.213  33.007  Condition  Local  Measure  ID1(seconds)  1D2(seconds)  1D3(seconds)  1D4(seconds)  Mean  118.536  137.363  163.409  203.443  SD  33.567  32.254  51.196  20.343  5.4.1.1 Within-subjects effect The means and standard deviations of performance time for each index of difficulty are presented in Table 5.8. There was a significant main effect for total point-and-click time among four different indices of difficulty: ID1, 1D2, 1D3, 1D4 (F (3,54)  =  48.249, p  <  .001, r  =  .900). This confirmed the Fitts’s Law  hypothesis that as target sizes increase, the time taken to acquire a target decreases. Table 5.8 Descriptive statistics of total time spent in each task block corresponding to each index of difficulty. ID1(seconds)  1D2(seconds)  1D3(seconds)  1D4(seconds)  Mean  107.533  139.891  154.031  189.202  SD  28.707  42.334  43.458  41.750  107  5.4.1.2 Between-subjects effect  The means and standard deviations for each STUDY condition are presented in Table 5.9. There was no significant main effect in the target acquisition time between remote and local conditions (F (1,18)  =  1.382 p  =  .255,  ii2  =  .071). The  magnitude of the effect size was moderate. This indicates that although data did not demonstrate any statistical significant difference, in practical terms, it took SuBJECTs  moderately more time on average to complete all twelve tasks in the  local condition than the same tasks in the remote condition, as shown in Figure 5.2 Table 5.9 Descriptive statistics of total time spent to complete the all of the twelve tasks  Remote (seconds)  Local (seconds)  Mean  139.641  155.683  SD  75.863  58.693  5.4.1.3 Interaction  According to the operational hypothesis Hal .3, we predicted that there might be an interaction between how long it took to run the block of point-and-click tasks for each index of difficulty and for each STUDY condition. A mixed between within subjects ANOVA was conducted to analyze the total amount of time SUBJECTS  spent in each of the four blocks between remote and local conditions.  108  Preliminary analysis for within-subject conditions (four indices of difficulty) and between-subject conditions (remote and local STUDY condition) was done to determine if there was any violation of the normality, equal variance, and sphericity assumption. Sphericity was not satisfied. We therefore report the multivariate test results rather than an ANOVA as recommended by statistical textbooks. There was no significant interaction between the indices of difficulty and the STUDY conditions (F (3,54)  =  1.193, p  =  .034, Wilks’ Lambda = .8 17).  Figure 5.2 Total time spent to complete the all of the twelve tasks Estimated Marginal Means of MEASURE_I condition  220000.0000  200000.0000C 180000.0000-  .  160000.0000-  I  140000.0000-  120000.0000-  1 00000.0000-  80000.0000-  ID Remote  Local  109  5.4.2 Total time to complete the ExPERIMENT in the remote and local conditions An independent samples t-test was conducted to compare the total amount of time taken to conduct the twelve point-and-click tasks in remote and local conditions. Descriptive statistics of the data are shown in Table 5.10. Preliminary analysis was performaned to check normality and equal variance. None was found. There was no statistically significant difference in total task completion time between the remote condition (mean (mean =747.741, SD  =  effect is moderate (r 2 time to complete the  =  640.261, SD  154.010, F(18) =  =  =  150.51) and the local condition  —1.494, p  =  .153). The magnitude of the  .120). This suggests that local  ExPERiMENT  than remote  SUBJECTS  SUBJECTS  did take more  did.  Table 5.10 Descriptive statistics of total time spent in remote and local EXPERIMENTS  Remote (seconds)  Local (seconds)  Mean  640.261  747.741  SD  150.51  154.010  110  5.5 Questionnaire analysis 5.5.1 Interaction Survey A quantitative analysis was made of the subjective measures collected by questionnaire (Appendix A) of PARTICIPANTS’ experience of the two roles in the remote and local conditions. For each question, PARTICIPANTS are asked to rank their experience from 1 to 5, with 1 means strongly agree and 5 means strongly disagree. The survey results were analyzed using SPSS (version 15.0). Each of the questions measured a single aspect. The descriptive statistics were calculated for each question separately, as shown in E.l in Appendix E. In this section, we discuss only questions that presented a statistical or practical difference. Differences were looked for between SUBJECTs and ExPERIMENTERs within a condition, as well as across conditions. Each STUDY session was designed to accommodate one EXPERIMENTER and one SUBJECT  as they were interacting with  one another. It was predicted that they would share similar experiences of a STUDY and  thus form a comparable pair. The pair-wise responses from the  PARTICIPANTs can be found in Table  E.2 Appendix E.  The EXPERIMENTERs in the local and remote conditions formed two independent groups, as did the SUBJECTs. A mixed between-within subjects ANOVA should have been applied to the questions answered by EXPERIMENTERS and SUBJECTS in both conditions; however, none of the results of these questions satisfied the assumptions of normality or equal variance. Because of this, instead of  111  conducting an ANOVA, a paired-sample t-test was conducted for each of the twelve questions asked of both SuBJEcTs and EXPERIMENTERS. PARTiCIPANTs playing the same role in the remote and the local conditions had their survey results compared by two-sample independent t-tests, listed in E.3 in Appendix E. For remote EXPERIMENTER-SUBJECT pairs, the survey results were analyzed by  paired sample t-tests for matched-up questions. The same applies to local STUDY pairs. The results are shown in table E.3.l and E.3.2 Appendix E.  5.5.1.1 Measures between conditions based on roles Only Questions 3 and 8 showed any significant differences related to our hypotheses. Further analysis shows that there is a large practical difference between the responses of ExPERIMENTERs in Question 3 and 8.  Question 3: “It was easy to talk to and hear the SuBJEcT/ExPERIMENTER  “.  PARTICIPANTS answered on a scale of 1-5, with 1 being a high (good) ranking, and  5 being a low (bad) ranking. This question was ranked low by remote EXPERIMENTERS, with a mean of 4.6, while local EXPERIMENTERS ranked the same question high, with a mean of 1.6.  There is a significant and practical difference between the remote and local EXPERIMENTERS (Z (18)  =  -2.006, p  =  .045,  92 =  .180 (large effect).  112  Question 3 ranks the ease of talking to and hearing the other party in the EXPERIMENT.  As expected, the results show that there is a significant difference  on the EXPERIMENTERS’  side. In the local condition, PARTICIPANTs communicated  with each other face-to-face, as oral communication was not obstructed by any factor. In the remote condition, PARTICIPANTS communicated with each other via microphones. Although the audio equipment was inspected beforehand to ensure that both parties could hear each other clearly, the results of Question 3 revealed that UBC PARTICIPANTs still experienced difficulty when listening to their remote party. As an observation, one factor that may have contributed to this is that some UQAM SUBJECTS spoke English with a French accent, which might have posed some communication barriers to the UBC PARTICIPANTS.  While this might Suggest that co-presence is a richer interaction condition than the non co-presence, the confound of some remote SuBJECTs speaking with an accent means we cannot draw any definitive conclusions from these results.  Question 8: “Overall, from my own point of view, how well did the  EXPERiMENT  go?” PART1CIPANT5 again answered on a scale of ranking, and  1-5, with 1 being a high (good)  5 being a low (bad) ranking. The rankings for this question by both  remote and local EXPERIMENTERS was EXPERIMENTERS  high, with mean of 1.9 for remote  and 1.5 for local EXPERIMENTERs. There is a larger practical  113  difference for remote and local effect size  =  EXPERIMENTERS  (Z (18)  =  -1.523, p  =  .128, and  .130 (large effect).  Question 8 inquires about the overall satisfaction of the EXPERIMENTERs  ExPERIMENT.  Local  ranked this question with a higher score than the remote ones.  This suggests that  EXPERIMENTERs  experienced more satisfaction in the local  condition. This suggests that the richness of the face-to-face condition enables more interaction and that it leads to higher participation satisfaction than the remote condition.  5.5.1.2  PARTICIPANTS’  experience within each condition  Among the pair-wise measurements respectively in the remote and the local conditions, only Question 6 showed any significant differences related to our hypotheses. The analysis of Question 6 shows a large practical difference (though not a statistically significant difference).  Question 6: “I was aware ofmy EXPERIMENTER /SUBJECT’s presence all the time.” In the remote condition, the ranking of this question by  EXPERIMENTERs was  equal to that of  EXPERIMENTERs and  SUBJECTs,  with a mean of 2.1 for both  SuBJECTs. In the local condition, than their  SUBJECTS,  EXPERIMENTERs  ranked this question higher  with a mean of 1.4 for EXPERIMENTERs and 2.6 for  There was a larger practical difference between the local  SUBJECTS.  EXPERIMENTERs  and  114  SUBJECTS  (Z (9)  =  -1.897, p  =  .058, effect Size  =  .280 (large effect), which was  almost statistically significant at the pc05 level.  Question 6 measures the awareness of the presence of the other party. Local EXPERIMENTERS  indicated that they were more aware of their  than were remote SUBJECTS  EXPERIMENTERS.  did lead to  SUBJECTs’  presence  This suggests that the physical presence of  EXPERIMENTERS being  more aware of them.  In summary, the results of these questions confirm one of our initial hypotheses (Ha ) 2 : that the PARTICIPANTS did experience richer human-to-human interaction in  the co-present or face-to-face condition than in the remote condition.  5.6 Interview analysis Qualitative analysis was also performed on the feedback collected from postSTUDY  interview about  PARTICIPANTs’  experience in the remote and local  conditions. Not only did we discover important facts that test our hypotheses, but also other interesting factors about the multimedia aids used to support our STUDY.  Question 1: “How was your experience ofbeing an ExPERIMENTER”? Question 2: “What kind of problems did you experience when interacting with your SUBJECT?”  115  Interview question 1 and 2 ask about EXPERIMENTERs’ rnteraction experience. From these questions, we typically discovered evidence that refutes our third hypothesis  (Ha3). EXPERIMENTERs’ responses to this question varied. Regardless  of the video, audio, and EXPERIMENTER ExPERIMENTER. found  that  would  VNC issues, our pre-STuDY assumption was that a local experience  less  attention  shift  than  a  remote  However, when interviewing the local EXPERIMENTERS, we  6 out of 10 EXPERIMENTERs commented that being a local  EXPERIMENTER  was  EXPERIMENTERS, only  very  stressful.  When  interviewing  the  remote  1 out of 10 EXPERIMENTERs made the same comment.  During the interviewS, local EXPERIMENTERS explained the stressful aspects  of  their experience. The EXPERIMENTERS complained that it was hard for them to  read the instructions and look at their SUBJECTS at the same time. They often commented that when delivering instructions, they had to interpret the facial expression of the SUBJECTs as it provided them with the feedback of instruction comprehension.  However, after checking their SUBJECT’S facial expression, it  was hard for them to go back to their instruction sheet and continue from where they left off. This suggests that local EXPERIMENTERs have to split their attention between two tasks at the same time: one was delivering the instructions by paying attention to the instruction sheet, and the other was interpreting their SUBJECTs’ facial expression.  Somehow, the remote condition allowed a virtual distance  between PARTICIPANTS and reduced the amount of attention shift between  116  EXPERIMENTERS was  and  SUBJECTs  —  one remote  EXPERIMENTER  commented that it  easier to play the role of an ExPERIMENTER.  Question 3: “Between audio and video, what media served a more important role in communication during the ExPERiMENT?” (For remote EXPERIMENTERs) Question 3 asks about the importance of audio versus video. In contrast to pre STUDY  assumptions, most  PARTICIPANTs  medium than video. PostPARTICIPANTs  playing the  STuDY  treated audio as a much more important  interviews showed that 9 out of 10 remote  EXPERIMENTER’s  role found the audio channel more  expressive than the video, and that information about interaction flowed in a more understandable way via audio.  Question 4: “What was the most challenging part of setup?” (For remote EXPERIMENTERS)  PARTICIPANTs  came up with the following responses:  Loading the interface  PARTICIPANTs  complained about their frustration of loading up LORIT’s interface  on UBC’s workstations.  PARTICIPANTs  explained that they had to go through three  extra steps before getting access to the website. After entering the website address in a web browser, besides providing a user name and a password for the login dialog box,  PARTICIPANTs  had to install the ActiveX component, cancel a  117  Microsoft NetMeeting invitation, and click twice on a green button before the interface showed up on screen. In addition, the web interface was only supported by Microsoft Internet Explorer version 6.0 or higher, which mean the interface cannot be loaded by Mozilla Firefox. There were several cases when PARTICIPANTS  attempted to load the web interface on Mozilla Firefox and failed.  5.7 Summary The results of our  STuDY  provide useful information about the use of remote  experimentation to support research on tasks similar to Fitts’s Law. Table 5.11 summaries our results by hypotheses.  118  Table 5.11: Summary of results Hypothesis  Expected Results  Actual Results  Hal .1  Average r 2 measured in the local condition is better than Average r 2 measured in the remote  Average r 2 measured in the local condition is NO better than Average r 2 measured in the remote  Hal.2  Performance of local SUBJECTS Performance of local SUBJECTs is better (faster and less error- is NO better (faster and less prone) than that of remote error-prone) than that of remote SUBJECTs in the Fitts’s Law SUBJECTs in the Fitts’s Law  Hal.3  ExPERIMENT  EXPERIMENT  Local SUBJEcTs spend less time to conduct all the tasks in each task block and overall they spend less time to complete all the Fitts’s Law tasks than that of remote SUBJECTS in the Fitts’s Law  Local and remote SUBJECTs spent equal amount of time to conduct all the tasks in each task block and to complete all the Fitts’s Law tasks  EXPERiMENT.  2 Ha  Interaction experience is richer Interaction experience was in the co-present condition than richer in the co-present the non co-present condition. condition than in the non co present condition.  Ha3  Local EXPERIMENTERS would Local EXPERIMENTERs experience a less attention shift experienced more attention shift than remote than the remote EXPERIMENTERs the EXPERIMENTERS  We believe that many of the experiments that are conducted by doctoral students by to evaluate educational technology fall into this category. The next chapter discusses some interesting interaction patterns observed from our  STuDY  and in  Chapter 7 we draw a number of conclusions and recommendations that arise from our research.  119  Chapter 6  Discussion In this chapter, we discuss some interesting interaction patterns drawn from the real STUDY.  Patterns of Interaction There are several patterns of interaction observed in remote and local conditions, and they are listed in Table 6.1. Table 6.1 Patterns of interaction Patterns of Interaction  Local  Remote  SUBJECT <-> ExPERIMENTER  observed  observed  ExPERIMENTER <->  EXPERIMENTER <->  UBC lab assistant  UQAM lab assistant  EXPERIMENTER <->  lab  assistant  EXPERIMENTER <->  UBC lab assistant SUBJECT <->  lab assistant  UBC Lab assistant <-> UQAM lab assistant  SUBJEcT <->  UBC lab  SuBJECT <->  UQAM  assistant  lab assistant  N/A  UBC lab assistant <-> UQAM lab assistant  1). Interaction between ExPERiMENTER and lab assistants  120  As predicted, remote  STUDY  sessions suffered from large overheads for  videoconference set-up and media sharing over the network. Thus, more interaction was observed in remote sessions than local sessions. In the training phase of a remote Fitts’s  STUDY  EXPERIMENT  session, the UQAM lab assistant walked through the  with a UBC  UBC lab assistant assisted a  PARTICIPANT.  PARTICIPANT  In the camera-setting phase, the  in loading and opening the remote lab  web interface and responding to any related questions. These questions ranged from lab equipment configuration, to activation status of equipment, and identifying the widgets that could be manipulated. When moving on to the actual EXPERIMENT  with  phase, both UQAM and UBC lab assistants intervened or interacted  EXPERIMENTERs  Compared  to  the  based on the context and occurrence of problems. remote  condition,  local  sessions  required  neither  videoconference setup nor camera setting. As in a remote session, a local ExPERIMENTER  needed to go through a training phase with a lab assistant.  Observation showed that local  EXPERIMENTERs  lab assistant for  related concerns in a minor way.  ExPERIMENT  EXPERIMENT  phase, a local  ExPERIMENTER  only communicated with the UBC In the actual  only interacted with the local lab  assistant.  2). Interaction between SUBJECTS and ExPERIMENTERs In terms of interaction between  SuBJECTs and EXPER1MENTER5,  remote sessions  resulted in more pair-wise interaction then the local. As mentioned previously,  121  audio served a more important role than video. Thus, poor audio directly encouraged  EXPERIMENTERS  and  SUBJECTs  to proactively communicate with each  other to ensure a correct mutual understanding. Communication forms included asking a remote party to repeat an instruction, multiple checking of certain progress checkpoints, and reminding the obedience of the interaction protocol. Unlike remote sessions, local sessions did not require a dedicated audio channel. Thus, participatory pairs did not need as much interaction as the remote pairs, except for inquiring about what the  SUBJECTs  thought about the  ExPERIMENT.  3). Interaction between SUBJECTS and lab assistants co-located with their lab assistants in both conditions.  SUBJECTs were  remote  STUDY,  when problems or errors occurred, instead of directly asking their  EXPERIMENTERs, SUBJECTS responses.  In the  Co-location  preferred to approach their co-located lab assistant for  did provide  PARTICIPANTs  with the assurance of being  able to receive instant feedback from a knowledgeable individual. This indicates that the presence of another person encouraged interpersonal communication. In the local arose, them.  STuDY,  SUBJECTs  this interaction pattern did not manifest frequently. When issues waited for their  However,  if  SUBJECTs  EXPERIMENTERs  sensed  their  to tell them how to deal with EXPERIMENTERs  were  not  knowledgeable about the answer to a question or not aware of certain mistakes and a lab assistant had to interfere in some urgent context, then the interaction pattern showed up. If this situation occurred,  EXPERIMENTERs  would lose their  122  credibility of being a knowledgeable partner. Consequently,  SUBJECTS  preferred  to turn to the lab assistant for any concerns or questions thereafter. We call this situation a role shift.  4). Interaction between two lab assistants This pattern only applied to the remote condition, in which lab assistants had to communicate with each other to check videoconference setup, including video and audio channel configuration. They also needed to communicate with one another to ensure that the STuDY  PARTIcIPANTs  were making progressing correctly in a  session.  123  Chapter 7  Conclusions and Future Work 7.1 Conclusions We conducted an  EXPERIMENT  Based on the results of our  STUDY,  performance, co-presence of SUBJECTs’  under both co-present and distributed conditions. we conclude that in terms of experimental  ExPERIMENTERs  does not necessarily affect  performance. In fact, SuBJEcTs’ performance did not show any  statistically significant differences under either co-present or non co-present conditions. However, local EXPERIMENT  than remote  SUBJECTs  SUBJECTs,  practically spent more time doing the confirming Hauber et al.’s finding that  “adding spatiality is capable of creating a collaborative context that is closer to face-to-face, but at the same time loses the efficiency of a task-focused twodimensional interface.” (2006, p. 420). In terms of interaction, the interpersonal experience differed between  ExPERiMENTERS  and  SUBJEcTs  only in certain  aspects, indicating that co-presence is richer but not necessarily more pleasant than non co-presence. In fact, those  PARTICIPANTS  acting as  EXPERIMENTERs  in  the face-to-face condition experienced higher pressure.  124  When building our  STUDY,  we looked for ways to bring together commercial  off-the-shelf software and available hardware and combine them to form a research infrastructure. Working with our research partners at UQAM, we lifted that environment to the point of supporting remote collaboration. Our design did not strive to provide our remote  PARTICIPANTs  STuDY  with an illusory face-to-  face experience, as such a focus would have missed the substantive point of promoting inventive forms of remote experimentation (Birnholtz & Horn, 2007). Instead, we designed our  STuDY  to make our  PARTICIPANTs  remote experience while still fulfilling the goal of our successfully conducted our  STUDY  STuDY.  conscious of the  Given that we have  in a distributed condition, this confirms that  traditional research experiments can indeed be conducted in a remote way when researchers are not co-located, or when equipment access is restricted due to geographical separation or facility shortcomings.  7.2 Lessons learned from our STUDY Our  STUDY  revealed that the  PARTICIPANTS playing  the  EXPERIMENTER’s role  had  not acquired adequate knowledge of their tasks and of the procedure of the Fitts’s Law  EXPERIMENT;  PARTICIPANTs’  feedback revealing the importance of  comprehensive familiarity with the experimental materials.  More thorough  training is essential; this includes practicing the experimental procedure many times until one clearly understands the expectations for procedure and progress. Furthermore, it would be worthwhile to run through the experimental instructions  125  several times and then encourage the ExPERIMENTERs to deliver them in their own words  —  in other words, more time should have been allocated to training.  We predicted initially that capturing a clear and proper view of PARTICIPANTS  ifl  the remote STUDY was most important, and that audio would serve only as an auxiliary channel of communication. Our qualitative evaluation provided us with the completely opposite result; it was evident that audio was a more expressive and important medium than video, and that a high-quality audio channel was needed far more than video.  Although the camera view of Fitts’s Law  ExPERIMENT,  PARTICIPANTs  STUDY  partners did not affect the results of our  we consider that it is still worthwhile to provide  with a clear view of their remote party. Since there are three auto-  controlled cameras in LORIT, we believe that letting preferred camera, could affect the remote  In addition, during the  STUDY,  PARTICIPANTs  PARTICIPANTS’  choose their  interaction experience.  a screen-saver repeatedly appeared on the  secondary workstation at UBC, introducing a degree of interference. Each time we woke up the  ExPERIMENT  Station from the screen-saver, video windows were  frozen and had to be re-activated and re-sized to ensure a proper view of the remote party. Thus, turning off the screen saver is an essential step in the remote condition.  126  7.3 Contribution This work has contributed to the emerging field of online learning and research in the following ways: •  We formally evaluated the relative efficacy of conducting an in both co-present and non-co-present conditions. Although  EXPERIMENT  PARTICIPANTs  remained conscious of the geographic distance, the results are still encouraging. Our experimental outcome shows that a non-interactive ExPERIMENT  can still be conducted and achieve expected results, even in  the absence of an  ExPERIMENTER.  However, this requires appropriate  cooperation and communication between the human parties and proper support from high-quality software and hardware.  •  We used an experimental research lab, LORIT, and its web interface, Econtrol, to conduct our  STuDY  in the non-co-present condition. By  making use of LORIT, we evaluated its usability in a genuinely distributed experimental context as LORIT is a technological module designed to support doctoral students to conduct educational research in an e-learning environment.  •  In the remote condition, we used the lightpath network connection service to support the video-conferencing software, and we evaluated the performance of Lightpath by running multi-media applications (Virtual network computing and Access Grid 3.0) and our own experimental  127  software (developed in Java) in parallel.  With Lightpath’s high  performance, we were able to conduct our remote  EXPERIMENT  in real  time. Thus, the lightpath network’s dedicated, secure, and confidential service was critical to the success of our STuDY.  7.4 Challenges We encountered some challenges when preparing our remote  The most  STUDY.  challenging factor is that the two cities are in different time zones; Vancouver is three hours behind Montreal. This temporal difference imposed some complexity in scheduling  PARTICIPANTS  and booking lab facilities at both UBC and UQAM.  Moreover, the availability of LORIT also changes from day to day. Due to these obstacles, we could only arrange a maximum of two remote  sTuDies  per weekday,  since the setup for remote studies takes longer than that for local studies.  7.5 Future work The problems in our our  STUDY  STUDY  leave room for future research, while the results of  indicate a number of areas that we need to improve.  7.5.1 STUDY  design  We chose a simple and robust classical  EXPERIMENT,  the Fitts’s Law  verify our research hypothesis. The next version of the  STuDY  STUDY,  to  could undertake a  more complex and sophisticated research experiment. Also, on this occasion, we  128  only accommodated one SUBJECT in each STuDY session. In reality, some research experiments are designed to accommodate multiple SUBJECTs and observe the behavior among them. Therefore, further research could be performed using that perspective.  7.5.2 Use of Chat, Video, and Audio Although text messaging or chat was completely ignored in our STUDY, there is potential in extending the research into the text-messaging medium, as some research has suggested that chat is a non-obtrusive channel compared to audio, and that it protects people’s privacy under open-context conditions (Scholl, McCarthy, & Harr, 2006). In our case, if the population of each SUBJECT group increases, some other communication issues may emerge, which might require an additional channel, such as chat, to broaden the range of communication among several PARTICIPANTs.  7.5.3 Configuration of Equipment In our STUDY, the UBC venue did not have an equipment configuration symmetrical to that of LORIT. Consequently, the SUBJECTs at LORIT could not adjust the UBC camera in the same way that the UBC ExPERIMENTERs could adjust the LORIT camera. If our STUDY were to be extended, it would beneficial to equip the UBC lab with a fixed and auto-controllable camera, so that UQAM  129  PARTICIPANTS  can have freedom of camera adjustment equal to that possible from  UBC to LORIT.  Our remote  STuDY  revealed that audio quality was satisfactory in only one  direction. The audio quality from LORIT to UBC was barely average. Added to the issue of the remote UBC  PARTICIPANTs.  PARTICIPANTs’  accents, audio posed some challenge to  The next version of the  STuDY  preparation should pay more  attention to tweaking the audio quality for both parties. It would be beneficial to use a microphone directly attached to a  PARTICIPANT,  or even a microphone  station allowing flexible position configuration.  7.5.4 Possible modification of remote lab web interface Our  STUDY took advantage of  the remote lab, LORIT, and its web interface for  remotely controlling its devices and equipment. However, several unexpected problems were also caused by the interface. One problem was a mismatch of people’s mental model of the icon symbolizing the exit from LORIT. One PARTICIPANT  accidentally shut down the entire remote lab by clicking this icon.  Modifications should certainly be made to this interface component  —  Another  problem was related to French labels on the interface. Although our UBC PARTICIPANTs  were able to manage to use the Econtrol interface despite its French  labels, it might still have posed a certain amount of difficulty to  PARTICIPANTs  130  who could not understand French. One suggestion would be to have an alternative Econtrol interface version with English labels.  131  Bibliography (2007). Netmeeting for Beginners, Retrieved November 4, 2007, from http://www .ncsu.edu/it/multimedialnetmeeting!netmeeting.html  —  Accot, J. & Zhai, S. (1997). Beyond Fitts’ Law: Models for trajectory-based HCI tasks. In Proceedings ofthe SJGCHJ Conference on Human Factors in Computing Systems. (CHI ‘97), (pp. 295 302). New York, NY: ACM Press. —  Abbot, Costello, (1939), Who is on first?. Retrieved February 22, 2008, from www.youtube.comlwatch?v=sShMA85pv8M Berry, L. (2005). Role-Based Control of Shared Application Views. (Master dissertation, the University of British Columbia, 2005). Retrieve from http ://www.cs.ubc.caJgrads/resources/thesis/NovO5/ Birnholtz, J.P., & Horn, D.B. (2007). Shake, rattle and roles: Lessons from experimental earthquake engineering for incorporating remote users in large-scale e-science experiments., Journal of Computer-Mediated Communication, 12(2), article 17. Retrieved February 23, 2008, from http://icmc.indiana.edu/voli2/issue2/birnholtz. html Bourdeau, J., Henri, F., Dufresne, A., Tchetagni, J., & Ben Ali, R. (2006). Design of a doctoral training environment: five challenges. Retrieved February 2, 2008, from http://www.lornet.org/Portals/i O/12L0R06/22_Design%20oJ%20a%2ODoctoral %2OTraining-%2OFive%2OCahallenges.pdf Bourdeau, J., Dufresne, A., Prom Tep, S., & Turcotte, S. (2005). Test et validation des environnements de Téléapprentissage, in Pierre, S. (Ed.), Développement, integration et evaluation des technologies de formation et d’apprentissage, chap. 16. Montréal : Les Presses internationales de Polytechnique, 479-499. English version: A Scientific Instrument for the Engineering of Telelearning Environments. Chen, W., Towles, H., Nyland, L., Welch, G., & Fuchs, H. (2000). Toward a compelling sensation of telepresence: Demonstrating a portal to a distant (Static) office. In Proceedings ofthe 11th IEEE Visualization 2000 Conference (VIS 2000) (October 08 13, 2000). VISUALIZATION. IEEE Computer Society, Washington, DC -  132  Coventry, L. (n.d.) Video conferencing in higher education. Retrieve February 20, 2008, from http ://www.agocg.ac.uk/reports/mmedia/video3/video3 .pdf Fitts, M. P. (1954). The information capacity of the human motor system in controlling the amplitude of movement. Journal ofExperimental Psychology (pp. 262-269). (Reprinted from Journal of Experimental Psychology, 47, 381-391, 1954). Retrieved February 20, 2008, from http://www.cs.princeton.edu/courses/archive/fallO7/cos436/FittsJEP 1 954.pdf Foss, B. A., Eikaas, T. I., & Hovd, M. (2000). Merging physical experiments back into the learning arena. Proceedings ofthe American Control Conference (pp. 2944-2948). Chicago: AACC Press. Gu, R., Zhu, M. L., Dong, Y. B., Shi, D.D., & Wang, Y. G., (2005, July). A case study of virtual circuit laboratoryfor undergraduate student courses. Paper presented at ITHET 6th Annual International Conference, Juan Dolio, Dominican Republic, (T2B-21-T2B-24). Retrieved February 20, 2007, from IEEE Explore digital library. Hasan, S. M., Lewis, G.J., Alexandrov, V. N., Dove, M.T., & Tucker, M.G. (2007). Multicast application sharing tool for the Access Grid toolkit., Retrieved November 4, 2007, from http://www.allhands.org.uk/2005/proceedings/papers/493.pdf Hauber, J., Regenbrecht, H., Billinghurst, M., & Cockburn, A. (2006). Spatiality in videoconferencing: Trade-offs between efficiency and social presence. Proceedings ofthe 2004 ACM conference on Computer Supported Collaborative Work, (pp. 4 13-422). New York, NY: ACM Press. Heeler, P., & Hardy, C. (2005). A preliminary report on the use of video technology in online courses. Journal of Computing Sciences in Colleges, 20(4), 127-133. Huck, S. W. (2004). Reading statistics and research. Boston: Pearson Education Inc. Hollan, J., & Stometta, S. (1992). Beyond being there. In Proceedings of the ACM Conference on Human Factors in Computing Systems. (pp. 119-125). New York, NY, USA: ACM Press. Kauff, P. & Schreer, 0. (2002). An immersive 3D video-conferencing system using shared virtual team user environments. In Proceedings of the 4th international conference on Collaborative virtual environments (CVE 2002) (pp. 105-116). New York, NY, USA: ACM Press.  133  Li, Y. Y., Esche, S., & Chassapis, C. (2007). AC 2007-2564: An architecture for real-time remote laboratories. Retrieved February 20, 2008, from http ://www. icee.usm.edu/icee/conferences/asee2007/papers/2564ANARCHITE CTURE_FOR_REAL_TIME_REMOTE_LAB.pdf MacKenzie, I. S. (1991). Fitts’ Law as a Performance Model in Human-Computer Interaction. (Doctoral applied dissertation, the University of York, 1991). Retrieve from http://www.yorku. ca/mack/phd MacKenzie, I. S. (1992) Movement time prediction in human-computer interfaces. Readings in human-computer interaction (211 ed.) (pp. 483-493). Retrieved February 20, 2008, from http ://www.yorku.ca/mack/G192 .html MacKenzie, I. S., & Buxton, W. (1992). Extending Fitts’s law to two-dimensional tasks. Proceedings of the ACM Conference on Human Factors in Computing Systems CHI ‘92, (pp. 2 19-226). New York, NY, USA: ACM Press. -  Matsukura, R., Koyamada, K., Tan, Y., Karube, Y., & Morlya, M. (2004). VizGrid: Collaborative visualization grid environment for natural interaction between remote researchers. The Fujitsu Scientific and Technical Journal, 40(2), 205-2 16. Retrieved September 8, 2007, from http://wwwfujitsu.com/downloads/MA G/vol4O-2/paper04.pdf Moløkken-østvold, K. (2005). Ethical concerns when increasing realism in controlled experiments with industrial participants. Retrieved from IEEE Explore digital library Murata, A. & Iwase, H. (2001). Extending Fitts’ law to a three-dimensional pointing task. Retrieved February 4, 2008, from Science Direct website: http://www.sciencedirect. com/science?_ob =A rticle URL&_udi=B6V8T445GW332&_userl 022551&_rdoc=1 &Jmt=&_origsearch&_sort=d&viewc&_acct= C000050484& version = 1 & un Version 0& usenid=1 022551 &md5 = 70c35f5J94 bad9d48l4fc2el8fcb83] 0 Nguyen, D., & Canny, J. (2005). MultiView: Spatially Faithful Group Video Conferencing. Proceedings of the 2005 Conference on Human Factors in Computing Systems (CHI’05) 512-521. Portland, Oregon. Patrick, A. (2007, March 9). Privacy practicesfor HCI research. Retrieved Dec 15, 2007, from http ://www.andrewpatrick.ca/essays/privacy-practices-for-hci research!  134  Preece, J., Rogers, Y., & Sharp, H. (2002). Interaction design. New York: John Wiley & Sons, Inc. Ranjan, A., Birnholtz, J.P., & Balakrishnan, R. (2006). An exploratory analysis of partner action and camera control in a video-mediated collaborative task. Proceedings ofthe 2006 20th anniversary conference on Computer Supported Cooperative Work, (pp. 403-422). New York, NY, USA: ACM Press. Rohrig, C., & Jochheim, A. (n.d.). Java-basedframeworkfor remote access to laboratory experiments. Retrieved Dec 7, 2007, from University of Hagen, Department of Electrical Engineering website: http://prtfernuni hagen. de/virtlab/ace2k/ace2k.pdf Schedlbauer, M., Pastel, R., Heines, J. (1995). An extensible and interactive research platform for exploring Fitts’ Law. Retrieved February 3, 2008, from http://teaching. cs. uml. edu/’-heines/techrpts/Papers/Schedlbauer-TR-ECIS MTE.pdf Schmid, C., Eikaas, T.I., Foss, B.A., & Gillet D. (2001). A remote laboratory experimentation network. Retrieved February 20, 2008 from Scientific Commons database Scholl, J., McCarthy, J., & Harr, R. (2006). A comparison of chat and audio in media rich environments. Proceedings of the Computer Supported Collaborative Work (CSCW 2006), (pp. 323-332). New York, NY, USA: ACM Press. Schreer, 0., Brandenburg, N., Askar, S., & Kauff P. (2001). Hybrid recursive matching and segmentation-basedpostprocessing in real-time immersive video conferencing. Paper presented at the conference on VMV ‘01, Stuttgart, Germany. Retrieved November 20, 2007, from CiteSeerX digital library. Sun, J. & Razdan, A. (n.d.). Remote control & visualization of scanning probe microscopes via web. Retrieved February 20, 2008, from IEEE Explore digital library Svedkauskaite, A., Reza-Hernandez, L., & Mary Clifford (2003, June 24). Critical Issue: Using technology to support Limited- English-Proficient (LEP) students’ learning experiences. Retrieved February 23, 2008, from Learning Point Associates, North Central Regional Educational Laboratory website: http://www. ncrel. org/sdrs/areas/issues/methods/technlgy/te900. htm Taylor, R.M., Robinett, W., Chi, V.L., Brooks, F.P., Wright, W.V., Williams R.S., & Snyder, E.J. (1993). The nanomanipulator: A virtual-reality interface to a scanning tunneling microscope. Proceedings of the 20th annual conference on  135  Computer graphics and interactive techniques, (pp. 127 -134). New York, NY, USA: ACM Press. Treloar, A. (n.d.). DART: Building the new collaborative e-research infrastructure. Retrieved March 13, 2008, from The Council of Australian University Directors of Information Technology website: http://www.caudit.edu. au/educauseaustralasiaO7/authors_papers/Treloar-1 83.pdf Vertegaal, R., Weevers, I., Sohn, C., & Cheung, C. (2003). GAZE-2: Conveying eye contact in group video conferencing using eye-controlled camera direction. Proceedings ofthe SIGCHI conference on Human factors in computing systems (CH12003), (pp. 521-528). New York, NY, USA: ACM Press. Wei Chao, C., Towles, H., Nyland, L., Welch, G., & Fuchs. H. (2000). Toward a compelling sensation of telepresence: demonstrating a portal to a distant (static) office. Proceedings ofthe conference on Visualization ‘00 (IEEE visualization 2000), (pp. 327- 333). Los Alamitos, CA, USA: IEEE Computer Society Press. Wei-Chao, C., Towles, H., Nyland, L., Welch, G. & Fuchs, H. (2000). Toward a Compelling Sensation of Telepresence: Demonstrating a portal to a distant (static) office. IEEE Visualization, Proceedings ofthe conference on Visualization ‘00, Salt Lake City, Utah, United States, Pages: 327 333. http://delivery.acm.org/10.1i45/3800OO/375263/p327chen.pdJ?key] =375263&key2=2138735021&collGUIDE&dlGUIDE&CFID= 59095509& CFTOKEN=342991 21 —  Zaimovice-Uzunovice, N., Leme, S., & Petkovice, D. (2001, August). Virtual instruments a chance to teach engineering at a distance. Paper presented at International Conference on Engineering Education, Oslo, Norway, (7D1-247D1-26). Retrieve February 20, 2008, from http://www.ineer.org/Events/ICEE200 1/Proceedings/authors!. ./papers/43 7.pdf —  Zhang , J. Z., Ball, A. K., Extine, M. C., & Extine, W. (2007). Design ofa real time remote-access engineering laboratory using integrated web service and wireless technologyfor distance learners. Retrieved February 23, 2008, from http://www.eng.monash.edu. au/uicee/worldtransactions/WordTransA bstracts Vo14 No2/1 5-Zhang23.pdf Zhao, H. (2002, October) Fitts’ Law: Modeling movement time in HCI. Retrieved November 12, 2007, from the University of Maryland, Department of Computer Science web site: http://www.cs.umd.edu/class/fall2002/cmsc838s/tichi/fitts. html  136  Appendices  Appendix A Questionnaire The following questionnaire was administered to all  PARTICIPANTS  in our  STuDY.  The results were analyzed and are reported in Chapter 5 of this thesis.  The  STUDY  was conducted under the ethics certificate 1107-00128 issued by the  Behavioral Ethics Research Board of the University of British Columbia.  137  Remote Experimenter Questionnaire: Personal information: 1. What age group do you fall into: 15—19 I 20—24 125—29 I 30—34 I 35—39  I 40 and higher  2. Are you familiar with the idea of Fitts Law?  3. Have you done any point-and-click experiment before?  Interaction: 1. As an experimenter, my overall feeling about interacting with the subject was pleasant and comfortable. Rank from a scale of 1 to 5 (1 strongly agree, 5 strongly disagree)  2. Did I feel that I have delivered my instruction in a way that was clear and comprehensible to a subject? Select from a scale of 1 to 5:  138  a. My subject understood all instructions clearly when they were stated only once. b. My subject understood some instructions clearly if they were stated once. c. I had to repeat the instructions multiple times before my subjects could understand them d. My instructions were extremely poorly delivered and not understood by subjects at all so I had to repeat multiple times e. others: please specify_____________________________________  3. It was easy to talk to and hear the subject. Rank from a scale of 1 to 5 (1 strongly agree, 5 strongly disagree)  4. When delivering instructions and explanations, I frequently used hand gesture to interact with a subject. Rank from a scale of 1 to 5 (1 strongly agree, 5 strongly disagree)  5. I looked at my subject frequently to make sure he / she was making correct progress on the experiment. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree)  139  6. I was aware of my subject’s presence all the time. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree)  7. Rank from a scale of 1 to 5 (1 means agree, 5 means disagree) a. I confused my subject very often b. My subject confused me very often  8. Overall, from my own point of view, how well did the experiment go? Please rank based on a scale of 1 to 5 (1 means very well and smoothly, 5 means very difficult and badly).  9. How much sense of control did I feel over the experiment? Please rank on a scale of 1 to 5 (1 means totally in control over the experiment, 5 means not at all in control over the experiment)  10. I DID NOT need much interaction with my subject when doing the tasks since all of them were non-interactive. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree)  11. I was making eye contact with my subject all the time.  140  Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree)  Training: 12. After being trained by my assistant, how confident did I feel about playing the role of an experimenter and running the study with a subject? Rank from a scale of 1 to 5 (1 means most confident, 5 not confident at all)  Video-conferencing: 13. The frequent interruption from a research lab assistant was pleasant and helpful. Rank from a scale of 1 to 5 (1 means strongly agree, and 5 means strongly disagree)  14. Using VNC (virtual network connection), I could control my subject’s screen and interfere with the subject’s interaction, how did I feel about this kind of control? Rank from on a scale of 1 to 5: (1 means most pleasant and not disturbed or annoyed at all, 5 means very unpleasant and outrageously disturbed and annoyed).  141  15. At the time I got access to my subject’s PC via VNC, I immediately knew where to look for on subject’s screen. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree)  16. If my subject made a mistake, I used oral instructions and VNC to intervene the subject. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree)  17. I felt very comfortable seeing my own face on screen. Rank from a scale of 1 to 5 (1 means strongly agree, and 5 means strongly disagree)  18. What kind of screen presentation of videos made I feel that I am in control of an experiment (gave me the sense of observing my subject)? El Seeing both subject’s and my own face on the pc screen El Seeing just the subject’s face on the screen  Set-up: 19. Were the camera icons intuitive enough to be recognized? Please rank from a scale of 1 to 5 (1 means “recognized it right away”, 5 means “did  142  not know it represented a camera”).  20. By looking at the layout of the remote lab, could I somehow get the idea where my subject was located in the remote lab? Please rank from a scale of 1 to 5 (1 means “I knew where my subject was in the lab”, 5 means “I had no idea where my subject was in the lab”).  21. How did I feel about the response time of camera adjustment? Please rank from 1 to 5 (1 means very quickly and instantly, 5 means very slowly and long delayed).  22. Overall, how was my experience about using the econtrol interface to adjust the cameras in remote? Please rank from a scale of 1 to 5 (1 means very easy to use, 5 means very difficult to use).  23. Please select all the media that were used to help with the camera set-up: Video, Audio, VNC, Chat  Other Comments:  143  Remote Subject Questionnaire: Personal information: 1. What age group do you fall into: [1 15—19 [1 20 —24 [125—29 [130—34 [135—39  [140 and higher  2. Are you familiar with the idea of Fitts Law?  3. Have you done any point-and-click experiment before?  Interaction: 1. As a subject, my overall feeling about interacting with the experimenter was pleasant and comfortable. Rank from a scale of 1 to 5 (1 strongly agree, 5 strongly disagree)  2. Did I feel that I have received my experimenter’s instruction in a way that was clear and comprehensible? Select from a scale of 1 to 5:  144  b. I understood all instructions clearly when they were stated only once. c. I understood some instructions clearly if they were stated once. d. I had to ask my experimenter to repeat the instructions multiple times before I could understand them. e. My experimenter delivered the instructions extremely poorly and could not be understood at all so I had to ask my experimenter to repeat multiple times f.  others: please specif’______________________________________  3. It was easy to talk to and hear the experimenter. Rank from a scale of 1 to 5 (1 strongly agree, 5 strongly disagree)  4. When listening to instructions and explanations, I frequently used hand gesture to interact with the experimenter. Rank from a scale of 1 to 5 (1 strongly agree, 5 strongly disagree)  5. I looked at my experimenter frequently to ensure that I was making correct progress on the experiment. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree)  145  6. I was aware of my experimenter’s presence all the time. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree)  7. Rank from a scale of 1 to 5 (1 means agree, 5 means disagree) a. I confused my experimenter very often b. My experimenter confused me very often  8. Overall, from my own point of view, how well did the experiment go? Please rank based on a scale of 1 to 5 (1 means very well and smoothly, 5 means very difficult and badly).  9. How much sense of BEING controlled did I feel over the experiment? Please rank on a scale of 1 to 5 (1 means totally being controlled over the experiment, 5 means not at all being controlled over the experiment)  10. 1 DID NOT need much interaction with my experimenter when doing the tasks since all of them were non-interactive. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree)  11. I was making eye contact with my experimenter all the time.  146  Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree)  Training: 12. After being trained by a research lab assistant, how much confidence did I sense about the other participant playing the role of an experimenter and running the study with me? Rank from a scale of 1 to 5 (1 means most confident, 5 not confident at all)  Video-conferencing: 13. The frequent interruption from a research lab assistant was pleasant and helpful. Rank from a scale of 1 to 5 (1 means strongly agree, and 5 means strongly disagree)  14. Using VNC (virtual network connection), my experimenter could control my screen and interfere with my interaction, how did I feel about this kind of control? Rank from on a scale of 1 to 5: (1 means most pleasant and not disturbed or annoyed at all, 5 means very unpleasant and outrageously disturbed and annoyed).  147  15. At the time my experimenter got access to my PC via VNC, he/she immediately knew where to look for on my screen. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree)  16. If I made a mistake, my experimenter used oral instructions and VNC to intervene me. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree)  17. I felt very comfortable seeing my own face on screen. Rank from a scale of 1 to 5 (1 means strongly agree, and 5 means strongly disagree)  18. What kind of screen presentation of videos made me feel that the study was controlled by my experiment (gave me the sense that my experimenter was observing me)? L Seeing both the experimenter’s face and my own face on the pc screen LI Seeing just the experimenter’s face on the screen  148  Other Comments:  Local Experimenter Questionnaire: Personal information: 1. What age group do you fall into: 15  —  19  D 20 —24 25—29 30—34 D 35—39  D 40 and higher  2. Are you familiar with the idea of Fitts Law?  3. Have you done any point-and-click experiment before?  Interaction: 1. As an experimenter, my overall feeling about interacting with the subject was pleasant and comfortable.  149  Rank from a scale of 1 to 5 (1 strongly agree, 5 strongly disagree)  2. Did I feel that I have delivered my instruction in a way that was clear and comprehensible to a subject? Select from a scale of 1 to 5: a. My subject understood all instructions clearly when they were stated only once. b. My subject understood some instructions clearly if they were stated once. c. I had to repeat the instructions multiple times before my subjects could understand them d. My instructions were extremely poorly delivered and not understood by subjects at all so I had to repeat multiple times e. others: please specify___________________________________  3. It was easy to talk to and hear the subject. Rank from a scale of 1 to 5 (1 strongly agree, 5 strongly disagree)  4. When delivering instructions and explanations, I frequently used hand gesture to interact with a subject. Rank from a scale of 1 to 5 (1 strongly agree, 5 strongly disagree)  150  5. I looked at my subject frequently to make sure he / she was making correct  progress on the experiment. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree)  6. I was aware of my subject’s presence all the time. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree)  7. Rank from a scale of 1 to 5 (1 means agree, 5 means disagree) a. I confused my subject very often b. My subject confused me very often  8. Overall, from my own point of view, how well did the experiment go? Please rank based on a scale of 1 to 5 (1 means very well and smoothly, 5 means very difficult and badly).  9. How much sense of control did I feel over the experiment? Please rank on a scale of 1 to 5 (1 means totally in control over the experiment, 5 means not at all in control over the experiment)  151  10. I DID NOT need much interaction with my subject when doing the tasks since all of them were non-interactive. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree)  11. 1 was making eye contact with my subject all the time. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree)  Training: 12. After being trained by my assistant, how confident did I feel about playing the role of an experimenter and running the study with a subject? Rank from a scale of 1 to 5 (1 means most confident, 5 not confident at all)  Other comments:  152  Remote Subject Questionnaire: Personal information: 1. What age group do you fall into: LI 15—19 LI 20 —24 LI 25—29 LI 30—34  LI 35—39  LI 40 and higher  2. Are you familiar with the idea of Fitts Law?  3. Have you done any point-and-click experiment before?  Interaction: 1. As a subject, my overall feeling about interacting with the experimenter was pleasant and comfortable. Rank from a scale of 1 to 5 (1 strongly agree, 5 strongly disagree)  2. Did I feel that I have received my experimenter’s instruction in a way that was clear and comprehensible? Select from a scale of 1 to 5:  153  a. I understood all instructions clearly when they were stated only once. b. I understood some instructions clearly if they were stated once. c. I had to ask my experimenter to repeat the instructions multiple times before I could understand them. d. My experimenter delivered the instructions extremely poorly and could not be understood at all so I had to ask my experimenter to repeat multiple times e. others: please specify______________________________________  3. It was easy to talk to and hear the experimenter. Rank from a scale of 1 to 5 (1 strongly agree, 5 strongly disagree)  4. When listening to instructions and explanations, I frequently used hand gesture to interact with the experimenter. Rank from a scale of 1 to 5 (1 strongly agree, 5 strongly disagree)  5. I looked at my experimenter frequently to ensure that I was making correct progress on the experiment. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree)  154  6. I was aware of my experimenter’s presence all the time. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree)  7. Rank from a scale of 1 to 5 (1 means agree, 5 means disagree) a. I confused my experimenter very often b. My experimenter confused me very often  8. Overall, from my own point of view, how well did the experiment go? Please rank based on a scale of 1 to 5 (1 means very well and smoothly, 5 means very difficult and badly).  9. How much sense of BEING controlled did I feel over the experiment? Please rank on a scale of 1 to 5 (1 means totally being controlled over the experiment, 5 means not at all being controlled over the experiment)  10. I DID NOT need much interaction with my experimenter when doing the tasks since all of them were non-interactive. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree)  11. I was making eye contact with my experimenter all the time.  155  Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree)  Training: 12. After being trained by a research lab assistant, how much confidence did I sense about the other participant playing the role of an experimenter and running the study with me? Rank from a scale of 1 to 5 (1 means most confident, 5 not confident at all)  Video-conferencing: 13. The frequent interruption from a research lab assistant was pleasant and helpful. Rank from a scale of 1 to 5 (1 means strongly agree, and 5 means strongly disagree)  14. Using VNC (virtual network connection), my experimenter could control my screen and interfere with my interaction, how did I feel about this kind of control? Rank from on a scale of 1 to 5: (1 means most pleasant and not disturbed or annoyed at all, 5 means very unpleasant and outrageously disturbed and annoyed).  156  15. At the time my experimenter got access to my PC via VNC, he/she  immediately knew where to look for on my screen. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree)  16. If I made a mistake, my experimenter used oral instructions and VNC to intervene me. Rank from a scale of 1 to 5 (1 means strongly agree, 5 means strongly disagree)  17. I felt very comfortable seeing my own face on screen. Rank from a scale of 1 to 5 (1 means strongly agree, and 5 means strongly disagree)  18. What kind of screen presentation of videos made me feel that the study was controlled by my experiment (gave me the sense that my experimenter was observing me)? L Seeing both the experimenter’s face and my own face on the pc screen Li Seeing just the experimenter’s face on the screen  Other Comments:  157  Appendix B Interview The following interview questions were administered to STUDY.  The  EXPERIMENTERS lfl  our  The results were analyzed and are reported in Chapter 5 of this thesis.  STUDY  was conducted under the ethics certificate H07-00128 issued by the  Behavioral Ethics Research Board of the University of British Columbia.  158  Interview Script: 1. How was your experience of being an experimenter? 2. What kind of problems did you experience when interacting with your subject? For experimenters under the remote condition only:  3. Between audio and video, what media served as a more important role in communication during the experiment? 4. What was the most challenging part of setup?  159  Appendix C Ethics Certificates The following are the ethics certificates issued by the Behavioral Ethics Research Board of the University of British Columbia and the Comité d’éthique de la recherche de la Faculté des arts et des sciences of L’Université du Québec a Montréal.  160  CERTIFICATE OF APPROVAL- MINIMAL RISK RENEWAL PRINCIPAL INVESTIGATOR: Kellogg S. Booth  DEPARTMENT:  USC BREB NUMBER:  UBCIScience/ComputerScience  :HO7OO1 28  INSTITUTION(S) WHERE RESEARCH WILL BE CARRIED OUT: Institu ion  UBC  Site  Vancouver (excludes UBC Hospital)  Other locations where the research will be conducted: A laboratory at the Unwersity of Montreal under the direction of Dr Aude Dutresne LORIT Laboratopre Observatoire de Recherohe air le Télé-Apprentissage en collaboration avoc Is LICEF/TELUO-UQAM.  CO-INVESTIGATOR(S): Ying Zhang  SPONSORING AGENCiES: Natural Sciences and Engineering Research Council of Canada (NSERC) ‘Network for effective collaboration technologies through advanced research (NECTAR) researcfi Tele-universite- ‘Network Agreement: Canarie Advanced Applications Program: BEST Projecr -  PROJECT TITLE: Conducting experiments with human subjects in local and remote laboratories  EXPIRY DATE OF THISAPPROVAL: March31, 2009 APPROVAL DATE: March 31, 2008 The Annual Renewal for Study have been reviewed and the procedures were found to be acceptable on ethical grounds for research involving human subjects.  Approval is Issued on behalf of the Behavioural Research Ethics Board Dr. M. Judith Lynani, Chair  Dr. Ken Craig, Chair Dr. Jim Rupert. Associate Chair Dr Laurie Ford Associate Chair Dr Daniel Salhani Associate Chair Dr Anita Ho Associate Chair  161  Université de Montréal Comité d’ethique de Ia recherche de Ia Facuttê des arts et des sciences (CERFAS)  DIRECTIVES ET QUESTiONNAIRE EVALUATION ETHIQUE DES PROJETS DE RECHERCHE  DIRECTIVES 1)  Veuillez rerriptir le questionnaire cl-joint. Vous pouvez Ic dactylographier ou le remplir directementà lécran sur Ic site Web http:/iwwwfasumentrealcaifasinfoiforrnulaire_ethiguehtm puis limprirner; n’oubliez pas de Ic signer. Vous devez en foumir 4 exemplaires (Poriginal et trois copies).  2)  Veuiflez joindre aux 4 exemplaires du questionnaire 4 copies de Ta description du projet. S’iI sagit d’une demande de subvention: partie de Ta dernande de subvention décrivant Ic projet et Ta methodologie; ne pas inclure a bibliographie, les sections budgétaires, Ic curriculum vitae. Vous pouvez ajouter tout autre renseignement susceptible de faciliter le travail des membres du Comité dethique.  3)  Veuillez egalement joindre 4 copies du(des) formulaire(s) de consentement (ou du synopsis des in formations données aux participants afin d’obtenir leur consentement verbal). Le formulaire de consernement (ou Ic synopsis le cas échéant) est trôs important et dolt être fait scion les riormes. Avant de Ic produire, veuiUez consulter le Document d’in formation sur les formulaires de consentement disponible sur Ic site Web: htto:Jfwww.fasumontreal ca’fasinfo/formuiaire_ethiue.htrn Vous peuvez ajouter tout autre renseignement susceptible de facitter Ic travail des membres du  Comité dethique (questionnaires, schema d’entrevue etc.). 4)  Veuillez retourner Ic questionnaire et les documents en quatre exemptaires  a:  Si vous êtes tudiant-e:  TechnicIenne en gestion des dossiers étudiants au secretariat de votre departernent ou école Si vous êtes professeur-e ou chercheur-e:  Vice-décanat a Ia recherche FacultC des arts et des sciences Pavilion Lione!-Groulx Bureau C-9109 UniversitC de Montréal -  162  Unlversftë de Montréal QUESTIONNAIRE EVALUATION ETHIQUE DES PROJETS DE RECHERCHE  Comitê d’éthique de Ia recherche de Ia Faculté des arts et des sciences (CERFAS) Dact1ogFapher ou remplir clirectement stw Ic site http/www.fas.umontreaLca/fasinfo/formulaire ethiguehtm  NOM TITRE  : DUFRESNE Professeure titulaire  PRENOM: Aude DEPARTEMENT: Communication  Etuchant-e-s: DIPLÔME POSTULE: DIRECTEUR-TRICE DE RECHERCHE: ADRESSE COURRIEL DE L’ETUDIANT-E ETNUMERODETELEPHONE SOURCE DE FINANCEMENT (Ic cas échéant): (apparaitra sur le certificat déthique) COCHERCHEUR(S) (le cas óchóant): PETIT Gregory, REISS Stéphane, ZHANG Ying  TITRE EXACT DU PROJET (sera inscrit tel quel sur Je certificat déthique): Human-Computer Interaction Research Project (CPSC 549A) : Remote Interaction  DATE DU DEBUT DE LA COLLECTE DES DONNEES 30 avrtl 2007  SECTiON A OUI 1.  Les participants(ou une partie d’entre eux) sont-ils ages de moms de 18 ans?  2.  Les participants (ou une partie d’entre eux) sont-ils inaptes (ineapacitC a donner un consentement Cclairé par exemple en raison de curs capacitCs mentales)?  NON  Si OUt a a Question 1 su 2 Le consentement dun parent ou d’un tuteur sera-t-il obtenu par écrit? 3..  Les participants (ou une partie dentre eiix) proviennent-ils dune population <<captive ou dCpendante (ex personnes en miteu carcéral ou de protection)?  C  163  OUI 4.  Le protocole de recherche prévoit-il que des groupes de participants doivent ètre exclus de Ia recherche en raison de risques pour leur sante (par exemple en raison de grossesse, de troubles cardio-vasculaires ou respiratoires, etc.) 7  0  5.  Les participants (ou une partie d’entre eux) sont-ils soumis urie procedure d’ordre medical dans le cadre de Ia recherche (par exemple prise de sang, utilisation de médicaments. etc.)?  Q  6.  Les participants peuverit-ils faire lobjet dun signalernent obligatoire en vertu de Ia Loi sur Jo protection de Ia jeunesse (violence envers les enfants>?  U  7.  Au-deI dun signalement obligatoire, les inforrnations de nature nominative ou confidentielle données par les participants aux chercheurs peuvent-elles ètre transmises a d’autres personnes ou organismes?  8.  Le protocole de recherche nécessite-t-iI de ne divulguer aux participants que des renseignements partiels sur létude ou de les induire temporairement en erreur?  9.  La recherche se déroulera-t-elle dons un autre pays que le Canada? Si oui, le(s)quel(s):  10. La participation a a recherche peut-elle entrainer des risques sérieux pour Ia sante mentale ou physique des participants (ou pour une partie d’entre eux)?  NON  U U  11. La participation a a recherche peut-elle entrainer involontairement des prejudices aux participants dans le cadre de leur milieu familial (relations conjugales paren tales, etc.) ou tie leur milieu de travail (par exemp[e face a lemployeur, au syndicat ou aux collCgues)?  SECTION B Vous pouvez répondre sur ces pages ou utiliser des pages distinctes pour répondre a cette section, en reproduisant Ic numéro et 1€ titre de La question; ne pas oublier de signer le document. 1. Résumé du projet (DCcrire en 10 lignes es objectifs du projet) The purpose of this experiment is to test the hypothesis that with correct equipment settings, experiments conducted remotely should not aflect performance in a simple 1-itt’s law (1 here is a relationsnip between me wiatn of a target and me time requirea to acquire the targer The participant will be asked to use a visioconterence to perform tasks. He/she will be selected to play the roie as either a subject or an experiment in one session. As a subject, he/she wili be asked to perform a pointand-click experiment; as an experimenter, he/she will supervise me subject and interview him. I he Interaction will be recoraed and analyzed (quantitative and qualitative). I he participant will be asked to participate in only one session, each lasting no more than 1 hour. Recordings will only be used for analysis and with the subject consent it may also be used br class project and other research presentations at the University of British Columbia or University of Montreal. 2. Quelles soot les caractéristiques des participants qui prendront part a Ia recherche (ige, sexe, milieu, etc.)? Préciser le nombre de participants. Préciser Cgalement s’il y a des critéres d’exclu sion (caractéristiques empécharit des personnes tie participer a la recherche) Les participants seront des étudiants de Montréal et des étudiants de Vancouver.  164  3. De queue facon es participants seront-ils recrutés? Une annonce sera passée au sein des ditférentes universités de Montréal pour recruter des participants parlant correctement anglais.  4.  A quelleCs)  actMtés(s) participeront les sujets (entrevues, questionnaires, tests, etc.)? Les participants interviendront dans lexpérimentation, soit en tant que sujet, soit en tant qu’expérimentateur, et devront repondre a un questionnaire a La fin de ‘experimentation. Les participants seront flumes pendant rexpérimentation mais pourront refuser l’enreistrement s’ils Ic souhaitent.  5. Les participants encourent-ils des risques ou inconvénients? Queues mesures seront prises pour contrer ces risques ou inconvénients? Aucun risque ou inconvenient ne sera encouru.  6. Queue forme de consentement le chercheur demande-t-il aux participants? Avant de remplir cette section, veuillez consulter Ic document formulaires de consentement  Informations relatives aux  Consentement ècrit Joindre 4 copies du(des) formulaire(s) de consentement -  Consentement verbal Joindre 4 copies d’un synopsis des informations données verbalement aux participants aim d’obtenir teur consentement verbal. —  Le recoLirs au seul consentement verbal constitue une exception. ci-dessous.  Signalure:  Veuillez justifier ce choix  Date: 17 avril 2007  165  Appendix D D.1: Scatter plots for remote conditions  120000  /e30:2322/  .00  2.00  300  IOU  4  2.00  3.00  4.00  Subject 2  Subject I  0  1200.00  —  6O%7/ ,J3;6166?1?3?9I  1.00 IOU  2.00  Su1ject 3  300  4.00  200  3.00  4.00  Subject 5.  166  1100.00  1660.004  Time_C  =  519.99 .233.97 * I  hI  i  U0O/  100  200  300  400  1600.00  ,,J91s+22osaI  200  300  2.00  400  577.09  +  233.91  *  3.00  200  +  281.10  300  400  Subject 9  1300.00  /  Time_12 60969 P-Square = 0.99  ::  4.00  +  174.49 ID  /  1.00 100  300  * /  160  400  SubjectS  Time Ii  2.00  Time_9 = 525.96 R-Square = 0.07  /  100  .00  Subjeàt7  Subject 6  2.00  3.00  /  4.00  Subject 12 Subject 11  167  D.2: Scatter plots for local conditions  1600.00  1300.00 re7f,126Y/  1000.00  ISO  2.00  I  3.00  ::  I  1.00  4.00  9  .OO.J 1320  I  Time 3  .00  2.00  * /  9  1000.00’  .J  4.00  Subject 2  Subject I  I300.00  3M  2.00  1211.13 +23.38* IIY’  3.00  4.00  Subject 3  Time 4 = 412.81 P-Square = 1.88  1.00  2.00  +  3.00  281.64 Iii  4.00  Subject 4  //t816320h1  1.0  2.00  Subject 5  3.00  4.00  1.0  2.00  350  4.00  Subject 6  168  140000  0  i.:::  1 00  200  3.00  4.D0  IOU  2.00  3.00  4.00  Subject S  Subject7  100000  E:0H/2bb956nu1  2.00  Subject 9  3.00  4.00  1.00  2.00  3.00  4.00  Subject 10  169  Appendix E E.1: Mean ranks and standard deviations of each question in the questionnaires on a 5-point, agreeable scale ranging from 1(strongly agree) to 5(strongly disagree). Question  Local experimenter  Local subject  Remote experimenter  Remote subject  1  Mean = 1.400, SD = .699  Mean = 2.300, SD 1.418  Mean 2.400, SD = 1.075  Mean = 2.100, SD = 1.433  2  Mean =2.050, SD1.212  Mean= 1.30, SD.675  Mean 1.600, SD=.516  Mean= 1.600, sd=.844  3  Mean = 1.600, SD = .843  Mean = 1.900, SD 1.449  Mean 4.600, SD = .699  Mean = 1.900, SD = .316  4  Mean = 3.400, SD = 1.647  Mean = 4.000, SD 1.414  Mean 4.600, SD = .699  Mean = 4.900, SD = 3.162  5  Mean = 2.300, SD = 1.160  Mean = 3.200, SD = 1.476  Mean = 2.500, SD = 1.501  Mean = 3.900, SD = 1.287  6  Mean = 1.400, SD = .966  Mean = 2.600, SD 1.647  Mean = 2.100, SD = 1.370  Mean = 2.100, SD = 1.449  7a  Mean = 3.900, SD = 1.449  Mean = 4.100, SD 1.287  Mean 4.000, SD = .943  Mean = 4.500, SD = .707  7b  Mean = 4.300, SD = 1.059  Mean = 4.500, SD .707  Mean = 4.600, SD = .699  Mean = 4.100, SD = 1.101  8  Mean = 1.500, SD=.850  Mean = 1.850, SD=.818  Mean 1.900, SD=.568  Mean = 1.800, SD=.918  9  Mean = 1.950, SD = .599  Mean = 2.700, SD 1.494  Mean = 2.200, SD = .632  Mean = 2.400, SD = .966  10  Mean = 2.600, SD = 1.350  Mean = 2.300, SD = 1.059  Mean 2.900, SD = 1.197  Mean = 1.800, SD = .919  11  Mean = 3.300, SD = 1.337  Mean = 3.850, SD .747  Mean 3.900, SD = 1.287  Mean = 3.800, SD = 1.229  Number Interaction  170  Question Number  Local experimenter  Local subject  Remote experimenter  Remote subject  Mean=2.000, SD = .943  Mean= 1.800, SD = .919  Mean=2.500, SD = 1.080  Mean2.700, SD = 1.337  13  N/A  N/A  Mean= 1.800, SD=1.033  Mean= 1.900, SD=1.101  14  N/A  N/A  Mean= 1.900, SD = .738  Mean=2.300, SD = 1.337  15  N/A  N/A  Mean = 2.400, SD = 1.265  Mean = 2.600, SD = 1.173  16  N/A  N/A  Mean = 2.300, SD=1.418  Mean = 3.00, SD=1.944  17  N/A  N/A  Mean = 1.500, SD = 1.080  Mean = 2.200, SD = .790  18  N/A  N/A  Mean = 1.300, SD = .480  Mean = 1.6, SD = .520  1  N/A  N/A  Mean=1.820, SD= 1.080  N/A  2  N/A  N/A  Mean=2.360, SD= 1.120  N/A  3  N/A  N/A  Mean =2.180, SD=.870  N/A  4  N/A  N/A  Mean = 1.720, SD = .900  N/A  5  N/A  N/A  N/A  N/A  Training 12  Video conferencing  Set up  171  E.2 Pair-wise experimenter vs. subject (condition ignored) Question  Experimenter  Subject  difference  Independent samples T-test results  1  Mean = 1.400, SD = .681  Mean = 2.300, SD= 1.380  Mean = 1.100, SD =1.165  T (df= 19) = -3.018, p = .007, effect size =.324  2  Mean=1.825, SD =.935  Mean= 1.450, SD =.759  Mean= .875, SD =.971  =  Mean=3.100, SD =1.7 13  Mean= 3.400, SD= 1.846  Mean= .900 SD =1.165  T(df= l9)-.922,p = .368, effect size = .043  4  Mean = 4.000, SD =1.376  Mean = 4.450, SD = 1.099  Mean = .950, SD = 1.099  T(df 19) = -1.443, p = 1.652, effect size = .098  5  Mean = 2.400, SD = 1.3 13  Mean = 3.550, SD =1.394  Mean = 2.050, SD =1.099  T(df= 19) = -2.489, p = .022, effect size = .246  6  Mean=1.750, SD = 1.208  Mean= 2.350, SD =1.531  Mean= 1.500, SD =1.468  T(df=19)=-1.318, p = .203, effect size = .084  7a  Mean = 3.950, SD = 1.190  Mean = 4.300, SD =1.031  Mean = 1.050, SD =1.099  T(df= 19) = -1.046, p = .309, effect size =.054  7b  Mean = 4.450, SD = .887  Mean = 4.300, SD =.923  Mean = .950, SD =.944  =  Mean= 1.700, SD = .732  Mean= 1.925, SD =.847  Mean= .825, SD =.654  T(df= l9)-.488,p = .63 1, effect size = .012  number  Interaction  3  8  ,  T(df19)=1.325,p .2 10, effect size = .084  T(df= 19) = .497, p .625, effect size = .012  172  Question number  Experimenter  Subject  difference  Independent samples T-test results  9  Mean =2.075, SD = .612  Mean= 2.550, SD = 1.234  Mean= .875, SD =.944  T(df= 19)=-1.758, p = .095, effect size = .139  10  Mean=2.750, SD = 1.251  Mean= 1.050, SD =.998  Mean= 1.300, SD =1.080  T(df= l9)2.OO8,p = .059, effect size = .175  11  Mean=3.600, SD = 1.313  Mean= 3.825, SD =.990  Mean= 1.0750, SD =1.079  =  Mean = 2.250, SD = 1.019  Mean = 2.250, SD = 1.208  Mean = 1.100 SD =.967  =  12  ,  T(df=19)=-.659,p .518, effect size = .022  T(df= 19) = .00 1, p .900, effect size =.001  173  E.3 :Independent t-test results: Experimenter vs. experimenter; subject vs. subject Question Number  Remote experimenter vs. local experimenter  Remote subject vs. local subject  1(significant)  Z(l8) = -2.471, p .250, large effect  2  Z(18)= -i362,p= .508, r12 .024, small effect  3( significant)  Z(18) = -2.006, p = .045, i2 = .180, large effect  4  Z(18) = -1.745, p .140, large effect  5  T(18) = .332, p = .743, i2 .006, small effect  6  Z(18) = -1.378, p = .168, r2 .006, small effect  7a  Z(18)=-.32l,p=.748,r2= .006, small effect  Z(18)-.589,p=.556,2=.019, small effect  7b  Z(18)=-.580,p=.562,2= .0 18, small effect  Z(18)=-.753,p=.452,2=.031, small effect  8  Z(18) = -1.523, p = .128, ii2 = .130, moderate effect  Z(18) = -.161, p small effect  9  Z(18)= -.8l6,p = .414, ri2 .042, small effect  Z(18)= .196, p small effect  10  Z(18) = -.798, p = .425, r2 .031, small effect  11  Z(18)-l.331,p.183,ri2= .002, small effect  T(l8)=-.l10,p=.914,r2=.001, small effect  12  Z(l8)-l.109,p=.268,ri2= .064, moderate effect  Z(18)-1.461,p=.l44,Ti2=.llO, moderate effect  .013, i12  =  =  =  T(18) = -.209, p small effect  =  =  =  =  .002,  Z(18) = -1.693, p large effect  Z(18) = -.677, p small effect  .541, M 2  =  Z(18) = -1.107, p small effect  =  .737, r2  Z(18)=-.936,p= .349, r2= .046, moderate effect Z(18) = -.612, p small effect  .081, ri2  =  =  .090, ri2  =  .140,  =  .268, ri2  =  .060,  =  Z(18) = -1.071, p small effect  .020,  .498, ri2  =  =  .020,  =  .872, M 2  .845,i2  =  =  =  .284, ri2  .001,  .002,  =  .056,  174  E.3.1: Paired-sample t-test for remote experimenter vs. subject, p  =  .025  Question Number  Experimenter  Subject  Difference  Paired-sample T-test  1  Mean = 2.400, SD = 1.075  Mean = 2.100, SD 1.433  Mean = 1.300, SD = 1.060  Z(9) = -.3 12, p .755, i2 =.010 small effect  2  Mean = 1.600, SD = .516  Mean = 1.600, SD = .844  Mean = .600, SD .516  Z(9) = .001, p = .90, ii2 < .001, small effect  3  Mean = 4.600, SD=.699  Mean = 4.900, SD=.316  Mean = .500, SD=.707  Z(9) = -1.134, p .257,r2=.l20, moderate  4  Mean = 4.600, SD = .699  Mean = 4.900, SD = 3.162  Mean = .500, SD = .707  Z(9) = -1.134, p .257, i2 = .120 moderate effect  5  Mean=2.500, SD 1.501  Mean=3.900, SD = 1.287  Mean=2.000, SD = 1.333  Z(9)-1.679,p .090,112 = .240, large effect  6  Mean=2.100, SD = 1.370  Mean=2.l00, SD = 1.450  Mean= 1.600, SD = 1.506  Z(9)=.085,p=.932, r2 = .001 small effect  7a  Mean = 4.0 10, SD=.943  Mean = 4.500, SD=.707  Mean = .900, SD=1.10l  Z(9) = -1.089, p = .276,i2=.116 (moderate effect)  7b  Mean = 4.600, SD = .699  Mean = 4.100, SD = 1.101  Mean = .900, SD = .994  Z(9) = -1.186, p = .236, ri2 = .130 (moderate effect)  8  Mean = 1.900, SD = .568  Mean = 1.800, SD = .918  Mean = .100, SD = .483  Z(9) -.378, p = .705, ii2 = .015 small effect  9  Mean = 2.200, SD = .632  Mean = 2.400, SD .966  Mean = .600, SD = .699  Z(9) = -.707, p .480, ri2 = .001 small effect  10  Mean=2.9, SD 1.197  Mean= 1.8, SD =.919  Mean= 1.1, SD =1.197  Z(9)=-2.232,p= .026,i2=.356large effect  11  Mean=3.900, SD = 1.287  Mean3.800, SD = 1.229  Mean= 1.100, SD = .876  Z(9)-.l73,p=.862, i2 = .003, small effect  12  Mean=2.500, SD = 1.080  Mean=2.700, SD = 1.337  Mean= 1.400, SD = .966  F(9)=- l,p= 36 . .726, i2 = .014 small effect  Mean = 1.800, SD = 1.033  Mean = 1.900, SD = 1.101  Mean = 900, SD = .994  Z(9) = -.141, p .888,112 = .003  13  =  =  =  ,  =  175  Question Number  Experimenter  Subject  Difference  Paired-sample T-test  14  Mean = 1.900, SD = .738  Mean = 2.300, SD = 1.337  Mean = .800, SD = .844  Z(9) = -.966, p = .334, 1i2 = .090 small effect  15  Mean 2.400, sd 1.265  Mean = 2.600, SD = 1.173  Mean = .800, SD = .632  Z(9) = -.632, p = .527 r2 = .040 small effect  16  Mean2.300, SD=l.418  Mean=3.000, SD=1.944  Mean= 1.500, SD=1.509  F(9)=-1.076,p= .310,i2=.llO moderate effect’  17  Mean = 1.500, sd = 1.080  Mean = 2.200, SD = .790  Mean = 1.300, SD = .950  Z(9) = -1.425, p = .154, ii2 = .180 large effect  18  Mean = 1.300, SD = .480  Mean = 1.600, SD = .520  Mean = .300 SD = .480  Z(9) = -1.732, p = .083, i2 = .250, large effect  ,  176  E.3.2: Paired-sample t-test for local experimenter vs. subject: Question Number  Local experimenter  Local subject  Difference  Local experimenter vs. subject  1  Mean 1.400, SD .699  Mean=2.300, SD = 1.418  Mean= 1.100, SD = 1.200  Z(9)=-1.930,p= .054, i2 = .290 large effect  2  Mean = 2.050, SD = 1.2 12  Mean = 1.300, SD = .675  Mean = 1.150, SD = 1.250  Z(9) = -1.450, p .147,112 = .190, large effect  3  Mean = 1.600, SD= .843  Mean = 1.900, SD= 1.449  Mean = 1.300, SD= 1.420  Z(9) = -.422, p .673,ri2=.020, small effect  4  Mean = 3.400, SD = 1.647  Mean = 4.000, SD 1.414  Mean = 1.400, SD = 1.260  Z(9) = -.862, p = .389,112 = .070, moderate effect  5  Mean = 2.300, SD= 1.159  Mean = 3.200, SD= 1.475  Mean = 2.100, SD=.880  T(9) = -1.034, p 12S,i2.=.llO moderate effect  6  Mean = 1.400, SD = .966  Mean = 2.600, SD = 1.647  Mean = 1.400, SD = 1.520  Z(9) = -1.897, p .058, q2 = .280 (large effect)  7a  Mean = 3.900, SD = 1.449  Mean = 4.100, SD = 1.287  Mean = 1.200, SD = 1.400  Z(9) = -.259, p = .796, 112 = .007,moderate effect  7b  Mean = 4.300, SD = 1.059  Mean = 4.500, SD = .707  Mean = 1.000, SD = .940  Z(9) = -.35 1, p = .726, i2 = .010, small effect  8  Mean= 1.500, SD = .850  Mean= 1.850, SD = .818  Mean= 1.150, SD = .750  Z(9)=-.855,p= .393,112 = .070, moderate effect  9  Mean= 1.950, SD=.597  Mean=2.700, SD=l.494  Mean= 1.150, SD=1.100  Z(9)-1.552,p .12l,112=.210, large effect  10  Mean = 2.600, SD = 1.350  Mean = 2.300, SD = 1.059  Mean = 1.500, SD = .970  Z(9) = -.423, p = .672,112 = .019, small effect  =  =  =  177  Question Number  Local experimenter  Local subject  Difference  Local experimenter vs. subject  11  Mean = 3.300, SD= 1.337  Mean = 3.85, SD=.747  Mean = 1.050, SD= 1.300  Z(9) = -1.089, p .276,ri2=.116, moderate effect  12  Mean=2.000, SD = .943  Mean= 1.800, SD = .919  Mean=.800, SD = .920  Z(9)=-.333,p= .739, i2 = .010, small effect  =  178  

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
http://iiif.library.ubc.ca/presentation/dsp.24.1-0051242/manifest

Comment

Related Items