UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Using haptics to address mobile interaction design challenges : prototyping and user evaluation with… Luk, Joseph Kurachi 2006

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata

Download

Media
831-ubc_2006-0555.pdf [ 17.06MB ]
Metadata
JSON: 831-1.0051517.json
JSON-LD: 831-1.0051517-ld.json
RDF/XML (Pretty): 831-1.0051517-rdf.xml
RDF/JSON: 831-1.0051517-rdf.json
Turtle: 831-1.0051517-turtle.txt
N-Triples: 831-1.0051517-rdf-ntriples.txt
Original Record: 831-1.0051517-source.json
Full Text
831-1.0051517-fulltext.txt
Citation
831-1.0051517.ris

Full Text

U s i n g H a p t i c s t o A d d r e s s M o b i l e I n t e r a c t i o n D e s i g n C h a l l e n g e s Protoyping and User Evaluation with a Handheld Tactile Display • by Joseph Kurachi Luk B.S., Cognitive Science, University of California - San Diego, 1999 A THESIS SUBMITTED IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE DEGREE OF Master of Science in The Faculty of Graduate Studies (Computer Science) The University Of British Columbia July 2006 © Joseph Kurachi Luk, 2006 11 Abstract Current user interfaces for mobile and handheld computing platforms pr in-cipal ly offer user interaction through the visual and audi tory modalit ies. However, mobile devices are often used in contexts where vision and hearing are impaired. A t the same time, more and more functionality is being layered upon mobile devices, while the physical size of the display and keypad has remained small . T h i s l imits the rate of information that can be exchanged between the user and the system, and poses an interaction design challenge. Hapt ics offers a potential solution by providing an addi t ional modal i ty that is also especially well-suited to the demands of portable, personal devices that are in contact wi th the user's skin. In this work we identify ways that interaction through the sense of touch can enhance mobile user interfaces. We describe the synergistic process of design of user interaction concepts and novel handheld tactile display hard-ware based on the principle of piezoelectric actuated lateral skin stretch. Fol lowing the realization of the prototype hardware, we performed percep-tua l characterization studies to determine the expressive capabilities of the new device in the hands of a human user. Informed by the results from the in i t i a l user studies, we buil t and tested a handheld browser applica-t ion w i t h tactile enhancement. The results of user testing wi th the browser Abstract m applicat ion suggest that the current implementation of directional tactile s t imulat ion alone is not sufficient to enhance performance (task time) in spatial navigation; however, the user study also brought to light some en-couraging qualitative feedback and ways to improve the interaction design and haptic feedback. B y conducting a full i teration of a user-centred design process in hap-tics, we have provided a case study to inform future development efforts, a flexible platform for prototyping, and an indicat ion of promising future directions for using haptics to solve mobile interaction design challenges. Joseph K u r a c h i L u k iv Contents Abstract i i C o n t e n t s iv List of Tables x i i List of Figures x i i i Glossary x v i Acknowledgements x i x 1 Introduction 1 1.1 Problems wi th Contemporary Mob i l e Interaction Design . . . 2 1.1.1 Sensory B a n d w i d t h L i m i t a t i o n 2 1.1.2 Envi ronmenta l Compet i t ion for V i s u a l and A u d i t o r y At tent ional Resources 4 1.2 Thesis Research Questions 7 1.3 Thesis Approach and Overview 8 2 Related W o r k 12 2.1 Mob i l e Interface Design Challenges 12 2.2 Hapt ic Augmenta t ion for M u l t i m o d a l Enhancement 13 Contents v 2.3 Handheld Hapt ics 14 2.4 Perceptual Eva lua t ion of Hap t i c Devices 16 3 D e s i g n C o n c e p t s 18 3.1 Electronic B o o k Reader w i t h Vibro tac t i l e Feedback 19 3.2 E a r l y 1-D Navigat ion Concepts 21 3.3 App l i ca t i on Concepts 26 3.3.1 L i s t selection: Ringer mode applicat ion 28 3.3.2 Scroll ing: Browser applicat ion 31 3.3.3 Direc t ion signalling: Assisted navigation applicat ion . 33 3.3.4 Display of background status information and alerts . 34 3.3.5 M i n i m a l l y Intrusive Interface for R i c h Naviga t ion of M u s i c 35 4 H a n d h e l d P r o t o t y p e D e v e l o p m e n t 39 4.1 Design Phi losophy 40 4.1.1 Linear slide-mounted tactile display using piezoelec-tr ic actuated lateral skin stretch 40 4.1.2 Use of off-the-shelf hardware components 41 4.1.3 Handheld operation while connected to a host P C . . 42 4.1.4 Author ' s Cont r ibu t ion 42 4.2 System Overview 43 4.3 Output Transducers 43 4.3.1 Tacti le Output Device (Tactile Display) 43 4.3.2 Video Display 48 4.3.3 Author ' s Cont r ibu t ion 48 4.4 Sensors 49 Contents v i 4.4.1 Slider Posi t ion Sensor 49 4.4.2 P u s h to Select Sensor 50 4.4.3 Author ' s Cont r ibu t ion 50 4.5 Interface Electronics 50 4.5.1 Author ' s Cont r ibu t ion 51 4.6 Power Supplies 51 4.6.1 Author ' s Cont r ibu t ion 52 4.7 Cont ro l Software 52 4.7.1 Input and Outpu t T imings 53 4.7.2 Author ' s Cont r ibu t ion 57 4.8 Tacti le F l o w Rendering 57 4.9 Visua l iza t ion of Tact i le S t imul i 61 4.9.1 P rob lem 61 4.9.2 Novel Graph ica l Representations 62 4.9.3 Shaded G r a p h for Voltage Signal 62 4.9.4 S k i n Stretch Image 64 4.9.5 Automated Tools for Design 66 4.9.6 gif2hapticon Tool 68 4.9.7 Author ' s Cont r ibu t ion 70 5 Perceptual Characterization 72 5.1 Introduction 72 5.2 Au thor ' s Cont r ibu t ion 73 5.3 Study 1 - Range of Perceivable Stimulus Speed 73 5.3.1 Speed Study - Exper iment Design 74 5.3.2 Speed Study - Procedure 74 5.3.3 Speed Study - Results 76 Contents v i i 5.3.4 Speed Study - Discussion 77 5.4 Study 2 - Hap t i c Icon Discr imina t ion Exper iment 77 5.4.1 M D S Study - Exper imenta l Design 78 5.4.2 M D S Study - Procedure 79 5.4.3 M D S Study - Results 80 5.5 S tudy 3 - Subgroup M D S Exper iment 82 5.5.1 Subgroup M D S - Exper imenta l Design 83 5.5.2 Subgroup M D S - Procedure 83 5.5.3 Subgroup M D S - Results 83 5.6 Summary of Perceptual Character izat ion Findings 85 5.6.1 Qual i ta t ive Findings 86 5.7 Perceptual characterization findings for appl icat ion design . . 87 5.7.1 L i s t selection 87 5.7.2 Scroll ing 87 5.7.3 Direc t ion signalling 88 5.7.4 Aler t s and background status indicators 88 6 Browser Prototype 90 6.1 Design Goals 91 6.2 Low-F ide l i t y Prototype: Image-Based Browser 92 6.2.1 Design 92 6.2.2 Implementation 94 6.2.3 Image Browser User Test 95 6.3 Hapt ic Display of Web Pages 96 6.3.1 The Hapt ic Page M a p 96 6.3.2 M a p p i n g Hapt ic Icons to Page Elements 97 6.3.3 Spat ia l Layout 99 Contents v i i i 6.4 Navigat ion M o d e l 100 6.4.1 Cursor Posi t ion 100 6.4.2 Rendering Hapt ic Icons 101 6.4.3 Page Element Focusing 102 6.4.4 Graphica l Display Scroll ing 102 6.4.5 Cont ro l of Cursor Movement 103 6.4.6 Spring return to centre 105 6.4.7 H y b r i d Veloci ty / Pos i t ion Con t ro l M o d e l 108 6.4.8 Reduct ion of Slider J i t ter 113 6.4.9 Reduct ion of High-Ampl i tude , High-Frequency O u t p u t s l l 5 6.4.10 Speed L i m i t a t i o n 116 6.5 Browser Software Archi tecture 123 6.5.1 Browser Cl ient 126 6.5.2 Browser Server 127 6.5.3 Interprocess Communica t ion and T i m i n g 131 6.5.4 Browser Hap t i c Icons 132 6.6 K n o w n Software Issues and Caveats 132 6.6.1 Support for Element Height 132 6.6.2 Opportuni t ies for further software opt imizat ion . . . . 133 7 B r o w s e r U s e r E v a l u a t i o n 134 7.1 A i m s 135 7.2 Study Design 136 7.2.1 Study Variables 136 7.2.2 Normal iza t ion for Task Difficulty 137 7.3 Methodology 139 7.3.1 Recrui tment of Study Part ic ipants 140 Contents ix 7.3.2 User Test Environment 141 7.3.3 Briefing and Col lect ion of Demographic D a t a 144 7.3.4 Task Blocks 145 7.3.5 Tra in ing Sessions 148 7.3.6 M a i n D a t a Col lect ion Session 148 7.3.7 Post-Task Assessment 148 7.4 P i lo t S tudy 148 7.5 S t imul i Used in the Study 149 7.6 Dis t rac t ion Task 152 7.7 Browser Exper iment Software 154 7.8 Quanti ta t ive Results 158 7.8.1 Effect of Condi t ion on Task T i m e 158 7.8.2 Individual Subject Differences in Performance 158 7.8.3 Effect of Task on Task T i m e 160 7.8.4 Effect of Task x Condi t ion on Task T i m e 160 7.8.5 Val ida t ion of Task Difficulty Normal iza t ion 163 7.8.6 Analys is Us ing Normal ized Task T i m e 165 7.8.7 Learning / Pract ice Effects 166 7.8.8 Quant i ta t ive Val ida t ion of Dis t rac t ion Task 168 7.9 Qual i ta t ive Results 168 7.9.1 Pre-Task At t i tudes Survey 168 7.9.2 Qual i ta t ive Evalua t ion of the Dis t rac t ion Task . . . . 171 7.9.3 Qual i ta t ive Evalua t ion of the Navigat ion Task . . . . 176 7.10 Discussion 177 8 Conclusion and Future Work 182 8.1 Summary of K e y Contr ibut ions 182 Contents x 8.1.1 Identification of a novel mul t imoda l approach to ad-dressing l imitat ions in mobile user interfaces 182 8.1.2 Development of a new handheld haptics hardware plat-form 183 8.1.3 Evo lu t ion of applicat ion design concepts based on user studies and hardware development 184 8.1.4 M e t h o d for rapid prototyping and graphical represen-tat ion of tactile s t imuli 184 8.1.5 Perceptual characterization of a novel miniature piezo-electric tactile display 185 8.1.6 Handheld browser applicat ion w i t h tactile enhancement 185 8.1.7 M e t h o d for usabil i ty testing of mobile applications . . 186 8.1.8 Case study of a user interaction design process for haptics 186 8.2 Research Questions 187 8.3 Future Work: App l i ca t ion Designs for Further Investigation . 191 8.3.1 Appl ica t ions Involving Shape Rendering 192 8.3.2 General Hapt ic Icon Appl ica t ions 193 8.3.3 Spat ial Signall ing 193 8.3.4 Browser Improvements 194 8.4 Future Work: Hardware Improvements 195 8.5 Conclusion 196 Bibliography 199 A Browser User Evaluation Documents 207 A . l Task Inventory 212 Contents x i B Browser User Study Supplemental D a t a 217 C g i f 2 h a p t i c o n Code 222 D Browser Prototype Code 227 D . l Tact i le I / O Loop 227 D . l . l Da taUpdateThread .h 227 D.1.2 DataUpdateThread .cpp 228 D . l . 3 Hap t i cPageMap .h 230 D.1.4 Hapt icPageMap.cpp 232 D.1.5 BrowserShared.h 242 D.1.6 BrowserShared.cpp 243 D . l . 7 Browser X M L B i t s . h 244 D . l . 8 B r o w s e r X M L B i t s . c p p 245 D . l . 9 main.cpp 250 D . 2 V i s u a l Browser Component 254 D.2.1 browser.js 255 D.2.2 webscroller.css 264 D.2.3 browser.xul 264 D.2.4 readydialog.xul 265 D.2.5 I M y C o m p o n e n t . i d l 265 D.2.6 M y Component , cpp 266 E Browser Exper iment Software C o d e 269 E . l taskloop.js 270 E .2 taskloop.html 277 E.3 ajaxcomponent.js 279 E .4 reinforce.js 281 x i i Lis t of Tables 5.1 S t imul i used in the M D S studies 78 6.1 Values of hybr id velocity / posi t ion control model parameters. 119 6.2 Settings for the slider smoothing function using historical av-eraging 120 7.1 Browser Study Par t ic ipant Demographics 140 7.2 Measured Task T i m e by C O N D I T I O N 159 7.3 Measured Task T i m e by Subject 162 7.4 Normal ized Task T i m e by C O N D I T I O N 166 7.5 Dis t rac t ion Task Performance D a t a 170 x i i i List of Figures 1.1 Example Navigat ion Tree Prob lem 5 1.2 Thesis Research Overview 9 3.1 eBook Reader w i th Vibro tac t i le Feedback 20 3.2 Examples of Ex i s t i ng Linear Touch Input Devices 23 3.3 Bidi rec t ional Linear Touch Sensor / Tacti le Outpu t Schematic 24 3.4 Simulated Force Feedback Using a M o v i n g B u m p 27 3.5 App l i ca t ion Design Scenarios 28 3.6 L is t Selection App l i ca t i on 29 3.7 Low-F ide l i t y Foam M o c k u p 30 3.8 Browser App l i ca t i on 32 3.9 M u s i c Navigat ion App l i ca t ion 37 4.1 Hardware Overview 44 4.2 Hardware Overview, M a r c h 2005 45 4.3 Tact i le Display 46 4.4 Pos i t ion and Push-to-Select Sensors 49 4.5 E m p i r i c a l T i m i n g Measurements 54 4.6 Cont ro l Software Flowchart 56 4.7 Sk in Stretch Patterns in Na tu ra l and Ar t i f i c ia l Tacti le F low St imul i 58 List of Figures x iv 4.8 A Mi l l ipede 60 4.9 Visua l iza t ion of Tacti le S t imul i 63 4.10 Photoshop Cus tom Fi l te r settings for automated stretch i m -age generation 67 4.11 Image-Based Hap t i c Icon Design Workflow 71 5.1 Examples of S t imul i Used for the Speed Study 75 5.2 Speed Study Results 76 5.3 Waveforms used in the M D S Studies 79 5.4 Results from the M D S Study 81 5.5 Results from the Subgroup M D S Study 84 6.1 Image Browser 93 6.2 Hapt ic Page M a p 98 6.3 Scroll ing Margins 104 6.4 Slider Cont ro l Modes 106 6.5 Force Feedback Us ing Springs I l l 6.6 Veloci ty / Posi t ion Cont ro l State Diagram 114 6.7 Subtaxel Rendering Technique 121 6.8 Effective Icon Design for Subtaxel Rendering 122 6.9 Browser Software Archi tecture 125 6.10 Hap t ic Page M a p and Icons M o d e l 130 7.1 Browser User Test Environment 142 7.2 Hap t ic Icons Used in the Browser User Study 150 7.3 Effect of Task on Task T i m e 161 7.4 Effect of Task and Condi t ion on Task T i m e 164 List of Figures x v 7 .5 Relat ionship of Task T i m e wi th Presentation Order (Set of 3 Blocks) 1 6 7 7.6 Relat ionship of Task T i m e wi th P R E S E N T A T I O N O R D E R . . . 1 6 9 7 . 7 Results from the Pre-Task At t i tudes Survey 1 7 2 7.8 Qual i ta t ive Feedback on the Dis t rac t ion Task 1 7 3 7 .9 Qual i ta t ive Feedback on the Navigat ion Task 1 7 5 x v i Glossary haptic icon A l so known as a hapticon or tacton [10], a haptic icon is a brief, distinctive signal delivered to the user through the haptic apparatus, or its software representation. H i g h fidelity prototype A s compared to low-fidelity prototyping, a methodology for expedit ing the involvement of users in an interaction design process by shifting the development effort from detailed imple-mentation of features to early usabil i ty testing. In contrast, a high-fidelity prototype is one which is relatively close to the final product in terms of functionality and the level of interact ivi ty which is supported. piezo, piezo actuator Refers to an ind iv idua l piezoelectric bending motor element. position control, or pContro l In this mode, the scrolling mot ion of the cursor or page follows the slider posit ion directly. Th i s type of control is also used in jog dial or mouse wheel controls. slider In this document,' refers to the mechanism that allows the tactile display to be moved up and down. stretch image A graphical representation of the amount of gap between adjacent piezo actuators at a given point i n t ime. Assuming the user's Glossary x v i i finger has been in constant contact w i th the T D since prior to the application of signal, the levels depicted in each location of the stretch image correspond to the amount of relative skin displacement ("skin stretch") at that locat ion on the T D . Described further in section 4.9. subpixel rendering A s used in this document, refers to the technique of rendering graphics into an offscreen buffer w i t h higher resolution than the physical resolution of the display, then downsampling to the dis-play resolution using anti-aliasing filters. For graphical displays, this increases the effectively usable resolution at the cost of some sharp-ness. In this document, this term does not refer to the technique of using an L C D ' s red-green-blue subpixels to increase resolution. subtaxel rendering Equivalent to subpixel rendering for a tactile display. tactile flow The perception of the movement of a tactile stimulus across the skin over time. See Section 4.8. tactile window T h e por t ion of the page map that is currently being ren-dered to the T D . See Section 6.4.1. taxel Just as a pixel is a logical element of a multi-element graphical display, a taxel is a logical element of a multi-element tactile display. In the case of the device prototype discussed in this thesis, a taxel represents the voltage delivered to a single piezo actuator. T h e T D described in this thesis therefore represents eight taxels. T D Tacti le display. tweening In animation, refers to the creation of motion by adding inter-mediate frames that depict a gradual progression from one state to Glossary xviii another. For the tactile display described in this thesis, "tweening" refers to the successive displacement of a pattern of piezo activation (voltage levels) across adjacent piezo actuators, to create the sensation of tactile flow. velocity control, or v C o n t r o l In this mode, the slider position is mapped to the scrolling velocity. Therefore, as long as the user keeps the slider in one place, the cursor or page moves with constant velocity. This type of control is also used in joysticks and shuttle controls. voltage image (volt image) 1 A graphical representation of the voltage applied across the array of piezo actuators at a given point in time, as described in section 4.9. voltage image (volt image) 2 A n object in the STReSS library A P I which contains data in the form of voltage levels. X M L Extensible Markup Language. A generic markup language, stan-dardized by the W3C (World Wide Web Consortium). In the current project, extensible documents conforming to the X M L syntax are used for various purposes including representing tactile stimuli. X P C O M Cross Platform Component Object Model. Part of the open-source Mozilla application platform. Used in the tactile enhanced H T M L browser (Chapter 6) for inter-process communications. X U L X M L User Interface Language. A markup language for specifying the visual user interface components in the Mozilla application platform. In the current project, X U L is used to customize the Mozilla browser for use as a handheld application. x ix Acknowledgements I would like to acknowledge the guidance of my supervisor, D r . K a r o n M a c L e a n , who was always a champion for her students, working tirelessly to encourage us to .do quality, meaningful research, and to overcome our personal challenges, perfectionism included. M u c h of the research described in this thesis was the result of an ex-ceptional collaboration between laboratories at the Univers i ty of B r i t i s h C o l u m b i a and M c G i l l Univers i ty in Mont rea l . In particular, Jerome Pasquero and I part icipated i n an extended exchange which resulted in numerous learning experiences since we each brought a different, sometimes opposite, point of view to the research problems. I am grateful to Jerome and other members of the M c G i l l Hapt ics Lab , especially Professor Vincent Hayward , Vincent Levesque, and Q i Wang, for their patience i n engaging me in end-less scholarly debates, their warm hospitality, and their dedication to the project. The contributions of D o n Pavlasek and Jozsef B o k a to the mechan-ical design and fabrication are also sincerely appreciated. T h e members of my thesis committee, D r . Rodger Lea , D r . R o n Rensink, and especially D r . Steven Wolfman, gave valuable feedback on this research. F ina l ly , I would like to express my application to M a r i o Enr iquez for help w i t h the M D S testing methodology, and to Steve Yohanan for sharing his operating system expertise. 1 Chapter 1 Introduction Mobi le , portable, and handheld computing environments offer significant promise for the future of interactive computat ion. Compared to a typica l desktop computer, a mobile device enables opportunist ic uses for informa-t ion that stretch the boundaries of the popular definition of data: instead of working wi th documents and files, mobile devices are called upon to man-age an increasingly varied collection of data, from voice calls to photos and location based information snippets. T h e increasing popular i ty of mobile de-vices also represents a continued shift away from the physical instantiat ion of data and towards an increasingly miniatur ized and networked medium. Information that was once tangible, in the form of paper, magnetic tapes or other media, is now invis ibly contained wi th in the device used to access it or s imply delivered over ubiquitous, wireless networks. T h e dissociation of information from its physical encapsulation is driven by pressures to improve por tabi l i ty by reducing device weight and volume. However, interactive applications can never be completely removed from physical constraints due to the necessity of producing a human sensory per-cept . 1 Furthermore, the history of computer user interfaces shows that ^rain-computer interfaces enable the possibility of a completely non-physical user interface, but are too poorly understood at the present time to consider them a viable alternative in mobile device interaction. Chapter 1. Introduction 2 significant progress in expanding the breadth of accessibility to computers can be achieved when interfaces, such as the graphical user interface ( G U I ) , leverage the intuit ive physics [38] that evolved in human beings as a means for dealing w i t h the environment. T h e design of effective mobile information devices is thus confronted w i t h a classic di lemma: balancing the power of abstraction and freedom from physical constraints against the usabil i ty advantages of a system that interacts effectively wi th human beings on a natural physical scale. A l l too often, increased mobi l i ty comes at the cost of decreased usability. 1.1 Problems with Contemporary Mobile Interaction Design The user interfaces of contemporary mobile devices suffer from sensory band-width limitation and environmental competition for visual and auditory at-tentional resources. Ar t i f i c i a l haptics, incorporat ing synthetic tactile sig-nall ing, may be a useful tool for solving these problems. 1.1.1 Sensory B a n d w i d t h L i m i t a t i o n Throughout most of the 1990s, there was a steady decrease i n the size (vol-ume and weight) of personal mobile information devices such as mobile phones, personal digi ta l assistants ( P D A s ) , media players and laptop com-puters, spurred by steady technological progress in integration of electronic circuits, portable power, and digi ta l wireless transmission. However, the trend towards minia tur iza t ion has since leveled off i n many categories of devices, despite continued progress in hardware integration and anecdotal Chapter 1. Introduction 3 evidence from users that the por tabi l i ty of ever smaller devices is highly desirable. The conventional wisdom is that input and output devices such as keypads and screens can not be made smaller without negatively impact-ing usability. Thus the factors l imi t ing device por tabi l i ty have ceased to be technical i n nature; instead, user interface considerations now dictate the pract ical l imits to minia tur iza t ion of today's mobile devices. For the increasingly complex tasks being performed on mobile devices, the information capacity of the smal l slice of the visual field covered by a smal l L C D display has become a l imi ta t ion . Various head-mounted displays such as eyepieces and goggles have been developed to increase the effective display area while retaining portabil i ty, but thus far they have not proven widely pract ical because of their intrusiveness. Similar ly, most mobile information devices today include an audible transducer and microphone input, but aside from making phone calls and l imi ted command-based voice recognition, information-rich auditory user in -terfaces have yet to be demonstrated on mobile devices. T h e use of mult iple simultaneous sensory modalities, known as multi-modal interaction, offers a method to increase the available user interface bandwidth, or information volume per unit t ime that can be exchanged be-tween user and device. [12] W i t h visual and auditory modalit ies already re-ceiving widespread research interest, and taste and smell being thus far relatively impract ica l as user interface modalities, the domain of touch ap-pears to be a useful area for study as an addi t ional modal i ty to enrich the information carrying capacity of the user interface of a mobile device. A n abstract example of an interaction design problem created by l imi ted sensory bandwidth in mobile devices is shown in Figure 1.1. For a given in-terface w i th a number of functions, "porting" the applicat ion to a mobile Chapter 1. Introduction 4 device from a more conventional personal computer imposes l imitat ions in the amount of information that can be pract ical ly presented at any one level of navigation hierarchy. Designers wishing to preserve functionality are therefore forced to implement "deeper" navigation hierarchies requiring more user interaction to access the desired function. T h e increased naviga-t ion overhead, in turn , decreases the pract ical i ty of the applicat ion in the hands of a busy, mobile user. W h i l e this example may seem simplist ic, it is a relatively accurate description of the problems wi th the first genera-t ion of mobile phone applications based on the wireless applicat ion protocol ( W A P ) . 1 . 1 . 2 E n v i r o n m e n t a l C o m p e t i t i o n f o r V i s u a l a n d A u d i t o r y A t t e n t i o n a l R e s o u r c e s Haptics may be especially useful for mul t imodal interaction in a mobile use context because it does not share the action-at-a-distance property of visual and audi tory signalling, and is a more "parallel" sensory modal i ty than vision and hearing. W h e n moving through the environment, a person typica l ly uses visual and auditory modalities for navigation, avoiding obstacles, and learning about the environment. A n y system that requires those modalit ies for user interaction must compete for the user's attention wi th information originat-ing from the environment. Compet i t ion for l imi ted visual attention can present serious safety issues: for example, whereas use of a mobile device while walking on a crowded street might be marginal ly acceptable, use while dr iv ing is becoming an increasing public safety concern, and use while landing an aircraft might be Chapter 1. Introduction 5 large-screen navigation tree equivalent small-screen navigation tree ; ; ? ' .* * x ill 111 Figure 1.1: Examples of deep versus wide navigation trees. The limited sensory bandwidth available at any level of the hierarchy forces a "deep" configuration, which requires more effort to navigate. Chapter 1. Introduction 6 considered completely unacceptable [49]. In practice, the user is only able to devote a por t ion of their visual attention to a mobile device, and repeatedly shifts their attention between device and environment [40]. It follows that a mobile user interface can not be guaranteed to deliver a visual cue to the user unless the information is retained onscreen long enough for the user to direct their attention to the display. Th i s reduces the interface bandwidth relative to desktop computers even further than a simple comparison of screen size would suggest. A u d i t o r y interfaces suffer from similar l imitat ions. A very noisy envi-ronment such as a crowded street presents challenges in making the sounds produced by the device audible, as well as recognizing sounds produced by the user. A u d i t o r y interfaces therefore scale poorly to large numbers of densely packed users. O n the other hand, in a very quiet environment, such as a meeting or library, use of an auditory interface might be completely un-acceptable because it would disturb others. Headphones and microphones that isolate exterior noise improve user-device communicat ion, but only at the expense of user-environment communicat ion, causing similar concerns as visual impairment. In comparison to vision or hearing, the sense of touch is relatively un-obstructed in a mobile use context. The vestibular system is used for bal-ance, and the feet or hands may be in contact w i th control surfaces, but other areas of skin are likely to be available for interaction. Unl ike vision or hearing, which must be sensed through discrete organs, touch receptors are dis tr ibuted throughout the body and active simultaneously — thus, the system can be characterized as more "parallel" in nature. Tact i le signals are only active at the point of direct contact, enabling discreet interaction. In addit ion, a portable device is also in contact w i th Chapter 1. Introduction 7 the user's body more often than a desktop computer, making it a good platform for tactile interaction. Indeed, a non-multimodal, purely haptic interface is conceivable and is currently used in limited contexts such as vibrotactile signalling of incom-ing calls. However, haptics can not be considered a generic replacement for visual or auditory interfaces, since each modality has its own profile of ap-plications for which it is optimally suited. Work is ongoing to define novel application spaces for haptics, including affective, subconscious, and inter-personal interfaces. Within the scope of this thesis work, however, the po-tential benefits of haptic augmentation to existing mobile applications with demonstrated, proven utility will be explored. This will enable effort to be focused on implementing a high-fidelity prototype, including hardware de-velopment, application conceptualization, and usability testing, completing a full iteration of an interaction design cycle within the scope of a master's thesis. 1.2 Thesis Research Questions The goal of this thesis research is to increase the understanding of the ap-plication of haptics to a mobile computing context. The following research questions are addressed: 1 . What are the problems with existing mobile user interfaces that may be addressed using haptics? 2. How can one implement haptics on a mobile device despite power, size, and weight restrictions? Chapter 1. Introduction 8 3. What are the expressive capabilities of the hardware prototype in the hands of a human user? 4. What are the engineering challenges associated with building a high-fidelity hardware and software prototype with handheld use in mind? 5. Is tactile flow an effective aid for user navigation? 6. How can user-centred design methodology be applied to haptics, where hardware technology is still the primary determinant of user interface capabilities? These questions are revisited in the Conclusion (Chapter 8) which sum-marizes the research findings. 1.3 Thesis Approach and Overview T h e approach that was used to address the research questions identified in the previous section was to create a handheld haptics system in a user-centred, iterative fashion by incorporat ing user input as early as possible i n the design process. The stages of this process are shown in Figure 1.2. Because the a im of the thesis research was to use this process to better un-derstand the potential for haptics in a mobile user interface, rather than to produce a fully engineered product, pr ior i ty was given to creating a handheld haptic user experience over a fullly-functional device; if its usefulness could be demonstrated, then future work could address the engineering challenges related to true mobili ty, such as battery power and custom integrated cir-cuits. Therefore, early in the design process the heuristic was adopted that the prototype should attempt to replicate as much of the handheld user ex-perience as pract ical wi th in the resources available for the research, while Chapter 1. Introduction 9 Background Present Work Future Work User Needs Knowledge of — i Existing Hardware Usage Scenarios & Application Concepts Hardware Design Perceptual Characterization Further Hardware Development i Further •*• Stimulus Discovery and Perceptual Characterization Application Design I Application Usability Testing • New Applications I ^ Further Refinement of Usability Testing Practices Conclusions and Recommendations J for Future Application Design Figure 1.2: Overview of the design process used for this thesis research. Chapter 1. Introduction 10 remaining tethered to a host P C and easily customizable; however, it should have a power, weight, and volume profile such that future integration into a wireless mobile device could be conceivable using known engineering tech-niques. Further details of this design decision are provided in Chapter 4, Hardware. Rather than specialized haptic-only applications or, applications for users wi th visual or auditory impairment, we have chosen to focus on the general area of haptic augmentation of applications similar to those currently em-ployed on mobile devices, because there is a clearer understanding of user needs and an opportuni ty to contribute to ongoing mobile development pro-grammes. Therefore, for the purpose of the studies described in this thesis, the users are a general audience. T h e hardware development and perceptual characterization stages were performed in collaboration w i t h Professor Vincent Hayward , and graduate students Jerome Pasquero, Vincent Levesque, and Qi Wang of the M c G i l l Univers i ty Hapt ics Laboratory; and undergraduate research student Shannon L i t t l e of U B C . T h e contributions of each collaborator are noted in appro-priate sections throughout the thesis. T h i s thesis is organized into the following sections: C h a p t e r 1 - I n t r o d u c t i o n describes the problem that is addressed in this thesis, and the motivat ion for the research. C h a p t e r 2 - R e l a t e d W o r k discusses the existing research in mobile and haptic interfaces. C h a p t e r 3 - D e s i g n C o n c e p t s describes the in i t i a l conceptual design work that was performed to better understand the research domain Chapter 1. Introduction 11 and to determine the appropriate applicat ion and hardware specifica-tions for further development. Chapter 4 - Hardware Prototype Development covers the engineer-ing process that was performed to create a handheld haptics platform wi th an integrated piezoelectric tactile display. In addi t ion to elec-tromechanical design, this chapter also includes a consideration of the method for designing and representing tactile outputs, and a descrip-t ion of the control software architecture. Chapter 5 - Perceptual Characterization describes user experiments conducted before detailed applicat ion development, to understand the capabilities and l imitat ions of the device in producing a salient tactile percept. Chapter 6 - Browser Prototype describes an in-depth, high-fidelity pro-totype of a mobile browsing applicat ion that was buil t using the hand-held haptics platform. Th i s chapter also includes the results of an in i -t ia l exploratory user test, and hardware customizations for the brows-ing applicat ion. Chapter 7 - Browser User Test presents a controlled user study designed to assess the performance characteristics of the hapt ical ly enhanced browsing applicat ion. New software created to manage the usabil i ty test, and a distraction task model, are also described. Chapter 8 - Conclusion returns to the research questions identified i n the previous section and considers how they were addressed through the activities conducted for this thesis research. 12 Chapter 2 Related W o r k In this chapter we describe the previous research that informs the present study as it concerns mobile interface design challenges, haptic augmentation for mul t imoda l enhancement, handheld haptic technologies, and perceptual evaluation of haptic devices. The decisions to focus on haptic augmentation for general mobile devices, to use piezoelectric lateral skin-stretch technol-ogy, and to use the perceptual characterization methodology are justified based on the existing knowledge in the aforementioned domains. 2.1 Mobile Interface Design Challenges The hypothesis that l imi ted sensory bandwidth due to smal l screens and keypads restricts mobile interface designs is based largely on widespread anecdotal evidence and the experience of the author i n designing mobile in -terfaces professionally. A s mobile devices are a field of high commercial in -terest and development, many usabil i ty studies are only published privately wi th in companies [53]. A t the same time, every research paper relating to mobile interface design begins wi th a description of the challenges posed by l imi ted user interface bandwidth , resulting in the need for reduced and restructured content relative to desktop interfaces [6]. T h e need for more expressive interfaces that use physical, tactile affordances to support a con-Chapter 2. Related Work 13 stellation of computing devices was art iculated in the famous Tangible B i t s paper by Ishii and Ul lmer [24]. L i m i t e d visual attention in mobile use contexts has been described in [40]. B y following mobile users in realistic usage environments, Oulasv i r t a et a l . observed rapid and frequent visual attention switching between device and environment as users attempted to navigate while using a mobile browser. Based on broad field evidence, a related paper by Ro to and Oulasv i r t a [47], suggests that non-visual, and par t icular ly haptic, interfaces are needed i n mobile applications to overcome the problems associated wi th loss of visual attention during system wait states. 2.2 Haptic Augmentation for Multimodal Enhancement Given the attention-demanding nature of tasks performed on a handheld device in a mobile environment, studies of mul t imoda l enhancement of in -terfaces under workload are relevant. In particular, C h a n et al . [12] demon-strated that haptic signals could be learned and recognized despite distrac-t ion that placed a demand on the users' attentional resources. T a n et al . [50] demonstrated that performance in an attention-demanding vehicular navi-gation task could me improved wi th haptic feedback applied to a person's back. In a navigation task, Dennerlein [17] demonstrated that haptic force-feedback could be used to improve targeting performance. Similar ly, a patent by Novin t Technologies [7] covers arbi trary force feedback produced when the user pushes "into" a screen boundary, for non-visual feedback of scrolling. Directional stimulation is a key component of these applications, Chapter 2. Related Work 14 and is achieved in existing systems either through force feedback or spa-t ia l ly dis tr ibuted vibrotacti le s t imulat ion [50]. Us ing mult iple actuators to provide a directional signal provides significant increased u t i l i ty over simple vibration-([27, 31] and Immersion Corp . patent [9]). These examples support the feasibility of the applicat ion scenarios that are proposed later, including the use of directional and tactile feedback (but not through spatial patterns of vibration) to potential ly enhance targeting and spatial navigation in an interface. 2.3 Handheld Haptics W h i l e there is promise for the use of haptics on a mobile device, there are relatively few examples of functioning implementations. Some underlying difficulties are listed below. 1 • Lack of mechanical grounding. A p p l y i n g force-feedback to a user re-quires a fixed mechanical ground. In a mobile context, the forces must be created relative to the user, which imposes constraints on the physical design and force output capabilities. A n alternative is tactile display, which generates no net force on the user, but consequently l imits the scale of sensations transmitted. • Stringent power, size, and weight constraints apply in mobile contexts. Use of a conventional motor for force-feedback introduces a significant impact on a l l three. [44] • Since relatively few instances of integrated, r ich haptic feedback exist today, it is difficult to justify inclusion in a mobile device unt i l there 1 Portions of this section have been previously published in [34]-Chapter 2. Related Work 15 is a better understanding of the added value it creates for the user. T h e most common occurrence of haptic feedback i n mobile devices to-day is the ubiquitous mobile phone or pager vibrator . Patterns of v ibra t ion are typical ly used to indicate various alerts, such as an a larm or incom-ing cal l . Recently there has also been commercial and research interest i n put t ing v ibra t ion in more sophisticated applications [13, 14, 32]. Generally, vibrotact i le s t imul i are produced globally (across the entire device) and wi th only two levels (on or off), vibrotacti le devices generally do not afford b id i -rectional interaction in the sense of the user actively exploring information through movement and the sense of touch [23, 48]. Devices that are capable of delivering grounded forces to the user have the potential for greater expressive capacity than vibra t ion. Designs are restricted to min ima l degrees of freedom (DoF) [35], yet must create enough added value to justify the power, size, and weight tradeoffs. Piezoelectric actuation offers significant promise for mobile applications because it can achieve a smaller form factor without coils and magnets. Poupyrev et al . of Sony's Computer Science Laboratories used piezo el-ements to produce vibrotacti le actuation of handheld devices or parts of them [46]. In the case of a touch screen [45], the user typica l ly experiences the i l lusion of local actuation although the entire screen moves; this type of vibrotact i le actuation of a flat surface is also mentioned in a patent by New Transducers L i m i t e d [8]. Creat ing true mult iple loci of actuation on a smal l scale is significantly more complicated using vibrotact i le signals [46]. Piezoelectric actuators may be configured i n a way that also produces non-vibrotactile, low-frequency skin s t imulat ion [23]. W h e n the user places his /her finger on an array of actuators which collectively comprise a mul t i -Chapter 2. Related Work 16 element tactile display, the relative motion of the ind iv idua l piezo tips stretches the skin locally, act ivating skin mechanoreceptors. A p p l y i n g spe-cific patterns of distr ibuted skin deformation can create the i l lusion of touch-ing small-scale shapes and textures. A device based on this technology, called the V i r t u a l Bra i l l e Display ( V B D ) [30], has been used to render legi-ble Bra i l l e dots using only lateral stretching of the skin. S imi lar sensations can be achieved using technologies that push into the skin [51], but the lateral skin-stretch configuration is mechanically simpler and makes the most efficient use of the range of motion of commercial ly avail-able piezoelectric bending motors [15], resulting in favourable power, size, and weight profiles. Such a configuration also provides internal mechanical grounding, as forces are generated between adjacent piezo elements. We thus eventually chose lateral skin-stretch as the most promising con-figuration for the hardware prototype. Our approach uses the same basic principle as the V B D , but miniatur ized and embedded in a handheld form factor wherein the skin-stretch site is displayed to the users thumb, and mounted on a slider. The device is described in further detail later. 2.4 Perceptual Evaluation of Haptic Devices W h e n a novel haptic transducer is created, prior to the development of full-b lown applications it is first helpful to understand the perceptual character-istics of its output, so that signals can be created that best use the device's expressive capabilities. The perceptual evaluation procedure described in Chapter 5 is based on work described by Enriquez and M a c L e a n [18], using the mult idimensional scaling ( M D S ) technique to categorize haptic icons, and the subgroup M D S technique has been further analyzed in [42]. The Chapter 2. Related Work 17 procedure used in this thesis for designing the haptic signals also loosely follows the work by Brewster and B r o w n on tactons [10, 11]. 18 Chapter 3 Design Concepts T h i s section describes applicat ion concept development that followed from the background research into the problem domain and existing haptic ap-proaches. 1 Design concept development proceeded i n stages over the course of a year; the existence of the piezoelectrically actuated lateral skin stretch tech-nology was not known unt i l after in i t i a l hardware and applicat ion concept development had been done. Instead of creating applications for a part icular hardware specification (technology-centred design), we sought to leave the hardware specification as open-ended as possible while focusing our efforts on understanding how different applicat ion designs could meet user needs. O f course, no application, especially haptic, can exist in a vacuum wi th no dependency on hardware. Therefore, dur ing applicat ion concept develop-ment we also gave consideration to the minimum set of hardware features necessary to make the specified interaction possible. For the present project, we also had to keep in mind several pract ical considerations such as access to manufacturing facilities and setting an appropriate scope for a master's thesis project. In these ways, the interaction design and hardware design Sections 3.3.1 through 3.3.4 have been previously published in [34] and represent the collaborative work of authors on that paper. Figures 3.6, 3.8, 3.9, and Section 3.3.5 have not been previously published. Chapter 3. Design Concepts 19 evolved synergistically, w i th the interaction needs being the pr imary driver. In this chapter we describe the maturat ion of design concepts from an open-ended concept of s imulat ing the haptic experience of reading a book, to a sl ightly more t ight ly specified concept of 1-D navigation wi th haptic feed-back. Conceptual prototypes, including low-fidelity physical mockups and detailed usage scenarios, were used to drive the design process by elucidat-ing details of the user interaction. F ina l ly , we converged on a design based on 1-D interaction model, w i th piezoelectric lateral skin-stretch technology used to enable the application concepts. 3.1 Electronic Book Reader with Vibrotactile Feedback A n early design effort focused on haptic augmentation of the experience of using a mobile, handheld electronic book reader. T h e promise of a portable electronic device for reading information has been long recognized [26], but adoption of electronic readers has been hampered by poor display quality, weight and power, and slow navigation compared to t radi t ional books. W i t h the display quality, weight and power considerations being slowly mit igated due to technological advances, what remains is the user interface problem. F l i p p i n g through a paper book is a r ich haptic experience, t ransmit t ing information such as the relative location of a page in the book, the speed of movement through pages, and even the usage history (books tend to natural ly flip open to well worn sections). We explored the concept of artificial haptic signalling to support book navigation by constructing the prototype shown i n Figure 3.1. A vibrotact i le transducer (a smal l speaker) was mounted on a rubber block that was shaped Chapter 3. Design Concepts 20 Figure 3.1: (A) Device for providing vibrotacti le feedback to simulate the experience of page-flipping. The rubber nipper is mounted on a mini-joystick and can articulate in the up /down direction w i t h spring return to centre. (B) T h e device being held in the hand. Note that the rubber edge meets the thumb at an angle s imilar to that of the edge of a book when pages are being flipped. Chapter 3. Design Concepts 21 to simulate the angle of a book's edge when the device was held in the hand. The rubber "flipper" was mounted on a spring-loaded miniature joystick found in a typica l P C game controller, and its mot ion restricted by a custom-machined plate so that the flipper could only move up and down. Fina l ly , the apparatus was enclosed in a soft formed foam hand-grip (Crayola M o d e l Mag ic ) . T h e concept was that the user would use his/her thumb to t i l t the rubber flipper up and down, producing a velocity signal which, relayed through the off-the-shelf U S B game controller, would cause repeated page-up or page-down signals to be sent to electronic book reader software. Feedback would be provided for each page-turning event in the form of a perceptible click caused by the vibrotacti le transducer, allowing the user to feel their move-ment through the pages of an electronic book. F ina l ly , well worn pages would be simulated by a combination of stronger clicks accompanied by appropriate delays in the navigation model . T h i s concept was a first-iteration rapid prototype intended to study the ergonomic experience of a device wi th potential for integration into a mobile / handheld environment. T h e speaker itself d id not generate sufficient force for a factually perceptible click. A l though it could have been upgraded wi th a voice coi l or similar device w i t h stronger force, attention shifted to a more versatile 1-D scroll ing strip concept that could incorporate more expressive feedback. 3.2 Early 1-D Navigation Concepts In a typ ica l feature-packed mobile device, much of the chassis that faces the user (the X — Y plane) is covered wi th input and output transducers Chapter 3. Design Concepts 22 such as keypads, displays, and speakers. Furthermore, there is pressure to minimize the thickness of such devices in the Z direction, in order to mainta in a form factor that fits unobtrusively against the body in a pocket or handbag. These factors make it difficult to incorporate a haptic device on the front or back surface of a mobile device. More accessible locations are the side surfaces, which are both relatively unused and also somewhat mitigates concerns about the physical depth of a haptic transducer, which may extend somewhat inside the case without significantly altering the typica l aspect ratio of a handheld device. In a typica l box-shaped handheld device, this locat ion is accessible to the user's thumb, which sweeps out an arc that can be used for 1-degree of freedom, linear input . Recently, several commercial products have been introduced that uti l ize capacitive touch sensors to digitize 1-D input (Figure 3.2). T h i s provided the inspirat ion to use the side surface of a device as a bidirect ional haptic transducer. A conceptual schematic of a bidirectional touch input / tactile output system is shown i n Figure 3.3. The tactile transducer component is assumed to be of the type that produces physical deformations perpendicular to the surface of the finger, and could be implemented using any of a variety of technologies. T h e following technologies were considered as candidates for the tactile transducer: • E l e c t r o m a g n e t i c a l l y a c t u a t e d p i n s produce local deflection and and form the basis for conventional Bra i l l e displays. W i t h an appro-priately chosen flexible covering acting as a low-pass filter, they can also be used to render smooth shapes. Investigation was made into using the mechanism from a dot-matr ix printhead to create minia-Chapter 3. Design Concepts 23 Figure 3.2: Examples of existing products u t i l iz ing linear capacitive touch sensors. Chapter 3. Design Concepts 24 Capacitative Tactile Click Position Transducer Switch Sensor Figure 3.3: Schematic diagram of a concept bidirect ional haptic device for 1-D linear input and tactile output. A flexible sensor for determining the posi t ion of the user's finger is layered on top of a deformation-producing tactile output transducer, and mounted on a movable apparatus so that the force of the user's finger pressing into the device can be detected for actions such as selection. (Components not drawn to scale) <:o:o:o:o:o:o:»—| o o QQQ000O—| O O Chapter 3. Design Concepts 25 ture, high-resolution deflections. Unfortunately, the drawbacks to elec-tromechanical actuation are high power consumption and the weight of the coils. • Just as piezoelectric speakers can be used as alternatives to electro-magnetic speakers, a flat, segmented array of p i e z o e l e m e n t s could produce the tactile deflections. However, conventional piezo elements are made for producing audio, which has both higher frequency and lower ampli tude than required for producing tactile sensations beyond vibrat ion, which is poorly suited to dense spatial signalling. • Smal l quantities of special m a g n e t o - o r e l e c t r o - r h e o s t a t i c fluids may be configured as an array of separated cells. W h e n a current or magnetic field is applied to a cell , the encapsulated l iqu id hardens, cre-at ing a tactile sensation when the user explores the array. Concerns wi th this approach include long-term rel iabi l i ty of the fluids and mem-brane, exotic micro-machining requirements, response time, crosstalk between cells (especially w i th the magnetorheostatic approach), and potential ly l imi ted amplitude. • S h a p e m e m o r y a l l o y s ( S M A s ) may be used in any number of configurations, such as b imorph benders, to produce local deflections. However, because the motion is temperature-related, crosstalk, heat-sinking, and especially reaction t ime remain significant concerns for a spatial array configuration. Other considerations relate to the posit ion sensing layer, which must be flexible if used on top of a deflection producing layer. Depending on the actuator technology, it must also be able to cope wi th magnetic and/or elec-Chapter 3. Design Concepts 26 trie fields, which would cause problems for existing capacitive touch sensors. T h i s could potential ly be cancelled out in software, though at a cost of effec-tive resolution. A n alternative configuration consists of put t ing the sensing layer underneath (i.e., separated from the user's finger by) the actuation layer, which could eliminate the flexibility requirement but potential ly ex-acerbate the interference problem. For the purposes of rapid prototyping for experimentation, preparations were made to mount a posit ion sensing coil on the user's finger and to use a distal ly mounted drawing tablet (which works by radio frequency sensing at a distance of up to a few centimeters). Leaving the specific choice of actuator and sensor technology open for the moment, interaction concepts were developed; an example is shown in Figure 3.4. Assuming a device that can produce localized orthogonal deflec-t ion while t racking the user's finger posit ion w i t h reasonable t ime accuracy, force-feedback may be simulated without resort to motors that physical ly push against the user's finger in the lateral direction. The concept is mod-eled on the sport of surfing, where a continuously moving ramp provides a lateral force on the rider due to the action of gravity and slippage along the surface. Similarly, a ramp that tracks the user's finger and provides contin-uous "uphi l l " resistance may be used to provide force feedback for a variety of interaction designs such as those described in [17] that improve steering and navigation. 3.3 Application Concepts Under the working assumption of a linear, 1-degree of freedom bidirectional haptic device capable of delivering several distinguishable tactile s t imul i combined w i t h finger posit ion sensing, several applicat ion concepts were Chapter 3. Design Concepts 27 i i i i m Bump tracks position and moves to stay ahead of finger i i i i i i i i n Figure 3.4: Simulated force feedback using a moving bump that tracks the user's finger, providing continuous "uphi l l " resistance. Chapter 3. Design Concepts 28 developed to better understand how such a device might be useful in mobile interaction design. For ease of understanding each concept, a contextual scenario wi th a fictitious example user is used to motivate the application. Hyperlink sport. iu*dji»t> Football Hockey Basketball Auto Racing Curling Cricket ATLANTA - The BravftS ended a (out-game losing GtrsaK today with a 2-1 wtn over Detroit in over-time. The stadium was (b) M o v e Target forward X (c) Iff ® Dean « offline (d) Figure 3.5: Overview of four application design scenarios, (a) Lis t selection, (b) Scroll ing, (c) Direct ion signalling, (d) Background status notification. T h e figures shown as callouts represent haptic icons. Figure by Jerome Pasquero and Joseph L u k . 3.3.1 L i s t s e l ec t ion : R i n g e r m o d e a p p l i c a t i o n (Figure 3.5, a, and Figure 3.6) Linda is in a meeting and wants to set her phones ringer mode discreetly. Grasping her phone inside her purse, she explores the ringer mode menu by moving the selection highlight while receiving tactile feedback. Each menu item feels unique, like touching objects with different shape and texture, and she recognizes the sensation of the silent menu item because she has used this function before. She selects the silent mode and receives tactile feedback as confirmation. The scenario illustrates one way we can employ haptic icons [18], or Chapter 3. Design Concepts 29 tactons — [10] brief, artificial tactile s t imuli to provide assistance in mak-ing selections. A unique tactile stimulus is assigned to each i tem in a list menu; w i th repeated feedback, users quickly associate functional meanings to abstract haptic icons [11, 12]. ooo Tactile stimuli Figure 3.6: Example mul t imoda l screen and haptic design for the list se-lection applicat ion concept. Dis t inct haptic icons are given an onscreen representation in this example as an associative aid. T h e piezo tactile display technology described previously is capable of displaying smal l simulated surface features, such as bumps and gratings, w i t h arbi trary motion relative to the users finger. It promises a r ich vocab-Chapter 3. Design Concepts 30 Figure 3.7: Low-fideli ty foam mockup used for early user evaluation of the menu selection application concept. There is a textured plastic tactile strip mounted on the left side of the mockup. Chapter 3. Design Concepts 31 ulary of haptic icons, which are later characterized in this document. B y mount ing the tactile display on a slider that is also sensitive to thumb pressure, it becomes an input device. T h e user can select items i n a vert ical list menu by moving the display up and down. A s the selection highlight is moved, the haptic icon associated wi th the selected list i tem is felt. Kines-thetic awareness of finger posit ion allows the user to operate the device without looking, and to make a selection using the tactile display. To gather information on user's impressions of this appl icat ion concept, we constructed a low-fidelity foam mockup (Figure 3.7) and asked people to hold it i n their hands and to imagine using the tactile sensations to help them make menu selections. The subjective feedback was generally positive and helped guide our prototype hardware development (Chapter 4). 3.3.2 S c r o l l i n g : B r o w s e r a p p l i c a t i o n (Figure 3.5, b , and Figure 3.8) Bob checks the sports news and scores many times each day. He didn't like using his old mobile phone's browser for this because he had to scroll around a lot to view content, which made him often lose his place. Bob accesses a sports website using his new haptically-enabled phone and scrolls down into a news story. He feels the sensation of his finger sliding over a textured surface while the text of the story moves up the screen. As he continues to scroll, he feels the headline of the next story (a distinct bump) and some links (each vibrates gently as it is highlighted). All the stimuli move smoothly past his finger in sync with the scrolling movement. Having scanned the page, Bob scrolls back up and quickly locates his area of interest (his home team's standings) aided by the memory of what that part of the Chapter 3. Design Concepts 32 Figure 3.8: Example mul t imodal screen and haptic design for the browser applicat ion concept, (a), The port ion of the page currently visible is shown as a highlighted region of the scroll bar. (b), Salient page elements are represented as haptic icons and shown in the scroll bar as an associative aid. Chapter 3. Design Concepts 33 page feels like. Small-screen mobile devices typical ly require more scrolling and/or selec-t ion actions to navigate a deep rather than wide information layout. B o t h place demands on the users visual attention. Hapt ic augmentation as v i -brotactile feedback has been shown to improve performance in a handheld scrolling task [46]. However, a compact multiple-element tactile display of-fers addi t ional capabilities such as smooth tactile flow rendering (a sensation moving across the skin). Different page elements, such as headings, images, and links, can be rendered as haptic icons that are played when the user scrolls over them. Thus, each page has an associated haptic map that reflects its structure. Users learn to recognize familiar pages and can quickly scroll to desired sections or l inks. Improvements in scrolling efficiency would encourage user behaviours such as scanning to understand page structure and context, and increase the amount of information that can pract ical ly be presented on a page. 3.3.3 Direction signalling: Assisted navigation application (Figure 3.5, c) Mary is looking for a toy shop at a large, crowded shopping mall. Her location- and orientation-aware mobile device helps her find the shop with an active map and directions. The device also provides haptic feedback so she doesn't have to constantly look at the screen, allowing her to keep her eyes and ears on her surroundings. Mary holds the device discreetly at her side, with her thumb resting on the tactile display and pointing forward. The tactile display repeatedly strokes her thumb in the reverse direction (towards her back), indicating that the Chapter 3. Design Concepts 3 4 device is pointed in the opposite direction from her destination. As she turns around, the sensation gradually weakens, then begins to build again in the opposite, forward direction; she is now correctly oriented. Mary starts walking while continuing to hold the device. The stroking becomes faster, indicating that she is approaching her destination. A n y applicat ion that assists the user in finding a spatial target could uti l ize an expressive tactile display to convey a direction cue. O n a macro scale, this includes vehicle-based or walking navigation tasks, where the user must travel to a destination. O n a small scale, the user could receive haptic assistance to orient a mobile device camera so image-recognition software can read a barcode or scene. Appl ica t ions in between include finding wireless access points or other active distr ibuted information devices, or people i n a search-and-rescue scenario. Vibro tac t i l e s t imulat ion at distr ibuted points of the body has been con-sidered for navigation [20], but in a non-intrusive handheld form factor, the display of tactile flow can be used to indicate 1-D direction. Other parame-ters (e.g. speed, ampli tude, and wave shape) add information dimensions. 3.3.4 D i s p l a y o f b a c k g r o u n d s ta tus i n f o r m a t i o n a n d a le r t s (Figure 3 . 5 , c) Albert always feels in touch with his friends because they all share "presence" [39] and location information with each other via their mobiles, with status notifications as they become busy or free. Albert is composing a text message to a buddy. His fingers are busy entering text, but occasionally he places his thumb on the tactile display to move the cursor, and feels a subtle repeating haptic icon that indicates his friend Steve has come online. Albert can continue with his task, aware of Chapter 3. Design Concepts 35 the status change. Later, Albert is on the phone when his friend Dean goes offline. Albert feels a different haptic icon and is aware of Dean's status without having to interrupt his conversation or to remove the phone from his ear to look at the display. Hapt ic alerts are commonly used on mobile devices, signalling events such as incoming or dropped calls [16]. Simple, high-amplitude signals such as v ibra t ion can be perceived through clothing and on various areas of the body, but more expressive tactile s t imulat ion requires direct contact w i th sensitive parts of the skin, such as the fingertips. Therefore, it is best suited to situations where the device is being actively used and held in the hand, where the haptic feedback provides background status information. If the active appl icat ion also makes use of haptics, the s t imul i used for background notification must be distinct from the foreground applications haptic signals. Examples such as this underscore the importance of designing haptic icons in the larger context of their anticipated usage, and employing empir ical da ta relating to their group perceptual characteristics. 3.3.5 M i n i m a l l y I n t r u s i v e In ter face for R i c h N a v i g a t i o n o f M u s i c (Figure 3.9) Rich enjoys listening to music on his mobile device while on the go. He recently downloaded 30 songs onto his mobile music player, and wants to categorize them into songs he likes and songs he doesn't like. This is rather time consuming when he has to listen to the introduction of the songs, so he is in the habit of using the fast-forward/cue feature to skip to the "main", or chorus section of the song. Unfortunately, on his old music Chapter 3. Design Concepts 36 player this was a difficult task — since he never knew where the chorus section began, he often overshot it or was forced to listen to less relevant parts of the song before getting to the desired section. Moreover, since there were very few buttons on his miniature mobile music player, he had to press and hold the same button used to change tracks while visually monitoring the position counter, which was very difficult to do while walking or driving. Using his new haptically enabled mobile music player, Rich can literally feel the structure of the songs in his collection. He runs his thumb up and down the slider and feels bumps and textures where chorus sections begin and end. Stronger musical themes feel like more significant tactile features, while subthemes feel like weaker features. He can easily cue to any of these features without looking, by moving his thumb to a position with a tactile feature and pressing to select. With this method, he quickly sorts through his newly downloaded songs by repeatedly loading the next track, advancing to the chorus section, and listening to the main musical theme. He uses the device's rating feature to make note of which songs he likes, so he can enjoy their full length later. This scenario is based on the work of Goto [21], who described a method to automatical ly parse and identify musical themes (chorus sections) in pop-ular music. One application of the technique was in a retail music listening kiosk, where customers needed to sample mult iple tracks on a C D and make a purchase decision wi th in a l imi ted amount of t ime. B y providing cues as to the structure of the music — including main repeating themes as well as subthemes — the kiosk could help customers advance to the most salient, relevant, or memorable sections of music to make decisions about how much they like the selection. The goal of rapidly scanning tracks to identify and/or categorize them is shared by both users of listening kiosks Chapter 3. Design Concepts 3 7 Chorus sections — Audio Player — •i) Volume I / \ Prev Track U2 Vertigo 3:10 4/6 \y Next Track ( ) r" >— r \ ) i ) ( 1 .) C ) C D C ( J ( \ ( J c Figure 3 . 9 : Example mul t imoda l screen and haptic design for the r ich music navigation concept, (a), T h e por t ion of the page currently visible is shown as a highlighted region of the scroll bar. (b), Salient page elements are represented as haptic icons and shown in the scroll bar as an associative aid. Chapter 3. Design Concepts 38 and users of portable music players. Mob i l e music players are further chal-lenged by l imi ted space for physical buttons, making it less pract ical to add more controls than it is to add richer controls. Navigat ion of subthemes, ma in musical themes, entire tracks, and entire albums or playlists forms a cont inuum of salience which can be mapped to various levels of haptic interaction using a transducer w i th sufficiently wide bandwidth . 39 Chapter 4 Handheld Prototype Development Work ing from the in i t i a l applicat ion concepts and hardware profile, we began an effort to create a high-definition, active hardware prototype that would enable significant investigation into the effectiveness of a portable haptic interface in the hands of a user. T h e piezoelectric actuated lateral skin stretch technology introduced in Chapter 2 was selected for this project based on sui tabi l i ty for the problem domain and interaction designs, as well as pract ical considerations such as the matur i ty of the technology (allowing for a prototype to be realized wi th in the scope of a master's thesis) and an existing relationship between the author's research group and the group developing the technology at M c G i l l University. Since much of the work in this chapter is the result of a highly collabo-rative effort between research groups at the Univers i ty of B r i t i s h C o l u m b i a and M c G i l l University, each section in this chapter includes a description of the present author's ind iv idua l contr ibution to that aspect of the project. T h e hardware prototype, including the piezoelectric lateral skin-stretch con-cept, is described in detail in [43]. In this thesis we shall focus on the aspects that are relevant for the overall interaction design process of mobile haptics. Chapter 4. Handheld Prototype Development 40 4.1 Design Philosophy Because the options for specific implementations of the concept of handheld haptics are v i r tua l ly unl imited, we followed a design philosophy that was informed by a combination of the conceptual design work discussed earlier ( including user needs and application concepts), resources available for con-duct ing the research, and general heuristics about pract ical implementat ion of the hardware concept. 4.1.1 L i n e a r s l i d e - m o u n t e d t a c t i l e d i s p l a y u s i n g p i e z o e l e c t r i c a c t u a t e d l a t e r a l s k i n s t r e t c h A s mentioned in previous chapters, the hardware concept consists of a linear (1-degree-of-freedom), slide-mounted tactile display based on the piezoelec-tr ic actuated lateral skin stretch technology, mounted in a handheld case and positioned under the user's thumb. We believe this option represents the best compromise for adding significant new haptic functionality without diverging too far from contemporary mobile device norms. It would be conceivable to add mult iple degree-of-freedom movement and addi t ional features, but the addit ional costs in terms of size, weight, complexity, and power consumption must be justified against potential ben-efits. Since the benefits of r ich mobile haptics have yet to be demonstrated conclusively, we felt it was best to adopt a conservative approach first; if significant benefits could be demonstrated, further iterations of the design could increase functionality i n the areas where there would be an expected benefit. T h e T D is placed on the left side of the device (accessible by the thumb on the left hand) because it is the most common locat ion in existing products Chapter 4. Handheld Prototype Development 41 for side-mounted controls such as buttons or jog wheels. T h e convention appears to be due to a preference, since most people are right-handed, for leaving the right hand available for jo t t ing notes while ta lk ing on the phone, or wr i t ing on the screen of a P D A . 4.1.2 U s e o f off - the-shelf h a r d w a r e c o m p o n e n t s All of the components of the hardware prototype should be readily available, mass produced parts. Th i s design philosophy allows us to reduce costs and increase the speed of development, by focusing on the system integration issues rather than detailed engineering of components. O u r final hardware prototype may be replicated by any facility w i th basic mechanical and electronic prototyping equipment such as a hand-operated m i l l and soldering tools, at a final b i l l of materials cost of around $380 U . S . dollars, excluding the F P G A and P C used for control. T h e major component of this cost, retail piezo actuators ($160), declines significantly (by a factor of 4 or more) if ordered in quantity. The final size, weight, and power profiles of the prototype tactile display mechanism make it not currently suitable for deployment as a mass pro-duced consumer product. However, each of the cr i t ical components may be minia tur ized wi th industr ia l micro-manufacturing facilities. In particular, the piezo elements, which currently require 50 V D C and extend into the device w i t h a depth of 31.8 mm, may be specially manufactured as layered benders that could reduce the voltage requirement to 8-10V while s imulta-neously increasing r igidi ty and thus reducing the required depth [45]. In addit ion, the comparator and amplifier sections of the control electronics would be vastly simplified since reduced voltage would be well w i th in the Chapter 4. Handheld Prototype Development 42 range of mass produced microminiature D C - D C converters and integrated amplifiers. 4.1.3 Handheld operation while connected to a host P C A significant part of the complexity of producing mobile devices derives from integrating batteries and wireless communications, and programming embedded computers — issues that are secondary to the haptic interface questions we are t ry ing to investigate. Therefore, we have retained a tether to a host P C and power supply, while packaging the input and output trans-ducers in a handheld form factor. T h i s allows us to investigate key aspects of the handheld user experience using a method similar to the " W i z a r d of O z " paradigm: to the user, it appears that they are using a mobile device wi th a few subtle wires added. Despite the tether, the overall form factor generally affords mobi l i ty and users are able to conceptually dist inguish the wires as additions possibly related to the testing status. 4.1.4 Author's Contribution The part of the design philosophy related to the form factor, especially the thumb-mounted posit ioning, is pr imar i ly the work of the author. Other aspects (off-the-shelf components and tethered operation) were heuristics that were also shared in parallel by the collaborator, Jerome Pasquero, and the thesis supervisor, prior to the beginning of the project. Chapter 4. Handheld Prototype Development 43 4.2 System Overview The hardware components of the system, including labelled signal paths, are shown in Figure 4.1. The major components of this system are described i n the sections below. Since this thesis discusses a design process, it is instructive to examine how the hardware specification evolved over t ime. W i t h i n a month of begin-ning the hardware project, many of the components had been established (Figure 4.2). A l o n g the way to the final design, the desk mount (which was expected to increase rel iabi l i ty and simplify cable management, but was not necessary because the cables were not as intrusive as expected) was el im-inated, as was the provision for using an off-the-shelf game controller to digitize analog posit ion and but ton signals, and addi t ional pushbuttons on the device (though they could be added at any t ime if applications warrant). 4.3 Output Transducers 4.3.1 Tactile Output Device (Tactile Display) The tactile display ( T D ) has an active area of 8.7 x 6.4 m m and contains eight stacked piezoelectric bending motors separated by th in (0.5 mm) brass rods. The bending "motors" (Piezo Systems model T215-H4-203Y) are th in (0.38 m m thick) rectangular sheets of relatively r igid ceramic-like material measuring 6.4 x 31.8 m m . Internally they consist of two bonded layers of alternately polarized piezoelectric material , such that when an electric field is applied to the motor, one layer contracts while the other expands, resulting in a bending deformation of the entire slab. The piezo elements are mounted in a new dual-pinning configuration that Chapter 4. Handheld Prototype Development 44 Figure 4.1: Overview of hardware components of the handheld prototype (final version used for user testing). LO PC NTSC Composite Video LO o o CM Xi o in CO a o a o o CM desk stand Chapter 4. Handheld Prototype Development 46 Figure 4.3: The tactile display, (a) Mechanism of action, (b) Assembled view, (c) Exp loded view. Diagrams by Jerome Pasquero. Chapter 4. Handheld Prototype Development 47 optimizes the force delivered to the user's finger and is more efficient than the cantilever mount used in the V i r t u a l Bra i l le Display [30, 43]. Because the distal ends of the piezo elements are fixed (i.e., mechanically grounded to the chassis that is being held in the user's hand), the proximal ends of the piezo elements that are in contact w i th the user's finger move when an electric field is applied. T h e por t ion of the user's skin covering the region between the ends of two adjacent piezo elements is stretched when the elements move away from one another, and it is compressed when they move towards one another. T h i s activates the strain-sensitive mechanoreceptors in the user's skin, producing a tactile percept. F inal ly , it has been demonstrated that this produces the i l lusion of the sensation of something pushing into the skin, as occurs when the finger is placed on a surface feature such as a smal l bump [23]. R a w piezo elements are shipped as layered slabs wi th three electrodes: the two outer surfaces and an internal, central electrode. Preparat ion in-volves electrically connecting the two outer electrodes while creating an area to access the central electrode. Us ing a technique developed by the M c G i l l collaborators, we abraded a smal l section of outer electrode and applied con-ductive tape, conductive paint, and insulat ing tape to create the required connections. Special solder flux provided by the piezo vendor was used to attach 30-gauge wire to each bending motor. Note that al though the piezos we used require relatively high voltage (+ / - 50V) , the current draw is on the order of a few mil l iamps and thus they are not high power devices. Because the piezo bender essentially consists of two oppositely charged electrodes separated by a gap, its electrical behaviour at sub-resonant frequencies is essentially that of a sl ightly lossy capacitor. A s such, its steady state current draw is almost n i l , and producing a deflection Chapter 4. Handheld Prototype Development 48 by charging and discharging the plates also does not require high current. Further details of the tactile display design can be found in [43]. 4.3.2 Video Display T h e video display is a conventional 2.5-inch colour T F T L C D monitor that includes driver c i rcui t ry to accept N T S C composite input signals. In the handheld prototype, the video display is mounted in the top half of the casing wi th the driver board positioned alongside, rather than under, the L C D panel. A n 8-pin r ibbon cable carrying the video signals emerges from the bo t tom of the casing and is routed around the top of the casing to jo in the power and signal wires. A second driver / interface board accepts the N T S C composite input v i a an R C A jack and 12V D C power (from a standard A C adapter) and sends the signals out on the r ibbon cable. After long term use the video display experienced hardware problems unrelated to its usage i n the haptic prototype, and it was not possible to procure a t imely replacement. Therefore, for the user tests related to the browser it was necessary to use a slightly larger 3.5-inch monitor affixed to the top of the case. W h i l e this d id not produce an appreciably different screen image (both use N T S C composite input) , it d id cause a significant increase i n the weight of the top half of the device. 4.3.3 Author's Contribution T h e tactile display was designed mostly by Jerome Pasquero, while the au-thor was responsible for preparing the piezoelectric components, and par t ic i -pat ing and providing feedback i n design discussions. The video monitor is an off-the-shelf component; however, the author is responsible for its inclusion, Chapter 4. Handheld Prototype Development 49 selection, and procurement. 4.4 Sensors 1 Figure 4.4: Loca t ion of input sensors, (a) Resistive posit ion sensor, (b) Push-to-select sensor. Diagrams by Jerome Pasquero. 4.4.1 Slider Pos i t ion Sensor T h e entire T D is mounted on P T F E (Teflon) bushings that slide over 2.4-mm diameter steel rods, wi th a travel of 1 1 m m . Slider posit ion is acquired using a standard analog 1 0 K ohm potentiometer-type resistive position sensor. A custom A / D module (Section 4.5) digitizes the signal and sends it to the Chapter 4. Handheld Prototype Development 50 control circuitry. 4 . 4 . 2 P u s h to Se lec t Sensor To facilitate a "click to select" behaviour, the entire assembly can articulate in an arc w i t h a travel of approximately 1 m m when the user pushes into the T D (i.e., perpendicular to the active surface). The movement is sensed by a standard 3 m m pushbutton switch mounted on the distal end of the mechanism. The switch is connected v i a the A / D module to the control circuitry. 4 . 4 . 3 A u t h o r ' s C o n t r i b u t i o n The author was responsible for the idea of the selection functionality, and component selection for the position sensor. A major part of the effort of integration of the sensor components was in achieving the right mechanical design for the slider and ar t iculat ing mechanism; this was done by Jerome Pasquero i n cooperation wi th M c G i l l mechanical workshop collaborators Joe B o k a and D o n Pavlasek. 4.5 Interface Elect ronics Cont ro l is achieved v i a a series of custom and off-the-shelf electronic mod-ules. T h e tactile output signal begins at the P C running the applicat ion software, which sends out packets v i a a Universal Serial Bus ( U S B ) con-nection to a field programmable gate array ( F P G A , Constel la t ion board 1 0 K 5 0 E from Nova Engineering) running custom firmware. The function of the F P G A is to convert the P C output into an 8-channel 156.25-kHz pulse w id th modulated signal. The P W M signal is then fed through a comparator, Chapter 4. Handheld Prototype Development 51 low-pass filter, and high voltage amplifier, resulting in an 8-channel analog signal that varies from + / - 50V. (see Figure 4.1). Note that any equipment capable of generating a P W M signal may be used as the input to the comparator, filter, and amplifier stages. If desired, a standard I / O card, or embedded controller, may be connected in lieu of the P C and F P G A , e l iminat ing the most costly and bu lky components of the system and allowing for flexibil i ty in prototyping. O n the input side, a standard 8-bit analog to digi ta l ( A / D ) converter is used to digitize the resistive posit ion signal. • 4.5.1 Author ' s Con t r ibu t ion Jerome Pasquero was pr imar i ly responsible for the design and specification of the interface electronics. T h e author was responsible for par t ic ipat ing in design discussions (mostly to advocate the m a x i m u m level of flexibility for rapid interaction design prototyping), sourcing of some parts, and assembly of the custom electronics. 4.6 Power Supplies T h e power supply for the tactile output consists of a pair of standard regu-lated 48V, 2 5 W power supplies (Kaga Components L t d . Par t # SP30U-48S) normal ly used for telephone equipment. T h e fine-tuning voltage adjustment feature was used to tweak the output voltage to 50V. A d d i t i o n a l power sup-plies include manufacturer-supplied standard A C adapters for the F P G A and L C D , and an internal + / - 15V power supply derived from the main + / -50V source v i a voltage regulator. Chapter 4. Handheld Prototype Development 52 4.6.1 A u t h o r ' s C o n t r i b u t i o n The design and integration of the main power supply is the work of the author. 4.7 C o n t r o l Software The system is controlled by custom software modules on the F P G A and host P C . T h e F P G A software consists of device specific firmware that handles the translat ion of U S B packets into P W M output, and 8-bit A / D input to U S B packets. It addi t ional ly offers buffer management features and can opt ional ly support a non-hosted mode in which the F P G A operates i n a closed-loop fashion, w i t h piezo output dependent on a cached slider posit ion function. T h e P C software layer is based on the " S T R e S S L i b r a r y " , 1 [29] a cross-platform A P I wri t ten in C + + , featuring buffered input and output functions and support functions. It is implemented using various open-source libraries, including the ACE component l ibrary for X M L parsing, threading, and other support functions, and the l i b u s b l ibrary for I / O . A l l applications described i n this thesis were implemented on a 1 G H z Pen t ium class P C running SuSe L i n u x . Internally, the + / - 50V output signal is represented using 12 bits (4096 levels) on the F P G A , and the P C control software is able to handle both 12-bit and 8-bit signals. In practice, it was found that there was no readily 1The API is so named because it shares a large part of the code with an earlier project [41]. Chapter 4. Handheld Prototype Development 53 perceivable difference 2 between 12-bit and 8-bit resolution output, so the 8-bit (256 levels) signal was used for a l l the applications described i n this thesis. Th i s allowed for more efficient use of resources, since t iming was a concern (see below). 4.7.1 Input and Output T imings The basic software I / O loop runs at a rate of 333 H z . Once every 3 ms, new information (consisting of voltage levels for each of the eight piezos) is writ-ten to the F P G A ' s output buffer, and the input buffer (consisting of slider posit ion and click status) is read into the P C . A l l implementations described in this thesis were run on standard L i n u x without real-time extensions, so the actual 1/O timings are subject to system load conditions. Experiments and performance optimizations were performed to ensure stable real-world performance under load. Figure 4.5 shows the measured output at the amplifier (with the piezo load disconnected), while the system is running the full interactive browser prototype described in Chapter 6. In addi t ion to generating piezo output, the system must read slider input, compute the output tactile image, and update the visual browser display. Effect of output t iming on qualitative experience The top por t ion of Figure 4.5 shows the output of an early version of the software, which was able to produce signals at an irregular rate of approx-imately 12 H z , while the bot tom port ion of Figure 4.5 shows the output 2Although this observation is based on the subjective experience of the author and other researchers, it is reasonable to assume that since the total piezo travel must be less than 1.0 mm, a 12-bit (0.2 / i m maximum) resolution is well beyond the limit of human perception, which is roughly 1 /im for small surface features [25]. Chapter 4. Handheld Prototype Development 54 Figure 4.5: Measured outputs under conditions of load (running the browser applicat ion). Above, prior to opt imizat ion. Below, after opt imizat ion, show-ing two adjacent output channels at once. Chapter 4. Handheld Prototype Development 55 after a series of software performance optimizat ions improved the rate to a relatively stable 83 H z . A t the low (12 Hz) sampling rate, the sensations produced by the T D were perceived by the author and other observers as rough, vibrat ion-l ike s t imul i without a clear sense of tactile flow, while the high (83 Hz) rate stimulus produced a much smoother sensation. O n l y the high sample rate stimulus could be compared to the sensation of touching a small bump. It is l ikely that the problems wi th low sample rate st im-ulus were the result of aliasing — undesirable high frequency transients are produced at the square edges of the wave, which become notably more pronounced at the lower sample rate. Contro l software performance optimization Software performance opt imizat ion to improve the output rate consisted of iterative development of both the F P G A firmware and P C software to optimize the buffering algori thm for rapid, interactive, closed-loop opera-t ion. T h i s was especially important for the browser applicat ion (Chapter 6) which, unlike the earlier applications used for simple, triggered playback of haptic icons and tactile movies (Chapter 4), could not "anticipate" and cache significant amounts of output for smooth buffered delivery. Further performance improvement was achieved by rewri t ing the appli-cations to the number of threads, as interprocess communicat ion overhead was found to be a significant source of irregular delays. T h e final implemen-ta t ion of the browser applicat ion uses only two "user" threads, plus one I / O thread in addi t ion to the various threads internal to the M o z i l l a browser. F ina l ly , as described in more detail in Chapter 6, the browser applicat ion software is designed to take advantage of the fact that the visual sense Chapter 4. Handheld Prototype Development 56 • " ^ Read Input (a) Slider Position Slider Model Smoothed Slider Position Velocity Model Page Map Position Document Model Highlighted Element Display Model Visual Output Highlighted Element Haptic Icon Model 8-Channel Voltage Image Output Smoother Smoothed Voltage Image I/O Layer i USB FPGA I Analog Amplifier I Analog Tactile Output (b) Read Input Slider Position Slider Model Smoothed Slider Position Velocity Model Page Map Position Position Page Map Model I Page Map New Position Page Map Haptic Icon Model 8-Channel Voltage Image Output Smoother Smoothed Voltage Image I/O Layer I use FPGA I Analog Amplifier I Analog Tactile Output Document Model Highlighted: Element Display Model Visual Output Figure 4.6: Implementation alternatives for the control software, (a), large single loop, (b), asynchronous design for improved tactile performance. The computation-intensive M o z i l l a por t ion is enclosed in a dotted line. Chapter 4. Handheld Prototype Development 57 is less sensitive to t iming irregularities than the tactile sense. Instead of one large I / O loop incorporat ing the M o z i l l a browser and its document model together w i t h the tactile computat ion modules, the final software uses separate, asynchronous loops for tactile and visual components (Figure 4.6). T h e tactile loop, a s l im 500K C + + program, runs at a higher pr ior i ty than the visual loop (the approximately 2 0 M B M o z i l l a suite), and includes its own internal, simplified representation of the web page so that it does not need to wait for the visual component. T h e final resource usage of the full-blown control software stack, includ-ing browser, averages around 60% of the 1 G H z C P U , as measured by the L i n u x "top" command. 4.7.2 Au tho r ' s Con t r ibu t ion The core control software architecture and implementat ion was done by Vincent Levesque. The author was involved in helping test and verify the system, including the measurements and optimizations (with the exception of the firmware update) described in Section 4.7.1. 4.8 Tactile Flow Rendering The concept of tactile flow is described in various previous publications on related piezoelectric actuated lateral-skin stretch tactile displays [23, 28, 30, 41] and publications on the present prototype [34, 43]; however, since it forms the basis for many investigations throughout this thesis, we shall review it briefly here. W h e n a finger is swept over a small surface feature, patterns of skin Chapter 4. Handheld Prototype Development 5 8 Natural Touch t=o t=3 Synthetic Tactile Flow L E G E N D Compression Zone W\J Stretch Zone -5oV ' 0V T 5 O V Figure 4.7: M o v i n g patterns of skin stretch create a sensation of tactile flow. Left, the shear pattern associated wi th dragging a finger over a small surface feature. Right, synthetic re-creation of the pattern using the piezoelectric lateral skin-stretch T D . Parts are not drawn to scale. Chapter 4. Handheld Prototype Development 59 stretch and compression are produced [28].3 For example, the drag of the feature against the skin creates shear compression along the por t ion of skin behind the leading edge, and since the skin is a continuous elastic material , the area ahead of the leading edge undergoes a complementary shear expan-sion, or stretch (Figure 4.7, left). Furthermore, this pattern of skin stretch and compression travels across the finger as it is moved over the feature. Th i s moving pattern of skin s t imulat ion is detected by mechanoreceptors and produces a sensory percept, which we have termed tactile flow. It is possible to simulate an analogous moving pattern of skin stretch and compression using the tactile display in our handheld prototype. F i g -ure 4.7, right, shows how the voltage-actuated bending of the piezo elements creates regions of shear stretch and compression. Note that at t ime t = 3, the pattern of voltage actuation is the same as at t ime t = 0, except that the signals are shifted to the left. In outward visual appearance, the sequential movement of adjacent piezo actuators during tactile flow display looks sim-ilar to biological peristalsis (sequential muscle contraction), or the pattern of movement of mill ipede legs (Figure 4.8). Tactile flow is commonly associated wi th spatial movement, such as dragging one's finger over a surface to explore its texture, or losing one's grip on an object. It is a directional signal; moreover, it is more natural (in terms of being more commonly encountered in dai ly life, as one touches and explores objects) than spatially distr ibuted patterns of v ibra t ion [9, 50]. Therefore, it is hypothesized that tactile flow may be useful as a spatial cue for use in navigating computer interfaces. Th i s topic is covered i n more detail in Chapter 7, Browser User Study. 3 One can simulate this experience by sliding a finger over the tactile "keyboard hints" — small bumps present on most computer keyboards at the F and J key positions. Chapter 4. Handheld Prototype Development 60 Figure 4.8: A millipede. Its legs move in a pattern that looks outwardly similar to the movement of the TD's piezoelectric actuators when displaying a tactile flow signal, except that the T D does not normally move relative to the user's finger. Chapter 4. Handheld Prototype Development 61 4.9 V i sua l i z a t i on of Tact i le S t i m u l i 4.9.1 P r o b l e m Depicting and communicating the tactile display output visually was a no-toriously difficult problem. The output electrical signal is an eight-channel time-varying voltage ranging from +50V to -50V DC. Thus there is the problem of depicting Voltage x Time x 8 Channels using two-dimensional media. The output may be represented by drawing the eight signal traces in a stacked fashion on the same time axis, like the display of a hypotheti-cal eight-channel oscilloscope (Figure 4.5 and Figure 4.9, a). This "parallel wave graph" approach is used in previous publications [30, 41]. However, this representation has several problems, including: • It only depicts the time-dependency aspect of the stimulus; it does not readily show the output as a function of slider position, which is the key feature that enables bidirectional interactivity and exploration of virtual texture that extends beyond the 8-taxel display window. • The presentation format is somewhat difficult to understand for a lay audience. It is most easily understood by a technical audience that is accustomed to reading signal trace graphs. • Because the vertical axis encodes two separate variables — piezo num-ber and voltage — the spatial relationship between piezos is broken up by local axes (voltage). Thus the graph does not visually convey the propagation of a signal from one piezo to another that characterizes "peristaltic" motion or tactile flow. • The "parallel wave graph" format requires a lot of space to depict Chapter 4. Handheld Prototype Development 62 even a simple stimulus. A more compact representation would thus be desirable. 4 . 9 . 2 N o v e l G r a p h i c a l R e p r e s e n t a t i o n s To address these issues, a new way of th inking graphical ly about the piezo output was developed as part. of the hardware development effort for the handheld tactile display. T h i s includes zeroth-order shaded voltage map and first-order skin stretch image representations, and a haptic icon design process that leverages the graphical representations for rapid prototyping of tactile s t imul i . 4 . 9 . 3 S h a d e d G r a p h f o r V o l t a g e S i g n a l B y replacing the height of a voltage trace wi th a greyscale shading value, it is possible to achieve a more compact representation that eliminates the overloading of the vert ical axis wi th both voltage and spatial information. A s i l lustrated in Figure 4.9, b, the shaded graph enables more rapid compre-hension by visual inspection of the "shape profile" and spatial displacement pattern of the stimulus, as compared to the parallel wave graph. Another key feature of the shaded graph is that instead of s imply rep-resenting eight piezo outputs, the vert ical axis may now be considered to encode taxels, or arbitrary spatial units along which tactile s t imul i are orga-nized. T h e T D can be considered a display window, spanning exactly eight taxels, into this larger tactile space. Therefore, the shaded representation serves a dual purpose: to visualize the output of the eight piezo channels over t ime, and also to visualize a tactile feature map, opt ional ly one that evolves over t ime. T h e latter representation is used throughout the remainder of Chapter 4. Handheld Prototype Development 63 +50V -50 V L +50V -50VL +50V I I I I I I I I I I I I I 0 Time 12 (b) LEGEND +50V stretch N ov neutral H -50V compression Voltage Image Stretch Image I I I I I I I I 0 Time I I 12 Figure 4.9: Graphica l representations of a stimulus consisting of three small bumps that travel up, and then down, the T D . (a) Conventional parallel wave graph, used i n previous publications, (b) Shaded graphs showing volt-age and skin stretch (delta) images. The t ime axis is expressed in arbi trary units. Brass end-plates are not included in this stretch analysis. Chapter 4. Handheld Prototype Development 64 this thesis. 4.9.4 Skin Stretch Image T h e raw voltage level at a given t ime s imply corresponds to the static dis-placement of the piezo element at that t ime. The tactile percept, on the other hand, is produced by transient patterns of skin stretch that activate mechanoreceptors in the user's finger. Th i s may be caused by: 1. Sk in stretch or compression produced by the movement of two piezos relative to each other. The area of skin i n the gap between the piezos undergoes the stretch or compression. 2. Sk in stretch or compression produced by the movement of a flanking piezo relative to the brass end-plates. 3. H i g h frequency vibrotacti le activation of a piece of skin, i n which forces are produced against the inert ia of the skin. 4. Ac t ive exploration of the static T D surface caused by the user moving or sl ipping their finger over the T D . O f these various causes, (4) is unrelated to the synthetic haptics being discussed here, and (3) is subject to the l imitat ions of vibrotact i le s t imul i that we described in Chapter 2, and as such we consider it secondary to the r ich lower-frequency activations made especially possible by the current T D design. B y plot t ing the difference between voltages applied to adjacent piezo elements using a similar shading convention to the above, we can generate a graphical picture of the skin activation pattern due to causes (1) and (2). In the present work, we have chosen to use "bright" shading for "stretch" Chapter 4. Handheld Prototype Development 65 (or expansion of the gap between two adjacent piezo-actuators), and "dark" shading for "compression" (narrowing of the gap). The resulting visual representation is known as a stretch image, and is contrasted wi th the raw voltage image, sometimes abbreviated volt image. The stretch image reflects more direct ly the user's tactile experience of the stimulus. Stretch Image Caveats The stretch image is an attempt to more accurately depict the tactile sen-sations produced by the device, and it is applicable to a l l of the s t imul i described in this thesis, but it is s t i l l not a completely perfect picture. The following cases should be noted: 1. A s mentioned in the previous section, the T D has sufficiently rapid response so as to be capable of producing vibrotact i le s t imulat ion that is not due to skin stretch created between adjacent piezo elements and therefore is not captured by the skin stretch representation. For example, if a l l eight piezo elements are vibrated in synchrony, the skin stretch image representation would be a flat grey, equivalent to no activation at a l l , since the piezos are not moving relative to one another. For this reason, stretch images are always presented alongside the source volt image in this work, as vibrotact i le transients can be visual ly identified as areas of high horizontal contrast in the volt images even when they are not present in the stretch images. 2. Patterns of static stretch do not produce a tactile percept. For ex-ample, the piezos could be moved into a part icular configuration and kept there for a long t ime (seconds), producing a momentary sensation when they are first moved, but one that dies out as the mechanorecep-Chapter 4. Handheld Prototype Development 66 tors adapt to the pattern of stretch. T h i s adaptat ion is not reflected in the stretch image. T h e potential problem of too low-frequency s t imulat ion should be kept in m i n d during tactile experience design. Further discussion of image-based compensation for mechanoreceptor adaptat ion can be found in the following section. 4.9.5 Automated Tools for Design W h e n tactile images are represented as an array of eight greyscale pixels, they can be created, manipulated, and analyzed using readily available tools for dealing w i t h visual (bitmap or raster) images. Us ing a paint and image manipula t ion program such as Adobe Photoshop, tactile images can be eas-i ly painted using the mouse, and many convolution filters may be applied wi th quali tat ively similar effects in both visual and tactile domains: blur, sharpen, add noise, contrast, and many others fall into this category. Since the image format provides an on-line visual preview of the tactile stimulus, the visual transformation produces a related tactile transformation. For ex-ample, b lurr ing an image visually, smooths ("blurs") the tactile sensation as well . Furthermore, stretch images can be automatical ly and rapidly generated in one click using Photoshop's "Custom" Filter. Th i s is a generic convolution filter that accepts a user-specified kernel up to 5 x 5 in size, and may be used to generate a wide variety of visual effects. W h e n applied using the settings shown in Figure 4.10, it takes the difference between two vert ical ly adjacent pixels, normalizes it to the stretch image scale previously described, and outputs the image. Because the stretch image is one pixel smaller than Chapter 4. Handheld Prototype Development 67 the volt image, the top pixel needs to be cropped off after using this filter. Alternat ively, i f desired, the interaction of the two end piezos (#1 and # 8 ) wi th the brass end-plates can be investigated s imply by adding two rows of neutral grey pixels to the voltage image prior to convolution. The resulting stretch image, after cropping, would then be 9 pixels high. i; Jiiiiu; ;.i jjiiiiiiiiij I liliJij I. liii !l ijliiii i liJii I Ii  iill i lijiil i i liUjli liJi i I liljjl lj Ni I !i ;• C l J S t 0 m JMM$^K^t$ _ ' O K ( Cancel ') f ' toad... j ( Sawe... ') 3 Preview Scale: |2~~ ~] Offset: 127 Figure 4.10: Photoshop Cus tom Fi l t e r settings to automate creation of stretch images. The depicted convolution kernel instructs the program to do the following: Take the sum of the current pixel, multiplied by 1, and the pixel immediately above it, multiplied by -1. Scale the total by dividing by 2, then add 127 (half of the full-scale value of 255). More sophisticated algorithms could be employed to improve the accu-racy of the stretch image by taking into account adaptation: local areas of skin stretch that are held for long periods of time could gradually "fade to grey" by detecting non-changing pixels (stretch zones) and pushing them towards grey on a timescale that is consistent w i th empir ical ly measured adaptat ion phenomena. Chapter 4. Handheld Prototype Development 68 F r o m a practical standpoint, it was decided not to implement adaptat ion compensation in the image processing for several reasons. F i rs t , the adap-ta t ion rate is not constant and is dependent on a number of factors such as psychological state, skin properties and ind iv idua l factors. Second, adding a real-time scale to the horizontal axis of the images introduces significant complicat ion that undermines their purpose as a rapid prototyping and v i -sualization tool . F ina l ly , it is quite easy to visual ly detect and avoid long periods of unintentional lack of motion in haptic icon designs; it is far more l ikely that such artifacts would result from higher-level effects such as inter-action between haptic icons and the speed of the user's navigation. For these reasons it was decided that the plain, uncompensated stretch image repre-sented a good compromise between accuracy (correspondence of the visual representation wi th the felt tactile sensation) and ut i l i ty as a prototyping and visual izat ion tool . 4.9.6 gif2hapticon Tool Pr io r to the introduct ion of the gif 2hapticon tool , mult ichannel output for piezo tactile displays was usually constructed algorithmically, w i t h tables produced in M a t l a b or similar tools through summat ion of sine functions, and used by applications in a simple spatially modulated lookup loop. A l -ternatively, using a Java-based application, the user could specify temporal output (a "tactile movie") by setting the value for each time-sample using onscreen slider widgets. The output from this applicat ion was an X M L - l i k e f i le 4 encoding the sample "frames", which was read by a tactile movie player 4 X M L was used primarily because of its extensible nature, which ensures that future functionality, such as more complex time/space relationships and support for different TDs, could be added without having to make major revisions to existing software. The Chapter 4. Handheld Prototype Development 69 program and output in sequence to the hardware. B o t h existing methods required considerable t ime and effort to generate a new tactile stimulus. In practice, the s t imuli that were created tended to be selected on the basis of convenience of the generating algori thm, rather than the quali ty of the tactile experience — for example, a lgori thmical ly "neat" simple sine waves or, when using the slider approach, square or linear transitions. Addi t iona l ly , the algori thmic method restricts the design of haptic signals to a technical audience that is familiar and comfortable w i t h mathematical functions. A method to allow more direct, rapid prototyping of haptic signals, leveraging the visualizations described previously, would enable more iterative design of s t imuli and exploration of the stimulus space. Leveraging the existing X M L framework, a tool called g i f 2 h a p t i c o n was developed by the author to rapidly transcode raster images in the G I F 8 9 a [2] format to the tactile movie X M L files, or to the newer "haptic icon" file format (also X M L based) described in Chapter 6. It is wr i t ten in C + + and uses the free ImageMagick [4] l ibrary for reading G I F images. W i t h the g i f 2 h a p t i c o n uti l i ty, any software that is normal ly used for creating animated G I F images for the Web may also be used to generate tactile movies. A sample workflow is shown in 4 .11 . Perhaps more impor-tantly, the first steps of this workflow, where haptic icons are designed in a creative and exploratory process, are nearly identical to the creation of ani-mated G I F icons for the Web. Therefore, anyone wi th experience in creating simple Web graphics can participate i n haptic icon design, thus enabling a actual implementation does not include a formal Document Type Definition (DTD) for reasons of expediency (although one could be added trivially in the future), therefore it is "XML-like" in its current state. 5These methods were developed and used by the group at McGill University. Chapter 4. Handheld Prototype Development 70 much wider audience than those who are familiar w i t h creating matrices i n M a t l a b . T h e code for the g i f 2 h a p t i c o n tool is shown in A p p e n d i x C . It is a cross-platform A N S I C + + application, as is the required ImageMagick library. The general command line usage is: g i f 2 h a p t i c o n - t < i n f i l e . g i f > < o u t f i l e . x m l > where - t specifies tactile movie mode and i n f i l e . g i f and o u t f i l e . x m l are the input and output filenames, respectively. g i f 2 h a p t i c o n is discussed further in Browser Prototype, Section 6, as it also features a mode for creating icons that are read by that applicat ion. 4.9.7 Author's Contribution The above section is p r imar i ly the work of the author. P r io r to the au-thor's involvement w i th the project, the "parallel wave graphs" described in Section 4.9.1 were developed by the M c G i l l collaborators, as well as a graph-ical representation, used in a Java tactile movie tool developed by Jerome Pasquero, that showed the relative angular orientations of ind iv idua l piezo elements (not discussed i n the body text above for reasons of brevity) . Chapter 4. Handheld Prototype Development 71 Photoshop Paint / Draw Image Typical Web Workflow Imageready Add Motion gif2hapticon XML Figure 4.11: The process for creating a haptic icon using the image-based workflow is similar to. the process of creating graphics for the Web. In this example, a l l creative steps are done in Adobe Photoshop and ImageReady, and the conversion to tactile signals is handled by automated tools, w i th the output of the Photoshop stage being a static G I F and the output from the ImageReady stage being an animated G I F . Tacti le images of arbi trary size are supported. For the purposes of a simple tactile movie, the size would be 1 x 8 pixels, while a haptic icon (Chapter 6) or other spatial representation can have a greater height. A l l images represented here are volt images. Chapter 5 72 Perceptual Characterization 5.1 Introduction The development of the in i t i a l applicat ion concepts and handheld tactile display hardware was guided by an understanding of the general capabilities of the lateral skin-stretch technology, and ideas for how it could address user needs i n mobile contexts. To proceed to the next stage of more detailed applicat ion design, we needed to quantify how users perceive the haptic signals generated by the new hardware. We then mapped some of the regions of the haptic "vocabulary" (range of s t imul i that the device could generate), al lowing us to assess sui tabi l i ty of the envisioned applications, and what s t imul i would best match the roles specified in our concept designs. 1 We used a similar approach to perceptual characterization as [12]. The core stimulus salience quantification method ut i l ized mult idimensional scal-ing ( M D S ) , a tool for analyzing perception in complex stimulus spaces [21]. G i v e n a range of s t imul i , M D S analysis produces maps of how the perceptual space is organized. : A version of this chapter has been published. Luk, J., Pasquero, J., Little, S., MacLean, K. , Hayward, V. , Levesque, V . (2006). Haptics as a Solution for Mobile Inter-action Challenges: Initial Design Using a Handheld Tactile Display Prototype, in Proceed-ings of ACM Conference on Human Factors in Computing Systems, CHI '06, Montreal, Canada, April 2006. Chapter 5. Perceptual Characterization 73 Our new hardware can generate moving s t imuli , but the range of de-tectable movement speeds was not known. We therefore performed a study to estimate this range. Th i s enabled us to select speeds for icons for later M D S analysis. 5.2 A u t h o r ' s C o n t r i b u t i o n T h i s chapter was previously published in [34], and represents the collabo-rative effort of authors on that paper, including Jerome Pasquero, Shannon Li t t l e , K a r o n M a c L e a n , Vincent Levesque, and Vincent Hayward . The au-thor part icipated in every phase of this research, including software develop-ment, haptic icon design, running the experiments, data analysis, presenting and wr i t ing up the findings, decision making and project management. A l l user trials in the subgroup M D S experiment and the in i t i a l data analysis for the speed study were conducted by the author. 5.3 S tudy 1 - Range of Perceivable S t imulus Speed T h e purpose of the speed study was to determine the available perceptual bandwidth in one possible dimension that could be used as a parameter for haptic icon design. T h e question we sought to answer was: "What is the upper l imi t on the movement speed of a v i r tua l 'shape' that people are able to perceive?" To estimate the range of useable stimulus speed we hypothesized that the users' abi l i ty to perceive the movement direction would decrease as speed increased. Chapter 5. Perceptual Characterization 74 5.3.1 S p e e d S t u d y - E x p e r i m e n t D e s i g n We used a simple moving stimulus consisting of a square waveform that was "tweened" across the tactile display to achieve a sense of mot ion (Figure 5.1). T w o waveforms were used, producing either a moving region of skin expansion ("stretching") followed by compression ("pinching"), or compres-sion followed by expansion. The max imum stimulus speed was l imi ted by the hardware sampling frequency to 3.40 m / s (taking 2.56 ms to cross the display). We conducted a simple pilot s tudy among the authors to determine the approximate appropriate speed range for testing, setting the lower speed bound to a region where stimulus detection accuracy reached a plateau. The independent variables were: speed (0.17 to 3.40 m/s ) ; direction (up or down); and wave type (stretch-compress or compress-stretch). The dependent variables, measured w i t h a forced-choice method, were: perceived direction (up or down), yielding an accuracy measure when compared to the actual direction, and confidence level (confident or guess). 5.3.2 S p e e d S t u d y - P r o c e d u r e The trials were conducted on a L i n u x P C w i t h the tactile device attached. O n each t r ia l , the computer selected random values for each independent variable. T h e user pressed a G U I but ton labeled "P lay" to feel the stimulus, which was repeated three times w i t h an intervening delay of 0.7 second. The user was then required to input the perceived direction and confidence level before proceeding to the next t r ia l . There were five "training" trials where the user was informed of the actual direction v i a a modal dialog box just after entering their responses, followed by 40 "test" trials where the user received no notification. Chapter 5. Perceptual Characterization 75 (a) piezo #8 base waveform 3.5 m/s, down time (msec) +50V -50V LEGEND Figure 5.1: Examples of s t imuli used for the speed study, (a) Voltage signal for one piezo element, (b) Pat tern of lateral skin stretch produced wi th the 3.5 m / s stimulus, (c) Pat tern of lateral skin stretch produced wi th the 1.8 m/s stimulus. The highlighted area represents one tactile frame in which there is the sensation of stretching and compression at opposite ends of the display. Chapter 5. Perceptual Characterization 76 5 . 3 . 3 Speed Study - Results 8 right-handed volunteers (5 male, 3 female, aged 20-40 years old) part ici-pated in the user study. Each user took approximately 5-10 minutes to run the study. O- O- O- O- O y O- O- O- O- \ - *b-S p e e d (m/s) Figure 5.2: Results from the investigation of perceivable range of stimulus speed. The heavy line is the polynomial trend line; measured data points are in grey. T h e overall accuracy results from the speed study are shown in Figure 5.2. The relationship of accuracy and speed was stat is t ical ly significant w i t h (x2 =43.00, p<0.01), support ing the experimental hypothesis. Accuracy fell to approximately chance levels at the max imum speed of 3.40 m/s , but ap-proached 90% at 0.34 m / s using a polynomia l regression. The measured accuracy at 0.19 m/s and 0.31 m / s appears to be lower than the surround-ing data points. W h i l e l ikely due to random variation, this observation is Chapter 5. Perceptual Characterization 77 being further investigated. A t . t h e higher speeds, users reported that the stimulus felt like a "click" or smal l v ibra t ion and that direction was difficult to ascertain. N o significant effect was found for wave type (x2 = 1-87, p>0.01). User-reported confidence level decreased as the speed was increased (x2 = 165.49, p<0.01). 5.3.4 S p e e d S t u d y - D i s c u s s i o n The results from the speed study show that the device is capable of signalling the direction of stimulus movement over a large range of speeds. The sensa-t ion experienced is comparable to sl iding one's finger across a surface wi th a smal l bump. It thus seems feasible to use a directional "tactile flow" signal in applications such as assisted navigation. In addit ion, the results suggest that speeds lower than approximately 0.34 m/s would be appropriate for designing abstract haptic icons that convey the sense of mot ion. 5.4 Study 2 - Haptic Icon Discrimination Experiment T h e purpose of the haptic icon discr iminat ion experiment was to assess the range and dis t r ibut ion of perceivable difference of some specific haptic icons rendered wi th this device. The mult idimensional scaling ( M D S ) technique was used to map the organization of the stimulus space. Chapter 5. Perceptual Characterization 78 Factor Levels waveform t r i , ro l l , saw, bump, edge amplitude ± 5 0 V ("full"), ± 2 5 V ("half") speed 0.34, 0.23, 0.17 (m/s) duration (Calculated from waveform and speed) t r i : {480, 720, 960} (milliseconds) ro l l : {221, 331, 442} saw: {86, 130, 173} bump: {74, 110, 147} edge: {74, 110, 147} Table 5.1: S t imul i used i n the M D S studies 5.4.1 M D S Study - Experimental Design The s t imuli were selected according to a 5 waveforms x 2 amplitudes x 3 speeds factorial combination, resulting in 30 haptic s t imul i (Table 5.1 and Figure 5.3). These factors roughly correspond to stimulus components used i n prior studies for tactile displays [11, 19]. The waveforms were chosen to represent quali tat ively different tactile experiences based on first-pass experimentation wi th different signals, and included both repeating and non-repeating waveforms. For the speed parameter, we chose a range that produced an accuracy rate approaching 90% in the prior speed study. A fourth "meta-parameter", duration, was calculated from the speed and waveform parameters, and represents the total amount of t ime the stimulus is present under the user's finger. We hypothesized that this parameter might be perceptually relevant and tracked it in later analyses. Chapter 5. Perceptual Characterization 79 +50V -50V +50V -50V +50V -50V +50V -50V +50V -50V 1 20 40 60 80 100 120 140 sample (x 320LLS) Figure 5.3: Waveforms used in the M D S studies. 5.4.2 M D S S t u d y - P r o c e d u r e The participants completed five stimulus-sorting blocks in a method similar to that used in [36] and [52]. The sorting method is a way to efficiently measure perceptual s imilar i ty between pairs of s t imul i . Par t ic ipants were seated at a workstat ion and operated the mouse wi th the right hand while holding the device in their left h a n d 2 w i th the thumb resting on the tactile display. Slider posit ion was ignored. Part icipants used a G U I that presented the 30 s t imul i in a gr id of approximately 1 c m 2 tiles. They could trigger 2 Only left handed operation is supported on the handheld prototype. This is also the case with almost all commercial handheld products having side-mounted controls. See Section 4.1. Chapter 5. Perceptual Characterization 80 stimulus playback by cl icking a tile w i th the left mouse button, and used the right mouse but ton to pick up, move, and drop the tiles into approximately 7 c m 2 regions on the screen, which represented clusters. O n the first block, they could adjust the number of clusters using onscreen + / - buttons. In subsequent blocks, they were required to produce 3, 6, 9, 12, or 15 clusters, presented i n random order; the number of clusters closest to the user-selected for the first block was excluded. We also collected qualitative feedback from users in a post-task interview, seeded wi th the following questions: • "How would you describe the tactile sensations you experienced to someone who had not experienced them?" • "What aspects of the device felt comfortable or natural to use, and what aspects d id not?" • " C a n you suggest any applications of the tactile sensations for a mobile device?" 5.4.3 M D S S t u d y - R e s u l t s Ten right-handed individuals (7 male, 3 female, aged 19-31 years old) par-t icipated in the study, and were compensated $10 Canadian dollars. A l l subjects completed the tasks wi th in one hour. We performed an M D S analysis on the data obtained from the sorting task. S t imul i that are sorted together into a cluster were assigned pairwise s imi lar i ty scores proport ional to the total number of clusters in a given sort, because it is reasoned that when a user has more clusters from which to choose, the significance of placing two s t imul i together in a cluster is Chapter 5. Perceptual Characterization 81 increased. ' 7 N ill** 13 N / 96 \ 1 7 * ; # 7 7 ! / ' ^ \ J A72\ \ 7 , / x \ \ • / v / _> LEGEND * A 77?/ OROLL >=*EDGE OSAW OBUMP • half amplitude •/«// amplitude number : duration (xlOms) Figure 5.4: Results from the M D S analysis of haptic icons. Each point represents a stimulus, and dotted lines il lustrate st imulus groupings. The axes may be rotated arbitrarily. T h e results from a two-dimensional M D S performed wi th ordinal , untied data are shown in Figure 5.4. Analyses i n 3-D and higher dimensions d id not y ie ld any addi t ional s tructural information about the data. T h e graph clearly indicates that users tend to structure the stimulus space in terms of waveform, wi th the t r i s t imul i clearly distinguished, and rol l s t imul i also being separated from the non-repeating waveforms bump, Chapter 5. Perceptual Characterization 82 edge, and saw. T h e s t imul i formed by the three non-repeating waveforms — bump, edge, and saw — were less clearly distinguished on the graph, indicat ing that users d id not consistently sort them separately from one another. Th i s suggests that the differences between these waveforms are not perceptually salient, possibly due to l imitat ions of the hardware or skin sensitivity. Addi t iona l ly , because the experimental paradigm uses relative perceptual data, the dom-inance of the repeating / non-repeating waveform difference may obscure subtle differences among the non-repeating waveforms [7]. A closer examinat ion of the graph suggests that durat ion and ampli tude may also be salient perceptual dimensions, but their organization in the overall M D S graph is not consistent. However, when subsets of the data were analyzed one waveform at a t ime, most of the graphs exhibited clear durat ion and ampli tude structure along the x— and y—axes. Because the data was collected in a task where users were required to sort a l l stimulus factors at once, we hypothesized that because the less salient dimensions are perceived quali tat ively differently depending on waveform, a global M D S so-lu t ion was unable to represent them al l consistently. We therefore performed an addi t ional experiment to determine the val idi ty of the subset analysis. 5.5 Study 3 - Subgroup M D S Experiment T h e purpose of the subgroup M D S experiment was to determine whether more subtle stimulus factors could be detected when the waveform was not var ied . 3 3 A detailed analysis of the subgroup methodology is published in [42]. Chapter 5. Perceptual Characterization 83 5.5.1 Subgroup M D S - Exper imenta l Design The subgroup M D S experiment consisted of four trials: a control t r ia l s im-ilar to the first M D S experiment, and three subgroup trials where users performed sorting tasks using indiv idual waveform subgroups. We chose the t r i , ro l l , and edge waveforms for further analysis because the earlier M D S analysis and qualitative reports indicated that they were judged to be the least s imilar . 5.5.2 Subgroup M D S - Procedure To avoid fatigue resulting from the increased number of trials, we reduced the number of sorting blocks per t r ia l . For the control t r ia l w i th 30 s t imuli , subjects performed three sorts w i th a user-selectable number of clusters i n the first sort, and 5, 10, or 15 clusters in subsequent sorts, presented i n random order w i th the number of clusters closest to the user-selected number of clusters in the first sort excluded. For the waveform subgroup trials using 6 s t imul i , after the first t r ia l the clusters were 2, 3, or 4 clusters using the same presentation and exclusion criteria. T h e control t r ia l was presented first, followed by the three waveform subgroup trials in random presentation order. A l l other data collection methods were the same as i n the first M D S experiment. 5.5.3 Subgroup M D S - Results Five right-handed people (3 male, 2 female, aged 19 to 35) part icipated i n the subgroup experiment. None had part icipated in a previous experiment w i t h the device. Part icipants were paid C A D $20 for a 90-minute session. T h e subgroup M D S results confirmed the findings from the earlier sub-Chapter 5. Perceptual Characterization 84 Figure 5.5: . Results from the subgroup M D S study, (a) Cont ro l t r ia l w i t h al l 30 s t imul i , (b) t r i s t imul i (c) ro l l s t imul i (d) edge s t imul i . The results exhibit organization along the dimensions of durat ion and amplitude. Chapter 5. Perceptual Characterization 85 set analysis, w i t h durat ion and amplitude being clearly employed by users to organize the stimulus space. Figure 5.5 indicates no clearly discernible dura t ion/ampl i tude organization in the control t r ia l graph wi th a l l 30 s t im-ul i , but when ind iv idua l waveforms were tested separately, the organization became apparent. In the subgroup graphs, durat ion is aligned vert ical ly and ampli tude horizontally. Addi t iona l ly , the data from the control t r ia l exhibi ted the same overall structure as the data from the first M D S study, providing further confir-mat ion of the original results and the robustness of the technique despite differences in the number of clusters used in the sorting task. Taken to-gether, the results indicate that durat ion and amplitude, while secondary to some differences between waveforms, are nevertheless discernible and useful as salient parameters for haptic icon design in this environment. 5.6 S u m m a r y of Perceptua l Charac te r iza t ion F ind ings The results from the three perceptual characterization studies suggest that users are capable of dist inguishing a wide variety of s t imul i produced by the hardware prototype. Direct ion, certain waveforms, durat ion, and am-pli tude are salient parameters that may be used i n designing haptic icons for use in applications. The three-way grouping we observed among waveforms was especially interesting, because it empir ical ly suggests how our first-pass parameterization model of haptic icons could be improved; for example, in -stead of treating waveform as a single parameter, in subsequent designs one could consider non-periodic versus periodic waveforms, and further subdi-vide the periodic group into different wave shapes (e.g., t r i versus ro l l i n the Chapter 5. Perceptual Characterization 86 present experiment). 5.6.1 Qualitative Findings D u r i n g user evaluation we were also able to learn how people perceive the device qualitatively. Th i s information is especially useful for determining how users would perceive the value of the proposed applications. The key findings are summarized as follows: • Universa l ly (N=15/15) , participants d id not find the s t imuli annoying or disruptive. M a n y participants reported that they preferred them to their mobile phone's v ibra t ion mode. A variety of reasons were given, including quiet operation and moderate stimulus amplitudes. • M a n y (N=8/15) participants volunteered that they would find this type of tactile stimulus useful for alerts and notifications, such as iden-tification of who is call ing, information about a wai t ing message, or an alarm. • Some (N=3/15) participants experienced m i l d tactile fatigue, usually expressed as numbness, which was overcome by repositioning the finger to use a different part of skin, or taking a brief (approx. one minute) break. • In general, participants said they found the device comfortable to hold and ergonomically suitable for the tasks. Since the sl iding function was not used in the perceptual characterization studies, it is not known whether this report would be affected by using the slider for input. Chapter 5. Perceptual Characterization 87 5.7 Percep tua l character izat ion findings for appl ica t ion design W i t h some quantitative and qualitative data on low-level user perception of the prototype device, we can now consider whether the applications origi-nal ly envisioned for the device are indeed appropriate, and to proceed w i t h the next steps of applicat ion design. 5.7.1 L i s t s e l e c t i o n Judging from the results of the perceptual characterization, haptic icons de-signed along the dimensions of waveform (periodic or non-periodic), dura-t ion, and direction are candidates for dist inguishing items in a list. Because the most salient parameters are the direction and speed of the stimulus, it is important to decouple this rendered motion from illusions of relative stimulus motion generated as a result of the voluntary thumb movements to produce control input to the system. One way of avoiding this confound is to signify a discrete command such as scrolling an i tem up or down w i t h a larger but mechanically grounded gesture that incorporates pressing the slider against an end-stop. 5.7.2 S c r o l l i n g A s original ly envisioned, the browsing application uses rendered speed and direction parameters to provide haptic feedback to the user about the move-ment of the point of focus w i th in the page. Hapt ic shape (waveform) is the only parameter available to provide information about the selected i tem (link, image, heading, etc.) However, the two M D S studies suggest that the Chapter 5. Perceptual Characterization 88 user's abi l i ty to discriminate haptic shape wi th this device may be some-what l imi ted when using non-periodic signals. It is possible to bu i ld and test the browser applicat ion using the currently identified set of haptic icons, but its usefulness may be l imi ted by the relatively narrow choices of icons. Al ternat ive next steps include (a) reiterating the haptic icon design and perceptual characterization stage to discover more choices for haptic shape; (b) re-examining the rendering method and electronic I / O characteristics to minimize electronic and mechanical filtering that may be reducing the resolution and bandwidth of the haptic signal output; and (c) reconsider-ing the mechanical construction of the tactile display itself w i th the a im of further amplifying signal strength and thus, presumably, the potential distinctiveness of different waveforms. 5.7.3 Direction signalling The location-finding applicat ion concept relies on the tactile display's abi l i ty to convey direction information to the user. The user studies confirmed that direction of tactile flow is clearly distinguishable across a useful range of speeds. Intensity, waveform and rhy thm of repeating s t imul i may be used to provide addi t ional information about the distance to the target, status, or movement of the target. Our results thus encourage prototyping and usabil i ty testing for this application according to the original design concept. 5.7.4 Alerts and background status indicators User feedback obtained during interviews following the perceptual charac-terization sessions indicated strong potential for using the device for alerts, Chapter 5. Perceptual Characterization 89 based on the judgment that it would be pleasant and non-intrusive compared to currently available vibrotacti le displays. D a t a from the perceptual characterization suggests a hierarchy of salience that could be mapped to the relative importance or urgency of an alert. For example, a periodic signal would be useful for important alerts due to its high saliency. Less important changes in background status, such as the movement of passively monitored "buddies", could be conveyed w i t h non-repeating signals. F ina l ly , i f background status indicators are to be mult iplexed wi th other haptic signals generated by the foreground (currently in-use) applicat ion, one of the dimensions identified in the user studies could be allocated for this display. For example, if the speed dimension was allocated to background status indicators, slow moving s t imuli could be used for the foreground appli-cation, while fast-moving s t imuli could indicate background alerts. However, because of the l imi ted set of currently known salient haptic s t imul i , it would be advisable to perform another i teration of haptic icon discovery before al locat ing a large chunk of the "vocabulary" to background indicators. 9 0 Chapter 6 Browser Prototype Having constructed a hardware prototype and characterized its expressive capabilities in the hands of users, we were ready to begin prototyping and testing one of the applicat ion concepts described in Chapter 3. We chose the browser concept to investigate first, because it is the most generalizable in terms of an abstract scrolling / navigation model, unlike more specialized interfaces such as the music player. A browser w i th tactile enhancement is also a significant departure from existing approaches, unlike the notification scenario, which could be considered an evolutionary improvement of existing vibrotacti le notifications. F ina l ly , the concept of s imulat ing the tactile sen-sation of spatial movement using tactile flow rendering, as an aid to cursor navigation, has not been previously studied. T w o levels of browser prototypes are described i n this chapter. In Sec-t ion 6.2 we describe a simple image browser,1 al lowing viewing and scrolling of an image w i t h an associated "tactile track", and an informal user evalu-ation conducted to roughly determine the expected user experience w i t h a higher fidelity prototype. The remainder of the chapter is devoted to the design and implementat ion of the high-fidelity tactile web browser, which is 1The image browser was implemented by Vincent Levesque and Jerome Pasquero based on a collaborative design, and the user study was conducted by the author with contri-butions to the experimental design from the McGill collaborators and thesis supervisor. The image browser is also described in [42]. Chapter 6. Browser Prototype 91 capable of reading web pages, automatical ly generating haptic maps based on web content, and support ing interactive navigation w i t h both visual and haptic feedback. A formal user evaluation of this browser is described in the following chapter. 6.1 Design Goals The purpose of bui lding the browser prototype is (1) to explore one of the aforementioned applicat ion areas in depth, (2) to discover what challenges exist for implementing effective haptics in this appl icat ion area using the piezo tactile display device, and (3) to assess its effectiveness w i t h usabil i ty testing. A s mentioned in Chapter 4, the a im was to develop, test, and validate an applicat ion in the handheld, tethered environment before going to the much more difficult and expensive mobile, untethered context. A s such, the browser software does not need to be concerned w i t h wireless issues and variable content delivery relative to signal quality. Web standards such as H T M L , C S S , and JavaScript are used instead of wireless standards such as W M L , M I D P , and Brew. W h i l e this decision potential ly l imits the un-tethered capabilities using the current browser architecture, it also greatly simplifies the design and eliminates costly technology licensing, allowing use of a fully open source software architecture. Us ing Web standards means that the browser is capable of interacting w i t h live content from remote servers. However, for the purposes of usabil i ty testing, local ly cached web pages are used. T h e browser uses standard web formatt ing and display constructs, but does not attempt to automatical ly reformat pages original ly designed for P C screens. Just as w i t h mobile Chapter 6. Browser Prototype 92 devices buil t around existing Web standards, content creators w i l l use the same techniques for authoring but must optimize their content for specific devices. Design for tactile browsing can be considered an extension of this process. 6.2 Low-Fidelity Prototype: Image-Based Browser A prototype browser was implemented that takes a graphical image as in -put and allows interactive scrolling of the image, w i th synchronized tactile output. The software was tested informally w i th users. T h e image browser is described in detail in [43]. Therefore, it is only briefly described here. 6.2.1 D e s i g n The design concept, described previously i n Chapter 2, is the work of the author. A key component of this design is the provision of a tactile track, represented graphical ly alongside the rendered page, w i th icons spatial ly arranged at locations corresponding to the vertical positions of elements such as headings, l inks, images, text, etc. (Figure 6.1). Furthermore, the height of the icons represented in the tactile track corresponds to the height of the page elements. W h e n the user scrolls the page, the tactile track moves in synchrony wi th the page. These concepts were used to bui ld the image-based browser. Chapter 6. Browser Prototype 93 (a) header haptic icon icon image haptic icon -(b) THMB The Tactile Handhe ld Miniature B imoda l dev ice features a liquid crystal d isplay and a tactile display. II is held in o n e hand with the thumb on the tactile d isp lay which is comb ined with a scrol l button. Tactile Display The display uses a stack of p iezoelect r ic benders lo deform the sk in in the tangential direct ion. Tactile Rendering This technology can render p rogrammable tactile sensat ions such as moving smal l sca le features. Demonstrative Application To demonst ra te its capabi l i t ies a s imple appl icat ion w a s deve loped whereby a c c e s s to a document w a s provided by s imul ta tneous visual and tactile explorat ion in a scrol l ing m o d e . © McGill University Haptics Laboratory ;ind University of British Columbia SPIN Laboratory (C) - a . c T3 / CD,' Figure 6.1: Overview of main features of the image browser, (a) "Tactile track" wi th callouts for icons, (b) Source image, (c) The viewable area is determined by the screen size and height of the tactile display ("tactile window") . D iag ram by Jerome Pasquero. Chapter 6. Browser Prototype 9 4 6.2.2 Implementation Implementation was done by the team at M c G i l l , pr incipal ly Vincent Levesque. The image browser application is implemented i n C + + and uses the G T K l ibrary [3] for graphics display, and the S T R e S S l ibrary (Section 4 . 7 ) for T D input and output. T h e browser software is capable of generating its own haptic icons para-metrically. It supports three types of icons, whose positions and types are manual ly encoded in the input image: 1. Headers 2. B o d y Text 3 . Embedded Images In the image browser prototype, embedded images are represented by a high-frequency "grating", body text by sparse "dots", and headers by ei-ther two closely spaced "dots", or a single dot w i t h a superimposed high frequency grating w i t h adjustable magnitude. Based on informal user eval-uat ion of the s t imul i , the two-dot configuration for headers was chosen as the best candidate for being distinguishable from the other s t imul i . Scrol l ing is accomplished by pushing the slider up or down, which causes the page to continue scrolling as long as the slider was held in that posit ion, i n proport ion to the distance from the centre of the slider travel (velocity control). For this test, the slider was equipped wi th a spring-return mecha-nism that centred the slider in the middle of its travel. There was no action associated wi th the T D ' s "click" mechanism. Chapter 6. Browser Prototype 95 6.2.3 Image Browser User Test A n informal user evaluation was carried out by the author to collect impres-sions from people who were unfamiliar w i th the device. The full results are reported in [43]. Eva lua t ion wi th seven participants produced the following findings: 1. W h i l e the high-frequency "image" icon and the low-frequency "body text" icons were clearly distinguishable, 6 out of 7 participants had difficulty dist inguishing the "header" icon, even when it was displayed graphically (in the tactile track) i n a way that was visual ly distinguish-able from the other icons, and they were asked expl ic i t ly to attend to the number of s t imul i they could distinguish. 2. A l l (7/7) participants said they wanted to be able to access the ends of the page which were inaccessible due to the tactile window being fixed in the middle of the visual scrolling window. 3. 4 out of 7 participants reported that they could easily perceive the direction of movement of the tactile s t imul i , and that it was in syn-chronization w i t h the movement of the page. 4. 3 out of 6 participants reported frustration wi th the use of the slider mechanism for scrolling and targeting. One part icipant said that in -stead of velocity control, it would be nice if the device worked like a jog dial , support ing repetitive pushing motions in position-control mode 5. Par t ic ipants ' free-response suggestions included being able to make selections by pushing down on the T D ("click" action), page flipping, Chapter 6. Browser Prototype 96 eyes-free browsing, and notification. 6.3 Haptic Display of Web Pages T h e previous section described a browser which operated on graphical mockup images of web pages, w i th the accompanying tactile s t imul i being generated algori thmical ly from one of three templates, w i t h placement and type manual ly encoded in the image. L i n k selection, reading of web source files ( H T M L , etc.), and other typical features of a web browser were not supported. The goal of the image-based browser was to serve as a first-pass low-fidelity prototype to obtain early user feedback as part of the ongoing process of developing the browser application. We w i l l now discuss an automated method for creating a haptic repre-sentation of a web page from its H T M L source files. The key features of this approach are: 1. Simultaneous visual and haptic rendering of H T M L elements. 2. A n object-oriented, extensible framework for the haptic page map. 3. Support for an arbi trary variety of haptic icons associated wi th page elements. 4. Support for animated (i.e., t ime-varying) haptic icons. 6.3.1 The Haptic Page M a p T h e output of the rendering algori thm is a haptic page map, which is a model (in the object-oriented programming sense) of the page i n the haptic domain. (Figure 6.2) In the haptic page map, icons are la id out spatial ly Chapter 6. Browser Prototype 97 along a one-dimensional page map coordinate, which corresponds to the ver-t ica l dimension in the visual representation of the web page. 2 . D u r i n g navigation, the user moves a cursor along the page map coordinate, as de-scribed in section 6.4. 6.3.2 Mapping Haptic Icons to Page Elements Page elements, such as links, images, and headings, are associated wi th haptic icons according to the following algori thm: 1. If the element contains the H T M L name attribute, that property is used as the haptic icon filename, followed by an . xml extension. 2. Otherwise, the default icons l i n k . x m l , i m a g e . x m l , h i . x m l , h2.xml, etc. are used. T h i s approach allows page authors to specify haptic icons easily wi th in the H T M L markup. For example, the X H T M L tag: <img s r c = ' c o u p o n . j p g ' n a m e = ' h a p t i c o n A ' \> places an image in the page wi th filename c o u p o n . j p g and associates it w i th the haptic icon h a p t i c o n A . x m l . Combined wi th the animated G I F based workflow described previously in Section 4 .9 .6 , this H T M L - b a s e d markup scheme means that no special tools are required to add haptics to an existing web page, and page authors can leverage their existing skills in graphics creation and H T M L markup. 2 A coordinate transformation may apply in converting between graphical vertical co-ordinates and page map coordinates. See Section 6.4.10. Chapter 6. Browser Prototype 98 XHTML Source ca h r e f = ' l i n k l . h t m l ' name='hump' >Link l < / a x b r \> <irag src='happy.gif \><br \> <a href=' l ink2.html' name='plateau'>Link 2</a> Gecko1 Graphically Rendered Page HapticPageMap int height = 400 Hapticlcon 'icons 1 Hapticlcon: :Spatial int position = 150 Hapticlcon "next float *data Hapticlcon::Animated int position = 200 Hapticlcon "next float *data[frames] Hapticlcon ::Spatial int position = 250 Hapticlcon "next • float *data Spatial Representation of Haptic Page Map 0 k g 2 0 -o 0> <n CD 0> "O O o o Q-01 400 Figure 6.2: The ( X ) H T M L source is rendered graphically by the Gecko engine and hapt ical ly using the H a p t i c P a g e M a p model , shown here in a simplified schema. T h e resulting spatial arrangement of icons is also shown. T h e H a p t i c P a g e M a p object includes a list of H a p t i c l c o n s , which may be of type S p a t i a l l c o n or A n i m a t e d l c o n . Together, the objects form a complete representation of the web page i n the haptic domain. Chapter 6. Browser Prototype 9 9 6 . 3 . 3 Spatial Layout A simple spatial layout a lgori thm is used to assign positions to the hap-t ic icons. T h e haptic icon's top coordinate corresponds to the top pixel-coordinate of the rendered element in the graphical representation of the web page. Various approaches to deal w i t h the case when mult iple page elements are la id out horizontally, and thus occupy the same vert ical space on the graphic display, were tested. There is functionality buil t into the layout en-gine for dynamical ly reconfiguring the haptic icons to avoid overlap Ini t ia l experimentation wi th this approach revealed that the mismatch between graphical and haptic representations created unsettl ing navigation inconsis-tencies. A t the same time, it was found that horizontal ly la id out elements are generally rare on smal l screen displays, especially mobile phones wi th a "portrait" (as opposed to "landscape") oriented display. Tak ing both these factors into account, it was decided not to take advantage of the horizontal layout functionality in the final prototype. Currently, the layout a lgori thm does not make use of the rendered pixel-height of the element; haptic icons are not dynamical ly resized to match the visual element height. Th i s was considered a low pr ior i ty for l inks and headings, since their heights are usually specified globally i n a style sheet and do not often vary from element to element. B o t h the variable-height haptic icon and horizontal layout features would be rather t r iv ia l to implement in the current framework, and represent op-portunities for future improvement (Section 8.3.4). Further details of the haptic page map generation algori thm are provided i n Section 6.5. Chapter 6. Browser Prototype 100 6.4 Nav iga t ion M o d e l T h i s section describes the user interaction model for the web page and its associated haptic page map. 6.4.1 C u r s o r P o s i t i o n T h e navigation model maintains a cursor posit ion in both the graphic and haptic domains. T h e coordinates may be the same, or they may be related by a linear transformation, as described in Section 6.4.10. T h e haptic cursor posit ion is logically an integer number in the haptic page map coordinate space. Signals are rendered to the T D from a port ion of the page map that is centred on the cursor posit ion. T h i s is known as the tactile window. The height of the tactile window is normal ly eight units high but may be increased wi th subtaxel rendering, described i n Section 6.4.10. Because there are an even number of piezos, by convention the cursor posit ion is rendered slightly above centre, or piezo #3 if no transformation is applied. T h e graphic cursor is logically a single row of pixels comprising the verti-cal point of focus. It is displayed as a semi-transparent yellow bar, achieved w i t h the alpha channel features of the P N G image file format, centred on the logical cursor posit ion, w i th height corresponding to the height of the tactile window in the current coordinate transformation scheme; it is eight pixels high when there is no transformation. Therefore, the graphic cursor pro-vides the user w i t h a visual display of the area of the page which is currently "visible" (or, more accurately, tactable) through the tactile display. T h e graphic and haptic cursor positions are kept synchronized in soft-ware. T h e term cursor position may be used to refer to both. Chapter 6. Browser Prototype 101 6.4.2 Rendering Haptic Icons A s mentioned in the previous section, the tactile window, or por t ion of the haptic page map that is "viewable" by the tactile display at a given posit ion, is rendered to the T D continuously. T h e following algori thm is used: 1. Every I / O loop cycle, the program traverses the list of haptic icons that is l inked from the HapticPageMap object. 2. If a haptic icon is found whose extent lies w i th in the tactile window (i.e., is within focus), the object's rendering function is called, which renders the icon (places voltage values) to a buffer. 3. The loop repeats unt i l a l l icons have been checked. 3 4. The port ion of the buffer that is wi th in the tactile window is output to the device. Therefore, the case of overlapping icons (normally an error condition), icons that are lower i n the list of icons may overwrite their predecessors. A n i m a t e d icons are handled impl ic i t ly by the rendering function of the Hapticlcon: : Animatedlcon object, which maintains a real-time counter and renders different "frames" when it is called at different times. Under the current implementation, the counter starts from the first animat ion frame when the icon is first brought into focus (its extent is overlapped by the tactile window), but is not reset subsequently; the icon's animat ion may be 3 Since the number of haptic icons is small (typically, less than 20) and is limited by the amount that the user is willing to scroll, the overhead associated with a simple iteration through the list of icons is negligible on the current system. Further improvement could be made by using a sorted data structure, such as a tree, to reduce search cycles. Chapter 6. Browser Prototype 102 considered to continue playing in the background dur ing the t ime it is out of focus. 6 . 4 . 3 Page Element Focusing The M o z i l l a browser keeps track of a focused, or highlighted, page element. If the user presses the selection but ton (usually a mouse button, but handled abstractly in the current implementation), the focused element is selected. For example, in the case of a l ink, the browser loads the new l ink. T h e page element that intersects w i t h the graphical cursor posit ion re-ceives the current focus. In simpler terms, the user can highlight an i tem by posi t ioning the cursor over i t . Because the cursor occupies a full row, the current design does not support selection of mult iple horizontal ly distr ibuted elements; it s imply defaults to selecting the rightmost element. Because the cursor is logically a single row, but is displayed as the height of the tactile window which occupies eight or more rows, it is possible for the cursor to be par t ia l ly intersecting an element both in the visual and tactile space without the element being focused. 6 . 4 . 4 Graphical Display Scrolling Because pages are often taller than the graphical display, cursor movement must also control scrolling of the display. A push-to-scroll model is used; when the cursor hits a boundary of the display, the display begins scrolling if necessary. O n a P C , this type of model is commonly used in word processors and text-entry boxes, where vert ical ly moving the cursor beyond the display boundary causes the display to scroll one line at a t ime. Because the cursor is also being used for focusing, it is desirable for the Chapter 6. Browser Prototype 103 user to have some way of looking ahead before h i t t ing an element. Therefore, scroll margins are implemented, as shown in Figure 6.3. F r o m its in i t i a l posi t ion at the top of the display, the cursor moves down unt i l it hits the scroll margin. If the user continues scrolling down, the cursor's displayed posit ion remains fixed while the page scrolls under i t . If the user reverses direction and moves the cursor up, the page does not scroll un t i l the cursor hits the top scroll margin (hysteretic motion) . W h e n the display can scroll no more, the cursor moves past the scroll margin and to the top or bot tom of the page. In this way, the whole page is accessible to the cursor, unlike the previously described image based browser. 6.4.5 C o n t r o l o f C u r s o r M o v e m e n t In the simplest case, the "throw" of the slider may be mapped to the overall height of the page map, such that if the user positions the slider at the top of its range, the cursor is positioned at the top of the page, and i f the user positions the slider at the bot tom of its range, the cursor is positioned at the bo t tom of the page. T h i s is a direct position control mode; the cursor posit ion corresponds direct ly to the slider posit ion. Posi t ion control is the model used in jog dials, including mouse scroll wheels. 4 E a r l y experimentation wi th the device revealed that there was insuffi-cient spatial resolution to display more than two or three distinguishable haptic icons using direct posit ion control across the short slider range (11 mm) . Furthermore, precise posit ioning would require extremely high dex-terity, and may even be beyond the precision of the analog posit ion sensor; 4 D e p e n d i n g on the a p p l i c a t i o n , the mouse sc ro l l whee l m a y also ope ra te in a d isc re te pseudo -pos i t i on con t ro l m o d e , s i m u l a t i n g the effect o f a user repea ted l y p ress ing a sc ro l l key. Chapter 6. Browser Prototype 104 1 t Top Margin j l " 1 1/5 A 0 What's Um "r IW'e'afliiei'i Focal Area 1 Transit \ I& Movies 3/5 If M Bottom Margin II1 "ill il"' 4 Figure 6.3: Scrol l ing margins for the graphical display, located at 20% from the top and bot tom of the display. The display begins scroll ing when the cursor reaches a margin. Chapter 6. Browser Prototype 105 for a typ ica l web page 600 pixels high, the necessary resolution would be approximately 1390 dots-per-inch, far in excess of typical mouse hardware resolution. 5 Because of the l imi ted range of slider motion, it was necessary to use velocity control, which is the model used in joysticks and shuttle controls. In the simplest case (pure velocity control), the cursor movement velocity corresponds to the slider posit ion, w i th the centre of the slider treated as zero velocity. A smal l "dead zone" may be used in the centre to reduce unwanted cursor motion. Th i s method was used in the image-based browser (Section 6.2). A more sophisticated model was used for the web browser (Section 6.4.7), and a summary of the three cursor control models is shown i n Figure 6.4. 6.4.6 S p r i n g r e t u r n t o cen t re T y p i c a l velocity control input devices have a spring-loaded return to centre mechanism which provides a resistance force when the user pushes on the device to cause motion. Informal user testing of the device w i th and without spring feedback revealed that users strongly preferred spring feedback when the device was used i n simple velocity control mode. A number of approaches were tr ied to implement spring feedback, in -cluding elastic foam, elastic bands, and compression springs. Due to the mechanical design of the slider, use of compression springs in a coil-over-shaft configuration was the most parsimonious and reliable approach. (Fig-ure 6.5) T h i s required finding springs w i th a m i n i m u m inner diameter of 2.4 m m and appropriate length and spring constant. Springs w i th appropriate 5The Microsoft Intellimouse Optical, for instance, has a resolution of 400 dots-per-inch. [5] Chapter 6. Browser Prototype 106 TD/ Slider Simple Position Control o ~D CD (£2 CD CD "O O o o a '^ CD Simple Velocity Control "Dead i Zone" l 600 •100 < o o Hybrid Position / Velocity TJ O w .—* Q 3 -20 -100 I o o 0 +20 100 100 Figure 6.4: Slider control modes. In simple posit ion control, the slider posit ion is mapped directly to the cursor position, e.g., for a page wi th height 600. In simple velocity control, the slider posit ion relative to centre is mapped to the velocity of cursor movement, w i th an optional central "dead zone" to make centering easier. In hybr id posi t ion/veloci ty control, a region of the slider travel functions in direct posit ion control mode; if the user pushes the sider beyond that region, velocity control is active. Chapter 6. Browser Prototype 107 physical dimensions could be found in common retractable ball-point pens, and 16 different pens were acquired and disassembled to find the one wi th the most appropriate spring constant. However, due to the fact that typ-ical use of a retractable ball-point pen involves compressing the spring by pushing direct ly on its end, versus lateral force applied i n the case of the T D , most ball-point pen springs were too stiff for use wi th the device. C o n -sequences of excessively stiff springs included user fatigue and diminished tactile stimulus intensity, due to the piezos being bent laterally against one another by the force against the spring-loaded slider. Softer springs were found in multi-colour ball-point pens that also have lateral actuation. The softest spring was found in a Mi t sub i sh i four-colour combinat ion pen from Japan, and this spring was cut to size and used i n the device. The spring was actually too soft to produce a reliable and accurate return to centre against the friction of the slider and the inert ia of the T D ; therefore, two slightly overlength springs in a mutua l ly preloaded configuration were used to improve the centering properties. Unfortunately, this also increased the force necessary to push the slider a l l the way to its top or bo t tom stop. T h i s configuration was used in the image-based browser tests (Section 6.2) and early pilot testing of the H T M L - b a s e d browser. W h i l e it was clearly necessary for velocity control (users reported strong preferences for the pres-ence of the springs), extended use caused fatigue due to the stiffness of the springs, and premature failure of the insulat ing varnish coat due to the strong shear forces between the thumb and the T D . A n extensive survey of spring vendors revealed that no softer springs could be found in the 2.4 m m inner diameter configuration, possibly due to metal lurgical l imitat ions in the m i n i m u m thickness of wire that can be used while remaining in its Chapter 6. Browser Prototype 108 elastic zone. Softer springs are available in a torsion configuration, but major re-engineering of the slider mechanism would have been required to accommodate them, and it is unlikely that such low spring rates would be sufficient to overcome the friction associated wi th the mass of the T D . Due to these l imitat ions, it was decided to remove the springs prior to further testing of the browser prototype, and to defer broad re-engineering of the slider mechanism unt i l the cursor control model could be studied in more detail . 6.4.7 H y b r i d V e l o c i t y / P o s i t i o n C o n t r o l M o d e l Through informal user evaluation, it was found that velocity control alone d id not afford the bidirect ional tactile exploration that was effective in cre-at ing the i l lusion of exploring smal l shapes wi th the V i r t u a l Bra i l le Display [30]. Furthermore, it was desirable to make effective use of the centre of the slider travel for precise posit ioning instead of s imply having a "dead zone". A hybr id velocity / posit ion control model was adopted to meet these goals, as shown i n Figure 6.4. W h e n the slider is in the posit ion control zone, velocity is zero and the user can move the cursor using posit ion control w i th in a local offset range. Outside of this zone, velocity control is active. The actual values of the parameters used are listed in Section 6.4.7. To the user, the experience of navigation using the hybr id control model is of a smal l active area i n the centre of the slider range wi th in which they can interactively "browse" a haptic icon as if they were touching a small shape, moving the finger up and down to explore its edges. Push ing the T D further causes scrolling motion. If the T D is moved very fast (for example, rapidly from its bottom-stop to its top-stop), the cursor may rapidly move across its Chapter 6. Browser Prototype 109 posit ion control range. If the m a x i m u m velocity setting is lower than this velocity, there may be a somewhat unexpected b imodal cursor behaviour in which it rapidly jumps and then moves more slowly under velocity control. However, in real-world usage, the user is unlikely to move the slider rapidly from end-stop to end-stop; the more common usage pattern involves moving the slider away from its centre detent and holding it there un t i l the target is achieved, followed by smal l fine-tuning adjustments in the case of overshoot, undershoot, or local exploration of adjacent elements. H y b r i d Contro l M o d e Feedback and Spring R e t u r n W h i l e the hybr id control model allows fine-grained exploration wi th in the central posi t ion control zone, users s t i l l reported discomfort using the device i n its velocity control zone without spring feedback. In addit ion, in the absence of any haptic cues such as spring force feedback, it was difficult to determine where w i th in the slider travel the mode switch occurred, which also resulted in negative user observations. To deal w i th the latter concern, synthetic tactile notification of mode switching was implemented. Whenever the user crossed a mode switch boundary, the current tactile signal was overlaid w i th a rapid vibrotact i le signal consisting of a half-height (50V peak-to-peak) cycle, repeated twice over 4 samples. T h e overlaid samples were cl ipped to + 5 0 V / - 5 0 V if neces-sary. Th i s provided a "cl icking" or "bumping" sensation as the slider was moved across the mode boundaries. The sensation was judged effective i n early pilot testing for notifying mode switches, but only if the slider was moved very slowly across its travel. W i t h rapid movement, the user could move the slider across its 11 m m travel wi th in a fraction of a second, causing Chapter 6. Browser Prototype 110 the two mode switch notifications to blend into a generic vibratory signal w i t h min ima l spatial local izat ion relevance. Furthermore, it d id not address the need for proport ional force feedback in velocity control mode. T h e eventual solution was to adopt hardware springs in a hybr id con-figuration. Figure 6.5 shows the design after several testing rounds wi th various materials and thicknesses. Smal l pieces of latex foam were affixed to the slider stops using industr ial double-sided tape. The extent of the foam pieces was approximately one-quarter of the total slider travel each (3 mm) . Because of the compliance of the foam, the point at which the slider hits the foam is essentially imperceptible; however, after the foam is com-pressed more than halfway, it starts to provide perceptible resistance. There is approximately 1 m m of travel wi th in this range, dur ing which the force is perceived as essentially isometric. To the user, the experience is of mov-ing the slider wi th in a free-movement zone causing the cursor to track the slider posit ion, followed by pushing the slider against its stops w i t h variable force resulting i n velocity control. Specific parameters were determined after informal opt imizat ion wi th three pilot users, and are listed in the section below. Hybr id Control Delays Further testing of the device under the hybr id control system revealed ad-di t ional areas needing improvement for user comfort. F i r s t , when users wanted to stop cursor movement by moving the slider out of the velocity control zone, they typical ly released the slider, al lowing the springs to move it back into the posit ion control zone, or they actively sl id the slider back towards centre. In either of these cases, the slider was unlikely to end up Chapter 6. Browser Prototype 111 Figure 6.5: Force feedback using springs. The standard coil-over-shaft spring configuration and the hybr id spring configuration is shown in various slider positions. Chapter 6. Browser Prototype 1 1 2 exactly at the posit ion control boundary; it was more l ikely to be somewhere wi th in the posit ion control zone. The effect of this on cursor movement was an unsettl ing "snap back" effect that made targeting difficult because the user would acquire a target using posit ion control, release the device, and the cursor would move backwards a distance equivalent to the distance be-tween the posit ion control zone boundary and the final slider posit ion, thus bringing the intended target out of focus. T h e second problem was unintended cursor movement due to acciden-ta l ly moving the slider into the velocity control zone. Th i s s i tuat ion occurred when the user was targeting a point just outside of the posit ion control range. The user would approach the target by continuing to move the slider i n the direction of the target, cross the zone boundary unintentionally, and cause unexpected continuous cursor movement. F ina l ly , due to the short velocity control range, and the low amount of force that the user could apply to the device laterally without causing fatigue or piezo deflection, there was a l imi ted dynamic range available for use wi th velocity control. Users s t i l l wanted to move the cursor rapidly to traverse a web page quickly; however, if the m a x i m u m speed parameter was increased excessively, there was insufficient precision at low speeds for precise targeting. These three problems were addressed by adding t ime controlled elements into the control software. The resulting control system's state diagram is shown in Figure 6.6. To address the first cause of unexpected cursor move-ment due to "snap back", a delay was implemented i n the transi t ion from velocity control to posit ion control. Since the system remains in velocity control dur ing this delay, the rapid return of the slider to its centre detent posit ion s imply causes the velocity to drop to zero, "absorbing" the large Chapter 6. Browser Prototype 113 negative posit ion displacement. W h e n the user replaces his/her finger on the display and starts moving it again after the delay has expired, the sys-tem is in the posit ion control state, enabling interactive local exploration of tactile features. T h e second concern of accidentally moving the slider into the the velocity control zone was addressed using a delay in the t ransi t ion from posit ion control to velocity control. L o c a l exploratory behaviour in posit ion-control mode consists of relatively rapid, back-and-forth small scale sweeps of the v i r tua l surface. To make the transi t ion to velocity control using the delay model the user must "push and hold" , a more intentional action, before the system started scrolling under velocity control. F ina l ly , a linear acceleration model was added to the velocity control to enable increased scrolling speeds if the user held the slider in the velocity control zone for an extended time. Hybr id Control Settings After performing the above mentioned adjustment process, the final param-eters used in the browser prototype are listed in table 6.1. 6.4.8 Reduction of Slider Jitter Al though the slider uses a high-precision resistive posi t ion sensor, there is considerable analog sensor noise. If unfiltered, this noise produces contin-uous high-frequency piezo mot ion that completely destroys the abi l i ty to render shapes using low-frequency piezo deflections. A historical averaging model was therefore implemented in the software to reduce j i t ter. R a w slider input is first passed to a S l i d e r S m o o t h e r object in soft-Chapter 6. Browser Prototype 114 Figure 6.6: State transi t ion diagram for the four velocity / posit ion control states. ware, which outputs the "smoothed slider posi t ion". It takes two param-eters: c a c h e _ s i z e and t h r e s h o l d , whose final values are shown in Table 6.2. W h e n the slider is stationary or being moved slowly, an average of the last c a c h e _ s i z e number of samples is taken and used as the final slider posit ion, thus 'cancell ing out any small , high-frequency noise. For high val-ues of c a c h e _ s i z e and large slider displacements, this can take a noticeable t ime to converge on the slider posit ion, which is perceivable as a "rubber band" effect on the cursor. Therefore, an escape parameter, t h r e s h o l d , is implemented. If the user moves the slider rapidly, historical averaging is abandoned and the current slider posit ion is used directly. Furthermore, Chapter 6. Browser Prototype 115 the cache of values is reset so that averaging re-starts at the current slider posit ion when its velocity drops below the threshold value. 6.4.9 Reduc t ion of H i g h - A m p l i t u d e , High-Frequency Outputs W h e n a user is scrolling very rapidly through the page map, it is possible for two sequential samples to be taken from very distant areas of the page map. W h e n there is a large voltage spread between adjacent samples, it creates a large, sudden displacement of the piezo that is perceived by the user as a rough "cl icking" sensation. T h e high-frequency transients caused by rapid discrete movement of the piezo are especially bothersome, since they tend to result in user reports of undesirable "vibrat ion" , "zapping" or "electric shocks". It is also potential ly damaging to the piezo motor itself. To combat this undesirable effect, two measures are used. Fi rs t , the slider input smoothing algori thm can spread displacements over t ime using its historical averaging algori thm, as described in the previous section. How-ever, to reduce unwanted "rubber band" cursor behaviour, large slider dis-placements cause a buffer abort. Furthermore, when velocity control is being used w i t h high max imum velocity settings, high-frequency, high-amplitude actuator transients can be produced even when the slider is held station-ary in a high-velocity posit ion. Therefore, an addi t ional output smoothing algori thm was implemented to reduce sudden large piezo movements. T h e algori thm first checks whether a given piezo actuator w i l l have a large voltage change, as measured between the "current" sample and the immediately previous sample. If the threshold (currently set at 50V in either direction) is exceeded, then the actual output voltage delivered to the piezo Chapter 6. Browser Prototype 116 is "smoothed" out by averaging the current sample and the immediately previous sample. (We nicknamed this method "lookback" smoothing). Because this tends to have the effect of smoothing out peaks and there-fore l imi t ing .the dynamic range of the T D during high speed movement, there is also a feedforward component designed to "recover" part of the peak voltage in future samples. W h e n averaging is used to produce the out-put of a given sample, the sample's original (pre-averaging) value is stored. T h a t value is used to compute the next sample by averaging the cached and current sample values on the next timestep. T h e threshold test is then ap-plied and, if necessary, the previous actual sample voltage is averaged back in using the "lookback" algori thm. The net effect of these smoothing functions is to create the sensation of rapidly scanning across tactile features, while min imiz ing aliasing caused by sampling delay. 6.4.10 Speed L i m i t a t i o n In the perceptual characterization phase (Chapter 5), a m a x i m u m speed of 0.34 m / s was found to elicit a reliably discernible sense of directional mot ion among users. We may use this figure as a reasonable estimate of an upper bound for the speed at which a user can traverse the haptic page map while maintaining a continuous tactile flow percept. 6 A t m a x i m u m speed, given 6 The speed study measured non-interactive tactile flow signals, and it is possible that the actual upper speed bound for an interactive (i.e., user-modulated) tactile flow signal may be slightly higher. An experienced user may also be able to detect higher speed signals. However, for the purposes of our initial calculation, it is clear that even with a 30% margin of error, requiring the user to spend 8 to 33 seconds to traverse a typical web page is still unacceptably long and requires countermeasures. Chapter 6. Browser Prototype 117 the physical dimensions of the T D it takes 30 ms for a haptic icon to move from one piezo actuator to the next. If spat ial layout of the haptic map is performed using a one-to-one map-ping between pixels and taxels, it would require 18 seconds at max imum speed to go from the top to the bot tom of a typical 600 pixel-high web page. Pages in the testing corpus for the browser ranged from 424 to 1698 pixels high, so it would require considerable t ime (12 to 50 seconds) to traverse the web page at speeds necessary to stay wi th in the range of perceivable direction. T w o techniques were used to accomplish the decoupling of the scrolling mot ion on the screen and the T D : 1. Page map shrinking, and 2. Subtaxel rendering These methods effectively comprise a cascading transformation scheme for the page map coordinates that can reduce the logical height of the haptic page map up to eight times relative to its graphical equivalent. If desired, it may be possible to further increase m a x i m u m speed using other techniques i n combinat ion or separately. These possibilities are discussed in Section 8.3.4. Page M a p Shrinking T h e browser client supports a mode that transforms al l page coordinates and requested icon heights to create a page map that is half the pixel-height of the web page. Us ing this technique, it is possible to double the effective speed l imi t of scrolling traversal, but because icons must be half-height, they must be simpler, thus l imi t ing the differentiability of icons. Chapter 6. Browser Prototype 118 User testing wi th the page map shrinking feature suggested that the loss of icon fidelity negatively impacted the perception of tactile flow because it required designing more sharp-edged icons. Users perceived rapid changes in the voltage applied to actuators as "vibrat ion" rather than tactile flow. Th i s undesirable perception of high-frequency aberrations is analogous to the problem of harsh edges and other anomalies that occurs in computer graphics when images are downsampled excessively using simple nearest-neighbour scaling algorithms. Further page map shrinkage beyond a factor of two was deemed im-pract ical because the haptic icon height for a l ink icon, normal ly 21 page map units for the corpus of pages used in the user test (Chapter 7), would drop below 10, thus severely l imi t ing the expressive capabilities of the icons and increasing the risk of delivering unintended high-frequency vibrotacti le signals to the user. S u b t a x e l R e n d e r i n g To reduce the movement speed of the tactile stimulus while s t i l l retaining good fidelity, a subtaxel rendering method was implemented in the browser server component. Us ing this method, the internal representation of the page map retains the same height as in the one-to-one case. However, when the signal is rendered to the T D , two or three (depending on a configurable mode setting) taxels from the internal map are averaged to form one output taxel . In this way, the effective "viewport size" of the tactile display (the tactile window) increases two- or three-fold, to 16 or 24 taxels, respectively. Because the locat ion of the T D ' s window into the overall page map is main-tained in the internal high-resolution coordinates, smooth movement results. Chapter 6. Browser Prototype 119 Parameter Description Value Units maxP The page map range accessible in position control mode. 50 page map units maxV The maximum velocity at full acceleration. 200 page map units per second posControlZone The portion of the slider travel assigned to position control (the remainder is used for veloc-ity control). Determined by the point at which the latex foam begins to provide notice-able feedback. 0.75 (ratio) timeBeforeVControl The timeout before entering ve-locity control mode. Used to prevent accidental activation of velocity controlled movement during exploratory behaviour. 0.05 seconds vControlAcceleration The time between entering ve-locity control mode and reach-ing the maximum velocity for a given slider position. 1.0 seconds timeAfterVControl The timeout before entering po-sition control mode. Prevents "snap-back" of the cursor when the user releases the TD. 0.05 seconds Table 6.1: Values of hybrid velocity / position control model parameters. Chapter 6. Browser Prototype 120 Parameter Descr ipt ion Value c a c h e _ s i z e The number of slider values (collected once ev-ery 3 ms) to average to find the current slider posit ion. 10 t h r e s h o l d If, w i th in the 3 ms sample interval, the slider is moved a distance greater than this value (in raw 12-bit slider posit ion units), historical averaging is abandoned and the current slider posit ion is used directly. 5 Table 6.2: Settings for the slider smoothing function using historical aver-aging. Chapter 6. Browser Prototype 121 High-Resolution (Subtaxel) Image Rendered Output t = 0 t = 1 +10 +50 o -20 +10 +50 Figure 6.7: The subtaxel rendering technique. In this example, three taxels from the internal high-resolution representation are averaged to form one output taxel. The stimulus is moving up across the display, and two time samples are shown. Chapter 6. Browser Prototype 122 Effective Input Suboptimal Output Volt. Stretch Input Output Volt. Stretch to CD E 100 200 300 400 H 100 200 300 400 h LEGEND +50V stretch 0V • neutral -50V H compression Piezo #2 Output Piezo #2 Output • • • 0 100 200 300 400 Time (ms)* 0 100 200 300 400 Time (ms)* * When user is moving the cursor at a constant speed of 10 pixels per second. Figure 6.8: Effective icon design for subtaxel rendering. High-resolution page map representations are shown in the left column of the group, and the output of the subtaxel rendering algorithm is shown on the right column in both raw voltage signal and skin-stretch representations. Five time sam-ples are shown as the stimulus is tweened upwards across the display at a constant speed, illustrating how the subtaxel algorithm provides fine spatial information in the form of temporal signal changes. Note that exceeding the resolution limit of the TD when designing high-resolution icons tends to create blurry results. Below the pictures, the output from one piezo (#2 from the top) is graphed over time. In the case of a well-designed icon, the piezo reaches full-scale values. Chapter 6. Browser Prototype 123 The method is, in effect, using temporal data to encode spatial data — using the high temporal resolution of the sense of touch [19] to compensate for the low spatial resolution of the tactile display device. The technique is shown in Figure 6.7. Because data is lost during the downsampling, haptic icons must be designed carefully to ensure that they create the intended sensations. Figure 6.8 illustrates some examples of effective and suboptimal haptic icon design with respect to downsampling. By analogy, when visual images are heavily downsampled to low pixel counts using averaging, they tend to lose contrast and become blurry, and the same danger applies to the haptic icon images. Especially since the sensations produced by the T D tend to be quite subtle, it is important to retain as much dynamic range as possible when designing icons. Fortunately, the effects of downsampling may be quickly previewed dur-ing icon design in Adobe Photoshop by scaling the image by 33% and using the "bilinear" setting, which is equivalent to the averaging method used by the software. 6.5 Browser Software Arch i t ec tu re The browser software architecture is shown in overview form in Figure 6.9. A logical flow diagram is also presented in Figure 4.6. Briefly, the principal components of this architecture are as follows: 1. A client component , built on the open-source Mozilla A P I , that han-dles parsing of the H T M L and CSS source documents, generation of the page map X M L file, keyboard and mouse input, graphical display, loading of new pages, and logging data for the usability evaluation. Chapter 6. Browser Prototype 124 Described further in Section 6.5.1. 2. A server component, implemented in high speed C + + code and running a continuous I / O loop, generating the tactile signals from its internal representation of the page map, and translat ing analog slider input to page map coordinates and forwarding the input to the client component. Described further in Section 6.5.2. 3. Interactive haptic icon design tools, previously described in Sec-t ion 4.9.6. 4. (Not shown on the diagram) W e b page generation tools for cre-ation of H T M L and C S S source content. Hapt ic markup is opt ional ly added to the H T M L source as described in Section 6.3.2. 5. A file-based interprocess communication system using "posi-t ion" and "click" files to transmit information from the server compo-nent to the client component, and a "flag" file for the client component to signal to the server component to load a new HapticPageMap. De-scribed further in Section 6.5.3. Author 's Contr ibut ion Ini t ia l work on specifying the architecture of the browser client was per-formed by undergraduate research student Shannon L i t t l e w i th support from the author. The current X P C O M and X U L components share approx-imately 50% of the code from that effort [33], which is in turn significantly derived from online X P C O M and X U L tutorials; the current JavaScript component, where the majori ty of functionality is implemented, shares less than 10% of code. Chapter 6. Browser Prototype 125 Figure 6.9: The browser software architecture, including software and hard-ware components and the data that is exchanged between them. Chapter 6. Browser Prototype 126 The I / O port ion of the browser server is based on example applications provided by Vincent Levesque, a graduate student at M c G i l l university, as documentation for the S T R e S S L i b r a r y [29] A P I . Ear l ie r examples of the " T H M B L ib ra ry" co-written by Vincent Levesque and Shannon L i t t l e based on this code were not used; the code was rewrit ten from scratch for more precise t iming control. Where open-source software was used, it is mentioned in the following sections. A l l other portions, including the aforementioned haptic page map and navigation functionality, cursor display, usabil i ty testing support, page con-tent, and style sheets are the work of the author. 6.5.1 Browser Cl ient T h e browser client component is buil t on the open-source M o z i l l a A P I and its Gecko rendering engine. Since M o z i l l a is the most popular extensible, open-source platform for web page browsing as of this wri t ing , it ensures good compat ib i l i ty w i th Web standards and good por tabi l i ty of the software. T h e browser client component consists of: 1. A JavaScript application, 2. A n X P C O M (Cross P la t fo rm Component Object Model ) component, 3. A X U L ( X M L User Interface Language) "chrome" component, and 4. T h e M o z i l l a A P I , its Document Object M o d e l and security modules, and the Gecko parsing and rendering engine. Mos t of the browser client component is implemented in the JavaScript applicat ion, because it has access to the M o z i l l a browser's Document Ob-Chapter 6. Browser Prototype 127 ject M o d e l ( D O M ) , al lowing read access for the spatial layout of web page elements, H T M L tags including specialized haptic markup, and mouse and keyboard events. The JavaScript applicat ion also writes to the D O M when it loads a new page, displays the cursor, highlights elements that are in-focus, or provides usabil i ty testing related displays to the user. Data-logging is ac-complished using the standard terminal dumpO command from JavaScript . The main I / O loop consists of reading the slider posit ion and click count (via the X P C O M layer, described below), updat ing the cursor posit ion based on the slider posit ion, and comparing the click count to a previous value — if an increase is detected, the highlighted page element is selected. T h e X U L layer consists of small wrapper code to run the JavaScript applicat ion wi th in the M o z i l l a framework while suppressing standard P C browser features such as toolbars. T h e X P C O M component handles file input and output, which is nor-mal ly unavailable to JavaScript because of security restrictions. It con-sists of various functions to read the posit ion and click files, and write the HapticPageMap file and its associated update-flag file. T h e file input and output functions are implemented in 0 + + , and a compiled I D L (Interface Descr ipt ion Language) wrapper is used to create the abstract function in-terface that allows JavaScript to call the C+-1- functions. F ina l ly , the M o z i l l a A P I itself is based on M o z i l l a 1.01 browser, and is the work of various open-source authors. 6 . 5 . 2 B r o w s e r S e r v e r A l l tacti le device input and output routines are encapsulated i n the browser server component, which is a separate executable running at the highest Chapter 6. Browser Prototype 128 pr ior i ty available in the L i n u x kernel. Th i s architecture ensures rapid, con-tinuous updat ing of the tactile device, reducing the risk of irregular tactile signals that are perceived as undesirable vibrations or roughness. T h e browser server begins by loading an X M L - l i k e 7 HapticPageMap file generated by the client component after reading an H T M L source file. The HapticPageMap file contains links to the various X M L - l i k e haptic icon files that are associated wi th page elements. The browser server parses a l l of these files using the open source A C E 8 L ib ra ry ' s X M L parsing engine. The result is an infernal representation of the haptic structure of the page in the form of an object model buil t around the HapticPageMap object (Figure 6.2). Since the object model exists wi th in the compiled C + + component, it can be read rapidly during the I / O loop. Device input and output is handled through Vincent Levesque's S T R e S S L ib ra ry ' s A P I [29], which in tu rn makes use of libusb and firmware onboard the F P G A for hardware input and output. T h e browser server's I / O loop 9 performs the following actions every cy-cle: 1. Check the page map flag, and reload the HapticPageMap if necessary. 2. Read the slider posit ion and click count from hardware and perform the translat ion into page map coordinates. 3. Compute the eight-taxel tactile image based on the cursor posit ion and the HapticPageMap. 7See the footnote to section 4.9.6. 8 A D A P T I V E Communication Environment [1]. 9The browser is not currently implemented on a real-time operating system, so its update interval can not be guaranteed to be 3 ms. See Section 4.7.1 for more information on measured timings. Chapter 6. Browser Prototype 129 4. Output the tactile image to the hardware. 5. Output the cursor posit ion to a p o s i t i o n file. 6. Output the click count to a c l i c k file. T h e browser server implements a dual-threaded architecture as a further performance consideration. Fi lesystem operations are handled outside the main I / O thread, and mutex locks (via the A C E library) are used for data synchronization. Browser Server Haptic Page Map Model A key part of the design of the browser server component is its object orientation. Just as graphical web browsers have developed a r ich Docu -ment Object M o d e l around the H T M L source, the browser server's haptic model is intended to be flexible and extensible using object-oriented tools. T h e base object (equivalent to the d o c u m e n t object i n the D O M ) is the H a p t i c P a g e M a p object, which contains a list of H a p t i c l c o n s (equivalent to the d o c u m e n t . e l e m e n t s [] array in the D O M ) . Currently, two types of hap-tic icons, S p a t i a l l c o n and A n i m a t e d l c o n , are supported, each wi th their own content data stored in a class inherited from the H a p t i c I c o n C o n t e n t abstract base class (Figure 6.10). T h i s architecture facilitates adding addit ional types of haptic icons. For example, one could imagine a haptic icon that varies its posi t ion based on t ime and/or slider posit ion. It could be rapidly implemented as another subclass of H a p t i c l c o n , or S p a t i a l l c o n . A s long as it supports its own polymorphic R e n d e r l c o n O method, no changes to the core browser server code are necessary. In fact, the A n i m a t e d l c o n class was added after the browser server architecture was complete using the method just described. Chapter 6. Browser Prototype 130 HapticPageMap int height Hapticlcon *icons • CurrentMapPos() ComputeDetentPosFromVelocity() Computelmage() RenderAlllconsQ 1 Hapticlcon abstract base class char 'type, 'name Renderlcon() HapticlconContent abstract base class Spatiallcon type="Spatiallcon" Spatial IconContent SpatiallconContent double 'data Animatedlcon type="Animatedlcon" AnimatedlconContent * f AnimatedlconContent double *data[frames] Figure 6.10: Inheritance and encapsulation diagram for the HapticPageMap object, H a p t i c l c o n abstract base class and its inherited classes, and HapticlconContent abstract base class and its inherited classes. Chapter 6. Browser Prototype 131 Further functionality could be achieved by scripting the HapticPageMap model (using JavaScript), in a similar fashion as is currently done with the D O M . This would enable a range of advanced haptic functionality and interactivity leveraging the existing core input/output code. 6.5.3 Interprocess Communication and T iming Communication between the browser server and browser client components is achieved through the use of small files. This has the advantages of simplic-ity and leveraging the filesystem's implicit locking mechanisms to prevent simultaneous access and data corruption. It has the disadvantage of poten-tially low performance due to disk overhead. Timing studies were therefore required to validate the filesystem approach. As described in Section 4.7.1, various web pages were run with an oscillo-scope connected to the amplifier outputs, and a series of software optimiza-tions achieved an empirically stable output rate. No specific timing studies were done on the GUI (Mozilla) loop, but it was observed that position and c l i c k files were not accumulating in the filesystem, indicating that they were being read in a timely manner by the client. No delays were observable between slider input and GUI output. As mentioned before, various optimizations were done for high perfor-mance, including multithreading and careful task prioritization. Placing the file-exchange directory on a R A M disk did not have any performance effect, probably because the files were already being exchanged via the filesystem's memory cache rather than being written to and read from disk. Chapter 6. Browser Prototype 132 6.5.4 Browser Haptic Icons A s discussed above in Section 4.9.6, a graphical image-based workflow allows for interactive icon design and rapid analysis using the stretch image tech-nique. T h e g i f 2 h a p t i c o n tool not only supports the tactile movie output mode, but also a haptic icon output mode wi th the - i command line option (the default). The X M L - l i k e format is similar to the tactile movie syntax but is more opt imized for haptic icons for use wi th web pages. B o t h static ( S p a t i a l l c o n ) and t ime-varying ( A n i m a t e d l c o n ) formats are supported, and the type is automatical ly determined by checking whether there are mult iple frames stored in the G I F input . Icons are stored in a icon l ibrary directory accessible by the browser server and loaded at runtime when the H a p t i c P a g e M a p is parsed. 6.6 Known Software Issues and Caveats 6.6.1 Support for Element Height Currently, haptic icons are X M L - l i k e files that are named under the conven-t ion [name] - [ h e i g h t ] . x m l . The page rendering engine selects the height value based on the onscreen height of the associated element. Therefore, a separate haptic icon file (or at least a symbolic l ink to a val id file) must be present for every possible rendered height of the elements. T h i s l imits the generalizability somewhat, especially for images, which may have vari-ous heights. A possible solution would be to generate icons algori thmical ly ( including repeated t i l ing) , to use the closest-available height icon, or some combinat ion of the two approaches. The X M L format should then be re-considered somewhat, moving the height parameter out of the filename and Chapter 6. Browser Prototype 133 into a separate tag, such as <height>, much as the <at> tag is currently used. M u l t i p l e height representations could also be combined into one file. 6.6.2 Opportunities for further software optimization The browser prototype was implemented wi th the goal of el ici t ing user feed-back on the applicat ion concept as expeditiously as possible. There are opportunities for improving the software architecture if the goal is to bu i ld a "production" version wi th more headroom for future expansion of features: • A formal implementat ion of the X M L standard, including a Docu -ment T y p e Defini t ion ( D T D ) would support future extensibil i ty while retaining compat ibi l i ty w i th the existing " X M L - l i k e " implementat ion. • The inter-process communicat ion method, which currently uses high-overhead but simple to implement filesystem access, could be revised using more lightweight, opt imized communicat ion methods for greater performance. • The applicat ion could be ported to a real-time operating system for more precise t iming control. • The modular i ty of the JavaScript code base could be opt imized for better maintainabil i ty. A s mentioned earlier, while the software is far from being exhaustively opt imized, none of the aforementioned optimizations would affect the oper-at ion of the browser in its current form. The current version of the browser has been tested to support the application concept and to deliver stable tac-tile performance. A s such, it is ready for user evaluation, which is covered in the following chapter. 134 Chapter 7 Browser User Evaluation T h i s chapter describes a formal evaluation that was performed on the browser prototype, aimed at measuring its performance in the hands of a human user. We measured the t ime required to browse web pages to find a piece of in-formation, such as a weather forecast, and compared the performance in three conditions: (1) the handheld prototype w i t h tactile feedback, (2) the prototype w i t h o u t tactile feedback, and (3) a mouse scroll wheel used as a discrete navigation control. To simulate a perceptually demanding mo-bile environment, s tudy participants were asked to simultaneously perform a distraction task (not identified to the participants as such) using pedals to respond to video cues. The data from the study, after normalizat ion to compensate for differences in difficulty between the various tasks, d id not demonstrate a significant performance difference between the two handheld prototype conditions, although the mouse scroll wheel was faster as ex-pected. Qual i ta t ive reaction to the tactile feedback was generally positive, w i th a small proport ion of users reporting min ima l conscious awareness of the s t imulat ion. In this chapter, the volunteers that part icipated in the study are referred to as users, and the terms user evaluation and user test are equivalent. Chapter 7. Browser User Evaluation 135 7.1 A i m s The purpose of the user test was to evaluate the effects of tactile feedback on performance and qualitative user experience in a handheld context using the previously described browser prototype. The browser prototype imple-ments a number of novel concepts, including the use of artificial tactile flow rendering to provide spatial cues. The results from the user test could serve to validate these concepts, and the u t i l i ty of tactile feedback on a hand-held or mobile device i n general, by providing a working demonstration and performance statistics. T h e s tudy addresses the following questions: 1. Does the presence of tactile feedback, in the form of a synthetic tactile flow signal, affect the t ime required to complete a web navigation task? 2. How does the experience of browsing and navigation on the handheld prototype compare to a more common paradigm (the mouse scroll wheel)? 3. W h a t are users' subjective reactions to the use of tactile feedback in this application? 4. W h a t are users' subjective reactions to the handheld prototype hard-ware and browser software? 5. A r e there any correlations between user demographics, such as age, gender, and previous mobile experience, and their performance in the browsing task wi th or without tactile feedback? Chapter 7. Browser User Evaluation 136 7.2 Study Design Formally, the user study is designed as a task time performance measure w i t h one pr imary within-subjects factor having three levels. T h e selected alpha level was 0.05. T h e full list of study variables is outl ined below. 7.2.1 S t u d y V a r i a b l e s P r i m a r y Dependent Variable TASK TIME : T h e amount of t ime (in milliseconds) required for the user to complete the task. The task begins when the user signals that they are ready to begin the task (and the browser screen is presented on the display). The task ends when the user signals that they have found the requested information. P r i m a r y Independent Variable (condition) 3 levels: SLIDER+ (slider with tactile feedback), SLIDER- (slider without tactile feedback), MOUSE (mouse scroll wheel used for navigation). Addi t iona l Dependent Variables T h e following were also measured: • Performance on the distraction task (described below in Section 7.6). • Individual l ink access times, which yielded the amount of t ime spent on each page. • A qualitative assessment battery (described below). Chapter 7. Browser User Evaluation 137 Addit ional Independent Variables The following factors were randomized: • T h e T A S K T Y P E (3 levels: W E A T H E R , T R A N S I T , and M O V I E S ) • T h e specific T A S K : - W E A T H E R group: 18 levels - T R A N S I T group: 12 levels - M O V I E S group: 10 levels - (total 40 different tasks) • Various parameters related to the distraction task (see Section 7.6 below). • User demographics. In addit ion, a variable, P R E S E N T A T I O N O R D E R , is defined as the presen-tat ion sequence number after flattening for C O N D I T I O N . More information on P R E S E N T A T I O N O R D E R is given in Section 7.3.4. 7.2.2 N o r m a l i z a t i o n for T a s k D i f f i c u l t y Each of the 40 T A S K S is similar in the sense that they a l l involve searching for information i n a web page hierarchy using identical navigation methods. However, because the information that the user is requested to find is sl ightly different in each T A S K , each T A S K w i l l have an intrinsic task completion time, referred to here as the T A S K D I F F I C U L T Y . The task difficulty is influenced by a number of factors, including the depth of the hierarchy at which the information is found, the posit ion of l inks on the web page (i.e., the distance Chapter 7. Browser User Evaluation 1 3 8 that the user must scroll to access the l ink) , and the complexi ty of the cues (e.g., l ink text) employed by the user during navigation. Since T A S K D I F F I C U L T Y is a confounding variable i n our analysis of the effect of C O N D I T I O N on task time, it is desirable to decouple the measured task times from the relative differences among tasks, using a normal izat ion procedure. The first step in this procedure requires the computat ion of a statist ical estimate of relative T A S K D I F F I C U L T Y . Tasks that are relatively more "difficult" (i.e., have a higher intrinsic task complet ion time) would have their measured task completion times depressed by the T A S K D I F F I -C U L T Y score, and vice-versa for relatively "easier" tasks. Theoret ical models such as F i t t s ' L a w and G O M S could provide pre-dictive estimates of task difficulty as a normalizat ion score, but there is no single existing published model that takes into account a l l the factors influencing task difficulty i n the specific context of the current experiment. Furthermore, it is not w i th in the scope of the present work to derive and val-idate an in-depth user model of browsing using the hybr id posi t ion/veloci ty control system on a handheld device using the specific content and layout that was developed for the user study. Therefore, to maximize accuracy, an empirical T A S K D I F F I C U L T Y score was used, calculated from the measured data according to the following procedure. A normalization vector was computed, having a normal izat ion score for each of the 40 T A S K S . For each T A S K , an average score (across a l l subjects and conditions) was computed and compared to the average for a l l T A S K S . The ratio of current task t ime to average overall t ime score was used as the "difficulty" score, and the normalizat ion factor was computed by taking its inverse. B y deriving the normalizat ion score from a pooled statistic across a l l Chapter 7. Browser User Evaluation 139 users, the task difficulty normalizat ion does not take into account indiv idual user differences. Sources for these differences might include variations in familiari ty w i t h certain tasks versus others, or differences i n the relative effects of various factors that influence the task t ime (navigation distance, verbal complexity, etc.). The val idi ty of the normalizat ion procedure in light of ind iv idua l subject differences w i l l be explored i n Section 7.8.5. 7.3 Methodology A tota l of 16 volunteers were recruited for the study (three pilot subjects and thirteen main study subjects), which was performed under the auspices of U B C Ethics Approva l #B01-0470 (see A p p e n d i x A ) . E a c h study session took approximately 50 minutes and consisted of: 1. Briefing and collection of demographic data. 2. A pre-task attitudes survey. 3. Verbal delivery of task instructions. 4. A training, or practice session, minus the distractor task. 5. A second training session, including the distractor task. 6. The ma in study task (described further below). 7. A post-task attitudes survey. 8. A post-task interview. 9. Compensat ion ($10 per subject). Chapter 7. Browser User Evaluation 140 Gender 38% female Age median: 28.5 std. dev: 4.6 Right handed 81% Previous experience wi th a musical instrument 77% A m o n g subjects w i th previous musical instrument ex-perience, number of years played median: 7 std. dev: 4.0 Regular user of a mobile phone 77% Regular user of a P D A 15% Regular user of a game controller w i th vibrotacti le feedback 15% Regular user of a mouse w i t h scroll wheel 94% Table 7.1: Demographic characteristics of the subject pool (16 subjects) that was used for the browser user evaluation, including both pilot subjects and ma in study subjects. Due to var iabi l i ty in the experimental procedure dur ing the three pilot runs (mostly related to fixing bugs in the computer setup), only data from the 13 main study subjects were used i n the quantitative and qualitative analyses of the experiment. 7.3.1 Recruitment of Study Participants Part ic ipants were recruited through the H C I @ U B C subjects database, and through friends and associates of the author who met the study criteria. The conditions for par t ic ipat ing in the study were a good command of Engl i sh and no prior experience wi th the browser prototype or the present research Chapter 7. Browser User Evaluation 141 i n general. T h e final corpus had demographic characteristics as described i n Table 7.1. T h e subject pool appears to be a typica l random selection of potential users of general mobile data services, sl ightly skewed towards younger participants due to the university setting. Since no dis t inct ion was made between pilot subjects and main study subjects other than the order in which they booked their appointments, bo th groups of subjects can be considered to be randomly drawn from the dis t r ibut ion described i n Table 7.1. 7.3.2 User Test Environment A l l user tests were conducted in a dedicated, soundproof usabil i ty testing lab at the Univers i ty of B r i t i s h C o l u m b i a during normal business hours or in the early evening. The room was arranged as shown in Figure 7.1. The following devices were arranged in front of the study participant: 1. The tactile display prototype, including an embedded L C D display. Because the original ly specified 2.5-inch L C D panel malfunctioned prior to the start of the experiment and could not be prompt ly re-placed, it was necessary to use a 3.5-inch N T S C T F T colour L C D monitor, attached to the top case w i t h double-sided tape. Th i s re-sulted in an increased weight, especially i n the top part of the device. 2. A mouse (Logitech Scrol l Mouse) w i t h scroll wheel. 3. A standard 101-key P C keyboard. The two largest keys, E N T E R and S P A C E , were used for signalling readiness to begin the task and completion of the task,. respectively, and were labelled " S T A R T " and " S T O P " , using 7.5-cm wide yellow sticky notes. Chapter 7. Browser User Evaluation 142 F i g u r e 7.1: B r o w s e r user tes t e n v i r o n m e n t . Chapter 7. Browser User Evaluation 143 4. A second P C keyboard (not shown in the diagram), connected to the experiment P C and used by the facilitator for resetting the distractor task. 5. A 17-inch colour T F T L C D P C monitor running at 1280x1024 res-olut ion. P rompts to the user were displayed i n 120-pixel high type, centred in the display on a white background. 6. A set of pedals, of the type used commonly in home game systems for dr iv ing games ( A C T L A B S , Performance Pedals). Part ic ipants were instructed to place one foot over each pedal. The pedals themselves were modified to function as left and right mouse buttons, and were connected to the P C v i a a U S B mouse. 7. A browser PC, running the Linux-based browser prototype software and connected to the tactile display, L C D , mouse w i t h scroll wheel, keyboard, and local-area network. 8. A n experiment PC, running Windows 2000 and the browser experi-ment software (described in Section 7.7), and connected to the P C monitor, pedals, and local-area network. 9. The driver electronics, connected between the browser P C and tactile display prototype. D u r i n g the experiment, the facilitator remained in the room wi th the par-ticipant, delivering instructions read from the verbal protocol (Appendix A ) and verbally collecting the information the part icipant was asked to retrieve. The experimenter also remained logged in to the browser P C v i a a remote terminal connection from a separate laptop, performing such functions as Chapter 7. Browser User Evaluation UA start ing and stopping the browser prototype software, changing configura-t ion settings (for example, S L I D E R + to M O U S E ) , and moni tor ing the data collection process. T h e experimenter also used a keyboard connected to the experiment P C to signal task completion and to present the next task to the part icipant when they verbally indicated that they were ready to proceed. In the S L I D E R - condit ion, a l l software settings were the same as the S L I D E R + condit ion, except the final amplifier electronics were powered off. T h i s ensured that there were no confounding variables, such as t iming differ-ences, introduced as a result of changing conditions between S L I D E R + and S L I D E R - . 7.3.3 B r i e f i n g a n d C o l l e c t i o n of D e m o g r a p h i c D a t a Study participants were first asked to read and sign the standard U B C Eth ics Consent form (included i n A p p e n d i x A ) , and each part icipant was given a copy for their records. A l l participants were verbally reminded that they could abort their part icipat ion in the study at any t ime without penalty (i.e., they would s t i l l receive their compensation). Par t ic ipants were also reminded that their frank and honest opinions were requested, and that the purpose of the study was to obtain a performance assessment of the device, not the study participant. A l l s tudy participants then completed the demographic survey and pre-test attitudes survey; the results are shown in Table 7.1 and Figure 7.7. A s noted in the verbal protocol (Appendix A ) , a l l participants were instructed to hold the device in their left h a n d 1 w i t h their thumb resting 1 Only left handed operation is supported on the handheld prototype. This is also the case with almost all commercial handheld products having side-mounted controls. See Section 4.1. Chapter 7. Browser User Evaluation 1 4 5 on the tactile display. Part ic ipants were to ld not to abrade or pick at the surface of the T D , and were reminded when necessary to mainta in a hand posit ion such that the thumb was in full contact w i th a l l 8 elements of the T D . W h e n the mouse scroll wheel was used as an input device, the user was also asked to use their left hand to operate the mouse. In that case, the device (sti l l required for visual display) was placed on the table and propped up at an an angle for better visibi l i ty. 7.3.4 T a s k B l o c k s The study is organized into B L O C K S of 6 T A S K S each. E a c h B L O C K is per-formed under one C O N D I T I O N ( S L I D E R + , S L I D E R - , or M O U S E ) . W i t h i n a group of three B L O C K S , each condit ion is presented; and each subject re-ceives a sequence that is randomly drawn from the 6 possible permutations of condit ion presentation order. Between B L O C K S , s tudy participants get a one-minute break to relax and stretch. W i t h i n each B L O C K , the 6 T A S K S are also presented i n a randomized order. T A S K S may be categorized into T A S K T Y P E S , consisting of W E A T H E R , T R A N S I T , and M O V I E S , each of which is represented twice per block. There are 6 possible permutations of the presentation order of 3 T A S K T Y P E S . For each B L O C K , a presentation order is drawn at random from this set, and the sequence is repeated twice to form a 6 - T A S K B L O C K . F ina l ly , for each T A S K T Y P E , the presentation order of specific T A S K S is randomized. The presentation sequence of an example run through the main part of the study can be visualized as follows (six blocks shown): ( S L I D E R + ) A 1 6 . C 5 . B 8 . A 1 1 . C 9 . B 1 — > (break) —> ( M 0 U S E ) C 3 . B 4 . A 3 . C 2 . B 5 . A 7 — • (break) — • Chapter 7. Browser User Evaluation 1 4 6 ( S L I D E R - ) B 3 . A 1 5 . C 1 . B 2 . A 1 7 . C 4 — > (break) ( S L I D E R + ) A 2 . B 1 . C 8 . A 5 . B 1 1 . C 6 — > (break) ( M O U S E ) C 1 . A 1 8 . B 9 . C 7 . A 1 . B 9 — • (break) -( S L I D E R - ) B 1 0 . C 5 . A 4 . B 6 . C 9 . A 1 0 — • (break) where each letter ( A , B , or C ) represents a T A S K T Y P E , the number following the letter represents a specific T A S K , and each line is a B L O C K . Note that the sequence of T A S K S does not repeat un t i l a l l tasks of a part icular T A S K T Y P E have been exhausted; the sequence for that T A S K T Y P E then repeats in the same order as before. When , for example, a l l 1 2 tasks of T A S K T Y P E A have been exhausted, the next T A S K drawn from T A S K T Y P E A is the same as the first task that was drawn during the first pass through the sequence. T h e meta-variable P R E S E N T A T I O N O R D E R is the ordinal T A S K x B L O C K presentation sequence number, w i th C O N D I T I O N S "flattened" out. Formal ly : POtask = ttask + T(ceil(btask/3) - 1 ) ( 7 . 1 ) Where: • POtask is the P R E S E N T A T I O N O R D E R number for the given T A S K , • ttask is the sequential presentation number for the T A S K w i th in its B L O C K (ranging from 1 to 6 ) • hask is the sequential presentation number of the B L O C K to which the T A S K belongs (ranging from 1 to 9 ) • T is the number of T A S K S per B L O C K ( 6 ) Chapter 7. Browser User Evaluation 147 In the example above, T A S K S A16 , C 3 , a n d B 3 each have a P R E S E N -T A T I O N O R D E R of 1, and T A S K S C 5 , B 4 , a n d A 1 5 have a P R E S E N T A T I O N O R D E R of 2. The sequence continues through T A S K S A 2 , C l , a n d BIO, which have a P R E S E N T A T I O N O R D E R of 7. B y grouping T A S K S that are pre-sented at approximately the same ordinal posit ion in different C O N D I T I O N S under a single P R E S E N T A T I O N O R D E R score, a within P R E S E N T A T I O N O R D E R analysis of variance of task times is enabled. The results from this analysis are presented in Section 7.8.1. — The full task inventory is included i n A p p e n d i x A ; however, some examples are: A l W h a t is the weather in London today? A 2 W h a t w i l l the weather be like in London tomorrow? A 3 W h a t w i l l the weather be like in London the day after tomorrow? A 4 W h a t is the weather in Paris today? B l If you take the 99 B- l ine from U B C at 1pm, when w i l l you arrive at Broadway station? C l W h e n is the movie "L 'Enfan t" playing at the Ridge Theatre? T h e various randomizations and permutations are handled automatical ly by the JavaScript component of the Test Software (described in Section 7.7). E r ro r cases, defined as the user cl icking on the wrong l ink, becoming distracted (in ways other than typical ly induced by the distraction task), or experiencing a problem, were recorded but excluded from the analysis of task times. W h e n a participant made an error, he/she was asked to inform the facilitator of the error but complete the task anyway. A "back Chapter 7. Browser User Evaluation 148 to previous page" l ink was provided in a standardized locat ion on a l l pages of the sample corpus for this purpose. 7.3.5 Training Sessions Part ic ipants first completed two t ra ining sessions to become familiar w i t h the tasks. In the first t ra ining session, subjects completed 3 shortened B L O C K S of 2 T A S K S each, in each of the three conditions ( S L I D E R + , S L I D E R - , or M O U S E ) . In the second t ra ining session, subjects also completed 3 short-ened B L O C K S of 2 T A S K S each, but wi th the addi t ion of the distract ion task (described below i n Section 7.6). 7.3.6 Main Data Collection Session In the main part of the study, subjects completed 9 full B L O C K S of 6 T A S K S each, or 54 T A S K S i n total . 7.3.7 Post-Task Assessment Fol lowing the main data collection task, subjects were given a one-minute break and then asked to fil l out a post-task attitudes survey. The survey is included in A p p e n d i x A , and the results are discussed in Section 7.9. Fol lowing the survey, the facilitator conducted an interview using the protocol documented in Append ix A . T h e participants were then given their compensation and were free to go after signing a receipt. 7.4 P i l o t S tudy A pilot study was first conducted wi th three participants to assess, debug, and optimize the experiment protocol and browser prototype design. Pa -Chapter 7. Browser User Evaluation 149 rameters i n the browser prototype software were continuously opt imized during the pilot phase using a part ic ipatory design model (the final values are documented in Table 6.1). In addit ion, the decision to implement the hybr id control spring design (Figure 6.5) was made based on user feedback from the pilot study. 7.5 S t i m u l i U s e d i n the S tudy The final selection of haptic icons used for the study are shown in Figure 7.2. T w o types of haptic icons were used. Large headings were marked wi th the high-frequency grating icon, and links were marked wi th the modified single-bump crater icon. To arrive at these haptic icon selections, more than 20 variants of haptic icon representations were created using the image-based rapid prototyping method (Section 4.9.6) and part ic ipatory design feedback was obtained during the pilot phase and used immediately to tweak and optimize the icons in an on-line fashion. Icons were selected for high salience (perceived amplitude), dist inguishabil i ty from one another, and clari ty of the directional tactile flow signal. To achieve these design goals, several points were taken into considera-t ion. Fi rs t , the results from both the M D S haptic icon discr iminabi l i ty study (Section 5.4.1) and the Image Browser User Test (Section 6.2.3) indicated that the most salient dimension employed by users for dist inguishing haptic icons was whether the icons were high frequency and periodic or low fre-quency and relatively non-periodic, therefore, the two selections were made on opposite ends of this spectrum. The decision to use only two icons was based on the following justification: the browser user study was intended pr imar i ly to address the effectiveness of tactile flow as a design concept; pre-Chapter 7. Browser User Evaluation 150 crater -50V +50V grating -50V +50V Figure 7.2: Hapt ic icons used in the browser user study. For each of the two icons, the volt image representation, voltage trace, and crop of an associated page element are shown. T h e height of the crater representation (21 page map units) is the same as the height of a l ink (in pixels) when rendered i n the browser window. T h e rendered height of headings may differ slightly from the height of the associated grating icon; in that case, the haptic icon is top-justified. Chapter 7. Browser User Evaluation 151 vious studies had already addressed the dist inguishabil i ty of a wide variety of haptic icons; and the use of a large corpus of haptic icons might intro-duce undesired var iabi l i ty in the salience of the tactile flow signal which the present study was not powered to evaluate. O n the other hand, i f the effec-tiveness of tactile flow as a spatial cue was proven, a subsequent study could examine the detailed design of an opt imal ly sized corpus of haptic icons for use w i t h the browser. A n i m a t e d icons containing t ime-varying moving components were ex-plored, but since the goal of the experiment was to demonstrate the useful-ness of the tactile flow signal as a spatial cue, it was felt that the icons at this stage of development should not interfere w i th the tactile flow signal by superimposing addi t ional motion. Therefore, static icons were selected. T h e decision to use the grating icon for headings and the crater icon ' for l inks, as opposed to vice-versa, was informed by the observation that the directional sensation of tactile flow was much stronger for the non-periodic signals, and links were more frequent and irregularly distr ibuted than headings in the test web page corpus, thus the importance of effective tactile flow display for links was higher. F ina l ly , the icons were designed such that, using the 3- to- l subtaxel rendering setting (Section 6.4.10), ind iv idual piezos would s t i l l reach max-i m u m (or near-maximum) amplitude displacement. To avoid icon overlap and because pilot testing d id not reveal any significant advantages i n terms of tactile flow perception, the page map shrinking feature of the browser prototype software was not used. Chapter 7. Browser User Evaluation 152 7.6 Distraction Task Div ided attention between the information appliance and the user's envi-ronment is a key feature of mobile device usage in the real world [40]. To simulate this experience while retaining the benefits of a controlled labora-tory testing environment and without incurr ing the costs of engineering the prototype for true mobil i ty, a visual distraction task was implemented as part of the experiment protocol. Eve ry seven seconds (beginning seven seconds after the start of the browsing task), the user was prompted to perform an independent task wi th his /her feet, so as to not affect the hand-based interaction wi th the mobile device. The prompt was displayed on a P C monitor on the desk, requiring the user to shift his/her gaze from the hands (below eye level) to horizontal eye level. The prompt was displayed in large (120-pixel high) type, such that a change could be observed using peripheral vision while looking at the device held i n the hand; however, the prompt was displayed i n A L L C A P I T A L letters in a serif font to minimize preattentive processing of the message. Since the various prompts were morphologically similar and complex, the user was forced to read the message each t ime. There were six distraction task variants, each of which involved respond-ing to one of the following commands: 1. Press the left pedal once. 2. Press the left pedal twice. 3. Press the right pedal once. 4. Press the right pedal twice. Chapter 7. Browser User Evaluation 153 5. Press the left, then the right pedal. 6. Press the right, then the left pedal. To balance in i t i a l difficulty against increasing user famil iar i ty as they acquired experience w i t h the task, only the first four distract ion tasks were used dur ing the first half of the experiment (3 out of 6 blocks in t ra ining phase 2, and 4 out of 9 blocks in the main experiment); after that, the full selection of 6 tasks were used. Feedback was given at seven-second intervals together w i t h a refresh of the task prompt. A cumulative positive reinforcement model was used, s im-ilar to a typical video game design. Successive correct performance resulted in the onscreen presentation of a reward image, consisting of a briefly ( 1 second) animated character such as a smiley face, w i t h increasing valence. For example, the smileys became larger, more numerous, more animated and more embellished to reinforce a string of correct performance. Six levels of reinforcement were supported. If the user provided an incorrect input dur-ing a seven-second interval, the correct-response counter was reset to zero and the background of the prompt area was changed from white to red. A correct response during the next seven-second interval resulted in the back-ground reverting to white, the counter being incremented by one, and the appropriate reward animation being presented. T h e effectiveness of the distraction task is assessed from a quantitative perspective in Section 7.8.8 and from a qualitative perspective in Section 7.9.2. Chapter 7. Browser User Evaluation 154 7.7 Browser Exper imen t Software A s described in Section 7.3.2, User Test Environment , two P C s were used dur ing the experiment, one running the browser prototype software, and the other running the experiment software. The experiment software provides the following functionality: 1. Randomiz ing the selection of tasks i n blocks, using the procedure de-scribed in Section 7.3.4. 2. Disp lay ing task prompts to the user (e.g., "What is the weather like in Paris today?") . 3. Disp lay ing the t imed distractor task prompts to the user (e.g., "Press the left pedal twice."). Described further in Section 7.6. 4. P rov id ing feedback to the user on their performance in the distractor task. 5. Recording the distractor task events from the pedals, and tagging the data w i t h t iming information. 6. Sending the collected data to the facilitator's email account v i a the network. 7. Reading the keyboard events from the facilitator. 8. P rompt ing the user to take a one-minute break. In addi t ion, for the most accurate t iming, the browser prototype software also includes several functions related to the experiment: 1. Receiving events from the keyboard when the user signals the start and end of the task. Chapter 7. Browser User Evaluation 155 2. Disp lay ing a modal dialog box blocking interaction wi th the browser unt i l the user signals the start of the task. 3. Support ing the mouse scroll wheel as a configurable alternate input device. 4. Resett ing the browser upon user-signalled completion of the task. 5. D u m p i n g t iming data to the terminal (recorded on the experiment facilitator's remote laptop). T h e browser experiment software is designed for any standards-based web browser (it was run on M o z i l l a Firefox 1.0 under Windows 2000 for the experiment) and consists of the following components (included in Append ix E): 1. A set of X H T M L files w i t h embedded style sheet (CSS 1.0) informa-tion, comprising the presentation layer of the software. 2. A set of JavaScript files, comprising the logic layer. 3. A C G I (Common Gateway Interface 1.1) script, based on the free " F o r m M a i l " script, running on an off-site server that accepts form data and forwards it to the facilitator's email account. The general architecture is that of a web applicat ion using the modern A J A X (Asynchronous JavaScript and X M L ) methodology, running mostly on the client side. It uses the X M L H t t p R e q u e s t object supported in a l l current (as of 2006) browsers to send data to a server-based C G I script in an online fashion, without reloading the page or causing a visual interrup-t ion to the user. T h i s allows for a highly interactive, portable experiment Chapter 7. Browser User Evaluation 156 management applicat ion while providing convenient email-based collection of da ta w i th redundant copies distr ibuted across networked computers for m a x i m u m safety. To the author's knowledge, the present software is the first known applicat ion of this technique for usabil i ty experiment management. T h e logical flow of the browser experiment software is as follows: 1. The appropriate H T M L file (training phase 1, t ra ining phase 2, or main experiment) is loaded into the browser by the facilitator. 2. The H T M L file instructs the browser to load and executes the associ-ated JavaScript code. 3. A start screen is presented, prompt ing the facilitator to enter a part ici-pant number, and in the main experiment version, an optional number of blocks (default is 9). The participant number is provided s imply for convenience and is appended to a l l data output by the program. 4. W h e n the participant indicates to the facilitator that he/she is ready to begin the first task, the facilitator presses a form submit but ton. 5. The JavaScript component creates an array for each of the three T A S K T Y P E S . In the main experiment version, the arrays are randomly per-muted, and the tasks are sequentially assigned to B L O C K S that are stored as associative arrays. 6. A display refresh is performed without reloading the page, accom-plished by manipula t ing layer v is ib i l i ty parameters in the style sheet. 7. The first task prompt is displayed to the user. The software waits for a keyboard input from the facilitator, indicat ing that the part icipant Chapter 7. Browser User Evaluation 157 has read the prompt and has pressed the S T A R T key, signalling the browser prototype component to start the task. 8. The software enters a loop, presenting a distractor task prompt to the user at seven-second intervals. The software checks to ensure that no two sequentially presented distractor tasks are alike. 9. Pedal input from the user is read and stored. W h e n the seven-second interval expires, the input is checked against the task requirement and feedback (correct or incorrect) is provided to the user. 10. W h e n the facilitator presses a key on the keyboard, the task is finished, the distractor task prompt is removed, and the next task prompt is presented on the screen. 11. W h e n a block has been completed, the software sends an inter im copy of the data log to the server-based C G I script (and thus to the facil-i tator 's email account). The user is prompted to take a one-minute break v i a a modal dialog. 12. W h e n al l blocks have been completed, the software sends the final copy of the data log to the server and presents a modal dialog thanking the user for his/her part icipat ion. Since the page is never refreshed, the data is retained in a text input field that is hidden below the scroll boundary of the window. Should the email transmission fail, the data can s t i l l be recovered from the page using copy and paste on the client P C . Chapter 7. Browser User Evaluation 158 7.8 Quantitative Results A total of 645 val id, error-free observations of T A S K T I M E were gathered i n each of the 3 C O N D I T I O N S across the 40 T A S K S and 13 participants, and used i n the quantitative analysis of performance. 7.8.1 Effect of Condit ion on Task Time The results of a repeated-measures A N O V A on the task t ime data across a l l 13 non-pilot subjects is shown in Table 7.2. For this analysis, T A S K T I M E was compared wi th in P A R T I C I P A N T and P R E S E N T A T I O N O R D E R variables. If an error occurred in any of the three C O N D I T I O N S for a given level of P R E -S E N T A T I O N O R D E R for a given subject, the data for that level of level of P R E S E N T A T I O N O R D E R was not considered i n this analysis; only those com-binations of task x subject w i th full repeated measures across a l l conditions were used, resulting in a slightly lower number of observations (N) than the overall data set. T h e results show a stat ist ically significant difference in T A S K T I M E across the three different C O N D I T I O N S . However, when the C O N D I T I O N S are exam-ined in a pairwise fashion, there is a stat ist ically significant difference be-tween the M O U S E condi t ion and the two S L I D E R conditions, but not among the two S L I D E R conditions (i.e., w i th and without active tactile feedback). 7.8.2 Individual Subject Differences in Performance C o u l d the lack of a stat is t ical ly significant difference between the two S L I D E R conditions be due to ind iv idua l differences in the way users respond to the presence or absence of tactile feedback? The data were analyzed separately for each subject (with appropriate Bonferroni correction for the groupwise Chapter 7. Browser User Evaluation 159 Condition Mean Task Time Std. Error N M O U S E 12166.34 4415.81 181 S L I D E R - 18200.52 5453.77 181 S L I D E R + 18445.01 4682.06 181 Overall A N O V A Conditions F p Overal l 132.750 0.000 Pairwise Comparison (Bonferroni Corrected) 95% Confidence Interval Conditions A M e a n Low High p M O U S E vs S L I D E R - -6034.18 -7182.19 -4886.19 0.000 M O U S E VS S L I D E R + -6278.67 -7292.42 -5264.93 0.000 S L I D E R - VS S L I D E R + -244.49 -1373.88 884.91 1.000 Table 7.2: Results from a within-subjects, w i t h i n - P R E S E N T A T l O N O R D E R analysis of measured task time. A l l measurement units are in milliseconds. error rate), and the results are shown in Table 7.3. A s shown in Table 7.3, the magnitude of the difference between S L I D E R + and S L I D E R - conditions was small for each participant, and none of the differences was stat ist ically significant. The evidence therefore does not support the hypothesis that opposing effects of C O N D I T I O N i n ind iv idua l subjects was responsible for the overall lack of statist ical significance. Chapter 7. Browser User Evaluation 160 7 . 8 . 3 Effect of Task on Task T i m e The relationship between T A S K and measured T A S K T I M E is shown in F i g -ure 7.3. T h e graph clearly shows variat ion in the task times that suggest differences in task difficulty. The task requiring the most t ime to complete was: C I O W h a t rat ing d id the movie "The Promise" receive in its review in the Washington Post? T h e task requiring the shortest t ime to complete was: A 1 5 W h a t w i l l the weather be like i n San Francisco the day after tomorrow? E x a m i n i n g the web pages in the test corpus, we see that task C I O re-quires a m i n i m u m traversal distance of 156 + 852 + 563 + 497 = 2068 pixels or page map units: the longest in the corpus. B y comparison, task A 1 5 requires traversal of 99 + 484 + 254 + 176 = 1013 pixels or page map units: the shortest in the corpus. The observations are therefore generally i n accordance wi th G O M S and F i t t s ' L a w principles about the effects of distance and keystroke input on task completion time. 7 . 8 . 4 Effect of Task x Cond i t ion on Task T i m e C o u l d the lack of a stat ist ically significant difference between the two S L I D E R conditions be due to opposing effects in different T A S K S ? Figure 7.4 shows the T A S K T I M E data sorted by T A S K across each of the conditions. A l t h o u g h the data is somewhat noisier due to a lower number of samples i n each condi t ion versus the overall task time, each of the three conditions follows the same general trend wi th regard to variat ion in task difficulty; we do not observe any large interaction effects between C O N D I T I O N and T A S K T I M E Chapter 7. Browser User Evaluation 161 Time vs Task I 30000 Task Figure 7.3: The 40 T A S K S are sorted according to measured T A S K T I M E and displayed here wi th standard error bars. Chapter 7. Browser User Evaluation 162 A v e r a g e T a s k T i m e v s C o n d i t i o n S u b j . N a l l c o n d MOUSE SLIDER- SLIDER+ P 4 49 18878 16282 20527 19986 1.000 5 54 16294 13487 17358 18036 1.000 6 53 15748 10841 18227 18314 1.000 7 49 16892 10384 19566 19960 1.000 8 47 17775 13170 20226 19558 1.000 9 53 13339 • 9155 15875 15084 1.000 1 0 51 13203 10654 13763 15376 0.968 1 1 46 15513 11392 17517 17963 1.000 1 2 49 17509 13987 19053 19529 1.000 1 3 48 13650 9055 15613 16283 0 .932 1 4 46 18171 12587 21419 20913 1.000 1 5 50 21640 17069 23146 24026 1.000 1 6 50 13187 9248 15413 15031 1.000 A L L 645 15287 12056 18245 18499 1.000 T a b l e 7.3: M e a s u r e d t a s k t i m e for e a c h o f t h e 13 m a i n tes t p a r t i c i p a n t s . " A l l c o n d " s t a n d s for " a l l c o n d i t i o n s " . T h e p -va lues are t h e resu l t o f B o n f e r -r o n i c o r r e c t e d t - tes ts b e t w e e n t h e t w o SLIDER c o n d i t i o n s . A l l m e a s u r e m e n t u n i t s a re i n m i l l i s e c o n d s . F u r t h e r s t a t i s t i c s a re p r o v i d e d i n A p p e n d i x B . Chapter 7. Browser User Evaluation 163 across the TASKS. Furthermore, al though the performance i n the MOUSE condit ion in the most difficult TASK (CIO) is worse than the performance i n the SLIDER- condi t ion i n the least difficult TASK (A15), when compared i n a TASK-matched fashion, the performance in the MOUSE condi t ion is always better than the performance in either of the SLIDER conditions. T h i s does not support the hypothesis that the lack of an observed difference i n the two SLIDER conditions was due to an interaction between task difficulty and CONDITION. 7.8.5 Validation of Task Difficulty Normalization Before proceeding w i t h the task difficulty normalizat ion as described in Sec-t ion 7.2.2, we should verify that the observed variat ion in performance on a task-by-task basis was indeed due to intrinsic TASK-related sources, and not due to the effect of the study independent variables. A s discussed i n Section 7.8.3, bo th the empir ical data and theoretical models support the hypothesis that variat ion i n task difficulty is associated wi th navigational distance. Furthermore, as Figure 7.4 indicates, the general t rend of increas-ing difficulty w i t h certain tasks is observed i n all three CONDITIONS i n a slightly noisy, but mostly monotonic fashion. Combined , these observations lend val idi ty to the concept of task difficulty determined by intrinsic factors, which may be "normalized out" using an empir ical model of task difficulty. Considerat ion was given as to whether to use a single, pooled normal-izat ion score from across the three conditions for each TASK (i.e., a nor-mal izat ion vector), or to use separate normalizat ion scores for each TASK x CONDITION (i.e., a 3 x 40 normalizat ion matrix). There was no reason to reject the hypothesis of similar effects of task difficulty across al l three Chapter 7. Browser User Evaluation 164 Time vs Task vs Condition Task Figure 7.4: The coloured lines show measured TASK TIME vs. TASK in each of the three conditions, while the heavy black line shows the overall (averaged) t ime across a l l conditions. The TASKS are sorted in order of overall measured TASK TIME. Po lynomia l curve fit lines are also shown for each of the conditions. The data are noisy due to a low and variable number of observations at each point due to the randomizat ion of task assignments. T h e following data points had zero observations and are reflected i n the graph as smooth lines drawn between adjacent observations: Tasks A 1 3 and B l l in the SLIDER- condit ion, and task A 1 7 in the MOUSE condi t ion. Chapter 7. Browser User Evaluation 165 c o n d i t i o n s b a s e d o n t h e d a t a i n F i g u r e 7.4. F u r t h e r m o r e , d u e t o t h e r a n -d o m i z a t i o n o f t a s k s , s o m e ce l l s i n t h e 3 x 40 m a t r i x w o u l d h a v e h a d z e r o o b s e r v a t i o n s ( the m a x i m u m n u m b e r o f o b s e r v a t i o n s i n . a c e l l was 13) ; u s i n g t h e o v e r a l l d a t a t h e lowest n u m b e r o f o b s e r v a t i o n s w a s 9, a n d t h e h i ghes t w a s 24 (see A p p e n d i x B ) . F o r these reasons , i t was d e c i d e d t o d e t e r m i n e t h e n o r m a l i z a t i o n sco re for a g i v e n t a s k b y c o n s i d e r i n g t h e d a t a ac ross a l l t h ree c o n d i t i o n s . T h e n o r m a l i z a t i o n scores were a p p l i e d t o t h e measured t a s k t i m e s t o d e t e r m i n e t h e normalized t a s k t i m e , a n d r a n g e d f r o m 0.733 for t h e t a s k r e q u i r i n g t he m o s t t i m e t o c o m p l e t e ( C I O ) , to 1.474 for t h e fas tes t t a s k ( A 1 5 ) . 7.8.6 Analysis Using Normalized Task T i m e O v e r a l l E f f e c t o f C o n d i t i o n o n N o r m a l i z e d T a s k T i m e A r e v i s e d v e r s i o n o f T a b l e 7.2, u s i n g t i m e s n o r m a l i z e d for t a s k d i f f i cu l t y , is s h o w n i n T a b l e 7.4. D a t a v a r i a t i o n d u e t o d i f fe rences i n t a s k d i f f i c u l t y is r e d u c e d , as e v i d e n c e d b y the lower s t a n d a r d e r r o r a n d i n c r e a s e d F s t a t i s t i c ; h o w e v e r , d e s p i t e i n c r e a s i n g t h e d e t e c t i o n p o w e r t h r o u g h n o r m a l i z a t i o n , t h e o v e r a l l r esu l t s b e t w e e n t h e t w o SLIDER c o n d i t i o n s a re s t i l l no t s t a t i s t i c a l l y s i g n i f i c a n t . I n d i v i d u a l S u b j e c t D i f f e r e n c e s i n N o r m a l i z e d T a s k T i m e x C o n d i t i o n N o r m a l i z i n g t h e t a s k scores d i d n o t af fect t h e s i g n i f i c a n c e o f t h e o b s e r v a t i o n s o n a n i n d i v i d u a l user b a s i s . F u l l s t a t i s t i c s a re p r o v i d e d i n A p p e n d i x B . Chapter 7. Browser User Evaluation 166 Condition Mean Task Time Std. Error N M O U S E 11474.34 3957.91 181 S L I D E R - 17011.57 4455 .32 181 S L I D E R + 17179.28 3653 .15 181 Overall A N O V A Conditions F p O v e r a l l 163.275 0.000 Pairwise Comparison (Bonferroni Corrected) 95% Confidence Interval Conditions AMean Low High p M O U S E v s S L I D E R - -5537 .25 -6456 .06 -4618.41 0.000 M O U S E v s S L I D E R + -5704 .94 -6499 .99 -4909 .89 0.000 S L I D E R - v s S L I D E R + -167 .707 -945 .67 610.26 1.000 T a b l e 7.4: R e s u l t s f r o m a w i t h i n - s u b j e c t s , w i th in - P R E S E N T A T l O N ORDER a n a l y s i s o f normalized t a s k t i m e . U n i t s a re i n m i l l i s e c o n d s . 7.8.7 L e a r n i n g / P r a c t i c e Effects E v e n i f t h e d i f fe rence b e t w e e n t h e t w o S L I D E R c o n d i t i o n s was no t s t a t i s t i -c a l l y s i gn i f i can t ac ross t h e en t i r e e x p e r i m e n t , c o u l d t h e p resence o f t a c t i l e f e e d b a c k have a p o s i t i v e o r adve rse effect o n l e a r n i n g t h e t a s k a n d o p e r a -t i o n o f t he d e v i c e ? T o add ress t h i s q u e s t i o n , we sp l i t t he d a t a i n t o b i n s fo r e a c h o f t h e t h r e e t h r e e - B L O C K sets c o m p r i s i n g a f u l l r u n t h r o u g h e a c h o f t h e t h ree C O N D I T I O N S . T h e d a t a are p resen ted i n F i g u r e 7.5. A t y p i c a l Chapter 7. Browser User Evaluation 167 asymptotic performance improvement is observed in each of the three C O N -DITIONS; however, it is unclear whether there are significant interactions i n the two S L I D E R conditions. The shape of the curves suggests that more fa-mi l ia r i ty w i t h the device or practice w i t h the experimental paradigm would not have produced a significant difference between the two S L I D E R condi-tions, although the present data do not suggest an expected outcome for a long-term, longi tudinal study. Time vs Block-of-Blocks 21000 1 — 9000 • , 1 1 2 3 Block-of-Blocks Figure 7.5: The measured (mouse, slider-, slider+) and normalized (nmouse, nslider-, nslider+) task times are plotted versus the presentation order se-quence of a set of 3 B L O C K S ("block-of-blocks") comprising a full run through each of the three C O N D I T I O N S . Chapter 7. Browser User Evaluation 168 F o r i n c r e a s e d r e s o l u t i o n , t h e d a t a were a n a l y z e d b y t h e p r e v i o u s l y de -s c r i b e d P R E S E N T A T I O N O R D E R m e t a - v a r i a b l e . T h e resu l t s w i t h b o t h m e a -s u r e d a n d n o r m a l i z e d t a s k t i m e s a re s h o w n i n F i g u r e 7.6. A g a i n , t y p i c a l l e a r n i n g c u r v e s a re o b s e r v e d , w i t h a s l i gh t i nc rease i n t a s k t i m e at t h e e n d l i k e l y a t t r i b u t a b l e t o f a t i gue . T h e d a t a d o no t i n d i c a t e a n y effect o f t a c t i l e f eedback o n l e a r n i n g ; h o w e v e r , t h e r e l a t i v e f la tness a n d c o n s i s t e n c y o f t h e l e a r n i n g c u r v e s l e n d con f i dence t o t h e v a l i d i t y o f t h e o b s e r v a t i o n s . 7.8.8 Quantitative Validation of Distraction Task T h e d a t a g a t h e r e d b y t h e e x p e r i m e n t P C were a n a l y z e d t o ensu re t h a t p a r t i c i p a n t s were p e r f o r m i n g t h e d i s t r a c t i o n t a s k p e r t h e e x p e r i m e n t p r o t o -c o l , a n d t h e resu l t s a re s u m m a r i z e d i n T a b l e 7.5. W h i l e s o m e p a r t i c i p a n t s s h o w e d h i g h e r a c c u r a c y t h a n o t h e r s o n t h e d i s t r a c t i o n t ask , t h e o v e r a l l a c c u -r a c y r a t e was 8 1 % , s h o w i n g t h a t p a r t i c i p a n t s were a t t e n d i n g t o a n d p e r f o r m -i n g t he d i s t r a c t i o n t a s k . T h e q u a l i t a t i v e f i n d i n g s r e l a t e d t o t h e d i s t r a c t i o n t a s k a re d i s c u s s e d i n S e c t i o n 7.9.2. 7.9 Qual i ta t ive Resul ts 7.9.1 Pre-Task Attitudes Survey T h e resu l t s f r o m the p r e - t a s k a t t i t u d e s s u r v e y a re s h o w n i n F i g u r e 7.7. A l l q u e s t i o n s were m e a s u r e d o n a L i k e r t sca le . Q u e s t i o n s 1, 5, 2, a n d 3 were i n t e n d e d t o assess p a r t i c i p a n t s ' a t t i t u d e s a b o u t p e r f o r m i n g m u l t i p l e t a s k s a t o n c e o n a h a n d h e l d d e v i c e . Q u e s t i o n s 4 a n d 7 were i n t e n d e d t o g a t h e r s o m e i ns i gh t i n t o p a r t i c i p a n t s ' se l f -assessment o f t a c t i l e a p t i t u d e a n d p re fe rence , w h i c h p e o p l e have r a r e l y c o n s i d e r e d be fo re a n d the re fo re c a n Chapter 7. Browser User Evaluation 169 Measured Time vs Presentation Order 15000 £ i„„ mouse slider-• slider+ •Poly. (slider+) -Poly, (slider-) •Poly, (mouse) 5000 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 Task / Block Normalized Time vs Presentation Order 20000 15000 "**.."" "U p o E J: IOOOO ,.<"*x, *y\. ninimm" 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 Task / Block nmouse nslider-nslider+ • Poly, (nslider-) •Poly. (nslider+) • Poly, (nmouse) Figure 7.6: Task t ime and PRESENTATION ORDER, w i th polynomial curve fit lines added. Chapter 7. Browser User Evaluation 170 P a r t i c i p a n t P e d a l P r o m p t s P e d a l C o r r e c t % C o r r e c t 4 70 67 96% 5 67 59 88% 6 58 57 98% 7 69 58 84% 8 72 31 43% 9 54 33 61% 10 46 42 91% 11 53 37 70% 12 71 62 87% 13 47 37 79% 14 73 50 68% 15 97 95 98% 16 62 55 89% A L L 839 683 81% Table 7.5: Performance on the distraction task by individual study partici-pants. The table shows the total number of task prompts that were presented to the user, compared with the number of periods between prompts where the user made a correct response according to the prompt. Chapter 7. Browser User Evaluation 171 no t be m e a s u r e d u s i n g d i r e c t q u e s t i o n i n g . F i n a l l y , q u e s t i o n 6 was a d d e d b a s e d o n q u a l i t a t i v e f eedback o b t a i n e d d u r i n g ea r l i e r s t u d i e s i n w h i c h users e x p r e s s e d a s t r o n g p re fe rence for t h e p i e z o e l e c t r i c T D ve rsus a t y p i c a l m o b i l e p h o n e v i b r a t o r d u e t o l ower no ise a n d i n t r u s i v e n e s s ; for t h i s e x p e r i m e n t , i t w a s h y p o t h e s i z e d t h a t p r e - e x i s t i n g a t t i t u d e s a b o u t m o b i l e p h o n e v i b r a t i o n m i g h t b e c o r r e l a t e d t o p e r f o r m a n c e w i t h t h e T D . U n f o r t u n a t e l y , b e c a u s e no s i gn i f i can t effect was o b s e r v e d for t h e p res -ence o r a b s e n c e o f t a c t i l e f eedback , i t is i m p o s s i b l e t o c o r r e l a t e t h e m e a s u r e d a t t i t u d e s w i t h i n d i v i d u a l s u b j e c t s ' p e r f o r m a n c e o n t h e t ask . H o w e v e r , t h e p resence o f t h i s d a t a a n c h o r s t h e d e m o g r a p h i c p i c t u r e o f t h e s u b j e c t p o p -u l a t i o n a n d revea ls a s i gn i f i can t d i v e r s i t y i n a t t i t u d e s a b o u t m u l t i t a s k i n g w h i l e u s i n g a h a n d h e l d d e v i c e , w h i c h l e n d s e c o l o g i c a l v a l i d i t y t o t h e s t u d y . 7.9.2 Qualitative Evaluation of the Distraction Task T h e g o a l o f t h e d i s t r a c t i o n t a s k w a s t o c a u s e t h e p a r t i c i p a n t t o s p l i t h i s / h e r v i s u a l a t t e n t i o n resources i n a m a n n e r s i m i l a r t o t h a t o f a p e r s o n u s i n g a m o -b i l e d e v i c e i n a r e a l - w o r l d usage c o n t e x t . D a t a f r o m t h e e x p e r i m e n t i n d i c a t e s t h a t t h e t a s k w a s m o s t l y s u c c e s s f u l i n t h i s r e g a r d . T h e f a c i l i t a t o r o b s e r v e d t h a t a l l p a r t i c i p a n t s f r e q u e n t l y s h i f t e d t h e i r v i s u a l a t t e n t i o n b e t w e e n t h e h a n d h e l d b r o w s e r ' s L C D a n d the P C m o n i t o r (where t h e d i s t r a c t i o n t a s k p r o m p t s were d i s p l a y e d ) . V i d e o a n a l y s i s o f t he a u t h o r a n d s e v e r a l r esea rch co l l eagues p e r f o r m i n g t h e d i s t r a c t i o n t a s k t oge the r w i t h t h e m a i n e x p e r i -m e n t a l t a s k f u r t h e r c o n f i r m e d t h a t t h e i n t e r v a l b e t w e e n a t t e n t i o n a l sh i f t s was o n t h e o r d e r o f a few (2 - 10) seconds a n d o f t en c o i n c i d e d w i t h t i m e s w h e n t h e w e b page was b e i n g r e l o a d e d , w h i c h m a t c h e s w h a t w a s o b s e r v e d i n p r e v i o u s field s t u d i e s o f m o b i l e users [40]. Chapter 7. Browser User Evaluation 172 1.1 am a multitasker. H i i i i i Strongly Agree Agree Neutral Disagree Strongly Disagree 3. Sometimes it's necessary to use a mobile phone while driving. Strongly Agree Agree Neutral Disagree Strongly Disagree 5. I prefer to work on one thing at a time. 1 Strongly Agree Neutral Disagree Strongly Agree Disagree 7. When I am shopping, I often pick up objects to touch them even though I know I won't buy them. Jiiii 11 Strongly Agree Agree Neutral Disagree Strongly Disagree 2. Mobile phones are useful for more than just talking. IIIII • Strongly Agree Agree Neutral Disagree Strongly Disagree 4. I'm good at working with my hands. Strongly Agree Agree Neutral Disagree Strongly Disagree 6. The vibration function on a mobile phone is annoying. Strongly Agree Neutral Disagree Strongly Agree Disagree Figure 7.7: Results from the pre-task attitudes survey for the 13 main study participants. Chapter 7. Browser User Evaluation 173 8.1 was able to keep my eyes on the screen of the handheld device during the tasks. Strongly Agree Neutral Strongly Disagree 10. The task of pushing the pedals in response to the instructions was challenging. Strongly Agree Neutral Strongly Disagree F i gure 7.8: Results of the post-task attitudes assessment questions pertain-ing to the distraction task. Chapter 7. Browser User Evaluation 174 T h e q u a l i t a t i v e d a t a g a t h e r e d f r o m t h e p a r t i c i p a n t s a f te r t h e y h a d fin-i s h e d t h e t a s k s ( F i g u r e 7.8) m o s t l y s u p p o r t s t h e h y p o t h e s i s t h a t t h e d i s t r a c -t i o n t a s k was success fu l i n a c h i e v i n g i t s goa l s . T h e r e were m i x e d responses t o Q u e s t i o n 8 ("I w a s ab le t o keep m y eyes o n t h e sc reen o f t h e h a n d h e l d d e v i c e d u r i n g t h e t a s k s . . . " ) , p o s s i b l y d u e t o a l a c k o f awareness o f w h e r e v i s u a l a t t e n t i o n w a s d i r e c t e d ; i n d e e d , t h e p a r t i c i p a n t s w e r e n o t a s k e d t o p a y a t t e n t i o n t o w h e r e t h e y were l o o k i n g u n t i l a f te r a l l t h e t a s k s h a d b e e n c o m p l e t e d . H o w e v e r , m o s t s u b j e c t s ag reed w i t h t h e p r o m p t i n Q u e s t i o n 10 ( " T h e t a s k o f p u s h i n g the p e d a l s i n r esponse t o t he i n s t r u c t i o n s was c h a l -l e n g i n g . . . " ) , i n d i c a t i n g t h a t t h e d i s t r a c t i o n t a s k was at leas t p l a c i n g s o m e a t t e n t i o n a l d e m a n d s o n t h e s u b j e c t . I n p o s t - t a s k i n t e r v i e w s , seve ra l p a r t i c i p a n t s e x p r e s s e d f r u s t r a t i o n w i t h t h e l eve l o f s y s t e m feedback o n the d i s t r a c t i o n t ask . F e e d b a c k was o n l y g i v e n a t 7 - second i n t e r v a l s w h e n t h e p r o m p t w a s c h a n g e d , no t i m m e d i a t e l y f o l -l o w i n g t h e user i n p u t , so users were s o m e t i m e s c o n f u s e d as t o w h e t h e r t h e y h a d s u c c e e d e d i n e n t e r i n g the r e q u e s t e d i n p u t . I t was e s p e c i a l l y d i f f i cu l t for p a r t i c i p a n t s d u e t o t h e use o f t w o s i m u l t a n e o u s t a s k s i n a n a t t e n t i o n - s t a r v e d p a r a d i g m ( the f a c i l i t a t o r o b s e r v e d t h a t t h i s c a u s e d s u b j e c t s t o fo rget w h i c h i n p u t s t h e y h a d m a d e ) , a n d t h e p o o r m e c h a n i c a l f e e d b a c k o f t h e p e d a l h a r d -w a r e , w h i c h w a s o r i g i n a l l y d e s i g n e d for c o n t i n u o u s , a n a l o g u e i n p u t . H o w -ever , t h e d i s t r a c t i o n t a s k was i n t e n d e d t o r equ i r e users t o d i v i d e t h e i r o w n a t t e n t i o n w i t h i n s e v e n - s e c o n d b l o c k s r a t h e r t h a n t o i n t e r r u p t o r p r e e m p t t h e user a t m a c h i n e - s p e c i f i e d t i m e s ; i m m e d i a t e , a s y n c h r o n o u s f eedback w o u l d h a v e i n t e r f e red w i t h t h i s m o d e l s o m e w h a t . Chapter 7. Browser User Evaluation 175 9. The task of finding information on the page was challenging. : ,1 i • • _ . Strongly Agree Neutral Strongly Disagree 12.1 would pay 5% more for a mobile phone that had this kind of tactile display. Strongly Agree Neutral Strongly Disagree 14. Overall, I found the device easy to use. Strongly Agree Neutral Strongly Disagree 11. The tactile feedback was helpful in locating items on the page. Strongly Agree Strongly Disagree 13. The tactile feedback I experienced today was pleasant. Strongly Agree Strongly Disagree Figure 7.9: Results of the post-task attitudes assessment questions pertain-ing to the navigation task. Chapter 7. Browser User Evaluation 176 7.9.3 Qualitative Evaluation of the Navigation Task The questionnaire data gathered from the participants on the main (i.e., navigation) task are shown i n Figure 7.9. Addi t iona l ly , the following feed-back was obtained during the post-task interviews: • Mos t (12/13) participants felt that their performance was best i n the MOUSE condit ion. Hardware-related reasons included more familiar-i ty w i th the mouse scroll wheel model (5/12) and better ergonomic placement of the hands (6/12). • F ive out of 13 participants said they thought their performance was better w i th active tactile feedback. Reasons cited included a qualita-tive preference for being able to feel the page elements. • A lmos t a l l of the users cri t icized the velocity-control scrolling model compared to the mouse scroll wheel model. T h e most commonly cited reason was that continuous scrolling was less efficient than discrete (i.e., l ink-to- l ink) scrolling, although 3/13 users also expressed dis-comfort w i th the irregular jumping motion of the mouse scrolling in areas where l inks were sparse. • . Surprisingly, 4/13 users reported no awareness of the tactile feedback unt i l they encountered the post-task assessment questionnaire. One user said they d id not notice the tactile feedback unt i l about halfway through the study. Th i s was despite a l l users having been informed that they may experience tactile feedback during the study. The per-ceptual load due to the distractor task may have had an effect on this lack of awareness. Chapter 7. Browser User Evaluation 177 • Miscellaneous comments and suggestions included improving the hand posit ion for better ergonomics, reducing the inert ia of the slider for lower fatigue, and improving the contrast and clar i ty of the embedded L C D screen. 7.10 Discussion Return ing to the questions posed at the start of the study: 1. Does the presence of tactile feedback, in the form of a synthetic tactile flow signal, affect the time required to complete a web navigation task? It does not appear that tactile flow feedback aids targeting and naviga-t ion performance in the browser model used. B y carefully inspecting the data along various dimensions (subject, PRESENTATION ORDER, etc.), and increasing the power of the experiment through the task difficulty normalizat ion, we can more confidently state that there does not appear to be an effect of tactile feedback in the measurements taken. W h i l e this is clearly disappointing in light of the hypotheses formulated at the start of the experiment, the findings enable us to focus on areas for improvement of the device rather than on issues wi th the experimental design. 2. How does the experience of browsing and navigation on the hand-held prototype compare to a more common paradigm (the mouse scroll wheel)? T h e familiari ty and ergonomic comfort, as well as the discrete scrolling model, of the mouse scroll wheel were preferred. Chapter 7. Browser User Evaluation 178 3. What are users' subjective reactions to the use of tactile feedback in this application? Part ic ipants were mostly either ambivalent about the tactile feedback or reacted positively. 4. What are users' subjective reactions to the handheld prototype hard-ware and browser software? The prototype was mi ld ly cri t icized for its ergonomic profile and weight. 5. Are there any correlations between user demographics, such as age, gender, and previous mobile experience, and their performance in the browsing task with or without tactile feedback? Due to a lack of overall stat ist ical significance between the two SLIDER conditions, it was not possible to obtain correlative data on these de-mographic characteristics. W h y was performance seemingly unaffected by the tactile flow rendering setting? Pr io r to the start of the experiment, we hypothesized that the following factors, i f satisfied, may influence the measured performance in the tactile feedback condit ion: 1. T h e tactile rendering could be perceived as a sense of tactile flow by the user. 2. T h e tactile flow signal would simulate moving spatial cues that occur in the real world when the user is touching objects. 3. T h e simultaneous, mul t imodal input of both visual and haptic cues would allow the user to more quickly gain a sense of spatial location in a v i r tua l space (i.e., web page). Chapter 7. Browser User Evaluation 179 4. G iven a more rapid, more confident acquisit ion of spatial location, targeting performance would be increased. 5. A n y performance changes would be measured by the experimental paradigm. T h e first factor should be true, based on the fact that the parameters used were w i th in the l imits established by the Perceptual Character izat ion studies, and the haptic icon design effort to maximize the ampli tude of the percept created by the T D . However, the fact that several users d id not notice the presence or absence of tactile feedback under conditions of high perceptual load suggests that the ampli tude may be on the low (i.e., weak) side. Since the signals were already mostly opt imized, the remaining l imitat ions are pr imar i ly in the hardware and the physical constraints of the piezoelectric elements used in the display. Weak s t imulat ion could be responsible for the lack of a statist ically significant performance difference, but it seems unlikely that no performance difference would be observed due to low ampli tude alone, since users are universally able to detect the tactile feedback when their attention is directed towards i t . Regarding the second and th i rd factors, the synthetic tactile flow ren-dering differs from typica l real-world touch in a significant way: i n natural touch, a tactile flow percept is generated when the fingertip is moved (i.e., slid) across a textured surface, usually accompanied by the proprioceptive sensation of the directed movement of the body part. In the browser model, the tactile signal is decoupled from its typica l proprioceptive accompaniment and controlled wi th a novel force- and position-sensitive v i r tua l scrolling model . Moreover, in the world of natural touch, externally modulated sl id-ing (tactile flow) signals are usually l imi ted to accidentally losing one's grip Chapter 7. Browser User Evaluation 180 on an object, or an insect crawling across the fingertip, where the semantic interpretation of the tactile signal is not one of spatial posit ion, but of an alert. A synthetic, decoupled tactile flow signal, therefore, has a significant challenge to overcome in being a novel way of providing a spatial cue. T h e fourth factor is l ikely to be true on the basis of general psycholog-ical principles of semantic activation and directed movement. In addit ion, a thorough analysis of the experiment, combined wi th the fact that perfor-mance differences were observed in the M O U S E condit ion, lends confidence to the fifth condit ion. Therefore, it seems that the most l ikely explanation for the lack of performance improvement was that the concept of decou-pled tactile flow signal as a spatial cue was not validated by the current experiment. How can the browser be improved? In addi t ion to the performance data, a significant amount of qualitative data about the browser in the hands of a human user was collected. The study suggests that user comfort can be increased by improving the ergonomic characteristics of the device, including thumb posit ion, size, and weight. Instead of continuous velocity-based scrolling, the navigation model could be discretized, either par t ia l ly (as in the case of the mouse scroll wheel) or whol ly (as in the case of scroll keys), al lowing users to more easily target l inks. A discretized navigation model could use different haptic signals, including icons that use tactile flow for identification rather than spatial cueing, to widen the vocabulary of tactile signals and perhaps become useful not as spatial cues, but as eyes-free semantic cues as to the content of the web page. These opportunities for further development are considered in further detail in the conclusion in the following chapter. Chapter 7. Browser User Evaluation 181 In summary, the key contributions of the Browser User Exper iment are that, having thoroughly tested the applicat ion of tactile flow in a browser model , we can rule out further opt imizat ion of haptic icons, scroll model pa-rameters, browsing content, and experimental design, and instead direct our efforts towards applicat ion designs that do not rely on decoupled tactile flow rendering as a spatial cue. Qual i ta t ive feedback obtained during the study gives clues about how to significantly improve the hardware and navigation models. We have also established an experimental paradigm, including a dis-t ract ion task and novel, web-based experiment management software, which can be used to evaluate further progress in handheld, mul t imoda l interfaces. Chapter 8 182 Conclus ion and Future W o r k In this chapter we summarize the key contributions of this work, revisit the research questions posed earlier, and outline promising future directions enabled by this work. 8.1 Summary of Key Contributions This thesis makes the following contributions to the body of knowledge about interaction design for mobile haptics. 8.1.1 Identification of a novel multimodal approach to addressing limitations in mobile user interfaces Conventional approaches to mobile interaction design using existing hard-ware suffer from l imi ted interface bandwidth and poor sui tabi l i ty for mobile conditions of visual and/or auditory impairment. Contr ibut ions of this work include the ar t iculat ion of this need, a survey of existing haptic technologies and the selection of the piezoelectric actuated lateral skin-stretch technol-ogy, mounted in a simplified 1-D slider configuration, as a tractable method for addressing the requirements of mobile haptics. Compared to exist ing vibrotact i le approaches, our concept of mobile haptics enables richer tac-tile experiences while managing cost, power, size, and weight tradeoffs. The Chapter 8. Conclusion and Future Work 183 val idi ty of this selection has been demonstrated by the realization of a hand-held prototype support ing interaction in both haptic and visual modalities. 8.1.2 Development of a new handheld haptics hardware platform Having selected the problem domain and the tactile display technology, we 1 proceeded wi th implementat ion of a hardware and software platform for prototyping and testing handheld haptic applications. T h i s inexpensive, reproducible prototype, including the various technical developments incor-porated wi th in , represents another contr ibution of this research project. Even after the major engineering effort on the prototype (carried out at M c G i l l Universi ty) was completed, the hardware and software specifica-t ion evolved through insights obtained by using the platform for applica-t ion development and conducting user tests. For example, the need for a spring-based return-to-centre feature for velocity control was identified dur-ing applicat ion development. The specification of the prototype platform at the t ime of wr i t ing represents the result of this iterative development. The existence of the prototype enables further research in handheld haptic applications without the need for development of entirely new hardware. 1 The hardware prototype development was a collaborative effort with members of the McGill University Haptics Lab. The present author was a contributor at all stages of the prototype development, but the electronic and mechanical design was lead by Jerome Pasquero, and the control software design was led by Vincent Levesque. For more infor-mation, please see Chapter 4. Chapter 8. Conclusion and Future Work 184 8 . 1 . 3 E v o l u t i o n o f a p p l i c a t i o n d e s i g n c o n c e p t s b a s e d o n u s e r s t u d i e s a n d h a r d w a r e d e v e l o p m e n t T h i s thesis describes a suite of novel application designs that are matched to the capabilities and l imitat ions of the handheld prototype. In addi t ion to the detailed in i t i a l specification based upon a design philosophy and back-ground research, this work presents evolutionary development of the applica-t ion concepts based on insights learned during the prototyping process. For example, the size of the "vocabulary" of haptic icons available for use wi th the browser applicat ion — and thus the range of haptic icon functionality — was discovered through perceptual characterization experiments (Chap-ter 5). 8 . 1 . 4 M e t h o d f o r r a p i d p r o t o t y p i n g a n d g r a p h i c a l r e p r e s e n t a t i o n o f t a c t i l e s t i m u l i A s described in Section 4.9, novel approaches for visual iz ing voltage outputs, spatial volt images, and skin stretch images were developed to address the previously cumbersome process of creating, managing, and analyzing tactile s t imul i . In addit ion, an accessible prototyping procedure using standard web-based graphical tools, and support util i t ies for automated parsing of the visual representations, are introduced. These methods can be generalized to any tactile display that produces spatial ly dis tr ibuted patterns of skin s t imulat ion. Chapter 8. Conclusion and Future Work 185 8.1.5 P e r c e p t u a l charac ter i za t ion of a novel m i n i a t u r e piezoelectr ic tact i le d i sp lay Rather than immediately developing applications based on the developers' own vague and subjective impressions of the capabilities of the new system, we first conducted a detailed user evaluation to determine its "vocabulary" of signals that human users are capable of perceiving. Both the results of these user studies, and the validation of this method as a useful component of user-centred design for novel output hardware, represent contributions of this work. We determined that directional tactile flow can be perceived reliably at speeds up to 0.34 metres per second, and that haptic icons may be designed along dimensions of — in order of decreasing saliency level — repeating ver-sus nonrepeating waveform, direction, speed, and amplitude. These findings may be put to use by anyone who is interested in developing applications for a tactile display with similar physical parameters to the one we used. 8.1.6 H a n d h e l d browser a p p l i c a t i o n w i t h tact i le enhancement As an example of one high-fidelity application prototype, we constructed a multimodal browser capable of reading web pages and displaying them si-multaneously with visual and haptic representations. The method for trans-lating web markup to a haptic page map, and the navigation model for a 1-D slider are examples of novel contributions of this effort. Usability testing of the handheld browser did not demonstrate a per-formance (task time) advantage for navigation with directional tactile cues (tactile flow), although the qualitative feedback obtained from the users sug-Chapter 8. Conclusion and Future Work 186 gests that there may be a subjective preference for the tactile confirmation of target acquisit ion. The results may be used to guide further development of applications (Section 8.3, below), and to highlight areas where hardware improvements are l ikely to have the most significant effect on usabil i ty (Sec-t ion 8.4). 8.1.7 M e t h o d for usabil i ty testing of mobile applications A contr ibut ion of this thesis is the method used for testing a handheld device i n the laboratory while s imulat ing some of the cognitive and perceptual load aspects of mobile interaction in the real world . T h e system incorporates a pedal-based distraction task that requires the user to engage the feet and visual system without interfering wi th the hands, similar to the scenario of a person walking while using a device. A n experiment manager software application, buil t using modern web technologies, provides cumulative reinforcement feedback to the user and collects data in a reliable, distr ibuted fashion. Th i s software is available for re-use by any researcher who wishes to conduct a user experiment using simulated mobile conditions. 8.1.8 Case study of a user interaction design process for haptics Final ly , the activities described in this thesis can be considered, from a case study perspective, as one full i teration of a user-centred design process (Figure 1.2) that begins wi th an analysis of user needs and proceeds in a stepwise cascade of discovery and increasing understanding of the interaction between the system and the user. The results from each stage of this process Chapter 8. Conclusion and Future Work 187 provide input for further iterative improvement. T h i s work presents a specialized case of the classic user-centred design archetype, one that is specifically tailored to the needs of contemporary research in haptics. B y demonstrating that it is possible to follow the user-centred approach even wi th nascent technology and heavy dependency on hardware configuration, this work lends val idi ty to the concept of incorpo-rat ing users early in a design process in order to focus on achieving good usabil i ty and to minimize system- or technology-centric design. 8.2 Research Questions The research questions posed at the beginning of this thesis were addressed i n the following ways. 1. What are the problems with existing mobile user interfaces that may be addressed using haptics? In Chapter 1 we identified two fundamental problems of existing mo-bile interfaces: sensory bandwidth limitation and environmental (con-textual) competition for visual and auditory attentional resources. Hap-tics is well suited to address these challenges through parallel feedback in a sensory modal i ty that is s t i l l mostly unused in mobile interaction. In Chapter 3 we described the in i t i a l stages of an iterative applicat ion design process that has identified 2. How can one implement haptics on a mobile device despite power, size, and weight restrictions? In Chapter 2 we discussed existing approaches for haptic transduc-ers and identified the piezoelectric lateral skin-stretch technology as a Chapter 8. Conclusion and Future Work 188 promising candidate for further investigation. The resulting piezo tac-tile display represents a compromise: it is smaller, lighter, consumes less power, and avoids the mechanical grounding challenges of motors used for force feedback, but it is also more complicated than a typical mobile phone vibrator. However, the technology we used also offers a significantly richer haptic experience than mere vibration. In addition to the selection of the tactile display itself, our investigation into various alternatives for mobile haptics hardware resulted in the choice of a linear slide-mounted tactile display positioned on the side of the device as a working compromise between complexity (and thus, size and weight) and functionality. After application prototyping and user testing, we now know that features such as a spring return-to-centre and increased slider travel (perhaps enabled by a different mounting configuration) should be considered for the next iteration of design, in order to improve the usability of the hardware when used for cursor navigation. Since our choice of technology was partially influenced by practical considerations such as access to manufacturing facilities, it is conceiv-able that if the process was replicated by someone else, a different hardware focus might have been chosen. For example, someone with access to M E M S (Micro-Electro-Mechanical Systems) facilities, or spe-cialized intellectual property, might have been able to create a hard-ware prototype just as rapidly as we were, without using off-the-shelf parts. However, the interaction designs we proposed prior to the selec-tion of the hardware technology are generalizable in the sense that they do not assume large grounded forces (which would be impractical on Chapter 8. Conclusion and Future Work 189 a mobile device regardless of haptic display technology), and provide a "bridge" between existing mobile user needs and a rough hardware specification that may be implemented using a variety of technologies. 3. What are the expressive capabilities of the hardware prototype in the hands of a human user? In Chapter 5 we described the results of three perceptual characteriza-t ion experiments that revealed the usable stimulus speed range, dist in-guishable haptic icon design parameters, and hierarchy of salience of various parameters such as speed, direction, waveform and amplitude. 4. What are the engineering challenges associated with building a high-fidelity hardware and software prototype with handheld use in mind? Through the process of designing and bui ld ing the prototype platform and its applicat ion software, we discovered a variety of new information related to the realization of a handheld haptic device. For example, on the hardware side we were the first to validate the use of a stiffer double-pinning configuration for the piezo benders as a key factor en-abling minia tur iza t ion (Section 4.3.1). 2 Through experiments w i th the control software a lgori thms, 3 we discovered that a 12-Hz refresh rate was insufficient, while an 83-Hz refresh rate was sufficient for produc-ing smooth tactile sensations on this T D . Final ly , we encountered the problem of visual iz ing and prototyping haptic icons, and addressed it using the methods described in Section 4.9. 2The double-pinning configuration was invented by project collaborators Jerome Pasquero and Qi Wang. 3 The control software development and optimization was carried out in collaboration with Vincent Levesque. Chapter 8. Conclusion and Future Work 190 B y bui ld ing a representative software applicat ion (the browser) we learned that, under the present system configuration, an asynchronous design wi th pr ior i ty given to the tactile process over the visual process was necessary for good haptic performance. The need for a sophis-ticated cursor navigation model was also recognized, w i t h iterative development eventually resulting in a hybr id posi t ion/veloci ty control model that promotes active tactile exploration wi th in the central re-gion of the slider travel while al lowing traversal of larger areas than would be possible w i t h posit ion control alone. 5. Is tactile flow an effective aid for user navigation? T h e user studies conducted wi th the browser prototype d id not demon-strate a significant performance (task time) advantage for tactile flow as an aid to navigation or target acquisition in the browsing context. However, qualitative feedback from the same studies was often posi-tive, w i th many users indicat ing a subjective preference for the extra tactile feedback. Further user studies could perhaps better elucidate the ways this mul t imoda l interaction could be used to better support user goals other than more rapid navigation. 6. How can user-centred design methodology be applied to haptics, where hardware technology is still the primary determinant of user interface capabilities? It is a common misconception that emerging fields like haptics require designers to do a lot of up-front effort on the display technology before involving users, due to the complexity and lack of standardization of the interface hardware. User testing, when done, is typica l ly ut i l ized Chapter 8. Conclusion and Future Work 191 for fine-tuning the software applications, or s imply val idat ing the use-fulness of a part icular approach. A t the same time, it is well known that incorporat ing user studies early in the design process is essential for achieving m a x i m u m usabil i ty benefit per development effort [22]. Some of the methods we used to incorporate user testing early in the design process, such as perceptual characterization, have also been used in previous studies (e.g., [12]). However, in the present study, we have conducted a full cycle of development, i n which usabil i ty consider-ations were the pr imary driver at each stage — from concept work that began w i t h an inquiry about user needs and delayed the detailed spec-ification of hardware unt i l after usage scenarios had been determined, to the further development of the applicat ion concepts informed by perceptual characterization studies, and finally, to the two-stage de-velopment of the browser applicat ion consisting of a low-fidelity pro-totype followed by user testing, and a high-fidelity prototype which was then formally evaluated. Therefore, we have demonstrated that, even when the user interface is highly dependent on technological l imitat ions, it is possible to follow a user-centred design process and to benefit from the streamlined path towards good usabil i ty that this approach allows. 8.3 Future Work: Application Designs for Further Investigation Informed by the results of the hardware prototyping, perceptual characteri-zation, and high-fidelity software prototype and user test w i th one selected Chapter 8. Conclusion and Future Work 1 9 2 applicat ion, we may now return to the applicat ion design stage to consider the feasibility of further applications based on the same handheld haptics concept. 8.3.1 A p p l i c a t i o n s Invo lv ing S h a p e R e n d e r i n g Appl ica t ions involving shape rendering are those that take advantage of the piezoelectric tactile display's abi l i ty to mimic patterns of skin stretch analogous to those created by touching small objects and surface features. Handhe ld Brai l le Reader T h e abi l i ty of a piezoelectric lateral skin-stretch tactile display to render leg-ible Bra i l l e dots v i a active tactile exploration has been previously demon-strated in [30]. A mobile version has various applications for b l ind users including discreet text messaging or reading printed characters w i t h opt ical character recognition ( O C R ) . The only hardware change that would be nec-essary for pract ical implementat ion would be the use of two rows of piezo arrays for each of the two vertical Bra i l le dot positions. T h e existing hybr id posit ion / velocity scroll model could be used to overcome slider travel l i m -itations, but further testing w i l l have to be done to determine the op t imal parameters for legibil i ty of Bra i l le . M e d i c a l Appl icat ion: Displaying Small Shapes Al though not s t r ic t ly a mobile application, the abi l i ty to digi ta l ly transmit and reproduce smal l shapes could enable a remotely located doctor to factu-al ly palpate an area on a patient which is sensed by a high resolution tactile sensor such as the one described in [37]. A n example might be detailed ex-Chapter 8. Conclusion and Future Work 193 aminat ion of the results of a tine test, which produces distinctive patterns of swelling depending on ant ibody status, which are often much more difficult to dist inguish visual ly than factually. 8.3.2 General Haptic Icon Applications In the following group of applications, the T D is used as a fixed output transducer, w i t h min ima l direct coupling between user input and tactile output. T h i s type of interaction was tested and verified in the M D S and speed study experiments, and does not assume untested concepts such as spatial acquisit ion by tactile flow rendering. Therefore, it is expected to have a high probabi l i ty of successful deployment. However, factors such as environmental distract ion and matching between the information content of the output and the T D ' s capabilities must s t i l l be considered. Instant Messaging / Notification A s described in Section 3.3.4, tactile notification of presence status changes could enrich the experience of using instant messaging or other social appli-cations. It is expected that the effects of haptic enhancement would be most pronounced under conditions of workload [12] or environmental distraction. 8.3.3 Spatial Signalling T h i s group of applications makes use of the T D ' s capacity to display a d i -rectional signal. A s the M D S analysis demonstrated, users can distinguish at least two directions, three speeds, and two amplitudes in a op t imal con-trolled laboratory setting. The semantic usefulness of those parameters in a real-world context of mult iple distractions and variable tactile engagement Chapter 8. Conclusion and Future Work 194 s t i l l needs to be determined. Navigat ion A s described in Section 3.3.3, a directional signal may be ut i l ized for nav-igation feedback. Based on the results of the browser user study, it is not clear that the directional signalling w i l l produce an intui t ive directional re-sponse in the user. Rather , it is perhaps more l ikely that the user would come to associate a specific haptic icon (in this case a directional signal) w i th a semantic meaning, and produce responses appropriately. However, this hypothesis requires further testing. 8 . 3 . 4 Browser Improvements W h i l e the concept of tactile flow rendering to improve spatial navigation was not validated in the present study, there is room for numerous improvements in the browser prototype. These include: • General software opt imizat ion to improve I / O performance, perhaps moving to a real-time operating system and/or embedded controller for the tactile loop. • Dynamic haptic icon rendering based on the pixel height of page ele-ments. • A n algori thm to properly handle horizontal ly la id out elements that overlap in the vert ical domain. T h i s would require a more complex 2D (visual page map) to I D (haptic page map) transformation engine than is currently implemented. Chapter 8. Conclusion and Future Work 1 9 5 • M a x i m u m usable scrolling speed can perhaps be further increased by adding a "high speed mode" 4 : when the user is scrolling at high speed, the normal haptic icons can be dynamical ly substi tuted wi th alter-nate representations opt imized for m a x i m u m amplitude, and perhaps "stretched" so that they remain under the finger longer, or by display-ing only the most semantically important elements. Th i s favours speed over "vocabulary". The range of distinguishable haptic icons would be reduced while the system is in high speed mode, since the amplitude parameter and some waveforms would no longer be available. Such a modification, therefore, must be considered carefully w i t h respect to the desired haptic tagging scheme. 8.4 Future Work: Hardware Improvements O f course, numerous opportunities exist for further improvement in the pro-totype hardware specification. M a n y of the challenges associated wi th bui ld-ing working applications using the prototype are related to the slider, which despite being auxi l iary to the tactile output device has a major effect on the user experience. Increasing the slider travel (perhaps by reconsidering the mount ing location), and finding appropriately stiff springs for the hardware return-to-centre feature could improve the scrolling model . T h e first 1-D piezo tactile display, the V i r t u a l Bra i l l e Display, had a slider travel that was an order of magnitude longer than the present miniature tac-tile display. Th i s enabled a simple position-control mode, but impor tant ly also allowed the user to use proprioception to sense the slider posit ion wi th -out having to mentally integrate velocity and time. Neither of these usabil i ty 4This idea was contributed by thesis reader Dr. Steve Wolfman. Chapter 8. Conclusion and Future Work 196 advantages are present in the current handheld version. F ina l ly , presently the push to select, or "cl ick", feature of the device is implemented as a switch. B y replacing the switch wi th an analog force sensor such as a force sensitive resistor ( F S R ) , and perhaps reconsidering the mount ing appropriately to deliver the force l inearly to the sensor, addi t ional interaction models may be considered that modulate the stimulus ampli tude based on how strongly the user is pressing down on the T D , enabling richer tactile exploration. 8.5 Conclusion In this thesis we have described one full i teration of an interaction design and hands-on inquiry process (Figure 1.2). We began by identifying a problem wi th broad relevance: the usabil i ty l imitat ions of contemporary mobile de-vices. Under the working hypothesis that haptics might be a good candidate for addressing this problem, we conducted conceptual design work informed by an inquiry into existing usage patterns and user needs. User require-ments and system requirements emerged, concurrently, eventually al lowing us to proceed wi th the development of a high-fidelity, "deep" hardware pro-totype using a number of novel technologies, in partnership w i t h an expert collaborator. Once some of the device's expressive capabilities were known, we proceeded wi th the development of a high-fidelity browser applicat ion prototype. F ina l ly , it was then possible to use more conventional usabil i ty techniques to assess the performance characteristics of the browser. Through the exercise of designing, bui lding, and testing the mobile hap-tics concept, we learned that user-centred interaction design for haptic inter-faces requires the use of slightly different methodologies than conventional Chapter 8. Conclusion and Future Work 1 9 7 user-centred interaction design for pr imar i ly visual interfaces. F i r s t , be-cause r ichly expressive haptics — especially in the mobile domain — is s t i l l i n its infancy, the hardware used for rendering computer signals into hu-man sensory percepts is far more varied and weakly standardized than for the visual domain. Th i s implies that it is not possible to do "pure" user-centred interaction design that is entirely system (and hardware) agnostic. Instead, applicat ion and hardware design must go hand-in-hand; however, in our approach we have striven to follow the spirit of good user-centred design techniques by beginning the inquiry into user needs and applicat ion scenarios as early as possible, using the m i n i m u m set of assumptions about the specific hardware implementation. A s design work proceeded, both the target hardware profile and the application designs evolved synergistically, informed by increasing knowledge about user needs and candidate hardware designs. Another special characteristic of designing for mobile haptics is the u t i l -i ty of formal perceptual characterization based on user studies. W h i l e the sensory gamut of a visual display might be readily characterizable using a set of standardized measures and a simple visual inspection, haptic signals are often far more subtle, complex, and difficult to characterize. Therefore, when a new haptic output device is developed, it is helpful to understand its sensory gamut — or measure of expressiveness — using the techniques described in this thesis, to inform the design of applications that best take advantage of the tactile display. T h e identification and val idat ion of practical interaction design tech-niques for mobile haptics, concept designs for applications based on a min-imal ly assumptive hardware profile, high-fidelity hardware prototype, high-fidelity browser applicat ion prototype, and user study findings that shed Chapter 8. Conclusion and Future Work 198 light on the characteristics of a miniature lateral skin-stretch haptic system represent major contributions of this work. For future work, i t is reason-able to expect that another i teration of the same process, informed by the findings from each step of the previous i teration, w i l l produce further i m -provements in hardware and applicat ion usability. Such predictabi l i ty en-ables researchers and engineers to engage in mobile haptics development in a planned fashion, and it is hoped that this method w i l l increase the accessibil-ity, usefulness, and usability, put t ing this exceedingly promising technology i n the hands of users. 199 Bibliography [1] T h e adaptive communicat ion environment (ace). h t tp : / /www.cs .wus t l . edu / s c h m i d t / A C E . h t m l . [2] Gif89a specification. h t t p : / / w w w . w 3 . o r g / G r a p h i c s / G I F / s p e c -gif89a.txt. [3] T h e gimp toolki t (gtk) library, h t t p : / /www.g tk .o rg / . [4] Imagemagick l ibrary home page, h t tp : / /www. imagemagick .org / . [5] Microsoft intellimouse optical specification, http: / /www.microsof t .com/hardware . [6] C o r i n R . Anderson, Pedro Domingos, and Danie l S. Weld . Personalizing web sites for mobile users. Proceedings of the 10th Conference on the World Wide Web (WWW10), 2001. [7] T . G . Anderson. Human-computer interface including hapt ical ly con-trolled interactions. Un i t ed States Patent 6,954,899, October 11 2005. [8] R . B o w n . Mul t i - funct ional vibro-acoustic device. U n i t e d States Patent 6,911,901, June 28 2005. [9] A . C . B raun , L . B . Rosenberg, D . F . Moore , K . M . M a r t i n , and A . S . G o l d -enberg. Direct ional tactile feedback for haptic feedback interface de-vices. Un i t ed States Patent 6,864,877, M a r c h 8 2005. Bibliography 200 [10] Stephen Brewster and L o r n a M . Brown . Tactons: structured tactile messages for non-visual information display. In CRPIT '04-' Proceed-ings of the fifth conference on Australasian user interface, pages 15-23, Darl inghurst , Aus t ra l i a , Aus t ra l ia , 2004. Aus t ra l i an Computer Society, Inc. [11] L . M . Brown , S. A . Brewster, and H . C . Purchase. A first investigation into the effectiveness of tactons. pages 167-176, 2005. [12] A . C h a n , K . M a c L e a n , and J . McGrenere . Learning and identifying haptic icons under workload, pages 432-439, 2005. [13] Ange la Chang, Sile O ' M o d h r a i n , R o b Jacob, E r i c Gunther , and Hiroshi Ishii . Comtouch: design of a vibrotacti le communicat ion device. In DIS '02: Proceedings of the conference on Designing interactive systems, pages 312-320, New York , N Y , U S A , 2002. A C M Press. [14] Ange la Chang and Conor O 'Su l l ivan . Audio-hapt ic feedback in mobile phones. In CHI '05: CHI '05 extended abstracts on Human factors in computing systems, pages 1264-1267, New Yor k , N Y , U S A , 2005. A C M Press. [15] R . Cholewiak and C . Sherrick. A computer-controlled mat r ix system for presentation to skin of complex spatiotemporal pattern. Behavior Research Methods and Instrumentation, 13(5):667-673, 1981. [16] Immersion Corporat ion. Vibe tonz system. Produc t Literature, M a r c h 2005. [17] Jack T i g h Dennerlein, D a v i d B . M a r t i n , and Chris topher Hasser. Force-feedback improves performance for steering and combined steering-Bibliography 201 targeting tasks. In CHI '00: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 423-429, New York , N Y , U S A , 2000. A C M Press. [18] M . Enr iquez and K . E . M a c L e a n . C o m m o n onset masking of vibrotact i le s t imul i - poster. In Proc. World Haptics. I E E E , 2005. [19] F . Geldard . Some neglected possibilities of communicat ion. Science, 131(3413):1583-1588, 1960. [20] F . Gemperle, N . O ta , and D . Siewiorek. Design of a wearable tactile display, pages 5-12, 2001. [21] Masa taka Goto . Smartmusickiosk: Mus i c listening station wi th chorus-search function. Proceedings of the 16th Annual ACM Symposium on User Interface Software and Technology (UIST 2003), pages 31-40, 2003. [22] J . D . G o u l d . How to design usable systems. In Readings in Human-Computer Interaction: Toward the Year 2000, pages 93-121. 1995. [23] V . Hayward and M . Cruz-Hernandez. Tact i le display device using dis-t r ibuted lateral skin stretch. In Proc. of the Haptic Interfaces for Vir-tual Environment and Teleoperator Systems Symposium (IMECE2000), volume DSC-69-2 , pages 1309-1314. A S M E , 2000. [24] H . Ishii and B . Ul lmer . Tangible bits: Towards seamless interfaces between people, bits and atoms. In A C M , editor, Proc. of CHI'97, pages 234-241, 1997. Bibliography 202 [25] R . S . Johansson and R . H . L a M o t t e . Tacti le detection thresholds for a single asperity on an otherwise smooth surface. Somatosensory Re-search, 1:21-31, 1983. [26] A l a n Kay . Personal dynamic media. IEEE Computer, 10(3):31-41, 1977. [27] R . L . K l a t z k y and S. J . Lederman. How well can we encode spatial layout from sparse kinesthetic contact? In Proceedings of the 11th Symposium on Haptic interfaces For Virtual Environment and Teleop-erator Systems (Haptics'03), page 179, Washington, D C , M a r c h 2003. I E E E Computer Society. [28] Vincent Levesque and Vincent H a y ward. Exper imenta l evidence of lat-eral skin s train during tactile exploration. Proc. Eurohaptics 2003, pages 261-275, 2003. [29] Vincent Levesque and Jerome Pasquero. Stress library.' h t t p : / / w w w .c im .mcg i l l . c a / hap t ic / . [30] Vincent Levesque, Jerome Pasquero, Vincent Hayward , and Maryse Legault . Display of v i r tua l braille dots by lateral skin deformation: feasibility study. ACM Trans. Appl. Percept., 2(2):132-149, 2005. [31] Rober t W . Lindeman, John L . Sibert, E r i c k Mendez-Mendez, Sachin P a t i l , and Danie l Phifer. Effectiveness of directional vibrotact i le cuing on a building-clearing task. In CHI '05: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 271-280, New York , N Y , U S A , 2005. A C M Press. Bibliography 203 [32] J u k k a L in j ama and Top i Kaaresoja. Novel , minimal is t haptic gesture interaction for mobile devices. In NordiCHI '04: Proceedings of the third Nordic conference on Human-computer interaction, pages 457-458, New York , N Y , U S A , 2004. A C M Press. [33] Shannon L i t t l e . Theory, software, and psychophysical studies for the tactile handheld miniature b imodal device. Technical Repor t TR-2005-21, Univers i ty of B r i t i s h Columbia , Department of Computer Science, 2005. [34] Joseph L u k , Jerome Pasquero, Shannon Li t t l e , K a r o n E . M a c L e a n , Vincent Levesque, and Vincent Hayward . A role for haptics in mobile interaction: In i t ia l design using a handheld tactile display prototype. In CHI '06: Proceedings of the SIGCHI conference on Human factors in computing systems, New York , N Y , U S A , 2006. A C M Press. [35] K . E . M a c L e a n , M . J . Shaver, and D . K . P a i . Handheld haptics: a usb media controller w i th force sensing, pages 311-318, 2002. [36] K . E . M a c L e a n and M . Enriquez. Perceptual design of haptic icons. In Proc. EuroHaptics 2003. I E E E , 2003. [37] V i v e k Maheshwari and R a v i F . Saraf. High-resolution thin-f i lm device to sense texture by touch. Science, 312(5779): 1501-1504, June 9 2006. [38] Michae l McCloskey . Intuit ive physics. Scientific American, 248:122-130, A p r i l 1983. [39] B . A . Nard i , S. Whi t taker , and E . Bradner . Interaction and outeraction: instant messaging in action. Proceedings of the 2000 ACM Conference .on Computer Supported Cooperative Work, pages 79-88, 2000. Bibliography 204 [40] A n t t i Oulasvi r ta , Sakari Tamminen, V i r p i Roto , and Jaana Kuore lah t i . Interaction in 4-second bursts: the fragmented nature of attentional resources in mobile hci . In CHI '05: Proceedings of the SIGCHI con-ference on Human factors in computing systems, pages 919-928, New York , N Y , U S A , 2005. A C M Press. [41] J . Pasquero and V . Hayward . S T R E S S : A pract ical tacti le display system w i t h one mil l imeter spatial resolution and 700 H z refresh rate. In Proc. Eurohaptics 2003, pages 94-110, 2003. [42] J . Pasquero, J . L u k , S. L i t t l e , and K . E M a c L e a n . Perceptual analysis of haptic icons: an investigation into the val idi ty of cluster sorted mds. In Proc. Of Symp. On Haptic Interfaces for Virtual Environment and Teleoperator Systems (IEEE-VR). I E E E , 2006. [43] Jerome Pasquero, Joseph L u k , Vincent Levesque, Q i Wang , Vincent Hayward , and K a r o n M a c L e a n . Dis t r ibu ted tactile display for r ich haptic interaction wi th a handheld infomation display. To appear in IEEE Multimedia, 2006. [44] C A . Perez, A . J . Santibafiez, C A . Holzmann , P . A . Estevez, and C M . He ld . Power requirements for vibrotact i le piezo-electric and electrome-chanical transducers. Medical and Biological Engineering and Comput-ing, 41:718-726, November 2003. [45] Ivan Poupyrev and Shigeaki Maruyama . Tact i le interfaces for small touch screens. In UIST '03: Proceedings of the 16th annual ACM sym-posium on User interface software and technology, pages 217-220, New York , N Y , U S A , 2003. A C M Press. Bibliography 205 [46] Ivan Poupyrev, Shigeaki Maruyama , and J u n Rekimoto . Ambien t touch: designing tactile interfaces for handheld devices. In UIST '02: Proceedings of the 15th annual ACM symposium on User interface soft-ware and technology, pages 51-60, New York , N Y , U S A , 2002. A C M Press. [47] V i r p i Ro to and A n t t i Oulasvir ta . Need for non-visual feedback wi th long response times i n mobile hci . In WWW '05: Special interest tracks and posters of the 14-th international conference on World Wide Web, pages 775-781, New York , N Y , U S A , 2005. A C M Press. [48] K . B . Shimoga. Finger force and touch feedback issues in dexterous telemanipulation. pages 159-178, 1992. [49] D a v i d L . Strayer, Frank A . Drews, and Dennis J . Crouch . A comparison of the cell phone driver and the drunk driver. Human Factors: The Journal of the Human Factors and Ergonomics Society, 48(2):381-391, Summer 2006. [50] Hong Z . Tan , R o b Gray, J . Jay Young, and R y a n Traylor . A haptic back display for attentional and directional cueing. Haptics-e: The Electronic Journal of Haptics Research, 3(1):20, June 11 2003. [51] C . R . Wagner, S.J . Lederman, and R . D . Howe. Design and performance of a tactile display using rc servomotors. Haptics-e Electronic Journal of Haptics Research (www.haptics-e.org), 3(4), September 1999. [52] L . W a r d . Mul t id imens iona l scaling of the molar physical environment. Multivariate Behavioral Research, 12:23-42, 1977. Bibliography 206 [53] Scott Weiss. A n alternative business model for addressing usabili ty: Subscript ion research for the telecom industry. ACM interactions, pages 62-63, Ju ly-Augus t 2005. 207 Appendix A Browser User Evaluation Documents The following documents related to the browser user evaluation are included in this Append ix : 1. Par t ic ipant Demographic D a t a Col lect ion F o r m and Qual i ta t ive A s -sessment Bat te ry 2. Verbal Pro toco l (Experiment Instructions) 3. Post -Study Interview D a t a Col lect ion F o r m 4. Task Inventory 5. Examples of Test Web Pages 203 UBC Physical and multimodal user interfaces - usability and psychophysics Study Questionnaire Form UBC Behavioural Research Ethics Approval #B01-0470 Instructions Please respond to all of the items listed below. If you have any questions, please ask the facilitator. Part 1 : Background 1. Age years 2. Sex: female or male 2. First languaqe(s): Part 2 : Survey 1. a) Did you play a musical instrument as a child? yes or no (please circle one) b) If 'yes', for how many years did you play? years c) If 'yes', which instrument(s) did you play? 2. Do you regularly use any of the following: (check all that apply) Q Mobile phone G PDA (personal digital assistant), including BlackBerry • Game controller with vibration or force feedback (PlayStation Dual Shock, etc.) Q Mouse with a scroll wheel 3. Please respond to the following statements by placing an X in the box according to the following scale: Strongly Agree Agree Neutral Disagree Strongly Disagree 1. I am a multitasker. 2. Mobile phones are useful for more than just talking. 3. Sometimes it's necessary to use a mobile phone while driving. 4. I'm good at working with my hands. 5. I prefer to work on one thing at a time. 6. The vibration function on a mobile phone is annoying. 7. When I am shopping, I often pick up objects to touch them even though I know I won't buy them. Thank you! Please inform the facilitator when you are done. UBC Physical and multimodal user interfaces - usability and psychophysics Study Questionnaire UBC Behavioural Research Ethics Approval #B01-0470\ Instructions Please respond to all of the items listed below. If you have any questions, please ask the facilitator. Part 3 - Post-Study Survey 4. Please respond to the following statements by placing an X in the box according to the following scale: Strongly Agree Agree Neutral Disagree Strongly Disagree 8. I was able to keep my eyes on the screen of the handheld device during the tasks 9. The task of finding information on the page was challenging. 10. The task of pushing the pedals in response to the instructions was challenging. 11. The tactile feedback was helpful in locating items on the page 12. I would pay 5% more for a mobile phone that had this kind of tactile display. 13. The tactile feedback I experienced today was pleasant 14. Overall, I found the device easy to use 5. The facilitator will now ask you some brief questions. 6. Additional comments: Thank you for participating in this study! Experiment instructions Thank you for participating in this experiment today. Please review these instructions and if you have any questions, ask the facilitator. If you feel uncomfortable, you may quit the experiment at any time without penalty. Holding the device Hold the device in your left hand with your thumb resting on the side-mounted tactile display (TD). You can slide the TD up and down, and you will feel some spring resistance. Do not rub, pick, or otherwise abrade the surface of the TD. For best results, try to keep the TD centred under your thumb as you operate the device. Tasks First, you will have a chance to "practice" to get familiar with the controls and tasks. Then you will do a series of tasks that involve finding some information on a web page. Every sixth task, you may be asked to change devices - for example, using the mouse instead of the TD. If you are using the TD, you might feel tactile feedback, or you might not. There will be a one-minute break after each block of six tasks. In addition to browsing web pages, you may be asked to do another task simultaneously. Every 7 seconds, an instruction will appear on the computer screen. You must press the pedals according to the instructions within 7 seconds, before the next instruction appears. Operating the browser Depending on the task, you can move the cursor using either the TD or the mouse scroll wheel. Highlight the link you wish to select, and select it using the ENTER key. Timing Timing information is being collected, so it is important that you do all the tasks as fast as you can. The procedure is as follows: 1. The facilitator will explain the next task. 2. When you are ready to begin the task, press the ENTER key. Timing starts from this point. 3. When you find the information requested, press the SPACE bar as soon as possible. This stops the timer. 4. Inform the facilitator of the answer. Again, it is important that you press the SPACE bar as soon as you see the answer. Do this before you tell the answer to the facilitator. If you make a mistake On the bottom of each web page, there is a link labeled "Back" which will take you to the previous page. If you make a mistake, please continue trying to complete the task as quickly as possible. When you are finished, please inform the facilitator that a mistake was made during the task. 1\ Physical and multimodal user interfaces - usability and psychophysics Stock Inquiry Questions About this document The experimenter will converse with the participant using the following 'stock' questions to guide the inquiry. 1. During the experiment, you were asked to find information using either the mouse, or the handheld device with or without tactile feedback. Did you feel that your performance was better in any of the three conditions? If so, why? 2. Aside from the task of pushing the pedals and the general experimental setup, do you have any suggestions for improvement, for example in the way the handheld device is used, or the way tactile feedback is given? Ref. #: cslMnterview 2004 07 05 v1 O.doc Appendix A. Browser User Evaluation Documents 212 A . l T a s k Inventory A l W h a t is the weather i n London today? A 2 W h a t w i l l the weather be like in London tomorrow? A 3 W h a t w i l l the weather be like in London the day after tomorrow? A 4 W h a t is the weather i n Par is today? A 5 W h a t w i l l the weather be like in Paris tomorrow? A 6 W h a t w i l l the weather be like in Paris the day after tomorrow? A 7 W h a t is the weather in Tokyo today? A 8 W h a t w i l l the weather be like in Tokyo tomorrow? A 9 W h a t w i l l the weather be l ike i n Tokyo the day after tomorrow? A 1 0 W h a t is the weather in Hong K o n g today? A l l W h a t w i l l the weather be like i n H o n g K o n g tomorrow? A 1 2 W h a t w i l l the weather be like in Hong K o n g the day after tomorrow? A 1 3 W h a t is the weather i n San Francisco today? A 1 4 W h a t w i l l the weather be like in San Francisco tomorrow? A 1 5 W h a t w i l l the weather be like in San Francisco the day after tomorrow? A 1 6 W h a t is the weather in Toronto today? A 1 7 W h a t w i l l the weather be like i n Toronto tomorrow? A 1 8 W h a t w i l l the weather be like in Toronto the day after tomorrow? Appendix A. Browser User Evaluation Documents 213 B l If you take the 99 B- l ine from U B C at 1pm, when w i l l you arrive at Broadway station? B2 If you take the 99 B- l ine from U B C at 1pm, when w i l l you arrive at Granvi l le? B 3 If you take the 99 B- l ine from U B C at 1pm, when w i l l you arrive at M a i n ? B 4 If you take the 99 B- l ine from U B C at 9pm, when w i l l you arrive at Broadway station? B 5 If you take the 99 B- l ine from U B C at 9pm, when w i l l you arrive at Granvi l le? B 6 If you take the 99 B- l ine from U B C at 9pm, when w i l l you arrive at M a i n ? B 7 If you take the #44 bus from U B C at 10am, when w i l l you arrive at Macdonald? B 8 If you take the #44 bus from U B C at 10am, when w i l l you arrive at Davie? B 9 If you take the #44 bus from U B C at 10am, when w i l l you arrive at Waterfront station? BIO If you take the #44 bus from U B C at 5pm, when w i l l you arrive at Macdonald? B l l If you take the #44 bus from U B C at 5pm, when w i l l you arrive at Davie? Appendix A. Browser User Evaluation Documents 214 B 1 2 If you take the #44 bus from U B C at 5pm, when w i l l you arrive at Waterfront station? C I W h e n is the movie "L 'Enfan t" playing at the Ridge Theatre? C 2 W h e n is the movie "L 'Enfan t" playing at F i f t h Avenue Cinemas? C 3 W h e n is the movie "L 'Enfan t" playing at Empi re Oakridge? C 4 W h a t rat ing d id the movie "L 'Enfan t" receive in its review in the Los Angeles Times? C 5 W h a t rat ing d id the movie "L 'Enfan t" receive in its review in the New Y o r k Post? C 6 W h e n is the movie "The Promise" playing at the Ridge Theatre? C 7 W h e n is the movie "The Promise" playing at the Pacific Cinematheque? C 8 W h e n is the movie "The Promise" playing at the Si lverci ty Riverpor t? C 9 W h a t rat ing d id the movie "The Promise" receive in its review in the Globe and M a i l ? C I O W h a t rat ing d id the movie "The Promise" receive in its review in the Washington Post? O W h a t ' s N e w W e a t h e r *t T r an s i t 1© Mov ie s I F i n a n c e |§) Spo r t s H N e w s i 1 M a n a g e A c c o u n t Browser-content version 29-mar-o6 Terms of Service Privacy Statement Weather SPRING OUTLOOK Canadians eagerly anticipate spring each year. Find out how soon you can shed the winter layers and expect the warmer weather to arrive. >> More... LOCAL WEATHER Vancouver [Change] 13°C 15°C 10°C Today Tomorrow Tuesday TRAVEL FORECAST • No r th A m e r i c a • E u r o p e • A s i a • A f r i c a • S o u t h A m e r i c a • An ta r c t i c a Back to Home orth America NORTH AMERICA WEATHER Bos ton C a l g a r y C h i c a g o D e n v e r E d m o n t o n Los A n g e l e s M o n t r e a l O t t a w a S a n F ranc i s co Sea t t l e T o r o n t o V a n c o u v e r W a s h i n g t o n , D.C. W i n n i p e g Back to Weather Back to Home Weather - Travel Forecast SAN FRANCISCO, CALIFORNIA 12°C 12°C 11°C Today Tomorrow Tuesday > > Extended Forecast Back to North America Back to Weather Back to Home 2iS —i TRANSIT INFORMATION • News • Planned Maintenance • Company Information BUSES AT UBC 004 - Phibbs Exch/Powell/Downtown/ UBC 009 - Bdrv/Bwav Stn/Gran/Alma/ UBC 017 - Qak/Downtown/UBC 025 - Brentwood Stn/UBC 041 - Joyce Stn/Crown/UBC 043 - Joyce Stn/UBC 044 - UBC/Downtown 049 - Metrotown Stn/Dunbar Loop/ UBC 099 - Broadway Stn/UBC (B-Line) 258 - UBC/West Vancouver 480 - UBC/Richmond Centre N17 - Downtown/UBC Niqhtbus Back to Home Transit - 44 Downtown DEPARTURE TIME 9:00 am 10:00 am 11 :00 am 12:00 pm 1:00 pm 2:00 pm 3:00 pm 4 :00 pm 5:00 pm 6:00 pm 7:00 pm 8:00 pm 9:00 pm Back to Transit Back to Home Transit - 44 Stops 44 U B C / D O W N T O W N UBC BUS LOOP BAY 7 5:04 p ALMA ST 5:18 p MACDONALD ST 5:22 p VINE ST 5:24 p BURRARD ST / 3RD AV 5:26 p PACIFIC ST 5:30 p DAVIE ST 5:31 p R O B S O N S T 5:33 p PENDER ST 5:34 p WATERFRONT STATION 5:35 p Back to 44 UBC/Downtown Back to Transit Back to Home L ' E N F A N T RIDGE THEATRE 1:30, 4:30, 8:00, 10:30 FIFTH AVENUE CINEMAS 5:00, 7:30 EMPIRE OAKRIDGE 12:30, 6:00, 8:00 >> MORE THEATRES News Vi Trailers 1^ FEATURED FILMS •1*193 if, Basic Instinct 2 - : •**' 11 L'Enfant ygK:.l .1 || Idlewild I i" i i i t i i K i n iiiii 'PKflJP' P R O M I S E The Promise ii«iiii,ii..inii i, i. Movies - L'Enfant ,'"'.! Dispossessed twenty-year o l d B r u n o (Jeremie Renier) lives w i t h his eighteen-year-old g i r l f r i e n d S o n i a ( D e b o r a h Francois) i n Seraing, a n eastern Belg ian steel t o w n . T h e y l ive off Sonia 's u n e m p l o y m e n t benefits a long w i t h the p a n h a n d l i n g a n d petty thefts c o m m i t t e d by B r u n o a n d his gang. T h e i r lives change forever w h e n S o n i a gives b i r t h to t h e i r c h i l d , J i m m y . NOW PLAYING Get Showtimes View Trailer Reviews F o r e i g n , D r a m a R a t i n g : N o t yet rated J e a n - P i e r r e , L u c Dardenne (dir .) J e r e m i e R e n i e r D e b o r a h Francois Back to Movies Back to Home Back to L'Enfant Back to Movies Back to Home 217 Appendix B Browser User Study Supplemental Data T h i s a p p e n d i x c o n t a i n s s u p p l e m e n t a l d a t a c h a r t s r e l a t e d t o t h e s t a t i s t i c a l a n a l y s i s o f t h e b r o w s e r user s t u d y . T h e f o l l o w i n g a re i n c l u d e d : • V a r i a n c e i n f o r m a t i o n for t h e i n d i v i d u a l p a r t i c i p a n t a n a l y s i s , i n c l u d i n g c h a r t s a n d s t a n d a r d d e v i a t i o n t a b l e s for b o t h t h e m e a s u r e d a n d n o r -m a l i z e d d a t a . N o r m a l i z e d m e a n s a re a lso p r o v i d e d . T h e m e a n s for t h e m e a s u r e d d a t a were r e p o r t e d i n T a b l e 7.3. • D a t a t a b l e s for t a s k d i f f i cu l t y a n a l y s i s , i n c l u d i n g scores u s e d for t h e n o r m a l i z a t i o n o f m e a s u r e d t a s k t i m e s . Appendix B. Browser User Study Supplemental Data 218 Normalized Task Time vs Condition Participant N allcond mouse slider- slider+ 10 51 12280 10307 12644 14027 16 50 12316 8788 14144 14124 9 53 12442 8480 14918 14016 13 48 12727 8434 14738 15008 11 46 14653 11130 15969 17268 6 53 14817 10341 17252 16994 5 54 15270 12683 16499 16628 7 49 15722 9817 17858 18795 12 49 16390 13050 18126 18181 8 47 16592 12205 18689 18596 14 46 17270 11912 20302 19975 4 49 17523 15036 19414 18275 15 50 20170 15933 21820 22142 ALL 645 15211 11354 17056 17247 Appendix B. Browser User Study Supplemental Data 219 Normalized Task Time By Participant And Condition 10 16 9 13 11 6 S 7 12 8 14 4 15 Participant Std.Dev. Norm. Task Time vs Cond. Participant N allcond mouse slider- slider+ 10 51 2987 3218 1821 2451 16 50 3414 2879 1595 2232 9 53 5329 2336 7056 2533 13 48 3709 1772 1726 2731 11 46 4302 4266 3201 2564 6 53 4505 3527 3147 2855 5 54 3195 2798 2807 2344 7 49 4900 2983 2797 2977 12 49 3827 3209 2190 3121 8 47 5183 3920 5397 2793 14 46 5450 4141 3968 3361 4 49 3609 4184 2721 2041 15 50 4311 3428 3704 2880 ALL 645 4822 4822 4358 3573 Appendix B. Browser User Study Supplemental Data 220 Measured Task Time By Participant And Condition 30000 T ~ - ; - ™.,.,.™ „ „., 25000 10 16 9 13 11 6 5 7 12 8 14 4 15 ALL Std.Dev. Task Time vs Condition Participant N allcond mouse slider- slider+ 10 51 4032 3445 3238 3966 16 50 4152 3058 3064 3096 9 53 6388 2907 8625 3799 13 48 4440 2059 2772 3979 11 46 5095 4452 3671 4445 6 53 5265 3712 4109 4063 5 54 3726 2990 3021 3530 7 49 5734 3148 3708 4349 12 49 4533 4516 2867 3528 8 47 6386 4386 7791 2825 14 46 6058 4490 4476 4632 4 49 4504 4872 3596 3850 15 50 5431 4849 4622 4339 ALL 645 6692 4489 5298 4566 Task Difficulty Analysis G r o u p T a s k a l l c o n d m o u s e s l i d e r - s l i d e r + f a s t e r a l l c o n d ( N ) m o u s e ( N ) s l i d e r - ( N ) s l i d e r + ( N ) D I F F I C U L T Y N O R M S C O R E A 0 1 4 1 0 5 7 6 4 2 1 4 6 0 9 1 4 7 2 3 - 1 2 1 3 8 0 . 9 2 3 1 . 0 8 4 A 1 1 5 1 3 0 8 4 6 0 1 7 4 6 0 1 6 3 6 1 + 1 3 3 7 3 0 . 9 9 0 1 . 0 1 0 A 2 1 1 2 9 0 9 2 1 3 1 1 7 6 5 1 4 1 6 9 - 1 2 6 2 4 0 . 7 3 9 1 . 3 5 4 A 3 1 6 0 2 0 5 4 2 1 1 5 9 1 8 1 7 6 0 7 - 1 3 1 5 7 1 . 0 4 8 0 . 9 5 4 A 4 1 3 8 2 8 9 7 2 9 1 4 9 6 6 1 6 0 3 1 - 1 1 3 5 3 0 . 9 0 5 1 . 1 0 5 A 5 1 4 2 9 0 1 2 4 0 0 1 5 1 6 0 1 5 0 6 9 + 1 3 4 6 3 0 . 9 3 5 1 . 0 7 0 A 6 1 3 4 5 7 1 0 8 6 0 1 9 5 7 0 1 4 0 6 7 • + 1 5 8 3 4 0 . 8 8 0 1 . 1 3 6 A 7 1 4 5 8 5 1 1 9 8 7 1 7 3 4 7 1 7 0 7 4 + 1 0 5 2 3 0 . 9 5 4 1 . 0 4 8 A 8 1 2 8 7 3 1 0 2 6 9 1 2 4 3 0 1 7 7 6 4 - 1 4 7 3 4 0 . 8 4 2 1 . 1 8 8 A 9 1 3 0 5 1 6 0 6 2 1 3 5 8 3 1 4 1 4 0 - 1 1 1 7 3 0 . 8 5 4 1 . 1 7 1 A 1 0 1 2 3 9 4 9 4 1 9 1 1 9 2 4 1 4 4 6 2 - 1 1 3 3 5 0 . 8 1 1 1 . 2 3 3 A 1 1 1 3 4 2 5 1 1 4 4 1 1 3 7 4 7 1 4 5 9 1 - 1 1 3 4 4 0 . 8 7 8 1 . 1 3 9 A 1 2 1 3 9 2 9 9 1 2 4 1 7 4 1 4 1 4 5 9 0 + 1 6 4 4 8 0 . 9 1 1 1 . 0 9 7 A 1 3 1 1 9 4 6 9 0 9 6 # D I V / 0 ! 1 5 3 6 7 1 1 6 0 5 0 . 7 8 1 1 . 2 8 0 A 1 4 1 0 3 7 1 8 5 9 3 1 5 9 2 1 1 2 0 7 7 + 1 0 6 1 3 0 . 6 7 8 1 . 4 7 4 A 1 5 1 2 5 4 1 7 0 8 4 1 5 8 9 2 1 5 0 8 1 + 1 1 4 5 2 0 . 8 2 0 1 . 2 1 9 A 1 6 1 1 9 6 9 1 0 2 7 4 1 3 0 8 7 1 5 3 7 8 - 1 1 6 3 2 0 . 7 8 3 1 . 2 7 7 A 1 7 1 7 0 9 4 # D I V / 0 ! 1 5 1 7 1 1 9 4 9 9 - 9 0 5 4 1 . 1 1 8 0 . 8 9 4 B 0 1 6 7 5 1 1 1 7 8 8 1 9 0 4 6 1 8 6 5 7 + 1 7 5 5 7 1 . 0 9 6 0 . 9 1 3 B 1 1 6 5 2 3 1 5 3 8 9 1 6 3 3 3 2 0 3 6 8 - 1 9 9 7 3 1 . 0 8 1 0 . 9 2 5 B 2 1 7 9 8 3 1 4 2 1 2 1 8 8 4 0 1 6 5 0 8 + 1 9 1 1 3 5 1 . 1 7 6 0 . 8 5 0 B 3 1 9 9 7 4 1 5 7 6 6 2 1 0 9 3 2 1 3 1 7 - 2 2 5 8 9 1 . 3 0 7 0 . 7 6 5 B 4 1 7 4 1 9 1 5 2 4 0 1 7 9 1 6 1 9 0 2 3 - 1 8 5 9 4 1 . 1 3 9 0 . 8 7 8 B 5 1 8 4 7 8 1 7 0 9 5 2 1 1 6 0 1 8 5 4 1 + 1 3 6 3 4 1 . 2 0 9 0 . 8 2 7 B 6 1 5 0 7 4 1 1 0 2 5 1 7 2 3 8 1 4 3 9 5 + 2 2 4 1 0 8 0 . 9 8 6 1 . 0 1 4 B 7 1 4 6 6 0 1 1 7 8 9 1 7 0 1 4 1 6 9 1 8 + 1 8 8 4 6 0 . 9 5 9 1 . 0 4 3 B 8 1 4 3 1 9 8 9 0 9 1 7 1 6 1 1 7 8 3 4 - 1 6 6 4 6 0 . 9 3 7 1 . 0 6 8 B 9 1 6 0 5 8 1 4 5 3 8 1 6 9 7 9 1 7 0 6 4 - 1 8 7 5 6 1 . 0 5 0 0 . 9 5 2 B 1 0 1 7 9 3 3 1 4 5 8 6 2 1 6 3 8 1 8 5 4 6 + 2 0 7 5 8 1 . 1 7 3 0 . 8 5 2 B 1 1 1 5 8 9 4 1 1 6 9 2 # D I V / 0 ! 2 1 2 9 6 1 6 9 0 7 1 . 0 4 0 0 . 9 6 2 C 0 1 6 1 4 5 8 9 2 7 1 8 5 6 2 1 9 7 4 3 - 1 7 5 6 6 1 . 0 S 6 0 . 9 4 7 C 1 1 6 8 6 2 1 2 5 6 4 1 9 4 5 4 1 9 5 5 4 - 2 1 8 6 7 1 . 1 0 3 0 . 9 0 7 c 2 1 8 3 2 9 1 3 4 1 5 1 9 3 1 8 2 2 0 1 0 - 2 0 7 5 8 1 . 1 9 9 0 . 8 3 4 c 3 1 7 6 9 4 1 3 7 8 8 1 8 5 6 8 1 9 9 5 6 - 2 1 5 1 2 4 1 . 1 5 7 0 . 8 6 4 c 4 2 0 8 3 5 1 6 9 3 9 2 0 1 6 1 2 4 0 3 2 - 2 3 6 8 9 1 . 3 6 3 0 . 7 3 4 c 5 1 7 6 2 3 1 3 7 0 7 1 9 1 5 4 2 0 5 6 6 - 2 2 8 7 7 1 . 1 5 3 0 . 8 6 7 c 6 1 7 8 7 9 1 1 9 3 8 2 4 9 9 4 2 3 0 6 4 + 2 1 1 1 7 3 1 . 1 7 0 0 . 8 5 5 c 7 2 0 0 1 1 1 4 4 7 8 2 0 8 3 1 2 3 3 9 1 - 2 3 7 6 1 0 1 . 3 0 9 0 . 7 6 4 c 8 1 8 1 1 2 9 8 9 2 2 6 4 6 2 2 1 3 3 4 + 2 4 9 5 1 0 1 . 1 8 5 0 . 8 4 4 c 9 2 0 8 6 9 1 4 3 6 3 2 4 9 4 1 2 3 3 0 3 + 2 1 7 7 7 1 . 3 6 5 0 . 7 3 3 A L L A L L 1 5 2 8 7 1 2 0 5 6 1 8 2 4 5 1 8 4 9 9 - 6 4 5 2 1 6 2 1 0 2 1 9 1 . 0 0 0 1 . 0 0 0 A A L L 1 3 4 7 9 9 7 5 3 1 5 1 7 1 1 5 4 7 2 - 2 1 4 7 1 6 8 7 5 0 . 8 8 2 1 . 1 3 4 B A L L 1 6 7 9 0 1 3 4 7 5 1 8 4 9 5 1 8 3 5 4 + 2 1 8 7 2 7 3 7 3 1 . 0 9 8 0 . 9 1 1 C A L L 1 8 S 0 8 1 2 8 9 7 2 1 0 1 0 2 1 8 4 6 - 2 1 3 7 3 6 9 7 1 1 . 2 1 1 0 . 8 2 6 222 Appendix C gif2hapticon Code { / / GIF2HAPTICON / / U t i l i t y for c o n v e r t i n g ( o p t i o n a l l y , animated) GIF f i l e s to hapt i c icon XML / / format for the THMB device / / (c) 2006, Joseph Luk / / Compile command: gec - I S H O M E / ImagcMagick - 6 .1.9 -LSHOME/ImagcMagick - 6.1.9/ magick / . 1 i b s —IMagick g i f 2 h a p t i c o n . c —o g i f 2 h a p t i c o n / / ensure LD_LIBRARY_PATH=/Volumes/ U s r / admin /Image Magick — 6 . 1 . 9 / l t d l / - l i b s : / Volumes / Usr / admin / ImagcMagick — 6 . 1 . 9 / m a g i c k / . l i b s # i n c l u d c < u n i s t d . h> #i n c1u d e < c t y p c . h > ^ i n c l u d e < s t d i o . h > # i n c l u d c < s t d l i b . h > #i n e lude < s t r i n g . h> ^ i n c l u d e < a s s c r t . h> # i n c l u d c < m a g i c k / a p i . h > bool o u t p u t T act i l c M o vie ( F I L E * f i l c , Image * image, unsigned char * p i x c l s , int numframes ) ; bool o u t p u t H apt ic I co n ( F I L E * f i l c , Image * imagc, unsigned char * p i x c l s , int numframes ) ; int main ( in t argc , char* argv [] ) / / ImagcMagick s t u f f Ex c c p t i o n I n f o except ion ; Image *imagc ; Imagelnfo * i m a g c . i n f o ; unsigned char * p i x c l s ; int x , y , i ; F I L E * f i l e ; unsigned long numframes; unsigned long f r a m c s i z c ; enum output .modes { t a c t i l c m o v i c , h a p t i c i c o n } output-mode; int ch ; char * o u t f i l c ; char d e f a u l t o u t f i l e [ ] = " o u t . x m l " ; output .modc = h a p t i c i c o n ; if (argc > 1 &&; ! strcmp ( argv [1] ," help") ) { p r i n t f ("Usage: g i f 2 h a p t i c o n <output modc> <animatcd GIF f i l c n a m c > <XML output f i lename >\n" ) ; p r i n t f (" \ t<output modc> opt ions can b c : \ n " ) ; p r i n t f ( " \ t \ t —t Output t a c t i l e m o v i c \ n " ) ; p r i n t f ( " \ t \ t - i ( lowercase l e t t e r i ) Output hapt i c icon (DEFAULT) \ n" ) ; p r i n t f ( " \ t d c f a u l t s to < i n . g i f > and <out .xml> if no arguments arc g i v e n . \ n " ) ; p r i n t f ( " \ t N o t e : a l l frames must be the same s ize (no crazy opt imized G I F s ) . \ -" ) ; p r i n t f ( " \ t l f you have problems , go to ImageReady animat ion pa le t te —>Opt im ize A n i m a t i o n . . . and dese lec t e v e r y t h i n g , \ n" ) ; ex i t (1 ) ; } while ( (ch = getopt ( a r g c , a r g v , " t i " ) ) != —1) Appendix C. gif2hapticon Code 223 swi tch (ch) { case ' t ' : output-mode = t a c t i 1 e m o v i c ; break ; case ' i ' : output .modc = h a p t i c i c o n ; break ; case ' ? ' : i f ( i s p r i n t ( o p t o p t ) ) f p r i n t f ( s t d c r r , " Unknown opt ion ' — % c ' . \ n " , o p t o p t ) ; else f p r i n t f ( s t d c r r , "Unknown opt ion c h a r a c t e r '\\x%x ' . \ n " , o p t o p t ) ; r e t u r n 1 ; d e f a u l t : abort () ; } argc —= o p t i n d ; a r g v += o p t i n d ; / / Load the image I n i t i a l i z c M a g i c k ( * a r g v ) ; G c t E x c e p t i o n I n f o ( & e x c c p t i o n ) ; image_info = C l o n d m a g c I n f o (( I mage In fo * )NULL) ; i f (argc >= 1) ( void ) C o p y M a g i c k S t r i n g ( i m a g c . i n f o —>f i len am c , ar g v [0] , MaxTextExtcnt ) ; else ( void ) C o p y M a g i c k S t r i n g ( i m a g c . i n f o —>f i len am c ," in . g i f" . M a x T e x t E x t c n t ) ; i magc=Rcad I mage ( image- info , ^ e x c e p t i o n ) ; i f ( e x ce p t io n . s c v c r i t y != U n def i ncd Ex ccp t io n ) C a t c h E x c c p t i o n (^except ion ) ; i f (image = (Image *) NULL) { f p r i n t f ( s t d c r r , " C o u l d n ' t open the image f i l e . Please ensure you have an i mage named '%s ' in the c u r r e n t d i r e c t o r y . \ n " , i m a g c . i n f o —>f i 1 cnamc ) cx it ( 1 ) ; } / / Deal with the animated GIF numframcs = G c 11 m age List Len gt h ( i m age ) ; f r a m c s i z c = (image—>rows ) * (i mage —>co 1 u m n s ) * s i z c o f (unsigned int ) ; i f (!( p i x e l s = (uns igned char *) mal loc (( image—>rows )*( image—>colu m ns )* s i z c o f ( unsigned i n t ) ) ) ) { f p r i n t f ( s t d c r r , " C o u l d n ' t a l l o c a t e image memory %u by %u . \ n " , i mage—> columns , image—>rows ) ; e x i t ( 1 ) ; } / / Done l o a d i n g image. p r i n t f ("Loaded image %s (%u x %u) p i x e l s by %u f r a m e s . \ n " , i m a g e . i n f o - > f i lename , image—>columns , image—>rows , numframcs) ; / / Open output f i l e if (argc == 2 ) o u t f i l e = argv [ 1 ] ; else o u t f i l e = d c f a u 11 o u t f i 1 c ; f i l e = fopen ( o u t f i l e , "w"); if (! f i l e ) { f p r i n t f ( s t d c r r , " C o u l d n ' t open output f i l e %s for w r i t i n g . \ n " , o u t f i l e ) ; ex i t (1); } / / Dump the XML switch (output-mode) { Appendix C. gif2hapticon Code 224 case t a c t i l e m o v i e : p r i n t f (" E x p o r t i n g o u t p u t T a c t i l c M o v i c p r i n t f ( " Done . \ n " ) ; break ; case h a p t i c i c o n : p r i n t f ( " E x p o r t i n g to o u t p u t H a p t i d c o n ( f i l e p r i n t f ( " Done . \ n " ) ; break ; d e f a u l t : abort ( ) ; to f i l e ( f i l e , _\"%s\" image , t a c t i l e movie format . . . \ n" ixels , numframes) ; f i l e \"%s\" , , image , p ixe hapt ic icon format . 1 s , numframes ) ; • \ n " o u t f i 1 c ) ; o u t f i 1 c ) ; f c l o s c ( f i l e ) ; free ( p i x e l s ) ; D c s t r o y l m a g e l n f o ( i m a g c . i n f o ) ; D c s t r o y E x c c p t i o n l n f o f & c x c e p t i o n ) ; Destroy Magick ( ) ; re turn 1 ; bool o u t p u t T a c t i l c M o v i c ( F I L E * f i l c , Image *imagc, unsigned char * p i x c l s , int numframcs ) { assert ( f i l e ) ; assert (image) ; assert ( p i x e l s ) ; Image * working image ; E x c c p t i o n l n f o except ion ; f l oa t s c a l c d p i x c l v a l u c ; f p r i n t f ( f i l e , "<?xml v e r s i o n =\" 1.0\" e n c od i n g = \ " U T F - 8 \ " ? >\n " ) ; f p r i n t f ( f i l e , " < IDOCTYPE T a c t i l c M o v i c SYSTEM \ " T ac t i 1 e M o v i c - 0 . 2 . d t d \ " > \ n " ) ; f p r i n t f ( f i l e , " < T a c t i 1 c M o v i c V e r s i o n =\"0.2\" F r a'meS izc Y=\"%u \ " F ramcS izcX = \"%u \ " NhFrarncs=\"%u\">\n" , image—>rows , i m age— >co 1 u m ns , numframes) ; for ( i n t i=0; i< n urn fr amcs; iH—\-) { f p r i n t f ( f i l e , " <Frame Index=\"%u\" >\n" , i ) ; wor k ingimagc=GctImagcFrom List ( image, i ) ; / / get the i t h image if ( ! ( E x p o r t I m a g c P i x c I s (workingimage , 0 ,0 , image—>columns , image— >rows , "I" , C h a r P i x c l , (vo id * ) p i x c l s , ^ e x c e p t i o n ) ) ) { f p r i n t f ( s t d c r r , " C o u l d n ' t acqu ire image p i x e l s for frame %d . \ n" , i ) ; oxi t (1) ; } for ( i n t y=0; y<working image —> rows; y + + ) { f p r i n t f ( f i l e , " <Row r =\"%u\" > " ,y ) ; for ( i n t x=0; x<workingimagc—>columns ; x+4-) { s c a l c d p i x c l v a l u c = (*( p i x e l s + y*workingimagc —> columns + x )) / 255.0; f p r i n t f ( f i l e , "%1.3f", s c a l c d p i x c l v a l u c ) ; if ( x < (workingimage—>columns — 1)) f p r i n t f ( f i l e ," "); f p r i n t f ( f i l e , "</Row>\n"); } f p r i n t f ( f i l e , " < /Framc>\n") ; } f p r i n t f ( f i l e , " < / T a c t i l c M o v i e >\n" ) ; Appendix C. gif2hapticon Code 225 bool o u t p u t H a p t i c I c o n ( F I L E * f i l c , Image * image, unsigned char *p ixe l s , int numframes ) { assert ( f i l e ) ; assert (image) ; assert ( p i x e l s ) ; E x c e p t io n Info except ion ; f l oa t s c a l c d p i x c l v a l u e ; Image * workingimage ; if ( image—>columns != 1) { f p r i n t f ( s t d e r r , "Warning: only the f i r s t column of p i x e l s w i l l be e x p o r t e d . \ ; } f p r i n t f ( f i l e , "<?xml v e r s i o n =\" 1 . 0 \ " c n co d i n g =\" U T F - 8 \ " ?> \ n " ) ; f p r i n t f ( f i l e , " < IDOCTYPE H a p t i c l c o n >\n\n" ) ; f p r i n t f ( f i l e , "<icon>\n"); if (numframcs 1) { / / s t a t i c icon if ( ! ( Ex po rt Im ageP ixcls ( image, 0 ,0 , image—>columns , image—>rows, " I " , Char P i x e l , (vo id * ) p i x c l s , ^ e x c e p t i o n ) ) ) { f p r i n t f ( s t d c r r , " C o u l d n ' t acqu ire image p i x e l s . \ n " ) ; cx i t ( 1 ) ; } < t y p c > S p a t i a l I c o n < / t y p c > \ n " ) ; <hcight>%d</hcight >\n" , image->rows ) ; " <elcmcn t >\n " ) ; " <indcx>%d</indcx >\n" , y) ; va lue to f l o a t between 0 and 1 ic = (*( p i x e l s + y* i mage—>colum ns + x )) / 255.0; value to 0 -> ( -50V) , 1 -> ( + 50V) ic = s c a l c d p i x c l v a l u e * 100 — 50.0; " <value >%.f </value >\n" , s c a l c d p i x e l v a l u e ) ; " < / c l c m c n t > \ n " ) ; } } else { / / animated icon f p r i n t f ( f i l e , " <ty pc > Tern po ral Ico n < / 1 y pc >\n " ) ; f p r i n t f ( f i l e , " <hcight >%d</h c igh t >\n " , im age—>rows ) ; for ( i n t i =0; i <numframcs ; i + + ) { f p r i n t f ( f i l e , " <Framc>\n"); f p r i n t f ( f i l e , " < F ramcIdx>%d</Fr ameldx >\n " , i ) ; workingimagc=GetImagcFroniList (image , i ) ; / / get the i th image if ( ! ( E x p o r t l m a g c P i x c I s (working image , 0 , 0 , image—>columns , image—>rows , " I " , C har Pixel , (vo id * ) p i xc 1 s , ^ e x c e p t i o n ) ) ) { f p r i n t f ( s t d e r r , " C o u l d n ' t acqu ire image p i x e l s for frame %d . \ n" , i ) ; f p r i n t f ( f i l e f p r i n tf ( f i l e int x = 0; for ( int y = 0; { fpr i n t f ( f i fpr i n t f ( fi / / sca le pi sea 1 c d p i x c 1 / / sca le pi sea 1 e d p i x e 1 fpr i n t f ( f i fpr i n t f ( fi Appendix C. gif2hapticon Code 226 exi t ( 1 ) ; } int x = 0; for ( i n t y=0; y<im age—>ro w s ; y + + ) { f p r i n t f ( f i l e , " <clcmcnt>\n"); f p r i n t f ( f i l e , " < i n d cx >%cl </i n dcx >\n " , y ) ; / / sca le p i x e l value to f l oa t between 0 and 1 s c a l c d p i x c l v a l u c = (*( p i x e l s -f- y * i inage—>co 1 umns + x )) / 255.0; / / sca le p i x e l value to 0 -> ( -50V) , 1 -> (-f-50V) s c a l c d p i x c l v a l u c = s c a l c d p i x c l v a l u c * 100 — 50.0; f p r i n t f ( f i l e , " < value >%. f </val uc >\n " , s c a 1 e d p i x e 1 v a 1 u e ) ; f p r i n t f ( f i l e , " </c lcmcnt >\n" ) ; } f p r i n t f ( f i l e , " </Frame >\n" ) ; } } f p r i n t f ( f i l e , " < / i co n >\n " ) ; 227 Appendix D Browser Prototype Code D . l Tactile I /O Loop T h e browser prototype applicat ion software layer for the tactile I / O loop consists of several modules, which are listed below. Input and output func-tions are handled by the S T R e S S L i b r a r y [29] by Vincent Levesque. • DataUpdateThread - T h e input /ou tput loop thread. • HapticPageMap - Functions related to the haptic page map and its haptic icons. • BrowserShared - Shared memory data structure for inter-thread com-municat ion. • BrowserXMLBits - Functions related to parsing the X M L haptic icon and page map files. • main - the ma in executable. D . l . l DataUpdateThread.h # i f n d e f DATAUPDATE_THREADJi #dcf ine DATAUPDATE_THREAD_H # i n c l u d c < ios trcam> ^ i n c l u d e <fstrcam> # i n c l u d c < s t d i o . h > # i n c l u d c <s t d 1 i b . h> # i n c l u d c < s t r i n g . h> # i n c l u d c < d i r c n t . h> # i n c l u d e <math . h> ^ i n c l u d e < a s s c r t . h> # i n c l u d e < s y s / t i m c . h> ^ i n c l u d e " s t r c s s d / Fir m ware Info X M L . h" Appendix D. Browser Prototype Code 228 ^ i n c l u d e " s t r c s s d / Dcv iccInfoXML . h" # i n c l u d c " s t r c s s d / S treamDcv icc . h" # i n c l u d c " s t r e s s d / P a c k c t L o g g e r . h " ^ i n c l u d e " s t r e s s d / i n i t .h" # i n c l u d e " BrowscrSharcd . h" / * TODO Move f i l e I / O r o u t i n e s into yet another t h r e a d , running at lower p r i o r i t y . . . th i s thread s h o u l d : — Check if there is a new page map — If so , create a new i n s t a n c e of the page map v a r i a b l e and load it in — Handle p u t t i n g the p o s i t i o n back to the browser? * / c las s DataUpdateThread { p u b l i c : DataUpdateThread ( BrowscrSharcd * s , s t r c s s d : : S t reamDcv icc* d) ; v i r t u a l " DataUpdateThread () ; s t a t i c void * S t a r t ( v o i d *p) ; p r o t e c t e d : void S t a r t - ( ) ; p r i v a t e : BrowscrSharcd * shared ; s t r c s s d : : S t reamDcv icc* dev ; S l i d e r S m o o t h e r smoother ; }; # c n d i f D . l .2 DataUpdateThread.epp / / DataUpdateThread . epp / / Joseph Luk , July 2006. Based on code by Vincent Levesque and Shannon L i t t l e . ^ i n c l u d e " DataUpdateThread . h" # i n c l u d c " BrowscrXMLBits . h" DataUpdateThread : : DataUpdateThread ( BrowscrSharcd* s , s t r e s s d : : S treamDcvicc* d ) { shared = s; dev = d ; } DataUpdateThread : : ~ DataUpdateThread ( ) { } void * DataUpdateThread :: S t a r t (vo id *p) { ( (DataUpdateThread *) p )-> S t a r t . ( ) ; r e t u r n 0; } void DataUpdateThread : : S t a r t . () { Appendix D. Browser Prototype Code 229 f l o a t s l i d c r . p o s ; int map.pos = 0; f l oa t norm.map.pos ; int 1 ast _m ap-pos = 0; int c l i c k . c o u n t ; s t r c s s d : : V o l t I m g img ( dev—>Nb ActuatorsX () , dev—>Nb Actuators Y () ) ; s t r c s s d ;: S t a t u s l n f o i n f o ; / / s l i d e r parameters const f l oa t s l i d c r . m i n = 1700; / /was 1615 const f l oa t s l i d e r . m a x = 2770; / /was 2799 p r i n t f (" S t a r t i while (1) { Data update t h r e a d . . . \ n " ) ; / / read s l i d e r dev—> RcadS tat us ( i n f o ) ; / / c u r r e n t l y takes e x a c t l y 3 ms / / ( for c a l i b r a t i o n ) // // // // // // p r i n t f ("%d\n" , info . s l i d c r D a t a . ) ; f ( i n f o , s l i d e r D a t a . < s l i d c r . m i n ) s l i d c r . p o s = 0.0; lsc i f ( i n f o . s 1 i d c r D a t a _ > s l i d e r . m a x ) s l i d c r . p o s = 100.0; 1 s c s l i d c r . p o s = ( info , s l i d c r D a t a . — s l i d c r . m i n ) / ( s l i d e r . m a x — s l i d c r . m i n ) * 100.0; l i d e r . p o s = 100.0 — s l i d c r . p o s ; l i d c r - p o s = smoother. S m o o t h S l i d c r P o s ( s l i d c r . p o s ) ; f ( shared->GctPageMapRcady () ) map.pos = shared —>pagcmap—>Currcn t Ma pP os ( s l i d c r . p o s ) ; / / Get c l i c k count c l i c k . c o u n t = info , c l i c k C o u n t . ; / / Export data to shared memory shared —>SctCl icks ( c l i c k - c o u n t ) ; i f ( map.pos { l a s t . m a p . p o s ) / / Export data to shared memory shared—>SctMapPos ( map-pos ) ; / / wr i te to device p r i n t f ("Compute i m a g c \ n " ) ; shared— >pagemap—>Com put elm age (&img , map.pos) ; p r i n t f ("Write Imagc\n"); dev—>WriteImagc(&img ) ; p r i n t f ( " D o n e . \ n " ) ; l a s t . m a p . p o s = map.pos; / / cache last value // } } / / if ( sharod->GetPageMapRcady () ) } / / w h i l o ( l ) Appendix D. Browser Prototype Code 230 D . l . 3 Hapt icPageMap.h # i f n d c f HAPTICPAGEMAPJi #def ine H APTICP AG EM A P_H c lass Hapt icPageMap; / / def ined below c lass H a p t i c l c o n ; / / de f ined below # i n c l u d e #i n c1u d e #i n c1u d c # i n c l u d c # i n c l u d c st ressd / FirmwarcInfoXML . h" s t r e s s d / D c v i c c I n f o X M L . h" st rcssd / S treamDcvicc .h" s t r c s s d / P a c k c t L o g g e r . h " s t r c s s d / i n it .h" # i n c l u d c " BrowscrXMLBit s . h" / / Support f u n c t i o n s for time s t u f f / / r e t u r n the time ( in seconds) s ince t imeStamp, with microsecond p r e c i s i o n double T i m c S i n c c (const s t r u c t t i m c v a l *t imcStamp, s t r u c t t i m e v a l * currcntT imcStamp = NULL) ; / / r e turns the d i f f e r e n c e in the t imcstamps , in m i l l i s e c o n d s ( tvl—tv2) double T i m c D i f f (const s t r u c t t i m c v a l * t v l , const s t r u c t t i m c v a l * t v 2 ) ; / / stamp the time into a t i m c v a l s t r u c t void TimeStamp ( s t r u c t t i m c v a l *timeStamp) ; c la s s H a p t i c l c o n C o n t e n t { p u b l i c : H a p t i c l c o n * icon parent ; char * t y p c ; char *namc; v i r t u a l bool R c n d c r l c o n (double * b u f _ p t r ) = 0; / / W rite th i s hapt i c icon to the buffer at the zero p o s i t i o n r t u a l bool A l l o c a t c D a t a ( i n t s i z e ) = 0; r t u a l double GctData ( i n t index) = 0; r t u a l bool Sc tData ( i n t index , double va lue) = 0; }; c lass S p a t i a l l c o n C o n t e n t : p u b l i c H a p t i c l c o n C o n t e n t { p u b l i c : S pat i al Ico n Co n tc n t ( int height = 0 ); v i r t u a l ~ S p a t i a l I c o n C o n t c n t ( ) ; double *da ta ; bool Rcndcr lcon (double * b u f _ p t r ) ; bool A l l o c a t c D a t a ( i n t s i z e ) ; double GctData ( i n t i n d e x ) ; bool Sc tData ( i n t index , double v a l u e ) ; } ; c lass An imated lconContent : p u b l i c H a p t i c l c o n C o n t e n t { p u b l i c : An imated lconContent ( int height = 0, int frames = 1 ); v i r t u a l " S p a t i a l l c o n C o n t c n t () ; double ** data ; bool R c n d c r l c o n (double * b u f . p t r ) ; bool A l l o c a t c D a t a ( i n t s i z e ) ; Appendix D. Browser Prototype Code 231 double GctData ( i n t i n d e x ) ; / / index = frame* height bool SetData ( i n t index , double va lue) ; i n t numFramcs ; int c u r r e n t F r a m c ; s t r u c t t i m c v a l i n i t T i m c S t a m p ; int fp s ; }; c lass H a p t i c l c o n { p u b l i c : H a p t i c l c o n ( const char * f i lename ); v i r t u a l ~ H a p t i c I c o n ( ) ; int at ; / / Y p o s i t i o n of the icon int h e i g h t ; / / extent of the icon in Y taxe l s H a p t i c I c o n C o n t c n t * content ; H a p t i c l c o n *ncxt ; v i r t u a l bool RcndcrlconToMap (HapticPageMap *map_ptr ) ; / / Wri te th i s hapt i c icon to the page map "at" i t s l o c a t i o n c lass HapticPageMap { p u b l i c : HapticPageMap (const char * f i l e n a m e ) ; v i r t u a l " HapticPageMap ( ) ; int CurrcntMapPos (const f l oa t s l i d c r . p o s ) ; / / update with the s l i d e r p o s i t i o n and re turn the current p o s i t i o n f l oa t C o m p u t c D c t c n t P o s F r o m V c l o c i t y ( ) ; / / r e turn the new detent p o s i t i o n based on the current v e l o c i t y . bool Computclmagc ( s t r c s s d : : V o l t l m g *outputImg , int map.pos) ; void R e n d e r A H I c o n s ( v o i d ) ; / / pre —render a l l icons to the page map, mostly use fu l for t e s t i n g p u rposes H a p t i c l c o n * i co n s ; double* voltmap ; / / in vol ts , temporary while we only support s p a t i a l icons int h e i g h t ; / / s ize of the map array ( in t a x e l u n i t s ) p r i v a t e : / / v e l o c i t y model s t u f f bool s i m p l e C o n t r o l M o d c ; / / c o n f i g u r a t i o n parameter; set to TRUE i f you want s imple p o s i t i o n c o n t r o l cnum V C o n t r o l S t a t c s { p C o n t r o l , v C o n t r o l W a i t i n g , v C o n t r o l A c t i v c , v C o n t r o l F i n i s h e d }; V C o ntro lS tatcs v Co n t r o 1St ate ; s t r u c t t i m c v a l v Co n t r ol S t ateT i m eSt am p ; / / time entered c u r r e n t v C o n t r o l S t a t c / / except ion : always zero when in p C o n t r o l int c u r r c n t V c 1 o c i ty ; / / UNITS: map uni t s per second ; up is negative , down is p o s i t i v e f l oa t l a s t D c t c n t P o s ; int p O f f s c t ; s t r u c t t i m c v a l 1 as tTimcStamp ; s t r u c t t i m c v a l s tar tTimeStamp ; Appendix D. Browser Prototype Code 232 }; f l oa t lookahcad [8] ; # d c f i n c LOOKAHEAD-NOVALUE -1000 f l oa t lookback [ 8 ] ; int CropPos ( i n t i n p u t P o s ) ; / / he lper f u n c t i o n to crop the p o s i t i o n to the bounds of the map f l oa t CropPos ( f l o a t i n p u t P o s ) ; c la s s S 1 idcr S moot her { / / Used to reduce noise and j i t t e r from the s l i d e r . / / Caches the las t three values of the s l i d e r and averages them. / / If the new s l i d e r p o s i t i o n is c lose to the average , don ' t update . / / Otherwise , r e t u r n the new p o s i t i o n . / / c u r r e n t l y only set up for f l oa t s l i d e r p o s i t i o n s / / probably should be t c m p l a t i z c d in the f u t u r e . p u b l i c : S l i d c r S m o o t h c r () ; v i r t u a l " S l i d c r S m o o t h c r () ; f l oa t Smoot h S 1 idcr Pos ( f l o a t rawpos); p r i v a t e : int c a c h c . s i z c ; f l o a t t h r e s h o l d ; f l oa t * s l i d c r _ c a c h e ; int s t a r t i d x ; #c n d i f D . 1.4 HapticPageMap. epp / / Hapt ic Page Map support f u n c t i o n s / / Ju ly 2006, Joseph Luk ^ i n c l u d e " HapticPageMap . h" / / # d c f i n c DEBUG / / # d c f i n c CVTEST 70 / / constant v e l o c i t y test f lag / / # d c f i n c CLICKONVCONTROL / / create " c l i c k i n g " t a c t i l e feedback for v e l o c i t y c o n t r o l / / # d c f i n c SUBTAXELDOUBLE / / use b i l i n e a r s c a l i n g to reduce image s ize by 1/2 (slows down t a c t i l e flov makes an e f f e c t i v e l y 16 — t a x e l wide d i s p l a y ) # d c f i n c SUBTAXELTRIPLE / / use b i l i n e a r s c a l i n g to reduce image s ize by 1/3 (slows down t a c t i l e flov makes an e f f e c t i v e l y 24—taxel wide d i s p l a y ) / / Time a r i t h m e t i c support f u n c t i o n s (C— s t y l e ) / / r e t u r n the time ( in seconds) s ince t imeStamp, with microsecond p r e c i s i o n / / also puts the current time in cur rent TimeStam p , i f prov ided double T i m c S i n c c (const s t r u c t t i m c v a l *t imcStamp, s t r u c t t i m c v a l * currcntTimcStamp ) { Appendix D. Browser Prototype Code 233 } double t i m e d i f f ; s t r u c t t i m c v a l tern p_tv; s t r u c t t i m c v a l *tv = currcntTimcStam p ? cu rrentTimeStamp : & t c m p . t v ; g c t t i m c o f d a y ( tv , N U L L ) ; / / s u b t r a c t the current time from the cached time double u s c c d i f f ; t i m c d i f f = tv—>tv_scc — timeStamp—> t v _s c c ; if ( 0-0 > ( u s c c d i f f = (double ) (tv—>tv_uscc - t imeSt am p—> t v _u s c c ) / 1000000.0 ) ) { u s c c d i f f = 1.0 -f~ u s e c d i f f ; t i m c d i f f ; } t i m c d i f f += u s c c d i f f ; r e t u r n ( t i m c d i f f ) ; / / r e t u r n s m i l l i s e c o n d s tvl—tv2 double T i m c D i f f (const s t r u c t t i m c v a l * t v l , const s t r u c t t i m c v a l *tv2) { double u s c c d i f f ; double t i m c d i f f ; t i m c d i f f = t v l —> t v . s c c — tv2 —> t v _ s c c ; if ( 0.0 > ( u s c c d i f f = (doublc ) ( tv l—>tv_uscc - tv2 ->tv_usec ) / 1000000.0 ) ) { u s c c d i f f = 1.0 -f u s c c d i f f ; t i m c d i f f ; } t i m c d i f f += u s c c d i f f ; } re turn ( t i m c d i f f ) ; / / stamp the time into a t i m c v a l s t r u c t void TimeStamp ( s t r u c t t i m c v a l *timcStamp) { s t r u c t t i m c v a l tv ; g c t t i m c o f d a y ( &tv , N U L L ) ; timeStamp —>tv.sec ~ tv . t v . s c c ; timcS tamp—> t v . u s c c = tv . t v . u s c c ; } H a p t i c l c o n :: H a p t i c l c o n ( const char * f i lename — NULL ) { height = 0; at = 0; if ( f i l e n a m e ) { Rawlcon P arscr * p a r s c r = new Rawlcon Parser ( f i lename , t h i s ) ; de lete p a r s e r ; i f ( c o n t e n t ) { con ten t — >name = new ch a r [ s t r 1 c n ( f i 1 e n a m c ) ] ; s t r c p y ( content— >namc , f i l e n a m e ) ; } } else content = NULL; > H a p t i c l c o n : : ~ H a p t i c l c o n () { i f ( c o n t e n t ) de le te content ; } bool H a p t i c l c o n :: RcndcrlconToMap ( HapticPageMap *map_ptr) { / / i f ( ! c o n t e n t ) r e t u r n f a l s e ; / / no i c o n ! Appendix D. Browser Prototype Code 234 assert ( c o n t e n t ) ; assert ( m a p . p t r ) ; if (at > map.pt r— > h c igh t ) r e t u r n f a l s e ; int bottom = at 4- height ; i f ( bottom > ( map.ptr—>hcight ) ) { # i f d e f DEBUG / / cout « " C r o p p i n g hapt i c icon of type " « con ten t —>ty pe « « at « " height " « height « c n d l ; #end i f / / needs cro pp i ng bool r c t v a l ; int d i f f = map.ptr—>hcight — at; / / render into a temp buffer f i r s t and then copy to the page map double *buf = new double [ this —>hcight ] ; if ( r e t v a l = t h i s —> co n t e n t —> Rc n de r Ico n ( b u f) ) { for ( i n t i = 0 ; K d i f f ; i++) { / / assert , remove l a t e r i f necessary assert ( at + i < map.ptr — >hcight ) ; } map.ptr—>voltmap [ at+i ] = b u f [ i ] ; } dele te [] buf; r e t u r n r c t v a l ; } else r e t u r n ( t h i s —>con ten t —>Rcndcr Icon ( ( map.ptr—>voltmap ) + a t ) ) ; S p a t i a l I c o n C o n t c n t : : S p a t i a l I c o n C o n t e n t ( int height { type = new c h a r [ 1 2 ] ; s t r c p y ( t y p e , " S p a t i a l l c o n " ) ; if ( h e i g h t ) data = new dou blc [ h c i g h t ] ; e lse data = NULL; } name = NULL; i c o n p a r e n t = NULL; S p a t i a l l c o n C o n t e n t : : ~ S p a t i a l I c o n C o n t c n t ( ) { # i f d e f DEBUG p r i n t f ( " S p a t i a l l c o n C o n t e n t D c s t r u c t o r \ n " ) ; # c n d i f de l e t e [ ] da ta ; de le te [] type; de le te [ ] name ; } bool S p a t i a l l c o n C o n t e n t :: Rcnder lcon (double * b u f _ p t r ) { assert ( i c o n p a r c n t ) ; for ( i n t i = 0 ; i< i c o n p a r c n t - > h e i g h t ; i ++) b u f . p t r { i ] = data [ i ] ; r e t u r n t r u e ; } bool S p a t i a l l c o n C o n t e n t :: A l l o c a t e D a t a ( i n t s i z e ) { data = new double [ s ize ] ; r e t u r n t r u e ; } double S p a t i a l l c o n C o n t e n t ; : GctData ( i n t index) { re turn data [ index ] ; } Appendix D. Browser Prototype Code 235 bool S p a t i a l l c o n C o n t c n t : : Sc tData ( i n t i n d e x , double va lue) { assert ( d a t a ) ; d a t a [ i n d e x ] = va lue ; re turn t r u e ; } A n i m a t e d l c o n C o n t e n t : : Animated lconContent ( int h e i g h t , int frames ) { type = new c h ar [ 1 2 ] ; s t r c p y ( t y p e , " Animatcd lcon " ) ; numFramcs = frames ; if ( h e i g h t ) data = new double [ height * frames ] ; e lse data = NULL; name — NULL; i c o n p a r c n t = NULL; c u r r c n t F r a m c = 0; fps = 30; / / d e f a u l t value g c t t i m c o f d a y (&; i n i t T i mcStamc ) ; } Animated lconContent : : ~ Animated lconContent () { # i f d c f DEBUG p r i n t f (" An imated lconContent D e s t r u c t o r \ n " ) ; # c n d i f de le te [] da ta ; de le te [] type; de le te [ ] name ; } bool An imated lconContent : : R c n d c r l c o n (double * b u f _ p t r ) { assert ( i c o n p a r e n t ) ; c u r r c n t F r a m c = (fps / T i me S i n c c ( in i t T i me S tarn p ) ) % numFramcs; for ( i n t i =0; i< i co n p ar e n t —> h c i g h t ; i + + ) b u f . p t r [ i ] — data [ i - f - (he ight*currentFramc) ] ; r e turn t r u e ; } bool A n i m a t e d l c o n C o n t e n t : : A l l o c a t c D a t a ( i n t s i z e ) { data = new double [ s i z e ] ; re turn t r u e ; } double An imated lconContent : : GctData ( int index) { r e t u r n d a t a [ i n d e x ] ; } bool A n i m a t e d l c o n C o n t e n t : : SctData ( i n t in d ex , double va lue) { assert ( d a t a ) ; d a t a [ i n d e x ] = v a l u e ; re turn t r u e ; } HapticPageMap :: HapticPageMap (const char * f i 1 e n a m c = NULL) { icons = NULL; if ( f i l e n a m e ) { PagcM ap Parser * p a r s c r = new PagcMapParser ( f i l e n a m e , t h i s ) ; de le te p a r s e r ; } Appendix D. Browser Prototype Code 236 else height =100; / / d e f a u l t vo l t map height / / in it the array with base value vo 11 m ap = new double [ height ] ; for ( i n t i=0; i < h e i g h t ; i+ + ) voltmap [ i ] = —50.0; 1 a s t D e t c n t P o s = 0; las tTimcSt am p . t v ~s e c = 0; las tTimcStamp . t v . u s e c = 0; c u r r c n t V c l o c i t y = 0; v C o n t r o l S t a t c = p C o n t r o l ; v C o n t i o l S t a t c T i m c S t a m p . t v . s c c = 0; v C o n t r o l S t a t c T i m e S t a m p . t v . u s c c = 0; for ( i n t i=0; i<8; i++) lookahcad [ i ] = LOOKAHEADJJOVALUE; for ( i n t i=0; i<8; i ++) l o o k b a c k [ i ] = 0; si m p lcCon trol M ode = f a l s e ; TimcStamp (&st a r t T i meSt am p ) ; } HapticPageMap : : " HapticPageMap () { dele te [] vo l tmap; H a p t i c l c o n * i c o n = i c o n s ; H a p t i c l c o n * n e x t i c o n ; whi le ( i c o n ) { noxt icon = icon—>ncxt; de le te i c o n ; icon = n e x t i c o n ; } } { int H a p t i c P £ v g c M a p : : Cu r ren t M ap Pos ( f loa t s l i d c r . p o s ) / / r e turn the cursor p o s i t i o n in the hapt ic map / / i f using v e l o c i t y mode, may depend on prev ious v a l u e s ! / / s l i d c r . p o s should be expressed as f l oa t from 0.0 to 100.0 /* l—i—l-s l i d c r . p o s map s ize >| |< range >| l< >l d i s p l a y window */ i f ( fa l se ) { i f ( T i else i e lse i e lse i e lse i e lse i e lse i e lse i else i else i / / automated movement program for demo movie ncSince(&startTimcStamp ) ( T i m e S i n c c ( & s t a r t T i m c S t (TimcSince(&cstartTimcSt ( T i m e S i n c c ( & s t a r t T i m c S t (TimeSince(&:startTimcSt (TimeSince(&:startTimeSt (TimeSincc(&sstartTimcSt ( T i m e S i n c c ( & s t a r t T i m c S t ( T i m e S i n c c ( & s t a r t T i m c S t ( T i m c S i n c e f & s t a r t T i m c S t < 0.1) amp ) < amp ) < amp ) < amp ) < amp ) < amp ) < amp ) < amp ) < amp ) < l a s t D e t c n t P o s 0) 5) 0 ) 6 ) 0) 6 ) .0) . 6 ) c u r r c n t V c l o c c u r r c n t V c l o c c u r r c n t V c l o c c u r r c n t V c l o c c u r r c n t V c l o c c u r r c n t V c l o c C u r r c n t V c l o c c u r r c n t V c l o c ) c u r r c n t V c l i ty = i ty = i ty = i ty = i ty = i ty = i ty = i ty = o c i t y 6 2 ; 50; 0; - 17; 0 ; 17; 0; -17; 0; Appendix D. Browser Prototype Code 237 else if ( T i m e S i n c e ( & s t a r t T i m e S t a m p ) < 11.6 ) c u r r c n t V c l o c i t y = 17; else i f ( T i m c S i n c e ( & s t a r t T i m e S t a m p ) < 12.0 ) c u r r c n t V c l o c i t y = 0; else i f (T imcSincc(&:s tar tTimcStamp ) < 13.0 ) c u r r c n t V c l o c i t y = —50; else i f ( T i m c S i n c c ( & s t a r t T i m e S t a m p ) < 14.9 ) c u r r c n t V c l o c i t y = -80; else i f ( T i m c S i n c c ( & s t a r t T i m c S t a m p ) < 17 ) TimeStamp(&;startTimeStamp ) ; ComputcDetent PosFrom Veloci ty ( ) ; r e t u r n (CropPos (( i n t ) round ( l a s t D e t e n t P o s ) ) ) ; if ( s i m p l c C o n t r o l M o d c ) { f l o a t d isp lay _w indo w = 8.0 / ( f 1 o a t ) h e i g h t ; / / d i sp lay-window is the percentage of the page " v i s i b l e " through the / / t a c t i l e d i s p l a y . int range = height — 16; / / range is the p o r t i o n of the map where the centre of the s l i d e r / / may be p laced int w i n d o w . s t a r t = ( i n t ) ((100.0 — s l i d e r - p o s ) / 1 0 0 . 0 * ( f l oa t ) range ) ; / / s t a r t index in the hapt i c page map l a s t D e t e n t P o s = ( f l oa t ) w indow.s tar t ; r e t u r n ( w indow.s tar t ) ; } else / / v e l o c i t y c o n t r o l { f l oa t n o r m S l i d c r P o s = ( 1.0 - ( s l i d c r . p o s / 50.0) ); / / normal i zed to between —1.0 and +1.0 f l oa t c u r r c n t A c c c l c r a t i o n ; / / A few convenient macros / / # d c f i n c SIMPLEVELOCITY # d c f i n c VELOCITYATSTOPS / / cons tants # i f def ined VELOCITY ATSTOPS int maxP = 50; f l oa t maxV = 200.0; f l o a t - p o s C o n t r o l Z o n e = 0.75; / / Use th i s many percent of s l i d e r area for pos c o n t r o l zone f l oa t t imcB cforc V C o n t r o l = 0.05; / / Number of seconds before v e l o c i t y c o n t r o l a c t i v a t e s f l oa t v Co nt r o 1 A ccc 1 cr at io n = 1; / / Number of seconds before v e l o c i t y reaches maxV af ter s t a r t i n g V—control f l o a t t imc A ft cr V C o n tro l = 0.05; / / Number of seconds af ter d i sengag ing v e l o c i t y c o n t r o l , dur ing which time user input is ignored so that they can r e t u r n the s l i d e r to centre # e l i f de f ined SIMPLEVELOCITY int maxP = 10; f l oa t maxV = 3 0 0.0; f l oa t p o s C o n t r o l Z o n e = 0.20; / / Use th i s many percent of s l i d e r area for pos c o n t r o l zone f l oa t t i m c B c f o r c V C o n t r o l = 0.0; / / Number of seconds before v e l o c i t y c o n t r o l a c t i v a t e s f l oa t v Co nt r ol A ccc l c r a t i o n = 0.2; / / Number of seconds before v e l o c i t y reaches maxV af ter s t a r t i n g V— c o n t r o l f l oa t t i m e A f t e r V C o n t r o l — 0.01; / / Number of seconds a f ter d i s engag ing v e l o c i t y c o n t r o l , dur ing which time user input is ignored so that they can r e t u r n the s l i d e r to centre t^e 1 s e i n t maxP = 5 0; f l oa t maxV = 100.0; f l oa t p o s C o n t r o l Z o n e = 0.95; / / Use t h i s many percent of s l i d e r area for pos c o n t r o l zone f l o a t t im eB cfo re V Co ntro 1 — 0 .01; / / Number of seconds before v e l o c i t y c o n t r o l a c t i v a t e s f l o a t v C o n t r o l A c c e l e r a t i o n — 1; / / Number of seconds before v e l o c i t y reaches maxV af ter s t a r t i n g V— c o n t r o l f l oa t t i m e A f t e r V C o n t r o l = 0.01; / / Number of seconds af ter d i sengag ing v e l o c i t y c o n t r o l , d u r i n g which time user input is ignored so that they can r e t u r n the s l i d e r to centre Appendix D. Browser Prototype Code 238 # c n d i f if ( fabs ( n o r m S l i d c r P o s ) < p o s C o n t r o l Z o n c ) { / / P o s i t i o n C o n t r o l Zone c u r r c n t V c 1 o c i t y — 0 ; switch ( v C o n t r o l S t a t e ) { case v C o n t r o l W a i t i n g : / / abort w a i t i n g for v C o n t r o l , r e turn to p C o n t r o l v C o n t r o l S t a t e — p C o n t r o l ; / / cont inue to p C o n t r o l case p C o n t r o l : pOffset = ( i n t )( round ( n o r m S l i d c r P o s / p os Co n t r o 1Z o n e * ( f l o a t ) maxP ) ) ; break ; case v C o n t r o l A c t i v e : / / enter v C o n t r ol F i n ished ; s t a r t t imer v C o n t r o l S t a t e = v C o n t r o 1Fin i shed ; TimeStamp ( & v C o n t r o l S t a t e T i m e S t a m p ) ; break ; case v C o n t r o l F i n i s h c d : / / check t i m e r ; i f passed , then enter p C o n t r o l i f ( T imeS i nee (& v C o n t r o l S t a t e T i m e S t a m p ) > t i m c A f t e r V C o n t r o l ) { / / re —zero detent p o s i t i o n to avoid moving the h i g h l i g h t int newPOffsct = ( int )( round ( nor mSl i der Pos / posContro lZone * ( f 1 o a t ) maxP ) ) ; l a s t D c t c n t P o s += pOffset — newPOffset; pOffset = newPOffset; v C o n t r o l S t a t e = p C o n t r o l ; } break ; } / / switch ( v C o n t r o l S t a t e ) } else { / / V e l o c i t y C o n t r o l Zone switch ( v C o n t r o l S t a t e ) { case p C o n t r o l : case v C o n t r o l F i n i s h c d : / / enter w a i t i n g s t a t e , s t a r t t imer v C o n t r o l S t a t e = v C o n t r o l W a i t i n g ; TimeStamp ( ^ v C o n t r o l S t a t e T i m e S t a m p ) ; pOffset = ( i n t )( round ( norm Slider Pos / p o s C o n t r o l Z o n e * ( f l o a t ) maxP) ) ; break ; case v C o n t r o l W a i t i n g : / / check to sec i f the t imer has exceeded the t i m cBcfo r c V Co n t ro1 , i f so , go to next s ta te pOffset — ( i n t )( round ( nor mSl idcr Pos / p o s C o n t r o l Z o n e * ( f l o a t ) maxP ) ) ; i f ( T i m c S i n c c ( & v C o n t r o l S t a t e T i m e S t a m p ) > t i m e B e f o r c V C o n t r o l ) { TimeStamp (&cvControlStateTimeStamp ) ; v C o n t r o l S t a t e = v C o n t r o l A c t i v e ; } break ; case v C o n t r o l A c t i v e : i f ( n o r m S l i d c r P o s > 0 ) { pOffset — maxP ; # i f d c f CVTEST c u r r c n t V c l o c i t y = CVTEST; ^ e l s e c u r r c n t V c l o c i t y = ( int } (round (( n o r m S l i d c r P o s p o s C o n t r o l Z o n e ) / ( 1 . 0 — p o s C o n t r o l Z o n e ) * maxV) ) ; # end i f Appendix D. Browser Prototype Code 239 else { > pOffsc t = —maxP ; # i f d c f CVTEST c u r r c n t V c l o c i t y = - C V T E S T ; #c 1 s c c u r r c n t V c l o c i t y = ( i n t )( round (( nor m S l idcr Pos + pos C o n t r o l Zone ) / (1.0 — posCo n t ro lZone ) * maxV) ) ; # c n d i f / / add a c c e l e r a t i o n f a c t o r c u r r e n t A c c e l e r a t i o n = ( T i m c S i n c c ( & v C o n t r o l S t a t c T i m c S t a m p ) ) / ( v C o n t r o I A c c e l e r a t i o n ) ; i f ( c u r r e n t A c c e l e r a t i o n < 1.0) c u r r e n t V e l o c i t y = ( int ) (round ) ( ( f l o a t ) c u r r e n t V c l o c i t y * c u r r e n t A c c e l c r a t i o n ) ; break ; } / / switch ( v C o n t r o l S t a t c ) } / / v e l o c i t y c o n t r o l zone i f ( p O f f s c t > maxP) pOffset = maxP ; e lse if ( p O f f s c t < —maxP) pOffset = —maxP; C o m p u t c D c t c n t P o s F r o m V e l o c i t y () ; # i f d c f DEBUG p r i n t f ("CurrcntMapPos : v e l o c i t y C o n t r o l p O f f se t=%d , c u r r c n t V c l o c i t y=% d , detent Pos=%f , mapPos=%d\n" , pOffsct , c u r r e n t V e l o c i t y , l a s t D c t c n t P o s , CropPos (( int ) round ( l a s t D c t c n t P o s )-fpOffsct ) ) ; # c n d i f # i f d c f CVTEST if ( l a s t D c t c n t P o s >= height - 8) l a s t D c t c n t P o s = 0.0; #end i f r e t u r n (CropPos ( ( in t ) round ( l a s t D c t c n t P o s ) + p O f f s c t ) ) ; f l o a t HapticPageMap ; : C o m p u t c D c t c n t P o s F r o m V e l o c i t y ( ) { / / r e turn the cursor p o s i t i o n in the hapt ic map if ( s i m p l c C o n t r o l M o d c ) r e t u r n l a s t D c t c n t P o s ; else { / / compute the current p o s i t i o n based on the las t heading f l oa t ncwDctcntPos ; double t i m c d i f f ; s t r u c t t i m c v a l t v ; ge t t imcofday ( &tv , N U L L ) ; / / s u b t r a c t the current time from the cached time double u s c c d i f f ; t i m c d i f f = tv . t v . s c c — last T imcS tamp . t v _sc c ; if ( 0.0 > ( u s c c d i f f = ( double ) ( t v . t v .u sec — las tT imcStamp . t v .u sec ) / 1000000.0 ) ) { u s c c d i f f = 1.0 u s c c d i f f ; t i m c d i f f ; } t i m c d i f f += u s c c d i f f ; las tTimcStamp , t v . s c c = t v . t v . s e c ; las tTimcStamp . t v . u s c c = tv . t v . u s e c ; / / i f t i m c d i f f is s p e c i f i e d Appendix D. Browser Prototype Code 240 // { / / UNTESTED ! ! ! / / l a s tT imcStamp . t v . s c c ( t i m e - t ) t r u n c ( t i m c d i f f ) ; / / l a s tTimcStamp . t v . u s c c += ( s uscconds_t ) ( ( modf ( t i m cd i f f , NULL) ) * 1000000 ) ; / / i f ( l a s tTimcStamp . t v . u s c c >= 1000000) // { / / l a s tTimcStamp . t v . u s c c - = 1000000; / / last T i m c S tarn p . t v _s ec H—h; // ncwDctcntPos = l a s t D c t c n t P o s + t i m c d i f f * ( f l o a t ) c u r r c n t V c l o c i t y ; ncwDctcntPos = CropPos ( ncwDetentPos ) ; l a s t D c t c n t P o s = ncwDctcntPos; r e t u r n ncwDctcntPos; } } int HapticPageMap CropPos ( i n t i n p u t P o s ) { if ( inputPos < 0) r c t u r n ( O ) ; if ( i n p u t P o s > height—8) r c t u r n ( h c i g h t — 8) ; re turn i n p u t P o s ; } f l oa t HapticPageMap :: CropPos ( f l o a t i n p u t P o s ) { i f ( i n p u t P o s < 0.0) r e t u r n ( 0 . 0 ) ; # i f def ined SUBTAXELTRIPLE if ( i n p u t P o s > ( f l oa t )( h e i g h t - 8 ) ) r o t u r n (( f 1 o a t ) ( ho igh t - 24) ) ; # o l i f def ined SUBTAXELDOUBLE i f ( i n p u t P o s > ( f lo a t ) ( heigh t -8) ) r e t u r n ( ( f 1 o a t ) ( heigh t — 16) ) ; #c 1 s c if ( i n p u t P o s > ( f lo at ) ( height -8) ) ret u rn ( ( f 1 o a t ) ( heigh t - 8 ) ) ; # c n d i f re turn i n p u t P o s ; bool HapticPageMap : : Computclmagc ( s t r c s s d : : V o l t l m g *outputImg , int map.pos) { / / compute the t a c t i l e output image at cursor p o s i t i o n map.pos / / and place the r e s u l t in output lmg / / r e t u rn s true i f e v e r y t h i n g was dandy; fa l se o therwise / / uncommcnt one block of code below / / crop to map bounds map.pos = CropPos (map.pos) ; / / or assert map bounds / / assert (map.pos >= 0 && map.pos < height —8) ; / / or r e t u r n fa l se i f out s ide of map bounds / / i f (map.pos < 0 | | map.pos > height—8) r e t u r n f a l s e ; double t i m c d i f f ; f loa t v a l u c A t ; r e g i s t e r int index; H a p t i c l c o n * i c o n ; icon = i c o n s ; while ( i c o n ) { / / render only the area t h a t ' s " v i s i b l e " # i f def ined SUBTAXELTRIPLE const int v i s i b l c a r c a = 2 4; # c l i f def ined SUBTAXELDOUBLE const int v i s i b l c a r c a = 16; Appendix D. Browser Prototype Code 241 #c l sc const int v i s i b l c a r c a — 8; # c n d i f if ( (icon—>at <— m a p . p o s + v i s i b l c a r c a ) ( icon — >at-ficon —>hcight >= map_pos) ) { icon —>RenderIconToMap ( t h i s ) ; } icon = icon —> next; } / / copy data from the page map into the output lmg for ( i n t x=0; x<8; x++) { # i f def ined SUBTAXELTRIPLE index = x*3 4- map-pos ; v a l u c A t = ( voltmap [ index ] + voltmap [ index+1] + vol t m ap [ i n d e x + 2] ) / 3.0; # c l i f def ined SUBTAXELDOUBLB index = x*2 -f- map.pos; v a l u c A t = ( voltmap [ index ] -f vol t map [ i n dcx + 1 ]) / 2; #o lsc v a l u c A t = voltmap [ x + map.pos ] ; #ond i f / / Do the lookahcad and lookback smoothing if ( lookahcad [x] != LOOKAHEADJMOVALUE) v a l u c A t = ( v a l u c A t + lookahcad [ x ] ) / 2.0; if ( fabs ( v a l u c A t — lookback [x] } > 50.0) / / t h r e s h o l d value { lookahcad[x] = v a l u c A t ; v a l u c A t = ( v a l u c A t -f lookback [x] ) / 2.0; } else lookahcad fx] = LOOKAHEAD-NOVALUE; l ookback[x ] = v a l u c A t ; / / o p t i o n a l l y , prov ide t a c t i l e feedback of t r a n s i t i o n between v e l o c i t y c o n t r o l and p o s i t i o n c o n t r o l modes # i f d c f CLICKONVCONTROL i f ( v C o n t r o l S t a t e = v C o n t r o l W a i t i n g ) { t i m c d i f f = T i m c S i n c c ( & v C o n t r o l S t a t c T i m c S t a m p ) ; i f ( t i m c d i f f < 0.005) v a l u c A t += 40; else if ( t i m c d i f f < 0.01) v a l u c A t - = 40; if ( v a l u c A t < —50) v a l u c A t = —50; else if ( v a l u c A t > 50) v a l u c A t = 50; } # e n d i f outputlmg —> S c t (7 — x , 0, v a l u e A t ) ; } r e t u r n t r u e ; } void HapticPageMap : : R c n d c r A l l I c o n s ( v o i d ) { H a p t i c l c o n * icon ; icon = i c o n s ; while ( i c o n ) { icon —> Ren der I con T o Map ( t h i s ) ; icon — icon —> next; } Appendix D. Browser Prototype Code 242 S l i d c r S m o o t h c r : : S l i d c r S m o o t h c r () { c a c h e . s i z c = 10; t h r e s h o l d = 5.0; / / when s l i d e r is moved a large d i s tance (above th i s t h r e s h o l d ) , / / abandon averag ing and jus t jump to the new s l i d e r p o s i t i o n s l i d e r . c a c h c = new f l oa t [ c a c h e . s i z c ] ; for ( i n t i =0; i < c a c h c _ s i z c ; iH—h) s l i d e r . c a c h c [ i ] = 0.0; s t a r t i d x =0; S l i d c r S m o o t h c r : : ~ S l i d c r S m o o t h c r () de le te [] s l i d e r . c a c h c ; l oa t S l i d e r S m o o t h c r :: SmoothS l idcrPos ( f l o a t rawpos) / / Takes the current s l i d e r pos and re turns the "smoothed" value of the s l i d e r pos / / Uses s imple averag ing of the last ( c a c h e . s i z c -1 ) p o s ' s , and the current rawpos f l oa t avgpos = 0.0; f l oa t r c t v a l ; int i ; / / add the value to the a r r a y , c l o b b e r i n g the o ldes t value s t a r t i d x + + ; s t a r t i d x %= c a c h e . s i z c ; s l i d e r _ c a c h c [ s t a r t i d x ] = rawpos; for ( i =0; i < c a c h c _s i z c ; i 4-+) avgpos += s l i d e r . c a c h c [ i ] ; avgpos /= c a c h e . s i z c ; if ( fabs(rawpos — avgpos) >= t h r e s h o l d ) { r c t v a l = rawpos; for ( i=0; i < c a c h c _ s i z c ; iH—(-) s l i d e r . c a c h c [ i ] = rawpos; } else r c t v a l = avgpos; r e t u r n ( r c t v a l ) ; D . 1.5 B r o w s e r S h a r e d , h # i f n d c f BROWSERSH ARED_H #def ine BROWSERSHARED-H # i n c l u d c " acc /Mutcx . h" # i n c l u d c " HapticPageMap . h" ^ i n c l u d e " BrowscrXMLBit s . h" c lass BrowscrSharcd { p u b l i c : BrowscrSharcd () ; Appendix D. Browser Prototype Code 243 int GctMapPos ( ) ; void Sc tMapPos ( in t pos ) ; int G c t C l i c k s ( ) ; vo id S c t C l i c k s ( i n t c l i c k _ c o u n t _ p a r a m ) ; bool GctPageMapRcady ( ) ; void SctPagcMapRcady (bool r e a d y . f l a g ) ; / / PUBLIC hapt ic page map s t r u c t u r e / / Rules : before r e a d i n g , check if GctPageMapRcady () is true or not / / before w r i t i n g , SctPagcMapRcady ( f a l s e ) HapticPageMap *pagcmap ; p r i v a t e : int c l i c k . c o u n t ; i n t map.pos ; bool pagcmap.ready ; ACE.Mutcx mutcx ; }; # c n d i f D . l . 6 BrowserShared.cpp / / B ro wscrSh ared . epp ^ i n c l u d e " BrowserSharcd . h" B rowscr Shared : : BrowserSharcd () { c l i c k . c o u n t = 0; map.pos — 0 ; pagcmap.ready = 0; pagemap = NULL; } int BrowserSharcd : : GctMapPos () { mutcx. acqu ire () ; int r c t v a l — map.pos ; mutcx. re l ease () ; r e t u r n r c t v a l ; } void BrowserSharcd : : SctMapPos ( int pos) { mutcx . a c q u i r e () ; map.pos = pos ; mutcx. re l ease () ; r e turn ; } int BrowserSharcd : : G e t C l i c k s () { mutcx. acqu ire () ; int r c t v a l = c l i c k . c o u n t ; mutcx. re lease () ; r e turn r c t v a l ; } void B r o w s c r S h a r e d : : S c t C l i c k s ( i n t c l i c k . c o u n t . p a r a m ) { mutcx. a c q u i r e () ; c l i c k - c o u n t = c l i c k . c o u n t . p a r a m ; Appendix D. Browser Prototype Code 244 mutex. re l ease () ; r e turn ; } bool BrowscrSharcd : : Get Page Map Ready ( ) { mutex . a c q u i r e () ; int r c t v a l = pagcmap.rcady ; mutex. re l ease () ; r e t u r n r e t v a l ; } void BrowscrSharcd : : Set P age M ap Ready (bool r e a d y - f l a g ) { mutex. acqu ire () ; pagcmap.rcady = r e a d y - f l a g ; mutex. re lease () ; r e turn ; > D . l . 7 B r o w s e r X M L B i t s . h / / Browser XML Bits , modi f ied from o r i g i n a l f i l e by Vincent Levesque (GPL) / / Joseph Luk , July 2006 # i f n d c f BROWSERXMLBITS_H # d c f i n c BROWSERXMLBITS_H ^ i n c l u d e < s t r i n g > ^ i n c l u d e < v c c t o r > ^ i n c l u d e "ACEXML/common/ Defaul t Handler .h" c lass PagcMapParscr : p u b l i c A C E X M L - D c f a u l t H a n d l e r { p u b l i c : P agcM ap Parser ( s t d : : s t r i n g f i l e , HapticPageMap* d a t a ) ; v i r t u a l ~ PagcMapParscr ( ) ; vo id P a r s c F i t c ( s t d : : s t r i n g f i l e N a m c ) ; v i r t u a l void c h a r a c t e r s (const ACEXML_Char * c h , int s t a r t , int length ACEXML_ENV_ARG_DECL) ACE.THROW.SPEC (( A C E X M L - S A X E x c c p t i o n ) ) ; v i r t u a l void cndDocumcnt (ACEXML-ENV_SINGLEJVRGX>ECL) ACE_THROW_SPEC((ACEXML-SAXExccpt ion) ) ; v i r t u a l void cndElcmcnt (const ACEXML-Char *namcspaccURI , const ACEXML-Char * l o c a l N a m c , const ACEXML-Char *qNamc ACEXML_ENV_ARG_DECL) ACE-THROW_SPEC(( ACEXML-SAXExccpt ion ) ) ; v i r t u a l void set D oc u me n t Lo cat o r ( A C E X M L - L o c a t o r * l o c a t o r ) ACE_THROWJ3PEC((ACEXML_SAXException) ) ; v i r t u a l void star tDocu men t (ACEXML_ENV_SINGLE_ARG_DECL) ACE_THROW_SPEC((ACEXML_SAXExccption) ) ; v i r t u a l void s t a r t E l c m e n t (const ACEXML-Char * namcspaccU Rl , const ACEXML-Char *localName , const ACEXML_Char *qNamc, A C E X M L - A t t r i b u t c s * at ts A C E X M L_ENV_ARG_D ECL) ACE_THROW_SPEC(( A C E X M L . S A X E x c c p t i o n ) ) ; p r i v a t e : A C E X M L - L o c a t o r * l o c a t o r . ; , s t d : : v c c t o r < s t d : : s t r i n g > t a g s . ; HapticPageMap * data ; int c u r r A t ; }; Appendix D. Browser Prototype Code 245 c lass RawIconParscr : p u b l i c A C E X M L.Dcfault H and lcr { p u b l i c : RawIconParscr f std :: s t r i n g f i l e , H a p t i c l c o n * i c o n p t r ) ; v i r t u a l ~ Ra wlco n P ar sc r () ; vo id P a r s e F i l c ( s t d :: s t r i n g f i l e N a m c ) ; v i r t u a l void c h a r a c t e r s (const ACEXML.Char * c h , int s t a r t , int length ACEXML_ENV_ARG_DECL) ACE_THROW_SPEC((ACEXML_SAXException) ) ; v i r t u a l void end Document (ACEXML-ENV_SINGLE-ARG-DECL) ACE_THROW.SPEC((ACEXML_SAXExcept ion) ) ; v i r t u a l void cndElcmcnt (const ACEXML.Char *namcspaccURI, const A C E X M L . C h a r * localNamc , const ACEXML.Char *qNamc ACEXML.ENV.ARGJDECL) ACE_THROW_SPEC(( A C E X M L . S A X E x c c p t i o n ) ) ; v i r t u a l void set Doc umcntLocator ( A C E X M L . L o c a t o r * l o c a t o r ) ACE_THROW_SPEC( ( A C E X M L . S A X E x c c p t i o n ) ) ; v i r t u a l void startDocurncnt (ACEXML_ENV_SINGLE_ARG_DECL) ACE_THROW_SPEC( ( ACEXML_SAXExccpt ion ) ) ; v i r t u a l void s t a r t E l c m c n t (const ACEXML.Char * namcspaccU R l , const ACEXML.Char • localNamc , const ACEXML.Char *qNamc, A C E X M L . A t t r i b u t c s *a t t s ACEXML_ENV_ARG_DECL) ACE_THROW_SPEC((ACEXML.SAXExcept ion) ) ; p r i v a t e : A C E X M L . L o c a t o r * l o c a t o r . ; std : : v c c t o r < s t d : : s t r i n g > t a g s . ; H a p t i c l c o n * icon ; int c u r r l n d c x ; int currFrame ; }; # c n d i f / /BROWSERXMLBITS-H D . l . 8 B r o w s e r X M L B i t s . c p p / / Browser XML Bits , modi f ied from o r i g i n a l / / Joseph Luk, July 2006 # i n c l u d c < ios trcam> ^ i n c l u d e <sstream> ^ i n c l u d e < a s s e r t . h> ^ i n c l u d e ^ i n c l u d e #i n c l u d c # i n c l u d c #i n c l u d c ^ i n c l u d e # i n c 1 u d c # i n c 1 u d c a c c / A C E . h " a c c / L o g . M s g . h" ACEXML/common/ F i l c C har S t rcam . h" ACEXML/common / HttpCharStream . h" ACEXML/common/StrCharStrcam . h" ACEXML/ p a r s c r / p a r s e r / P a r s c r .h" a c c / G c t . O p t . h " ace / A u t o . P t r . h" # i n c l u d c " HapticPageMap . h" #i n c1u d c " B r o w s c r X M L B i t s . h " f i l e by V i n c e n t Levesque (GPL) int average = —1; void PagcMapParscr : : P a r s c F i l c ( s td : : s t r i n g f i l eNamc) { A C E X M L . F i l o C h a r S t r o a r a t f s tm = now A C E X M L . F i l e C h a r S t r e a m ; Appendix D. Browser Prototype Code 246 i f ( fst m—>opcn ( f i l eNamc . c . s t r () ) != 0) { s t d : : s t r i n g s t r ( " F a i l e d to open XML f i l e : " ) ; s t r A-= f i leNamc ; / /STRESSDJTHROW-EX_DESC( F i l c E x PagcMapParscr : : P a r sc F i 1 c " , st r ) ; / / TODO: throw the c o r r e c t except ion ACEXML_TRY_NEW_ENV { A C E X M L . P a r s c r p a r s e r ; A C E X M L . l n p u t S o u r c c input ( fstm ) ; parser . s c t C o n t c n t H a n d l c r ( th is ) ; parser . s c t F c a t u r c ( " h t t p : / / x m l . o r g / s a x / f e a t u r c s / v a l i d a t i o n " , f a l s e ) ; / / todo p a r s e r , parse (&input ACEXMI^ENVJUIGJARAMETER) ; ACEXML.TRY-CHECK; } ACEXM L_C ATCH ( A C E X M L . S A X E x c o p t i o n . ex) { ACE_UNUSED_ARG(ox ) ; ACE-DEBUG ((LM_ERROR, A C E - T E X T ("Except ion o c c u r r e d . Ex i t i n g . . . \ n " ) ) ) ; cx . p r i n t () ; } ACEXML-ENDTRY; PagcMapParscr c h a r a c t e r s (const ACEXML.Char * c d a t a , int s tart , int end ACEXML_ENV_ARG_DECL J N O T . U S E D ) A C E _ T H R O W . S P E C ( ( A C E X M L . S A X E x c c p t i o n ) ) { int l t a g = t a g s . . s i z e ( ) — 1; / / p r i n t f ( " I t a g : %d\n" , l t a g ) ; i f ( t a g s . . b a c k { ) == "at"){ i f ( t a g s . , at ( l tag— 1) =— "icon") { c u r r A t = a t o i ( c d a t a ) ; # i f d c f DEBUG p r i n t f ("Tag <at> value %d\n" , c u r r A t ) ; #end i f } } else i f ( t ags _ . back ( ) "namc"){ if ( tags_ . at ( ltag—1) == " icon") { # i f d c f DEBUG p r i n t f (" Tag <namc> value %s , c u r r A t = %d\n" , c d a t a , c u r r A t ) ; # e n d i f H a p t i c l c o n *tcmp = data —> icons ; char *pathnamc = new char [ s t r l c n (" icons /") 4* s t r l c n ( c d a t a ) ] ; s t r c p y (pathname , " icons / " ) ; s t r c a t (pathname, c d a t a ) ; d at a—> icons = new H a p t i c l c o n (pathname).; de lete [] pathname; data—>icons —>ncxt = temp; data —> icons —> at = c u r r A t ; c u r r A t = 0; } } else i f ( t ag s _ . back ( ) == "he ight") { int height = a t o i ( c d a t a ) ; # i f d e f DEBUG p r i n t f ("Tag <hc ight> value %d\n" , h e i g h t ) ; #end i f data—>hcight = h e i g h t ; } Appendix D. Browser Prototype Code 247 else { # i f d o f DEBUG s t d : : s t r i n g thetag = t a g s _ . b a c k ( ) ; p r i n t f (" Unknown tag %s \ n" , t h e t a g . c _ s t r ( ) ) ; #end i f } } void P a g c M a p P a r s c r : : cndDocumcnt (ACEXMLJENV_SINGLE-ARG_DECL_NOT_USED) ACE.THROWJ3PEC ( ( A C E X M L . S A X E x c c p t i o n ) ) { } void PagcMapParscr : : cndElcmcnt ( const ACEXML.Char * uri , const ACEXML.Char *name, const ACEXML.Char *qName A C E X M L . E N V_ARG_DECL_NOT_USED) ACE_THROW_SPEC( ( A C E X M L . S A X E x c c p t i o n ) ) { t a g s . . p o p - b a c k ( ) ; } void PagcMapParscr : : sc t Do c u m c n t Loc at o r ( A C E X M L.Locator * l o c a t o r ) ACE_THROW_SPEC(( A C E X M L . S A X E x c c p t i o n ) ) { this —>locator . = l o c a t o r ; } void PagcMapParscr : : s tar tDocumcnt (ACEXML.ENVJ3INGLE_ARG-DECL-NOT.USED) ACE.THROW-SPEC (( A C E X M L . S A X E x c c p t i o n ) ) { } void PagcMapParscr : : s t a r t E l c m c n t ( const ACEXML.Char * u r i , const ACEXML.Char *namc , const ACEXML.Char *qNamc, A C E X M L . A t t r i b u t c s * a 1 i s t ACEXML_ENV_ARG JDECL3IOT.USED) ACE.THROWJ3PEC ( ( A C E X M L . S A X E x c c p t i o n ) ) { t a g s . . push_back(name ) ; } P age M ap P arse r : : P a g e M a p P a r s e r ( s t d : : s t r i n g f i l e , H apt ic PageMap * data) { th is —> data = da ta ; c u r r A t ~ —1; # i f d c f DEBUG p r i n t f ( " PagcMapParser about to c a l l p a r s e f i 1 c \ n " ) ; # c n d i f P a r s c F i l c ( f i l c ) ; } PagcMapParscr : : ~ PageMapParscr( ) { } void R a w I c o n P a r s c r : : P a r s e F i 1 c ( s t d :: s t r i n g f i l eName) { Appendix D. Browser Prototype Code 248 A C E X M L . F i l c C h a r S t r c a m *fstm = new A C E X M L_F ilcCharStrcam ; i f ( fstm —>opcn ( f i l cNamc . c . s t r () ) != 0) { s t d : : s t r i n g s t r ( " F a i l e d to open XML f i l e : " ) ; s t r += f i l cNamc ; / /STRESSD JTHROW-EX_D ESC ( F i l c E x ," Rawlcon Parser : : P a r s c F i l c " , str ) ; } / / TODO: throw the c o r r e c t except ion ACEXML_TRY_NEW_ENV { A C E X M L - P a r s e r p a r s e r ; ACEXML-InputSourcc input ( fstm ) ; p ar s c r . set Con t cn t H an d lc r ( t h i s ) ; p a r s e r . s c t F c a t u r c ( " h t t p : / / x m l . o r g / s a x / f c a t u r c s / v a l i d a t i o n " , f a l s e ) ; / / todo p a r s e r , parse (&input ACEXMUENV-ARG-PARAMETER) ; ACEXML.TRY.CHECK; } ACEXML_CATCH ( A C E X M L - S A X E x c c p t i o n , ox ) { ACE_UNUSED^\RG ( ex ) ; ACE-DEBUG ((LM_ERROR, A C E - T E X T ("Except ion o c c u r r e d . E x i t i n g . . . \ n" ))) ; ex. p r i n t () ; } A C E X M L _ E N D T R Y ; void Raw Ic o n P ar s e r : : c h a r a c t e r s ( const ACEXML-Char *cdata , int s tar t , int end ACEXML-ENV_ARG_DECL_NOT_USED) ACE-THROW-SPEC( ( ACEXML-SAXExccpt ion ) ) { int l t a g = t a g s . . s i z e ( ) — 1; / / p r i n t f (" l t a g : %d\n" , l t a g ) ; i f ( t a g s . . b a c k ( ) " index " ) { / / i f ( t a g s . , at ( l tag— 1) == "clement") { c u r r l n d e x = a t o i ( c d a t a ) ; assert ( icon —>height) ; asser t ( c u r r l n d e x < icon —>height) ; } } else i f ( t a g s _ . b a c k ( ) == "value"){ / / i f ( t a g s . , at ( l tag— 1) == "element") { assert ( icon —> content ) ; i f ( ! strcmp ( icon —> content —> type , " S p a t i a l l c o n " ) ) { # i f d c f DEBUG p r i n t f (" Tag < val uc > = %f , c u r r l n d c x=%d\n " , a t o f ( c d a t a ) , c u r r l n d e x ) ; # c n d i f icon —> content —> SctData ( curr lndex , a t o f ( c d a t a ) ) ; } i f ( ! strcmp ( icon —>con t c n t —>ty pc , " A n i m a t c d l c o n " ) ) { # i f d e f DEBUG p r i n t f ("Tag <valuc> = %f , c u r r I ndcx=9cd \ n " , a t o f ( c d a t a ) , c u r r l n d e x ) ; #end i f i f ( t ags _ . a t ( 11 ag — 1) == "frame") { icon —>con t c n t —>Sct Dat a ( c u r r l n d c x + currFrame , a t o f ( c d a t a ) ) ; } else { # i f d o f DEBUG p r i n t f (" ERROR: value given out s ide of frame context in A n i m a t c d l c o n . \ n " ) ; #end i f } } else { Appendix D. Browser Prototype Code 249 # i f d o f DEBUG p r i n t f (" Unknown icon type for c i c m c n t / v a 1 u c %s\n" , i co n —>co n t e n t —> t y pe ) #cndi'f } } lsc i f ( t a g s . . back () == "frames") if ( ! s t r c m p ( i c o n —>content —>type , " A n i m a t c d l c o n " ) ) { icon —>co n t e n t —>num Frames = a t o i ( c d a t a ) ; } lsc i f ( t a g s . . b a c k ( ) == "f p s " ) if ( ! s t r c m p ( i c o n —>con t cn t —>ty pe , " A n i m a t c d l c o n " ) ) { icon —> content —> fps = a t o i ( c d a t a ) ; } lsc i f ( t ags _ . back ( ) == "frame") if ( ! s t rcmp( i con —>con t cn t —>ty pe , " A n i m a t c d l c o n " ) ) { c u r r c n t F r a m c = a t o i ( c d a t a ) ; } lsc i f ( t a g s _ . b a c k ( ) == "type") if ( ! s t r c m p ( c d a t a , " S p a t i a l l c o n " ) ) { icon —>c o n t c n t = new S p a t i a l l c o n C o n tent () ; icon —> content —> i c o n p a r e n t = i c o n ; } else i f (! strcmp ( cdata , " A n i m a t c d l c o n " ) ) { icon —>c o n t c n t = new Animated lconContent () ; icon —> content —> i c o n p a r e n t — i c o n ; } else { # i f d c f DEBUG p r i n t f (" Unknown icon type %s \ n" , c d a t a ) ; #ond i f } } else if ( t ags _ . back () == "he ight") { assert ( icon —>content) ; icon —> height = a t o i ( c d a t a ) ; if ( ! s t rcmp( i co n —> c o n t c n t —>ty pc , " S p a t i a l l c o n " ) ) { # i f d o f DEBUG p r i n t f ( " A l l o c a t i n g data for height =%d \ n " , i c o n —> height ) ; #ond i f icon —>content —>AllocatcData ( icon —>height) ; } else i f ( ! s trcmp( icon —>con t en t —>ty pc , " A n i m a t c d l c o n " ) ) { # i f d c f DEBUG p r i n t f ( " A l l o c a t i n g data for h c i g h t=%d \ n " , i co n — > h c i g h t ) ; # c n d i f icon —> content —> A l l o c a t c D a t a ( icon —> height * icon —> content —> numFramcs) ; } else { # i f d c f DEBUG p r i n t f ("Unknown icon type for clement / value %s \ n " , icon —>co n t c n t —> t y p e ) #cndi'f } Appendix D. Browser Prototype Code 250 void RawIconParscr : : endDocumcnt (ACEXML_ENV_SINGLE_ARG-DECL_NOT_USED) ACE-THROW-SPEC ( ( A C E X M L . S A X E x c c p t i o n ) ) { } void RawIconParscr : : cndElcmcnt ( const ACEXML.Char * uri , const ACEXML.Char *namc, const ACEXML.Char *qName ACEXML_ENV_ARG_J3ECL_-NOT-USED) ACE.THROW-SPEC ( ( A C E X M L . S A X E x c c p t i o n ) ) { t a g s . . p o p . b a c k ( ) ; } void RawIconParscr :: s c tDocumcntLocator ( A C E X M L - L o c a t o r * l o c a t o r ) ACE_THROW_SPEC( ( ACEXML-SAXExcept ion ) ) { this —>locator . = l o c a t o r ; } void RawIconParscr : : s tartDocument (ACEXML-ENV-SINGLE-ARG.DECL_NOT.USED) ACE.THROWJ3PEC( ( A C E X M L . S A X E x c c p t i o n ) ) { } void RawIconParscr : : s t a r t E l e m c n t ( const ACEXML.Char * ur i , const ACEXML.Char *namc, const A C E X M L . C h a r *qNamc, A C E X M L . A t t r i b u t c s * a l i s t ACEXML_ENV_ARGJDECL_NOT_USED) ACE_THROW.SPEC( ( A C E X M L . S A X E x c c p t i o n ) ) { t a g s . . push-back(name ) ; } RawIconParscr : : RawIconParscr ( std : ; s t r i n g f i l e , H a p t i c l c o n * i c o n p t r ) { this —> icon = i c o n p t r ; c u r r l n d e x = — 1; P a r s e F i l c ( f i l e ) ; } RawIconParscr : : ~ RawIconParscr () { } D . l . 9 ma in .cpp / / Browser V2 . 0 / / Rewrite to reduce overhead — s i n g l e thread only / / Joseph Luk, July 2006 / / based on code by V i n c e n t Levesque and Shannon L i t t l e # i n c l u d e #i n cl u de #i n cl u de ^ i n c l u d e # i n c l u d c s t r c s s d / FirmwarcInfoXML . h" st r c s sd / Dcv iccIn foXML . h" s t r e s s d / S t r c amD cv icc . h " s t r c s s d / P a c k c t L o g g c r . h " s t r c s s d / i n i t . h " #i n c1u d c < s t d i o . h> ^ i n c l u d e < ios trcam> ^ i n c l u d e < s i g n a l . h> #i n c1ud c < s y s / s t a t . h> ^ i n c l u d e " a c e / T h r e a d . h" Appendix D. Browser Prototype Code 251 # i n c l u d c " DataUpdateThread . h" #def inc FIRMWARE-FILE " d of a u 11 . sd f " #dof ino HARDWARE-FILE " d cf au 11 . sdh " / / These must be kept s y n c h r o n i z e d with MyComponcnt. epp ! #dc f ine WORKING-DIR " /home/ lu k / b r o w s e r - w o r k i n g - d i r " #def ino PAGEMAP-FILENAME " pagemap . xml" # d c f i n c PAGEMAP-FLAGFILENAME "nowpagomap" int t c r m f l a g ; void s i g i n t _h a n d 1 c r ( i n t signum) { t c r m f l a g = 1; } / / I / O r o u t i n e s to communicate with browser / / c u r r e n t l y uses f i l e I / O , should probably be improved l a t e r bool W r i t c S 1 i d c r P o s i t io n ( i n t map.pos, HapticPageMap *pageMap); bool W r i t c C l i c k s ( i n t c l i c k - c o u n t ) ; bool WasPagcRcloadcd ( v o i d ) ; bool f i l c E x i s t s (char * f i l c N a m c ) ; int main ( i n t a r g c , char* argv [] ) { s t r u c t BrowscrSharcd s h a r c d D a t a ; int c l i c k . c o u nt , l a s t - c l i c k s ; int map.pos ; f l oa t norm.map.pos ; t c r m f l a g = 0; s i g n a l (SIGINT, & s i g i n t _ h a n d l c r ) ; try { s t r c s s d :: i n i t ( ) ; std : : cout « "Load ing f irmware d e f i n i t i o n from " « FIRMWARE-FILE « " . . . \ n " ; s t r c s s d : : FirmwarcInfoXML fwlnfo (FIRMWARE-FILE) ; s t d : : cout « "Load ing hardware d e f i n i t i o n from " « HARDWARE_FILE « " . . . \ n " ; s t r c s s d : : Devi c e l n f o X M L dev lnfo (HARDWARE-FILE) ; std : : cout « " C r e a t i n g s treaming STRcSS device with buf fer s ize of l . . . \ n " ; s t r c s s d : : S trcamDcvice d e v ( d e v I n f o , f w l n f o , d e v l n f o , 1); std : : cout « "Connec t ing and programming device ( t h i s may take a few seconds) dev . Connect( ) ; Was Page Reloaded () ; / / de lete page update f i l e , i f it e x i s t s std : : cout « " S t a r t i n g high — p r i o r i t y Data UPDATE thread . . . \ n " ; DataUpdateThread dat aU pdThread (&s h ar cd D a t a , & d e v ) ; A C E . h t h r o a d . t hThread; A C E - t h r c a d - t t - i d ; A C E - T h r o a d :: spawn ( DataUpdateThread : : Start , &dataUpdThread , THR-NEWJLWP | THR-JOINABLE, & t . i d , &hThread , ACE-Schcd-Params : : p r i o r i t y . m a x (ACE-SCHED-OTHER) Appendix D. Browser Prototype Code 252 do / / th i s loop runs once per new page map { shared Data . SctPagcMapRcady ( f a l s e ) ; / / stop thread from read ing from page map char pagemap-path [ 1 00] ; s p r i n t f (pagcmap.path , "%s/%s", WORKINGJ3IR, PAGEMAP.FILENAME) ; p r i n t f ("Loading page map f i l e %s . \ n " J pagcmap.path) ; HapticPageMap pagcMap ( pagcmap.path ) ; p r i n t f ("Done l o a d i n g page map. Rendering i c o n s . . . \ n" ) ; pagcMap . R c n d c r A l l I c o n s () ; p r i n t f ("Done r e n d e r i n g i c o n s . \ n " ) ; p r i n t f ( " \ n S t a t i c pagemap dump:\n" ) ; for ( i n t pos=0; pos<pagcMap . h c i g h t ; pos++) { p r i n t f ("%2.0f ," , pagcMap . voltmap [ pos ] ) ; } p r i n t f (" \n \n") ; sharcdData . pagemap = &pagcMap ; sharcdData . SctPagcMapRcady ( true ) ; / / s t a r t thread read ing from page map us lccp (500) ; l a s t . c l i c k s = s h a r c d D a t a . G c t C l i c k s f ) ; / / th i s should probably be moved in to a l o w — p r i o r i t y thread . . . p r i n t f ("Type C T R L - C to q u i t . \ n " ) ; while ( t c r m f l a g == 0) { us l ccp (100); map.pos = shared Data . GctMapPos ( ) ; c l i c k . c o u n t = s h a r c d D a t a . G e t C l i c k s f ) ; # i f d c f DEBUG p r i n t f ("Current p o s i t i o n at %d\n" , map.pos) ; # c n d i f W r i t c S l i d c r P o s i t i o n (map.pos , &pageMap) ; / / W r i t c C l i c k s ( c l i c k . c o u n t ) ; i f ( WasPageReloadcd () ) break; } / / while ( t c r m f l a g —— 0) / / c lean up page map s t u f f here , i f necessary } while ( t c r m f l a g —— 0) ; p r i n t f ( " K i l l i n g t a c t i l e loop t h r e a d . \ n"); A C E . T h r c a d : : cance l ( hThrcad ) ; std : : cout « " D i s c o n n e c t i n g device . . . \ n " ; dev . Disconnect () ; } catch ( s t r c s s d :: Except ion & cx) { std : : cout « " l i b s t r c s s d threw an except ion : \ n " std : ; cout « cx . E r r S t r () « " \ n " ; } s t r c s s d :: f i n i () ; r e t u r n 0 ; Appendix D. Browser Prototype Code 253 } b o o l W r i t c S 1 i d c r P o s i t i o n ( i n t m a p . p o s , HapticPageMap *pageMap) { / / d e l e t e a l l e x i s t i n g .pos f i l e s DIR* pd ir = o pen d i r (WORKING-DIR) ; s t r u c t d i r c n t *pcnt ; i f ( ! p d i r ) { p r i n t f (" opendir () f a i l u r e ; t e r m i n a t i n g " ) ; cx i t (1) ; } s t d : : s t r i n g i c o n . t y pc = ""; while ( (pcnt = r e a d d i r ( p d i r ) ) ) { s t d : : s t r i n g f i l e = pent —>d_name ; int l ength = f i l e , length ( ) ; / / f i l t e r out .pos (pos) f i l e s only i f ( f i l e . r f i n d (" .pos" , l eng th ) == length — 4) { / / d e l e t e f i l e char t y p c . s t r [50] ; s t rcpy ( t y p c - s t r , WORKING-DIR) ; s t r c a t ( t y p c - s t r ,"/") ; s t r c a t ( t y p c . s t r , f i l c . c _ s t r ( ) ) ; r c m o v c ( t y p C - S t r ) ; } } c l o s c d i r ( p d i r ) ; / / w r i t e new pos f i l e char t y p c . s t r [50]; s t r c p y ( t y p c - s t r ,WORKING-DIR) ; s t r c a t ( t y p c - s t r ,"/") ; char s l i d c r . s t r [10]; / / f l oa t norm.map.pos = ( f l o a t ) map.pos / ( f 1 o a t ) pagcMap—> h c i g h t * 100.0; / / s p r i n t f ( s l i d c r . s t r , "%f", norm.map.pos) ; s p r i n t f ( s l i d c r . s t r , "%d" , map.pos) ; s t r c a t ( t y p c . s t r , s l i d c r . s t r ) ; s t r c a t ( t y p c . s t r , " . p o s " ) ; std : : o f s trcam o u t p u t F i l e ; o u t p u t F i l c .open ( t y p c . s t r , std : : ofstream : : out) ; o u t p u t F i l e . c lose () ; r e t u r n t r u e ; } bool W r i t c C l i c k s ( i n t n c w . c l i c k s ) { s t a t i c int l a s t . c l i c k s — 0; p r i n t f ( " l a s t . c l i c k s = %d , n c w . c l i c k s = %d \ n", l a s t . c l i c k s , n c w . c l i c k s ) ; int c l i c k . d i f f = n c w . c l i c k s — l a s t . c l i c k s ; if ( c l i c k . d i f f > 0){ for ( i n t c = 0; c < c l i c k . d i f f ; c ++){ / / w r i t e new c l i c k f i l e char t y p c . s t r [50] ; s t r c p y ( t y p c - s t r ,WORKING-DIR) ; s t r c a t ( t y p c . s t r >"/") ; char s l i d c r . s t r [10]; s p r i n t f ( s l i d c r _ s t r , "%d", l a s t . c l i c k s + c ) ; Appendix D. Browser Prototype Code 254 s t r c a t ( t y p c . s t r , s l i d c r . s t r ) ; s t r c a t ( t y p c . s t r , " . c l i c k " ) ; std : : o f s trcam o u t p u t F i l e ; o u t p u t F i l e . o p e n ( t y p c . s t r , std : : o fs trcam : : out ) ; o u t p u t F i l e . c lose () ; } } l a s t . c l i c k s = n c w . c l i c k s ; } / / Check the p age in a p _f 1 agf i 1 c to sec i f it e x i s t s ; i f it does , de le te it . bool WasPagcRcloadcd ( v o i d ) { / / DIR. pd ir = opondir (WORKINGJDIR) ; / / s t r u c t d i r c n t . p e n t ; // / / i f ( ! p d i r ) { / / p r i n t f ( " o p e n d i r f ) f a i l u r e ; t e r m i n a t i n g " ) ; / / c x i t ( l ) ; // } // / / bool found = f a l s e ; / / while ( (pcnt = r c a d d i r ( p d i r ) ) && found == fa l s e ) { / / s t d : : s t r i n g f i l e = pc nt —>d_name ; / / int l ength — f i l e , l ength ( ) ; / / / / f i l t e r out only the PAGEMAP.FLAGFILENAME / / i f ( f i 1 c . r f i n d (PAGEMAP.FLAG FILENAME, l eng th ) != s t d : : s t r i n g : : npos) { / / found = t r u e ; / / / / d e l e t e f i l e / / char pathname [ 1 0 0 ] ; / / s p r i n t f (pathname, "%s/%s", WORKINGJDIR, PAGEMAP.FLAGFILENAME) ; / / rcmovc(pathnamc); // } // } / / c l o s e d i r ( p d i r ) ; // / / r e turn ( f o u n d ) ; / / L e t ' s try a more e f f i c i e n t way s t a t i c char path [100]; s p r i n t f ( p a t h , "%s/%s", WORKINGJDIR, PAGEMAP.FLAGFILENAME) ; if ( f i l c E x i s t s ( p a t h ) ) { remove ( path ) ; r e t u r n t r u e ; } else r e t u r n f a l s e ; bool f i l c E x i s t s (char * f i l c N a m c ) { s t r u c t s tat buf; int r c t v a l = s tat ( f i leName , &buf ); r e t u r n ( r c t v a l == 0 ? true : f a l s e ) ; > D .2 V i s u a l Browser Componen t The visual browser component extends the Mozilla platform using a combi-nation of custom JavaScript, X U L , and X P C O M files. It is based partially on code developed by Shannon Little [33] . In the interest of brevity, some Appendix D. Browser Prototype Code 255 automatical ly generated and t r iv ia l configuration files are omit ted. • browser, j s - the main interactive JavaScript routine. • w e b s c r o l l e r . ess - Cascading Style Sheet description of the default browser presentation settings. • browser.xul - onscreen user interface description file. • r e a d y d i a l o g . x u l - user interface description for a moda l dialog used as a prompt during user evaluation. • IMyComponent. i d l - I D L (Interface Descr ipt ion Language) stub header for the X P C O M I / O routines. • MyComponent. epp - the file input and output routines, implemented in the X P C O M framework. D.2.1 b rowser . j s / / browser . j s / / Joseph Luk, Ju ly 2006 / / Based on framework code by Shannon L i t t l e / / G l o b a l s / / uncommcnt one of the f o l l o w i n g input modes / / var input .modc = "mouse"; / / var input _m ode = " s l i d e r — " ; var input .modc = " s l i d c r + " ; var i n i t . p a g c = " f i l e : / / / home / l u k / b r o w s c r —content / index . html" const t a c t i l c . s c a l c f a c t o r = 1; / / should be 1 ( f u l l - s c a l e ) or 2 (ha l f — s c a l e ) var myBrowscr = n u l l ; var appCorc = n u l l ; var top = 0; var current = 0; var search Range ; var s t a r t P t ; var cndPt ; var l i n k T c x t s ; var l i n k U R L s ; var h i g h l i g h t e d = n u l l ; var pageheight = 0; var pagewidth = 0; var addheight = 0; var l i n k s ; Appendix D. Browser Prototype Code 256 var numlinks = 0; / / t o t a l number of l i n k s in the page var n u m _1 i n ks _a t _y pos ; var component l inker ; / / study l ogg ing g l o b a l s var 1 og .reset Co u n t = 0; var l o g . s t a r t T i m e = ""; var l o g . c n d T i m e — ""; var l o g . p a g c s = new A r r a y ( ) ; / / page load l i n k s var l o g . t i m c s = new A r r a y ( ) ; / / page load times var f i r s t l o a d = t r u e ; var debug = t r u e ; const PREFS.CID = " @mozi 11 a . org / p r ef erences ; 1 " ; const P R E F S . I . P R E F = " n s I P r c f " ; const PREFJ3TRING = "browser , dom . window . dump . enabled " ; try { var P r c f = new Components . C o n s t r u c t o r (PREFS.CID , P R E F S . I . P R E F ) ; var p r c f = new P r c f () ; p r c f . S c t B o o l P r c f ( PREFJ3TRING , true ) ; }catch(c){> f u n c t i o n i n i t B r o w s e r () { i f (debug) l o a d P a g c . ( " j a v a s c r i p t : " ) ; / / load debug page i f a p p l i c a b l e myBrowser = document. g c t E l c m c n t B y I d ( " browser —content"); appCorc = Components, c la s se s [" © m o z i l l a . o r g / a p p s h c l l / component /browser / ins tance;1"] . c r c a t c l n s t a n c c (Components, i n t e r f a c e s . n s I B r o w s e r l n s t a n c c ) ; appCorc . sc tWcbShcl lWindow( window) ; var content = document . g c t E l c m e n t B y l d (" browser — content" ) ; i f ( c o n t e n t ) { content . a d d E v c n t L i s t c n c r (" load" , setup , t r u e ) ; } try { netscape . s e c u r i t y . P r i v i l c g c M a n a g e r . e n a b l c P r i v i l c g c (" U n i v e r s a l X P C o n n c c t " ) ; const c id = " @ my domain . com / XPCOMSample / My Component ; 1 " ; component l inker = Components, c lasses [ c i d ] , c r c a t c l n s t a n c c ( ) ; component l inker = c o m p o n e n t l i n k e r . Q u e r y l n t c r f a c c (Components, i n t e r f a c e s . IMyComponent) ; } catch ( e r r ) { a l e r t ( e r r ) ; re turn ; } / / Go fu l l—screen mode / * netscape . s e c u r i t y . P r i v i l c g c M a n a g e r . e n a b l c P r i v i l c g c (" U n i v e r s a l X P C o n n c c t " ) ; */ / * const MEDIATOR.CONTRACTID=" © m o z i l l a . o r g / a p p s h c l l / w i n d o w - m e d i a t o r ; 1 " ; * / / * const nsIWindowMcdiator=Components. i n t e r f a c e s . nsIWindowMcdiator; * / / * var windowManagcr= * / / » Components . c lasses [ MEDIATOR-CO NTRACTID ] . g c t S o r v i c o ( nsIWindowMcdiator) ; * / /* try { */ / * netscape . s e c u r i t y . P r i v i l c g c M a n a g e r . e n a b l c P r i v i l c g c (" U n i v e r s a l X P C o n n c c t " ) ; */ / * main Window = windo w Manager . get Most Recent Window (" n a v i g a t o r : browser") ; * / / • } * / / * c a t c h ( c ) { * / Appendix D. Browser Prototype Code 257 h a l e r t ( c ) ; * / I* } * / 1* i f ( main Window . s i d c b a r _ i s _ h i d d c n ( ) ) * / h h i d c S i d e b a r = f a 1 s e ; * / h i f ( h i d c S i d c b a r ) * / /* main Window .S idcbarShowHide ( ) ; */ /* mainWindow. B ro w scr F u 1 IS cr ce n ( ) ; */ /* window . f u l l S c r c e n = t r u e ; * / 1* window, l o c a t i o n b a r . v i s i b l e = fa l se * / /* window, t o o l b a r . v i s i b l c = f a l s c ; * / var browser W in = document . g c t E l c m c n t B y l d ( " browserwin " ) ; i f ( input .modc = "mousc"){ / / a t t a c h l i s t e n e r s browscrWin . a d d E v c n t L i s t c n c r ( ' DOMMouscScrol l ' , s c r o l l , t r u e ) ; browscrWin . a d d E v c n t L i s t c n c r ( ' c l i c k ' , PushBut tonHandlcr , true ) ; } / / a t tach keyboard se lec t event l i s t e n e r (workaround for n o n f u n c t i o n a l s e l ec t button ) browscrWin . a d d E v c n t L i s t c n c r ( ' key down ' , P u s h B u t t o n H a n d l c r , t r u e ) ; / / Load the page , t r i g g e r i n g the setup f u n c t i o n loadPagc_( in i t . p a g c ) ; } f u n c t i o n l o a d P a g c . ( page ) { / / glob only the las t part of the URL var p a g e s t r i n g = page . t o S t r i n g () ; var pagc f i l cnamc = p a g e s t r i n g . s u b s t r i n g ( p a g e s t r i n g . l a s t I n d c x O f ( " / " ) + 1 , p a g e s t r i n g . l ength ) ; / / dump ( " l o a d P a g c : " 4- pagcf i lename " \ n" ) ; / / push the page load into the log array log-pages [ log-pages . l ength ] = p a g c f i l e n a m e ; var datcObj = new Datc( ) ; l o g - t i m e s [ l o g - t i m e s . length ] = datcObj . getTime () ; const nsl W cb Navigat ion = Components, i n t e r f a c e s , nsl W cbNavigat ion ; gc tBrowscr () . wcbNavigat ion . loadURI(page , nsIWcbNavigat ion . LOAD_FLAGS_NONE, nul l , nu l l , n u l l ) ; } f u n c t i o n setup (event ){ myBrowscr = event . o r i g i n a l T a r g e t ; myBrowscr. c o l l a p s e = t r u e ; current = 0; h i g h l i g h t e d = n u l l ; document . command D ispatchcr . focused Window = window ; document . commandD ispatchcr . focused Element = myBrowscr . l i n k s . i t cm( current ) ; i f ( input .modc . i n d c x O f (" s l i d e r " ) != — 1){ p a g c h c i g h t a l t = c o n t e n t . i n n c r H e i g h t 4- content . s c r o l l M a x Y ; p a g c w i d t h a l t = c o n t e n t . i n n c r W i d t h 4- content . s c r o l l M a x X ; if (debug) dump ("Page dimensions ( v i a content . i n n c r H c i g h t + s c r o l l M a x Y ) " + pagchcight -f- " X " + pagewidth 4- " \ n " ) ; pagchc ight = gctBrowscr () . contentDocument ; body . s c r o l l H e i g h t ; pagewidth = gctBrowscr () . contentDocument . body . s c r o l l W i d t h ; i f (debug) dump ("Page dimensions ( v i a S c r o l l H e i g h t ) " + p a g c h c i g h t a l t + " X " + p a g c w i d t h a l t + " \ n " ) ; if (debug) dump ("window. i n n c r H e i g h t = " + window . i n n c r H e i g h t -f- " \ n " ) ; i f (debug) dump (" content . i n n c r H e i g h t = " 4- c o n t c n t . i n n e r H c i g h t 4- "\n") Appendix D. Browser Prototype Code 258 numlinks = myBrowscr. l i n k s , l e n g t h ; i f (debug) dump ("Number of l i n k s = " + numlinks + " \ n " ) ; / / pagchc ight = gctBrowscr () . contentDocument . f i r s t C h i l d . o f f s c t H c i g h t ; / / pagewidth = getBrowser () . contentDocument . f i r s t C h i l d . o f f s e t W i d t h ; n u m . l i n k s . a t . y p o s = new A r r a y ( p a g e h c i g h t ) ; / / f i n d rows where there is more than one l i n k and the same y pos for (j = 0; j < numl inks ; ++j ) { y = f indPosY ( gc tBrowscr ( ) . contentDocument . l i n k s , i t e m ( j ) ) ; i f (! n u m . l i n k s . a t . y p o s [ y ] ) { n u m . l i n k s . a t . y p o s [y] = 1; } c 1 s c { n u m . l i n k s . a t . y p o s [y] + + ; page height += gctBrowscr () .contentDocument . l i n k s . i t e m ( j ) . o f f s e t H e i g h t ; } } / / c r e a t e l i n k s array l i n k s = new Array ( pagewidth ) ; for (j = 0; j < pagewidth; j++){ 1 i n k s [ j ] = new A r r a y ( pagchc ight ) ; for (k = 0; k < pagohe ight ; k++){ l i n k s [ j ] [ k] = - 1 ; } / / S t a r t g e n e r a t i n g the pagemap f i l e c o m p o n c n t l i n k e r . Delete Page M ap ( ) ; c o m p o n c n t l i n k c r . Write PagcMap ( ' < ? xml v e r s i o n ="1.0" c n co d i ng = "UTF—8" ?> \ n \ n ' ) ; c o m p o n c n t l i n k e r . WritePageMap ( ' < pagemap>\n ' ) ; var seal cd Page H eight = pagchc ight / t a c t i 1 e _ s c a 1 e f a c t o r ; c o m p o n c n t l i n k e r . WritePageMap (' < h c i g h t > ' + s c al e d P age H e i g h t + '< / height >\n') ; riugh a l l the page l i n k s < numlinks ; iH—h) i = 0; / / i t e r a t e thr for ( i = 0; i { l i n k = getBrowser ( ) . contentDocument . l i y = f indPosY ( l i n k ) ; x = f indPosX ( l i n k ) ; range = 1ink . o f f s e t H e i g h t ; item ( i ) ; / * i f ( l i n k , name ) * / iconname = l i n k .name; * / else * / iconname = " b i gsqu ar e . xml " ; i f (debug) dump ("Link na if (debug) dump ("Y = ") ; i f (debug) dump ( y ) ; if (debug) dump (" Range i f (debug) dump ( r a n g e ) ; if (debug) dump (" \n") ; e=" + H var sca ledY = y / t a c t i 1 c _ s c a 1 c f a c t o r ; c o m p o n c n t l i n k e r . WritePageMap (" < icon>\n") ; c o m p o n c n t l i n k c r . W r i t e P a g e M a p (" <at>" + sca ledY -f- " </at >\n" ) ; / / com po nen t l i nkcr . WritePageMap (" <range>" + range + " </r a n ge >\n " ) ; co m ponent 1 i nke r . WritePageMap (" <name>" + iconname + " </namc>\n " ) ; c o m p o n c n t l i n k c r .WritePageMap (" < / icon>\n") ; found = fa l s e ; for (j = 0; (j < pagewidth [found) ; / / i f there is a l ready a l i n k at th i s y , extend new l ink 's y coord by i t s height i f ( l i n k s [ j ] [ y ] >= 0 ) { y = y + range; found = t r u e ; } Appendix D. Browser Prototype Code 259 } for (k = y; k < y + range; kH—h){ l i n k s [x] [ k] = i ; } } component l inker . Write Page Map ( ' < / pagemap>\n ' ) ; / / t r i g g e r re load from t a c t i l e loop / / d e p r e c a t e d , wait u n t i l user s t a r t s the task before doing t h i s } / / o t h e r w i s e , assume mouse else { h i g h l i g h t e d = gctBrowscr () . contentDocumcnt . l i n k s . item ( c u r r e n t ) ; gc tBrowscr () . contentDocumcnt . g c t E l c m c n t B y l d (" h i g h l i g h t d i v " ) . s t y l e . v i s i b i l i t y = " hidden " ; } / / sca le the h i g h l i g h t image as needed to represent the cursor s ize var imagetag = "<img s r c ='h i g h 1 ig h t—image . png ' i d =' h i g h 1 i g h t i m a g c ' width="; imagctag += pagewidth ; imagetag += " h c i g h t =" ; / / imagctag += 8 * t a c t i l c . s c a l c f a c t o r ; (without b i l i n e a r i n t e r p o l a t i o n ) imagctag -\-= 16 * t a c t i l e . s c a l c f a c t o r ; imagetag +— " />" ; gc tBrowscr () . contentDocumcnt . g c t E l c m c n t B y l d ( " h i g h l i g h t d i v " ) . inner H T M L = imagctag ; i f ( f i r s t l o a d ) { window. openDia log (" chrome:/ / w c b s c r o l l e r / contents / r c a d y d i a l o g . x u l " ,"showmorc" , " chrome , modal " ) ; f i r s t l o a d — f a l s e ; l o g _ r e s e t C o u n t + + ; l og .pages = new Array () ; l o g . t i m c s = new A r r a y ( ) ; log_endTime = 0; i f ( input .modc . indexOf (" s l i d e r ") != — 1) { component l inker . SetPageMapFlag ( ) ; u p d a t e - f r o m . d e v i c c () ; } var datcObj = new Date() ; 1 og.st ar t T i m c = dateObj . gctTime () ; > else { var datcObj = new Date() ; i f ( input .modc . indexOf (" s l i d e r " ) != —1){ component l inker . SetPageMapFlagf ) ; u p d a t e . f r o m . d e v i c e ( ) ; } i f (debug) dump ("Page loaded at " + datcObj . gctTime () + " \ n " ) ; } / / n e e d s focus in order to process keyboard gctBrowscr () .contentDocumcnt . l i n k s . i tem(0) . focus () ; / / s e t bottom text bar Appendix D. Browser Prototype Code 260 document . g c t E l e m e n t B y l d (" current — l in k " ) ; f u n c t i o n goBack ( ) { var wcbNavigat ion = getBrowser () . wcbNavigat ion ; i f ( w c b N a v i g a t i o n . canGoBack) w c b N a v i g a t i o n . go Back () ; } f u n c t i o n goForward ( ) { var wcbNavigat ion = gctBrowscr ( ) . w c b N a v i g a t i o n ; i f (wcbNavigat ion .canGoForward) wcbNavigat ion . goForward () ; } f u n c t i o n UpdatcBackForwardBut tons ( ) { var b a c k B r o a d c a s t c r = document. g c t E l c m c n t B y l d ("can G o B a c k " ) ; var f o r w a r d B r o a d c a s t e r = document. g c t E l c m c n t B y l d ("canGoForward"); var wcbNavigat ion = gctBrowscr ( ) . wcbNav iga t ion ; var b a c k D i s a b l c d = ( b a c k B r o a d c a s t c r . get A t t r i b u t e ( " d i s a b l e d " ) —= "true") ; var f o r w a r d D i s a b l e d = ( f o r w a r d B r o a d c a s t e r . g c t A t t r i b u t c ( " d i s a b l e d " ) == " true") if ( b a c k D i s a b l c d == webNavigat ion . canGoBack ) b a c k B r o a d c a s t c r . set A t t r i b u t e (" d i s a b l e d " , ! b a c k D i s a b l c d ) ; i f ( f o r w a r d D i s a b l e d == wcbNavigat ion . canGoForward ) f o r w a r d B r o a d c a s t e r . set A t t r i b u t e (" d i s a b l e d " , ! f o r w a r d D i s a b l e d ) ; } f u n c t i o n gctBrowscr () { re turn document . g c t E l e m e n t B y l d (" browser —content " ) ; > f u n c t i o n BrowscrContcnt () { re turn gc tBrowscr () . contentDocumcnt ; / / Implement push —to—s c r o 11 f u n c t i o n f u n c t i o n Scrol l W indow ( c u r s o r Y P o s ) { / / Constants var margintop = 100; / / S c r o l l i n g begins when cursor h i t s margin var marginbottom = margintop ; / / L o c a l vars var c u r r t o p = gctBrowscr ( ) . contentDocumcnt . body. s c r o l l T o p ; var curr bottom = c u r r t o p + content . i n n c r H c i g h t ; var newtop = c u r r t o p ; / / Code if ( c u r s o r Y P o s < c u r r t o p + margintop) { newtop = cursorYPos — margintop ; i f (newtop < 0) newtop = 0; } else i f ( c u r s o r Y P o s > currbot tom — marginbottom) { newtop = cursorYPos -f- marginbottom — content . i n n c r H c i g h t ; i f (newtop > c o n t e n t , scroll M a x Y ) newtop = c o n t e n t , scroll M a x Y ; > gctBrowscr () . contentDocumcnt . body . s c r o l l T o p = newtop ; Appendix D. Browser Prototype Code 261 / / f o r input-mode = s l i d e r f u n c t i o n updatc_ from_dev icc () { if ( in put .m ode. indexOf (" s l i d e r " ) == — 1){ a 1 c r t (" ERROR: method \ ' u p d a t c - f r o m _d c v i c e \ ' should not be c a l l e d unless using s l i d e r d e v i c e . \ n Set s t r i n g in l ine 2 of browser, js to \ " s l i d e r \ " .") ; r e t u r n ; } c 1 s c { c l i c k = c o m p o n e n t l i n k c r . G c t C l i c k ( ) ; if ( c l i c k h i g h l i g h t e d != n u l l ) { / * document . g c t E l c m e n t B y l d (" URLBar" ) . i n s e r t l t e m A t (0 , h i g h l i g h t e d , h i g h l i g h t e d ) ; * / / * document . g c t E l c m e n t B y l d (" URLBar" ) . s el e c te d In d c x =; 0; * / myHigh l ight ( h i g h l i g h t e d , " ye l low") ; / / g ° T o ( ) ; } c 1 s c { var s c a l c d Y P o s = c o m p o n c n t l i n k c r , R e a d S l i d e r P o s ( ) ; var ypos = s c a l c d Y P o s * t a c t i l e . s c a l e f a c t o r ; / / var ypos = Math . round ( ( c o m p o n c n t l i n k c r . R e a d S l i d c r P o s ( ) / 100) * p a g c h c i g h t ) ; / * var canvas = document . g c t E l c m e n t B y l d (" canvas" ) ; * / / * var ctx = c a n v a s . g c t C o n t c x t ( " 2d") ; * / / * ctx . f i 11Sty1c = "rgba(150 , 150, 0, 0 .5)"; * / / * c t x . f i l l R c c t (0 , ypos — 1, pagewidth , ypos + 1); * / / / move the h ighl igh—div clement to show the ypos / / TODO: need to adjust to correspond to t a c t i l e window var c u r s o r t o p = ypos — 4; gctBrowscr () . contentDocument . g c t E l c m e n t B y l d (" h i g h l i g h t d i v " ) . s t y l e . top= c u r s o r t o p -f-" px " ; S c r o l l W i n d o w ( y p o s ) ; var i == 0; var found = f a l s e ; for ( i ; ( i < pagewidth && ! found) ; i++){ if ( l i n k s [ i ] [ ypos] >= 0) { / / check i f h i g h l i g h t e d is d i f f e r e n t from p r e v i o u s ; i f so , play i c o n . . / * i f (gc tBrowscr () . contentDocument , l i n k s . i t c m ( l i n k s [ i ] [ ypos] ) ! = h i g h l i g h t e d ) * / / * { * / / * c o m p o n c n t l i n k e r . W r i t c I c o n ( " l i n k " ) ; * / / * } * / i f ( h i g h l i g h t e d != n u l l ) { myHigh l ight ( h i g h l i g h t e d , n u l l ) ; / / c l e a r prev ious h i g h l i g h t i n g } else { / * c o m p o n c n t l i n k c r . W r i t c l c o n f " l i n k " ) ; * / } h i g h l i g h t e d = gctBrowscr () . contentDocument . l i n k s . item ( l i n k s [ i ] [ ypos] ) ; h i g h l i g h t e d , focus () ; myHigh l ight ( h i g h l i g h t e d , "#f55"); found = t r u e ; } } i f (! found && h i g h l i g h t e d != n u l l ) { myHigh l ight ( h i g h l i g h t e d , n u l l ) ; h i g h l i g h t e d = n u l l ; / / c o m p o n c n t l i n k c r . W r i t c I c o n ( " n u l l ") ; } s c t T i m c o u t ( ' u p d a t e . f r o m . d c v i c c () 20); / /TODO — tweak th i s value Appendix D. Browser Prototype Code 262 } } / / f o r input .modc = mouse f u n c t i o n s c r o l l ( c v c n t ) { var d i r e c t i o n = event , d e t a i l ; / / d i r e c t i o n > 0 > UP / / d i r e c t i o n < 0 > DOWN / / i f we're not a lready at the top or bottom, then s c r o l l i f ( ( d i r e c t i o n > 0 current < (myBrowscr. l i n k s , l ength — 1)) | | ( d i r e c t i o n < 0 c u r r e n t > 0)){ m y H i g h l i g h t ( h i g h l i g h t e d , n u l l ) ; / / c l e a r prev ious h i g h l i g h t i n g if ( d i r e c t i o n > 0) c u r r e n t — current + 1; else c u r r e n t = current — 1; 0 h i g h l i g h t e d = ge tBrowscr ( ) . contentDocumcnt . l i n k s . item ( c u r r e n t ) ; h i g h l i g h t e d . focus () ; myHigh l ight ( h i g h l i g h t e d , "#f55"); s c t C u r r L a b c l () ; } event . p r c v c n t D e f a u l t () ; / / d o n ' t a c t u a l l y s c r o l l the page — focus () w i l l do that for us f u n c t i o n PushBut tonHandlcr ( event ) { var l i n k S c l c c t c d = f a l s e ; i f ( event , type == " c l i c k " ) { button = e v e n t . w h i c h ; event . p r c v c n t D e f a u l t ( ) ; / / cance l d e f a u l t event handler } else i f ( event , type == "kcydown " ) { var dateObj = new Datc( ) ; i f ( S t r i n g . fromCharCodc ( even t . key Code ) = " ") { iog_cndTimc = d a t c O b j . g c t T i m c ( ) ; i f (debug) dump ("User s i g n a l l e d task comple t ion at : " + dateObj . gctTime () + "\n") ; S t u d y L o g ( ) ; } else i f ( S t r i n g . fromCharCodc ( event . keyCode) == "B") gctBrowscr () . h i s t o r y . back() ; / / untested else i f ( S t r i n g . fromCharCodc ( even t . key Code ) == "R" ) { i f (debug) dump ("Reset at : " + d a t c O b j . gctTime () -f- " \ n " ) ; f i r s t l o a d = t r u e ; loadPagc_( i n i t . p a g c ) ; } else / / TODO: test for re turn key press here { l i n k S c l c c t c d = t r u e ; } event . p r e v e n t D c f a u l t () ; / / cance l d e f a u l t event handler } i f ( l i n k S e l c c t e d ) { / * document . g c t E l c m c n t B y l d (" URLBar" ) . i n s c r t l t c m A t (0 , h i g h l i g h t e d , h i g h l i g h t e d ) ; * / / * document . get Element By Id (" URLBar" ) . se lec t c d I n d ex = 0; * / var dateObj = new Date () ; i f (debug) dump("Link se l ec ted at " + datcObj . gctTime () 4- ": " + h i g h l i g h t e d + " \ n " ) ; Appendix D. Browser Prototype Code 263 m y H i g h l i g h t ( h i g h l i g h t e d , " y e l l o w " ) ; loadPage_( h i g h l i g h t e d ) ; } } // * * * * * * * f u n c t i o n s e t C u r r L a b c l () { / / curr = document. g c t E l e m e n t B y l d (" current — l i n k " ) ; / / text = h i g h l i g h t e d ; > f u n c t i o n my Highl ight ( node , c o l o r ) { / / i f (debug) dump(node); if ( c o l o r == n u11){ node. r e m o v c A t t r i b u t c (" s t y l e " ) ; } else { node. s c t A t t r i b u t c ( " s t y l e " , "background — c o l o r : " + co lor + " ;") ; } } / / n o t used f u n c t i o n rcmovelmagcs ( ) { var images = gctBrowscr () . contentDocumcnt . ge tElementsByTagNamc( ' img' ) for ( var i = 0; i < images, l e n g t h ; i -\—h) { parent — images [ i ] . parcntNodc ; parent . r c m o v c C h i l d ( images [ i ] ) ; } } f u n c t i o n s h r i n k l m a g c s ( ) { / / f o r smal l screen v iewing var s c r c c n . h c i g h t = 160; var s c r e e n - w i d t h = 120; var 1 = getBrowser () . contentDocumcnt . gctElcmcntsByTagName( 'img ' ) ; f o r ( v a r i = 0; i < 1. l e n g t h ; i++) { i f ( 1 [ i ]. width > sc r ce n -w i d t h ) { 1 [ i ] . height *= s c r e e n - h e i g h t /1 [ i ] . w idth; 1 [ i ] . width = s c r c e n _ w i d t h ; } else i f ( 1 [ i ] . natur a l W i d t h > s c r ee n _ w i d t h ) { var c = s c r e e n - w i d t h / l [ i ] . n a t u r a l W i d t h ; 1 [ i ] . height = l [ i ] . n a t u r a l H c i g h t * e; 1 [ i ] . width = s c r e e n - w i d t h ; } } } f u n c t i o n f i n d P o s X ( o b j ) { var c u r l e f t = 0; i f (obj . o f f s e t P a r e n t ) { while (obj . o f f s e t P a r e n t ) { c u r l e f t += obj . o f f s e t L e f t obj = ob j . o f f s ct P a r e n t ; } • } else i f ( o b j . x ) c u r l e f t += obj . x ; r e t u r n c u r l e f t ; } f u n c t i o n f i n d P o s Y ( o b j ) { var curtop = 0; i f (obj . o f f s e t P a r e n t ) { while (obj . o f f s e t P a r e n t ) { curtop +— obj . o f f s e t T o p Appendix D. Browser Prototype Code 264 obj = o b j . o f f sc t P a rc n t ; } } else i f ( o b j . y ) curtop += o b j . y; re turn c u r t o p ; / / Dump log for study / / TODO: add f i l e output would probably be best f u n c t i o n StudyLog( ) { / / log f i l e format : / / LOG : subj , b lock# ,cond , t a s k # , t a s k t y p c , task , s t a r t t i m c , endtime , c lapsedt ime , pagc laddr , p a g c l t i m c , pagc2addr , pagc2timc , . . . . / / aka LOG: input .modc , l o g . r c s c t C o u n t , , , l o g . s t a r t T i m c , l o g . c n d T i m c ,( c a l c u l a t e d ) , contents of l o g . p a g c s and l o g . t i t m c s arrays var logo u t p u t = "LOG: "; nput_mode ; o g . r e s c t C o u n t ; o g - s t a r t T i m c ; l og -cndTime ; logoutput += logoutput += logoutput += logoutput 4-= logoutput 4-= logoutput += logoutput 4-= logoutput 4-= if ( l o g . s t a r t T i m c && l o g . c n d T i m c ) { } l ogoutput 4-= l o g . c n d T i m c l o g . s t a r t T i m c ; for (var i = 0; i < l o g . p a g c s . l e n g t h ; i++) { logoutput 4-= " , " ; logoutput 4-= log-pages [ i ] ; l ogoutput 4-= " , " ; logoutput 4~= l o g - t i m e s [ i ] ; } l ogoutput 4-= " \ n " ; dump ( l o g o u t p u t ) ; D.2.2 webscroller.css window b a c k g r o u n d - c o l o r : white ; . h i g h l i g h t b a c k g r o u n d - c o l o r : #CCFFFF; . normal background —colo r : white ; . s e l e c t e d b a c k g r o u n d - c o l o r : red ; D .2.3 browser, x u l Appendix D. Browser Prototype Code 265 <?xml v e r s i o n ="1.0"?> <?xml — s t y l e s h e e t href =" chrome: / / g l o b a l / sk in" ty pe=" text / css"?> <?xml— s t y l e s h e e t h r e f - ' c h r o m e : / / n a v i g a t o r / s k i n " t y p c - ' t e x t / c s s " ? > <?xml—stylesheet href =" f i l e : / / / home / l u k / b r o w s e r — u i / w e b s c r o l l e r / sk in / w e b s c r o l l e r . ess" type=" text / css"?> < window i d =" browscrwin" t i t l e = " T a c t i l e browser" x m 1 n s - ' h 11 p : / / www. m o z i l l a . org /keymas ter / gatekeeper / there . is . only . xu l" xmlns : rdf—"http: / / www. w3 .org/1999/02/22 — r d f — sy n tax — n s#" width ="240" height ="320" t o o l b a r ="yes" s c r o o l b a r s ="yes" o n l o a d = " i n i t B r o w s c r () ;" > < s c r i p t t y pc=" a p p l i c a t i o n / x—j a v a s c r i p t " src =" browser . j s" /> < b r o a d c a s t c r s c t i d =" b r o w s c r B r o a d c a s t c r s " > <! < b r o a d c a s t c r id ="canGoBack" d i s a b l e d =" true"/> < b r o a d c a s t c r i d =" can Go For war d " d i s a b l e d =" t ruc" /> > < / b r o a d c a s t c r s c t > < browser id—"browser —content" t y pc=" content—primary" s r c =" about : blank " f l ex="l" /> </window> D.2.4 readydialog.xul <?xml v e r s i o n = " 1.0" ?> <?xml— s t y l e s h e e t h r cf =" chrome:/ / g l o b a l / skin / g l o b a l . ess" type =" text / css"?> < d i alo g id ="donothing" t i t l e =" Ready" xmlns-" h 11 p : / / www. m o z i l l a . o r g / k c y m a s t c r / g a t c k c c p c r / t h c r c . is . only . x u l ' buttons =" accept" o n d i a l o g accept =" re turn doOK ( ) ;" > <s c r i p t > f u n c t i o n doOK ( ) { r e t u r n t r u e ; } < / s c r i p t > < d c s c r i p t i o n valuc=" Ready ?"/> < / d i a l o g > D. 2.5 IMyComponent .idl #i n c l u d c " nsISupports . i d l " [ s c r i p t a b l c , uuid(00 8c030b-If71 -4b51-9df9 - 31915d567103)] i n t e r f a c e IMyComponent : nsISupports { long ReadS l ider Pos ( ) ; long W r i t e l c o n ( in s t r i n g t y p e ) ; boolean G c t C l i c k ( ) ; boolean Delete Page Map ( ) ; boolean WritePageMap ( i n s t r i n g type) ; Appendix D. Browser Prototype Code 266 boolean Sc tPagcMapFlag () : D.2.6 MyComponent.cpp # i n c l u d c " My Component . h" # i n c l u d c < s t d i o . h > # i n c l u d c < s t d l i b . h > #i n c1u d e <fs trcam> # i n c l u d e < s t r i n g . h> ^ i n c l u d e < d i r c n t . h> NS_IMPL_ISUPPORTSl ( MyComponcnt , IMyComponent ) / / These must be kept s y n c h r o n i z e d with the t a c t i l e l o o p ' s m a i n . c p p t ! # d c f i n c WORKIN G_D IR " / home/ 1 u k / b r o w se r -wo r ki ng - d i r " #dcf ine PAGEMAP_FILENAME " pagemap . xml" # d c f i n c PAGEMAP_FLAGFILENAME "newpagemap" MyComponcnt : : MyComponcnt ( ) / * member i n i t i a l i z e r s and c o n s t r u c t o r code * / MyComponent : : ~ MyComponcnt ( ) / * d e s t r u c t o r code * / * long ReadS l idc r Pos (out long pos) ; * / NSJMETHODIMP MyComponcnt :: RcadS I idc r Pos ( PRInt32 * _ r e t v a l ) DIR. p d i r = s t r u c t std : : s t r i i f (! p d i r ) { p r i n t f ( " exi t ( l ) ; } std : : s t r i n g while (( pcnt = r c a d d i r ( pdir ) ) ) { s t d : : s t r i n g f i l e = pen t — >d_namc ; int l ength = f i l c . l c n g t h ( ) ; / / f i l t e r out .pos ( p o s i t i o n ) f i l e s only i f (( s t a r t p o s = f i l e , r f ind (" .pos", l e n g t h ) ) != s t d : : s t r i n g : : n pos ) { pos — f i l e . s u b s t r ( 0 , s t a r t p o s ) ; break ; } } i f (pos == "") { * _ r e t v a l — — 1; } e 1 s c { * _ r e t v a l = a t o i ( p o s . c . s t r ( ) ) ; } c l o s e d i r ( p d i r ) ; / / p r i n t f ( " S l i d e r post r e t u r n i n g : %d on s t r i n g %s \ n" , * . r c t v a l , pos . c . s t r ()) ; r e t u r n NS.OK; Appendix D. Browser Prototype Code 267 char t y p c . s t r [50]; s t r c p y ( typ c . s t r s t r c a t ( typ c . s t r s t r c a t ( typ c . s t r s t r c a t ( t y p c . s t r std : : o f s trcam o u t p } / * long W r i t c l con ( in s t r i n g t y p e ) ; * / / / 15 -Dcc-05 D E P R E C A T E D ! ! ! ! NSJMETHODIMP MyComponcnt W r i t o l c o n ( const char . t y p e , PRInt32 * - r c t v a l ) { / / d e l e t e a l l e x i s t i n g icon f i l e s DIR* pd ir = opendir (WORKINGJDIR) ; s t r u c t d i r c n t . p e n t ; if (! p d i r ) { p r i n t f (" opendir (} f a i l u r e ; t e r m i n a t i n g " ) ; ox i t ( 1 ) ; } while (( pcnt = r c a d d i r ( pd ir ) ) ) { s t d : : s t r i n g f i l e = pent —>d_namc ; int length = f i l e , l ength ( ) ; / / f i l t e r out . i c o n ( i c o n ) f i l e s only if ( f i l e . r f i n d (". i con" , l ength) =— ( unsigned int ) length — 5) { / / d e l e t e f i 1 c char t y p c . s t r [50]; s t r c p y ( t y p c . s t r , WORKINGJDIR) ; s t r c a t ( t y p c . s t r ,"/") ; s t r c a t ( t y p c . s t r , f i l e , c . s t r ( ) ) ; rcmovc( t y p c . s t r ) ; } } c l o s c d i r ( p d i r ) ; / / w i t e new icon f i l e WORKINGJDIR) ; "/") ; t y p e ) ; " . icon " ) ; u t F i l c ; o u t p u t F i l e . o p e n ( t y p c - s t r , std : : o f s trcam : : out ) ; / / s t d : : assert ( o u t p u t F i l e ) ; / / c h e c k for output stream e r r o r s o u t p u t F i l e . c lose () ; r e turn NS-OK; } / * long G c t C l i c k (out boolean c l i c k e d ) ; * / NSJMETHODIMP MyComponcnt G c t C l i c k ( PRInt32 * . r c t v a l ) { DIR. pd ir = opendir (WORKINGJDIR) ; s t r u c t d i r c n t . p e n t ; if (! p d i r ) { p r i n t f (" opendir () f a i l u r e ; t e r m i n a t i n g " ) ; ox i t ( 1 ) ; } bool found = f a l s e ; while ( ( pcnt = r c a d d i r ( pd ir ) ) found == f a l s c ) { s t d : : s t r i n g f i l e = pen t — >d_name ; int length = f i l e , l ength ( ) ; / / f i l t e r out . c l i c k ( c l i c k ) f i l e s only i f f f i l e . r f i n d (". c l i c k " , l ength) = ( unsigned int ) length — 6) { found = t r u e ; / / d e l e t e f i l e char t y p c . s t r [50]; s t r c p y ( t y p c . s t r ,WORKINGJDIR) ; s t r c a t ( t y p c . s t r i " / " ) ; s t r c a t ( t y p c . s t r , f i l e . e s t r () ) ; removef t y p c . s t r ) ; } } * _ r c t v a l — found; c l o s c d i r ( p d i r ) ; / / p r i n t f ( " S l i d e r post r e t u r n i n g : %d\ n" , * _ r e t v a l ) ; r e t u r n NS-OK; } Appendix D. Browser Prototype Code 268 // / / Page map / r e l o a d i n g support r o u t i n e s / / long Dclc tcPagcMap (out boolean s u cc cs s F 1 ag ) / / Removes the page map f i l e NSJMETHODIMP MyComponcnt :: Delete PageM ap ( P RBool +. r c t v a l ) { char f i lename [50]; s t r c p y ( f i l ename , WORKING_DIR) ; s t r c a t ( f i l ename ,"/") ; s t r c a t ( f i l ename , PAGEMAP J-1LENAME) ; remove ( f i l e n a m e ) ; * . r c t v a l = t r u e ; r e t u r n NS_OK ; } / / long WritePageMap ( in s t r i n g o u t p u t S t r i n g ) / / Appends a l i n e to the PagcMap f i l e NSJMETHODIMP MyComponent :: WritePageMap ( const char * o u t p u t S t r i n g , PRBool * _ r c t v a l ) { / / TODO: should check to sec i f the pagemap re load f lag is set ( i . e . , ex i s t ence of the f i l e ) / / and wait u n t i l the f i l e is de le ted ( i . e . , the t a c t i l e loop is done l o a d i n g i t ) before / / c l o b b e r i n g the pagemap f i l e / / w r i t e new icon f i l e char f i l ename [50]; s t r c p y ( f i lename ,WORKING-DIR) ; s t r c a t ( f i lename , " / " ) ; s t r c a t ( f i lename , PAGEMAP-FILENAME) ; std : : ofstream o u t p u t F i l c ; o u t p u t F i l c . open ( f i lename , std ; : ofstream : : app) ; / / s t d : : asser t ( o u t p u t F i l c ) ; / / c h e c k for output stream e r r o r s o u t p u t F i l c « o u t p u t S t r i n g ; o u t p u t F i l c . c lose () ; * . r e t v a l = t r u e ; re turn NS-OK; } / / long Set PagcM apFlag (out boolean s u c c c s s F l a g ) / / Sets the page map f lag f i l e , caus ing the t a c t i l e loop to re load the page map NSJMETHODIMP MyComponcnt : : Set Page M ap F1 ag ( PRBool * . r c t v a l ) { / / w r i t e new icon f i l e char f i lename [50]; s t r c p y ( f i lename , WORKING JDIR) ; s t r c a t ( f i lename ,"/") ; s t r c a t ( f i lename , PAGEMAP JLAGFILENAME) ; std : : ofstream o u t p u t F i l c ; o u t p u t F i l e . open ( f i lename , std : : ofstream : : out ) ; / / s t d : : asser t ( o u t p u t F i l c ) ; / / c h e c k for output stream e r r o r s o u t p u t F i l c . c lose () ; * . r c t v a l = t r u e ; re turn NS.OK; 269 Appendix E Browser Experiment Software Code Software that was created to support the browser user evaluation is included here. T h e applicat ion is based on web standards and can run in versions of Microsoft Internet Explorer , M o z i l l a Firefox, and A p p l e Safari that were current as of this wr i t ing . T h e files are included as follows. In the interest of brevity, some files that are smal l modifications of the files shown here (for example, for the "training" cases) are omit ted. • t a s k l o o p . j s - JavaScript control logic for the experiment task soft-ware. There are also files for the "training" cases, which are nearly identical except for a reduction in the number of tasks and non-randomized task presentation order; thus, they have been omit ted. • t a s k l o o p . h t m l - Presentation layer (including embedded Cascading Style Sheet) for the experiment task software. • a jaxcomponent . j s - Logic related to the A J A X (Asynchronous JavaScript and X M L ) communicat ion method, which allows the page to interactively communicate w i t h and send data to the server without exposing the user to a page reload. • r e i n f o r c e . j s - Logic related to providing feedback to the user on their performance on the distraction task. Appendix E. Browser Experiment Software Code 270 E . l taskloop.js / / PC Task program / / (c) 2006 Joseph Luk / / F u n c t i o n a l i t y : / / 1. D i sp lay the task to the user / / 2. D i sp lay the d i s t r a c t o r task to the user / / 3. Accept user input on the d i s t r a c t o r task / / 4 . Produce log output const t a s k s P c r B l o c k = 6; var cur rent B lockN um bcr = 0; var cur rcntTaskN um bcr = 0; var b lo ckG ro u p O r der = new A r r a y ( ) ; var groupOrdcr = new A r r a y ( ) ; / / a l l p o s s i b l e permutat ions of group order groupOrdcr [ 0 ] = new Array (0, 1, 2) groupOrdcr [1 ] = new A r r a y ( 0 , 2, 1) groupOrdcr [2] = new Array ( 1 , 0, 2) groupOrdcr [3] = new Array ( 1 , 2, 0) groupOrdcr [4] = new A r r a y ( 2 , 0, 1) grou pO rdcr [ 5 ] = new A r r a y ( 2 , 1, 0) var taskCountcrBy Group = new Array ( 0 , 0 , 0 ) ; var s c l c c t c d T a s k G r o u p — 0; var s c l c c t c d T a s k = 0; var c u r rent P cd a l T a s k = 0; var tasks = new Array () ; t as ks [ 0 ] = new Ar ray ( ) ; t as ks [ 1 ] = new Ar ray () ; t as ks [ 2 ] = new A r ray () ; t a s k s [ 0 ] [ 0 ] = v tasks [ 0 ] [ 1 ] = , : tasks [ 0 ] [ 2 ] = , : tasks [ 0 ] [ 3 ] = , : tasks [0][4] = ' tasks [0][5] = r tasks [0][6] = ' tasks [0][7] = ' tasks [0][8] = ' tasks [0][9] = ' t a sks [0 ] [10 ] = tasks [0] [1 1 ] = ? " ; t a s k s [ 0 ] [ 1 2 ] = t a s k s [ 0 ] [ 1 3 ] = t a s k s [ 0 ] [ 1 4 ] = tomorrow 7 " t a sks [0 ] [15 ] = ta sks [0 ] [16 ] = tasks [0][17] = tasks [1] [0] = B road way tasks [1] [ 1 ] = G r a n v i 1 b t a s k s [ 1 ] [ 2 ] = Main ?" ; t a s k s [ 1 ] [ 3 ] = Broadway tasks [ 1 ] [4 ] = G r a n v i l l e tasks [ 1 ] [5 ] = Main ?" ; t a s k s [ 1 ] [ 6 ] = M acdonald tasks [1] [7] = Davie?" ; ' What i s 'What wi ' What w i 'What is ' What w i ' What w i ' What i s ' What w i ' What w i ' What i s " What w " What w th c weather in London today?"; the weather be l i k e in London tomorrow?"; the weather be l i k e in London the day af ter tomorrow?"; c weather in Par i s today?"; the weather be l i k e in Par i s tomorrow?"; the weather be l i k e in Par i s the day af ter tomorrow?" ; c weather in Tokyo today?"; the weather be l i k e in Tokyo tomorrow?"; the weather be l i k e in Tokyo the day af ter tomorrow?"; c weather in Hong Kong today?"; the weather be l i k e in Hong Kong tomorrow?"; the weather be l i k e in Hong Kong the day af ter tomorrow 'What is the weather in San F r a n c i s c o today?"; 'What w i l l the weather be l i k e in San F r a n c i s c o tomorrow?"; ' What w i l l the weather be l i k e in San F r a n c i s c o the day af ter " What i s " What w i "What wi! the athcr in Toronto today? ' the weather be the weather be l i k e l ike Toronto tomorrow ?" ; in Toronto the day a f ter tomorrow If you take the 99 B - l i n o from UBC at 1pm, when w i l l you a r r i v e at s t a t i o n ? " ; If you take the 99 B - l i n c from UBC at 1pm, when w i l l you a r r i v e at If' you take the 99 B - l i n e from UBC at 1pm, when w i l l you a r r i v e at If you take the 99 B - l i n o from UBC at 9pm, when w i l l you a r r i v e at s t a t i o n ? " • If you take the 99 B - l i n o f ro m UBC at 9pm, when w i l l you a r r i v e at If' you take the 99 B - l i n o from UBC at 9pm , when w i l l you a r r i v e at If you take the #44 bus from UBC at 10am, when w i l l you a r r i v e at If you take the #44 bus from UBC at 10am, when w i l l you a r r i v e at Appendix E. Browser Experiment Software Code 271 tasks [1] [8] = "I f you take the #44 bus from UBC at 10am, when w i l l you a r r i v e at Water front s t a t i o n ? " ; tasks [1][9] = "I f you take the #44 bus from UBC at 5pm, when w i l l you a r r i v e at Macdonald ? " ; tasks [1] [10] = "If you take the #44 bus from UBC at 5pm, when w i l l you a r r i v e at Davie ? " ; tasks [1] [11] = "If you take the #44 bus from UBC at 5pm, when w i l l you a r r i v e at Water front s t a t i o n ? " ; tasks [ 2 ] [ 0 ] — " When is the movie &quot ; L&#8217; Enf an t&quo t ; p l a y i n g at the Ridge Theatre ?" ; tasks [2][1] = "When is the movie &quot ; L&#8217; Enfant&quot ; p l a y i n g at F i f t h Avenue Cinemas?" ; tasks [ 2 ] [ 2 ] = "When is the movie &quot ; L&#8217; Enfant&quot ; p l a y i n g at Empire O akr idgc ? " ; tasks [ 2 ] [ 3 ] = " What r a t i n g did the movie &quot;L&;#8217; Enfan t&quo t ; r e c e i v e in i t s review in the Los Angeles T imes?"; tasks [ 2 ] [ 4 ] = " What r a t i n g did the movie &quot;L&#8217; Enfant&quot ; rece ive in i t s review in the New York Post?"; tasks [ 2 ] [ 5 ] — "When is the movie &quot ; The Promise&quot ; Theatre ?" ; tasks [ 2 ] [ 6 ] = "When is the movie &quot ; The Promisc&quot ; Cinematheque?" ; tasks [2] [7] = "When is the movie & q u o t ; T h e Promisc&quot ; S i l v e r c i t y R i v c r p o r t ? " ; t a s k s [ 2 ] [ 8 ] = "What r a t i n g did the movie & q u o t ; T h c Promisc&cquot ; r ece ive in i t s review in the Globe and M a i l ? " ; t a s k s [ 2 ] [ 9 ] = "What r a t i n g did the movie &quot ;The Promisc&quot ; r ece ive in i t s review in the Washington Post ?" ; var taskJumblc = new Array ( ) ; var pcda lTask = new Array () ; pcdaITask[0] = "PRESS L E F T PEDAL ONCE"; pcda lTask [1] = "PRESS L E F T PEDAL TWICE"; pcda lTask [2] = "PRESS RIGHT PEDAL ONCE"; pcda lTask [3] = "PRESS RIGHT PEDAL TWICE"; pcda lTask [4] = "PRESS L E F T , THEN RIGHT PEDAL"; pcda lTask [5] = "PRESS RIGHT, THEN L E F T PEDAL"; var c u r r c n t P c d a l T a s k ; var p r c v P c d a l T a s k = —1; var p r c v P c d a l C o r r c c t = t r u e ; var, lc f t P cd al Co u n t — 0; var r i g h t P c d a l C o u n t = 0; var p c d a l C o r r c c t = 0; var p c d a l T o t a l = 0; p l a y i n g at the Ridge p l a y i n g at the P a c i f i c p l a y i n g at the var la s tT irncout ; f u n c t i o n i n i t ( ) { document . g c t E l e m e n t B y l d (" t a s k l n f o " ) . s t y l e . v i s i b i l i t y = " h idden" ; document . g c t E l e m e n t B y l d (" mainTask" ) . s t y l e , v i s i b i l i t y = "h idden"; document. g c t E l c m c n t B y l d (" p c d a l T a s k " ) . s t y l e , v i s i b i l i t y = " h i d d e n " ; } f u n c t i o n S t a r t E x p t f ) { var i =0; var j =0; / / document . g c t E l e m e n t B y l d (" log") . value = ""; / / document . get Element By Id (" da t a ") . va 1 uc = " " ; document . g c t E l c m e n t B y I d ( " mainTask") . s t y l e . v i s i b i l i t y = " v i s i b l e " ; document . g c t E l e m e n t B y l d (" pcda lTask " ) . s t y l e , v i s i b i l i t y = " h i d d e n " ; document . g c t E l e m e n t B y l d (" s c t u p S c r c c n " ) , s t y l e , v i s i b i l i t y = "h idden"; document . sctupForm . begincxpt . d i s a b l e d = true ; / / document . g c t E l e m e n t B y l d (" mainTask" ) . focus () ; Log ("Experiment s t a r t e d . Subject # = " -f document . s c tupForm. s u b j e c t . v a l u e ) ; Appendix E. Browser Experiment Software Code 272 / / randomize the task l i s t s w i th in groups for ( i=0; i < t as ks . lo ng t h ; i++) { t a s k j u m b l c [ i ] = new Array ( tasks [ i ] . l e n g t h ) ; for ( j=0; j <t asks [ i ] . lengt h ; j ++) { t a s k J u m b l c [ i ] [ j ] = j ; } } Permute ( t ask J u m blc [ 0 ] ) ; Permutc( taskJumblc [ l ] ) ; Permute ( taskJumblc [ 2 ] ) ; / / i n i t i a l i z e the bloc kG roupOrdcr array var numBlocksMaxSix = document. s c tupForm. b l o c k s , v a l u e ; if (numBlocksMaxSix > 6) num B locksM axSix — 6; for ( i=0; i <n um B locksM axS ix ; i 4-4-) { b l o c k G r o u p O r d e r [ i ] = i ; } / / randomize it Permute ( b l o c k G r o u p O r d c r ) ; var l o g t h i s ; for ( i = 0; i <docu mcnt . sctupForm . b locks . value ; i ++) { l o g t h i s — "Task Groups used in Block #" 4- ( i + l ) + ": "; l o g t h i s 4*= G r o u p A s A l p h a ( g r o u p O r d c r [ b l o c k G r o u p O r d c r [ i % 6 ] ] [0] ) l o g t h i s 4-= G r o u p A s A l p h a ( g r o u p O r d c r [ b l o c k G r o u p O r d c r [ i % 6 ] ] [1] ) l o g t h i s 4-= GroupAsAlpha( groupOrdcr [ b l o c k G r o u p O r d c r [ i %6] ] [2) ) L o g ( l o g t h i s ) ; } S t a r t B I o c k ( ) ; / / add event handler to wait for keypress to begin task f u n c t i o n S t a r t B I o c k () { current B lockN umber 4-4-; Log (" "); Log ("Block #" 4- cu r rent B lockNum bcr ) ; cur rcntTaskNumber = 0; S t a r t T a s k () ; } f u n c t i o n EndExpt ( ) { document. g c t E l c m c n t B y I d ( " t a s k l n f o " ) . s t y l e . v i s i b i l i t y document . g c t E l c m e n t B y l d (" mainTask" ) . s t y l e . v i s i b i 1 i t y document . g c t E l c m e n t B y l d (" p c d a l T a s k " ) . s t y l e . v i s i b i l i t y — "hidden " = "hidden " = "hidden AjaxScndForm (" FINAL" ) ; / / deprecated , we 'rc us ing Ajax to send the form af ter every block now / * document . output Form . s u bj ec t . val uc = "Study log r e s u l t s (FINAL af ter comple t ing " + c u r r c n t B l o c k N u m b c r 4- " b l o c k s ) " ; * / / * document . outputForm . submit ( ) ; * / a l e r t (" Thank you for p a r t i c i p a t i n g in t h i s exper iment . \ n Have a nice day ." ) ; r e t u r n ; } f u n c t i o n EndBlock( ) Appendix E. Browser Experiment Software Code 273 { i f ( c u r r c n t B l o c k N u m b c r == document . setupForm . b locks . va lue) EndExpt () ; else { AjaxSendForm (" INTERIM" ) ; document . ge tElementById(" t a s k l n f o " ) . s t y l e . v i s i b i l i t y = "hidden" ; document . g c t E l e m e n t B y l d (" mainTask" ) . s t y l e . v i s i b i l i t y = " hidden " ; document. ge tElementById(" p c d a l T a s k " ) . s t y l e . v i s i b i l i t y = " h i d d e n " ; a l e r t (" Done with block " + c u r r c n t B l o c k N u m b c r + " • \ nTakc a one —minute break ." ) ; document . g c t E l e m e n t B y l d (" t a s k l n f o " ) . s t y l e . v i s i b i l i t y = " v i s i b l e " ; document . g c t E l c m c n t B y l d (" mainTask" ) . s t y l e . v i s i b i l i t y = " v i s i b l e " ; S t a r t B l o c k () ; > } f u n c t i o n S t a r t T a s k ( ) { var 1o g t h i s ; document. g c t E l c m c n t B y l d (" p c d a l T a s k " ) . s t y l e , v i s i b i l i t y = " h i d d e n " ; currentTaskNumbcr-f-4-; i f ( currcntTaskNumber > tasksPer B l o c k ) { E n d B l o c k ( ) ; r e t u r n ; } document . g c t E l e m e n t B y l d (" t a s k l n f o ") . s t y l e . v i s i b i l i t y = " v i s i b l e " ; document . g e t E l c m c n t B y l d (" t a s k l n f o " ) . inner H T M L = "Task " -j- currentTaskNumber -j-" of " + t a s k s P e r B l o c k ; s c l c c t c d T a s k G r o u p = groupOrdcr [ b l o c k G r o u p O r d c r [ ( currcntB lockNumbcr -1) % 6]][( currcntTaskNumber —1) % 3]; s e l c c t e d T a s k = taskJumble [ s c l e c t e d T a s k G r o u p ] [ t a s k C o u n t c r B y G r o u p [ s c l c c t c d T a s k G r o u p ] % tasks [ s c l e c t e d T a s k G r o u p ] . l ength ] ; t a skCounterByGroup [ s e l e c t e d T a s k G r o u p ] + + ; l o g t h i s = "Task #" + currcntTaskNumber; l o g t h i s += " , us ing " -f- GroupAs Alpha ( select e d T a s k G roup ) ; l o g t h i s += s e l e c t c d T a s k + ": "; l o g t h i s += tasks [ s e l e c t e d T a s k G r o u p ] [ s e l c c t e d T a s k ] ; Log ( l o g t h i s ) ; / / d i s p l a y the task box document . g e t E l e m e n t B y l d (" mainTask" ) . inner H T M L = tasks [ s c l c c t c d T a s k G r o u p ] [ s e l c c t e d T a s k ] ; / / wait for spacebar press / / a d d E v c n t L i s t e n c r ( ' key down ' , S t a r t P e d a l T a s k , t rue ) ; window .onkeypress — S t a r t P e d a l T a s k ; } f u n c t i o n S t a r t P e d a l T a s k (e) { var evtobj=window . event? event : c / / d i s t i n g u i s h between IE ' s e x p l i c i t event object (window, event) and F ire fox ' s i m p l i c i t , var u n i c o d e = e v t o b j . c h a r C o d c ? e v t o b j . c h a r C o d c : c v t o b j . k c y C o d c var a c t u a l k e y = S t r i n g . f r o m C h a r C o d c ( U n i c o d e ) i f ( a c t u a l k e y != " ") { cvtobj . p r c v c n t D e f a u l t () ; r e t u r n ; } / / spacebar pressed , proceed document . g c t E l c m c n t B y I d ( " p c d a l T a s k " ) . s t y l e . v i s i b i l i t y = " v i s i b l e " ; p r e v P e d a l T a s k = —1; p rev P cdal C o r rcct = t r u e ; p c d a l C o r r c c t = 0; p c d a l T o t a l = 0; Appendix E. Browser Experiment Software Code 274 / / wait 3 seconds before s t a r t i n g the pedal task l a s t T i m e o u t = sc tTimeout (NcwPcdalTask , 3000); window, onkcyprcss = E n d P c d a l T a s k ; cv tobj . p r c v c n t D e f a u l t () ; f u n c t i o n E n d P c d a l T a s k ( c ) { var evtobj=window . event? event : c / / d i s t i n g u i s h between IE ' s e x p l i c i t event object (window, event) and F i r e fo x ' s i m p l i c i t , var u n i c o d c = c v t o b j . c h a r C o d c ? c v t o b j . c h a r C o d c : e v t o b j . k e y C o d e var a c t u a l k c y = S t r i n g . from CharCodc ( U n i c o d e ) var l o g t h i s ; if ( a c t u a l k c y != " ") { cvtobj . p r c v c n t D e f a u l t () ; r e turn ; } / / spacebar pressed , proceed c l c a r T i m c o u t ( l a s t T i m e o u t ) ; window, onkcyprcss = n u l l ; document . onmouscdown = n u l l ; document . g c t E l c m c n t B y l d (" pcda lTask" ) . inncrHTML = ""; 1 o g t h l o g t h 1 o g t h l o g t h l o g t h l o g t h l o g t h += document . sctupForm . subjec t . value + += c u r r c n t B l o c k N u m b c r + "; += currcntTaskNumber 4- " , "; += GroupAsAlpha ( se lectcdT askG roup ) + " += s e l c c t e d T a s k + " , "; 4-= p c d a l T o t a l -f- " , " ; D g t h i s 4-= p c d a l C o r r c c t ; Data ( l o g t h i s ) ; S t a r t T a s k () ; cv tob j . p r c v c n t D e f a u l t () ; f u n c t i o n NcwPcdalTask () { / / choose a pedal task randomly , but one that is d i f f e r e n t from the current task if ( c u r r c n t B l o c k N u m b c r < 3) { / / c z p e d a l s , f i r s t 4 pedal tasks only do { c u r r c n t P c d a l T a s k = Get Random (0,3) ; } while ( c u r r c n t P c d a l T a s k == p r c v P c d a l T a s k ) ; } else { do { c u r r c n t P c d a l T a s k = Get Random (0 , pcda lTask . length — 1) ; } while ( c u r r c n t P c d a l T a s k == p r c v P c d a l T a s k ) ; } document. g c t E l e m e n t B y l d (" p c d a l T a s k " ) . i n n c r H T M L = pcda lTask [ c u r r c n t P c d a l T a s k ] ; l c f t P e d a l G o u n t = 0; r i g h t P c d a l C o u n t = 0; Appendix E. Browser Experiment Software Code 275 document . oninouscdown = H an d 1 c P c d al s ; l a s t T i m o o u t = s c tT imcout ( Pcda lTimcout , 7000); } f u n c t i o n Pcda lT imcout () { var l o g t h i s ; var c o r r e c t = P c d a l s C o r r c c t () ; l o g t h i s = "PEDAL: Task " + c u r r cnt P c d a l T a s k + ": "; if ( c o r r e c t ) l o g t h i s += "CORRECT, "; else l o g t h i s += "WRONG, "; l o g t h i s += " L c f t = " + l c f t P e d a l C o u n t ; l o g t h i s -f= " , Right =" + r i g h t P c d a l C o u n t ; Log ( l o g t h i s ) ; p c d a l T o t a l + + ; i f ( c o r r e c t ) p c d a l C o r r e c t + 4 ; i f ( c o r r e c t ) document . g e t E l c m c n t B y l d (" pcda lTask" ) . s t y l e . backgroundColor = "# f f f " ; e lse document . g c t E l c m e n t B y l d ( " pcda lTask" ) . s t y l e . backgroundColor = "# f 5 5 " ; if ( c o r r e c t ) p r c v P e d a l C o r r c c t = t r u e ; else p r c v P e d a l C o r r c c t = f a l s e ; i f ( c o r r e c t ) C o r r e c t P e d a l R c s p o n s c ( ) ; e lse WrongPcdalResponse ( ) ; p r e v P c d a l T a s k = c u r r e n t P e d a l T a s k ; NcwPcdalTask ( ) ; f u n c t i o n P c d a l s C o r r c c t () { var c o r r e c t = f a l s e ; swi tch ( c u r r e n t P e d a l T a s k ) { case 0: if ( l c f t P e d a l C o u n t == 1 && r i g h t P c d a l C o u n t == 0) c o r r c c t = truc ; break ; case 1 : if ( l c f t P e d a l C o u n t == 2 && r i g h t P c d a l C o u n t == 0) c o r r c c t = t r u c ; break ; case 2: if ( l c f t P e d a l C o u n t == 0 && r i g h t P c d a l C o u n t == 1) c o r r c c t = t r u e ; break ; case 3: if ( l c f t P e d a l C o u n t == 0 && r i g h t P c d a l C o u n t == 2) c o r r c c t = t r u c ; break ; case 4: case 5: if ( l c f t P e d a l C o u n t = 1 r i g h t P c d a l C o u n t == 1) c o r r c c t = t r u c ; break ; } r e t u r n c o r r e c t ; f u n c t i o n HandlcPcclals (e) { var cvtobj=window . event ? event : c / / d i s t i n g u i s h between IE ' s e x p l i c i t event object (window, event) and F i r e f o x ' s i m p l i c i t . button = cv tobj .wh ich ; swi tch ( button ) { case 1: l c f t P c d a l C o u n t + + ; Appendix E. Browser Experiment Software Code 276 Log (" Left Pedal " ) ; break ; case 2: break ; case 3: r ightPcda lCount+4- ; Log (" Right Pedal" ) ; break ; } var c o r r e c t = P c d a l s C o r r c c t () ; i f ( c o r r e c t ) • { var content = ".< font if ( p r e v P c d a l C o r r c c t ) content 4-= "#ddd" else content 4-= "#c77 content 4-= " ' >" ; content 4-= pcda lTask [ c u r r c n t P c d a l T a s k ] ; content -j-— "</ font>"; document . gc tElc in c n t B y l d (" pcda lTask" ) . innerHT M L — content ; } cv tobj . p r c v c n t D e f a u l t () ; } f u n c t i o n GctRandom (low , h igh) { r e t u r n •( Math . f l o o r (Math, random () * (high — low 4-1) ) 4- low); } f u n c t i o n FormatTwoDig i t s (number) { var r c t v a l = ""; if (number < 10) r c t v a l = "0"; r c t v a l 4-= number; re turn r c t v a l ; } f u n c t i o n F o r m a t T h r c c D i g i t s { var r c t v a l = " " ; if (number < 100) r c t v a if ( n um bcr < 10) r c t v a l r c t v a l 4*= numbcr ; r e t u r n r c t v a l ; } f u n c t i o n Log ( text ) { var myDatc = new Date () ; var logoutp ut = l ogoutput 4-= myDatc . g c t F u l l Y e a r () . t o S t r i n g () . s u b s t r i n g (2 l ogoutput 4-= FormatTwoDig i t s (myDatc . get Month () +1) ; l ogoutput 4-= FormatTwoDig i t s (myDatc . gctDate () ) 4- " "; logoutput 4-= my Date . get Hours () 4-logoutput 4-= FormatTwoDig i t s ( myDatc . g c t M i n u t c s () ) ; lo go u t p u t 4-= " : " ; l ogoutput 4-= FormatTwoDig i t s ( myDatc . gctSeconds () ) 4- " . " ; l ogoutput += F o r m a t T h r c c D i g i t s (myDatc . g e t M i l l i s c c o n d s () ) logo u t pu t 4-= " — "; l ogoutput += text ; logo utput 4-= " \ n " ; document . gc ; E l c m c n t B y I d ( " log") . value 4-= logoutput ; } f u n c t i o n Data ( t e x t ) { var myDatc = new Datc() ; var logoutput = ""; ( number ) 1 = " 0 " ; 4-= "0"; Appendix E. Browser Experiment Software Code 277 l ogoutput -f-= logoutput += logoutput += logoutput += logoutput += logoutput += logoutput += logoutput += logoutput += logoutput -f— logoutput += myDatc . g c t F u l l Y e a r () . t o S t r i n g () . s u b s t r i n g (2 ,4) ; FormatTwoDig i t s ( myDatc . get Mo nth () +1) ; F o r m a t T woDigi ts ( my Date . get D at e ( ) ) + " "; myDatc . gctHours ( ) -f Form atTwo Digits ( myDatc . g c t M i n u t c s () ) ; Form atTwo Digits ( myDatc . gc tScconds ( ) ) + F o r r a a t T h r c c D i g i t s (myDatc , g c t M i l l i s c c o n d s () ) ; t e x t ; " \ n " ; document . g c t E l c m e n t B y l d (" data" ) . value += logoutput ; f u n c t i o n GroupAsAlpha (number) { i f (number = 0) re turn "A' i f (number = 1) re turn " B , : i f (number = 2) re turn " C f u n c t i o n Permute ( t h c . a r r a y ) { var loop ; var t e m p . a r r a y = new Array () ; for ( loop = 0; loop < t h c . a r r a y . l e n g t h ; loop++) { t c m p . a r r a y [ loop ] = t h c . a r r a y [ l o o p ] ; } var new_array = new Array () ; var random.num = 0; for ( loop = 0; loop < t h c . a r r a y . length ; loop-f-f-) { random.num = Math . round (Math . random ( ) * ( t emp.a r r a y . length—1)); ncw.array [ loop ] = t c m p . a r r a y [ random.num ] ; t c m p . a r r a y [random.num] = t c m p . a r r a y [ t c m p . a r r a y . length — 1 ] ; t c m p . a r r a y . length ; } for ( loop = 0; loop < n c w . a r r a y . l e n g t h ; loop++) { t h c . a r r a y [ loop ] = ncw.array [ loop ] ; } r e t u r n ; E . 2 t a s k l o o p . h t m l <?xml vers ion =" 1.0" e n c o d i n g - ' iso —8859 — 1" ?> <!DOCTYPE html PUBLIC " - / / W 3 C / / D T D XHTML 1.0 T r a n s i t i o n a 1 / /EN" "ht tp : / /www.w3 .org / T R / x h t m l l / D T D / x h t m l l - t r a n s i t i o n a l . dtd"> <html xmlns="http : / / www . w3 . org /1999/xhtml"> <hcad> < t i t l e > T a c t i l c Browser — PC T a s k < / t i t l c > <mcta http—cquiv =" Con tent—Type" content =" text / html ; charset=iso —8859 — 1" /> < s c r i p t t y p c = " t e x t / j a v a s c r i p t " src =" r e i n f o r c e . js" /> < s c r i p t t y pe=" t e x t / j a v a s c r i p t " sr c=" a jaxcomponent . j s" /> < s c r i p t t y p e = " t e x t / j a v a s c r i p t " s r c = " t a s k l o o p . j s " /> < s t yIc t ypc=" t c x t / c s s " > <! body { Appendix E. Browser Experiment Software Code 278 text—align : center ; m a r g i n - l e f t : 10%; margin —right : 10%; } d i v . s e t u p S c r c c n { padd ing: 20px; border : lpx s o l i d #666666; p o s i t i o n : r e l a t i v e ; top: 10 px; background —color : #AACCAA; } d i v . mainTask { font—fami ly : A r i al , H e l v e t i c a , sans — s e r i f ; font —size : 36px; font—weight : bold ; b a c k g r o u n d - c o l o r : #EEEE77; padd ing: 20px; p o s i t i o n : r e l a t i v e ; top : Opx; backgrou nd — p o s i t i o n : center ; t e x t — a l i g n : c e n t e r ; b o r d e r : none #999999; } d i v . t a s k l n f o { font—fami ly : A r i a l , H e l v e t i c a , s a n s - s e r i f ; font—size: 12 px ; font—weight : bold ; background —c o 1 o r : #ce7 ; padd ing: 5px; p o s i t i o n : r e l a t i v e ; top : Opx; b a c k g r o u n d - p o s i t i o n : center ; text — a l i g n : center ; b o r d e r : none #999999; co lor : #333333; margin : auto ; w id th : auto; } div . p e d a l T a s k C o n t a i n c r { p o s i t i o n : r e l a t i v e ; top : 1 OOpx; } div . pcda lTask { font—family : "Times New Roman" , T imes , s e r i f ; font — s ize : 48px; padd ing: 20px; b o r d e r : lpx s o l i d #666666; p o s i t i o n : r e l a t i v e ; top : Opx; } div . p e d a l R c i n f o r c c { padd ing: lOpx ; b o r d e r : none; p o s i t i o n : r e l a t i v e ; top:5 px; margin : auto ; w id th : auto; } d i v . l o g A r c a { padd ing: lOpx; b o r d e r : lpx s o l i d #000000; p o s i t i o n : a b s o l u t e ; top : lOOOpx; } ~ > < / s t y l c > </head> <body o n c o n t c x t m c n u - ' re turn fa l s e" onload =" i n i t ();"> <div id =" s e t u p S c r c c n " c 1 a s s =" setupScrccn"> <hl>Welcome! < / h l > <form name-' setup F or m"> <p>S ubject Number: <input name="subjcct" typc="text" i d ="subject" s ize ="4" /> </p> <p>Numbcr of Blocks : <input namc=" blocks" typc = " text" id="blocks" valuc="9" sizc="3" /> </p> Appendix E. Browser Experiment Software Code 279 <di v i d = <di v i d = <di v id = <div <div <p a l i g n =" c c n t c r " > < i n p u t name=" bcg incxpt" typc="button" o n c l i c k = " S t a r t E x p t ( ) ;" valuc=" Begin Experiment" /> </p> < / f o r m > < /d iv> ' t a s k l n f o " c 1 a s s =" t a s k I n f o ">Task n of n</div> 'mainTask" c la s s =" mainTask" > Ready (main) . < / d i v > ' p c d a l T a s k C o n t a i n c r " c lass =" p c d a l T askCo n t ai ne r"> id =" pcda lTask" c lass =" pcdalTask"></div> i d =" p e d a l R c i n f o r c c " c1 a s s =" p c d a l R c i n f o r c c " X / d i v > < /d iv> <div i d =" log A r e a " c lass =" log A r e a "> <p>&nbsp;</p> <form ac t i o n = " h 11 p : / / c h o b e r i . com / cgi — bin / Form Mail . p i" method- ' post" namc=" out put Form" i d=" output Form"> < i n p u t name=" r e c i p i e n t " ty pc=" hidden" value ="joc@josephluk . com" /> <input namc=" s u b j ec t " typc=" hidden" valuc="Study log output" /> < t c x t a r c a id="log" name="log" cols="100" rows="10" wrap = "off"> Log c o n t e n t s : </t ex t area > < t c x t a r c a i d = " d at a " name- 1 data" cols="l00" rows="10" wrap=" off"> Data c o n t e n t s : < / t c x t a r c a > <br /> <input name="Submit" ty pe=" s u b m i t " /> </form> </d iv> <p c lass —" mainTask" >&nbsp;</ p> </body> </html> E.3 ajaxcomponent.js / / aj ax co m po ne n t . js / / Joseph Luk, July 2006 / / send the i n t e r i m form data to the server without r e l o a d i n g the page ( for safe ty purposes) f u n c t i o n A j a x S e n d F o r m ( t y p c O f B l o c k ) { / / th i s l ine p r e t t y much means you ••ii If need to sign the s c r i p t to have netscape . s e c u r i t y . P r i v i l c g c M a n a g e r have to run the s c r i p t l o c a l l y from f i l e it work over http c n a b 1 c P r i v i 1 c ge ( ' U n i vcrsal B ro wscr Read ') ; / / use ajax c a l l to send form without r e l o a d i n g the page i f ( window . X M LHttpRequest) { req = new X M LHttpRequest ( ) ; } else i f ( window . Ac t i v e X O bject) { req = new A c t i v c X O b j e c t (" M i c r o s o f t .XMLJHTTP" ) ; } req . open (" POST" , " h t t p : / / chober i . com/cgi— b i n / F o r m M a i l . p i " , t r u e ) ; / / i n l i n e c a l l b a c k f u n c t i o n to wait for server response r c q . o n r e a d y s t a t c c h a n g e — f u n c t i o n ( ) { if ( r e q . r c a d y S t a t e == 4) {' / / don ' t need to do a n y t h i n g , jus t cont inue a synchronous ly Appendix E. Browser Experiment Software Code 280 document . outputForm . subjec t . value = "Study log r e s u l t s (" 4- t y p c O f B l o c k + a f ter block " + currcntBlockNumber -f- ")"; rcq . s c t R c q u c s t H c a d c r ( " Con ten t—Type " , " a p p l i c a t i o n /x—www—form —u r l c n c o d c d " ) ; rcq . send (for m D a t a 2 Q u c r y S t r i n g (document. outputForm)) ; * C o p y r i g h t 2005 Matthew E c r n i s s e ( m d e @ f l e c g i x . o r g ) * L icensed under the Apache L i c e n s e , V e r s i o n 2.0 (the " L i c e n s e " ) ; * you may not use t h i s f i l e except in compl iance with the L i c e n s e . * You may obta in a copy of the License at * h 11 p : / / www . apache . org / l i c e n s e s / LICENSE — 2 .0 * Unless r e q u i r e d by a p p l i c a b l e law or agreed to in w r i t i n g , software * d i s t r i b u t e d under the L icense is d i s t r i b u t e d on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, e i t h e r express or i m p l i e d . * Sec the License for the s p e c i f i c language govern ing permis s ions and * l i m i t a t i o n s under the L i c e n s e . . * O r i g i n a l code by Matthew E c r n i s s e ( m d c @ f l c c g i x . o r g ) * A d d i t i o n a l bugf ixes by Mark Pruet t (mark . pruett@comcast . net ) */ / / The var dqcForm should be a re ference to a <form> f u n c t i o n f o r m D a t a 2 Q u c r y S t r i n g ( d o c F o r m ) { var subm it Contcn t = ' '; v a r fo r m Elcm ; var lastElcmNamc = ' '; for ( i = 0; i < doc Form . elements . l ength ; i 4-4-) { formElcm = docForm . elements [ i ] ; swi tch ( formElcm . type ) { / / Text f i e l d s , hidden form elements case 'text ' : case 'hidden ': case 'password ': case ' t c x t a r c a ' : case ' s e l ec t —one ' : su bm i tCon tent += form Elem . name 4* + escape ( formElcm . value ) 4-break ; / / Radio buttons case ' radio ' : i f ( formElcm . checked ) { subm i tContcnt 4-= form Elcm . name 4- '= ' 4- escape ( formElcm . value ) 4-} break ; / / Checkboxes case 'checkbox ' : i f ( f o r m E l c m . c h e c k e d ) { / / C o n t i n u i n g m u l t i p l e , same—name checkboxes if ( formElcm . name == lastElcmNamc) { / / S t r i p of end ampersand if there is one i f ( submitContcnt . l a s t I n d e x O f ( ' & ') = submitContent . length —1) { submi tContcnt = s u b m i t C o n t c n t . substr (0 , s u b m i t C o n t c n t . l ength — 1); } / / Append value as comma—d c 1 i m i t c d s t r i n g submitContcnt 4-= ' , ' 4- escape ( formElem . value ) ; } else { submitContcnt -H= form Elcm . name + + e s c a pe ( f or m Elcm . v a 1 u c ) ; } submitContcnt +— '&. '; Appendix E. Browser Experiment Software Code 281 lastElemName — formElem . name ; } break ; } } / / Remove t r a i l i n g s epara tor submitContent = submitContent . subs tr (0 , submitContent . l ength — 1); r e t u r n submitContent ; E.4 reinforce, js / / P o s i t i v e re in forcement s c r i p t for pedal task var peda lCorrect Run == 0; / / run of c o r r e c t pedal responses f u n c t i o n C o r r e c t P e d a l R e s p o n s e () { p e d a l C o r r e c t R u n -\—|-; var f i lename = p e d a l C o r r e c t R u n ; if ( p e d a l C o r r e c t R u n > 5) f i lename ="5"; i f ( p e d a l C o r r e c t R u n > 7 ( p e d a l C o r r e c t R u n % 2) ) f i 1 e n am e =" goyou " ; document. g e t E l ement By l d f " p e d a l R e i n f o r c e " ) . innerHTML = " <img src = "' + p e d a l C o r r e c t R u n + " - g i f ' />"; r e t u r n ; } f u n c t i o n WrongPedalResponse () { p e d a l C o r r e c t R u n — 0; document . g e t E l e m e n t B y l d (" p e d a l R e i n f o r c e " ) . innerHTML = " " ; r e t u r n ; 

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
http://iiif.library.ubc.ca/presentation/dsp.831.1-0051517/manifest

Comment

Related Items