UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Using haptics to address mobile interaction design challenges : prototyping and user evaluation with… Luk, Joseph Kurachi 2006

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata

Download

Media
831-ubc_2006-0555.pdf [ 17.06MB ]
Metadata
JSON: 831-1.0051517.json
JSON-LD: 831-1.0051517-ld.json
RDF/XML (Pretty): 831-1.0051517-rdf.xml
RDF/JSON: 831-1.0051517-rdf.json
Turtle: 831-1.0051517-turtle.txt
N-Triples: 831-1.0051517-rdf-ntriples.txt
Original Record: 831-1.0051517-source.json
Full Text
831-1.0051517-fulltext.txt
Citation
831-1.0051517.ris

Full Text

Using Haptics to Address Mobile InteractionDesign  Challenges  Protoyping and User Evaluation with a Handheld Tactile Display  •  by  Joseph Kurachi Luk B.S., Cognitive Science, University of California - San Diego, 1999  A THESIS SUBMITTED IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE DEGREE OF Master of Science in The Faculty of Graduate Studies (Computer Science)  The University Of British Columbia July 2006 © Joseph Kurachi Luk, 2006  11  Abstract C u r r e n t user interfaces for mobile and handheld c o m p u t i n g platforms p r i n cipally offer user interaction t h r o u g h the visual and a u d i t o r y modalities. However, mobile devices are often used i n contexts where vision and hearing are i m p a i r e d . A t the same time, more and more functionality is being layered u p o n mobile devices, while the physical size of the display and keypad has remained s m a l l . T h i s limits the rate of information that can be exchanged between the user and the system, and poses an interaction design challenge. H a p t i c s offers a potential solution by p r o v i d i n g an a d d i t i o n a l m o d a l i t y that is also especially well-suited to the demands of portable, personal devices that are i n contact w i t h the user's skin.  In this work we identify ways that interaction through the sense of touch can enhance mobile user interfaces. W e describe the synergistic process of design of user interaction concepts and novel handheld tactile display hardware based on the principle of piezoelectric actuated lateral s k i n stretch. F o l l o w i n g the realization of the prototype hardware, we performed percept u a l characterization studies to determine the expressive capabilities of the new device i n the hands of a h u m a n user.  Informed by the results from  the i n i t i a l user studies, we b u i l t and tested a handheld browser applicat i o n w i t h tactile enhancement. T h e results of user testing w i t h the browser  Abstract  m  a p p l i c a t i o n suggest that the current implementation of directional tactile s t i m u l a t i o n alone is not sufficient to enhance performance  (task time) i n  spatial navigation; however, the user study also brought to light some encouraging qualitative feedback and ways to improve the interaction design and haptic feedback.  B y conducting a full iteration of a user-centred design process i n haptics, we have provided a case study to inform future development  efforts,  a flexible p l a t f o r m for prototyping, and an i n d i c a t i o n of promising future directions for using haptics to solve mobile interaction design challenges.  Joseph K u r a c h i L u k  iv  Contents Abstract  ii  Contents  iv  L i s t of Tables  xii  L i s t of F i g u r e s  xiii  Glossary  xvi  Acknowledgements  xix  1  Introduction  1  1.1  P r o b l e m s w i t h C o n t e m p o r a r y M o b i l e Interaction D e s i g n . . .  2  1.1.1  Sensory B a n d w i d t h L i m i t a t i o n  2  1.1.2  E n v i r o n m e n t a l C o m p e t i t i o n for V i s u a l and A u d i t o r y A t t e n t i o n a l Resources  2  4  1.2  Thesis Research Questions  7  1.3  Thesis A p p r o a c h and O v e r v i e w  8  Related W o r k  12  2.1  M o b i l e Interface Design Challenges  12  2.2  H a p t i c A u g m e n t a t i o n for M u l t i m o d a l Enhancement  13  Contents  3  v  2.3  Handheld Haptics  14  2.4  Perceptual E v a l u a t i o n of H a p t i c Devices  16  Design Concepts  18  3.1  E l e c t r o n i c B o o k Reader w i t h V i b r o t a c t i l e Feedback  19  3.2  E a r l y 1-D N a v i g a t i o n Concepts  21  3.3  A p p l i c a t i o n Concepts  26  3.3.1  L i s t selection: R i n g e r mode a p p l i c a t i o n  28  3.3.2  Scrolling: Browser a p p l i c a t i o n  31  3.3.3  D i r e c t i o n signalling: Assisted navigation a p p l i c a t i o n  .  33  3.3.4  D i s p l a y of background status i n f o r m a t i o n a n d alerts  .  34  3.3.5  M i n i m a l l y Intrusive Interface for R i c h N a v i g a t i o n of Music  4  35  Handheld Prototype Development  39  4.1  40  Design P h i l o s o p h y 4.1.1  L i n e a r slide-mounted tactile display using piezoelectric actuated lateral s k i n stretch  40  4.1.2  Use of off-the-shelf hardware components  41  4.1.3  H a n d h e l d operation while connected to a host P C  4.1.4  Author's Contribution  . .  42 42  4.2  System O v e r v i e w  43  4.3  O u t p u t Transducers  43  4.3.1  Tactile O u t p u t Device (Tactile D i s p l a y )  43  4.3.2  Video Display  48  4.3.3  Author's Contribution  48  4.4  Sensors  49  Contents  4.5  4.6  4.7  5  vi  4.4.1  Slider P o s i t i o n Sensor  49  4.4.2  P u s h to Select Sensor  50  4.4.3  Author's Contribution  50  Interface Electronics  50  4.5.1  51  Author's Contribution  Power Supplies  51  4.6.1  52  Author's Contribution  C o n t r o l Software  52  4.7.1  Input and Output Timings  53  4.7.2  Author's Contribution  57  4.8  Tactile F l o w R e n d e r i n g  57  4.9  V i s u a l i z a t i o n of T a c t i l e S t i m u l i  61  4.9.1  Problem  61  4.9.2  N o v e l G r a p h i c a l Representations  62  4.9.3  Shaded G r a p h for Voltage Signal  62  4.9.4  S k i n Stretch Image  64  4.9.5  A u t o m a t e d Tools for Design  66  4.9.6  gif2hapticon Tool  68  4.9.7  Author's Contribution  70  Perceptual Characterization  72  5.1  Introduction  72  5.2  Author's Contribution  73  5.3  S t u d y 1 - R a n g e of Perceivable Stimulus Speed  73  5.3.1  Speed S t u d y - E x p e r i m e n t Design  74  5.3.2  Speed S t u d y - Procedure  74  5.3.3  Speed S t u d y - Results  76  Contents 5.3.4 5.4  5.5  5.6  5.7  Speed S t u d y - Discussion  vii 77  S t u d y 2 - H a p t i c Icon D i s c r i m i n a t i o n E x p e r i m e n t  77  5.4.1  M D S S t u d y - E x p e r i m e n t a l Design  78  5.4.2  M D S S t u d y - Procedure  79  5.4.3  M D S S t u d y - Results  80  S t u d y 3 - Subgroup M D S E x p e r i m e n t  82  5.5.1  Subgroup M D S - E x p e r i m e n t a l Design  83  5.5.2  Subgroup M D S - Procedure  83  5.5.3  Subgroup M D S - Results  83  S u m m a r y of Perceptual C h a r a c t e r i z a t i o n F i n d i n g s  85  5.6.1  86  Qualitative Findings  Perceptual characterization findings for a p p l i c a t i o n design . .  87  5.7.1  L i s t selection  87  5.7.2  Scrolling  87  5.7.3  D i r e c t i o n signalling  88  5.7.4  A l e r t s and background status indicators  88  6 Browser Prototype  90  6.1  Design Goals  91  6.2  L o w - F i d e l i t y P r o t o t y p e : Image-Based Browser  92  6.2.1  Design  92  6.2.2  Implementation  94  6.2.3  Image Browser User Test  95  6.3  H a p t i c D i s p l a y of W e b Pages  96  6.3.1  T h e H a p t i c Page M a p  96  6.3.2  M a p p i n g H a p t i c Icons to Page Elements  97  6.3.3  Spatial Layout  99  Contents  6.4  6.5  6.6  7  viii  Navigation Model  100  6.4.1  Cursor Position  100  6.4.2  R e n d e r i n g H a p t i c Icons  101  6.4.3  Page Element Focusing  102  6.4.4  G r a p h i c a l D i s p l a y Scrolling  102  6.4.5  C o n t r o l of C u r s o r M o v e m e n t  103  6.4.6  S p r i n g return to centre  105  6.4.7  H y b r i d Velocity / Position Control M o d e l  108  6.4.8  R e d u c t i o n of Slider J i t t e r  113  6.4.9  R e d u c t i o n of H i g h - A m p l i t u d e , High-Frequency O u t p u t s l l 5  6.4.10 Speed L i m i t a t i o n  116  Browser Software A r c h i t e c t u r e  123  6.5.1  Browser C l i e n t  126  6.5.2  Browser Server  127  6.5.3  Interprocess C o m m u n i c a t i o n and T i m i n g  131  6.5.4  Browser H a p t i c Icons  132  K n o w n Software Issues and Caveats  132  6.6.1  Support for E l e m e n t Height  132  6.6.2  O p p o r t u n i t i e s for further software o p t i m i z a t i o n . . . .  133  Browser User Evaluation  134  7.1  Aims  135  7.2  S t u d y Design  136  7.2.1  S t u d y Variables  136  7.2.2  N o r m a l i z a t i o n for Task Difficulty  137  7.3  Methodology  139  7.3.1  140  R e c r u i t m e n t of S t u d y P a r t i c i p a n t s  Contents 7.3.2  User Test E n v i r o n m e n t  141  7.3.3  Briefing and C o l l e c t i o n of D e m o g r a p h i c D a t a  144  7.3.4  Task B l o c k s  145  7.3.5  T r a i n i n g Sessions  148  7.3.6  M a i n D a t a C o l l e c t i o n Session  148  7.3.7  Post-Task Assessment  148  7.4  Pilot Study  148  7.5  S t i m u l i U s e d i n the S t u d y  149  7.6  Distraction Task  152  7.7  Browser E x p e r i m e n t Software  154  7.8  Q u a n t i t a t i v e Results  158  7.8.1  Effect of C o n d i t i o n on Task T i m e  158  7.8.2  I n d i v i d u a l Subject Differences i n Performance  158  7.8.3  Effect of T a s k on T a s k T i m e  160  7.8.4  Effect of Task x C o n d i t i o n on Task T i m e  160  7.8.5  V a l i d a t i o n of Task Difficulty N o r m a l i z a t i o n  163  7.8.6  A n a l y s i s U s i n g N o r m a l i z e d Task T i m e  165  7.8.7  L e a r n i n g / P r a c t i c e Effects  166  7.8.8  Q u a n t i t a t i v e V a l i d a t i o n of D i s t r a c t i o n Task  168  7.9  8  ix  Q u a l i t a t i v e Results  168  7.9.1  P r e - T a s k A t t i t u d e s Survey  168  7.9.2  Q u a l i t a t i v e E v a l u a t i o n of the D i s t r a c t i o n Task  . . . .  171  7.9.3  Q u a l i t a t i v e E v a l u a t i o n of the N a v i g a t i o n Task  . . . .  176  7.10 Discussion  177  Conclusion and Future Work  182  8.1  182  S u m m a r y of K e y C o n t r i b u t i o n s  Contents 8.1.1  x  Identification of a novel m u l t i m o d a l approach to addressing l i m i t a t i o n s i n mobile user interfaces  8.1.2  182  Development of a new handheld haptics hardware platform  8.1.3  183  E v o l u t i o n of a p p l i c a t i o n design concepts based on user studies and hardware development  8.1.4  184  M e t h o d for r a p i d p r o t o t y p i n g a n d graphical represent a t i o n of tactile s t i m u l i  8.1.5  184  P e r c e p t u a l characterization of a novel m i n i a t u r e piezoelectric tactile display  185  8.1.6  H a n d h e l d browser a p p l i c a t i o n w i t h tactile enhancement 185  8.1.7  M e t h o d for usability testing of mobile applications  8.1.8  Case s t u d y of a user interaction design process for haptics  8.2  Research Questions  8.3  Future W o r k : A p p l i c a t i o n Designs for Further Investigation  . . 186  186 187 . 191  8.3.1  A p p l i c a t i o n s I n v o l v i n g Shape R e n d e r i n g  192  8.3.2  G e n e r a l H a p t i c Icon A p p l i c a t i o n s  193  8.3.3  S p a t i a l Signalling  193  8.3.4  Browser Improvements  194  8.4  F u t u r e W o r k : H a r d w a r e Improvements  195  8.5  Conclusion  196  Bibliography  199  A Browser User Evaluation Documents  207  A.l  Task Inventory  212  Contents  xi  B  Browser User Study Supplemental D a t a  217  C  g i f 2 h a p t i c o n Code  222  D  Browser Prototype C o d e  227  D.l  Tactile I / O Loop  227  D.l.l  DataUpdateThread.h  227  D.1.2  DataUpdateThread.cpp  228  D.l.3  HapticPageMap.h  230  D.1.4  HapticPageMap.cpp  232  D.1.5  BrowserShared.h  242  D.1.6  BrowserShared.cpp  243  D.l.7  Browser X M L B i t s . h  244  D.l.8  BrowserXMLBits.cpp  245  D.l.9  main.cpp  250  D. 2  E  V i s u a l Browser C o m p o n e n t  254  D.2.1  browser.js  255  D.2.2  webscroller.css  264  D.2.3  browser.xul  264  D.2.4  readydialog.xul  265  D.2.5  IMyComponent.idl  265  D.2.6  M y C o m p o n e n t , cpp  266  B r o w s e r E x p e r i m e n t Software C o d e  269  E. l  taskloop.js  270  E.2  taskloop.html  277  E.3  ajaxcomponent.js  279  E.4  reinforce.js  281  xii  L i s t of Tables 5.1  S t i m u l i used i n the M D S studies  78  6.1  Values of h y b r i d velocity / position control m o d e l parameters. 119  6.2  Settings for the slider s m o o t h i n g function using h i s t o r i c a l averaging  120  7.1  Browser S t u d y P a r t i c i p a n t Demographics  140  7.2  M e a s u r e d Task T i m e by C O N D I T I O N  159  7.3  M e a s u r e d Task T i m e by Subject  162  7.4  N o r m a l i z e d Task T i m e by C O N D I T I O N  166  7.5  D i s t r a c t i o n Task Performance D a t a  170  xiii  List of Figures 1.1  E x a m p l e N a v i g a t i o n Tree P r o b l e m  5  1.2  Thesis Research O v e r v i e w  9  3.1  e B o o k Reader w i t h V i b r o t a c t i l e Feedback  20  3.2  E x a m p l e s of E x i s t i n g L i n e a r T o u c h Input Devices  23  3.3  B i d i r e c t i o n a l L i n e a r T o u c h Sensor / Tactile O u t p u t Schematic  24  3.4  S i m u l a t e d Force Feedback U s i n g a M o v i n g B u m p  27  3.5  A p p l i c a t i o n Design Scenarios  28  3.6  L i s t Selection A p p l i c a t i o n  29  3.7  Low-Fidelity Foam Mockup  30  3.8  Browser A p p l i c a t i o n  32  3.9  Music Navigation Application  37  4.1  Hardware Overview  44  4.2  Hardware Overview, M a r c h 2005  45  4.3  Tactile Display  46  4.4  P o s i t i o n a n d Push-to-Select Sensors  49  4.5  E m p i r i c a l T i m i n g Measurements  54  4.6  C o n t r o l Software Flowchart  56  4.7  S k i n Stretch P a t t e r n s i n N a t u r a l and A r t i f i c i a l Tactile F l o w Stimuli  58  List of Figures  xiv  4.8  A Millipede  60  4.9  V i s u a l i z a t i o n of Tactile S t i m u l i  63  4.10 P h o t o s h o p C u s t o m F i l t e r settings for a u t o m a t e d stretch i m age generation  67  4.11 Image-Based H a p t i c Icon D e s i g n Workflow  71  5.1  E x a m p l e s of S t i m u l i U s e d for the Speed S t u d y  75  5.2  Speed S t u d y Results  76  5.3  Waveforms used i n the M D S Studies  79  5.4  Results from the M D S S t u d y  81  5.5  Results from the Subgroup M D S S t u d y  84  6.1  Image Browser  93  6.2  H a p t i c Page M a p  98  6.3  Scrolling M a r g i n s  104  6.4  Slider C o n t r o l M o d e s  106  6.5  Force Feedback U s i n g Springs  Ill  6.6  V e l o c i t y / P o s i t i o n C o n t r o l State D i a g r a m  114  6.7  S u b t a x e l R e n d e r i n g Technique  121  6.8  Effective Icon Design for S u b t a x e l R e n d e r i n g  122  6.9  Browser Software A r c h i t e c t u r e  125  6.10 H a p t i c Page M a p and Icons M o d e l  130  7.1  Browser User Test E n v i r o n m e n t  142  7.2  H a p t i c Icons U s e d i n the Browser User S t u d y  150  7.3  Effect of Task on Task T i m e  161  7.4  Effect of Task and C o n d i t i o n on Task T i m e  164  List of Figures 7.5  xv  R e l a t i o n s h i p of Task T i m e w i t h Presentation O r d e r (Set of 3 Blocks)  167  7.6  R e l a t i o n s h i p of Task T i m e w i t h P R E S E N T A T I O N O R D E R  . . . 169  7.7  Results from the P r e - T a s k A t t i t u d e s Survey  172  7.8  Q u a l i t a t i v e Feedback on the D i s t r a c t i o n Task  173  7.9  Q u a l i t a t i v e Feedback on the N a v i g a t i o n Task  175  xvi  Glossary haptic icon A l s o k n o w n as a hapticon or tacton [10], a haptic icon is a brief, distinctive signal delivered to the user through the haptic apparatus, or its software  representation.  H i g h fidelity p r o t o t y p e A s  compared  to  low-fidelity  prototyping,  a  methodology for expediting the involvement of users i n an interaction design process by shifting the development effort from detailed implementation of features to early u s a b i l i t y testing. I n contrast, a highfidelity prototype is one w h i c h is relatively close to the final product i n terms of functionality and the level of i n t e r a c t i v i t y w h i c h is supported. piezo, piezo actuator Refers to an i n d i v i d u a l piezoelectric bending motor element. p o s i t i o n control, or p C o n t r o l In this mode, the scrolling m o t i o n of the cursor or page follows the slider p o s i t i o n directly. T h i s type of control is also used i n jog dial or mouse wheel controls. slider In this document,' refers to the mechanism that allows the tactile display to be moved up and down. stretch image A graphical representation of the amount of gap between adjacent piezo actuators at a given point i n time. A s s u m i n g the user's  Glossary  xvii  finger has been i n constant contact w i t h the T D since prior to the application of signal, the levels depicted i n each location of the stretch image correspond to the amount of relative skin displacement ("skin stretch") at that l o c a t i o n on the T D . D e s c r i b e d further i n section 4.9. s u b p i x e l r e n d e r i n g A s used i n this document, refers to the technique of rendering graphics into an offscreen buffer w i t h higher resolution t h a n the physical resolution of the display, then d o w n s a m p l i n g to the display resolution using anti-aliasing filters. F o r graphical displays, this increases the effectively usable resolution at the cost of some sharpness. In this document, this t e r m does not refer to the technique of using an L C D ' s red-green-blue subpixels to increase resolution. subtaxel r e n d e r i n g E q u i v a l e n t to subpixel rendering for a tactile display. tactile flow T h e perception of the movement of a tactile stimulus across the s k i n over time. See Section 4.8. tactile window T h e p o r t i o n of the page m a p that is currently being rendered to the T D . See Section 6.4.1. taxel Just as a pixel is a logical element of a multi-element graphical display, a taxel is a logical element of a multi-element tactile display. In the case of the device prototype discussed i n this thesis, a taxel represents the voltage delivered to a single piezo actuator. T h e T D described i n this thesis therefore represents eight taxels. T D T a c t i l e display. tweening I n a n i m a t i o n , refers to the creation of m o t i o n by adding intermediate frames that depict a gradual progression from one state to  Glossary  xviii  another. For the tactile display described in this thesis, "tweening" refers to the successive displacement of a pattern of piezo activation (voltage levels) across adjacent piezo actuators, to create the sensation of tactile flow.  velocity control, or v C o n t r o l In this mode, the slider position is mapped to the scrolling velocity. Therefore, as long as the user keeps the slider in one place, the cursor or page moves with constant velocity. This type of control is also used in joysticks and shuttle controls. voltage image (volt i m a g e )  1  A graphical representation of the voltage  applied across the array of piezo actuators at a given point in time, as described in section 4.9. voltage image (volt i m a g e )  2  A n object in the STReSS library A P I which  contains data in the form of voltage levels. X M L Extensible Markup Language.  A generic markup language, stan-  dardized by the W3C (World Wide Web Consortium). In the current project, extensible documents conforming to the X M L syntax are used for various purposes including representing tactile stimuli. X P C O M Cross Platform Component Object Model. source Mozilla application platform.  Part of the open-  Used in the tactile enhanced  H T M L browser (Chapter 6) for inter-process communications. X U L X M L User Interface Language. A markup language for specifying the visual user interface components in the Mozilla application platform. In the current project, X U L is used to customize the Mozilla browser for use as a handheld application.  xix  Acknowledgements I w o u l d like to acknowledge the guidance of m y supervisor, D r .  Karon  M a c L e a n , who was always a c h a m p i o n for her students, w o r k i n g tirelessly to encourage us t o . d o quality, meaningful research, and to overcome our personal challenges, perfectionism included. M u c h of the research described i n this thesis was the result of an exceptional c o l l a b o r a t i o n between laboratories at the U n i v e r s i t y of B r i t i s h C o l u m b i a a n d M c G i l l U n i v e r s i t y i n M o n t r e a l . I n particular, Jerome Pasquero a n d I p a r t i c i p a t e d i n an extended exchange w h i c h resulted i n numerous learning experiences since we each brought a different, sometimes opposite, point of view to the research problems. I a m grateful to Jerome and other members of the M c G i l l H a p t i c s L a b , especially Professor V i n c e n t H a y w a r d , V i n c e n t Levesque, and Q i W a n g , for their patience i n engaging me i n endless scholarly debates, their w a r m hospitality, a n d their dedication to the project. T h e contributions of D o n Pavlasek and Jozsef B o k a to the mechanical design a n d fabrication are also sincerely appreciated. T h e members of m y thesis committee, D r . Rodger L e a , D r . R o n R e n s i n k , a n d especially D r . Steven W o l f m a n , gave valuable feedback on this research. F i n a l l y , I w o u l d like to express m y a p p l i c a t i o n to M a r i o E n r i q u e z for help w i t h the M D S testing methodology, and to Steve Y o h a n a n for sharing his operating system expertise.  1  Chapter 1 Introduction M o b i l e , portable, and handheld c o m p u t i n g environments offer significant promise for the future of interactive c o m p u t a t i o n .  C o m p a r e d to a t y p i c a l  desktop computer, a mobile device enables o p p o r t u n i s t i c uses for informat i o n that stretch the boundaries of the popular definition of data: instead of w o r k i n g w i t h documents and files, mobile devices are called u p o n to m a n age an increasingly varied collection of data, from voice calls to photos and location based i n f o r m a t i o n snippets. T h e increasing p o p u l a r i t y of mobile devices also represents a continued shift away from the physical i n s t a n t i a t i o n of d a t a and towards an increasingly m i n i a t u r i z e d and networked m e d i u m . Information that was once tangible, i n the form of paper, magnetic tapes or other media, is now i n v i s i b l y contained w i t h i n the device used to access it or s i m p l y delivered over ubiquitous, wireless networks. T h e dissociation of information from its physical encapsulation is driven by pressures to improve p o r t a b i l i t y by reducing device weight and volume. However, interactive applications can never be completely removed from physical constraints due to the necessity of p r o d u c i n g a h u m a n sensory percept.  1  Furthermore, the history of computer user interfaces shows that  ^rain-computer interfaces enable the possibility of a completely non-physical user interface, but are too poorly understood at the present time to consider them a viable alternative in mobile device interaction.  Chapter  1.  Introduction  2  significant progress i n expanding the breadth of accessibility to computers can be achieved when interfaces, such as the g r a p h i c a l user interface ( G U I ) , leverage the i n t u i t i v e physics [38] that evolved i n h u m a n beings as a means for dealing w i t h the environment. T h e design of effective mobile information devices is thus confronted w i t h a classic d i l e m m a : balancing the power of abstraction and freedom from p h y s i c a l constraints against the usability advantages of a system that interacts effectively w i t h h u m a n beings on a n a t u r a l physical scale. A l l too often, increased m o b i l i t y comes at the cost of decreased usability.  1.1  Problems with Contemporary Mobile Interaction Design  T h e user interfaces of contemporary mobile devices suffer from sensory band-  width limitation and environmental competition for visual and auditory attentional resources. A r t i f i c i a l haptics, i n c o r p o r a t i n g synthetic tactile signalling, may be a useful t o o l for solving these problems.  1.1.1  Sensory B a n d w i d t h L i m i t a t i o n  T h r o u g h o u t most of the 1990s, there was a steady decrease i n the size (volume and weight) of personal mobile information devices such as mobile phones, personal d i g i t a l assistants ( P D A s ) , m e d i a players a n d laptop computers, spurred by steady technological progress i n integration of electronic circuits, portable power, and d i g i t a l wireless transmission.  However, the  trend towards m i n i a t u r i z a t i o n has since leveled off i n m a n y categories of devices, despite continued progress i n hardware integration and anecdotal  Chapter  1.  Introduction  3  evidence from users that the p o r t a b i l i t y of ever smaller devices is h i g h l y desirable. T h e conventional w i s d o m is that input a n d o u t p u t devices such as keypads and screens can not be made smaller w i t h o u t negatively impacting usability. T h u s the factors l i m i t i n g device p o r t a b i l i t y have ceased to be technical i n nature; instead, user interface considerations now dictate the p r a c t i c a l l i m i t s to m i n i a t u r i z a t i o n of today's mobile devices. For the increasingly complex tasks being performed on mobile devices, the information capacity of the s m a l l slice of the v i s u a l field covered by a s m a l l L C D display has become a l i m i t a t i o n . V a r i o u s head-mounted displays such as eyepieces and goggles have been developed to increase the effective display area while retaining portability, but thus far they have not proven w i d e l y p r a c t i c a l because of their intrusiveness. Similarly, most mobile information devices t o d a y include an audible transducer and microphone input, but aside from m a k i n g phone calls and l i m i t e d command-based voice recognition, information-rich a u d i t o r y user i n terfaces have yet to be demonstrated on mobile devices. T h e use of m u l t i p l e simultaneous sensory modalities, k n o w n as multimodal interaction, offers a m e t h o d to increase the available user interface bandwidth, or information volume per unit time t h a t can be exchanged between user a n d device. [12] W i t h v i s u a l and a u d i t o r y modalities already receiving widespread research interest, and taste and smell being thus far relatively i m p r a c t i c a l as user interface modalities, the d o m a i n of touch appears to be a useful area for s t u d y as an a d d i t i o n a l m o d a l i t y to enrich the information c a r r y i n g capacity of the user interface of a mobile device. A n abstract example of an interaction design p r o b l e m created by l i m i t e d sensory b a n d w i d t h i n mobile devices is shown i n F i g u r e 1.1. F o r a given i n terface w i t h a number of functions, "porting" the a p p l i c a t i o n to a mobile  Chapter  1.  Introduction  4  device from a more conventional personal computer imposes l i m i t a t i o n s i n the amount of information that can be p r a c t i c a l l y presented at any one level of navigation hierarchy.  Designers w i s h i n g to preserve functionality  are therefore forced to implement "deeper" navigation hierarchies requiring more user interaction to access the desired function. T h e increased navigat i o n overhead, i n t u r n , decreases the p r a c t i c a l i t y of the a p p l i c a t i o n i n the hands of a busy, mobile user. W h i l e this example m a y seem simplistic, it is a relatively accurate description of the problems w i t h the first generat i o n of mobile phone applications based on the wireless a p p l i c a t i o n p r o t o c o l (WAP).  1.1.2  E n v i r o n m e n t a l A t t e n t i o n a l  C o m p e t i t i o n  for V i s u a l  a n d  A u d i t o r y  R e s o u r c e s  H a p t i c s m a y be especially useful for m u l t i m o d a l interaction i n a mobile use context because it does not share the action-at-a-distance property of v i s u a l and a u d i t o r y signalling, and is a more "parallel" sensory m o d a l i t y t h a n vision and hearing. W h e n m o v i n g through the environment, a person t y p i c a l l y uses v i s u a l and auditory modalities for navigation, avoiding obstacles, and learning about the environment. A n y system that requires those modalities for user interaction must compete for the user's attention w i t h information originating from the environment. C o m p e t i t i o n for l i m i t e d v i s u a l attention can present serious safety issues: for example, whereas use of a mobile device while w a l k i n g on a crowded street might be m a r g i n a l l y acceptable, use while d r i v i n g is becoming an increasing p u b l i c safety concern, and use while l a n d i n g an aircraft might be  Chapter 1. Introduction  large-screen navigation tree  5  equivalent small-screen navigation tree  ;;?'  .* * x  ill 111  Figure 1.1:  Examples of deep versus wide navigation trees. The limited  sensory bandwidth available at any level of the hierarchy forces a "deep" configuration, which requires more effort to navigate.  Chapter  1.  Introduction  6  considered completely unacceptable [49]. In practice, the user is o n l y able to devote a p o r t i o n of their v i s u a l attention to a mobile device, and repeatedly shifts their attention between device and environment [40]. It follows that a mobile user interface can not be guaranteed to deliver a v i s u a l cue to the user unless the information is retained onscreen long enough for the user to direct their attention to the display. T h i s reduces the interface b a n d w i d t h relative to desktop computers even further t h a n a simple comparison of screen size w o u l d suggest. A u d i t o r y interfaces suffer from similar l i m i t a t i o n s . A very noisy environment such as a crowded street presents challenges i n m a k i n g the sounds p r o d u c e d by the device audible, as well as recognizing sounds produced by the user.  A u d i t o r y interfaces therefore scale p o o r l y to large numbers of  densely packed users. O n the other hand, i n a very quiet environment, such as a meeting or library, use of an a u d i t o r y interface might be completely unacceptable because it w o u l d disturb others.  Headphones and microphones  t h a t isolate exterior noise improve user-device c o m m u n i c a t i o n , but o n l y at the expense of user-environment c o m m u n i c a t i o n , causing similar concerns as visual i m p a i r m e n t . In comparison to vision or hearing, the sense of touch is relatively unobstructed i n a mobile use context. T h e vestibular system is used for balance, and the feet or hands may be i n contact w i t h control surfaces,  but  other areas of s k i n are likely to be available for interaction. U n l i k e vision or hearing, w h i c h must be sensed through discrete organs, touch receptors are d i s t r i b u t e d throughout the b o d y and active simultaneously — thus, the system can be characterized as more "parallel" i n nature. T a c t i l e signals are o n l y active at the point of direct contact, enabling discreet interaction. In a d d i t i o n , a portable device is also i n contact w i t h  Chapter 1.  Introduction  7  the user's body more often than a desktop computer, making it a good platform for tactile interaction. Indeed, a non-multimodal, purely haptic interface is conceivable and is currently used in limited contexts such as vibrotactile signalling of incoming calls. However, haptics can not be considered a generic replacement for visual or auditory interfaces, since each modality has its own profile of applications for which it is optimally suited. Work is ongoing to define novel application spaces for haptics, including affective, subconscious, and interpersonal interfaces. Within the scope of this thesis work, however, the potential benefits of haptic augmentation to existing mobile applications with demonstrated, proven utility will be explored. This will enable effort to be focused on implementing a high-fidelity prototype, including hardware development, application conceptualization, and usability testing, completing a full iteration of an interaction design cycle within the scope of a master's thesis.  1.2  Thesis Research Questions  The goal of this thesis research is to increase the understanding of the application of haptics to a mobile computing context. The following research questions are addressed: 1 . What are the problems with existing mobile user interfaces that may be addressed using haptics? 2. How can one implement haptics on a mobile device despite power, size, and weight restrictions?  Chapter  1.  Introduction  8  3. What are the expressive capabilities of the hardware prototype in the hands of a human user? 4. What are the engineering challenges associated with building a highfidelity hardware and software prototype with handheld use in mind? 5. Is tactile flow an effective aid for user navigation?  6. How can user-centred design methodology be applied to haptics, where hardware technology is still the primary determinant of user interface capabilities? These questions are revisited i n the C o n c l u s i o n (Chapter 8) w h i c h summarizes the research findings.  1.3  Thesis Approach and Overview  T h e approach t h a t was used to address the research questions identified i n the previous section was to create a handheld haptics system i n a usercentred, iterative fashion by i n c o r p o r a t i n g user i n p u t as early as possible i n the design process. T h e stages of this process are shown i n F i g u r e 1.2. Because the a i m of the thesis research was to use this process to better understand the p o t e n t i a l for haptics i n a mobile user interface, rather t h a n to produce a fully engineered product, p r i o r i t y was given to creating a handheld haptic user experience over a fullly-functional device; if its usefulness could be demonstrated, then future work could address the engineering challenges related to true mobility, such as battery power and c u s t o m integrated circuits. Therefore, early i n the design process the heuristic was adopted that the prototype should attempt to replicate as m u c h of the handheld user experience as p r a c t i c a l w i t h i n the resources available for the research, while  Chapter  Background  1.  Introduction  Present Work  9  Future Work  User Needs  Usage Scenarios & Knowledge of — i Application Concepts Existing Hardware Further Hardware Development  Hardware Design  Perceptual Characterization  Application Design  Application Usability Testing  i Further •*• Stimulus Discovery and Perceptual Characterization •  I I  New Applications  ^ Further Refinement of Usability Testing Practices  Conclusions and Recommendations for Future Application Design F i g u r e 1.2: O v e r v i e w of the design process used for this thesis research.  J  Chapter 1. Introduction  10  r e m a i n i n g tethered to a host P C and easily customizable; however, it should have a power, weight, and volume profile such that future integration into a wireless mobile device could be conceivable using k n o w n engineering techniques.  Further details of this design decision are provided i n C h a p t e r 4,  Hardware. R a t h e r t h a n specialized haptic-only applications or, applications for users w i t h v i s u a l or a u d i t o r y impairment, we have chosen to focus on the general area of haptic augmentation of applications similar to those currently employed on mobile devices, because there is a clearer understanding of user needs a n d an o p p o r t u n i t y to contribute to ongoing mobile development programmes. Therefore, for the purpose of the studies described i n this thesis, the users are a general audience. T h e hardware development and perceptual characterization stages were performed i n collaboration w i t h Professor V i n c e n t H a y w a r d , and graduate students Jerome Pasquero, V i n c e n t Levesque, and Q i W a n g of the M c G i l l U n i v e r s i t y H a p t i c s L a b o r a t o r y ; and undergraduate research student S h a n n o n L i t t l e of U B C . T h e contributions of each collaborator are noted i n appropriate sections throughout the thesis. T h i s thesis is organized into the following sections:  C h a p t e r 1 - I n t r o d u c t i o n describes the p r o b l e m that is addressed i n this thesis, and the m o t i v a t i o n for the research. Chapter 2 - Related Work  discusses the existing research i n mobile and  haptic interfaces. C h a p t e r 3 - D e s i g n C o n c e p t s describes  the  i n i t i a l conceptual  design  work t h a t was performed to better understand the research d o m a i n  Chapter 1. Introduction  11  and to determine the appropriate a p p l i c a t i o n a n d hardware specifications for further development.  Chapter 4 - Hardware Prototype Development covers the engineering process that was performed to create a handheld haptics platform w i t h an integrated piezoelectric tactile display. I n a d d i t i o n to electromechanical design, this chapter also includes a consideration of the m e t h o d for designing a n d representing tactile outputs, a n d a description of the control software architecture.  Chapter 5 - Perceptual Characterization describes user experiments conducted  before detailed a p p l i c a t i o n development, t o understand the  capabilities a n d l i m i t a t i o n s of the device i n p r o d u c i n g a salient tactile percept.  Chapter 6 - Browser Prototype describes an in-depth, high-fidelity prototype of a mobile browsing a p p l i c a t i o n that was b u i l t using the handheld haptics platform. T h i s chapter also includes the results of an i n i t i a l exploratory user test, a n d hardware customizations for the browsing a p p l i c a t i o n .  Chapter 7 - Browser User Test presents  a  controlled  user  study  designed to assess the performance characteristics of the h a p t i c a l l y enhanced browsing a p p l i c a t i o n . N e w software created to manage the usability test, a n d a distraction task model, are also described.  Chapter 8 - Conclusion returns to the research questions identified i n the previous section a n d considers how they were addressed through the activities conducted for this thesis research.  12  Chapter 2 Related Work In this chapter we describe the previous research t h a t informs the present s t u d y as it concerns mobile interface design challenges, haptic augmentation for m u l t i m o d a l enhancement, handheld haptic technologies, a n d perceptual evaluation of haptic devices. T h e decisions to focus on h a p t i c augmentation for general mobile devices, to use piezoelectric lateral skin-stretch technology, a n d to use the perceptual characterization methodology are justified based on the existing knowledge i n the aforementioned domains.  2.1  Mobile Interface Design Challenges  T h e hypothesis that l i m i t e d sensory b a n d w i d t h due to s m a l l screens and keypads restricts mobile interface designs is based largely on widespread anecdotal evidence and the experience of the author i n designing mobile i n terfaces professionally. A s mobile devices are a field of high c o m m e r c i a l i n terest and development, m a n y u s a b i l i t y studies are only published privately w i t h i n companies [53]. A t the same time, every research paper relating to mobile interface design begins w i t h a description of the challenges posed by l i m i t e d user interface b a n d w i d t h , resulting i n the need for reduced and restructured content relative to desktop interfaces [6]. T h e need for more expressive interfaces that use physical, tactile affordances to support a con-  13  Chapter 2. Related Work  stellation of c o m p u t i n g devices was articulated i n the famous Tangible B i t s paper b y Ishii a n d U l l m e r [24]. L i m i t e d visual attention i n mobile use contexts has been described i n [40]. B y following mobile users i n realistic usage environments, O u l a s v i r t a et a l . observed r a p i d a n d frequent v i s u a l attention s w i t c h i n g between device a n d environment as users attempted to navigate while using a mobile browser. Based o n broad field evidence, a related paper b y R o t o a n d O u l a s v i r t a [47], suggests t h a t non-visual, a n d p a r t i c u l a r l y haptic, interfaces are needed i n mobile applications to overcome the problems associated w i t h loss of visual attention d u r i n g system wait states.  2.2  Haptic Augmentation for Multimodal Enhancement  G i v e n the attention-demanding nature of tasks performed o n a h a n d h e l d device i n a mobile environment, studies of m u l t i m o d a l enhancement of i n terfaces under w o r k l o a d are relevant. In particular, C h a n et a l . [12] demonstrated that haptic signals could be learned a n d recognized despite distraction that placed a d e m a n d on the users' attentional resources. T a n et a l . [50] demonstrated that performance i n an attention-demanding vehicular navigation task could me improved w i t h h a p t i c feedback a p p l i e d to a person's back. In a n a v i g a t i o n task, D e n n e r l e i n [17] demonstrated t h a t haptic forcefeedback could be used to improve targeting performance.  Similarly, a  patent b y N o v i n t Technologies [7] covers a r b i t r a r y force feedback produced when the user pushes "into" a screen boundary, for non-visual feedback of scrolling. Directional stimulation is a key component of these applications,  Chapter  2. Related Work  14  and is achieved i n existing systems either through force feedback or spat i a l l y d i s t r i b u t e d vibrotactile s t i m u l a t i o n [50]. U s i n g m u l t i p l e actuators to provide a directional signal provides significant increased u t i l i t y over simple vibration-([27, 31] and Immersion C o r p . patent [9]). These examples support the feasibility of the a p p l i c a t i o n scenarios that are proposed later, i n c l u d i n g the use of directional and tactile feedback (but not through spatial patterns of vibration) to potentially enhance targeting a n d spatial navigation i n an interface.  2.3  Handheld Haptics  W h i l e there is promise for the use of haptics on a mobile device, there are relatively few examples of functioning implementations. difficulties are listed b e l o w .  Some u n d e r l y i n g  1  • L a c k of mechanical grounding. A p p l y i n g force-feedback to a user requires a fixed mechanical ground.  In a mobile context, the forces  must be created relative to the user, w h i c h imposes constraints on the physical design and force output capabilities. A n alternative is tactile display, w h i c h generates no net force on the user, but l i m i t s the scale of sensations  consequently  transmitted.  • Stringent power, size, and weight constraints a p p l y i n mobile contexts. Use of a conventional motor for force-feedback introduces a significant impact on a l l three. [44] • Since relatively few instances of integrated, rich haptic feedback exist today, it is difficult to justify inclusion i n a mobile device u n t i l there 1  Portions of this section have been previously published in [34]-  Chapter  2. Related Work  15  is a better understanding of the added value it creates for the user. T h e most c o m m o n occurrence of haptic feedback i n mobile devices today is the ubiquitous mobile phone or pager v i b r a t o r . Patterns of v i b r a t i o n are t y p i c a l l y used to indicate various alerts, such as an a l a r m or i n c o m ing call. R e c e n t l y there has also been c o m m e r c i a l a n d research interest i n p u t t i n g v i b r a t i o n i n more sophisticated applications [13, 14, 32]. Generally, v i b r o t a c t i l e s t i m u l i are produced globally (across the entire device) and w i t h only two levels (on or off), v i b r o t a c t i l e devices generally do not afford b i d i rectional interaction i n the sense of the user actively e x p l o r i n g information t h r o u g h movement and the sense of touch [23, 48]. Devices that are capable of delivering grounded forces to the user have the potential for greater expressive capacity t h a n v i b r a t i o n .  Designs are  restricted to m i n i m a l degrees of freedom ( D o F ) [35], yet must create enough added value to justify the power, size, and weight tradeoffs. Piezoelectric actuation offers significant promise for mobile applications because it can achieve a smaller form factor w i t h o u t coils and P o u p y r e v et al.  magnets.  of Sony's C o m p u t e r Science Laboratories used piezo el-  ements to produce v i b r o t a c t i l e actuation of handheld devices or parts of t h e m [46]. In the case of a touch screen [45], the user t y p i c a l l y experiences the illusion of local actuation although the entire screen moves; this type of v i b r o t a c t i l e a c t u a t i o n of a flat surface is also mentioned i n a patent by N e w Transducers L i m i t e d [8]. C r e a t i n g true m u l t i p l e loci of a c t u a t i o n on a s m a l l scale is significantly more complicated using v i b r o t a c t i l e signals [46]. Piezoelectric actuators may be configured i n a way that also produces non-vibrotactile, low-frequency skin s t i m u l a t i o n [23]. W h e n the user places h i s / h e r finger on an array of actuators w h i c h collectively comprise a m u l t i -  Chapter  2. Related Work  16  element tactile display, the relative m o t i o n of the i n d i v i d u a l piezo tips stretches the skin locally, a c t i v a t i n g skin mechanoreceptors.  A p p l y i n g spe-  cific patterns of d i s t r i b u t e d skin deformation can create the illusion of touching small-scale shapes and textures.  A device based on this technology,  called the V i r t u a l B r a i l l e D i s p l a y ( V B D ) [30], has been used to render legible B r a i l l e dots using o n l y lateral stretching of the skin. S i m i l a r sensations can be achieved using technologies that push into the s k i n [51], but the lateral skin-stretch configuration is mechanically simpler and makes the most efficient use of the range of m o t i o n of commercially available piezoelectric bending motors [15], resulting i n favourable power, size, and weight profiles. Such a configuration also provides internal mechanical grounding, as forces are generated between adjacent piezo elements. W e thus eventually chose lateral skin-stretch as the most p r o m i s i n g configuration  for the hardware prototype.  O u r approach uses the same basic  principle as the V B D , but m i n i a t u r i z e d and embedded i n a handheld form factor wherein the skin-stretch site is displayed to the users t h u m b , and m o u n t e d on a slider. T h e device is described i n further detail later.  2.4  Perceptual Evaluation of Haptic Devices  W h e n a novel haptic transducer is created, prior to the development of fullb l o w n applications it is first helpful to understand the perceptual characteristics of its output, so that signals can be created that best use the device's expressive capabilities. T h e perceptual evaluation procedure described i n C h a p t e r 5 is based on work described by E n r i q u e z and M a c L e a n [18], using the m u l t i d i m e n s i o n a l scaling ( M D S ) technique to categorize haptic icons, and the subgroup M D S technique has been further analyzed i n [42]. T h e  Chapter  2. Related Work  17  procedure used i n this thesis for designing the h a p t i c signals also loosely follows the w o r k b y Brewster a n d B r o w n o n tactons [10, 11].  18  Chapter 3 Design Concepts T h i s section describes a p p l i c a t i o n concept development that followed from the b a c k g r o u n d research into the p r o b l e m d o m a i n a n d existing h a p t i c approaches.  1  Design concept development proceeded i n stages over the course of a year; the existence of the piezoelectrically actuated lateral s k i n stretch technology was not k n o w n u n t i l after i n i t i a l hardware a n d a p p l i c a t i o n concept development h a d been done. Instead of creating applications for a p a r t i c u l a r hardware specification (technology-centred design), we sought to leave the hardware specification as open-ended as possible while focusing our efforts on understanding how different a p p l i c a t i o n designs could meet user needs. O f course, no a p p l i c a t i o n , especially haptic, can exist i n a v a c u u m w i t h no dependency on hardware.  Therefore, d u r i n g a p p l i c a t i o n concept develop-  ment we also gave consideration to the minimum set of hardware features necessary to make the specified interaction possible. For the present project, we also h a d to keep i n m i n d several p r a c t i c a l considerations such as access to manufacturing facilities a n d setting an appropriate scope for a master's thesis project.  In these ways, the interaction design a n d hardware design  Sections 3.3.1 through 3.3.4 have been previously published in [34] and represent the collaborative work of authors on that paper. Figures 3.6, 3.8, 3.9, and Section 3.3.5 have not been previously published.  Chapter  3. Design Concepts  19  evolved synergistically, w i t h the interaction needs being the p r i m a r y driver. In this chapter we describe the m a t u r a t i o n of design concepts from an open-ended concept of s i m u l a t i n g the haptic experience of reading a book, to a slightly more t i g h t l y specified concept of 1-D navigation w i t h haptic feedback. C o n c e p t u a l prototypes, i n c l u d i n g low-fidelity physical mockups and detailed usage scenarios, were used to drive the design process by elucidating details of the user interaction. F i n a l l y , we converged on a design based on 1-D interaction model, w i t h piezoelectric lateral skin-stretch technology used to enable the a p p l i c a t i o n concepts.  3.1  Electronic Book Reader with Vibrotactile Feedback  A n early design effort focused on haptic augmentation of the experience of using a mobile, handheld electronic book reader. T h e promise of a portable electronic device for reading information has been long recognized [26], but adoption of electronic readers has been hampered by poor display quality, weight and power, and slow navigation compared to t r a d i t i o n a l books. W i t h the display quality, weight and power considerations being slowly m i t i g a t e d due to technological advances, what remains is the user interface p r o b l e m . F l i p p i n g t h r o u g h a paper book is a rich haptic experience, t r a n s m i t t i n g information such as the relative l o c a t i o n of a page i n the book, the speed of movement t h r o u g h pages, and even the usage history (books t e n d to n a t u r a l l y flip open to well w o r n sections). W e explored the concept of artificial haptic signalling to support book navigation by constructing the prototype shown i n F i g u r e 3.1. A v i b r o t a c t i l e transducer (a s m a l l speaker) was mounted on a rubber block that was shaped  Chapter 3. Design Concepts  F i g u r e 3.1:  20  ( A ) Device for p r o v i d i n g vibrotactile feedback to simulate the  experience of page-flipping. T h e rubber nipper is mounted o n a mini-joystick a n d can articulate i n the u p / d o w n direction w i t h spring r e t u r n to centre. (B) T h e device being held i n the h a n d .  N o t e that the rubber edge meets  the t h u m b at an angle s i m i l a r to that of the edge of a book w h e n pages are being flipped.  Chapter 3. Design Concepts  21  to simulate the angle of a book's edge when the device was h e l d i n the h a n d . T h e rubber "flipper" was m o u n t e d on a spring-loaded m i n i a t u r e joystick found i n a t y p i c a l P C game controller, and its m o t i o n restricted by a custommachined plate so that the flipper could only move up and d o w n . F i n a l l y , the apparatus was enclosed i n a soft formed foam hand-grip ( C r a y o l a M o d e l Magic). T h e concept was that the user w o u l d use h i s / h e r t h u m b to t i l t the rubber flipper up and down, p r o d u c i n g a velocity signal which, relayed t h r o u g h the off-the-shelf U S B game controller, w o u l d cause repeated page-up or pagedown signals to be sent to electronic book reader software. Feedback w o u l d be p r o v i d e d for each page-turning event i n the form of a perceptible click caused by the v i b r o t a c t i l e transducer, allowing the user to feel their movement t h r o u g h the pages of an electronic book.  F i n a l l y , well w o r n pages  w o u l d be simulated by a c o m b i n a t i o n of stronger clicks accompanied by appropriate delays i n the navigation m o d e l . T h i s concept was a  first-iteration  r a p i d prototype intended to s t u d y the  ergonomic experience of a device w i t h p o t e n t i a l for integration into a mobile / h a n d h e l d environment. T h e speaker itself d i d not generate sufficient force for a f a c t u a l l y perceptible click. A l t h o u g h it could have been upgraded w i t h a voice coil or similar device w i t h stronger force, attention shifted to a more versatile 1-D scrolling strip concept that could incorporate more expressive feedback.  3.2  Early 1-D Navigation Concepts  In a t y p i c a l feature-packed mobile device, m u c h of the chassis that faces the user (the X — Y plane) is covered w i t h i n p u t and output  transducers  Chapter  3. Design Concepts  such as keypads, displays, a n d speakers.  22  Furthermore, there is pressure  to m i n i m i z e the thickness of such devices i n the Z direction, i n order to m a i n t a i n a form factor that fits unobtrusively against the b o d y i n a pocket or handbag. These factors make it difficult to incorporate a haptic device on the front or back surface of a mobile device. M o r e accessible locations are the side surfaces, w h i c h are b o t h relatively unused a n d also somewhat mitigates concerns about the physical depth of a haptic transducer, w h i c h m a y extend somewhat inside the case w i t h o u t significantly altering the t y p i c a l aspect ratio of a handheld device. In a t y p i c a l box-shaped handheld device, this l o c a t i o n is accessible to the user's thumb, w h i c h sweeps out a n arc t h a t c a n be used for 1-degree of freedom, linear i n p u t . Recently, several c o m m e r c i a l products have been introduced that utilize capacitive touch sensors to digitize 1-D input (Figure 3.2). T h i s provided the i n s p i r a t i o n to use the side surface of a device as a bidirectional haptic transducer. A conceptual schematic of a b i d i r e c t i o n a l touch i n p u t / tactile output system is shown i n F i g u r e 3.3. T h e tactile transducer component is assumed to be of the type that produces physical deformations perpendicular to the surface of the finger, a n d could be implemented using any of a variety of technologies. T h e following technologies were considered as candidates for the tactile transducer:  • E l e c t r o m a g n e t i c a l l y a c t u a t e d p i n s produce local deflection a n d and form the basis for conventional B r a i l l e displays. W i t h a n appropriately chosen flexible covering acting as a low-pass filter, they c a n also be used to render s m o o t h shapes. Investigation was made into using the mechanism from a d o t - m a t r i x printhead to create m i n i a -  Chapter  F i g u r e 3.2: sensors.  3.  Design  Concepts  23  E x a m p l e s of existing products u t i l i z i n g linear capacitive touch  Chapter  3. Design Concepts  Capacitative Tactile Position Transducer Sensor  24  Click Switch  <:o:o:o:o:o:o:»—|  QQQ000O—|  F i g u r e 3.3:  o o  O O  Schematic d i a g r a m of a concept b i d i r e c t i o n a l haptic device for  1-D linear i n p u t and tactile output. A flexible sensor for determining the p o s i t i o n of the user's finger is layered on top of a deformation-producing tactile o u t p u t transducer, and mounted on a movable apparatus so that the force of the user's finger pressing into the device can be detected for actions such as selection. (Components not d r a w n to scale)  Chapter  3. Design Concepts  25  ture, high-resolution deflections. Unfortunately, the drawbacks to electromechanical actuation are h i g h power c o n s u m p t i o n a n d the weight of the coils. • Just as piezoelectric speakers c a n be used as alternatives t o electromagnetic speakers, a flat, segmented array of p i e z o e l e m e n t s c o u l d produce the tactile deflections. However, conventional piezo elements are made for p r o d u c i n g audio, w h i c h has b o t h higher frequency a n d lower a m p l i t u d e t h a n required for p r o d u c i n g tactile sensations beyond v i b r a t i o n , w h i c h is p o o r l y suited to dense spatial signalling. • S m a l l quantities of special m a g n e t o - o r e l e c t r o - r h e o s t a t i c  fluids  may be configured as a n array of separated cells. W h e n a current or magnetic field is applied to a cell, the encapsulated l i q u i d hardens, crea t i n g a tactile sensation when the user explores the array.  Concerns  w i t h this approach include long-term reliability of the fluids a n d membrane, exotic m i c r o - m a c h i n i n g requirements, response time, crosstalk between cells (especially w i t h the magnetorheostatic  approach), a n d  p o t e n t i a l l y l i m i t e d amplitude. •  Shape  memory  alloys  (SMAs)  m a y be used i n any n u m b e r of  configurations, such as b i m o r p h benders, to produce local deflections. However, because the m o t i o n is temperature-related,  crosstalk, heat-  sinking, a n d especially reaction t i m e remain significant concerns for a spatial array configuration.  O t h e r considerations relate to the position sensing layer, w h i c h must be flexible i f used o n t o p of a deflection p r o d u c i n g layer.  D e p e n d i n g on the  actuator technology, it must also be able to cope w i t h magnetic a n d / o r elec-  Chapter 3. Design Concepts  26  trie fields, w h i c h w o u l d cause problems for existing capacitive touch sensors. T h i s c o u l d p o t e n t i a l l y be cancelled out i n software, t h o u g h at a cost of effective resolution. A n alternative configuration consists of p u t t i n g the sensing layer underneath  (i.e., separated from the user's finger by) the actuation  layer, w h i c h could eliminate the  flexibility  requirement but p o t e n t i a l l y ex-  acerbate the interference problem. F o r the purposes of r a p i d p r o t o t y p i n g for experimentation, preparations were made to mount a p o s i t i o n sensing coil on the user's finger and to use a d i s t a l l y mounted d r a w i n g tablet (which works by radio frequency sensing at a distance of up to a few centimeters). L e a v i n g the specific choice of actuator and sensor technology open for the moment, interaction concepts were developed; an example is shown i n F i g u r e 3.4. A s s u m i n g a device that can produce localized orthogonal deflect i o n w h i l e t r a c k i n g the user's finger p o s i t i o n w i t h reasonable t i m e accuracy, force-feedback m a y be simulated w i t h o u t resort to motors that p h y s i c a l l y p u s h against the user's finger i n the lateral direction. T h e concept is m o d eled on the sport of surfing, where a continuously m o v i n g r a m p provides a lateral force on the rider due to the action of g r a v i t y and slippage along the surface. Similarly, a r a m p that tracks the user's finger a n d provides continuous " u p h i l l " resistance m a y be used to provide force feedback for a variety of interaction designs such as those described i n [17] that improve steering and navigation.  3.3  Application Concepts  U n d e r the w o r k i n g assumption of a linear, 1-degree of freedom b i d i r e c t i o n a l h a p t i c device capable of delivering several distinguishable tactile s t i m u l i combined w i t h finger position sensing, several a p p l i c a t i o n concepts  were  Chapter 3. Design Concepts  27  Bump tracks position and moves to stay ahead of finger  ii i i  m  i i i ii i i n i  F i g u r e 3.4: S i m u l a t e d force feedback using a m o v i n g b u m p that tracks the user's finger, p r o v i d i n g continuous " u p h i l l " resistance.  Chapter 3. Design Concepts  28  developed to better understand how such a device might be useful i n mobile interaction design.  For ease of understanding each concept, a contextual  scenario w i t h a fictitious example user is used to motivate the application.  Hyperlink sport. iu*dji»t>  M  Football Hockey  o  v  Target X  e  forward  Basketball  Iff  Auto Racing Curling Cricket  ® Dean « offline  ATLANTA - The BravftS ended a (out-game losing GtrsaK today with a 2-1 wtn over Detroit in overtime. The stadium was  (c)  (b)  (d)  F i g u r e 3.5: Overview of four application design scenarios, (a) L i s t selection, (b) Scrolling, (c) D i r e c t i o n signalling, (d) B a c k g r o u n d status notification. T h e figures shown as callouts represent haptic icons.  F i g u r e by Jerome  Pasquero and Joseph L u k .  3.3.1  List selection: R i n g e r mode application  (Figure 3.5, a, and Figure 3.6) Linda is in a meeting and wants to set  her phones ringer mode discreetly.  Grasping her phone inside her purse,  she explores the ringer mode menu by moving the selection highlight while receiving tactile feedback. Each menu item feels unique, like touching objects with different shape and texture, and she recognizes the sensation of the silent menu item because she has used this function before. She selects the silent mode and receives tactile feedback as confirmation. T h e scenario illustrates one way we can employ haptic icons [18], or  Chapter 3. Design Concepts  29  tactons — [10] brief, artificial tactile s t i m u l i to provide assistance i n making selections. A unique tactile stimulus is assigned to each i t e m i n a list menu; w i t h repeated feedback, users quickly associate functional meanings to abstract haptic icons [11, 12].  ooo  Tactile stimuli  F i g u r e 3.6:  E x a m p l e m u l t i m o d a l screen and haptic design for the list se-  lection a p p l i c a t i o n concept.  D i s t i n c t haptic icons are given an onscreen  representation i n this example as an associative a i d .  T h e piezo tactile display technology described previously is capable of displaying s m a l l simulated surface features, such as bumps and gratings, w i t h a r b i t r a r y m o t i o n relative to the users finger. It promises a rich vocab-  Chapter 3. Design Concepts  F i g u r e 3.7:  30  Low-fidelity foam m o c k u p used for early user evaluation of the  menu selection a p p l i c a t i o n concept. T h e r e is a textured plastic tactile strip mounted on the left side of the m o c k u p .  Chapter 3. Design Concepts  31  ulary of haptic icons, w h i c h are later characterized i n this document. B y m o u n t i n g the tactile display on a slider that is also sensitive to t h u m b pressure, it becomes an input device. T h e user can select items i n a v e r t i c a l list menu by m o v i n g the display up and down. A s the selection highlight is moved, the haptic icon associated w i t h the selected list i t e m is felt. K i n e s thetic awareness of finger p o s i t i o n allows the user to operate the device w i t h o u t looking, and to make a selection using the tactile display. T o gather i n f o r m a t i o n on user's impressions of this a p p l i c a t i o n concept, we constructed a low-fidelity foam m o c k u p (Figure 3.7) and asked people to hold it i n their hands and to imagine using the tactile sensations to help t h e m make menu selections. T h e subjective feedback was generally positive and helped guide our prototype hardware development ( C h a p t e r 4).  3.3.2  Scrolling: Browser application  (Figure 3.5, b , a n d F i g u r e 3.8) Bob checks the sports news and scores many  times each day. He didn't like using his old mobile phone's browser for this because he had to scroll around a lot to view content, which made him often lose his place. Bob accesses a sports website using his new haptically-enabled phone and scrolls down into a news story. He feels the sensation of his finger sliding over a textured surface while the text of the story moves up the screen. As he continues to scroll, he feels the headline of the next story (a distinct bump) and some links (each vibrates gently as it is highlighted).  All the stimuli  move smoothly past his finger in sync with the scrolling movement. Having scanned the page, Bob scrolls back up and quickly locates his area of interest (his home team's standings) aided by the memory of what that part of the  Chapter 3. Design Concepts  F i g u r e 3.8:  32  E x a m p l e m u l t i m o d a l screen and h a p t i c design for the browser  a p p l i c a t i o n concept, (a), T h e p o r t i o n of the page currently visible is shown as a highlighted region of the scroll bar.  (b),  Salient page elements are  represented as haptic icons and shown i n the scroll bar as an associative aid.  Chapter  3. Design Concepts  33  page feels like. Small-screen mobile devices t y p i c a l l y require more scrolling a n d / o r select i o n actions to navigate a deep rather t h a n wide i n f o r m a t i o n layout. B o t h place demands on the users visual attention.  H a p t i c augmentation as v i -  brotactile feedback has been shown to improve performance i n a h a n d h e l d scrolling task [46]. However, a compact multiple-element tactile display offers a d d i t i o n a l capabilities such as s m o o t h tactile flow rendering (a sensation m o v i n g across the skin). Different page elements, such as headings, images, a n d links, can be rendered as h a p t i c icons that are played when the user scrolls over t h e m . T h u s , each page has a n associated haptic m a p t h a t reflects its structure. Users learn to recognize familiar pages a n d c a n quickly scroll to desired sections or links. Improvements i n scrolling efficiency w o u l d encourage user behaviours such as scanning to understand page structure a n d context, a n d increase the amount of information that c a n p r a c t i c a l l y be presented o n a page.  3.3.3  D i r e c t i o n signalling: Assisted navigation application  (Figure 3.5, c) Mary is looking for a toy shop at a large, crowded shopping mall. Her location- and orientation-aware  mobile device helps her find the  shop with an active map and directions.  The device also provides haptic  feedback so she doesn't have to constantly look at the screen, allowing her to keep her eyes and ears on her surroundings. Mary holds the device discreetly at her side, with her thumb resting on the tactile display and pointing forward.  The tactile display repeatedly strokes  her thumb in the reverse direction (towards her back), indicating that the  Chapter  3. Design Concepts  34  device is pointed in the opposite direction from her destination. As she turns around, the sensation gradually weakens, then begins to build again in the opposite, forward direction; she is now correctly oriented. Mary starts walking while continuing to hold the device. The stroking becomes faster, indicating that she is approaching her destination. A n y a p p l i c a t i o n that assists the user i n finding a spatial target could utilize an expressive tactile display to convey a direction cue. O n a macro scale, this includes vehicle-based or w a l k i n g navigation tasks, where the user must travel to a destination. O n a s m a l l scale, the user could receive haptic assistance to orient a mobile device camera so image-recognition software can read a barcode or scene.  A p p l i c a t i o n s i n between include finding wireless  access points or other active d i s t r i b u t e d information devices, or people i n a search-and-rescue scenario. V i b r o t a c t i l e s t i m u l a t i o n at d i s t r i b u t e d points of the b o d y has been considered for navigation [20], but i n a non-intrusive handheld form factor, the display of tactile flow can be used to indicate 1-D direction. O t h e r parameters (e.g. speed, a m p l i t u d e , and wave shape) a d d information dimensions.  3.3.4  D i s p l a y of b a c k g r o u n d status i n f o r m a t i o n a n d alerts  (Figure 3 . 5 ,  c) Albert always feels in touch with his friends because they  all share "presence" [39] and location information with each other via their mobiles, with status notifications as they become busy or free. Albert is composing a text message to a buddy. His fingers are busy entering text, but occasionally he places his thumb on the tactile display to move the cursor, and feels a subtle repeating haptic icon that indicates his friend Steve has come online. Albert can continue with his task, aware of  Chapter 3. Design Concepts  35  the status change. Later, Albert is on the phone when his friend Dean goes offline. Albert feels a different haptic icon and is aware of Dean's status without having to interrupt his conversation or to remove the phone from his ear to look at the display. H a p t i c alerts are c o m m o n l y used on mobile devices, signalling events such as i n c o m i n g or dropped calls [16]. Simple, h i g h - a m p l i t u d e signals such as v i b r a t i o n can be perceived t h r o u g h c l o t h i n g and on various areas of the body, but more expressive tactile s t i m u l a t i o n requires direct contact w i t h sensitive parts of the skin, such as the fingertips. Therefore, it is best suited to situations where the device is being actively used and h e l d i n the hand, where the haptic feedback provides background status i n f o r m a t i o n . If the active a p p l i c a t i o n also makes use of haptics, the s t i m u l i used for background notification must be distinct from the foreground applications h a p t i c signals. E x a m p l e s such as this underscore the importance of designing h a p t i c icons i n the larger context of their anticipated usage, and e m p l o y i n g e m p i r i c a l d a t a relating to their group perceptual characteristics.  3.3.5  M i n i m a l l y I n t r u s i v e I n t e r f a c e for R i c h N a v i g a t i o n o f Music  (Figure 3.9) Rich enjoys listening to music on his mobile device while on the go. He recently downloaded 30 songs onto his mobile music player, and wants to categorize them into songs he likes and songs he doesn't like. This is rather time consuming when he has to listen to the introduction of the songs, so he is in the habit of using the fast-forward/cue  feature to skip to  the "main", or chorus section of the song. Unfortunately, on his old music  Chapter 3. Design Concepts  36  player this was a difficult task — since he never knew where the chorus section began, he often overshot it or was forced to listen to less relevant parts of the song before getting to the desired section. Moreover, since there were very few buttons on his miniature mobile music player, he had to press and hold the same button used to change tracks while visually monitoring the position counter, which was very difficult to do while walking or driving. Using his new haptically enabled mobile music player, Rich can literally feel the structure of the songs in his collection. He runs his thumb up and down the slider and feels bumps and textures where chorus sections begin and end. Stronger musical themes feel like more significant tactile features, while subthemes feel like weaker features. He can easily cue to any of these features without looking, by moving his thumb to a position with a tactile feature and pressing to select. With this method, he quickly sorts through his newly downloaded songs by repeatedly loading the next track, advancing to the chorus section, and listening to the main musical theme. He uses the device's rating feature to make note of which songs he likes, so he can enjoy their full length later. T h i s scenario is based on the work of G o t o [21], who described a m e t h o d to a u t o m a t i c a l l y parse a n d identify musical themes (chorus sections) i n popular music. O n e application of the technique was i n a retail music listening kiosk, where customers needed to sample m u l t i p l e tracks o n a C D a n d make a purchase decision w i t h i n a l i m i t e d amount of t i m e . B y p r o v i d i n g cues as to the structure of the music — i n c l u d i n g m a i n repeating themes as well as subthemes — the kiosk could help customers advance to the most salient, relevant, or memorable sections of music to make decisions about how m u c h they like the selection. T h e goal of r a p i d l y scanning tracks to identify a n d / o r categorize t h e m is shared by b o t h users of listening kiosks  Chapter 3. Design Concepts  —  Audio Player  37  —  •i) Volume I / \ Prev Track  Chorus sections  3:10  U2 Vertigo  4/6  \y Next Track  (  )  r" >— .) r \ )) i ) ( 1  Figure 3 . 9 :  (  J  ( C C D C(  \  c  J  E x a m p l e m u l t i m o d a l screen and haptic design for the r i c h music  navigation concept, (a), T h e p o r t i o n of the page currently visible is shown as a highlighted region of the scroll bar.  (b),  Salient page elements are  represented as haptic icons and shown i n the scroll bar as a n associative aid.  Chapter  3. Design Concepts  38  and users of portable music players. M o b i l e music players are further challenged b y l i m i t e d space for physical buttons, m a k i n g it less p r a c t i c a l to add more controls t h a n it is to add richer controls. N a v i g a t i o n of subthemes, m a i n musical themes, entire tracks, and entire albums or playlists forms a c o n t i n u u m of salience w h i c h can be m a p p e d to various levels of haptic interaction using a transducer w i t h sufficiently wide b a n d w i d t h .  39  Chapter 4  Handheld Prototype Development W o r k i n g from the i n i t i a l a p p l i c a t i o n concepts and hardware profile, we began an effort to create a high-definition, active hardware prototype that w o u l d enable significant investigation into the effectiveness of a portable  haptic  interface i n the hands of a user. T h e piezoelectric actuated lateral skin stretch technology introduced i n C h a p t e r 2 was selected for this project based on s u i t a b i l i t y for the p r o b l e m d o m a i n a n d interaction designs, as well as p r a c t i c a l considerations such as the m a t u r i t y of the technology (allowing for a prototype to be realized w i t h i n the scope of a master's thesis) and an existing relationship between  the  author's research group and the group developing the technology at M c G i l l University. Since m u c h of the work i n this chapter is the result of a h i g h l y collaborative effort between research groups at the U n i v e r s i t y of B r i t i s h C o l u m b i a and M c G i l l University, each section i n this chapter includes a description of the present author's i n d i v i d u a l c o n t r i b u t i o n to that aspect of the project. T h e hardware prototype, i n c l u d i n g the piezoelectric lateral skin-stretch concept, is described i n detail i n [43]. I n this thesis we shall focus on the aspects that are relevant for the overall interaction design process of mobile haptics.  Chapter 4. Handheld Prototype Development  4.1  40  Design Philosophy  Because the options for specific implementations of the concept of h a n d h e l d haptics are v i r t u a l l y u n l i m i t e d , we followed a design philosophy that was informed by a c o m b i n a t i o n of the conceptual design w o r k discussed earlier (including user needs and a p p l i c a t i o n concepts), resources available for cond u c t i n g the research, a n d general heuristics about p r a c t i c a l i m p l e m e n t a t i o n of the hardware concept.  4.1.1  Linear slide-mounted tactile display using piezoelectric actuated lateral skin stretch  A s mentioned i n previous chapters, the hardware concept consists of a linear (1-degree-of-freedom), slide-mounted tactile display based on the piezoelectric actuated lateral skin stretch technology, mounted i n a h a n d h e l d case and positioned under the user's t h u m b . W e believe this o p t i o n represents the best compromise for adding significant new h a p t i c functionality w i t h o u t diverging too far from contemporary mobile device norms. It w o u l d be conceivable to add m u l t i p l e degree-of-freedom  movement  and a d d i t i o n a l features, but the a d d i t i o n a l costs i n terms of size, weight, complexity, and power c o n s u m p t i o n must be justified against potential benefits. Since the benefits of rich mobile haptics have yet to be demonstrated conclusively, we felt it was best to adopt a conservative approach first; if significant benefits c o u l d be demonstrated, further iterations of the design could increase functionality i n the areas where there w o u l d be an expected benefit. T h e T D is placed on the left side of the device (accessible by the t h u m b on the left hand) because it is the most c o m m o n l o c a t i o n i n existing products  Chapter 4. Handheld Prototype Development  41  for side-mounted controls such as buttons or j o g wheels. T h e convention appears to be due to a preference, since most people are right-handed, for leaving the right h a n d available for j o t t i n g notes while t a l k i n g on the phone, or w r i t i n g on the screen of a P D A .  4.1.2  U s e of off-the-shelf h a r d w a r e  components  All of the components of the hardware prototype should be readily available, mass produced parts. T h i s design philosophy allows us to reduce costs and increase the speed of development, by focusing on the system integration issues rather t h a n detailed engineering of components. O u r final hardware prototype m a y be replicated by any facility w i t h basic mechanical and electronic p r o t o t y p i n g equipment such as a hand-operated m i l l and soldering tools, at a final b i l l of materials cost of around $380 U . S . dollars, e x c l u d i n g the F P G A and P C used for control. T h e major component of this cost, retail piezo actuators ($160), declines significantly (by a factor of 4 or more) if ordered i n quantity. T h e final size, weight, and power profiles of the prototype tactile display mechanism make it not currently suitable for deployment as a mass produced consumer product. However, each of the c r i t i c a l components m a y be m i n i a t u r i z e d w i t h i n d u s t r i a l micro-manufacturing facilities.  I n particular,  the piezo elements, w h i c h currently require 50 V D C and extend into the device w i t h a depth of 31.8 m m , m a y be specially manufactured as layered benders t h a t could reduce the voltage requirement to 8-10V while s i m u l t a neously increasing r i g i d i t y and thus reducing the required depth [45]. I n a d d i t i o n , the comparator and amplifier sections of the control electronics w o u l d be vastly simplified since reduced voltage w o u l d be well w i t h i n the  Chapter 4. Handheld Prototype Development  42  range of mass produced m i c r o m i n i a t u r e D C - D C converters and integrated amplifiers.  4.1.3  H a n d h e l d operation while connected to a host P C  A significant part of the c o m p l e x i t y of p r o d u c i n g mobile devices derives from integrating batteries and wireless communications, and p r o g r a m m i n g embedded computers — issues that are secondary to the h a p t i c interface questions we are t r y i n g to investigate. Therefore, we have retained a tether to a host P C and power supply, while packaging the input and output transducers i n a handheld form factor. T h i s allows us to investigate key aspects of the h a n d h e l d user experience using a m e t h o d similar to the " W i z a r d of O z " p a r a d i g m : to the user, it appears t h a t they are using a mobile device w i t h a few subtle wires added. Despite the tether, the overall form factor generally affords m o b i l i t y a n d users are able to conceptually distinguish the wires as additions possibly related to the testing status.  4.1.4  Author's  Contribution  T h e part of the design philosophy related to the form factor, especially the t h u m b - m o u n t e d p o s i t i o n i n g , is p r i m a r i l y the work of the author.  Other  aspects (off-the-shelf components and tethered operation) were heuristics t h a t were also shared i n parallel by the collaborator, Jerome Pasquero, and the thesis supervisor, prior to the beginning of the project.  Chapter  4.2  4. Handheld Prototype Development  43  System Overview  T h e hardware components of the system, i n c l u d i n g labelled signal paths, are shown i n F i g u r e 4.1. T h e major components of this system are described i n the sections below. Since this thesis discusses a design process, it is instructive to examine how the hardware specification evolved over time. W i t h i n a m o n t h of beginn i n g the hardware project, m a n y of the components h a d been established (Figure 4.2). A l o n g the way to the final design, the desk mount (which was expected to increase reliability and simplify cable management, but was not necessary because the cables were not as intrusive as expected) was eliminated, as was the provision for using a n off-the-shelf game controller to digitize analog position and b u t t o n signals, and a d d i t i o n a l pushbuttons on the device (though they could be added at any time if applications warrant).  4.3 4.3.1  Output Transducers Tactile O u t p u t Device (Tactile Display)  T h e tactile display ( T D ) has an active area of 8.7 x 6.4 m m and contains eight stacked piezoelectric bending motors separated by t h i n (0.5 m m ) brass rods. T h e bending "motors" (Piezo Systems m o d e l T 2 1 5 - H 4 - 2 0 3 Y ) are t h i n (0.38 m m thick) rectangular sheets of relatively rigid ceramic-like m a t e r i a l measuring 6.4 x 31.8 m m .  Internally they consist of two bonded layers  of alternately polarized piezoelectric material, such that when an electric field is applied to the motor, one layer contracts while the other expands, resulting i n a bending deformation of the entire slab. T h e piezo elements are m o u n t e d i n a new d u a l - p i n n i n g configuration that  Chapter  F i g u r e 4.1:  4. Handheld Prototype  Development  44  O v e r v i e w of hardware components of the handheld prototype  (final version used for user testing).  LO  LO  o o  CM Xi  o in  CO  PC  a o  a o o  CM  NTSC Composite Video  desk stand  Chapter  4. Handheld Prototype  F i g u r e 4.3: T h e tactile display,  Development  (a) M e c h a n i s m of action,  view, (c) E x p l o d e d view. Diagrams b y Jerome Pasquero.  46  (b) Assembled  Chapter  4. Handheld Prototype Development  47  optimizes the force delivered to the user's finger and is more efficient t h a n the cantilever mount used i n the V i r t u a l B r a i l l e D i s p l a y [30, 43]. Because the d i s t a l ends of the piezo elements are fixed (i.e., mechanically grounded to the chassis that is being held i n the user's hand), the p r o x i m a l ends of the piezo elements that are i n contact w i t h the user's finger move w h e n an electric field is applied. T h e p o r t i o n of the user's s k i n covering the region between the ends of two adjacent piezo elements is stretched when the elements move away from one another, and it is compressed when they move towards one another. T h i s activates the strain-sensitive mechanoreceptors i n the user's skin, p r o d u c i n g a tactile percept.  F i n a l l y , it has been demonstrated  that  this produces the illusion of the sensation of something pushing into the skin, as occurs when the finger is placed on a surface feature such as a s m a l l b u m p [23]. R a w piezo elements are shipped as layered slabs w i t h three electrodes: the two outer surfaces and an internal, central electrode.  Preparation in-  volves electrically connecting the two outer electrodes w h i l e creating an area to access the central electrode. U s i n g a technique developed by the M c G i l l collaborators, we abraded a s m a l l section of outer electrode a n d applied conductive tape, conductive paint, and i n s u l a t i n g tape to create the required connections. S p e c i a l solder flux provided by the piezo vendor was used to attach 30-gauge wire to each bending motor. N o t e that although the piezos we used require relatively h i g h voltage ( + / - 5 0 V ) , the current draw is on the order of a few m i l l i a m p s and thus they are not high power devices. Because the piezo bender essentially consists of two oppositely charged electrodes separated by a gap, its electrical behaviour at sub-resonant frequencies is essentially that of a slightly lossy capacitor. A s such, its steady state current draw is almost n i l , and p r o d u c i n g a deflection  Chapter  4. Handheld Prototype Development  48  by charging a n d discharging the plates also does not require high current. F u r t h e r details of the tactile display design can be found i n [43].  4.3.2  V i d e o Display  T h e video display is a conventional 2.5-inch colour T F T L C D m o n i t o r that includes driver c i r c u i t r y to accept N T S C composite i n p u t signals. In the handheld prototype, the video display is mounted i n the top half of the casing w i t h the driver b o a r d positioned alongside, rather t h a n under, the L C D panel. A n 8-pin r i b b o n cable c a r r y i n g the video signals emerges from the b o t t o m of the casing and is routed around the top of the casing to j o i n the power and signal wires. A second driver / interface b o a r d accepts the N T S C composite input v i a an R C A jack and 1 2 V D C power (from a standard A C adapter) a n d sends the signals out on the r i b b o n cable. After long t e r m use the video display experienced hardware problems unrelated to its usage i n the haptic prototype, and it was not possible to procure a t i m e l y replacement.  Therefore, for the user tests related to the  browser it was necessary to use a slightly larger 3.5-inch m o n i t o r affixed to the top of the case.  W h i l e this d i d not produce an appreciably different  screen image (both use N T S C composite i n p u t ) , it d i d cause a significant increase i n the weight of the top half of the device.  4.3.3  Author's  Contribution  T h e tactile display was designed m o s t l y by Jerome Pasquero, while the author was responsible for preparing the piezoelectric components, and p a r t i c i p a t i n g and p r o v i d i n g feedback i n design discussions. T h e video m o n i t o r is an off-the-shelf component; however, the author is responsible for its inclusion,  Chapter 4. Handheld Prototype Development  49  selection, and procurement.  4.4  Sensors  1  F i g u r e 4.4:  L o c a t i o n of i n p u t sensors,  (a) Resistive position sensor,  (b)  Push-to-select sensor. D i a g r a m s by Jerome Pasquero.  4.4.1  Slider P o s i t i o n Sensor  T h e entire T D is mounted on P T F E (Teflon) bushings that slide over 2.4-mm diameter steel rods, w i t h a travel of 1 1 m m . Slider position is acquired using a standard analog 1 0 K o h m potentiometer-type resistive position sensor. A c u s t o m A / D module (Section 4.5) digitizes the signal and sends it to the  Chapter 4. Handheld Prototype Development  50  control circuitry.  4.4.2  P u s h to Select Sensor  T o facilitate a "click to select" behaviour, the entire assembly can articulate i n an arc w i t h a travel of a p p r o x i m a t e l y 1 m m when the user pushes into the T D (i.e., perpendicular to the active surface). T h e movement is sensed by a standard 3 m m p u s h b u t t o n switch mounted on the distal end of the mechanism.  T h e switch is connected v i a the A / D m o d u l e to the control  circuitry.  4.4.3  Author's Contribution  T h e author was responsible for the idea of the selection functionality, and component selection for the position sensor.  A major part of the effort of  integration of the sensor components was i n achieving the right mechanical design for the slider and a r t i c u l a t i n g mechanism; this was done by Jerome Pasquero i n cooperation w i t h M c G i l l mechanical workshop collaborators Joe B o k a and D o n Pavlasek.  4.5  Interface E l e c t r o n i c s  C o n t r o l is achieved v i a a series of c u s t o m and off-the-shelf electronic m o d ules.  T h e tactile o u t p u t signal begins at the P C r u n n i n g the a p p l i c a t i o n  software, w h i c h sends out packets v i a a U n i v e r s a l Serial B u s ( U S B ) connection to a field programmable gate array ( F P G A , C o n s t e l l a t i o n b o a r d 1 0 K 5 0 E from N o v a Engineering) r u n n i n g custom firmware. T h e function of the F P G A is to convert the P C output into an 8-channel 156.25-kHz pulse w i d t h m o d u l a t e d signal. T h e P W M signal is then fed t h r o u g h a comparator,  Chapter  4. Handheld Prototype  Development  51  low-pass filter, and high voltage amplifier, resulting i n an 8-channel analog signal t h a t varies from + / - 5 0 V . (see F i g u r e 4.1). N o t e t h a t any equipment capable of generating a P W M signal may be used as the input to the comparator, filter, and amplifier stages. If desired, a standard I / O card, or embedded controller, m a y be connected i n lieu of the P C a n d F P G A , e l i m i n a t i n g the most costly and b u l k y components of the system and allowing for flexibility i n p r o t o t y p i n g . O n the i n p u t side, a standard 8-bit analog to d i g i t a l ( A / D ) converter is used to digitize the resistive p o s i t i o n signal.  4.5.1  •  Author's Contribution  Jerome Pasquero was p r i m a r i l y responsible for the design and specification of the interface electronics. T h e author was responsible for p a r t i c i p a t i n g i n design discussions (mostly to advocate the m a x i m u m level of  flexibility  for  r a p i d interaction design p r o t o t y p i n g ) , sourcing of some parts, and assembly of the c u s t o m electronics.  4.6  P o w e r Supplies  T h e power supply for the tactile output consists of a pair of standard regulated 4 8 V , 2 5 W power supplies ( K a g a C o m p o n e n t s L t d . P a r t # S P 3 0 U - 4 8 S ) n o r m a l l y used for telephone equipment. T h e fine-tuning voltage adjustment feature was used to tweak the output voltage to 5 0 V . A d d i t i o n a l power supplies include manufacturer-supplied standard A C adapters for the  FPGA  and L C D , and an internal + / - 1 5 V power s u p p l y derived from the m a i n + / 5 0 V source v i a voltage regulator.  Chapter 4. Handheld Prototype Development 4.6.1  52  Author's Contribution  T h e design a n d integration of the m a i n power supply is the work of the author.  4.7  C o n t r o l Software  T h e system is controlled by custom software modules on the F P G A  and  host P C . T h e F P G A software consists of device specific firmware that handles the translation of U S B packets into P W M output, and 8-bit A / D input to U S B packets.  It a d d i t i o n a l l y offers buffer management features and can  o p t i o n a l l y support a non-hosted mode i n w h i c h the F P G A operates i n a closed-loop fashion, w i t h piezo output dependent on a cached slider position function. T h e P C software layer is based on the " S T R e S S L i b r a r y " ,  1  [29] a cross-  p l a t f o r m A P I w r i t t e n i n C + + , featuring buffered input a n d output functions and support functions. It is implemented using various open-source libraries, i n c l u d i n g the ACE component l i b r a r y for X M L parsing, threading, and other support functions, and the l i b u s b l i b r a r y for I / O . A l l applications described i n this thesis were implemented on a 1 G H z P e n t i u m class P C r u n n i n g SuSe Linux. Internally, the + / - 5 0 V output signal is represented using 12 bits (4096 levels) on the F P G A , and the P C control software is able to handle b o t h 12-bit and 8-bit signals. In practice, it was found that there was no readily 1  The API is so named because it shares a large part of the code with an earlier project  [41].  Chapter 4. Handheld Prototype Development perceivable difference  2  53  between 12-bit and 8-bit resolution output, so the  8-bit (256 levels) signal was used for a l l the applications described i n this thesis. T h i s allowed for more efficient use of resources, since t i m i n g was a concern (see below).  4.7.1  Input a n d O u t p u t T i m i n g s  T h e basic software I / O loop runs at a rate of 333 H z . Once every 3 ms, new information (consisting of voltage levels for each of the eight piezos) is written to the F P G A ' s output buffer, and the input buffer (consisting of slider position and click status) is read into the P C . A l l implementations described i n this thesis were r u n on standard L i n u x w i t h o u t real-time extensions, so the actual 1/O timings are subject to system load conditions. E x p e r i m e n t s and performance o p t i m i z a t i o n s were performed to ensure stable real-world performance under load. F i g u r e 4.5 shows the measured o u t p u t at the amplifier ( w i t h the piezo load disconnected), while the system is r u n n i n g the full interactive browser prototype described i n C h a p t e r 6. In a d d i t i o n to generating piezo output, the system must read slider input, compute the output tactile image, and update the visual browser display.  Effect of o u t p u t t i m i n g on qualitative experience T h e top p o r t i o n of F i g u r e 4.5 shows the output of an early version of the software, w h i c h was able to produce signals at an irregular rate of approxi m a t e l y 12 H z , while the b o t t o m p o r t i o n of F i g u r e 4.5 shows the output 2  Although this observation is based on the subjective experience of the author and  other researchers, it is reasonable to assume that since the total piezo travel must be less than 1.0 mm, a 12-bit (0.2 / i m maximum) resolution is well beyond the limit of human perception, which is roughly 1 /im for small surface features [25].  Chapter 4. Handheld Prototype Development  54  F i g u r e 4.5: M e a s u r e d outputs under conditions o f load (running the browser application). Above, prior to o p t i m i z a t i o n . Below, after o p t i m i z a t i o n , showing two adjacent o u t p u t channels at once.  Chapter  4. Handheld Prototype  Development  55  after a series of software performance o p t i m i z a t i o n s i m p r o v e d the rate to a relatively stable 83 H z . A t the low (12 H z ) s a m p l i n g rate, the sensations produced by the T D were perceived by the author and other observers as rough, v i b r a t i o n - l i k e s t i m u l i w i t h o u t a clear sense of tactile flow, while the high (83 H z ) rate stimulus produced a m u c h smoother sensation. O n l y the high sample rate stimulus could be compared to the sensation of touching a small b u m p .  It is likely that the problems w i t h low sample rate s t i m -  ulus were the result of aliasing — undesirable high frequency  transients  are produced at the square edges of the wave, w h i c h become n o t a b l y more pronounced at the lower sample rate.  C o n t r o l software performance o p t i m i z a t i o n Software performance o p t i m i z a t i o n to improve the o u t p u t rate consisted of iterative development of b o t h the F P G A firmware and P C software to o p t i m i z e the buffering a l g o r i t h m for r a p i d , interactive, closed-loop operat i o n . T h i s was especially i m p o r t a n t for the browser a p p l i c a t i o n (Chapter 6) w h i c h , unlike the earlier applications used for simple, triggered playback of haptic icons and tactile movies (Chapter 4), could not "anticipate" and cache significant amounts of output for smooth buffered delivery. F u r t h e r performance improvement was achieved by r e w r i t i n g the applications to the number of threads, as interprocess c o m m u n i c a t i o n overhead was found to be a significant source of irregular delays. T h e final i m p l e m e n t a t i o n of the browser a p p l i c a t i o n uses only two "user" threads, plus one I / O thread i n a d d i t i o n to the various threads internal to the M o z i l l a browser. F i n a l l y , as described i n more detail i n C h a p t e r 6, the browser a p p l i c a t i o n software is designed to take advantage of the fact that the visual sense  Chapter 4.  •"^  Read Input Slider Position  Handheld Prototype Development  (a) (b)  56  Read Input Slider Position  Slider Model Smoothed Slider Position  Slider Model Smoothed  Velocity Model  Slider Position  Page Map Position  Velocity Model  Page Map Position  Document Model  Position Highlighted  Highlighted  Element  Element  Display Model  Page Map Model  I  Page Map Position  Document Model New Page Map  Highlighted: Element  Haptic Icon Model Display Model 8-Channel  Visual Output  Voltage Image  Haptic Icon Model  Output Smoother Smoothed  8-Channel Voltage Image  Output Smoother Smoothed Voltage Image  Visual Output  Voltage Image  I/O Layer  I  use  I  Analog  FPGA  I/O Layer  i  USB  Amplifier  FPGA  I  Analog  I  Analog  I  Analog  Tactile Output  Amplifier  Tactile Output  F i g u r e 4.6:  I m p l e m e n t a t i o n alternatives for the control software, (a),  single loop, (b), asynchronous design for improved tactile performance. computation-intensive M o z i l l a p o r t i o n is enclosed i n a dotted line.  large The  Chapter 4. Handheld Prototype Development  57  is less sensitive to t i m i n g irregularities t h a n the tactile sense.  Instead of  one large I / O loop i n c o r p o r a t i n g the M o z i l l a browser and its  document  m o d e l together w i t h the tactile c o m p u t a t i o n modules, the  final  software  uses separate, asynchronous loops for tactile a n d v i s u a l components (Figure 4.6). T h e tactile loop, a slim 500K C + + program, runs at a higher p r i o r i t y t h a n the v i s u a l loop (the a p p r o x i m a t e l y 2 0 M B M o z i l l a suite), and includes its o w n internal, simplified representation of the web page so that it does not need to wait for the v i s u a l component. T h e final resource usage of the full-blown control software stack, i n c l u d ing browser, averages around 60% of the 1 G H z C P U , as measured by the L i n u x "top" c o m m a n d .  4.7.2  Author's Contribution  T h e core control software architecture and i m p l e m e n t a t i o n was done by V i n c e n t Levesque. T h e author was involved i n helping test and verify the system, i n c l u d i n g the measurements and o p t i m i z a t i o n s ( w i t h the exception of the firmware update) described i n Section 4.7.1.  4.8  Tactile Flow Rendering  T h e concept of tactile flow is described i n various previous publications on related piezoelectric actuated lateral-skin stretch tactile displays [23, 28, 30, 41] and publications on the present prototype [34, 43]; however, since it forms the basis for m a n y investigations throughout this thesis, we shall review it briefly here. W h e n a finger is swept over a s m a l l surface feature,  patterns of s k i n  Chapter  4. Handheld Prototype  Natural Touch  Development  58  Synthetic Tactile Flow  t=o  t=3  LEGEND Compression Zone W\J  -5oV ' 0V T 5 O V  Stretch Zone  F i g u r e 4.7: M o v i n g patterns of s k i n stretch create a sensation of tactile flow. Left, the shear pattern associated w i t h dragging a finger over a s m a l l surface feature.  Right, synthetic re-creation of the pattern using the piezoelectric  lateral skin-stretch T D . P a r t s are not d r a w n to scale.  Chapter  4. Handheld Prototype  stretch a n d compression are produced [28].  3  Development  59  For example, the drag of the  feature against the skin creates shear compression along the p o r t i o n of s k i n b e h i n d the leading edge, and since the skin is a continuous elastic m a t e r i a l , the area ahead of the leading edge undergoes a complementary shear expansion, or stretch (Figure 4.7, left). Furthermore, this p a t t e r n of skin stretch and compression travels across the finger as it is moved over the feature. T h i s m o v i n g p a t t e r n of skin s t i m u l a t i o n is detected by mechanoreceptors and produces a sensory percept, w h i c h we have termed tactile flow. It is possible to simulate an analogous m o v i n g pattern of s k i n stretch a n d compression using the tactile display i n our handheld prototype.  Fig-  ure 4.7, right, shows how the voltage-actuated bending of the piezo elements creates regions of shear stretch and compression. N o t e t h a t at t i m e t = 3, the pattern of voltage a c t u a t i o n is the same as at t i m e t = 0, except that the signals are shifted to the left. I n o u t w a r d visual appearance, the sequential movement of adjacent piezo actuators d u r i n g tactile flow display looks similar to biological peristalsis (sequential muscle contraction), or the pattern of movement of millipede legs (Figure 4.8). Tactile flow is c o m m o n l y associated w i t h spatial movement, such as dragging one's finger over a surface to explore its texture, or losing one's grip on an object. It is a directional signal; moreover, it is more n a t u r a l (in terms of being more c o m m o n l y encountered i n d a i l y life, as one touches and explores objects) t h a n spatially d i s t r i b u t e d patterns of v i b r a t i o n [9, 50]. Therefore, it is hypothesized that tactile flow m a y be useful as a spatial cue for use i n n a v i g a t i n g computer interfaces. T h i s topic is covered i n more detail i n C h a p t e r 7, Browser User Study. 3  One can simulate this experience by sliding a finger over the tactile "keyboard hints"  — small bumps present on most computer keyboards at the F and J key positions.  Chapter  Figure 4.8:  4. Handheld Prototype Development  60  A millipede. Its legs move in a pattern that looks outwardly  similar to the movement of the T D ' s piezoelectric actuators when displaying a tactileflowsignal, except that the T D does not normally move relative to the user's finger.  Chapter  4.9 4.9.1  4. Handheld Prototype  Development  61  V i s u a l i z a t i o n of T a c t i l e S t i m u l i Problem  Depicting and communicating the tactile display output visually was a notoriously difficult problem. The output electrical signal is an eight-channel time-varying voltage ranging from +50V to -50V DC. Thus there is the problem of depicting Voltage x Time x 8 Channels using two-dimensional media. The output may be represented by drawing the eight signal traces in a stacked fashion on the same time axis, like the display of a hypothetical eight-channel oscilloscope (Figure 4.5 and Figure 4.9, a). This "parallel wave graph" approach is used in previous publications [30, 41]. However, this representation has several problems, including: • It only depicts the time-dependency aspect of the stimulus; it does not readily show the output as a function of slider position, which is the key feature that enables bidirectional interactivity and exploration of virtual texture that extends beyond the 8-taxel display window. • The presentation format is somewhat difficult to understand for a lay audience. It is most easily understood by a technical audience that is accustomed to reading signal trace graphs. • Because the vertical axis encodes two separate variables — piezo number and voltage — the spatial relationship between piezos is broken up by local axes (voltage). Thus the graph does not visually convey the propagation of a signal from one piezo to another that characterizes "peristaltic" motion or tactile flow. • The "parallel wave graph" format requires a lot of space to depict  Chapter  4. Handheld Prototype  Development  62  even a simple stimulus. A more compact representation w o u l d thus be desirable.  4.9.2  N o v e l  G r a p h i c a l  R e p r e s e n t a t i o n s  T o address these issues, a new way of t h i n k i n g g r a p h i c a l l y about the piezo output was developed as part. of the hardware development effort for the handheld tactile display.  T h i s includes zeroth-order shaded voltage m a p  and first-order skin stretch image representations, a n d a haptic icon design process that leverages the graphical representations for r a p i d p r o t o t y p i n g of tactile s t i m u l i .  4.9.3  S h a d e d  G r a p h  f o rV o l t a g e  Signal  B y replacing the height of a voltage trace w i t h a greyscale shading value, it is possible to achieve a more compact representation that eliminates the overloading of the v e r t i c a l axis w i t h b o t h voltage a n d spatial information. A s illustrated i n F i g u r e 4.9, b, the shaded graph enables more r a p i d comprehension b y v i s u a l inspection of the "shape profile" a n d spatial displacement pattern of the stimulus, as compared t o the parallel wave graph. A n o t h e r key feature of the shaded graph is that instead of s i m p l y representing eight piezo outputs, the v e r t i c a l axis m a y n o w be considered to encode taxels, or a r b i t r a r y spatial units along w h i c h tactile s t i m u l i are organized. T h e T D can be considered a display window, spanning exactly eight taxels, into this larger tactile space. Therefore, the shaded  representation  serves a d u a l purpose: to visualize the output of the eight piezo channels over time, a n d also to visualize a tactile feature map, o p t i o n a l l y one that evolves over time. T h e latter representation is used throughout the remainder of  Chapter 4. Handheld Prototype Development  63  +50V -50 V L +50V  (b)  -50VL +50V  LEGEND +50V  stretch  N  ov I I I I I I I I I I I I I 0  Time  -50V  12  Voltage Image  neutral  H  compression  Stretch Image  I I I I I I I I 0  Time  I I 12  F i g u r e 4.9: G r a p h i c a l representations of a stimulus consisting of three small bumps that travel up, and then down, the T D . (a) Conventional parallel wave graph, used i n previous publications, (b) Shaded graphs showing voltage and s k i n stretch (delta) images. T h e t i m e axis is expressed i n a r b i t r a r y units. Brass end-plates are not included i n this stretch analysis.  Chapter 4. Handheld Prototype Development  64  this thesis.  4.9.4  Skin Stretch Image  T h e raw voltage level at a given time s i m p l y corresponds to the static displacement of the piezo element at that time. T h e tactile percept, on the other hand, is produced by transient patterns of s k i n stretch that activate mechanoreceptors i n the user's finger. T h i s may be caused by: 1. S k i n stretch or compression produced by the movement of two piezos relative to each other. T h e area of skin i n the gap between the piezos undergoes the stretch or compression. 2. S k i n stretch or compression produced by the movement of a flanking piezo relative to the brass end-plates. 3. H i g h frequency v i b r o t a c t i l e activation of a piece of s k i n , i n w h i c h forces are produced against the i n e r t i a of the skin. 4. A c t i v e e x p l o r a t i o n of the static T D surface caused by the user m o v i n g or slipping their finger over the T D . O f these various causes, (4) is unrelated to the synthetic haptics being discussed here, and (3) is subject to the l i m i t a t i o n s of v i b r o t a c t i l e s t i m u l i t h a t we described i n C h a p t e r 2, and as such we consider it secondary to the rich lower-frequency activations made especially possible by the current T D design. B y p l o t t i n g the difference between voltages applied to adjacent  piezo  elements using a similar shading convention to the above, we can generate a graphical picture of the s k i n activation pattern due to causes (1) and (2). In the present work, we have chosen to use "bright" shading for "stretch"  Chapter 4. Handheld Prototype Development  65  (or expansion of the gap between two adjacent piezo-actuators), a n d "dark" shading for "compression" (narrowing of the gap). T h e resulting visual representation is k n o w n as a stretch image, a n d is contrasted w i t h the raw voltage image, sometimes abbreviated volt image. T h e stretch image reflects more d i r e c t l y the user's tactile experience of the stimulus.  S t r e t c h Image Caveats T h e stretch image is a n attempt to more accurately depict the tactile sensations produced b y the device, a n d it is applicable t o a l l of the s t i m u l i described i n this thesis, b u t it is still not a completely perfect picture. T h e following cases should be noted: 1. A s mentioned i n the previous section, the T D has sufficiently r a p i d response so as to be capable of p r o d u c i n g v i b r o t a c t i l e s t i m u l a t i o n that is not due to s k i n stretch created between adjacent piezo elements and therefore is not captured b y the s k i n stretch representation. F o r example, i f a l l eight piezo elements are v i b r a t e d i n synchrony, the s k i n stretch image representation w o u l d be a flat grey, equivalent to no a c t i v a t i o n at a l l , since the piezos are not m o v i n g relative to one another. F o r this reason, stretch images are always presented alongside the source volt image i n this work, as v i b r o t a c t i l e transients can be visually identified as areas of h i g h horizontal contrast i n the volt images even when they are not present i n the stretch images. 2. P a t t e r n s of static stretch do not produce a tactile percept.  F o r ex-  ample, the piezos could be moved into a p a r t i c u l a r configuration a n d kept there for a long time (seconds), p r o d u c i n g a m o m e n t a r y sensation when they are first moved, b u t one that dies out as the mechanorecep-  Chapter  4. Handheld Prototype  Development  66  tors adapt to the pattern of stretch. T h i s a d a p t a t i o n is not reflected  in the stretch image. T h e p o t e n t i a l p r o b l e m of too low-frequency s t i m u l a t i o n should be kept i n m i n d d u r i n g tactile experience design. F u r t h e r discussion of image-based compensation for mechanoreceptor a d a p t a t i o n can be found i n the following section.  4.9.5  Automated Tools for Design  W h e n tactile images are represented as a n array of eight greyscale pixels, they can be created, m a n i p u l a t e d , a n d analyzed using readily available tools for dealing w i t h visual ( b i t m a p or raster) images. U s i n g a paint a n d image m a n i p u l a t i o n p r o g r a m such as A d o b e Photoshop, tactile images c a n be easi l y painted using the mouse, a n d m a n y convolution filters m a y be applied w i t h q u a l i t a t i v e l y similar effects i n b o t h visual a n d tactile domains: blur, sharpen, add noise, contrast, a n d m a n y others fall into this category. Since the image format provides a n on-line visual preview of the tactile stimulus, the visual transformation produces a related tactile transformation. F o r example, b l u r r i n g an image visually, smooths ("blurs") the tactile sensation as well. Furthermore, stretch images can be a u t o m a t i c a l l y a n d r a p i d l y generated i n one click using Photoshop's "Custom" Filter. T h i s is a generic convolution filter t h a t accepts a user-specified kernel up to 5 x 5 i n size, a n d m a y be used to generate a wide variety of visual effects.  W h e n a p p l i e d using the  settings shown i n F i g u r e 4.10, it takes the difference between two v e r t i c a l l y adjacent pixels, normalizes it to the stretch image scale previously described, and outputs the image. Because the stretch image is one pixel smaller t h a n  Chapter 4. Handheld Prototype Development the  67  volt image, the top pixel needs to be cropped off after using this filter.  A l t e r n a t i v e l y , i f desired, the interaction of the two end piezos (#1 and # 8 ) w i t h the brass end-plates can be investigated s i m p l y by a d d i n g two rows of neutral grey pixels to the voltage image prior to convolution. T h e resulting stretch image, after cropping, w o u l d then be 9 pixels high. i;  Jiiiiu; ;.i jjiiiiiiiiij  I liliJij I. liii !l ijliiii i liJii I Ii iill i lijiil i i liUjli liJi i I liljjl lj Ni I !i ;•  C  l  J  S  t  0  m  JMM$^K^t$  _  ' O K (  Cancel ')  f ' toad... (  j  Sawe... ')  3 Preview  Scale: |2~~ ~] Offset: 127  F i g u r e 4.10:  P h o t o s h o p C u s t o m F i l t e r settings to automate creation of  stretch images. T h e depicted convolution kernel instructs the program to do the following:  Take the sum of the current pixel, multiplied by 1, and the  pixel immediately above it, multiplied by -1. Scale the total by dividing by 2, then add 127 (half of the full-scale value of 255).  M o r e sophisticated algorithms could be employed to improve the accuracy of the stretch image by t a k i n g into account adaptation: local areas of skin stretch that are held for long periods of time c o u l d g r a d u a l l y "fade to grey" by detecting non-changing pixels (stretch zones) and pushing t h e m towards grey on a timescale that is consistent w i t h e m p i r i c a l l y measured adaptation  phenomena.  Chapter 4. Handheld Prototype Development  68  F r o m a p r a c t i c a l standpoint, it was decided not to implement adaptation compensation i n the image processing for several reasons. F i r s t , the adapt a t i o n rate is not constant and is dependent on a number of factors such as psychological state, s k i n properties and i n d i v i d u a l factors. Second, a d d i n g a real-time scale to the horizontal axis of the images introduces significant c o m p l i c a t i o n that undermines their purpose as a r a p i d p r o t o t y p i n g and v i sualization t o o l . F i n a l l y , it is quite easy to v i s u a l l y detect and avoid long periods of unintentional lack of m o t i o n i n haptic icon designs; it is far more likely that such artifacts w o u l d result from higher-level effects such as interaction between haptic icons and the speed of the user's navigation. For these reasons it was decided that the p l a i n , uncompensated stretch image represented a good compromise between accuracy (correspondence of the v i s u a l representation w i t h the felt tactile sensation) and u t i l i t y as a p r o t o t y p i n g and v i s u a l i z a t i o n t o o l .  4.9.6  g i f 2 h a p t i c o n Tool  P r i o r to the i n t r o d u c t i o n of the gif 2hapticon t o o l , m u l t i c h a n n e l o u t p u t for piezo tactile displays was usually constructed algorithmically, w i t h tables produced i n M a t l a b or similar tools through s u m m a t i o n of sine functions, and used by applications i n a simple spatially m o d u l a t e d lookup loop. A l ternatively, using a Java-based application, the user c o u l d specify t e m p o r a l o u t p u t (a "tactile movie") by setting the value for each time-sample using onscreen slider widgets. T h e output from this a p p l i c a t i o n was an X M L - l i k e file encoding the sample "frames", w h i c h was read by a tactile movie player 4  4  X M L was used primarily because of its extensible nature, which ensures that future  functionality, such as more complex time/space relationships and support for different TDs, could be added without having to make major revisions to existing software. The  Chapter  4. Handheld Prototype  Development  69  p r o g r a m and output i n sequence to the hardware. B o t h existing methods required considerable t i m e and effort to generate a new tactile stimulus. I n practice, the s t i m u l i that were created tended to be selected on the basis of convenience of the generating a l g o r i t h m , rather t h a n the q u a l i t y of the tactile experience — for example, a l g o r i t h m i c a l l y "neat" simple sine waves or, w h e n using the slider approach, square or linear transitions.  A d d i t i o n a l l y , the algorithmic m e t h o d restricts the design of  haptic signals to a technical audience that is familiar and comfortable w i t h m a t h e m a t i c a l functions. A m e t h o d to allow more direct, r a p i d p r o t o t y p i n g of h a p t i c signals, leveraging the visualizations described previously, w o u l d enable more iterative design of s t i m u l i and exploration of the stimulus space. Leveraging the existing X M L framework, a t o o l called g i f 2 h a p t i c o n was developed by the author to r a p i d l y transcode raster images i n the G I F 8 9 a [2] format to the tactile movie X M L files, or to the newer "haptic icon" file format (also X M L based) described i n C h a p t e r 6. It is w r i t t e n i n C + + and uses the free I m a g e M a g i c k [4] l i b r a r y for reading G I F images. W i t h the g i f 2 h a p t i c o n utility, any software that is n o r m a l l y used for creating animated G I F images for the W e b m a y also be used to generate tactile movies. A sample workflow is shown i n 4 . 1 1 . Perhaps more i m p o r tantly, the first steps of this workflow, where haptic icons are designed i n a creative and exploratory process, are nearly identical to the creation of anim a t e d G I F icons for the W e b . Therefore, anyone w i t h experience i n creating simple W e b graphics can participate i n haptic icon design, thus enabling a actual implementation does not include a formal Document Type Definition (DTD) for reasons of expediency (although one could be added trivially in the future), therefore it is "XML-like" in its current state. 5  These methods were developed and used by the group at McGill University.  Chapter  4.  Handheld  Prototype  Development  70  m u c h wider audience t h a n those who are familiar w i t h creating matrices i n Matlab. T h e code for the  g i f 2 h a p t i c o n t o o l is shown i n A p p e n d i x C . It is a cross-  platform A N S I C + + application, as is the required I m a g e M a g i c k library. T h e general c o m m a n d line usage is:  g i f 2 h a p t i c o n -t <infile.gif>  <outfile.xml>  where - t specifies tactile movie mode and i n f i l e . g i f  and o u t f i l e . x m l  are the i n p u t and o u t p u t filenames, respectively. gif  2 h a p t i c o n is discussed further i n Browser P r o t o t y p e , Section 6, as  it also features a mode for creating icons that are read by that a p p l i c a t i o n .  4.9.7  Author's  Contribution  T h e above section is p r i m a r i l y the work of the author.  P r i o r to the au-  thor's involvement w i t h the project, the "parallel wave graphs" described i n Section 4.9.1 were developed by the M c G i l l collaborators, as well as a graphical representation, used i n a J a v a tactile movie t o o l developed by Jerome Pasquero, that showed the relative angular orientations of i n d i v i d u a l piezo elements (not discussed i n the b o d y text above for reasons of b r e v i t y ) .  Chapter  4. Handheld Prototype Development  71  Photoshop Paint / Draw Image  Imageready Add Motion Typical Web Workflow  F i g u r e 4.11:  XML  gif2hapticon  T h e process for creating a haptic icon using the image-based  workflow is similar to. the process of creating graphics for the W e b . I n this example, a l l creative steps are done i n A d o b e P h o t o s h o p a n d ImageReady, and the conversion to tactile signals is h a n d l e d b y a u t o m a t e d tools, w i t h the output of the P h o t o s h o p stage being a static G I F a n d the output from the ImageReady stage being a n animated G I F . T a c t i l e images of a r b i t r a r y size are supported. F o r the purposes of a simple tactile movie, the size w o u l d be 1 x 8 pixels, while a haptic icon ( C h a p t e r 6) or other s p a t i a l representation can have a greater height. A l l images represented here are volt images.  72  Chapter 5  Perceptual Characterization 5.1  Introduction  T h e development of the i n i t i a l a p p l i c a t i o n concepts and handheld tactile display hardware was guided by an understanding of the general capabilities of the lateral skin-stretch technology, and ideas for how it c o u l d address user needs i n mobile contexts.  T o proceed to the next stage of more detailed  a p p l i c a t i o n design, we needed to quantify how users perceive the haptic signals generated by the new hardware. W e then m a p p e d some of the regions of the haptic "vocabulary" (range of s t i m u l i that the device could generate), allowing us to assess s u i t a b i l i t y of the envisioned applications, and what s t i m u l i w o u l d best m a t c h the roles specified i n our concept designs.  1  W e used a similar approach to perceptual characterization as [12]. T h e core stimulus salience quantification m e t h o d u t i l i z e d m u l t i d i m e n s i o n a l scaling ( M D S ) , a t o o l for a n a l y z i n g perception i n complex stimulus spaces [21]. G i v e n a range of s t i m u l i , M D S analysis produces maps of how the perceptual space is organized. :  A version of this chapter has been published.  Luk, J., Pasquero, J., Little, S.,  MacLean, K . , Hayward, V., Levesque, V . (2006). Haptics as a Solution for Mobile Interaction Challenges: Initial Design Using a Handheld Tactile Display Prototype, in Proceedings of ACM Conference on Human Factors in Computing Systems, CHI '06, Montreal, Canada, April 2006.  Chapter 5. Perceptual Characterization  73  O u r new hardware can generate m o v i n g s t i m u l i , but the range of detectable movement speeds was not k n o w n . W e therefore performed a study to estimate this range. T h i s enabled us to select speeds for icons for later M D S analysis.  5.2  Author's Contribution  T h i s chapter was previously published i n [34], a n d represents the collaborative effort of authors on that paper, i n c l u d i n g Jerome Pasquero, Shannon L i t t l e , K a r o n M a c L e a n , V i n c e n t Levesque, a n d V i n c e n t H a y w a r d . T h e author p a r t i c i p a t e d i n every phase of this research, i n c l u d i n g software development, h a p t i c icon design, r u n n i n g the experiments, d a t a analysis, presenting and w r i t i n g up the findings, decision m a k i n g a n d project management. A l l user trials i n the subgroup M D S experiment a n d the i n i t i a l d a t a analysis for the speed study were conducted by the author.  5.3  S t u d y 1 - R a n g e of P e r c e i v a b l e S t i m u l u s Speed  T h e purpose of the speed s t u d y was to determine the available perceptual b a n d w i d t h i n one possible dimension that could be used as a parameter for h a p t i c icon design. T h e question we sought to answer was: " W h a t is the upper l i m i t on the movement speed of a v i r t u a l 'shape' t h a t people are able to perceive?"  T o estimate the range of useable stimulus speed  we hypothesized t h a t the users' ability to perceive the movement direction w o u l d decrease as speed increased.  Chapter 5.3.1  5. Perceptual  Characterization  74  Speed Study - Experiment Design  W e used a simple m o v i n g stimulus consisting of a square waveform that was "tweened" across the tactile display to achieve a sense of m o t i o n (Figure 5.1). T w o waveforms were used, p r o d u c i n g either a m o v i n g region of s k i n expansion ("stretching") followed by compression ("pinching"), or compression followed by expansion. T h e m a x i m u m stimulus speed was l i m i t e d by the hardware s a m p l i n g frequency to 3.40 m / s (taking 2.56 ms to cross the display). W e conducted a simple pilot s t u d y among the authors to determine the approximate appropriate speed range for testing, setting the lower speed b o u n d to a region where stimulus detection accuracy reached a plateau. T h e independent variables were:  speed (0.17 to 3.40 m / s ) ; d i r e c t i o n  (up or down); and wave type (stretch-compress or compress-stretch).  The  dependent variables, measured w i t h a forced-choice m e t h o d , were: perceived direction (up or down), yielding an accuracy measure w h e n compared to the actual direction, and confidence level (confident or guess).  5.3.2  Speed Study - Procedure  T h e trials were conducted on a L i n u x P C w i t h the tactile device attached. O n each t r i a l , the computer selected r a n d o m values for each independent variable. T h e user pressed a G U I b u t t o n labeled " P l a y " to feel the stimulus, w h i c h was repeated three times w i t h a n intervening delay of 0.7 second. T h e user was then required to input the perceived direction a n d confidence level before proceeding to the next t r i a l . T h e r e were five " t r a i n i n g " trials where the user was informed of the actual direction v i a a m o d a l dialog box just after entering their responses, followed by 40 "test" trials where the user received no notification.  Chapter 5. Perceptual Characterization  (a) piezo #8  75  base waveform +50V -50V  3.5 m/s, down  LEGEND  time (msec)  F i g u r e 5.1: E x a m p l e s of s t i m u l i used for the speed study, (a) Voltage signal for one piezo element, (b) P a t t e r n of lateral skin stretch produced w i t h the 3.5 m / s stimulus, (c) P a t t e r n of lateral s k i n stretch produced w i t h the 1.8 m / s stimulus. T h e highlighted area represents one tactile frame i n w h i c h there is the sensation of stretching and compression at opposite ends of the display.  Chapter 5.3.3  5. Perceptual  Characterization  76  Speed Study - Results  8 right-handed volunteers (5 male, 3 female, aged 20-40 years old) p a r t i c i pated i n the user study. E a c h user took a p p r o x i m a t e l y 5-10 minutes to r u n the study.  O-  O-  O-  O-  O  y  O-  Speed  F i g u r e 5.2:  O-  O-  O-  \ - *b-  (m/s)  Results from the investigation of perceivable range of stimulus  speed. T h e heavy line is the p o l y n o m i a l trend line; measured d a t a points are i n grey.  T h e overall accuracy results from the speed s t u d y are shown i n F i g u r e 5.2. T h e relationship of accuracy and speed was s t a t i s t i c a l l y significant w i t h (x  2  =43.00, p<0.01), s u p p o r t i n g the experimental hypothesis. A c c u r a c y fell  to a p p r o x i m a t e l y chance levels at the m a x i m u m speed of 3.40 m / s , but approached 90% at 0.34 m / s using a p o l y n o m i a l regression.  T h e measured  accuracy at 0.19 m / s and 0.31 m / s appears to be lower t h a n the surrounding d a t a points. W h i l e likely due to r a n d o m variation, this observation is  Chapter  5. Perceptual  being further investigated.  Characterization  77  A t . t h e higher speeds, users reported that the  stimulus felt like a "click" or s m a l l v i b r a t i o n and t h a t direction was difficult to ascertain. N o significant effect was found for wave type (x  2  = 1-87, p>0.01). User-  reported confidence level decreased as the speed was increased (x  2  = 165.49,  p<0.01).  5.3.4  Speed Study - Discussion  T h e results from the speed s t u d y show that the device is capable of signalling the direction of stimulus movement over a large range of speeds. T h e sensation experienced is comparable to sliding one's finger across a surface w i t h a s m a l l b u m p . It thus seems feasible to use a directional "tactile flow" signal i n applications such as assisted navigation. In addition, the results suggest that speeds lower t h a n a p p r o x i m a t e l y 0.34 m / s w o u l d be appropriate for designing abstract haptic icons that convey the sense of m o t i o n .  5.4  Study 2 - Haptic Icon Discrimination Experiment  T h e purpose of the haptic icon d i s c r i m i n a t i o n experiment was to assess the range and d i s t r i b u t i o n of perceivable difference of some specific haptic icons rendered w i t h this device. T h e m u l t i d i m e n s i o n a l scaling ( M D S ) technique was used to m a p the organization of the stimulus space.  Chapter  5. Perceptual  Characterization  Factor  Levels  waveform  t r i , roll, saw, b u m p , edge  amplitude  ± 5 0 V ("full"), ± 2 5 V ("half")  speed  0.34, 0.23, 0.17 (m/s)  duration  ( C a l c u l a t e d from waveform and speed)  78  t r i : {480, 720, 960} (milliseconds) roll: {221, 331, 442} saw: {86, 130, 173} b u m p : {74, 110, 147} edge: {74, 110, 147} Table 5.1:  5.4.1  S t i m u l i used i n the M D S studies  M D S Study - E x p e r i m e n t a l Design  T h e s t i m u l i were selected according to a 5 waveforms x 2 amplitudes x 3 speeds factorial combination, resulting i n 30 haptic s t i m u l i (Table 5.1 and F i g u r e 5.3). These factors roughly correspond to stimulus components used i n prior studies for tactile displays [11, 19]. T h e waveforms were chosen to represent q u a l i t a t i v e l y different tactile experiences based on first-pass experimentation w i t h different signals, and included b o t h repeating and non-repeating waveforms. For the speed parameter, we chose a range that produced an accuracy rate approaching 90% i n the prior speed study. A fourth "meta-parameter", duration, was calculated from the speed and waveform parameters, and represents the t o t a l amount of t i m e the stimulus is present under the user's finger.  W e hypothesized that this  might be perceptually relevant and tracked it i n later analyses.  parameter  Chapter 5. Perceptual Characterization  79  +50V  -50V +50V -50V +50V -50V +50V -50V +50V -50V  1  20  40  60  80  100  120  140  sample (x 320LLS) F i g u r e 5.3:  5.4.2  Waveforms used i n the M D S studies.  M D SStudy - Procedure  T h e participants completed five stimulus-sorting blocks i n a m e t h o d similar to that used i n [36] and [52]. T h e sorting m e t h o d is a way to efficiently measure perceptual s i m i l a r i t y between pairs of s t i m u l i .  P a r t i c i p a n t s were  seated at a w o r k s t a t i o n and operated the mouse w i t h the right h a n d while h o l d i n g the device i n their left h a n d  2  w i t h the t h u m b resting on the tactile  display. Slider position was ignored. P a r t i c i p a n t s used a G U I that presented the 30 s t i m u l i i n a g r i d of a p p r o x i m a t e l y 1 c m 2  2  tiles. T h e y could trigger  Only left handed operation is supported on the handheld prototype. This is also the  case with almost all commercial handheld products having side-mounted controls. See Section 4.1.  Chapter 5. Perceptual  Characterization  80  stimulus playback by clicking a tile w i t h the left mouse b u t t o n , a n d used the right mouse b u t t o n to pick up, move, a n d drop the tiles into a p p r o x i m a t e l y 7 c m regions on the screen, w h i c h represented clusters. O n the first block, 2  they could adjust the number of clusters using onscreen + / - buttons.  In  subsequent blocks, they were required to produce 3, 6, 9, 12, or 15 clusters, presented i n r a n d o m order; the number of clusters closest to the user-selected for the first block was excluded. W e also collected qualitative feedback from users i n a post-task interview, seeded w i t h the following questions: •  " H o w w o u l d y o u describe the tactile sensations y o u experienced to someone who h a d not experienced them?"  •  " W h a t aspects of the device felt comfortable or n a t u r a l to use, a n d what aspects d i d not?"  •  " C a n y o u suggest any applications of the tactile sensations for a mobile device?"  5.4.3  M D S Study - Results  T e n right-handed i n d i v i d u a l s (7 male, 3 female, aged 19-31 years old) part i c i p a t e d i n the study, a n d were compensated $10 C a n a d i a n dollars. A l l subjects completed the tasks w i t h i n one hour. W e performed an M D S analysis on the d a t a obtained from the sorting task. S t i m u l i that are sorted together into a cluster were assigned pairwise s i m i l a r i t y scores p r o p o r t i o n a l to the t o t a l number of clusters i n a given sort, because it is reasoned that w h e n a user has more clusters from w h i c h to choose, the significance of placing two s t i m u l i together i n a cluster is  Chapter  5.  Perceptual  81  Characterization  increased.  '7  N  13  ill**  1  7  *  \  N  /  ! / ' ^  # 7 7 ;  7  \  96  J  , /  x  •  /  A72\  \  \  •  *  OROLL  half amplitude  F i g u r e 5.4:  /  v  LEGEND  A 77?/  \  •/«//  >=*EDGE amplitude  _>  OSAW OBUMP  number : duration  (xlOms)  Results from the M D S analysis of haptic icons. E a c h point  represents a stimulus, and dotted lines illustrate stimulus groupings.  The  axes m a y be rotated arbitrarily.  T h e results from a two-dimensional M D S performed w i t h o r d i n a l , untied d a t a are shown i n F i g u r e 5.4. A n a l y s e s i n 3-D and higher dimensions d i d not y i e l d any a d d i t i o n a l s t r u c t u r a l information about the data. T h e graph clearly indicates that users tend to structure the stimulus space i n terms of waveform, w i t h the t r i s t i m u l i clearly distinguished, and roll s t i m u l i also being separated from the non-repeating waveforms b u m p ,  Chapter  5. Perceptual  Characterization  82  edge, and saw. T h e s t i m u l i formed by the three non-repeating waveforms — b u m p , edge, and saw — were less clearly distinguished on the graph, i n d i c a t i n g that users d i d not consistently sort t h e m separately from one another.  T h i s suggests  that the differences between these waveforms are not perceptually salient, possibly due to l i m i t a t i o n s of the hardware or s k i n sensitivity. A d d i t i o n a l l y , because the experimental p a r a d i g m uses relative perceptual data, the d o m inance of the repeating / non-repeating waveform difference may obscure subtle differences among the non-repeating waveforms [7]. A closer e x a m i n a t i o n of the graph suggests t h a t d u r a t i o n and a m p l i t u d e m a y also be salient perceptual dimensions, but their organization i n the overall M D S graph is not consistent.  However, when subsets of the d a t a  were analyzed one waveform at a time, most of the graphs e x h i b i t e d clear d u r a t i o n and a m p l i t u d e structure along the x— and y—axes.  Because the  d a t a was collected i n a task where users were required to sort a l l stimulus factors at once, we hypothesized that because the less salient dimensions are perceived qualitatively differently depending on waveform, a global M D S sol u t i o n was unable to represent t h e m a l l consistently. W e therefore performed an a d d i t i o n a l experiment to determine the validity of the subset analysis.  5.5  Study 3 - Subgroup M D S Experiment  T h e purpose of the subgroup M D S experiment was to determine whether more subtle stimulus factors could be detected when the waveform was not varied. 3  3  A detailed analysis of the subgroup methodology is published in [42].  Chapter 5. Perceptual  5.5.1  Characterization  83  Subgroup M D S - Experimental Design  T h e subgroup M D S experiment consisted of four trials: a c o n t r o l t r i a l similar to the first M D S experiment, a n d three subgroup trials where users performed sorting tasks using i n d i v i d u a l waveform subgroups. W e chose the t r i , roll, a n d edge waveforms for further analysis because the earlier M D S analysis a n d qualitative reports i n d i c a t e d that they were j u d g e d to be the least similar.  5.5.2  Subgroup M D S - Procedure  T o avoid fatigue resulting from the increased number of trials, we reduced the number of sorting blocks per t r i a l . F o r the control t r i a l w i t h 30 s t i m u l i , subjects performed three sorts w i t h a user-selectable number of clusters i n the first sort, a n d 5, 10, or 15 clusters i n subsequent sorts, presented i n r a n d o m order w i t h the number of clusters closest to the user-selected number of clusters i n the first sort excluded. F o r the waveform subgroup trials using 6 s t i m u l i , after the first t r i a l the clusters were 2, 3, or 4 clusters using the same presentation a n d exclusion criteria. T h e control t r i a l was presented first, followed by the three waveform subgroup trials i n r a n d o m presentation order. A l l other d a t a collection methods were the same as i n the first M D S experiment.  5.5.3  Subgroup M D S - Results  F i v e right-handed people (3 male, 2 female, aged 19 to 35) p a r t i c i p a t e d i n the subgroup experiment. N o n e h a d p a r t i c i p a t e d i n a previous experiment w i t h the device. P a r t i c i p a n t s were p a i d C A D $20 for a 90-minute session. T h e subgroup M D S results confirmed the findings from the earlier sub-  Chapter  F i g u r e 5.5:  5.  Perceptual  Characterization  84  . Results from the subgroup M D S study, (a) C o n t r o l t r i a l w i t h  all 30 s t i m u l i , (b) t r i s t i m u l i (c) r o l l s t i m u l i (d) edge s t i m u l i . T h e results exhibit organization along the dimensions of d u r a t i o n and a m p l i t u d e .  Chapter  5. Perceptual  Characterization  85  set analysis, w i t h d u r a t i o n and amplitude being clearly employed by users to organize the stimulus space. F i g u r e 5.5 indicates no clearly discernible d u r a t i o n / a m p l i t u d e organization i n the control t r i a l graph w i t h a l l 30 s t i m u l i , but w h e n i n d i v i d u a l waveforms were tested separately, the organization became apparent. I n the subgroup graphs, d u r a t i o n is aligned v e r t i c a l l y and a m p l i t u d e horizontally. A d d i t i o n a l l y , the d a t a from the control t r i a l e x h i b i t e d the same overall structure as the d a t a from the first M D S study, p r o v i d i n g further confirm a t i o n of the original results and the robustness of the technique despite differences i n the number of clusters used i n the sorting task.  T a k e n to-  gether, the results indicate that d u r a t i o n and amplitude, w h i l e secondary to some differences between waveforms, are nevertheless discernible and useful as salient parameters for haptic icon design i n this environment.  5.6  S u m m a r y of P e r c e p t u a l C h a r a c t e r i z a t i o n Findings  T h e results from the three perceptual characterization studies suggest that users are capable of distinguishing a wide variety of s t i m u l i produced by the hardware prototype.  D i r e c t i o n , certain waveforms, d u r a t i o n , and am-  plitude are salient parameters that may be used i n designing haptic icons for use i n applications. T h e three-way grouping we observed a m o n g waveforms was especially interesting, because it e m p i r i c a l l y suggests how our first-pass parameterization m o d e l of haptic icons could be improved; for example, i n stead of treating waveform as a single parameter, i n subsequent designs one could consider non-periodic versus periodic waveforms, a n d further s u b d i vide the periodic group into different wave shapes (e.g., t r i versus r o l l i n the  Chapter  5. Perceptual  Characterization  86  present experiment).  5.6.1  Qualitative Findings  D u r i n g user evaluation we were also able to learn how people perceive the device qualitatively. T h i s information is especially useful for determining how users w o u l d perceive the value of the proposed applications. T h e key findings are s u m m a r i z e d as follows: • U n i v e r s a l l y ( N = 1 5 / 1 5 ) , participants d i d not find the s t i m u l i annoying or disruptive. M a n y participants reported that they preferred t h e m to their mobile phone's v i b r a t i o n mode. A variety of reasons were given, i n c l u d i n g quiet operation and moderate stimulus amplitudes. • M a n y ( N = 8 / 1 5 ) participants volunteered that they w o u l d find this type of tactile stimulus useful for alerts a n d notifications, such as identification of who is calling, information about a w a i t i n g message, or an a l a r m . • Some ( N = 3 / 1 5 ) participants experienced m i l d tactile fatigue, usually expressed as numbness, w h i c h was overcome by repositioning the finger to use a different part of skin, or t a k i n g a brief (approx. one minute) break. • In general, participants said they found the device comfortable to hold and ergonomically suitable for the tasks. Since the sliding function was not used i n the perceptual characterization studies, it is not k n o w n whether this report w o u l d be affected by using the slider for input.  Chapter 5. Perceptual Characterization  5.7  87  P e r c e p t u a l c h a r a c t e r i z a t i o n findings for a p p l i c a t i o n design  W i t h some quantitative a n d qualitative d a t a on low-level user perception of the prototype device, we can now consider whether the applications origin a l l y envisioned for the device are indeed appropriate, a n d to proceed w i t h the next steps of a p p l i c a t i o n design.  5.7.1  List selection  J u d g i n g from the results of the perceptual characterization, haptic icons designed along the dimensions of waveform (periodic or non-periodic), durat i o n , a n d direction are candidates for distinguishing items i n a list. Because the most salient parameters are the direction and speed of the stimulus, it is i m p o r t a n t to decouple this rendered m o t i o n from illusions of relative stimulus m o t i o n generated as a result of the v o l u n t a r y t h u m b  movements  to produce control input to the system. One way of avoiding this confound is to signify a discrete c o m m a n d such as scrolling an i t e m up or d o w n w i t h a larger but mechanically grounded gesture that incorporates pressing the slider against an end-stop.  5.7.2  Scrolling  A s o r i g i n a l l y envisioned, the browsing application uses rendered speed and direction parameters to provide haptic feedback to the user about the movement of the point of focus w i t h i n the page. H a p t i c shape (waveform) is the only parameter available to provide information about the selected i t e m (link, image, heading, etc.) However, the two M D S studies suggest that the  Chapter  5.  Perceptual Characterization  88  user's a b i l i t y to discriminate haptic shape w i t h this device may be somewhat l i m i t e d when using non-periodic signals. It is possible to b u i l d and test the browser a p p l i c a t i o n using the currently identified set of haptic icons, but its usefulness m a y be l i m i t e d by the relatively narrow choices of icons. A l t e r n a t i v e next steps include (a) reiterating the haptic icon design and perceptual characterization stage to discover more choices for haptic shape; (b) re-examining the rendering m e t h o d and electronic I / O  characteristics  to m i n i m i z e electronic and mechanical filtering that may be reducing the resolution and b a n d w i d t h of the haptic signal output; and (c) reconsidering the mechanical construction of the tactile display itself w i t h the a i m of further amplifying signal strength and thus, presumably, the potential distinctiveness of different waveforms.  5.7.3  Direction signalling  T h e location-finding a p p l i c a t i o n concept relies on the tactile display's a b i l i t y to convey direction information to the user.  T h e user studies confirmed  that direction of tactile flow is clearly distinguishable across a useful range of speeds.  Intensity, waveform and r h y t h m of repeating s t i m u l i m a y be  used to provide a d d i t i o n a l information about the distance to the target, status, or movement of the target. O u r results thus encourage p r o t o t y p i n g and usability testing for this application according to the original design concept.  5.7.4  Alerts and background status indicators  User feedback obtained d u r i n g interviews following the perceptual characterization sessions indicated strong potential for using the device for alerts,  Chapter  5. Perceptual  Characterization  89  based on the judgment that it w o u l d be pleasant and non-intrusive compared to currently available vibrotactile displays. D a t a from the perceptual characterization suggests a hierarchy of salience that could be m a p p e d to the relative importance or urgency of an alert. F o r example, a periodic signal w o u l d be useful for i m p o r t a n t alerts due to its high saliency.  Less i m p o r t a n t changes i n background status, such as the  movement of passively m o n i t o r e d "buddies", c o u l d be conveyed w i t h nonrepeating signals. F i n a l l y , i f background status indicators are to be m u l t i p l e x e d w i t h other haptic signals generated by the foreground (currently in-use) a p p l i c a t i o n , one of the dimensions identified i n the user studies could be allocated for this display. F o r example, if the speed dimension was allocated to background status indicators, slow m o v i n g s t i m u l i could be used for the foreground application, while fast-moving s t i m u l i could indicate background alerts. However, because of the l i m i t e d set of currently k n o w n salient haptic s t i m u l i , it w o u l d be advisable to perform another iteration of haptic icon discovery before allocating a large chunk of the "vocabulary" to background indicators.  90  Chapter 6  Browser Prototype H a v i n g constructed a hardware prototype a n d characterized its expressive capabilities i n the hands of users, we were ready to begin p r o t o t y p i n g a n d testing one of the a p p l i c a t i o n concepts described i n C h a p t e r 3. W e chose the browser concept to investigate first, because it is the most generalizable i n terms of a n abstract scrolling / navigation model, unlike more specialized interfaces such as the music player. A browser w i t h tactile enhancement is also a significant departure from existing approaches, unlike the notification scenario, w h i c h could be considered an evolutionary improvement of existing vibrotactile notifications. F i n a l l y , the concept of s i m u l a t i n g the tactile sensation of spatial movement using tactile flow rendering, as an a i d to cursor navigation, has not been previously studied. T w o levels of browser prototypes are described i n this chapter. I n Sect i o n 6.2 we describe a simple image browser, allowing v i e w i n g a n d scrolling 1  of a n image w i t h a n associated "tactile track", a n d a n i n f o r m a l user evaluation conducted to roughly determine the expected user experience w i t h a higher fidelity prototype.  T h e remainder of the chapter is devoted to the  design a n d i m p l e m e n t a t i o n of the high-fidelity tactile web browser, w h i c h is 1  The image browser was implemented by Vincent Levesque and Jerome Pasquero based  on a collaborative design, and the user study was conducted by the author with contributions to the experimental design from the McGill collaborators and thesis supervisor. The image browser is also described in [42].  Chapter  6. Browser Prototype  91  capable of reading web pages, a u t o m a t i c a l l y generating h a p t i c maps based on web content, and s u p p o r t i n g interactive n a v i g a t i o n w i t h b o t h visual and h a p t i c feedback. A formal user evaluation of this browser is described i n the following chapter.  6.1  Design Goals  T h e purpose of b u i l d i n g the browser prototype is (1) to explore one of the aforementioned a p p l i c a t i o n areas i n depth, (2) to discover what challenges exist for i m p l e m e n t i n g effective haptics i n this a p p l i c a t i o n area using the piezo tactile display device, and (3) to assess its effectiveness w i t h u s a b i l i t y testing. A s mentioned i n C h a p t e r 4, the a i m was to develop, test, and validate a n a p p l i c a t i o n i n the handheld, tethered environment before going to the m u c h more difficult and expensive mobile, untethered context. A s such, the browser software does not need to be concerned w i t h wireless issues and variable content delivery relative to signal quality. W e b standards such as H T M L , C S S , and JavaScript are used instead of wireless standards such as W M L , M I D P , and B r e w . W h i l e this decision p o t e n t i a l l y l i m i t s the untethered capabilities using the current browser architecture, it also greatly simplifies the design a n d eliminates costly technology licensing, allowing use of a fully open source software architecture. U s i n g W e b standards means that the browser is capable of interacting w i t h live content from remote servers. However, for the purposes of u s a b i l i t y testing, l o c a l l y cached web pages are used. T h e browser uses s t a n d a r d web formatting and display constructs, but does not a t t e m p t to a u t o m a t i c a l l y reformat pages originally designed for P C screens.  Just as w i t h mobile  Chapter  6. Browser Prototype  92  devices b u i l t around existing W e b standards, content creators w i l l use the same techniques for a u t h o r i n g but must o p t i m i z e their content for specific devices. D e s i g n for tactile browsing can be considered an extension of this process.  6.2  Low-Fidelity Prototype: Image-Based Browser  A prototype browser was implemented that takes a g r a p h i c a l image as i n put and allows interactive scrolling of the image, w i t h synchronized tactile output. T h e software was tested informally w i t h users. T h e image browser is described i n detail i n [43]. Therefore, it is only briefly described here.  6.2.1  Design  T h e design concept, described previously i n C h a p t e r 2, is the work of the author. A key component of this design is the provision of a tactile track, represented g r a p h i c a l l y alongside the rendered page, w i t h icons s p a t i a l l y arranged at locations corresponding to the vertical positions of elements such as headings, links, images, text, etc.  (Figure 6.1). Furthermore, the  height of the icons represented i n the tactile track corresponds to the height of the page elements.  W h e n the user scrolls the page, the tactile track  moves i n synchrony w i t h the page. These concepts were used to b u i l d the image-based browser.  Chapter 6. Browser Prototype  (C)  (b)  (a)  93  THMB T h e Tactile H a n d h e l d Miniature B i m o d a l d e v i c e features a liquid crystal display a n d a tactile display.  header  haptic icon  -a c  .  T3 / CD,'  -  II is h e l d in o n e h a n d with the thumb o n the tactile d i s p l a y w h i c h is c o m b i n e d with a scroll button.  Tactile Display  icon  T h e display u s e s a stack of p i e z o e l e c t r i c b e n d e r s l o d e f o r m the s k i n in the tangential direction.  Tactile Rendering  image haptic icon  This technology can render p r o g r a m m a b l e tactile s e n s a t i o n s s u c h as moving small scale features.  Demonstrative Application To d e m o n s t r a t e its capabilities a s i m p l e application w a s developed whereby a c c e s s to a d o c u m e n t w a s provided by s i m u l t a t n e o u s visual a n d tactile exploration in a scrolling m o d e . © McGill University Haptics Laboratory ;ind University of British Columbia S P I N Laboratory  F i g u r e 6.1: O v e r v i e w of m a i n features of the image browser, (a) "Tactile track" w i t h callouts for icons, (b) Source image, (c) T h e viewable area is determined by the screen size and height of the tactile display ("tactile w i n d o w " ) . D i a g r a m by Jerome Pasquero.  Chapter 6. Browser Prototype 6.2.2  94  Implementation  Implementation  was done by the team  at M c G i l l ,  principally Vincent  Levesque. T h e image browser a p p l i c a t i o n is implemented i n C + + and uses the G T K l i b r a r y [3] for graphics display, and the S T R e S S l i b r a r y (Section 4 . 7 ) for T D i n p u t and output. T h e browser software is capable of generating its own haptic icons parametrically. It supports three types of icons, whose positions and types are m a n u a l l y encoded i n the i n p u t image: 1. Headers 2. B o d y T e x t 3 . E m b e d d e d Images In the image browser prototype, embedded images are represented by a high-frequency "grating", b o d y text by sparse "dots", a n d headers by either two closely spaced "dots", or a single dot w i t h a superimposed high frequency grating w i t h adjustable magnitude. B a s e d on i n f o r m a l user evalu a t i o n of the s t i m u l i , the two-dot configuration for headers was chosen as the best candidate for being distinguishable from the other s t i m u l i . Scrolling is accomplished by pushing the slider up or down, w h i c h causes the page to continue scrolling as long as the slider was h e l d i n that position, i n p r o p o r t i o n to the distance from the centre of the slider travel  (velocity  control). For this test, the slider was equipped w i t h a spring-return mechan i s m t h a t centred the slider i n the m i d d l e of its travel. T h e r e was no action associated w i t h the T D ' s "click" mechanism.  Chapter 6. Browser Prototype 6.2.3  95  Image Browser User Test  A n informal user evaluation was carried out b y the author to collect impressions from people w h o were unfamiliar w i t h the device. T h e full results are reported i n [43]. E v a l u a t i o n w i t h seven participants produced the following findings: 1. W h i l e the high-frequency "image" icon a n d the low-frequency "body text" icons were clearly distinguishable, 6 out of 7 participants h a d difficulty distinguishing the "header" icon, even when it was displayed graphically (in the tactile track) i n a way that was v i s u a l l y distinguishable from the other icons, a n d they were asked e x p l i c i t l y to attend to the number of s t i m u l i they could distinguish. 2. A l l (7/7) participants said they wanted to be able to access the ends of the page w h i c h were inaccessible due to the tactile w i n d o w being fixed i n the m i d d l e of the v i s u a l scrolling window. 3. 4 out of 7 participants reported that they could easily perceive the direction of movement of the tactile s t i m u l i , a n d that it was i n synchronization w i t h the movement of the page. 4. 3 out of 6 participants reported frustration w i t h the use of the slider mechanism for scrolling a n d targeting.  O n e participant said that i n -  stead of velocity control, it w o u l d be nice i f the device worked like a j o g dial, s u p p o r t i n g repetitive pushing motions i n position-control mode 5. P a r t i c i p a n t s ' free-response suggestions included being able to make selections by pushing down o n the T D ("click" action), page flipping,  Chapter  6. Browser Prototype  96  eyes-free browsing, a n d notification.  6.3 The  Haptic Display of Web Pages previous section described a browser w h i c h operated o n graphical  mockup images of web pages, w i t h the accompanying tactile s t i m u l i being generated a l g o r i t h m i c a l l y from one of three templates, w i t h placement a n d type m a n u a l l y encoded i n the image. L i n k selection, reading of web source files ( H T M L , etc.), a n d other t y p i c a l features of a web browser were not supported. T h e goal of the image-based browser was to serve as a first-pass low-fidelity prototype to o b t a i n early user feedback as part of the ongoing process of developing the browser a p p l i c a t i o n . W e w i l l now discuss a n automated m e t h o d for creating a haptic representation of a web page from its H T M L source files. T h e key features of this approach are: 1. Simultaneous visual a n d haptic rendering of H T M L elements. 2. A n object-oriented, extensible framework for the h a p t i c page m a p . 3. Support for a n a r b i t r a r y variety of haptic icons associated w i t h page elements.  4. Support for animated (i.e., time-varying) h a p t i c icons.  6.3.1 The  T h e H a p t i c Page M a p  o u t p u t of the rendering a l g o r i t h m is a haptic page map, w h i c h is a  model (in the object-oriented p r o g r a m m i n g sense) of the page i n the haptic domain.  (Figure 6.2) I n the haptic page map, icons are laid out s p a t i a l l y  Chapter 6. Browser Prototype  97  along a one-dimensional page map coordinate, w h i c h corresponds to the vert i c a l dimension i n the v i s u a l representation of the web page.  2  .  During  navigation, the user moves a cursor along the page m a p coordinate, as described i n section 6 . 4 .  6.3.2  M a p p i n g H a p t i c Icons to Page Elements  Page elements, such as links, images, a n d headings, are associated w i t h h a p t i c icons according to the following a l g o r i t h m : 1. If the element contains the H T M L name attribute, that property is used as the haptic icon filename, followed by an . x m l extension. 2. Otherwise, the default icons l i n k . x m l , i m a g e . x m l , h i . x m l , h2.xml, etc. are used. T h i s approach allows page authors to specify haptic icons easily w i t h i n the H T M L m a r k u p . F o r example, the X H T M L tag: <img s r c = ' c o u p o n . j p g '  name='hapticonA'  \>  places an image i n the page w i t h filename c o u p o n . j p g a n d associates it w i t h the haptic icon h a p t i c o n A . x m l . C o m b i n e d w i t h the a n i m a t e d G I F based workflow described previously i n Section 4 . 9 . 6 , this H T M L - b a s e d m a r k u p scheme means that no special tools are required to a d d haptics to an existing web page, a n d page authors can leverage their existing skills i n graphics creation a n d H T M L m a r k u p . 2  A coordinate transformation may apply in converting between graphical vertical co-  ordinates and page map coordinates. See Section 6.4.10.  Chapter  6. Browser  Prototype  98  XHTML Source ca h r e f = ' l i n k l . h t m l ' name='hump' >Link l < / a x b r \> <irag s r c = ' h a p p y . g i f  \><br \>  Spatial Representation of Haptic Page Map  <a href='link2.html' name='plateau'>Link 2</a>  HapticPageMap int height = 400 Hapticlcon 'icons  0  -o  0>  <n CD  1  Gecko  1  Graphically Rendered Page  0> "O  Hapticlcon: :Spatial  O o o  int position = 150 Hapticlcon "next float *data  Hapticlcon::Animated  int position = 200 Hapticlcon "next float *data[frames]  Q01  0 kg  2  Hapticlcon ::Spatial  int position = 250 Hapticlcon "next • float *data 400  F i g u r e 6.2:  T h e ( X ) H T M L source is rendered graphically by the Gecko  engine a n d h a p t i c a l l y using the H a p t i c P a g e M a p m o d e l , shown here i n a simplified schema. T h e resulting spatial arrangement of icons i s also shown. T h e H a p t i c P a g e M a p object includes a list of H a p t i c l c o n s , w h i c h m a y be of type S p a t i a l l c o n or A n i m a t e d l c o n . Together, the objects form a complete representation of the web page i n the haptic d o m a i n .  Chapter 6. Browser Prototype 6.3.3  99  Spatial Layout  A simple spatial layout a l g o r i t h m is used t o assign positions to the haptic icons. T h e haptic icon's t o p coordinate corresponds to the t o p pixelcoordinate of the rendered element i n the graphical representation of the web page. V a r i o u s approaches to deal w i t h the case when m u l t i p l e page elements are laid out horizontally, a n d thus occupy the same v e r t i c a l space on the graphic display, were tested. T h e r e is functionality b u i l t into the layout engine for d y n a m i c a l l y reconfiguring the haptic icons to avoid overlap I n i t i a l experimentation w i t h this approach revealed that the m i s m a t c h between graphical a n d haptic representations created unsettling navigation inconsistencies. A t the same time, it was found t h a t h o r i z o n t a l l y l a i d out elements are generally rare o n s m a l l screen displays, especially mobile phones w i t h a "portrait" (as opposed to "landscape") oriented display. T a k i n g b o t h these factors into account, it was decided not to take advantage of the horizontal layout functionality i n the final prototype. Currently, the layout a l g o r i t h m does not make use of the rendered pixelheight of the element; haptic icons are not d y n a m i c a l l y resized to m a t c h the visual element height. T h i s was considered a low p r i o r i t y for links a n d headings, since their heights are usually specified globally i n a style sheet and do not often vary from element to element. B o t h the variable-height haptic icon a n d horizontal layout features w o u l d be rather t r i v i a l to implement i n the current framework, a n d represent opportunities for future improvement (Section 8.3.4). F u r t h e r details of the haptic page m a p generation a l g o r i t h m are p r o v i d e d i n Section 6.5.  Chapter  6.4  6. Browser Prototype  100  Navigation Model  T h i s section describes the user interaction m o d e l for the web page a n d its associated h a p t i c page m a p .  6.4.1  Cursor Position  T h e n a v i g a t i o n m o d e l maintains a cursor p o s i t i o n i n b o t h the graphic a n d h a p t i c domains. T h e coordinates m a y be the same, or they m a y be related by a linear transformation, as described i n Section 6.4.10. T h e h a p t i c cursor p o s i t i o n is logically an integer number i n the h a p t i c page m a p coordinate space. Signals are rendered to the T D from a p o r t i o n of the page m a p that is centred on the cursor p o s i t i o n . T h i s is k n o w n as the tactile window. T h e height of the tactile w i n d o w is n o r m a l l y eight units high but m a y be increased w i t h subtaxel rendering, described i n Section 6.4.10. Because there are a n even number of piezos, b y convention the cursor position is rendered slightly above centre, or piezo #3 i f no transformation is applied. T h e graphic cursor is logically a single row of pixels c o m p r i s i n g the vertical point of focus. It is displayed as a semi-transparent yellow bar, achieved w i t h the a l p h a channel features of the P N G image file format, centred o n the logical cursor position, w i t h height corresponding to the height of the tactile w i n d o w i n the current coordinate transformation scheme; it is eight pixels high w h e n there is no transformation.  Therefore, the graphic cursor pro-  vides the user w i t h a visual display of the area of the page w h i c h is currently "visible" (or, more accurately, tactable) t h r o u g h the tactile display. T h e graphic a n d h a p t i c cursor positions are kept synchronized i n software. T h e t e r m cursor position m a y be used to refer to b o t h .  Chapter 6.4.2  6. Browser  Prototype  101  R e n d e r i n g H a p t i c Icons  A s mentioned i n the previous section, the tactile window, or p o r t i o n of the haptic page m a p that is "viewable" by the tactile display at a given position, is rendered to the T D continuously. T h e following a l g o r i t h m is used: 1. E v e r y I / O loop cycle, the p r o g r a m traverses the list of haptic icons that is linked from the HapticPageMap object. 2. If a haptic icon is found whose extent lies w i t h i n the tactile w i n d o w (i.e., is within focus), the object's rendering function is called, w h i c h renders the icon (places voltage values) to a buffer. 3. T h e loop repeats u n t i l a l l icons have been checked.  3  4. T h e p o r t i o n of the buffer that is w i t h i n the tactile w i n d o w is o u t p u t to the device. Therefore, the case of overlapping icons (normally a n error condition), icons that are lower i n the list of icons m a y overwrite their predecessors. A n i m a t e d icons are handled i m p l i c i t l y b y the rendering function of the H a p t i c l c o n : : Animatedlcon object, w h i c h maintains a real-time counter and renders different "frames" when it is called at different times. U n d e r the current implementation, the counter starts from the first a n i m a t i o n frame when the icon is first brought into focus (its extent is overlapped by the tactile w i n d o w ) , but is not reset subsequently; the icon's a n i m a t i o n m a y be 3  Since the number of haptic icons is small (typically, less than 20) and is limited by the  amount that the user is willing to scroll, the overhead associated with a simple iteration through the list of icons is negligible on the current system. Further improvement could be made by using a sorted data structure, such as a tree, to reduce search cycles.  Chapter  6. Browser  Prototype  102  considered to continue p l a y i n g i n the background d u r i n g the t i m e it is out of focus.  6.4.3  Page Element Focusing  T h e M o z i l l a browser keeps track of a focused, or highlighted, page element. If the user presses the selection b u t t o n (usually a mouse b u t t o n , but handled abstractly i n the current implementation), the focused element is selected. For example, i n the case of a link, the browser loads the new link. T h e page element that intersects w i t h the g r a p h i c a l cursor p o s i t i o n receives the current focus. In simpler terms, the user can highlight an i t e m by p o s i t i o n i n g the cursor over it. Because the cursor occupies a full row, the current design does not support selection of m u l t i p l e h o r i z o n t a l l y d i s t r i b u t e d elements; it s i m p l y defaults to selecting the rightmost element. Because the cursor is logically a single row, but is displayed as the height of the tactile w i n d o w w h i c h occupies eight or more rows, it is possible for the cursor to be p a r t i a l l y intersecting an element b o t h i n the visual and tactile space w i t h o u t the element being focused.  6.4.4  Graphical Display Scrolling  Because pages are often taller t h a n the graphical display, cursor movement must also control scrolling of the display. A push-to-scroll m o d e l is used; when the cursor hits a b o u n d a r y of the display, the display begins scrolling if necessary. O n a P C , this type of m o d e l is c o m m o n l y used i n w o r d processors and text-entry boxes, where v e r t i c a l l y m o v i n g the cursor beyond the display b o u n d a r y causes the display to scroll one line at a time. Because the cursor is also being used for focusing, it is desirable for the  Chapter  6. Browser Prototype  103  user to have some way of looking ahead before h i t t i n g a n element. Therefore, scroll margins are implemented, as shown i n F i g u r e 6.3. F r o m its i n i t i a l position at the t o p of the display, the cursor moves d o w n u n t i l it hits the scroll m a r g i n . If the user continues scrolling down, the cursor's displayed position remains fixed while the page scrolls under i t . If the user reverses direction a n d moves the cursor up, the page does not scroll u n t i l the cursor hits the t o p scroll m a r g i n (hysteretic m o t i o n ) . W h e n the display can scroll no more, the cursor moves past the scroll m a r g i n a n d to the t o p or b o t t o m of the page. I n this way, the whole page is accessible to the cursor, unlike the previously described image based browser.  6.4.5  Control of Cursor Movement  In the simplest case, the "throw" of the slider m a y be m a p p e d to the overall height of the page m a p , such that i f the user positions the slider at the t o p of its range, the cursor is positioned at the t o p of the page, a n d i f the user positions the slider at the b o t t o m of its range, the cursor is positioned at the b o t t o m of the page. T h i s is a direct position control mode; the cursor position corresponds d i r e c t l y to the slider p o s i t i o n . P o s i t i o n control is the m o d e l used i n j o g dials, i n c l u d i n g mouse scroll wheels.  4  E a r l y experimentation w i t h the device revealed that there was insufficient spatial resolution to display more t h a n two or three distinguishable h a p t i c icons using direct position control across the short slider range (11 m m ) . Furthermore, precise p o s i t i o n i n g w o u l d require extremely h i g h dexterity, a n d m a y even be beyond the precision of the analog p o s i t i o n sensor; 4  D e p e n d i n g o n t h e a p p l i c a t i o n , t h e mouse scroll wheel m a y also operate i n a discrete  p s e u d o - p o s i t i o n c o n t r o l m o d e , s i m u l a t i n g t h e effect o f a user r e p e a t e d l y p r e s s i n g a s c r o l l key.  Chapter 6.  Top Margin  Browser Prototype  104  t  1 " 1 1/5  j l  A  0  What's Um  "r IW'e'afliiei'i Focal Area  3/5  Transit  1  \  I& Movies If M  Bottom Margin  II  1  "ill il"'  4  F i g u r e 6.3: Scrolling margins for the graphical display, located at 20% from the t o p a n d b o t t o m of the display. T h e display begins scrolling when the cursor reaches a m a r g i n .  Chapter 6. Browser Prototype  105  for a t y p i c a l web page 600 pixels high, the necessary resolution w o u l d be a p p r o x i m a t e l y 1390 dots-per-inch, far i n excess of t y p i c a l mouse hardware resolution.  5  Because of the l i m i t e d range of slider m o t i o n , it was necessary to use velocity control, w h i c h is the m o d e l used i n joysticks and shuttle controls. In the simplest case (pure velocity control), the cursor movement velocity corresponds to the slider position, w i t h the centre of the slider treated as zero velocity. A s m a l l "dead zone" may be used i n the centre to reduce unwanted cursor m o t i o n . T h i s m e t h o d was used i n the image-based browser (Section 6.2).  A more sophisticated m o d e l was used for the web browser  (Section 6.4.7), and a s u m m a r y of the three cursor control models is shown i n F i g u r e 6.4.  6.4.6  S p r i n g return to centre  T y p i c a l velocity control i n p u t devices have a spring-loaded return to centre mechanism w h i c h provides a resistance force when the user pushes on the device to cause m o t i o n . Informal user testing of the device w i t h and w i t h o u t spring feedback revealed that users strongly preferred spring feedback when the device was used i n simple velocity control mode. A number of approaches were tried to implement spring feedback, i n c l u d i n g elastic foam, elastic bands, and compression springs.  D u e to the  mechanical design of the slider, use of compression springs i n a coil-overshaft configuration was the most parsimonious and reliable approach. ( F i g ure 6.5) T h i s required finding springs w i t h a m i n i m u m inner diameter of 2.4 m m and appropriate length and spring constant. Springs w i t h appropriate 5  The Microsoft Intellimouse Optical, for instance, has a resolution of 400 dots-per-  inch. [5]  Chapter 6. Browser Prototype  TD/ Slider  Simple Position Control  Simple Velocity Control  o  •100  <  ~D CD  (£2  o o  CD  CD "O  O o o  a ^'  106  Hybrid Position / Velocity TJ  O w .—* Q 3  -20  -100  I o o  0  "Dead i Zone" l  CD  +20  600  F i g u r e 6.4:  Slider control modes.  100  100  In simple position control, the slider  position is m a p p e d directly to the cursor position, e.g., for a page w i t h height 600. In simple velocity control, the slider position relative to centre is m a p p e d to the velocity of cursor movement, w i t h an optional  central  "dead zone" to make centering easier. In h y b r i d p o s i t i o n / v e l o c i t y control, a region of the slider travel functions i n direct position control mode; if the user pushes the sider beyond that region, velocity control is active.  Chapter  6. Browser  Prototype  107  physical dimensions could be found i n c o m m o n retractable ball-point pens, and 16 different pens were acquired and disassembled to find the one w i t h the most appropriate spring constant.  However, due to the fact that t y p -  ical use of a retractable ball-point pen involves compressing the spring by pushing d i r e c t l y on its end, versus lateral force applied i n the case of the T D , most ball-point pen springs were too stiff for use w i t h the device. C o n sequences of excessively stiff springs included user fatigue a n d d i m i n i s h e d tactile stimulus intensity, due to the piezos being bent laterally against one another by the force against the spring-loaded slider. Softer springs were found i n multi-colour ball-point pens t h a t also have lateral actuation. T h e softest spring was found i n a M i t s u b i s h i four-colour c o m b i n a t i o n pen from J a p a n , and this spring was cut to size and used i n the device.  T h e spring was a c t u a l l y too soft to produce a reliable and  accurate r e t u r n to centre against the friction of the slider and the i n e r t i a of the T D ; therefore, two slightly overlength springs i n a m u t u a l l y preloaded configuration were used to improve the centering properties. Unfortunately, this also increased the force necessary to push the slider a l l the way to its top or b o t t o m stop. T h i s configuration was used i n the image-based browser tests (Section 6.2) and early pilot testing of the H T M L - b a s e d browser. W h i l e it was clearly necessary for velocity control (users reported strong preferences for the presence of the springs), extended use caused fatigue due to the stiffness of the springs, and premature failure of the i n s u l a t i n g varnish coat due to the strong shear forces between the t h u m b and the T D . A n extensive survey of spring vendors revealed that no softer springs could be found i n the 2.4 m m inner diameter configuration, possibly due to m e t a l l u r g i c a l l i m i t a t i o n s i n the m i n i m u m thickness of wire that can be used while r e m a i n i n g i n its  Chapter 6. Browser Prototype elastic zone.  108  Softer springs are available i n a torsion configuration,  but  major re-engineering of the slider mechanism w o u l d have been required to accommodate them, and it is unlikely that such low spring rates w o u l d be sufficient to overcome the friction associated w i t h the mass of the T D . D u e to these l i m i t a t i o n s , it was decided to remove the springs prior to  further  testing of the browser prototype, and to defer b r o a d re-engineering of the slider mechanism u n t i l the cursor control m o d e l could be studied i n more detail.  6.4.7  H y b r i d Velocity / Position Control M o d e l  T h r o u g h informal user evaluation, it was found that velocity control alone d i d not afford the b i d i r e c t i o n a l tactile exploration that was effective i n creating the illusion of exploring s m a l l shapes w i t h the V i r t u a l B r a i l l e D i s p l a y [30]. Furthermore, it was desirable to make effective use of the centre of the slider travel for precise positioning instead of s i m p l y h a v i n g a "dead zone". A h y b r i d velocity / position control model was adopted to meet these goals, as shown i n F i g u r e 6.4.  W h e n the slider is i n the position control  zone, velocity is zero and the user can move the cursor using position control w i t h i n a l o c a l offset range. Outside of this zone, velocity control is active. T h e actual values of the parameters used are listed i n Section 6.4.7. T o the user, the experience of navigation using the h y b r i d control model is of a s m a l l active area i n the centre of the slider range w i t h i n w h i c h they can interactively "browse" a haptic icon as if they were touching a s m a l l shape, m o v i n g the finger up a n d down to explore its edges. P u s h i n g the T D further causes scrolling m o t i o n . If the T D is moved very fast (for example, r a p i d l y from its bottom-stop to its top-stop), the cursor may r a p i d l y move across its  Chapter  6. Browser  Prototype  109  position control range. If the m a x i m u m velocity setting is lower t h a n this velocity, there may be a somewhat unexpected b i m o d a l cursor behaviour i n w h i c h it r a p i d l y j u m p s and then moves more slowly under velocity control. However, i n real-world usage, the user is unlikely to move the slider r a p i d l y from end-stop to end-stop; the more c o m m o n usage p a t t e r n involves m o v i n g the slider away from its centre detent and holding it there u n t i l the target is achieved, followed by s m a l l fine-tuning adjustments i n the case of overshoot, undershoot, or l o c a l exploration of adjacent elements.  H y b r i d C o n t r o l M o d e Feedback a n d S p r i n g R e t u r n W h i l e the h y b r i d control m o d e l allows fine-grained exploration w i t h i n the central position control zone, users still reported discomfort using the device i n its velocity control zone w i t h o u t spring feedback.  In a d d i t i o n , i n the  absence of any haptic cues such as spring force feedback, it was difficult to determine where w i t h i n the slider travel the mode switch occurred, w h i c h also resulted i n negative user observations. T o deal w i t h the latter concern, synthetic tactile notification of mode switching was implemented.  Whenever the user crossed a mode switch  boundary, the current tactile signal was overlaid w i t h a r a p i d vibrotactile signal consisting of a half-height (50V peak-to-peak) cycle, repeated twice over 4 samples. T h e overlaid samples were c l i p p e d to + 5 0 V / - 5 0 V if necessary. T h i s provided a "clicking" or " b u m p i n g " sensation as the slider was moved across the mode boundaries.  T h e sensation was judged effective i n  early pilot testing for notifying mode switches, but only if the slider was moved very slowly across its travel. W i t h r a p i d movement, the user could move the slider across its 11 m m travel w i t h i n a fraction of a second, causing  Chapter  6. Browser  Prototype  110  the two mode switch notifications to blend into a generic v i b r a t o r y signal w i t h m i n i m a l spatial l o c a l i z a t i o n relevance. Furthermore, it d i d not address the need for p r o p o r t i o n a l force feedback i n velocity control mode. T h e eventual solution was to adopt hardware springs i n a h y b r i d configuration.  F i g u r e 6.5 shows the design after several testing rounds w i t h  various materials and thicknesses.  S m a l l pieces of latex foam were affixed  to the slider stops using i n d u s t r i a l double-sided tape.  T h e extent of the  foam pieces was a p p r o x i m a t e l y one-quarter of the t o t a l slider travel each (3 m m ) . Because of the compliance of the foam, the point at w h i c h the slider hits the foam is essentially imperceptible; however, after the foam is compressed more t h a n halfway, it starts to provide perceptible resistance.  There  is a p p r o x i m a t e l y 1 m m of travel w i t h i n this range, d u r i n g w h i c h the force is perceived as essentially isometric. T o the user, the experience is of moving the slider w i t h i n a free-movement zone causing the cursor to track the slider position, followed by pushing the slider against its stops w i t h variable force resulting i n velocity control. Specific parameters were determined after informal o p t i m i z a t i o n w i t h three pilot users, and are listed i n the section below.  H y b r i d Control Delays F u r t h e r testing of the device under the h y b r i d control system revealed add i t i o n a l areas needing improvement for user comfort.  F i r s t , when users  wanted to stop cursor movement by m o v i n g the slider out of the velocity control zone, they t y p i c a l l y released the slider, allowing the springs to move it back into the p o s i t i o n control zone, or they actively slid the slider back towards centre.  I n either of these cases, the slider was u n l i k e l y to end up  Chapter  6. Browser  Prototype  111  F i g u r e 6.5: Force feedback using springs. T h e standard coil-over-shaft spring configuration and the h y b r i d spring configuration is shown i n various slider positions.  Chapter  6. Browser  Prototype  112  exactly at the p o s i t i o n control boundary; it was more likely to be somewhere w i t h i n the p o s i t i o n control zone. T h e effect of this on cursor movement was an unsettling "snap back" effect that made targeting difficult because the user w o u l d acquire a target using p o s i t i o n control, release the device, and the cursor w o u l d move backwards a distance equivalent to the distance between the p o s i t i o n control zone b o u n d a r y and the final slider position, thus b r i n g i n g the intended target out of focus. T h e second p r o b l e m was unintended cursor movement due to accident a l l y m o v i n g the slider into the velocity control zone. T h i s s i t u a t i o n occurred when the user was targeting a point just outside of the p o s i t i o n control range. T h e user w o u l d approach the target by continuing to move the slider i n the direction of the target, cross the zone b o u n d a r y unintentionally, and cause unexpected continuous cursor movement. F i n a l l y , due to the short velocity control range, and the low amount of force that the user could a p p l y to the device laterally w i t h o u t causing fatigue or piezo deflection, there was a l i m i t e d d y n a m i c range available for use w i t h velocity control.  Users still wanted to move the cursor r a p i d l y  to traverse a web page quickly; however, if the m a x i m u m speed parameter was increased excessively, there was insufficient precision at low speeds for precise targeting. These three problems were addressed by a d d i n g time controlled elements into the control software.  T h e resulting control system's state d i a g r a m is  shown i n F i g u r e 6.6. T o address the first cause of unexpected cursor movement due to "snap back", a delay was implemented i n the t r a n s i t i o n from velocity control to p o s i t i o n control. Since the system remains i n velocity control d u r i n g this delay, the r a p i d return of the slider to its centre detent position s i m p l y causes the velocity to drop to zero, "absorbing" the large  Chapter  6. Browser  negative p o s i t i o n displacement.  Prototype  113  W h e n the user replaces h i s / h e r finger on  the display and starts m o v i n g it again after the delay has expired, the syst e m is i n the p o s i t i o n control state, enabling interactive local exploration of tactile features. T h e second concern of accidentally m o v i n g the slider into the the velocity control zone was addressed using a delay i n the t r a n s i t i o n from position control to velocity control. L o c a l exploratory behaviour i n position-control mode consists of relatively r a p i d , back-and-forth s m a l l scale sweeps of the v i r t u a l surface. T o make the t r a n s i t i o n to velocity control using the delay m o d e l the user must "push and h o l d " , a more intentional action, before the system started scrolling under velocity control. F i n a l l y , a linear acceleration m o d e l was added to the velocity control to enable increased scrolling speeds if the user held the slider i n the velocity control zone for an extended time.  H y b r i d Control Settings After performing the above mentioned adjustment process, the final p a r a m eters used i n the browser prototype are listed i n table 6.1.  6.4.8  R e d u c t i o n of Slider Jitter  A l t h o u g h the slider uses a high-precision resistive p o s i t i o n sensor, there is considerable analog sensor noise. If unfiltered, this noise produces continuous high-frequency piezo m o t i o n t h a t completely destroys the a b i l i t y to render shapes using low-frequency piezo deflections. A historical averaging model was therefore implemented i n the software to reduce jitter. R a w slider i n p u t is first passed to a S l i d e r S m o o t h e r object i n soft-  Chapter  F i g u r e 6.6:  6. Browser  Prototype  114  State t r a n s i t i o n d i a g r a m for the four velocity / position control  states.  ware, w h i c h outputs the "smoothed slider p o s i t i o n " . It takes two parameters: c a c h e _ s i z e and t h r e s h o l d , whose final values are shown i n T a b l e 6.2.  W h e n the slider is stationary or being moved slowly, an average of  the last c a c h e _ s i z e number of samples is taken and used as the final slider position, thus'cancelling out any small, high-frequency noise. F o r high values of c a c h e _ s i z e and large slider displacements, this can take a noticeable t i m e to converge on the slider position, w h i c h is perceivable as a "rubber b a n d " effect on the cursor. Therefore, an escape parameter, t h r e s h o l d , is implemented.  If the user moves the slider rapidly, h i s t o r i c a l averaging is  abandoned and the current slider position is used directly.  Furthermore,  Chapter  6.  Browser  Prototype  115  the cache of values is reset so that averaging re-starts at the current slider position when its velocity drops below the threshold value.  6.4.9  R e d u c t i o n of H i g h - A m p l i t u d e , H i g h - F r e q u e n c y Outputs  W h e n a user is scrolling very r a p i d l y t h r o u g h the page map, it is possible for two sequential samples to be taken from very distant areas of the page m a p . W h e n there is a large voltage spread between adjacent samples, it creates a large, sudden displacement of the piezo that is perceived by the user as a rough "clicking" sensation. T h e high-frequency transients caused by r a p i d discrete movement of the piezo are especially bothersome, since they tend to result i n user reports of undesirable " v i b r a t i o n " , "zapping" or "electric shocks". It is also potentially damaging to the piezo m o t o r itself. T o combat this undesirable effect, two measures are used.  F i r s t , the  slider input s m o o t h i n g a l g o r i t h m can spread displacements over time using its h i s t o r i c a l averaging a l g o r i t h m , as described i n the previous section. H o w ever, to reduce unwanted "rubber b a n d " cursor behaviour, large slider displacements cause a buffer abort. Furthermore, w h e n velocity control is being used w i t h high m a x i m u m velocity settings, high-frequency, h i g h - a m p l i t u d e actuator transients can be produced even when the slider is held stationary i n a high-velocity p o s i t i o n . Therefore, an a d d i t i o n a l  output s m o o t h i n g  a l g o r i t h m was implemented to reduce sudden large piezo movements. T h e a l g o r i t h m first checks whether a given piezo actuator w i l l have a large voltage change, as measured between the "current" sample and the i m m e d i a t e l y previous sample. If the threshold (currently set at 5 0 V i n either direction) is exceeded, then the actual output voltage delivered to the piezo  Chapter  6. Browser Prototype  116  is "smoothed" out by averaging the current sample a n d the i m m e d i a t e l y previous sample. (We n i c k n a m e d this m e t h o d "lookback" smoothing). Because this tends to have the effect of s m o o t h i n g out peaks a n d therefore l i m i t i n g .the d y n a m i c range of the T D d u r i n g high speed movement, there is also a feedforward component designed to "recover" part of the peak voltage i n future samples. W h e n averaging is used to produce the output of a given sample, the sample's original (pre-averaging) value is stored. T h a t value is used to compute the next sample by averaging the cached and current sample values on the next timestep. T h e threshold test is then applied and, if necessary, the previous actual sample voltage is averaged back i n using the "lookback" a l g o r i t h m . T h e net effect of these s m o o t h i n g functions is to create the sensation of r a p i d l y scanning across tactile features, while m i n i m i z i n g aliasing caused by s a m p l i n g delay.  6.4.10  Speed L i m i t a t i o n  In the perceptual characterization phase ( C h a p t e r 5), a m a x i m u m speed of 0.34 m / s was found to elicit a reliably discernible sense of d i r e c t i o n a l m o t i o n among users. W e m a y use this figure as a reasonable estimate of an upper b o u n d for the speed at w h i c h a user can traverse the haptic page m a p while m a i n t a i n i n g a continuous tactile flow percept. 6  6  A t m a x i m u m speed, given  The speed study measured non-interactive tactile flow signals, and it is possible that  the actual upper speed bound for an interactive (i.e., user-modulated) tactile flow signal may be slightly higher. An experienced user may also be able to detect higher speed signals. However, for the purposes of our initial calculation, it is clear that even with a 30% margin of error, requiring the user to spend 8 to 33 seconds to traverse a typical web page is still unacceptably long and requires countermeasures.  Chapter  6. Browser Prototype  117  the p h y s i c a l dimensions of the T D it takes 30 ms for a h a p t i c icon to move from one piezo actuator to the next. If s p a t i a l layout of the h a p t i c m a p is performed using a one-to-one mapp i n g between pixels and taxels, it w o u l d require 18 seconds at m a x i m u m speed to go from the top to the b o t t o m of a t y p i c a l 600 pixel-high web page. Pages i n the testing corpus for the browser ranged from 424 to 1698 pixels high, so it would require considerable time (12 to 50 seconds) to traverse the web page at speeds necessary to stay w i t h i n the range of perceivable direction. T w o techniques were used to accomplish the decoupling of the scrolling m o t i o n o n the screen and the T D : 1. Page m a p s h r i n k i n g , and 2. S u b t a x e l rendering These methods effectively comprise a cascading transformation scheme for the page m a p coordinates that can reduce the logical height of the haptic page m a p up to eight times relative to its graphical equivalent. If desired, it m a y be possible to further increase m a x i m u m speed using other techniques i n c o m b i n a t i o n or separately.  These possibilities are discussed in Section  8.3.4.  Page M a p Shrinking T h e browser client supports a mode that transforms a l l page coordinates and requested icon heights to create a page m a p that is half the pixel-height of the web page. U s i n g this technique, it is possible to double the effective speed l i m i t of scrolling traversal, but because icons must be half-height, they must be simpler, thus l i m i t i n g the differentiability of icons.  Chapter  6. Browser  Prototype  118  User testing w i t h the page m a p s h r i n k i n g feature suggested that the loss of icon fidelity negatively i m p a c t e d the perception of tactile flow because it required designing more sharp-edged icons. Users perceived r a p i d changes i n the voltage applied to actuators as " v i b r a t i o n " rather t h a n tactile flow. T h i s undesirable perception of high-frequency aberrations is analogous to the p r o b l e m of harsh edges and other anomalies that occurs i n computer graphics when images are downsampled excessively using simple nearestneighbour scaling algorithms. F u r t h e r page m a p shrinkage beyond a factor of two was deemed i m p r a c t i c a l because the haptic icon height for a l i n k icon, n o r m a l l y 21 page m a p units for the corpus of pages used i n the user test (Chapter 7), w o u l d drop below 10, thus severely l i m i t i n g the expressive capabilities of the icons and increasing the risk of delivering unintended high-frequency vibrotactile signals to the user.  Subtaxel  Rendering  T o reduce the movement speed of the tactile stimulus while s t i l l retaining good fidelity, a subtaxel rendering m e t h o d was implemented i n the browser server component.  U s i n g this m e t h o d , the internal representation  of the  page m a p retains the same height as i n the one-to-one case. However, when the signal is rendered to the T D , two or three (depending on a configurable mode setting) taxels from the internal m a p are averaged to form one o u t p u t taxel. I n this way, the effective "viewport size" of the tactile display (the tactile window) increases two- or three-fold, to 16 or 24 taxels, respectively. Because the l o c a t i o n of the T D ' s w i n d o w into the overall page m a p is m a i n tained i n the internal high-resolution coordinates, s m o o t h movement results.  Chapter  6. Browser  Prototype  119  Parameter  Description  Value  Units  maxP  The page map range accessible  50  page map  in position control mode. maxV  The maximum velocity at full  units 200  acceleration.  page map units per second  posControlZone  The portion of the slider travel  0.75  (ratio)  0.05  seconds  1.0  seconds  0.05  seconds  assigned to position control (the remainder is used for velocity control).  Determined by  the point at which the latex foam begins to provide noticeable feedback. timeBeforeVControl  The timeout before entering velocity control mode.  Used to  prevent accidental activation of velocity controlled movement during exploratory behaviour. vControlAcceleration  The time between entering velocity control mode and reaching the maximum velocity for a given slider position.  timeAfterVControl  The timeout before entering position control mode. Prevents "snap-back" of the cursor when the user releases the T D .  Table 6.1: Values of hybrid velocity / position control model parameters.  Chapter 6. Browser Prototype  120  Parameter  Description  Value  cache_size  T h e number of slider values (collected once ev-  10  ery 3 ms) to average to find the current slider position. threshold  If, w i t h i n the 3 ms sample interval, the slider  5  is moved a distance greater t h a n this value (in raw 12-bit slider p o s i t i o n units), historical averaging is abandoned a n d the current slider position is used directly. T a b l e 6.2: aging.  Settings for the slider s m o o t h i n g function using historical aver-  Chapter 6. Browser Prototype  High-Resolution (Subtaxel) Image  121  Rendered Output  +10  t =0  +50  o -20  +10  +50  t=1  F i g u r e 6.7: T h e subtaxel rendering technique. In this example, three taxels from the internal high-resolution representation are averaged to form one o u t p u t taxel. T h e stimulus is m o v i n g up across the display, and two time samples are shown.  Chapter 6. Browser Prototype  Effective Input  122  Suboptimal  Output Volt. Stretch  Input  Output Volt. Stretch  LEGEND +50V 100  100  200  200  300  300  stretch  to  CD  0V •  E  -50V H 400 H  compression  400 h  Piezo #2 Output  Piezo #2 Output  •  0  neutral  100 200 300 400 Time (ms)*  0  •  •  100 200 300 400 Time (ms)*  * When user is moving the cursor at a constant speed of 10 pixels per second.  Figure 6.8: Effective icon design for subtaxel rendering. High-resolution page map representations are shown in the left column of the group, and the output of the subtaxel rendering algorithm is shown on the right column in both raw voltage signal and skin-stretch representations. Five time samples are shown as the stimulus is tweened upwards across the display at a constant speed, illustrating how the subtaxel algorithm provides fine spatial information in the form of temporal signal changes. Note that exceeding the resolution limit of the TD when designing high-resolution icons tends to create blurry results. Below the pictures, the output from one piezo (#2 from the top) is graphed over time. In the case of a well-designed icon, the piezo reaches full-scale values.  Chapter  6. Browser Prototype  123  The method is, in effect, using temporal data to encode spatial data — using the high temporal resolution of the sense of touch [19] to compensate for the low spatial resolution of the tactile display device. The technique is shown in Figure 6.7. Because data is lost during the downsampling, haptic icons must be designed carefully to ensure that they create the intended sensations. Figure 6.8 illustrates some examples of effective and suboptimal haptic icon design with respect to downsampling. B y analogy, when visual images are heavily downsampled to low pixel counts using averaging, they tend to lose contrast and become blurry, and the same danger applies to the haptic icon images. Especially since the sensations produced by the T D tend to be quite subtle, it is important to retain as much dynamic range as possible when designing icons. Fortunately, the effects of downsampling may be quickly previewed during icon design in Adobe Photoshop by scaling the image by 33% and using the "bilinear" setting, which is equivalent to the averaging method used by the software.  6.5  B r o w s e r Software A r c h i t e c t u r e  The browser software architecture is shown in overview form in Figure 6.9. A logical flow diagram is also presented in Figure 4.6. Briefly, the principal components of this architecture are as follows: 1. A client c o m p o n e n t , built on the open-source Mozilla A P I , that handles parsing of the H T M L and CSS source documents, generation of the page map X M L file, keyboard and mouse input, graphical display, loading of new pages, and logging data for the usability evaluation.  Chapter  6. Browser Prototype  124  D e s c r i b e d further i n Section 6.5.1. 2. A server c o m p o n e n t , implemented i n high speed C + + code a n d r u n n i n g a continuous I / O loop, generating the tactile signals from its internal representation of the page map, a n d t r a n s l a t i n g analog slider input to page m a p coordinates a n d forwarding the i n p u t to the client component. D e s c r i b e d further i n Section 6.5.2. 3. Interactive h a p t i c icon design tools, previously described i n Sect i o n 4.9.6. 4. (Not shown on the diagram) W e b page generation tools for creation of H T M L a n d C S S source content. H a p t i c m a r k u p is o p t i o n a l l y added to the H T M L source as described i n Section 6.3.2. 5. A file-based interprocess c o m m u n i c a t i o n s y s t e m using "posit i o n " a n d "click" files to transmit information from the server component to the client component, a n d a "flag" file for the client component to signal to the server component to load a new HapticPageMap. D e scribed further i n Section 6.5.3.  Author's  Contribution  I n i t i a l work on specifying the architecture of the browser client was performed by undergraduate  research student S h a n n o n L i t t l e w i t h support  from the author. T h e current X P C O M a n d X U L components share approxi m a t e l y 50% of the code from that effort [33], w h i c h is i n t u r n significantly derived from online X P C O M a n d X U L tutorials; the current J a v a S c r i p t component, where the m a j o r i t y of functionality is implemented, shares less t h a n 10% of code.  Chapter  6. Browser Prototype  125  Figure 6.9: The browser software architecture, including software and hardware components and the data that is exchanged between them.  Chapter  6. Browser  Prototype  126  T h e I / O p o r t i o n of the browser server is based on example applications provided by V i n c e n t Levesque, a graduate student at M c G i l l university, as documentation for the S T R e S S L i b r a r y [29] A P I . E a r l i e r examples of the " T H M B L i b r a r y " co-written by V i n c e n t Levesque and S h a n n o n L i t t l e based on this code were not used; the code was rewritten from scratch for more precise t i m i n g control. W h e r e open-source software was used, it is mentioned i n the following sections. A l l other portions, i n c l u d i n g the aforementioned haptic page m a p and n a v i g a t i o n functionality, cursor display, usability testing support, page content, and style sheets are the work of the author.  6.5.1  Browser Client  T h e browser client component is built on the open-source M o z i l l a A P I and its Gecko rendering engine.  Since M o z i l l a is the most p o p u l a r extensible,  open-source platform for web page browsing as of this w r i t i n g , it ensures g o o d c o m p a t i b i l i t y w i t h W e b standards and good p o r t a b i l i t y of the software. T h e browser client component consists of: 1. A JavaScript application, 2. A n X P C O M (Cross P l a t f o r m C o m p o n e n t Object M o d e l ) component, 3. A X U L ( X M L User Interface Language) "chrome" component, and 4. T h e M o z i l l a A P I , its D o c u m e n t Object M o d e l and security modules, and the Gecko parsing and rendering engine. M o s t of the browser client component is implemented i n the JavaScript a p p l i c a t i o n , because it has access to the M o z i l l a browser's D o c u m e n t O b -  Chapter  6. Browser  Prototype  127  ject M o d e l ( D O M ) , allowing read access for the spatial layout of web page elements, H T M L tags i n c l u d i n g specialized haptic m a r k u p , and mouse and k e y b o a r d events. T h e JavaScript a p p l i c a t i o n also writes to the D O M when it loads a new page, displays the cursor, highlights elements t h a t are in-focus, or provides u s a b i l i t y testing related displays to the user. Data-logging is accomplished using the standard t e r m i n a l dumpO c o m m a n d from J a v a S c r i p t . T h e m a i n I / O loop consists of reading the slider p o s i t i o n and click count (via the X P C O M layer, described below), u p d a t i n g the cursor p o s i t i o n based on the slider position, and c o m p a r i n g the click count to a previous value — if an increase is detected, the highlighted page element is selected. T h e X U L layer consists of s m a l l wrapper code to r u n the JavaScript a p p l i c a t i o n w i t h i n the M o z i l l a framework while suppressing standard P C browser features such as toolbars. T h e X P C O M component handles file i n p u t and output, w h i c h is norm a l l y unavailable to JavaScript because of security restrictions.  It con-  sists of various functions to read the position and click files, and w r i t e the HapticPageMap file and its associated update-flag file. T h e file i n p u t a n d o u t p u t functions are implemented i n 0 + + ,  and a compiled I D L (Interface  D e s c r i p t i o n Language) wrapper is used to create the abstract function i n terface that allows JavaScript to call the C+-1- functions. F i n a l l y , the M o z i l l a A P I itself is based on M o z i l l a 1.01 browser, and is the work of various open-source  6.5.2  B r o w s e r  authors.  Server  A l l tactile device i n p u t and output routines are encapsulated i n the browser server component, w h i c h is a separate executable r u n n i n g at the highest  Chapter  6. Browser Prototype  128  p r i o r i t y available i n the L i n u x kernel. T h i s architecture ensures r a p i d , continuous u p d a t i n g of the tactile device, reducing the risk of irregular tactile signals that are perceived as undesirable vibrations or roughness. T h e browser server begins by l o a d i n g an X M L - l i k e  7  HapticPageMap file  generated by the client component after reading an H T M L source file. T h e HapticPageMap file contains links to the various X M L - l i k e haptic icon files t h a t are associated w i t h page elements. these files using the open source A C E  8  T h e browser server parses a l l of  L i b r a r y ' s X M L parsing engine. T h e  result is an infernal representation of the haptic structure of the page i n the form of an object m o d e l built around the HapticPageMap object (Figure 6.2). Since the object m o d e l exists w i t h i n the c o m p i l e d C + + component, it can be read r a p i d l y d u r i n g the I / O loop. D e v i c e input a n d output is handled t h r o u g h V i n c e n t Levesque's S T R e S S L i b r a r y ' s A P I [29], w h i c h i n t u r n makes use of l i b u s b and firmware onboard the F P G A for hardware i n p u t and output. T h e browser server's I / O l o o p performs the following actions every cy9  cle: 1. C h e c k the page m a p flag, and reload the HapticPageMap if necessary. 2. R e a d the slider p o s i t i o n and click count from hardware and perform the t r a n s l a t i o n into page m a p coordinates. 3. C o m p u t e the eight-taxel tactile image based on the cursor position and the HapticPageMap. 7  8  9  See the footnote to section 4.9.6. A D A P T I V E Communication Environment [1]. The browser is not currently implemented on a real-time operating system, so its  update interval can not be guaranteed to be 3 ms. See Section 4.7.1 for more information on measured timings.  Chapter  4.  6. Browser  129  Prototype  O u t p u t the tactile image to the hardware.  5. O u t p u t the cursor p o s i t i o n to a 6. O u t p u t the click count to a The  position  click  file.  file.  browser server implements a dual-threaded architecture as a further  performance consideration. F i l e s y s t e m operations are h a n d l e d outside the m a i n I / O thread, and mutex locks ( v i a the A C E l i b r a r y ) are used for d a t a synchronization.  Browser Server Haptic Page M a p Model A  key part of the design of the browser server component is its object  orientation.  Just as graphical web browsers have developed a rich D o c u -  ment Object M o d e l around the H T M L source, the browser server's haptic m o d e l is intended to be flexible and extensible using object-oriented tools. The  base object (equivalent to the  HapticPageMap  object i n the D O M ) is the  Hapticlcons  object, w h i c h contains a list of  the  document. elements  tic  icons,  own  document  Spatiallcon  [] array i n the D O M ) . and  Animatedlcon,  (equivalent to  C u r r e n t l y , two types of hapare supported, each w i t h their  content d a t a stored i n a class inherited from the  Haptic  I  conContent  abstract base class (Figure 6.10). T h i s architecture facilitates adding a d d i t i o n a l types of h a p t i c icons. F o r example, one could imagine a h a p t i c icon that varies its p o s i t i o n based on t i m e a n d / o r slider position. It could be r a p i d l y implemented as another subclass of  Hapticlcon,  polymorphic  or  RenderlconO  code are necessary.  Spatiallcon.  A s long as it supports its o w n  m e t h o d , no changes to the core browser server  In fact, the  Animatedlcon  class was added after the  browser server architecture was complete using the m e t h o d just described.  Chapter 6. Browser Prototype  130  HapticPageMap int height Hapticlcon *icons  •  1  CurrentMapPos() ComputeDetentPosFromVelocity()  Hapticlcon  Computelmage()  abstract base class  RenderAlllconsQ  char 'type, 'name Renderlcon()  HapticlconContent  Spatiallcon  Animatedlcon  type="Spatiallcon"  type="Animatedlcon" AnimatedlconContent *  Spatial IconContent  f  abstract base class  F i g u r e 6.10: object,  SpatiallconContent  AnimatedlconContent  double 'data  double *data[frames]  Inheritance a n d e n c a p s u l a t i o n d i a g r a m for the H a p t i c P a g e M a p  H a p t i c l c o n abstract  base class  and  its i n h e r i t e d classes,  H a p t i c l c o n C o n t e n t abstract base class a n d its i n h e r i t e d classes.  and  Chapter 6. Browser Prototype  131  Further functionality could be achieved by scripting the HapticPageMap model (using JavaScript), in a similar fashion as is currently done with the D O M . This would enable a range of advanced haptic functionality and interactivity leveraging the existing core input/output code.  6.5.3  Interprocess C o m m u n i c a t i o n  and T i m i n g  Communication between the browser server and browser client components is achieved through the use of small files. This has the advantages of simplicity and leveraging the filesystem's implicit locking mechanisms to prevent simultaneous access and data corruption. It has the disadvantage of potentially low performance due to disk overhead. Timing studies were therefore required to validate the filesystem approach. As described in Section 4.7.1, various web pages were run with an oscilloscope connected to the amplifier outputs, and a series of software optimizations achieved an empirically stable output rate. No specific timing studies were done on the G U I (Mozilla) loop, but it was observed that p o s i t i o n and c l i c k files were not accumulating in the filesystem, indicating that they were being read in a timely manner by the client. No delays were observable between slider input and G U I output. As mentioned before, various optimizations were done for high performance, including multithreading and careful task prioritization. Placing the file-exchange directory on a R A M disk did not have any performance effect, probably because the files were already being exchanged via the filesystem's memory cache rather than being written to and read from disk.  Chapter 6.5.4  6. Browser Prototype  132  Browser H a p t i c Icons  A s discussed above i n Section 4.9.6, a g r a p h i c a l image-based workflow allows for interactive icon design a n d r a p i d analysis using the stretch image technique. T h e g i f 2 h a p t i c o n tool not only supports the tactile movie output mode, but also a h a p t i c icon output mode w i t h the - i c o m m a n d line o p t i o n (the default). T h e X M L - l i k e format is similar to the tactile movie syntax but is more o p t i m i z e d for h a p t i c icons for use w i t h web pages. B o t h static ( S p a t i a l l c o n ) a n d t i m e - v a r y i n g ( A n i m a t e d l c o n ) formats are supported, and  the type is a u t o m a t i c a l l y determined by checking whether there are  m u l t i p l e frames stored i n the G I F i n p u t . Icons are stored i n a icon l i b r a r y directory accessible by the browser server a n d loaded at runtime when the H a p t i c P a g e M a p is parsed.  6.6 6.6.1  Known Software Issues and Caveats Support for Element Height  Currently, h a p t i c icons are X M L - l i k e files t h a t are n a m e d under the convent i o n [name] - [ h e i g h t ] . x m l . T h e page rendering engine selects the height value based on the onscreen height of the associated element.  Therefore,  a separate haptic icon file (or at least a s y m b o l i c l i n k to a v a l i d file) must be present for every possible rendered height of the elements. T h i s limits the generalizability somewhat, especially for images, w h i c h m a y have various heights. A possible solution w o u l d be to generate icons a l g o r i t h m i c a l l y (including repeated t i l i n g ) , to use the closest-available height icon, or some c o m b i n a t i o n of the two approaches.  T h e X M L format should then be re-  considered somewhat, m o v i n g the height parameter out of the filename a n d  Chapter  6. Browser Prototype  133  into a separate tag, such as <height>, m u c h as the <at> tag is currently used. M u l t i p l e height representations could also be combined into one file.  6.6.2  Opportunities for further software optimization  T h e browser prototype was implemented w i t h the goal of eliciting user feedback on the a p p l i c a t i o n concept as expeditiously as possible.  T h e r e are  opportunities for i m p r o v i n g the software architecture if the goal is to b u i l d a " p r o d u c t i o n " version w i t h more headroom for future expansion of features: • A formal i m p l e m e n t a t i o n of the X M L standard, i n c l u d i n g a D o c u ment T y p e Definition ( D T D ) w o u l d support future extensibility while retaining c o m p a t i b i l i t y w i t h the existing " X M L - l i k e " i m p l e m e n t a t i o n . • T h e inter-process c o m m u n i c a t i o n m e t h o d , w h i c h c u r r e n t l y uses highoverhead but simple to implement filesystem access, could be revised using more lightweight, o p t i m i z e d c o m m u n i c a t i o n methods for greater performance. • T h e a p p l i c a t i o n could be ported to a real-time operating system for more precise t i m i n g control. • T h e m o d u l a r i t y of the J a v a S c r i p t code base could be o p t i m i z e d for better m a i n t a i n a b i l i t y . A s mentioned earlier, while the software is far from being exhaustively o p t i m i z e d , none of the aforementioned o p t i m i z a t i o n s w o u l d affect the opera t i o n of the browser i n its current form. T h e current version of the browser has been tested to support the a p p l i c a t i o n concept and to deliver stable tactile performance. A s such, it is ready for user evaluation, w h i c h is covered i n the following chapter.  134  Chapter 7  Browser User Evaluation T h i s chapter describes a formal evaluation that was performed on the browser prototype, aimed at measuring its performance i n the hands of a h u m a n user. W e measured the t i m e required to browse web pages to find a piece of i n formation, such as a weather forecast, a n d compared the performance i n three conditions: (1) the handheld prototype w i t h tactile feedback, (2) the prototype w i t h o u t tactile feedback, a n d (3) a mouse scroll wheel used as a discrete n a v i g a t i o n control. T o simulate a perceptually d e m a n d i n g m o bile environment, s t u d y participants were asked to simultaneously perform a distraction task (not identified to the participants as such) using pedals to respond to video cues. T h e d a t a from the study, after n o r m a l i z a t i o n to compensate for differences i n difficulty between the various tasks, d i d not demonstrate a significant performance difference between the two h a n d h e l d prototype conditions, although the mouse scroll wheel was faster as expected. Q u a l i t a t i v e reaction to the tactile feedback was generally positive, w i t h a small p r o p o r t i o n of users reporting m i n i m a l conscious awareness of the s t i m u l a t i o n . In this chapter, the volunteers that p a r t i c i p a t e d i n the s t u d y are referred to as users, a n d the terms user evaluation a n d user test are equivalent.  Chapter 7.1  7. Browser User Evaluation  135  Aims  T h e purpose of the user test was to evaluate the effects of tactile feedback on performance and qualitative user experience i n a h a n d h e l d context using the previously described browser prototype. T h e browser prototype implements a number of novel concepts, i n c l u d i n g the use of artificial tactile flow rendering to provide s p a t i a l cues. T h e results from the user test could serve to validate these concepts, and the u t i l i t y of tactile feedback on a handheld or mobile device i n general, by p r o v i d i n g a w o r k i n g demonstration and performance statistics. T h e s t u d y addresses the following questions: 1. Does the presence of tactile feedback, i n the form of a synthetic tactile flow signal, affect the time required to complete a web navigation task? 2. H o w does the experience of browsing and navigation on the handheld prototype compare to a more c o m m o n p a r a d i g m (the mouse scroll wheel)? 3. W h a t are users' subjective reactions to the use of tactile feedback i n this application? 4. W h a t are users' subjective reactions to the h a n d h e l d prototype hardware and browser software? 5. A r e there any correlations between user demographics, such as age, gender, and previous mobile experience, and their performance i n the browsing task w i t h or w i t h o u t tactile feedback?  Chapter  7.2  7. Browser User Evaluation  136  Study Design  Formally, the user study is designed as a task time performance measure w i t h one p r i m a r y within-subjects factor h a v i n g three levels. T h e selected a l p h a level was 0.05. T h e full list of study variables is o u t l i n e d below.  7.2.1  Study Variables  P r i m a r y Dependent Variable TASK T I M E :  T h e amount of time (in milliseconds) required for the user  to complete the task. T h e task begins when the user signals that they are ready to begin the task (and the browser screen is presented on the display). T h e task ends when the user signals that they have found the  requested  information.  P r i m a r y Independent V a r i a b l e (condition) 3 levels: S L I D E R + (slider w i t h tactile feedback), SLIDER- (slider w i t h o u t tactile feedback), MOUSE (mouse scroll wheel used for navigation).  Additional Dependent Variables T h e following were also measured: • Performance on the d i s t r a c t i o n task (described below i n Section 7.6). • I n d i v i d u a l link access times, w h i c h yielded the amount of time spent on each page. • A qualitative assessment battery (described below).  Chapter  7. Browser User Evaluation  137  A d d i t i o n a l Independent Variables T h e following factors were randomized: •  T h e T A S K T Y P E (3 levels: W E A T H E R , T R A N S I T , a n d  •  T h e specific T A S K :  -  W E A T H E R group: 18 levels  -  T R A N S I T group: 12 levels  -  M O V I E S group: 10 levels  -  (total 40 different tasks)  MOVIES)  • V a r i o u s parameters related to the distraction task (see Section 7.6 below). • User demographics. In a d d i t i o n , a variable, P R E S E N T A T I O N O R D E R , is defined as the present a t i o n sequence number after flattening for C O N D I T I O N . M o r e information on P R E S E N T A T I O N O R D E R is given i n Section 7.3.4.  7.2.2  N o r m a l i z a t i o n for T a s k Difficulty  E a c h of the 40 T A S K S is similar i n the sense t h a t they a l l involve searching for information i n a web page hierarchy using identical navigation methods. However, because the information that the user is requested t o find is slightly different i n each T A S K , each T A S K w i l l have an intrinsic task completion time, referred to here as the T A S K D I F F I C U L T Y . T h e task difficulty is influenced by a number of factors, i n c l u d i n g the depth of the hierarchy at w h i c h the information is found, the position of links on the web page (i.e., the distance  Chapter  7. Browser User Evaluation  138  that the user must scroll to access the l i n k ) , and the c o m p l e x i t y of the cues (e.g., l i n k text) employed by the user d u r i n g navigation. Since T A S K D I F F I C U L T Y is a confounding variable i n our analysis of the effect of C O N D I T I O N  on task time, it is desirable to decouple the measured  task times from the relative differences among tasks, using a n o r m a l i z a t i o n procedure.  T h e first step i n this procedure requires the c o m p u t a t i o n of a  statistical estimate of relative T A S K D I F F I C U L T Y .  Tasks t h a t are relatively  more "difficult" (i.e., have a higher intrinsic task c o m p l e t i o n time) w o u l d have their measured task c o m p l e t i o n times depressed by the T A S K  DIFFI-  C U L T Y score, and vice-versa for relatively "easier" tasks. T h e o r e t i c a l models such as F i t t s ' L a w a n d G O M S could provide predictive estimates of task difficulty as a n o r m a l i z a t i o n score, but there is no single existing published m o d e l that takes into account a l l the factors influencing task difficulty i n the specific context of the current experiment. Furthermore, it is not w i t h i n the scope of the present w o r k to derive and validate an in-depth user m o d e l of browsing using the h y b r i d p o s i t i o n / v e l o c i t y control system on a handheld device using the specific content and layout that was developed for the user study. Therefore, to m a x i m i z e accuracy, an  empirical  TASK DIFFICULTY  score was used, calculated from the measured  d a t a according to the following procedure. A normalization  vector was computed, h a v i n g a n o r m a l i z a t i o n score for  each of the 40 T A S K S . F o r each T A S K , an average score (across a l l subjects and conditions) was c o m p u t e d and compared to the average for a l l T A S K S . The  ratio of current task time to average overall time score was used as the  "difficulty" score, and the n o r m a l i z a t i o n factor was c o m p u t e d by t a k i n g its inverse. By  d e r i v i n g the n o r m a l i z a t i o n score from a pooled statistic across a l l  Chapter  7. Browser User Evaluation  139  users, the task difficulty n o r m a l i z a t i o n does not take into account i n d i v i d u a l user differences.  Sources for these differences might include variations i n  familiarity w i t h certain tasks versus others, or differences i n the relative effects of various factors that influence the task time (navigation distance, verbal complexity, etc.). T h e v a l i d i t y of the n o r m a l i z a t i o n procedure i n light of i n d i v i d u a l subject differences w i l l be explored i n Section 7.8.5.  7.3  Methodology  A t o t a l of 16 volunteers were recruited for the s t u d y (three pilot subjects and thirteen m a i n s t u d y subjects), w h i c h was performed under the auspices of U B C E t h i c s A p p r o v a l #B01-0470 (see A p p e n d i x A ) . E a c h s t u d y session took a p p r o x i m a t e l y 50 minutes and consisted of: 1. Briefing and collection of demographic data. 2. A pre-task attitudes survey. 3. V e r b a l delivery of task instructions. 4. A training, or practice session, minus the distractor task. 5. A second training session, i n c l u d i n g the distractor task. 6. T h e m a i n study task (described further below). 7. A post-task attitudes survey. 8. A post-task interview. 9. C o m p e n s a t i o n ($10 per subject).  Chapter  7. Browser User Evaluation  140  Gender  38% female  Age  median: 28.5 std. dev: 4.6  R i g h t handed  81%  P r e v i o u s experience w i t h a m u s i c a l instrument  77%  A m o n g subjects w i t h previous m u s i c a l instrument ex-  median: 7  perience, number of years played  std. dev: 4.0  R e g u l a r user of a mobile phone  77%  Regular user of a P D A  15%  Regular user of a game controller w i t h v i b r o t a c t i l e  15%  feedback R e g u l a r user of a mouse w i t h scroll wheel T a b l e 7.1:  94%  D e m o g r a p h i c characteristics of the subject p o o l (16 subjects)  that was used for the browser user evaluation, i n c l u d i n g b o t h pilot subjects and m a i n s t u d y subjects.  D u e to v a r i a b i l i t y i n the experimental procedure d u r i n g the three pilot runs (mostly related to fixing bugs i n the computer setup), o n l y d a t a from the 13 m a i n study subjects were used i n the quantitative a n d qualitative analyses of the experiment.  7.3.1  Recruitment of Study Participants  P a r t i c i p a n t s were recruited t h r o u g h the H C I @ U B C subjects database, and t h r o u g h friends a n d associates of the author who met the study criteria. T h e conditions for p a r t i c i p a t i n g i n the study were a good c o m m a n d of E n g l i s h and no prior experience w i t h the browser prototype or the present research  Chapter  7. Browser User Evaluation  141  i n general. T h e final corpus had demographic characteristics as described i n Table 7.1. T h e subject p o o l appears to be a t y p i c a l r a n d o m selection of p o t e n t i a l users of general mobile d a t a services, slightly skewed towards younger participants due to the university setting. Since no d i s t i n c t i o n was made between pilot subjects and m a i n study subjects other t h a n the order i n w h i c h they booked their appointments, b o t h groups of subjects can be considered to be r a n d o m l y d r a w n from the d i s t r i b u t i o n described i n Table 7.1.  7.3.2  User Test Environment  A l l user tests were conducted i n a dedicated, soundproof usability testing lab at the U n i v e r s i t y of B r i t i s h C o l u m b i a d u r i n g n o r m a l business hours or i n the early evening. T h e r o o m was arranged as shown i n F i g u r e 7.1. T h e following devices were arranged i n front of the study participant: 1. T h e tactile display prototype, i n c l u d i n g an embedded L C D display. Because the originally specified 2.5-inch L C D panel malfunctioned prior to the start of the experiment and could not be p r o m p t l y replaced, it was necessary to use a 3.5-inch N T S C T F T colour L C D monitor, attached to the top case w i t h double-sided tape.  T h i s re-  sulted i n an increased weight, especially i n the top part of the device. 2. A mouse (Logitech Scroll Mouse) w i t h scroll wheel. 3. A standard  101-key P C keyboard.  T h e two largest keys,  ENTER  a n d S P A C E , were used for signalling readiness to begin the task and completion of the task,. respectively, and were labelled " S T A R T " and " S T O P " , using 7.5-cm wide yellow sticky notes.  Chapter 7. Browser User Evaluation  F i g u r e 7.1:  B r o w s e r user test  environment.  142  Chapter  7. Browser User Evaluation  143  4. A second P C keyboard (not shown i n the diagram), connected t o the experiment P C a n d used b y the facilitator for resetting the distractor task. 5. A 17-inch colour T F T L C D P C m o n i t o r r u n n i n g at 1280x1024 reso l u t i o n . P r o m p t s to the user were displayed i n 120-pixel h i g h type, centred i n the display o n a white background. 6. A set of pedals, of the type used c o m m o n l y i n home game systems for d r i v i n g games ( A C T L A B S , Performance Pedals). P a r t i c i p a n t s were instructed to place one foot over each pedal. T h e pedals themselves were modified to function as left a n d right mouse buttons, a n d were connected to the P C v i a a U S B mouse. 7. A browser PC, r u n n i n g the L i n u x - b a s e d browser prototype software and connected t o the tactile display, L C D , mouse w i t h scroll wheel, keyboard, a n d local-area network. 8. A n experiment PC, r u n n i n g W i n d o w s 2000 a n d the browser experiment software (described i n Section 7.7), a n d connected t o the P C monitor, pedals, a n d local-area network. 9. T h e driver electronics, connected between the browser P C a n d tactile display prototype.  D u r i n g the experiment, the facilitator remained i n the r o o m w i t h the participant, delivering instructions read from the verbal p r o t o c o l ( A p p e n d i x A ) a n d verbally collecting the information the p a r t i c i p a n t was asked to retrieve. T h e experimenter also remained logged i n to the browser P C v i a a remote t e r m i n a l connection from a separate laptop, performing such functions as  Chapter  7. Browser  User  Evaluation  UA  s t a r t i n g and stopping the browser prototype software, changing configurat i o n settings (for example,  SLIDER+  to  MOUSE),  and m o n i t o r i n g the d a t a  collection process. T h e experimenter also used a k e y b o a r d connected to the experiment P C to signal task completion and to present the next task to the p a r t i c i p a n t when they verbally indicated that they were ready to proceed. In the  SLIDER-  c o n d i t i o n , a l l software settings were the same as the  c o n d i t i o n , except the final amplifier electronics were powered off.  SLIDER+  T h i s ensured that there were no confounding variables, such as t i m i n g differences, i n t r o d u c e d as a result of changing conditions between  SLIDER+  and  SLIDER-.  7.3.3  Briefing a n d Collection of D e m o g r a p h i c  Data  S t u d y participants were first asked to read and sign the s t a n d a r d U B C E t h i c s Consent form (included i n A p p e n d i x A ) , and each p a r t i c i p a n t was given a copy for their records.  A l l participants were verbally r e m i n d e d that they  could abort their p a r t i c i p a t i o n i n the s t u d y at any time w i t h o u t penalty (i.e., they w o u l d still receive their compensation).  P a r t i c i p a n t s were also  r e m i n d e d that their frank and honest opinions were requested, and that the purpose of the study was to o b t a i n a performance assessment of the device, not the s t u d y participant. A l l s t u d y participants then completed the demographic survey and pretest attitudes survey; the results are shown i n Table 7.1 and F i g u r e 7.7. A s noted i n the verbal p r o t o c o l ( A p p e n d i x A ) , a l l participants were instructed to h o l d the device i n their left h a n d 1  1  w i t h their t h u m b resting  Only left handed operation is supported on the handheld prototype. This is also the  case with almost all commercial handheld products having side-mounted controls. Section 4.1.  See  Chapter  7. Browser User Evaluation  145  on the tactile display. P a r t i c i p a n t s were t o l d not to abrade or pick at the surface of the T D , a n d were reminded when necessary to m a i n t a i n a h a n d position such t h a t the t h u m b was i n full contact w i t h a l l 8 elements of the T D . W h e n the mouse scroll wheel was used as a n i n p u t device, the user was also asked to use their left h a n d to operate t h e mouse. I n that case, the device (still required for visual display) was placed o n t h e table and propped up at a n an angle for better visibility.  7.3.4 The  Task Blocks  s t u d y is o r g a n i z e d i n t o B L O C K S of 6 T A S K S each. E a c h B L O C K is per-  formed u n d e r one C O N D I T I O N  (SLIDER+,  SLIDER-,  or M O U S E ) .  Within  a  group of three B L O C K S , each c o n d i t i o n is presented; a n d each subject receives a sequence that is r a n d o m l y d r a w n from the 6 possible permutations of c o n d i t i o n presentation order. Between B L O C K S , s t u d y participants get a one-minute break to relax and stretch. W i t h i n each B L O C K , the 6 T A S K S are also presented i n a r a n d o m i z e d order. T A S K S m a y be categorized i n t o T A S K T Y P E S , c o n s i s t i n g of W E A T H E R ,  T R A N S I T , a n d M O V I E S , each of w h i c h is represented twice per block. T h e r e are 6 possible p e r m u t a t i o n s of the p r e s e n t a t i o n order of 3 T A S K T Y P E S . F o r  each B L O C K , a presentation order is d r a w n at r a n d o m from this set, and the sequence is repeated twice to form a 6 - T A S K B L O C K . F i n a l l y , for each T A S K T Y P E , the p r e s e n t a t i o n order of specific T A S K S is r a n d o m i z e d .  T h e presentation sequence of a n example r u n t h r o u g h the m a i n part of the study can be visualized as follows (six blocks shown): (SLIDER+)A16.C5.B8.A11.C9.B1 (M0USE)C3.B4.A3.C2.B5.A7  — > (break) —>  — • (break) — •  Chapter  7. Browser User Evaluation  (SLIDER-)B3.A15.C1.B2.A17.C4  146  — > (break)  ( S L I D E R + ) A 2 . B 1 . C 8 . A 5 . B 1 1 . C 6 — > (break) ( M O U S E ) C 1 . A 1 8 . B 9 . C 7 . A 1 . B 9 — • (break) (SLIDER-)B10.C5.A4.B6.C9.A10  -  — • (break)  where each letter ( A , B , or C ) represents a T A S K T Y P E , the number following the letter represents a specific T A S K , a n d each line is a B L O C K . N o t e t h a t the sequence of T A S K S does not repeat u n t i l a l l tasks of a p a r t i c u l a r T A S K T Y P E have been exhausted; the sequence for that T A S K T Y P E then repeats i n the same order as before. W h e n , for example, a l l 1 2 tasks of T A S K T Y P E A have been exhausted, the next T A S K d r a w n from T A S K T Y P E A is the same as the first task that was d r a w n d u r i n g the first pass t h r o u g h the sequence. T h e meta-variable P R E S E N T A T I O N  O R D E R is the o r d i n a l T A S K  x  BLOCK  presentation sequence number, w i t h C O N D I T I O N S "flattened" out. F o r m a l l y :  PO  task  =t  task  + T(ceil(b /3) task  - 1)  (7.1)  Where:  • POtask is the P R E S E N T A T I O N  ORDER  number for the given T A S K ,  • ttask is the sequential presentation number for the  TASK  w i t h i n its  B L O C K ( r a n g i n g from 1 to 6 )  • hask is the sequential presentation number of the T A S K belongs (ranging from 1 to 9 ) •  T is the n u m b e r of T A S K S per B L O C K ( 6 )  BLOCK  to w h i c h the  Chapter 7. Browser User Evaluation  147  In the example above, T A S K S A 1 6 , C 3 , a n d B 3 each have a P R E S E N T A T I O N O R D E R of 1, and  T A S K S C5,  B4,  and  A15  have a  PRESENTATION  O R D E R of 2. T h e sequence continues t h r o u g h T A S K S A 2 , C l , a n d B I O , w h i c h have a P R E S E N T A T I O N O R D E R of 7. B y grouping T A S K S that are presented at a p p r o x i m a t e l y the same o r d i n a l position i n different C O N D I T I O N S under a single P R E S E N T A T I O N O R D E R score, a  within P R E S E N T A T I O N  ORDER  analysis of variance of task times is enabled. T h e results from this analysis are presented i n Section 7.8.1.  — T h e full task inventory is i n c l u d e d i n  A p p e n d i x A ; however, some examples are: Al  W h a t is the weather i n L o n d o n today?  A 2 W h a t w i l l the weather be like i n L o n d o n tomorrow? A 3 W h a t w i l l the weather be like i n L o n d o n the day after tomorrow? A 4 W h a t is the weather i n P a r i s today? Bl  If y o u take the 99 B - l i n e from U B C at 1pm, w h e n w i l l you arrive at B r o a d w a y station?  Cl  W h e n is the movie " L ' E n f a n t " p l a y i n g at the R i d g e T h e a t r e ? T h e various randomizations and permutations are h a n d l e d a u t o m a t i c a l l y  by the J a v a S c r i p t component of the Test Software (described i n Section 7.7). E r r o r cases, defined as the user clicking on the w r o n g link, b e c o m i n g distracted (in ways other t h a n t y p i c a l l y induced b y the d i s t r a c t i o n task), or experiencing a problem, were recorded but excluded from the analysis of task times.  W h e n a participant made an error, he/she was asked to  inform the facilitator of the error but complete the task anyway. A "back  Chapter  7. Browser User Evaluation  148  to previous page" l i n k was p r o v i d e d i n a standardized l o c a t i o n on a l l pages of the sample corpus for this purpose.  7.3.5  Training Sessions  P a r t i c i p a n t s first completed two t r a i n i n g sessions to become familiar w i t h the tasks.  In the first t r a i n i n g session, subjects completed 3 shortened  B L O C K S of 2 T A S K S each, i n each of the three conditions ( S L I D E R + ,  SLIDER-,  or M O U S E ) . In the second t r a i n i n g session, subjects also completed 3 shortened B L O C K S of 2 T A S K S each, but w i t h the a d d i t i o n of the d i s t r a c t i o n task (described below i n Section 7.6).  7.3.6  Main Data Collection Session  In the m a i n part of the study, subjects completed 9 full B L O C K S of 6 T A S K S each, or 54 T A S K S i n t o t a l .  7.3.7  Post-Task Assessment  F o l l o w i n g the m a i n d a t a collection task, subjects were given a one-minute break and then asked to fill out a post-task attitudes survey. T h e survey is included i n A p p e n d i x A , and the results are discussed i n Section 7.9. F o l l o w i n g the survey, the facilitator conducted an interview using the protocol documented i n A p p e n d i x A . T h e participants were then given their compensation and were free to go after signing a receipt.  7.4  Pilot Study  A pilot study was first conducted w i t h three participants to assess, debug, a n d o p t i m i z e the experiment protocol and browser prototype design. P a -  Chapter  7. Browser User Evaluation  149  rameters i n the browser prototype software were continuously o p t i m i z e d d u r i n g the pilot phase using a p a r t i c i p a t o r y design m o d e l (the final values are documented i n T a b l e 6.1). I n a d d i t i o n , the decision to implement the h y b r i d control spring design (Figure 6.5) was made based o n user feedback from the pilot study.  7.5  S t i m u l i U s e d i n the S t u d y  T h e final selection of haptic icons used for the s t u d y are shown i n F i g u r e 7.2. T w o types of h a p t i c icons were used. L a r g e headings were m a r k e d w i t h the high-frequency grating icon, a n d links were m a r k e d w i t h the modified singleb u m p crater icon. T o arrive at these haptic icon selections, more t h a n 20 variants of h a p t i c icon representations were created using the image-based r a p i d p r o t o t y p i n g m e t h o d (Section 4.9.6) a n d p a r t i c i p a t o r y design feedback was obtained d u r i n g the pilot phase a n d used i m m e d i a t e l y to tweak a n d o p t i m i z e the icons i n an on-line fashion. Icons were selected for high salience (perceived amplitude), distinguishability from one another, a n d clarity of the directional tactile flow signal. T o achieve these design goals, several points were taken into considerat i o n . F i r s t , the results from b o t h the M D S haptic icon d i s c r i m i n a b i l i t y study (Section 5.4.1) a n d the Image Browser User Test (Section 6.2.3) i n d i c a t e d t h a t the most salient dimension employed by users for d i s t i n g u i s h i n g haptic icons was whether the icons were high frequency a n d periodic or low frequency a n d relatively non-periodic, therefore, the two selections were made on opposite ends of this spectrum. T h e decision to use only two icons was based o n the following justification: the browser user s t u d y was intended p r i m a r i l y to address the effectiveness of tactile flow as a design concept; pre-  Chapter  7. Browser  User  Evaluation  crater  -50V  +50V  150  grating  -50V  +50V  F i g u r e 7.2: H a p t i c icons used i n the browser user study. For each of the two icons, the volt image representation, voltage trace, a n d crop of an associated page element are shown. T h e height of the crater representation (21 page m a p units) is the same as the height of a l i n k (in pixels) when rendered i n the browser w i n d o w . T h e rendered height of headings m a y differ slightly from the height of the associated grating icon; i n that case, the h a p t i c i c o n is top-justified.  Chapter  7. Browser User Evaluation  151  vious studies h a d already addressed the d i s t i n g u i s h a b i l i t y of a wide variety of haptic icons; and the use of a large corpus of haptic icons might introduce undesired v a r i a b i l i t y i n the salience of the tactile flow signal w h i c h the present s t u d y was not powered to evaluate. O n the other hand, i f the effectiveness of tactile flow as a spatial cue was proven, a subsequent s t u d y could examine the detailed design of an o p t i m a l l y sized corpus of haptic icons for use w i t h the browser. A n i m a t e d icons containing t i m e - v a r y i n g m o v i n g components were explored, but since the goal of the experiment was to demonstrate the usefulness of the tactile flow signal as a spatial cue, it was felt t h a t the icons at this stage of development should not interfere w i t h the tactile flow signal b y superimposing a d d i t i o n a l m o t i o n . Therefore, static icons were selected. T h e decision to use the grating icon for headings and the crater icon ' for links, as opposed to vice-versa, was informed by the observation that the directional sensation of tactile flow was m u c h stronger for the nonperiodic signals, and links were more frequent a n d irregularly d i s t r i b u t e d t h a n headings i n the test web page corpus, thus the i m p o r t a n c e of effective tactile flow display for links was higher. F i n a l l y , the icons were designed such that, using the 3 - t o - l subtaxel rendering setting (Section 6.4.10), i n d i v i d u a l piezos w o u l d s t i l l reach maxi m u m (or near-maximum) amplitude displacement.  T o avoid icon overlap  and because pilot testing d i d not reveal any significant advantages i n terms of tactile flow perception, the page m a p s h r i n k i n g feature of the browser prototype software was not used.  Chapter  7.6  7. Browser User Evaluation  152  Distraction Task  D i v i d e d attention between the information appliance a n d the user's environment is a key feature of mobile device usage i n the real w o r l d [40]. T o simulate this experience while retaining the benefits of a controlled laborat o r y testing environment and w i t h o u t i n c u r r i n g the costs of engineering the prototype for true mobility, a visual d i s t r a c t i o n task was implemented as part of the experiment p r o t o c o l . E v e r y seven seconds (beginning seven seconds after the start of the browsing task), the user was p r o m p t e d to perform a n independent  task  w i t h h i s / h e r feet, so as to not affect the hand-based interaction w i t h the mobile device. T h e p r o m p t was displayed on a P C m o n i t o r on the desk, requiring the user to shift h i s / h e r gaze from the hands (below eye level) to horizontal eye level.  T h e p r o m p t was displayed i n large (120-pixel high)  type, such that a change could be observed using peripheral vision while l o o k i n g at the device held i n the hand; however, the p r o m p t was displayed i n A L L C A P I T A L letters i n a serif font to m i n i m i z e preattentive processing of the message. Since the various p r o m p t s were m o r p h o l o g i c a l l y similar and complex, the user was forced to read the message each t i m e . T h e r e were six d i s t r a c t i o n task variants, each of w h i c h involved responding to one of the following commands:  1. Press the left p e d a l once. 2. Press the left pedal twice. 3. Press the right pedal once. 4. Press the right pedal twice.  Chapter  7. Browser User Evaluation  153  5. Press the left, then the right pedal. 6. Press the right, then the left pedal. T o balance i n i t i a l difficulty against increasing user familiarity as they acquired experience w i t h the task, only the first four d i s t r a c t i o n tasks were used d u r i n g the first half of the experiment (3 out of 6 blocks i n t r a i n i n g phase 2, and 4 out of 9 blocks i n the m a i n experiment); after that, the full selection of 6 tasks were used. Feedback was given at seven-second intervals together w i t h a refresh of the task p r o m p t . A cumulative positive reinforcement m o d e l was used, similar to a t y p i c a l video game design. Successive correct performance resulted i n the onscreen presentation of a reward image, consisting of a briefly ( 1 second) a n i m a t e d character such as a smiley face, w i t h increasing valence. For example, the smileys became larger, more numerous, more a n i m a t e d and more embellished to reinforce a string of correct performance. S i x levels of reinforcement were supported. If the user provided an incorrect i n p u t during a seven-second interval, the correct-response counter was reset to zero and the background of the p r o m p t area was changed from white to red. A correct response d u r i n g the next seven-second interval resulted i n the backg r o u n d reverting to white, the counter being incremented by one, and the appropriate reward a n i m a t i o n being presented. T h e effectiveness of the d i s t r a c t i o n task is assessed from a quantitative perspective i n Section 7.8.8 and from a qualitative perspective i n Section 7.9.2.  Chapter  7.7  7. Browser User Evaluation  154  B r o w s e r E x p e r i m e n t Software  A s described i n Section 7.3.2, User Test E n v i r o n m e n t , two P C s were used d u r i n g the experiment, one r u n n i n g the browser prototype software, and the other r u n n i n g the experiment software. T h e experiment software provides the following functionality: 1. R a n d o m i z i n g the selection of tasks i n blocks, using the procedure described i n Section 7.3.4. 2. D i s p l a y i n g task prompts to the user (e.g., " W h a t is the weather like in Paris today?"). 3. D i s p l a y i n g the t i m e d distractor task prompts to the user (e.g., "Press the left pedal twice."). D e s c r i b e d further i n Section 7.6. 4. P r o v i d i n g feedback to the user on their performance i n the distractor task. 5. R e c o r d i n g the distractor task events from the pedals, and tagging the d a t a w i t h t i m i n g information. 6. Sending the collected d a t a to the facilitator's e m a i l account v i a the network. 7. R e a d i n g the k e y b o a r d events from the facilitator. 8. P r o m p t i n g the user to take a one-minute break. In a d d i t i o n , for the most accurate t i m i n g , the browser prototype software also includes several functions related to the experiment: 1. R e c e i v i n g events from the k e y b o a r d when the user signals the start a n d end of the task.  Chapter  7. Browser User Evaluation  155  2. D i s p l a y i n g a m o d a l dialog box b l o c k i n g interaction w i t h the browser u n t i l the user signals the start of the task. 3. S u p p o r t i n g the mouse scroll wheel as a configurable alternate input device. 4. Resetting the browser u p o n user-signalled completion of the task. 5. D u m p i n g t i m i n g d a t a to the t e r m i n a l (recorded on the  experiment  facilitator's remote laptop). T h e browser experiment software is designed for any standards-based web browser (it was r u n on M o z i l l a Firefox 1.0 under W i n d o w s 2000 for the experiment) and consists of the following components (included i n A p p e n d i x  E): 1. A set of X H T M L files w i t h embedded style sheet ( C S S 1.0) information, comprising the presentation layer of the software. 2. A set of JavaScript files, comprising the logic layer. 3. A C G I ( C o m m o n G a t e w a y Interface  1.1) script, based on the  free  " F o r m M a i l " script, r u n n i n g on an off-site server that accepts form d a t a and forwards it to the facilitator's e m a i l account. T h e general architecture is that of a web a p p l i c a t i o n using the m o d e r n A J A X (Asynchronous JavaScript and X M L ) methodology, r u n n i n g m o s t l y on the client side.  It uses the X M L H t t p R e q u e s t object supported i n a l l  current (as of 2006) browsers to send d a t a to a server-based C G I script i n an online fashion, w i t h o u t reloading the page or causing a v i s u a l interruption to the user. T h i s allows for a h i g h l y interactive, portable experiment  Chapter  7. Browser User Evaluation  156  management a p p l i c a t i o n while p r o v i d i n g convenient email-based collection of d a t a w i t h redundant copies d i s t r i b u t e d across networked computers for m a x i m u m safety. T o the author's knowledge, the present software is the first k n o w n a p p l i c a t i o n of this technique for usability experiment management. T h e logical flow of the browser experiment software is as follows: 1. T h e appropriate H T M L file (training phase 1, t r a i n i n g phase 2, or m a i n experiment) is loaded into the browser by the facilitator. 2. T h e H T M L file instructs the browser to load and executes the associated J a v a S c r i p t code. 3. A start screen is presented, p r o m p t i n g the facilitator to enter a p a r t i c i pant number, and i n the m a i n experiment version, an o p t i o n a l number of blocks (default is 9). T h e participant number is p r o v i d e d s i m p l y for convenience and is appended to a l l d a t a output by the p r o g r a m . 4. W h e n the participant indicates to the facilitator t h a t he/she is ready to begin the first task, the facilitator presses a form s u b m i t b u t t o n . 5. T h e J a v a S c r i p t component creates an array for each of the three T A S K T Y P E S . In the m a i n experiment version, the arrays are r a n d o m l y permuted, and the tasks are sequentially assigned to B L O C K S that are stored as associative arrays. 6. A display refresh is performed w i t h o u t reloading the page, accomplished by m a n i p u l a t i n g layer v i s i b i l i t y parameters i n the style sheet. 7. T h e first task p r o m p t is displayed to the user. T h e software waits for a keyboard i n p u t from the facilitator, i n d i c a t i n g that the p a r t i c i p a n t  Chapter  7. Browser User Evaluation  157  has read the p r o m p t and has pressed the S T A R T key, signalling the browser prototype component to start the task. 8. T h e software enters a loop, presenting a distractor task p r o m p t to the user at seven-second intervals. T h e software checks to ensure that no two sequentially presented distractor tasks are alike. 9. P e d a l input from the user is read and stored. W h e n the seven-second interval expires, the input is checked against the task requirement and feedback (correct or incorrect) is provided to the user. 10. W h e n the facilitator presses a key on the keyboard, the task is finished, the distractor task p r o m p t is removed, and the next task p r o m p t is presented on the screen. 11. W h e n a block has been completed, the software sends an i n t e r i m copy of the d a t a log to the server-based C G I script (and thus to the facilitator's e m a i l account). T h e user is p r o m p t e d to take a one-minute break v i a a m o d a l dialog. 12. W h e n a l l blocks have been completed, the software sends the final copy of the d a t a log to the server and presents a m o d a l dialog t h a n k i n g the user for h i s / h e r p a r t i c i p a t i o n . Since the page is never refreshed, the d a t a is retained i n a text input field that is h i d d e n below the scroll b o u n d a r y of the w i n d o w . S h o u l d the e m a i l transmission fail, the d a t a can still be recovered from the page using copy and paste on the client PC.  Chapter  7.8  7. Browser User Evaluation  158  Quantitative Results  A t o t a l of 645 valid, error-free observations of T A S K T I M E were gathered i n each of the 3 C O N D I T I O N S across the 40 T A S K S and 13 participants, and used in the quantitative analysis of performance.  7.8.1 The  Effect of C o n d i t i o n on Task T i m e  results of a repeated-measures A N O V A on the task t i m e d a t a across a l l  13 non-pilot subjects is shown i n T a b l e 7.2. F o r this analysis, T A S K T I M E was c o m p a r e d w i t h i n P A R T I C I P A N T and P R E S E N T A T I O N O R D E R variables. If  an error occurred i n any of the three C O N D I T I O N S for a given level of P R E S E N T A T I O N O R D E R for a given subject, the d a t a for that level of level of P R E S E N T A T I O N O R D E R was not considered i n this analysis; o n l y those combinations of task x subject w i t h full repeated measures across a l l conditions were used, resulting i n a slightly lower number of observations (N) t h a n the overall d a t a set. T h e results show a s t a t i s t i c a l l y significant difference i n T A S K T I M E across the three different C O N D I T I O N S . However, w h e n the C O N D I T I O N S are examined i n a pairwise fashion, there is a s t a t i s t i c a l l y significant difference between the M O U S E c o n d i t i o n and the two S L I D E R conditions, but not among the two S L I D E R conditions (i.e., w i t h and w i t h o u t active tactile feedback).  7.8.2  Individual Subject Differences in Performance  C o u l d the lack of a s t a t i s t i c a l l y significant difference between the two S L I D E R conditions be due to i n d i v i d u a l differences i n the way users respond to the presence or absence of tactile feedback? T h e d a t a were analyzed separately for each subject ( w i t h appropriate B o n f e r r o n i correction for the groupwise  Chapter 7. Browser User Evaluation Condition  M e a n Task Time  Std. Error  N  MOUSE  12166.34  4415.81  181  SLIDER-  18200.52  5453.77  181  SLIDER+  18445.01  4682.06  181  159  Overall A N O V A Conditions Overall  F  p  132.750  0.000  Pairwise Comparison (Bonferroni Corrected) 95% Confidence Interval Conditions  AMean  Low  High  p  M O U S E vs S L I D E R -  -6034.18  -7182.19  -4886.19  0.000  M O U S E VS S L I D E R +  -6278.67  -7292.42  -5264.93  0.000  S L I D E R - VS S L I D E R +  -244.49  -1373.88  884.91  1.000  Table 7.2:  Results from a within-subjects, w i t h i n - P R E S E N T A T l O N O R D E R  analysis of measured task time. A l l measurement units are i n milliseconds.  error rate), and the results are shown i n T a b l e 7.3. A s shown i n T a b l e 7.3, the magnitude of the difference between S L I D E R + and S L I D E R - conditions was small for each participant, a n d none of the differences was s t a t i s t i c a l l y significant.  T h e evidence therefore does not  support the hypothesis that opposing effects of C O N D I T I O N i n i n d i v i d u a l subjects was responsible for the overall lack of statistical significance.  Chapter 7. Browser User Evaluation  7.8.3  160  Effect of Task on Task T i m e  T h e relationship between T A S K a n d measured T A S K T I M E is shown i n F i g ure 7.3. T h e graph clearly shows v a r i a t i o n i n the task times that suggest differences i n task difficulty. T h e task requiring the most t i m e to complete was: C I O W h a t r a t i n g d i d the movie " T h e P r o m i s e " receive i n its review i n the Washington Post? T h e task requiring the shortest t i m e to complete was: A 1 5 W h a t w i l l the weather be like i n San Francisco the day after tomorrow? E x a m i n i n g the web pages i n the test corpus, we see that task C I O requires a m i n i m u m traversal distance of 156 + 852 + 563 + 497 = 2068 pixels or page m a p units: the longest i n the corpus. B y comparison, task A 1 5 requires traversal of 99 + 484 + 254 + 176 = 1013 pixels or page m a p units: the shortest i n the corpus. T h e observations are therefore generally i n accordance w i t h G O M S and F i t t s ' L a w principles about the effects of distance and keystroke i n p u t on task c o m p l e t i o n time.  7.8.4  Effect of Task x C o n d i t i o n o n Task T i m e  C o u l d the lack of a s t a t i s t i c a l l y significant difference between the two S L I D E R conditions be due to opposing effects i n different T A S K S ? F i g u r e 7.4 shows the T A S K T I M E d a t a sorted by T A S K across each of the conditions. A l t h o u g h the d a t a is somewhat noisier due to a lower number of samples i n each c o n d i t i o n versus the overall task time, each of the three conditions follows the same general trend w i t h regard to v a r i a t i o n i n task difficulty; we do not observe any large interaction effects between C O N D I T I O N and T A S K  TIME  Chapter  7. Browser  User  Evaluation  161  Time vs Task I 30000  Task  F i g u r e 7.3:  T h e 40 T A S K S are sorted according to measured T A S K T I M E a n d  displayed here w i t h s t a n d a r d error bars.  Chapter  7. Browser User Evaluation  Average Task T i m e vs  162  Condition  allcond  MOUSE  SLIDER-  SLIDER+  49  18878  16282  20527  19986  1.000  5  54  16294  13487  17358  18036  1.000  6  53  15748  10841  18227  18314  1.000  7  49  16892  10384  19566  19960  1.000  8  47  17775  13170  20226  19558  1.000  9  53  13339 •  9155  15875  15084  1.000  10  51  13203  10654  13763  15376  0.968  11  46  15513  11392  17517  17963  1.000  12  49  17509  13987  19053  19529  1.000  13  48  13650  9055  15613  16283  0.932  14  46  18171  12587  21419  20913  1.000  15  50  21640  17069  23146  24026  1.000  16  50  13187  9248  15413  15031  1.000  645  15287  12056  18245  18499  1.000  Subj.  N  4  A L L T a b l e 7.3:  P  M e a s u r e d t a s k t i m e for e a c h o f t h e 13 m a i n t e s t  participants.  " A l l c o n d " s t a n d s for " a l l c o n d i t i o n s " . T h e p - v a l u e s a r e t h e r e s u l t o f B o n f e r r o n i c o r r e c t e d t - t e s t s b e t w e e n t h e t w o SLIDER c o n d i t i o n s . A l l m e a s u r e m e n t u n i t s are i n milliseconds. F u r t h e r statistics are p r o v i d e d i n A p p e n d i x B .  Chapter across the TASKS.  7. Browser User Evaluation  163  Furthermore, although the performance i n the MOUSE  c o n d i t i o n i n the most difficult TASK (CIO) is worse t h a n the performance i n the SLIDER- c o n d i t i o n i n the least difficult TASK (A15), w h e n compared i n a T A S K - m a t c h e d fashion, the performance i n the MOUSE c o n d i t i o n is always better t h a n the performance i n either of the SLIDER conditions. T h i s does not support the hypothesis that the lack of a n observed difference i n the two SLIDER conditions was due to a n interaction between task difficulty a n d CONDITION.  7.8.5  Validation of Task Difficulty  Normalization  Before proceeding w i t h the task difficulty n o r m a l i z a t i o n as described i n Sect i o n 7.2.2, we s h o u l d verify that the observed v a r i a t i o n i n performance o n a task-by-task basis was indeed due t o intrinsic TASK-related sources, a n d not due t o the effect of the study independent variables. A s discussed i n Section 7.8.3, b o t h the e m p i r i c a l d a t a a n d theoretical models support the hypothesis that v a r i a t i o n i n task difficulty is associated w i t h n a v i g a t i o n a l distance. Furthermore, as F i g u r e 7.4 indicates, the general t r e n d of increasing difficulty w i t h certain tasks is observed i n all three CONDITIONS i n a slightly noisy, b u t m o s t l y monotonic fashion. C o m b i n e d , these observations lend v a l i d i t y t o the concept of task difficulty determined by intrinsic factors, w h i c h m a y be "normalized out" using a n e m p i r i c a l m o d e l of task difficulty. C o n s i d e r a t i o n was given as to whether to use a single, p o o l e d n o r m a l i z a t i o n score from across the three conditions for each TASK (i.e., a norm a l i z a t i o n vector), or to use separate n o r m a l i z a t i o n scores for each TASK x CONDITION (i.e., a 3 x 40 n o r m a l i z a t i o n matrix).  T h e r e was no reason  to reject the hypothesis of s i m i l a r effects of task difficulty across a l l three  Chapter  7. Browser User Evaluation  164  Time vs Task vs Condition  Task  F i g u r e 7.4:  T h e coloured lines show measured TASK TIME vs.  TASK i n  each of the three conditions, while the heavy black line shows the overall (averaged) t i m e across a l l conditions.  T h e TASKS are sorted i n order of  overall measured TASK TIME. P o l y n o m i a l curve fit lines are also shown for each of the conditions. T h e d a t a are noisy due to a low and variable number of observations at each point due to the r a n d o m i z a t i o n of task assignments. The  following d a t a points h a d zero observations a n d are reflected i n the  graph as s m o o t h lines d r a w n between adjacent observations: Tasks A 1 3 and B l l i n the SLIDER- c o n d i t i o n , a n d task A 1 7 i n the MOUSE c o n d i t i o n .  Chapter  7. Browser  User  Evaluation  165  c o n d i t i o n s b a s e d o n t h e d a t a i n F i g u r e 7.4. F u r t h e r m o r e , d u e t o t h e r a n d o m i z a t i o n of tasks, some cells i n t h e 3 x 40 m a t r i x w o u l d have h a d zero observations (the m a x i m u m n u m b e r of observations i n . a cell was 13); using t h e o v e r a l l d a t a t h e l o w e s t n u m b e r o f o b s e r v a t i o n s w a s 9, a n d t h e h i g h e s t w a s 2 4 (see A p p e n d i x B ) . F o r t h e s e r e a s o n s , i t w a s d e c i d e d t o d e t e r m i n e t h e n o r m a l i z a t i o n score for a given task b y considering the d a t a across a l l three conditions. The normalization determine requiring  the  s c o r e s w e r e a p p l i e d t o t h e measured t a s k t i m e s  normalized  to  task t i m e , a n d r a n g e d f r o m 0.733 for t h e task  the most time to complete  ( C I O ) , t o 1.474 f o r t h e f a s t e s t  task  (A15).  7.8.6 Overall  Analysis U s i n g Normalized Task T i m e Effect of Condition  o n N o r m a l i z e d  Task  T i m e  A r e v i s e d v e r s i o n o f T a b l e 7.2, u s i n g t i m e s n o r m a l i z e d for t a s k difficulty, is s h o w n i n T a b l e 7.4. D a t a v a r i a t i o n d u e t o differences i n t a s k d i f f i c u l t y is r e d u c e d , as e v i d e n c e d b y t h e lower s t a n d a r d error a n d i n c r e a s e d F statistic; however, despite increasing the detection power through normalization, the o v e r a l l r e s u l t s b e t w e e n t h e t w o SLIDER c o n d i t i o n s a r e s t i l l n o t s t a t i s t i c a l l y significant.  Individual  Subject  Differences i n N o r m a l i z e d Task  T i m e  x  Condition  N o r m a l i z i n g t h e t a s k scores d i d n o t affect t h e s i g n i f i c a n c e o f t h e o b s e r v a t i o n s o n a n i n d i v i d u a l user basis. F u l l statistics are p r o v i d e d i n A p p e n d i x B .  Chapter  Condition  7. Browser User Evaluation  166  Mean Task Time  Std. Error  N  MOUSE  11474.34  3957.91  181  SLIDER-  17011.57  4455.32  181  SLIDER+  17179.28  3653.15  181  Overall A N O V A Conditions Overall  F  p  163.275  0.000  Pairwise Comparison (Bonferroni Corrected) 95% Confidence Interval Conditions  AMean  M O U S E vs S L I D E R -  -5537.25  M O U S E vs S L I D E R + S L I D E R - vs S L I D E R +  T a b l e 7.4: analysis of  7.8.7  Low  High  p  -6456.06  -4618.41  0.000  -5704.94  -6499.99  -4909.89  0.000  -167.707  -945.67  610.26  1.000  R e s u l t s from a within-subjects, within-PRESENTATlON  normalized  ORDER  task time. U n i t s are i n milliseconds.  Learning / Practice  Effects  E v e n if the difference between the two S L I D E R c o n d i t i o n s was not  statisti-  cally significant across the entire experiment, could the presence of tactile f e e d b a c k h a v e a p o s i t i v e o r a d v e r s e effect o n l e a r n i n g t h e t a s k a n d o p e r a t i o n of the device?  T o address this question, we split the d a t a into bins  for e a c h o f t h e three t h r e e - B L O C K sets c o m p r i s i n g a f u l l r u n t h r o u g h e a c h of the three C O N D I T I O N S .  T h e d a t a are p r e s e n t e d i n F i g u r e 7.5. A  typical  Chapter  7. Browser User Evaluation  167  a s y m p t o t i c performance improvement is observed i n each of the three DITIONS;  the two  CON-  however, it is unclear whether there are significant interactions i n  SLIDER  conditions. T h e shape of the curves suggests that more fa-  m i l i a r i t y w i t h the device or practice w i t h the experimental p a r a d i g m w o u l d not have produced a significant difference between the two  SLIDER  condi-  tions, although the present d a t a do not suggest an expected outcome for a long-term, l o n g i t u d i n a l study.  Time vs Block-of-Blocks 21000 1  —  9000  •  1  ,  1  2  3  Block-of-Blocks  F i g u r e 7.5: T h e measured (mouse, slider-, slider+) and normalized (nmouse, nslider-, nslider+) task times are p l o t t e d versus the presentation order sequence of a set of 3 each of the three  BLOCKS  ("block-of-blocks") comprising a full r u n t h r o u g h  CONDITIONS.  Chapter  7. Browser User Evaluation  168  F o r increased resolution, the d a t a were a n a l y z e d b y the previously described PRESENTATION ORDER meta-variable. T h e results w i t h b o t h meas u r e d a n d n o r m a l i z e d t a s k t i m e s are s h o w n i n F i g u r e 7.6. A g a i n ,  typical  learning curves are observed, w i t h a slight increase i n task t i m e at t h e e n d l i k e l y a t t r i b u t a b l e t o f a t i g u e . T h e d a t a d o n o t i n d i c a t e a n y effect o f t a c t i l e feedback o n learning; however, t h e relative flatness a n d consistency of t h e learning curves lend confidence to the validity of the observations.  7.8.8  Quantitative Validation of Distraction Task  T h e data gathered b y the experiment  P C were a n a l y z e d t o ensure t h a t  participants were performing t h e distraction task p e r the experiment protoc o l , a n d t h e results are s u m m a r i z e d i n T a b l e 7.5. W h i l e s o m e p a r t i c i p a n t s showed higher accuracy t h a n others o n the distraction task, the overall accuracy rate was 8 1 % , showing t h a t participants were a t t e n d i n g t o a n d performing the distraction task. T h e qualitative findings related to the distraction t a s k a r e d i s c u s s e d i n S e c t i o n 7.9.2.  7.9 7.9.1  Qualitative Results P r e - T a s k Attitudes Survey  T h e results f r o m t h e p r e - t a s k a t t i t u d e s s u r v e y are s h o w n i n F i g u r e 7.7. A l l questions were m e a s u r e d o n a L i k e r t scale.  Q u e s t i o n s 1, 5 , 2 , a n d 3  w e r e i n t e n d e d t o assess p a r t i c i p a n t s ' a t t i t u d e s a b o u t p e r f o r m i n g tasks at once o n a handheld device.  multiple  Questions 4 a n d 7 were intended t o  gather some insight into participants' self-assessment of tactile a p t i t u d e a n d preference, w h i c h people have rarely considered before a n d therefore c a n  Chapter  7. Browser User Evaluation  169  Measured Time vs Presentation Order  mouse slider• slider+ •Poly. (slider+) -Poly, (slider-) •Poly, (mouse)  15000  i„„  £  5000  1  2  3  4  5  6  1  2  3  4  5  6  1  2  3  4  5  6  Task / Block  Normalized Time vs Presentation Order  20000  "**.."" "U p nmouse nslidernslider+ • Poly, (nslider-) •Poly. (nslider+) • Poly, (nmouse)  15000  ,.<"*x, *y\.  o E J: IOOOO  ninimm"  1  2  3  4  5  6  1  2  3  4  5  6  1  2  3  4  5  6  Task / Block  F i g u r e 7.6: Task time a n d PRESENTATION ORDER, w i t h p o l y n o m i a l curve fit lines added.  Chapter  Participant  7. Browser User Evaluation  P e d a l P r o m p t s  PedalCorrect  170  % C o r r e c t  4  70  67  96%  5  67  59  88%  6  58  57  98%  7  69  58  84%  8  72  31  43%  9  54  33  61%  10  46  42  91%  11  53  37  70%  12  71  62  87%  13  47  37  79%  14  73  50  68%  15  97  95  98%  16  62  55  89%  839  683  81%  ALL  Table 7.5:  Performance on the distraction task by individual study partici-  pants. The table shows the total number of task prompts that were presented to the user, compared with the number of periods between prompts where the user made a correct response according to the prompt.  Chapter  7. Browser User Evaluation  not be measured using direct questioning.  171  Finally, question 6 was added  based o n qualitative feedback o b t a i n e d d u r i n g earlier studies in w h i c h users e x p r e s s e d a s t r o n g p r e f e r e n c e for t h e p i e z o e l e c t r i c T D v e r s u s a t y p i c a l m o b i l e p h o n e v i b r a t o r d u e to lower noise a n d intrusiveness; for t h i s e x p e r i m e n t , it was hypothesized that pre-existing attitudes about mobile phone vibration might be correlated to performance w i t h the T D . U n f o r t u n a t e l y , b e c a u s e n o s i g n i f i c a n t effect w a s o b s e r v e d f o r t h e p r e s e n c e o r a b s e n c e o f t a c t i l e f e e d b a c k , it is i m p o s s i b l e t o c o r r e l a t e t h e m e a s u r e d attitudes w i t h i n d i v i d u a l subjects' performance o n the task. However, the presence of this d a t a anchors the demographic picture of the subject p o p u l a t i o n a n d reveals a significant diversity in attitudes a b o u t  multitasking  while using a h a n d h e l d device, w h i c h lends ecological validity to the study.  7.9.2  Qualitative Evaluation of the Distraction Task  T h e goal of the distraction task was to cause the participant to split his/her visual attention resources in a m a n n e r similar to that of a person using a m o bile device i n a real-world usage context. D a t a from the experiment indicates that the task was mostly successful i n this regard. T h e facilitator observed that all participants frequently shifted their visual attention between the h a n d h e l d browser's L C D a n d the P C monitor (where the distraction task p r o m p t s were displayed). V i d e o analysis of the a u t h o r a n d several research colleagues performing the distraction task together w i t h the m a i n experimental task further confirmed that the interval between attentional  shifts  w a s o n t h e o r d e r o f a f e w (2 - 10) s e c o n d s a n d o f t e n c o i n c i d e d w i t h t i m e s when the web page was being reloaded, w h i c h matches what was observed i n p r e v i o u s field s t u d i e s o f m o b i l e u s e r s [40].  Chapter  7. Browser User Evaluation  1.1 am a multitasker.  Strongly Agree  Agree  H  Neutral  iiiii  Disagree  2. Mobile phones are useful for more than just talking.  Agree  Neutral  Disagree  Strongly Disagree  Strongly Disagree  5. I prefer to work on one thing at a time.  Strongly Agree  Agree  1  Neutral  Disagree  •  IIIII  3. Sometimes it's necessary to use a mobile phone while driving.  Strongly Agree  172  Strongly Disagree  Strongly Agree  Agree  Neutral  Disagree  Strongly Disagree  4. I'm good at working with my hands.  Strongly Agree  Agree  Neutral  Disagree  Strongly Disagree  6. The vibration function on a mobile phone is annoying.  Strongly Agree  Agree  Neutral  Disagree  Strongly Disagree  7. When I am shopping, I often pick up objects to touch them even though I know I won't buy them.  Jiiii 111  Strongly Agree  Agree  Neutral  Disagree  Strongly Disagree  Figure 7.7: Results from the pre-task attitudes survey for the 13 main study participants.  Chapter  7. Browser User Evaluation  8.1 was able to keep my eyes on the screen of the handheld device during the tasks.  Strongly Agree  Neutral  Strongly Disagree  173  10. The task of pushing the pedals in response to the instructions was challenging.  Strongly Agree  Neutral  Strongly Disagree  F i gure 7.8: Results of the post-task attitudes assessment questions pertaining to the distraction task.  Chapter  7. Browser User Evaluation  174  T h e qualitative d a t a gathered f r o m the p a r t i c i p a n t s after t h e y h a d  fin-  i s h e d t h e t a s k s ( F i g u r e 7.8) m o s t l y s u p p o r t s t h e h y p o t h e s i s t h a t t h e d i s t r a c t i o n task was successful in achieving its goals. T h e r e were m i x e d responses t o Q u e s t i o n 8 ("I  w a s able to keep m y eyes o n the screen of the h a n d h e l d  device d u r i n g the t a s k s . . . " ) , p o s s i b l y d u e to a lack of awareness of where visual attention was directed; indeed, the participants were not asked to p a y a t t e n t i o n to w h e r e t h e y were l o o k i n g u n t i l after all the t a s k s h a d b e e n c o m p l e t e d . H o w e v e r , m o s t s u b j e c t s a g r e e d w i t h t h e p r o m p t i n Q u e s t i o n 10 ( " T h e task of p u s h i n g the pedals in response to the instructions was chall e n g i n g . . . " ) , i n d i c a t i n g t h a t t h e d i s t r a c t i o n t a s k w a s at least p l a c i n g s o m e attentional demands on the subject. In post-task interviews, several participants expressed frustration  with  the level of s y s t e m feedback o n the d i s t r a c t i o n task. F e e d b a c k was o n l y given at 7-second intervals w h e n the p r o m p t was c h a n g e d , not i m m e d i a t e l y foll o w i n g t h e u s e r i n p u t , s o u s e r s w e r e s o m e t i m e s c o n f u s e d as t o w h e t h e r t h e y h a d s u c c e e d e d i n e n t e r i n g t h e r e q u e s t e d i n p u t . It w a s e s p e c i a l l y d i f f i c u l t f o r p a r t i c i p a n t s d u e to the use of two s i m u l t a n e o u s tasks i n a n a t t e n t i o n - s t a r v e d p a r a d i g m (the facilitator o b s e r v e d t h a t this c a u s e d s u b j e c t s to forget w h i c h i n p u t s they h a d m a d e ) , a n d the p o o r mechanical feedback of the p e d a l h a r d w a r e , w h i c h w a s o r i g i n a l l y d e s i g n e d for c o n t i n u o u s , a n a l o g u e i n p u t . ever, the d i s t r a c t i o n task was intended to require users to d i v i d e their  Howown  attention w i t h i n seven-second blocks rather t h a n to interrupt or preempt the user at machine-specified times; i m m e d i a t e , a s y n c h r o n o u s feedback w o u l d have interfered w i t h this m o d e l somewhat.  Chapter  7. Browser User Evaluation  9. The task of finding information on the page was challenging.  ,1  :  i  Strongly  •  •  Neutral  Agree  11. The tactile feedback was helpful in locating items on the page.  _. Strongly  Strongly  Strongly  Disagree  Agree  Disagree  12.1 would pay 5% more for a mobile phone that had this kind of tactile display.  Strongly  Neutral  Agree  175  13. The tactile feedback I experienced today was pleasant.  Strongly  Strongly  Strongly  Disagree  Agree  Disagree  14. Overall, I found the device easy to use.  Strongly  Neutral  Agree  F i g u r e 7.9:  Strongly Disagree  Results of the post-task attitudes assessment questions pertain-  ing to the navigation task.  Chapter 7. Browser User Evaluation 7.9.3  176  Qualitative Evaluation of the Navigation Task  T h e questionnaire d a t a gathered from the participants on the m a i n (i.e., navigation) task are shown i n F i g u r e 7.9. A d d i t i o n a l l y , the following feedback was obtained d u r i n g the post-task interviews: • M o s t (12/13) participants felt that their performance was best i n the MOUSE condition.  Hardware-related reasons included more familiar-  i t y w i t h the mouse scroll wheel m o d e l (5/12) a n d better ergonomic placement of the hands (6/12). • F i v e out of 13 participants said they thought their performance was better w i t h active tactile feedback. Reasons cited included a qualitative preference for being able to feel the page elements. • A l m o s t a l l of the users criticized the velocity-control scrolling m o d e l compared to the mouse scroll wheel m o d e l . T h e most c o m m o n l y cited reason was that continuous scrolling was less efficient t h a n discrete (i.e., l i n k - t o - l i n k ) scrolling, although 3/13 users also expressed discomfort w i t h the irregular j u m p i n g m o t i o n of the mouse scrolling i n areas where links were sparse. • . Surprisingly, 4/13 users reported no awareness of the tactile feedback u n t i l they encountered the post-task assessment questionnaire.  One  user said they d i d not notice the tactile feedback u n t i l about halfway t h r o u g h the study. T h i s was despite a l l users h a v i n g been informed that they m a y experience tactile feedback d u r i n g the study. T h e perceptual load due to the distractor task m a y have h a d an effect on this lack of awareness.  Chapter  7. Browser User Evaluation  177  • Miscellaneous comments and suggestions i n c l u d e d i m p r o v i n g the h a n d position for better ergonomics, reducing the i n e r t i a of the slider for lower fatigue, and i m p r o v i n g the contrast and c l a r i t y of the embedded L C D screen.  7.10  Discussion  R e t u r n i n g to the questions posed at the start of the study: 1. Does the presence of tactile feedback, in the form of a synthetic tactile flow signal, affect the time required to complete a web navigation task? It does not appear that tactile flow feedback aids targeting and navigat i o n performance i n the browser m o d e l used. B y carefully inspecting the d a t a along various dimensions (subject, PRESENTATION ORDER, etc.), and increasing the power of the experiment t h r o u g h the task difficulty n o r m a l i z a t i o n , we can more confidently state t h a t there does not appear to be an effect of tactile feedback i n the  measurements  taken. W h i l e this is clearly d i s a p p o i n t i n g i n light of the hypotheses formulated at the start of the experiment, the findings enable us to focus on areas for improvement of the device rather t h a n on issues w i t h the experimental design. 2. How does the experience of browsing and navigation on the handheld prototype compare to a more common paradigm (the mouse scroll wheel)? T h e familiarity and ergonomic comfort, as well as the discrete scrolling model, of the mouse scroll wheel were preferred.  Chapter  7. Browser User Evaluation  3. What are users' subjective this  reactions  178  to the use of tactile feedback in  application?  P a r t i c i p a n t s were m o s t l y either ambivalent about the tactile feedback or reacted positively. 4. What are users' subjective ware and browser  reactions  to the handheld  prototype  hard-  software?  T h e prototype was m i l d l y criticized for its ergonomic profile and weight. 5. Are there any correlations gender, and previous  between user demographics,  mobile experience,  browsing task with or without tactile  such as age,  and their performance  in the  feedback?  D u e to a lack of overall statistical significance between the two SLIDER conditions, it was not possible to o b t a i n correlative d a t a on these demographic characteristics. W h y was performance seemingly unaffected by the tactile flow rendering setting?  P r i o r to the start of the experiment, we hypothesized that the  following factors, i f satisfied, may influence the measured performance i n the tactile feedback c o n d i t i o n : 1. T h e tactile rendering could be perceived as a sense of tactile flow by the user. 2. T h e tactile flow signal w o u l d simulate m o v i n g spatial cues that occur i n the real w o r l d when the user is touching objects. 3. T h e simultaneous, m u l t i m o d a l i n p u t of b o t h visual and haptic cues w o u l d allow the user to more q u i c k l y gain a sense of spatial l o c a t i o n i n a v i r t u a l space (i.e., web page).  Chapter  7. Browser User Evaluation  179  4. G i v e n a more r a p i d , more confident acquisition of spatial location, targeting performance w o u l d be increased. 5. A n y performance changes w o u l d be measured by the experimental paradigm. T h e first factor should be true, based on the fact that the parameters used were w i t h i n the l i m i t s established by the P e r c e p t u a l C h a r a c t e r i z a t i o n studies, a n d the haptic icon design effort to m a x i m i z e the a m p l i t u d e of the percept created by the T D . However, the fact that several users d i d not notice the presence or absence of tactile feedback under conditions of h i g h perceptual load suggests that the a m p l i t u d e m a y be on the low (i.e., weak) side. Since the signals were already m o s t l y o p t i m i z e d , the r e m a i n i n g l i m i t a t i o n s are p r i m a r i l y i n the hardware and the physical constraints of the piezoelectric elements used i n the display. W e a k s t i m u l a t i o n could be responsible for the lack of a statistically significant performance difference, but it seems unlikely that no performance difference w o u l d be observed due to low a m p l i t u d e alone, since users are universally able to detect the tactile feedback when their attention is directed towards it. R e g a r d i n g the second and t h i r d factors, the synthetic tactile flow rendering differs from t y p i c a l real-world touch i n a significant way: i n n a t u r a l touch, a tactile flow percept is generated when the fingertip is moved (i.e., slid) across a textured surface, usually accompanied by the proprioceptive sensation of the directed movement of the b o d y part. In the browser model, the tactile signal is decoupled from its t y p i c a l proprioceptive accompaniment and controlled w i t h a novel force- and position-sensitive v i r t u a l scrolling m o d e l . Moreover, i n the w o r l d of n a t u r a l touch, externally m o d u l a t e d sliding (tactile flow) signals are usually l i m i t e d to accidentally losing one's grip  Chapter  7. Browser User Evaluation  180  on an object, or an insect crawling across the fingertip, where the semantic interpretation of the tactile signal is not one of s p a t i a l position, but of an alert. A synthetic, decoupled tactile flow signal, therefore, has a significant challenge to overcome i n being a novel way of p r o v i d i n g a spatial cue. T h e fourth factor is likely to be true on the basis of general psychological principles of semantic activation and directed movement. In a d d i t i o n , a thorough analysis of the experiment, combined w i t h the fact that performance differences were observed i n the M O U S E c o n d i t i o n , lends confidence to the fifth c o n d i t i o n . Therefore, it seems that the most likely e x p l a n a t i o n for the lack of performance improvement was that the concept of decoupled tactile flow signal as a s p a t i a l cue was not validated by the current experiment. H o w can the browser be improved?  I n a d d i t i o n to the  performance  data, a significant amount of qualitative d a t a about the browser i n the hands of a h u m a n user was collected. T h e study suggests that user comfort can be increased by i m p r o v i n g the ergonomic characteristics of the device, i n c l u d i n g t h u m b position, size, and weight. Instead of continuous velocitybased scrolling, the navigation m o d e l could be discretized, either p a r t i a l l y (as i n the case of the mouse scroll wheel) or w h o l l y (as i n the case of scroll keys), allowing users to more easily target links. A discretized n a v i g a t i o n m o d e l could use different h a p t i c signals, i n c l u d i n g icons t h a t use tactile flow for identification rather t h a n spatial cueing, to w i d e n the v o c a b u l a r y of tactile signals and perhaps become useful not as s p a t i a l cues, but as eyesfree semantic cues as to the content of the web page. These opportunities for further development are considered i n further detail i n the conclusion i n the following chapter.  Chapter  7. Browser User Evaluation  181  In summary, the key contributions of the Browser User E x p e r i m e n t are that, having thoroughly tested the a p p l i c a t i o n of tactile flow i n a browser m o d e l , we can rule out further o p t i m i z a t i o n of haptic icons, scroll m o d e l parameters, browsing content, and experimental design, and instead direct our efforts towards a p p l i c a t i o n designs that do not rely on decoupled tactile flow rendering as a spatial cue. Q u a l i t a t i v e feedback obtained d u r i n g the study gives clues about how to significantly improve the hardware and navigation models. W e have also established an experimental p a r a d i g m , i n c l u d i n g a dist r a c t i o n task and novel, web-based experiment management software, w h i c h can be used to evaluate further progress i n handheld, m u l t i m o d a l interfaces.  182  Chapter 8  C o n c l u s i o n and F u t u r e W o r k In this chapter we summarize the key contributions of this work, revisit the research questions posed earlier, and outline p r o m i s i n g future directions enabled by this work.  8.1  Summary of Key Contributions  T h i s thesis makes the following contributions to the b o d y of knowledge about interaction design for mobile haptics.  8.1.1  Identification of a novel multimodal approach to addressing limitations in mobile user interfaces  C o n v e n t i o n a l approaches to mobile interaction design using existing hardware suffer from l i m i t e d interface b a n d w i d t h and poor s u i t a b i l i t y for mobile conditions of v i s u a l a n d / o r a u d i t o r y impairment. C o n t r i b u t i o n s of this work include the a r t i c u l a t i o n of this need, a survey of existing haptic technologies a n d the selection of the piezoelectric actuated lateral skin-stretch technology, mounted i n a simplified 1-D slider configuration, as a tractable m e t h o d for addressing the requirements of mobile haptics.  C o m p a r e d to existing  v i b r o t a c t i l e approaches, our concept of mobile haptics enables richer tactile experiences while managing cost, power, size, and weight tradeoffs.  The  Chapter  8. Conclusion and Future Work  183  v a l i d i t y of this selection has been demonstrated by the realization of a handheld prototype s u p p o r t i n g interaction i n both haptic and visual modalities.  8.1.2  Development of a new h a n d h e l d haptics hardware platform  H a v i n g selected the p r o b l e m d o m a i n and the tactile display technology, w e  1  proceeded w i t h i m p l e m e n t a t i o n of a hardware and software p l a t f o r m for p r o t o t y p i n g and testing handheld h a p t i c applications.  T h i s inexpensive,  reproducible prototype, i n c l u d i n g the various technical developments incorporated w i t h i n , represents another c o n t r i b u t i o n of this research project. E v e n after the major engineering effort on the prototype (carried out at M c G i l l University) was completed, the hardware and software specificat i o n evolved t h r o u g h insights obtained by using the p l a t f o r m for applicat i o n development a n d c o n d u c t i n g user tests. For example, the need for a spring-based return-to-centre feature for velocity control was identified during a p p l i c a t i o n development.  T h e specification of the prototype platform  at the time of w r i t i n g represents the result of this iterative development. T h e existence of the prototype enables further research i n h a n d h e l d haptic applications w i t h o u t the need for development of entirely new hardware. 1  The hardware prototype development was a collaborative effort with members of the  McGill University Haptics Lab. The present author was a contributor at all stages of the prototype development, but the electronic and mechanical design was lead by Jerome Pasquero, and the control software design was led by Vincent Levesque. For more information, please see Chapter 4.  Chapter 8.1.3  E v o l u t i o n user  8. Conclusion and Future Work  o f a p p l i c a t i o n  studies  design  a n d h a r d w a r e  c o n c e p t s  184  b a s e d  o n  d e v e l o p m e n t  T h i s thesis describes a suite of novel application designs that are matched to the capabilities a n d l i m i t a t i o n s of the handheld prototype. I n a d d i t i o n to the detailed i n i t i a l specification based u p o n a design philosophy a n d backg r o u n d research, this work presents evolutionary development of the applicat i o n concepts based o n insights learned d u r i n g the p r o t o t y p i n g process. F o r example, the size of the "vocabulary" of haptic icons available for use w i t h the browser a p p l i c a t i o n — a n d thus the range of haptic icon functionality — was discovered t h r o u g h perceptual characterization experiments ( C h a p ter 5).  8.1.4  M e t h o d  f o r r a p i d  r e p r e s e n t a t i o n  p r o t o t y p i n g  o f tactile  a n d g r a p h i c a l  stimuli  A s described i n Section 4.9, novel approaches for v i s u a l i z i n g voltage outputs, spatial volt images, a n d skin stretch images were developed to address the previously cumbersome process of creating, managing, a n d a n a l y z i n g tactile stimuli.  I n a d d i t i o n , an accessible p r o t o t y p i n g procedure using standard  web-based g r a p h i c a l tools, a n d support utilities for automated parsing of the visual representations,  are introduced.  These methods c a n be generalized  to any tactile display that produces s p a t i a l l y d i s t r i b u t e d patterns of s k i n stimulation.  Chapter 8.1.5  8. Conclusion and Future Work  185  P e r c e p t u a l c h a r a c t e r i z a t i o n of a novel m i n i a t u r e piezoelectric tactile display  Rather than immediately developing applications based on the developers' own vague and subjective impressions of the capabilities of the new system, we first conducted a detailed user evaluation to determine its "vocabulary" of signals that human users are capable of perceiving. Both the results of these user studies, and the validation of this method as a useful component of user-centred design for novel output hardware, represent contributions of this work. We determined that directional tactile flow can be perceived reliably at speeds up to 0.34 metres per second, and that haptic icons may be designed along dimensions of — in order of decreasing saliency level — repeating versus nonrepeating waveform, direction, speed, and amplitude. These findings may be put to use by anyone who is interested in developing applications for a tactile display with similar physical parameters to the one we used.  8.1.6  Handheld  browser application w i t h tactile  enhancement  As an example of one high-fidelity application prototype, we constructed a multimodal browser capable of reading web pages and displaying them simultaneously with visual and haptic representations. The method for translating web markup to a haptic page map, and the navigation model for a 1-D slider are examples of novel contributions of this effort. Usability testing of the handheld browser did not demonstrate a performance (task time) advantage for navigation with directional tactile cues (tactile flow), although the qualitative feedback obtained from the users sug-  Chapter  8. Conclusion and Future Work  186  gests that there m a y be a subjective preference for the tactile confirmation of target acquisition. T h e results m a y be used to guide further development of applications (Section 8.3, below), and to highlight areas where hardware improvements are likely to have the most significant effect on u s a b i l i t y (Sect i o n 8.4).  8.1.7  M e t h o d for u s a b i l i t y testing of m o b i l e applications  A c o n t r i b u t i o n of this thesis is the m e t h o d used for testing a h a n d h e l d device i n the laboratory while s i m u l a t i n g some of the cognitive and perceptual load aspects of mobile interaction i n the real w o r l d . T h e system incorporates a pedal-based d i s t r a c t i o n task that requires the user to engage the feet and v i s u a l system w i t h o u t interfering w i t h the hands, similar to the scenario of a person w a l k i n g while using a device. A n experiment manager software a p p l i c a t i o n , b u i l t using m o d e r n web technologies, provides cumulative reinforcement feedback to the user and collects d a t a i n a reliable, d i s t r i b u t e d fashion.  T h i s software is available  for re-use by any researcher who wishes to conduct a user experiment using simulated mobile conditions.  8.1.8  Case s t u d y of a user i n t e r a c t i o n design process for haptics  F i n a l l y , the activities described i n this thesis can be considered, from a case study perspective, as one full iteration of a user-centred design process (Figure 1.2) that begins w i t h an analysis of user needs a n d proceeds i n a stepwise cascade of discovery and increasing understanding of the interaction between the system and the user. T h e results from each stage of this process  Chapter  8.  Conclusion and Future Work  187  provide i n p u t for further iterative improvement. T h i s work presents a specialized case of the classic user-centred design archetype, one that is specifically tailored to the needs of contemporary research i n haptics. B y demonstrating that it is possible to follow the usercentred approach even w i t h nascent technology a n d heavy dependency o n hardware configuration, this work lends v a l i d i t y to the concept of incorpor a t i n g users early i n a design process i n order to focus o n achieving good usability a n d to m i n i m i z e system- or technology-centric design.  8.2  Research Questions  T h e research questions posed at the beginning of this thesis were addressed i n the following ways. 1. What are the problems with existing mobile user interfaces that may be addressed using haptics? In C h a p t e r 1 we identified two fundamental problems of existing mobile interfaces: sensory bandwidth limitation a n d environmental (contextual) competition for visual and auditory attentional resources. H a p tics is well suited to address these challenges t h r o u g h parallel feedback i n a sensory m o d a l i t y that is still m o s t l y unused i n mobile interaction. In C h a p t e r 3 we described the i n i t i a l stages of an iterative a p p l i c a t i o n design process that has identified 2. How can one implement haptics on a mobile device despite power, size, and weight restrictions? In C h a p t e r 2 we discussed existing approaches for h a p t i c transducers a n d identified the piezoelectric lateral skin-stretch technology as a  Chapter  8. Conclusion and Future Work  188  promising candidate for further investigation. The resulting piezo tactile display represents a compromise: it is smaller, lighter, consumes less power, and avoids the mechanical grounding challenges of motors used for force feedback, but it is also more complicated than a typical mobile phone vibrator. However, the technology we used also offers a significantly richer haptic experience than mere vibration. In addition to the selection of the tactile display itself, our investigation into various alternatives for mobile haptics hardware resulted in the choice of a linear slide-mounted tactile display positioned on the side of the device as a working compromise between complexity (and thus, size and weight) and functionality. After application prototyping and user testing, we now know that features such as a spring return-to-centre and increased slider travel (perhaps enabled by a different mounting configuration) should be considered for the next iteration of design, in order to improve the usability of the hardware when used for cursor navigation. Since our choice of technology was partially influenced by practical considerations such as access to manufacturing facilities, it is conceivable that if the process was replicated by someone else, a different hardware focus might have been chosen. For example, someone with access to M E M S (Micro-Electro-Mechanical Systems) facilities, or specialized intellectual property, might have been able to create a hardware prototype just as rapidly as we were, without using off-the-shelf parts. However, the interaction designs we proposed prior to the selection of the hardware technology are generalizable in the sense that they do not assume large grounded forces (which would be impractical on  Chapter  8. Conclusion and Future Work  189  a mobile device regardless of haptic display technology), and provide a "bridge" between existing mobile user needs and a rough hardware specification that m a y be implemented using a variety of technologies.  3. What are the expressive capabilities of the hardware prototype in the hands of a human user? In C h a p t e r 5 we described the results of three perceptual characterizat i o n experiments that revealed the usable stimulus speed range, d i s t i n guishable haptic icon design parameters, and hierarchy of salience of various parameters such as speed, direction, waveform a n d amplitude.  4. What are the engineering challenges associated with building a highfidelity hardware and software prototype with handheld use in mind? T h r o u g h the process of designing and b u i l d i n g the prototype platform and its a p p l i c a t i o n software, we discovered a variety of new information related to the realization of a handheld haptic device. F o r example, on the hardware side we were the first to validate the use of a stiffer double-pinning configuration for the piezo benders as a key factor enabling m i n i a t u r i z a t i o n (Section 4.3.1).  2  T h r o u g h experiments w i t h the  control software a l g o r i t h m s , we discovered that a 12-Hz refresh rate 3  was insufficient, while an 83-Hz refresh rate was sufficient for producing s m o o t h tactile sensations on this T D . F i n a l l y , we encountered  the  p r o b l e m of v i s u a l i z i n g and p r o t o t y p i n g haptic icons, and addressed it using the methods described i n Section 4.9. 2  The double-pinning configuration was invented by project collaborators Jerome  Pasquero and Qi Wang. The control software development and optimization was carried out in collaboration 3  with Vincent Levesque.  Chapter  8. Conclusion and Future Work  190  B y b u i l d i n g a representative software a p p l i c a t i o n (the browser) we learned that, under the present system configuration, an asynchronous design w i t h p r i o r i t y given to the tactile process over the visual process was necessary for good h a p t i c performance.  T h e need for a sophis-  ticated cursor n a v i g a t i o n m o d e l was also recognized, w i t h iterative development eventually resulting i n a h y b r i d p o s i t i o n / v e l o c i t y control model t h a t promotes active tactile e x p l o r a t i o n w i t h i n the central region of the slider travel while allowing traversal of larger areas t h a n w o u l d be possible w i t h p o s i t i o n control alone. 5. Is tactile flow an effective aid for user  navigation?  T h e user studies conducted w i t h the browser prototype d i d not demonstrate a significant performance (task time) advantage for tactile  flow  as an a i d to n a v i g a t i o n or target acquisition i n the browsing context. However, qualitative feedback from the same studies was often positive, w i t h m a n y users i n d i c a t i n g a subjective preference for the e x t r a tactile feedback. F u r t h e r user studies could perhaps better elucidate the ways this m u l t i m o d a l interaction could be used to better support user goals other t h a n more r a p i d navigation. 6. How can user-centred hardware technology  design methodology is still the primary  be applied to haptics, determinant  of user  where interface  capabilities? It is a c o m m o n misconception that emerging fields like haptics require designers to do a lot of up-front effort on the display technology before involving users, due to the c o m p l e x i t y and lack of s t a n d a r d i z a t i o n of the interface hardware. User testing, when done, is t y p i c a l l y u t i l i z e d  Chapter  8. Conclusion and Future Work  191  for fine-tuning the software applications, or s i m p l y v a l i d a t i n g the usefulness of a p a r t i c u l a r approach. A t the same time, it is well k n o w n that i n c o r p o r a t i n g user studies early i n the design process is essential for achieving m a x i m u m usability benefit per development effort [22]. Some of the methods we used to incorporate user testing early i n the design process, such as perceptual characterization, have also been used i n previous studies (e.g., [12]). However, i n the present study, we have conducted a full cycle of development, i n w h i c h u s a b i l i t y considerations were the p r i m a r y driver at each stage — from concept work that began w i t h an i n q u i r y about user needs and delayed the detailed specification of hardware u n t i l after usage scenarios had been determined, to the further development of the a p p l i c a t i o n concepts informed by perceptual characterization studies, and finally, to the two-stage development of the browser a p p l i c a t i o n consisting of a low-fidelity prototype followed by user testing, and a high-fidelity prototype w h i c h was then formally evaluated. Therefore, we have demonstrated that, even when the user interface is h i g h l y dependent on technological l i m i t a t i o n s , it is possible to follow a user-centred design process and to benefit from the streamlined p a t h towards good u s a b i l i t y that this approach allows.  8.3  Future Work: Application Designs for Further Investigation  Informed by the results of the hardware p r o t o t y p i n g , perceptual characterization, a n d high-fidelity software prototype and user test w i t h one selected  Chapter  8. Conclusion and Future Work  192  application, we m a y now return to the a p p l i c a t i o n design stage to consider the feasibility of further applications based on the same handheld haptics concept.  8.3.1  Applications Involving Shape Rendering  A p p l i c a t i o n s i n v o l v i n g shape rendering are those that take advantage of the piezoelectric tactile display's a b i l i t y to m i m i c patterns of skin stretch analogous to those created by touching s m a l l objects and surface features.  H a n d h e l d Braille Reader T h e a b i l i t y of a piezoelectric lateral skin-stretch tactile display to render legible B r a i l l e dots v i a active tactile exploration has been previously demonstrated i n [30]. A mobile version has various applications for b l i n d users i n c l u d i n g discreet text messaging or reading printed characters w i t h o p t i c a l character recognition ( O C R ) . T h e o n l y hardware change t h a t w o u l d be necessary for p r a c t i c a l i m p l e m e n t a t i o n w o u l d be the use of two rows of piezo arrays for each of the two vertical B r a i l l e dot positions. T h e existing h y b r i d position / velocity scroll m o d e l could be used to overcome slider travel l i m itations, but further testing w i l l have to be done to determine the o p t i m a l parameters for legibility of B r a i l l e .  Medical Application:  D i s p l a y i n g S m a l l Shapes  A l t h o u g h not s t r i c t l y a mobile application, the a b i l i t y to d i g i t a l l y transmit and reproduce s m a l l shapes c o u l d enable a remotely located doctor to factually palpate an area on a patient w h i c h is sensed by a h i g h resolution tactile sensor such as the one described i n [37]. A n example might be detailed ex-  Chapter 8. Conclusion and Future Work  193  a m i n a t i o n of the results of a tine test, w h i c h produces distinctive patterns of swelling depending on a n t i b o d y status, w h i c h are often much more difficult to distinguish v i s u a l l y t h a n factually.  8.3.2  General H a p t i c Icon Applications  In the following group of applications, the T D is used as a fixed o u t p u t transducer,  w i t h m i n i m a l direct c o u p l i n g between user i n p u t and tactile  output. T h i s type of interaction was tested and verified i n the M D S and speed study experiments, and does not assume untested concepts such as spatial acquisition by tactile flow rendering.  Therefore, it is expected to  have a high p r o b a b i l i t y of successful deployment. However, factors such as environmental d i s t r a c t i o n and m a t c h i n g between the i n f o r m a t i o n content of the o u t p u t a n d the T D ' s capabilities must still be considered.  Instant M e s s a g i n g / N o t i f i c a t i o n A s described i n Section 3.3.4, tactile notification of presence status changes could enrich the experience of using instant messaging or other social applications. It is expected that the effects of haptic enhancement w o u l d be most pronounced under conditions of w o r k l o a d [12] or environmental distraction.  8.3.3  Spatial Signalling  T h i s group of applications makes use of the T D ' s capacity to display a d i rectional signal. A s the M D S analysis demonstrated, users can distinguish at least two directions, three speeds, and two amplitudes i n a o p t i m a l controlled l a b o r a t o r y setting. T h e semantic usefulness of those parameters i n a real-world context of m u l t i p l e distractions and variable tactile engagement  Chapter 8. Conclusion and Future Work  194  still needs to be determined.  Navigation A s described i n Section 3.3.3, a d i r e c t i o n a l signal m a y be u t i l i z e d for navigation feedback. B a s e d on the results of the browser user study, it is not clear that the d i r e c t i o n a l signalling w i l l produce an i n t u i t i v e directional response i n the user.  R a t h e r , it is perhaps more likely that the user w o u l d  come to associate a specific h a p t i c icon (in this case a directional signal) w i t h a semantic meaning, and produce responses appropriately. However, this hypothesis requires further testing.  8.3.4  Browser Improvements  W h i l e the concept of tactile flow rendering to improve spatial n a v i g a t i o n was not validated i n the present study, there is r o o m for numerous improvements i n the browser prototype. These include: • G e n e r a l software o p t i m i z a t i o n to improve I / O performance,  perhaps  m o v i n g to a real-time operating system a n d / o r embedded controller for the tactile loop. • D y n a m i c haptic icon rendering based on the pixel height of page elements. • A n a l g o r i t h m to properly handle h o r i z o n t a l l y l a i d out elements that overlap i n the v e r t i c a l d o m a i n . T h i s w o u l d require a more complex 2D (visual page map) to I D (haptic page map) transformation engine t h a n is currently implemented.  Chapter  8. Conclusion and Future Work  195  • M a x i m u m usable scrolling speed can perhaps be further increased by a d d i n g a "high speed m o d e " : when the user is scrolling at high speed, 4  the n o r m a l haptic icons can be d y n a m i c a l l y substituted w i t h alternate representations o p t i m i z e d for m a x i m u m a m p l i t u d e , and perhaps "stretched" so that they r e m a i n under the finger longer, or by displaying o n l y the most semantically i m p o r t a n t elements. T h i s favours speed over "vocabulary". T h e range of distinguishable haptic icons w o u l d be reduced while the system is i n high speed mode, since the amplitude parameter a n d some waveforms w o u l d no longer be available. Such a modification, therefore, must be considered carefully w i t h respect to the desired haptic tagging scheme.  8.4  Future Work: Hardware Improvements  O f course, numerous opportunities exist for further improvement i n the prototype hardware specification. M a n y of the challenges associated w i t h b u i l d ing w o r k i n g applications using the prototype are related to the slider, w h i c h despite being a u x i l i a r y to the tactile output device has a major effect on the user experience. Increasing the slider travel (perhaps by reconsidering the m o u n t i n g location), and finding appropriately stiff springs for the hardware return-to-centre feature could improve the scrolling m o d e l . T h e first 1-D piezo tactile display, the V i r t u a l B r a i l l e Display, h a d a slider travel that was an order of magnitude longer t h a n the present m i n i a t u r e tactile display. T h i s enabled a simple position-control mode, but i m p o r t a n t l y also allowed the user to use proprioception to sense the slider p o s i t i o n w i t h out h a v i n g to m e n t a l l y integrate velocity and time. N e i t h e r of these usability 4  This idea was contributed by thesis reader Dr. Steve Wolfman.  Chapter  8. Conclusion and Future Work  196  advantages are present i n the current handheld version. F i n a l l y , presently the push to select, or "click", feature of the device is implemented as a switch. B y replacing the switch w i t h an analog force sensor such as a force sensitive resistor ( F S R ) , and perhaps reconsidering the m o u n t i n g appropriately to deliver the force linearly to the sensor, a d d i t i o n a l interaction models m a y be considered that m o d u l a t e the stimulus a m p l i t u d e based on how strongly the user is pressing down on the T D , enabling richer tactile e x p l o r a t i o n .  8.5  Conclusion  In this thesis we have described one full iteration of an interaction design and hands-on i n q u i r y process (Figure 1.2). W e began by identifying a p r o b l e m w i t h broad relevance: the usability l i m i t a t i o n s of contemporary mobile devices. U n d e r the w o r k i n g hypothesis that haptics might be a good candidate for addressing this problem, we conducted conceptual design w o r k informed by an i n q u i r y into existing usage patterns and user needs.  User require-  ments and system requirements emerged, concurrently, eventually allowing us to proceed w i t h the development of a high-fidelity, "deep" hardware prototype using a number of novel technologies, i n partnership w i t h an expert collaborator. Once some of the device's expressive capabilities were k n o w n , we proceeded w i t h the development of a high-fidelity browser a p p l i c a t i o n prototype. F i n a l l y , it was then possible to use more conventional u s a b i l i t y techniques to assess the performance characteristics of the browser. T h r o u g h the exercise of designing, b u i l d i n g , and testing the mobile haptics concept, we learned t h a t user-centred interaction design for haptic interfaces requires the use of slightly different methodologies t h a n conventional  Chapter  8. Conclusion and Future Work  user-centred interaction design for p r i m a r i l y visual interfaces.  197  F i r s t , be-  cause r i c h l y expressive haptics — especially i n the mobile d o m a i n — is s t i l l i n its infancy, the hardware used for rendering computer signals into hum a n sensory percepts is far more varied a n d weakly standardized t h a n for the v i s u a l d o m a i n . T h i s implies that it is not possible to do "pure" usercentred interaction design that is entirely system (and hardware) agnostic. Instead, a p p l i c a t i o n a n d hardware design must go hand-in-hand; however, i n our approach we have striven to follow the spirit of good user-centred design techniques b y beginning the i n q u i r y into user needs a n d a p p l i c a t i o n scenarios as early as possible, using the m i n i m u m set of assumptions about the specific hardware i m p l e m e n t a t i o n . A s design work proceeded, b o t h the target hardware profile a n d the a p p l i c a t i o n designs evolved synergistically, informed b y increasing knowledge about user needs a n d candidate hardware designs. A n o t h e r special characteristic of designing for mobile haptics is the u t i l i t y of formal perceptual characterization based o n user studies. W h i l e the sensory gamut of a visual display might be readily characterizable using a set of standardized measures a n d a simple visual inspection, haptic signals are often far more subtle, complex, a n d difficult to characterize. Therefore, when a new haptic output device is developed, it is helpful to understand its sensory gamut — or measure of expressiveness — using the techniques described i n this thesis, to inform the design of applications t h a t best take advantage of the tactile display. T h e identification a n d v a l i d a t i o n of p r a c t i c a l interaction design techniques for mobile haptics, concept designs for applications based on a m i n i m a l l y assumptive hardware profile, high-fidelity hardware prototype, highfidelity browser a p p l i c a t i o n prototype, a n d user s t u d y findings that shed  Chapter 8. Conclusion and Future Work  198  light on the characteristics of a m i n i a t u r e lateral skin-stretch h a p t i c system represent major contributions of this work. F o r future work, it is reasonable to expect t h a t another iteration of the same process, informed by the findings  from each step of the previous iteration, w i l l produce further i m -  provements i n hardware a n d a p p l i c a t i o n usability. S u c h p r e d i c t a b i l i t y enables researchers and engineers to engage i n mobile haptics development i n a planned fashion, and it is hoped that this m e t h o d w i l l increase the accessibility, usefulness, and usability, p u t t i n g this exceedingly p r o m i s i n g technology i n the hands of users.  199  Bibliography [1] T h e  adaptive  communication  environment  (ace).  http://www.cs.wustl.edu/ s c h m i d t / A C E . h t m l . [2] G i f 8 9 a  specification.  http://www.w3.org/Graphics/GIF/spec-  gif89a.txt. [3] T h e g i m p t o o l k i t (gtk) library, h t t p : / / w w w . g t k . o r g / . [4] Imagemagick l i b r a r y home page, h t t p : / / w w w . i m a g e m a g i c k . o r g / . [5] Microsoft  intellimouse  optical  specification,  http: / / w w w . m i c r o s o f t . c o m / h a r d w a r e . [6] C o r i n R . A n d e r s o n , Pedro Domingos, a n d D a n i e l S. W e l d . P e r s o n a l i z i n g web sites for mobile users. Proceedings of the 10th Conference on the  World Wide Web (WWW10), 2001. [7] T . G . A n d e r s o n . H u m a n - c o m p u t e r interface i n c l u d i n g h a p t i c a l l y controlled interactions. U n i t e d States P a t e n t 6,954,899, October 11 2005. [8] R . B o w n . M u l t i - f u n c t i o n a l vibro-acoustic device. U n i t e d States P a t e n t 6,911,901, June 28 2005. [9] A . C . B r a u n , L . B . Rosenberg, D . F . M o o r e , K . M . M a r t i n , a n d A . S . G o l d enberg.  D i r e c t i o n a l tactile feedback for haptic feedback interface de-  vices. U n i t e d States P a t e n t 6,864,877, M a r c h 8 2005.  Bibliography  200  [10] Stephen Brewster a n d L o r n a M . B r o w n .  Tactons:  structured  messages for non-visual information display. I n CRPIT  tactile  '04-' Proceed-  ings of the fifth conference on Australasian user interface, pages 15-23, D a r l i n g h u r s t , A u s t r a l i a , A u s t r a l i a , 2004. A u s t r a l i a n C o m p u t e r Society, Inc. [11] L . M . B r o w n , S. A . Brewster, a n d H . C . Purchase. A first investigation into the effectiveness of tactons. pages 167-176, 2005. [12] A . C h a n , K . M a c L e a n , a n d J . M c G r e n e r e . L e a r n i n g a n d identifying haptic icons under workload, pages 432-439, 2005. [13] A n g e l a C h a n g , Sile O ' M o d h r a i n , R o b Jacob, E r i c G u n t h e r , a n d H i r o s h i Ishii. C o m t o u c h : design of a vibrotactile c o m m u n i c a t i o n device. I n DIS '02: Proceedings of the conference on Designing interactive systems, pages 312-320, N e w Y o r k , N Y , U S A , 2002. A C M Press. [14] A n g e l a C h a n g a n d C o n o r O ' S u l l i v a n . A u d i o - h a p t i c feedback i n mobile phones. I n CHI '05: CHI '05 extended abstracts on Human factors in computing systems, pages 1264-1267, N e w Y o r k , N Y , U S A , 2005. A C M Press. [15] R . C h o l e w i a k a n d C . Sherrick. A computer-controlled m a t r i x system for presentation t o skin of complex spatiotemporal pattern. Research Methods and Instrumentation,  Behavior  13(5):667-673, 1981.  [16] Immersion C o r p o r a t i o n . V i b e t o n z system. P r o d u c t L i t e r a t u r e , M a r c h 2005. [17] Jack T i g h Dennerlein, D a v i d B . M a r t i n , a n d C h r i s t o p h e r Hasser. Forcefeedback  improves performance  for steering a n d combined  steering-  Bibliography  targeting tasks.  201  I n CHI '00: Proceedings of the SIGCHI  conference  on Human factors in computing systems, pages 423-429, N e w Y o r k , N Y , U S A , 2000. A C M Press. [18] M . E n r i q u e z a n d K . E . M a c L e a n . C o m m o n onset m a s k i n g of v i b r o t a c t i l e s t i m u l i - poster. I n Proc. World Haptics. I E E E , 2005. [19] F . G e l d a r d . Some neglected possibilities of c o m m u n i c a t i o n .  Science,  131(3413):1583-1588, 1960. [20] F . Gemperle, N . O t a , a n d D . Siewiorek. Design of a wearable tactile display, pages 5-12, 2001. [21] M a s a t a k a G o t o . Smartmusickiosk: M u s i c listening station w i t h chorussearch function. Proceedings of the 16th Annual ACM Symposium on User Interface Software and Technology (UIST 2003), pages 31-40, 2003. [22] J . D . G o u l d .  H o w to design usable systems.  Computer Interaction:  I n Readings in Human-  Toward the Year 2000, pages 93-121. 1995.  [23] V . H a y w a r d a n d M . C r u z - H e r n a n d e z . T a c t i l e display device using dist r i b u t e d lateral skin stretch. I n Proc. of the Haptic Interfaces for Virtual Environment and Teleoperator Systems Symposium  (IMECE2000),  volume D S C - 6 9 - 2 , pages 1309-1314. A S M E , 2000. [24] H . Ishii a n d B . U l l m e r .  Tangible bits:  between people, bits a n d atoms. pages 234-241, 1997.  Towards seamless  interfaces  I n A C M , editor, Proc. of CHI'97,  Bibliography  202  [25] R . S . Johansson a n d R . H . L a M o t t e . Tactile detection thresholds for a single asperity o n a n otherwise smooth surface.  Somatosensory  Re-  search, 1:21-31, 1983. [26] A l a n K a y .  Personal d y n a m i c media.  IEEE  Computer,  10(3):31-41,  1977. [27] R . L . K l a t z k y a n d S. J . L e d e r m a n .  H o w well c a n we encode spatial  layout from sparse kinesthetic contact?  I n Proceedings of the 11th  Symposium on Haptic interfaces For Virtual Environment and Teleoperator Systems (Haptics'03), page 179, W a s h i n g t o n , D C , M a r c h 2003. I E E E C o m p u t e r Society. [28] V i n c e n t Levesque a n d V i n c e n t H a y ward. E x p e r i m e n t a l evidence of lateral s k i n s t r a i n d u r i n g tactile exploration.  Proc. Eurohaptics 2003,  pages 261-275, 2003. [29] V i n c e n t  Levesque  and  Jerome  Pasquero.  Stress  library.'  http://www.cim.mcgill.ca/ haptic/. [30] V i n c e n t Levesque, Jerome Pasquero, V i n c e n t H a y w a r d , a n d M a r y s e Legault.  D i s p l a y of v i r t u a l braille dots b y lateral s k i n deformation:  feasibility study. ACM Trans. Appl. Percept., 2(2):132-149, 2005. [31] R o b e r t W . L i n d e m a n , J o h n L . Sibert, E r i c k M e n d e z - M e n d e z , Sachin P a t i l , a n d D a n i e l Phifer. Effectiveness of directional v i b r o t a c t i l e cuing on a building-clearing task. I n CHI '05: Proceedings of the  SIGCHI  conference on Human factors in computing systems, pages 271-280, N e w Y o r k , N Y , U S A , 2005. A C M Press.  Bibliography  203  [32] J u k k a L i n j a m a a n d T o p i K a a r e s o j a . N o v e l , m i n i m a l i s t haptic gesture interaction for mobile devices.  I n NordiCHI  third Nordic conference on Human-computer  '04: Proceedings of the interaction,  pages 4 5 7 -  458, N e w Y o r k , N Y , U S A , 2004. A C M Press. [33] S h a n n o n L i t t l e . Theory, software, a n d psychophysical studies for the tactile handheld m i n i a t u r e b i m o d a l device. Technical R e p o r t T R - 2 0 0 5 21, U n i v e r s i t y of B r i t i s h C o l u m b i a , Department of C o m p u t e r Science, 2005. [34] Joseph L u k , Jerome Pasquero, Shannon L i t t l e , K a r o n E . M a c L e a n , V i n c e n t Levesque, a n d V i n c e n t H a y w a r d . A role for haptics i n mobile interaction: I n i t i a l design using a handheld tactile display prototype. In CHI '06: Proceedings of the SIGCHI  conference on Human factors  in computing systems, N e w Y o r k , N Y , U S A , 2006. A C M Press. [35] K . E . M a c L e a n , M . J . Shaver, and D . K . P a i . H a n d h e l d haptics: a usb m e d i a controller w i t h force sensing, pages 311-318, 2002. [36] K . E . M a c L e a n a n d M . E n r i q u e z . P e r c e p t u a l design of haptic icons. In Proc. EuroHaptics 2003. I E E E , 2003. [37] V i v e k M a h e s h w a r i a n d R a v i F . Saraf. High-resolution thin-film device to sense texture b y touch. Science, 312(5779): 1501-1504, June 9 2006. [38] M i c h a e l M c C l o s k e y .  Intuitive physics.  Scientific American,  248:122-  130, A p r i l 1983. [39] B . A . N a r d i , S. W h i t t a k e r , a n d E . B r a d n e r . Interaction a n d outeraction: instant messaging i n action. Proceedings of the 2000 ACM  Conference  .on Computer Supported Cooperative Work, pages 79-88, 2000.  Bibliography  204  [40] A n t t i O u l a s v i r t a , Sakari T a m m i n e n , V i r p i R o t o , and J a a n a K u o r e l a h t i . Interaction i n 4-second bursts:  the fragmented nature of attentional  resources i n mobile h c i . In CHI '05: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 919-928, N e w Y o r k , N Y , U S A , 2005. A C M Press. [41] J . Pasquero and V . H a y w a r d .  STRESS:  A p r a c t i c a l tactile display system  w i t h one m i l l i m e t e r spatial resolution and 700 H z refresh rate. In Proc. Eurohaptics 2003, pages 94-110, 2003. [42] J . Pasquero, J . L u k , S. L i t t l e , and K . E M a c L e a n . P e r c e p t u a l analysis of haptic icons: an investigation into the v a l i d i t y of cluster sorted mds. In Proc. Of Symp. On Haptic Interfaces for Virtual Environment Teleoperator Systems (IEEE-VR).  and  I E E E , 2006.  [43] Jerome Pasquero, Joseph L u k , V i n c e n t Levesque, Q i W a n g , V i n c e n t H a y w a r d , and K a r o n M a c L e a n .  D i s t r i b u t e d tactile display for rich  haptic interaction w i t h a handheld infomation display. IEEE Multimedia,  To appear in  2006.  [44] C A . Perez, A . J . Santibafiez, C A . H o l z m a n n , P . A . Estevez, and C M . H e l d . Power requirements for v i b r o t a c t i l e piezo-electric and electromechanical transducers. Medical and Biological Engineering and Computing, 41:718-726, November 2003. [45] Ivan P o u p y r e v and Shigeaki M a r u y a m a .  T a c t i l e interfaces for s m a l l  touch screens. In UIST '03: Proceedings of the 16th annual ACM symposium on User interface software and technology, pages 217-220, N e w Y o r k , N Y , U S A , 2003. A C M Press.  Bibliography  205  [46] Ivan P o u p y r e v , Shigeaki M a r u y a m a , a n d J u n R e k i m o t o .  Ambient  touch: designing tactile interfaces for handheld devices. In UIST '02: Proceedings of the 15th annual ACM symposium on User interface software and technology, pages 51-60, N e w Y o r k , N Y , U S A , 2002. A C M Press. [47] V i r p i R o t o a n d A n t t i O u l a s v i r t a . N e e d for non-visual feedback w i t h long response times i n mobile h c i . In WWW and posters of the 14-th international  '05: Special interest tracks  conference on World Wide Web,  pages 775-781, N e w Y o r k , N Y , U S A , 2005. A C M Press. [48] K . B . Shimoga. F i n g e r force a n d touch feedback issues i n dexterous telemanipulation. pages 159-178, 1992. [49] D a v i d L . Strayer, F r a n k A . Drews, a n d D e n n i s J . C r o u c h . A comparison of the cell phone driver a n d the d r u n k driver.  Human Factors:  The  Journal of the Human Factors and Ergonomics Society, 48(2):381-391, S u m m e r 2006. [50] H o n g Z . T a n , R o b Gray, J . J a y Y o u n g , a n d R y a n T r a y l o r . A haptic back display for attentional a n d directional cueing.  Haptics-e:  The  Electronic Journal of Haptics Research, 3(1):20, June 11 2003. [51] C . R . Wagner, S . J . L e d e r m a n , a n d R . D . Howe. Design a n d performance of a tactile display using rc servomotors. Haptics-e Electronic Journal of Haptics Research (www.haptics-e.org), 3(4), September 1999. [52] L . W a r d . M u l t i d i m e n s i o n a l scaling of the molar physical environment. Multivariate  Behavioral Research, 12:23-42, 1977.  Bibliography [53] Scott Weiss.  206  A n alternative business model for addressing usability:  Subscription research for the telecom industry. ACM interactions, pages 62-63, J u l y - A u g u s t 2005.  207  Appendix A  Browser User Evaluation Documents T h e following documents related to the browser user evaluation are included i n this A p p e n d i x : 1. P a r t i c i p a n t D e m o g r a p h i c D a t a C o l l e c t i o n F o r m and Q u a l i t a t i v e A s sessment B a t t e r y 2. V e r b a l P r o t o c o l ( E x p e r i m e n t  Instructions)  3. P o s t - S t u d y Interview D a t a C o l l e c t i o n F o r m 4. Task Inventory 5. E x a m p l e s of Test W e b Pages  203 UBC  Physical  and multimodal  Study Questionnaire  user interfaces  Form  - usability  and  psychophysics  UBC Behavioural Research Ethics Approval #B01-047  Instructions Please respond to all of the items listed below. If you have any questions, please ask the facilitator.  Part 1 : Background 1.  2.  years  Age  Sex:  female  or  male  2. First languaqe(s):  Part 2 : Survey 1. a) Did you play a musical instrument as a child? b) If 'yes', for how many years did you play?  yes  or  no  (please circle one)  years  c) If 'yes', which instrument(s) did you play? 2. Do you regularly use any of the following: (check all that apply) Q  Mobile phone  G  P D A (personal digital assistant), including BlackBerry  •  G a m e controller with vibration or force feedback (PlayStation Dual Shock, etc.)  Q  Mouse with a scroll wheel  3. Please respond to the following statements by placing an X in the box according to the following scale: Strongly Agree  Agree  Neutral  Disagree  Strongly Disagree  1.  I am a multitasker.  2.  Mobile phones are useful for more than just talking.  3.  Sometimes it's necessary to use a mobile phone while driving.  4.  I'm good at working with my hands.  5.  I prefer to work on one thing at a time.  6.  The vibration function on a mobile phone is annoying.  7.  When I am shopping, I often pick up objects to touch them even though I know I won't buy them.  Thank you! Please inform the facilitator when you are done.  UBC  Physical and multimodal user interfaces - usability and psychophysics Study Questionnaire  UBC Behavioural Research Ethics Approval #B01-0470\  Instructions Please respond to all of the items listed below. If you have any questions, please ask the facilitator. Part 3 - Post-Study Survey 4. Please respond to the following statements by placing an X in the box according to the following scale: Strongly Agree  8.  Agree  Neutral  Disagree  Strongly Disagree  I was able to keep my eyes on the screen of the handheld device during the tasks  9. The task of finding information on the page was challenging. 10. The task of pushing the pedals in response to the instructions was challenging. 11. The tactile feedback was helpful in locating items on the page 12. I would pay 5% more for a mobile phone that had this kind of tactile display. 13. The tactile feedback I experienced today was pleasant 14. Overall, I found the device easy to use  5. The facilitator will now ask you some brief questions.  6. Additional comments:  Thank you for participating in this study!  Experiment instructions  Thank you for participating in this experiment today. Please review these instructions and if you have any questions, ask the facilitator. If you feel uncomfortable, you may quit the experiment at any time without penalty. Holding the device  Hold the device in your left hand with your thumb resting on the side-mounted tactile display (TD). You can slide the TD up and down, and you will feel some spring resistance. Do not rub, pick, or otherwise abrade the surface of the TD. For best results, try to keep the TD centred under your thumb as you operate the device. Tasks  First, you will have a chance to "practice" to get familiar with the controls and tasks. Then you will do a series of tasks that involve finding some information on a web page. Every sixth task, you may be asked to change devices - for example, using the mouse instead of the TD. If you are using the TD, you might feel tactile feedback, or you might not. There will be a one-minute break after each block of six tasks. In addition to browsing web pages, you may be asked to do another task simultaneously. Every 7 seconds, an instruction will appear on the computer screen. You must press the pedals according to the instructions within 7 seconds, before the next instruction appears. Operating the browser  Depending on the task, you can move the cursor using either the TD or the mouse scroll wheel. Highlight the link you wish to select, and select it using the E N T E R key. Timing  Timing information is being collected, so it is important that you do all the tasks as fast as you can. The procedure is as follows: 1. The facilitator will explain the next task. 2. When you are ready to begin the task, press the E N T E R key. Timing starts from this point. 3. When you find the information requested, press the SPACE bar as soon as possible. This stops the timer. 4. Inform the facilitator of the answer. Again, it is important that you press the SPACE bar as soon as you see the answer. Do this before you tell the answer to the facilitator. If you make a mistake  On the bottom of each web page, there is a link labeled "Back" which will take you to the previous page. If you make a mistake, please continue trying to complete the task as quickly as possible. When you are finished, please inform the facilitator that a mistake was made during the task.  1\ Physical  and multimodal  Stock Inquiry  user interfaces - usability and  psychophysics  Questions  A b o u t this d o c u m e n t The experimenter will converse with the participant using the following 'stock' questions to guide the inquiry.  1.  During the experiment, you were asked to find information using either the mouse, or the handheld device with or without tactile feedback. Did you feel that your performance was better in any of the three conditions? If so, why?  2. Aside from the task of pushing the pedals and the general experimental setup, do you have any suggestions for improvement, for example in the way the handheld device is used, or the way tactile feedback is given?  Ref. #: cslMnterview 2004 07 05 v1 O.doc  Appendix  A.l A l  Task  A. Browser User Evaluation  Documents  212  Inventory  W h a t is the weather i n L o n d o n today?  A 2 W h a t w i l l the weather be like i n L o n d o n tomorrow? A 3 W h a t w i l l the weather be like i n L o n d o n the day after tomorrow? A 4 W h a t is the weather i n P a r i s today? A5  W h a t w i l l the weather be like i n P a r i s tomorrow?  A 6 W h a t w i l l the weather be like i n P a r i s the day after tomorrow? A 7 W h a t is the weather i n T o k y o today? A 8 W h a t w i l l the weather be like i n T o k y o tomorrow? A 9 W h a t w i l l the weather be like i n T o k y o the day after tomorrow? A 1 0 W h a t is the weather i n H o n g K o n g today? A l l  W h a t w i l l the weather be like i n H o n g K o n g tomorrow?  A 1 2 W h a t w i l l the weather be like i n H o n g K o n g the day after tomorrow? A 1 3 W h a t is the weather i n S a n Francisco today? A 1 4 W h a t w i l l the weather be like i n San Francisco tomorrow? A 1 5 W h a t w i l l the weather be like i n San Francisco the day after tomorrow? A 1 6 W h a t is the weather i n Toronto today? A 1 7 W h a t w i l l the weather be like i n Toronto tomorrow? A 1 8 W h a t w i l l the weather be like i n Toronto the day after tomorrow?  Appendix Bl  A. Browser User Evaluation  Documents  213  If y o u take the 99 B - l i n e from U B C at 1pm, when w i l l you arrive at B r o a d w a y station?  B 2 If y o u take the 99 B - l i n e from U B C at 1pm, when w i l l you arrive at Granville? B 3 If y o u take the 99 B - l i n e from U B C at 1pm, when w i l l you arrive at Main? B 4 If y o u take the 99 B - l i n e from U B C at 9 p m , when w i l l y o u arrive at B r o a d w a y station? B 5 If y o u take the 99 B - l i n e from U B C at 9 p m , when w i l l you arrive at Granville? B 6 If y o u take the 99 B - l i n e from U B C at 9 p m , when w i l l you arrive at Main? B 7 If y o u take the #44 bus from U B C at 10am, when w i l l you arrive at Macdonald? B 8 If y o u take the #44 bus from U B C at 10am, when w i l l y o u arrive at Davie? B 9 If y o u take the #44 bus from U B C at 10am, when w i l l y o u arrive at Waterfront BIO  station?  If y o u take the #44 bus from U B C at 5pm, when w i l l you arrive at Macdonald?  Bll  If y o u take the #44 bus from U B C at 5pm, when w i l l y o u arrive at Davie?  Appendix  A. Browser User Evaluation Documents  214  B 1 2 If you take the #44 bus from U B C at 5pm, when w i l l y o u arrive at Waterfront station? CI  W h e n is the movie " L ' E n f a n t " p l a y i n g at the R i d g e T h e a t r e ?  C 2 W h e n is the movie " L ' E n f a n t " playing at F i f t h A v e n u e Cinemas? C 3 W h e n is the movie " L ' E n f a n t " playing at E m p i r e Oakridge? C 4 W h a t r a t i n g d i d the movie " L ' E n f a n t " receive i n its review i n the L o s Angeles T i m e s ? C 5 W h a t rating d i d the movie " L ' E n f a n t " receive i n its review i n the N e w Y o r k Post? C 6 W h e n is the movie "The P r o m i s e " p l a y i n g at the R i d g e T h e a t r e ? C 7 W h e n is the movie " T h e P r o m i s e " p l a y i n g at the Pacific Cinematheque? C 8 W h e n is the movie "The P r o m i s e " p l a y i n g at the S i l v e r c i t y R i v e r p o r t ? C 9 W h a t r a t i n g d i d the movie "The P r o m i s e " receive i n its review i n the G l o b e and M a i l ? C I O W h a t r a t i n g d i d the movie " T h e P r o m i s e " receive i n its review i n the W a s h i n g t o n Post?  2iS —i TRANSIT INFORMATION  orth America O  What's N e w  NORTH AMERICA  Weather  WEATHER  BUSES AT UBC  Transit  *t  Boston Calgary Chicago Denver Edmonton Los Angeles Montreal Ottawa San Francisco Seattle Toronto Vancouver W a s h i n g t o n , D.C. Winnipeg  1© M o v i e s IFinance |§) S p o r t s News  H  • News • Planned Maintenance • Company Information  i 1 Manage Account Browser-content version 29-mar-o6 Terms of Service Privacy Statement  Weather  Back to Weather Back to Home  >> More...  Weather - Travel Forecast SAN  017 025 041 043 044 049 UBC  - Qak/Downtown/UBC - Brentwood Stn/UBC - Joyce Stn/Crown/UBC - Joyce Stn/UBC - UBC/Downtown - Metrotown Stn/Dunbar Loop/  099 - Broadway Stn/UBC (B-Line) 258 - UBC/West Vancouver 480 - UBC/Richmond Centre N17 - Downtown/UBC Niqhtbus  Back to Home  S P R I N G OUTLOOK Canadians eagerly anticipate spring each year. Find out how soon you can shed the winter layers and expect the warmer weather to arrive.  004 - Phibbs Exch/Powell/Downtown/ UBC 009 - Bdrv/Bwav Stn/Gran/Alma/ UBC  FRANCISCO,  CALIFORNIA  Transit - 4 4 Downtown DEPARTURE TIME 9:00 am 10:00 am  LOCAL W E A T H E R Vancouver [Change]  11:00 am  12°C 12°C 11°C Today  Tomorrow  Tuesday  > > Extended Forecast  12:00 pm 1:00 p m 2:00 pm  13°C 15°C 10°C Today  Tomorrow  Tuesday  T R A V E L FORECAST • • • • • •  North America Europe Asia Africa South America Antarctica  Back to Home  Back to North America Back to Weather  3:00 pm  Back to Home  4:00 pm 5:00 pm 6:00 pm 7:00 pm 8:00 pm 9:00 pm  Back to Transit Back to Home  Movies - L'Enfant  Transit - 44 Stops 44  UBC/DOWNTOWN  News  Vi  Trailers  ^1  ,'"'.!  U B C B U S L O O P BAY 7  5:04  p  FEATURED FILMS  ALMA ST  5:18  •1*193  p  MACDONALD ST  5:22  p  VINE ST  5:24  Basic Instinct 2  p  BURRARD ST / 3RD AV  5:26  if,  p  PACIFIC S T  5:30  p  - :  DAVIE ST  5:31  •**'  p  11  ROBSON ST  5:33  p  PENDER ST  5:34  L'Enfant  p  WATERFRONT STATION  5:35  p  Back  to 44  Back  to  Transit  Back  to  Home  NOW PLAYING  UBC/Downtown  Get S h o w t i m e s V i e w Trailer ygK:.l .1 ||  Idlewild  L'ENFANT  iiitiiKin  i"  iiiii  RIDGE T H E A T R E  8:00,  FIFTH A V E N U E C I N E M A S  5:00,  7:30  'PKflJP' PROMISE ii«iiii,ii..inii  EMPIRE O A K R I D G E  12:30, 6:00,  >>  MORE  Back  to  L'Enfant  Back  to  Movies  Back  to  Home  8:00  THEATRES  i, i.  Reviews Foreign, D r a m a R a t i n g : N o t yet r a t e d Jean-Pierre, Luc Dardenne (dir.) Jeremie Renier Deborah Francois  I 1:30, 4:30, 10:30  D i s p o s s e s s e d twenty-year o l d B r u n o ( J e r e m i e R e n i e r ) lives w i t h his eighteen-year-old girlfriend Sonia (Deborah Francois) i n Seraing, an eastern B e l g i a n steel t o w n . T h e y l i v e off Sonia's u n e m p l o y m e n t benefits a l o n g w i t h the p a n h a n d l i n g a n d petty thefts c o m m i t t e d b y B r u n o a n d h i s gang. T h e i r lives change forever w h e n S o n i a gives b i r t h to t h e i r child, J i m m y .  The Promise  Back  to  Movies  Back  to  Home  217  Appendix B  Browser User Study Supplemental Data T h i s appendix contains supplemental d a t a charts related to the statistical analysis of t h e browser user study. T h e following are i n c l u d e d :  •  V a r i a n c e i n f o r m a t i o n for the i n d i v i d u a l participant analysis, i n c l u d i n g charts a n d s t a n d a r d d e v i a t i o n tables for b o t h t h e m e a s u r e d a n d normalized data.  N o r m a l i z e d means are also p r o v i d e d .  T h e means for  the m e a s u r e d d a t a were r e p o r t e d i n T a b l e 7.3.  •  D a t a tables for task difficulty analysis, i n c l u d i n g scores used for t h e normalization of measured task times.  Appendix  B. Browser User Study Supplemental Data  218  Normalized Task Time v s Condition Participant 10 16 9 13 11 6 5 7 12 8 14 4 15 ALL  N 51 50 53 48 46 53 54 49 49 47 46 49 50 645  allcond slidermouse slider+ 12280 10307 12644 14027 12316 8788 14144 14124 12442 8480 14918 14016 12727 8434 14738 15008 14653 11130 15969 17268 14817 10341 17252 16994 15270 12683 16499 16628 15722 9817 17858 18795 16390 13050 18126 18181 16592 12205 18689 18596 17270 11912 20302 19975 17523 15036 19414 18275 20170 15933 21820 22142 15211 11354 17056 17247  Appendix  B. Browser User Study Supplemental Data  219  Normalized Task Time By Participant And Condition  10  16  9  13  11  6  S  7  12  8  14  4  15  Participant  Std.Dev. N o r m . Task Time v s C o n d . Participant 10 16 9 13 11 6 5 7 12 8 14 4 15 ALL  N  allcond 51 50 53 48 46 53 54 49 49 47 46 49 50 645  slider-  mouse 2987 3414 5329 3709 4302 4505 3195 4900 3827 5183 5450 3609 4311 4822  3218 2879 2336 1772 4266 3527 2798 2983 3209 3920 4141 4184 3428 4822  slider+ 1821 1595 7056 1726 3201 3147 2807 2797 2190 5397 3968 2721 3704 4358  2451 2232 2533 2731 2564 2855 2344 2977 3121 2793 3361 2041 2880 3573  Appendix  B. Browser User Study Supplemental Data  220  Measured Task Time By Participant And Condition 30000 T ~  -  ;  -  ™.,.,.™  „  „.,  25000  10  16  9  13  11  6  5  7  12  8  14  4  15  ALL  Std.Dev. Task T i m e v s Condition Participant 10 16 9 13 11 6 5 7 12 8 14 4 15 ALL  N  allcond 51 50 53 48 46 53 54 49 49 47 46 49 50 645  mouse 4032 4152 6388 4440 5095 5265 3726 5734 4533 6386 6058 4504 5431 6692  slider+  slider3445 3058 2907 2059 4452 3712 2990 3148 4516 4386 4490 4872 4849 4489  3238 3064 8625 2772 3671 4109 3021 3708 2867 7791 4476 3596 4622 5298  3966 3096 3799 3979 4445 4063 3530 4349 3528 2825 4632 3850 4339 4566  Task Difficulty Analysis Group  Task  all  cond  mouse  slider-  slider+  A  0  14105  7642  14609  14723  A  1  15130  8460  17460  16361  A  2  11290  9213  11765  14169  A  3  16020  5421  15918  17607  A  4  13828  9729  14966  16031  A  5  14290  12400  15160  15069  A  6  13457  10860  19570  14067  A  7  14585  11987  17347  17074  A  8  12873  10269  12430  17764  A  9  13051  6062  13583  14140  A  10  12394  9419  11924  14462  A  11  13425  11441  13747  14591  A  12  13929  9124  17414  14590  A  13  11946  9096  A  14  10371  8593  15921  #DIV/0!  12077  A  15  12541  7084  15892  15081  16  11969  10274  13087  15378  A  17  17094  15171  19499  B  0  16751  11788  19046  18657  B  1  16523  15389  16333  20368  B  2  17983  14212  18840  16508  B  3  19974  15766  21093  21317  B  4  17419  15240  17916  19023  B  5  18478  17095  21160  18541  B  6  15074  11025  17238  14395  B  7  14660  11789  17014  16918  B  8  14319  8909  17161  17834  B  9  16058  14538  16979  17064  B  10  17933  14586  21638  18546  B  11  15894  11692  C  0  16145  8927  18562  19743  C  1  16862  12564  19454  19554 22010  c c c c c c c c  #DIV/0!  + + •+ + +  15367  A  #DIV/0!  faster  + + + + + + + +  21296  2  18329  13415  19318  3  17694  13788  18568  19956  4  20835  16939  20161  24032  5  17623  13707  19154  20566  6  17879  11938  24994  23064  7  20011  14478  20831  23391  8  18112  9892  26462  21334 23303  9  20869  14363  24941  ALL  ALL  15287  12056  18245  18499  A  ALL  13479  9753  15171  15472  B  ALL  16790  13475  18495  18354  C  ALL  18S08  12897  21010  21846  + + + + -  all c o n d  (N) m o u s e 12  (N)  slider-  slider+  (N)  DIFFICULTY  NORMSCORE  8  0.923  1.084  7  3  0.990  2  4  0.739  1.354  5  7  1.048  0.954  (N)  1  3  13  3  12  6  13  1  1.010  11  3  5  3  0.905  1.105  13  4  6  3  0.935  1.070  15  8  3  4  0.880  1.136  10  5  2  3  0.954  1.048  14  7  3  4  0.842  1.188  11  1  7  3  0.854  1.171  11  3  3  5  0.811  1.233  11  3  4  4  0.878  1.139  16  4  4  8  0.911  1.097  11  6  0  5  0.781  1.280  10  6  1  3  0.678  1.474  11  4  5  2  0.820  1.219  11  6  3  2  0.783  1.277  0  5  4  1.118  0.894  17  5  5  7  1.096  0.913  19  9  7  3  1.081  0.925  9  19  1  13  5  1.176  0.850  22  5  8  9  1.307  0.765  18  5  9  4  1.139  0.878  13  6  3  4  1.209  0.827  22  4  10  8  0.986  1.014  18  8  4  6  0.959  1.043  16  6  4  6  0.937  1.068  18  7  5  6  1.050  0.952  20  7  5  8  1.173  0.852  16  9  0  7  1.040  0.962  17  5  6  6  1.0S6  0.947  21  8  6  7  1.103  0.907  20  7  5  8  1.199  0.834  21  5  12  4  1.157  0.864  23  6  8  9  1.363  0.734  22  8  7  7  1.153  0.867  21  11  7  3  1.170  0.855  23  7  6  10  1.309  0.764  24  9  5  10  1.185  0.844  21  7  7  7  1.365  0.733  645  216  210  219  1.000  1.000  214  71  68  75  0.882  1.134  218  72  73  73  1.098  0.911  213  73  69  71  1.211  0.826  222  Appendix C  gif2hapticon Code // // // //  GIF2HAPTICON U t i l i t y for c o n v e r t i n g ( o p t i o n a l l y , a n i m a t e d ) f o r m a t for the THMB d e v i c e ( c ) 2006, J o s e p h Luk  //  C o m p i l e command: gec - I S H O M E / ImagcMagick - 6 .1.9 - L S H O M E / I m a g c M a g i c k - 6 . 1 . 9 / magick / . 1 i b s —IMagick g i f 2 h a p t i c o n . c —o g i f 2 h a p t i c o n e n s u r e L D _ L I B R A R Y _ P A T H = / V o l u m e s / U s r / admin /Image Magick — 6 . 1 . 9 / l t d l / - l i b s : / Volumes / Usr / admin / ImagcMagick — 6 . 1 . 9 / m a g i c k / . l i b s  //  GIF f i l e s  to  haptic  i c o n XML  # i n c l u d c < u n i s t d . h> #i n c1u d e < c t y p c . h > ^include <stdio.h> #includc <stdlib.h> #i n e l u d e < s t r i n g . h> ^ i n c l u d e < a s s c r t . h> #includc <magick/api.h> bool bool  int {  o u t p u t T act i l c M o vie ( F I L E * f i l c , Image * i m a g e , u n s i g n e d c h a r * p i x c l s , i n t numframes ) ; o u t p u t H a p t i c I co n ( F I L E * f i l c , Image * i m a g c , u n s i g n e d c h a r * p i x c l s , i n t numframes ) ; main ( i n t  argc , c h a r *  argv []  )  / / ImagcMagick s t u f f Ex c c p t i o n I n f o exception ; Image *imagc ; Imagelnfo * i m a g c . i n f o ; unsigned char * p i x c l s ; int x , y , i ; FILE * f i l e ; u n s i g n e d l o n g numframes; unsigned long f r a m c s i z c ;  enum o u t p u t . m o d e s { t a c t i l c m o v i c , h a p t i c i c o n i n t ch ; char * o u t f i l c ; char d e f a u l t o u t f i l e [ ] = "out.xml"; output.modc = h a p t i c i c o n ; if {  (argc  > 1 &&; ! s t r c m p  ( argv [1] ,"  }  output-mode;  help") )  p r i n t f ( " U s a g e : g i f 2 h a p t i c o n < o u t p u t modc> < a n i m a t c d G I F f i l c n a m c > <XML o u t p u t f i l e n a m e >\n" ) ; p r i n t f ( " \ t < o u t p u t modc> o p t i o n s can b c : \ n " ) ; p r i n t f ( " \ t \ t —t Output t a c t i l e movic\n"); p r i n t f ( " \ t \ t - i (lowercase letter i ) O u t p u t h a p t i c i c o n (DEFAULT) \ n" ) ; printf ("\tdcfaults to < i n . g i f > and < o u t . x m l > i f no arguments a r c g i v e n . \ n " ) ; p r i n t f ( " \ t N o t e : a l l frames must be the same s i z e (no c r a z y o p t i m i z e d G I F s ) . \  -");  printf  ( " \ t l f you have p r o b l e m s , go to ImageReady A n i m a t i o n . . . and d e s e l e c t e v e r y t h i n g , \ n" ) ; ex i t (1 ) ; }  while  ((ch = getopt  (argc,  argv,  "ti"))  != —1)  animation  p a l e t t e —>Opt im ize  Appendix  C. gif2hapticon  Code  switch (ch) { case ' t ' : output-mode = t a c t i 1 e m o v i c ; break ; case ' i ' : output.modc = h a p t i c i c o n ; break ; case '? ' : if ( i s p r i n t (optopt)) fprintf ( s t d c r r , " Unknown o p t i o n ' — % c ' . \ n " , else fprintf (stdcrr , "Unknown o p t i o n c h a r a c t e r '\\x%x ' . \ n " , optopt); return 1 ; default : a b o r t () ;  223  optopt);  } a r g c —= o p t i n d ; a r g v += o p t i n d ;  //  Load the image InitializcMagick(*argv); GctExceptionInfo(&exccption ) ; i m a g e _ i n f o = C l o n d m a g c I n f o (( I mage In fo * )NULL) ; i f ( a r g c >= 1) ( v o i d ) C o p y M a g i c k S t r i n g ( i m a g c . i n f o —>f i len am c , ar g v [0] , M a x T e x t E x t c n t ) ; else ( v o i d ) C o p y M a g i c k S t r i n g ( i m a g c . i n f o —>f i l e n am c ," in . g i f " . M a x T e x t E x t c n t ) ; i magc=Rcad I mage ( i m a g e - i n f o , ^ e x c e p t i o n ) ; i f ( e x ce p t io n . s c v c r i t y != U n def i ncd Ex ccp t io n ) CatchExccption (^exception ) ; i f (image = (Image *) NULL) { fprintf ( s t d c r r , " C o u l d n ' t open the image f i l e . P l e a s e e n s u r e you have an i mage named '%s ' i n the c u r r e n t d i r e c t o r y . \ n " , i m a g c . i n f o —>f i 1 c n a m c ) cx it ( 1 ) ; } / / D e a l w i t h the a n i m a t e d GIF numframcs = G c 11 m age L i s t L e n gt h ( i m age ) ; f r a m c s i z c = (image—>rows ) * (i mage —>co 1 u m n s ) * s i z c o f ( u n s i g n e d if  (!( p i x e l s = ( u n s i g n e d unsigned int))))  char  int ) ;  *) mal loc (( image—>rows )*( image—>colu m ns )* s i z c o f (  { fprintf ( s t d c r r , "Couldn't columns , image—>rows ) ; exit(1);  allocate  image  memory %u by %u . \ n " , i mage—>  } / / Done l o a d i n g i m a g e . p r i n t f ( " L o a d e d image %s (%u x %u) p i x e l s by %u f r a m e s . \ n " , f i l e n a m e , image—>columns , image—>rows , numframcs) ; / / Open o u t p u t file i f ( a r g c == 2 ) o u t f i l e = argv [ 1 ] ; e l s e o u t f i l e = d c f a u 11 o u t f i 1 c ; f i l e = fopen ( o u t f i l e , "w"); i f (! f i l e ) { fprintf ( s t d c r r , " C o u l d n ' t open o u t p u t exit (1); } / / Dump the X M L switch (output-mode) {  file  %s  for  image.info- >  writing.\n",  outfile);  Appendix C. gif2hapticon  Code  224  case t a c t i l e m o v i e : p r i n t f (" E x p o r t i n g to f i l e _\"%s\" t a c t i l e movie f o r m a t . . . \ n" o u t fi 1 c ) ; o u t p u t T a c t i l c M o v i c ( f i l e , image , i x e l s , numframes) ; p r i n t f ( " Done . \ n " ) ; break ; case h a p t i c i c o n : p r i n t f ( " E x p o r t i n g to f i l e \"%s\" , h a p t i c i c o n f o r m a t . • \ n " o u t fi 1 c ) ; o u t p u t H a p t i d c o n ( f i l e , image , p i x e 1 s , numframes ) ; p r i n t f ( " Done . \ n " ) ; break ; default : abort ( ) ;  fclosc (file); free (pixels); Dcstroylmagelnfo ( imagc.info ) ; DcstroyExccptionlnfof&cxception ) ; Destroy Magick ( ) ; return 1 ;  bool {  outputTactilcMovic numframcs )  assert assert assert  (FILE  * f i l c , Image  *imagc,  unsigned  char  *pixcls ,  int  (file); (image) ; (pixels);  Image * w o r k i n g image ; Exccptionlnfo exception ; float s c a l c d p i x c l v a l u c ; fprintf ( f i l e , "<?xml v e r s i o n = \ " 1.0\" e n c od i n g = \ " U T F - 8 \ " ? > \ n " ) ; fprintf ( f i l e , " < IDOCTYPE T a c t i l c M o v i c SYSTEM \ " T ac t i 1 e M o v i c - 0 . 2 . d t d \ " > \ n " ) ; fprintf ( f i l e , " < T a c t i 1 c M o v i c V e r s i o n = \ " 0 . 2 \ " F r a'meS izc Y=\"%u \ " F ramcS izcX = \"%u \ " NhFrarncs=\"%u\">\n" , image—>rows , i m age— >co 1 u m ns , numframes) ; for {  (int  i = 0 ; i < n urn fr amcs; iH—\-)  fprintf  (file  , "  <Frame Index=\"%u\" >\n" , i ) ;  wor k i n g i m a g c = G c t I m a g c F r o m L i s t ( i m a g e , if  (!(ExportImagcPixcIs  i);  //  get  the  ith  image  ( w o r k i n g i m a g e , 0 , 0 , image—>columns , image— >rows , "I" , C h a r P i x c l , ( v o i d * ) p i x c l s , ^ e x c e p t i o n ) ) )  { fprintf (stdcrr, oxi t (1) ;  "Couldn't  acquire  image  pixels  for  frame %d . \ n" , i ) ;  } for {  (int  y=0;  fprintf for {  y < w o r k i n g image —> rows;  (file  (int  , "  x=0;  y+ + )  <Row r =\"%u\" > " ,y ) ;  x<workingimagc—>columns ;  x+4-)  s c a l c d p i x c l v a l u c = (*( p i x e l s + y * w o r k i n g i m a g c —> columns + x f p r i n t f ( f i l e , "%1.3f", s c a l c d p i x c l v a l u c ) ; if ( x < (workingimage—>columns — 1)) f p r i n t f ( f i l e ," " ) ; fprintf  (file,  "</Row>\n");  } fprintf  (file  , "  </Framc>\n");  } fprintf  (file,  " < / T a c t i l c M o v i e >\n" ) ;  ))  /  255.0;  Appendix  bool  outputHapticIcon (FILE numframes )  C. gif2hapticon  *filc  , Image  * image,  225  Code  unsigned  char  *pixels ,  int  { assert assert assert  (file); (image) ; (pixels);  E x c e p t io n Info e x c e p t i o n ; float s c a l c d p i x c lvalue ; Image * w o r k i n g i m a g e ; if {  ( image—>columns fprintf  !=  (stderr,  1)  "Warning:  only  the  first  column  of  pixels  will  be  exported.\  ;  } fprintf fprintf fprintf if {  , "<?xml v e r s i o n = \ " 1 . 0 \ " c n co d i n g =\" U T F - 8 \ " ?> \ n " ) ; , " < IDOCTYPE H a p t i c l c o n > \ n \ n " ) ; , "<icon>\n");  (file (file (file  (numframcs  1)  //  static  if  ( ! ( E x po rt Im ageP i x c l s  {  icon  fprintf (stdcrr, cxi t ( 1 ) ;  ( i m a g e , 0 , 0 , image—>columns , image—>rows, "I", Char P i x e l , (void * ) p i x c l s , ^exception ) ) )  "Couldn't  acquire  image  pixels.\n");  } f p r i ntf f p r i n tf  (file (file  <typc>SpatialIcon</typc>\n"); < h c i g h t > % d < / h c i g h t >\n" , image->rows ) ;  i n t x = 0; for ( i n t y = 0; { fpr i n t f (fi f p r i n t f ( fi  " "  //  v a l u e to f l o a t between 0 and 1 ic = (*( p i x e l s + y* i mage—>colum ns + x )) v a l u e to 0 - > ( - 5 0 V ) , 1 - > ( + 50V) ic = s c a l c d p i x c l v a l u e * 100 — 5 0 . 0 ;  scale pi sea 1 c d p i x c 1 / / scale pi sea 1 e d p i x e 1 fpr i n t f fpr i n t f  <elcmcn t >\n " ) ; <indcx>%d</indcx >\n" , y) ;  (fi ( fi  " "  } } else { / / animated icon fprintf (file , " fprintf (file, "  < v a l u e >%.f < / v a l u e >\n" , </clcmcnt>\n");  /  255.0;  scalcdpixelvalue ) ;  <ty pc > Tern po ral Ico n < / 1 y pc >\n " ) ; < h c i g h t >%d</h c igh t >\n " , im age—>rows ) ;  for ( i n t i =0; i <numframcs ; i + + ) { fprintf (file , " <Framc>\n"); fprintf (file , " < F ramcIdx>%d</Fr ameldx >\n " , i ) ; w o r k i n g i m a g c = G e t I m a g c F r o n i L i s t (image , if {  ( !( ExportlmagcPixcIs fprintf  (stderr,  i);  //  get  the  ith  image  ( w o r k i n g i m a g e , 0 , 0 , image—>columns , image—>rows , " I " , C har P i x e l , ( v o i d * ) p i xc 1 s , ^ e x c e p t i o n ) ) )  "Couldn't  acquire  image  pixels  for  frame %d . \ n" , i ) ;  Appendix C. gif2hapticon Code  226  exit ( 1 ) ; } i n t x = 0; for ( i n t y=0; y<im age—>ro w s ; y + + ) { fprintf (file , " <clcmcnt>\n"); fprintf (file, " < i n d cx >%cl </i n dcx >\n " , y ) ; / / s c a l e p i x e l v a l u e to f l o a t between 0 and 1 s c a l c d p i x c l v a l u c = (*( p i x e l s -f- y * i inage—>co 1 umns + x )) / / s c a l e p i x e l v a l u e to 0 - > ( - 5 0 V ) , 1 - > (-f-50V) s c a l c d p i x c l v a l u c = s c a l c d p i x c l v a l u c * 100 — 5 0 . 0 ; fprintf fprintf  (file , " (file , "  }  (file,  "  </Frame >\n" ) ;  }  fprintf  (file  255.0;  < v a l u e >%. f < / v a l uc >\n " , s c a 1 e d p i x e 1 v a 1 u e ) ; < / c l c m c n t >\n" ) ;  } fprintf  /  , " < / i co n >\n " ) ;  227  Appendix D  Browser Prototype Code D.l  Tactile I / O Loop  T h e browser prototype a p p l i c a t i o n software layer for the tactile I / O loop consists of several modules, w h i c h are listed below. Input a n d output functions are handled b y the S T R e S S L i b r a r y [29] b y V i n c e n t Levesque. • D a t a U p d a t e T h r e a d - T h e i n p u t / o u t p u t loop thread. • HapticPageMap - Functions related to the haptic page m a p a n d its haptic icons. • B r o w s e r S h a r e d - Shared m e m o r y d a t a structure for inter-thread communication. • B r o w s e r X M L B i t s - Functions related to parsing the X M L haptic icon a n d page m a p files. • main - the m a i n executable.  D.l.l #ifndef #dcfine  DataUpdateThread.h DATAUPDATE_THREADJi DATAUPDATE_THREAD_H  #includc ^include #includc #includc #includc #includc #include ^include #include  <iostrcam> <fstrcam> <stdio.h> <s t d 1 i b . h> < s t r i n g . h> < d i r c n t . h> <math . h> < a s s c r t . h> < s y s / t i m c . h>  ^include  " s t r c s s d / Fir m ware Info X M L . h"  Appendix  ^include #includc #includc ^include  " s t r c s s d / D c v i c c I n f o X M L . h" " s t r c s s d / S t r e a m D c v i c c . h" "stressd /PackctLogger.h" " s t r e s s d / i n i t .h"  #include  " B r o w s c r S h a r c d . h"  /*  228  D. Browser Prototype Code  TODO  Move  file  I/O routines  into  yet  another  this thread should: — Check i f t h e r e is a new page map — If s o , c r e a t e a new i n s t a n c e of the — Handle p u t t i n g the p o s i t i o n back to  thread,  running  page map v a r i a b l e the b r o w s e r ?  at  lower  and  load  priority...  it  in  */ class DataUpdateThread { public : DataUpdateThread ( BrowscrSharcd * s , strcssd : : StreamDcvicc*  d) ;  v i r t u a l " D a t a U p d a t e T h r e a d () ; s t a t i c void * S t a r t ( v o i d *p); protected: void Start-(); private: BrowscrSharcd  * shared ;  strcssd : : StreamDcvicc* SliderSmoother  dev ;  smoother ;  }; #cndif  D.l.2 // //  DataUpdateThread.epp  D a t a U p d a t e T h r e a d . epp J o s e p h L u k , J u l y 2006.  ^include #includc  Based  on  code  by  Vincent  " D a t a U p d a t e T h r e a d . h" " B r o w s c r X M L B i t s . h"  DataUpdateThread : : DataUpdateThread ( BrowscrSharcd* s t r e s s d : : StreamDcvicc* d ) { s h a r e d = s; dev = d ; } DataUpdateThread : : ~ DataUpdateThread ( ) { } void {  * D a t a U p d a t e T h r e a d :: S t a r t ( v o i d ((DataUpdateThread r e t u r n 0;  *)  }  void {  *p)  p )-> S t a r t . ( ) ;  D a t a U p d a t e T h r e a d : : S t a r t . ()  s,  Levesque  and  Shannon  Little.  Appendix  float slidcr.pos ; i n t map.pos = 0; float norm.map.pos ; i n t 1 ast _m a p - p o s = int c l i c k . c o u n t ;  D. Browser Prototype Code  0;  s t r c s s d : : V o l t I m g img ( dev—>Nb A c t u a t o r s X () , dev—>Nb A c t u a t o r s Y () ) ; s t r c s s d ;: S t a t u s l n f o info; / / slider parameters const f l o a t s l i d c r . m i n = const f l o a t s l i d e r . m a x = p r i n t f (" S t a r t i while // //  (for  //  (1)  Data  1700; 2770;  update  //was //was  1615 2799  thread...\ n");  {  read s l i d e r dev—> R c a d S tat us ( i n f o ) ; / /  currently  takes  exactly  3 ms  calibration) p r i n t f ("%d\n" , i n f o . s l i d c r D a t a . ) ; f ( info, slider D a t a . < slidcr.pos lsc  =  slidcr.min )  0.0;  i f ( i n fo . s 1 i dc r Da t a _ >  slider.max )  slidcr.pos  =  100.0;  slidcr.pos  =  ( info , s l i d c r D a t a . — s l i d c r . m i n ) / ( s l i d e r . m a x — s l i d c r . m i n ) * 100.0;  1sc  lider.pos lidcr-pos f  = 100.0 — s l i d c r . p o s ; = smoother. SmoothSlidcrPos( s l i d c r . p o s ) ;  ( s h a r e d - > G c t P a g e M a p R c a d y () ) map.pos //  = s h a r e d —>pagcmap—>Currcn t M a pP os ( s l i d c r . p o s ) ;  Get c l i c k count c l i c k . c o u n t = info , c l i c k C o u n t . ;  / / E x p o r t d a t a to s h a r e d memory shared — > S c t C l i c k s ( c l i c k - c o u n t ) ;  // //  if {  ( map.pos  last.map.pos )  / / E x p o r t d a t a to s h a r e d memory shared—>SctMapPos ( map-pos ) ; //  w r i t e to d e v i c e p r i n t f ("Compute i m a g c \ n " ) ; shared— >pagemap—>Com put elm age (&img , p r i n t f ("Write Imagc\n"); dev—>WriteImagc(&img ) ; printf ("Done.\n" ) ;  // // //  last.map.pos  // } }  //  = map.pos;  //  } //  if  whilo(l)  ( s h a r o d - > G e t P a g e M a p R c a d y () )  cache  map.pos) ;  last  value  229  Appendix D.l.3  HapticPageMap.h  #ifndcf #define class class  HAPTICPAGEMAPJi H APTICP AG EM A P_H  HapticPageMap; Hapticlcon; / / st r e s s d stressd st r c s s d strcssd strcssd  #include #i n c1u d e #i n c1u d c #includc #includc #includc //  230  D. Browser Prototype Code  / / d e f i n e d below d e f i n e d below  / F i r m w a r c I n f o X M L . h" / D c v i c c I n f o X M L . h" / S t r e a m D c v i c c .h" / PackctLogger.h" / i n it . h "  " B r o w s c r X M L B i t s . h"  Support  functions  for  time  stuff  / / r e t u r n the time ( i n s e c o n d s ) s i n c e t i m e S t a m p , with m i c r o s e c o n d double T i m c S i n c c (const s t r u c t t i m c v a l *timcStamp, s t r u c t t i m e v a l c u r r c n t T i m c S t a m p = NULL) ;  precision *  / / r e t u r n s the d i f f e r e n c e in the t i m c s t a m p s , in m i l l i s e c o n d s (tvl—tv2) double T i m c D i f f (const s t r u c t t i m c v a l * t v l , const s t r u c t t i m c v a l * t v 2 ) ; / / stamp the time i n t o v o i d TimeStamp ( s t r u c t  a timcval struct t i m c v a l *timeStamp) ;  class HapticlconContent { public : Hapticlcon char char  * icon parent ;  *typc; *namc;  v i r t u a l bool / / W rite rtual rtual rtual  Rcndcrlcon (double *buf_ptr) = t h i s h a p t i c i c o n to the b u f f e r  0; at  b o o l A l l o c a t c D a t a ( i n t s i z e ) = 0; d o u b l e G c t D a t a ( i n t i n d e x ) = 0; bool S c t D a t a ( i n t i n d e x , double v a l u e )  the  =  zero  0;  }; class  Spatiallcon Content  :  public  HapticlconContent  { public : S p a t i al Ico n C o n tc n t ( i n t h e i g h t virtual ~SpatialIconContcnt(); double bool  = 0  );  *data;  Rcndcrlcon  (double  *buf_ptr) ;  bool A l l o c a t c D a t a ( i n t size); double GctData ( i n t index); bool S c t D a t a ( i n t i n d e x , double  value);  };  class AnimatedlconContent : public HapticlconContent { public : A n i m a t e d l c o n C o n t e n t ( i n t h e i g h t = 0, i n t frames = 1 v i r t u a l " S p a t i a l l c o n C o n t c n t () ; double  ** data ;  bool  Rcndcrlcon  bool  AllocatcData  (double (int  * buf.ptr ) ; size);  );  position  Appendix  D. Browser Prototype Code  double GctData ( i n t i n d e x ) ; // bool S e t D a t a ( i n t index , d o u b l e  231  index = frame* h e i g h t value) ;  i n t numFramcs ; int currentFramc ; struct timcval initTimcStamp; i n t fp s ; };  class  Hapticlcon  {  public : H a p t i c l c o n ( const char virtual ~HapticIcon() ;  * filename  int  at;  of  int  height;  / / Y position //  extent  HapticIconContcnt Hapticlcon  the  of  );  icon  the  icon  in Y  taxels  * content ;  *ncxt;  v i r t u a l bool / / Write  RcndcrlconToMap (HapticPageMap *map_ptr); t h i s h a p t i c i c o n to the page map "at" i t s  location  class HapticPageMap { public : HapticPageMap (const char * f i l e n a m e ) ; v i r t u a l " HapticPageMap ( ) ; int  CurrcntMapPos (const float slidcr.pos); / / update w i t h the s l i d e r p o s i t i o n and r e t u r n  float ComputcDctcntPosFromVclocity (); / / r e t u r n the new d e t e n t p o s i t i o n based  bool void  on  Computclmagc ( s t r c s s d : : V o l t l m g * o u t p u t I m g , RenderAHIcons (void) ; / / pre —render a l l i c o n s p u rposes  Hapticlcon  to  the  page map,  the  the  current  current  int  mostly  position  velocity.  map.pos) ; useful  for  testing  support  spatial  * i co n s ;  double* voltmap ; // int height; / / size  in of  v o l t s , t e m p o r a r y w h i l e we o n l y the map a r r a y ( i n t a x e l units)  private : / / v e l o c i t y model s t u f f bool s i m p l e C o n t r o l M o d c ; / / configuration simple p o s i t i o n control  parameter;  set  to T R U E  if  cnum V C o n t r o l S t a t c s { p C o n t r o l , v C o n t r o l W a i t i n g , v C o n t r o l A c t i v c , vControlFinished }; V C o n t r o l S t a t c s v C o n t r o 1St ate ; s t r u c t t i m c v a l v Co n t r ol S t a t e T i m eSt am p ; / / time e n t e r e d c u r r e n t vControlStatc / / e x c e p t i o n : always zero when pControl c u r r c n t V c 1 o c i ty ; positive float lastDctcntPos; int pOffsct ; int  //  UNITS:  struct  timcval  1 astTimcStamp ;  struct  timcval  startTimeStamp ;  map u n i t s  per  second ; up  is  you  icons  want  in  n e g a t i v e , down  is  Appendix  D. Browser Prototype  f l o a t l o o k a h c a d [8] ; # d c f i n c LOOKAHEAD-NOVALUE float lookback [ 8 ] ; int  232  Code  -1000  CropPos ( i n t i n p u t P o s ) ; // bounds of the map CropPos ( f l o a t inputPos);  helper  function  to  crop  the  position  to  the  float };  c l a s s S 1 i d c r S moot her { / / Used to reduce n o i s e and j i t t e r from the s l i d e r . / / Caches the l a s t t h r e e v a l u e s of the s l i d e r and a v e r a g e s them. // If the new s l i d e r p o s i t i o n is c l o s e to the a v e r a g e , d o n ' t u p d a t e . / / O t h e r w i s e , r e t u r n the new p o s i t i o n . // //  c u r r e n t l y o n l y set up for f l o a t slider positions p r o b a b l y s h o u l d be t c m p l a t i z c d i n the f u t u r e .  public : S l i d c r S m o o t h c r () ; v i r t u a l " S l i d c r S m o o t h c r () ; Smoot h S 1 idcr Pos  float  (float  rawpos);  private : int c a c h c . s i z c ; float threshold; float *slidcr_cache; int startidx;  #c n d i f  D . 1.4 // //  H a p t i c P a g e M a p . epp  H a p t i c Page Map s u p p o r t J u l y 2006, J o s e p h Luk  ^include  " H a p t i c P a g e M a p . h"  //  #dcfinc  //  # d c f i n c CVTEST 70 / / constant v e l o c i t y  // //  functions  DEBUG  test  flag  # d c f i n c CLICKONVCONTROL / / create " c l i c k i n g " t a c t i l e  feedback  for  SUBTAXELDOUBLE b i l i n e a r s c a l i n g to r e d u c e image s i z e by 1/2 makes an e f f e c t i v e l y 16 — t a x e l wide d i s p l a y ) # d c f i n c SUBTAXELTRIPLE / / use b i l i n e a r s c a l i n g to r e d u c e image s i z e by 1/3 makes an e f f e c t i v e l y 24—taxel wide d i s p l a y ) / /  #dcfinc / / use  velocity  Time  arithmetic  support  functions  control (slows  down  tactile  flov  (slows  down  tactile  flov  (C— s t y l e )  / / r e t u r n the time ( i n s e c o n d s ) s i n c e t i m e S t a m p , w i t h m i c r o s e c o n d / / a l s o puts the c u r r e n t time in cur rent T i m e S t a m p , i f p r o v i d e d double T i m c S i n c c (const s t r u c t t i m c v a l *timcStamp, s t r u c t t i m c v a l currcntTimcStamp ) {  precision *  233  Appendix D. Browser Prototype Code  double  timed iff;  struct  timcval  tern p_tv;  struct  timcval  *tv  = c u r r c n t T i m c S t a m p ? cu r r e n t T i m e S t a m p  : &tcmp.tv ;  g c t t i m c o f d a y ( tv , N U L L ) ; / / s u b t r a c t the c u r r e n t time from the cached time double u s c c d i f f ; t i m c d i f f = tv—>tv_scc — timeStamp—> t v _s c c ; if ( 0-0 > ( u s c c d i f f = ( d o u b l e ) (tv—>tv_uscc - t imeSt am p—> t v _u s c c )  /  1000000.0  ) )  {  usccdiff timcdiff  =  1.0  -f~  usecdiff;  ;  } timcdiff }  return  +=  usccdiff ;  (timcdiff);  / / returns milliseconds tvl—tv2 double T i m c D i f f (const s t r u c t t i m c v a l { double usccdiff; double t i m c d i f f ;  *tvl ,  const  t i m c d i f f = t v l —> t v . s c c — tv2 —> t v _ s c c ; if ( 0.0 > ( u s c c d i f f = ( d o u b l c ) ( t v l — > t v _ u s c c { u s c c d i f f = 1.0 -f u s c c d i f f ; timcdiff ; } t i m c d i f f += u s c c d i f f ; return  struct  -  timcval  tv2->tv_usec)  *tv2)  /  (timcdiff);  } / / stamp the time i n t o a t i m c v a l s t r u c t v o i d TimeStamp ( s t r u c t t i m c v a l * t i m c S t a m p ) { s t r u c t t i m c v a l tv ; g c t t i m c o f d a y ( &tv , N U L L ) ; timeStamp —>tv.sec ~ tv . t v . s c c ; timcS tamp—> t v . u s c c = tv . t v . u s c c ; } H a p t i c l c o n :: H a p t i c l c o n ( { h e i g h t = 0; at = 0; if {  const  char  * filename  — NULL )  (filename) R a w l c o n P arscr * p a r s c r = new delete parser; if {  } } else  R a w l c o n Parser  (filename ,  (content) con ten t — >name = new ch a r [ s t r 1 c n ( f i 1 e n a m c ) ] ; s t r c p y ( content— >namc , f i l e n a m e ) ; content  = NULL;  >  H a p t i c l c o n : : ~ H a p t i c l c o n () { if ( c o n t e n t ) d e l e t e content; } bool { //  H a p t i c l c o n :: R c n d c r l c o n T o M a p if  (!content)  return  false;  ( HapticPageMap *map_ptr) //  no  icon!  this) ;  1000000.0  )  )  Appendix  assert assert if  (at  int if {  (content); (map.ptr); > m a p . p t r— > h c igh t )  bottom (  = at  bottom >  return  false;  4- h e i g h t ;  ( map.ptr—>hcight ) )  # i f d e f DEBUG cout « " C r o p p i n g h a p t i c i c o n of type " « at « " height " « height « cndl; #endif / / needs c r o p p i n g bool r c t v a l ; i n t d i f f = map.ptr—>hcight — a t ; / / r e n d e r i n t o a temp b u f f e r f i r s t and then copy d o u b l e * b u f = new d o u b l e [ t h i s —>hcight ] ; if ( r e t v a l = t h i s —> co n t e n t —> Rc n de r Ico n ( b u f) ) { for (int i = 0 ; K d i f f ; i++) { / / a s s e r t , remove l a t e r i f n e c e s s a r y a s s e r t ( at + i < m a p . p t r — > h c i g h t ) ;  // «  234  D. Browser Prototype Code  map.ptr—>voltmap [ a t + i ] =  con ten t —>ty pe  to  the  page map  buf[i];  }  } else  } delete return  [] b u f ; rctval;  return  ( t h i s —>con ten t —>Rcndcr Icon (  SpatialIconContcnt::SpatialIconContent( { type  =  strcpy  new  int  ( map.ptr—>voltmap ) +  height  char[12];  (type,  "Spatiallcon");  if ( h e i g h t ) d a t a = new e l s e d a t a = NULL;  dou blc [ h c i g h t ] ;  name = NULL; i c o n p a r e n t = NULL; } SpatiallconContent ::~ SpatialIconContcnt() { # i f d e f DEBUG printf (" SpatiallconContent Dcstructor\n" ) ; #cndif delete[] data; d e l e t e [] type; d e l e t e [ ] name ; } bool {  S p a t i a l l c o n C o n t e n t :: R c n d e r l c o n  (double  assert (iconparcnt); for (int i = 0 ; i< iconparcnt->height b u f . p t r { i ] = data [ i ] ; return true;  ;  *buf_ptr)  i ++)  } bool {  S p a t i a l l c o n C o n t e n t :: A l l o c a t e D a t a d a t a = new d o u b l e return true;  (int  size)  [ size ] ;  } double S p a t i a l l c o n C o n t e n t ; : GctData ( i n t { r e t u r n d a t a [ index ] ; }  index)  at));  «  Appendix  bool {  235  D. Browser Prototype Code  S p a t i a l l c o n C o n t c n t : : SctData  (int  index,  double  value)  assert (data); data[index] = value; return true; } AnimatedlconContent:: AnimatedlconContent ( { type = new c h ar [ 1 2 ] ; strcpy (type, " Animatcdlcon " ) ;  int  height,  int  frames  )  numFramcs = frames ; if ( h e i g h t ) d a t a = new e l s e d a t a = NULL;  d o u b l e [ h e i g h t * frames ] ;  name — NULL; i c o n p a r c n t = NULL; c u r r c n t F r a m c = 0; fps = 30; / / default value g c t t i m c o f d a y (&; i n i t T i m c S t a m c ) ; } A n i m a t e d l c o n C o n t e n t : : ~ A n i m a t e d l c o n C o n t e n t () { # i f d c f DEBUG p r i n t f (" A n i m a t e d l c o n C o n t e n t D e s t r u c t o r \ n " ) ; #cndif d e l e t e [] d a t a ; d e l e t e [] type; d e l e t e [ ] name ; } bool {  AnimatedlconContent : : Rcndcrlcon assert  (double  *buf_ptr)  (iconparent);  currcntFramc  =  (fps  /  T i me S i n c c ( in i t T i me S tarn p ) ) % numFramcs;  for  ( i n t i =0; i< i co n p ar e n t —> h c i g h t ; i + + ) b u f . p t r [ i ] — data [ i-f-(height*currentFramc) ] ; return true; } bool {  AnimatedlconContent:: AllocatcData d a t a = new d o u b l e return true;  (int  size)  [size];  } double  {  AnimatedlconContent : : GctData  return  ( int  index)  data[index];  } bool {  AnimatedlconContent:: SctData  (int  index,  double  value)  assert (data); data[index] = value; return true; }  H a p t i c P a g e M a p :: H a p t i c P a g e M a p ( c o n s t { icons if { }  char  *fi 1 en am c =  NULL)  = NULL;  (filename) P a g c M ap Parser * p a r s c r delete parser;  = new  PagcMapParser  (filename,  this);  Appendix  else  h e i g h t =100;  //  D. Browser Prototype Code  default  volt  236  map h e i g h t  / / i n i t the a r r a y w i t h base v a l u e vo 11 m ap = new d o u b l e [ h e i g h t ] ; for ( i n t i = 0 ; i < h e i g h t ; i+ + ) v o l t m a p [ i ] = —50.0; 1 a s t D e t c n t P o s = 0; l a s t T i m c S t am p . t v ~s e c = 0; l a s t T i m c S t a m p . t v . u s e c = 0; c u r r c n t V c l o c i t y = 0; vControlStatc = pControl; v C o n t i o l S t a t c T i m c S t a m p . t v . s c c = 0; v C o n t r o l S t a t c T i m e S t a m p . t v . u s c c = 0; for for  (int (int  i=0; i=0;  i<8; i<8;  i++) i ++)  si m p l c C o n t r o l M ode = TimcStamp  l o o k a h c a d [ i ] = LOOKAHEADJJOVALUE; l o o k b a c k [ i ] = 0;  false;  (&st a r t T i meSt am p ) ;  } HapticPageMap : : " HapticPageMap { d e l e t e [] v o l t m a p ;  } int {  ()  Hapticlcon * ico n = icons; Hapticlcon *nexticon ; while ( i c o n ) { n o x t i c o n = icon—>ncxt; delete icon; icon = n e x t i c o n ; }  H a p t i c P £ v g c M a p : : Cu r ren t M ap Pos  ( float  slidcr.pos )  // //  r e t u r n the c u r s o r if using v e l o c i t y  p o s i t i o n in the h a p t i c map mode, may depend on p r e v i o u s  //  slidcr.pos  be e x p r e s s e d  /*  should  as  float  from  values!  0.0  to  100.0  l—i—lslidcr.pos  map s i z e range  |<  l<  >l  display  >|  >|  window  */ if {  ( false )  //  automated  movement  program  f o r demo  movie  if ( T i n c S i n c e ( & s t a r t T i m c S t a m p ) < 0.1) l a s t D e t c n t P o s ( T i m e S i n c c ( & s t a r t T i m c S t amp ) < else i 0) c u r r c n t V c l o c i t y = ( T i m c S i n c e ( & c s t a r t T i m c S t amp ) < else i 5) c u r r c n t V c l o c i t y = ( T i m e S i n c c ( & s t a r t T i m c S t else i amp ) < 0) c u r r c n t V c l o c ity = ( T i m e S i n c e ( & : s t a r t T i m c S t amp ) < else i 6) c u r r c n t V c l o c ity = ( T i m e S i n c e ( & : s t a r t T i m e S t amp ) < 0) c u r r c n t V c l o c i t y = else i ( T i m e S i n c c ( & s s t a r t T i m c S t amp ) < 6) else i c u r r c n t V c l o c ity = ( T i m e S i n c c ( & s t a r t T i m c S t amp ) < .0) C u r r c n t V c l o c i t y = else i ( T i m e S i n c c ( & s t a r t T i m c S t amp ) < .6) else i c u r r c n t V c l o c ity = ( T i m c S i n c e f & s t a r t T i m c S t amp ) < else i ) c u r r c n t V c l ocity  6 2 ;  50; 0; - 17; 0; 17; 0; -17; 0;  Appendix  else else else else else  if if if if if  D. Browser Prototype Code  (TimeSince(&startTimeStamp) (TimcSince(&startTimeStamp ) (TimcSincc(&:startTimcStamp ) (TimcSincc(&startTimeStamp ) (TimcSincc (&startTimcStamp)  < < < < <  237  11.6 ) c u r r c n t V c l o c i t y = 17; 12.0 ) c u r r c n t V c l o c i t y = 0; 13.0 ) c u r r c n t V c l o c i t y = —50; 14.9 ) c u r r c n t V c l o c i t y = - 8 0 ; 17 ) T i m e S t a m p ( & ; s t a r t T i m e S t a m p ) ;  ComputcDetent PosFrom Velocity ( ) ; return  ( C r o p P o s (( i n t ) round ( l a s t D e t e n t P o s ) ) ) ;  if ( s i m p l c C o n t r o l M o d c ) { f l o a t d isp lay _w i n d o w = 8.0 / / / d i s p l a y - w i n d o w is the / / tactile display. int  int  range = h e i g h t — 16; / / range is the p o r t i o n / / may be p l a c e d window.start = / / s t a r t index  lastDetentPos return  =  ( f1 oat ) height ; p e r c e n t a g e of the  of  the  map where  page  the  ( i n t ) ((100.0 — s l i d e r - p o s ) / 1 0 0 . 0 in the h a p t i c page map  "visible"  centre  of  the  through  the  slider  * ( float ) range);  ( float ) window.start ;  ( window.start ) ;  } else {  //  velocity  control  f l o a t n o r m S l i d c r P o s = ( 1.0 - ( s l i d c r . p o s / / / n o r m a l i z e d to between —1.0 and +1.0 float c u r r c n t A c c c l c r a t i o n ;  50.0)  );  / / A few c o n v e n i e n t macros / / # d c f i n c SIMPLEVELOCITY # d c f i n c VELOCITYATSTOPS // #if  constants d e f i n e d V E L O C I T Y ATSTOPS i n t maxP = 50; f l o a t maxV = 2 0 0 . 0 ; f l o a t - p o s C o n t r o l Z o n e = 0.75; / / Use t h i s many p e r c e n t of s l i d e r a r e a for pos c o n t r o l zone f l o a t t i m c B cforc V C o n t r o l = 0 . 0 5 ; / / Number of s e c o n d s b e f o r e velocity control activates f l o a t v C o nt r o 1 A ccc 1 cr at io n = 1; / / Number of s e c o n d s b e f o r e velocity r e a c h e s maxV a f t e r s t a r t i n g V — c o n t r o l f l o a t t imc A ft cr V C o n t r o l = 0 . 0 5 ; / / Number of s e c o n d s a f t e r d i s e n g a g i n g v e l o c i t y c o n t r o l , d u r i n g which time user i n p u t is i g n o r e d so t h a t they can r e t u r n the s l i d e r to c e n t r e # e l i f d e f i n e d SIMPLEVELOCITY i n t maxP = 10; f l o a t maxV = 3 0 0.0; f l o a t p o s C o n t r o l Z o n e = 0.20; / / Use t h i s many p e r c e n t of s l i d e r a r e a for pos c o n t r o l zone f l o a t t i m c B c f o r c V C o n t r o l = 0.0; / / Number of s e c o n d s b e f o r e velocity control activates f l o a t v C o nt r ol A c c c l c r a t i o n = 0.2; / / Number of s e c o n d s b e f o r e v e l o c i t y r e a c h e s maxV a f t e r s t a r t i n g V— c o n t r o l f l o a t t i m e A f t e r V C o n t r o l — 0.01; / / Number of s e c o n d s a f t e r d i s e n g a g i n g v e l o c i t y c o n t r o l , d u r i n g which time user i n p u t is i g n o r e d so t h a t they can r e t u r n the s l i d e r to c e n t r e ^te 1 s e i n t maxP = 5 0; f l o a t maxV = 1 0 0 . 0 ; f l o a t p o s C o n t r o l Z o n e = 0.95; / / Use t h i s many p e r c e n t of s l i d e r a r e a for pos c o n t r o l zone f l o a t t im eB cfo re V C o n t r o 1 — 0 . 0 1 ; / / Number of s e c o n d s b e f o r e velocity control activates f l o a t v C o n t r o l A c c e l e r a t i o n — 1; / / Number of s e c o n d s b e f o r e velocity r e a c h e s maxV a f t e r s t a r t i n g V— c o n t r o l f l o a t t i m e A f t e r V C o n t r o l = 0.01; / / Number of s e c o n d s a f t e r d i s e n g a g i n g v e l o c i t y c o n t r o l , d u r i n g which time user i n p u t is i g n o r e d so t h a t they can r e t u r n the s l i d e r to c e n t r e  Appendix  D. Browser Prototype Code  238  #cndif if {  ( fabs ( n o r m S l i d c r P o s )  <  posControlZonc)  / / P o s i t i o n C o n t r o l Zone c urrcntVc1ocity — 0 ; switch (vControlState) { case v C o n t r o l W a i t i n g : / / a b o r t w a i t i n g for v C o n t r o l , r e t u r n to p C o n t r o l vControlState — pControl; / / c o n t i n u e to p C o n t r o l case p C o n t r o l : p O f f s e t = ( i n t )( round ( n o r m S l i d c r P o s / p os Co n t r o 1Z o n e * ( f l o a t ) maxP ) ) ; break ; case v C o n t r o l A c t i v e : / / e n t e r v C o n t r ol F i n i s h e d ; s t a r t timer vControlState = vContro 1Finished ; TimeStamp ( & v C o n t r o l S t a t e T i m e S t a m p ) ; break ; case v C o n t r o l F i n i s h c d : / / check t i m e r ; i f p a s s e d , then e n t e r p C o n t r o l i f ( T imeS i nee (& v C o n t r o l S t a t e T i m e S t a m p ) > t i m c A f t e r V C o n t r o l ) { / / re —zero d e t e n t p o s i t i o n to a v o i d moving the highlight i n t n e w P O f f s c t = ( i n t )( round ( nor m S l i der Pos / p o s C o n t r o l Z o n e * ( f 1 o a t ) maxP ) ) ; l a s t D c t c n t P o s += p O f f s e t — n e w P O f f s e t ; pOffset = newPOffset;  } } else  //  vControlState = pControl; } break ; switch (vControlState)  { //  Velocity  Control  Zone  switch (vControlState) { case p C o n t r o l : case v C o n t r o l F i n i s h c d : / / enter waiting s t a t e , s t a r t timer vControlState = vControlWaiting; TimeStamp ( ^ v C o n t r o l S t a t e T i m e S t a m p ) ; p O f f s e t = ( i n t )( round ( n o r m S l i d e r Pos / p o s C o n t r o l Z o n e * ( f l o a t ) maxP) ) ; break ; case v C o n t r o l W a i t i n g : / / check to sec i f the t i m e r has e x c e e d e d the t i m c B c f o r c V C o n t ro1 , i f so , go to next s t a t e p O f f s e t — ( i n t )( round ( nor m S l i d c r Pos / p o s C o n t r o l Z o n e * ( f l o a t ) maxP ) ) ; if ( T i m c S i n c c ( & v C o n t r o l S t a t e T i m e S t a m p ) > t i m e B e f o r c V C o n t r o l ) { TimeStamp (&cvControlStateTimeStamp ); vControlState = vControlActive; } break ; case v C o n t r o l A c t i v e : {i f ( n o r m S l i d c r P o s > 0 ) p O f f s e t — maxP ; # i f d c f CVTEST c u r r c n t V c l o c i t y = CVTEST; ^else c u r r c n t V c l o c i t y = ( i n t } ( r o u n d (( n o r m S l i d c r P o s posControlZone) / ( 1 . 0 — posControlZone) * maxV) ) ; #endif  Appendix  else {  D. Browser Prototype Code  p O f f s c t = —maxP ; # i f d c f CVTEST c u r r c n t V c l o c i t y = -CVTEST; #c 1 s c c u r r c n t V c l o c i t y = ( i n t )( round (( nor m S l i d c r Pos pos C o n t r o l Zone ) / (1.0 — p o s C o n t ro lZone ) * maxV) ) ; #cndif  239  +  >  / / add a c c e l e r a t i o n factor current Acceleration = (TimcSincc(&vControlStatcTimcStamp)) / ( vControIAcceleration ) ; if ( c u r r e n t A c c e l e r a t i o n < 1.0) c u r r e n t V e l o c i t y = ( int ) (round ) ( ( f l o a t ) c u r r e n t V c l o c i t y * currentAccelcration) ; break ; } }  //  //  switch  velocity  (vControlStatc)  control  zone  i f ( p O f f s c t > maxP) p O f f s e t = maxP ; e l s e i f ( p O f f s c t < —maxP) p O f f s e t = —maxP; C o m p u t c D c t c n t P o s F r o m V e l o c i t y () ; # i f d c f DEBUG p r i n t f ( " C u r r c n t M a p P o s : v e l o c i t y C o n t r o l p O f f se t=%d , c u r r c n t V c l o c i t y=% d , d e t e n t Pos=%f , mapPos=%d\n" , p O f f s c t , c u r r e n t V e l o c i t y , l a s t D c t c n t P o s , C r o p P o s (( i n t ) round ( l a s t D c t c n t P o s ) - f p O f f s c t ) ) ; #cndif # i f d c f CVTEST if ( l a s t D c t c n t P o s >= #endif return  h e i g h t - 8)  lastDctcntPos  (CropPos ( ( int ) round ( l a s t D c t c n t P o s )  +  =  0.0;  pOffsct));  f l o a t HapticPageMap ; : C o m p u t c D c t c n t P o s F r o m V e l o c i t y ( ) { / / r e t u r n the c u r s o r p o s i t i o n in the h a p t i c map if ( s i m p l c C o n t r o l M o d c ) r e t u r n lastDctcntPos; else { / / compute the c u r r e n t p o s i t i o n based on f l o a t ncwDctcntPos ; double timcdiff; struct timcval tv; g e t t i m c o f d a y ( &tv ,  the  last  heading  NULL);  / / s u b t r a c t the c u r r e n t time from the cached time double usccdiff; t i m c d i f f = tv . t v . s c c — last T i m c S tamp . t v _sc c ; if ( 0.0 > ( u s c c d i f f = ( d o u b l e ) ( t v . t v . u s e c — l a s t T i m c S t a m p . t v . u s e c ) 1000000.0 ) ) { u s c c d i f f = 1.0 usccdiff; timcdiff ; } t i m c d i f f += u s c c d i f f ; lastTimcStamp , t v . s c c = t v . t v . s e c ; l a s t T i m c S t a m p . t v . u s c c = tv . t v . u s e c ; //  if  timcdiff  is  specified  /  Appendix  //  240  D. Browser Prototype Code  { / /  // // 1000000  UNTESTED ! ! ! lastTimcStamp . tv.scc lastTimcStamp . tv.uscc  +=  ( time-t ) trunc ( t i m c d i f f ) ; ( s u s c c o n d s _ t ) ( ( modf ( t i m cd i f f , NULL) ) *  );  //  if  //  {  ( lastTimcStamp . tv.uscc  // //  >=  1000000)  lastTimcStamp . tv.uscc - = last T i m c S tarn p . t v _s ec H—h;  1000000;  // ncwDctcntPos = l a s t D c t c n t P o s + t i m c d i f f ncwDctcntPos = CropPos ( ncwDetentPos ) ;  * (float) currcntVclocity;  l a s t D c t c n t P o s = ncwDctcntPos; return ncwDctcntPos; } } int {  HapticPageMap  CropPos  (int  inputPos)  if ( i n p u t P o s < 0) r c t u r n ( O ) ; if ( i n p u t P o s > height—8) r c t u r n ( h c i g h t — 8) ; return inputPos; } float {  H a p t i c P a g e M a p :: C r o p P o s  (float  inputPos)  i f ( i n p u t P o s < 0.0) return(0.0); # i f defined SUBTAXELTRIPLE if ( i n p u t P o s > ( f l o a t )( h e i g h t - 8 ) ) #olif d e f i n e d SUBTAXELDOUBLE i f ( i n p u t P o s > ( f lo a t ) ( heigh t - 8 ) ) #c 1 s c if ( i n p u t P o s > ( f lo at ) ( h e i g h t - 8 ) ) #cndif return  bool {  r o t u r n (( f 1 o a t ) ( ho igh t - 24) ) ; r e t u r n ( ( f 1 o a t ) ( heigh t — 16) ) ; ret u r n ( ( f 1 o a t ) ( heigh t - 8 ) ) ;  inputPos;  H a p t i c P a g e M a p : : Computclmagc ( s t r c s s d : : V o l t l m g // //  compute the t a c t i l e o u t p u t image at and p l a c e the r e s u l t in o u t p u t l m g  //  returns  //  uncommcnt one b l o c k of code below / / crop to map bounds map.pos = C r o p P o s ( m a p . p o s ) ;  true  if  everything  was  cursor  dandy;  // //  or a s s e r t map bounds assert (map.pos >= 0 && map.pos <  // //  or if  return false (map.pos < 0  if ||  false  position  h e i g h t —8) ;  *icon;  icon = i c o n s ; while (icon)  { / / r e n d e r o n l y the a r e a t h a t ' s " v i s i b l e " #if d e f i n e d SUBTAXELTRIPLE c o n s t i n t v i s i b l c a r c a = 2 4; #clif d e f i n e d SUBTAXELDOUBLE c o n s t i n t v i s i b l c a r c a = 16;  int  map.pos  otherwise  o u t s i d e of map bounds map.pos > height—8) r e t u r n  double t i m c d i f f ; float valucAt ; r e g i s t e r int index; Hapticlcon  *outputImg ,  false;  map.pos)  Appendix  #clsc const #cndif if  (  {  int  D. Browser Prototype Code  visiblcarca  241  — 8;  (icon—>at <— m a p . p o s + v i s i b l c a r c a ) ( icon — >at-ficon —>hcight >= map_pos)  )  i c o n —>RenderIconToMap ( t h i s ) ;  }  icon  }  =  icon —> n e x t ;  / / copy d a t a from the page map i n t o the o u t p u t l m g for ( i n t x=0; x<8; x++) { #if defined SUBTAXELTRIPLE index = x*3 4- map-pos ; v a l u c A t = ( v o l t m a p [ index ] + v o l t m a p [ i n d e x + 1 ] + vol t m ap [ i n d e x + 2] ) 3.0; #clif d e f i n e d SUBTAXELDOUBLB index = x*2 -f- m a p . p o s ; v a l u c A t = ( v o l t m a p [ index ] -f vol t map [ i n dcx + 1 ]) / 2; #olsc v a l u c A t = v o l t m a p [ x + map.pos ] ; #ondif // if  Do the l o o k a h c a d and l o o k b a c k smoothing ( l o o k a h c a d [x] != LOOKAHEADJMOVALUE) v a l u c A t = / 2.0;  if {  ( fabs ( v a l u c A t — l o o k b a c k [x] } >  50.0)  lookahcad[x] = valucAt; v a l u c A t = ( v a l u c A t -f l o o k b a c k [x] )  // /  (valucAt  threshold  +  /  lookahcad [ x ] )  value  2.0;  } else  l o o k a h c a d fx]  lookback[x]  =  = LOOKAHEAD-NOVALUE;  valucAt;  //  o p t i o n a l l y , p r o v i d e t a c t i l e f e e d b a c k of t r a n s i t i o n between c o n t r o l and p o s i t i o n c o n t r o l modes # i f d c f CLICKONVCONTROL if ( v C o n t r o l S t a t e = vControlWaiting ) { timcdiff = TimcSincc(&vControlStatcTimcStamp) ; i f ( t i m c d i f f < 0.005) v a l u c A t += 40; e l s e i f ( t i m c d i f f < 0.01) valucAt -= if ( v a l u c A t < —50) v a l u c A t = —50; e l s e i f ( v a l u c A t > 50) v a l u c A t = 50; } #endif o u t p u t l m g —> S c t (7 — x ,  0,  valueAt);  }  return  true;  } void {  HapticPageMap : : R c n d c r A l l I c o n s Hapticlcon  (void)  * icon ;  icon = i c o n s ; while (icon) { icon —> Ren der I con T o Map ( t h i s ) ; i c o n — icon —> n e x t ; }  40;  velocity  Appendix  242  D. Browser Prototype Code  S l i d c r S m o o t h c r : : S l i d c r S m o o t h c r () { c a c h e . s i z c = 10; t h r e s h o l d = 5.0; / / when s l i d e r is moved a l a r g e d i s t a n c e (above t h i s / / abandon a v e r a g i n g and j u s t jump to the new s l i d e r  threshold) , position  s l i d e r . c a c h c = new f l o a t [ cache.sizc ] ; for ( i n t i =0; i < c a c h c _ s i z c ; iH—h) s l i d e r . c a c h c [ i ] = 0.0; s t a r t i d x =0;  S l i d c r S m o o t h c r : : ~ S l i d c r S m o o t h c r () delete  loat  []  slider.cachc;  S l i d e r S m o o t h c r :: S m o o t h S l i d c r P o s  //  Takes the c u r r e n t s l i d e r slider pos Uses s i m p l e a v e r a g i n g of rawpos  //  f l o a t avgpos = float rctval; int i ;  (float  pos  and  the  last  rawpos)  returns  for ( i =0; i < c a c h c _s i z c ; i 4-+) avgpos += s l i d e r . c a c h c [ i ] ; avgpos / = cache.sizc; (  fabs(rawpos  — a v g p o s ) >=  threshold  r c t v a l = rawpos; for ( i = 0 ; i < c a c h c _ s i z c ; iH—(-) s l i d e r . c a c h c [ i ] = rawpos; } else rctval return  D . 1.5 #ifndcf #define  =  avgpos;  (rctval);  Browser Shared, h BROWSERSH ARED_H BROWSERSHARED-H  #includc  " a c c / M u t c x . h"  #includc ^include  " H a p t i c P a g e M a p . h" " B r o w s c r X M L B i t s . h"  class  "smoothed" pos's,  0.0;  / / add the v a l u e to the a r r a y , c l o b b e r i n g s t a r t i d x ++; s t a r t i d x %= c a c h e . s i z c ; slider_cachc[startidx] = rawpos;  if {  the  (cache.sizc -1)  BrowscrSharcd {  public : B r o w s c r S h a r c d () ;  )  the  oldest  value  value and  of  the  the current  Appendix  i n t GctMapPos ( ) ; void SctMapPos(int int  D. Browser Prototype Code  pos);  GctClicks();  void  SctClicks(int  bool  GctPageMapRcady ( ) ;  click_count_param) ;  void  SctPagcMapRcady  (bool  ready.flag) ;  / / PUBLIC h a p t i c page map s t r u c t u r e / / R u l e s : b e f o r e r e a d i n g , check i f GctPageMapRcady () / / before w r i t i n g , SctPagcMapRcady ( f a l s e ) H a p t i c P a g e M a p *pagcmap ; private : int c l i c k . c o u n t ; i n t map.pos ; bool  pagcmap.ready ;  ACE.Mutcx  mutcx ;  };  #cndif  D.l.6 //  BrowserShared.cpp  B ro wscrSh ared . epp  ^include  " B r o w s e r S h a r c d . h"  B rowscr Shared : : B r o w s e r S h a r c d { c l i c k . c o u n t = 0; map.pos — 0 ; p a g c m a p . r e a d y = 0; pagemap = NULL; }  ()  int {  ()  B r o w s e r S h a r c d : : GctMapPos mutcx. a c q u i r e () ; i n t r c t v a l — map.pos ; mutcx. r e l e a s e () ; return r c t v a l ;  } void  B r o w s e r S h a r c d : : SctMapPos  ( int  pos)  { mutcx . a c q u i r e () ; map.pos = pos ; mutcx. r e l e a s e () ; return ; } int {  BrowserSharcd : : G e t C l i c k s  ()  mutcx. a c q u i r e () ; int r c t v a l = c l i c k . c o u n t ; mutcx. r e l e a s e () ; return r c t v a l ; } void {  243  BrowscrShared::SctClicks  (int  click.count.param )  mutcx. a c q u i r e () ; click-count = click.count.param;  is  true  or  not  Appendix  D. Browser Prototype Code  244  mutex. r e l e a s e () ; return ; } bool {  B r o w s c r S h a r c d : : Get Page Map Ready ( ) mutex . a c q u i r e () ; int r c t v a l = pagcmap.rcady ; mutex. r e l e a s e () ; return r e t v a l ;  } void {  B r o w s c r S h a r c d : : Set P age M ap Ready  (bool  ready-flag)  mutex. a c q u i r e () ; pagcmap.rcady = r e a d y - f l a g ; mutex. r e l e a s e () ; return ;  >  D.l.7 // //  BrowserXMLBits.h  Browser X M L B i t s , m o d i f i e d Joseph L u k , J u l y 2006  #ifndcf #dcfinc ^include ^include ^include  original  file  by  Vincent  Levesque  (GPL)  BROWSERXMLBITS_H BROWSERXMLBITS_H <string> <vcctor> "ACEXML/common/ D e f a u l t H a n d l e r . h "  class PagcMapParscr : public { public : P agcM ap Parser ( s t d : : s t r i n g v i r t u a l ~ PagcMapParscr ( ) ; void  from  ParscFitc(std:: string  ACEXML-DcfaultHandler file  ,  HapticPageMap*  data);  fileNamc);  v i r t u a l v o i d c h a r a c t e r s ( c o n s t ACEXML_Char * c h , ACEXML_ENV_ARG_DECL) ACE.THROW.SPEC (( A C E X M L - S A X E x c c p t i o n ) ) ;  int  start,  int  length  v i r t u a l v o i d cndDocumcnt (ACEXML-ENV_SINGLEJVRGX>ECL) ACE_THROW_SPEC((ACEXML-SAXExccption) ); v i r t u a l v o i d c n d E l c m c n t ( c o n s t A C E X M L - C h a r *namcspaccURI , c o n s t l o c a l N a m c , c o n s t A C E X M L - C h a r *qNamc ACEXML_ENV_ARG_DECL) ACE-THROW_SPEC(( A C E X M L - S A X E x c c p t i o n ) ) ; v i r t u a l v o i d set D oc u me n t Lo cat o r ( A C E X M L - L o c a t o r * l o c a t o r ) ACE_THROWJ3PEC((ACEXML_SAXException) ) ;  ACEXML-Char *  v i r t u a l v o i d star t D o c u men t (ACEXML_ENV_SINGLE_ARG_DECL) ACE_THROW_SPEC((ACEXML_SAXExccption) ); v i r t u a l v o i d s t a r t E l c m e n t ( c o n s t A C E X M L - C h a r * namcspaccU R l , c o n s t A C E X M L - C h a r * l o c a l N a m e , c o n s t ACEXML_Char *qNamc, A C E X M L - A t t r i b u t c s * at ts A C E X M L_ENV_ARG_D ECL) ACE_THROW_SPEC(( A C E X M L . S A X E x c c p t i o n ) ) ; private : ACEXML-Locator* locator. ; std::vcctor<std::string> tags. ; HapticPageMap * data ; int currAt; };  ,  Appendix  class RawIconParscr { public :  :  public  R a w I c o n P a r s c r f std : : s t r i n g v i r t u a l ~ R a wlco n P ar sc r () ; void  P a r s e F i l c ( s t d :: s t r i n g  D. Browser Prototype Code  245  A C E X M L . D c f a u l t H and lcr  file  ,  Hapticlcon*  iconptr) ;  fileNamc);  v i r t u a l void characters (const ACEXML.Char *ch, ACEXML_ENV_ARG_DECL) ACE_THROW_SPEC((ACEXML_SAXException) ) ;  int  start,  int  length  v i r t u a l v o i d end Document ( A C E X M L - E N V _ S I N G L E - A R G - D E C L ) ACE_THROW.SPEC((ACEXML_SAXException) ) ; v i r t u a l void cndElcmcnt (const ACEXML.Char *namcspaccURI, const l o c a l N a m c , c o n s t A C E X M L . C h a r *qNamc A C E X M L . E N V . A R G J D E C L ) ACE_THROW_SPEC(( A C E X M L . S A X E x c c p t i o n ) ) ; v i r t u a l v o i d set Doc u m c n t L o c a t o r ( A C E X M L . L o c a t o r *locator) ACE_THROW_SPEC( ( A C E X M L . S A X E x c c p t i o n ) ) ;  ACEXML.Char *  v i r t u a l v o i d s t a r t D o c u r n c n t (ACEXML_ENV_SINGLE_ARG_DECL) ACE_THROW_SPEC( ( A C E X M L _ S A X E x c c p t i o n ) ) ; v i r t u a l v o i d s t a r t E l c m c n t ( c o n s t A C E X M L . C h a r * namcspaccU R l , c o n s t A C E X M L . C h a r • l o c a l N a m c , c o n s t A C E X M L . C h a r *qNamc, A C E X M L . A t t r i b u t c s * a t t s ACEXML_ENV_ARG_DECL) ACE_THROW_SPEC((ACEXML.SAXException) ) ; private : ACEXML.Locator* l o c a t o r . ; std : : v c c t o r < s t d : : s t r i n g > t a g s . ; H a p t i c l c o n * icon ; int c u r r l n d c x ; int currFrame ; }; #cndif  //BROWSERXMLBITS-H  D.l.8 // //  BrowserXMLBits.cpp  Browser XML B i t s , m o d i f i e d Joseph L u k , J u l y 2006  #includc ^include ^include  from  original  file  by  Vincent  <iostrcam> <sstream> < a s s e r t . h>  ^include ^include  acc/ACE.h" a c c / L o g . M s g . h"  #i n c l u d c #includc #i n c l u d c ^include # inc1udc # inc1udc  ACEXML/common/ F i l c C har S t r c a m . h" ACEXML/common / H t t p C h a r S t r e a m . h" A C E X M L / c o m m o n / S t r C h a r S t r c a m . h" ACEXML/ p a r s c r / p a r s e r / P a r s c r . h " acc/Gct.Opt.h" ace / A u t o . P t r . h"  # i n c l u d c " H a p t i c P a g e M a p . h" #i n c1u d c " B r o w s c r X M L B i t s . h "  int  a v e r a g e = —1;  void PagcMapParscr : : P a r s c F i l c (std : : s t r i n g fileNamc) { A C E X M L . F i l o C h a r S t r o a r a t f s t m = now A C E X M L . F i l e C h a r S t r e a m ;  Levesque  (GPL)  Appendix  i f {  ( fst m—>opcn  D. Browser Prototype Code  ( f i l e N a m c . c . s t r () )  !=  246  0)  s t d : : s t r i n g s t r ( " F a i l e d to open X M L f i l e : " ) ; s t r A-= f i l e N a m c ; //STRESSDJTHROW-EX_DESC( F i l c E x P a g c M a p P a r s c r : : P a r sc F i 1 c " ,  st r ) ;  / / TODO: throw the c o r r e c t e x c e p t i o n ACEXML_TRY_NEW_ENV { ACEXML.Parscr parser; A C E X M L . l n p u t S o u r c c i n p u t ( fstm ) ; parser . s c t C o n t c n t H a n d l c r ( this ) ; parser . s c t F c a t u r c ( " h t t p : / / x m l . o r g / s a x / f e a t u r c s / v a l i d a t i o n " , f a l s e ) ; / / p a r s e r , parse (&input ACEXMI^ENVJUIGJARAMETER) ; ACEXML.TRY-CHECK; } A C E X M L_C A T C H ( A C E X M L . S A X E x c o p t i o n . ex) { ACE_UNUSED_ARG(ox ) ; ACE-DEBUG ((LM_ERROR, A C E - T E X T ( " E x c e p t i o n o c c u r r e d . E x i t i n g . . . \ n " ) ) ) ; cx . p r i n t () ; } ACEXML-ENDTRY;  PagcMapParscr  characters  (const  ACEXML.Char * c d a t a , int start , int end ACEXML_ENV_ARG_DECL J N O T . U S E D ) ACE_THROW.SPEC ( (ACEXML.SAXExccption) )  { int l t a g = t a g s . . s i z e ( ) — 1; / / p r i n t f ( " I t a g : %d\n", ltag); if if  ( t a g s . . b a c k { ) == " a t " ) { ( t a g s . , at ( l t a g — 1) =— " i c o n " ) { currAt = a t o i ( c d a t a ) ; # i f d c f DEBUG p r i n t f ("Tag <at> value % d \ n " , #endif }  } else  if if {  ( t ags _ . back ( ) ( t a g s _ . at ( ltag—1) #ifdcf printf #endif  currAt);  "namc"){ == " i c o n " )  DEBUG (" T a g <namc>  v a l u e %s ,  currAt  = %d\n",  H a p t i c l c o n *tcmp = d a t a —> i c o n s ; c h a r *pathnamc = new c h a r [ s t r l c n (" i c o n s /") s t r c p y (pathname , " i c o n s / " ) ; s t r c a t (pathname, c d a t a ) ; d at a—> i c o n s = new H a p t i c l c o n (pathname).; d e l e t e [] pathname; data—>icons —>ncxt = temp; d a t a —> i c o n s —> at = c u r r A t ; currAt  =  0;  } } else { int  }  if  ( t ag s _ . back ( ) == height  =  "height")  atoi(cdata);  # i f d e f DEBUG p r i n t f ("Tag < h c i g h t > #endif data—>hcight = h e i g h t ;  value %d\n",  height);  cdata,  4* s t r l c n  currAt);  (cdata)];  todo  Appendix D. Browser Prototype Code  247  else { # i f d o f DEBUG s t d : : s t r i n g thetag = t a g s _ . b a c k ( ) ; p r i n t f (" Unknown tag %s \ n " , t h e t a g . c _ s t r ( ) ) ; #endif } } void P a g c M a p P a r s c r : : cndDocumcnt (ACEXMLJENV_SINGLE-ARG_DECL_NOT_USED) ACE.THROWJ3PEC ( ( A C E X M L . S A X E x c c p t i o n ) ) { } void PagcMapParscr : : cndElcmcnt  ( const  ACEXML.Char * uri , c o n s t A C E X M L . C h a r *name, c o n s t A C E X M L . C h a r *qName A C E X M L . E N V_ARG_DECL_NOT_USED) ACE_THROW_SPEC( ( A C E X M L . S A X E x c c p t i o n ) )  { tags..pop-back(); } void P a g c M a p P a r s c r : : sc t Do c u m c n t L o c at o r ( A C E X M L.Locator * ACE_THROW_SPEC(( A C E X M L . S A X E x c c p t i o n ) ) { this —>locator. = l o c a t o r ; }  locator )  void PagcMapParscr : : startDocumcnt (ACEXML.ENVJ3INGLE_ARG-DECL-NOT.USED) ACE.THROW-SPEC (( A C E X M L . S A X E x c c p t i o n ) ) { } void PagcMapParscr : : s t a r t E l c m c n t  ACEXML.Char * u r i , c o n s t A C E X M L . C h a r *namc , c o n s t A C E X M L . C h a r *qNamc, ACEXML.Attributcs *a1 ist ACEXML_ENV_ARG JDECL3IOT.USED) ACE.THROWJ3PEC ( ( A C E X M L . S A X E x c c p t i o n ) )  {  ( const  t a g s . . push_back(name ) ;  } P age M ap P arse r : : P a g e M a p P a r s e r ( s t d : : s t r i n g f i l e , H apt ic PageMap * { t h i s —> d a t a = d a t a ; c u r r A t ~ —1; # i f d c f DEBUG p r i n t f ( " P a g c M a p P a r s e r about to c a l l p a r s e f i 1 c \ n " ) ; #cndif ParscFilc(filc); } PagcMapParscr : : ~ PageMapParscr() { }  void {  R a w I c o n P a r s c r : : P a r s e F i 1 c ( s t d :: s t r i n g  fileName)  data)  Appendix  ACEXML.FilcCharStrcam if {  ( fstm —>opcn  D. Browser Prototype Code  *fstm  = new  ( f i l c N a m c . c . s t r () )  248  A C E X M L_F ilcCharStrcam ; !=  0)  s t d : : s t r i n g s t r ( " F a i l e d to open XML f i l e : " ) ; s t r += f i l c N a m c ; / / S T R E S S D JTHROW-EX_D ESC ( F i l c E x ," R a w l c o n Parser : : P a r s c F i l c " ,  }  str ) ;  / / TODO: throw the c o r r e c t exception ACEXML_TRY_NEW_ENV { ACEXML-Parser parser; A C E X M L - I n p u t S o u r c c i n p u t ( fstm ) ; p ar s c r . set C o n t cn t H an d lc r ( t h i s ) ; parser. sctFcaturc(" h t t p : / / x m l . o r g / s a x / f c a t u r c s / v a l i d a t i o n " , false); / / p a r s e r , parse ( & i n p u t ACEXMUENV-ARG-PARAMETER) ; ACEXML.TRY.CHECK; } ACEXML_CATCH ( A C E X M L - S A X E x c c p t i o n , ox ) { ACE_UNUSED^\RG ( ex ) ; ACE-DEBUG ((LM_ERROR, A C E - T E X T ( " E x c e p t i o n o c c u r r e d . E x i t i n g . . . \ n" ) ) ) ; ex. p r i n t () ; } ACEXML_ENDTRY  todo  ;  void Raw Ic o n P ar s e r : : c h a r a c t e r s  ACEXML-Char *cdata , int start , i n t end ACEXML-ENV_ARG_DECL_NOT_USED) ACE-THROW-SPEC( ( A C E X M L - S A X E x c c p t i o n ) )  {  i n t l t a g = t a g s . . s i z e ( ) — 1; / / p r i n t f (" l t a g : % d \ n " , l t a g ) ; if  //  }  (tags..back() " index " ) { i f ( t a g s . , at ( l t a g — 1) == " c l e m e n t " ) { currlndex = atoi(cdata); a s s e r t ( i c o n —>height) ; a s s e r t ( c u r r l n d e x < icon —>height) ; }  else //  ( const  i f ( t a g s _ . b a c k ( ) == " v a l u e " ) { i f ( t a g s . , at ( l t a g — 1) == " e l e m e n t " ) { a s s e r t ( i c o n —> c o n t e n t ) ; if {  ( ! strcmp ( icon —> c o n t e n t —> type ,  "Spatiallcon"))  # i f d c f DEBUG p r i n t f (" Tag < v a l uc > = %f , c u r r l n d c x=%d\n " , a t o f ( c d a t a ) , # cndif icon —> c o n t e n t —> S c t D a t a ( c u r r l n d e x , } if {  currlndex);  atof(cdata)) ;  ( ! strcmp ( icon —>con t c n t —>ty pc , " A n i m a t c d l c o n " ) ) # i f d e f DEBUG p r i n t f ("Tag < v a l u c > = %f , c u r r I ndcx=9cd \ n " , a t o f ( c d a t a ) , c u r r l n d e x ) ; #endif i f ( t ags _ . a t ( 11 ag — 1) == "frame") { icon —>con t c n t —>Sct Dat a ( c u r r l n d c x + c u r r F r a m e , a t o f ( c d a t a ) ) ;  }  else  } else  } {  { # i f d o f DEBUG p r i n t f (" ERROR: v a l u e g i v e n Animatcdlcon.\n") ; #endif  outside  of  frame  context  in  Appendix  D. Browser Prototype Code  # i f d o f DEBUG p r i n t f (" Unknown i c o n  type  for  249  c i c m c n t / v a 1 u c %s\n" , i co n —>co n t e n t —> t y pe )  #cndi'f } } lsc  if  ( t a g s . . back ()  if {  ( ! s t r c m p ( i c o n —>content —>type ,  "frames")  icon —>co n t e n t —>num Frames =  } lsc  ==  ==  "Animatcdlcon"))  atoi(cdata);  if  (tags..back()  if {  ( ! s t r c m p ( i c o n —>con t cn t —>ty pe , " A n i m a t c d l c o n " ) )  "f p s " )  icon —> c o n t e n t —> fps  =  atoi(cdata);  } lsc  if  ( t ags _ . back ( ) ==  if {  ( ! s t r c m p ( i con —>con t cn t —>ty pe , " A n i m a t c d l c o n " ) ) currcntFramc  =  "frame")  atoi(cdata);  } lsc  if  (tags_.back()  if  (!strcmp(cdata,  ==  "type") "Spatiallcon"))  { icon —>c o n t c n t = new S p a t i a l l c o n C o n t e n t () ; icon —> c o n t e n t —> i c o n p a r e n t = i c o n ; } else {  if  (! s t r c m p ( c d a t a ,  "Animatcdlcon"))  i c o n —>c o n t c n t = new A n i m a t e d l c o n C o n t e n t () ; i c o n —> c o n t e n t —> i c o n p a r e n t — i c o n ; } else { # i f d c f DEBUG p r i n t f (" Unknown i c o n #ondif } else {  t y p e %s \ n " ,  cdata);  } if  ( t ags _ . back () assert  "height")  ( i c o n —>content) ;  i c o n —> h e i g h t if {  ==  =  atoi(cdata);  ( ! s t r c m p ( i co n —> c o n t c n t —>ty pc ,  "Spatiallcon"))  # i f d o f DEBUG p r i n t f ( " A l l o c a t i n g d a t a for h e i g h t =%d \ n " , i c o n —> h e i g h t ) ; #ondif icon —>content — > A l l o c a t c D a t a ( i c o n —>height) ; } else {  if  ( ! s t r c m p ( i c o n —>con t en t —>ty pc ,  "Animatcdlcon"))  # i f d c f DEBUG p r i n t f ( " A l l o c a t i n g d a t a for h c i g h t=%d \ n " , i co n — > h c i g h t ) ; #cndif i c o n —> c o n t e n t —> A l l o c a t c D a t a ( i c o n —> h e i g h t * icon —> c o n t e n t —> numFramcs) ;  } else { # i f d c f DEBUG p r i n t f ("Unknown #cndi'f }  icon  type  for  clement / value  %s \ n " , i c o n —>co n t c n t —> t y p e )  Appendix D. Browser Prototype Code  void R a w I c o n P a r s c r : : endDocumcnt (ACEXML_ENV_SINGLE_ARG-DECL_NOT_USED) ACE-THROW-SPEC ( ( A C E X M L . S A X E x c c p t i o n ) ) { } void RawIconParscr : : cndElcmcnt  ACEXML.Char * uri , c o n s t A C E X M L . C h a r *namc, c o n s t A C E X M L . C h a r *qName ACEXML_ENV_ARG_J3ECL_-NOT-USED) ACE.THROW-SPEC ( ( A C E X M L . S A X E x c c p t i o n ) )  {  ( const  tags..pop.back();  } void R a w I c o n P a r s c r :: s c t D o c u m c n t L o c a t o r ( A C E X M L - L o c a t o r ACE_THROW_SPEC( ( A C E X M L - S A X E x c e p t i o n ) ) { this —>locator. = l o c a t o r ; }  *  locator)  void RawIconParscr : : startDocument (ACEXML-ENV-SINGLE-ARG.DECL_NOT.USED) ACE.THROWJ3PEC( ( A C E X M L . S A X E x c c p t i o n ) ) { } void RawIconParscr : : s t a r t E l e m c n t  ( const  ACEXML.Char * uri , c o n s t A C E X M L . C h a r *namc, c o n s t A C E X M L . C h a r *qNamc, ACEXML.Attributcs * alist ACEXML_ENV_ARGJDECL_NOT_USED) ACE_THROW.SPEC( ( A C E X M L . S A X E x c c p t i o n ) )  { t a g s . . push-back(name ) ; } RawIconParscr : : RawIconParscr ( std : ; s t r i n g { t h i s —> i c o n = i c o n p t r ; c u r r l n d e x = — 1; ParseFilc ( file ) ; }  file  ,  Hapticlcon*  R a w I c o n P a r s c r : : ~ R a w I c o n P a r s c r () { }  D.l.9 // // // //  main.cpp  Browser V2 . 0 R e w r i t e to reduce o v e r h e a d — s i n g l e t h r e a d o n l y Joseph L u k , J u l y 2006 based on code by V i n c e n t Levesque and Shannon L i t t l e  #include #i n cl u de #i n cl u de ^include #includc #i n c1u d c ^include ^include #i n c1ud c ^include  s t r c s s d / F i r m w a r c I n f o X M L . h" st r c s s d / D c v i c c I n f o X M L . h" stressd / StrcamDcvicc . h " strcssd / PackctLoggcr.h" strcssd/init.h" < s t d i o . h> <iostrcam> < s i g n a l . h> < s y s / s t a t . h> " a c e / T h r e a d . h"  iconptr)  250  Appendix  #includc #definc #dofino  251  D. Browser Prototype Code  " D a t a U p d a t e T h r e a d . h" FIRMWARE-FILE " d of a u 11 . sd f " HARDWARE-FILE " d cf au 11 . sdh "  / / These must be kept s y n c h r o n i z e d w i t h MyComponcnt. epp ! # d c f i n e WORKING-DIR " / h o m e / l u k / b r o w s e r - w o r k i n g - d i r " # d e f i n o PAGEMAP-FILENAME " pagemap . xml" # d c f i n c PAGEMAP-FLAGFILENAME "nowpagomap"  int  tcrmflag;  void {  s i g i n t _h a n d 1 c r tcrmflag  =  (int  signum)  1;  } / / I / O r o u t i n e s to communicate w i t h browser / / c u r r e n t l y uses f i l e I / O , s h o u l d p r o b a b l y be i m p r o v e d l a t e r b o o l W r i t c S 1 i d c r P o s i t io n ( i n t m a p . p o s , H a p t i c P a g e M a p * p a g e M a p ) ; bool W r i t c C l i c k s ( i n t click-count); bool WasPagcRcloadcd ( v o i d ) ; bool int  filcExists main ( i n t  (char  argc,  *  char*  filcNamc); argv [] )  { struct BrowscrSharcd sharcdData; i n t c l i c k . c o u nt , l a s t - c l i c k s ; i n t map.pos ; float norm.map.pos ; t c r m f l a g = 0; s i g n a l (SIGINT, & s i g i n t _ h a n d l c r ) ; try { s t r c s s d :: i n i t ( ) ; std : : cout « " L o a d i n g f i r m w a r e d e f i n i t i o n from " « s t r c s s d : : F i r m w a r c I n f o X M L f w l n f o (FIRMWARE-FILE) ;  FIRMWARE-FILE «  ".. .\ n" ;  s t d : : cout « " L o a d i n g hardware d e f i n i t i o n from " « s t r c s s d : : Devi c e l n f o X M L d e v l n f o (HARDWARE-FILE) ;  HARDWARE_FILE «  "...\n";  std : : cout « " C r e a t i n g s t r e a m i n g STRcSS d e v i c e with strcssd : : StrcamDcvice dev(devInfo, fwlnfo, devlnfo,  buffer 1);  std : : cout  «  "Connecting  and programming d e v i c e  (this  size  may take  of a few  dev . C o n n e c t ( ) ; Was Page Reloaded () ;  //  delete  page  update  file ,  if  it  exists  std : : cout « " S t a r t i n g high — p r i o r i t y Data UPDATE t h r e a d . . . \ n " ; D a t a U p d a t e T h r e a d dat aU p d T h r e a d (&s h ar cd D a t a , & d e v ) ; A C E . h t h r o a d . t hThread; ACE-thrcad-t t-id ; A C E - T h r o a d :: spawn ( DataUpdateThread : : Start , &dataUpdThread , THR-NEWJLWP | T H R - J O I N A B L E , &t.id , &hThread , A C E - S c h c d - P a r a m s : : p r i o r i t y . m a x (ACE-SCHED-OTHER)  l...\n"; seconds)  Appendix  do {  //  this  loop  D. Browser Prototype  runs  once  per new  page  shared D a t a . S c t P a g c M a p R c a d y ( f a l s e ) ; page map  252  Code  map //  stop  thread  from  reading  from  char pagemap-path [ 1 00] ; s p r i n t f ( p a g c m a p . p a t h , "%s/%s", WORKINGJ3IR, PAGEMAP.FILENAME) ; p r i n t f ( " L o a d i n g page map f i l e %s . \ n " J p a g c m a p . p a t h ) ; H a p t i c P a g e M a p pagcMap ( p a g c m a p . p a t h ) ; p r i n t f ("Done l o a d i n g page map. R e n d e r i n g pagcMap . R c n d c r A l l I c o n s () ; p r i n t f ("Done r e n d e r i n g i c o n s . \ n " ) ;  i c o n s . . . \ n" ) ;  p r i n t f ( " \ n S t a t i c pagemap d u m p : \ n " ) ; for ( i n t pos=0; pos<pagcMap . h c i g h t ; pos++) { p r i n t f ("%2.0f ," , pagcMap . v o l t m a p [ pos ] ) ; } p r i n t f ("\n\n") ; s h a r c d D a t a . pagemap = &pagcMap ; s h a r c d D a t a . SctPagcMapRcady ( true ) ; map u s l c c p (500) ;  //  start  l a s t . c l i c k s = sharcdData. G c t C l i c k s f ) ; / / t h i s s h o u l d p r o b a b l y be moved i n t o printf  ("Type C T R L - C  w h i l e ( t c r m f l a g == { uslccp (100);  to  thread  reading  a low—priority  quit.\n");  0)  map.pos = shared D a t a . GctMapPos ( ) ; click.count = sharcdData. GetClicksf); # i f d c f DEBUG p r i n t f ("Current #cndif  position  at % d \ n " ,  map.pos);  W r i t c S l i d c r P o s i t i o n (map.pos , &pageMap) ; //  WritcClicks( click.count );  if } // }  while  //  ( W a s P a g e R e l o a d c d () )  while  clean  (tcrmflag  (tcrmflag  up page  break;  —— 0)  map s t u f f  here,  if  necessary  —— 0) ;  p r i n t f ( " K i l l i n g t a c t i l e loop ACE.Thrcad : : cancel ( hThrcad ) ; s t d : : cout « "Disconnecting dev . D i s c o n n e c t () ;  t h r e a d . \ n");  device  ...\n";  } c a t c h ( s t r c s s d :: E x c e p t i o n & cx) { s t d : : cout « " l i b s t r c s s d threw an e x c e p t i o n : \ n " s t d : ; cout « cx . E r r S t r () « " \ n" ; } s t r c s s d : : f i n i () ; return  0;  from  thread . . .  page  Appendix  D. Browser Prototype Code  253  }  b o o l  W ritcS  1  idcrPosition  (int  map.pos,  H a p t i c P a g e M a p *pageMap)  { //delete all existing .pos files DIR* p d i r = o pen d i r (WORKING-DIR) ; s t r u c t d i r c n t *pcnt ; if  (!pdir){ p r i n t f (" o p e n d i r () c x i t (1) ;  failure;  terminating");  } s t d : : s t r i n g i c o n . t y pc = " " ; while ((pcnt = r e a d d i r ( p d i r ) ) ) { s t d : : s t r i n g f i l e = pent —>d_name ; int length = f i l e , length (); / / f i l t e r out . p o s ( p o s ) f i l e s only i f ( f i l e . r f i n d (" .pos" , l e n g t h ) == l e n g t h — 4) { //delete file c h a r t y p c . s t r [50] ; s t r c p y ( t y p c - s t r , WORKING-DIR) ; s t r c a t ( t y p c - s t r ,"/") ; strcat (typc.str , f i l c . c _ s t r ( ) ) ; rcmovc(typC-Str)  ;  }  } closcdir  (pdir);  / / w r i t e new pos file char t y p c . s t r [50]; s t r c p y ( t y p c - s t r ,WORKING-DIR) ; s t r c a t ( t y p c - s t r ,"/") ; char s l i d c r . s t r [ 1 0 ] ; // //  f l o a t n o r m . m a p . p o s = ( f l o a t ) map.pos / ( f 1 o a t ) pagcMap—> h c i g h t s p r i n t f ( s l i d c r . s t r , "%f", n o r m . m a p . p o s ) ; sprintf strcat strcat  *  100.0;  ( s l i d c r . s t r , "%d" , map.pos) ; (typc.str , s l i d c r . s t r ) ; (typc.str ,".pos");  std : : o f s t r c a m o u t p u t F i l e ; o u t p u t F i l c .open ( t y p c . s t r , o u t p u t F i l e . c l o s e () ; return true;  std : : ofstream : : out) ;  }  bool {  WritcClicks  (int  ncw.clicks)  static  int  printf  ("last.clicks  last.clicks  — 0;  = %d ,  ncw.clicks  = %d \ n " ,  int c l i c k . d i f f = n c w . c l i c k s — l a s t . c l i c k s ; i f ( c l i c k . d i f f > 0){ for ( i n t c = 0; c < c l i c k . d i f f ; c ++){ / / w r i t e new c l i c k file c h a r t y p c . s t r [50] ; s t r c p y ( t y p c - s t r ,WORKING-DIR) ; s t r c a t ( t y p c . s t r >"/") ; c h a r s l i d c r . s t r [10]; s p r i n t f ( s l i d c r _ s t r , "%d", l a s t . c l i c k s  last.clicks  +  c);  ,  ncw.clicks);  Appendix D. Browser Prototype Code  254  strcat (typc.str , s l i d c r . s t r ) ; strcat (typc.str ,". click"); std : : o f s t r c a m o u t p u t F i l e ; o u t p u t F i l e . o p e n ( t y p c . s t r , s t d : : o f s t r c a m : : out ) ; o u t p u t F i l e . c l o s e () ; } } last.clicks  =  ncw.clicks;  }  / / Check the p age in a p _f 1 a g f i 1 c bool WasPagcRcloadcd ( v o i d )  to  sec  if  it  exists ;  if  it  does ,  delete  it .  { // //  //  D I R . p d i r = o p o n d i r (WORKINGJDIR) ; struct dircnt .pent;  // // //  if  // // // // // // // // // // //  b o o l found = f a l s e ; w h i l e ( ( p c n t = r c a d d i r ( p d i r ) ) && found == f a l s e ) { std::string f i l e = pc nt —>d_name ; int length — f i l e , length (); //filter out o n l y the PAGEMAP.FLAGFILENAME i f ( f i 1 c . r f i n d (PAGEMAP.FLAG FILENAME, l e n g t h ) != s t d : : s t r i n g : : npos) { found = t r u e ; //delete file char pathname [ 1 0 0 ] ; s p r i n t f ( p a t h n a m e , "%s/%s", WORKINGJDIR, PAGEMAP.FLAGFILENAME) ; rcmovc(pathnamc);  // //  // // //  // / /  }  }  (!pdir){ printf ("opendirf) cxit(l);  failure;  }  closedir return  (pdir); (found);  / / L e t ' s t r y a more e f f i c i e n t s t a t i c c h a r path [ 1 0 0 ] ; sprintf if {  (path,  way  "%s/%s", WORKINGJDIR,  PAGEMAP.FLAGFILENAME) ;  (filcExists(path)) remove return  } else  bool {  terminating");  return  filcExists  ( path ) ; true; false;  (char  *  filcNamc)  s t r u c t stat buf; i n t r c t v a l = s t a t ( f i l e N a m e , &buf ) ; r e t u r n ( r c t v a l == 0 ? t r u e : f a l s e ) ; >  D.2  Visual Browser Component  The visual browser component extends the Mozilla platform using a combination of custom JavaScript, X U L , and X P C O M files. It is based partially on code developed by Shannon Little [33]. In the interest of brevity, some  Appendix  D. Browser Prototype Code  255  a u t o m a t i c a l l y generated and t r i v i a l configuration files are o m i t t e d . • b r o w s e r , j s - the m a i n interactive J a v a S c r i p t routine. • w e b s c r o l l e r . e s s - C a s c a d i n g Style Sheet description of the default browser presentation settings. • b r o w s e r . x u l - onscreen user interface description file. • r e a d y d i a l o g . x u l - user interface description for a m o d a l dialog used as a p r o m p t d u r i n g user evaluation. • IMyComponent. i d l - I D L (Interface D e s c r i p t i o n Language) stub header for the X P C O M I / O routines. • MyComponent. epp - the file input and output routines, implemented in the X P C O M  D.2.1  framework.  browser.js  // //  browser . j s Joseph Luk,  //  Based  on  July  2006  framework  code  by  Shannon  // Globals / / uncommcnt one of the f o l l o w i n g / / var i n p u t . m o d c = "mouse"; / / var i n p u t _m ode = " s l i d e r — " ; var i n p u t . m o d c = " s l i d c r + " ; var  init.pagc  const var var var var var var var var var var var var var var  Little  input  modes  = " f i l e : / / / home / l u k / b r o w s c r — c o n t e n t / i n d e x . html"  tactilc.scalcfactor  =  1;  / / s h o u l d be 1 ( f u l l - s c a l e ) myBrowscr = n u l l ; appCorc = n u l l ; top = 0; c u r r e n t = 0; search Range ; startPt; cndPt ; linkTcxts; linkURLs ; highlighted = null; p a g e h e i g h t = 0; p a g e w i d t h = 0; a d d h e i g h t = 0; links;  or  2  (half — scale)  Appendix D. Browser Prototype Code var var var  n u m l i n k s = 0; / / total n u m _1 i n ks _a t _y pos ; componentlinker ;  / / study l o g g i n g globals var 1 og . r e s e t C o u n t = 0; var l o g . s t a r t T i m e = " " ; var l o g . c n d T i m e — " " ; var l o g . p a g c s = new A r r a y ( ) ; var l o g . t i m c s = new A r r a y ( ) ;  var var  number  // //  of  page page  links  load load  in  the  256  page  links times  firstload = true; debug = t r u e ;  const const  P R E F S . C I D = " @ m o z i 11 a . org / p r ef e r e n c e s ; 1 " ; PREFS.I.PREF = "nsIPrcf";  const  PREFJ3TRING = " b r o w s e r , dom . window . dump . e n a b l e d " ;  try { var P r c f = new Components . C o n s t r u c t o r ( P R E F S . C I D , P R E F S . I . P R E F ) ; var p r c f = new P r c f () ; p r c f . S c t B o o l P r c f ( PREFJ3TRING , t r u e ) ; }catch(c){> f u n c t i o n i n i t B r o w s e r () { if (debug) l o a d P a g c . ( " j a v a s c r i p t : " ) ; / / load debug page i f  applicable  myBrowser = document. g c t E l c m c n t B y I d ( " browser — c o n t e n t " ) ; a p p C o r c = Components, c l a s s e s [" © m o z i l l a . o r g / a p p s h c l l / c o m p o n e n t / b r o w s e r / instance;1"] . c r c a t c l n s t a n c c (Components, i n t e r f a c e s . n s I B r o w s e r l n s t a n c c ) ; a p p C o r c . s c t W c b S h c l l W i n d o w ( window) ; var c o n t e n t = document . g c t E l c m e n t B y l d (" browser — c o n t e n t " ) ; if(content){ c o n t e n t . a d d E v c n t L i s t c n c r (" l o a d " , setup , t r u e ) ; } try  {  n e t s c a p e . s e c u r i t y . P r i v i l c g c M a n a g e r . e n a b l c P r i v i l c g c (" U n i v e r s a l X P C o n n c c t " ) ; const  c i d = " @ my domain . com / XPCOMSample / My Component ; 1 " ;  componentlinker  = Components, c l a s s e s [ c i d ] , c r c a t c l n s t a n c c  ();  c o m p o n e n t l i n k e r = c o m p o n e n t l i n k e r . Q u e r y l n t c r f a c c (Components, i n t e r f a c e s . IMyComponent) ; }  catch  (err)  {  alert(err) ; return ; } //  Go f u l l — s c r e e n mode n e t s c a p e . s e c u r i t y . P r i v i l c g c M a n a g e r . e n a b l c P r i v i l c g c (" U n i v e r s a l X P C o n n c c t " ) ;  /*  */ /* /* /* /»  */  /*  c o n s t MEDIATOR.CONTRACTID=" © m o z i l l a . o r g / a p p s h c l l / w i n d o w - m e d i a t o r ; 1 " ; * / const nsIWindowMcdiator=Components. i n t e r f a c e s . nsIWindowMcdiator; * / var windowManagcr= * / Components . c l a s s e s [ MEDIATOR-CO NTRACTID ] . g c t S o r v i c o ( n s I W i n d o w M c d i a t o r ) ; try  /* / * /• /*  {  */  n e t s c a p e . s e c u r i t y . P r i v i l c g c M a n a g e r . e n a b l c P r i v i l c g c (" U n i v e r s a l X P C o n n c c t " ) ;  */  main Window = windo w Manager . get Most Recent Window (" n a v i g a t o r : b r o w s e r " ) ; * / } */ catch(c) { * /  Appendix  D. Browser Prototype  257  Code  alert(c) ; */  h I* 1* h h  }  */  i f ( main Window . s i d c b a r _ i s _ h i d d c n ( ) ) */ hidcSidebar=fa1se; */ if(hidcSidcbar) */ main Window . S i d c b a r S h o w H i d e ( ) ; */ mainWindow. B ro w scr F u 1 IS cr ce n ( ) ; */ window . f u l l S c r c e n = t r u e ; * / window, l o c a t i o n b a r . v i s i b l e = f a l s e */ window, t o o l b a r . v i s i b l c = f a l s c ; * /  /* /* /*  1*  /*  var if  browser W in  = document . g c t E l c m c n t B y l d ( " b r o w s e r w i n " ) ;  (input.modc = "mousc"){ //attach listeners browscrWin . a d d E v c n t L i s t c n c r ( ' DOMMouscScroll ' , s c r o l l , t r u e ) ; browscrWin . a d d E v c n t L i s t c n c r ( ' c l i c k ' , P u s h B u t t o n H a n d l c r , true ) ;  } //  a t t a c h k e y b o a r d s e l e c t event l i s t e n e r ( w o r k a r o u n d for n o n f u n c t i o n a l button ) b r o w s c r W i n . a d d E v c n t L i s t c n c r ( ' key down ' , P u s h B u t t o n H a n d l c r , t r u e ) ; / / Load the page , t r i g g e r i n g l o a d P a g c _ ( in i t . p a g c ) ;  the  setup  select  function  } f u n c t i o n l o a d P a g c . ( page ) { / / glob o n l y the l a s t p a r t of the URL var p a g e s t r i n g = page . t o S t r i n g () ; var p a g c f i l c n a m c = p a g e s t r i n g . s u b s t r i n g ( p a g e s t r i n g . l a s t I n d c x O f ( " / " ) pagestring . length ) ; / /  dump ( " l o a d P a g c :  " 4- p a g c f i l e n a m e  +  1,  " \ n" ) ;  / / push the page l o a d i n t o the l o g a r r a y log-pages [ log-pages . length ] = pagcfilename; var d a t c O b j = new D a t c ( ) ; l o g - t i m e s [ l o g - t i m e s . l e n g t h ] = d a t c O b j . g e t T i m e () ; c o n s t nsl W cb N a v i g a t i o n = C o m p o n e n t s , i n t e r f a c e s , nsl W c b N a v i g a t i o n ; g c t B r o w s c r () . w c b N a v i g a t i o n . l o a d U R I ( p a g e , n s I W c b N a v i g a t i o n . LOAD_FLAGS_NONE, null , null , n u l l ) ; } function  s e t u p ( e v e n t ){  myBrowscr = e v e n t . o r i g i n a l T a r g e t ; myBrowscr. c o l l a p s e = t r u e ; c u r r e n t = 0; highlighted = null; document . command D ispatchcr . focused Window = window ; document . c o m m a n d D ispatchcr . focused E l e m e n t = myBrowscr . l i n k s . i t c m ( c u r r e n t ) ; if  ( i n p u t . m o d c . i n d c x O f (" s l i d e r " )  !=  — 1){  p a g c h c i g h t a l t = c o n t e n t . i n n c r H e i g h t 4- c o n t e n t . s c r o l l M a x Y ; p a g c w i d t h a l t = c o n t e n t . i n n c r W i d t h 4- c o n t e n t . s c r o l l M a x X ; if ( d e b u g ) dump ("Page d i m e n s i o n s ( v i a c o n t e n t . i n n c r H c i g h t + s c r o l l M a x Y ) " p a g c h c i g h t -f- " X " + p a g e w i d t h 4- " \ n " ) ; p a g c h c i g h t = g c t B r o w s c r () . c o n t e n t D o c u m e n t ; body . s c r o l l H e i g h t ; p a g e w i d t h = g c t B r o w s c r () . c o n t e n t D o c u m e n t . body . s c r o l l W i d t h ; i f (debug) dump ("Page d i m e n s i o n s ( v i a S c r o l l H e i g h t ) " + p a g c h c i g h t a l t " + pagcwidthalt + "\n"); if if  (debug) (debug)  dump ("window. i n n c r H e i g h t = " + window . i n n c r H e i g h t -f- " \ n " ) ; dump (" c o n t e n t . i n n c r H e i g h t = " 4- c o n t c n t . i n n e r H c i g h t 4- " \ n " )  +  + " X  Appendix D. Browser Prototype Code numlinks = myBrowscr. l i n k s , l e n g t h ; i f (debug) dump ("Number of l i n k s = " + // //  numlinks +  258  "\n");  p a g c h c i g h t = g c t B r o w s c r () . c o n t e n t D o c u m e n t . f i r s t C h i l d . o f f s c t H c i g h t ; p a g e w i d t h = g e t B r o w s e r () . c o n t e n t D o c u m e n t . f i r s t C h i l d . o f f s e t W i d t h ;  num.links.at.ypos  = new  A rray(pagehcight) ;  / / f i n d rows where t h e r e is more than one l i n k and the same y pos for (j = 0; j < n u m l i n k s ; ++j ) { y = findPosY ( gctBrowscr ( ) . contentDocument. l i n k s , i t e m ( j ) ) ; i f (! n u m . l i n k s . a t . y p o s [ y ] ) { n u m . l i n k s . a t . y p o s [y] = 1; } c1sc { n u m . l i n k s . a t . y p o s [y] + + ; page h e i g h t += g c t B r o w s c r () . c o n t e n t D o c u m e n t . l i n k s . i t e m ( j ) . o f f s e t H e i g h t ; } } / / c r e a t e l i n k s array l i n k s = new A r r a y ( p a g e w i d t h ) ; for (j = 0; j < p a g e w i d t h ; j++){ 1 i n k s [ j ] = new A r r a y ( p a g c h c i g h t ) ; for (k = 0; k < p a g o h e i g h t ; k++){ l i n k s [ j ] [ k] = - 1 ; }  / / S t a r t g e n e r a t i n g the pagemap f i l e c o m p o n c n t l i n k e r . Delete Page M ap ( ) ; c o m p o n c n t l i n k c r . Write PagcMap ( ' < ? xml v e r s i o n ="1.0" c n co d i ng = "UTF—8" ?> \ n \ n '); c o m p o n c n t l i n k e r . W r i t e P a g e M a p ( ' < pagemap>\n ' ) ; var seal cd P a g e H eight = p a g c h c i g h t / t a c t i 1 e _ s c a 1 e f a c t o r ; c o m p o n c n t l i n k e r . W r i t e P a g e M a p (' < h c i g h t > ' + s c al e d P age H e i g h t + ' < / height >\n') ; i = 0; / / i t e r a t e t h r riugh a l l the page l i n k s for ( i = 0; i < n u m l i n k s ; iH—h) { link = getBrowser (). contentDocument. li y = findPosY ( link ) ; x = findPosX ( l i n k ) ; range = 1ink . o f f s e t H e i g h t ; if  /*  if if if if if if  //  item ( i ) ;  ( l i n k , name ) * / iconname = l i n k .name; * / else * / iconname = " b i g s q u ar e . xml " ;  (debug) (debug) (debug) (debug) (debug) (debug)  dump dump dump dump dump dump  ( " L i n k na ("Y = ") ; (y); (" Range (range); ("\n");  e=" + H  var s c a l e d Y = y / t a c t i 1 c _ s c a 1 c f a c t o r ; c o m p o n c n t l i n k e r . W r i t e P a g e M a p (" <icon>\n"); c o m p o n c n t l i n k c r . W r i t e P a g e M a p (" <at>" + s c a l e d Y -f- " < / a t >\n" ) ; com po nen t l i n k c r . W r i t e P a g e M a p (" <range>" + range + " < / r a n ge >\n " ) ; co m p o n e n t 1 i nke r . W r i t e P a g e M a p (" <name>" + iconname + " </namc>\n " ) ; c o m p o n c n t l i n k c r . W r i t e P a g e M a p (" </icon>\n") ; found = f a l s e ; for (j = 0; (j < p a g e w i d t h [found) ; //if t h e r e is a l r e a d y a l i n k at t h i s y , height i f ( l i n k s [ j ] [ y ] >= 0 ) { y = y + range; found = t r u e ;  }  extend  new  l i n k 's  y coord  by  its  Appendix D. Browser Prototype Code } for (k = y; k < y + l i n k s [x] [ k] = i ; }  range;  259  kH—h){  } c o m p o n e n t l i n k e r . Write Page Map // //  ( ' < / pagemap>\n ' ) ;  t r i g g e r r e l o a d from t a c t i l e d e p r e c a t e d , wait u n t i l user  loop starts  the t a s k b e f o r e d o i n g t h i s } / / o t h e r w i s e , assume mouse else { h i g h l i g h t e d = g c t B r o w s c r () . c o n t e n t D o c u m c n t . l i n k s . item ( c u r r e n t ) ; g c t B r o w s c r () . c o n t e n t D o c u m c n t . g c t E l c m c n t B y l d (" h i g h l i g h t d i v " ) . s t y l e . v i s i b i l i t y = " hidden " ; } / / s c a l e the h i g h l i g h t image as needed to r e p r e s e n t the c u r s o r s i z e var i m a g e t a g = "<img s r c ='h i g h 1 ig h t—image . png ' i d =' h i g h 1 i g h t i m a g c ' w i d t h = " ; i m a g c t a g += p a g e w i d t h ; i m a g e t a g += " h c i g h t =" ; // i m a g c t a g += 8 * t a c t i l c . s c a l c f a c t o r ; (without bilinear interpolation) i m a g c t a g -\-= 16 * t a c t i l e . s c a l c f a c t o r ; i m a g e t a g +— " / > " ; g c t B r o w s c r () . c o n t e n t D o c u m c n t . g c t E l c m c n t B y l d ( " h i g h l i g h t d i v " ) . inner H T M L = imagctag ;  if {  ( firstload )  window. o p e n D i a l o g (" c h r o m e : / / w c b s c r o l l e r / c o n t e n t s / r c a d y d i a l o g . x u l " ,"showmorc" , " chrome , modal " ) ; firstload — false; log_resetCount++; l o g . p a g e s = new A r r a y () ; l o g . t i m c s = new A r r a y ( ) ; l o g _ e n d T i m e = 0; if  ( i n p u t . m o d c . i n d e x O f (" s l i d e r ") != — 1) { componentlinker . SetPageMapFlag ( ) ; u p d a t e - f r o m . d e v i c c () ;  } var d a t c O b j = new D a t e ( ) ; 1 o g . s t ar t T i m c = d a t e O b j . g c t T i m e () ;  >  else { var d a t c O b j = new D a t e ( ) ; i f ( i n p u t . m o d c . i n d e x O f (" s l i d e r " ) != —1){ componentlinker . SetPageMapFlagf) ; update.from.device () ; } i f (debug) dump ("Page l o a d e d at " + d a t c O b j . g c t T i m e () }  / / n e e d s f o c u s in o r d e r to p r o c e s s k e y b o a r d g c t B r o w s c r () . c o n t e n t D o c u m c n t . l i n k s . i t e m ( 0 ) . f o c u s () ; //set  bottom  text  bar  +  "\n");  Appendix D. Browser Prototype Code  260  document . g c t E l e m e n t B y l d (" c u r r e n t — l i n k " ) ;  f u n c t i o n goBack ( ) { var w c b N a v i g a t i o n = g e t B r o w s e r () . w c b N a v i g a t i o n ; if ( w c b N a v i g a t i o n . canGoBack) w c b N a v i g a t i o n . go Back () ; } f u n c t i o n goForward ( ) { var w c b N a v i g a t i o n = g c t B r o w s c r ( ) . if (wcbNavigation .canGoForward) w c b N a v i g a t i o n . g o F o r w a r d () ; }  wcbNavigation;  function UpdatcBackForwardButtons() { var b a c k B r o a d c a s t c r = document. g c t E l c m c n t B y l d ("can G o B a c k " ) ; var f o r w a r d B r o a d c a s t e r = document. g c t E l c m c n t B y l d ( " c a n G o F o r w a r d " ) ; var w c b N a v i g a t i o n = g c t B r o w s c r ( ) . w c b N a v i g a t i o n ; var var  b a c k D i s a b l c d = ( b a c k B r o a d c a s t c r . get A t t r i b u t e ( " d i s a b l e d " ) —= " t r u e " ) ; f o r w a r d D i s a b l e d = ( f o r w a r d B r o a d c a s t e r . g c t A t t r i b u t c ( " d i s a b l e d " ) == " t r u e " )  if  ( b a c k D i s a b l c d == w e b N a v i g a t i o n . canGoBack ) b a c k B r o a d c a s t c r . set A t t r i b u t e (" d i s a b l e d " ,  if  ( f o r w a r d D i s a b l e d == w c b N a v i g a t i o n . c a n G o F o r w a r d ) f o r w a r d B r o a d c a s t e r . set A t t r i b u t e (" d i s a b l e d " , ! f o r w a r d D i s a b l e d ) ;  ! backDisablcd) ;  } f u n c t i o n g c t B r o w s c r () { r e t u r n document . g c t E l e m e n t B y l d (" browser —content " ) ; >  f u n c t i o n B r o w s c r C o n t c n t () { r e t u r n g c t B r o w s c r () . c o n t e n t D o c u m c n t ;  / / Implement push —to—s c r o 11 f u n c t i o n f u n c t i o n S c r o l l W indow ( c u r s o r Y P o s ) { // Constants var m a r g i n t o p = 100; / / Scrolling var m a r g i n b o t t o m = m a r g i n t o p ;  begins  when  cursor  hits  margin  / / Local vars var c u r r t o p = g c t B r o w s c r ( ) . c o n t e n t D o c u m c n t . b o d y . s c r o l l T o p ; var c u r r b o t t o m = c u r r t o p + c o n t e n t . i n n c r H c i g h t ; var newtop = c u r r t o p ; //  Code  if {  (cursorYPos <  currtop  +  margintop)  newtop = c u r s o r Y P o s — m a r g i n t o p ; i f (newtop < 0) newtop = 0; } else {  if  (cursorYPos >  currbottom — marginbottom)  newtop = c u r s o r Y P o s -f- m a r g i n b o t t o m — c o n t e n t . i n n c r H c i g h t ; i f (newtop > c o n t e n t , s c r o l l M a x Y ) newtop = c o n t e n t , s c r o l l M a x Y ; >  g c t B r o w s c r () . c o n t e n t D o c u m c n t . body . s c r o l l T o p = newtop ;  Appendix  D. Browser Prototype Code  261  //for input-mode = s l i d e r f u n c t i o n u p d a t c _ f r o m _ d e v i c c () { if  ( in p u t . m o d e . i n d e x O f (" s l i d e r " ) == — 1){ a 1 c r t (" ERROR: method \ ' u p d a t c - f r o m _d c v i c e \ ' s h o u l d not be c a l l e d u n l e s s u s i n g s l i d e r d e v i c e . \ n Set s t r i n g in l i n e 2 of b r o w s e r , js to \ " s l i d e r \ " .") ; return ;  } c  1sc{  click  /* /*  = componentlinkcr.GctClick();  if ( c l i c k h i g h l i g h t e d != n u l l ) { document . g c t E l c m e n t B y l d (" URLBar" ) . i n s e r t l t e m A t (0 , h i g h l i g h t e d , highlighted); * / document . g c t E l c m e n t B y l d (" URLBar" ) . s el e c te d In d c x =; 0; * / myHighlight ( highlighted , " yellow") ; // g°To() ; } c1sc { var var  //  scalcdYPos = componcntlinkcr ,ReadSliderPos(); ypos = s c a l c d Y P o s * t a c t i l e . s c a l e f a c t o r ;  var ypos = Math . round ( pagchcight) ;  /* /* /* /*  (componcntlinkcr. ReadSlidcrPos()  /  100)  *  var canvas = document . g c t E l c m e n t B y l d (" c a n v a s " ) ; * / var ctx = c a n v a s . g c t C o n t c x t ( " 2d") ; * / ctx . f i 1 1 S t y 1 c = " r g b a ( 1 5 0 , 150, 0, 0 . 5 ) " ; * / c t x . f i l l R c c t ( 0 , ypos — 1, p a g e w i d t h , ypos + 1); * / / / move the h i g h l i g h — d i v clement to show the ypos / / TODO: need to a d j u s t to c o r r e s p o n d to t a c t i l e window var c u r s o r t o p = ypos — 4; g c t B r o w s c r () . c o n t e n t D o c u m e n t . g c t E l c m e n t B y l d (" h i g h l i g h t d i v " ) . s t y l e . top= c u r s o r t o p -f-" px " ; ScrollWindow(ypos) ; var var for  /* /* /*  /*  i == 0; found = f a l s e ; ( i ; ( i < p a g e w i d t h && ! found) ; i++){ if ( l i n k s [ i ] [ ypos] >= 0) { / / check i f h i g h l i g h t e d is d i f f e r e n t from p r e v i o u s ; i f s o , p l a y i c o n . . i f ( g c t B r o w s c r () . c o n t e n t D o c u m e n t , l i n k s . i t c m ( l i n k s [ i ] [ y p o s ] ) ! = highlighted) */ { */ componcntlinker. WritcIcon(" l i n k " ) ; */  } */  if  /*  ( h i g h l i g h t e d != n u l l ) { myHighlight (highlighted ,  } else {  null);  //clear  componcntlinkcr. Writclconf" l i n k " ) ;  previous  highlighting  */  } h i g h l i g h t e d = g c t B r o w s c r () . c o n t e n t D o c u m e n t . l i n k s . item ( l i n k s [ i ] [ ypos] ) ; h i g h l i g h t e d , f o c u s () ; m y H i g h l i g h t ( h i g h l i g h t e d , "#f55"); found = t r u e ; } } if  }  (! found && h i g h l i g h t e d != n u l l ) { myHighlight ( highlighted , null ) ; highlighted = null; / / c o m p o n c n t l i n k c r . W r i t c I c o n ( " n u l l ") ;  s c t T i m c o u t ( ' u p d a t e . f r o m . d c v i c c ()  20);  / /TODO — tweak  this  value  Appendix  }  D. Browser Prototype Code  262  }  //for i n p u t . m o d c = mouse function scroll(cvcnt){ var d i r e c t i o n = e v e n t , d e t a i l ; //direction > 0 > UP //direction < 0 > DOWN //if w e ' r e not a l r e a d y at  the  top  or  bottom,  then  scroll  if  ((direction > 0 c u r r e n t < ( m y B r o w s c r . l i n k s , l e n g t h — 1)) || (direction < 0 current > 0)){ myHighlight( highlighted , null) ; / / c l e a r previous highlighting if ( d i r e c t i o n > 0) c u r r e n t — c u r r e n t + 1; else c u r r e n t = c u r r e n t — 1; 0 h i g h l i g h t e d = g e t B r o w s c r ( ) . c o n t e n t D o c u m c n t . l i n k s . item ( c u r r e n t ) ; h i g h l i g h t e d . f o c u s () ; m y H i g h l i g h t ( h i g h l i g h t e d , "#f55"); s c t C u r r L a b c l () ; } e v e n t . p r c v c n t D e f a u l t () ; / / d o n ' t a c t u a l l y s c r o l l the page — f o c u s () w i l l do t h a t for us  f u n c t i o n P u s h B u t t o n H a n d l c r ( event ) var l i n k S c l c c t c d = f a l s e ; if {  ( e v e n t , type button  =  ==  " click")  event.which;  event. p r c v c n t D e f a u l t (); } else {  if  {  ( e v e n t , type  ==  //  cancel  default  event  handler  "kcydown " )  var d a t e O b j = new D a t c ( ) ; if ( S t r i n g . f r o m C h a r C o d c ( even t . key Code ) = " ") { iog_cndTimc = d a t c O b j . g c t T i m c ( ) ; i f (debug) dump ("User s i g n a l l e d task c o m p l e t i o n a t : " + d a t e O b j . g c t T i m e () + "\n") ; StudyLog(); } e l s e i f ( S t r i n g . f r o m C h a r C o d c ( event . keyCode) == "B") g c t B r o w s c r () . h i s t o r y . back() ; / / untested e l s e i f ( S t r i n g . f r o m C h a r C o d c ( even t . key Code ) == "R" ) { if (debug) dump ("Reset a t : " + d a t c O b j . g c t T i m e () -f"\n"); f i r s t l o a d = true; loadPagc_( i n i t . p a g c ) ; } else  {  / / TODO:  linkSclcctcd  test =  for  return  key  press  here  true;  } event . p r e v e n t D c f a u l t () ; / /  cancel  default  event  handler  } if { /* /*  ( linkSelccted ) document . g c t E l c m c n t B y l d (" URLBar" ) . i n s c r t l t c m A t (0 , h i g h l i g h t e d , highlighted) ; */ document . get E l e m e n t By Id (" URLBar" ) . se lec t c d I n d ex = 0; * / var d a t e O b j = new Date () ; i f ( d e b u g ) d u m p ( " L i n k s e l e c t e d at " + d a t c O b j . g c t T i m e () 4- ": " + highlighted + "\n");  Appendix D. Browser Prototype Code myHighlight( highlighted , loadPage_( h i g h l i g h t e d ) ;  263  "yellow");  } }  //  ****** *  function // curr // text  s e t C u r r L a b c l () { = d o c u m e n t . g c t E l e m e n t B y l d (" c u r r e n t — l i n k " ) ; = highlighted;  >  f u n c t i o n my H i g h l i g h t ( node , c o l o r ) { // i f (debug) d u m p ( n o d e ) ; i f ( c o l o r == n u11){ node. r e m o v c A t t r i b u t c (" s t y l e " ) ; } else { node. s c t A t t r i b u t c ( " s t y l e " , " b a c k g r o u n d — c o l o r : } } //not used f u n c t i o n rcmovelmagcs ( ) { var images = g c t B r o w s c r () . c o n t e n t D o c u m c n t . for ( var i = 0 ; i < i m a g e s , l e n g t h ; i -\—h) {  " +  color  +  ";");  getElementsByTagNamc('img')  p a r e n t — images [ i ] . p a r c n t N o d c ; p a r e n t . r c m o v c C h i l d ( images [ i ] ) ; } } function shrinklmagcs ( ) { / / f o r small screen viewing var s c r c c n . h c i g h t = 160; var s c r e e n - w i d t h = 120; var 1 = g e t B r o w s e r () . c o n t e n t D o c u m c n t . g c t E l c m c n t s B y T a g N a m e ( 'img ' ) ; f o r ( v a r i = 0 ; i < 1. l e n g t h ; i++) { i f ( 1 [ i ]. width > sc r ce n -w i d t h ) { 1 [ i ] . h e i g h t *= s c r e e n - h e i g h t /1 [ i ] . w i d t h ; 1 [ i ] . width = scrcen_width; } e l s e i f ( 1 [ i ] . n a t u r a l W i d t h > s c r ee n _ w i d t h ) { var c = s c r e e n - w i d t h / l[i].naturalWidth; 1 [ i ] . h e i g h t = l [ i ] . n a t u r a l H c i g h t * e; 1 [ i ] . width = screen-width; } } } function findPosX(obj) { var c u r l e f t = 0; if (obj . o f f s e t P a r e n t ) { while (obj . o f f s e t P a r e n t ) { c u r l e f t += obj . o f f s e t L e f t obj = ob j . o f f s ct P a r e n t ; }• } else if ( o b j . x ) c u r l e f t += obj . x ; return curleft ; } function findPosY(obj ) { var c u r t o p = 0; if (obj . o f f s e t P a r e n t ) { while (obj . o f f s e t P a r e n t ) { c u r t o p +— obj . o f f s e t T o p  Appendix D. Browser Prototype Code  264  obj = o b j . o f f sc t P a rc n t ; }  } else if (obj.y) c u r t o p += o b j . y ; return curtop;  / / Dump l o g for s t u d y / / TODO: add f i l e o u t p u t would p r o b a b l y be best function StudyLog() { / / log f i l e f o r m a t : / / LOG : subj , b l o c k # , c o n d , t a s k # , t a s k t y p c , task , s t a r t t i m c , endtime , c l a p s e d t i m e , p a g c l a d d r , p a g c l t i m c , pagc2addr , pagc2timc , . . . . / / aka LOG: i n p u t . m o d c , l o g . r c s c t C o u n t , , , l o g . s t a r t T i m c , l o g . c n d T i m c ,( c a l c u l a t e d ) , c o n t e n t s of l o g . p a g c s and l o g . t i t m c s a r r a y s var  logo u t p u t = "LOG:  ";  l o g o u t p u t += nput_mode ; l o g o u t p u t += l o g o u t p u t += o g . r e s c t C o u n t ; l o g o u t p u t 4-= l o g o u t p u t 4-= o g - s t a r t T i m c ; l o g o u t p u t += l o g o u t p u t 4-= l o g - c n d T i m e ; l o g o u t p u t 4-= i f ( l o g . s t a r t T i m c && l o g . c n d T i m c ) {  logoutput  } for ( v a r i = 0 ; i logoutput logoutput logoutput logoutput }  4-= l o g . c n d T i m c  < log.pagcs. length; 4-= " , " ; 4-= l o g - p a g e s [ i ] ; 4-= " , " ; 4~ l o g - t i m e s [ i ] ; =  l o g o u t p u t 4-= " \ n " ; dump ( l o g o u t p u t ) ;  D.2.2  webscroller.css  window background-color :  white ;  .highlight b a c k g r o u n d - c o l o r : #CCFFFF; . normal background —colo r :  white ;  .selected background-color :  D.2.3  log.startTimc ;  red ;  browser, x u l  i++)  {  Appendix D. Browser Prototype Code  <?xml  265  v e r s i o n ="1.0"?>  <?xml — s t y l e s h e e t h r e f =" chrome: / / g l o b a l / s k i n " ty pe=" t e x t / css"?> <?xml— s t y l e s h e e t h r e f - ' c h r o m e : / / n a v i g a t o r / s k i n " t y p c - ' t e x t / c s s " ? > < ? x m l — s t y l e s h e e t h r e f =" f i l e : / / / home / l u k / b r o w s e r — u i / w e b s c r o l l e r / s k i n / w e b s c r o l l e r . ess" type=" t e x t / css"?> < window i d =" b r o w s c r w i n " t i t l e = " T a c t i l e browser" x m 1 n s - ' h 11 p : / / www. m o z i l l a . o r g / k e y m a s t e r / g a t e k e e p e r / t h e r e . is . o n l y . x u l " xmlns : r d f — " h t t p : / / www. w3 . o r g / 1 9 9 9 / 0 2 / 2 2 — r d f — sy n tax — n s#" w i d t h ="240" h e i g h t ="320" t o o l b a r ="yes" s c r o o l b a r s ="yes" o n l o a d = " i n i t B r o w s c r () ;" > < scr ipt  t y pc=" a p p l i c a t i o n / x—j a v a s c r i p t "  src =" browser . j s " / >  <broadcastcrsct i d =" b r o w s c r B r o a d c a s t c r s " > <! <broadcastcr id ="canGoBack" d i s a b l e d =" t r u e " / > <broadcastcr i d =" can Go For war d " d i s a b l e d =" t r u c " / >  >  </broadcastcrsct > < browser  id—"browser —content" t y pc=" c o n t e n t — p r i m a r y " s r c =" about : b l a n k " f l e x = " l " />  </window>  D.2.4 <?xml  readydialog.xul  v e r s i o n = " 1.0" ?>  <?xml— s t y l e s h e e t < d i alo g  h r cf =" c h r o m e : / / g l o b a l / s k i n / g l o b a l . ess"  id = " d o n o t h i n g " t i t l e =" Ready" x m l n s - " h 11 p : / / www. m o z i l l a . o r g / k c y m a s t c r / g a t c k c c p c r / t h c r c . is . o n l y . x u l ' b u t t o n s =" a c c e p t " o n d i a l o g a c c e p t =" r e t u r n doOK ( ) ;" >  <s c r i p t > f u n c t i o n doOK ( ) { return true; } </script > <dcscription  v a l u c = " Ready ?"/>  </dialog >  D. 2.5 #i n c l u d c  type =" t e x t / css"?>  IMyComponent .idl " nsISupports . i d l "  [ s c r i p t a b l c , u u i d ( 0 0 8 c 0 3 0 b - I f 7 1 - 4 b 5 1 - 9 d f 9 - 31915d567103)] i n t e r f a c e IMyComponent : n s I S u p p o r t s { l o n g R e a d S l i d e r Pos ( ) ; l o n g W r i t e l c o n ( in s t r i n g boolean G c t C l i c k ( ) ; boolean boolean  type);  Delete Page M a p ( ) ; WritePageMap ( i n s t r i n g  type) ;  Appendix  boolean  D.2.6  D. Browser Prototype Code  266  S c t P a g c M a p F l a g () :  MyComponent.cpp  # i n c l u d c " My Component . h" #includc <stdio.h> #includc <stdlib.h> #i n c1u d e < f s t r c a m > # i n c l u d e < s t r i n g . h> ^ i n c l u d e < d i r c n t . h> N S _ I M P L _ I S U P P O R T S l ( MyComponcnt , IMyComponent )  / / These must be kept s y n c h r o n i z e d w i t h the t a c t i l e loop's # d c f i n c WORKIN G_D IR " / home/ 1 u k / b r o w se r - w o r ki ng - d i r " # d c f i n e PAGEMAP_FILENAME " pagemap . xml" # d c f i n c PAGEMAP_FLAGFILENAME "newpagemap"  main.cppt!  MyComponcnt : : MyComponcnt ( ) /*  member  initializers  and  constructor  code  */  MyComponent : : ~ MyComponcnt ( ) /*  destructor  code  */  * l o n g R e a d S l i d c r Pos (out l o n g p o s ) ; * / NSJMETHODIMP MyComponcnt :: R c a d S I idc r Pos ( PRInt32 DIR.  *_retval)  pdir = stru ct  std : : s t r i if  (! p d i r ) { pri n t f ( " exi t ( l ) ;  } std : : s t r i n g w h i l e (( pcnt = r c a d d i r ( p d i r ) ) ) { std::string f i l e = pen t — >d_namc ; int length = filc.lcngth(); //filter out . p o s ( p o s i t i o n ) files  } if  only  i f (( s t a r t p o s = f i l e , r f i n d ( " . p o s " , l e n g t h ) ) pos — f i l e . s u b s t r ( 0 , startpos); break ; }  !=  s t d : : s t r i n g : : n pos ) {  (pos == "") { * _ r e t v a l — — 1;  } e1sc { *_retval  =  atoi(pos.c.str());  } closedir //  (pdir);  printf(" Slider return  NS.OK;  post  r e t u r n i n g : %d on  s t r i n g %s \ n " ,  *.rctval ,  pos . c . s t r ( ) ) ;  Appendix  D. Browser Prototype  267  Code  } / * long W r i t c l c o n (in s t r i n g t y p e ) ; * / / / 15-Dcc-05 DEPRECATED!!!! NSJMETHODIMP MyComponcnt W r i t o l c o n ( c o n s t { //delete all existing icon files DIR* p d i r = o p e n d i r (WORKINGJDIR) ; struct dircnt .pent; if  (! p d i r ) { p r i n t f (" o p e n d i r (} ox i t ( 1 ) ;  failure;  char  .type,  PRInt32  *- rctval)  terminating");  } w h i l e (( pcnt = r c a d d i r ( p d i r ) ) ) { s t d : : s t r i n g f i l e = pent —>d_namc ; int length = f i l e , length (); //filter out . i c o n ( i c o n ) f i l e s only i f ( f i l e . r f i n d (". i c o n " , l e n g t h ) =— ( u n s i g n e d //delete fi 1 c char t y p c . s t r [50]; s t r c p y ( t y p c . s t r , WORKINGJDIR) ; strcat ( t y p c . s t r ,"/") ; strcat (typc.str , f i l e , c.str ()); rcmovc( t y p c . s t r ) ; } } closcdir  i n t ) l e n g t h — 5) {  (pdir);  /c/hw a r i t tey pnew c . s t ri c[o5n0 ] ; f i l e strcpy ( typ c.str strcat ( t y p c . s t r WORKINGJDIR) ; s t r c a t ( t y p c . s t r "/") ; strcat (typ c.str type); std : : o f s t r c a m o u t p " . i c o n " ) ; utFilc ; o u t p u t F i l e . o p e n ( t y p c - s t r , std : : o f s t r c a m : : out ) ; / / s t d : : a s s e r t ( o u t p u t F i l e ) ; / / c h e c k for o u t p u t s t r e a m o u t p u t F i l e . c l o s e () ; r e t u r n NS-OK; } / * l o n g G c t C l i c k (out b o o l e a n c l i c k e d ) ; * / NSJMETHODIMP MyComponcnt G c t C l i c k ( PRInt32 { D I R . p d i r = o p e n d i r (WORKINGJDIR) ; struct dircnt .pent; if  (! p d i r ) { p r i n t f (" o p e n d i r () ox i t ( 1 ) ;  failure;  errors  *.rctval)  terminating");  } b o o l found = f a l s e ; w h i l e ( ( pcnt = r c a d d i r ( p d i r ) ) found == f a l s c ) { s t d : : s t r i n g f i l e = pen t — >d_name ; int length = f i l e , length (); //filter out . c l i c k ( c l i c k ) files only i f f f i l e . r f i n d (". c l i c k " , l e n g t h ) = ( unsigned found = t r u e ; //delete file char t y p c . s t r [50]; s t r c p y ( t y p c . s t r ,WORKINGJDIR) ; strcat (typc.str i " / " ); s t r c a t ( t y p c . s t r , f i l e . e s t r () ) ; removef t y p c . s t r ) ; } } * _rctval  — found;  closcdir (pdir); / / p r i n t f ( " Slider r e t u r n NS-OK; }  i n t ) l e n g t h — 6) {  post  r e t u r n i n g : %d\ n " ,  *_retval);  Appendix D. Browser Prototype Code  // //  Page map /  reloading  support  268  routines  / / l o n g D c l c t c P a g c M a p (out b o o l e a n s u cc cs s F 1 ag ) / / Removes the page map f i l e NSJMETHODIMP MyComponcnt :: Delete PageM ap ( P RBool +. r c t v a l ) { c h a r f i l e n a m e [50]; s t r c p y ( f i l e n a m e , WORKING_DIR) ; s t r c a t ( f i l e n a m e ,"/") ; s t r c a t ( f i l e n a m e , PAGEMAP J-1LENAME) ; remove  (filename) ;  * . r c t v a l = true; r e t u r n NS_OK ; }  / / long WritePageMap (in s t r i n g o u t p u t S t r i n g ) / / Appends a l i n e to the PagcMap f i l e NSJMETHODIMP MyComponent : : WritePageMap ( c o n s t c h a r * o u t p u t S t r i n g , P R B o o l * _ r c t v a l ) { / / TODO: s h o u l d check to sec i f the pagemap r e l o a d f l a g is set (i.e., e x i s t e n c e of the file) / / and w a i t u n t i l the f i l e is d e l e t e d ( i . e . , the t a c t i l e l o o p is done l o a d i n g it) before / / c l o b b e r i n g the pagemap f i l e / / w r i t e new i c o n file char f i l e n a m e [50]; s t r c p y ( f i l e n a m e ,WORKING-DIR) ; strcat (filename , " / " ) ; s t r c a t ( f i l e n a m e , PAGEMAP-FILENAME) ; std : : ofstream o u t p u t F i l c ; o u t p u t F i l c . open ( f i l e n a m e , s t d ; : o f s t r e a m : : app) ; // s t d : : a s s e r t ( o u t p u t F i l c ) ; / / c h e c k for o u t p u t s t r e a m outputFilc  «  errors  outputString ;  o u t p u t F i l c . c l o s e () ; * . r e t v a l = true; r e t u r n NS-OK; } / / l o n g Set P a g c M a p F l a g (out b o o l e a n succcssFlag) / / Sets the page map f l a g f i l e , c a u s i n g the t a c t i l e loop to r e l o a d NSJMETHODIMP MyComponcnt : : Set Page M ap F1 ag ( PRBool * . r c t v a l ) { / / w r i t e new i c o n file char f i l e n a m e [ 5 0 ] ; s t r c p y ( f i l e n a m e , WORKING JDIR) ; s t r c a t ( f i l e n a m e ,"/") ; s t r c a t ( f i l e n a m e , PAGEMAP J L A G F I L E N A M E ) ; std : : ofstream o u t p u t F i l c ; o u t p u t F i l e . open ( f i l e n a m e , s t d : : o f s t r e a m : : out ) ; // s t d : : assert ( o u t p u t F i l c ) ; / / c h e c k for o u t p u t stream e r r o r s o u t p u t F i l c . c l o s e () ; * . r c t v a l = true; r e t u r n NS.OK;  the  page  map  269  Appendix E  Browser Experiment Software Code Software that was created to support the browser user evaluation is included here.  T h e a p p l i c a t i o n is based on web standards a n d can r u n i n versions  of Microsoft Internet E x p l o r e r , M o z i l l a Firefox, a n d A p p l e Safari that were current as of this w r i t i n g . T h e files are included as follows. I n the interest of brevity, some files that are s m a l l modifications of the files shown here (for example, for the " t r a i n i n g " cases) are o m i t t e d . • t a s k l o o p . j s - J a v a S c r i p t control logic for the experiment task software. T h e r e are also files for the " t r a i n i n g " cases, w h i c h are nearly identical except for a reduction i n the number of tasks and nonr a n d o m i z e d task presentation order; thus, they have been o m i t t e d . • t a s k l o o p . h t m l - Presentation layer (including embedded C a s c a d i n g Style Sheet) for the experiment task software. • ajaxcomponent. j s  -  L o g i c related to the  AJAX  (Asynchronous  J a v a S c r i p t a n d X M L ) c o m m u n i c a t i o n m e t h o d , w h i c h allows the page to interactively communicate w i t h a n d send d a t a to the server w i t h o u t exposing the user to a page reload. • r e i n f o r c e . j s - L o g i c related to p r o v i d i n g feedback to the user on their performance on the d i s t r a c t i o n task.  Appendix  E.l  E. Browser Experiment Software Code  taskloop.js  / / PC Task program / / ( c ) 2006 J o s e p h // Functionality: / / 1. D i s p l a y the / / 2. D i s p l a y the / / 3. A c c e p t user / / 4 . Produce l o g const  tasksPcrBlock  Luk task to the user distractor task to the i n p u t on the d i s t r a c t o r output  user task  = 6;  var var  cur rent B l o c k N um bcr = 0; cur r c n t T a s k N um bcr = 0;  var  b lo c k G ro u p O r der = new A r r a y ( ) ;  var g r o u p O r d c r = new A r r a y ( ) ; / / a l l possible permutations g r o u p O r d c r [ 0 ] = new A r r a y ( 0 , g r o u p O r d c r [1 ] = new A r r a y ( 0 , g r o u p O r d c r [2] = new A r r a y ( 1 , g r o u p O r d c r [3] = new A r r a y ( 1 , g r o u p O r d c r [4] = new A r r a y ( 2 , grou p O rdcr [ 5 ] = new A r r a y ( 2 , var var var var  270  of 1, 2, 0, 2, 0, 1,  group 2) 1) 2) 0) 1) 0)  order  t a s k C o u n t c r B y G r o u p = new A r r a y ( 0 , 0 , 0 ) ; s c l c c t c d T a s k G r o u p — 0; s c l c c t c d T a s k = 0; c u r rent P cd a l T a s k = 0;  var t a s k s t as ks [ 0 ] t as ks [ 1 ] t as ks [ 2 ]  = = = =  new new new new  A r r a y () ; A r ray ( ) ; A r ray () ; A r ray () ;  tasks[0][0] = ' What i s th c weather in London t o d a y ? " ; the weather be l i k e in London t o m o r r o w ? " ; t a s k s [ 0 ] [ 1 ] = 'What wi the weather be l i k e in London the day a f t e r tomorrow?"; t a s k s [ 0 ] [ 2 ] = ' What w i c weather in P a r i s t o d a y ? " ; t a s k s [ 0 ] [ 3 ] = 'What is the weather be l i k e in P a r i s tomorrow?"; t a s k s [0][4] = ' ' What w i the weather be l i k e in P a r i s the day a f t e r tomorrow?" ; t a s k s [0][5] = ' What w i c weather in Tokyo t o d a y ? " ; t a s k s [0][6] = ' ' What i s the weather be l i k e in Tokyo t o m o r r o w ? " ; t a s k s [ 0 ] [ 7 ] = ' ' What w i the weather be l i k e in Tokyo the day a f t e r tomorrow?"; t a s k s [ 0 ] [ 8 ] = ' ' What w i ' What i s c weather in Hong Kong t o d a y ? " ; tasks [0][9] = ' the weather be l i k e in Hong Kong t o m o r r o w ? " ; t a s k s [ 0 ] [ 1 0 ] = " What w the weather be l i k e in Hong Kong the day a f t e r tomorrow t a s k s [0] [1 1 ] = " What w ?" ; 'What is the weather in San F r a n c i s c o today?"; tasks[0][12] = tomorrow?"; t a s k s [ 0 ] [ 1 3 ] = 'What w i l l the weather be l i k e in San F r a n c i s c o t a s k s [ 0 ] [ 1 4 ] = ' What w i l l the weather be l i k e in San F r a n c i s c o the day a f t e r tomorrow 7 " athcr in T o r o n t o t o d a y ? ' t a s k s [ 0 ] [ 1 5 ] = " What i s the t a s k s [ 0 ] [ 1 6 ] = " What w i the weather be l i k e T o r o n t o tomorrow ?" ; t a s k s [ 0 ] [ 1 7 ] = "What w i ! in T o r o n t o the day a f t e r tomorrow the weather be l i k e v  ,:  ,: ,:  r  If you t a s k s [1] [0] = B road way s t a t i o n t a s k s [1] [ 1 ] = If you G ran vi1b If' you tasks[1][2] = Main ?" ; tasks[1][3] = If you Broadway s t a t i o n t a s k s [ 1 ] [4 ] = If you Granville t a s k s [ 1 ] [5 ] = If' you Main ?" ; tasks[1][6] = If you M acdonald t a s k s [1] [7] = If you Davie?" ;  the  99 B - l i n o  from  UBC at  1pm,  when  will  you  arrive  at  take  the  99 B - l i n c  from  UBC at  1pm,  when  will  you  arrive  at  take  the  99 B - l i n e  from UBC at  1pm,  when  will  you  arrive  at  take  take ? " ;  the  99 B - l i n o  from UBC at  9pm,  when  will  you  arrive  at  take  the  99 B - l i n o  f ro m UBC at  9pm,  when  will  you  arrive  at  take  the  99 B - l i n o  from  9pm ,  take  the  #44  bus  from UBC at  10am,  when  will  you  arrive  at  take  the  #44  bus  from UBC at  10am,  when  will  you  arrive  at  ?" •  UBC at  when  will  you  arrive  at  Appendix  E. Browser Experiment Software Code  271  t a s k s [1] [8] = " I f you t a k e the #44 bus from UBC at 10am, when w i l l you a r r i v e at Waterfront station?"; t a s k s [1][9] = " I f you t a k e the #44 bus from UBC at 5pm, when w i l l you a r r i v e at Macdonald ? " ; t a s k s [1] [10] = " I f you take the #44 bus from UBC at 5pm, when w i l l you a r r i v e at Davie ? " ; t a s k s [1] [11] = " I f you take the #44 bus from UBC at 5pm, when w i l l you a r r i v e at Waterfront station?"; t a s k s [ 2 ] [ 0 ] — " When is the movie &quot ; L&#8217; E n f an t&quo t ; p l a y i n g at the Ridge T h e a t r e ?" ; t a s k s [ 2 ] [ 1 ] = "When is the movie &quot ; L&#8217; E n f a n t & q u o t ; p l a y i n g at F i f t h Avenue C i n e m a s ? " ; t a s k s [ 2 ] [ 2 ] = "When is the movie &quot ; L&#8217; E n f a n t & q u o t ; p l a y i n g at E m p i r e O akr idgc ? " ; t a s k s [ 2 ] [ 3 ] = " What r a t i n g d i d the movie &quot;L&;#8217; E n f a n t&quo t ; r e c e i v e in i t s review in the Los A n g e l e s T i m e s ? " ; t a s k s [ 2 ] [ 4 ] = " What r a t i n g d i d the movie & q u o t ; L & # 8 2 1 7 ; E n f a n t & q u o t ; r e c e i v e in i t s review in the New York P o s t ? " ; t a s k s [ 2 ] [ 5 ] — "When is the movie &quot ; The P r o m i s e & q u o t ; p l a y i n g at the Ridge T h e a t r e ?" ; t a s k s [ 2 ] [ 6 ] = "When is the movie &quot ; The P r o m i s c & q u o t ; p l a y i n g at the P a c i f i c Cinematheque?" ; t a s k s [2] [7] = "When is the movie & q u o t ; T h e P r o m i s c & q u o t ; p l a y i n g at the Silver city Rivcrport?"; t a s k s [ 2 ] [ 8 ] = "What r a t i n g d i d the movie & q u o t ; T h c Promisc&cquot ; r e c e i v e in i t s review in the G l o b e and M a i l ? " ; t a s k s [ 2 ] [ 9 ] = "What r a t i n g d i d the movie & q u o t ; T h e P r o m i s c & q u o t ; r e c e i v e in i t s r e v i e w in the W a s h i n g t o n Post ?" ; var  taskJumblc  = new A r r a y ( ) ;  var p c d a l T a s k = new A r r a y () ; p c d a I T a s k [ 0 ] = "PRESS L E F T PEDAL O N C E " ; p c d a l T a s k [1] = "PRESS L E F T PEDAL T W I C E " ; p c d a l T a s k [2] = "PRESS RIGHT PEDAL O N C E " ; p c d a l T a s k [3] = "PRESS RIGHT PEDAL T W I C E " ; p c d a l T a s k [4] = "PRESS L E F T , THEN RIGHT P E D A L " ; p c d a l T a s k [5] = "PRESS RIGHT, THEN L E F T P E D A L " ; var c u r r c n t P c d a l T a s k ; var p r c v P c d a l T a s k = —1; var p r c v P c d a l C o r r c c t = t r u e ; var, lc f t P cd al C o u n t — 0; var r i g h t P c d a l C o u n t = 0; var var  p c d a l C o r r c c t = 0; p c d a l T o t a l = 0;  var  lastTirncout ;  function init() { document . g c t E l e m e n t B y l d (" t a s k l n f o " ) . s t y l e . v i s i b i l i t y = " h i d d e n " ; document . g c t E l e m e n t B y l d (" mainTask" ) . s t y l e , v i s i b i l i t y = " h i d d e n " ; d o c u m e n t . g c t E l c m c n t B y l d (" p c d a l T a s k " ) . s t y l e , v i s i b i l i t y = " h i d d e n " ; } function StartExptf) { var i =0; var j =0; // //  document . g c t E l e m e n t B y l d (" l o g " ) . v a l u e = " " ; document . get E l e m e n t B y Id (" da t a " ) . va 1 uc = " " ; document document document document  //  . g c t E l c m e n t B y I d ( " mainTask") . s t y l e . v i s i b i l i t y = " v i s i b l e " ; . g c t E l e m e n t B y l d (" p c d a l T a s k " ) . s t y l e , v i s i b i l i t y = " h i d d e n " ; . g c t E l e m e n t B y l d (" s c t u p S c r c c n " ) , s t y l e , v i s i b i l i t y = " h i d d e n " ; . sctupForm . begincxpt . disabled = true ;  document . g c t E l e m e n t B y l d (" mainTask" ) . f o c u s () ; Log  ("Experiment  started.  Subject  #=  " -f d o c u m e n t . s c t u p F o r m .  subject.value);  Appendix E. Browser Experiment Software Code  272  / / r a n d o m i z e the task l i s t s w i t h i n groups for ( i = 0 ; i < t as ks . lo ng t h ; i++) { t a s k j u m b l c [ i ] = new A r r a y ( t a s k s [ i ] . l e n g t h ) ; for ( j = 0 ; j <t asks [ i ] . l e n g t h ; j ++) { taskJumblc[i][j] = j; } } Permute ( t ask J u m blc [ 0 ] ) ; Permutc( taskJumblc [l] ) ; Permute ( t a s k J u m b l c [ 2 ] ) ; // initialize the bloc kG r o u p O r d c r a r r a y var n u m B l o c k s M a x S i x = document. s c t u p F o r m . b l o c k s , v a l u e ; i f ( n u m B l o c k s M a x S i x > 6) num B locksM axSix — 6; for {  (i=0;  i <n um B locksM axS ix ;  i 4-4-)  blockGroupOrder[i] = i ; } / / r a n d o m i z e it Permute ( b l o c k G r o u p O r d c r ) ; var for {  logthis; ( i = 0; i <docu mcnt . s c t u p F o r m . b l o c k s . v a l u e ;  i ++)  l o g t h i s — "Task Groups used in B l o c k #" 4- ( i + l ) + l o g t h i s 4*= G r o u p A s A l p h a ( g r o u p O r d c r [ b l o c k G r o u p O r d c r l o g t h i s 4-= G r o u p A s A l p h a ( g r o u p O r d c r [ b l o c k G r o u p O r d c r l o g t h i s 4-= G r o u p A s A l p h a ( g r o u p O r d c r [ b l o c k G r o u p O r d c r Log(logthis);  ": "; [ i % 6 ] ] [0] ) [ i % 6 ] ] [1] ) [ i %6] ] [2) )  } StartBIock(); //  add  event  handler  to  wait  for  keypress  to  begin  task  f u n c t i o n S t a r t B I o c k () { c u r r e n t B l o c k N umber 4-4-; Log (" "); Log ( " B l o c k #" 4- cu r rent B l o c k N u m bcr ) ; cur r c n t T a s k N u m b e r = 0; S t a r t T a s k () ; } f u n c t i o n EndExpt ( ) { document. g c t E l c m c n t B y I d ( " t a s k l n f o " ) . s t y l e . v i s i b i l i t y — " h i d d e n " document . g c t E l c m e n t B y l d (" mainTask" ) . s t y l e . v i s i b i 1 i t y = " h i d d e n " document . g c t E l c m e n t B y l d (" p c d a l T a s k " ) . s t y l e . v i s i b i l i t y = " h i d d e n A j a x S c n d F o r m (" F I N A L " ) ; // /* /*  d e p r e c a t e d , w e ' r c u s i n g Ajax to send the form a f t e r e v e r y b l o c k now document . output Form . s u bj ec t . v a l uc = "Study l o g r e s u l t s ( F I N A L a f t e r c o m p l e t i n g " + c u r r c n t B l o c k N u m b c r 4- " b l o c k s ) " ; */ document . o u t p u t F o r m . s u b m i t ( ) ; * /  alert  (" Thank  you  return ; }  function  EndBlock()  for  participating  in  this  e x p e r i m e n t . \ n Have a n i c e  day ." ) ;  273  Appendix E. Browser Experiment Software Code {  }  i f ( c u r r c n t B l o c k N u m b c r == document . s e t u p F o r m . b l o c k s . v a l u e ) E n d E x p t () ; else { A j a x S e n d F o r m (" INTERIM" ) ; document . g e t E l e m e n t B y I d ( " t a s k l n f o " ) . s t y l e . v i s i b i l i t y = " h i d d e n " ; document . g c t E l e m e n t B y l d (" m a i n T a s k " ) . s t y l e . v i s i b i l i t y = " h i d d e n " ; document. g e t E l e m e n t B y I d ( " p c d a l T a s k " ) . s t y l e . v i s i b i l i t y = " h i d d e n " ; a l e r t (" Done w i t h b l o c k " + c u r r c n t B l o c k N u m b c r + " • \ n T a k c a one —minute break ." ) ; document . g c t E l e m e n t B y l d (" t a s k l n f o " ) . s t y l e . v i s i b i l i t y = " v i s i b l e " ; document . g c t E l c m c n t B y l d (" m a i n T a s k " ) . s t y l e . v i s i b i l i t y = " v i s i b l e " ; S t a r t B l o c k () ; >  function StartTask() var 1o g t h i s ;  {  d o c u m e n t . g c t E l c m c n t B y l d (" p c d a l T a s k " ) . s t y l e , v i s i b i l i t y  =  "hidden";  currentTaskNumbcr-f-4-; if  (currcntTaskNumber >  tasksPer Block)  {  EndBlock();  return;  }  document . g c t E l e m e n t B y l d (" t a s k l n f o ") . s t y l e . v i s i b i l i t y = " v i s i b l e " ; document . g e t E l c m c n t B y l d (" t a s k l n f o " ) . inner H T M L = " T a s k " -j- c u r r e n t T a s k N u m b e r -j" of " + t a s k s P e r B l o c k ; s c l c c t c d T a s k G r o u p = groupOrdcr [ blockGroupOrdcr [ ( currcntBlockNumbcr -1) c u r r c n t T a s k N u m b e r —1) % 3];  %  6]][(  selcctedTask = taskJumble [ sclectedTaskGroup ] [ taskCountcrByGroup [ s c l c c t c d T a s k G r o u p ] % tasks [sclectedTaskGroup ] . length ] ; taskCounterByGroup [ selectedTaskGroup ] + + ; l o g t h i s = " T a s k #" + currcntTaskNumber; l o g t h i s += " , u s i n g " -f- G r o u p A s A l p h a ( select e d T a s k G roup ) ; l o g t h i s += s e l e c t c d T a s k + ": "; l o g t h i s += t a s k s [ s e l e c t e d T a s k G r o u p ] [ s e l c c t e d T a s k ] ; Log ( l o g t h i s ) ; / / d i s p l a y the task box document . g e t E l e m e n t B y l d (" mainTask" ) . inner H T M L = selcctedTask ] ; //  / / wait for s p a c e b a r press a d d E v c n t L i s t e n c r ( ' key down ' , S t a r t P e d a l T a s k , window . o n k e y p r e s s — S t a r t P e d a l T a s k ;  tasks [ s c l c c t c d T a s k G r o u p ] [  true) ;  } function  S t a r t P e d a l T a s k (e)  { var var var if {  evtobj=window . e v e n t ? event : c / / d i s t i n g u i s h between I E ' s o b j e c t (window, e v e n t ) and F i r e f o x ' s i m p l i c i t , unicode=evtobj.charCodc? evtobj.charCodc : cvtobj.kcyCodc actualkey = String.fromCharCodc( U n i c o d e ) (actualkey  !=  "  explicit  ")  c v t o b j . p r c v c n t D e f a u l t () ; return ; } / / s p a c e b a r pressed , p r o c e e d document . g c t E l c m c n t B y I d ( " p c d a l T a s k " ) . s t y l e . v i s i b i l i t y p r e v P e d a l T a s k = —1; p rev P c d a l C o r r c c t = t r u e ; p c d a l C o r r c c t = 0; p c d a l T o t a l = 0;  = "visible " ;  event  Appendix E. Browser Experiment Software Code  / / wait 3 s e c o n d s b e f o r e lastTimeout = sctTimeout window, o n k c y p r c s s  =  starting the p e d a l t a s k (NcwPcdalTask , 3000);  EndPcdalTask;  c v t o b j . p r c v c n t D e f a u l t () ; function EndPcdalTask(c) { var evtobj=window . e v e n t ? e v e n t : c / / d i s t i n g u i s h between IE ' s o b j e c t (window, e v e n t ) and F i r e fo x ' s implicit, var u n i c o d c = c v t o b j . c h a r C o d c ? cvtobj.charCodc : evtobj.keyCode var a c t u a l k c y = S t r i n g . from C h a r C o d c ( U n i c o d e ) var if {  explicit  event  logthis; (actualkcy  !=  "  ")  cvtobj . prcvcntDefault return ;  () ;  } / / spacebar pressed , proceed clcarTimcout ( lastTimeout) ; window, o n k c y p r c s s = n u l l ; document . onmouscdown = n u l l ; document . g c t E l c m c n t B y l d (" p c d a l T a s k " ) . i n n c r H T M L = " " ; 1ogth logth 1ogth logth logth logth logth D g t h  Data  += document . s c t u p F o r m . s u b j e c t . v a l u e + += c u r r c n t B l o c k N u m b c r + "; += c u r r c n t T a s k N u m b e r 4- " , "; += G r o u p A s A l p h a ( s e l e c t c d T a s k G roup ) + " += s e l c c t e d T a s k + " , "; 4-= p c d a l T o t a l -f- " , " ; i s 4-= p c d a l C o r r c c t ; (logthis);  S t a r t T a s k () ; c v t o b j . p r c v c n t D e f a u l t () ;  f u n c t i o n N c w P c d a l T a s k () { / / choose a p e d a l t a s k task if {  (currcntBlockNumbcr // do { }  cz  pedals,  first  randomly, <  but  one  that  is  different  from  the  current  3) 4  pedal  tasks  only  c u r r c n t P c d a l T a s k = Get Random ( 0 , 3 ) ; w h i l e ( c u r r c n t P c d a l T a s k == p r c v P c d a l T a s k ) ;  } else { do { }  c u r r c n t P c d a l T a s k = Get Random (0 , p c d a l T a s k . l e n g t h — 1) ; w h i l e ( c u r r c n t P c d a l T a s k == p r c v P c d a l T a s k ) ;  }  d o c u m e n t . g c t E l e m e n t B y l d (" p c d a l T a s k " ) . i n n c r H T M L = l c f t P e d a l G o u n t = 0; r i g h t P c d a l C o u n t = 0;  pcdalTask [currcntPcdalTask] ;  274  Appendix E. Browser Experiment Software Code document . oninouscdown = lastTimoout  275  H an d 1 c P c d al s ;  = sctTimcout ( PcdalTimcout ,  7000);  } f u n c t i o n P c d a l T i m c o u t () { var l o g t h i s ; var c o r r e c t = P c d a l s C o r r c c t () ; l o g t h i s = "PEDAL: Task " + c u r r cnt P c d a l T a s k if ( c o r r e c t ) l o g t h i s += "CORRECT, "; e l s e l o g t h i s += "WRONG, "; l o g t h i s += " L c f t = " + l c f t P e d a l C o u n t ; l o g t h i s -f= " , R i g h t =" + r i g h t P c d a l C o u n t ; Log (logthis);  +  ":  ";  pcdalTotal+ +; if ( c o r r e c t ) pcdalCorrect+4; if  ( c o r r e c t ) document . g e t E l c m c n t B y l d (" p c d a l T a s k " ) . s t y l e . b a c k g r o u n d C o l o r = "# fff " ; document . g c t E l c m e n t B y l d ( " p c d a l T a s k " ) . s t y l e . b a c k g r o u n d C o l o r = "# f 5 5 " ;  else  if ( c o r r e c t ) p r c v P e d a l C o r r c c t = else p r c v P e d a l C o r r c c t = f a l s e ;  true;  if ( c o r r e c t ) CorrectPedalRcsponsc(); else WrongPcdalResponse ( ) ; prevPcdalTask = NcwPcdalTask ( ) ;  function { var  currentPedalTask;  P c d a l s C o r r c c t () correct  switch  =  false;  (currentPedalTask)  { case  0: if ( l c f t P e d a l C o u n t break ; 1 : if ( l c f t P e d a l C o u n t break ; 2: if ( l c f t P e d a l C o u n t break ; 3: if ( l c f t P e d a l C o u n t break ; 4: 5: if ( l c f t P e d a l C o u n t break ;  case case case case case } return  ==  1 && r i g h t P c d a l C o u n t  ==  0)  c o r r c c t = truc ;  ==  2 && r i g h t P c d a l C o u n t  ==  0)  corrcct = truc;  ==  0 && r i g h t P c d a l C o u n t  ==  1)  corrcct = true;  ==  0 && r i g h t P c d a l C o u n t  ==  2)  corrcct = truc;  =  1  ==  1)  corrcct = truc;  rightPcdalCount  correct;  f u n c t i o n H a n d l c P c c l a l s (e) { var c v t o b j = w i n d o w . e v e n t ? e v e n t : c / / d i s t i n g u i s h between o b j e c t (window, e v e n t ) and F i r e f o x ' s i m p l i c i t . button  =  cvtobj .which;  switch ( button ) { case 1: lcftPcdalCount+ +;  IE ' s  explicit  event  Appendix E. Browser Experiment Software Code Log (" L e f t P e d a l " ) ; break ; case 2: break ; case 3: rightPcdalCount+4-; Log (" R i g h t P e d a l " ) ; break ; } var c o r r e c t = if (correct) •  P c d a l s C o r r c c t () ;  {  v a r c o n t e n t = ".< font if (prevPcdalCorrcct) c o n t e n t 4-= "#ddd" e l s e c o n t e n t 4-= "#c77 content  4-= " ' >" ;  c o n t e n t 4-= p c d a l T a s k [ c u r r c n t P c d a l T a s k ] ; c o n t e n t -j-— " < / f o n t > " ; document . g c t E l c in c n t B y l d (" p c d a l T a s k " ) . innerHT M L — c o n t e n t ; } c v t o b j . p r c v c n t D e f a u l t () ; } f u n c t i o n GctRandom (low , h i g h ) { r e t u r n •( Math . f l o o r ( M a t h , random () }  * ( h i g h — low 4-1) ) 4-  low);  function F o r m a t T w o D i g i t s (number) { var r c t v a l = " " ; i f (number < 10) r c t v a l = "0"; r c t v a l 4-= number; return rctval; } function F o r m a t T h r c c D i g i t s ( number ) { var r c t v a l = " " ; i f (number < 100) rctva 1 = " 0 " ; i f ( n um bcr < 10) r c t v a l 4-= "0"; r c t v a l 4*= numbcr ; return rctval ; } function Log ( t e x t ) { var myDatc = new Date () ; var l o g o u t p ut = logoutput logoutput logoutput logoutput logoutput lo go u t p u t logoutput logoutput logo u t pu t  4-= myDatc . g c t F u l l Y e a r () . t o S t r i n g () . s u b s t r i n g (2 4-= F o r m a t T w o D i g i t s (myDatc . get M o n t h () +1) ; 4-= F o r m a t T w o D i g i t s (myDatc . g c t D a t e () ) 4- " "; 4-= my Date . get Hours () 44-= F o r m a t T w o D i g i t s ( myDatc . g c t M i n u t c s () ) ; 4-= " : " ; 4-= F o r m a t T w o D i g i t s ( myDatc . g c t S e c o n d s () ) 4- " . " ; += F o r m a t T h r c c D i g i t s (myDatc . g e t M i l l i s c c o n d s () ) 4-= " — ";  l o g o u t p u t += t e x t ; logo u t p u t 4-= " \ n " ; document . gc ; E l c m c n t B y I d ( " l o g " ) . v a l u e 4-= } f u n c t i o n Data ( t e x t ) { var myDatc = new D a t c ( ) ; var l o g o u t p u t = " " ;  logoutput ;  276  Appendix  E. Browser Experiment Software Code  logoutput logoutput logoutput logoutput logoutput logoutput logoutput logoutput logoutput  -f-= myDatc . g c t F u l l Y e a r () . t o S t r i n g () . s u b s t r i n g (2 ,4) ; += F o r m a t T w o D i g i t s ( myDatc . get Mo nth () +1) ; += F o r m a t T w o D i g i ts ( my Date . get D at e ( ) ) + " "; += myDatc . g c t H o u r s ( ) -f += F o r m a t T w o D i g i t s ( myDatc . g c t M i n u t c s () ) ; += += F o r m a t T w o D i g i t s ( myDatc . g c t S c c o n d s ( ) ) + += F o r r a a t T h r c c D i g i t s (myDatc , g c t M i l l i s c c o n d s () ) ; +=  logoutput logoutput  -f— t e x t ; += " \ n " ;  document . g c t E l c m e n t B y l d (" d a t a " ) . v a l u e  f u n c t i o n G r o u p A s A l p h a (number) { i f (number = 0) r e t u r n " A ' i f (number = 1) r e t u r n " B i f (number = 2) r e t u r n " C  +=  277  logoutput ;  , :  f u n c t i o n Permute ( t h c . a r r a y ) { var loop ; var t e m p . a r r a y = new A r r a y () ; for ( l o o p = 0; loop < t h c . a r r a y . l e n g t h ; { t c m p . a r r a y [ loop ] = t h c . a r r a y [ l o o p ] ; }  loop++)  var new_array = new A r r a y () ; var random.num = 0; for ( l o o p = 0; loop < t h c . a r r a y . l e n g t h ; loop-f-f-) { random.num = Math . round (Math . random ( ) * ( t e m p . a r r a y . l e n g t h — 1 ) ) ; n c w . a r r a y [ loop ] = t c m p . a r r a y [ random.num ] ; t c m p . a r r a y [random.num] = t c m p . a r r a y [ t c m p . a r r a y . l e n g t h — 1 ] ; t c m p . a r r a y . length ; } for ( l o o p = 0; l o o p < n c w . a r r a y . l e n g t h ; loop++) { t h c . a r r a y [ loop ] = n c w . a r r a y [ l o o p ] ; } return ;  E.2  taskloop.html  <?xml v e r s i o n =" 1.0" e n c o d i n g - ' iso —8859 — 1" ?> <!DOCTYPE h t m l PUBLIC " - / / W 3 C / / D T D XHTML 1.0 T r a n s i t i o n a 1 / / E N " "http://www.w3.org / T R / x h t m l l / D T D / x h t m l l - t r a n s i t i o n a l . dtd"> <html x m l n s = " h t t p : / / www . w3 . o r g / 1 9 9 9 / x h t m l " > <hcad> < t i t l e > T a c t i l c Browser — P C T a s k < / t i t l c > <mcta h t t p — c q u i v =" Con tent—Type" c o n t e n t =" t e x t / html ; c h a r s e t = i s o —8859 — 1" /> < scr ipt typc="text/javascript" s r c =" r e i n f o r c e . j s " /> < s c r i p t t y pe=" t e x t / j a v a s c r i p t " sr c=" a j a x c o m p o n e n t . j s " /> < sc ript type="text/javascript" src="taskloop.js" /> < s t yIc t ypc=" t c x t / c s s " > <! body {  Appendix E. Browser Experiment Software Code  text—align : center ; m a r g i n - l e f t : 10%; margin —right : 10%; } div.setupScrccn { p a d d i n g : 20px; b o r d e r : l p x s o l i d #666666; position : relative ; t o p : 10 px; b a c k g r o u n d —color : #AACCAA; } d i v . mainTask { f o n t — f a m i l y : A r i al , H e l v e t i c a , sans — s e r i f ; font —size : 36px; font—weight : b o l d ; b a c k g r o u n d - c o l o r : #EEEE77; p a d d i n g : 20px; position: relative; top : Opx; backgrou nd — p o s i t i o n : c e n t e r ; text—align: center; b o r d e r : none #999999; } div.tasklnfo { font—family: A r i a l , H e l v e t i c a , sans-serif; f o n t — s i z e : 12 px ; font—weight : b o l d ; b a c k g r o u n d —c o 1 o r : #ce7 ; p a d d i n g : 5px; position : relative ; top : Opx; background-position : center ; text — a l i g n : center ; b o r d e r : none #999999; c o l o r : #333333; margin : auto ; width: auto; } div . p e d a l T a s k C o n t a i n c r { position : relative ; top : 1 OOpx; } div . p c d a l T a s k { f o n t — f a m i l y : "Times New Roman" , T i m e s , serif; font — s i z e : 48px; p a d d i n g : 20px; b o r d e r : l p x s o l i d #666666; position : relative ; top : Opx; } div . p e d a l R c i n f o r c c { p a d d i n g : lOpx ; b o r d e r : none; position : relative ; t o p : 5 px; margin : auto ; width: auto; } div.logArca { padding: lOpx; b o r d e r : l p x s o l i d #000000; position: absolute; top : lOOOpx; }~> </stylc > </head> <body o n c o n t c x t m c n u - ' r e t u r n f a l s e " o n l o a d =" i n i t ();"> < d i v id =" s e t u p S c r c c n " c 1 a s s =" s e t u p S c r c c n " > <hl>Welcome! < / h l > <form n a m e - ' setup F or m"> <p>S u b j e c t Number: < i n p u t n a m e = " s u b j c c t " t y p c = " t e x t " i d = " s u b j e c t " s i z e ="4" /> </p> <p>Numbcr of B l o c k s : < i n p u t namc=" b l o c k s " t y p c = " t e x t " i d = " b l o c k s " v a l u c = " 9 " s i z c = " 3 " </p>  278  />  Appendix E. Browser Experiment Software Code 279 <p  a l i g n =" c c n t c r " > < i n p u t name=" b c g i n c x p t " E x p e r i m e n t " /> </p> < / fo r m > </div> i d = 'tasklnfo" i d = 'mainTask"  <di v <di v  typc="button"  o n c l i c k = " S t a r t E x p t ( ) ;"  valuc=" Begin  c 1 a s s =" t a s k I n f o ">Task n of n < / d i v > c l a s s =" mainTask" > Ready (main) . < / d i v >  id = ' p c d a l T a s k C o n t a i n c r " c l a s s =" p c d a l T a s k C o n t ai ne r"> < d i v id =" p c d a l T a s k " c l a s s =" p c d a l T a s k " > < / d i v > < d i v i d =" p e d a l R c i n f o r c c " c1 a s s =" p c d a l R c i n f o r c c " X / d i v > </div>  <di v  < d i v i d =" log A r e a " c l a s s =" log A r e a "> <p>&nbsp;</p> <form ac t i o n = " h 11 p : / / c h o b e r i . com / cgi — b i n / Form M a i l . p i " m e t h o d - ' post" namc=" out put F o r m " i d=" output Form"> < i n p u t name=" r e c i p i e n t " ty pc=" h i d d e n " v a l u e = " j o c @ j o s e p h l u k . com" /> < i n p u t namc=" s u b j ec t " t y p c = " h i d d e n " v a l u c = " S t u d y l o g o u t p u t " /> < t c x t a r c a i d = " l o g " name="log" cols="100" rows="10" wrap = "off"> Log c o n t e n t s : </t ex t a r e a > < t c x t a r c a i d = " d at a " n a m e - d a t a " Data c o n t e n t s :  cols="l00"  1  </tcxtarca> <br /> < i n p u t name="Submit" </form> </div>  ty pe=" s u b m i t "  rows="10"  wrap=" off">  />  <p c l a s s —" mainTask" > & n b s p ; < / p> </body> </html>  E.3 // // //  ajaxcomponent.js  aj ax co m po ne n t . js J o s e p h L u k , J u l y 2006 send the i n t e r i m form safety purposes)  function //  data  to  the  server  without  reloading  the  page  (for  AjaxSendForm(typcOfBlock) {  this  line  pretty  much means  you  have  to  run the  script  locally  ••ii  from  file  If need to s i g n the s c r i p t to have i t work over h t t p n e t s c a p e . s e c u r i t y . P r i v i l c g c M a n a g e r c n a b 1 c P r i v i 1 c ge ( ' U n i v c r s a l B ro wscr R e a d ') ; / / use ajax c a l l to send form w i t h o u t r e l o a d i n g the i f ( window . X M LHttpRequest) { req = new X M LHttpRequest ( ) ; } e l s e i f ( window . A c t i v e X O b j e c t ) { req = new A c t i v c X O b j e c t (" M i c r o s o f t .XMLJHTTP" ) ; }  page  req . open (" POST" , " h t t p : / / c h o b e r i . com/cgi— b i n / F o r m M a i l . p i " , t r u e ) ; / / i n l i n e c a l l b a c k f u n c t i o n to wait for s e r v e r response rcq.onreadystatcchange — function() { i f ( r e q . r c a d y S t a t e == 4) {' / / d o n ' t need to do a n y t h i n g ,  just  continue  asynchronously  Appendix E. Browser Experiment Software Code  document . o u t p u t F o r m . s u b j e c t . v a l u e = "Study l o g a f t e r b l o c k " + c u r r c n t B l o c k N u m b e r -f- ")";  results  ("  4- t y p c O f B l o c k  280  +  rcq . s c t R c q u c s t H c a d c r ( " Con ten t—Type " , " a p p l i c a t i o n /x—www—form —u r l c n c o d c d " ) ; rcq . send (for m D a t a 2 Q u c r y S t r i n g ( d o c u m e n t . o u t p u t F o r m ) ) ;  *  Copyright  2005  Matthew  Ecrnisse  (mde@flecgix.org)  * L i c e n s e d under the Apache L i c e n s e , V e r s i o n 2.0 * you may not use t h i s f i l e e x c e p t in c o m p l i a n c e * You may o b t a i n a copy of the L i c e n s e at *  (the with  "License"); the L i c e n s e .  h 11 p : / / www . apache . org / l i c e n s e s / L I C E N S E — 2 .0  * U n l e s s r e q u i r e d by a p p l i c a b l e law or agreed to in w r i t i n g , s o f t w a r e * d i s t r i b u t e d under the L i c e n s e is d i s t r i b u t e d on an "AS IS" B A S I S , * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, e i t h e r e x p r e s s or i m p l i e d . * Sec the L i c e n s e for the s p e c i f i c l a n g u a g e g o v e r n i n g p e r m i s s i o n s and * l i m i t a t i o n s under the License.. * *  O r i g i n a l code by Matthew E c r n i s s e (mdc@flccgix.org) A d d i t i o n a l b u g f i x e s by Mark P r u e t t (mark . p r u e t t @ c o m c a s t . net )  */ //  The var  function  dqcForm s h o u l d  be  a  reference  formData2QucryString(docForm)  to  a <form>  {  var subm it C o n t c n t = ' '; v a r fo r m E l c m ; var l a s t E l c m N a m c = ' '; for  ( i =  0;  i < doc Form . e l e m e n t s . l e n g t h ;  formElcm = docForm . e l e m e n t s [ i ] ; switch ( formElcm . type ) { / / T e x t f i e l d s , h i d d e n form e l e m e n t s case ' t e x t ' : case ' h i d d e n ' : case 'password ' : case ' t c x t a r c a ' : case ' s e l e c t —one ' : su bm i t C o n tent += form Elem . name 4* break ;  i 4-4-)  {  + escape ( f o r m E l c m . v a l u e ) 4-  / / Radio b u t t o n s case ' r a d i o ' : i f ( f o r m E l c m . checked ) { subm i t C o n t c n t 4-= form Elcm . name 4- ' = ' } break ;  4- e s c a p e ( f o r m E l c m . v a l u e ) 4-  // Checkboxes case 'checkbox ' : if ( f o r m E l c m . c h e c k e d ) { / / C o n t i n u i n g m u l t i p l e , same—name c h e c k b o x e s i f ( f o r m E l c m . name == l a s t E l c m N a m c ) { / / S t r i p of end ampersand i f t h e r e is one i f ( s u b m i t C o n t c n t . l a s t I n d e x O f ( ' & ') = s u b m i t C o n t e n t . l e n g t h —1) { s u b m i t C o n t c n t = s u b m i t C o n t c n t . s u b s t r (0 , s u b m i t C o n t c n t . l e n g t h — 1); } / / Append v a l u e as comma—d c 1 i m i t c d s t r i n g s u b m i t C o n t c n t 4-= ' , ' 4- e s c a p e ( formElem . v a l u e ) ; } else { s u b m i t C o n t c n t -H= form E l c m . name + + e s c a pe ( f or m E l c m . v a 1 u c ) ; } s u b m i t C o n t c n t +— '&. ';  Appendix  281  E. Browser Experiment Software Code  l a s t E l e m N a m e — f o r m E l e m . name ; } break ; } } / / Remove t r a i l i n g separator s u b m i t C o n t e n t = s u b m i t C o n t e n t . s u b s t r (0 , s u b m i t C o n t e n t . l e n g t h return submitContent ;  E.4 // var  reinforce, js  Positive  reinforcement  script  p e d a l C o r r e c t R u n == 0; / / run of c o r r e c t p e d a l  if if  filename  for  pedal  task  responses  f u n c t i o n C o r r e c t P e d a l R e s p o n s e () p e d a l C o r r e c t R u n -\—|-; var  — 1);  {  = pedalCorrectRun;  ( p e d a l C o r r e c t R u n > 5) (pedalCorrectRun > 7  filename ="5"; ( p e d a l C o r r e c t R u n % 2)  )  f i 1 e n am e =" goyou " ;  document. g e t E l e m e n t B y l d f " p e d a l R e i n f o r c e " ) . i n n e r H T M L = " <img pedalCorrectRun + " - g i f ' />"; return; } f u n c t i o n W r o n g P e d a l R e s p o n s e () { p e d a l C o r r e c t R u n — 0; document . g e t E l e m e n t B y l d (" p e d a l R e i n f o r c e " ) . i n n e r H T M L = return ;  "" ;  src = "'  +  

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share