Open Collections

UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

The annotators' perspective on co-authoring with structured annotations Htun, Yamin 2007

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata

Download

Media
831-ubc_2007-0437.pdf [ 8.59MB ]
Metadata
JSON: 831-1.0052060.json
JSON-LD: 831-1.0052060-ld.json
RDF/XML (Pretty): 831-1.0052060-rdf.xml
RDF/JSON: 831-1.0052060-rdf.json
Turtle: 831-1.0052060-turtle.txt
N-Triples: 831-1.0052060-rdf-ntriples.txt
Original Record: 831-1.0052060-source.json
Full Text
831-1.0052060-fulltext.txt
Citation
831-1.0052060.ris

Full Text

The Annotators ' Perspective on Co-authoring with Structured Annotations by Yamin Htun B.Sc, Texas Christian University, 2005 A THESIS SUBMITTED IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE DEGREE OF Master of Science in The Faculty of Graduate Studies (Computer Science) The University of British Columbia September 2007 © Yamin Htun 2007 11 Abstract In asynchronous collaborative writing, annotations play an important role as a communication medium among co-authors. Research has shown that grouping related annotations together can help those who review an anno-tated document by reducing their workload and raising the accuracy of their reviewing. Less is known about the impact on users who create such struc-tured annotations — the annotators. The research reported in this thesis had three goals: (1) to better understand current annotation creation prac-tices, (2) to explore how structuring would be used by annotators, both the structuring process and the resulting types of structure, and (3) to evaluate the impact on annotators of having to create structured annotations. We conducted three studies to address each of these goals in turn. The first study was an observational study which strengthened our understanding of how annotators use existing tools to communicate document-related infor-mation in the form of annotations. That study revealed annotation practices that could benefit from additional structuring support, such as annotators describing how annotations in a document relate to each other. Our second study used a paper prototype system that supported annotation grouping to investigate how annotators would structure annotations, if given the option. Common behaviour that emerged was the grouping of thematically related annotations, as well as the grouping of annotations specifically targeted to a given co-author. The study also uncovered a range of temporal approaches to structuring annotations, such as top-down and bottom-up grouping. Based on the first two studies, we incorporated a light-weight implicit structuring approach based on tagging into our annotation model and then implemented an extended version of a high-fidelity prototype that supports structured an-notation, including tagging. We used the prototype in our third study, a controlled experiment, which compared the impact of structured annotations relative to unstructured annotations. Participants in that study perceived structured annotations to be worth the additional workload required. The study further suggested that the bottom-up grouping approach complements the top-down approach in describing relationships amongst annotations in a document. Contents Abstract ii Contents iii List of Tables vi List of Figures vii Acknowledgements viii 1 Introduction 1 1.1 Research Motivation 1 1.2 Research Goals 2 1.3 Research Approach 3 1.4 Research Contributions 4 1.5 Overview 5 2 Related Work 6 2.1 Collaborative Writing 6 2.2 Annotation Support in Co-authoring Contexts 8 2.3 Information Organization Approaches 10 3 Observational Study to Understand the Annotating Process 13 3.1 Study Goals 13 3.2 Methodology 13 3.3 Results 14 3.3.1 Multiple Passes 14 3.3.2 Justifications 15 3.3.3 Local versus Global Comments 16 3.3.4 Tagging-like Behaviors 16 3.3.5 Prioritization 17 3.3.6 Reviewing Methods 17 Contents i v 3.3.7 O t h e r C o m m u n i c a t i o n M e t h o d s 18 3.4 S u m m a r y 19 4 Paper Prototype Study to Understand Annotation Struc-turing 22 4.1 S t u d y G o a l s : 22 4.2 M e t h o d o l o g y 22 4.2.1 P a r t i c i p a n t s 23 4.2.2 T a s k 23 4.2.3 P r o c e d u r e 25 4.3 R e s u l t s 26 4.3.1 T e m p o r a l A p p r o a c h e s for C r e a t i n g G r o u p s 26 4.3.2 S e m a n t i c s o f A n n o t a t i o n G r o u p s 27 4.3.3 S t r u c t u r e s o f G r o u p s 28 4.3.4 S t r o n g S u p p o r t for S t r u c t u r i n g 29 4.4 S u m m a r y a n d A d d i t i o n a l C o m m e n t s 29 5 Incorporating Tagging into Structured Annotations 31 5.1 I m p l i c a t i o n s a n d M o t i v a t i o n s for G r o u p i n g a n d T a g g i n g . . . 31 5.2 I n t e g r a t i n g S t r u c t u r i n g a n d T a g g i n g i n t o A n n o t a t i o n s . . . . 32 5.3 E x t e n d e d B u n d l e E d i t o r 34 6 Experiment to Examine the Impact of Structuring Annota-tions 37 6.1 S t u d y G o a l s 37 6.2 M e t h o d o l o g y 39 6.2.1 P a r t i c i p a n t s 40 6.2.2 T a s k 41 6.2.3 P r o c e d u r e 42 6.2.4 D e s i g n 43 6.2.5 M e a s u r e s 43 6.2.6 H y p o t h e s e s 44 6.3 R e s u l t s 45 6.3.1 C o m m u n i c a t i o n 45 6.3.2 Self-assessed W o r k l o a d 47 6.3.3 C o s t - B e n e f i t 50 6.3.4 S u m m a r y o f R e s u l t s 52 6.3.5 O t h e r M e a s u r e s : U s a g e o f S t r u c t u r e s 54 6.3.6 A d d i t i o n a l Q u a l i t a t i v e F e e d b a c k 57 6.4 D i s c u s s i o n 58 Contents v 7 Conclusions and Future Work 60 7.1 Limitations 60 7.2 Future Work 61 7.2.1 Streamlining the Additional Workload 61 7.2.2 Evaluating Structured Annotations in Iterative Col-laboration 62 7.2.3 Incorporating Structured Annotations into E-mail . . 62 Bibliography 63 A Observational Study Resources 68 A . l Consent Form 68 A.2 Study Document 70 A. 3 Interview Questions 75 B Paper Prototype Study Resources 78 B. l Consent Form 78 B.2 Demographic Questionnaire 80 B.3 Training Task 82 B.4 Instructions to Participants 82 B. 5 Study Document with Annotations 83 C Experiment Study Resources 90 C. l Consent Form 90 C.2 Questionnaires 92 C.2.1 Demographic Questionnaire 92 C.2.2 System-specific Questionnaire: Bundle System . . . . 95 C.2.3 System-specific Questionnaire: Simple System 99 C.2.4 Comparative Questionnaire 103 C.3 Study Documents 105 C.3.1 Training Document 105 C.3.2 Task Documents 107 C.3.3 Manipulations Planted in the Task Documents . . . . 110 C.4 Tasks 116 C.4.1 Training Tasks 116 C.4.2 Experiment Tasks 120 D U B C Research Ethics Board Certificate 122 v i List of Tables 2.1 C o l l a b o r a t i v e w r i t i n g w o r k m o d e s 7 3.1 E x a m p l e s o f a n n o t a t i o n s c r e a t e d b y p a r t i c i p a n t s u s i n g M i -crosoft W o r d 20 3.2 E x a m p l e s o f a n n o t a t i o n s c r e a t e d b y p a r t i c i p a n t s u s i n g a p e n o n a d o c u m e n t p r i n t e d o n p a p e r 21 4.1 T e m p o r a l p a t t e r n s o f s t r u c t u r i n g a n n o t a t i o n s . ( N = 8 ) 27 5.1 A n n o t a t i o n m o d e l 33 6.1 C o m p a r i s o n o f B u n d l e s y s t e m a n d S i m p l e s y s t e m 40 6.2 P r o b l e m s p l a n t e d i n e a c h ta sk d o c u m e n t 42 6.3 M e a n responses t o s e l f - repor ted m e a s u r e s 50 6.4 F a c t o r s used for m e a s u r i n g cost a n d benef i t a n d t h e i r C r o n -bach ' s a l p h a va lues 51 6.5 D e s c r i p t i v e s u m m a r y o f b u n d l e s a n d tags ( N = 1 2 ) 57 List of Figures 4.1 A p a r t i c i p a n t p e r f o r m i n g the t a s k w i t h the p a p e r p r o t o t y p e . 24 4.2 S a m p l e pi les o f a n n o t a t i o n g r o u p s 25 4.3 S a m p l e a n n o t a t i o n g r o u p s 28 5.1 B u n d l e E d i t o r 34 5.2 F i l t e r i n g f u n c t i o n 36 5.3 C o m m e n t d i a l o g . 36 6.1 T h e B u n d l e E d i t o r u s e d i n the E x p e r i m e n t 38 6.2 T h e S i m p l e E d i t o r u s e d i n the E x p e r i m e n t 39 6.3 B o x - p l o t d i s p l a y i n g t h e n u m b e r o f s ingle a n n o t a t i o n s ( N = 1 2 ) . 46 6.4 B o x - p l o t d i s p l a y i n g the n u m b e r o f m e t a - c o m m e n t s ( N = 1 2 ) . . 48 6.5 O v e r a l l w o r k l o a d a s s o c i a t e d w i t h e a c h s y s t e m ( N = 1 2 ) 49 6.6 B o x - p l o t o f costs a n d benef i ts m e a s u r e s for e a c h s y s t e m . ( N = 1 2 ) 53 6.7 A l l b u n d l e s c r e a t e d b y 11 p a r t i c i p a n t s 55 6.8 B u n d l e s as a f u n c t i o n of n u m b e r o f u n i q u e i n h e r i t e d tags a n d n u m b e r o f a n n o t a t i o n s 58 V l l l Acknowledgements F i r s t , I w o u l d l ike to t h a n k m y s u p e r v i s o r s , D r . J o a n n a M c G r e n e r e a n d D r . K e l l o g g B o o t h . T h i s w o r k w o u l d n o t h a v e b e e n p o s s i b l e w i t h o u t t h e i r g u i d a n c e a n d s u p p o r t . I a m g r a t e f u l to J o a n n a for a l w a y s p r o v i d i n g d e t a i l e d a n d t h o r o u g h feedback . H e r u n d e r s t a n d i n g a n d p e r s o n a l g u i d a n c e has b e e n o f great v a l u e for m e . I feel v e r y f o r t u n a t e a n d h o n o r e d to h a v e h a d K e l l y as a c o - s u p e r v i s o r . H i s w i d e k n o w l e d g e a n d great ideas h a v e h a d a r e m a r k a b l e in f luence o n th i s w o r k . I a m also g r a t e f u l to m y s e c o n d r e a d e r D r . S teve W o l f m a n for h is i n s i g h t f u l f eedback . I w o u l d l ike to a c k n o w l e d g e m y peers a n d f r i ends for t h e i r f eedback a n d s u p p o r t . I n p a r t i c u l a r , I w o u l d l ike to t h a n k C l a r e n c e a n d Z e p h y r , w h o were w i l l i n g to l e n d a n ear a n d offer a d v i c e . I w o u l d a lso l ike t o express m y g r a t i t u d e t o M y o , H l a i n g a n d M a r k , whose presence a n d f r i e n d s h i p h e l p e d m a k e m y life o u t s i d e o f w o r k t r u l y en joyab le . M y o deserves s p e c i a l t h a n k s for a l w a y s b e i n g there for m e , a n d for b e i n g e s p e c i a l l y u n d e r s t a n d i n g d u r i n g the pas t few m o n t h s . L a s t , ,most c e r t a i n l y n o t least , m y t h a n k s go to m y f a m i l y for t h e i r c o n s t a n t e n c o u r a g e m e n t a n d u n c o n d i t i o n a l love. I a m v e r y g r a t e f u l to m y p a r e n t s , w h o have a l w a y s b e l i e v e d i n m e , a n d i n s p i r e d a l l m y a c h i e v e m e n t s . It is t o t h e m t h a t I d e d i c a t e th i s w o r k . T h e r e search r e p o r t e d i n th i s thesis was s u p p o r t e d b y T h e U n i v e r s i t y o f B r i t i s h C o l u m b i a a n d b y the N a t u r a l Sc iences a n d E n g i n e e r i n g R e s e a r c h C o u n c i l o f C a n a d a u n d e r the D i s c o v e r y G r a n t s p r o g r a m a n d t h e S t r a t e g i c N e t w o r k G r a n t s p r o g r a m as a c o m p o n e n t o f N E C T A R , t h e N e t w o r k for E f f e c t i v e C o l l a b o r a t i o n T e c h n o l o g i e s t h r o u g h A d v a n c e d R e s e a r c h . 1 Chapter 1 Introduction 1.1 Research Motivation Collaborative writing refers to the process of two or more people working together to create a document. In most collaborative writing tasks, co-authors typically annotate drafts and pass the annotated documents back and forth. Annotations play an important role as a central communication medium. Most word processing systems, however, only support simple anno-tations (basic edits and comments), forcing valuable communication among group members to take place outside the shared document, most often in the bodies of emails to which the document is attached and sent between co-authors. This results in communication being disconnected from the doc-ument, causing unnecessary overhead and inefficiencies [8]. For example, co-authors often need to provide explicit navigation statements such as "see page 2, paragraph 3," or they need to copy and paste some referenced text from the document into e-mail messages. This separation of artifacts means that valuable information can easily get disregarded or misplaced. These difficulties can increase dramatically with only a few reviewing cycles. To address the shortcomings with current annotation tools, Zheng et al. developed an annotation model that unifies all document-related com-munication: single annotations are anchored at a specific place in the doc-ument, general comments are essentially anchored to the whole document, and structured annotations are a grouping of one or more single annotations or general comments [43]. Structured annotations may have hierarchical structure (groups within groups). The structuring is intended to commu-nicate meta-information, i.e., act as meta-comments, about the group of annotations (e.g., summaries of edits). Zheng et al. evaluated the effects Chapter 1. Introduction 2 of structured annotations on users reviewing an annotated document (i.e., the "recipients"), and found efficiency and accuracy benefits when compared to unstructured meta-comments written in e-mail messages. The effects of structured annotations on "annotators," those who create the annotations, had not yet been explored, and is the focus of this thesis. 1.2 Research Goals Our target population is distributed groups collaborating asynchronously during the editing and reviewing stages of co-authoring, and creating a large volume of annotations to communicate document-related information. Be-fore assessing the impact of working with structured annotations on annota-tors, it was fundamental and essential to our research efforts to first attain a better understanding of annotators' work practices with existing tools, in-cluding pen-on-paper markup. We wanted to examine how annotations are used, both the annotating process and the resulting types of annotations to communicate document-related issues. As an initial investigation of structured annotations' usability, we next had to understand how annotators would go about structuring annotations if structure was provided to them as an option. We wanted to explore how annotators would use structure, for example, to thematically group related annotations, or perhaps to group annotations that were specifically targeted to a given co-author. Additionally, we wondered whether annotators would create complex hierarchical structures or non-hierarchical "flat" structures, and how much complexity the additional structures would add to annotating tasks. In terms of creating structured annotations, Zheng et al.'s work assumed a relatively heavy-weight approach, where users would create explicit anno-tation groups called "bundles" [43]. Bundles make structure explicit and are especially helpful in creating annotation groups associated with precon-ceived structures and categories. This has been called top-down processing in the literature [18]. The recent explosion of interest in tagging systems [22], however, suggests that a more light-weight approach to information or-Chapter 1. Introduction 3 ganization might be preferable or at least complementary. Tags allow groups to be formed more implicitly and are helpful in bottom-up processing where structure emerges rather than having to be pre-defined. We decided to see if tagging could be an appropriate way to structure annotations, allowing for more light-weight structuring than other methods. Assuming that users were going to structure their annotations to com-municate meta-information about annotations, we wondered how this would impact the overall amount of information communicated among co-authors. We wanted to investigate whether users who had a tool for structuring an-notations would create more meta-comments, our assumption being that creating structured annotations would be an easier way to provide meta-comments than doing so separately in the body of an e-mail. In addition to the amount of communication, the impact of structured annotations on annotators' workload had to be explored. It was not known whether the overall workload for structuring annotations would be similar to users providing unstructured annotations and then having to compose a detailed e-mail with the equivalent meta-information. We also had to in-vestigate whether annotators would perceive structured annotations to be worth the effort. It was crucial to investigate the costs (effort) and bene-fits tradeoff associated with structured annotations for annotators because, as Grudin noted, tools that have high costs will, not be used unless those perceived costs are balanced by high perceived benefits to the people doing additional work [14]. 1.3 Research Approach In order to address the above research goals, we conducted our research in three phases. First, we conducted an observational study to better under-stand annotators' workflow with existing annotation tools, none of which provide any explicit support for structuring annotations. In particular, we observed the use of traditional pen-on-paper markup and the use of M i -crosoft Word, a popular commercial word processor. In the second phase, we sought to identify different structuring approaches, and the types of Chapter 1. Introduction 4 s t r u c t u r e s t h a t a n n o t a t o r s w o u l d use. W e e x p l o r e d these issues i n d e p e n d e n t o f the too l s t o create s t r u c t u r e d r e l a t i o n s h i p s ( a n d t h e i r p o t e n t i a l u s a b i l i t y issues) b y c o n d u c t i n g a n e x p l o r a t o r y s t u d y w i t h a p a p e r p r o t o t y p e . B a s e d o n the first two s tudies , we t h e n e x t e n d e d a n ear l ier p r o t o t y p e d e s i g n e d to s t r u c t u r e a n n o t a t i o n s [43] b y i n c o r p o r a t i n g b o t t o m - u p s t r u c t u r i n g (i.e., t a g g i n g ) . L a s t l y , we c o n d u c t e d a f o r m a l e x p e r i m e n t i n w h i c h t h e p r o t o t y p e s y s t e m was c o m p a r e d t o a n e q u i v a l e n t s y s t e m t h a t d i d n o t s u p p o r t s t r u c -t u r e , a n d i n v e s t i g a t e d the i m p a c t o f s t r u c t u r e d a n n o t a t i o n s o n w o r k l o a d a n d t h e a m o u n t o f i n f o r m a t i o n c o m m u n i c a t e d . T h e O b s e r v a t i o n a l S t u d y a n d t h e E x p e r i m e n t were c o n d u c t e d w i t h w r i t -i n g t u t o r s at T h e U n i v e r s i t y o f B r i t i s h C o l u m b i a w h o p r o f e s s i o n a l l y a n n o -ta te d o c u m e n t s t o h e l p s t u d e n t s w i t h t h e i r w r i t i n g ski l l s . H a v i n g r e v i e w e d a n d a n n o t a t e d n u m e r o u s d o c u m e n t s o f v a r i o u s l e n g t h s a n d t y p e s , t h e y h a v e e x p e r i e n c e c o m m u n i c a t i n g d o c u m e n t - r e l a t e d i n f o r m a t i o n . W e u s e d expe -r i e n c e d a n n o t a t o r s b e c a u s e we w a n t e d t o u n d e r s t a n d h o w s t r u c t u r i n g a n -n o t a t i o n s w o u l d b e u s e d to a d d r e s s different t y p e s o f e r r o r s (e.g., s y n t a x , s e m a n t i c s ) ; less e x p e r i e n c e d w r i t e r s t e n d to focus o n l y o n s y n t a c t i c errors w h e n r e v i e w i n g d o c u m e n t s [24]. F o r t h e P a p e r P r o t o t y p e S t u d y , g r a d u a t e s t u d e n t s were u s e d for r e c r u i t i n g efficiency. I n t h a t s t u d y , a n n o t a t i o n s d i d n o t n e e d to be c r e a t e d , o n l y o r g a n i z e d ; t h e leve l o f e x p e r i e n c e r e q u i r e d was r e l a t i v e l y lower t h a n i n o u r o t h e r two s tudies . 1.4 Research Contributions T h i s thes is d o c u m e n t s w o r k d o n e to e x a m i n e the effects o f s u p p o r t i n g s t r u c -t u r e o n users w h o create a n n o t a t i o n s . F r o m the O b s e r v a t i o n a l S t u d y , we h a v e s t r e n g t h e n e d o u r u n d e r s t a n d i n g o f h o w a n n o t a t o r s use e x i s t i n g too l s ( b o t h d i g i t a l a n d p e n - o n - p a p e r m a r k u p ) to c o m m u n i c a t e d o c u m e n t - r e l a t e d i n f o r m a t i o n i n the f o r m of a n n o t a t i o n s . W e h a v e also ident i f i ed c o m m o n uses o f s t r u c t u r e d a n n o t a t i o n s a n d different a p p r o a c h e s t o s t r u c t u r i n g a n -n o t a t i o n s t h r o u g h the P a p e r P r o t o t y p e S t u d y . O u r c o n t r i b u t i o n s i n c l u d e a l i gh t -we ight i m p l i c i t s t r u c t u r i n g a p p r o a c h b a s e d o n t a g g i n g , a n d a n ex-t e n d e d v e r s i o n o f a n i n t e r a c t i v e p r o t o t y p e t h a t s u p p o r t s s t r u c t u r i n g a n n o -Chapter 1. Introduction 5 t a t i o n s . W i t h t h e p r o t o t y p e , we e v a l u a t e d the use o f s t r u c t u r e d a n n o t a t i o n s u p p o r t for a n n o t a t i n g d o c u m e n t s a n d c o m m u n i c a t i n g d o c u m e n t - r e l a t e d is-sues. F r o m the E x p e r i m e n t , we s h o w e d t h a t s t r u c t u r e d a n n o t a t i o n s are p e r c e i v e d to be w o r t h the effort de sp i t e the a d d i t i o n a l w o r k l o a d , a n d t h a t b o t t o m - u p a n d t o p - d o w n a p p r o a c h e s to s t r u c t u r i n g a n n o t a t i o n s are c o m p l e -m e n t a r y so b o t h s h o u l d b e s u p p o r t e d . 1.5 Overview T h i s thes i s c o m p r i s e s d e s c r i p t i o n s o f t h r e e s tud ies t h a t were d e s i g n e d to a d -dress the re sear ch goals d e s c r i b e d ear l ier i n th i s c h a p t e r . P r e v i o u s w o r k rele-v a n t to th i s r e sear ch is s u m m a r i z e d i n C h a p t e r 2. I n C h a p t e r 3, we d e s c r i b e t h e O b s e r v a t i o n a l S t u d y , w h i c h was d e s i g n e d to b e t t e r u n d e r s t a n d a n n o -t a t i o n c r e a t i o n prac t i ce s . C h a p t e r 4 descr ibes t h e P a p e r P r o t o t y p e S t u d y , w h i c h was o u r i n i t i a l i n v e s t i g a t i o n o f h o w users m i g h t go a b o u t s t r u c t u r i n g a n n o t a t i o n s . C h a p t e r 5 s u m m a r i z e s the k e y f ind ings a n d i m p l i c a t i o n s t h a t we d e r i v e d f r o m t h e first two s tudies , a n d discusses m o t i v a t i o n s for s t r u c -t u r i n g a n d t a g g i n g . It t h e n descr ibes a n e x t e n d e d v e r s i o n o f a h i g h f ide l i ty p r o t o t y p e t h a t s u p p o r t s s t r u c t u r e d a n n o t a t i o n s i n c l u d i n g t a g g i n g . C h a p t e r 6 presents the E x p e r i m e n t , w h i c h e x a m i n e d the i m p a c t o f s t r u c t u r e d a n n o -t a t i o n s o n w o r k l o a d a n d the a m o u n t o f i n f o r m a t i o n c o m m u n i c a t e d . F i n a l l y , C h a p t e r 7 s u m m a r i z e s t h e m a i n resul t s i n t h e thesis a n d discusses severa l areas for f u t u r e re search . S u b s t a n t i a l p o r t i o n s o f th i s thes is a p p e a r i n a conference p a p e r s u b m i s -s i o n j o i n t l y a u t h o r e d w i t h m y s u p e r v i s o r s , D r . J o a n n a M c G r e n e r e a n d D r . K e l l o g g B o o t h . Chapter 2 Related Work 6 I n t h i s c h a p t e r , we first d e s c r i b e g e n e r a l b a c k g r o u n d o n c o l l a b o r a t i v e w r i t -i n g re search . W e t h e n p r o v i d e a m o r e f o c u s e d d i s c u s s i o n o f r e search o n d o c u m e n t - r e l a t e d c o m m u n i c a t i o n i n the f o r m of a n n o t a t i o n s a m o n g co-a u t h o r s , a n d we s u r v e y s y s t e m s t h a t p r o v i d e s u p p o r t for s t r u c t u r i n g a n -n o t a t i o n s . O u r p r o j e c t is b a s e d o n o r g a n i z i n g a n n o t a t i o n s ; hence , we also h i g h l i g h t k e y a p p r o a c h e s to i n f o r m a t i o n o r g a n i z a t i o n , w i t h a focus o n t a g -g i n g . 2.1 Collaborative Writing C o l l a b o r a t i v e w r i t i n g refers t o the process o f two o r m o r e p e o p l e w o r k i n g t o g e t h e r to create a d o c u m e n t . C o l l a b o r a t i v e w r i t i n g is a c o m m o n p r a c -t ice; r e search has s h o w n t h a t m o r e a c c u r a t e a n d b e t t e r q u a l i t y d o c u m e n t s c a n be a c h i e v e d w i t h the co l l ec t ive k n o w l e d g e a n d e x p e r t i s e f r o m c o - a u t h o r s [10, 26]. A s p a r t o f c o l l a b o r a t i v e w o r k , c o l l a b o r a t i v e w r i t i n g c a n b e s u m m a -r i z e d i n four m o d e s b a s e d o n the degree o f p r o x i m i t y , i.e., w h e t h e r m e m b e r s w o r k f r o m the s a m e l o c a t i o n , a n d t h e degree o f s y n c h r o n i c i t y , i.e., w h e t h e r m e m b e r s w o r k o n the d o c u m e n t at t h e s a m e t i m e . T a b l e 2.1 i l lus tra te s t h e four m o d e s o f c o l l a b o r a t i v e w r i t i n g . R e s e a r c h e r s h a v e i n v e s t i g a t e d t h e o v e r a l l c o - a u t h o r i n g proces s [29, 31], a n d ident i f i ed different ac t iv i t i e s i n v o l v e d i n c o - a u t h o r i n g : b r a i n s t o r m i n g , r e s e a r c h i n g , p l a n n i n g , w r i t i n g , e d i t i n g a n d r e v i e w i n g [29]. M a n y c o l l a b o r a -t ive w r i t i n g too l s h a v e b e e n d e v e l o p e d to s u p p o r t these di f ferent ac t iv i t i e s . C l a s s i c c o l l a b o r a t i v e w r i t i n g s y s t e m s a p p e a r i n g i n the r e s e a r c h l i t e r a t u r e i n c l u d e S A S S E [6], P R E P [25], a n d Q u i l t [12]. M o r e de ta i l s a b o u t these a n d o t h e r c o - a u t h o r i n g s y s t e m s c a n be f o u n d i n [29]. W i t h i n c r e a s i n g access ib i l -Chapter 2. Related Work 7 . Same Location Proximity Synchr Same Time onicity Different Time F a c e t o F a c e A s y n c h r o n o u s S a m e - P l a c e Different Location S y n c h r o n o u s D i s t r i b u t e d A s y n c h r o n o u s D i s t r i b u t e d T a b l e 2.1: C o l l a b o r a t i v e w r i t i n g w o r k m o d e s ( a d a p t e d f r o m E l l i s et a l . [11]) i t y a n d pervas iveness o f t h e In terne t , c o m m e r c i a l w e b - b a s e d c o l l a b o r a t i v e a u t h o r i n g too l s h a v e also b e e n d e v e l o p e d s u c h as: C o l l a b o r a t u s [21], W r i t e -B o a r d [4], a n d G o o g l e D o c s [3]. D e s p i t e s igni f icant d e v e l o p m e n t w o r k over t h e p a s t d e c a d e , c o l l a b o r a -t ive w r i t i n g too l s r e m a i n u n d e r u s e d , a c c o r d i n g t o s tud ies c o n d u c t e d b y K i m a n d E k l u n d h [17], a n d N o e l a n d R o b e r t [26]. T h e i r s tud ie s f o u n d t h a t m o s t p e o p l e w r i t e a s y n c h r o n o u s l y w i t h i n s m a l l g r o u p s , a n d m e m b e r s use p e r s o n a l w o r d processors , i n s t e a d o f s p e c i a l i z e d c o - a u t h o r i n g too l s to w r i t e c o l l a b o r a -t ive d o c u m e n t s . G r o u p m e m b e r s c o m m u n i c a t e w i t h o n e a n o t h e r u s i n g n a -t ive a n n o t a t i o n t o o l s u p p o r t w i t h i n w o r d processors , a n d e x t e r n a l c o m m u n i -c a t i o n c h a n n e l s s u c h as e - m a i l . H o w e v e r , c o m m u n i c a t i n g d o c u m e n t - r e l a t e d issues s e p a r a t e l y i n e - m a i l messages c a n cause ineff iciencies as d i s c u s s e d i n C h a p t e r 1. T h e r e f o r e , the g o a l o f t h e w o r k r e p o r t e d i n t h i s thesis was t o p r o v i d e c o m p r e h e n s i v e a n n o t a t i o n s u p p o r t t h a t w o u l d a l l o w c o - a u t h o r s t o c o m m u n i c a t e d o c u m e n t - r e l a t e d i n f o r m a t i o n w i t h i n a c o l l a b o r a t i v e d o c u -m e n t . I n o u r r e v i e w o f t h e l i t e r a t u r e , we focus p r i m a r i l y o n d o c u m e n t - r e l a t e d c o m m u n i c a t i o n i n the f o r m of a n n o t a t i o n s a m o n g c o - a u t h o r s . Chapter 2. Related Work 8 2.2 Annotation Support in Co-authoring Contexts W o j a h n et a l . s t u d i e d the effects o f a n n o t a t i o n interfaces [41] o n c o m m u -n i c a t i o n a m o n g c o - a u t h o r s . N o t s u r p r i s i n g l y , t h e y f o u n d t h a t d i f f i cu l ty i n p r o d u c i n g a n n o t a t i o n s o f ten resul t s i n b r i e f a n n o t a t i o n s w i t h less e l a b o r a -t i o n . C h u r c h i l l et a l . [8] d e v e l o p e d a l i ght -we ight c o m m u n i c a t i o n t o o l c a l l e d " A n c h o r e d C o n v e r s a t i o n s . " T h e t o o l s u p p o r t s r e a l - t i m e c o m m u n i c a t i o n i n the c o n t e x t o f c o l l a b o r a t i v e d o c u m e n t s b y a l l o w i n g c o n v e r s a t i o n s c r i p t s t o b e a n c h o r e d i n t o speci f ic p a r t s o f a d o c u m e n t . T h i s m e r g e s s h a r e d d i scuss ions a n d d o c u m e n t ar t i fac t s , f a c i l i t a t i n g t h e e s t a b l i s h m e n t o f c o n t e x t i n f o r m a t i o n for d o c u m e n t - r e l a t e d c o n v e r s a t i o n s . H o w e v e r , we suspec t t h a t v e r b o s i t y o f f u l l c o n v e r s a t i o n s m a y o v e r l o a d a u t h o r s w h e n r e t r i e v i n g the i n f o r m a t i o n . C u r r e n t c o m m e r c i a l s y s t e m s (such as A d o b e A c r o b a t a n d M i c r o s o f t W o r d ; see [42] for de ta i l s ) , a n d the c o l l a b o r a t i v e w r i t i n g s y s t e m s d e s c r i b e d ear l i er ( such as S A S S E a n d P R E P ) p r o v i d e s i m p l e a n n o t a t i o n features on ly . T o e n h a n c e a n n o t a t i o n s u p p o r t , r i cher a n n o t a t i o n m o d e l s h a v e b e e n de-v e l o p e d [37, 43]. A n a c t i v i t y - o r i e n t e d a n n o t a t i o n m o d e l [37] was d e v e l o p e d a n d i m p l e m e n t e d i n a w e b - b a s e d c o l l a b o r a t i v e w r i t i n g t o o l ( ca l l ed " P C A T " ) d e s i g n e d spec i f i ca l ly for c o - a u t h o r i n g a n d r e v i e w i n g c l i n i c a l t r i a l p r o t o c o l s [38, 39]. I n P C A T , a n n o t a t i o n s c a n have p r o p e r t i e s s u c h as re sponse d e a d l i n e a n d u r g e n c y . U s e r s c a n also as s ign e a c h a n n o t a t i o n to o n e o f t h e m o d e l ' s p r e -de f ined categor ies s u c h as "quest ion" or "reply." T h e s y s t e m also s u p p o r t s g e n e r a l c o m m e n t s t h a t are a t t a c h e d to t h e w h o l e d o c u m e n t a n d t h r e a d e d a n -n o t a t i o n s for i t e ra t ive d i scuss ions a m o n g c o - a u t h o r s . A l t h o u g h the m o d e l e x t e n d s b a s i c a n n o t a t i o n features , we suspec t t h a t p r e - d e f i n e d categor ies m a y be t o o r i g i d a n d l i m i t e d to c a p t u r e m a n y o f the ac t iv i t i e s i n v o l v e d i n ' c o - a u t h o r i n g . T h e a n n o t a t i o n m o d e l d e v e l o p e d b y Z h e n g et a l . , w h i c h m o t i v a t e d o u r research , unif ies a l l d o c u m e n t - r e l a t e d c o m m u n i c a t i o n b y s u p p o r t i n g s t r u c -t u r e d a n n o t a t i o n s [43]. T h e a n n o t a t i o n m o d e l is i m p l e m e n t e d i n a re search p r o t o t y p e c a l l e d t h e " B u n d l e E d i t o r . " S i m i l a r t o P C A T , the B u n d l e E d i t o r ( Chapter 2. Related Work 9 s u p p o r t s t h r e a d e d a n n o t a t i o n s a n d g e n e r a l c o m m e n t s . T h e s y s t e m a l lows users to g r o u p a n n o t a t i o n s , a n d c o m m u n i c a t e m e t a - i n f o r m a t i o n i n t h e f o r m of a n n o t a t i o n g r o u p s s u c h as a l ist o f t o - d o i t e m s , o r a s u m m a r y o f edi ts . G r o u p s i n the " B u n d l e E d i t o r " are n o t pre -de f ined; users c a n crea te a n y k i n d o f a n n o t a t i o n g r o u p , a n d t h e y c a n a d d a n a m e a n d a n o p t i o n a l n o t e t o i n d i c a t e the n a t u r e o f t h e g r o u p i n g . A l t h o u g h t h i s m o d e l is the first to s u p p o r t a n n o t a t i o n g r o u p i n g i n c o - a u t h o r i n g contex t s , t h e c o n c e p t o f a n -n o t a t i o n g r o u p i n g is n o t e n t i r e l y new; a n n o t a t i o n s y s t e m s w i t h g r o u p i n g s u p p o r t have b e e n d e v e l o p e d i n o t h e r contex t s . W e d e s c r i b e s o m e o f these s y s t e m s i n the f o l l o w i n g sec t ion . Structured Annotation Support in Other Contexts T h e K n o w l e d g e W e a s e l s y s t e m [20] features c o l l a b o r a t i v e a n n o t a t i o n w i t h t h e p u r p o s e o f c a p t u r i n g s t r u c t u r a l k n o w l e d g e — the k n o w l e d g e o f r e l a t i o n -sh ips a m o n g a set o f d o c u m e n t s . T h e a n n o t a t i o n s i n t h e s y s t e m serve as l i n k s b e t w e e n the files (one as a n n o t a t e d source , a n d t h e o t h e r as a n n o t a t i n g t a r g e t ) . T h e s y s t e m a l lows users to n a v i g a t e a n n o t a t i o n s l o c a l l y b y fo l low-i n g l i n k s , o r g l o b a l l y b y q u e r y i n g the a n n o t a t i o n s ' a t t r i b u t e s . It also a l lows c o l l a b o r a t o r s to g r o u p r e l a t e d a n n o t a t i o n s t o f o r m a h y p e r l i n k n e t w o r k o f r e l a t e d i n f o r m a t i o n resources . O v s i a n n i k o v d e v e l o p e d a s y s t e m c a l l e d A n n o t a t o r , a n a n n o t a t i o n t o o l for t a k i n g notes o n p u b l i s h e d web pages or H T M L d o c u m e n t s [27]. T h e s y s t e m a l lows users to s tore a n n o t a t i o n s i n t h e f o r m o f a t o m s , c l u m p s a n d notes . A n a t o m is s i m i l a r to a s ingle a n n o t a t i o n i n o u r s t r u c t u r e d a n n o t a t i o n m o d e l — se lec ted text w i t h a c o m m e n t a t t a c h e d to i t . A n o t e is s i m i l a r to a g e n e r a l c o m m e n t a n d a t t a c h e d to the w h o l e d o c u m e n t . A c l u m p is de f ined t o be a set o f s e m a n t i c a l l y r e l a t e d a t o m s l i n k e d to the s a m e a n n o t a t i o n . U s e r s c a n re tr i eve a n n o t a t i o n s f r o m different d o c u m e n t s a n d genera te n e w c l u m p s w i t h q u e r y resul t s . H e n c e , i n A n n o t a t o r , t h e g r o u p i n g is l a r g e l y across H T M L d o c u m e n t s . I n sof tware d e v e l o p m e n t contex t s , a t o o l n a m e d T a g S E A was d e v e l o p e d t o s u p p o r t c o l l a b o r a t i v e a n n o t a t i o n s a m o n g g r o u p m e m b e r s [35]. T h e t o o l Chapter 2. Related Work 10 provides a light-weight mechanism to organize annotations by associating them with the same "tags" or keywords. For example, users can add key-words such as "bugs" or "performance" to annotations anchored at different sections of program files to group the annotations together. Users can then browse annotations through hierarchies of tags or filtering based on partic-ular tags. The goal of this tool is to enhance coordination among members and capture important knowledge about source code. To our knowledge, no formal evaluations have been reported for any of the above systems. Hence, little is known about the impact on users of having to create structured annotations. Our research is the first to assess the impact of supporting structure on users who create annotations in a collaborative writing context. 2.3 Information Organization Approaches Our research also fits within the broader research area of information or-ganization because we focus on organizing annotations. We provide a brief survey of the literature on information organization and discuss different organization approaches including tagging, which we identified earlier as a potentially more light-weight approach for structuring annotations. Researchers have studied how people organize and manage information for future retrieval in different contexts such as e-mail [40], web bookmarks [5], and files [16]. Hierarchical or tree-like structural systems are the most common paradigm for filing and categorizing information [30]. However, hierarchical systems present several challenges in both the categorization and retrieval of information. Although multiple categorizations are supported in existing operating systems (with "shortcuts," "aliases," and "symbolic links"), they require extra effort and are not pervasive throughout systems. Hence, multiple categorization features are not commonly used [30]. The act of categorizing into a single group is cognitively difficult [30, 34]. Moreover, the path order dependence enforced by hierarchical systems restricts the way in which the information can be retrieved and requires users to remember the exact path [7]. Chapter 2. Related Work 11 In order to enhance the retrieval process, search engines have been de-veloped to find desired information by typing in just a few keywords (e.g., Google Desktop, Apple's Spotlight). Despite the possibility of such search engines returning perfect results with a small set of keywords, the search tools require users to specify the keywords inherent in the desired informa-tion content, which is difficult, especially in collaborative contexts where information is created by other individuals. A diary study conducted by Teevan et al. [36] reported that keyword-based search engines were not com-monly used when searching for information; a browsing strategy was used instead because it offered better understanding of the desired information and a sense of location during searches. Jones et al. [16] found in their study that categories helped users see the relationships within their information and to have a sense of control over the grouping and location of information. Recently, tagging has grown popular as an alternative to hierarchical structuring and keyword searching for organizing information resources, es-pecially in collaborative contexts. For example, del.icio.us [1] is a web-based tool for organizing and sharing bookmarks based on tagging, and flickr [2] is a similar tool for digital images. In such tagging systems, users assign meta-data or keywords to information resources. Traditionally meta-data is created by professionals (catalogers or authors) [22], but tagging systems allow ordinary users to describe and organize content with any vocabulary they choose. Tagging systems offer two major capabilities: (1) they allow users to add tags to information in a light-weight manner, and (2) tags serve as navigational aids for users to find and organize the information later [28]. The prevalence of a given tag in a system is visible to users through the dis-play of its frequency. Users can easily access the annotations labeled with the same tag through a single click, thereby serving as a navigational aid. Tagging systems emphasize user-defined keywords as "a fundamental organizational construct" [22]. Unlike hierarchical or tree-like structural systems, tagging systems do not require users to develop and agree on a hierarchy of structure to organize and retrieve information; instead they just need to have a shared understanding of a tag's meaning to achieve co-operation and shared value. Collaboration is encouraged by browsing and Chapter 2. Related Work 12 searching shared tags that are relevant to one's interests. Moreover, navi-gation of information resources through shared tags is similar to conducting keyword-based searches, except that users are not restricted to the exact keyword terms inherent in information resources [13]. Tagging is claimed to require less cognitive workload from users than other information organiza-tion schemes [34]. Hence, information organization using a tagging approach could offer higher benefits at lower costs, and might impose fewer barriers for collaboration than other approaches. Chapter 3 13 Observational Study to Understand the Annotating Process 3.1 Study Goals In this chapter, we describe a small qualitative observational study con-ducted to help us understand annotators' workflow with existing annotation tools: Microsoft Word (a popular commerical word processor), and tradi-tional pen-on-paper markup. Zheng et al., before developing their structured annotation model, conducted a study in which they collected collaborative artifacts retrospectively from users (co-authors of academic papers), and analyzed meta-comments in the e-mails [43]. They did not analyze annota-tions embedded in the documents, nor investigate the workflow involved in creating annotations. Little evidence existed about: (1) what process users use to annotate the document, and (2) what kinds of annotations are cre-ated in the document. We sought to understand these issues by observing users creating annotations in a document, and by analyzing the annotations created. We first describe the methodology of our Observational Study and then discuss the behaviors and practices observed. 3.2 Methodology We conducted the study with writing tutors from The University of British Columbia. Recruited through online mailing lists, a total of 5 tutors (4 Chapter 3. Observational Study to Understand the Annotating Process 14 females) participated in the study. All participants had been tutoring aca-demic writing skills for more than three years. Two participants had pre-viously reviewed more than 200 documents, and three between 50 to 100 documents. The study was designed for a single two-hour session. At the beginning of the study, participants were asked to review and annotate a four-page essay-style document (see Appendix A.2) with approximately 1,500 words using their method of choice; two participants annotated with a pen on the printed document, and three used Microsoft Word with its track changes and commenting functions. Participants were instructed to provide their feed-back on the document as they would normally do in their tutoring practice. A simulated e-mail message window was also provided to all participants, allowing for additional document-related communication directed to the hy-pothetical recipient. Reviewing was followed by a semi-structured interview to probe the annotating practices observed. Appendix A.3 shows the inter-view questions. Two hours were required for each participant to complete the study. Participants were paid $40 for their participation. 3.3 Results We report our findings from the study. We observed and took notes on all behaviors related to document reviewing and annotation creation. This allowed us to understand the process of creating annotations. We also col-lected the annotated documents at the end of the task and examined the annotations (example annotations are shown in Tables 3.1 and 3.2 at the end of this chapter). In addition, qualitative feedback provided during the interview was transcribed. 3.3.1 Multiple Passes All participants made at least two passes through the document while re-viewing. They made annotations about syntax issues (e.g., grammar) on their first pass. Then, they took another full pass or quick skim to check Chapter 3. Observational Study to Understand the Annotating Process 15 semantic issues (e.g., argument structure) and to achieve an overview of document status, errors made, and remaining work to be done. They then wrote comments on those issues at the end of the document or in an e-mail message. For example, general comments such as "Your writing is clear and easy to follow . . . ," and "I found the piece well-written, and well-structured. Most changes are minor ...," were written at the end of the printed document or in the accompanying e-mail messages. One participant wrote a summary of her review on each section of the document in an e-mail message. She included section titles of the document such as: "Introduction," "Body," and "Conclusion," and wrote overall feed-back on each section such as "Introduction goes from general to specific, good. But you wait too long to introduce groupware." Summaries of repeated errors that occurred in the document were also included, such as "GRAMMATI-CAL — some tendency to use unnecessary words, and word combinations ... Examples, 'decisions have to be made,' instead of 'decisions are made. "' However, when asked during the interview, all participants mentioned that they did not want to spend a lot of time summarizing their review, which essentially involved repeating and referring to what they had already noted in the document. 3.3.2 Justifications We found that all participants not only made suggestions for changes, but also occasionally provided an explanatory comment along with their anno-tations, particularly when a problem was encountered for the first time. For example, as shown in Table 3.1 (i), an edit annotation suggesting a verb tense correction^ was accompanied by a comment explaining "Stay in the same tense as the rest of the sentence." Al l participants revealed that ex-planatory comments were added to help annotation recipients better under-stand the errors and the changes made to the document. When the same error was repeated in the document, participants made the edit changes to correct each instance of the error, but did not add an explanatory comment again. They expected recipients to refer back to the annotation at a previ-Chapter 3. Observational Study to Understand the Annotating Process 16 ous occurrence of the problem. For example, one participant mentioned in the interview that, "// there is no comment accompanying a [edit] change, it is because I have previously changed the same error and commented on it before." We note that this is one place where structure may be beneficial: all instances of the same problem can be linked together to reduce ambiguity. 3.3.3 Local versus Global Comments All participants gave two major types of feedback on the documents: "local" or sentence-level feedback, and "global" or document-level feedback. One of the participants explained: "There are two different types of comments [that I usually make] — comments that are specific for a specific sentence — why this sentence is not working, or what have you done wrong, or things like that [that] need a little bit explanation. But gener-ally comments are about larger issues — say for example, ... whether or not the language is fitting to the general audience. So, it is more of a general comment, so maybe I want to highlight a few ideas that relate to that comment." All participants created single annotations (annotations embedded in the document with a single anchor to specific content) to address sentence level or local issues. Regarding global issues, participants typically inserted a comment such as "Example of non-academic language — read through for this sort of language" as a single annotation to the place in the document where the problem was first realized. We note that with no explicit addi-tional pointers, recipients would not necessarily be able to see all instances of a problem. This is another place where additional structure may be ben-eficial. 3.3.4 Tagging-like Behaviors Two participants used a keyword association technique to efficiently point out errors in the documents. For example, one participant defined a keyword Chapter 3. Observational Study to Understand the Annotating Process 17 coding as "WC=Word Choice" and added the keyword "WC" to every place where she found a wording problem, instead of writing more verbose com-ments repeatedly. We note that this is an instance of tagging-like behaviors where the tags are symbols or abbreviated notes. 3.3.5 Prioritization Two participants occasionally highlighted the text anchors of some annota-tions using, for example, a yellow highlighter or italicized fonts. One partic-ipant explained that she did so to indicate the higher degree of urgency and importance of those annotations. The participants thus used highlighting to ensure that recipients would pay specific attention to the most important annotations that might otherwise be "buried" among other annotations in the document, a common problem with heavily annotated documents. 3.3.6 Reviewing Methods Participants explained their method of choice (Microsoft Word vs. pen-on-paper) as follows. Three participants said that they preferred to annotate documents electronically because they could conveniently send or receive electronic copies of documents via e-mail. The two other participants mostly annotated printed copies of documents because they did not have access to electronic copies at the time of tutoring or because it was easier for them to draw and visualize some ideas or outlines of the documents on paper. As the example in Table 3.1 shows, all three participants who used Mi-crosoft Word recorded the edit changes they made to the document by using the "Track Changes" (TRK) feature, which automatically creates edit anno-tations. When adding comments to the document, we observed that these participants used two different methods: (1) using the commenting function of Microsoft Word, and (2) typing directly into the document text using special fonts (such as italics). One participant exclusively used the first method to add comments while another participant exclusively used the second method (he explained during the interview that he was not aware of the commenting feature in Microsoft Word). The remaining third partici-Chapter 3. Observational Study to Understand the Annotating Process 18 pant used both methods; she felt that it was sometimes easier for her to type directly into the text than to click the "Insert Comment" button and add a comment. Annotations (both edits and comments) in Microsoft Word are displayed with a markup balloon in the right margin of the document near the anchored text. Participants described their concern with the display of annotation balloons in Microsoft Word; they were concerned that visual clutter resulting from displaying all annotations in the document might have recipients overwhelmed and intimidated with the amount of annotations in the documents. The two participants who annotated printed documents marked on the document text to indicate the suggested changes. These participants also wrote some comments in between the lines or in the margins with their anchored text circled, underlined or enclosed by parentheses. The examples in Table 3.2 show samples of annotations made in the printed documents by the participants. 3.3.7 Other Communication Methods The three participants who used Microsoft Word in our study mentioned that they had tried other methods such as online chatting and phone calls. Although these methods allowed for more i n t e r a c t i v e discussions and Conver-sations, our participants did not like them because (1) those communications happened outside the document, requiring additional explicit navigational statements such as "see page 3, paragraph 2" to build a context for conversa-tions, and (2) such interactive communication did not allow the participants to control their reviewing time sufficiently. For example, one participant who used MSN messenger shared her frustration with the online messaging method, "Once I gave them my MSN [id], they keep talking to me day after day and everyday ... and asked me questions about their draft. It gets really frustrating. And [I have] to be the one to say 'OK, I can't help you anymore. I've got my own work to do."' Chapter 3. Observational Study to Understand the Annotating Process 19 Hence, the participants chose not to use these methods anymore, and mainly used the annotation features of Microsoft Word and e-mail, in which they had more control over the length of time they allocated to reviewing tasks. 3 .4 Summary The Observational Study provided us with a better understanding of dif-ferent types of annotations and workflow involved in their creation. While reviewing documents, participants gave feedback not only to improve the quality of the reviewed document, but also to enhance the writing skills of recipients by including explanatory comments along with some of their suggested edits. The findings also confirmed that gaps exist between cur-rent methods of annotating (specifically pen-on-paper markup and Microsoft Word) and annotators' needs: (1) annotators lacked an efficient means for describing relationships between annotations, and (2) they also lacked a key-word annotation feature, such as tagging, to allow for efficient feedback. We also learned that participants would like to have a priority scheme for high-lighting important annotations, and an ink annotation feature for drawing and visualizing ideas in the document. These are features we had not in-vestigated in our study but should be considered in future studies. Another interesting finding was that participants preferred to have more control over their reviewing time, and to keep the interactivity with recipients low. Having identified different places where structure may be beneficial, our next step was to examine how annotators might actually go about structur-ing annotations if structuring was provided to them as an option. In the next chapter, we describe a qualitative study involving a paper prototype in which we explore this issue. Chapter 3. Observational Study to Understand the Annotating Process 20 application and its features offered. Moreover, its success could also be due to the fact that developers have learned and become aware of the problems and issues around such applications and jpvercomejniost of them. , ,-'* Comment [y22]: Say «titiaxat Is CSCW Ready for Theory? Since CSC\V has emereed. some theories have imoosed influence on CSCW. An Inserted: h»-« Inserted: an i . Comment and edits added using "Comment" balloons and "Track Changes." Without such indication, the reader, among other things, has no idea as to what kind of entity is involved here. The reagder needs to know what is this entity that you are discussing, at the very beginning of your text^ was originally coined by Irene Greif at a workshop in 1984 attended by a small group interested in using technology to Inserted: &«nlK£i!salttl j Inserted: i i . Comment added directly into the text using blue colored font and sur-rounded by parentheses. The term CSCWI was originauv coined bv Irene Greif at a workshop in 1984 C o n m » n [ » J l : Y « n a * a i » l I i » which was attended by a small group interested in using technology to help support a tofantmym.itoaM people with their work. This workshop eventually led to the first CSCW conference in 1986 [?] CSCW focuses on the study of tools and applications for support groups and their social, psychological, and organizational impacts. The term CSCW stands tor Computer Supported Cooperathe Work, (we needed to W/iow this earlier: see comment ~2) Tools and applications in the CSCW context mainly support the coordination and i i i . Comments created using "Comment" balloons (first line) and typing directly into the text (see the second last line) with parentheses used to distinguish the comment from the document text. Table 3.1: Examples of annotations created by participants using Microsoft Word. Chapter 3. Observational Study to Understand the Annotating Process 21 More often than not, organizations and groups are structured and responsibilities are divided so that the^pverhead forjtommunications will be minimized. Hby^evetf^s the size of the group grows biggerth\iiW overhead requirements for communication and coordination it demands. Some groupware applications are designed mainly to provide i . Marks and comments made in between lines of text applications in the CSCW context mainly,support the coordination and communication , , £ for small groups or organizations such as scheduling meetings, distributed decision^' 1 ' y y 1 ' " " , • • A - ' ' ' ,cj2 '..,t.l-T making, locatingcollcagues. collaborative work. etc. /SiricethejV: CSCW has evolved as {'t«'<Q iU'"^ a field with many influences from various disciplines"such as: Computer Science, • ! j u « s l ' . Sociology, Psycholoay, Ethnography. Anthropology. Oruanizational Studies and Communications. The field has accumulated a set of empirically based interdisciplinary studies, arid many new interestina arrolications to sunoort omuns and nrwmimilinns Tlw i i . Comments made in the margin Table 3.2: Examples of annotations created by participants using a pen on a document printed on paper. Chapter 4 22 P a p e r P r o t o t y p e S t u d y to U n d e r s t a n d A n n o t a t i o n S t r u c t u r i n g 4.1 Study Goals Having confirmed annotators' needs for more structured annotations, we sought to understand what process and what types of structure annotators would use when provided with structuring as an option. More specifically, we wanted to assess: (1) the semantics of the structures created, (2) the approaches taken to create structure (top-down, bottom-up, or otherwise), and (3) the complexity of the structuring created in terms of the size of an-notation groups and whether hierarchies (e.g., groups within groups) might be adopted. In order to mitigate the impact of any particular tool (and its potential usability issues), we elected to do a qualitative exploratory study with a paper prototype where grouping annotations amounted to essentially making little piles of paper annotations. 4.2 Methodology The study was qualitative and exploratory in nature. We did not have specific hypotheses at the outset of the study. Chapter 4. Paper Prototype Study to Understand Annotation Structuring 23 4.2.1 Participants A total of 8 people (5 females) participated in the study. Recruited through online mailing lists and newsgroups, they were all graduate students at The University of British Columbia: one from Zoology, three from Psychology, and four from Computer Science. A screening process ensured that all par-ticipants had co-authoring experience; five participants had co-authored more than ten documents, two between five and ten documents, and one less than five documents. Collectively, the participants had co-authored a wide range of documents including brochures, project reports, technical documentation, journal papers, and an encyclopedia chapter. Five participants used a word processor (mainly Microsoft Word) every-day, two every 2-3 days, and one did so once a month. They all felt very confident about using their word processor, and only one participant had never used any annotation functions. Participants were paid $15 for their participation. 4.2.2 Task Participants were asked to perform a task of organizing annotations in a document. They were instructed to assume the role of a co-author collab-oratively writing a given document with two other co-authors who had ex-pertise in different areas. The participants' task was to organize pre-existing annotations in the document, ones they had hypothetically just created, so that their co-authors could review the document efficiently and accurately. The document consisted of 932 words and 42 annotations. Because we were interested in variability among users' grouping approaches and an-notation groups, we provided the same document and annotations to all participants, who were asked not to add any new annotations. The scenario and annotations were designed with an outlook that different kinds of an-notation groups could be created (e.g., based on types {edits, comments}, themes {tone, clarity}, or targeted co-author). The document was about understanding the effects of different types of music and volume on stu-dents' ability to study. For the most part, the content was general enough Chapter 4. Paper Prototype Study to Understand Annotation Structuring 24 for a l l p a r t i c i p a n t s t o u n d e r s t a n d . It d i d c o n t a i n d e s c r i p t i o n s o f e x p e r i m e n t m e t h o d o l o g y a n d a n a l y s i s , w h i c h m i g h t b e m o r e access ib le t o P s y c h o l o g y s t u d e n t s , s o m e t h i n g we o n l y r e a l i z e d i n r e t r o s p e c t . A p p e n d i c e s B . 4 a n d B . 5 s h o w t h e i n s t r u c t i o n s g i v e n t o p a r t i c i p a n t s a n d t h e a n n o t a t e d d o c u m e n t u s e d i n t h e s t u d y . P a r t i c i p a n t s were g i v e n a p r i n t e d c o p y o f t h e d o c u m e n t w i t h a n n o t a t i o n s d i s p l a y e d o n t h e r ight m a r g i n a n d t h e i r a n c h o r e d t ex t h i g h l i g h t e d . S e p a -r a t e i d e n t i c a l copies o f e a c h a n n o t a t i o n p r i n t e d i n d i v i d u a l l y o n s m a l l p a p e r s t r i p s were m a d e a v a i l a b l e so t h a t p a r t i c i p a n t s c o u l d p i l e t h e s t r ips t o g e t h e r ( a n d o p t i o n a l l y p a p e r - c l i p t h e m ) t o m a k e a n n o t a t i o n g r o u p s . T o i d e n t i f y a p i l e , a pos t - i t sheet w a s p l a c e d o n t o p for w r i t i n g t h e a n n o t a t i o n g r o u p ' s n a m e a n d a n o p t i o n a l note . M u l t i p l e copies o f e a c h a n n o t a t i o n were m a d e a v a i l a b l e so t h a t p a r t i c i p a n t s c o u l d p l a c e a n a n n o t a t i o n i n t o m o r e t h a n o n e g r o u p . E a c h a n n o t a t i o n g r o u p was a l l o w e d to b e n e s t e d u n d e r o t h e r g r o u p s i n a n y h i e r a r c h i c a l s t r u c t u r e . F i g u r e 4.1 shows a p a r t i c i p a n t p e r f o r m i n g t h e a n n o t a t i o n - o r g a n i z a t i o n task , a n d F i g u r e 4.2 i l lu s t ra te s s a m p l e pi les o f a n n o t a t i o n g r o u p s t h a t were c r e a t e d . F i g u r e 4.1: A p a r t i c i p a n t p e r f o r m i n g t h e ta sk w i t h t h e p a p e r p r o t o t y p e . Chapter 4. Paper Prototype Study to Understand Annotation Structuring 25 F i g u r e 4.2: S a m p l e pi les o f a n n o t a t i o n g r o u p s c r e a t e d w i t h g r e e n p o s t - i t notes i d e n t i f y i n g the n a m e o f t h e g r o u p (or b u n d l e ) a n d a n o p t i o n a l note . 4.2.3 Procedure T h e s t u d y was d e s i g n e d for a s ing le o n e - a n d - a - h a l f - h o u r sess ion. A d e m o -g r a p h i c q u e s t i o n n a i r e was fo l lowed b y a n i n f o r m a t i o n sess ion o n g e n e r a l c o n c e p t s s u c h as c o l l a b o r a t i v e w r i t i n g a n d t h e n a t r a i n i n g sess ion o n h o w t o use t h e p a p e r m a t e r i a l s i n t h e task . W e were c o n c e r n e d t h a t d o c u m e n t s a n d a n n o t a t i o n s u s e d i n t h e t r a i n i n g sess ion m i g h t i n t e r a c t w i t h the t a s k p e r f o r -m a n c e ; hence a l ist o f s h o p p i n g i t e m s n e e d i n g t o b e o r g a n i z e d was u s e d i n i l l u s t r a t i n g t h e p a p e r m a t e r i a l s . A p p e n d i x B . 3 shows a d e t a i l e d d e s c r i p t i o n o f the t r a i n i n g task . P a r t i c i p a n t s were n e x t a s k e d to r e a d a n a n n o t a t i o n - f r e e v e r s i o n o f the t a s k d o c u m e n t , after w h i c h t h e y were g i v e n t h e a n n o t a t e d ver -s ion . T h e y h a d t o p e r f o r m the a n n o t a t i o n o r g a n i z a t i o n task . F o r t h e first pass over t h e a n n o t a t e d d o c u m e n t , p a r t i c i p a n t s were r e q u i r e d t o r e a d t h e a n n o t a t i o n s i n the o r d e r t h a t t h e y a p p e a r e d i n t h e d o c u m e n t so as t o s i m u -late t h a t t h e y h a d t h e m s e l v e s a n n o t a t e d t h e d o c u m e n t i n s e q u e n t i a l o r d e r . T h e y were, however , a l l o w e d t o s tar t g r o u p i n g a n n o t a t i o n s at a n y p o i n t Chapter 4. Paper Prototype Study to Understand Annotation Structuring 26 during the task. A short questionnaire and a debriefing session were used to gain better insight into the grouping behaviors and preferences observed. 4.3 Results We analyzed the recorded video of participants performing the task, and coded all behaviours related to the simulated reviewing (i.e., participants reading pre-existing annotations) and annotation group creation. This al-lowed us to understand the temporal patterns of annotation grouping. We also collected the "piles" of annotation groups at the end of the task to an-alyze their structure. In addition, qualitative feedback provided during the debriefing session was transcribed. 4.3.1 Temporal Approaches for Creating Groups Participants followed different temporal sequences for grouping annotations: four dominant patterns emerged (as shown in Table 4.1) and are distin-guished by the number of passes over the document that were made and when, with respect to those passes, annotation groups were created. The participant who used the pre-review approach mentioned that the co-authoring scenario informed him of the annotation groups he wanted to create. This may have been an artifact of our study design, or may simply represent a difference in style, as none of our other participants followed this approach exclusively. The three participants who used the post-review approach said that seeing all annotations in the document before group-ing helped them make their groups more consistent and manageable. They mentioned that they took mental notes of annotations of interest so that they could relocate them for grouping later. Two of these participants ex-ternalized their mental notes by adding keywords or notes to annotations on the printed document during their first read through, and then grouped annotations based on those keywords. The two participants who used the during-review approach stated that they grouped annotations "naturally" as occurred to them without explicitly having to think about grouping. Lastly, Chapter 4. Paper Prototype Study to Understand Annotation Structuring 27 Description Pre-review N u m o f pass: 1 (1 p a r t i c i p a n t ) P a r t i c i p a n t s f o r m u l a t e d a n n o t a t i o n g r o u p s p r i o r t o see ing a n n o t a t i o n s . A n n o t a t i o n s t h a t fit these p r e - d e f i n e d g r o u p s were la ter se lec ted a n d a s s o c i a t e d w i t h c o r r e s p o n d i n g g r o u p s . Post-review N u m o f pass: 2 (3 p a r t i c i p a n t s ) P a r t i c i p a n t s r e a d all a n n o t a t i o n s p r i o r to f o r m u l a t i n g a n y g r o u p s . O n c e g r o u p s were de f ined , re levant a n n o t a t i o n s were a s s o c i a t e d w i t h t h e g r o u p s . During-review N u m o f pass: 1 (2 p a r t i c i p a n t s ) P a r t i c i p a n t s o r g a n i z e d a n n o t a t i o n s i n t o g r o u p s w h i l e r e a d i n g a n n o t a t i o n s , c r e a t e d n e w g r o u p s w h e n e x i s t i n g ones were n o t a p p r o p r i a t e for a g i v e n a n n o t a t i o n . Hybrid N u m o f pass: 2 (2 p a r t i c i p a n t s ) Di f f erent g r o u p s o f a n n o t a t i o n s were cre-a t e d u s i n g different a p p r o a c h e s s t a t e d a b o v e (pre-, post- o r during- rev iews) T a b l e 4.1: T e m p o r a l p a t t e r n s o f s t r u c t u r i n g a n n o t a t i o n s . ( N = 8 ) the two p a r t i c i p a n t s w h o u s e d t h e hybrid a p p r o a c h m e n t i o n e d t h a t t h e y cre-a t e d "obvious" g r o u p s ( such as t y p o s a n d g r a m m a r ) b y u s i n g the p r e - r e v i e w a p p r o a c h a n d o t h e r g r o u p s b y u s i n g o t h e r a p p r o a c h e s . A f t e r g r o u p i n g a l l a n n o t a t i o n s , four o u t o f t h e e ight p a r t i c i p a n t s (2: hybrid, 1: post-review, 1: during-review) a l so reshuff led s o m e o f t h e i r g r o u p s b y m e r g i n g o r s p l i t t i n g . 4.3.2 Semantics of Annotation Groups A n n o t a t i o n g r o u p s r e s u l t e d f r o m o r g a n i z i n g d i s p a r a t e a n n o t a t i o n s t h r o u g h -o u t the d o c u m e n t t h a t were c o n c e p t u a l l y r e l a t e d . M o s t g r o u p s (82%) were problem-based: a n n o t a t i o n s were s i m i l a r i n the n a t u r e o f t h e p r o b l e m s t h a t t h e y a d d r e s s e d (e.g., the g r o u p n a m e d " T o n e " h a d a n n o t a t i o n s t h a t h i g h -l i g h t e d a n d d i s c u s s e d incons i s t en t t o n e t h r o u g h o u t t h e d o c u m e n t ) . T h e t h r e e m o s t c o m m o n t y p e s o f p r o b l e m - b a s e d g r o u p s c r e a t e d b y p a r t i c i p a n t s were c lar i ty , g r a m m a r , a n d s t r u c t u r a l o r o r g a n i z a t i o n a l p r o b l e m s . T h e re-m a i n i n g g r o u p s (18%) were recipient-based: a n n o t a t i o n s t h a t were to b e Chapter 4. Paper Prototype Study to Understand Annotation Structuring 28 r e v i e w e d b y a n i n t e n d e d p a r t i c u l a r c o a u t h o r (e.g., t h e g r o u p n a m e d " F o r J a n e " w i t h a n n o t a t i o n s t h a t s o l i c i t e d J a n e ' s e x p e r t k n o w l e d g e ) . 4.3.3 Structures of Groups Content F i g u r e 4.3: S a m p l e a n n o t a t i o n g r o u p s i n h i e r a r c h i c a l s t r u c t u r e . A 3 , A 7 , A 8 , A 1 3 , A 1 4 , A 1 7 represent s ingle a n n o t a t i o n s i n t h e d o c u m e n t . T h e g r o u p n a m e d " C o n t e n t " is the t o p - l e v e l g r o u p , o t h e r g r o u p s "Jane ," " N i c k , " a n d " T h i n g s t o c o n f i r m " are i n s t a n c e s o f s u b - g r o u p s . T h e average n u m b e r o f g r o u p s c r e a t e d p e r p a r t i c i p a n t was 8.3 ( s d = 2 . 9 , m i n = 5 , m a x = 1 4 ) . T h e average n u m b e r o f a n n o t a t i o n s p e r g r o u p was 8.2 ( sd=2 .6 , m i n = l , m a x = 2 2 ) . O v e r a l l , g r o u p s t r u c t u r i n g was n o t v e r y c o m -plex; 45% of t h e g r o u p s h a d a flat structure i n t h a t t h e y h a d n o s u b - g r o u p s n o r were t h e y w i t h i n s u p e r - g r o u p s , a n d t h e r e m a i n i n g 55% o f the g r o u p s were i n hierarchical structures (i.e., t h e y h a d at least o n e s u b - g r o u p or a s u p e r - g r o u p ) . W e a n a l y z e d t h e c o m p l e x i t y o f these h i e r a r c h i e s i n t e r m s o f he ight ( the l e n g t h o f t h e p a t h from the t o p - l e v e l g r o u p to t h e fur thes t s u b - g r o u p ) . F i g u r e 4.3 shows a s a m p l e a n n o t a t i o n g r o u p w i t h a h i e r a r c h i c a l s t r u c t u r e ; t h e s t r u c t u r e has he ight 2, w h i c h is the l e n g t h o f t h e p a t h f r o m t h e t o p - l e v e l g r o u p n a m e d " C o n t e n t " to t h e fur thes t s u b - g r o u p n a m e d " T h i n g s t o C o n f i r m . " W e f o u n d t h a t the average he ight o f hierarchical structures c r e a t e d b y p a r t i c i p a n t s was o n l y 1.4 ( sd=0 .5 , m i n = l , m a x = 2 ) . Jane Nick A14 A17 1 Things to confirm ] A F r3 A3 A7 A13 Chapter 4. Paper Prototype Study to Understand Annotation Structuring 29 Analysis on the sub-groups revealed that 44% of them were true sub-components of their higher-level groups (what we call a "proper hierarchy"), e.g., a sub-group named "Missing standard deviation" nested within a group "Missing Information" (since standard deviation is one of the information types presented in the document). The rest of the sub-groups did not re-flect proper subset relationships; the hierarchy seemed to result from the individual participant's decision about the relative importance of attributes (what we call an "arbitrary hierarchy"). For example, one participant cre-ated a group "For Jane" with a nested sub-group "Clarifications" because she wanted to emphasize and make the recipient-based information more salient. At the same time, another participant created the reversed struc-ture for these same annotations: "Clarifications" with a nested sub-group "For Jane," in which the problem-based information was emphasized. 4.3.4 Strong Support for Structuring Seven out of eight participants strongly agreed that they liked being able to organize annotations within a document. One of the participants com-mented that groups were "infinitely easier than the current annotation for-mat and [can be used] to delegate sections to different authors [which] reduces duplicate effort." Another participant mentioned that she would like to use annotation groups not only to facilitate her co-authors' workflow but also to manage her own workflow. 4.4 Summary and Addit ional Comments All participants structured annotations during the tasks, and perceived the benefits of supporting grouping in annotation tools. We found that partici-pants used different temporal patterns to organize annotations. We identi-fied common semantics of annotation groups, but we did not observe very complex group structuring. It could be that the number of annotations and groups were not large enough to call for complex structures. During the debriefing section, two participants commented that if the number of an-Chapter 4. Paper Prototype Study to Understand Annotation Structuring 30 n o t a t i o n s i n a g r o u p b e c a m e large ( m o r e t h a n a b o u t 50 a n n o t a t i o n s ) , t h e y w o u l d l ike ly have b r o k e n t h e i r g r o u p s d o w n i n t o s m a l l e r , m o r e m a n a g e a b l e s u b - g r o u p s . O n e o f the p a r t i c i p a n t s w h o c r e a t e d o n l y "flat" g r o u p s s a i d t h a t he d i d n o t create n e s t e d h i e r a r c h y b e c a u s e he felt t h a t t h e t o t a l n u m -b e r o f g r o u p s he c r e a t e d was m a n a g e a b l e , a n d h a d he h a d m o r e g r o u p s i n the d o c u m e n t , he m i g h t h a v e c r e a t e d a h i g h e r - l e v e l g r o u p t h a t h a d s o m e a n n o t a t i o n g r o u p s as n e s t e d s u b - g r o u p s . M o r e re search w i t h larger d o c u -m e n t s a n d m o r e i t e ra t ive cyc le s o f r e v i e w i n g w o u l d b e n e e d e d to assess t h e usefulness o f s u p p o r t i n g c o m p l e x s t r u c t u r i n g . T h e d o c u m e n t t o p i c a p p e a r e d to h a v e h a d a n i m p a c t o n the resul ts , w h i c h i n r e t r o s p e c t was n o t s u r p r i s i n g . T h e t h r e e p a r t i c i p a n t s f r o m P s y -c h o l o g y c r e a t e d e ight o r m o r e g r o u p s d u r i n g t h e t a s k w h i l e o t h e r p a r t i c i -p a n t s f r o m C o m p u t e r Sc i ence a n d Z o o l o g y c r e a t e d fewer g r o u p s . T h e m o s t l i k e l y e x p l a n a t i o n for t h i s is d e r i v e d f r o m the s o c i a l p s y c h o l o g y l i t e r a t u r e : large n u m b e r s o f categor ies ref lected users' f a m i l i a r i t y w i t h t h e s u b j e c t o f t h e d o c u m e n t a n d t h e i r t h o r o u g h u n d e r s t a n d i n g o f the s u b j e c t [33]. T h e s t r u c t u r e o f s o m e g r o u p s r e s e m b l e d a "d iv ide a n d c o n q u e r " p r o b l e m d e c o m -p o s i t i o n a p p r o a c h i n w h i c h a n n o t a t i o n g r o u p s a n d s u b - g r o u p s c o r r e s p o n d t o c o m p o n e n t s a n d s u b c o m p o n e n t s o f the w o r k r e m a i n i n g to b e d o n e i n t h e d o c u m e n t . B a s e d o n the resu l t s f r o m o u r P a p e r P r o t o t y p e S t u d y a n d the O b s e r v a -t i o n a l S t u d y ( d e s c r i b e d i n C h a p t e r 3), we saw m o t i v a t i o n s for b o t h s t r u c -t u r i n g a n d t a g g i n g o f a n n o t a t i o n s . W e d i scuss - these i m p l i c a t i o n s a n d m o t i -v a t i o n s i n the n e x t c h a p t e r . Chapter 5 31 Incorpora t ing Tagging in to S t ruc tu r ed A n n o t a t i o n s T h e O b s e r v a t i o n a l S t u d y a n d t h e P a p e r P r o t o t y p e S t u d y d e s c r i b e d i n t h e two p r e v i o u s c h a p t e r s r e v e a l e d t h a t the a b i l i t y t o o r g a n i z e a n n o t a t i o n s i n t o g r o u p s c o u l d benef i t c o - a u t h o r s i n severa l ways , n a m e l y b y f a c i l i t a t i n g c o m -m u n i c a t i o n , p r o b l e m d e c o m p o s i t i o n , a n d w o r k f l o w m a n a g e m e n t a m o n g a n -n o t a t o r s a n d c o - a u t h o r s . I n th i s c h a p t e r we d i scuss t h e i m p l i c a t i o n s a n d m o t i v a t i o n s for s t r u c t u r i n g a n d t a g g i n g t h a t were d e r i v e d f r o m t h e two s tud ies . 5.1 Implications and Motivations for Grouping and Tagging T h e different t e m p o r a l p a t t e r n s o b s e r v e d i n t h e P a p e r P r o t o t y p e S t u d y for o r g a n i z i n g a n n o t a t i o n s s e e m to reflect the t o p - d o w n a n d b o t t o m - u p a p -p r o a c h e s , as sugges ted b y the i n f o r m a t i o n p r o c e s s i n g l i t e r a t u r e [18]. W h a t we refer to as t o p - d o w n o c c u r s w h e n a user creates a g r o u p , o f ten a p r i -o r i , a n d t h e n a d d s a n n o t a t i o n s to t h a t g r o u p . B o t t o m - u p o c c u r s w h e n the user f o r m u l a t e s t h e g r o u p , o f ten after r e v i e w i n g a l l t h e a n n o t a t i o n s . W e s p e c u l a t e t h a t the pre-review a p p r o a c h is a f o r m o f top-down i n f o r m a t i o n p r o c e s s i n g [18] w h i l e post-review is s i m i l a r t o bottom-up. R e s h u f f l i n g o f a n -n o t a t i o n g r o u p s c a n be c o n s i d e r e d as middle-out p r o c e s s i n g . T o s u p p o r t these different a p p r o a c h e s , it s h o u l d be pos s ib l e to create a n n o t a t i o n g r o u p s at a n y t i m e , i.e., before , d u r i n g , o r after s ingle a n n o t a t i o n s are c r e a t e d . W e be l ieve t h a t m e c h a n i s m s for c r e a t i n g a n d m a n a g i n g a n n o t a t i o n g r o u p s Chapter 5. Incorporating Tagging into Structured Annotations 32 s h o u l d b e flexible a n d l i gh t -we ight . C o n s i s t e n t w i t h findings r e p o r t e d b y N e u w i r t h et a l . [24], we f o u n d t h a t s o m e p a r t i c i p a n t s o c c a s i o n a l l y m a d e mental notes a b o u t a n n o t a t i o n s t h a t t h e y w a n t e d to r e t u r n to for o r g a n i z i n g . H e n c e , p r o v i d i n g a l i gh t -we ight m e a n s to e x t e r n a l i z e s u c h m e n t a l notes a n d s u p p o r t for n a v i g a t i o n a l a ids s h o u l d r e d u c e users ' c o g n i t i v e l o a d . M o r e o v e r , t a g g i n g - l i k e b e h a v i o r s f o u n d i n o u r O b s e r v a t i o n a l S t u d y sugges ted t h a t tags c o u l d also b e use fu l i n p i n -p o i n t i n g p r o b l e m s i n t h e d o c u m e n t . W e saw d i v e r s i t y i n the degree o f g r o u p s t r u c t u r i n g ( p r o p e r a n d a r b i t r a r y h i erarch ie s , a n d flat s t r u c t u r e s ) . D i s a g r e e m e n t o r conf l i c t s a m o n g users i n d e f i n i n g s t r u c t u r e s a n d h i e r a r c h i e s o f a n n o t a t i o n g r o u p s m a y cause a m b i -gu i t i e s a n d ineff iciencies . W e a r g u e t h a t t a g g i n g is l ike ly a g o o d s o l u t i o n t o th i s issue. W i t h t a g g i n g , users d o n o t n e e d t o agree o n a p a r t i c u l a r h i -erarchy , i n s t e a d t h e y j u s t n e e d to h a v e a s h a r e d u n d e r s t a n d i n g o f a tag's m e a n i n g [22]. H i e r a r c h i e s s h o u l d s t i l l be s u p p o r t e d , however , to recogn ize p r o p e r subse t r e l a t i o n s h i p s a m o n g a n n o t a t i o n g r o u p s . A l t h o u g h we d i d n o t observe v e r y c o m p l e x g r o u p s t r u c t u r i n g , we i m a g i n e t h a t c o m p l e x s t r u c t u r e s m i g h t arise as the a m o u n t o f a n n o t a t i o n s o r t h e s ize o f a c o l l a b o r a t i v e ar t i fac t grows over t i m e . W e real ize t h a t h a v i n g c o m p l e x s t r u c t u r e s m i g h t h i n d e r the c o - a u t h o r i n g process , b e c a u s e o f the a d d i t i o n a l n a v i g a t i o n t i m e r e q u i r e d to r e a c h a h i g h l y n e s t e d a n n o t a t i o n a n d the a d d i t i o n a l effort r e q u i r e d to d e v e l o p a g r e e m e n t a m o n g c o l l a b o r a t o r s a b o u t the h i e r a r c h i c a l i n f o r m a t i o n . T h i s af f irms the i m p o r t a n c e o f r e d u c i n g c o m p l e x i t y i n the degree o f s t r u c t u r i n g . W e n o t e t h a t t a g g i n g c a n be a g o o d a p p r o a c h b e c a u s e it c a n a l low for i m p l i c i t s t r u c t u r e s . 5.2 Integrating Structuring and Tagging into Annotations B a s e d o n t h e i m p l i c a t i o n s above , we e x t e n d e d Z h e n g et al . 's p r e v i o u s s t r u c -t u r e d a n n o t a t i o n m o d e l [43] b y a d d i n g tags as o n e o f the o p t i o n a l a t t r i b u t e s o f a n n o t a t i o n s (see T a b l e 5.1). T a g s i n o u r m o d e l serve m u l t i f o l d p u r p o s e s . Chapter 5. Incorporating Tagging into Structured Annotations 33 Mandatory Attributes: Annotator - creator Timestamp - creation time Reviewing Status - unread/read and accepted/rejected Anchor - location in the document Optional Attributes: Name - text description Recipient - intended co-author Comment - free-form text note Modification - insert/delete/replace of text Substructure - list of other associated annotations Tag - keywords Table 5.1: Annotation model from [43] with our added Tag attribute. They: • efficiently associate a keyword with annotations, • externalize mental notes or act as navigational aids, • easily identify semantic concepts inherent in annotations, • facilitate workflow by allowing for bottom-up annotation grouping, and • simplify structures of annotation groups through implicit grouping. Tags are treated as meta-information about annotations that users can easily add as they review and annotate a document. Tags can be used as navigational aids; by filtering annotations on a particular tag, users can then jump easily between annotations in the selected set of annotations in the document. Users can also choose to save tag-based filter results as structured groups of annotations. Tags allow flexible classification of annotations based on their semantic concepts. Tags provide implicit groups for annotations because co-authors can easily see relationships among annotations labeled Chapter 5. Incorporating Tagging into Structured Annotations 34 ^ . File Edit f? Spell Check Review | Sam • 0 Show Reviewing Pane 1—1 comment Send uccessfully loaded. Parenting and H o n e s t y 3 o m e parent ing exper ts urge that pa ren ts to-tell their ch i ld ren the truth about their p a s t BMper imentexper ience with d rugs and sex, even w h e n they don ' t w a n t their ch i ld ren to do wha t they did- Others th ink pa ren ts s h o u l d c e n s o r w h a t they tell he i r ch i ld ren. T h i s p a p e r wi l l d e m o n s t r a t e that it is better for pa ren ts to c e n s o r w h a t they tell thei r ch i ld ren unti l the chi ldren are old and mature e n o u g h to u n d e r s t a n d One of the pr imary du t ies of paren t ing is to impar t the pa ren t s ' mo ra l v i e w s to their ch i ld ren Mora l v i e w s are d e v e l o p e d h r o u g h a l i fet ime. T h i s m e a n s that the activit ies that a parent e n g a g e d in, e s p e c i a l l y with d rugs a n d sex , l ong before he or she w a s a parent, f o r m e d h is or her m o r a l v i ews . T h i s d o e s not m e a n that they are s tuck with t h e s e v i e w s , but that they l e l p e d to s h a p e the pa ren ts ' v i ews . For Ins tance, a w o m a n w h o a x p e r i e n c e d e x p e r i m e n t e d with d rugs a s a t e e n a g e r a n d «i*8teregretted it wi l l p robab ly have d e v e l o p e d the m o r a l v i e w that t e e n a g e r s s h o u l d not take d r u g s , b a s e d on (not despi te) the fact that s h e d id it hersel f . C h i l d r e n a n d young t e e n s just don ' t get it, s o it w o u l d be harmfu l for the parent to l a v e t o h ide her d rug u s e wh i l e te l l ing her ch i ld ren not to try d r u g s . Ch i ld ren have s u c h a b lack -and-wh i te v iew of the wor ld that they canno t u n d e r s t a n d that a p e r s o n c o u l d do s o m e t h i n g and :hen regret tt later. C o n f e s s i n g all of o n e ' s young ind isc re t i ons w o u l d m a k e a p a r e n t ' s d i s c u s s i o n s with their ch i ld ren about d r u g s a n d s e x counterproduct ive. In fact, m a n y pa ren ts might c h o o s e not to d i s c u s s t h e s e s u b j e c t s at al l , rather ;han risk c o n f u s i n g their ch i ld ren or revea l ing facts thei r ch i ld ren are too y o u n g to know. O p p o n e n t s of th is v iew might a rgue that once a chi ld l e a r n s the truth abou t h i s or her pa ren ts ' act iv i t ies, he or s h e wi l l dpfpatthp. n a r p n t a n r l wi l l n n I n n n p r SPP t h p n a r p n t g ' v i p w g a g r r p r t i h l p M n u m v o r th ic ic n n t a I n r t i r a l a g g n m n t i n n s i n r p Manage! I ^ & Add to Bundles! List of items in the bundle Comment: WHIthe word "evolved" be morqapproprif... Tags: ^.c/io/oe, Tags: wort_choic*(2). he Add Bundles More S3VK Bursa Close All Annotations All Bundles ToClarify Figure 5.1: Bundle Editor with document and reviewing panes. with the same tags, even though the annotations are not explicitly grouped together. 5.3 Extended Bundle Editor We implemented an extended version of Bundle Editor, a prototype origi-nally developed by Zheng et al. with structured annotation functions [43]. The main interface to the Bundle Editor prototype (Figure 5.1) consists Chapter 5. Incorporating Tagging into Structured Annotations 35 of a document pane and a reviewing pane. The document pane serves as a document editor with basic functionalities such as insert, delete, comment, etc. The reviewing pane consists of multiple tabs, each of which displays a specific "bundle" or group of annotations. For detailed information about the Bundle Editor's basic functionalities, refer to Zheng's Master's thesis [42]. Our extensions to the Bundle Editor focus on different ways in which users can create bundles rather than on automatic bundle creation tech-niques because we believe that automation cannot fully capture the richness and complexity of the annotations used in discussions and workflow man-agement. The Bundle Editor facilitates multiple approaches to creating and man-aging the types of bundles that were previously described. The tool supports top-down grouping of annotations by allowing users to create a bundle (Fig-ure 5.1 A) and then explicitly select annotations to be grouped into that or other bundles (Figure 5.IB). Bottom-up grouping is supported by allowing annotations to be tagged with one or more keywords (Figure 5.1C), and then filtered or selected based on their tags (Figure 5.ID, Figure 5.2). Middle-out grouping is supported by allowing bundles and annotations to be easily added to or removed from existing bundles (Figure 5.IE). In order to achieve all the advantages of tags, as described above, we made tagging pervasive throughout the system. Users can easily associate an annotation with one or more tags simply by typing new tags into a textbox or selecting from a list of existing tags (Figure 5.3). Figure 5.2 shows the dialog box with which users can filter annotations based on AND/OR combinations of tags and other attributes such as annotator and annotation type. In the filter dialog box, the prevalence of a given tag is visible to users through the display of its frequency right next to the tag word. With this implementation of an extended version of the Bundle Editor that robustly supports basic document editing, annotating and structuring of annotations, we proceeded to conduct our final study, in which partic-ipants used our prototype for annotating documents and communicating document-related issues. Chapter 5. Incorporating Tagging into Structured Annotations 36 (Author • Sam • John • Mary |Type • insert • Delete • Replace i I Comment • General Comment |Status • Unread • Read • Accented • Rejected flacjs • toned) • unclear(2) • word_choice(3) Filter annotations in: 0 current tab • "All Annotations" tab Show filtering results in: O current tab • a new tab Cancel F i g u r e 5.2: F i l t e r i n g f u n c t i o n t h a t a l lows users t o f i l ter a n n o t a t i o n s b a s e d o n A N D / O R c o m b i n a t i o n s . Enter your comment: Enter comma separated tags: Or Click to add from the list: 0-iold CTRL key to select multiple items) [Try to u s e m o r e a c a d e m i c w o r d s . tone tone unclear word choice • Make this comment a general comment Cancel QK F i g u r e 5.3: C o m m e n t d i a l o g b o x w h e r e t h e user c a n t y p e i n a n a n n o t a t i o n ' s c o m m e n t text a n d tags . Chapter 6 37 E x p e r i m e n t to E x a m i n e the Impact of S t r u c t u r i n g A n n o t a t i o n s 6.1 Study Goals The Paper Prototype Study described in Chapter 4 showed users' strong support for structuring annotations in a collaborative document. How-ever, important questions regarding the cost-benefit tradeoff associated with structured annotations, and the impact of structuring on the amount of in-formation communicated still remained unadressed. We conducted a con-trolled experiment and compared the use of structured annotation support to unstructured annotation support for annotating documents and commu-nicating document-related issues. We wanted to examine: (1) if under a controlled comparison with no support for structure users would still per-ceive the benefits of structuring their annotations (as they had with the Paper Prototype Study); (2) if the overall workload for structuring anno-tations would be similar to users providing non-structured annotations and then having to communicate any meta information in a text email format (i.e., outside of the document); and (3) if the overall amount of information communicated would differ with structured annotations. Chapter 6. Experiment to Examine the Impact of Structuring Annotations 38 Successfully loaded. File Edit ^ Spell Check Sam v Show Reviewing Pane J^=) Comment Send Emend ing H igh S c h o o l to Five Y e a r s 1. Introduction ' S e n i o r i tch" Is an I n c e s s a n t irritation that h igh s c h o o l s tuden ts all crave to scratch- T h e c u r e ? Gradua t i on . A s s tuden ts p r o g r e s s through their h igh s c h o o l y e a r s g row ing in w i s d o m and maturity, they are d e s p e r a t e for f r e e d o m . Yet wha t they d e s i r e m o s t is not a lways wha t is b e s t for t h e m . A l though m o s t s tuden ts don ' t w i s h to adm i t it. ex tending the h igh s c h o o l ca ree r to five y e a r s w o u l d m a k e an important and bene f i c ia l affect on their future. T h e current four y e a r s of h igh s c h o o l educa t ion do not prov ide suff ic ient t ime for mot ivated s tuden ts to a c c o m p l i s h all of their g o a l s before co l l ege . Mere ly gett ing accep ted by a se lec t ive co l l ege or universi ty requ i res m u c h p r e - p l a n n e d effort, a n d s tuden ts w h o are p r e o c c u p i e d with their s t u d i e s and other act ivi t ies s imp l y do not have e n o u g h t ime. 2. Interest H igh s c h o o l s are a lways a d d i n g n e w a n d in terest ing c o u r s e s to the curr icu la w h i c h they offer. However , m a n y co l l ege -b o u n d s tuden ts do not have suff ic ient t ime to exp lore t h e s e different c l a s s e s . H i g h s c h o o l s tuden ts thus are fo rced to c h o o s e c l a s s e s that fit into their l im i ted s c h e d u l e s . O c c a s i o n a l l y c l a s s e s that perce ive to be e a s i e r are c h o s e n , mere ly to m a k e su re that their t ranscr ip ts s tand out in the a d m i s s i o n s poo l Fif teen y e a r s of age s e e m s too late to start m a k i n g c h o i c e s b a s e d on "career * , rather than c h o o s i n g c l a s s e s that they m i g h t f i n d in terest ing a n d fun. An extra year of h igh s c h o o l w o u l d provide s tuden ts with oppor tun i t ies to explore different s u b j e c t s and take c l a s s e s that they interest. C o n s e q u e n t l y , s tudents w o u l d enjoy their educa t ion a n d deve lop m o r e s t a k e s in s c h o o l . 3. Individuality r . n l l e n e s t h n r n u r i h l v c o n s i d e r t h p w a y s i n w h i r h a n a n n l i r . a n t h a s s n e n t h i s n r h e r f o u r y e a r s n f h i n h s r . h n n l S t u d e n t s w h n E -0 F/fteH ray Manage Add 10 Bundles All AnuoKitioiis All Bundles F i g u r e 6.1: T h e B u n d l e E d i t o r u s e d i n t h e E x p e r i m e n t . P a r t i c i p a n t s were a b l e t o p e r f o r m b a s i c e d i t i n g a n d a n n o t a t i n g tasks . T h e r e v i e w i n g p a n e l has two t a b s b y de fau l t : " A l l A n n o t a t i o n s " t h a t d i s p l a y s a l ist o f a l l s ing le a n n o t a t i o n s , a n d " A l l B u n d l e s " t h a t d i s p l a y s a l ist o f a l l b u n d l e s e m b e d d e d i n t h e d o c u m e n t . Chapter 6. Experiment to Examine the Impact of Structuring Annotations 39 Successfully loaded. & •Mia ta UK, JW* Extending High School to Five Years 1. Introduction "Senior itch" ts an incessant irritation that high school students all crave to scratch. The cure? Graduation As students progress through their high school years growing in wisdom and maturity, they are desperate for freedom. Yet what they desire most is not always what is best for them Although most students don't wish to admit it, extending the high school career to five years would make an important and beneficial affect on their future. The current four years of high school education do not provide sufficient time for motivated students to accomplish all of their goals before college. Merely getting accepted by a selective college or university requires much pre-planned effort, and students who are preoccupied with their studies and other activities simply do not have enough lime. 2. Interest High schools are always adding new and interesting courses to the curncula which they offer. However, many college-bound students do not have sufficient time to explore these different classes. High school students thus are forced to choose classes that fit into their limited schedules. Occasionally classes that perceive to be easier are chosen, merely to make sure that their transcripts stand out in the admissions pool. Fifteen years of age seems too late to start making choices based on "career*, rather than choosing classes that they might find interesting and fun. An extra year of high school would provide students with opportunities to explore different subjects and take classes that they Interest Consequently, students would en|oy their education and develop more stakes In school. 3. Individuality |[<BJj-fr|j: \Fitter\ ISortl i tt" Aimot.itions Figure 6.2: The Simple Editor used in the Experiment. Similar to the Bundle Editor, participants were able to perform basic editing and annotat-ing tasks. However, the Simple Editor's reviewing panel has only one tab: "All Annotations" that displays a list of all single annotations embedded in the document. A simulated email message window was also provided in the Simple system condition (shown at the right), allowing for additional document-related communication directed to hypothetical recipients. 6.2 Methodology We compared two annotation systems: a Bundle system which consists of our extended Bundle Editor (Figure 6.1) and a Simple system which consists of a Simple Editor (an equivalent system that did not support annotation structure), and an email message window (Figure 6.2). Table 6.1 summarizes the differences between the two systems. So as not to bias our participants' perceptions by labeling one as "simple", the two systems were referred to Chapter 6. Experiment to Examine the Impact of Structuring Annotations 40 Bundle System Simple System Interface Com-ponents D o c u m e n t p a n e l , multi-tabbed r e v i e w i n g p a n e l D o c u m e n t p a n e l , single pane r e v i e w i n g p a n e l Communication Support S i n g l e a n n o t a t i o n s w i t h optional user-defined tags, general comments, structured annotations e m b e d d e d i n the d o c u -m e n t a n d l i s t ed i n t h e r e v i e w i n g p a n e l . S i n g l e a n n o t a t i o n s e m -b e d d e d i n t h e d o c u m e n t a n d l i s t e d i n t h e rev iew-i n g p a n e l , a simulated email message window. Filtering Func-tions A N D / O R f i l t er ing o n A u t h o r , T y p e a n d Tag A N D / O R f i l t er ing o n A u t h o r a n d T y p e T a b l e 6.1: C o m p a r i s o n o f B u n d l e s y s t e m a n d S i m p l e s y s t e m . B o t h s y s t e m s were c r e a t e d b y m o d i f y i n g o u r B u n d l e E d i t o r so t h a t t h e y d i f fered o n l y i n the w a y t h e y s u p p o r t e d a n n o t a t i o n s . b y the e x p e r i m e n t e r as the B u n d l e s y s t e m a n d the F i l t e r s y s t e m . 6.2.1 Participants A s i n the O b s e r v a t i o n a l S t u d y , w r i t i n g t u t o r s were r e c r u i t e d t h r o u g h o n l i n e m a i l i n g l ists a n d n e w s g r o u p s . A t o t a l o f 13 p e o p l e (a l l f e m a l e s 1 ) p a r t i c i p a t e d i n the s tudy . T h e y were p a i d $80 for t h e i r t i m e . O n e o f the p a r t i c i p a n t s h a d never u s e d a M i c r o s o f t W i n d o w s s y s t e m . H e r m e d i c a l c o n d i t i o n (ar thr i t i s ) m a d e it d i f f icul t for her to use a t w o - b u t t o n m o u s e o r a k e y b o a r d . F o r th i s r e a s o n , her d a t a was n o t i n c l u d e d i n a n y o f the ana lyses . D a t a f r o m the o t h e r 12 p a r t i c i p a n t s were e x a m i n e d a n d a n a l y z e d . N o n e o f t h e p a r t i c i p a n t s h a d p r e v i o u s l y b e e n i n v o l v e d i n the two ear l ier s tudies . I n t e r m s o f b a c k g r o u n d , a l l twelve p a r t i c i p a n t s u s e d a w o r d p r o c e s s o r r e g u l a r l y ( m a i n l y M i c r o s o f t W o r d ) , a l t h o u g h s ix h a d never u s e d a n y o f i ts a n n o t a t i o n f u n c t i o n s . E l e v e n p a r t i c i p a n t s u s e d a w o r d p r o c e s s o r f r e q u e n t l y (four p a r t i c i p a n t s e v e r y d a y a n d seven e v e r y two t o t h r e e d a y s ) a n d felt v e r y conf ident i n t h e i r usage . T h e r e m a i n i n g p a r t i c i p a n t o n l y u s e d a w o r d p r o -xThe population that we recruited from consisted mostly of females, and only females responded to our call for participation. Chapter 6. Experiment to Examine the Impact of Structuring Annotations 41 cessor once a month and her confidence was relatively low. Al l participants had reviewed documents more than ten times. Seven participants had pre-viously been involved in collaborative authoring, three more than five times, and four fewer than five times. 6.2.2 Task Participants were asked to review and annotate two documents, one with each of the systems. They were asked to assume a role as a collaborator within a group of three co-authors. The given documents were assumed to be drafted by the other two co-authors with some sections explicitly noted as being jointly drafted by the two, and others separately by each. Unlike in our Paper Prototype Study, here participants were expected to create annotations, both comments as well as direct edit changes to the docu-ment text. They were also requested to provide: (1) their general impression of the writing, and (2) a brief summary of their review that indicated the overall status of the document to help their co-authors skim the document quickly and prioritize the remaining work. The requested feedback was rep-resentative of common meta-comments communicated between annotators and recipients, as found in our Observational Study. The task documents were based on sample essays for the writing sections of the GRE (General Record Examination)2 and ACT (American College Testing)3 tests; the contents were on topics of general interest. As shown in Appendix C.3.2, neither of the task documents included any annotations. The documents were manipulated to be isomorphic; they had the same number of problems planted at similar locations throughout the documents. Both syntax problems (e.g., grammar errors, incorrect verb usages) and semantic problems (e.g., unclear statements, unsupported arguments) were carefully planted with the expectation that participants would create both local (sentence-level) and global (document-level) annotations. 2 GRE is a standardized test for graduate school admissions in North America. The sample GRE essays were collected from http://www.west.net/stewart/ws/. 3 ACT is a standardized test for undergraduate college admissions. The sample ACT essays were collected from http://www.actstudent.org/writing/sample/index.html Chapter 6. Experiment to Examine the Impact of Structuring Annotations 42 Categories of planted problems Num of instances Unclear statements 6 Flawed logic 3 Semantic Title inconsistency 3 Unsupported argument 2 Subtotal: 14 Inconsistent wording 2 Incorrect use of active-passive verbs 4 Incorrect use of verb tense 5 Syntax/lexical Use of contractions 2 Inappropriate word choice 2 Subtotal: 19 Total: 33 Table 6.2: Problems planted in each task document. The nine categories are illustrated in Appendix C.3.3. Prior to the experiment, to test the manipulations, two independent raters with expertise in writing and reviewing documents were asked to identify problems in the documents. They independently identified 72% and 70% of the planted problems across both documents, respectively, with a 60% overlap in the problems identified. The problems that were not iden-tified by either of the two raters were removed from the task documents. In the end, each document had a total of 33 planted problems (as shown in Table 6.2). Appendix C.3.3 shows these problems in the context of the study documents. A third document was used during practice sessions. Be-cause the same practice document was used in every configuration, we did not control the number or the types of problems in the practice document. 6.2.3 Procedure Each participant had a single four-hour session. It began with a demo-graphic questionnaire to obtain past computer, co-authoring and reviewing experience. Participants then saw a training video on general concepts such as collaborative writing and annotations, and how to use their first assigned Chapter 6. Experiment to Examine the Impact of Structuring Annotations 43 system. To ensure that all participants would have a similar level of famil-iarity with a system's functionality, participants had 15 minutes to perform a set of guided annotation tasks to gain experience with that system. They were provided with a list of annotations to create in the practice document. After the 15-minute practice session, participants had one hour to perform the experimental task on their first document with their first system. A questionnaire followed to collect feedback on that system. Participants were given a short 10 minute break and were then shown a training video on how to use their second system, followed by a 15-minute practice session, then the experimental task on their second document with their second system. Another questionnaire was conducted to gather feedback on that system. To solicit comparative feedback between the two systems, a final questionnaire was administered. We will refer to this final questionnaire as the comparative questionnaire and the previous two questionnaires as system-specific ques-tionnaires. Appendix C.2 shows all the questionnaires used in the study. A short semi-structured interview to collect further information regarding preferences and perceived performance ended the sessions. 6.2.4 Design The experiment used a within-subjects (system type) factorial design. Docu-ment was a within-subjects control variable, and both system and document presentation orders were between-subject controls. To minimize learning ef-fects, we counterbalanced the order of presentation for both system type and document, resulting in four configurations. 6.2.5 Measures The amount of communication among co-authors was assessed by counting the number of (1) single annotations and (2) meta-comments. Single an-notations were the same in both systems: edits and comments. Counting meta-comments differed between the two systems. In the Bundle system, each unique bundle and tag as well as each general comment counted as one meta-comment. In the Simple system, email content was analyzed to extract Chapter 6. Experiment to Examine the Impact of Structuring Annotations 44 meta-information items; e.g., a statement saying "Try to use more academic words in the places I highlighted" was counted as one meta-comment. To ensure that all meta-comments that were created were counted, any sin-gle comments anchored at the beginning or the end of documents in both systems were also analyzed to see if they contained meta-information. Self-reported measures from the two system-specific questionnaires were used to assess subjective workload measures associated with each task using the NASA-TLX workload index4. We also included other measures such as performance satisfaction, time satisfaction, reviewing efficiency, reviewing accuracy, and ease of use at the end of each system-specific questionnaire to assess perceived benefits associated with each system. The comparative questionnaire included the same set of questions as in the system specific questionnaires. Each question asked participants to rate the two systems on a single 10-point Likert scale (l:low and 10:high) using different notations (e.g., using ' X ' for the Bundle system, and 'O' for the Simple system). Dur-ing the interview, participants were asked to comment on the cost-benefit tradeoff of using each system. 6.2.6 Hypotheses Communication Hypotheses: (a) Participants will create more meta-comments in the Bundle system than in the Simple system. (Creating struc-tured annotations is an easier way to provide meta-commentary than doing so separately in the body of an email.) (b) Participants will create similar numbers of single annotations in both systems. (Both systems have identi-cal support for single annotations except for tags, which are a type of single annotation that is only in the Bundle system.) Workload Hypothesis: Reviewing with the Bundle system will not re-quire significantly higher workload than reviewing with the Simple system. (The added effort to group and tag annotations will not be greater than 4 N A S A - T L X is a standardized multi-dimensional rating procedure that derives an over-all workload score based on a weighted average of ratings on six workload categories[15]. Chapter 6. Experiment to Examine the Impact of Structuring Annotations 45 that required to compose a detailed email message with the equivalent in-formation.) Cost-Benefit Hypothesis: Participants will perceive the net gain — the amount by which the benefit exceeds the cost — to be higher in the Bundle system than in the Simple system. 6.3 Results We report on the quantitative data along with the qualitative data and feedback provided during the interview. Before testing our hypotheses, we checked to make sure that there was no effect of document; we ran a series of 2 documents x 2 orders of systems x 2 orders of documents ANOVA tests on our dependent measures, which showed no significant main or interaction effects of document5. Satisfied that there was no effect of document, we then ran a series of 2 systems x 2 orders of systems x 2 orders of documents ANVOA tests to evaluate our hypotheses. Along with statistical significance for each of these ANOVA tests, we report partial eta-squared n2, a measure of effect size, which is often more informative than statistical significance in applied human-computer interaction research [19]. To interpret this value, 0.01 is a small effect size, 0.06 is medium, and 0.14 is large [9]. 6.3.1 Communication Participants created an average of 64.5 single annotations (sd=27.5) in the Bundle system and 76.1 (sd=40.2) in the Simple system, a difference that was not statistically significant (F(l,8)=2.20, p=0.18, r/2=0.22). Figure 6.3 shows the number of single annotations created by participants. Although 5Results from testing main effects on our three dependent measures were as follows: number of single annotations, (F(l,8)=1.12, p=0.32,7j2=0.12); number of meta-comments, (F(l,8)=1.46, p=0.26, ??2=0.16); and TLX workload measure, (f(l,8)=0.61, p=0.46, r)2=0.07). While none of these tests showed a significant effect of document, we note that there were large effect sizes for the number of single annotations and meta com-ments. Given that we fully counterbalanced the order of documents, this is not a big concern for our study, but it would be interesting to examine this further. Chapter 6. Experiment to Examine the Impact of Structuring Annotations 46 15CH Simple System Bundle System F i g u r e 6.3: B o x - p l o t o f t h e n u m b e r o f s ingle a n n o t a t i o n s c o m m u n i c a t e d i n e a c h s y s t e m ( N = 1 2 ) . T h e b a r s s h o w t h e r a n g e o f va lues , t h e b o x e s s p a n b e t w e e n t h e first a n d t h i r d q u a r t i l e s , the h o r i z o n t a l l ines w i t h i n t h e boxes i n d i c a t e t h e m e d i a n va lues . t h i s l a c k o f effect was e x p e c t e d a n d s u p p o r t s o u r c o m m u n i c a t i o n h y p o t h e s i s for these t y p e s o f a n n o t a t i o n s , we n o t e t h a t a l arge effect s ize was f o u n d , s u g g e s t i n g t h a t it m a y b e p r u d e n t t o v a l i d a t e t h i s finding w i t h f u r t h e r re-search . I n t e r m s o f m e t a - c o m m e n t s , two m a i n categor ies were o b s e r v e d : (1) r e c i p i e n t - b a s e d (e.g., a t o - d o list for N i c k ) a n d (2) p r o b l e m - b a s e d (e.g., t o n e o f t h e d o c u m e n t ) . T h e categor ies were n o t exc lus ive ; i n s o m e cases, t h e s a m e a n n o t a t i o n ( s ) was (were) c o u n t e d as b o t h t y p e s , e.g., a n a n n o t a t i o n t a g g e d w i t h t h e c o n t e n t - r e l a t e d info " a r g u m e n t " a n d a s s o c i a t e d w i t h a t o - d o b u n d l e " F o r M a r y , " o r a s t a t e m e n t s a y i n g " M a r y , y o u s h o u l d w a t c h o u t for Chapter 6. Experiment to Examine the Impact of Structuring Annotations 47 u n s u p p o r t e d c l a i m s i n p a r a g r a p h 3." P a r t i c i p a n t s c r e a t e d s ign i f i cant ly m o r e m e t a - c o m m e n t s ( F ( l , 8 ) = 1 3 . 0 9 , p = 0 . 0 1 , r / 2 =0 .62 ) w h e n r e v i e w i n g w i t h the B u n d l e s y s t e m ( a v g = 6 . 7 , s d = 3 . 3 ) t h a n r e v i e w i n g w i t h t h e S i m p l e s y s t e m ( a v g = 3 . 7 , s d = 2 . 2 ) , a lso s u p p o r t -i n g o u r c o m m u n i c a t i o n h y p o t h e s i s . F i g u r e 6.4 shows the n u m b e r o f m e t a -c o m m e n t s c r e a t e d b y p a r t i c i p a n t s . Interes t ing ly , we f o u n d a s ign i f i cant s y s t e m o r d e r effect o n t h e n u m -b e r o f m e t a - c o m m e n t s ( F ( l , 8 ) = 1 7 . 4 4 , p < 0 . 0 1 , r / 2 = 0 . 6 9 ) . P a r t i c i p a n t s w h o were e x p o s e d to t h e B u n d l e s y s t e m first i n c l u d e d s i g n i f i c a n t l y m o r e m e t a -c o m m e n t s across b o t h s y s t e m s (avg=7 .0 , s d = 2 . 4 ) t h a n those w h o u s e d the S i m p l e s y s t e m first ( a v g = 3 . 3 , s d = 1 . 6 ) . O n e e x p l a n a t i o n is t h a t t h e B u n d l e s y s t e m f a c i l i t a t e d m e t a - c o m m e n t s i n the first r e v i e w i n g ta sk , l e a v i n g p a r -t i c i p a n t s w i t h the i n c l i n a t i o n t o s i m i l a r l y p r o v i d e m o r e i n f o r m a t i o n i n t h e s e c o n d task . C o n s i s t e n t w i t h the q u a n t i t a t i v e d a t a , m a n y p a r t i c i p a n t s also s a i d t h a t t h e y were ab le to p r o v i d e a m o r e c o m p r e h e n s i v e r e v i e w u s i n g the B u n d l e s y s t e m . O n e p a r t i c i p a n t s a i d "[The Bundle system] maximizes the inter-action between the writers," a n o t h e r s a i d "[When] you need a more criti-cal approach [it] gives you the exact tools," a n d a n o t h e r p a r t i c i p a n t a d d e d "I could communicate more information [that] I think is important to get across. . . . 7 was not just correcting the problems; I had a chance to explain why, like to justify it." T h e s e q u a l i t a t i v e c o m m e n t s are cons i s t ent w i t h o u r c o m m u n i c a t i o n h y p o t h e s e s . 6.3.2 Self-assessed Workload P e r c e i v e d average w o r k l o a d w i t h t h e B u n d l e s y s t e m , as m e a s u r e d b y the N A S A - T L X , was 69.8 ( sd=8 .0 ) w h i l e t h a t a s s o c i a t e d w i t h t h e S i m p l e s y s t e m was 63.0 ( sd=11 .8 ) , w h i c h was a m a r g i n a l l y s ign i f i cant di f ference ( i r ( l , 8 ) = 4 . 5 3 , p = 0 . 0 7 a n d r)2—0.36). T h e N A S A - T L X m e a s u r e s w o r k l o a d o n a n o v e r a l l scale o f 0:low to 100:high. F i g u r e 6.5 shows e a c h i n d i v i d u a l p a r t i c i p a n t ' s r e p o r t e d T L X w o r k l o a d . It i n d i c a t e s t h a t e ight o u t o f twe lve p a r t i c i p a n t s a s s o c i a t e d a h i g h e r w o r k l o a d w i t h the B u n d l e s y s t e m , two the reverse , a n d Chapter 6. Experiment to Examine the Impact of Structuring Annotations 48 124 H , , ' Simple System Bundle System F i g u r e 6.4: B o x - p l o t o f n u m b e r o f m e t a - c o m m e n t s c o m m u n i c a t e d i n e a c h s y s t e m ( N = 1 2 ) . T h e b a r s s h o w t h e r a n g e o f va lues , t h e boxes s p a n b e t w e e n t h e first a n d t h i r d q u a r t i l e s , the h o r i z o n t a l l ines w i t h i n t h e boxes i n d i c a t e the m e d i a n va lues . t h e r e m a i n i n g two r a t e d w o r k l o a d equa l ly . F i n d i n g a di f ference i n w o r k l o a d was n o t cons i s tent w i t h o u r h y p o t h e s i s , b u t m a y b e e x p l a i n e d b y t h e fact t h a t p a r t i c i p a n t s i n c l u d e d m o r e m e t a - c o m m e n t i n f o r m a t i o n i n the B u n d l e s y s t e m , t h u s r e q u i r i n g m o r e w o r k ; a l t ernate ly , it c o u l d b e b e c a u s e o f u s a b i l -i t y issues t h a t were u n c o v e r e d w i t h t h e B u n d l e s y s t e m , as d e s c r i b e d l a t e r i n t h i s c h a p t e r . M o r e re search is r e q u i r e d t o tease t h i s a p a r t . R e s u l t s for e a c h i n d i v i d u a l w o r k l o a d c a t e g o r y a n d benef i t q u e s t i o n are d i s p l a y e d i n T a b l e 6.3, as a n a l y z e d u s i n g the W i l c o x o n S i g n e d - R a n k t e s t 6 . O n average , the B u n d l e s y s t e m r e q u i r e d s i g n i f i c a n t l y h i g h e r m e n t a l d e m a n d (p=0 .01) a n d m o r e effort (p=0 .03) t h a n t h e S i m p l e s y s t e m . T h e average t e m p o r a l d e m a n d a s s o c i a t e d w i t h the B u n d l e s y s t e m was m a r g i n a l l y signif-i c a n t l y h i g h e r t h a n t h a t a s s o c i a t e d w i t h t h e S i m p l e s y s t e m (p=0 .07) . T h e average r a t i n g s for p e r f o r m a n c e a n d proces s s a t i s f a c t i o n a s s o c i a t e d w i t h the S i m p l e s y s t e m were s t a t i s t i c a l l y s i g n i f i c a n t l y h i g h e r t h a n those a s s o c i a t e d w i t h t h e B u n d l e s y s t e m (p=0 .04 a n d p = 0 . 0 1 r e s p e c t i v e l y ) . P a r t i c i p a n t s a lso felt t h a t the S i m p l e s y s t e m was easier t o use t h a n t h e B u n d l e s y s t e m (p=0 .02 ) . T h e s e findings m a y b e e x p l a i n e d b y the fact t h a t t h e features a n d t h e i n t e r a c t i o n t e c h n i q u e s r e q u i r e d i n the B u n d l e s y s t e m were m o r e so-6 Neither a t-test nor A N O V A was used because data was not normally distributed. Chapter 6. Experiment to Examine the Impact of Structuring Annotations 49 Figure 6.5: Overall workload associated with each system (N=12). phisticated than those in the Simple system; hence, some participants might have needed more training and practice than they were given in the study, and felt less comfortable with the Bundle system. More research is needed to improve the usability of the Bundle system, and we should examine the workload associated with the system over an extended period of use. Chapter 6. Experiment to Examine the Impact of Structuring Annotations 50 Mean Std. Deviation Sig.' Simple Bundle Simple Bundle Mental Demand 6.33 8.25 1.97 1.77 .01* Physical Demand 3.67 5.00 1.97 2.70 .12 Temporal Demand 4.25 5.25 2.34 2.73 .07 Performance 7.83 7.08 1.59 1.56 .04* Effort 6.08 7.33 2.11 1.83 .03* Frustration 4.17 5.00 2.79 2.52 .40 Process Satisfaction 8.58 6.83 1.08 2.08 .01* Time Satisfaction 8.50 8.17 1.24 1.47 .49 Reviewing Efficiency 7.92 7.25 1.38 2.26 .55 Reviewing Accuracy 8.08 7.50 1.08 2.32 .91 Ease of Use 8.08 6.50 2.43 2.47 .02* Table 6.3: Mean responses to self-reported measures with the scale l:low to 10:high. A * indicates that the difference between the means are statistically significant with p<0.05 (N=12). 6.3.3 Cost-Benefit During the interview, participants were asked to comment on the cost-benefit tradeoff for using each system. Eleven of twelve participants found both sys-tems to be useful and to have positive net gain (the benefit outweighed the cost). Among these 11 participants, eight said the net gain was higher in the Bundle system than in the Simple system, and that they would defi-nitely like to use the Bundle system in their future annotating tasks. These participants acknowledged that even though the cost of using the Bundle system was higher than for the Simple system (as reflected in our workload measure), the Bundle system would return a much greater benefit, especially over iterative collaborations. For example, one participant explained that because she was able to provide the authors with a more comprehensive review using the Bundle system, "going forward, if I am working with [the same co-authors] again, they would already know what kind of things I am looking for. So it's like I do the front heavy loading [by putting in extra Chapter 6. Experiment to Examine the Impact of Structuring Annotations 51 Cost (a=0.87) Mental Demand Physical Demand Frustration Benefit (a=0.76) Reviewing Efficiency Reviewing Accuracy Table 6.4: Factors used for measuring cost and benefit and their Cronbach's alpha values. effort for this first iteration] ... As you front load the work, as you go through [over iterations], it only gets easier." The remaining three participants (out of eleven) mentioned that al-though the Bundle system's benefit was higher than the Simple system, the cost associated with the Bundle system was much higher, resulting in a lower net gain compared to the Simple system. Nonetheless, they said that they would use the added functionality in the Bundle system in some of their future annotating tasks when they needed to provide detailed and precise feedback on larger documents. The remaining one participant believed that the returned benefit was not worth the cost for either system. When forced to choose between the two systems for her future use, she chose the Simple system because she preferred to give free-form non-structured feedback similar to the verbal feedback to which she was more accustomed. Thus, while the majority of participants thought that structuring annotations was worth the effort, there was clearly some diversity of opinion on this point. From the comparative questionnaire data, we computed an estimate of the participants' perceived cost and benefit associated with one system rel-ative to the other. We used an average of the responses to the Mental De-mand, Physical Demand, and Frustration from the T L X index as a measure of cost, and the Reviewing Efficiency and Reviewing Accuracy as a measure of benefit (see Table 6.4). We did not include the other dimensions, Tem-poral Demand, Performance, Effort, Process Satisfaction, Time Satisfaction and Ease of Use in the calculations because of their lack of consistency with Chapter 6. Experiment to Examine the Impact of Structuring Annotations 52 other measures, as was evident from low Cronbach's alpha values7 (a), a reliability measure. As shown in Figure 6.6, perceived cost associated with the Bundle sys-tem was 7.4 (sd=1.8) while that associated with the Simple system was 5.0 (sd=2.2), a significant difference (i(ll)=6.06, p<0.00) according to a paired t-test. Perceived benefit associated with the Bundle system was 7.9 (sd=1.6) while that associated with the Simple system was 7.8 (sd=1.2), a non-significant difference (i(ll)=6.06, p=0.93). Thus, the benefit measures were similar in the two systems but the cost measure was higher in the Bundle system than in the Simple system. At first glance, these quantitative results from the comparative question-naire seem inconsistent with the qualitative feedback obtained during the interview. The most probable explanation, however, is that the quantitative questionnaire responses reflected the cost-benefit tradeoff with respect to the study tasks that participants had just finished for the given scenario, while the qualitative interview reflected the tradeoff with respect to participants' envisioned future use of an annotation system (on larger documents and over iterations). Clearly, additional research will be necessary to probe the cost-benefit tradeoff associated with the extended use of the Bundle system over time. 6.3.4 Summary of Results We summarize our results for each of our hypotheses. Communication Hypotheses: these were supported. Participants cre-ated more meta-comments in the Bundle system than in the Simple system, and similar numbers of single annotations in both systems. 7 Cronbach's alpha measures how well a set of items (or variables) measures a single uni-dimensional construct. When data have a multi-dimensional structure, Cronbach's alpha will usually be low. To interpret Cronbach's alpha values, 0.7 is considered to be an acceptable reliability coefficient although lower thresholds are sometimes used in the literature [32]. Chapter 6. Experiment to Examine the Impact of Structuring Annotations 53 1<H 8H 6H 4 H 2H oH Simple System Bundle System Costs Benefits Figure 6.6: Box-plot of costs and benefits measures for each system (N=12). The bars show the range of values, the boxes span between the first and third quartiles, the horizontal lines within the boxes indicate the median values. A circle identifies an outlier. Workload Hypothesis: this was not supported. Reviewing with the Bundle system showed significantly higher workload than reviewing with the Simple system. Cost-Benefit Hypothesis: this was partially supported. According to the qualitative feedback, the majority of participants thought that the Bun-dle system offered higher net gain, while the quantitative results suggested Chapter 6. Experiment to Examine the Impact of Structuring Annotations 54 otherwise. 6.3.5 Other Measures: Usage of Structures Bundles: Eleven out of twelve participants created bundles. Analysis of the bundles8 and their associated annotations revealed that bundles were used to communicate both problem-based (59%) and recipient-based (41%) types of meta-information. Problem-based bundles were created using four different approaches: (1) unique-tag based, (2) clustered-tags based, (3) tag-less, and (4) type based. In the unique-tag based approach, bundles were created by group-ing annotations based on a particular tag. For example, a bundle named "word choices" was created by manually selecting annotations tagged with "word choices" or by saving a filter result on the tag "word choices." In the clustered-tag based approach, bundles were created by manually selecting and aggregating all or a subset of annotations that were labeled with similar tags. For example, a bundle named "Argument" was created by grouping annotations that were labeled with tags named "argument" and "unclear." In the tag-less approach, bundles were created by manually selecting and grouping annotations that did not have any tags. Lastly, in the type based approach, a bundle was created by saving a filter result on a particular type of annotation, such as "General Comment." Figure 6.7 shows the number of bundles created using each approach. The majority of problem-based bun-dles were created based on a unique tag or a set of similar tags, using the "bottom-up" structuring approach previously described. The majority of participants mentioned that creating bundles using this approach was easier and less time-consuming than the top-down approach, which was used to create recipient-based bundles. Recipient-based bundles were created by manually selecting and group-ing annotations that had diverse sets of associated tags. We believe that these recipient-based bundles were created to help the hypothetical recipi-ents manage their workflows. This was explained explicitly by one partici-8One empty bundle, i.e., a bundle that did not have any associated annotations, was excluded from the analysis. Chapter 6. Experiment to Examine the Impact of Structuring Annotations 55 Total Bundles Created: 29 • Unique-tag based bundles (6 participants) • Clustered-tags based bundles (3 participants) • Tag-less bundles (1 participant) • Type-based (1 participant) F i g u r e 6.7: A l l b u n d l e s c r e a t e d b y 11 p a r t i c i p a n t s . p a n t , w h o ref lected o n o n e o f her p r e v i o u s c o l l a b o r a t i v e w r i t i n g exper i ences a n d s t a t e d t h a t : "Bundles would have been useful for addressing the co-authors' problems individually. They didn't have the same ... errors. So, being able to separate them out, say you need to work on this, you need to work on that. But then also for the things that they were working on together, [bundles would allow me] to be able to combine them as well. So it's a way of both separating them out Chapter 6. Experiment to Examine the Impact of Structuring Annotations 56 but then making [them] more cohesive at the same time." A n o t h e r p a r t i c i p a n t e x p l a i n e d t h a t w o r k f l o w - r e l a t e d i n f o r m a t i o n c o m -m u n i c a t e d t h r o u g h b u n d l e s c o u l d h e l p rec ip i en t s r e v i e w a n n o t a t i o n s effi-c i e n t l y b e c a u s e "everything that requires a certain way of dealing with is together [in bundles]" a n d hence , c o - a u t h o r s "don't have to keep switching their mindset from one thing to another." Tags: A l l twelve p a r t i c i p a n t s a d d e d tags to t h e i r a n n o t a t i o n s . T a g s were u s e d t o d e s c r i b e speci f ic p r o b l e m ( s ) o f the a n n o t a t i o n s t h a t t h e y were a d d e d to . F o r e x a m p l e , one a n n o t a t i o n t h a t p o i n t e d o u t a n a r g u m e n t p r o b l e m as " T h i s b r i n g s i n a w h o l e s e p a r a t e issue t h a t y o u ' r e n o t p r e p a r e d t o d e a l w i t h i n t h i s p a p e r , " h a d a t a g n a m e d "argument ." It was s u r p r i s i n g t h a t p a r t i c i p a n t s d i d n o t use tags at a l l t o c o m m u n i c a t e r e c i p i e n t - b a s e d m e t a - c o m m e n t s ; t h e y u s e d tags e x c l u s i v e l y t o c o m m u n i c a t e p r o b l e m - b a s e d k i n d s o f i n f o r m a t i o n . P a r t i c i p a n t s s a i d t h a t t h e i n f o r m a t i o n c o m m u n i c a t e d t h r o u g h tags b r o u g h t "awareness to patterns of problems in documents," a n d w o u l d a l low c o - a u t h o r s to see the s t r e n g t h s a n d weaknesses o f t h e i r w r i t i n g , a n d also to ach ieve a q u i c k o v e r v i e w o f t h e c u r r e n t d o c u m e n t s ta tus . O n e p a r t i c i p a n t f u r t h e r s t a t e d t h a t the i n f o r m a t i o n i n tags a l l o w e d her to ach ieve "a greater perspective on the reviewing process" t h a t she h a d gone t h r o u g h . W e f o u n d t h a t tags were s o m e t i m e s u s e d as a l t e r n a t i v e s to l o n g c o m m e n t s w h e n a d d r e s s i n g r e c u r r i n g p r o b l e m s . T h i s was e x p l a i n e d e x p l i c i t l y b y one p a r t i c i p a n t d u r i n g the i n t e r v i e w , "One can comment the first time one runs into a problem. But after that, [tags] are like reminders, almost to go back to that comment." Relationship between bundles and tags: T a b l e 6.5 p r o v i d e s a h i g h -leve l d e s c r i p t i o n o f the b u n d l e s a n d tags t h a t p a r t i c i p a n t s c r e a t e d . O n average , e a c h p a r t i c i p a n t g r o u p e d 31% o f her a n n o t a t i o n s i n t o one o r m o r e b u n d l e s , a n d l a b e l e d 23% o f t h e a n n o t a t i o n s w i t h o n e or m o r e tags; 10% o f the a n n o t a t i o n s were b o t h t a g g e d a n d b u n d l e d , w h i l e 56% were n e i t h e r t a g g e d n o r b u n d l e d . T h e average n u m b e r s o f b u n d l e s a n d u n i q u e tags cre-a t e d p e r p a r t i c i p a n t were 2.4 ( sd=1 .4 ) , a n d 4.8 ( sd=2 .8 ) , respec t ive ly . W h i l e Chapter 6. Experiment to Examine the Impact of Structuring Annotations 57 # of # of % of # of # of % of bundles anno anno unique anno anno in each bundled tags with tagged bundle each tag Average 2.42 7.68 30.85 4.75 3.32 23.01 Median 2.00 5.50 13.70 4.00 3.09 17.17 Std Dev 1.44 5.61 33.14 2.80 1.85 18.94 Min 0.0 0.0 0.0 1.0 1.14 9.09 Max 5.0 19.4 100.0 11.0 7.5 81.36 Table 6.5: Descriptive summary of bundles and tags (N=12). a greater number of unique tags were created than bundles, a higher per-centage of annotations were associated with bundles than with tags; the number of annotations per bundle (avg=7.7, sd=5.6) was higher than that per tag (avg=3.3, sd=1.9). This may be explained by the fact that tags were used more to label comments than edits, while bundles were used al-most equally to organize both comments and edits. On average, participants tended to create more edit annotations than comment annotations; 71% of all annotations were edits in each document. In order to better understand how bundles and tags were related, bundles are displayed in Figure 6.8 as a function of the number of unique inherited tags (tags attached to the grouped annotations in a bundle) and the number of annotations in each bundle. The number of unique inherited tags was small for most bundles. 6.3.6 Additional Qualitative Feedback Usability of the Bundle system: Six participants suggested that the Bundle system needed to be more intuitive and straightforward. The inter-action technique for adding/removing annotations to/from bundles was a bit cumbersome: a separate tab for each bundle had to be opened and more than one button clicks were required to add/remove each annotation. Im-proving the usability of the system would involve implementing more efficient Chapter 6. Experiment to Examine the Impact of Structuring Annotations 58 Figure 6.8: Bundles as a function of number of unique inherited tags and number of annotations; the size of each bubble shows the number of bundles at each coordinate. interaction techniques for annotation organization, such as drag-and-drop. Preference for Structuring: Four of the six participants who were ex-posed to the Bundle system first said that while performing the second task with the Simple system, they wished it had some of the Bundle system func-tionality (e.g., tagging or grouping annotations). They felt that they could not provide feedback "as precise and thorough as in the Bundle system" and they had difficulty "explaining how problems [were] connected and uniting comments." 6.4 Discussion Structured annotations are worth the effort: Almost all participants (11/12) believed that structuring annotations offered positive net gain; struc-tured annotations are worth the effort. Furthermore, the majority of partici-pants (8/12) stated a preference to use structured annotations in their future Chapter 6. Experiment to Examine the Impact of Structuring Annotations 59 w o r k b e c a u s e t h e y b e l i e v e d t h a t s t r u c t u r i n g a n n o t a t i o n s offered higher net g a i n re la t ive to s i m p l e a n n o t a t i o n s . F o r t h e r e m a i n i n g p a r t i c i p a n t s , t h e p e r c e i v e d benef i ts d i d n o t suff ic ient ly o u t w e i g h t h e a d d i t i o n a l w o r k l o a d as-s o c i a t e d w i t h s t r u c t u r e d a n n o t a t i o n s ; neverthe less t h e y s a i d t h a t t h e y w o u l d use s t r u c t u r e d a n n o t a t i o n s i n c e r t a i n c o n t e x t s w h e r e t h e y n e e d e d to a n n o -t a t e d o c u m e n t s t h o r o u g h l y . Top-down and bottom-up approaches complement each other: A l -t h o u g h the b o t t o m - u p a p p r o a c h was c o n s i d e r e d to b e less t i m e c o n s u m i n g a n d m o r e l i gh t -we ight , it d i d n o t d i s p l a c e t h e t o p - d o w n a p p r o a c h . B o t h a p -p r o a c h e s were u s e d b y a l m o s t a l l p a r t i c i p a n t s ( 1 1 / 1 2 ) . T h e o n e p a r t i c i p a n t w h o d i d n o t use b u n d l e s c o m m e n t e d d u r i n g t h e i n t e r v i e w t h a t she r a n o u t o f t i m e t o create b u n d l e s t o w a r d s t h e e n d o f t h e task . W e suspec t t h a t a t o p -d o w n g r o u p i n g a p p r o a c h was u s e d to create r e c i p i e n t - b a s e d b u n d l e s b e c a u s e the i n f o r m a t i o n r e g a r d i n g i n t e n d e d c o - a u t h o r (s) was k n o w n p r i o r to a n n o -t a t i n g t h e d o c u m e n t s . C o n v e r s e l y , a b o t t o m - u p a p p r o a c h was u s e d t o create p r o b l e m - b a s e d b u n d l e s p r o b a b l y b e c a u s e t h e s t r u c t u r e was f o r m u l a t e d o n l y after r e a l i z i n g r e l a t i o n s h i p s a m o n g a n n o t a t i o n s . H e n c e the t o p - d o w n a n d b o t t o m - u p a p p r o a c h e s s u p p o r t s t r u c t u r i n g a n n o t a t i o n s i n a c o m p l e m e n t a r y way, a n d b o t h s h o u l d b e s u p p o r t e d i n a n a n n o t a t i o n - s t r u c t u r i n g t o o l . Structured annotations promote valuable communication: P a r t i c -i p a n t s c r e a t e d m o r e m e t a - c o m m e n t s i n t h e B u n d l e s y s t e m t h a n i n t h e S i m -p le s y s t e m . P a r t i c i p a n t s ' r e m a r k s o n the usefulness o f b u n d l e s a n d tags as d e s c r i b e d ear l ier suggest t h a t the m e t a - i n f o r m a t i o n c o m m u n i c a t e d i n t h e f o r m of s t r u c t u r e d a n n o t a t i o n s was p e r c e i v e d t o b e v a l u a b l e a n d bene f i c ia l for b o t h the i n t e n d e d rec ip i en t s a n d the a n n o t a t o r s themse lves , b y a l l o w i n g for a c o m p r e h e n s i v e r e v i e w t h a t went b e y o n d s i m p l e a n n o t a t i o n s , p r o v i d -i n g for a greater p e r s p e c t i v e o n the r e v i e w i n g process , s u p p o r t i n g w o r k f l o w m a n a g e m e n t , a n d of fer ing a q u i c k o v e r v i e w o f the d o c u m e n t s ta tus . 60 Chapter 7 Conclus ions and Fu tu re W o r k We assessed the impact of supporting structured annotations on users who create annotations. The Observational Study strengthened our understand-ing of how annotators use existing tools to communicate document-related information in the form of annotations. That study also revealed annotation practices that could benefit from additional structuring support, such as an-notators describing how annotations in a document relate to each other. In the Paper Prototype Study, we investigated how annotators would struc-ture annotations, if given the option. Common behaviour that emerged was the grouping of thematically related annotations, as well as the grouping of annotations specifically targeted to a given co-author. The study also uncovered a range of temporal approaches to structuring annotations, such as top-down and bottom-up grouping. In the Experiment, we compared the use of structured annotation support to unstructured annotation support for annotating documents and communicating document-related issues. We found that structured annotations were perceived to be worth the effort de-spite the additional workload, and that the tag-based grouping approach that we added to an existing annotation model [43] complements the origi-nal hierarchical approach in describing relationships among annotations in the document. 7.1 Limitations In all three of our studies, all participants were provided with documents to perform the tasks. In order to keep the tasks manageable,we chose relatively Chapter 7. Conclusions and Future Work 61 simple documents in terms of length compared to the documents that our system would likely be used for in actual practice. The generalizability of our results to larger documents is thus somewhat limited. Moreover, we carried out our research as laboratory studies. As with any lab study, there is a trade-off between realism and generalizability for increased precision [23]. We tried to address the limitations and maintain a degree of ecological validity by designing the study scenarios to be representative of those com-monly found in co-authoring projects, and designing the task documents to have the types of errors common in writing. Another limitation lies in our choice of involving professional writing tu-tors as participants in the Observational Study and the Experiment. We recruited professional writing tutors as participants because of their expe-rience in annotating documents, as described in the Chapter 1. However, we realize that tutors are not necessarily identical in their work practices to small groups of co-authors. Further investigation may be required to better understand the generalizability of our results to co-authors, or to other user groups. 7.2 Future Work We envision our structured annotation model as being integrated into ex-isting word processors (such as Microsoft Word or Google Docs) to support document-embedded communication among co-authors. A few avenues for future research are summarized in this section. 7.2.1 Streamlining the Additional Workload Our findings from the Experiment indicate that creating structured anno-tations increases workload. It is possible that with more exposure and an extended period of use, the workload associated with the Bundle system might decrease. Nevertheless, further work is needed to explore how the additional workload can be streamlined. As discussed earlier, improving the usability of the structuring tool should decrease workload. Hence the Chapter 7. Conclusions and Future Work 62 usability issues described earlier need to be resolved; efficient and intuitive interaction techniques for grouping annotations need to be incorporated. 7.2.2 Evaluating Structured Annotations in Iterative Collaboration Zheng et al. assessed the impact of structured annotations on users who receive structured annotations [43]. We have now assessed the impact on users who create them. The next step will be to investigate the impact of structured annotations on the complete co-authoring workflow that involves iterative cycles of annotating and editing. More specific issues to explore include how users might go about managing and maintaining structured annotations as the document evolves with iterations of the reviewing cycle. 7.2.3 Incorporating Structured Annotations into E-mail The purpose of structured annotations is to unify all document-related com-munication into the document. The purpose is not to completely displace external communication methods such as email. Given its pervasiveness, communication through email is inevitable. Perhaps duplicating or summa-rizing information from structured annotations in the body of email mes-sages might appropriately serve as a detailed notification; this could allow co-authors to get a quick overview of the reviewed document and the embed-ded annotations without having to open the document. A major drawback to any use of e-mail is the seemingly inevitable urge users have to send a reply, which runs the risk of having document-related communication out-side the document. More research is definitely needed to explore ways in which email content can be generated automatically from the information communicated in structured annotations in a document and whether it is possible to automatically incorporate e-mail replies explicitly back into the document. 63 B i b l i o g r a p h y [1] Del.icio.us [online social bookmarking tool], 2007. Accessed May 2, 2007, from http://del. icio.us. [2] Flickr [online photo management and sharing tool], 2007. Accessed May 2, 2007, from http://flickr.com. [3] Google docs [online word processor tool], 2007. Accessed May 2, 2007, from http://docs. google. com. [4] Write board [online collaborative writing tool], 2007. Accessed May 2, 2007, from http: //www. writeboard. com. [5] D. Abrams, R. Baecker, and M. Chignell. Information archiving with bookmarks: Personal web space construction and organization. In Pro-ceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 41-48. A C M Press, 1998. [6] R. M . Backer, D. Nastos, I. R. Posner, and K. L. Mawby. The user-centered iterative design of collaborative writing software. In Proceed-ings of the SIGCHI Conference on Human Factors in Computing Sys-tems, pages 399-405. A C M Press, 1993. [7] S. Bloehdorn and M. Volkel. Tagfs — tag semantics for hierarchical file systems. In Proceedings of International Conference on Knowl-edge Management, 2006. Retrieved May 2, 2006, from http://semf s. ontoware.org/pubs/2006-09-iknow2006-tagfs.pdf. [8] E. Churchill, J. Trevor, S. Bly, L. Nelson, and D. Cubranic. Anchored conversations: Chatting in the context of a document. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 454-461. A C M Press, 2000. [9] J. Cohen. Eta-squared and partial eta-squared in communication sci-ence. Human Communication Research, 28:473-490, 1973. Bibliography 64 [10] L . E d e a n d A . L u n s f o r d . Singular Texts/Plural Authors: Perspectives on Collaborative Writing. S o u t h e r n I l l ino i s U n i v e r s i t y P r e s s , C a r b o n -da le , I L , 1990. [11] C . A . E l l i s , S. J . G i b b s , a n d G . L . R e i n . G r o u p w a r e : S o m e issues a n d exper iences . Communications of the ACM, 3 4 ( l ) : 3 9 - 5 8 , 1991. [12] R . S. F i s h , R . E . K r a u t , a n d M . D . L e l a n d . Q u i l t : a c o l l a b o r a t i v e t o o l for c o o p e r a t i v e w r i t i n g . I n Proceedings of the ACM SIGOIS and IEEECS TC-OA Conference on Office Information Systems, pages 3 0 -37. A C M P r e s s , 1988. [13] S. G o l d e r a n d B . H u b e r m a n . T h e s t r u c t u r e o f c o l l a b o r a t i v e t a g g i n g s y s t e m s . Journal of Information Science, pages 198-208 , 2006. [14] J . G r u d i n . W h y c s c w a p p l i c a t i o n s fai l : P r o b l e m s i n t h e d e s i g n a n d e v a l u a t i o n o f o r g a n i z a t i o n a l interfaces . I n Proceedings of the ACM Con-ference on Computer Supported Cooperative Work, pages 8 5 - 9 3 . A C M P r e s s , 1988. [15] S. G . H a r t a n d L . E . S t a v e l a n d . D e v e l o p m e n t o f n a s a - t l x ( task l o a d i n d e x ) : R e s u l t s o f e m p i r i c a l a n d t h e o r e t i c a l re search . Human Mental Workload, pages 139-183 , 1988. [16] W . J o n e s , A . J . P h u w a n a r t n u r a k , R . G i l l , a n d H . B r u c e . D o n ' t t a k e m y fo lders away! o r g a n i z i n g p e r s o n a l i n f o r m a t i o n t o get t h i n g s d o n e . In Proceedings of the SIGCHI Conference on Human Factors in Com-puting Systems, pages 1505-1508 . A C M P r e s s , 2005. [17] H . K i m a n d K . S. E k l u n d h . R e v i e w i n g p r a c t i c e s i n c o l l a b o r a t i v e w r i t -i n g . I n Proceedings of the ACM Conference on Computer Supported Cooperative Work, pages 247 -259 . A C M P r e s s , 2001. [18] R . A . K i n c h l a a n d J . M . W o l f e . T h e o r d e r o f v i s u a l p r o c e s s i n g : "top-d o w n " , " b o t t o m - u p " o r " m i d d l e - o u t " . Perception and Psychophysics, 25(3) :225-231 , 1979. [19] T . L a n d a u e r . Chapter 9: Behavorial research methods in human-computer interaction, pages 203 -227 . H a n d b o o k o f H u m a n - C o m p u t e r I n t e r a c t i o n . E l s e v i e r Sc ience , A m s t e r d a m , s e c o n d e d i t i o n , 1997. [20] D . L a w t o n a n d I. E . S m i t h . T h e k n o w l e d g e wease l h y p e r m e d i a a n -n o t a t i o n s y s t e m . In Proceedings of Hypertext and HyperMedia, pages 106-117 . A C M P r e s s , 1993. Bibliography 65 [21] P . B . L o w r y , C . C . A l b r e c h t , J . D . L e e , a n d J . F . N u n a m a k e r . U s e r s ex-per iences i n c o l l a b o r a t i v e w r i t i n g u s i n g c o l l a b o r a t u s , a n i n t e r n e t - b a s e d c o l l a b o r a t i v e w o r k . I n Proceedings of International Conference on Sys-tem Sciences, pages 2 1 - 3 0 . I E E E C o m p u t e r Soc ie ty , 2002. [22] A . M a t h e s . F o l k s o n o m i e s - c o o p e r a t i v e c la s s i f i ca t ion a n d c o m m u n i c a t i o n t h r o u g h s h a r e d m e t a d a t a . R e t r i e v e d J u n e 20, 2006, f r o m http://www.adammathes.com/academic/ computer-mediated-communication/folksonomies.html, 2004. [23] J . M c G r a t h . M e t h o d o l o g y m a t t e r s : D o i n g re search i n t h e b e h a v i o u r a l a n d s o c i a l sc iences . Readings in Human-Computer Interaction: To-wards the Year 2000, 1994. [24] C . N e u w i r t h , R . C h a n d h o k , D . C h a r n e y , P . W o j a h n , a n d L . K i m . D i s -t r i b u t e d c o l l a b o r a t i v e w r i t i n g : A c o m p a r i s o n o f s p o k e n a n d w r i t t e n m o d a l i t i e s for r e v i e w i n g a n d r e v i s i n g d o c u m e n t s . I n Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 5 1 - 5 7 . A C M P r e s s , 1994. [25] C . N e u w i r t h , D . S. K a u f e r , R . C h a n d h o k , a n d J . H . M o r r i s . Issues i n the d e s i g n o f c o m p u t e r s u p p o r t for c o - a u t h o r i n g a n d c o m m e n t i n g . I n Pro-ceedings of the ACM Conference on Computer Supported Cooperative Work, pages 183-195 . A C M P r e s s , 1990. [26] S. N o e l a n d J . M . R o b e r t . E m p i r i c a l s t u d y o n c o l l a b o r a t i v e w r i t i n g : W h a t d o c o - a u t h o r s d o , use a n d l ike? I n Proceedings of the ACM Con-ference on Computer Supported Cooperative Work, v o l u m e 13, pages 6 3 - 8 9 . A C M P r e s s , 2004. [27] I. O v s i a n n i k o v , M . A r b i b , a n d T . M c n e i l l . A n n o t a t i o n t echno logy . In-ternational Journal of Human-Computer Studies, 50:329-362 , 1999. [28] J . P o r t e r . F o l k s o n o m i e s : A u s e r - d r i v e n a p p r o a c h t o o r g a n i z i n g c o n -tent . R e t r i e v e d M a r c h 20, 2007, f r o m h t t p : //www. u i e . c o m / a r t i c l e s / folksonomies/, 2006. [29] I. R . P o s n e r a n d R . M . B a e c k e r . How People Write Together. M o r g a n K a u f m a n n , S a n M a t e o , C A , 1993. [30] D . Q u a n , K . B a k s h i , D . H u y n h , a n d D . K a r g e . U s e r interfaces for s u p -p o r t i n g m u l t i p l e c a t e g o r i z a t i o n . I n Proceedings of International Con-ference on Human-Computer Interaction, 2003. Bibliography 66 [31] R. Rimmershaw. Collaborative writing practices and writing support technologies. Instructional Science, 21:15-28, 1992. [32] J. R. Santos. Cronbach's alpha: A tool for assessing the reliability of scales. Journal of Extension, 37(2), 1999. [33] W. A. Scott. Cognitive complexity and cognitive flexibility. Journal Information for Sociometry, 25(4):405-414, 1962. [34] R. Sinha. A cognitive analysis of tagging. Thoughts on Technol-ogy, Design, and Cognition. Retrieved June 7, 2007, fromhttp: //www. rashmisinha.com/archives/05_09/tagging-cognitive.html, 2005. [35] M . Storey, L. Cheng, I. Bull, and P. Rigby. Shared waypoints and social tagging to support collaboration in software development. In Proceed-ings of the A CM Conference on Computer Supported Cooperative Work, pages 195-198. A C M Press, 2006. [36] J. Teevan, C. Alvarado, M . S. Ackerman, and D. R. Karger. The perfect search engine is not enough: A study of orienteering behavior in directed search. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. A C M Press, 2004. [37] C. Weng and J. H. Gennari. Asynchronous collaborative writing through annotations. In Proceedings of the ACM Conference on Com-puter Supported Cooperative Work, pages 564-573. A C M Press, 2004. [38] C. Weng, D. W. McDonald, and J. H. Gennari. A collaborative clinical trial protocol writing system. Journal of Medical Informatics, pages 1461-1465, 2004. [39] C. Weng, D. W. McDonald, and D. Sparks. Participatory design of a collaborative clinical trial protocol writing system. International Jour-nal of Medical Informatics, pages 245-251, 2006. [40] S. Whittaker and C. Sidner. Email overload: Exploring personal infor-mation management of email. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 276-283. A C M Press, 1996. [41] P. Wojahn, C. Neuwirth, and B. Bullock. Effects of interfaces for anno-tation on communication in a collaborative task. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 456-463. A C M Press, 1998. Bibliography 67 [42] Q. Zheng. Structured annotations to support collaborative writing workflow. Master's thesis, University of British Columbia, 2005. [43] Q. Zheng, K. S. Booth, and J. McGrenere. Co-authoring with struc-tured annotations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 131-140. A C M Press, 2006. Appendix A. Observational Study Resources 70 A.2 Study Document Challenges of C S C W The term CSCW was originally coined by Irene Greif at a workshop in 1984 attended by a small group interested in using technology to support people at their work, which led to the first CSCW conference in 1986 [7]. CSCW focuses on the study of tools and applications to support groups and their social, psychological, and organizational impacts. The term CSCW stands for Computer Supported Cooperative Work. Tools and applications in the CSCW context mainly support the coordination and communication for small groups or organizations such as scheduling meetings, distributed decision making, locating colleagues, collaborative work, etc. Since then, CSCW has evolved as a field with many influences from various disciplines such as: Computer Science, Sociology, Psychology, Ethnography, Anthropology, Organizational' Studies and Communications. The field has accumulated a set of empirically based interdisciplinary studies, and many new interesting applications to support groups and organizations. The design and development of the groupware tools involve challenges regarding not only technical issues, but also many other factors. Applications designed only with a technology-centered perspective and without much consideration for users can result in neither usable nor useful applications, no matter how innovative or constructive the technology is. The human computer interaction researchers address and solve such a problem by proposing user-centered interface design guidelines or heuristics for usable and efficient applications [13]. However, such individual-centered approaches fail to address issues involving multi-user applications. The CSCW field has proposed organization-centered approaches, by broadened design perspectives to address social and organizational contexts of technology use in groups and organizations. Examination and consideration of group's nature, work practice, interaction, and incentives become essential in designing and developing groupware applications. Designing and deploying a successful groupware application involves many challenges. In addition to be usable and efficient for the common goal of the group, the applications are expected to benefit the individuals of the group. The developers have to foresee and consider the potential reciprocal and co-evolutionary relationship between the technology and social context of the group: the use context is effected by the constraints Appendix A. Observational Study Resources 71 and affordances of the technology offered by the application and the technology also co-adapts to the environment in important ways [14]. In order to develop a usable and efficient groupware application, developers need to face and cope with some of the major fundamental challenges. A critical mass of use of the new technology or application is often required to gain the total benefit of technology [6]. As discussed earlier, most groupware require the critical mass of use of the application to succeed. However, most groups constitute heterogeneous mixtures of members with different backgrounds and preferences. Therefore, groupware should be designed so that it is appealing to users with various background and knowledge. One possible solution is the use of adaptable user interfaces so that every group member can customize the application to their needs and preferences [11]. This is a problem and we need to deal with it wisely. For single-user application, a word processor liked by one in five prospective customers could be a huge success [5]. However, for multi-user application, an application liked by only one member out of five in the group will be a failure since the decisions have to be made at the group level. Thus the groupware should be appealing to all the group members. Reduce Friction More often than not, organizations and groups are structured and responsibilities are divided so that the overhead for communications will be minimized. However, as the size of the group grows bigger the more overhead requirements for communication and coordination it demands. Some groupware applications are designed mainly to provide such communication and coordination needs among the group members. One of the examples is a groupware with features that support collaborative co-authoring, version tracking, and distribution of drafts for collaborative tasks. However, if a collaborative groupware and users' main application are not integrated together, the users will need to leave their core applications and launch such collaborative tools or visit collaborative platform to get the latest version of the shared artifact. Consequently, the "friction" or unnecessary delays for users' work [8]. Groupware should be designed to minimize or eliminate the extra work, and to give users maximum seamless working experience. Hence, groupware features will fare better if they are integrated with applications that users use for their primary tasks [5]. Appendix A. Observational Study Resources 72 Privacy, Social and Political Issues The technology and functionalities offered by some groupware can lead to an activity that causes conflicts, effects personal privacy of individuals, interferes with social dynamics, or threatens existing political structure of the group. In everyday situations, we play it by ear according to social protocols, temporal situations, tactical agendas and awareness of the personalities. However, groupware applications often require making such protocols and information explicit, leaving no room for tactical procedure or improvisation. For example: a groupware calendar system and automatic scheduling system that is open for the group members to view and that allows priority-based scheduling will result in failure because users would be reluctant to publicly acknowledge that some of their meetings with other group members are low priority [5]. Therefore, during the design process, groupware developers should consider those social and political aspects of groups and consequences that the applications can cause and how to prevent them. Failure of Intuitive Decision-Making Often, the responsible manager who comes to a decision about resources for an application development project relies heavily on intuition [4]. We can use intuition as a guide for decisions about single user applications because it might not be very difficult for the decision-maker to imagine ourselves with the target users. However, for groupware, it needs more than intuition; in fact, education and vigilance about the nature of groups and different interests and incentives of each member [4]. Additionally, the manager might even come across issues involving conflicts of interest or bias towards a certain subset of the user population because groupware applications are intended for users with different job titles, heterogeneous background and various interests. Evaluation Evaluation for CSCW applications requires a very different approach and methodology than single user applications. Obstacles to achieve generalizability, precision and realism exist due to various potential impacts such as social, physical, political, temporal, motivational and economic involved in a group, central to the group's performance and success [4]. Most evaluations and studies involve a very small group with constant membership, performing simple tasks arbitrarily assigned to them by the experimenter under "context-stripped" conditions [9]. Just as Gould's "How to Design Usable Appendix A. Observational Study Resources 73 Systems" [3] and other usability design guidelines [1], systematic and comprehensive design models and guidelines to include group-centered perceptive will be helpful for developers. Moreover, exploration and identification of appropriate research methodologies for better evaluations of groupware will result in more successful and usable groupware applications! CASE STUDY: Microsoft SCHEDULE* The collaborative meeting scheduler application SCHEDULE+ is widely used by employees at the Microsoft Corporation, as Grudin and Palen reported from their three month long study as a participant observer in a Microsoft development division [6]. It is successful now, but it was a failure when first introduced in the early 1980s [4]. The recent success of the application mainly contributes to the social context of the group adapted by the application, and better design and group-centered perspectives from the developers. Overall, SCHEDULE+ has evolved as a successful groupware application over ten years since it was first introduced. The users at Microsoft seem to have adapted well to the application and its features offered. Moreover, its success could also be due to the fact that developers have learned and become aware of the problems and issues around such applications and overcome most of them. 7s CSCW Ready for Theory? Since CSCW has emerged, some theories have imposed influence on CSCW. An example of such theories is Structuration Theory, which states that the social context is effected by the new technology introduced and that the technology implementation is in turn influenced by users [9]. Moreover, other theories have also been proposed to solve the problems and issues surrounding groupware [9]. The recently debating question is whether CSCW as a field is ready for theory: whether the technology and problems are mature enough to accommodate theory or whether theory is needed to advance the field [9]. Where is CSCW going? CSCW has first emerged as a field including people with different background and yet common interest to utilize technology to support group work. As Grudin noted for CSCW, Appendix A. Observational Study Resources 74 "People come from different places, and it is useful to know where each is from and why they have come. Each visitor can see what the others have to offer and can decide what is worth taking home. [...]• There is no assumption that everyone speaks the same language, only that they will try to work out some means of communicating. [...] When understood and respected, the differences form the core of richer, shared understandings." Consequently, C S C W has grown as a unique multidisciplinary field. Researchers have contributed multiple perspectives and insights for the growth of C S C W field and bigger success of CSCW applications. Appendix A. Observational Study Resources 75 A.3 Interview Questions 1. How long have you been tutoring? 2. How many documents have you reviewed as a tutor? 3. How many hours did you dedicate to tutoring (reviewing documents) on a weekly basis? 4. What types of documents were usually asked to review? (e.g., essays, term papers, thesis, publishable papers) 5. On average, how long were documents? (single spaced, with standard font size 9-12) 6. How do you usually review the documents? a. Print out (Rarely Frequently) i. Why? ii. Does this make the reviewing process easier or more efficient than other options? iii. Positive aspects? iv. Negative aspects? b. Directly edit using word processor (Rarely Frequently) i. Why? ii. Does this make the reviewing process easier or more efficient than other options? iii. Positive aspects? iv. Negative aspects? c. Use annotation function (Rarely Frequently) i. Why? ii. Does this make the reviewing process easier or more efficient than other options? iii. Positive aspects? iv. Negative aspects? d. Write email message (Rarely Frequently) i. Why? ii. Does this make the reviewing process easier or more efficient than other options? iii. Positive aspects? iv. Negative aspects? e. Online communication groupware (Rarely Frequently) i. Why? ii. Does this make the reviewing process easier or more efficient than other options? iii. Positive aspects? iv. Negative aspects? f. Other ? i. Why? ii. Does this make the reviewing process easier or more efficient than other options? iii. Positive aspects? iv. Negative aspects? 7. On average, how long did you take to review a document (per page)? Appendix A. Observational Study Resources 76 8. On average, how many passes did you take at a document in the process of reviewing it? 9. Is there any particular strategy that you used while reviewing a document? (e.g., grammar first, style issues second, etc.) 10 What aspects of documents did you mainly comment on? Did students have to request you to make those comments? Upon request only or "voluntary"? a. Grammar b. Style c. Structure d. Clarity issues e. Resolve conflicts f. Other 11 On average, after reviewing, how many comments and edits appeared per page? 12 After reviewing the document, what else did you communicate to the writer(s)? (e.g, high level comments, prioritized problems, etc.) 13 How did you communicate to the writer(s)? a. Email i. Why? ii. Does this make the reviewing process easier or more efficient than other options? iii. Positive aspects? iv. Negative aspects? b. Face to face i. Why? ii. Does this make the reviewing process easier or more efficient than other options? iii. Positive aspects? iv. Negative aspects? c. Document drop off i. Why? ii. Does this make the reviewing process easier or more efficient than other options? iii. Positive aspects? iv. Negative aspects? d. Phone i. Why? ii. Does this make the reviewing process easier or more efficient than other options? iii. Positive aspects? iv. Negative aspects? e. Others i. Why? ii. Does this make the reviewing process easier or more efficient than other options? iii. Positive aspects? iv. Negative aspects? Appendix A. Observational Study Resources 77 14. How long did it take to communicate with the writers? (longer than reviewing time, etc.) 15. When were the documents requested to be reviewed? a. Early draft b. Towards the middle of draft c. Almost ready draft 16. Were there due dates for the documents that you were requested to review? 17. How many iterations did you review a document? Any follow up? 18. Are there any (technological as well as intellectual) restrictions that were imposed by the program on how you review the documents? Appendix B. Paper Prototype Study Resources 80 B.2 Demographic Questionnaire UBC Using Structured Annotations in Collaborative Writing Study Questionnaire Form Instructions Please try to respond to all of the Items listed below. For those items that are not applicable, specify N/A. Past Computer and Writing Experience (To be completed before the study) 1. Which word processor do you currently use for writing documents (e.g. essays, reports, letters, conference papers, journal articles, etc.)? 2. How often do you use the word processor? • Once a month • Once a week • Every 2-3 days • Every day 3. How confident do you feel about using the word processor? (Not at all confident) 1 2 3 4 5 6 7 8 9 1 0 (Very confident) 4. Do you use the annotation functions in the word processor? (e.g. Track Changes and Commenting functions in Microsoft Word) • Yes. • No, please specify why: 5. Have you previously written or reviewed documents with other people? • None • less than 5 times • between 5-10 times • more than 10 times Word processor used in collaborative writing: 6. Please describe briefly the nature of the documents that you have co-authored (e.g., project reports for a course or job, conference papers, journal papers). (Continue on the next page) Appendix B. Paper Prototype Study Resources 81 7. How do you and your co-authors review a collaborative document? (Check all the items that apply.) Reviewing Method Frequency of use (Mark an 'X' along the scale) • Print out the document, mark on the document using a pen, and then hand back the marked document to co-author(s). 1 1 1 1 1 1 1 1 ! (Rarely) 1 i (Frequcnlly) • Directly edit on the document using a word processor. I 1 1 1 1  1 1 1 (Rarely) i i (Frequently) • Use the annotation function in the word processor to edit the document and add comments. 1 1 1 I 1  1 1 1 (Rarely) 1 1 (Frequently) • Write an email message that includes suggested changes and comments about the document to other co-authors. 1 1 1 i 1 1 1 1 1 (Rarely) I | (Frequently) • Use online communication groupware (e.g., Yahool Groups) to discuss about the changes to the document. 1 I 1 1 1 1 1 1 1 (Rarely) 1 1 (Frequenlly) • Other. 1 i 1 1 1 1 1 1 1 I 1 Please specify: 1 1 1 1 1 1 1 1 I (Rarely) 1 1 (Frequenlly) Do not turn over the page. Please notify the experimenter when you finish answering the above questions. Appendix B. Paper Prototype Study Resources 82 B.3 Training Task Scenario: The folio wings are the items you need to buy. 1. Beef 2. Broccoli 3. Grapes 4. Milk 5. Chicken 6. Cheese 7. Banana 8. Yogurt 9. Eggs Your roommate Nancy and Amber are going for grocery shopping and have kindly offered to help you pick up the items you need. So, your task is to organize the items so that they can efficiently do the shopping. B.4 Instructions to Participants Scenario: Imagine you are Sam, and are supervising Jane and Nick on a project about "music effects on memorization". You are not directly involved in the project, but rather you provide directions to Jane and Nick and supervise the project at a higher level. All three of you are currently collaborating on a document titled "The Effects of Music Type and Volume on Memorization". Thus, Jane and Nick are your co-authors, who are equally responsible for editing the document. Their roles in the project and in writing this document are as followings: Jane's role: Domain expert in effects of music on cognitive processes Nick's role: Expert in experimental design and analysis You have received their first draft of the document. Now, you have 5 minutes to skim through the document to familiarize yourself with what they have written. Do not annotate or comment on the document at this point. Annotations and comments will be provided to you after you have read the document. Please notify the experimenter when you finish reading the above scenario. Appendix B. Paper Prototype Study Resources 83 Scenario continued: After receiving the first draft of the document, you briefly reviewed and annotated it. (The annotations are provided.) Now you are about to send the annotated document back to Jane and Nick. The timing is tight for them to review all of your annotations and revise the document because the submission deadline is midnight today. So, in order to make the best use of their time and expertise, your task is to organize the annotations so that your co-authors can review the document efficiently and accurately without missing critical annotations. Instructions: In order for the experimenter to understand your thought processes, we ask that you please read out loud when you are reading the document and the annotations, and think out loud while you are organizing the annotations. You can read the document and annotations as many times as you wish or need. For the first read through of the annotations, please read them in the order that they appear in the document. However, you can start organizing annotations at any point during the task (i.e., you do not have to read all the annotations in the document before you start organizing the annotations). Document Display Format The deleted text in the document is displayed with strikethrough (Deleted). The inserted text in the document is displayed with a wavy underline (inserted). The replaced text has the deleted text followed by the replacement text (RoplooodKt'tJla^ s). The document might contain edit annotations (Deletes, Inserts, Replaces) and comment annotations. Each annotation is displayed in la balloon {in the right margin of thej)aper. The [anchor of each annotation is highlighted (and is connected to its annotation balloon Comment [Al]: Comment: This is • comment biMoon. ' with a dashed line. Comment [A2]: Comment: inch or of each annotation is highlighted" is the anchor of this comment B . 5 Study Document with Annotations T h e d o c u m e n t was se lec ted f r o m a n o n l i n e a r c h i v e o f m a n u s c r i p t s f r o m the N a t i o n a l U n d e r g r a d u a t e R e s e a r c h C l e a r i n g h o u s e o f M i s s o u r i W e s t e r n S t a t e U n i v e r s i t y : h t t p : / / c l e a r i n g h o u s e . m i s s o u r i w e s t e r n . e d u / Appendix B. Paper Prototype Study Resources 84 Title: The Effects of Music Type and Volume on Memorization One hundred years ago there were not as many distractions for students as compared to the present day students. In today's society, recreational and occupational background noise is typically causing hearing loss at early age levels and it is not exactiy certain how (this) is / [effecting affecrin^cognitive processes. [There has been Ktde_significant^ ^ J on how background music may or may not [effect affectstudy habits / ' and quality. Many students feel like background music helps them with their studying. Rauscher, Shaw, and Ky (1993) found that performance [improves improved] on abstract/spatialrea^sqning tests after participants listened to Mozart as opposed to a relaxation tape or silence. [No difference was noted with the latter two interventions]. J According to a study by Barber, McKenzie, and Helme (1997), the human brain [responds differendy to classical music than it does to rock style musicj. They measured brain actiyky from the_s_calp_with an _ / Comment [Al]: Comment: Not clew. Hearing loss or the noise? I Comment [A2]: Replaced: "effecting" with "affecting" . Comment [A3]: Comment: Motivation for the research sounds very Comment [A4]: Replaced: "effect" with "affect" j Comment [AS]: Replaced: I "improves" with "improved" Comment [A6]: Comment: Not clear. No performance difference between the litter 2 or no improvement difference compared to Mozart condition? Comment [A7J: Comment: Did they also measure how the different music affecK the cognitive processes as well? Appendix B. Paper Prototype Study Resources 85 electrode cap worn by the participant called an (qeeg), |qualitativej_ / ' electroencephalography. [The recentlyjpopular trend of individuals listening to Mozart music in / order to become smarter, all began with |Alfred Tomatisj. It is known as "the |S mart Music Effect)," but only one study showed significant y ' results and other studies have been unsuccessful in replicating the first finding (Halpern, 1997). This hypothesis has exploded onto the market place which contains an abundance of products that are purported to improve mental capacity, intelligence and relaxation. In an experiment conducted by Ison, and Agrawal (1998), mice were involved to gain a better understanding of the [affects etfects[of_the mice's age as well as [gap threshold] qntheauditorysYstemsJrespohse / ' level to noisej. _ _ _ __ Hall and Grose(1994) have generalized (these finds |to_hum_an / ' children, reinforcing that the sensitivity of the (asymptote deterrnines starde reflex, not time constants or the gaps of hoise[ J Comment [A8]: Comment: I thought it is quantitative. Check on the lerni to Comment [A9]: Comment: 1 ihink this paragraph should immediately follow the second paragraph where yon first mentioned about Mozart music. Comment [A10]: Comment: Isn't he a Dr? If so, we should acknowledge his Comment [All]: Comment: 1 -thought this was called "Mozart Effect" Check on the term to make sure. j Comment [A 12]: Replaced: "affects" I with "effects" Comment [A13]: Comment: Not clear which gap threshold you are talking | Comment [A14]: Comment: What is 1 thdr finding? IComment [A15]: Comment: Not clear which findings you are referring to? J Comment [A 16]: Comment: Not clear. Need to elaborate more on this. Appendix B. Paper Prototype Study Resources 86 Contradictory to the Rauscher etal. study (1993), the McFarland and Kennison study assumptions are that the right hemisphere of the brain processes the music. Participants require greater effort to successfully learn a task with the presence of music. Their assumptions were supported by the data they collected. Thus, music makes learning more difficult. [Therefore, the purpose of thisexperimentwiU betotest theeffects of music types (classical and rock) and volume (soft and loud) on the memorization of students. Hence, the experiment is a |2x2J (4 conditions) design] The participants in this study were 20 Physical Therapy Assistant students both |male and female (varying ages)|, and was conducted on_ the campus of Missouri Western State University. The college is [located in northwest Missouri in the city of St Joseph[ Participants [will be were [selected by voluntajy_rnethqd.lt wa^ s_ assumed mat all participants had [normal hearing [A17]: Comment: The irulysis of different findings by different researchers should be included here, instead ofjust quoting them. Comment [A 18]: Comment: Isn't this a randomized design? If so, should Comment [A19]: Comment! What, are our hypo dieses here? Should state them explicitly right up front. Comment [A20]: Comment: Might be useful to mention the number of males and females as well as their age ranges. Comment [A21]: Comment: How is this relevant? Comment [A22]: Replaced: "will be" with "were" Comment [A23]: Comment: Is this » safe assumption? Given the hearing loss -problem in the first paragraph and since one of our treatment conditions was the music level, a slight difference in hearing sensitivity could confound our results! Appendix B. Paper Prototype Study Resources 87 Items included in this experiment were a Realistic soundlevel meter 33-2080, Aiwa brand compact disk stereo player, 2 compact disks, and a memory test. The memory test consisted of |two| different word / lists of 25 capital cities in the United States. The [classical music [and the [heavy metal music (were played ^ Testing occurred in 4, twenty-minute sessions. Participants entered the test room with one of the possible four interventions [in progress|. / Heavy metal style music with volume levels soft or loud or classical style music with volume levels set at soft or loud. Soft music lb-was} played at 55 [dB] and loud music was played at 70 dB both musical sources were 10 feet away from the participants. A memory test was present, face down at the participant's seat. The researcher instructed the participants that on the cue they were to turn over the word list in front of them and given two minutes to review it. On the second cue they would return list face down and were then given three minutes [to recall jas niany_words_a^s_possible. ___ J Comment [A24]: Cammcni: Only two differeni fists? We hive 4 conditions, Unless the presentation of the Est is counterbalanced, this conld be a major confounding factor in our results!! Comment [A25]: Comment; State which song we used for classical In the experiment. Comment [A26]: ("ornmm t: State which song we used for heavy metal music in the experiment. Comment [A27]: Comment: Why ir progress? When are the ir to the subjects? Comment [A28]: Replaced: "is" with Comment [A29]: Comment: Try avoiding short forms - state the actual Comment [A30]: Comment: Was the music still playing during the recaD as well? Appendix B. Paper Prototype Study Resources 88 The |numbers of correct responses | made by the subjects were recorded with the) two [types of interventions that were present at test , time. A) 2X2 factorial JAN O V A were calculated comparing music type and / volume to memory test scores. The A N O V A test was calculated in order to compare music volume levels and music type on [correct test responses]. The main effect for volume level on correct responses was / not significant (JF(1,36)= 1.26,p= .27J)._Th_e_main_e^ ^^  , on the number of correct responses was not significant (JF(l,36)=.686,p = .413[)Jhe_volume by music type interaction was_ not significant (JF(1,36) =.116 p=.735J).Thus it appears that neither music type or volume has any effect on the number of correct responses on a memory test. Initially, it was [expected that music type and volume would have a strong relationship to students' performance on the memory.test[. Statistical analysis showed a weak relationship between music type and test performance. There was a slight decrease in the test scores of students exposed to high volume music levels. They had an increase Comment [A31]: Comment: Isn't it common in the literature to record both correct and incorrect responses for the memory recall tests? Comment [A32]: Comment: Two 01 Four? Comment [A33]: Comment: This is within subject design, right? Need to mention within subject or between subject explicitly here. Comment [A34]: Comment: I think it'll be useful to give the data about the average number of correct responses and errors made by the subjects. Comment [A3S]: Comment: Should report standard error. ' ; ... Comment [A36]: Comment: Should report standard error. Comment [A37]: Comment: Should report standard error. Comment [A38]: Comment: What theory is this hypothesis based on? We should explain why we expected that Appendix B. Paper Prototype Study Resources 89 in the number of errors compared to participants exposed to low volume levels. These [findings concur with the McFarland and Kennison (1987)^  which showed that no difference was found in the performance of participants on abstract/spatial reasoning tests after listening to Mozart, a relaxation tape or silence. [The McFarland and Kennison study, found that the right hemisphere of the brain processes music thus requiring greater effort to learn a task while music was in the background. The use of a soundproof room and longer exposure time of the musical interventions could possibility result in greater reliability and variable accuracy. Future research ideas could include cultural or personality differences of [concentration] and volume levels|._ Comment [A39]: Comment: Can we cone hide thai onr results concur with theirs? They arc measuring the reasoning test results while we are measuring the memory recall test results. • Comment [A40]: Comment: I'm not sure reiterating their theory is necessary once we already mentioned it earlier. Either leave this out or lake out the earlier paragraph about their study. Comment [A41]: Comment: How arc we going to measure the concentration? Comment [A42]: Comment: I fed like we have a weak conclusion here. Can | we elevate our pitch in the conclusion? Appendix C. Experiment Study Resources 92 C.2 Questionnaires C.2.1 Demographic Questionnaire lUBCl w Collaborative Writing with Annotations Study Questionnaire Form Instructions Please try to respond to all of the items listed below. For those items that are not applicable, specify N/A. Part 1: Past Computer, Reviewing and Co-authoring Experience (To bo completed betm the study) Computer Experience 1. Which word processor do you currently use for writing/reviewing documents (e.g. essays, reports, letters, conference papers, journal articles, etc.)? 2. How often do you use the word processor? • Once a month • Once a week • Every 2-3 days • Every day 3. How confident do you feel about using the word processor? (Not at all confident) 1 2 3 4 5 6 7 (Very confident) 4. Do you use the annotation functions in the word processor? (e.g., Track Changes and Commenting functions in Microsoft Word) • Yes. O No. please specify whv: Reviewing Experience 5. How many times have you previously reviewed documents written by other people? • None • less than 5 times • between 5-10 times • more than 10 times Word processor used in reviewina: 6. Please describe briefly the nature of the documents that you have reviewed (e.g., project reports for a course or job, conference papers, journal papers). Appendix C. Experiment Study Resources 93 7. How do you review documents? (Check all the items that apply.) Reviewing Method Frequency of use (Mark an 'X' along the scale) • Print out the document, mark on the document using a pen, and then hand back the marked document to author(s). 1 1 1 1 1  1 1 1 (Rarely) 1 1 (Frequenlly) • Directly edit on the document using a word processor. i i i i i 1 i i i (Rarely) 1 1 (Frequently) • Use the annotation functions in the word processor to edit the document and add comments. 1 1 1 1 1  1 ! 1 (Rarely) 1 | (Frequently) • Write an email message that includes . suggested changes and comments about specific parts of the document to author(s). 1 1 1 1 1 1 1 I 1 (Rarely) I | (Frequenlly) • Other. i i i i t 1 i i i i i Please specify: 1 1 1 1 1 1 1 1 1 (Rarely) 1 1 (Frequently) 8. How do you communicate/discuss the annotations and reviewing results with the author(s)? (check all the items that apply) Communication Method Frequency of use (Mark an 'X' along the scale) • Face to face meetings. 1 1 1 1 1 1 1 1 1 1 (Rarely) (Frequently) • Email. 1 1 1 1 1 1 :t . 1 1 1 (Rarely) (Frequently) • Instant Messaging. | | | | | 1 1 1 1 1 (Rarely) (Frequenlly) • Phone. 1 1 1 1 i 1 1 1 1 1 (Rarely) (Frequently) • Use online communication groupware (e.g., Yahoo! Groups) to discuss about the changes to the document. 1 1 1 1 1 1 1 1 1 1 1 (Rarely) (Frequently) • Other. 1 1 1 1 1  I 1 1 1 1 Please specifv: (Rarely) (Frequently) Appendix C. Experiment Study Resources 94 Co-authoring Experience 9. Have you previously co-authored (written and reviewed) documents with other people?? • None • less than 5 times • between 5-10 times • more than 10 times Word processor used in co-authoring: 10. Please describe briefly the nature of the documents that you have co-authored (e.g., project reports for a course or job, conference papers, journal papers). 11. How do you discuss about collaborative documents? (Check all the items that apply.) Reviewing Method Frequency of use (Mark an 'X' along the scale) • Print out the document, mark on the document using a pen, and then hand back the marked document to co-author(s). i i i i i I i i i i i (Rarely) (Frequently) • Directly edit on the document using a word I I I I I 1 :l 1 1 1 processor. (Rarely) (Frequently) • Use the annotation functions in the word processor to edit the document and add comments. I I I J J I I I I I I (Rarely) (Frequently) • Write an email message that includes suggested changes and comments about I i. i 1 i 1 i l l i i specific parts of the document to co-author(s). (Rarely) (Frequently) • Use online communication groupware (e.g., Yahool Groups) to discuss about the changes to the document. 1 1 1 1 1 1 1 i 1 ! 1 (Rarely) (Frequenlly) • Other. 1 1 1 1 • T 1 1 1 1 1 1 Please specifv: (Rarely) (Frequently) Do not turn over the page. Please notify the experimenter when you finish answering the above questions. Appendix C. Experiment Study Resources 95 C.2.2 System-specific Questionnaire: Bundle System Expe riGIICe (To be completed aftet completing the task using the bundle system) With respect to your experience performing the given task with the bundle system, please answer the following questions by ma rk i ng an ' X ' along the scale below the corresponding question. 1. How much mental and perceptual activity was required to perform the task (e.g., thinking, deciding, remembering, searching, etc.)? Mental Demand I "j I I I I "| I I j I I I I I I I I I I I I, (Low) (High) 2. How much physical activity was required (e.g., scrolling, clicking, typing, etc.)? Physical Demand I • I I I I I I "I 1 f 1 1 I 1 1 1 1 1 1 1 1 1 (Low) (High) 3. How much time pressure did you feel due to the rate or pace at which the tasks needed to be accomplished? Temporal Demand j 1 1 1 1 1 j 1 | j 1 f i l l 1 1 1 1 1 1 1 (Low) (High) 4. How successful do you think you were in accomplishing the goals of the task set by the experimenter? Performance 1 | 1 | j | 1 I I | j 1 I I I 1 J i l l 1 1 (Low) (High) 5. How hard did you have to work (mentally and physically) to accomplish your level of performance? Effort 1 1 1 1 I 1 1 | | i j 1 1 1 1 1 1 1 1 1 1 1 (Low) (High) 6. How insecure, discouraged, irritated, stressed and annoyed versus secure, gratified, content, relaxed and complacent did you feel during the task? Frustration i i i f i 1 i i i I I 1 1 1 1 1 1 1 1 1 1 1 (Low) (High) Appendix C. Experiment Study Resources 96 Importance of Different Workload Categories Please select the member of each pair that had the more significant effect on the overall workload for the task that you just performed using the filer system and the email. Descriptions of the categories can be found at the bottom of the page. Physical Demand • or • Mental Demand Temporal Demand • or • Mental Demand Performance • or • Mental Demand Effort or • Mental Demand Frustration Level • or • Mental Demand Temporal Demand • or • Physical Demand Performance • or • Physical Demand Effort • or • Physical Demand Frustration Level • or • Physical Demand Temporal Demand • or • Performance Temporal Demand • or • Effort Temporal Demand • or • Frustration Level Performance • or • Effort Performance • or • Frustration Level Frustration Level • or • Effort Mental Demand: How much mental and perceptual activity was required to perform the task (e.g., thinking, deciding, remembering, searching, etc.)? Physical Demand: How much physical activity was required (e.g., scrolling, clicking, typing, etc.)? Temporal Demand: How much time pressure did you feel due to the rate or pace at which the tasks needed to be accomplished? Performance: How successful do you think you were in accomplishing the goals of the task set by the experimenter (or yourself)? Effort: How hard did you have to work (mentally and physically) to accomplish your level of performance? Frustration: How insecure, discouraged, irritated, stressed and annoyed versus secure, gratified, content, relaxed and complacent did you feel during the task? Appendix C. Experiment Study Resources 97 7. How satisfied are you with the process taken to accomplish the task? Process Satisfaction I I -I I I i I 1 1 I I (Low) (High) 8. How satisfied are you with the time taken to accomplish the task? Time Satisfaction i 1 i i i 1 I i i I i: (Low) (High) 9. To what extent do you think your annotations and bundles will facilitate the efficient reviewing by your co-authors? Reviewing efficiency I I 1 I I 1 I J" -I f I (Low) (High) 10. To what extent do you think your annotations and bundles will facilitate the accurate reviewing by your co-authors? Reviewing accuracy 1 1 ...i i 1 1 i i i 1 i: (Low) (High) 11. How satisfied are you with the ease of using this writing system? Ease of system use I I I I -I I I I I i I. (Low) (High) Appendix C. Experiment Study Resources 98 Questions: 1. What was the most difficult task for you to complete using the system? 2. What particular aspect(s) of bundles did you like? 3. What particular aspect(s) of bundles did you dislike? 4. What particular aspect(s) of this collaborative writing system did you like? 5. What particular aspect(s) of this collaborative writing system did you dislike? Appendix C. Experiment Study Resources 99 C.2.3 System-specific Questionnaire: Simple System Experience: (To be completed after completing the task using the filter system) With respect to your experience performing the given task with the filter system and email, please answer the following questions by marking an 'X' along the scale below the corresponding question. 1. How much mental and perceptual activity was required to perform the task (e.g., thinking, deciding, remembering, searching, etc.)? Mental Demand i l I I | I | i i | | I I I I I I I I I I I (Low) (High) 2. How much physical activity was required (e.g., scrolling, clicking, typing, etc.)? Physical Demand l I I I I I I I I I I I I I I I I I I I I I (Low) (High) 3. How much time pressure did you feel due to the rate or pace at which the tasks needed to be accomplished? Temporal Demand I | | j' | I I I | I | I I I I I I I I I I I (Low) (High) 4. How successful do you think you were in accomplishing the goals of the task set by the experimenter? Performance i i t i i I i i i | i I I I I I I I I I I I (Low) (High) 5. How hard did you have to work (mentally and physically) to accomplish your level of performance? Effort I 1 "i i i I i I i | I I I I I I I I I I I I; (Low) (High) 6. How insecure, discouraged, irritated, stressed and annoyed versus secure, gratified, content, relaxed and complacent did you feel during the task? Frustration j I I | | I | | i | j I I I I I I I I I I Is (Low) (High) Appendix C. Experiment Study Resources 100 Importance of Different Workload Categories Please select the member of each pair that had the more significant effect on the overall workload for the task that you just performed using the filer system and the email. Descriptions of the categories can be found at the bottom of the page. Physical Demand • or • Mental Demand Temporal Demand • or • Mental Demand Performance • or • Mental Demand Effort • or • Mental Demand Frustration Level • or • Mental Demand Temporal Demand • or • Physical Demand Performance • or • Physical Demand Effort • or • Physical Demand Frustration Level • or • Physical Demand Temporal Demand • or • Performance Temporal Demand • or • Effort Temporal Demand • or 0 Frustration Level Performance • or • Effort Performance • or • Frustration Level Frustration Level • or • Effort Mental Demand: How much mental and perceptual activity was required to perform the task (e.g., thinking, deciding, remembering, searching, etc.)? Physical Demand: How much physical activity was required (e.g., scrolling, clicking, typing, etc.)? Temporal Demand: How much time pressure did you feel due to the rate or pace at which the tasks needed to be accomplished? Performance: How successful do you think you were in accomplishing the goals of the task set by the experimenter (or yourself)? Effort: How hard did you have to work (mentally and physically) to accomplish your level of performance? Frustration: How insecure, discouraged, irritated, stressed and annoyed versus secure, gratified, content, relaxed and complacent did you feel during the task? Appendix C. Experiment Study Resources 101 7. How satisfied are you with the process taken to accomplish the task? Process Satisfaction (Low) (High) 8. How satisfied are you with the time taken to accomplish the task? Time Satisfaction 'i 1 1 I .1: 1 I 1 .1 I I (Low) (High) 9. To what extent do you think your annotations and email message will facilitate the efficient reviewing by your co-authors? Reviewing efficiency (Low) (High) 10. t o what extent do you think your annotations and email message will facilitate the accurate reviewing by your co-authors? Reviewing accuracy 1 i i i i 1 i i i i i: (Low) (High) 11. How satisfied are you with the ease of using this writing system? Ease of system use I i 1 i i I i -I i i : i; (Low) (High) Appendix C. Experiment Study Resources 102 Questions: 1. What was the most difficult task for you to complete using the system? 2. What particular aspect(s) of this collaborative writing system did you like? 3. What particular aspect(s) of this collaborative writing system did you dislike? Appendix C. Experiment Study Resources 103 C.2.4 Comparative Questionnaire Comparative Experience: (Tobe completed alter completing both tasks) P lease answer the following questions by marking - an 'X ' along the scale with respect to your experience performing the task with the bundle system, - an ' 0 ' a long the scale with respect to your experience performing the task with the filter system and email. W e are interested in your exper iences with each system relative to the other. Hence, you do not have to worry about your answers in this questionnaire identical to your answers in the previous questionnaires. 1. How much mental and perceptual activity was required to perform the task (e.g., thinking, deciding, remembering, searching, etc.)? Mental Demand I | i | 1 I 1 i I 1 1 1 I I I 1 1 1 I 1 1 1 (Low) (High) 2. How much physical activity was required (e.g., scrolling, clicking, typing, etc.)? Physical Demand i I I I 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 (Low) (High) 3. How much time pressure did you feel due to the rate or pace at which the tasks needed to be accomplished? Temporal Demand 1 1 1 | 1 I I I 1 1 | 1 1 1 1 1 1 1 1 1 1 1 (Low) (High) 4. How successful do you think you were in accomplishing the goals of the task set by the experimenter? Performance 1 1 1 1 1 1 1 1 1 1 1 (Low) (High) 5. How hard did you have to work (mentally and physically) to accomplish your level of performance? Effort 1 | I | 1 1 1 1 | | | 1 1 i 1 1 1 1 1 1 1 1, (Low) (High) 6. How insecure, discouraged, irritated, stressed and annoyed versus secure, gratified, content, relaxed and complacent did you feel during the task? Frustration 1 1 | f i l l 1 1 I i 1 I 1 I I I 1 ! 1 1 1 (Low) (High) Appendix C. Experiment Study Resources 104 7. How satisfied are you with the process taken to accomplish the task? Process Satisfaction 1 I I 1 I I (Low) (High) 8. How satisfied are you with the time taken to accomplish the task? Time Satisfaction I I I I I 1 I I 1 1 I (Low) (High) 9. To what extent do you think your annotations and bundle / email message will facilitate the efficient reviewing by your co-authors? Reviewing efficiency (Low) (High) 10. To what extent do you think your annotations and bundle / email message will facilitate the accurate reviewing by your co-authors? Reviewing accuracy I i \ I ;i I i I \ I I (Low) (High) , 11. How satisfied are you with the ease of using this writing system? Ease of system use J 1 • -I I I 1 I I 1 I I (Low) (High) Appendix C. Experiment Study Resources 105 C . 3 Study Documents C.3.1 Training Document Parenting and Honesty Some parenting experts urge parents to tell their children the truth about their past experiment with drugs and sex, even when they don't want their children to do what they did. Others think parents should censor what they tell their children. This paper will demonstrate that it is better for parents to censor what they tell their children until the children are old enough to understand. One of the primary duties of parenting is to impart the parents' moral views to their children. Moral views are developed through a lifetime. This means that the activities that a parent engaged in, especially with drugs and sex, long before he or she was a parent, formed his or her moral views. This does not mean that they are stuck with these views, but that they helped to shape the parents' views. For instance, a woman who experienced with drugs as a teenager and regrets it will probably have developed the moral view that teenagers should not take drugs, based on (not despite) the fact that she did it herself. Children and young teens just don't get it, so it would be harmful for the parent to have to hide her drug use while telling her children not to try drugs. Children have such a black-and-white view of the world that they cannot understand that a person could do something and then regret it later. Confessing all of one's young indiscretions would make a parent's discussions with their children about drugs and sex counterproductive. In fact, many parents might choose not to discuss these subjects at all, rather than risk confusing their children or revealing facts their children are too young to know. Opponents of this view might argue that once a child learns the truth about his or her parents' activities, he or she will defeat the parent and will no longer see the parents' views as credible. However, this is not a logical assumption, since by the time the child finds out the truth, he or she will be old enough to understand why the parent lied. A 22-year-old who finds out that her mother smoked pot in college will understand why the mother did not tell her that when she was 12. An 18-year-old who discovers that his Appendix C. Experiment Study Resources 106 father has premarital sex will most likely feel very differently about that than he would have when he was 11. To conclude, parents need to adjust what they told their children to fit the child's age and developmental stage. When the children are young, they should be less concerned with telling the absolute truth about their own experiences than they should be with making sure their children know what their own moral views are. Later, once the children are old enough, the parents could choose to reveal the truth of their own experiments, but at this point the children will be past the danger point. Appendix C. Experiment Study Resources 107 C.3.2 Task Documents Document 1: D o c H Extending high school to five years 1. Introduction "Senior itch" is an incessant irritation that high school students all crave to scratch. The cure? Graduation. As students progress through their high school years growing in wisdom and maturity, they are desperate for freedom. Yet what they desire most is not always what is best for them. Although most students don't wish to admit it, extending the high school career to five years would make an important and beneficial affect on their future. The current four years of high school education do not provide sufficient time for motivated students to accomplish all of their goals before college. Merely getting accepted by a selective college or university requires much pre-planned effort, and students who are preoccupied with their studies and other activities simply do not have enough time. 2. Interest High schools are always adding new and interesting courses to the curricula which they offer. However, many college-bound students do not have sufficient time to explore these different classes. High school students thus are forced to choose classes that fit into their limited schedules. Occasionally classes that perceive to be easier are chosen, merely to make sure that their transcripts stand out in the admissions pool. Fifteen years of age seems too late to start making choices based on "career", rather than choosing classes that they might find interesting and fun. An extra year of high school would provide students with opportunities to explore different subjects and take classes that they interest. Consequently, students would enjoy their education and develop more stakes in school. 3. Individuality Colleges thoroughly consider the ways in which an applicant has spent his or her four years of high school. Students who take on leadership roles, display dictation to an organization, and show other evidence of being well-rounded appeal most to elite educational institutions. Although some high school students may desire to take on leadership positions numerous extra-curricular organizations, they are often faced with school regulations that limit the number of offices they may hold. Even when their school does not place restrictions on such involvement, students eventually reach the limits of what can be done in a 24-hour day. All too often, students will find that they cannot participate in as many extra-curricular activities as they would like, due to the fact that they just do not have enough time. An extra year of high school would enable such students to become involved in more than one activity, providing them with sufficient time to become fully engaged in each to a point where they might eventually take on leadership roles. Then, colleges would receive more articulated resumes from applicants, while various school organizations would benefit stronger student participation, and students themselves would receive greater recognition for their efforts. 4. College Applications Merely being accepted by a selective college or university requires much pre-planned effort, the time for which is literally unavailable to students already concerned with grades and other activities. 5. Grades Struggling to achieve leadership roles and become the well-rounded applicants that colleges prefer, many Appendix C. Experiment Study Resources 108 students find it difficult to maintain respectable grade-point-average during high school while remaining involved in activities that interest them. However, this tradeoff wil l not consider when colleges seek applicants with high grade-point-averages in their admissions pool. Lengthening the period of time spent in high school would allow more students to achieve both higher grades and fuller participation in extra-curricular activities on their lifeline. Rather than being forced to pick and choose among classes with the intention of evaluating their GPA, students could spread out their studies and make the most of every single year. With less pressure and more time, dedicated students could thus improve their grades. They would also enjoy studying these difficult subjects and interested in them. Some dropout students complete high school by means of an equivalency test and receive an alternative credential such as a General Educational Development certificate. 6. Individuality Aside from the educational benefits, many students find that they shall be provided more time to accomplish a variety of goals. For example, while still at school, students wil l acquire a job so that the money earned might help them pay their way through college. With such a short preparation period before college, students can hardly be expected to accumulate an appropriate amount of funds for this purpose. In addition, many students who are interested in performing community service prior to attending college do not have enough time for this during the regular four-year high school program. Colleges wil l prefer students who have a rich background in community service and can show evidence of personal responsibility such as holding a job; however, under current circumstances, it remains difficult for students to unbalance their workload, and find the hours to put into these tasks. 7. Technical Skills With classes, extracurricular activities, and part-time work, time is fleeting high school students. They have very little time to spend with their families, or for leisure activities. Even when some time wi l l be available to recuperate, they often devote these extra hours to trying to accomplish more. Teens need to do what is necessary for their mental and physical health. They need to learn how to process effectively. Given the current pace many students' lives, this generation is likely to be full of workaholics, who do not understand or value the importance of family relationships. 8. Dropout Rates Maintaining interest in their schoolwork is a trivial factor, if students are to succeed academically. Some high schools students drop out because they lose interest in school. Extending high school to five years in the way suggested here would reduce the number of students who drop out. 9. Conclusion High schools lay the foundation for the rest of our life. Ak in to money in the bank, the investment of an additional year when we're young can make an enormous difference. With the additional time that a five-year high school program would provide, motivated students would be able to become more involved in their studies, boost their grades, become gainfully employed, and engage in community service. Since colleges seek exactly these properties in successful applications for admission, high school students should therefore be provided with more time to take part in such endeavors. Students are clearly working hard to ensure their futures, and another year of high school would enable them to achieve this more effectively. Appendix C. Experiment Study Resources 109 Document 2: DocL Technology and the Lack of Leisure Time 1. Introduction In a society where most households have clocks, phones, and televisions in almost every room, and daily schedules are demarcated by minutes instead of hours, many people suffer from stress and constanUy whine that they don't have enough time to do everything they want to do. This complaint appears paradoxical when one considers the almost exponential development in technology. Computers are speedier and more powerful, with machines that do various tasks for us, while our means of transport takes us where we need to be much more quickly than in the past. Still, we have less time than ever to spend leisurely. This problem arises, not because technologies have failed to achieve the goal of improving efficiency, but rather because it has become a pursuit in itself, with our consumer society subjecting us to a basic ethical drive for "more". 2. Relaxing Pursuits While technological advances have made many things better and faster, they have also created more activities. We now have televisions, computers, palm pilots, stereos, DVDs, play stations, and cell phones to occupy our time. Furthermore, all of these things are within reach of the average Canadians. Meanwhile, pursuits that consider as relaxing are becoming more expensive and less accessible. For example, for most people relaxing massages are affordable, and as cities grow larger, nature walks are becoming harder to found. It is sometimes easier for people to just sit down in front of the T V than to do something loose. 3. Diversity Many Canadians complain that they do not have enough time. Most of us are caught up in a schedule of going to work each day, coming home late, then taking care of mandate details before finally falling into bed, only to get up early the next day to continue the routine. In most households, both parents work full-time, and are busy working throughout the week, which leaves them only the weekends to take care household duties such as cleaning and paying bills. As a result, many parents wil l feel ostracized by their children because the parents do not have enough time or energy to spend with their kids. Many people suffer from chronic stress because they do not take time out their busy lives simply to relax. 4. Effectiveness of Technology Our computers are faster and more powerful, we have more machines that do various tasks for us, and our means of transport takes us to our destination much more quickly. 5. Intrusion of Media What little free time people do have, they spend in front of the T V or on the computer, thus they are not relaxing, but are actively engaged in a cognitive process. Rather than relaxing, people experience tension or stress, because they have spent their entire time keeping up with the constant glimpse of images, storing information about characters, plots, themes, products and music, without realizing it. The intrusive aspect of media represents another reason why people do not have enough time. People fill their time with meditated technologies and caught up in their favorite shows or games rather than taking the time to perform other things that they have been "meaning to do". The use of email has been shown to eliminate a lot of visual and verbal cues we often use in communicating with one another. Appendix C. Experiment Study Resources 110 6. Diversity Canada was founded on the foundation of a conservative Protestant ethic which dictates that people shall work hard now so they may reap the rewards in the future. While this ethic is essential for productivity and pushes one to want more, it has become psychologically detrimental. People wil l push themselves to become more efficient so that they can accomplish more. However, there is the inclusion of more goals, thus, in effect, the work needed to achieve a goal never gets done. For example, people desire to buy more things to achieve higher social status for themselves. Hence, instead of saving for early retirement, they end up having to work longer to pay off their debts. In addition, people are so busy pushing themselves daily with the vague promise of retirement at the end, hoping to retire sooner in life that they wil l forget to stop and enjoy life as it is happening. In order to avoid working when they are older, and enjoy each day of their lives, people forgo daily enjoyments for the promise of future rewards. 7. Technology Addiction "Leisure" should be a matter of personal choice, not fill ing time made available technologically advanced, efficient tools. The time saved by efficient tools wil l not necessarily transformed into leisure time. Ultimately, a person's motivation, in addition to his or her personality and lifestyle choices determine how the time saved is used. It is up to individuals to award the prize of leisure. They can choose to use it leisure time or for putting in more hours to make more money to support living expenses, or to buy the new car which they absolutely must have. 8. Social Impacts Social interactions among people and the ways in which they communicate are affected by technology. Hence, there is no doubt that technological change prevents social change. Technologies leave little time for socialization and leisure. 9. Conclusion While advances in technologies have made production more efficient, it has also burdened us with more things to manage and accomplish. As a result, people find themselves running around endlessly, sometimes forgetting what they're seeking. The point of any leisure time should be not to demand a greater affinity and reliance to technology, but a refusal to allow technologies run our lives. We must learn to stop occasionally, breathe, and enjoy life as it comes. We must learn to achieve a balance between looking ahead to tomorrow and learning to enjoy today. C.3.3 Manipulations Planted in the Task Documents This section contains the task documents with planted problems identified. The following keywords are used within the documents to indicate each of the problem category planted. • US: Unclear Statements • FL: Flawed Logic • TI: Title Inconsistency • UA: Unsupported Argument • GI: Inconsistent Wording Appendix C. Experiment Study Resources 111 • AP: Incorrect Use of Active-Passive Verbs • VT: Incorrect Use of Verb Tense • MP: Missing Prepositions • UC: Use of Contractions • WC: Inappropriate Word Choice Appendix C. Experiment Study Resources 112 Document 1: DocH Extending high school to five years 1. Introduction "Senior i t ch " is an incessant irritation that high school students all crave to scratch. The cure? Graduation. As students progress through their high school years growing in wisdom and maturity, they are desperate for freedom. Yet what they desire most is not always what is best for them. Although most students don 'tmci i wish to admit it, extending the high school career to five years would make an important and beneficial affectum on their future. The current four years of high school education do not provide sufficient time for motivated students to accomplish all of their goals before college. Merely getting accepted by a selective college or university requires much pre-planned effort, and students who are preoccupied with their studies and other activities simply do not have enough time. 2. Interest H igh schools are always adding new and interesting courses to the curricula which they offer. However, many college-bound students do not have sufficient time to explore these different classes. High school students thus are forced to choose classes that fit into their limited schedules. Occasionally classes that perceiveiAiGi to be easier are chosen, merely to make sure that their transcripts stand out in the admissions pool. Fifteen years of age seems too late to start making choices based on "career"n-i,n , rather than choosing classes that they might find interesting and fun. A n extra year of high school would provide students with opportunities to explore different subjects and take classes that they in teres t iAivi i . Consequently, students would enjoy their education and develop more stakesmsti in school. 3. Individuality! rni Colleges thoroughly consider the ways in which an applicant has spent his or her four years of high school. Students who take on leadership roles, display dictation to an organizationiusm , and show other evidence of being well-rounded appeal most to elite educational institutions. Although some high school students may desire to take on leadership positionsiMjsi numerous extra-curricular organizations, they are often faced with school regulations that limit the number of offices they may hold. AU too often, students wil l ivi loi find that they cannot participate in as many extra-curricular activities as they would like, due to the fact that they just do not have enough time. A n extra year of high school would enable such students to become involved in more than one activity. Then, colleges would receive more aiticulatedtwcin resumes from applicants, while various school organizations would benefit strongeriMi student participation, and students themselves would receive greater recognition for their efforts. 4. College Applications Merely being accepted by a selective college or university requires much pre-planned effort, the time for which is literally unavailable to students already concerned with grades and other activities, IUAIJI 5. Grades Struggling to achieve leadership roles and become the well-rounded applicants that colleges prefer, many Appendix C. Experiment Study Resources 113 students find it difficult to maintain respectable grade-point-average during high school while reniaining involved in activities that interest them. However, this tradeoff will not consideriAPi-n when colleges seek applicants with high grade-point-averages in their admissions pool. Lengthening the period of time spent in high school would allow more students to achieve both higher grades and fuller participation in extra-curricular activities on their lifelineiusisi . Rather than being forced to pick and choose among classes with the intention of evaluating their GPAiusioi, students could spread out their studies and make the most of every single year. With less pressure and more time, dedicated students could thus improve their grades. They would also enjoy studying these difficult subjects and uiterestediACiil in them. Some dropout students complete high school by means of an equivalency test and receive an alternative credential such as a General Educational Development certificate.iFLini 6. Individualitymij] Aside from the educational benefits, many students find that they shallivnoi be provided more time to accomplish a variety of goals. For example, while still at school, students willivmi acquire a job so that the money earned might help them pay their way through college. In addition, many students who are interested in performing community service prior to attending college do not have enough time for this during the regular four-year high school program. Colleges willivrei prefer students who have a rich background in community service and can show evidence of personal responsibility such as holding a job; however, under current circumstances, it remains difficult for students to unbalance their workloadiusi.'i, and find the hours to put into these tasks. 7. Technical SkillsrnMl With classes, extracurricular activities, and part-time work, time is fleeting hiphiMPzsi school students. Even when some time willivra.i be available to recuperate, they often devote these extra hours to trying to accomplish more. Teens need to do what is necessary for their mental and physical health. They need to learn how to process effectively! us:?i. Given the current pace manyiMrcsi students' lives, this generation is likely to be full of workaholics, who do not understand or value the importance of family relationships. 8. Dropout Rates Maintaining interest in their schoolwork is a trivial factoring)), if students are to succeed academically. Some high schools students 101301 drop out because they lose interest in school. Extending high school to five years in the way suggested here would reduce the number of students who drop out. I I M J I I 9. Conclusion High schools lay the foundation for the rest of our life. Akin to money in the bank, the investment of an additional year when we'reitjozi young can make an enormous difference. With the additional time that a five-year high school program would provide, motivated students would be able to become more involved in their studies, boost their grades, become gainfully employed, and engage in community service. Since colleges seek exactly these propertiesiwo'ii in successful applications for admission, high school students should therefore be provided with more time to take part in such endeavors. Appendix C. Experiment Study Resources 114 Document 2: DocL Technology and the Lack of Leisure Time 1. Introduction In a society where most households have clocks, phones, and televisions in almost every room, and daily schedules are demarcated by minutes instead of hours, many people suffer from stress and constantly whine that they don'tiucn have enough time to do everything they want to do. This complaint appears paradoxical when one considers the almost exponential development in technology. Computers are speedier and more powerful, with machines that do various tasks for us, while our means of transport takes us where we need to be much more quickly than in the past. Still, we have less time than ever to spend leisurelyicm . This problem arises, not because technologies have failed to achieve the goal of improving efficiency, but rather because it has become a pursuit in itself, with our consumer society subjecting us to a basic ethical drive for "more". 2. Relaxing Pursuits While technological advances have made many things better and raster, they have also created more activities. We now have televisions, computers, palm pilots, stereos, DVDs, play stations, and cell phones to occupy our time. Furthermore, all of these things are within reach of the average Canadians. Meanwhile, pursuits that consideriAp.ii as relaxing are becoming more expensive and less accessible. For example, for most people relaxing massages are affordablein-n, and as cities grow larger, nature walks are becoming harder to foundiAPsi. It is sometimes easier for people to just sit down in front of the TV than to do something loose.ius6i 3. Diversitynm Many Canadians complain that they do not have enough time. Most of us are caught up in a schedule of going to work each day, coming home late, then taking care of mandate details before finally falling into bed.msai only to get up early the next day to continue the routine. In most households, both parents work full-time, and are busy working throughout the week, which leaves them only the weekends to take care household d u t i e s i M w i such as cleaning and paying bills. As a result, many parents willivTioi feel ostracizediwcin by their children because the parents do not have enough time or energy to spend with their kids. Many people suffer from chronic stress because they do not take time out their busy UvesiMPi2i simply to relax. 4. Effectiveness of Technology Our computers are faster and more powerful, we have more machines that do various tasks for us, and our means of transport takes us to our destination much more quicklv.iuAiai 5. Intrusion of Media What little free time people do have, they spend in front of the TV or on the computer, thus they are not relaxing, but are actively engaged in a cognitive process. Rather than relaxingiAPui, people experience tension or stress, because they have spent their entire time keeping up with the constant glimpse of Appendix C. Experiment Study Resources 115 images.iusiii storing information about characters, plots, themes, products and music, without realizing it. The intrusive aspect of media represents another reason why people do not have enough time. People fill their time with meditated technologiesiusi6i and caughtiAPi7i up in their favorite shows or games rather than taking the time to perform other things that they have been "meaning to do". The use of email has been shown to eliminate a lot of visual and verbal cues we often use in communicating with one another.iR.iai 6. DiversityiTiisi Canada was founded on the foundation of a conservative Protestant ethic which dictates that people shallivrzoi work hard now so they may reap the rewards in the future. While this ethic is essential for productivity and pushes one to want more, it has become psychologically detrimental. People wiUivriii push themselves to become more efficient so that they can accomplish more. However, there is the inclusion of more goals, thus, in effect, the work needed to achieve a goal never gets done. For example, people desire to buy more things to achieve higher social status for themselves. Hence, instead of saving for early retirement, they end up having to work longer to pay off their debts. In addition, people are so busy pushing themselves daily with the vague promise of retirement at the end, hoping to retire sooner in life that they W J I I I V T C T forget to stop and enjoy life as it is happening. In order to avoid working when they are older, and enjoy each day of their Uvesjusjsi people forgo daily enjoyments for the promise of future rewards. 7. Technology Addictionrm-ii "Leisure" should be a matter of personal choice, not filling time made available IMP25I technologically advanced, efficient tools. The time saved by efficient tools willrvTMi not necessarily transformed into leisure time. Ultimately, a person's motivation, in addition to his or her personality and lifestyle choices determine how the time saved is used. It is up to individuals to award the prize of leisure.asii They can choose to use it leisureiMP23i time or for putting in more hours to make more money to support living expenses, or to buy the new car which they absolutely must have. 8. Social Impacts Social interactions among people and the ways in which they communicate are affected by technology. Hence, there is no doubt that technological change prevents social change.IFL?9I Technologiesioijoi leave little time for socialization and leisure, M - U I I 9. Conclusion While advances in technologies have made production more efficient, it has also burdened us with more things to manage and accomplish. As a result, people find themselves running around endlessly, sometimes forgetting what thev'reiuozi seeking. The pointiwc3.n of any leisure time should be not to demand a greater affinity and reliance to technology, but a refusal to allow technologies run our lives. We must learn to stop occasionally, breathe, and enjoy life as it comes. We must learn to achieve a balance between looking ahead to tomorrow and learning to enjoy today. Appendix C. Experiment Study Resources 116 C.4 Tasks C.4.1 Training Tasks For Bundle System Practice Tasks 1. Edit the first paragraph so that it will read as follows (the text needed to be inserted, deleted and replaced are indicated with bold font): Some parenting experts urge that parents te tell their children the truth about their past cxpcrimcntcxperience with drugs and sex, even when they don't want their children to do what they did. Others think parents should censor what they tell their children. This paper will demonstrate that it is better for parents to censor what they tell their children until the children are old and mature enough to understand. 2. Create a comment annotation with a tag, anchored to the phrase "This paper" (paragraph 1, last sentence) as follows: Comment text: Shall we say "We will" instead of "This paper"? Tag: wordchoice 3. Create a comment annotation as a follow-up comment on the Insert annotation "and mature" (paragraph 1, last sentence) as follows: Comment text: 1 want to include "mature" in addition to the word "old". I believe maturity is more important than the age to understand certain things in life. 4. Create a comment annotation, anchored to the phrase "are developed" (paragraph 2, second sentence) as follows: Comment text: Will the word "evolved" be more appropriate than "developed" here? 5. Edit the second last sentence in the second paragraph so that it will read as follows (replacements are indicated with bold font): For instance, a woman who expcrienccdcxperimented with drugs as a teenager and rcgretsregretted it will probably have developed the moral view that teenagers should not take drugs, based on (not despite) the fact that she did it herself. 6. Create a new bundle named "Experiment vs. Experience", and add the annotations that replaced "experience" with "experiment" (i.e., annotations created in task 1 and 5). Add the following note to the bundle: Note: We are trying to say parents tried drugs as an experiment, aren't we? 7. Create a comment annotation with a tag, anchored to the phrase "Children and young teens just don't get it" (paragraph 2, last sentence) as follows: Comment text: Watch the tone here. Tag: tone Appendix C. Experiment Study Resources 117 8. Create a comment annotation, anchored to the word "hide" (paragraph 2, last sentence) as follows: Comment text: This contradicts the sentence logically. I think you meant to say "reveal". 9. Create a new bundle named "Critical comments", and add the comment annotation that you created in the previous task ("This contradicts the sentence logically..."). 10. Switch to "All Annotations" tab. Select the annotation that you created in the task 3: ("/ want to include mature in addition to the word "old"..."). Add the selected annotation to the bundle you created in the previous task "Critical comments" by clicking "Add to Bundles" button. 11. Edit the comment you created in the task 4 ("Will the word "evolved" be more appropriate...") by adding a tag named "ivordchoice". 12. From All Annotations, filter the comment annotations that have the tags "wordchoice" or "tone ". Save the filter results in a bundle named "Need rewrite". 13. Remove the comment annotation: "Shalt we say 'We will' instead of 'This paper' from the bundle named "Need rewrite". Then save the bundle to reflect the changes, and close the bundle tab. 14. To which bundles (if any) are associated with the annotation "Replace: experiment with experience" from the first paragraph. (Hint: you can use "View Associated Bundles" from the right click menu to check) 15. Delete the insert annotation "Inserted: and mature" from the document. (Hint: you might notice that you will get an error saying you can't delete nested annotations. Try to delete the follow-up comment of"/ want to include 'mature' in addition..." before deleting the insert annotation.) 16. Create a general comment as follows: Comment text: This paper seems on track. But you might want to add more supporting evidence for your argument. Appendix C. Experiment Study Resources 118 For Simple System Practice Tasks 1. Edit the first paragraph so that it will read as follows (the text needed to be inserted, deleted and replaced are indicated with bold font): Some parenting experts urge that parents te tell their children the truth about their past cxpcrimcntcxperience with drugs and sex, even when they don't want their children to do what they did. Others think parents should censor what they tell their children. This paper will demonstrate that it is better for parents to censor what they tell their children until the children are old and mature enough to understand. 2. Create a comment annotation, anchored to the phrase "This paper" (paragraph 1, last sentence) as follows: Comment text: Shall we say "We will" instead of "This paper"? 3. Create a comment annotation, anchored to the inserted words "and mature" (paragraph 1, last sentence) as follows: Comment text: J want to include "mature" in addition to the word "old". I believe maturity is more important than the age to understand certain things in life. 4. Create a comment annotation, anchored to the phrase "are developed" (paragraph 2, second sentence) as follows: Comment text: Will the word "evolved" be more appropriate than "developed" here? 5. Edit the second last sentence in the second paragraph so that it will read as follows (replacements are indicated with bold font): For instance, a woman who cxpericncedcxperimented with drugs as a teenager and rcgrctsregretted it will probably have developed the moral view that teenagers should not take drugs, based on (not despite) the fact that she did it herself. 6. Create a comment annotation, anchored to the phrase "Children and young teens just don't get it" (paragraph 2, last sentence) as follows: Comment text: Watch the tone here. 7. Create a comment annotation, anchored to the word "hide" (paragraph 2, last sentence) as follows: Comment text: This contradicts the sentence logically. I think you meant to say "reveal". 8. From All Annotations, filter the comment annotations that have been created. Appendix C. Experiment Study Resources 119 9. Edit the comment you created in the task 4 ("Will the word "evolved" be more appropriate...") by adding a phrase saying "since we are arguing that parents' views and values are changing over timeT' 10. Delete the comment annotation you created in task 3 "/ want to include "mature" in addition..." from the document. Then delete the insert annotation "and mature" from the document. Appendix C. Experiment Study Resources 120 C.4.2 Experiment Tasks For DocH Scenario: You have been recruited to participate in this study because you have expertise in reviewing documents and helping students improve their documents' quality and their writing skills. In this study, you will be asked to take a similar role, whose task involves reviewing documents. You will be playing the role of a co-author who is involved in a collaborative writing project that will result in a published paper (i.e., your name will be included as one of the co-authors for the publication). Your responsibilities in this project include reviewing the documents to improve the quality, and assisting your co-authors with the writing process. Imagine you are Sam, and you are collaborating with two other co-authors: Mary and John on an argumentative article, titled "Extending High School to Five Years". Mary and John are the primary authors of the given document and have written a complete draft of the document. They have divided the document into sections and each wrote different sections as followings (note that the paragraphs are numbered in the document as well): • l s > a n d o" 1 p a r a g r a p h s a r e c o - w r i t t e n b y b o t h M a r y a n d J o h n • 3 r d , 6 t h , 7th p a r a g r a p h s a r e w r i t t e n b y M a r y • 2 n d , 4 ' " , 5 " a n d Sm p a r a g r a p h s a r e w r i t t e n b y J o h n You are about to leave town for summer vacation tomorrow. They have requested that you review the document before you leave. Upon receiving your review, they will read your annotations and incorporate them into the document. Your task is to review the document; you can make direct edit changes to the document text or provide comments. Since you are not a primary writer in this project, you do not need to rewrite the document, but rather you need to point out the problems and/or make suggestions in the document. Along with your review, your co-authors requested that you provide the following specific feedback, but not limited to: • your general impression of the writing • brief summary of your review that provides an overall status of the document which will help your co-authors skim the document quickly and prioritize the remaining work. Please note that you can communicate your co-authors only via the tool(s) provided. The deadline for submitting the document is during your vacation period. They will not be able to access you before the submission once you leave tomorrow. Hence, please make sure that you complete the task (i.e., you need to finish reviewing the whole document and provide clear annotations along with responses to their requests.) You have an hour to complete the task. Please try to think out loud or narrate your thoughts while performing the task. Appendix C. Experiment Study Resources 121 For DocL Scenario: You have been recruited to participate in this study because you have expertise in reviewing documents and helping students improve their documents' quality and their writing skills. In this study, you will be asked to take a similar role, whose task involves reviewing documents. You will be playing the role of a co-author who is involved in a collaborative writing project that will result in a published paper (i.e., your name will be included as one of the co-authors for the publication). Your responsibilities in this project include reviewing the documents to improve the quality, and assisting your co-authors with the writing process. Imagine you are Sam, and you are collaborating with two other co-authors: Mary and John on an argumentative article, titled 'Technology and the Lack of Leisure Time". Mary and John are the primary authors of the given document and have written a complete draft of the document. They have divided the document into sections and each wrote different sections as followings (note that the paragraphs are numbered in the document as well): • l s l and 9th paragraphs are co-written by both Mary and John • 3"1, 6th, 7"1 paragraphs are written by Mary • 2nd, 4lh, 5lh and 8th paragraphs are written by John You are about to leave town for summer vacation tomorrow. They have requested that you review the document before you leave. Upon receiving your review, they will read your annotations and incorporate them into the document. Your task is to review the document; you can make direct edit changes to the document text or provide comments. Since you are not a primary writer in this project, you do not need to rewrite the document, but rather you need to point out the problems and/or make suggestions in the document. Along with your review, your co-authors requested that you provide the following specific feedback, but not limited to: • your general impression of the writing • brief summary of your review that provides an overall status of the document which will help your co-authors skim the document quickly and prioritize the remaining work. Please note that you can communicate your co-authors only via the tool(s) provided. The deadline for submitting the document is during your vacation period. They will not be able to access you before the submission once you leave tomorrow. Hence, please make sure that you complete the task (i.e., you need to finish reviewing the whole document and provide clear annotations along with responses to their requests.) You have an hour to complete the task. Please try to think out loud or narrate your thoughts while performing the task. A p p e n d i x D 122 U B C Research E t h i c s B o a r d Cer t i f ica te 77ie University of British Columbia Office of Research Services Behavioural Research Ethics Board Suite 102, 6190 Agronomy Road, Vancouver, B.C. V6T 1Z3 CERTIFICATE OF APPROVAL- MINIMAL RISK RENEWAL PRINCIPAL INVESTIGATOR: DEPARTMENT: UBC BREB NUMBER: K e l l o g g S. B o o t h H05-80494 INSTITUTION(S) WHERE RESEARCH WILL BE CARRIED OUT: Institution I Site UBC Point Grey Site Other locations where the research will be conducted: N / A CO-INVESTIGATOR(S): J o a n n a M c G r e n e r e Y a m i n H t u n SPONSORING AGENCIES: Natural Sciences and Engineering Research Council of Canada (NSERC) - "Network for Effective Collaboration Technology through Advanced Research" PROJECT TITLE: B05-0494 - Network for Effective Collaboration Technology through Advanced Research EXPIRY DATE OF THIS APPROVAL: June 5,2008 JAPPRQVAL DATE: June 5,2007 The Annual Renewal for Study have been reviewed and the procedures were found to be acceptable on ethical grounds for research involving human subjects. Approval Is issued on behalf of the Behavioural Research Ethics Board 

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
http://iiif.library.ubc.ca/presentation/dsp.831.1-0052060/manifest

Comment

Related Items