Science, Faculty of
Computer Science, Department of
DSpace
UBCV
Khan, Taslim Arefin
Master of Science - MSc
University of British Columbia
Listening to text using read-out-loud applications is a popular way for people to consume content when their visual attention is situationally impaired (e.g., commuting, walking, tired eyes). However, due to the linear nature of audio, such apps do not support skimming--a non-linear, rapid form of reading--essential for quickly grasping the gist and organization of difficult texts, like academic or professional documents. To support auditory skimming for situational impairment, we: (1) identified the user needs and challenges in auditory skimming through a formative study (N=20), (2) derived the concept of "eyes-reduced'' skimming that blends auditory and visual modes of reading, inspired by how participants mixed visual and non-visual interactions, (3) generated a set of design guidelines for eyes-reduced skimming, and (4) designed and evaluated a novel audio skimming app (Skimmer) that embodies the guidelines. Our in-situ preliminary observation study (N=6) suggests that participants are positive about our design and are able to auditorily skim documents. We discuss design implications for eyes-reduced reading, read-out-loud apps, and text-to-speech engines.
Designing an Eyes-Reduced Document SkimmingApp for Situational ImpairmentsbyTaslim Arefin KhanB. Sc., Bangladesh University of Engineering and Technology, 2015M. Sc., Bangladesh University of Engineering and Technology, 2017A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFMASTER OF SCIENCEinThe Faculty of Graduate and Postdoctoral Studies(Computer Science)THE UNIVERSITY OF BRITISH COLUMBIA(Vancouver)October 2019c© Taslim Arefin Khan, 2019The following individuals certify that they have read, and recommend to the Fac-ulty of Graduate and Postdoctoral Studies for acceptance, the thesis titled:Designing an Eyes-Reduced Document Skimming App for Situational Im-pairmentssubmitted by Taslim Arefin Khan in partial fulfillment of the requirements forthe degree of Master of Science in Computer Science.Examining Committee:Joanna McGrenere, Computer ScienceSupervisorDongwook Yoon, Computer ScienceCo-supervisor and Supervisory Committee MemberiiAbstractListening to text using read-out-loud applications is a popular way for people toconsume content when their visual attention is situationally impaired (e.g., com-muting, walking, tired eyes). However, due to the linear nature of audio, suchapps do not support skimming–a non-linear, rapid form of reading–essential forquickly grasping the gist and organization of difficult texts, like academic or pro-fessional documents. To support auditory skimming for situational impairment,we: (1) identified the user needs and challenges in auditory skimming through aformative study (N=20), (2) derived the concept of “eyes-reduced” skimming thatblends auditory and visual modes of reading, inspired by how participants mixedvisual and non-visual interactions, (3) generated a set of design guidelines foreyes-reduced skimming, and (4) designed and evaluated a novel audio skimmingapp (Skimmer) that embodies the guidelines. Our in-situ preliminary observationstudy (N=6) suggests that participants are positive about our design and are ableto auditorily skim documents. We discuss design implications for eyes-reducedreading, read-out-loud apps, and text-to-speech engines.iiiLay SummaryWhen bleary-eyed or in a bumpy bus, reading text can be difficult or even impos-sible. People in such situationally impaired conditions often listen to a text-to-speech (TTS) app that narrates the given text out loud. Listening to a machine-generated narration from start to end is a good way for understanding easy content,such as fiction. In contrast, when people read more difficult content such as re-search papers with their eyes, they tend to “skim read” by jumping between pagesand quickly glancing at selective sections. Existing TTS apps don’t fully supportaudio-based skimming.To support skim reading during situational impairments, we built Skimmer, anapp that helps the user to predominantly listen, but also enables looking at content(e.g., images) when necessary (hence “eyes-reduced” interaction). We evaluatedSkimmer in the context of riding a bus and found that users were able to skimdocuments in an eyes-reduced manner.ivPrefaceThe experiments described in this thesis were conducted with the approval of theUBC Behavioral Research Ethics Board (certifiate number H18-00664).A significant portion of this thesis has been submitted as a manuscript and isunder review in a top-tier conference while submitting this thesis. I am the leadauthor in that manuscript.Dr. Joanna McGrenere and Dr. Dongwook Yoon helped ideating and framingthe research since its inception in January 2018. Both of them provided supervi-sory assistance during the time of this research. They provided feedback on thedesign and development of the research. They also assisted me with the writingof the submitted manuscript mentioned above.Paul Bucci read this manuscript and provided feedback on writing.The eDAPT group led by Dr. Joanna McGrenere contributed with feedbackand early edits for parts of this manuscript. Several members of the MultimodalUser Experience (MUX) lab contributed in this research by participating as pi-lot participants in the studies, brainstorming ideas, and lending a hand in trou-bleshooting programming bugs. Ashish Chopra and Mohi Reza helped me ad-dress technical problems while implementing Skimmer. David Marino, GodfreyCheng, Matthew Chun, and Laura Cang helped me capture images and videosduring the study.vTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Research Questions and Approach . . . . . . . . . . . . . . . . . 21.2 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61.3 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.1 Situational Impairment and Eyes-free Interaction . . . . . . . . . 72.2 Human Listening and Audio Interfaces . . . . . . . . . . . . . . . 92.3 Studies on Skimming . . . . . . . . . . . . . . . . . . . . . . . . 102.3.1 Document Skimming . . . . . . . . . . . . . . . . . . . . 102.3.2 Audio Skimming . . . . . . . . . . . . . . . . . . . . . . 11vi3 Understanding User Needs for Auditory Skimming . . . . . . . . . . 163.1 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163.1.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . . 163.1.2 Task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173.1.3 Design of the Study . . . . . . . . . . . . . . . . . . . . . 173.1.4 Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . 173.1.5 Apparatus . . . . . . . . . . . . . . . . . . . . . . . . . . 183.1.6 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . 193.1.7 Quantitative Measures . . . . . . . . . . . . . . . . . . . 203.1.8 Data Collection and Data Analysis . . . . . . . . . . . . . 213.2 Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214 Design Guidelines for Eyes-Reduced Skimming . . . . . . . . . . . . 265 Design of Skimmer . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305.1 Eyes-Reduced Skimming Features of Skimmer . . . . . . . . . . 305.1.1 Basic Navigation for Eyes-Reduced Skimming . . . . . . 305.1.2 Designing for Easy to Understand Audio Narration . . . . 315.1.3 Navigating Between Discourse Markers . . . . . . . . . . 315.1.4 Understanding the High-level Organization . . . . . . . . 335.1.5 Opt-in to See Visual Content . . . . . . . . . . . . . . . . 335.1.6 Auditory Cues in Support of Skimming . . . . . . . . . . 345.1.7 Accommodating Individual Differences in Listening Rates 345.2 Design Process . . . . . . . . . . . . . . . . . . . . . . . . . . . 365.2.1 Conceptual Design . . . . . . . . . . . . . . . . . . . . . 365.2.2 Design and Prototyping Method Refinements . . . . . . . 395.2.3 Building a Usable Prototype . . . . . . . . . . . . . . . . 426 Preliminary Evaluation of Skimmer . . . . . . . . . . . . . . . . . . 446.1 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 446.2 Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45vii7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 498 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 549 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56A Study Materials Used in Need Finding Study . . . . . . . . . . . . . 68A.1 Call For Participation . . . . . . . . . . . . . . . . . . . . . . . . 68A.2 Consent Form for Participants . . . . . . . . . . . . . . . . . . . 70A.3 Study Protocol and Participant Training . . . . . . . . . . . . . . 74A.4 Questionnaires and Interview Questions . . . . . . . . . . . . . . 86B Study Materials Used in Preliminary Evaluation Study . . . . . . . 92B.1 Call For Participation . . . . . . . . . . . . . . . . . . . . . . . . 92B.2 Consent Form for Participants . . . . . . . . . . . . . . . . . . . 94B.3 Study Protocol and Participant Training . . . . . . . . . . . . . . 98B.4 Questionnaires and Interview Questions . . . . . . . . . . . . . . 100viiiList of Tables3.1 Articles selected for skimming task in need finding study. . . . . . 18ixList of Figures1.1 Basic user interface of Skimmer. . . . . . . . . . . . . . . . . . . 41.2 User interfaces of Skimmer showing how Skimmer handles num-bers and figures/tables in the text. . . . . . . . . . . . . . . . . . . 52.1 (Part 1) A comparison of the level of support for various skimmingfeatures in different read-out-loud apps. . . . . . . . . . . . . . . 132.2 (Part 2) A comparison of the level of support for various skimmingfeatures in different read-out-loud apps. . . . . . . . . . . . . . . 142.3 (Part 3) A comparison of the level of support for various skimmingfeatures in different read-out-loud apps. . . . . . . . . . . . . . . 153.1 Participant standing in front of a big screen where we projectedambient bus scenario. . . . . . . . . . . . . . . . . . . . . . . . . 195.1 Skimmer’s discourse marker navigation. . . . . . . . . . . . . . . 325.2 Low-fidelity paper prototype of Skimmer with different paper screens. 355.3 Mapping standard gestures to features. . . . . . . . . . . . . . . . 385.4 Refined paper prototype which is evaluated using a wizard-of-oztechnique. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 406.1 Field study to evaluate Skimmer. . . . . . . . . . . . . . . . . . . 45xAcknowledgementsI thank my supervisors - Dr. Joanna McGrenere and Dr. Dongwook Yoon, for theirencouragements, patience, and enthusiasm in my research. They have been verysupportive of me - often asking and taking good care of my emotional, mental,and financial troubles. I also thank them for deciding to co-supervise me and fundmy research all along.I thank my parents for their endless love and encouragement throughout myMasters. They have always kept me in their prayers and always believed in me. Ithank my brother and my sister for also keeping me in their prayers.I thank my beautiful wife, Saleema Najenin Siddiqua for all the supports, love,encouragement, persistence, and the surprise gifts that made me smile after longhectic days.I thank all the members from eDAPT group - Shareen Mahmud, FrancescoVitale, Yelim Kim, Izabelle Janzen, Jessalyn Alvina, and Hayley Guillou for theirfeedback and support all through my Masters. I also thank Paul Bucci, LauraCang, Soheil Kianzad, Matthew Chun, and Hanieh Shakeri for providing feedbackon my research from time to time.Finally, I thank Sheikh Tanvir for taking good care of me as a roommate andfor taking me to some of the most beautiful places in British Columbia and Al-berta. I also thank Asma Enayet for occasionally hosting delicious dinners, some-times at my request, so that I get enough variations in my diet.All these people kept me moving while miles away from home in Vancouver.xiChapter 1IntroductionPeople are consuming media auditorily more than ever before. This trend startedwith media such as podcasts–content specifically designed for auditory consump-tion that is typically listened to linearly and is based on pre-recorded humanspeech. As the quality of text-to-speech (TTS) technology has improved, sohas the popularity of listening to machine-generated narrations of textual contentusing read-out-loud apps (e.g., Mozilla Scout [1], Pocket [2], and Google Go).Using such apps is a good way to consume relatively straight-forward content(e.g., fiction and news articles) that has a linear narrative and can be understoodby listening to it from the beginning to the end. However, reading structured,more complex documents often involves a different, particular type of reading—skimming—that the existing read-out-loud apps fail to support.Skimming, in general, is a rapid, selective, and non-sequential form of reading[3, 4] that is not straightforward to support through the auditory modality whichis, in contrast, transient, sequential, and temporal [5]. Skimming requires thereader’s eyes to wander over a page [6]. It is also a goal-based reading strategy.People skim to learn the gist of a document, look for specific information in thedocument, and learn the structural organisation of the document. It involves rapideye movement over the text, spotting keywords, selectively reading sentences rel-evant to the goal, and jumping around the text, until the information need is met.1To date, skimming has not been well supported by existing voice reading apps thatuse TTS.Yet, skimming is the new norm for reading [7]. The amount of online contentalone increases rapidly each year [8], making it impossible for anyone to keep up,even with a small fraction of what is available. In several readerships includingbusiness executives, lawyers, doctors, etc., skimming is an important and frequentstyle of reading for consuming structured texts. Academics, just as one exampleclass of reader, have noted the challenge of keeping up with the literature, giventhe explosion of publications. Thus, it is not surprising that people are skimmingmore than ever before, often to assess whether content warrants the effort of a fullread [9]. A key restriction is that skimming today requires a person’s full visualattention which limits when and where it can be done. Thus, it is not possibleto skim in “on-the-go” situations, in the way that one can listen to podcasts oreven audiobooks, despite that there is strong evidence calling for better supportfor “on-the-go” reading [10, 4].1.1 Research Questions and ApproachIn this thesis, our research question is whether it is possible to design a systemto effectively support eyes-reduced skim reading, namely a system that combinesauditory and visual reading modes. It enables content, such as web pages to aca-demic papers, to be auditorily skimmable eyes-free, but also provides useful visualinformation should a user choose to selectively glance down at the screen to lookat content, such as an image in the text. We use the term eyes-reduced to jux-tapose with eyes-free, which has been explored in the literature [11]. Skimmingauditorily should help individuals who are situationally impaired, such as thosewho are on the go, using their visual attention for other things, such as walking orcommuting by bus/train but experience motion sickness while reading in a mov-ing vehicle. The commuting context is particularly motivating as commute timesare increasing substantially in many parts of the world [12]. At a broader level,2people with vision impairments should also get benefit from auditory skimming.3(a) (b) (c)Tap to jump to next sentenceTap to increase speech rateTap to jump to next discourse markerTap to decrease speech rateTap to jump to previous sentenceTap to jump to previous discourse markerFigure 1.1: (a) Skimmer’s user interface (what the user sees). (b) Annotated user interface of Skimmer showing sixregions on the screen for different tap-based gestures. (c) Overview page outlines the structural organization of adocument. The user switches between the Article page and the Overview page by swiping horizontally.4(a) (b) (c)Semi-transparent overlay displaying numbers when narratedFloating buttons giving the user opt-in and opt-out choicesFigure 1.2: (a) Skimmer sets out numbers and reads them in a rounded format to reinforce learning when the userglances at the screen. (b) Skimmer gently informs the user with haptic feedback when a figure/table is referred toin the text. The user has the choice to opt-in (blue button with caption “Show Figure 1”). Upon opt-in, Skimmershows the image and narrates the caption. (c) The user can resume back to main content at any time by tapping thered button.5To address our research question we:• conducted a user study (N=20) in a simulated commuting context to under-stand the needs of auditory skimming,• iterated extensively on design guidelines for eyes-reduced skimming,• designed and implemented an eyes-reduced skimming prototype app, Skim-mer (Figures 1.1, 1.2), that works with academic documents, and• conducted a preliminary evaluation (N = 6) on the bus.1.2 ContributionsOur work contributes the following:• the concept of multi-modal eyes-reduced skimming that can be used on-the-go,• design guidelines for eyes-reduced skimming that can be leveraged by otherdesigners, and• Skimmer, a system that implements those guidelines.1.3 OverviewPrevious work related to this research is summarized in Chapter 2. In Chapter 3,we describe the goal, methods, and results of our need finding study. Chapter 4outlines our proposed guidelines for eyes-reduced skimming. Chapter 5 illustratesthe design of our skimming application. In Chapter 6, we outline the results ofa preliminary evaluation of our design. Chapter 7 discusses results and designimplications building upon our research. Finally, in Chapters 8 and 9, we discussthe limitations of our research and summarize our research, respectively.6Chapter 2Related WorkOur study for designing an eyes-reduced document skimming application for sit-uational impairments is informed by an understanding of the notion of situationalimpairment, eyes-free and non-visual interaction systems, human listening capa-bilities, audio interfaces, and skimming text and audio content.2.1 Situational Impairment and Eyes-free Interac-tionThe proliferation of technology and devices in the lives of human beings havebrought about new dimensions in accessibility research. One such example issituational impairment. Situational Impairment (SI) is the temporary effect offactors (behavioral, environmental, attentional, affective, social, technological)on a person’s ability to interact with a computing device [13, 14].Some researchers have studied both situational impairment and mobile deviceinteraction. The study in [15] provides an overview of factors which cause SI andtheir impact on mobile device interaction. Such factors include - ambient temper-ature, mobile state, ambient noise, mood and stress, encumbrance, and ambientlight. Several studies have explored the impact of an on-the-go setting on mobile7device interaction. For example, walking [16], encumbrance [17] have negativeimpact on mobile device interaction, especially touch accuracy. Several reme-dies have been proposed in response to these problems, such as improving touchaccuracy by generating user-specific touch models [18], enlarging targets [19],CrashAlert [20] for eyes-busy mobile interaction while walking, and eyes-freeunimanual bezel-initiated gestures [21].Understanding needs for non-visual or eyes-free interaction and proposingsystems and interaction techniques, especially for reading in SI, have been ofinterest to researchers. Yi et al. [11] explored users’ motivation for eyes-free in-teraction on mobile devices, and found that certain environmental (difficulty read-ing from smartphone under extreme lighting condition), social (openly interactingwith a phone in a meeting), device specific (small screen causing inconvenience inreading), and personal reasons (self-enthusiasm) motivate people to interact eyes-free. As an example, Ghosh et al. [22] designed EDITalk, an eyes-free interactionfor editing word processing (highlight, comment, insert, delete, replace). Further-more, some studies explored reading in SI. The studies in [23, 24] have shown thatwalking has a negative impact on reading comprehension. In response, [24] pro-posed a speech-synthesis based audio interface to improve comprehension whilenavigating within the environment. SeeReader [25] is an eyes-free documentreader for mobile devices. It supports region-based (figures, tables, and para-graphs) navigation by detecting regions under the user’s finger while traversingin circular motion, reads out loud summary of regions, while also notifying pres-ence of images. Read4Me [26] is a prototype browser that leverages smartphonesensors to detect context switches and automatically onboards user to a hands-free mode by using TTS. While on the surface, SeeReader and our work mayseem similar, in our work, we systematically understand needs and interactiontechniques for skimming in SI, and explore how non-linear navigation (sentence,paragraph, section) can support skimming structured documents.Researchers have also investigated the potential of audio and haptics in eyes-free tasks, such as menu-selection and navigation. Metatla et al. [27] explored8audio-only workspace, empirically showing that audio can be used for non-visualcollaboration tasks (editing an audio menu). The study in [28] explored eyes-freemobile multi-tasking, showing the effectiveness of spatial audio when used withconcurrent audio streams in divided-attention (listening to podcast and browsingaudio menu) and selective-attention tasks (listening to music and browsing audiomenu). Researchers leveraged audio for eyes-free menu-selection tasks - 3D ra-dial pie menu with head gesture and 2D hand gesture [29], earPod [30], BlindSight[31]. Other studies have leveraged haptic feedback for eyes-free mobile device in-teraction and menu selection, such as [32, 33]. PocketMenu [34] is a non-visualinteraction technique with touchscreen menu items in an on-the-go setting, com-bining both audio (speech) and haptic (vibro-tactile) feedback. Researchers havealso explored eyes-free navigation systems in - (1) digital space - [35] by gesturetyping on an invisible and adjustable QWERTY keyboard, and (2) physical space- AudioGPS [36] using spatial audio for GPS navigation. In our work, we leverageboth audio and haptics to reinforce action completion in an eyes-reduced context,while doing non-linear navigation for document skimming.2.2 Human Listening and Audio InterfacesResearchers have explored the effects of TTS on both sighted and visually im-paired (VI) populations. In a recent study, Bragg et al. [37] performed a studyon intelligibility of fast, synthetic speech, on both sighted and VI participants.The study showed that VI people can listen to TTS at a higher rate than sightedpeople. They also showed that sighted people can listen to TTS at a rate higherthan the normal speaking rate. Mean listening rate for sighted participants was297 Words Per Minute (WPM), which is much higher than the typical humanspeaking rate of 120 - 180 WPM. Cohen et al. [38] recommended guidelines forVoice User Interface design based on the nuances of human listening perceptions.We take important design inspirations from these works, and also put forward ourown empirical observations from our study to complement their findings, such as9setting an optimum speed for skimming documents.In order to understand how state-of-the-art read-out-loud interfaces support ordo not support auditory skimming, we explored 13 such applications available inthe app stores, designed for both visually impaired and sighted users. Most ap-plications support either original layout (which keeps the original text formattingof the document) or a plain text layout (which gets rid off the document’s textformatting), or both. Applications differ by types of navigation (spatial, seman-tic, temporal) and granularity of each types (page, chapter, sentence, 10 seconds,30 seconds, and so on). We also observed differences in user interface designs -how basic controls are integrated, spatial organization of menus and sub-menus.Figures 2.1, 2.2, and 2.3 summarizes our comparative analysis on how well theexisting apps support different features for auditory skimming. The label “Sup-ported” means that the application fully supports a wide range of sub-features thatconstitute the feature. “Partially supported” indicates an limited feature support.“Not supported” means that the app does not support the features at all.Our work builds on top of these applications. We use these applications asa first-step to guide our understanding of read-out-loud feature, and shape ourunderstanding of how well eyes-reduced skimming is supported.2.3 Studies on SkimmingResearchers have explored two categories of source materials for skimming - textdocument and audio.2.3.1 Document SkimmingSeveral studies have explored how people skim a document. Masson [39] definedskimming as a “technique that is commonly associated with reading goals that in-volve the comprehension of only a subset of a story’s content”. The reader focuseson information relevant to the goal of skimming. Adler and Van Doren [9] defined10skimming as the first sub-level of inspectional reading. According to them, themain aim of skimming is to discover whether the material “requires a more care-ful reading.” Studies such as [40, 3] found that skim reading happens in patches,and the paragraph is the most plausible option as a patch. Therefore, readers movefrom paragraph to paragraph, sticks to reading a paragraph until rate of informa-tion gain drops below a threshold. Marshall and Bly [6] explored within-documentnavigational patterns in both physical and digital documents. They observed thatskimming, scanning, and glancing are common reading strategies among peo-ple reading longer documents, embodying lightweight navigation such as lookingahead, looking back to re-read for context, and narrowing or broadening focus toan area. Since the widespread use of mobile devices and tablets, researchers havealso explored reading behaviors in digital environments, and found evidence ofskimming as a popular reading strategy [41, 42, 4, 43]. Our study benefits fromthe empirical understanding of skimming from the aforementioned studies, con-firm their findings through empirical findings of our own, and contributes furtherunderstanding of skimming strategies using audio in eyes-reduced manner.Researchers have proposed novel interfaces to aid document skimming. Structure-Aware Touch-Based Scrolling (SATS) [44] assists the reader to perform non-linearnavigation. In SATS, the tablet screen is divided into four parts vertically, assign-ing each part to navigational units, such as chapter, section, sub-section, and page.Spotlight [45] is an attention-optimised skim reading tool which selects salientobjects (e.g., title, headings, a figure) and displays as a transparent overlay onthe screen as the user scrolls. In our work, we take advantage of gesture-basednavigation to support eyes-reduced skimming. In addition to Spotlight’s findings,we found that discourse markers are also salient objects, grounded from our firststudy in Section 3.2.3.2 Audio SkimmingSpeechSkimmer [5] is a speech-based skimming device. In SpeechSkimmer, userscan skim recorded speech at different levels by - time compression, pause short-11ening and structure identification (pauses inferring sentence boundaries, pitchchange inferring speaker change). SpeechSkimmer also uses non-speech soundsto provide navigational cues. SCAN [46] and [47] support audio skimming byusing transcripts as proxy to visually skim audio content and keyword spotting byconfidence shading respectively. Recent studies focused on non-visual skimmingfor screen reader users. Studies in [48, 49] support skimming online contents byautomatically summarizing texts in multile levels. They extended this study totouch screen devices [50], and built on top of VoiceOver’s default gestures, suchas pinch-in and out to control levels of summarization. In our work, we extendthe literature by contributing Skimmer. Skimmer uses non-linear navigation ex-ploiting semantic structure of long, structured documents. More importantly, it isdesigned to be used largely eyes-free by enabling the user to predominantly lis-ten to the text, but also supports selectively glancing at the screen, should the userchoose to do so, an interaction technique we termed as “eyes-reduced” interaction.12Figure 2.1: (Part 1) A comparison of the level of support for various skimming features in different read-out-loudapps. Our design, Skimmer at the top row, supports all features except annotation. VoiceDreamReader (partially)supports most features and hence was used in our need finding study (Chapter 3) and preliminary evaluation (Chap-ter 6). The rest of the applications are presented in an alphabetical order. Green, yellow, or red background depictswhether features relevant to supporting skimming are ‘supported,’‘partially supported,’ or ‘not supported’.13Figure 2.2: (Part 2) A comparison of the level of support for various skimming features in different read-out-loudapps.14Figure 2.3: (Part 3) A comparison of the level of support for various skimming features in different read-out-loudapps.15Chapter 3Understanding User Needs forAuditory SkimmingWe conducted a formative study to understand the needs for auditory skimming insituationally impaired context. In particular, we sought to understand how mightpeople skim auditorily, what are the challenges, what is working and not working,given the state-of-the-art read-out-loud applications (hence referred to as “NeedFinding Study” elsewhere in this thesis). We conducted the study in a controlledlab setting designed to mimic some of the conditions of a public transit bus. Weused a representative read-out-loud app as a probe. All user study materials forthis study can be found in Appendix A.3.1 Method3.1.1 ParticipantsWe recruited 20 university graduate students (10 male, 10 female) through con-venience sampling. Participants came from various backgrounds (e.g., ComputerScience, Kinesiology, Occupational Therapy, Business). We recruited only thosewho reported being familiar with reading research articles, because unfamiliarity16or inexperience might introduce bias in the results.3.1.2 TaskParticipants were asked to imagine that they are enrolled in a graduate-level courseand have reading assignments for a class later that day in which they have toparticipate in a peer discussion session. In order to contribute to the discussion,they should skim auditorily under a time constraint while riding a bus to school.We asked participants to minimize visual attention to the screen, pretending toexperience motion sickness. We also mentioned that it was acceptable to lookat the screen while issuing a command (e.g., increasing speech rate, or tapping abutton). However, we stressed that they should try to look away when listening.We told participants that their objective was to gain an understanding of - gist ofthe article, topic and problem statement, key takeaway, and structural organisationof the article. Our intention for participants was not that they give us ”the rightanswer.” Rather, we gave them an objective in order to induce them to try theirbest. Specifically, we told them that they would be asked the above-mentionedobjectives during post-task interviews.3.1.3 Design of the StudyOur study was a 2 (document formats) vs. 2 (article types) within-subjects de-sign. In total, participants completed the task 4 times, once in each condition.The order in which participants saw document formats and article types was fullycounterbalanced.3.1.4 ConditionsDocument formatting condition: Based on our empirical findings from existingread-out-loud applications, we found that most applications support either one orboth of the following layouts - plain text layout (displays only text and discards17Article # Words Reading ease level Time allocatedA1 [51] 8202 College graduate 8.5 minsA2 [52] 7260 College graduate 8 minsA3 [53] 4664 College 4.5 minsA4 [54] 4559 College 4.5 minsTable 3.1: Articles (in rows) selected for skimming task and properties (incolumns) that defined inclusion and exclusion criteria for skimming taskany formatting) and original layout (displays the original formatting condition ofthe document). We studied both of them to understand if one or both are favorableto auditory skimming.Article type condition: We chose documents that are isomorphic in terms oftopic, structure, and difficulty. For this study, we chose the following two articletypes - research paper (representative academic papers A1 and A2 in Table 3.1)and Pew report (representative professional reports A3 and A4 in Table 3.1). Asummary of the selected articles is given in Table 3.1. For reading ease level,we measured Flesch-Kincaid Reading Ease [55]. We followed the reading mate-rial selection rationale mentioned in [40]. ‘Social media’ was the common themeamong all four articles. In addition, having ‘college’ or ‘college graduate’ read-ing difficulty level ensured that the articles were suitable for university graduatestudents, regardless of their academic backgrounds.We imported PDF files of the articles in the reading app. All articles weresingle column typeset to maintain consistency. Before that, we fixed reading orderusing Adobe Acrobat Reader’s accessibility tool.3.1.5 ApparatusWe used an iPhone 5S device for this study. We explored 13 read-out-loud appli-cations in both mobile and desktop platforms (in Figure 2.1, 2.2, 2.3). We choseto use VoiceDreamReader (VDR) as a technology-probe for the study becauseof its comprehensive feature sets and positive ratings by its users in app stores.We demonstrated the following navigational features of VDR to the participants18Figure 3.1: Participant standing in front of a big screen where we projected ambi-ent bus scenario.- play/pause, scrolling, jump forward or backward by 15 seconds, double tappingto start reading from a place, and page jumps.VDR provides feature support for non-linear navigation such as sentence,paragraph, sections, however, these features did not work consistently for all 4documents. The temporal navigation worked reliably, therefore, we decided touse temporal navigation (15 seconds) only.3.1.6 ProcedureParticipants at first answered demographic questions (age, gender, discipline).Then we introduced VDR to the participants. We demonstrated the features andasked them to try them out. To put participants in a bus riding context in a con-trolled lab setting, we projected a video that captured window view from a bus19commuter’s perspective on a large screen, with high volume to emulate ambientnoise (in Figure 3.1).We determined the task time given for skimming an experimental material (seeTable 3.1) by adopting Duggan et al.’s approach [40] that calculates the ratio ofnumber-of-words in the material and the target reading speed for skimming. Inour study, we used the human listening rate as a base to determine the task times.Bragg et al. [37] reported that average listening rate for sighted people is 297words per minute (WPM). Our pilot tests showed that it was difficult for people tocomprehend when listening at 297 WPM. Setting the reading time following thisprotocol would result in very long reading time (27 minutes for A1), which couldhint to participants to listen linearly and not skim it. To solve these problems,we conducted more pilot tests at varying listening speeds, such as 180 WPM, 200WPM, 225 WPM and 250 WPM. We decided to set 225 WPM as default as pilotsagreed that it is a reasonable speed to support their listening comprehension. Also,participants were free to change the reading speed as needed. Based on pilot tests,we found that setting 8.5 minutes for A1 and 4.5 minutes for A3 was reasonable,and thus we set the time for the rest of the articles to be commensurate with A1and A3.After each reading, they answered perceived level of comprehension. Partici-pants verbally responded to our questions related to skimming objectives. Duringthe task, participants saw either original layout or plain text layout first. Aftereach layout, they answered NASA TLX [56] workload measurement for the task.3.1.7 Quantitative MeasuresWe measured the following: (1) NASA TLX - a six scale weighted measurementfor workload, and (2) perceived level of comprehension - a 5-point Likert scalemeasuring their agreement on how well they understood each skimming objectives(see Appendix A.4).203.1.8 Data Collection and Data AnalysisIn addition to quantitative measures, we collected qualitative measures such as ob-servation data, video record of the tasks, and audio record of post-task interviews.In the post-task interviews, we first asked questions about the gist and structuralorganization of the article to gauge their understanding of the article. Apart fromconventional questions pertaining to what worked well and did not work well,we also asked for more details on what, why, and how of particular instances.For analysing qualitative data, we maintained a codebook. We started line-by-line coding, trying to explain the data, looking at the actions, and their outcomes.The initial round of coding was in-vivo [57]. We kept memos defining the codes.Codes and memos were shared and discussed with the research team, and refinedaltogether. We analyzed quantitative data using repeated measures ANOVA andAligned Rank Transform [58] because of non-parametric multi-factor data in ouranalysis.3.2 FindingsThe central theme from our qualitative observation is that the pull to look at thedevice is very strong. Participants almost seemed compelled at times to lookdown when skimming. Analysis of the video footage shows that participants werevisually “scanning”, which they confirmed in the post-task interview. They indi-cated that they were visually foraging parts of text where they believed the gistof the paper’s content might be (e.g., end of Introduction, and places marked byphrases such “In this paper” or “Our contributions are” might appear.).This tendency to see the document was so strong that four participants wouldnot follow the study protocol requesting them to minimize their visual attention tothe screen. They read-along most of the time, meaning that they visually read thetext while just following the auditory narration as a secondary information chan-nel. Individual preferences seem to play into such behavior as some expressedthemselves as a “visual person” and explained that “[listening to] audio is not21learning.” (P10) We discarded their quantitative data to keep homogeneity in ouranalysis (below) but kept their qualitative data, as their responses add accounts forwhy the pull to look is so strong.Overall, the participants struggled between using two different modes of read-ing: eyes-free vs. visual reading. Skimming, as a particular type of reading be-havior, often requires frequent non-linear navigation and occasional glancing overto visual content, such as figure. However, the traditional read-out-loud apps, in-cluding VDR, are designed with the assumption of a linear consumption of audio.This challenge calls for us to design and develop a new auditory skimming appmotivated to support a new mode of text consumption: eyes-reduced reading thatflexibly blends auditory and visual modes of reading. The following findings spec-ify what are the concrete user needs and challenges for supporting skim-readingin a eyes-reduced way.Non-linear navigation is hard. Participants were in need of non-linear naviga-tional features that respect the semantic structure of the document (e.g., sentences,paragraphs) so they do not feel lost right after the jump, which is a frequent nav-igational pattern in skim reading. They did not find temporal jumps (10s, 15s,and so on) useful, because “it wasn’t clear where 15 seconds leap would take meto. Move by sentences is more meaningful.” (P8) In general, participants reportedbeing familiar with the structure of research papers and what to expect from them.“I wanted to navigate. I wanted to see the results, see the summary, to the sugges-tions. But it was not easy to navigate to them. You have to scroll and find.” (P4)Participants wanted to listen to the first one or two sentences of a paragraph andskip to next paragraph. However, they often did not skip because “double tappingthe next paragraph was not working always.” (P8) In addition, it required visualinput from them to make a selection. Navigation was more difficult with plaintext layout because “there was no structure and everything looked similar” (P16)and participants had to “scroll and scroll until at some point I became frustrated.”(P9)Listening and navigating at the same time is difficult. The traditional app keeps22narrating the text even after when the user starts the navigational interaction (e.g.,scroll to find where to jump into). Participants found this simultaneous stimuluscognitively taxing. It is known that listening and short-term memory is competingfor the shared cognitive resources [59]. Their work-around was to pause the audiobefore navigating although cumbersome.Some types of content are difficult to understand by listening. Some partici-pants preferred looking down at a figure or a table referenced in the text, becausethey were visual by nature: “It’s useless to say Figure 2 when you can’t actuallysee Figure 2.” (P7) Some of the textual content was also difficult to understand,even when the narration is reading it verbatim. Such items include complex num-bers, and acronym: “[It was] difficult to understand when the voice is telling pvalue is 0.3.” (P7) “[It was] wordy, the stat reports are about numbers, and hear-ing the voice reading the numbers was not useful.” (P13)Original layout better supports structural understanding. For understandingthe layout or structure of the article, participants rated that they understood betterwith original layout formatting (M = 3.81,SD = 0.44) than plain text layout for-matting (M = 2.38,SD = 0.35); F(1,15) = 28.31, p < .001, partial η2=.65. Inseveral instances, participants reported the benefits of original layout over plaintext layout, such as “the original layout was better. If the purpose is to skim thearticle, I would like to see the layout” (P4) and “I like the original layout more,because it was impossible to tell how the paper is laid out in the plain text view.”(P7) While doing the task, workload was significantly higher in plain text layout(M = 76.67,SD = 11.87) than using original layout (M = 63.97,SD = 15.22);t(15) =−4.42, p < .001.Dynamic speech rate control is needed but inaccessible. Participants reportedthat they preferred slow narration for important content and faster narration fortrivial content. Such selective reading pattern is similar to what is reported in theliterature on visual skim reading behaviors [3]. However, participants often couldnot increase or decrease the speech rate, even when they wanted to, because therate control was not accessible, buried deep in the menu. “But since every time23I have to touch it and decrease and increase the word speed, this deviated mefrom what I was hearing. That’s why I skipped increasing the speed later.” (P12)Moreover, setting a default speech rate was another barrier, because the unit ofspeech rate “words per minute” was not meaningful for estimating how fast (orslow) it would be. Therefore, participants listened to the audio to figure out whichspeed to set.Individual differences observed in reading order. We found reading ordersfor research papers varied among participants. This order seemed predeterminedfor a person, something they learned and refined over their experience of readingresearch papers. For example, the order of Abstract, Introduction, and Conclusionwas very popular. Some would go to the Discussion section depending on howwell they utilised their time. Only P16 and P19 said they would be interested tosee the figures first. For Pew reports, which were relatively new to many, theyeither (1) listened chronologically, (2) visually read section, sub-section titles andlistened to one or two paragraphs, or (3) listened to the first paragraph and lastparagraph of each section.Document formatting and article type impact comprehension. When it comesto understanding the topic, people perceived to have better understanding - (1)while using original layout (M = 4.16,SD = 0.13) than plain text layout (M =3.53,SD = 0.31); F(1,15) = 7.147, p < .05, partial η2=.32, and (2) when read-ing research papers (M = 4,SD = 0.35) than pew reports (M = 3.69,SD = 0.53);F(1,15) = 5.43, p < .05, partial η2=.27. For understanding the key takeaway,participants perceived to have better understanding (1) while using original layout(M = 3.31,SD = 0.80) than plain text layout (M = 2.63,SD = 0.53); F(1,15) =11.84, p< .05, partial η2=.44, and (2) when reading research papers (M = 3.44,SD=0.62) than pew reports (M = 2.5,SD = 0.35); F(1,15) = 14.27, p < .05, partialη2=.49.Create annotation for later consumption. Part of the motivation for skimmingis to decide whether the document needs detailed reading later. To this point,participants mentioned their interest to create annotations such as, highlighting,24putting starts, and bookmarking. Another motivation is to mark which content isimportant and go back to it sometime later.Skimming in other situationally impaired contexts. Participants imagined othersituationally impaired contexts where they could find auditory skimming useful.For example, P17 and P19 said they would skim while doing regular daily activi-ties, such as cooking, doing exercises. P11, a Kinesiology student said she mightskim an article auditorily during outdoor activities, as pulling up a laptop is notalways convenient. P16 and P20 mentioned that auditory skimming can be helpfulwhile waiting in a queue (e.g., boarding a plane).To summarize our findings from this study, auditory skimming required ‘eyes-reduced’ reading rather than entirely eyes-free. Participants blended visual (forcontent that is predominantly visual, such as images) and non-visual modes ofreading in order to skim using audio. Specifically, non-linear navigation was keyto successful skimming, but it is not supported well.25Chapter 4Design Guidelines for Eyes-ReducedSkimmingWe triangulated implications from multiple data sources to develop a set of robustand comprehensive design guidelines. The data sources include: (1) the resultsfrom the need finding study in Section 3, (2) existing theoretical and empiricalstudies of skim reading [3, 6, 9, 39], and (3) existing guidelines for creating voicenarrations and voice user interface design guideline [38, 60]. Here we present ourdesign guidelines:DG-1. Provide ways to navigate the structure of the article in a non-linearfashion and to localize the current position. This should be available to the useron demand, similar to using a Table of Contents. The primary support requiredhere is the ability to jump to sections and to easily know the current position inthe document.DG-2. Provide spatial and semantic navigation instead of temporal naviga-tion. Temporal navigation did not help the user to conceptualize their jump lo-cation. Scrolling as a spatial navigation is very common. Hence, it should besupported. Semantic navigations are associated with the meaning of words andunderlying structure of the article. Hence, there should be support for sentenceand paragraph level jumps. Moreover, informed by the need finding study and26discourse analysis on academic articles by Hyland and Tse [61], we recommendsupporting discourse marker jumps to facilitate skimming since they also act asimportant cues when skimming visually. The study in [61] illustrated two majortypes of discourse markers - interactive and interactional. Interactive are thosethat writer uses to manage information flow as a guide to the reader, such as ‘fi-nally’, ‘to conclude’, ‘in this paper’, ‘see Figure 1’. Interactional markers arethose that convey the author’s perspective towards the propositional information.For example, ‘it is clear that’, ‘note that’, ‘our contributions’.DG-3. Pause narration when the user is navigating. Narration should bepaused when the user is navigating, such as scrolling or dragging. This is becausewhile navigating, the user focuses on where to go next and does not necessarilyfocus on current narration.DG-4. Provide ways to adjust speech rate dynamically. The user should beable to adjust the speech rate dynamically without pausing or stopping audio.This option should be accessible to the user without having to go at sub-menulevel hierarchy. Moreover, a simple scale should represent speech rate, e.g., 1.0for normal, 2.0 for twice the normal speed, instead of ‘words per minute’.DG-5. Provide ways to refer back visually to text content from the auditorynarration and vice versa. A spatial point of reference can help the user to referback to text content at any moment while skimming. There are two ways to sup-port this. First, the spatial point of reference can remain static, and the content can‘flow’ around the static point. An example is to have a static point of referencemid-way down the smartphone screen along the left bezel, leaving the text contentscroll-able. Second, the spatial point of reference can move across static content.The user should be able to move the point of reference that will serve as the playhead of the narration.DG-6. Provide rich paralingual cues in narration for better listening experi-ence. For a better listening experience and to motivate understanding, the speechsynthesizer should support rich paralingual cues in narration. Some examples are- breakdown lists into groups of 3 or 4, narrate statistical values following verbal27reporting conventions, expand abbreviations (‘e.g.’, ‘i.e.’), insert breaks in longsentences to help user absorb information, avoid footnotes and citations in thetext. Moreover, the system should add context to help the user focus. For exam-ple, announcing which section is going to be narrated next.DG-7. Provide auditory or haptic feedback as non-visual confirmation. Fornon-visual affordances, auditory or haptic feedback should be given. For example,an auditory feedback can be given between end of a paragraph and start of a newparagraph, to nudge the user to pay attention to an important content, such as animage or a number.DG-8. Support opt-in visual engagement. Support opt-in or consensual pro-cesses when switching context (e.g., jumping from main text to figure/table cap-tion). Also, set out numerical texts from other texts when narrated to reinforcelearning when the user glances at the screen.DG-9. Support unimanual interaction for eyes-reduced skimming. Gener-ally speaking, user interface widges, such as menus and buttons, should be easilyreachable, assuming that the user will be primarily interacting with one hand.DG-10. Support individual differences in skimming strategies. There shouldbe support to preset a reading order and automatically listen through the text with-out the user explicitly navigating to parts of the text. This can happen when thedocument type/format is familiar to the reader and knows what to expect. Further-more, there should be support for choosing which part to listen to on the fly. Thiscan happen when the user is not familiar with the document.DG-11. Support annotation creation and consumption. The user should beable to create within-text telegraphic annotation [62], such as highlight, book-marks, or putting stars. User should be able to navigate to these annotations. Anadvanced support would be to create speech annotations. At the same time, thereshould be support to consume these annotations.In summary, we proposed a set of design guidelines for eyes-reduced skim-ming for situational impairments. We formulated these guidelines following ourneed finding study and our empirical and theoretical understandings of skimming,28voice narration, and voice user interface design. Our hope is that other designerswill also benefit from, and iterate and refine these guidelines.29Chapter 5Design of SkimmerWe built a novel read-out-loud app called Skimmer that embodies the majority ofthe design guidelines (Chapter 4) for eyes-reduced skimming. Skimmer featuresnavigational support for auditory and visual modes of reading, non-linear navi-gation, and auditory and haptic cues. In this chapter, we explain the design ofSkimmer first, and then we explain the iterative design process.5.1 Eyes-Reduced Skimming Features of SkimmerHere, we elaborate on how Skimmer supports eyes-reduced skimming for situa-tional impairment and our rationale behind its design.5.1.1 Basic Navigation for Eyes-Reduced SkimmingSkimmer predominantly supports eyes-free navigation. The user can skip sen-tences or re-listen to a missed passage with a lightweight, eyes-free tap gestureon the right or left side of screen respectively (Figure 1.1 (b)). Skimmer alsosupports sentence-level navigation using vision by traditional scrolling (DG-2).When skimming, this can be useful for localizing one’s reading position–peekingat how far one is in the text and checking the remaining length of passages. Skim-30mer also pauses the audio as soon as the user touches the screen for navigationalaction (DG-3). Paragraph navigation is done through up-or-down vertical flickinggestures. This is because scrolling is so intuitive to people and the content is flow-ing vertically, so Skimmer aims to promote positive transfer from visual scrollingto eyes-free gestural interactions.5.1.2 Designing for Easy to Understand Audio NarrationWe designed the way Skimmer narrates the written content to help listeners betterunderstand complex and difficult texts: (1) Written text and structural informationare narrated by two different voices to help distinguish meta-information fromverbatim content. This is done by using a low-pitch voice for narrating written textand a higher-pitch voice for narrating meta-information. The meta-informationis for providing context to the reader, such as “Section 1. Introduction”, where“Section” is the meta-information. (2) Skimmer breaks down long lists of itemsor bullets in groups of 3 or 4 items at a time, to help memory retention (DG-6). (3)Skimmer reads numbers in a rounded format (such as p < 2.2×10−16 narrated asp< .001), because complex numbers are hard to understand just by listening (DG-6). Skimmer also provides an overlay to reinforce learning as the user visuallyengages with the numbers (Figure 1.2 (a)). (4) To catch the user’s attention to adiscourse marker, Skimmer narrates it louder and slower. For example, for thegiven text “In this paper, we study Snapchat ...,” Skimmer can identify “In thispaper,” as a discourse marker, and add an tag of Speech SynthesisMarkup Language (SSML) around it so our TTS engine can emphasize the phrasewith a salient pitch and tone. For this, audio narrations in Skimmer were generatedusing a Google Wavenet-based TTS engine that supports SSML tags.5.1.3 Navigating Between Discourse MarkersTo help users find and navigate to important nuggets of information, Skimmersupports selective jumps between various discourse markers (DG-2) through eyes-31(a) (b)Skimmer is reading a sentence “Such concerns…’’12The user taps the ‘jump forward’ buttonSkimmer jumps to the adjacent discourse marker “In this paper’’3Figure 5.1: Skimmer’s discourse marker navigation. (a) Current reading positionof Skimmer. The user taps on the bottom right button. (b) Skimmer jumps to thenext discourse marker. In this case, it is “In this paper”.32free tap gesture (lower end of screen, Figure 5.1 (a)). When the user taps thediscourse marker navigation button on the bottom right (left), Skimmer moves theplay-head to the next (previous) adjacent discourse marker, narrates the discoursemarker first (e.g., “In this paper”), and then narrates the full sentence from thebeginning of the sentence for context (Figure 5.1 (b)). Both discourse markersand full sentences are content that is narrated by the same voice (low-pitch voice).To reinforce to the users that they have reached a discourse marker narration,Skimmer plays a subtle ambient earcon (a cricket sound) as background audio.5.1.4 Understanding the High-level OrganizationTo support structural glancing and navigation through high-level structure of thedocument (DG-1), Skimmer features a table-of-content style Overview page dif-ferent from the main document view (Figure 1.1 (c)). All content in Overviewis accessible eyes-free, as the allowed gestures between the main document viewand Overview view are consistent.5.1.5 Opt-in to See Visual ContentSkimmer’s design for eyes-reduced reading nudges users when they encountervisual content. When a narration content includes a figure or table referred to inthe text, Skimmer gently informs the user with haptic feedback so that they canlook at the visual content (DG-7, DG-8). Skimmer facilitates navigating backand forth between the main text and the visual content on-demand; note that wedesigned the transition to visual peeking to be opt-in; allowing the user to chooseto continue focusing on listening to the text or to jump to the visual content (seeFigure 1.2 (b)). The haptic vibration pattern was carefully chosen. From Seifi etal.’s [63] library of vibration patterns, we chose the five patterns that have at leastone of two relevant tags: “attention-catching” that catches the user’s attentionwhen there is a figure/table reference and “get ready” that urges the user to beready to see the figure/table. Among the five, we picked the one with the highest33pleasantness score.5.1.6 Auditory Cues in Support of SkimmingDrawing inspiration from SeeReader [25], throughout all these features, Skimmerleverages a limited set of distinctive earcons to provide subtle auditory cue/feedback(DG-7). It’s worth noting that we consciously limited the number of total distinc-tive earcons to be under 6, as we found that incorporating too many earcons cancause overlaps of aural characteristic between them, which in turn confuses users.Hence, we used earcons, which is a scarce design resource due to the limited num-ber we can leverage. The purpose of using earcons is to signify the system statuschanges at the critical path of skimming activities, such as switching betweensentences, paragraphs, and discourse markers, and navigation to Overview page–thus giving users clear feedback that Skimmer has processed their navigationalgestures.5.1.7 Accommodating Individual Differences in Listening RatesSkimmer accommodates individual differences by enabling the user to dynami-cally adjust the speech rate through an eyes-free tap gesture (DG-4). Skimmerprovides two buttons at the top, similar to the discourse marker buttons at the bot-tom in style and size (Figure 1.1 (b)). Although the top of the screen is often hardto reach, we made a design trade-off by prioritising discourse marker navigation,an oft-used navigational feature, over the rate changes.In summary, Skimmer embodies our proposed design guidelines to support eyes-reduced skimming. Skimmer implemented the majority of the design guidelines(DG-1 to DG-9). In Chapter 7, we discuss the rest of the design guidelines andour future work.34(a) (b) (c) (d) (e)Figure 5.2: (a) Initial screen that lets the user choose a document for skimming. (b) The next screen showsthumbnails of every page with section names appearing as overlays. (c) Default layout showing the double columnformat - the original layout of the document. (d) When the user engages with the text by looking at it, it reflowsthe text by zooming in on the current text segment. (e) When a figure/table is referenced, the application shows theimage in landscape orientation to provide a better view.355.2 Design ProcessWe designed and developed Skimmer through a three-phase iterative design pro-cess. The first phase was to outline conceptual designs. We ideated several distinctconceptual models that abide by the guidelines, and then created and evaluatedlow-fidelity prototypes. In the second phase, we refined our low-fidelity proto-type to simulate eyes-reduced skimming using a wizard-of-oz technique. Feed-back from this simulation motivated us to step into the next phase, where we builta usable and working mobile app to evaluate our design ideas and design guide-lines.5.2.1 Conceptual DesignThe first step of our design process was to create conceptual designs of a basicspatial layout and gesture interactions.Designing the spatial layoutDesign ideas for the spatial representation are grounded to findings from Chapter3 where we learned that original layout is preferable over plain text layout. There-fore, to flesh out our conceptual ideas into tangible design objects, we createda low-fidelity prototype using styrofoam and paper. In our first design iteration,we considered the original layout of the source document (e.g., double columnformat), as shown in Figure 5.2 (c). We borrowed the concept of space fillingthumbnails [64] to provide a high-level structural idea of the reading material tothe user (in Figure 5.2 (b)). When the user chooses a section to read, the app dis-plays the page containing that section (Figure 5.2 c). Since our design motivationwas eyes-reduced interaction, having a double column layout on the screen withsmall fonts was a design trade-off that we thought would also discourage the userfrom focusing on the screen for a long time. However, when the app detects thatthe user is focusing on the screen, the app reflows the text by joining both columnschronologically and zoom-in to the text segment currently narrated on the screen,36as shown in Figure 5.2 (d). Moreover, since documents such as research paperscome with content that require visual attention, such as figures or tables, our de-sign lets the user to visually engage with the figure or table once they are referredin the main text by displaying in landscape orientation on the screen (in Figure5.2 (e)).Mapping features to gesture setsOur target at the outset was to optimize the touch gesture vocabulary in such away that users can positively transfer easy and well-known interaction techniquesto Skimmer. For these reasons, we decided to use standard gestures [65] and mapthose gestures to our features (in Figure 5.3). Standard iOS gestures include: tap,drag, flick, swipe, double tap, pinch, touch and hold, shake, and rotate. We madesure that our design is consistent with existing design practices and mental modelsthat users are habituated with.37Figure 5.3: Mapping standard gestures to features.385.2.2 Design and Prototyping Method RefinementsWe iterated over our initial design and gathered feedback from pilot users. Basedon the feedback received, we refined our design and simulated eyes-reduced skim-ming using a PowerPoint slide show.Observations from our initial round of design indicated that users’ experiencewith our prototyping method was too slow. This happened because in our firstprototype (Figure 5.2), the interface stages were managed using paper screens.Pilot users would take off the current paper screen and put on next screen from adeck of paper screens. Another problem was that the users would mistakenly putthe current screen on top of the next paper screen deck. This created confusionand users would lose valuable context while using the paper prototype. To solvethis problem, we chained paper screens one after another in the order of theirappearance (Figure 5.4 (a)). Users would pull out from the top to bring the nextpaper screen to view.39(a) (b)Audio clipInstruction for the pilot user(c)Figure 5.4: (a) Refined paper prototype - paper screens chained chronologically in the order of appearance. (b)PowerPoint slide with embedded audio files and instructions for both pilot user (in red ink) and the facilitator (inblack ink). (c) Three-way setup for two pilot users and one facilitator in a session.40Simulating voice narrationTo simulate voice narration in auditory skimming while users interacted with thepaper prototype, we embedded audio clips in a PowerPoint slide show (Figure 5.4(b)). The audio clips were extracted from different parts of a selected researcharticle, such as the first few sentences of a paragraph, section or sub-section titles,sentences that referred to a figure or table, and captions of a figure or a table. Wealso outlined instructions on how to simulate voice narration in the slides.To experience skimming using our paper prototype and simulated voice nar-ration, we used a wizard-of-oz technique. Two members of our research teamacted as pilot users and the thesis author acted as the facilitator. During this three-way session (in Figure 5.4 (c)), the facilitator would walk through the PowerPointslides. Instructions in black ink were for the facilitator to read-out-loud to the pilotusers. Instructions in red ink were for the pilot users, mostly asking to change thepaper screens. Audio clips were embedded with these instructions, such that theywould play automatically after some time.Challenges of the low-fidelity prototypeOur main pain point while prototyping was synchronizing the audio with the phys-ical navigation and spatial layout. Consider the last instruction from Figure 5.4(a). The user had to flick from right to left and slide the paper screens to expe-rience the visual change in output, while listening to the audio at the same time.Listening to the facilitator, while also maintaining an awareness of listening to theaudio, seemed overwhelming to the pilot users. Instruction sequencing was alsochallenging for the facilitator too. Sometimes, audio clips would start and over-lap with the facilitator’s instruction, which created more confusion. To properlyevaluate whether users can listen to a given text in practice, we had to build aninteractive medium-fidelity prototype.415.2.3 Building a Usable PrototypeWe moved onto implementing an interactive medium-fidelity system that dynam-ically renders narrations and visual interfaces in response to user interactions on-the-fly. Figures 1.1 and 1.2 show snapshots of Skimmer.Skimmer was implemented as a Javascript-based web app running on a 6.4-inch mobile phone. We built Skimmer to support HTML-based e-publication for-mat as a source document type (e.g., an HTML version of a paper in the latestACM CHI Proceedings). A couple of reasons encouraged us to move forwardwith HTML-based source documents. First, from our need finding study in Chap-ter 3, we learned that although participants preferred original layout over plain textlayout, there are some benefits of plain text layout that can positively influenceeyes-reduced skimming. An HTML-based document is browse-able in a singlepage by scrolling, rather than paginated. Second, as the e-pub scene in the re-search community is moving from static PDF to such flexible document types forenhancing accessibility and device-compatibility, we consider using the HTMLversion of a document to be a future-facing decision.Skimmer’s development consisted of the following high-level modules:• Audio module. We used HTML5’s audio library to build a simple audiomodule. HTML5 Audio defines some DOM methods, properties, and eventsthat we used to in our audio module. For example, we used the basic play()and pause() to play and stop the audio. We used playbackRate property toset speech rate. The two most useful events are ontimeupdate and onended,because they played a crucial role to refresh and update the display moduleand signal the end of an audio clip respectively.• Data module. For each text unit, Skimmer has an audio file and a JSON filetimestamped at the word-level. We define text units as the section titles, aparagraph, or caption of a table or figure. We generated audio files by pre-processing text data manually. One example of text pre-processing is givenin Chapter 5.1.2. We passed audio files to a speech recognition engine,42which generated JSON files with word-level timestamps.• Display module. In the display module, we highlighted the sentence andthe word currently narrated. The display module is fired at each ontimeup-date. The word at that time instance is highlighted, including the sentencecontaining that word.• Touch gesture module. Our decision to use HTML-based e-publication for-mat implied that the application would eventually run on a web browser.The default web browser scrolling provides a rich scroll-based user experi-ence. We wanted to have the same experience in our application. However,there are also some other default features that we did not want in our appli-cation. For example, double tap and press and hold would highlight text forselection in a web browser. Double tap may also cause zoom-in and out insome cases. We wanted to customize the result of gesture interaction, forexample, tap on either side of the screen to navigate sentences, or flick leftto right to change view. Therefore, our needs were special in that simplycalling preventdefault() function to prevent the browser’s default behaviorand customize them based on our needs was not a straight-forward solu-tion. We tried and tested several Javascript-based gesture libraries - suchas ZingTouch, Hammer.JS, interact.js, and DeepTissue.js. After much de-liberation and testing, we used ZingTouch to handle drag, swipe, and scrollbased gestures, and Hammer.JS to handle tap based gestures, such as sin-gle tap, double tap, and press and hold. We included both of them becausecalling preventdefault() would break Hammer.JS and ZingTouch would notstop bubbling tap events.In summary, we designed a usable prototype called Skimmer through multiplerounds of designs, implementations, and pilot evaluations.43Chapter 6Preliminary Evaluation of SkimmerWe conducted a preliminary evaluation of Skimmer. It was a structured observa-tion on a public bus where participants experienced both Skimmer and VDR (alsoused in our need finding study in Chapter 3). We used VDR as a reference pointto ground participants’ audio skimming experience, not with an expectation for itto be evaluated “head to head” (as VDR was not designed for auditory skimming).Our primary evaluation goal was to assess qualitatively how the participants ex-perienced the unique features in Skimmer. All user study materials for this studycan be found in Appendix B.6.1 MethodSix graduate students (3 male, 3 female) from various disciplines took part (nonefrom our need finding study in Chapter 3.). For the task, we asked participantsto skim two documents in “eyes-reduced” manner using audio, one documentwith each application. The motivation, context, and protocol were similar to theprevious study. As reading materials, we chose two articles from CHI 2019, bothrelated to social media (hereafter referred to as A1 [66] and A2 [67]). Both articlesare isomorphic in style, structure, and difficulty (on the Flesch-Kinkaid ReadingEase scale [55]). Each participant experienced both articles; orders of both articles44Researcher ResearcherParticipant ParticipantResearcherParticipantFigure 6.1: Field study to evaluate Skimmer. The study was conducted in publicbuses. A researcher accompanied a participant and conducted the study in a bus.and apps was counterbalanced. Times allotted for skimming A1 and A2 was 9 and8 minutes respective, due to differences in length (10,275 vs. 7,745 words). Wefollowed the process for determining time to skim in Chapter 3. The procedurewas similar to the previous study, except run on the bus. For each app, we firstdemonstrated it to the participants and let them play with it to become comfort-able. The thesis author accompanied the participants in the bus, observed themcompleting their tasks and took notes (in Figure 6.1). Both Skimmer and VDRwere deployed in a Samsung A50 Android device. We used a two-way audio split-ter so that both the participant and the author could listen to the audio output fromthe app individually (in Figure 6.1). After each app+article, the participant andthe author got off the bus. The participant answered a small number of questionsand then sat for an interview (audio-recorded).6.2 FindingsAll 6 participants completed the tasks with both apps, in that they skim-read thearticles. Three participants offered without prompting that they typically experi-ence motion sickness.Skimmer can be used eyes-reduced, VDR requires near constant visual atten-tion. It was very clear in our observation that participants were able to largely45not look at the screen using Skimmer. There were intermittent episodes whereparticipants glanced at the screen: (1) in the Overview page to section jump, and(2) in the Article page to see interesting keywords (e.g., “Trump”), reacting to(unexpected) earcons (e.g., appearance of new section after next paragraph ges-ture), and responding to the haptic nudge when a figure or table was referred inthe text. P2, P4, and P5 looked at the screen in the Results section, skipping byparagraphs, often stopping for a brief moment “to find interesting results quickly”.The comparison to VDR was black and white: participants spent the majority oftheir time scrolling “all the way down to find Conclusion. I don’t have any otherway of doing that other than looking at the screen. But it was really easy with thisapplication (Skimmer).” (P2)Users appreciate Overview the most. Overview was called out as the mostuseful feature in Skimmer: “key difference between the two apps.” Participantsfrequently navigated to the Overview page by swiping to the right (except P1 tap-ping Overview at the top one time), scrolling the list of sections and sub-sectionsand looking at the screen intermittently to find a section and then swiping to theleft, eyes-free to complete a jump to that section. At one moment, P3 and P4were tapping instead of swiping to the left because they “forgot how to go back”,which also prompted them to look at the screen. In contrast, such section jumpsprompted visual attention, heavy and frequent scrolling in VDR.Auditory feedback and haptic affordance help to re-focus. While the intendedpurpose of the auditory and haptic cues was to subtly reinforce an interaction (e.g.,next sentence) or signal an affordance (e.g., image available to see), they alsohelped participants “to situate themselves and come back” when they lost focus.P5 said, “I would say, the earcon for Section was really helpful. I was looking out-side and maybe lost focus, that earcon helped me situate myself again and comeback.” However, some participants did not notice the earcons at all. This couldmean that our choice of earcons in our design was successful, in that participantsdid not feel that these earcons were rather annoying them or contributed to losingfocus from listening in any way.46Individual differences in navigation preferences are supported. P2 jumpedfrom Conclusion to the last sentence of Abstract by using a combination of sec-tion, paragraph, and sentence jumps as “shortcuts”. P1, P3, and P6 used sentencejumps frequently because “it was faster”, while P4 and P5 used drag for sentencejumps frequently and flick for paragraph jumps occasionally because “scrollingfelt natural”.Skimming numbers remains challenging for those who tried it. With the excep-tion of P5 and P6, other participants did not consider numbers important in theirskimming. P5 looked down upon numbers in the Results section and visually readwith both VDR and Skimmer because “it was difficult to follow numbers, and de-feated the purpose of audio skimming.” Both P2 and P5 resonated that “numbersshould come with context, for example, 1500 participants” instead of just “1500”.Users appreciate narration quality and disparate voices. Participants wereimpressed by the quality of narration in Skimmer, mentioning that it was “softand comfortable to listen to, unlike the stiff voice in the other (VDR)”. In contrast,P5 mentioned that she liked VDR’s narration because it was “natural, researchpapers are dry, you don’t need emotion here, a robotic voice is also okay.” Skim-mer’s use of disparate voices was “unique” but little noticed.Discourse markers are useful, but need more exposure. Participants describedthe discourse marker jump as a “great tool” and “helpful”. Only P3 and P5 used itin the actual study. Regardless, all of them experienced it during training too, andmentioned that they “needed more trust.” P1 said, “I may need to experience it justby myself alone before I know what really is in there”. In contrast, we observedsimilar patterns of “zoom-in, spot discourse markers” behavior from need findingstudy while using VDR.Figures/Tables are mostly ignored, but appreciated the idea of a haptic nudge.Participants thought that haptic nudge was a “a great way of reminding me thatsomething happening here.” (P3) Only P1 and P3 used this opt-in feature to viewa figure and a table respectively, while the rest of the participants ignored, citing“figures/tables not important in skimming”. P3 said, “I don’t do audio reading47because I am not good at keeping attention. But such vibrations are really useful.”In contrast, while using VDR, P1 and P3 momentarily stopped at a figure/tablewhile scrolling visually, zoomed-in and out, and then moved on.Participants appreciated Skimmer’s design concept. For example, we heard -“lots of reading for a PhD student, not a terrible idea to get gist for some... superhelpful in skimming qualitative papers”(P5), “Pocket for commuting”(P6), andeven the suggestion that it could be used for parallel visual and auditory reading“read-along for listening and following words”(P2). P6’s experience highlightsone of our key research motivations: she felt nauseated while using VDR and hadto stop momentarily, and by contrast she did not report feeling nauseated whileusing Skimmer and was able to complete the task eyes-reduced.Participants performed comparably on the reading comprehension test. Thestatistic show no significant differences on comprehension. This is not surprisinggiven that with VDR they predominantly used their uses to skim, whereas withSkimmer they predominantly listened.48Chapter 7DiscussionWe reflect on our key findings and their implications for design and our designprocess.We have made concrete progress towards eyes-reduced skimming. Our goalfor this research was to design an interface that would allow people to skim doc-uments in an eyes-reduced way. Given the complexity of skimming as a form ofreading [40, 3] that frequently mixes between non-linear reading, and glancing,supporting it via the audio channel is a non-trivial design objective. Especiallyfor an on-the-go setting, where distractions and situational impairment factors areprevalent, it poses more challenges. In our preliminary evaluation on the bus, wecould only ask participants to try not to use their eyes, but we could not force themto not look. We were pleasantly surprised at the extent to which participants did infact not use their eyes while using Skimmer, which was not possible with the tra-ditional read-aloud application. Further, our participants were positive about theoverall design concept of Skimmer, with 5 of them indicating that they would useit to skim documents. This suggests that we have indeed made positive progresstowards our design goal, that the Skimmer design is on the right track.Skimmer is a design snapshot of an artefact in its evolution. Even though re-sults from our preliminary evaluation were promising, there are opportunities toimprove usability and address the gaps in our design for supporting eyes-reduced49interaction. For example, TTS needs to go beyond simply narrating the giventext with verbatim speech. There is room for improvement not only in what theysay (content) but also how they say it (paralingual cues, including voice modula-tion, varying pitch, tone, and varying speech rate) for better comprehension. Weonly explored some of those dimensions in our design, such as numbers and lists.Further, new elements such as a means to navigate to and from metadata suchas headers, footers, and references, will need to be integrated such that they donot make the interaction overly complex. More substantive design steps, towardsfully eyes-free design, are described further below.Eyes-reduced design is a new concept. Our work has introduced the conceptof eyes-reduced design – it is intended for designs that can be used primarily eyes-free, but, importantly, do not preclude the user from using their eyes, when theywant. This can be differentiated from design that is created with the intention ofbeing used with one’s eyes, but also happens to support some degree of eyes-freeinteraction. For example, a person can use YouTube on a mobile device while onthe go. They can select a video while looking and then, to some extent, look awaywhile walking, toggling the pause/play to listen to the video by tapping on thescreen. This is challenging because the region that accepts the tap to pause/playis small, specified by a visual icon. Thus, while it can be used-eyes free for someinteractions, it is difficult to do so, and was not seemingly designed with thatintention.Fully eyes-free skimming is next step. While Skimmer was designed with theintention that it be usable largely eyes-free, we have not yet tested its eyes-freeefficacy. Fully eyes-free skimming is the next main target of our research. Ourgoal from the outset of this research was that Skimmer support both the needsof people experiencing situational impairments as well as those who have visualimpairments. We made the choice to start our design process with targeting situ-ational impairments, as we reckoned it may be the more difficult design problem– people in situational impairments still have an option to use their eyes (henceeyes-reduced) when they deem it necessary. So the design would need to support50fluidly moving back and forth between primarily using one’s eyes and one’s audi-tion. Now that we have made progress on the eyes-reduced front, we will turn ourattention to assessing the extent to which Skimmer can be used entirely eyes-free.Some extensions are immediately obvious, such as reading out figure descriptions(alt text), but then our work will continue with individuals with visual impair-ments, including a systematic needs analysis. Evaluation will undoubtedly revealthat some of Skimmer’s design elements need adjustment to support that popula-tion. Upon iterating on the design, we will loop back to evaluating with peoplein situational impairments so as to ensure that our design is maximally supportingboth eyes-reduced and eyes-free interaction.Preset reading order and eyes-reduced annotation capability are also nextsteps. We chose not to implement DG-10 and DG-11 in our first run of imple-mentation. This is because we wanted to evaluate our design first, before im-plementing features that bring added values to the very primary and basic needs.Some participants from our need finding study in Chapter 3 mentioned the needfor a preset reading order and annotation capability while skimming.Preset reading order means automatically reading through an order set by theuser, such as Abstract, Introduction, Conclusion, and Discussion. This is an ad-vanced feature. Users often have a preconceived idea of what to read and in whichorder to read when the document is familiar [43]. Our need finding study in Chap-ter 3 also showed evidence of this behavior among our participants. There are twoways this feature can be implemented. First, the app can learn user’s readingbehaviors and patterns and predict the future reading pattern for a similar classof document. This is similar to recommendation systems (ads, search engines)that learn from our past history. Second, the app can have a settings panel whereusers can select the relative ordering of chapters or sections. This is similar tocreating a playlist and listening in the order the user has selected. We anticipatesome challenges while implementing this feature. Preset reading order may notbe applicable to some documents, such as online news articles, which do not havea systematic structure. Structural outline varies among scientific documents too,51which requires further consideration (e.g., ACM CHI papers and Nature papershave different structural outline).One of the fundamental goals of skimming is assessing whether the readingmaterial warrants further attention from the reader [9]. This brings annotation ca-pability in skimming context, because people often highlight or comment (puttingstars or squiggle) texts within a document to indicate content that is important,or content that they need to revisit later [62]. Such kinds of annotation are calledwithin-text telegraphic annotation. In our study, we only evaluated a single passskimming episode to evaluate our design. Hence, we did not design for DG-11.In our future work, we will explore ways and means of eyes-reduced annotationcreation and consumption. Design ideas can include eyes-free annotation cre-ation, such as press and hold while narration is ongoing, indicating to the systemto highlight the underlying text. Another possibility can be speech recorded an-notation. This can be anchored to the text currently narrated, without requiringthe user to look at the screen. For consumption, there should be a way to nav-igate annotations. Further, one possibility is also to have a cloud-based system,where the user will be able to skim and annotate while on-the-go, for example,and then perhaps if she later decides to revisit the content again when she is backto her desk, she can access the same annotated content when making a detailedpass on it. We anticipate some challenges here too. For example, speech recordedannotation may be uncomfortable and undesirable in public places. Further, it isan interesting problem how to inform the user about an annotation in the currentcontent. Should the system vibrate or make an auditory feedback? This may workif annotations are sparse and limited in number. How should the system let theuser know if the current sentence is highlighted or underlined?Different environments/contexts can pose different kinds of situational impair-ment. We only explored a context where people could not fully use their vi-sual attention because of potential motion sickness. However, different environ-ments/contexts result in situational impairments that vary in degree. For example,one riding a commuting bus can occasionally glance their mobile, but it will be52nearly impossible for one driving a vehicle to use their eyes even momentarily.Also, a specific impairment context may pose challenges in using interactionmodalities other than eyes. Zeleznik et al., documented the ‘sandwich’ problemwhen a user is holding an object in one hand, which impairs one’s manual in-teraction capability [68]. We wonder about other types of situational impairmentthat variably affect different modalities. For example, what about when peoplecan see, but cannot touch the device (e.g., reading recipes when your hands areoily or wet)? Interfaces that address specific kinds of situational impairments is apromising direction. Beyond eyes-reduced. What about touch-reduced?Our design guidelines can inspire future design directions. We derived ourdesign guidelines in Chapter 4 from multiple research fronts. Our intention wasto explore relevant research areas and put up a robust design guideline whichcan motivate designers to come up with new design ideas, perhaps better thanSkimmer.We can think of some design directions which can positively influence userexperience of Skimmer. First, similar to how Ahmed et al., [48, 49, 50] lever-age semantic summary of text using Natural Language Processing (NLP) tech-niques, Skimmer can also leverage semantic summary of text and provide a sum-mary snapshot of the text besides skimming. Second, our eyes-reduced designcurrently acknowledges visual attention only in cases where a figure or table isreferred. What if the user wants to read-along? Using the front camera and eye-tracking technique, the system can enlarge the text where user’s eyes are focusedand facilitate read-along when happening.53Chapter 8LimitationsThere are limitations to our work. We focused on structured, academic/professionaldocuments. It remains to be seen how well Skimmer will work with documentsthat have a lot of visuals or ones with many links that require cross-documentnavigation, thus beyond non-linear. It is possible that a different design would beneeded altogether. Further, our evaluation of Skimmer only involved 6 graduatestudents, with a single exposure. A larger sample from a more varied population,together with a longer exposure time to Skimmer, will be needed to more deeplyassess the design elements.54Chapter 9ConclusionIn this work we tackled the design challenge of supporting eyes-reduced skim-ming of structured documents. Based on our need finding study, we generateddesign guidelines for eyes-reduced skimming, iterated extensively on the designbased on the guidelines, and then implemented Skimmer, a prototype app. Througha preliminary evaluation, we learned that people could use Skimmer eyes-reducedwhile riding the bus, only looking down at the app infrequently, as they navigateda CHI paper largely eyes-free, listening to the content.We are conducting our work within the framework of inclusive design [69],with planned next steps to involve people with visual impairments so that we canpush more strongly on eyes-free interaction. This will lead us one step closer toachieving a system that people who are experiencing impairments (either situa-tionally or more permanently) can use to skim documents.55Bibliography[1] R. Sandler, “How much data do we create every day? themind-blowing stats everyone should read,” June 2018, re-trieved September 19, 2019 from https://www.businessinsider.com/mozilla-scout-voice-controlled-browser-2018-6.[2] R. I. L. Inc., “Pocket,” 2019, retrieved September 19, 2019 from https://getpocket.com/.[3] G. B. Duggan and S. J. Payne, “Skim reading by satisficing: Evidencefrom eye tracking,” in Proceedings of the SIGCHI Conference onHuman Factors in Computing Systems, ser. CHI ’11. New York,NY, USA: ACM, 2011, pp. 1141–1150. [Online]. Available: http://doi.acm.org/10.1145/1978942.1979114[4] A. Thayer, C. P. Lee, L. H. Hwang, H. Sales, P. Sen, and N. Dalal,“The imposition and superimposition of digital reading technology:The academic potential of e-readers,” in Proceedings of the SIGCHIConference on Human Factors in Computing Systems, ser. CHI ’11.New York, NY, USA: ACM, 2011, pp. 2917–2926. [Online]. Available:http://doi.acm.org/10.1145/1978942.1979375[5] B. Arons, “Speechskimmer: Interactively skimming recorded speech,” inProceedings of the 6th Annual ACM Symposium on User Interface Software56and Technology, ser. UIST ’93. New York, NY, USA: ACM, 1993, pp.187–196. [Online]. Available: http://doi.acm.org/10.1145/168642.168661[6] S. Bly and C. C. Marshall, “Turning the page on navigation,” in Proceedingsof the 5th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL’05).IEEE, 2005, pp. 225–234.[7] M. Wolf, “Skim reading is the new normal. the effect on so-ciety is profound,” August 2018, retrieved September 18, 2019from https://www.theguardian.com/commentisfree/2018/aug/25/skim-reading-new-normal-maryanne-wolf.[8] B. Merr, “How much data do we create every day? the mind-blowing statseveryone should read,” May 2018, retrieved September 18, 2019 from https://bit.ly/2krBBrv.[9] M. J. Adler and C. Van Doren, How to read a book: The classic guide tointelligent reading. Simon and Schuster, 2014.[10] J. M. Hailpern and B. A. Huberman, “Odin: Contextual documentopinions on the go,” in Proceedings of the SIGCHI Conferenceon Human Factors in Computing Systems, ser. CHI ’14. NewYork, NY, USA: ACM, 2014, pp. 1525–1534. [Online]. Available:http://doi.acm.org/10.1145/2556288.2556959[11] B. Yi, X. Cao, M. Fjeld, and S. Zhao, “Exploring user motivations foreyes-free interaction on mobile devices,” in Proceedings of the SIGCHIConference on Human Factors in Computing Systems, ser. CHI ’12.New York, NY, USA: ACM, 2012, pp. 2789–2792. [Online]. Available:http://doi.acm.org/10.1145/2207676.2208678[12] C. Ingraham, “The astonishing human potential wasted on com-mutes,” February 2016, retrieved September 18, 2019 from57https://www.washingtonpost.com/news/wonk/wp/2016/02/25/how-much-of-your-life-youre-wasting-on-your-commute/?noredirect=on.[13] A. Sears, M. Lin, J. Jacko, and Y. Xiao, “When computers fade: Pervasivecomputing and situationally-induced impairments and disabilities,” in HCIinternational, vol. 2, no. 03, 2003, pp. 1298–1302.[14] J. O. Wobbrock, “Situationally-induced impairments and disabilities,” inWeb Accessibility. Springer, 2019, pp. 59–92.[15] Z. Sarsenbayeva, N. van Berkel, C. Luo, V. Kostakos, and J. Goncalves,“Challenges of situational impairments during interaction with mobiledevices,” in Proceedings of the 29th Australian Conference on Computer-Human Interaction, ser. OZCHI ’17. New York, NY, USA: ACM, 2017, pp.477–481. [Online]. Available: http://doi.acm.org/10.1145/3152771.3156161[16] M. Lin, R. Goldman, K. J. Price, A. Sears, and J. Jacko, “How do peopletap when walking? an empirical investigation of nomadic data entry,” In-ternational journal of human-computer studies, vol. 65, no. 9, pp. 759–769,2007.[17] A. Ng, S. A. Brewster, and J. H. Williamson, “Investigating the effects ofencumbrance on one-and two-handed interactions with mobile devices,” inProceedings of the SIGCHI Conference on Human Factors in ComputingSystems. ACM, 2014, pp. 1981–1990.[18] M. E. Mott and J. O. Wobbrock, “Cluster touch: Improving touch accuracyon smartphones for people with motor and situational impairments,” inProceedings of the 2019 CHI Conference on Human Factors in ComputingSystems, ser. CHI ’19. New York, NY, USA: ACM, 2019, pp. 27:1–27:14.[Online]. Available: http://doi.acm.org/10.1145/3290605.3300257[19] S. K. Kane, J. O. Wobbrock, and I. E. Smith, “Getting off the treadmill:Evaluating walking user interfaces for mobile devices in public spaces,”58in Proceedings of the 10th International Conference on Human ComputerInteraction with Mobile Devices and Services, ser. MobileHCI ’08.New York, NY, USA: ACM, 2008, pp. 109–118. [Online]. Available:http://doi.acm.org/10.1145/1409240.1409253[20] J. D. Hincapie´-Ramos and P. Irani, “Crashalert: Enhancing peripheralalertness for eyes-busy mobile interaction while walking,” in Proceedingsof the SIGCHI Conference on Human Factors in Computing Systems, ser.CHI ’13. New York, NY, USA: ACM, 2013, pp. 3385–3388. [Online].Available: http://doi.acm.org/10.1145/2470654.2466463[21] A. Bragdon, E. Nelson, Y. Li, and K. Hinckley, “Experimental analysis oftouch-screen gesture designs in mobile environments,” in Proceedings ofthe SIGCHI Conference on Human Factors in Computing Systems, ser. CHI’11. New York, NY, USA: ACM, 2011, pp. 403–412. [Online]. Available:http://doi.acm.org/10.1145/1978942.1979000[22] D. Ghosh, P. S. Foong, S. Zhao, D. Chen, and M. Fjeld, “Editalk:Towards designing eyes-free interactions for mobile word processing,” inProceedings of the 2018 CHI Conference on Human Factors in ComputingSystems, ser. CHI ’18. New York, NY, USA: ACM, 2018, pp. 403:1–403:10. [Online]. Available: http://doi.acm.org/10.1145/3173574.3173977[23] B. Schildbach and E. Rukzio, “Investigating selection and readingperformance on a mobile phone while walking,” in Proceedings of the 12thInternational Conference on Human Computer Interaction with MobileDevices and Services, ser. MobileHCI ’10. New York, NY, USA: ACM,2010, pp. 93–102. [Online]. Available: http://doi.acm.org/10.1145/1851600.1851619[24] K. Vadas, N. Patel, K. Lyons, T. Starner, and J. Jacko, “Reading on-the-go:A comparison of audio and hand-held displays,” in Proceedings of the8th Conference on Human-computer Interaction with Mobile Devices and59Services, ser. MobileHCI ’06. New York, NY, USA: ACM, 2006, pp.219–226. [Online]. Available: http://doi.acm.org/10.1145/1152215.1152262[25] S. Carter and L. Denoue, “Seereader: An (almost) eyes-free mobile richdocument viewer,” arXiv preprint arXiv:0909.2185, 2009.[26] C.-H. Yu and R. C. Miller, “Enhancing mobile browsing and reading,” inCHI ’11 Extended Abstracts on Human Factors in Computing Systems, ser.CHI EA ’11. New York, NY, USA: ACM, 2011, pp. 1783–1788. [Online].Available: http://doi.acm.org/10.1145/1979742.1979845[27] O. Metatla, N. Bryan-Kinns, and T. Stockman, ““i hear you”: Understandingawareness information exchange in an audio-only workspace,” inProceedings of the 2018 CHI Conference on Human Factors in ComputingSystems, ser. CHI ’18. New York, NY, USA: ACM, 2018, pp. 546:1–546:13. [Online]. Available: http://doi.acm.org/10.1145/3173574.3174120[28] Y. Vazquez-Alvarez and S. A. Brewster, “Eyes-free multitasking: The effectof cognitive load on mobile spatial audio interfaces,” in Proceedings ofthe SIGCHI Conference on Human Factors in Computing Systems, ser.CHI ’11. New York, NY, USA: ACM, 2011, pp. 2173–2176. [Online].Available: http://doi.acm.org/10.1145/1978942.1979258[29] S. Brewster, J. Lumsden, M. Bell, M. Hall, and S. Tasker, “Multimodal’eyes-free’ interaction techniques for wearable devices,” in Proceedings ofthe SIGCHI Conference on Human Factors in Computing Systems, ser. CHI’03. New York, NY, USA: ACM, 2003, pp. 473–480. [Online]. Available:http://doi.acm.org/10.1145/642611.642694[30] S. Zhao, P. Dragicevic, M. Chignell, R. Balakrishnan, and P. Baudisch,“Earpod: Eyes-free menu selection using touch input and reactive audiofeedback,” in Proceedings of the SIGCHI Conference on Human Factorsin Computing Systems, ser. CHI ’07. New York, NY, USA: ACM, 2007,60pp. 1395–1404. [Online]. Available: http://doi.acm.org/10.1145/1240624.1240836[31] K. A. Li, P. Baudisch, and K. Hinckley, “Blindsight: Eyes-freeaccess to mobile phones,” in Proceedings of the SIGCHI Conferenceon Human Factors in Computing Systems, ser. CHI ’08. NewYork, NY, USA: ACM, 2008, pp. 1389–1398. [Online]. Available:http://doi.acm.org/10.1145/1357054.1357273[32] D. Ashbrook, P. Baudisch, and S. White, “Nenya: Subtle and eyes-freemobile input with a magnetically-tracked finger ring,” in Proceedings ofthe SIGCHI Conference on Human Factors in Computing Systems, ser.CHI ’11. New York, NY, USA: ACM, 2011, pp. 2043–2046. [Online].Available: http://doi.acm.org/10.1145/1978942.1979238[33] J. Pasquero, S. J. Stobbe, and N. Stonehouse, “A haptic wristwatchfor eyes-free interactions,” in Proceedings of the SIGCHI Conferenceon Human Factors in Computing Systems, ser. CHI ’11. NewYork, NY, USA: ACM, 2011, pp. 3257–3266. [Online]. Available:http://doi.acm.org/10.1145/1978942.1979425[34] M. Pielot, A. Kazakova, T. Hesselmann, W. Heuten, and S. Boll,“Pocketmenu: Non-visual menus for touch screen devices,” in Proceedingsof the 14th International Conference on Human-computer Interactionwith Mobile Devices and Services, ser. MobileHCI ’12. New York,NY, USA: ACM, 2012, pp. 327–330. [Online]. Available: http://doi.acm.org/10.1145/2371574.2371624[35] S. Zhu, J. Zheng, S. Zhai, and X. Bi, “I’sfree: Eyes-free gesture typingvia a touch-enabled remote control,” in Proceedings of the 2019 CHIConference on Human Factors in Computing Systems, ser. CHI ’19. NewYork, NY, USA: ACM, 2019, pp. 448:1–448:12. [Online]. Available:http://doi.acm.org/10.1145/3290605.330067861[36] S. Holland, D. R. Morse, and H. Gedenryd, “Audiogps: Spatial audio nav-igation with a minimal attention interface,” Personal and Ubiquitous com-puting, vol. 6, no. 4, pp. 253–259, 2002.[37] D. Bragg, C. Bennett, K. Reinecke, and R. Ladner, “A large inclusivestudy of human listening rates,” in Proceedings of the 2018 CHIConference on Human Factors in Computing Systems, ser. CHI ’18. NewYork, NY, USA: ACM, 2018, pp. 444:1–444:12. [Online]. Available:http://doi.acm.org/10.1145/3173574.3174018[38] M. H. Cohen, M. H. Cohen, J. P. Giangola, and J. Balogh, Voice user inter-face design. Addison-Wesley Professional, 2004.[39] M. E. Masson, “Cognitive processes in skimming stories.” Journal of Exper-imental Psychology: Learning, Memory, and Cognition, vol. 8, no. 5, p. 400,1982.[40] G. B. Duggan and S. J. Payne, “Text skimming: The process andeffectiveness of foraging through text under time pressure.” Journal ofExperimental Psychology: Applied, vol. 15, no. 3, p. 228, 2009. [Online].Available: http://dx.doi.org/10.1037/a0016995[41] C. C. Marshall and C. Ruotolo, “Reading-in-the-small: A study ofreading on small form factor devices,” in Proceedings of the 2NdACM/IEEE-CS Joint Conference on Digital Libraries, ser. JCDL ’02.New York, NY, USA: ACM, 2002, pp. 56–64. [Online]. Available:http://doi.acm.org/10.1145/544220.544230[42] T. Hillesund, “Digital reading spaces: How expert readers handle books, theweb and electronic paper,” 2010.[43] J. Franze, K. Marriott, and M. Wybrow, “What academics want whenreading digitally,” in Proceedings of the 2014 ACM Symposium on Document62Engineering, ser. DocEng ’14. New York, NY, USA: ACM, 2014, pp.199–202. [Online]. Available: http://doi.acm.org/10.1145/2644866.2644894[44] D. Kim, G. Gweon, and G. Lee, “Sats: Structure-aware touch-basedscrolling,” ETRI Journal, vol. 38, no. 6, pp. 1104–1113, 2016.[45] B. Lee, O. Savisaari, and A. Oulasvirta, “Spotlights: Attention-optimized highlights for skim reading,” in Proceedings of the 2016 CHIConference on Human Factors in Computing Systems, ser. CHI ’16.New York, NY, USA: ACM, 2016, pp. 5203–5214. [Online]. Available:http://doi.acm.org/10.1145/2858036.2858299[46] S. Whittaker, J. Hirschberg, J. Choi, D. Hindle, F. Pereira, and A. Singhal,“Scan: Designing and evaluating user interfaces to support retrieval fromspeech archives,” in Proceedings of the 22nd annual international ACMSIGIR conference on Research and development in information retrieval.ACM, 1999, pp. 26–33.[47] S. Vemuri, P. DeCamp, W. Bender, and C. Schmandt, “Improving speechplayback using time-compression and speech recognition,” in Proceedingsof the SIGCHI Conference on Human Factors in Computing Systems, ser.CHI ’04. New York, NY, USA: ACM, 2004, pp. 295–302. [Online].Available: http://doi.acm.org/10.1145/985692.985730[48] F. Ahmed, Y. Borodin, Y. Puzis, and I. V. Ramakrishnan, “Why read ifyou can skim: Towards enabling faster screen reading,” in Proceedings ofthe International Cross-Disciplinary Conference on Web Accessibility, ser.W4A ’12. New York, NY, USA: ACM, 2012, pp. 39:1–39:10. [Online].Available: http://doi.acm.org/10.1145/2207016.2207052[49] F. Ahmed, Y. Borodin, A. Soviak, M. Islam, I. Ramakrishnan, andT. Hedgpeth, “Accessible skimming: Faster screen reading of web pages,” inProceedings of the 25th Annual ACM Symposium on User Interface Software63and Technology, ser. UIST ’12. New York, NY, USA: ACM, 2012, pp.367–378. [Online]. Available: http://doi.acm.org/10.1145/2380116.2380164[50] F. Ahmed, A. Soviak, Y. Borodin, and I. Ramakrishnan, “Non-visual skimming on touch-screen devices,” in Proceedings of the 2013International Conference on Intelligent User Interfaces, ser. IUI ’13.New York, NY, USA: ACM, 2013, pp. 435–444. [Online]. Available:http://doi.acm.org/10.1145/2449396.2449452[51] M. Bu¨chi, N. Just, and M. Latzer, “Caring is not enough: the importance ofinternet skills for online privacy protection,” Information, Communication &Society, vol. 20, no. 8, pp. 1261–1278, 2017.[52] A. Sulaiman, N. I. Jaafar, and A. Tamjidyamcholo, “Influence of personalitytraits on facebook engagement and their effects on socialization behavior andsatisfaction with university life,” Information, Communication & Society,vol. 21, no. 10, pp. 1506–1521, 2018.[53] M. Anderson and J. Jiang, “Teens, social media technology 2018,” PewResearch Center, May 2018, retrieved September 18, 2019 from https://www.pewinternet.org/2018/05/31/teens-social-media-technology-2018/.[54] A. Smith and M. Duggan, “Crossing the line: What counts asonline harassment?” Pew Research Center, January 2018, re-trieved September 18, 2019 from https://www.pewinternet.org/2018/01/04/crossing-the-line-what-counts-as-online-harassment/.[55] J. P. Kincaid, R. P. Fishburne Jr, R. L. Rogers, and B. S. Chissom, “Deriva-tion of new readability formulas (automated readability index, fog count andflesch reading ease formula) for navy enlisted personnel,” 1975.[56] S. G. Hart and L. E. Staveland, “Development of nasa-tlx (task load index):Results of empirical and theoretical research,” in Advances in psychology.Elsevier, 1988, vol. 52, pp. 139–183.64[57] K. Charmaz and L. L. Belgrave, “Grounded theory,” The Blackwell encyclo-pedia of sociology, 2007.[58] J. O. Wobbrock, L. Findlater, D. Gergle, and J. J. Higgins, “The alignedrank transform for nonparametric factorial analyses using only anovaprocedures,” in Proceedings of the SIGCHI Conference on Human Factorsin Computing Systems, ser. CHI ’11. New York, NY, USA: ACM, 2011, pp.143–146. [Online]. Available: http://doi.acm.org/10.1145/1978942.1978963[59] B. Shneiderman, “The limits of speech recognition,” Communications of theACM, vol. 43, no. 9, pp. 63–65, 2000.[60] D. Goldberg, “The voice actor’s performance guidebook,” 2019,retrieved September 19, 2019 from https://www.edgestudio.com/voice-over-performance-guidebook.[61] K. Hyland and P. Tse, “Metadiscourse in academic writing: A reappraisal,”Applied linguistics, vol. 25, no. 2, pp. 156–177, 2004.[62] C. C. Marshall, “Annotation: from paper books to the digital library,” in Pro-ceedings of the second ACM international conference on Digital libraries.ACM, 1997, pp. 131–140.[63] H. Seifi, K. Zhang, and K. E. MacLean, “Vibviz: Organizing, visualizingand navigating vibration libraries,” in 2015 IEEE World Haptics Conference(WHC). IEEE, 2015, pp. 254–259.[64] A. Cockburn, C. Gutwin, and J. Alexander, “Faster document navigationwith space-filling thumbnails,” in Proceedings of the SIGCHI Conferenceon Human Factors in Computing Systems, ser. CHI ’06. NewYork, NY, USA: ACM, 2006, pp. 1–10. [Online]. Available: http://doi.acm.org/10.1145/1124772.112477465[65] A. Inc., “Human interface guidelines,” retrieved September 18, 2019from https://developer.apple.com/design/human-interface-guidelines/ios/user-interaction/gestures/#standard-gestures.[66] H. Habib, N. Shah, and R. Vaish, “Impact of contextual factors onsnapchat public sharing,” in Proceedings of the 2019 CHI Conferenceon Human Factors in Computing Systems, ser. CHI ’19. NewYork, NY, USA: ACM, 2019, pp. 26:1–26:13. [Online]. Available:http://doi.acm.org/10.1145/3290605.3300256[67] F. Bentley, K. Quehl, J. Wirfs-Brock, and M. Bica, “Understandingonline news behaviors,” in Proceedings of the 2019 CHI Conferenceon Human Factors in Computing Systems, ser. CHI ’19. NewYork, NY, USA: ACM, 2019, pp. 590:1–590:11. [Online]. Available:http://doi.acm.org/10.1145/3290605.3300820[68] R. Zeleznik, A. Bragdon, F. Adeputra, and H.-S. Ko, “Hands-onmath: A page-based multi-touch and pen desktop for technical workand problem solving,” in Proceedings of the 23Nd Annual ACMSymposium on User Interface Software and Technology, ser. UIST ’10.New York, NY, USA: ACM, 2010, pp. 17–26. [Online]. Available:http://doi.acm.org/10.1145/1866029.1866035[69] P. J. Clarkson, R. Coleman, S. Keates, and C. Lebbon, Inclusive design:Design for the whole population. Springer Science & Business Media,2013.[70] J. Xu, S. Zhu, A. Balasubramanian, X. Bi, and R. Shilkrot, “Ultra-low-power mode for screenless mobile interaction,” in Proceedings of the 31stAnnual ACM Symposium on User Interface Software and Technology, ser.UIST ’18. New York, NY, USA: ACM, 2018, pp. 557–568. [Online].Available: http://doi.acm.org/10.1145/3242587.324261466[71] D. Ghosh, P. S. Foong, S. Zhao, D. Chen, and M. Fjeld, “Editalk:Towards designing eyes-free interactions for mobile word processing,” inProceedings of the 2018 CHI Conference on Human Factors in ComputingSystems, ser. CHI ’18. New York, NY, USA: ACM, 2018, pp. 403:1–403:10. [Online]. Available: http://doi.acm.org/10.1145/3173574.3173977[72] A. Sears, M. Lin, J. Jacko, and Y. Xiao, “When computers fade: Pervasivecomputing and situationally-induced impairments and disabilities,” in HCIinternational, vol. 2, no. 03, 2003, pp. 1298–1302.[73] M. Goel, L. Findlater, and J. Wobbrock, “Walktype: Using accelerometerdata to accomodate situational impairments in mobile touch screen textentry,” in Proceedings of the SIGCHI Conference on Human Factors inComputing Systems, ser. CHI ’12. New York, NY, USA: ACM, 2012,pp. 2687–2696. [Online]. Available: http://doi.acm.org/10.1145/2207676.2208662[74] Z. Liu, “Reading behavior in the digital environment: Changes in readingbehavior over the past ten years,” Journal of documentation, vol. 61, no. 6,pp. 700–712, 2005.[75] M. Viswanathan and M. Viswanathan, “Measuring speech quality for text-to-speech systems: development and assessment of a modified mean opinionscore (mos) scale,” Computer Speech & Language, vol. 19, no. 1, pp. 55–83,2005.[76] M. Tohidi, W. Buxton, R. Baecker, and A. Sellen, “Getting theright design and the design right,” in Proceedings of the SIGCHIConference on Human Factors in Computing Systems, ser. CHI ’06.New York, NY, USA: ACM, 2006, pp. 1243–1252. [Online]. Available:http://doi.acm.org/10.1145/1124772.112496067Appendix AStudy Materials Used in NeedFinding StudyIn the following, we provide materials from our first study, the need finding studyin Chapter 3. Specifically, we provide the following -• Call For Participation• Consent Form• Study Protocol and Participant Training• QuestionnairesA.1 Call For Participation68 Department of Computer Science University of British Columbia Vancouver, BC, V6T 1Z4 Are you a grad student? Is it difficult for you to read research articles using a smartphone while commuting? Come help us as we try to make it easier for you! Receive a $15 honorarium for this 1 hour study Who are we? We are researchers from the University of British Columbia: Dr. Joanna McGrenere, Dr. Dongwook Yoon, and Taslim Arefin Khan. We are conducting a study to better understand the needs and challenges of reading a document in situations where visual interaction with smartphone is challenging, e.g., commuting. We are looking for participants who ... • are graduate students • can understand and speak English fluently • are familiar with reading research articles published in conferences, journals, or online portals • are 18 years or older What is involved? You will read few research articles using a smartphone application in a lab setting that will simulate bus commuting environment. After that, you will be interviewed about your experience of using that application. The study will be conducted at the ICICS Building, UBC, Vancouver. Interested in Participating? If you are interested in participating or would like more information, please contact Taslim at ​takhan@cs.ubc.ca​ ​or​ ​778-321-6840​. 69A.2 Consent Form for Participants70THE UNIVERSITY OF BRITISH COLUMBIA Department of Computer Science 2366 Main Mall Vancouver, B.C., V6T 1Z4 Consent Form Non-visual document skimming on the go Principal Investigator: Joanna McGrenere, Professor, Department of Computer Science, University of British Columbia, ​joanna@cs.ubc.ca, 604 827-5201​. Co-Principal Investigator: Dongwook Yoon, Assistant Professor, Department of Computer Science, University of British Columbia, ​yoon@cs.ubc.ca, 604-822-5485 ​. Taslim Arefin Khan, MSc student, Department of Computer Science, University of British Columbia, ​takhan@cs.ubc.ca, 778-321-6840​. What the study is about: The overall purpose of this research is to ​identify problems, needs, and requirement elicitations of audio-based reading (listening) using smartphones in situations where visual interaction with the smartphone is challenging​. What you will be asked to do: ​After you have read this document, I will respond to any questions or concerns that you may have. Once you have signed this consent form, you will be asked to: - interact with a smartphone and read an audio-based reading material - answer interview questions This should take approximately 30 minutes to 1 hour and be completed in 1 session. The session may also be photographed, video, and/or audio recorded. You have the option not to be photographed/video/audio recorded. Inclusion criteria: ​We are looking for participants who: ● are graduate students ● can understand and speak English fluently ● are familiar with reading scholarly articles published in conferences or journals in their respective research fields ● are 18 years or older. Ethics Application ID H18-00664, Consent Form v1.2, July 04, 2018, SpeechSkimmer Page 1 of 3 71 Risks and Benefits: ​There are no known psychological, physical, and social risks involved in the proposed research. Participants may learn how to benefit from non-visual skim reading while commuting to accomplish a reading task. How the data collected will be used: Data collected (including any image, audio/video recordings) will be used for analysis. This analysis will help us extract requirements for a skim reading interface. The research findings from this data may also be used to publish scholarly articles in conferences or journals. Compensation: ​You will receive $15 for his one hour study. The compensation will be provided prior to the study as cash only. Confidentiality: ​The results of your participation will be reported without any reference to you specifically. All information that you provide will be stored in Canada. Only the researchers involved in this research will have access to the data. Your data may be used but will remain anonymous in any reports, research papers, thesis documents, and presentations that result from this work. Your name will never be used, only a code associated with the data. Any identifiable element in the images, videos, or audios will be blurred or obscured to prevent identification. The only link between the code associated and your actual name will be this consent form and a printed spreadsheet, that will be stored in a locked cabinet, in a research lab with controlled access at UBC. The file linking your name with the associated code will be kept separate from the data. Data Retention: All electronic files will be stored in an encrypted hard drive of a password protected laptop. Backups of the data will be stored on the UBC Computer Science secure department servers. Any handwritten notes and paper transcripts will be kept in a locked cabinet in the researchers' laboratory or the Principal Investigator's office with controlled access at UBC. Five years after the completion of the research, the electronic data will be deleted, and all physical media, audio/video records, and paper transcripts will be shredded by the Principal Investigator. Taking part is voluntary: ​Your participation is voluntary. You are free to stop your participation at any point. Refusal to participate or withdrawal of your consent or discontinued participation in the study will not result in any penalty or loss of benefits or rights to which you might otherwise be entitled. The Principal Investigator may at his/her discretion remove you from the study for a number of reasons. In such an event, you will not suffer any penalty or loss of benefits or rights which you might otherwise be entitled. It will not affect your current or future relationship with the University of British Columbia. Ethical Concerns and Participant Rights Ethics Application ID H18-00664, Consent Form v1.2, July 04, 2018, SpeechSkimmer Page 2 of 3 72If you have any concerns or complaints about your rights as a research participant and/or your experiences while participating in this study, contact the Research Participant Complaint Line in the UBC Office of Research Ethics at 604-822-8598 or if long distance e-mail RSIL@ors.ubc.ca or call toll free 1-877-822-8598. Please tick at least one of the following ❏ I consent to be audio recorded in this study ❏ I consent to be video recorded in this study ❏ I consent to be photographed in this study ❏ I do not consent to be photographed, audio, and video recorded in this study I, ________________________________, have read the explanation about this study. I have been given the opportunity to discuss it and my questions have been answered to my satisfaction. I hereby consent to take part in this study. However, I realize that my participation is voluntary and that I am free to withdraw at any time. ____________________________________________________ Participant’s Signature Date Ethics Application ID H18-00664, Consent Form v1.2, July 04, 2018, SpeechSkimmer Page 3 of 3 73A.3 Study Protocol and Participant Training74Speech Skimmer StudyTaslim Arefin Khan175Training with Voice Dream Reader App276Basic ControlsPlay/pause buttonJump forwardJump back377Following the SpeechCurrently read word highlighted by a red rectangleCurrently read line highlighted by a yellow rectangle478LayoutsOriginal layout Plain text layout579Read a Selected LineDouble tap on a line that you want to read680Jump to Another PageTap hereInput which page you want to go781Controlling Your SpeedYou can adjust the speech rate to your comfort using the slider, or the + and - buttonsInitially we will set speech rate to 250 WPM882Your TaskYou are a grad student. You are taking a grad course. For the next class, you have 4 reading assignments. You have to choose 2 articles which you will read in details. You want to participate in-class discussion where you have to reason why you have chosen those 2 articles. You have a commute to school and just have enough time to read 2 articles after reaching school. You would like to use this commute time and skim all 4 articles to decide which 2 you are going to read later.983Your Objective• You should skim each article to comprehend the following• Get a general idea or gist of the article• What is the article about? What problems are they working on?• What suggestions/takeaways have been discussed in the article?• Discover the structure of the material• How is the article structured? i.e., What is the outline in terms of sections?• Which parts/sections will help you most to understand the article when you read later?1084How You Gonna Do It?• You will perform each task in a standing posture• After each task, you will answer a survey• Your objective will be to finish each article under a time constraint• Minimize your visual attention to the screen as much as possible, else you will get motion sickness• You can look at the screen while putting a command, but look away while listening; if possible put the commands eyes-free1185A.4 Questionnaires and Interview Questions86 Non-visual Document Skimming On the Go Study Demographic questions ● Which of following age group are you a part of? ○ 18 - 24 ○ 25 - 34 ○ 35 - 44 ○ 45 - 54 ○ 55 and above ● To which gender identity do you most identify? ○ Male ○ Female ○ Prefer not to say ○ Prefer to self describe ______ ● What is your academic discipline (e.g., Computer Science, Psychology, Medicine, etc.)? ____________ ● What is the current status of your vision? ○ Normal or corrected to normal ○ Other ________________________ (briefly describe) Interview questions ● How often do you commute using a public transit system such as a bus or train? ● What is the duration of your most frequent daily commute? ● What do you do during this commute time in a bus or train? ● How often do you read scholarly articles published in conferences or journals in your daily life? ● In general, what is your approach or strategy of reading such scholarly articles when you see it for the first time? ● What is the usual setting for you (time, place, comfort, etc.) to accomplish a scholarly article reading task? ● How often do you accomplish this task in mobile devices (smartphone or tablet)? ● If your strategy of reading a scholarly article is different from that of reading in a desktop or printed paper format, what are those strategies and why are they different? ● Can you think of a time when you had to read any scholarly article while commuting? ● Can you elaborate on your strategies of reading a scholarly article while commuting? What is it like for you to read while commuting? ● If you don't prefer reading while commuting, what are the reasons? ● What are the challenges of reading while commuting? ● What could better support these challenges for reading while commuting? Observation probing Following are the questions based on observing participants during a reading task. Interview questions v1.1, June 14, 2018, SpeechSkimmer Page 1 of 2 87 ● Can you explain your experience of listening to scholarly articles published in conferences or journals by elaborating on what worked well and what did not work well? ● How easy or difficult was it to listen and comprehend this (task) article using this (software used for the task) application? ● What would be the challenges of completing this task in a standing posture? ● What would be the challenges of completing this task in a sitting posture? ● I noticed that you did <an interaction technique> while reading <moment or instance in the material>. Could you please explain what was happening at that moment? Interview questions v1.1, June 14, 2018, SpeechSkimmer Page 2 of 2 88Default Question BlockPlease tick the layout type of this taskPlease tick the article type of this taskBlock 1Please state to what extent you agree or disagree with the following statements.Block 2Plain text Original layoutResearch paper Pew report Stronglydisagree DisagreeNeither agreenor disagree AgreeStronglyagreeI clearly understood theproblem that this articleaddressed I clearly understood thekey takeaway message ofthis article I clearly understood howthis article is organised Qualtrics Survey Software https://ubc.ca1.qualtrics.com/Q/EditSection/Blocks/Ajax/GetS...1 of 2 13/10/19, 7:55 PM89Private Policy Terms of UsePowered by QualtricsPlease state to what extent you agree or disagree with the following statements. Stronglydisagree DisagreeNeither agreenor disagree AgreeStronglyagreeI clearly understood thetopic of this article I clearly understood thekey takeaway orconclusion of this article I clearly understood howthis article is organised Qualtrics Survey Software https://ubc.ca1.qualtrics.com/Q/EditSection/Blocks/Ajax/GetS...2 of 2 13/10/19, 7:55 PM9013/10/19 7(56 PMNASA Task Load IndexPage 1 of 1file:///Users/taslimarefinkhan/Downloads/NASA%20Task%20Load%20Index%20Tool.htmlTask Questionnaire - Part 1Participant: Session: Click on each scale at the point that best indicates your experience of the task Mental DemandLow HighHow much mental and perceptual activity was required (e.g. thinking, deciding,calculating, remembering, looking, searching, etc)? Was the task easy or demanding,simple or complex, exacting or forgiving?Physical DemandLow HighHow much physical activity was required (e.g. pushing, pulling, turning, controlling,activating, etc)? Was the task easy or demanding, slow or brisk, slack or strenuous,restful or laborious?Temporal DemandLow HighHow much time pressure did you feel due to the rate of pace at which the tasks ortask elements occurred? Was the pace slow and leisurely or rapid and frantic?PerformanceGood PoorHow successful do you think you were in accomplishing the goals of the task set bythe experimenter (or yourself)? How satisfied were you with your performance inaccomplishing these goals?EffortLow HighHow hard did you have to work (mentally and physically) to accomplish your level ofperformance?FrustrationLow HighHow insecure, discouraged, irritated, stressed and annoyed versus secure, gratified,content, relaxed and complacent did you feel during the task?Continue >>91Appendix BStudy Materials Used in PreliminaryEvaluation StudyIn the following, we provide materials from our second study, preliminary evalu-ation of Skimmer in Chapter 6. Specifically, we provide the following -• Call For Participation• Consent Form• Study Protocol• QuestionnairesB.1 Call For Participation92 Department of Computer Science University of British Columbia Vancouver, BC, V6T 1Z4 Are you a grad student? Are you interested in trying auditory skim reading on the bus? Who are we? We are researchers from the University of British Columbia: Dr. Joanna McGrenere, Dr. Dongwook Yoon, and Taslim Arefin Khan. We are conducting a study to understand the usefulness of auditory skim reading using a smartphone application. We are looking for participants who ... • are graduate students • can understand and speak English fluently • are familiar with reading scholarly articles published in conferences or journals in their respective research fields • are 18 years or older What is involved? You will be using a smartphone application in a bus commuting setting. You will meet us at the UBC Nest. Then we will walk towards the UBC bus loop where we will board bus #99. Once the study tasks are completed, we will get off at the next stop and you will sit for an interview. The study will take approximately 60 minutes to complete, but we encourage participants to hold 90 minutes of their time. Receive a $20 honorarium for this study Interested in Participating? If you are interested in participating or would like more information, please contact Taslim at ​takhan@cs​.ubc.ca​ ​or​ ​778-321-6840​. Call for Participation (Version 2.1 / June 26, 2019) 93B.2 Consent Form for Participants94THE UNIVERSITY OF BRITISH COLUMBIA Department of Computer Science 2366 Main Mall Vancouver, B.C., V6T 1Z4 Consent Form Eyes-reduced document skimming on the go Principal Investigator: Joanna McGrenere, Professor, Department of Computer Science, University of British Columbia, ​joanna@cs.ubc.ca, 604 827-5201​. Co-Principal Investigator: Dongwook Yoon, Assistant Professor, Department of Computer Science, University of British Columbia, ​yoon@cs.ubc.ca, 604-822-5485 ​. Taslim Arefin Khan, MSc student, Department of Computer Science, University of British Columbia, ​takhan@cs.ubc.ca, 778-321-6840​. What the study is about: The overall purpose of this research is to ​assess the usefulness of a text-to-speech skimming interface for eyes-reduced interaction in a bus commuting context. What you will be asked to do: ​After you have read this document, I will respond to any questions or concerns that you may have. Once you have signed this consent form, you will be asked to: - board on the bus #99 from UBC bus loop and find a seat - interact with a smartphone and listen to an audio-based reading material - get off the bus - answer interview questions This should take approximately 60 minutes (maximum 90 minutes) and be completed in 1 session. The session may also be photographed, video, and/or audio recorded. You have the option not to be photographed/video/audio recorded. Inclusion criteria: ​We are looking for participants who: ● are graduate students ● can understand and speak English fluently Ethics Application ID H18-00664, Consent Form v2.2, July 04, 2019, SpeechSkimmer++ Page 1 of 3 95● are familiar with reading scholarly articles published in conferences or journals in their respective research fields ● are 18 years or older. Risks and Benefits: ​There are no known psychological, physical, and social risks involved in the proposed research. Since visually reading in a commuting context induces motion sickness in some people, participants may feel nauseous. Participants are welcome to pause/stop the study at any moment they feel nauseous. Participants may learn how to benefit from eyes-reduced skim reading while commuting to accomplish a reading task. How the data collected will be used: Data collected (including any image, audio/video recordings) will be used for analysis. This analysis will help us understand our design and help us improve the design for future. The research findings from this data may also be used to publish scholarly articles in conferences or journals. Compensation: ​You will receive $20 for this study. The compensation will be provided prior to the study as cash only. Confidentiality: ​The results of your participation will be reported without any reference to you specifically. All information that you provide will be stored in Canada. Only the researchers involved in this research will have access to the data. Your data may be used but will remain anonymous in any reports, research papers, thesis documents, and presentations that result from this work. Your name will never be used, only a code associated with the data. Any identifiable element in the images, videos, or audios will be blurred or obscured to prevent identification. The only link between the code associated and your actual name will be this consent form and a printed spreadsheet, that will be stored in a locked cabinet, in a research lab with controlled access at UBC. The file linking your name with the associated code will be kept separate from the data. Data Retention: All electronic files will be stored in an encrypted hard drive of a password protected laptop. Backups of the data will be stored on the UBC Computer Science secure department servers. Any handwritten notes and paper transcripts will be kept in a locked cabinet in the researchers' laboratory or the Principal Investigator's office with controlled access at UBC. Five years after the completion of the research, the electronic data will be deleted, and all physical media, audio/video records, and paper transcripts will be shredded by the Principal Investigator. Taking part is voluntary: ​Your participation is voluntary. You are free to stop your participation at any point. Refusal to participate or withdrawal of your consent or discontinued participation in the study will not result in any penalty or loss of benefits or rights to which you might otherwise be entitled. The Principal Investigator may at his/her discretion remove you from the study for a number of reasons. In such an event, Ethics Application ID H18-00664, Consent Form v2.2, July 04, 2019, SpeechSkimmer++ Page 2 of 3 96you will not suffer any penalty or loss of benefits or rights which you might otherwise be entitled. It will not affect your current or future relationship with the University of British Columbia. Ethical Concerns and Participant Rights If you have any concerns or complaints about your rights as a research participant and/or your experiences while participating in this study, contact the Research Participant Complaint Line in the UBC Office of Research Ethics at 604-822-8598 or if long distance e-mail RSIL@ors.ubc.ca or call toll free 1-877-822-8598. Please tick at least one of the following ❏ I consent to be audio recorded in this study ❏ I consent to be video recorded in this study ❏ I consent to be photographed in this study ❏ I do not consent to be photographed, audio, and video recorded in this study I, ________________________________, have read the explanation about this study. I have been given the opportunity to discuss it and my questions have been answered to my satisfaction. I hereby consent to take part in this study. However, I realize that my participation is voluntary and that I am free to withdraw at any time. ____________________________________________________ Participant’s Signature Date Ethics Application ID H18-00664, Consent Form v2.2, July 04, 2019, SpeechSkimmer++ Page 3 of 3 97B.3 Study Protocol and Participant Training98How you are gonna do it: 1. We will board a bus. 2. You will do one reading with Skimmer within a fixed amount of time. Try not to minimize your attention to the screen, i.e., look at the screen as less as possible. 3. We will unboard the bus. 4. Answer some questions. 5. We will board another bus. 6. You will do the second and final reading with VoiceDreamReader within a fixed amount of time. Try not to minimize your attention to the screen, i.e., look at the screen as less as possible. 7. We will unboard the bus. 8. Answer some questions. What you should find:​ We want you to skim in a way so that you have answers to the following: 1. Gist of the article 2. Structural organisation of the article 3. What Research Questions were asked in the article 4. Main contributions of the article Training: ​We trained participants by showing live demos of both Skimmer and VoiceDreamReader. 99B.4 Questionnaires and Interview Questions100Observations on Key design elements in support of different sub-tasks 1. Sub-task: Listenability and listening experience a. Use of two voices b. Use of earcons to reinforce action-complete c. Embedding paralingual qualities i. Emphasizing important words/group of words ii. Emphasizing quotes iii. Itemized li5sts narration (pre-announcing number of items of follow for > 4 items) iv. Narrating statistical test results in the text 2. Sub-task: Navigation a. Sentence navigation i. Discreet jump ii. Continuous (or random) b. Discourse marker navigation i. Ambient earcon c. Paragraph navigation d. Section navigation i. Selection of a Section or Subsection ii. Transition (in/out) from the Overview page e. Spatial random access (start from anywhere) f. Speech rate variation 3. Sub-task: Eyes-free and momentary visual attention a. Single reading arrow head position in the middle b. Ephemeral display of numerical entity c. Current position notification d. Attention to visual entity i. Vibration nudge to catch attention ii. Spatial positioning of opt in/out button 101Interview Questions Open-ended questions: 1. What is your initial reaction or impression to using the app for audio skimming? In what ways did the app support / not support you?How did the app support skimming? 2. Is this app something that you can imagine yourself using in your life? Example questions following participant score in the questionnaires 1. Can you please elaborate the reasoning behind your rating here? 2. It looks like you rated Physical Demand as the factor that contributed more to overall workload when using this application.. Why do you think so? Example questions following observation on key design elements 1. How was your experience going to Conclusion from Introduction? How easy or difficult was it for you? 2. How was your experience of listening to statistical values? What kind of challenges did you face while listening to the numbers? 1023. How was your experience of seeing figures and tables in the paper? 4. How easy or difficult was sentence/paragraph/discourse navigation? What challenges did you face during this navigation? 5. How did the two different voices (male and female) worked or not for your skimming tasks? 10313/10/19 7(56 PMNASA Task Load IndexPage 1 of 1file:///Users/taslimarefinkhan/Downloads/NASA%20Task%20Load%20Index%20Tool.htmlTask Questionnaire - Part 1Participant: Session: Click on each scale at the point that best indicates your experience of the task Mental DemandLow HighHow much mental and perceptual activity was required (e.g. thinking, deciding,calculating, remembering, looking, searching, etc)? Was the task easy or demanding,simple or complex, exacting or forgiving?Physical DemandLow HighHow much physical activity was required (e.g. pushing, pulling, turning, controlling,activating, etc)? Was the task easy or demanding, slow or brisk, slack or strenuous,restful or laborious?Temporal DemandLow HighHow much time pressure did you feel due to the rate of pace at which the tasks ortask elements occurred? Was the pace slow and leisurely or rapid and frantic?PerformanceGood PoorHow successful do you think you were in accomplishing the goals of the task set bythe experimenter (or yourself)? How satisfied were you with your performance inaccomplishing these goals?EffortLow HighHow hard did you have to work (mentally and physically) to accomplish your level ofperformance?FrustrationLow HighHow insecure, discouraged, irritated, stressed and annoyed versus secure, gratified,content, relaxed and complacent did you feel during the task?Continue >>104
Thesis/Dissertation
eng
Computer Science
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Designing an eyes-reduced document skimming app for situational impairments
Text