UBC Faculty Research and Publications

The role of pre-existing highlights in reader–text interactions and outcomes Dodson, Samuel; Freund, Luanne; Kopak, Rick 2018

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


52383-Dodson_S_et_al_Role_preexsisting_highlights.pdf [ 465.73kB ]
JSON: 52383-1.0376433.json
JSON-LD: 52383-1.0376433-ld.json
RDF/XML (Pretty): 52383-1.0376433-rdf.xml
RDF/JSON: 52383-1.0376433-rdf.json
Turtle: 52383-1.0376433-turtle.txt
N-Triples: 52383-1.0376433-rdf-ntriples.txt
Original Record: 52383-1.0376433-source.json
Full Text

Full Text

The role of pre-existing highlights in reader–text interactionsand outcomesSamuel DodsonUniversity of British Columbiadodsons@mail.ubc.caLuanne FreundUniversity of British Columbialuanne.freund@ubc.caRick KopakUniversity of British Columbiar.kopak@ubc.caABSTRACTMany digital information environments enable sharing of readers’highlights and other annotations, despite the lack of clear evidenceof the effects on interaction behaviours and outcomes. We reporton an experimental user study (n=15) of the impact of pre-existinghighlights of varying quality on the digital reading process andoutcomes of participants with different cognitive styles. We foundthat highlight quality affects surface level comprehension, but notdeeper understanding. Participants were able to assess highlightquality and expressed different approaches to highlighting thatinfluenced their interpretation of pre-existing highlights. Resultsregarding the impact of cognitive style were inconclusive.CCS CONCEPTS• Information systems → Collaborative and social comput-ing systems and tools; Users and interactive retrieval; • Ap-plied computing→ Collaborative learning;KEYWORDSannotation, comprehension, highlighting, reading, social readingACM Reference Format:Samuel Dodson, Luanne Freund, and Rick Kopak. 2018. The role of pre-existing highlights in reader–text interactions and outcomes. In JCDL ’18:The 18th ACM/IEEE Joint Conference on Digital Libraries, June 3–7, 2018, FortWorth, TX, USA. ACM, New York, NY, USA, 4 pages. https://doi.org/10.1145/3197026.31970661 INTRODUCTIONAnnotation is widely considered to be an effective method to sup-port active and effective reading [19], and highlighting, the mostcommonly used form of annotation [14], is available as a social, orshared, feature in many reading and learning platforms [7]. Withinthe digital library research community, there is more than twodecades of research on annotation, and with much of that work fo-cused on the conceptual and technical implementation [e.g., 1, 12].Much less is known about the effects of shared and pre-existinghighlighting on reading and learning. There is an assumption thatsharing annotations can be useful for learning, as indicated by thePermission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than theauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, orrepublish, to post on servers or to redistribute to lists, requires prior specific permissionand/or a fee. Request permissions from permissions@acm.org.JCDL ’18, June 3–7, 2018, Fort Worth, TX, USA© 2018 Copyright held by the owner/author(s). Publication rights licensed to Associa-tion for Computing Machinery.ACM ISBN 978-1-4503-5178-2/18/06. . . $15.00https://doi.org/10.1145/3197026.3197066development of social annotation systems for this purpose [e.g., 8],but the evidence is far from conclusive. It is unclear, for example,how shared highlights affect reading processes, particularly acrossdifferent groups of readers. A recent study of the effects of pre-existing highlights on reading outcomes [3] found that highlightsdid not improve comprehension and may be a disadvantageousfeature of the text. To better understand this phenomenon and toinform the design of social reading systems and digital libraries,this study addresses the following research questions: i) What isthe impact of variable quality highlights (i.e., appropriate and in-appropriate) on comprehension? ii) How are such highlights usedand perceived by readers with different cognitive styles? iii) Whatfactors affect the use and usefulness of highlights for readers?2 PRIORWORKGiven the central role of reading in learning, and the impact ofreading comprehension on post-secondary education outcomes, itis essential to findways to support effective reader–text interactions[17]. There has been a trend to increase readers’ interactions withtexts through active reading, where readers continuously evaluatethe meaning of the text and its use for their task [21], or in deepreading, defined as reading with the aim of comprehension andlong term retention [16]. A related goal is to facilitate social andcollaborative reading and learning, through systems that enablemultiple reader–text interactions, often through shared annotationcapabilities, whether within private reading groups or public digitalreading systems, such as Hypothesis (https://web.hypothes.is).Though highlighting is one of the most commonly used readingstrategies [14], existing studies of highlighting are inconclusive as toits benefits. Dunlosky et al. [4] carried out a review of a wide rangeof learning techniques employed by students, and concluded thathighlighting, while widely practiced by students, was not foundto be effective in improving outcomes. They suggest that thesefindings are limited, in that prior studies have not controlled for theamount of text highlighted or the quality of the highlights, nor hasresearch focused on what the conditions for effective highlightingare. The majority of studies Dunlosky et al. reviewed were printbased, and do not address the concerns raised by digital readingpractices and the greater challenges in attention and deep reading.The strongest effects of highlighting seem to be those for di-recting attention to particular words and for recall of facts, ratherthan for comprehending concepts. Fowler and Barker [5] foundthat pre-existing highlights can have a positive effect on recall.Similarly, Chi, Gumbrecht, and Hong [2] found that readers wereable to answer reading questions faster and more accurately usinga digital text with pre-existing highlights than without. Silvers andKreiner [18] studied the effects of appropriate and inappropriateJCDL ’18, June 3–7, 2018, Fort Worth, TX, USA Samuel Dodson, Luanne Freund, and Rick Kopakpre-existing highlighting on comprehension and found that com-prehension levels were not improved by appropriate highlightsbut were significantly lower for inappropriate highlighting. Similareffects were found by [3] in a digital reading environment.Reading depends on bottom-up and top-down cognitive pro-cesses [22]. Integrating a theory of comprehension into our studyoffers insight into the cognitive processes that influence how read-ers create and restructure mental representations of the text. Theo-ries of comprehension that use an interaction of bottom-up and top-down processes, such as the Construction-Integration (CI) Model[9], “are currently considered the best frameworks for understand-ing individual differences in reading comprehension” [10]. Kintsch [9],for example, distinguishes between how readers generate represen-tations of a text at different levels: themicrostructure andmacrostruc-ture. The CI model shows that a reader can establish a microstruc-ture representation of a text that is sufficient to answer some ques-tions about the text without comprehending the deeper meaningof the text through the macrostructure.Messick [13] defines cognitive style as the “strategies determin-ing a person’s typical modes of perceiving, remembering, thinking,and problem solving.” One of the most studied cognitive styles isField Dependence-Independence (FDI) [23]. Depending on theirdegree of FDI, individuals process, encode, and recall informationdifferently. Field Dependents (FDs) use external cues when inter-acting with information, whereas Field Independents (FIs) rely oninternal cues [24]. When reading, for example, FDs are more likelyto use the pre-existing structure of a text than FIs, who tend ig-nore external cues. Wolfe [25] states, “annotations provide a criticalscaffolding that can support students’ critical thinking and argu-mentation activities.” We posit that FDI may play a role in shapinghow individuals perceive and use highlights.3 METHODSWe conducted a within-subjects study in which each of 15 par-ticipants, undergraduate students recruited using a listserv, readtwo articles with pre-existing highlights. The study employed eye-tracking, questionnaires, and interviews for data collection. Par-ticipants were asked to read two articles from Scientific American[15, 20] containing highlights created by the authors. Their instruc-tions asked them to read the articles as if they were assigned for anupcoming class, and noted that the highlights were left by previousreaders. Articles were selected so as to be interesting to under-graduate students across disciplines, and texts of approximately3,000 words were used, longer in comparison to those used by [18].Participants read the texts using a twenty-four inch liquid-crystaldisplay on a desktop computer equipped with a Tobii Pro X2-60eye-tracker.Articles were presented in two conditions, which we termedappropriate (APP) and inappropriate (INAPP) highlighting. In keep-ing with Kintsch’s CI model of comprehension [9], APP highlightsmarked key concepts and themes important to understanding theoverall gist or message of the article, and INAPP highlights markedcontent that was secondary or peripheral. To create these condi-tions, the three authors read the texts independently, highlightingappropriate content first, and on a second pass, highlighting inap-propriate content. The three sets of highlights were aggregated andspans of text that were highlighted by at least two of the authorswere selected to represent each condition. In the final versions ofthe articles, highlighting emphasized no more than fifteen percentof the text, in accordance with guidelines based on previous find-ings [11]. The instruments and highlights were identical to thoseused in a previous experiment [3], however, we eliminated the con-trol condition (no highlighting) as our purpose was to compare themore specific effects of APP and INAPP highlights on reading.The paper version of the Group Embedded Figures Test (GEFT)was administered to participants at the beginning of the study toidentify their cognitive style. Participants were classified as FD orFI using the median split used in [3]. The response variables wereparticipants’ comprehension scores and eye fixations while readingthe articles. Comprehension scores were determined using Term–DefinitionMatching, where participants were asked tomatch a termwith one of nine possible definitions, and a modified Cloze test,where participants were asked to fill-in the blanks of a paragraphsummary of each article. The quiz was designed to measure bothshallow (surface level recall) and deep (conceptual understandingand gist) levels of comprehension, in accordance with the CI model.Participants were given five minutes to read each article andsevenminutes to complete the quiz, and theywere informed of thesecontstraints in the instructions. Through pilot testing, we foundthis was enough time for participants to skim the articles whilealso encouraging use of the highlights. At the end of each sessionwe asked participants to rate their level of prior topical knowledge,topical interest, and perceptions of the highlights on a 5-point scale.After completing the reading tasks, a 10 minute semi-structuredinterview was carried out to learn more about participants’ task-related experiences, and their reading and highlighting practicesmore generally. Participants received a $20 honorarium.4 RESULTSQuantitative results are summarized first in three sections, followedby findings from the interview data. The assumptions necessary torun parametric tests were met, and an alpha value of .05 was used.4.1 Comprehension OutcomesAcross all participants, scores were significantly higher in the APPthan the INAPP condition for the Term–Definition Matching ques-tions (Table 1). While the mean comprehension was also higherin APP for the Cloze questions, the difference was not significant.Within FDI groups, there were no significant differences betweenAPP and INAPP comprehension scores, perhaps due to the smallergroup sizes. A t-test showed no difference between FD and FI par-ticipants (t=-0.620, df =9.873, p=.550), indicating that the impact ofthe APP and INAPP conditions was similar across these groups.4.2 Use and Perceptions of HighlightsA summary of the eye-tracking fixation data (Table 2) shows ahigher mean time spent looking at highlights in the APP conditioncompared to INAPP. This difference is significant for the FD par-ticipants (t(5)=3.921, p=.011), but not for FIs (t(8)=2.180, p=.061).While mean times spent looking at highlights are higher for FDsthan FIs in both conditions, t-tests indicate that differences are notsignificant: APP (t(6.801)=1.665, p=.141); INAPP (t(12.861)=0.353,The role of pre-existing highlights in reader–text interactions and outcomes JCDL ’18, June 3–7, 2018, Fort Worth, TX, USATable 1: Term–Definition Matching and Cloze score by FDI and condition.Term–Definition Matching ClozeFDI n APP M (SD) INAPP M (SD) APP–INAPP APP M (SD) INAPP M (SD) APP–INAPPFD 6 6.33 (2.42) 4.50 (2.07) t(5)=1.611, p=.168 3.50 (2.17) 3.83 (2.79) t(5)=-0.415, p=.695FI 9 7.22 (1.48) 5.56 (2.65) t(8)=1.826, p=.105 4.11 (2.93) 3.56 (3.17) t(8)=0.342, p=.741all 15 6.87 (1.88) 5.13 (2.42) t(14)=2.525, p=.024 3.87 (2.59) 3.67 (2.92) t(14)=0.199, p=.845p=.730). So, while there is some suggestion that FDs relied moreheavily on the highlights, the impact of the conditions on bothgroups was similar in terms of their fixation times.Table 2: Mean fixation duration (seconds) on highlights byFDI and condition.FDI n APP M (SD) INAPP M (SD) BothFD 6 117.18 (23.50) 88.00 (11.71) 102.59 (23.36)FI 9 99.84 (12.13) 85.44 (16.34) 92.64 (15.81)all 15 106.78 (18.94) 86.46 (14.25) 96.62 (19.44)Results indicate that, overall, highlights did not attract a dispro-portionate amount of attention in comparison to non-highlightedtext. Given that the Areas of Interest marking highlighted texts forthe eye-tracking analysis constituted about a third of the contentfor each condition, the baseline fixation time would be 100 secondsof the total reading time of 300 seconds (5 minutes). Only the FDsin the APP condition exceeded this baseline (Table 2).Participants’ perceptions of the usefulness of highlights are sum-marized in Table 3. Overall, participants rated the usefulness ofhighlights significantly higher in the APP condition than the INAPP.These differences are not significant within the FDI groups.Table 3:Meanusefulness scores of highlights by FDI and con-dition (5-point scale).FDI n APP M (SD) INAPP M (SD) APP–INAPPFD 5∗ 3.60 (0.89) 3.20 (0.84) t(4)=1.000, p=.374FI 9 3.89 (0.60) 3.22 (0.97) t(8)=2.000, p=.081all 14 3.79 (0.70) 3.21 (0.89) t(13)=2.280, p=.040∗Data missing from one participant.4.3 Role of Prior Knowledge and InterestTo investigate the association between comprehension and self-report measures we calculated nonparametric correlations. Re-sults show significant positive correlations between interest andCloze test scores in both the APP rs (12) = .564, p=.036 and INAPPrs (12)=.713, p=.004 conditions, with the strongest correlation ob-served in the latter. Prior knowledge and comprehension scoreswere not significantly correlated. This suggests that participants’level of interest in the content of the articles is a factor in theirability to grasp their overall meaning, or gist, as assessed throughthe Cloze test. This was particularly true in the INAPP condition.4.4 Findings From InterviewsThe interview data was coded inductively using QDA Miner. Re-sponses provide further insights on uses and perceptions of high-lights and factors that affect use. Two different approaches to high-lighting emerged. One group, includingmany FIs, viewed highlightsas a means to gain overall understanding: “the passages were toolong, so what I did was to only read the highlights because theyhelped me get the main idea of the articles” [P12]. This was basedon the idea that highlights convey “the gist of the paragraph” [P4].Another group, primarily FDs, expected highlights to signalinformation or facts of significance. P11 noted, “I found the ones thatwere highlighting a certain word or something. . . like definitions,I found those really helpful.” P5 clearly preferred such highlights:“in the first article. . . the highlights. . . skipped over the useful detailsthat I would have personally highlighted and they went over themain statement of the topic being discussed.” This preference mayhave been driven by the task (i.e. reading to prepare for a test),as P15 noted, “the more that [highlights] pertained to technicalterminology as opposed to whatever the overall theme. . . the moreI found it useful to answer the questions.”A theme that arose in the interviews is the tension between thevalue of highlights for focusing attention and the negative potentialfor distraction. P15 expressed this well: “[I] found it problematicbecause it would try and grab your attention to something thatI would try and think, ‘oh, that should be relevant. I should payattention to this,’ but it’s in the middle of something that A) hasno context — you don’t have any frame of reference to try andunderstand that and B) it kind of disorients the natural flow of yourreading, because you get drawn to this.”Participants identified intrinsic and extrinsic factors that affectedhighlight use. Chief among the former is the type of content high-lighted, notably the distinction between thematic and fact-basedhighlighting. Other factors included location and coverage. Issuesarose with highlights of partial sentences or those situated mid para-graph, due to a lack of context. Two extrinsic factors were noted:time, and prior knowledge. Many participants expressed that theusefulness of highlights increased under time pressure, when they“wouldn’t have enough time to read the entire thing word for word”[P4]. P3 noted that highlights “really helped guide my attentionto where, probably, the bulk of the information for the quiz wasgoing to be.” However, some were disappointed when they usedthis technique and the answers were not among the highlightedtext. A small number of participants indicated that when they aremore familiar with the content and vocabulary of a text, they wouldhave less use for highlights as a guiding or structuring feature.JCDL ’18, June 3–7, 2018, Fort Worth, TX, USA Samuel Dodson, Luanne Freund, and Rick Kopak5 DISCUSSION & CONCLUSIONResults clearly indicate that there are negative effects of exposingreaders to poor quality highlights in this type of time limited readingtask. This adds evidence to earlier findings [18]. Comprehensionscores were lower in the INAPP condition, although only the surfacelevel test was affected. Given that the APP highlights were designedto focus attention on the main themes and concepts, we expectedto see an effect on conceptual understanding as well, but this wasnot the case. Readers with higher levels of interest were betterable to reach a deeper conceptual understanding, perhaps becausethe interest enabled them to focus their attention even in the faceof inappropriate highlights. These results provide support for theconclusion in [4], that highlights serve better as signaling devicesthan as scaffolding for text comprehension.We found that most participants were aware of the quality vari-ation, and some clearly attended less to the INAPP highlights. Thissuggests that post-secondary readers have the capacity to recognizeand ignore less useful highlights that they may encounter in socialreading systems. The interview data suggests that ignoring suchhighlights was not always easy, and that the effort needed to assessand ignore distracting highlights might subtract from the effortavailable to engage in deep reading of challenging texts.We expected highlights would have more value for FDs, whotypically benefit from external structure. However, results showno clear differences between FD and FI readers. FDs did spendmore time viewing highlights in the APP conditions than in theINAPP condition, while FIs did not spend any more time in thehighlights than the rest of the text, perhaps because they had littleneed of them. This brings into question the hypothesis posed in [3]that FIs perform badly when faced with inappropriate highlights,because they spend valuable time trying to make sense of them.Interestingly, our initial analysis of the interview data suggests thatFDs and FIs may have used highlights differently, with FDs usingthem predominantly to focus on factual information and FIs to gaina general conceptual understanding.Clearly time is a key factor in the use of highlights, but moreresearch needs to be done to determine the extent to which high-lights can serve to increase reading efficiency, as suggested in [6]. Acentral finding of this work, is that for social highlighting to be ef-fective, systems should support diverse approaches to highlighting.While the thematic and factual approached identified here are notnew [e.g., 12], results suggest that there may be both personal andtask-based reasons to adopt one or the other. We can conclude thatin social reading systems, including digital libraries, there can bebenefit in creating guidelines for shared highlighting, to maximizepotential benefits.Highlight quality was shown to affect surface level text com-prehension, although readers were able to assess the quality ofhighlights and to adjust their reliance upon them accordingly. Re-sults indicate that readers approach highlighting differently. Assuch, there is danger in adopting a one size fits all approach inwhich shared highlights are aggregated and displayed to readers. Atask-based approach that adds information to highlights and allowsreaders to know why a highlight was created, by whom, or forwhat task, may be valuable, but the negative potential of distractionneeds to be addressed. We fully acknowledge the limitations of thiswork, which is small-scale, lab-based and therefore non-naturalistic.The time constraint on readers was also quite severe and resultsmay not generalize to more relaxed or lengthy reading tasks. Fu-ture work will move towards organic studies of social reading andlearning systems in use in university courses to validate and extendthese findings.REFERENCES[1] Maristella Agosti and Nicola Ferro. 2007. A formal model of annotations of digitalcontent. ACM Transactions on Information Systems 26, 1 (2007), 244–255.[2] Ed H Chi, Michelle Gumbrecht, and Lichan Hong. 2007. Visual foraging ofhighlighted text: an eye-tracking study. In Human-Computer Interaction: HCIIntelligent Multimodal Interaction Environments. Springer, 589–598.[3] Samuel Dodson, Luanne Freund, and Rick Kopak. 2017. Do Highlights AffectComprehension?: Lessons from a User Study. In Proceedings of the 2017 Conferenceon Conference Human Information Interaction and Retrieval. ACM, New York, NY,381–384.[4] John Dunlosky, Katherine A Rawson, Elizabeth J Marsh, Mitchell J Nathan, andDaniel T Willingham. 2013. Improving students’ learning with effective learningtechniques: promising directions from cognitive and educational psychology.Psychological Science in the Public Interest 14, 1 (2013), 4–58.[5] Robert L Fowler and Anne S Barker. 1974. Effectiveness of highlighting forretention of text material. Journal of Applied Psychology 59, 3 (1974), 358–364.[6] Luanne Freund, Rick Kopak, and Heather O’Brien. 2016. The effects of textualenvironment on reading comprehension: implications for searching as learning.Journal of Information Science 42, 1 (2016), 79–93.[7] Anis Kalboussi, Omar Mazhoud, and Ahmed Hadj Kacem. 2016. Functionalitiesprovided by annotation systems for learners in educational context: an overview.International Journal of Emerging Technologies in Learning 11, 2 (2016).[8] Nancy Kaplan and Yoram Chisik. 2005. In the company of readers: the digitallibrary book as "practiced place". In Proceedings of the 5th ACM/IEEE-CS JointConference on Digital Libraries. ACM, New York, NY, 235–243.[9] Walter Kintsch. 1998. Comprehension: a paradigm for cognition. CambridgeUniversity Press, Cambridge, United Kingdom.[10] Jacqueline Leighton and Mark Gierl. 2011. The learning sciences in educationalassessment: the role of cognitive models. Cambridge University Press, Cambridge,United Kingdom.[11] Robert F Lorch, Jr. 1989. Text-signaling devices and their effects on reading andmemory processes. Educational Psychology Review 1, 3 (1989), 209–234.[12] Catherine C Marshall. 2009. Reading and writing the electronic book. Morgan &Claypool Publishers, San Rafael, CA.[13] Samuel Messick. 1976. Individuality in learning. Jossey-Bass Publishers, SanFrancisco, CA.[14] Ilia A Ovsiannikov, Michael A Arbib, and Thomas H McNeill. 1999. Annotationtechnology. International Journal of Human-Computer Studies 50, 4 (1999), 329–362.[15] Matthieu Ricard, Antoine Lutz, and Richard J Davidson. 2014. Mind of themeditator. Scientific American 311, 5 (2014), 38–45.[16] Judith C Roberts and Keith A Roberts. 2008. Deep reading, cost/benefit, and theconstruction of meaning: Enhancing reading comprehension and deep learningin sociology courses. Teaching Sociology 36, 2 (2008), 125–140.[17] Jean-Francois Rouet. 2006. The skills of document use: from text comprehension toweb-based learning. Psychology Press, Hove, United Kingdom.[18] Vicki L Silvers and David S Kreiner. 1997. The effects of pre-existing inappropriatehighlighting on reading comprehension. Literacy Research and Instruction 36, 3(1997), 217–223.[19] Michele L Simpson and Sherrie L Nist. 1990. Textbook annotation: an effectiveand efficient study strategy for college students. Journal of Reading 34, 2 (1990),122–129.[20] Keith C Summa and FredW Turek. 2015. The clocks within us. Scientific American312, 2 (2015), 50–55.[21] Craig Tashman and W Keith Edwards. 2011. Active reading and its discontents:the situations, problems and ideas of readers. In Proceedings of the SIGCHI Confer-ence on Human Factors in Computing Systems. ACM, New York, NY, 2927–2936.[22] Ludo Verhoeven and Charles Perfetti. 2008. Advances in text comprehension:model, process and development. Applied Cognitive Psychology 22, 3 (2008),293–301.[23] Herman A Witkin, R B Dyk, H F Fattuson, Donald R Goodenough, and S A Karp.1962. Psychological differentiation: studies of development. Wiley, New York, NY.[24] Herman A Witkin and Donald R Goodenough. 1977. Field dependence revisited.ETS Research Bulletin Series 1977, 2 (1977), i–53.[25] Joanna Wolfe. 2008. Annotations and the collaborative digital library: Effects ofan aligned annotation interface on student argumentation and reading strategies.International Journal of Computer-Supported Collaborative Learning 3 (2008), 141–164.


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items