@prefix vivo: . @prefix edm: . @prefix ns0: . @prefix dcterms: . @prefix dc: . @prefix skos: . vivo:departmentOrSchool "Education, Faculty of"@en, "Language and Literacy Education (LLED), Department of"@en ; edm:dataProvider "DSpace"@en ; ns0:degreeCampus "UBCV"@en ; dcterms:creator "Arthur, Peter J."@en ; dcterms:issued "2009-03-09T13:15:46Z"@en, "2009"@en ; vivo:relatedDegree "Doctor of Philosophy - PhD"@en ; ns0:degreeGrantor "University of British Columbia"@en ; dcterms:description "The purpose of this study was to determine whether in the context of an article-critiquing assignment in a teacher education course, access to Open Journal Systems’ Reading Tools, significantly enhances university students’ 1) comprehension, 2) ability to evaluate the quality of the article, and 3) level of confidence in using the article, critically or supportively, as part of their present work as students at university or future work as teachers in a school. The sample for this experimental design consisted of 75 pre-service teacher education students who comprised two sections of the compulsory course entitled “Information Communication Technology for Secondary Teachers” were randomly assigned to either have access to the Reading Tools or no access to complete an authentic assignment as part of the course. As part of the assignment students were asked to create a list of 10-12 “talking points” for each of four articles prior to the class in which they would use the points as a basis of discussing the article with their group. These “talking points” were then marked by two independent markers using a rubric to arrive at a “comprehension” and “critique score.” An article usability score was based on survey questions that all students answered after completing each article assignment. Students in the treatment groups were compared to students who didn’t have access to the Reading Tools. Results of the mixed design ANOVA used to analyze the data indicated there were no significant differences found between the two groups of students in the areas of comprehension, ability to critique, or article utilization. However, student survey feedback indicated a positive perception for the Reading Tools ability to provide value to the online research reading environment. Reasons are considered for why the tools offered no advantage to search-wise students, while further studies are proposed for the development of reading tools."@en ; edm:aggregatedCHO "https://circle.library.ubc.ca/rest/handle/2429/5733?expand=metadata"@en ; dcterms:extent "4018793 bytes"@en ; dc:format "application/pdf"@en ; skos:note " Reading Tools: The Enhancement of an Online Scholarly Research Environment by PETER ARTHUR B.Ed., University of Saskatchewan, 1988 M.Ed., University of Saskatchewan, 2000 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY in THE FACULTY OF GRADUATE STUDIES (Language and Literacy Education) THE UNIVERSITY OF BRITISH COLUMBIA (Vancouver) March 2009 © Peter Arthur, 2009 ii ABSTRACT The purpose of this study was to determine whether in the context of an article-critiquing assignment in a teacher education course, access to Open Journal Systems’ Reading Tools, significantly enhances university students’ 1) comprehension, 2) ability to evaluate the quality of the article, and 3) level of confidence in using the article, critically or supportively, as part of their present work as students at university or future work as teachers in a school. The sample for this experimental design consisted of 75 pre-service teacher education students who comprised two sections of the compulsory course entitled “Information Communication Technology for Secondary Teachers” who were randomly assigned to either have access to the Reading Tools or no access to complete an authentic assignment as part of the course. As part of the assignment students were asked to create a list of 10-12 “talking points” for each of four articles prior to the class in which they would use the points as a basis of discussing the article with their group. These “talking points” were then marked by two independent markers using a rubric to arrive at a “comprehension” and “critique score.” An article usability score was based on survey questions that all students answered after completing each article assignment. Students in the treatment groups were compared to students who didn’t have access to the Reading Tools. Results of the mixed design ANOVA used to analyze the data indicated there were no significant differences found between the two groups of students in the areas of comprehension, ability to critique, or article utilization. However, student survey feedback indicated a positive perception for the Reading Tools ability to provide value to the online research reading environment. Reasons are considered for why the tools offered no advantage to search-wise students, while further studies are proposed for the development of reading tools. iii TABLE OF CONTENTS Abstract ................................................................................................................................... ii
 Table of Contents................................................................................................................... iii
 List of Tables......................................................................................................................... vii
 List of Figures ........................................................................................................................ ix
 Acknowledgements ..................................................................................................................x
 Chapter 1 Introduction and Overview of the Study ............................................................1
 Introduction … ......................................................................................................................1
 Background ….......................................................................................................................4
 History of Scholarly Journals ............................................................................................5
 New Technological Innovations Enhance Scholarly Communication ..............................6
 Article Navigational Linking .............................................................................................8
 Linking to Related Material ...............................................................................................8
 Reader, Author and Colleague Interactivity ....................................................................13
 Alerts................................................................................................................................16
 Citation Maps...................................................................................................................17
 Social Bookmarking ........................................................................................................18
 History of Online Journal Tools ......................................................................................19
 Reading Tools Description ..............................................................................................23
 Problem …….......................................................................................................................24
 Significance of the Study.....................................................................................................25
 Overview of Methodology...................................................................................................26
 Organization of Dissertation................................................................................................27
 Chapter 2 Review of the Literature .....................................................................................28
 iv Introduction …. ...................................................................................................................28
 Chapter Structure .................................................................................................................29
 Criteria for Identification and Selection of Studies .............................................................30
 Definitions ….. ....................................................................................................................31
 Theoretical Perspectives ......................................................................................................32
 Print Verses Hypertext.........................................................................................................33
 Nonlinearity .....................................................................................................................33
 Advantages and Disadvantages of Nonlinear Hypertext .................................................35
 Variables Impacting Knowledge Acquisition in Nonlinear Reading Environments...........41
 Reader / Learner Variables ..............................................................................................41
 System Variables .............................................................................................................47
 Discussion ….......................................................................................................................59
 Theoretical and Methodological Issues ...........................................................................60
 Research Gap Identification.............................................................................................61
 Chapter 3 Methods ................................................................................................................64
 Introduction … ....................................................................................................................64
 Research Perspective ...........................................................................................................64
 Research Question ...............................................................................................................67
 Hypotheses …......................................................................................................................68
 Sample and Setting ..............................................................................................................69
 Protection of Subjects ......................................................................................................70
 Research Variables ..............................................................................................................70
 Research Instruments...........................................................................................................72
 Assessment Rubric...........................................................................................................72
 Survey I and II .................................................................................................................79
 v Data Collection Procedures .................................................................................................82
 Methods of Data Analysis ...................................................................................................90
 Other Analysis .................................................................................................................94
 Limitations …......................................................................................................................95
 Chapter 4 Results: Do the Reading Tools Make A Difference?........................................97
 The Reading Tools and Comprehension .............................................................................99
 Hypothesis 1 … ...................................................................................................................99
 The Reading Tools and Article Evaluation/Critique .........................................................102
 Hypothesis 2 … .................................................................................................................102
 The Reading Tools and the Student’s Perceived Ability to Utilize the Article.................107
 Hypothesis Three ...............................................................................................................107
 Hypothesis 3A................................................................................................................107
 Hypothesis 3B................................................................................................................111
 Other Results .. ..................................................................................................................114
 Chapter 5 Discussion: Do the Reading Tools Make A Difference? ................................124
 Introduction … ..................................................................................................................124
 Discussion ….....................................................................................................................125
 Introduction....................................................................................................................125
 Literature Review Response ..........................................................................................126
 Discussion of Results.........................................................................................................129
 Student Interaction with the Reading Tools: Access versus Actual Use .......................131
 Did Students Use the Tools?..........................................................................................132
 Summary of Discussion .................................................................................................141
 Limitations of the Study ....................................................................................................142
 Future Research .................................................................................................................142
 vi References ............................................................................................................................146
 Appendix A Survey I ...........................................................................................................157
 Appendix B Survey II..........................................................................................................158
 Appendix C Assignment ....................................................................................................162
 Appendix D Assignment Rubric ........................................................................................164
 Appendix E Student Consent Form...................................................................................166
 Appendix F Research Ethics Approval .............................................................................169
 vii LIST OF TABLES Table 3.1 Landis and Koch (1977, p. 165) Kappa Statistics and Strength of Agreement.......77
 Table 3.2 Inter-rater Reliability on the Scoring of the Assignment - Average ICC Values....78
 Table 3.3 Article Descriptions for Trials 1 - 4 ........................................................................86
 Table 4.1 Comprehension Mean and Standard Deviation Scores by Group.........................101
 Table 4.2 Comprehension Scores Significance between Trials ............................................102
 Table 4.3 Critique Means and Standard Deviation Scores by Group....................................104
 Table 4.4 Critique Scores Significance between Trials.........................................................105
 Table 4.5 Critique Score Trial and Group Relationships ......................................................106
 Table 4.8 Hypothesis 3A Trial Comparisons within the Without Reading Tool Group.......110
 Table 4.9 Hypothesis 3A Simple Main Effects of Group within Trial .................................111
 Table 4.10 Hypothesis 3B Means and Standard Deviation Utilization Scores by Group.....113
 Table 4.11 Hypothesis 3B Significance between Trials........................................................114
 Table 4.12 Number of Participants Printing the Article - Survey II Part A Students Online Versus Print Reading Habits .................................................................................................115
 Table 4.13 Survey II Part A Number of Articles Students Printed .......................................115
 Table 4.14 Survey II Part A Reasons Students Printed .........................................................116
 viii Table 4.15 Survey II Part A Reasons Students Read Online ................................................116
 Table 4.16 Survey II Part B Question 1 Results....................................................................118
 Table 4.17 Reading Tools Access Group Perceptions of Reading Tools Contribution ........119
 Table 4.18 Reading Tools Impact on Ability to Focus .........................................................120
 Table 4.19 Survey II Part B Question 3 Reading Tool Usefulness .......................................121
 Table 4.20 Survey II Part B Optional Open Comments – All Comments Reported.............122
 Table 5.1 Participant Reading Tool Access Group Descriptive Results Survey II Part B Question 2..............................................................................................................................129
 Table 5.2 Reading Tools Access Group Perceptions of Reading Tools Contribution ..........130
 Table 5.3 Reading Tools use from the Reading Tools Group...............................................132
 Table 5.4 Reading Tools Group Rating of Each Tool...........................................................138
 Table 5.5 Reading Tool Use by Course Section ...................................................................139
 ix LIST OF FIGURES Figure 1.1 Open Journal Systems’ Reading Tools ....................................................................3
 Figure 1.2 Examples of Online Journal Publishing Toolbars ...................................................7
 Figure 1.3 PLoS One Scholarly Journal Article Internal Navigation Toolbar ..........................8
 Figure 1.4 HighWire Press Tools and Services – Forward Referencing.................................10
 Figure 1.5 Science Direct Tools and Services – Forward Referencing...................................11
 Figure 1.6 Journal of Biological Procedures - Online References ..........................................12
 Figure 1.7 PLoS ONE Discussion Tools.................................................................................15
 Figure 1.8 PLOS One Note Tool Provides Reader Interactivity .............................................16
 Figure 1.9 HighWire Press Citation Map Example.................................................................18
 Figure 1.10 HighWire Press Social Bookmarking Tools Screenshot .....................................19
 Figure 1.11 HighWire Press Content Box (Tools) 2004 and 2008 .........................................20
 Figure 1.12 Open Journal Systems Reading Tools 2004 and 2008.........................................22
 Figure 2.1 Linearity Lies on a Continuum ..............................................................................34
 x ACKNOWLEDGEMENTS I would like to extend my deepest gratitude to all the people who made the completion of this dissertation possible. I would like to especially thank my dissertation committee members who contributed greatly to the success of this endeavour. A special thanks goes to my supervisor, Dr. John Willinsky. His guidance and supervision were most valuable to me and his mentoring provided the insight that allowed me to explore technology and its use in online scholarly environments more thoroughly. A student could not have wished for a better supervisor. I would also like to thank Dr. Marlene Asselin for her positive support and insightful comments that provoked thought and helped focus my writing. I also acknowledge Dr. Robert Campbell who encouraged, and supported my doctoral work. I especially appreciated the many hours he spent going through my dissertation during the final phases of editing the work. Finally, I dedicate this dissertation to my partner Melissa, and my children Mistaya, Tabitha, and Penelope, all of whom stood by me and sacrificed so much. I could never have done this without their support and understanding. 1 CHAPTER 1 INTRODUCTION AND OVERVIEW OF THE STUDY Introduction The Internet can now support systems of “Reading Tools” that can enable readers of a research article to connect the article they are reading not only to related studies, but just as easily to current newspaper articles and government reports on the same topic, to sites with instructional materials for teaching the topic and to online forums where the topic is informally debated. --Willinsky (2006, p. 13) Willinsky (2006) points out how online technologies may be used to potentially enhance critical engagement with scholarly research and ultimately improve the online reading environment. Up until recently, academic journals placed articles online primarily to increase access; however, the creation of online reading tools has significantly changed the online scholarly environment. HighWire Press, PubMed, Open Journal Systems, ScienceDirect, Public Library of Science (PLoS) and other online scholarly publishing systems now provide “tools” that reside next to or within online scholarly journal articles. These tools offer readers convenient links to relevant resources. Given the current growth of the number of academic journals publishing online, in conjunction with the rapid development of online information retrieval tools, a new way has emerged in reading online scholarly journals. Therefore it is important to investigate the degree to which these online technologies improve or impede reading. This dissertation is a report on a quantitative study that assessed the degree to which the Open Journal Systems’ (journal management and publishing system) “Reading Tools”, 2 which reside next to an online scholarly journal article and provide contextual resources (as depicted in Figure 1.1), help university students critique an article for an in class assignment. Specifically, this study determines whether access to Reading Tools in the context of an article critiquing assignment significantly enhances university students’ 1) comprehension; 2) ability to evaluate the quality of the article; and 3) level of confidence in using the article, critically or supportively, as part of their present work as students at university or in their future work as teachers in a school. 3 Figure 1.1 Open Journal Systems’ Reading Tools This study established, through a natural experimental design involving a class assignment in which students were required to read research articles, that providing access to Reading Tools does not have any significant effect on students’ comprehension of the journal articles studied, ability to critique them, or their perception of being able to use the articles for their in class assignment. However, there was evidence that students used Reading Tools 4 and noted their usefulness in the course of completing their assignment. The results of this study will be discussed in detail in Chapter 5. Presented in this chapter is the background of the study, the problem investigated, an outline of the methodology, the study’s significance and provides an overview of the five chapters of the dissertation. Background In today’s knowledge society, the primary economic resource is no longer capital, or natural resources nor is it labour, it is knowledge; and the fundamental status of knowledge in our economy seems to be a certainty for the foreseeable future. Transfer of knowledge is important in knowledge based environments. Nonaka (1994) suggests that knowledge-based companies recognize the importance of intellectual capital and create environments that facilitate the transfer of knowledge within the company. The managers knowledge-based organizations recognize that the most valued assets are the “skills, competencies, and creativity of its staff, not its buildings, equipment, and manufacturing facilities. For example the worth of a software company, as reflected in its stock price, includes an estimate of the potential human creativity and innovation that becomes embodied in the software the company creates” (Schwen, Kalman, Hara, & Kisling,1998, p. 76). Choo et al., (2006) recognizes the importance of a knowledge based economy when he states that businesses are concerned with the reuse of explicit knowledge that is codified, stored and disseminated through the use of information technology, electronic document systems, and formal procedures. 5 Similarly, universities are concerned with the creation and dissemination of knowledge in the form of research and scholarship. One of the most important vehicles for disseminating this knowledge is the scholarly journal. There are now estimated to be more than 40,000 journals worldwide, representing a substantial site of production for knowledge (Tenopir, 2004). As Rowland (1997) points out scholars agree that scholarly journals serve four principle functions: quality control, dissemination of information, archiving, and academic credit. History of Scholarly Journals Modern day scholarly communication originated in the 1640s with a group of scholars, named the Invisible College (later renamed the Royal Society), who regularly met at Oxford University. Initially private letters served as the principle communication tool; however, in 1665 the Royal Society began publishing the first scholarly journal, titled the Philosophical Transactions of the Royal Society of London. This journal was created to disseminate their letters more widely. This innovation also provided a way to archive scholarly communication and bring all of the Society’s members into a single forum (Peek & Newby, 1996). The Royal Society’s scholarly journal significantly influenced scholarly communication. Since the 17th century, scholarly journals have served as the principle medium for communication among scholars (Kaur, 2007; Harter, 1998). Over the years, the publishing agent shifted from academic societies to for-profit and non-profit publishers (Peek & Newby, 2000). Another significant shift occurred in the 1980s when e-journals were first piloted. However, lack of standards and the high costs of delivery and storage were initial concerns. Adonis, one of the first e-journal experiments, appeared in pilot form at the end of 6 the 1980s and was subsequently launched as a commercial product in 1991(Kaur, 2007). Then in 1992 the first peer-reviewed electronic, full text e-journal that included graphics, The Online Journal of Current Clinical Trials, began publication (Keyhani, 1993). A new paradigm was emerging in Scholarly communication. Kaur (2007) noted the tremendous potential of the World Wide Web when it was introduced in the 1990s and Oppenheim, Greenhalgh, and Rowland (2000) highlighted how the Internet could provide the opportunity for transforming the way scholars work and communicate. In the 1990s the features of the first e-journals—that is, online journals—varied greatly. Some journals appeared in electronic form only. Other online journals were exact copies of their print counterparts. These online journals presented scanned images of the print version and were made available as PDF files (Mogge, 1999). However, some online journals also began to provide features that were not available in their print versions and these features are discussed in the following section. New Technological Innovations Enhance Scholarly Communication A number of online scholarly journals have evolved to become something significantly different from print journals (Luzon, 2007). Editors of scholarly journals have leveraged technology to provide additional value added features that enhance the online scholarly journal environment. Most journals place these value added services and tools on an accessible toolbar adjacent to the online article. Three examples are presented in Figure 1.2. These enhancements include internal hyperlinks that assist with navigation, links to related materials, tools for reader and author interactivity, publishers’ alerts to readers, citation maps, and social book marking tools which are discussed under separate subheadings below. 7 Figure 1.2 Examples of Online Journal Publishing Toolbars 8 Article Navigational Linking Some online scholarly journals provide hyperlinks within the article to enhance reader navigation between various parts of the article. For example, some journals provide hyperlinks from the citation in the body to the corresponding reference at the end of the article. Similarly, some journals provide a toolbar of links for quick access to the beginning of various sections of the article (Figure 1.3). This type of tool provides the reader with ease of navigation and commonly used in web page design. Figure 1.3 PLoS One Scholarly Journal Article Internal Navigation Toolbar Linking to Related Material Journal editors and developers have also created many ways for readers to find additional related materials, such as forward referencing, active references, links to other articles by the article’s author, and links to other articles on the same topic or related topics. Forward referencing provides links to articles published later that cite the paper being read (Boyce, Owens, & Biemesdefer, 1997). While traditional citations allow the reader to follow the knowledge backward to the author’s source, forward referencing provides the reader with 9 the ability to follow the development of a topic forward to new articles that have cited the current article. As Luzon (2007) put it, forward references enable readers to establish a network of knowledge. An example of an article repository that uses forward referencing is HighWire Press, which provides two forward referencing options: “Citing Articles via Highwire” and “Citing Articles via Google Scholar” (Figure 1.4). Both tools display links to articles that have cited the article currently being read. Science Direct also has two forward referencing options: “Cited by” and “Cited By in Scopus.” “Cited by” provides links to other Science Direct journal articles that cite the current article. “Cited By in Scopus” provides links to documents in Scopus’s large abstract and citation database that cite the current article (Figure 1.5). 10 Figure 1.4 HighWire Press Tools and Services – Forward Referencing 11 Figure 1.5 Science Direct Tools and Services – Forward Referencing Eason’s (2004) survey of Zetoc (the British Library’s Electronic Table of Contents of around 20,000 journals) readers found that the most important service users wanted was to move from a cited reference in an article to the corresponding full text through a seamless service that could be easily learned. This service can be provided efficiently through reference linking, which provides the ability to link automatically from the references in one article to the articles themselves (Beit-Arie et al., 2001). The Journal of Biological Procedures Online provides reference links beside each citation using either PubMed or CrossRef to link to the corresponding article (Figure 1.6). 12 Figure 1.6 Journal of Biological Procedures - Online References For readers, then, one of the most useful features of online scholarly journal article environments is the ability to link to other online journal articles (Luzon, 2007). Virtually all online journals incorporate search tools that give readers the ability to search within an article, across the entire journal, across all journals in a publisher’s database (Luzon, 2007), or even to any other online journal article. Online journals achieve this within the journal by incorporating search tools in the online journal publishing system, and leverage search tools 13 such as Google Scholar, PubMed, ERIC, and others so that related materials in other journals can be found. Some online journals have tools to search for other articles by the same author within a particular publishing system. For example, Biological Procedures Online provides an author search that queries PubMed (Figure 1.2). A broader tool that searches across the Internet may also be used to find additional articles by the author. This is exemplified in HighWire Press where a tool called “articles by” is available. This tool searches for other articles by the same author in Google Scholar (Figure 1.4). Similarly, journals frequently provide the ability to find related materials within a particular publishing system, or through external search tools. Biological Procedures Online Journal provides a link to search for related works within PubMed. Similarly, HighWire Press provides a “Search for Related Materials” through Google Scholar. Synergy’s online publishing offers a “related articles” tool that provides readers with a list ordered by relevancy of other articles from other journals within Synergy’s online publishing system. Reader, Author and Colleague Interactivity In addition to making it easier for a researcher to connect with information, online journals can also facilitate connections between people. Online journals can now provide tools for authors and readers to interact, using live chats, forums, email links to article authors (Luzon, 2007), the ability to add related links, as well as providing options for making comments or posing questions on an article. The link to email an author (or authors) of an article provides an easy way for readers to send feedback to the author and for the author and reader to interact. Some journals simply provide the author’s email address at the top of the article, while other online journals 14 provide a button or tool on the side that when clicked opens the reader’s email client with a new email message addressed to the author. In addition, many journals provide opportunities for readers to interact with authors and other readers through comments. For example, the Journal of Interactive Media in Education (JiME) enables readers to click on any comment icon in order to browse or contribute to the discussion. Similarly, the Ecology and Society Journal provides readers a link inviting them to respond to the article. The responses, if accepted by the journal editor, will be hyperlinked to the article, allowing other readers to read colleagues’ responses. The journal PLoS ONE provides the ability to start a discussion on the article (Figure 1.7). It has tools to add notes to the text of the article and to make general comments. Other readers can then add to these notes and comments (Figure 1.8). 15 Figure 1.7 PLoS ONE Discussion Tools 16 Figure 1.8 PLOS One Note Tool Provides Reader Interactivity Alerts Many online journals provide the option for readers to set up “alerts” that notify a reader when an event happens. Lapid and Kung (2006) outline some of the alert types, including publication alerts, tables of contents or “eTOCs,” content alerts, citation alerts, author alerts and saved search alerts. Publication alerts provide a link to the journal or current issue’s table of contents. Tables of content alerts send along the full table of contents with links to abstracts and full-text versions of each article. Content alerts send along links to new articles on a specific topic that the reader specifies. The reader can set up content alerts by setting keywords of interest within a new article’s title, keywords, or abstract. The journal will then send links to new articles on a specific topic based on the specified keywords, and citation alerts provide readers with notification when one of their specified articles is cited. 17 Author alerts provide notification when a selected author publishes new material. Finally, saved search alerts retrieve a prior search each time the publisher’s database is updated, and send a notification if there are new relevant results. HighWire Press has an additional alerts feature called CiteTrack Alerts. CiteTrack Alerts sends an email to readers about new journal content based on reader specified criteria such as author, keyword, or referenced articles (Cuddy, 2005). It also will send an email to the reader when a particular article is cited, if a correction in the article is posted, when an article matching search criteria is published, or when a specified journal is published. In the last case, CiteTrack Alerts sends the journal’s table of contents in the email to the reader. HighWire will also send links through PDA Channels to the reader’s Palm OS or Pocket PC handheld. At PubMed Central readers can sign up for MyNCBI accounts to save searches and receive alerts for new content, authors, journals and saved searches (Lapid and Kung, 2006). Moreover, Science Direct provides readers with a “Save as Citation Alert” tool that sends readers an email when the article is cited by new documents added to Science Direct. Citation Maps Another technology that helps readers organize and manage sources is citation maps. HighWire press provides readers with a citation map for the articles they are reading online. The map is a graphical representation of the articles citing or cited by the selected article. The map is based on references found in articles of HighWire publishing journals. The citation map may assist readers with developing reading lists on a topic, and with understanding the development of a topic (Figure 1.8). 18 Figure 1.9 HighWire Press Citation Map Example Social Bookmarking Finally, online journals are providing convenient links to social bookmarking services such as del.icio.us and Digg that allow readers to save and categorize personal collections of bookmarks to share with others. HighWire Press provides these links either at the bottom of the article (Figure 1.10) or at the bottom of their side content box. 19 Figure 1.10 HighWire Press Social Bookmarking Tools Screenshot History of Online Journal Tools Online journaling tools continually change and evolve as new technologies become available. HighWire Press was one of the first publishers to create tools and services in a toolbar adjacent to the online article. They called their toolbar of services and tools the “content box” (Sack, 2005). HighWire Press tools vary according to the journal, since the journal editor chooses which tools to place in his or her journal’s content box. Over time, the look and types of available tools evolved. For example, HighWire Press now leverages Google Scholar to search for other articles written by an author or for related articles. In addition, they have added social bookmarking tools that make it easy to bookmark and share articles. HighWire Press has also enhanced the number and type of Alert services and the types of forward referencing available (Figure 1.11). 20 Figure 1.11 HighWire Press Content Box (Tools) 2004 and 2008 21 Open Journal Systems has developed its Reading Tools through the Public Knowledge Project in an effort toward improving the scholarly and public quality of academic research through the development of innovative online publishing and knowledge- sharing environments. The Reading Tools drew on the example set by HighWire Press’ tools. The original Reading Tool was called a Research Support Tool, and it displayed 10 to 15 links depending on the disciplinary version selected by the journal managers (Willinsky, 2004). Through research this tool evolved to a tool that is better organized, and provides new tools for searching for related materials (Figure 1.12). Willinsky (2004), Willinsky and Quint-Rapoport (2007), and Blake, Leitch, Siemens, and Willinsky (2007) focused their research on determining to what degree the Reading Tools or Pub Med’s tools and services enhance readers’ experience in online scholarly research environments. The results of these studies were used to enhance the Reading Tools. Willinsky (2004), Willinsky and Quint- Rapoport (2007), and Blake et al.’s (2007) research will be further discussed in chapter 2. 22 Figure 1.12 Open Journal Systems Reading Tools 2004 and 2008 23 Reading Tools Description Willinsky (2006) proposed that online scholarly research environments may be enhanced by providing links to additional resources that add context to the research article. He argues that providing links will improve the reader’s quality of critical engagement with the online research. Based on models of reader links from HighWire Press (Figure 1.2), PubMed, and others, Willinsky (2006) has created a new version of Reading Tools (Figure 1.1) as part of the Open Journal System’s journal management and publishing system. The new version offers convenient links to additional resources beyond the typical set of links that other systems offer. For example, HighWire journals provide tools to support expert readers, where as the OJS Reading Tools are created to assist a wider range of readers. For example, inexperienced readers benefit from the definitions link that provides a convenient way of looking up any word that they are unsure of in the article. Moreover, the related items link helps readers establish a context for the article they are reading by providing access to relevant resources that they may not have considered, may not have known existed, or may not have been otherwise able to locate (Willinsky, 2006b). The “Reading Tools” (Figure 1.1) reside next to online scholarly journal articles and look similar to traditional bookmarks. At the top of the tools is an indicator that shows whether an article is peer reviewed, and below there is a set of links to items such as the article’s abstract, the journal’s review policy, information about the author, directions on how to cite the article, the article’s metadata, supplementary files, and a print version. There is also a definition tool that allows readers to look up any word in a free online dictionary. Another set of links connects to relevant materials. These links are organized under the heading “Related Items” and include Research Studies, Authors’ Other Works, Dissertations 24 and Theses, Government Web Sites, Press and Media Reports, and Instructional Sites (Willinsky, 2006a). Problem For the past twenty years, online reading environment research has focused on the factors that impact knowledge acquisition. The literature examined the treatment of what are regarded as the principal variables, including text structure and navigational aids (Jenkins, Corritore, & Wiedenbeck, 2003), disorientation (Shapiro & Niederhauser, 2004), user control (Zumbach, 2006), increased cognitive load (Niederhauser, Reynolds, Salmen, Skolmoski, 2000), Web experience, and expertise in the knowledge domain (Shapiro, 2004). Many researchers focused their efforts on how navigational aids such as menus, maps, links, and search engines may improve navigation for novices and experts in the knowledge domain and system, and consequently lower the negative effects disorientation and increased cognitive load have on the online reading experience (Jenkins et al., 2003; Brinkerhoff, Klein, & Koroghlanian, 2001; Hofman & van Oostendorp, 1999). The Reading Tools feature presents unique opportunities and challenges that have not been addressed in the literature. Research on the online scholarly research environment is in its infancy. Past research focused on traditional hypertext web sites and did not include the content or structure of formal peer reviewed research. Most importantly, the literature has not focused on tools that provide links designed to help readers find related contextual resources that may assist them in understanding an article. The problem this study will address is to assess to what degree access to the Open Journal Systems Reading Tools assists pre-service faculty of education university students with the following: 1) comprehension of an article; 25 b) ability to evaluate the quality of the article; and c) increases confidence in using the article, critically or supportively, as part of their level of confidence in using the article, critically or supportively, as part of their future work as teachers or university students. This will be done in the context of an authentic assignment that asks faculty of education university students to critique an article related to issues that will be discussed in class. Significance of the Study With the majority of research available online and a growing volume available to everyone via open access it is important that we leverage technologies to improve not only access but also the online scholarly reading environment. Currently, virtually every online scholarly environment has some type of tool that offers links in an effort to assist readers. The Open Journal Systems Reading Tools studied in this investigation were created to assist both novice and expert readers. This study builds on previous findings of the Public Knowledge Project studies by investigating a new target audience: that is, novices in the knowledge domain (i.e., university students). It uses a quantitative methodology to determine the degree to which access to an online “Reading Tool” providing contextual resources assists university students with critiquing an article as part of an assignment. The results of this study can be used to inform future development of online tools in online scholarly journals. This study provides insight into the usefulness of the Open Journal Systems Reading Tools for university students who are novice readers and how this group interacts with online research. In addition, it is recognized that online tools are developing at a fast pace. The Reading Tools were developed before tools such as Google Scholar arrived, although more recent versions of the tools include the use of Google Scholar. It is important 26 to determine whether students can complete a critiquing assignment without the tools by using other methods that may make the Reading Tools obsolete, such as accessing directly their academic library’s online portal, search engines, academic databases or Google Scholar. Overview of Methodology In a reaction to experimental research that relies on contrived learning environments, this study makes an effort to conduct the research in an authentic context. As described in detail in chapter 3, this study attempts to examine the research question in an authentic classroom setting that investigates students interacting with the Reading Tools for an assignment they would have completed with or without the research study. Seventy five participants in the course entitled “Information Communication Technology for Secondary Teachers” were randomly assigned to two groups. The experimental group was made up of 38 students who had access to the Reading Tools, while 37 students were part of a control who were, in effect, the Without Reading Tools group. However, the data from one of the participants in the Without Reading Tools group was excluded, since one of their assignments was missing. For Trial 1 both groups had access to the assigned article online, without the Reading Tools. For trials 2, 3, and 4, the Reading Tools group worked with an article that was accompanied by the Reading Tools and the Without Reading Tools group worked with the same article without the tools. As part of the assignment students were asked to create a list of 10-12 talking points for each article prior to the class in which they would use the points as a basis of discussing the article with their group. These talking points were then marked by two independent markers using a rubric (Appendix D) to arrive at a “comprehension” and “critique score.” The independent markers were not aware of which 27 group each student was in. After handing in the talking points and before discussing each article, students completed an online survey that asked, “How comfortable would you feel using this article in a class presentation, whether to criticize or draw on its perspective?” and “How comfortable would you feel using this article in working with other teachers in a school, whether to criticize or draw on its perspective?” These survey results were used to provide a quantitative score related to whether the Reading Tools increased the student’s perceived ability to utilize the article in their work in the schools. As described in Chapter 4, this data was analyzed using a mixed design Analysis of Variance (ANOVA) for all three hypotheses described in Chapter 3. Organization of Dissertation The remainder of the study is organized into five chapters, a bibliography, and appendixes in the following manner: Chapter 2 presents a review of the related research that precludes and demonstrates the need for the current study. Chapter 3 delineates the research design and methodology of the study, including the instrument used to gather the data, the procedures followed, and description of the sample selected for study are described. An analysis of the data and report of the findings are presented in Chapter 4. Chapter 5 contains the discussion of the findings, conclusions, limitations of this study, and recommendations for further study. The study concludes with a bibliography and appendixes. 28 CHAPTER 2 REVIEW OF THE LITERATURE Introduction The goal of this literature review is to examine the theoretical and empirical body of literature that provides the foundation for this study and contributes to its rationale. This investigation examines the degree to which access to Open Journal Systems’ Reading Tools assist university students with reading comprehension, with their ability to evaluate the quality of an online journal article, and with their confidence level using the article, critically or supportively, as part of their future work as teachers in a school or work as students at university. More specifically, this review examines the literature that investigates reader and hypertext variables that impacts knowledge acquisition in online information reading environments. The review highlights issues that pertain to reading hypertext in online environments and identifies what researchers consider the principal variables in online reading. These variables include: user control, disorientation, increased cognitive load, system experience and expertise in the knowledge domain, hypertext topology, navigational aids, and tools that enhance scholarly journal environments. Based on an understanding of reader and system variables, this review will inform the future development of nonlinear reading environments of electronic scholarly publishing systems. One of the major questions under investigation is which hypertext structures and navigational aids might help readers acquire knowledge more successfully. The knowledge gained will be used, according to one of the goals of this study, to identify research gaps which the current research study was intended to address. 29 Moreover, this literature review contributes a rationale for the current study by illustrating the importance of addressing the research problem, and by verifying that there is a lack of existing research which directly studies the problem. An examination is made of knowledge acquisition in online environments and how it may impact the effectiveness of Reading Tools. To date, researchers have conducted empirical and theoretical studies in a wide range of disciplines—notably, cognitive psychology, computer science, education, literacy studies, library and information science, and communications—about issues related to variables that impact knowledge acquisition in nonlinear online reading environments (Liu, 2005). Although the research has been conducted in a range of disciplines there is a common research theme which compares readers’ experience in nonlinear hypertext to their experiences in print. Specifically, the common research theme examine how various hypertext topologies, navigational aids, prior knowledge, and system experience affect a reader’s disorientation and cognitive load, and, ultimately, their knowledge acquisition. Chapter Structure In this chapter, the criteria for search procedures and literature inclusion for the literature review are identified. Definitions of key terms are also provided. After which the literature is analyzed. This analysis begins with the theoretical view of reading hypertext. Following this is a discussion of how reader and system variables impact knowledge acquisition while reading online. After this the Public Knowledge Project literature that investigates tools that enhance online scholarly journal reading environments is discussed. 30 Finally, general conclusions are offered, including how a current research gap has created a need for the current study. Criteria for Identification and Selection of Studies Three criteria guided the selection of studies for this review. The first criteria limited the selection to empirical and theoretical articles, conference proceedings, and book chapters published between 1987 and 2008. The start year of 1987 was chosen to coincide with Conklin’s article entitled “Hypertext: An Introduction and Survey.” In this landmark article, disorientation and cognitive overhead are identified as two variables that negatively impact online reading. Since then, many researchers have built upon Conklin’s research by investigating disorientation, cognitive load, text structure, and other variables that effect knowledge acquisition in online reading environments. The second criteria included articles for review on the basis of their relevance to a selection of variables that impact knowledge acquisition in nonlinear online reading environments. These variables include user control, disorientation, increased cognitive load, system experience and expertise in the knowledge domain, hypertext topology, navigational aids, and tools that enhance scholarly journal environments. The third criteria included general literature that examines the acquisition of knowledge in online information environments. To locate relevant literature, the following databases were searched: ERIC, PsycINFO, Academic Search Premier, Library, Information Science & Technology Abstracts (LISTA), Library and Information Science Abstracts (LISA), and Education Index Full Text. These databases were searched using combinations of the following search terms: “comprehension,” “reading,” “online,” “nonlinear text,” “web,” “hypertext,” 31 “disorientation,” “prior knowledge,” “hypermedia,” “user control,” “learning,” “knowledge acquisition,” “scholarly journals,” “journal environments,” “Reading Tools,” “Content Box,” “tools”, “HighWire Press,” and “cognitive load.” Google Scholar was also searched using similar search terms. When an article was found that met the criteria for selection, the citations were also checked, and these often led to other articles. The final sample consisted of 31 primary studies published in refereed journals, edited books, and conference proceedings. There are other studies that provided supporting information and these are presented and discussed where relevant. Definitions The terms reading online, hypertext, knowledge acquisition, and domain knowledge are used throughout this review. In much of the literature the terms hypermedia and hypertext are used interchangeably. Hypertext is generally defined as text that 1) is presented in a nonlinear structure that allows readers to control the pace of reading, 2) consists of nodes that are arranged in the form of a web page, and 3) can be selected dynamically in a nonlinear fashion (Cress & Knabel, 2003). Hypermedia is distinct from hypertext in that it combines the text with other media elements such as video, audio, and graphical representations. The common thread of these two terms is that both hypertext and hypermedia contain nodes (hyperlinks) that can be selected dynamically and allow the text to be read in a nonlinear way. For the purposes of this review, the term hypertext will be used with the assumption that it represents both hypertext and hypermedia. Many of the studies reviewed focus on how variables in online hypertext environments impact two other related concepts: knowledge acquisition and learning. 32 Knowledge acquisition and learning are often used synonymously in the literature. For the purpose of this review the term knowledge acquisition will be used and will encompass learning. The term domain knowledge refers to all known information related to a particular field of study, for example, all information related to one field, such as physics, mathematics, or psychology (Alexander & Jetton, 2000). In this review, the amount of domain knowledge a reader possesses will be included among the primary variables that impact knowledge acquisition in online reading environments. Theoretical Perspectives Bolter and Landow were among the very first theorists to examine the differences between hypertext and print. Bolter saw how new media are refashioned, he used the term “remediated,” to describe what happens when older media refashion themselves to answer the challenges of new media (Bolter & Grusin, 1999). Bolter (2001) saw the way our culture “reads” hypertext as a significant shift away from the way our culture has traditionally read print. Drawing from past literacy developments such as the printing press, he forecasted how hypertext would impact print and predicted that hypertext would “remediate” traditional linear print. However, Bolter also saw that print and hypertext (electronic writing) need to coexist. He asserts, “Print forms the tradition on which electronic writing depends, and electronic writing is that which goes beyond print. Print now depends on the electronic too, in the sense that printed materials find it necessary to compete against digital technologies in order to hold their readers” (Bolter, 2001, p.46). Landow (1997) presents a somewhat different view than Bolter. Landow looked at how hypertext had changed the respective roles 33 of reader and writer. He claimed that hypertext “obviously creates empowered readers, ones who have more power relative both to the texts they read and to the authors of these texts” (Landow, 1997). Overall, the work done by these two theorists recognized how hypertext empowers the reader and is actually changing or remediating traditional print. Print Verses Hypertext Nonlinearity The reading of text differs in print and online formats. The general view is that print suits linear reading while online hypertext suits nonlinear reading. Print, however, can also be nonlinear. For example, dictionaries and encyclopedias are meant to be read in a nonlinear fashion. One unique difference between linear and nonlinear text relates to the organization of the text and the manner in which it is read. Print consists of text organized in a linear manner and is generally designed to be read from the beginning to end, whereas the web is a nonlinear environment in which a reader may “easily choose to move through an article in any of a number of ways instead of a single order determined by the fixed structure” (Eveland & Dunwoody, 2002, p. 35). This nonlinear structure provides the reader with greater user control. Mishra, Spiro, and Feltovich, (1996) contrast linear and nonlinear text. In their view, the nature of linear text is bound and fixed, while “computer based hypertexts are nonlinear, unbounded, and dynamic. For example, they make possible fluid and nearly unlimited juxtaposition and linkage of elements, without regard to the physical location of the elements in contrast to conventional text” (p. 288). Theoretically, this gives all hypertext the potential to be linked to all other hypertext, which would give the reader the ability to read endlessly in any order he or she chooses. 34 When considering the differences between linear and nonlinear text it is important to consider that all printed text and hypertext have varying degrees of linearity/nonlinearity. Typical print text that is intended to be read from start to finish may be considered linear. However, there are online information sites that are also relatively linear. For example, online scholarly journal sites such as JSTOR, have taken print text and placed it online with very few links. JSTOR does not include any forward references, citation linking, links to online dictionaries, or links to related articles. In contrast, journals, such as those hosted by HighWire Press, have a dynamic hypertext environment that can be considered nonlinear due to the presence of many links that provide pathways and opportunities for readers to access related material and interact with colleagues and authors. Each specific online text has a varying degree of nonlinearity and could be represented somewhere on the continuum between linear and nonlinear (see Figure 2.1). The amount of nonlinearity has not been clearly quantified in the literature. It is important to note that the amount of nonlinearity has a direct impact on user control and, consequently, could create disorientation and cognitive load that may negatively impact a reader’s knowledge acquisition. Figure 2.1 Linearity Lies on a Continuum 35 Advantages and Disadvantages of Nonlinear Hypertext There are advantages as well as disadvantages to reading nonlinear hypertext. Hypertext provides the reader with greater learner control by providing the ability to explore and connect text according to individual need, and also by allowing greater immediacy of access, searching capability, and increased interactivity. Moreover, the integration of sound and video through the convergence of text and multimedia makes for a more enriched reading environment compared to the environment provided by traditional print. However, as will be discussed in this literature review, increased cognitive load and disorientation are potential disadvantages to reading in hypertext, especially for novices in the knowledge domain and those with limited or no system experience. Advantage: User Control The nonlinear structure of hypertext has advantages over print as a means for learning due to its capacity for increased user control over the pace, sequence, and content (kind of material and depth) of information presented. As will be explained in the discussion section of this literature review, increased user control allows readers who may have different needs and cognitive styles to engage in self-directed learning. It also gives readers opportunities to create learning environments that best suits their needs (Eveland, Marton, & Seo, 2004; Muller-Kalthoff & Moller, 2006; Zumbach, 2006). The ability to customize the reading process can also lead to increased learner motivation (Eveland & Dunwoody, 2002). 36 Disadvantages: Disorientation and Cognitive Load Primary Disadvantage: Disorientation. An extensive area of hypertext research examines the impact of reader disorientation as it effects knowledge acquisition (Shapiro & Niederhauser, 2004). Disorientation is often defined as a feeling of being “lost in hyperspace” (Conklin, 1987). It can also include a situation where the reader does not know one or more of the following: how they arrived at a particular node, their location within a hypertext, how they could navigate to another node or information, and whether or not the information sought is available within the hypertext on screen or the larger network (Zumbach, 2006; Conklin, 1987). The complexity of a reading task can affect a reader’s level of disorientation (McDonald and Stevenson, 1999). Generally, readers must allocate their cognitive resources to navigate through the text. They must also be able to read and understand the content, as well as actively integrate any new information acquired with their prior knowledge. Furthermore McDonald and Stevenson (1999), propose that the “keyhole phenomenon” also has an impact on disorientation. This phenomenon occurs when only a limited number of nodes of information are displayed at any one time. When this occurs it becomes more difficult for a reader to conceptualize the overall content and structure of the complete text. The result is inefficient navigation due to problems locating the current position in the document relative to the complete text (Woods, 1984; cited in McDonald & Stevenson, 1999). The structure of text and the amount of prior knowledge a reader possesses can greatly impact disorientation. This was demonstrated in a study by Muller-Kalthoff & Moller (2006). They investigated how prior knowledge impacts research participant’s knowledge 37 acquisition and their perceived disorientation when reading in either a reduced or free browsing hypertext condition. The authors hypothesized that disorientation could be lessened by reducing the complexity of the structure of the hypertext and restricting access to parts of the hypertext. This hypothesis was tested with thirty-six university students who read from hierarchically structured hypertext in either a reduced or free browsing condition. Results indicated that those participants with low prior knowledge found the reduced access condition was more conducive to reading and also improved comprehension. In addition, the reduced access helped lower participants’ perception of disorientation. This was measured using an eleven item response questionnaire. The researchers also found that perception of disorientation was also linked to the participants’ computer experience, computer ability and subject related interest. The study reinforced the idea that not all readers feel the effect of disorientation. However, it was demonstrated that readers with low prior knowledge found that the freedom of an unstructured complex nonlinear environment did increase disorientation. The discussion section of this review will examine how navigation tools, hypertext topology, and the reader’s amount of prior domain and system knowledge impact disorientation and reader knowledge acquisition. Related Disadvantage: Cognitive Load. Scholars are in agreement that cognitive load has the potential to impede knowledge acquisition while reading (Niederhauser et al., 2000). A number of factors impact whether or not a reader’s comprehension is negatively affected by cognitive load. One factor is that as the complexity of the online reading environment and of the task increases there is a greater potential for cognitive load to impede knowledge acquisition (DeStafano & LeFevre, 2007). 38 In addition, novices in the knowledge domain or those with limited system experience are more likely to be negatively impacted by cognitive load while reading. Reading in non-linear environments increases cognitive load when compared to reading in print. Niederhauser et al. (2000) examined whether the use of hypertext-based navigation features would impact knowledge acquisition for thirty-nine undergraduate educational computing students. As part of a regular class assignment students read an 83- screen hypertext that could be read either linearly or nonlinearly. The students’ reading ability, domain knowledge, system experience, time reading each screen, navigation patterns, and learning from the text were measured using a multiple choice test as well as an essay on the hypertext content. Results indicated that students who read in a linear fashion scored higher. The authors hypothesized that reading in the nonlinear version of the content increased cognitive load, and therefore the students who did so scored lower on the tests. The authors also concluded that “the combination of intrinsic and extraneous cognitive loads associated with learning complex concepts in a hypertext environment that requires sufficient reading comprehension, background knowledge, and computer operating skills may be overwhelming for students to the point that learning is compromised” (Niederhauser et al., 2000, p. 252). One factor that influences the amount of cognitive load readers experience while reading online is the number of hyperlinks that exist in the text. The decision of the reader to follow hyperlinks in the text is an example of an activity uniquely related to reading online. This increase in the number of decisions while reading places increased cognitive effort on the reader. In a literature review, Destafano & LeFevre (2007) examined studies that focused on hypertext features that impact cognitive load on comprehension and navigation while 39 reading online. From the author’s examination of the literature they recommended more research exploring the use of restricted numbers of links in order to decrease the reader’s cognitive load and potentially enhance the reader’s knowledge acquisition. Following up on Destafano & LeFevre’s recommendation, Madrid, Van Oostendorp and Meiguizo (2009), investigated the effect of limiting the number of hyperlinks on the navigation menu and how navigation support in the form of link suggestions impact a reader’s cognitive load and knowledge acquisition when reading online. In their study forty- five university students read a hypertext in either an 8 links hypertext navigation menu condition or a 3 links version. In addition participants in each navigation menu condition were either provided link suggestion navigation support or not. Link suggestion support was provided in the form of identification of which two hyperlinks in the menu bar are most closely related to the content being read and could be read next. After reading the hypertext participants then completed post-test questions to measure knowledge acquisition. The results of the study indicated that the link suggestions assisted readers with knowledge acquisition. However, the results of the study did not find support for the number of hyperlinks effecting reader cognitive load or knowledge acquisition. This lack of effect from the number hyperlinks may be partially explained by the fact the links were not embedded in the text, but were placed on a navigation menu to the side, which may less of an impact on the reader. In addition the variation in the number of links may have been too small to create an effect. There were only 3 or 8 links in total. In summary, considerable evidence supports the idea that cognitive load may impede knowledge acquisition. However, the amount of cognitive load that a particular hypertext document tends to create is not necessarily fixed. The literature points to the presence of 40 variables by which the negative effect of cognitive load can be either increased or minimized. As nonlinearity, learner control, and reading task complexity increase, so does the potential for cognitive burden. The complexity of the reading task may be increased if the reader is a novice in the system and knowledge domain. Novices spend more cognitive resources not only navigating the system but also understanding the content. Conversely, there are a number of ways to reduce cognitive load. Web designers can assist by using consistent hypertext structures and conventions, as this will reduce the cognitive load produced by metacognitive demands in unfamiliar structures (Schwartz, Andersen, Hong, Howard, & McGee, 2004). Readers may further reduce cognitive load through becoming more familiar with the online system (Macedo-Rouet, Rouet, Epstein, & Fayard, 2003). The Reading Tools examined in this study were created to assist a wide range of readers with a varied amount of prior knowledge, system experience, and access to related articles, by providing links to related materials that help establish a context for the content being read. The Reading Tools add to the nonlinearity of the text and potentially distract the reader; however, the hyperlinks are not embedded in the text and as Madrid, Van Oosteendorp and Meiguizo (2009) found, the number of links on a side menu did not impact the reader’s cognitive load. The Reading Tools links also have the potential to add to the reader’s background knowledge and enhance the reading experience. 41 Variables Impacting Knowledge Acquisition in Nonlinear Reading Environments Reader / Learner Variables Not all readers come to the online reading experience with the same amount of domain knowledge and system skills. Researchers are investigating whether variables such as a reader’s domain knowledge or their system experience have a significant impact on knowledge acquisition when reading in nonlinear online text environments (Eveland & Dunwoody, 2000; Alexander, Kulikowich, & Jetton, 1994; Macedo-Rouet et al., 2003). Domain Knowledge Novice. Readers who are novices in the knowledge domain are challenged because they have little or no prior knowledge to aid in the learning process (Shapiro, 1999). Alexander warns that readers who do not have the subject matter knowledge requisite for processing information effectively use less efficient general text-processing or problem-solving strategies (Alexander et al., 1994). This makes novice readers more susceptible to the negative impact of disorientation and cognitive load, and the reading task becomes more complex. For example Destafano & LeFevre (2007) review of the literature found that some hypertext features increased the reader’s cognitive load and were particularly detrimental to those online readers who were novice in the knowledge domain. 42 Proficient or Expert Readers Proficient or expert readers within their knowledge domain have fewer challenges in reading online. Additionally they also have the ability to leverage navigational aids to enhance their reading online. Alexander and Jetton (2000) states that “those who come to the reading task with a wealth of knowledge and a deep interest in the subject matter and whose goal is to exert strategic effort to gain a deep understanding of the reading are considered experts in a domain” ( p. 307). Readers who are experts in the knowledge domain possess prior knowledge that acts as a base to build upon, and therefore acquire and construct new knowledge more efficiently and are better able to connect and integrate new information with their existing knowledge base (Tremayne & Dunwoody, 2001; Muller-Kalthoff, & Moller, 2006). The literature supports the view that in hypertext learning environments the level of a reader’s prior knowledge is a significant predictor of performance in the areas of comprehension and knowledge acquisition. Muller-Kalthoff & Moller (2006) suggest that a reader’s prior knowledge serve as a controlled variable in studies relating to learning with hypertexts. Lawless and Brown (1998) examined the relationship between prior knowledge and structured and unstructured recall. In their study thirty-four undergraduates were given a 60 card HyperCard stack on the topic of Lyme disease. Their task was to read for general understanding. The HyperCard stack included supplemental information in the form of sound, animation, digital video, “hot words,” a compiled vocabulary list, and a menu navigation system which provided a number of potential reading paths. The authors recorded participants’ navigation through the HyperCard stack and found that higher prior knowledge readers used more efficient strategies in the nonlinear text environment than low prior 43 knowledge readers. Higher prior knowledge readers were able to compare and contrast information, and were able to efficiently pick out relevant information. The higher prior knowledge readers also demonstrated greater text recall. Additionally, there are studies that support the idea that domain expertise lessens the impact of disorientation and cognitive load. Last, O’Donnell, and Kelly (2001) studied the impact of prior knowledge and goal strength on learning from hypertext. Their study assigned twelve university students to either high or a low prior knowledge group. Further subgroups were created based on either strong or weak goal condition. For example, participants in the strong goal condition had to read to find the answers to specific questions while participants in the weak goal read for the general purpose of acquainting themselves with the material. Participants with lower prior knowledge and lower goal strength reported increased disorientation. This was expressed by participants who complained they felt lost and had a hard time navigating the hypertext. Conversely, participants with higher prior knowledge had a less negative experience and were somewhat more successful at completing the strong goal tasks of finding answers to specific questions. McDonald and Stevenson (1998b) examined the effects of domain knowledge on hypertext navigation and demonstrated that novices in the knowledge domain had more disorientation problems. The authors hypothesized that readers with more domain knowledge have deeper levels of understanding of the subject matter. This knowledge enables them to impose structure on the hypertext content which thereby allows them to enhance their navigation. In addition to impacting the level of disorientation, the amount of prior knowledge also influences a reader’s online information searching. Jenkins et al. (2003) examined 44 information seeking strategies in groups of nurses who had different combinations of domain and web expertise. The domain expert nurses were recruited from osteoporosis research centers and either had specialized training in osteoporosis, or they had performed research in this field. The domain novice nurses were nurse-generalists, recruited from the same institution but who had no special knowledge of osteoporosis. Both groups of nurses were asked to look for information on osteoporosis using search engines. Results suggested that expertise impacts searching patterns. The domain novices searched broadly across many different topics with little or no evaluation of the results. The domain experts searched less broadly across topics and evaluated the information more thoroughly. In summary, the amount of prior domain knowledge a reader possesses impacts how he or she acquires more knowledge. Low domain knowledge readers have more cognitive load and more disorientation problems, while high domain knowledge readers have prior knowledge to build upon, and they can more efficiently navigate and process hypertext. This has implications for website authors and designers who need to be aware of their target audience’s skill level. Readers who are experts in the knowledge domain can leverage tools and sites with increased user control to enhance the online reading experience, whereas novice domain readers may find tools and sites with more user control to be detrimental to their online reading experience. System Experience System experience in the literature is defined as the knowledge of how to navigate and operate within an online hypertext environment. Some scholars have found that readers with greater system experience have more efficient navigation strategies, reduced cognitive 45 load and less disorientation associated with acquiring knowledge from hypertext (Mitchell, Chen, & Macredie, 2005; Brinkerhoff et al., 2001; Muller-Kalthoff, & Moller, 2006). A number of studies suggest that a high level of system experience positively impacts knowledge acquisition in an online nonlinear environment. Eveland et al.’s (2004) experimental study looked at the impact of web experience on learning from hyperlinked online news articles versus non-hyperlinked online news articles. In this study 149 students from a Midwestern American university were randomly assigned to one of three conditions: a control group, a hyperlinked online newspaper group, and a non-hyperlinked online newspaper group. Participants from the hyperlinked and non-hyperlinked news article groups read their formatted online news for thirty minutes and afterwards were given a post-test. The subjects in the control group were not exposed to any news content; and only completed a shortened version of the post-test. The researchers examined a number of variables which included learning, information processing, and types of knowledge obtained from both types of news media—hyperlinked and non-hyperlinked. Eveland et al. (2004) reported that prior web experience was the determining factor for knowledge acquisition, and that readers with hypermedia experience would, not surprisingly; gain more from hypermedia than relative novices. Macedo-Rouet’s (2003) research concurred with Eveland et al.’s (2004) in suggesting that web expertise has a positive correlation with knowledge acquisition. Her study examined the effects of reading online verses print on a reader’s comprehension, and perception of cognitive load. Participants were first given a pre-test to determine their level of knowledge; then, using the scores of the pre-test, participants were assigned to either a hypertext group, a print group or a control group. The three groups were balanced for prior system experience 46 and prior domain knowledge. The hypertext group read a hypertext report and then completed a post-test, while the print group read the same report in print and answered the post-test. However, the control group completed the post-test without reading the report. Results of this study indicated that there was a positive correlation between the amount of prior system experience and a readers’ comprehension performance when reading hypertext (Macedo-Rouet et al., 2003). Macedo-Rouet went on to suggest that the “negative impact of hypertext on comprehension and cognitive load may be partly overcome through practice with the electronic medium” (Macedo-Rouet et al., 2003, p.123). In addition to learning, information processing, knowledge attainment, recognition, and cued recall performance, system experience also impacts the reader’s breadth and depth of online reading. Ford and Chen (2000) examined the effect of prior system experience on 65 post graduate students’ navigation behaviors and learning performance when using a hypermedia tutorial. Participants from this study were first given a multiple choice pre-test to assess their level of prior knowledge. Participants then used the hypermedia tutorial to learn how to design a web page. After completing the tutorial, students completed a post-test that was identical to their pre-test. The researchers found that users with a higher level of system experience could browse more pages and could reach more detailed levels of the subject content than those with lower levels of system experience. Altun’s (2000) study examined the kind of strategies experienced computer users develop and employ while navigating in hypertext. Altun’s findings support the notion that the amount of system experience impacts navigation behavior. He conducted a case study with two experienced computer users. The data was collected for a two month duration using interviews, observations, email discussions and a questionnaire. Altun discovered that highly 47 experienced system users do not necessarily always use the navigational aids provided by the author; instead, they develop their own navigation strategies that minimize disorientation and cognitive load. A general conclusion emerges from the literature that has investigated the variable system experience. System experience can alleviate the negative effects of disorientation and cognitive load. Readers who have experience within the hypertext system do not have to spend precious cognitive resources navigating: they can focus on reading. In addition, they tend not to get lost in the hypertext and are able to leverage navigation strategies that enhance the reading experience. Conversely, readers who have little system experience are more likely to become disoriented and suffer from cognitive load. Online content authors and designers have experimented with hypertext topologies and navigational aids in an effort to assist readers who are novices with the system. In addition, practice within the system may help novices. System Variables The literature points to a number of system variables that impact knowledge acquisition while reading online. The following section of this chapter examines the literature that investigates hypertext topologies, as well as various navigational aids that include online scholarly journal tools. Hypertext Topologies Hypertexts are non-linear and non-sequential semantic structures of information nodes that are linked together in a web-like structure (Cress & Knabel, 2003). However, not all hypertext is structured the same way. Text structure or topology impacts the way a reader 48 accesses and interacts with the information, and ultimately affects how readers acquire knowledge (Lee, 2005). Generally, three different topologies were compared and examined in the literature. The first is linear text that is used in either print or hypertext and provides a low level of user control. Graff (2002) created a hypertext linear architecture that resembled a book. The hypertext pages were linked so the reader could move only to the next or previous page. In addition, each page had a link to the first page of each of the four general sections. Calisir and Gurel (2003) used traditional print text as a linear topology for their study. The second type of topology is a hierarchical architecture. A hierarchical architecture connects nodes in a hierarchical order in which a node at one level can be accessed only from nodes directly above or below it (McDonald & Stevenson, 1998a). Scholars experimented with not just one form, but with variations of hierarchical topologies. For example, McDonald and Stevenson (1998a) and Calisir and Gurel (2003) examined a “mixed structure” which was hierarchical but contained a number of cross-referential links that allowed readers to jump across branches of the hierarchy. The third topology is termed “relational” (Graff, 2002) or “nonlinear” (McDonald & Stevenson, 1998a) and is characterized as a completely networked structure that connects a given node to any other related node. McDonald and Stevenson completed two studies looking at how hypertext topology impacts navigation performance. McDonald and Stevenson (1996) examined the effects of two hypertext topologies (hierarchical and nonlinear) on navigation performance compared to a linear version of the same document. The results showed that subjects had better speed and accuracy in finding nodes with the linear text than with the non-linear text, while performance on the hierarchical document fell between the two extremes. These results 49 suggest that as text nonlinearity increases, navigation performance decreases. They also support the notion that as text nonlinearity increases the reader’s cognitive load increases, which has a detrimental effect on navigation performance. McDonald and Stevenson (1998a) investigated how readers’ prior knowledge impacted navigation performance in a hierarchical topology, a nonlinear hypertext topology, or mixed (hierarchical structure with cross referential links) hypertext topology. Results indicated that readers in the mixed topology had superior performance. The authors hypothesized that a mixed topology provides readers with a good mixture of structure and user control. The mixed topology provides readers with a text structure that gives clues to conceptual content and links with semantic relationships, and still enough user control and freedom to navigate across nodes. Additionally, research suggests that different readers are suited to different types of topology. Calisir and Gurel (2003) investigated the interaction of the same three types of topologies as McDonald and Stevenson (1998a)—linear, hierarchical and mixed—with prior knowledge of the learner in hypertext learning. Thirty participants who comprised both experts and novices in the knowledge domain were tested. In each topology, participants read for forty minutes, then filled out a perceived control questionnaire followed by a reading comprehension test. Results indicated that for knowledgeable participants there was no significant difference in reading comprehension among readers in the three topology conditions. Novice knowledge readers had higher reading comprehension scores in the mixed condition than in the linear condition. However, after weighing all factors the authors suggested that novices were most successful in a hierarchical topology: they hypothesized that the hierarchical structure helps novices by providing a content organizational structure of the hypertext. 50 Calisir, Eryazici, and Lehto (2008) investigated effects of same three types of topologies as Calisir and Gurel (2003) --linear , hierarchical and mixed--- with prior knowledge of the learner in hypertext learning. However, this time the researchers added a fourth topology –generative text—in which participants had the ability to create notes about the text. Forty participants who comprised both experts and novices in the knowledge domain read the hypertext for one hour in the four conditions described above. Following the reading participants completed a reading comprehension test. Results indicated that participants who were novices in the knowledge domain who read in the hierarchical topology had significantly higher scores on their reading comprehension tests than those participants who read in the linear topology. The results also indicated that novices in the knowledge domain who read in the generative topology had significantly higher scores on their reading comprehension tests than those participants who read in the mixed or linear topology. Consistent with the results from Calisir and Gurel (2003) this study indicates that topology has an effect on reading comprehension of novices in the knowledge domain.
 The hypothesis that some topologies are better than others for different types of readers is supported by Graff (2002). He examined the effects of linear, hierarchical, and relational topologies and readers’ cognitive style on learning with fifty-three participants. Cognitive style was measured using the Cognitive Styles Index. Results suggested that in the context of recall performance it is beneficial to match a reader’s cognitive style to the appropriate topology in order to enhance learning. However, in the context of a deeper level of learning, no significant impact was found between cognitive style and topology. In summary, there is evidence to suggest that a hierarchical and mixed topology, as compared to linear and relational topology, offers readers who are novice in the knowledge 51 domain a structure that reveals the way topics relate to one another and that may, as a result, assist in knowledge acquisition. More research is needed to examine the impact of hypertext topology on knowledge acquisition for high and low knowledge readers. Navigational Aids To enhance online reading environments, navigational aids such as menus, maps, links, and search engines were designed to help limit the negative impact of disorientation and cognitive load (Jenkins et al., 2003). A number of studies examined how navigational aids impact knowledge acquisition for readers with low or high prior knowledge and system experience. Scholars have tested interactive overviews, cognitive maps, advanced organizers, link previews, and search engines and these studies are discussed in the following section. Interactive Overviews, Cognitive Maps and Advanced Organizers. Interactive overviews, cognitive maps, and advanced organizers help readers’ navigate content by providing a visual structure consisting of links that represent the content’s components and their relationships (Lee & Baylor, 2006). This helps readers understand the relationships between topics in the hypertext. De Jong and van der Hulst (2002) examined how navigational tools impact a reader’s knowledge acquisition. Forty-six participants were randomly assigned to one of three conditions. First, the visual condition participants were provided a navigational tool in the form of a graphical overview. The graphical overview provided the learner with the basic structure of the domain, and the layout was designed to encourage learners to choose any (nonlinear) sequence that followed the domain structure. Second, the hints condition was similar to the visual condition; however, the nodes were arranged randomly and highlighted 52 as a way of hinting at the sequence of the domain related layout presented in the visual condition. Finally, the control condition was the same as the hints condition, except hints were highlighted. Each group of participants spent a minimum of 25 minutes reading the hypertext. Participants then completed a definitional knowledge test, propositional knowledge test and a configural knowledge test. The definitional knowledge test measured the participant’s factual knowledge acquired during the reading portion of the experiment. The propositional knowledge test included questions that measured the relationship between concepts in the readings. The configural knowledge test measured how well participants understood the structure of the text. Results of the study indicated that participants in the graphical overview condition did not have better recall of the factual content; however, they displayed significantly better acquisition of configural knowledge than participants in the other two conditions. The benefit of including an overview in an online reading environment was supported by a study conducted by Brinkerhoff et al. (2001). The study examined how the provision of an overview and having previous computer experience affected knowledge acquisition in an online reading environment. Participants read a hypertext module that included either a structured overview, an unstructured overview, or no overview. The participants in the overview condition did not experience higher post-test scores. The authors suggested that an overview will provide little or no effect when reading in a relatively simple hypertext structure. They hypothesized that a more complex hypertext structure might benefit from an overview. Consequently, their study suggests that an overview may have a stronger effect in a nonlinear environment than a linear environment. Another important finding was that participants with a greater amount of system experience learned more than those who had 53 less system experience. This finding supports Macedo-Rouet et al. (2003) and Eveland et al. (2004), who found that system experience influences knowledge acquisition. Studies by Potelle and Rouet (2003) and Shapiro (1999, 2000) indicate that hierarchical maps assist readers in online reading environments. A hierarchical map displays the subordinate relationship between subtopics and topics, while network maps show semantic relationships between topics (Potelle & Rouet, 2003). Potelle and Rouet (2003) investigated the effect of a hierarchical map, a network map, and an alphabetic list on the comprehension of a reader with either low or high prior knowledge, in a hypertext environment. Forty-seven participants read and answered questions that required inferences based on the reading. Results indicated that a hierarchical map improved comprehension for low knowledge participants at the global level. In contrast, high prior knowledge readers were not affected. The hierarchical map’s positive effect on readers was supported by Shapiro’s (1999) investigation of interactive overviews. Shapiro (1999) investigated the effect of an advanced organizer--an interactive overview--on low prior knowledge readers. Undergraduate students read hypertext either using an interactive overview or without an interactive overview. Results indicated that the providing an interactive overview assisted low prior knowledge readers with comprehension. Shapiro cautioned hypertext designers that providing an interactive overview can impede learning when the overview does not match the reader’s learning goal. Shapiro (2000) in a later study examined how a reader’s learning goal and the provision of a structured interactive overview impacted performance in a HyperCard learning environment. The results suggested that the provision of interactive overviews can overshadow the learning goals of novices as they read information. Additionally, the 54 provision of interactive overviews has less of an effect on readers with some prior knowledge. One exception to the positive effect of providing overviews was revealed in a study conducted by Hofman & van Oostendorp (1999) who investigated whether a text structural overview leads to improve comprehension. In this study forty students read the same content in one of two hypertext structures: a structural overview or a list. The structural overview provided cause and effect relationships between sections of text. The list structure provided a topic list of text sections. Results indicated that the hypertext that provided a structural overview had a negative effect on comprehension for less knowledgeable readers. The authors hypothesized that the structural overview may impede knowledge acquisition for low knowledge readers because it focuses their attention on the macrostructure of the text at the expense of the microstructure. However, as noted by Rouet, Vidal-Abarca, Bert-Erboul, and Millogo (2001) and Shapiro & Niederhauser (2004), Hofman and van Oostendorp’s structural overview, which was a type of graphical organizer, was a rather complex network diagram with no clear orientation and several types of links for which the meaning was only suggested. This lack of clarity and the added complexity of the graphical organizer may also have negatively impacted less knowledgeable readers. Muller-Kalthoff and Moller (2003) investigated the impact of graphical overviews on learning. Eighty-two students read from a hypertext site either with an overview or without an overview. The results indicated that the graphical overview was associated with less disorientation and improved knowledge retention in participants who also were characterized with high prior knowledge and high self-concept of their computer abilities. According to the authors, these results might suggest that studies investigating the effects of navigational aids 55 should consider both prior knowledge and self concept of computer ability as variables that affect learning. The results also support the findings of other studies that found hierarchical maps assist low prior knowledge readers with knowledge acquisition. Link Previews. Link previews reduce structural disorientation. This finding is supported by Cress and Knabel (2003) who investigated how link previews impact searching and knowledge acquisition in a hypertext environment. In their study fifty students were tested in two different types of hypertext environments. One environment provided students with the option to click on a link and a popup preview window would appear. This window provided information about where the link would take them. The students could then choose to go to the linked page or close the pop-up preview window and stay on the current page. In the second environment, the linked page opened immediately when a link was clicked. The researchers found that in a pre-test–post-test scenario that the preview condition positively influenced knowledge acquisition. Participants who read in the link preview environment provided more correct answers. The link preview condition also enhanced navigation efficiencies. For example, previews dramatically reduced the readers’ need for backward navigation. In a study similar to Cress and Knabel’s, Maes, van Geel, and Cozijn (2006) investigated the effectiveness of mouse over hyperlink previews in three different types of hypertext environments: 1) without hypertext previews; 2) with content oriented, semantic previews; and 3) with task-oriented, pragmatic previews. In their study forty-five participants with different levels of background knowledge, average computer experience, and a neutral attitude towards the Internet took part. The subject content was fairly advanced. Results 56 supported Cress and Knabel’s (2003) study and showed that previews improved searching efficiency by assisting readers with the decision of whether to click on a link. Search Engines. Another pair of researchers, Farrell and Moore (2001), investigated the effect of navigational on attitude and learning achievement, based on participants’ reading abilities. One hundred forty-six eighth grade students were designated either low, middle, or high readers based on their reading achievement scores, and then randomly assigned to one of three conditions. The first was a linear tool condition where content was delivered one screen at a time. The second was a menu tool condition that provided students with a choice of information to access. The third condition required students to interact with a search engine that gave the results in the same fashion as the menu tool. Students read from their designated condition for 35 minutes. Results indicated that high ability students using the search engine navigation tool had significantly higher achievement scores than high ability students using either the menu format or the linear format. Achievement scores were based on a 20-item multiple choice post-test. Although there are inconsistent findings in the research, there is support for the notion that in complex nonlinear hypertext environments navigational aids enhance knowledge acquisition for readers who are novices in the knowledge domain. Navigational aids, with the exception of search engines, have less effect on readers with high prior knowledge. Moreover, to receive measurable effect researchers who examine the impact of navigational aids with nonlinear hypertext reading environment must use complex hypertext structures to ensure disorientation or cognitive load are factors. 57 Online Scholarly Journal Tools The inquiry undertaken here that investigates the Open Journal System’s Reading Tools is closely aligned to three other studies that investigated reading tools that reside next to online journal articles. Willinsky (2004), Willinsky & Quint-Rapoport (2007), and Blake et al. (2007) focused their research on determining to what degree the Reading Tools or Pub Med’s tools and services enhance readers’ experience in online scholarly research environments. The results of these studies were used to enhance the Reading Tools that were examined in the current study. In addition, these studies are precursors to the current study and inform its focus. Willinsky’s (2004) investigation exposed the need to refine the Reading Tools and to determine if the subsequent versions of the Reading Tools could be less confusing to users. His study examined whether Open Journal Systems’ Research Support Tool (the predecessor to “Reading Tools”) assisted thirteen policymakers in reading, interpreting, and utilizing research. Policy members were shown a working model of the Reading Tools and interviewed regarding its use. They communicated both appreciation of and confusion toward the tool. They liked the fact that the Research Support Tool was relatively academic. One participant alluded to the fact that the search results was more academic using Research Support Tools databases than using Google (in the era prior to Google Scholar). On the other hand, an example of confusion related to a statement on the Research Support Tool that said, “For this peer-reviewed article.” One participant thought it needed to be explicit that this was referring to the fact the article was peer reviewed; for example., it should have stated, “This is a Peer-Reviewed Article,” or “This particular article was peer-reviewed.” The author concluded that improvements would need to be made to the Research Support Tool before a 58 population could be tested to measure whether the device enhanced reading comprehension and the ability to evaluate the study being read, and whether the Research Support Tool gives greater confidence in use as a research tool. Subsequently, the Reading Support Tool was enhanced based on participant feedback. In another study, Willinsky & Quint-Rapoport (2007) investigated how three types of PubMed Tools and services—information retrieval, information access, and information management—contribute to the scholarly research article reading experience. The sample consisted of eighteen alternative medicine practitioners who were given a one-to-one, two hour introductory session on how to use PubMed’s tools. Participants were subsequently asked to provide feedback on the value of each tool in terms of their own need. The participant feedback indicated that the tools contributed to their engagement as well as their comprehension of the research article. Based on the results, the authors recognized that further study was needed to assess this type of tool’s impact on 1) user comprehension, 2) ability to evaluate, and 3) utilization of the article. The inquiry undertaken here investigates these three areas. Blake et al.’s (2007) investigation of whether Online Journal Systems’ Reading Tools assist humanities and computing scholars in comprehending, evaluating and utilizing the research literature in their respective disciplines reinforces the need to investigate other populations. The results indicated the tools enhanced participants’ reading experience to the degree the Reading Tools assisted in positioning an article and its author in relation to other work in the field, locating research materials, and the ability to take better advantage of existing resources. However, the tools had little impact on comprehension of the article. The authors attributed this in part to the fact that the readers already had critical methods in place 59 which the tools simply emulated. However, other populations may not have online critical reading procedures in place and may benefit from the Reading Tools’ ability to assist in finding related studies. Furthermore, participants in this study hypothesized that students would find many of the tools useful, which further reinforces the need to examine university students as a target population for use of the Reading Tools. The current study builds on findings from these online scholarly journal tools studies in three ways. First, it investigates the new Reading Tools that were enhanced based on these qualitative studies. The Reading Tools have been reorganized and new databases added to search for related materials. Second, the sample of users is shifted in this study from experts in the knowledge domain to students who are novice in the knowledge domain. As hypothesized by participants in the Blake et al. (2007) study, this population may benefit more from the tools. Finally, the current study attempts to measure quantitatively whether Reading Tools impact a student’s comprehension, a student’s ability to evaluate the author’s contribution to the field, and a student’s utilization of an article. Discussion Within the literature there is a diversity of experimental designs, particularly in ways to measure various variables. This makes it difficult to compare research results (Chen & Rada, 1996). For example, the hypertext environments used in experimental research have varying amounts of nonlinearity. The amount of nonlinearity in a reading environment impacts knowledge acquisition. Therefore, a methodology will help us to better quantify the nonlinearity. 60 Theoretical and Methodological Issues As noted earlier, the amount of nonlinearity in an online reading environment has a direct impact on user control and, consequently, on potential disorientation and cognitive load. As hypertext’s nonlinearity increases, the complexity of the text and potential for disorientation and cognitive load also increases. Therefore, it might be beneficial for scholars to quantify nonlinearity for research purposes. In the literature there does not appear to be any quantifiable measure of how nonlinear a hypertext is. However, link density could be one measure. Link density would be defined as the number of semantic or relational links per 100 words of text, in addition to regular navigational links. Another measure would be related to the type of reading task and how much additional information beyond the site itself is needed to complete the reading task. This would be important in a research environment testing disorientation and cognitive load, as nonlinearity and task complexity would need to be high enough to induce an effect on knowledge acquisition. Further research is needed to determine at what point the complexity of the task and link density would impact knowledge acquisition for readers with different levels of expertise, that is., groups varying from novice to expert in the knowledge domain and system experience. One criticism of much of the experimental research is the lack of authenticity in the hypertext and reading task. Many of the studies were undertaken in very controlled, unnatural hypertext reading environments specifically contrived for research purposes. In these environments, special attention was paid to the wording, reading level, amount of text the module contained, and the text’s organization. Often, when print and online hypertext were being compared, a print version of the module was read by one group and the same text was placed online with very little nonlinearity and no multimedia. The reading tasks for 61 participants (usually university students) were also usually contrived and out of context in relation to courses that the students were taking. Participants were unmotivated to complete the reading task possibly because they knew they were completing the task for experimental research. In many cases the lack of nonlinearity and lack of complex authentic reading task resulted in a reading experience where user control, cognitive load, and disorientation were difficult to measure. There is a need to study more naturally occurring hypertexts found on the Internet. Zumbach (2006) claims that in many studies comparisons of hypertext and linear text use “over-simplified learning material or learning sessions that only last for an hour or less. A generalization from such studies is problematic because they do not represent reading behaviour in daily educational use where much more time is spent on journal articles or textbooks” (p. 418). Within the literature contradictory research results have emerged. Standard measurements such as nonlinearity are elements that should make it easier to compare empirical research. In addition, researchers will need to pay attention to all variables when testing for knowledge acquisition in online reading environments. Both system variables (nonlinearity, topology) and reader variables such as prior knowledge and system experience need to be considered in future experimental research. Research Gap Identification The studies identified above have looked at how navigational aids impact knowledge acquisition; however, as innovations in online reading environments emerge, they need to be examined carefully using knowledge gained from previous online reading studies. Examining the Reading Tools presents a unique opportunity for it has not been sufficiently addressed in the literature. 62 Three important lessons drawn from the literature demonstrate the need for the current research. First, the Reading Tools provide the reader with a new type of mixed topology. Mixed topologies typically provide readers with increased user control and a structure that gives readers a content framework. The Reading Tools provide searching tools that increase the reader’s user control. This provides the reader with the ability to find background and related information that may be used to increase comprehension of the content. In addition, the tools provide a type of structured overview of the content. This is done by offering links to the abstract and to keywords in the metadata. As discussed above, the literature suggests that a mixed topology, more so than linear and relational topology, provides structure to readers who have low prior knowledge. This structure provides a content context, and that content context assists in knowledge acquisition. Second, the Reading Tools may be classified as a type of navigation tool, as they direct readers to additional related resources through search engines that leverage Boolean searches. As discussed above, there are inconsistent findings in the literature; however, there is support for the benefits of navigational aids such as search engines. Navigational aids, with the exception of search engines, have less effect on readers with high prior knowledge than on readers with low prior knowledge (Farrell & Moore, 2001). The literature points to the fact that readers with low prior knowledge have very little conceptual knowledge and that they benefit from navigational aids such as hierarchical maps which provide a global picture of the material (Chen, Jing-Ping, & Macredie, 2004). Moreover, for researchers examining the impact of navigational aids, the nonlinear hypertext reading environment must be complex in order for it to cause disorientation or cognitive load and receive a measurable effect. 63 Finally, the amount of prior domain knowledge a reader possesses impacts how he or she acquires it. This study sets out to determine whether the Reading Tools’ offer of additional contextual resources will in fact assist novices in the knowledge domain. The results of the literature reviewed point to the conclusion that readers with varying levels of expertise in the knowledge domain and varying levels of system experience need different types of hypertext structures and support to optimize their reading experience. The Open Journal System’s Reading Tools are created to assist a wide range of readers. For example, inexperienced readers benefit from the definitions link that provides a convenient way of looking up any words they are unsure of in the article. Moreover, the related items link helps readers establish a context for the article they are reading by providing access to relevant resources that they may not have considered, may not have known existed, or may not have been otherwise able to locate (Willinsky, 2006). 64 CHAPTER 3 METHODS Introduction The purpose of this chapter is to present the research methods and procedures that were utilized in this study. The chapter is divided into four main sections. The first section provides the research perspective, and the second section states the research question and hypotheses. The third section focuses on the research procedures, which include the sample and setting, research variables, research instruments, and data collection. Finally, the fourth section includes the methods of data analysis that seek to answer the three hypotheses. Research Perspective In many studies that examine knowledge acquisition in online environments, comprehension is measured using pre-test and post-test experimental design (Macedo-Rouet et al., 2003; Eveland et al., 2004). Research that uses this type of design commonly tests how an intervention, such as a navigational tool, affects comprehension and correlates the level of this comprehension to the impact of disorientation and cognitive load in online reading environments. Another common method used in the knowledge acquisition in online reading environment research is the Think Aloud Protocol (TAP). TAP is commonly used in the field of cognitive psychology to examine cognitive processing while problem solving (Branch, 2000; Cotton & Gresty, 2006); however, more recently this approach has been used to study human-computer interactions (Cotton & Gresty, 2006). TAP involves observing participants, 65 noting their behavior, and asking them to vocalize their thoughts as they complete a task. It is an excellent method for tapping a participant’s cognitive processes while he or she performs a specific task. For example, Yang (1997) used the think aloud protocol to study five university students as they sought information in an online hypertext system. This method provided insights into student behavior and resulted in the creation of a model and classification scheme for information seeking. Through the use of TAP to better understand participants’ behavior, researchers have the capacity to obtain a general understanding of online information retrieval patterns, web site usability and navigation. For example, to find out more about students’ cognitive processes while interacting with Reading Tools, investigators could observe one student at a time as each verbalize his or her thoughts while completing an authentic task using the Reading Tools. The investigators would have a better understanding how students perceive and use the Reading Tools. However, the research undertaken in this study is less interested in how participants use the Reading Tools than in whether or not the tools actually enhance the reading environment. This research is specifically interested in determining the degree to which access to the Open Journal Systems Reading Tools assists pre-service faculty of education students with reading comprehension of an article, ability to evaluate the quality of the article; and level of confidence in using the article; critically or supportively, as part of their future work as teachers or university students.. As discussed in chapter 2, much past experimental research involved a lack of authenticity in the hypertext and reading task. Many of the studies were conducted in a very controlled, unnatural hypertext reading environment contrived for research purposes. In these environments, special attention was paid to wording, reading level, the amount of text the 66 module contained, and the text’s organization. Often, when print and online hypertext were being compared, a print version of the module was read by one group and the same text was placed online with very little nonlinearity and no multimedia. For example, Macedo-Rouet et al. (2003) used a print and online version of a popular science magazine report. The reading tasks assigned to participants, which were usually university students, were also typically contrived and out of context in relation to courses that the students were taking. Participants in these cases were unmotivated to complete the reading task and knew the reason they were completing the task was for experimental research. Moreover, in many cases the lack of nonlinearity and the complexity of the authentic reading task resulted in a reading experience in which user control, cognitive load, and disorientation were difficult to measure. As a result, there is a need to study more naturally occurring hypertexts found on the Internet. Zumbach (2006) criticizes past studies comparing of hypertext and linear text, claiming that many used “over-simplified learning material or learning sessions that only last for an hour or less.” He argues that “A generalization from such studies is problematic because they do not represent reading behavior in daily educational use where much more time is spent on journal articles or textbooks” (p. 418). The research presented here attempted to create a reasonably authentic Reading Tools environment and task. The Reading Tools were tested by having students complete an authentic assignment that was integrated naturally as part of a regular course. The students, as part of a compulsory Information Communication Technology course for Secondary and Middle School pre-service teachers, read and critically evaluated the contribution four authors have made to four pressing technology issues facing teachers today. Each article was discussed in small groups during class for twenty minutes over four class sessions. In 67 preparation for each of these discussions, after reading the article and prior to class the students created a list of 10-12 talking points for each article. These points were intended to guide a student leading a class discussion of the article or to participate, as in this case, in a small-group discussion on the article. The talking points the students were asked to prepare were to address the following four questions: a)What are the author’s key points? b) Why should or should not the author be judged as having a contribution to make to this issue? c) Where and why you are inclined to support or challenge one or more aspects of the author’s position? and d) How does the author’s position relates to other ideas, work, and stances on this issue including, but not restricted to, your own. In addition, each talking point should be approximately three sentences: i) identify issue; ii) take a position; iii) present basis for position. Research Question This research study will assess the degree to which an online “Reading Tool” that provides contextual resources assists students with critiquing an article as part of an assignment. Specifically, the study intends to determine whether the set of reading tools utilized by OJS assists students in the context of an article critiquing assignment by contributing to the following: 1. Comprehension of an article, 2. Ability to evaluate the quality of the article, and 3. Augment the students’ level of confidence in using the article, critically or supportively, as part of their future work as teachers in a school or work as students at university. 68 Hypotheses In order to assess how effectively the Reading Tools contribute to the students’ experience, the following hypotheses were developed to guide this study: H1. University students who have access to the Reading Tools will have significantly higher article comprehension scores based on the scoring of their talking points. There will be an interaction between the Reading Tool group, the Without Reading Tool group, and the four trials. In trial one there should not be any significant difference as the groups are randomly distributed; however, for trials 2, 3, and 4 the Reading Tool condition should provide students with an enhanced way to access other articles that may provide a better context for the article being read. Therefore, the Reading Tool group should have significantly higher article comprehension scores based on the scoring of their talking points. H2 There will be an interaction between the Reading Tool group, the Without Reading Tool Access group and Trials. In Trial 1 there should not be any significant difference in evaluation scores between group; however, for trials 2, 3 and 4 the Reading Tool Access condition should have significantly higher article evaluation scores. H3 There will be an interaction between the Reading Tools group and the Without Reading Tool group such that in trial one there should not be any significant difference in the article utilization scores between groups, and in trials 2, 3, and 4 the Reading Tool Access condition should have significantly higher article utilization scores as defined as each student’s perception regarding the following: 69 H3A) How comfortable he or she would feel using this article in a class presentation, whether to criticize or draw on its perspective, and H3B) How comfortable he or she would feel using the article in the context of working with other teachers in a school, whether to criticize or draw on its perspective. Sample and Setting Subjects for this study were selected from 80 pre-service teachers enrolled in two sections of EDUC 453: Information Communication Technology for Secondary Teachers, at a small university campus in Western Canada. This course is required for all undergrad Secondary and Middle School Teacher Education Program students. Of the 80 students, 75 signed consent forms to participate in the study. The final sample was composed of 75 students, 44 female and 31 male. Of the 75 students, 23 of the students are math/science majors who have a B.Sc., 10 are French specialists with a B.A., 30 are middle school specialists with a B.A., B.F.A., and/or B.Sc., and 12 are trades specialists with 1 year of academic courses from a college and 2 years of trades and technology courses. However, the data from one of the participants was excluded, since one of their assignments was missing. In addition to matters of convenience, the proposed group was selected because it is a segment of the population central to using online literature in the pursuit of learning. In addition, as discussed in chapter 2, Blake et al. (2007) identified university students as novices in the knowledge domain and hypothesized that they may benefit from the Reading Tools. 70 The Information Communication Technology for Secondary Teachers course was delivered over a three week period during the summer of 2007. The classroom setting consisted of a meeting area with discussion table seating at one end and 30 networked computers with printing available at the other end. Students were also simultaneously completing another course, The Adolescent in the Classroom. Each class lasted three hours each day with students attending one course in the morning and a session of the other in the afternoon. Protection of Subjects The rights of all participants were protected in accordance with the policies and standards of the Institutional Ethics Board (Appendix E). All participants signed consent forms to participate in the study. Participation was optional and did not impact the students’ grades. Research Variables Based on the research hypotheses identified for this study, several dependent and independent variables have been identified. There were three dependent variables in this study: 1) reading comprehension, 2) the ability to assess the author’s contribution to the article (critique), and 3) each student’s perception of his or her ability to utilize the article in other educational contexts. The comprehension and critique variables were derived from two independent markers scoring the students’ talking points. The rubric described in Appendix D was used to assess how well each student appeared to understand the basic ideas and concepts in the assigned article and also each student’s ability to critique the article. The rubric’s scale scored comprehension between 0 and 10 and critique between 0 and 40. The 71 perception of utilization score was derived from each student’s survey response score, which was calculated from survey questions that participants answered after each trial. These questions related to the participant’s perception of utilization of the article. For H3A the survey question was, “How comfortable would you feel using this article in a class presentation, whether to criticize or draw on its perspective?” For H3B the survey question was, “How comfortable would you feel using this article in working with other teachers in a school, whether to criticize or draw on its perspective? Students responded to these two survey questions with a Likert scale from 1 “No Contribution” to 5 “High Contribution.” The primary independent variable investigated was access to Open Journal System’s Reading Tools. Access to the Reading Tools was chosen for this independent variable instead of use because it was this study’s intention to investigate the impact of the Reading Tools in a natural setting; that is, a setting in which a student would navigate to a journal article where the Reading Tools were available for use but were not required to be used. In this natural setting the student would not be forced to use the tools. The methods by which this independent variable was investigated and analyzed are described later in this chapter. Other variables had to be taken into account and controlled. As discussed in the literature review, researchers need to pay attention to both system and reader variables when testing for knowledge acquisition in online reading environments. Reader variables such as prior knowledge and system experience were controlled in this experiment through randomization of students into either the Reading Tools or Without Reading Tools group. In addition, system variables such as non-linearity, topology, and navigational aids were taken into consideration when designing the study. As described later in this chapter, all four articles used in the assignment portion of this study had a similar low level of non-linearity. 72 However, the Reading Tools increased the non-linearity of the text. Furthermore, the Reading Tools provided students with a new type of mixed topology that increased user control and navigation by offering more potential internal and external links to resources. Therefore, it is recognized that the group with access to Reading Tools worked in a different reading environment than the group without access to the Reading Tools. Research Instruments Assessment Rubric Creating the Rubric A rubric was developed by the researcher, in conjunction with Dr. Willinsky, to assess two constructs: student comprehension and the ability to critique the journal article (Appendix D). The rubric acted as a scoring guide to assess student performance for the assignment related to the research study. Specifically, the rubric measured comprehension and the ability to critique as these constructs relate to H1 and H2. This was done by scoring the talking points created by the students as part of their assignment. The following presents the steps that were taken while preparing the rubric. First, the researcher checked other research articles and rubric banks to determine if any rubrics existed to measure comprehension and the ability to critique an article in this context. No rubrics were found. Next, the comprehension construct and the critique construct were each broken into performance criteria. Comprehension was simply stated as “Displays Comprehension.” Critique was broken into the following four equally weighted performance criteria based on the assignment: 1) “Ability to identify strengths and weaknesses of author’s argument”; 2) “Supported argument with quality citations or relevant experience/prior 73 knowledge”; 3)“Number and quality of external sources to support argument (Academic Source = peer review or edited book and Non Academic Source = government site, article, organization site)”; and 4) “Critiqued whether author is in a position to write on the topic.” For each performance criteria, five performance levels were created out of 10 marks. The performance levels are: Exceptional 9-10, Excellent 8-9, Very Good 7-8, Good 6-7 and Poor below 6. Then, for each performance levels descriptions were created for each cell of the rubric matrix. This was because, as Tierney and Simon (2004) point out, the performance levels need to be clearly differentiated through clear descriptors to prevent misinterpretation of the rubric by the markers. The goal of each description was to make the differentiation of performance levels clear and as objective as possible. An example from the rubric is the performance criterion “Number and quality of external sources to support argument,” one of the corresponding descriptions was “at least three academic sources – 8 out of 10” (Appendix D). As recommended in the literature, the rubric was piloted with another sample of student work before grading the assignments related to the research (Bresciani, Zelna, & Anderson, 2004). The rubric was refined with a rubric piloting group consisting of two sections totaling 64 students who completed the same assignment but were not part of the research study. The rubric was tested to ensure that there were discrete distinctions between criterion choices. Based on the marking of the rubric pilot group, the rubric was further refined. One of the changes made was the creation of a hierarchy to quantify the construct “Number and Quality of external sources to support argument.” For example, the number scoring under the Excellent column, was specified to indicate that at least three academic 74 sources = 8 and 3 academic and 1 non-academic source = 8.5. The final revised rubric (Appendix D) was used to mark all 300 assignments from the current research study. Rubric Inter-rater Reliability The reliability of a rubric instrument is defined as the degree to which an instrument yields consistent results under repeated instrumentations. A number of steps were taken to improve the inter-rater reliability of marking the rubric. The literature recommends training markers to improve inter-rater reliability. For example, Maki (2004) recommends the following steps to establish inter-rater reliability: 1. Ask raters to independently score a set of student samples that reflects the range of texts students produce in response to a direct method. 2. Bring raters together to review their responses to identify patterns of consistent and inconsistent responses. 3. Discuss and then reconcile inconsistent responses. 4. Repeat the process of independent scoring on a new set of student samples 5. Again, bring all scorers together to review their responses to identify patterns of consistent and inconsistent responses. 6. Discuss and then reconcile inconsistent responses (p. 127). A process similar to the one Maki suggests was used in this study to establish inter- rater reliability. To begin with, the researcher taught the nuances of the rubric to an independent rater. The researcher and the independent rater then scored 20 assignments from Trial 1 and compared results. Consistent and inconsistent responses were discussed and reconciled. Another 10 assignments were then marked by the researcher and independent rater. This time more consistency was found. After confirming congruency, the researcher and independent rater each scored the remaining assignments using the rubric. 75 Another strategy employed to improve inter-rater reliability was the use of experienced markers. The first marker was the researcher, who is the Director for the Centre for Teaching and Learning at a small university campus in Western Canada and has 20 years teaching experience in the kindergarten to Grade 12 and post-secondary education systems. The second marker was a Learning Instructional Support Specialist who works in the Centre for Teaching and Learning at a small university in Western Canada and has 22 years teaching experience in the kindergarten to Grade 12 and post-secondary education systems. Both markers have experience and expertise in creating and marking rubrics. Intraclass correlation (ICC) was the method chosen to assess inter-rater reliability. ICC measures the proportion of variance attributable to objects of measurement (McGraw & Wong, 1996). This method was chosen because the data were continuous. A particular form of ICC used: the two-way random effects model, absolute agreement type. The two-way model was used because variation was expected within the targets (i.e., participants), as well as between raters. It differs from a one-way model in which the variability would only be attributed to the targets; therefore, with a one-way model it would not be possible to systematically attribute variability to the independent raters. A random model was chosen because it was rationalized that since a rubric was used, criteria for judgment were established externally. Therefore, the rubric could be given to two different raters and the results should be consistent. This model does assume that the ratings and process can be generalized, and thus differs from a fixed model in which the raters are considered to be the only two raters of importance, and using different raters or changing one of the raters would produce different results. Under a fixed model it is assumed the ratings and rating process cannot be generalized. For this study, an absolute agreement type was used, which assesses if 76 the raters are assigning the same absolute score. The absolute agreement type also assumes that variability due to raters will have an impact. If scores differ in absolute value, they are viewed as disagreements. This differs from the consistency type which measures whether raters constantly assign relevant ratings, and assumes that variability due to raters is not important. Under the consistency type scores that differ in absolute value are not considered as disagreements; rather, their relative standing to one another is considered. Finally, the version used was the average measure reliability (as reported in the Table 3.2), which gives the reliability of the mean of the ratings of all raters. The average measure reliability was used because the plan was to use the average of the ratings for further analyses. ICCs can be interpreted similar to Kappa (Garson, 2007). The ICC approaches a value of 1.0 when there is no variance between raters (i.e., when the raters assigned the exact same scores). A value of 0 indicates that there is no agreement between the raters. Regarding the interpretation of ICC results, Garson (2007) states that ICCs can be interpreted similar to Kappa. Landis and Koch (1977) suggest employing the following Kappa statistic benchmarks to assist in interpretation: 77 Table 3.1 Landis and Koch (1977, p. 165) Kappa Statistics and Strength of Agreement Kappa statistic Strength of agreement <0.00 Poor 0.00- 0.20 Slight 0.21- 0.40 Fair 0.41-0.60 Moderate 0.61- 0.80 Substantial 0.81-1.00 Almost Perfect Results of the study’s Intraclass Correlation calculation indicate very high values for total scores and total critique scores for all trials (Table 3.2). The reliability between raters was poor for assessing comprehension and critique 1 scores, which may be in part due to the subjectivity of the rubric for these two measures. This was especially true for trials 2 through 4. The low reliability is not a major concern for critique scores, because the analyses, which are described in chapter 4, were assessed not on single scores, but rather on the total score for all critique scores. Moreover, reliability among raters for the overall score and critique score was high on all of the trials. 78 Table 3.2 Inter-rater Reliability on the Scoring of the Assignment - Average ICC Values Measure ICC Trial 1 Comprehension .811 Critique 1 .737 Critique 2 .851 Critique 3 .937 Critique 4 .980 Critique score .950 Total scores .942 Trial 2 Comprehension .699 Critique 1 .582 Critique 2 .798 Critique 3 .865 Critique 4 .971 Critique score .934 Total scores .924 Trial 3 Comprehension .691 Critique 1 .591 Critique 2 .772 Critique 3 .896 Critique 4 .982 Critique score .942 Total scores .931 Trial 4 Comprehension .631 Critique 1 .509 Critique 2 .769 Critique 3 .941 Critique 4 .961 Critique score .943 Total scores .935 The validity of the rubric refers to the extent to which it measures what is intended to measure. Moskal and Leydens (2000) suggest reflecting on five particular questions that may improve content-related and construct validity. To examine content-related validity the 79 following three questions are suggested: “1) Do the evaluation criteria address any extraneous content? 2) Do the evaluation criteria of the scoring rubric address all aspects of the intended content? and 3) Is there any content addressed in the task that should be evaluated through the rubric, but is not? (p.6)” In addition they recommend asking the following two questions to examine construct validity: “1) Are all of the important facets of the intended construct evaluated through the scoring criteria? and 2) Is any of the evaluation criteria irrelevant to the construct of interest? (p.6)” These five questions were addressed in the development of the rubric in this study. Survey I and II Survey I and II were created by the researcher with the assistance of Dr. Willinsky. If an appropriate previously validated survey had been available to test our hypotheses we would have used it. It is recognized that the reliability and validity of Survey I and II are limited; however, we believe both surveys are useful due to their strong construct validity. Survey I is a survey questionnaire (Appendix A) and was intended to measure student strategies related to comprehending and critiquing the article, and perception of comfort level in relation to utilizing the article. This survey was placed online using the course’s online learning management system (WebCT), and participants completed it after submitting each trial’s assignment (Figure 3.1). Survey I was comprised of four questions. The first open response question, “If there were points or concepts in the article that you were not familiar with, what did you, if anything, do to figure them out?” sought to determine the student’s strategy for comprehending the article. Similarly, the second question, “What strategies did you use to evaluate the article, to establish the degree to which you should trust its claims?” was asked 80 to better determine the student’s strategy in evaluating or critiquing the article. The responses to these questions would help determine whether the Reading Tools played a role in helping students complete the assignment. Questions 3 and 4 used a five-point Likert scale ranging from Strongly Disagree to Strongly Agree to measure the students’ perceptions of their comfort levels in using the article. Question 3, “I would feel comfortable using this article in a class presentation, whether to criticize or draw on its perspective?” directly aligns with hypothesis 3A. Similarly, Question 4, “How comfortable would you feel using this article in working with other teachers in a school, whether to criticize or draw on its perspective?” aligns with hypothesis 3B. Both questions examine whether the Reading Tools influenced students’ article utilization as defined by the students’ perception of their comfort levels in the areas queried in Questions 3 and 4. 81 Figure 3.1 Screenshot of Survey I Online in WebCT 82 Survey II consisted of two parts (Appendix B). The first part was for all participants and looked at the students’ online reading versus print reading habits. This section was included in order to explore whether the intervention studied would impact whether students read online or in print. In addition, it was hypothesized that if a student printed the article they would be less likely to use the Reading Tools (intervention). The second part of the survey was created specifically for the participants who were in the group that received access to the Reading Tools. The first two questions used a 5 point Likert scale ranging from 1: No Contribution to 5: High Contribution. The first question investigated the student’s perception of the Reading Tools’ contribution in completing the assignment. The second question investigated whether the Reading Tools increased or decreased their ability to focus on the position of the article’s author. The third question asked students to rate each of the “Reading Tools”. This question used a 1 to 5 Likert Scale from no contribution to high contribution with the option of “Did not use.” These first three questions were asked to provide insight into which tools were used and how highly they were rated. The final question was an open question that provided students with the option to provide any additional feedback about the Reading Tools. Data Collection Procedures On the first day in the classroom with the students, the researcher introduced the project and its components. The researcher explained that the research centered on testing a new “reading environment” or way of presenting research articles. No demonstration or instruction related to the Reading Tools (intervention) or mention of any expectation to actually use the Reading Tools was conveyed. It was explained that the data collection would 83 focus on one of the course assignments, and whether they participated in the research or not they would still need to complete it. Students were reassured that the assignment would be marked pass/fail so that the research would not impact the student’s grade. Students were then provided with a detailed description of the assignment (Appendix C). After introductions and an overview of the research and assignment, the researcher distributed the student consent forms (Appendix E) to those students who chose to participate. Those students who consented were randomly assigned to one of two conditions. The Reading Tools group (the experimental group) would have full access to the Reading Tools (intervention), while the Without Reading Tools group would have identical access to the same article online, without access to the Reading Tools. Randomization was completed by giving each participant a number that corresponded to their position on the class list. Those numbers were then randomized using a randomization program (http://www.ranimizer.org/form.htm). In one section there was an odd number of students, and the extra person was randomly placed in the Reading Tools condition. For those students who felt they were being denied opportunity for better learning, it was explained that that is only a perception, because there is currently no evidence to support the idea that Reading Tools do, in fact enhance learning. The student assignment itself included reading and critiquing each of the four articles. The student critique centered on creating 10 – 12 talking points in preparation for a twenty minute small group discussion of the article. Each of the four articles represented a separate trial, and focused on four separate educational technology issues related to the course. The articles were all peer reviewed and available to the students in an online demo version of Open Journal Systems (Figure 3.2 and Figure 3.3). There were only small differences 84 between the four articles in relation to word count, the Flesch Reading Ease Test, and the Flesch-Kincaid Grade Level Score (Table 3.3). The Flesch Reading Ease Test was computed using Microsoft Word 2007. It rates documents on a 100-point scale, with 100 representing that the document was very easy to understand and 1 that it was very complex and difficult to understand. Similarly, the Flesch-Kincaid Grade Level Test was calculated using Microsoft Word 2007 and rates the text on a United States school grade level. Figure 3.2 View of Trial 2 Article – Reading Tools Condition 85 Figure 3.3 View of Trial 2 Article – Without Reading Tools Condition 86 Table 3.3 Article Descriptions for Trials 1 - 4 Trial Article Journal Year published Word count Flesch Reading Ease Test score Flesch- Kincaid Grade Level score 1 1 “What added value does a 1:1 student to laptop ratio bring to technology-supported teaching and learning?” Journal of Computer Assisted Learning 2007 8 812 29.6 14.3 2 2 “New bottle but old wine: A research of cyberbullying in schools” Computers in Human Behavior 2007 5 473 33.1 12.8 3 3 “The digital divide: The special case of gender” Journal of Computer Assisted Learning 2006 8 574 41.3 12.7 4 4 “Bridging between information literacy and information technology in Singapore schools: an exploratory study” Education, Knowledge and Economy 2007 5 915 34.1 13.6 87 The non-linearity and complexity of the four articles was relatively low. The link density of the four articles was low and consisted mainly of internal navigational hyperlinks from a table of contents to sections of the article and also from the citations to their references. Only occasional links to external related resources were included. The majority of participants in this study had an academic degree and all had completed at least 3 years of post-secondary schooling. However, in all four articles the reading level was below 14.3, which would not challenge the readers’ ability (Table 3.3). Other scholarly journal articles that may have been used for the assignment could potentially have had a higher reading level and more theoretical complexity. The first article was introduced on Tuesday, July 10 and was due on Friday, July 13. For this first article students from both the Reading Tools and the Without Reading Tools groups created 10 – 12 talking points but neither group had access to the Reading Tools. The Reading Tools were withheld from the Reading Tools group for the first article only. This was done to establish a baseline for comparing the outcomes of the Reading Tools group from trial 1 to the outcomes of the other three trials On July 13 all students were given time at the beginning of class to submit the talking points for the first article using the electronic assignment drop box of the online learning management system (WebCT). In addition, before participating in the small group discussion, all students were given time in class to complete Survey I which was available online on WebCT (Figure 3.1). After completing the first article and discussing it in small groups, the second article was introduced and assigned. All articles hereafter followed the same procedures undertaken in article one except that the Reading Tools Group had access to the Reading Tools to 88 complete the assignment and the Without Reading Tools group did not (Table 3.4). Table 3.5 describes the articles, introduction, and due dates for Trials 2, 3, and 4. Table 3.4 2 x 4 Group x Trial Analysis Design Trial 1 Trial 2 Trial 3 Trial 4 Reading Toolsª Article 1 (No Access to Reading Tools) Article 2 (Access to Reading Tools) Article 3 (Access to Reading Tools) Article 4 (Access to Reading Tools) Without Reading Toolsb Article 1 (No access to Reading Tools) Article 2 (No access to Reading Tools) Article 3 (No access to Reading Tools) Article 4 (No access to Reading Tools) an = 38. bn = 36 89 Table 3.5 Assignment Articles with Introduction and Completion Dates Trial Article Journal Introduction date Completion date 1 1 “What added value does a 1:1 student to laptop ratio bring to technology-supported teaching and learning?” Journal of Computer Assisted Learning Tuesday, July 10 Friday, July 13 2 2 “New bottle but old wine: A research of cyberbullying in schools” Computers in Human Behavior Friday, July 13 Tuesday, July 17 3 3 “The digital divide: The special case of gender” Journal of Computer Assisted Learning Tuesday, July 17 Friday, July 20 4 4 “Bridging between information literacy and information technology in Singapore schools: an exploratory study” Education, Knowledge and Economy Friday, July 20 Wednesday, July 25 On Wednesday, July 25 after the last assignment was handed in and discussed, all students completed Survey II Part A on paper, in class (Appendix A). The Reading Tools 90 group also completed Survey II Part B. This part of the survey was designed specifically to collect information about access to the Reading Tools (Appendix B). Methods of Data Analysis The quantitative analysis in this study was conducted using a mixed design ANOVA to analyze the data for all three hypotheses. The between-subjects variable (i.e. group) measures were based on students with access to the Reading Tools (treatment) and students without access to Reading Tools (control). The repeated measures were the four trials. To test each hypothesis, three effects that compose the mixed design ANOVA analysis were examined. First, analysis was conducted to investigate whether there was an interaction between trial and group. This analysis examined whether the pattern of results for the Reading Tools group across the four trials was different than the pattern of results for the Without Reading Tools group. Second, a main effect for Trial was investigated to determine if there were differences among the four trials regardless of group. Means for each trial, regardless of group, were compared to determine whether there were differences among the four trials (i.e., differences in the four articles). It is important to understand whether the order of a particular trial impacted the results because each trial used a different article, and the articles were unique in complexity and topic. These differences may impact whether access to the Reading Tools enhanced students’ ability to complete the assignment. For example, students reading a complex article may rely more on the Reading Tools than they would while reading a less complex article. Finally, a main effect for Group was investigated to see if there were differences between the two groups. This analysis determined if there were any significant differences between the Reading Tools and Without Reading Tools 91 Group. If an interaction between trial and group, group, or trial effect was discovered post hoc analyses were conducted to determine what significant differences occurred. Before completing all analyses, the appropriate assumptions of the procedure for each test were assessed. In particular, the sphericity assumption was checked with the Mauchly’s sphericity test. Unless otherwise stated these assumptions were supported and no data transformation was required. For all analyses, a .05 alpha level of significance was used. However, in some post hoc analyses the alpha level needed to be adjusted to control for multiple comparisons to protect against type I errors. This was done using the Bonferroni adjustment that divided the 0.05 significance level by the number of comparisons. Hypothesis 1: University students who have access to the Reading Tools will have significantly higher article comprehension scores based on the scoring of their talking points. There will be an interaction between the Reading Tool group, the Without Reading Tools group, and the four trials. In trial one there should not be any significant difference as the groups are randomly distributed; however, for trials 2, 3, and 4 the Reading Tool condition should provide students with an enhanced way to access other articles that may provide a better context for the article being read. Therefore, the Reading Tool group should have significantly higher article comprehension scores based on the scoring of their talking points. Analysis of the relevant data gathered to test this hypothesis was submitted to a mixed design ANOVA analysis. This ANOVA tested for an interaction between the within- subjects (repeated measure) variable (four trials) and the between-subjects variable (two groups). The students’ comprehension score was the dependent variable. 92 The comprehension score was derived from two independent markers scoring the student talking points using the rubric (Appendix D) to assess how well each student appeared to understand the basic ideas and concepts in the article. The scale used with the rubric provided students with a comprehension score between 0 and 10. The final comprehension score was an average of the two independent marker scores. Hypothesis 2: There will be an interaction between the Reading Tool group, the Without Reading Tool Access group and Trials. In Trial 1 there should not be any significant difference in evaluation scores between groups; however, for trials 2, 3 and 4 the Reading Tool Access condition should have significantly higher article evaluation scores. Analysis of hypothesis 2 was performed using similar procedures as hypothesis one. The analysis of the relevant data gathered to test this hypothesis was submitted to a mixed design ANOVA analysis. This ANOVA tested for an interaction between the within-subjects (repeated measure) variable (four trials) and the between-subjects variable (two groups). The student critique score was the dependent variable. The critique score was derived from two independent markers scoring the student talking points using the rubric shown in Appendix D to assess how well each student critiqued the article being read. The scale used with the rubric provided students with a possible critique score between 0 and 40. The final critique score was an average of the two independent marker scores. Hypothesis 3: There will be an interaction between the Reading Tools group and the Without Reading Tool group such that in trial one there should not be any significant difference in the article utilization scores between groups, and in trials 2, 3, and 4 the 93 Reading Tool Access condition should have significantly higher article utilization scores as defined as each student’s perception regarding the following: H3A) How comfortable he or she would feel using this article in a class presentation, whether to criticize or draw on its perspective, and H3B) How comfortable he or she would feel using the article in the context of working with other teachers in a school, whether to criticize or draw on its perspective. Analysis of the relevant data gathered to test H3A and H3B hypotheses were analyzed separately using a mixed design ANOVA. The analysis of relevant data gathered to investigate hypothesis 3A tested for interaction between the within-subjects (repeated measure) variable (four trials) and the between-subjects variable (group). The students’ perception of utilization of the article was the dependant variable. The students’ perception of utilization was derived from the students’ survey response score, which was derived from the survey questions that the participants answered after each trial. These survey questions related to their perception of their utilization of the article. For H3A, the corresponding survey question was “How comfortable would you feel using this article in a class presentation, whether to criticize or draw on its perspective?” The survey question provided students with a scale from 1“No Contribution” to 5“High contribution”. Analysis of the relevant data gathered to test hypothesis 3B was also submitted to a mixed design ANOVA that tested for an interaction between the within-subjects (repeated measure) variable (four trials) and the between-subjects variable (group). Again, the student’s perception of utilization of the article was the dependant variable, the student’s 94 perception of utilization of the article was derived from student’s survey response score, which was derived from the survey questions that the participants answered after each trial. These survey questions related to their perception of utilization of the article. For H3B, the corresponding survey question was “How comfortable would you feel using this article working with other teachers in a school, whether to criticize or draw on its perspective?” The survey question provided students with a scale from 1 “No Contribution” to 5 “High contribution”. Other Analysis A descriptive data analysis was performed on the first two questions of Survey I and all of Survey II. The first two questions of Survey I, “If there were points or concepts in the article that you were not familiar with, what did you, if anything, do to figure them out?” and “What strategies did you use to evaluate the article, to establish the degree to which you should trust its claims?” were used to assist in interpreting the results in chapter 5. Survey II consisted of two parts (Appendix B) and is presented in chapter 4. All participants completed the first part after handing in and discussing the last article as part of the assignment. This part investigated the students’ online reading versus print reading habits. Survey results were analyzed in three ways. First, the number of students who printed each article was tabulated. Second, the number of articles each student printed was calculated. Finally, the reasons students printed and read online were analyzed. Commonalities between responses were sought and responses were placed in categories. Categories and the number of corresponding responses were tabulated. The second part of the Survey II was created specifically for the participants who were in the Reading Tools group. This part consists of three questions with an optional space 95 at the end to add any additional comments. The first two questions use a 5 point Likert scale ranging from 1 No Contribution to 5 High Contribution. The first question investigated the students’ perception of the Reading Tools contribution to the completion of the assignment. Means and standard deviation were calculated for each sub-question. In addition, each contribution level for each sub-question was calculated. To determine how many students felt Reading Tools contributed highly, the number of participants who rated each sub question contribution as either a 4 or 5 was tabulated. The third question of Survey II Part B asked students to rate each of the “Reading Tools”. These questions were asked to provide insight into which tools were used and how highly they were rated. This question used a 1 to 5 Likert Scale from no contribution to high contribution with the option of “Did not use”. Analysis for this question included means and number of students rating each tool were tabulated. The final question was an open question that provided all students with the option to provide any additional feedback about the Reading Tools. The results of this question are used to support arguments made in the discussion in Chapter 5. Limitations Two limitations to the research methodology are identified. The first is that the researcher was the teacher of the students and, consequently, researcher bias may be unintentionally present. The researcher for the study is employed at the university and the teacher of the two sections for this study. Participants in the study may have unduly tried the Reading Tools to please the teacher/researcher. In fact the students may not have used the tools at all when completing the assignment in a non-experimental setting. In addition, the 96 accuracy of the self-reported survey data may be limited. Sometimes, a student will give a desirable response to please the instructor, who is also the researcher. An attempt to control for this was done through communication with students and the use of two assignment markers. The researcher communicated to students that the assignments would not in any way influence their final grade and all analyses of their assignments would be done after their final grades were submitted. In addition, the researcher and an independent marker scored all the assignments. Both markers were unaware of which group each assignment belonged to. Moreover, inter-rater reliability measures were calculated. A second limitation of the research design is the variance in the level of difficulty between the four articles. Although the assignment was consistent across trials, different articles were used for each trial. Since there were four trials it was necessary to use four different articles. This may have been partially controlled for by using four articles from the same journal; however, this would not have supported the principle of conducting this study in an authentic learning environment. It is recognized that there was a difference in complexity between the articles. This may have affected how each article interacted with the Reading Tools. This limitation was partially controlled for by having each group interact with the same article, and by choosing articles on the basis of content related to the course articles rather than how well they worked with the Reading Tools. However, theoretically each of the articles used may not have been complex enough for the Reading Tools to make a significant contribution to a better comprehension score. 97 CHAPTER 4 RESULTS: DO THE READING TOOLS MAKE A DIFFERENCE? This chapter describes the analyses of data collected for this study of Reading Tools. The effects of access to Reading Tools on student comprehension, ability to critique, and utilization perception are arranged and presented by hypothesis. As described in the methodology chapter of this study, students completed an authentic assignment that was integrated into their required Faculty of Education course EDUC 453: “Information Communication Technology for Secondary Teachers.” In this course students read and critically evaluated four authors’ contributions made to four pressing technology related issues that face teachers today. As is also described in the methodology section of this dissertation, seventy five participants in the course entitled “Information Communication Technology for Secondary Teachers” were randomly assigned to two groups: the experimental group made up of 38 students who had access to the Reading Tools, and the control group made up of 37 students who were, in effect, the Without Reading Tools group (Note: data from one of the participants in the Without Reading Tools group was excluded due to a missing assignment). For Trial 1 both groups had access to the assigned article online, without the Reading Tools. For trials 2, 3, and 4, the Reading Tools group worked with an article that was accompanied by the Reading Tools and the Without Reading Tools group worked with an article without the tools. As part of the assignment students were asked to create a list of 10-12 talking points for each article prior to the class. They would use the points as a basis for discussing the article with their groups (Appendix C). These talking points were then marked by two 98 independent markers using a rubric (Appendix D) to arrive at a “comprehension” and “critique” score. After submitting the talking points and before discussing each article, students completed an online survey with a Likert rating system that asked, “How comfortable would you feel using this article in a class presentation, whether to criticize or draw on its perspective?” and “How comfortable would you feel using this article in working with other teachers in a school, whether to criticize or draw on its perspective?” These survey results were used to provide a quantitative score related to whether or not the Reading Tools increased the students’ perceived ability to utilize the article in their work in schools. The quantitative analysis in this study was conducted using a mixed design ANOVA to analyze the data for all three hypotheses, which are presented in this chapter. The between-subjects variable (i.e., group) measures were based on students with access to the Reading Tools (treatment) and students without access to Reading Tools (control). The repeated measures were the four trials. To test each hypothesis, three effects that comprised the mixed design ANOVA analysis were examined. First, an analysis was conducted to investigate whether there was an interaction between trial and group. This analysis tested whether the pattern of results for the Reading Tools group across the four trials differed from that of the Without Reading Tools group. Second, a main effect for trial was investigated to determine if there were differences among the four trials regardless of group. Means for each trial, regardless of group, were compared to determine whether there were differences among the four trials (i.e., differences in the four articles). It is important to understand whether or not trial impacts the results, since each trial used a different article, and each article differed in complexity and topic. These differences may impact whether access to the Reading Tools enhanced the students’ ability to complete the assignment. For example, students reading a 99 complex article may rely more on the Reading Tools than they would reading a less complex one. Finally, a main effect for group was investigated to see if there were differences between the two groups. This analysis was performed to determine if there were any significant differences between the Reading Tools and Without Reading Tools Group. If an interaction between trial and group, group, or trial was discovered, post hoc analyses were conducted to determine what significant differences occurred. Before completing all analyses, the appropriate assumptions of the procedure for each test were assessed. In particular, the sphericity assumption was checked with the Mauchly’s sphericity test. Unless stated otherwise, these assumptions were supported and no data transformation was required. For all analyses discussed, a .05 alpha level of significance was used. However, in some post hoc analyses the alpha level needed to be adjusted to control for multiple comparisons and to protect against type I errors. This was done using the Bonferroni adjustment that divided the .05 significance level by the number of comparisons. The last section of this chapter presents the descriptive statistics from Survey II which consisted of two parts (Appendix B). The first part was completed by all participants and explored their online reading versus print reading habits. The second part was completed by the Reading Tools group only. It focused on whether the Reading Tools contributed to their experiences. It also provided them an opportunity to rate the Reading Tools. The Reading Tools and Comprehension Hypothesis 1 University students who have access to the Reading Tools will have significantly higher article comprehension scores based on the scoring of their talking points. There will 100 be an interaction between the Reading Tool group, the Without Reading Tools group, and the four trials. In trial one there should not be any significant difference as the groups are randomly distributed; however, for trials 2, 3, and 4 the Reading Tool condition should provide students with an enhanced way to access other articles that may provide a better context for the article being read. Therefore, the Reading Tool group should have significantly higher article comprehension scores based on the scoring of their talking points. Analysis of the relevant data to test this hypothesis was submitted to a mixed design ANOVA. This ANOVA tested for an interaction between the within-subjects (repeated measure) variable (four trials) and the between-subjects variable (two groups). The students’ comprehension scores were the dependent variable. The comprehension scores were derived from two independent markers who scored the student talking points. The markers used the rubric (Appendix D) to assess how well each student appeared to understand the basic ideas and concepts in the articles. The rubric scale was between 0 and 10 inclusively. To test hypothesis 1, three effects were investigated using a mixed design ANOVA to determine whether students who were provided access to the Reading Tools had higher comprehension scores than students without Reading Tools access (refer to Table 4.1 for descriptive statistics). First, an analysis was conducted to investigate whether there was an interaction between trial and group. The repeated measures ANOVA indicated that the pattern of mean comprehension scores across the trials was not significantly different for the Reading Tools group compared to the Without Reading Tools group. This outcome provides evidence that there was no significant interaction between trial and group, F (2.56, 184.44) = .999, p = .389, η2 = .014. 101 Table 4.4 Comprehension Mean and Standard Deviation Scores by Group Group Mean Standard Deviation Trial 1 7.71 .92 Reading Toolsa 7.74 .84 Without Reading Toolsb 7.68 .92 Trial 2 7.99 .75 Reading Toolsa 8.11 .74 Without Reading Toolsb 7.86 .76 Trial 3 7.93 .76 Reading Toolsa 7.93 .78 Without Reading Toolsb 7.92 .75 Trial 4 7.80 .59 Reading Toolsa 7.78 .57 Without Reading Toolsb 7.81 .62 an = 38. bn = 36 Second, a main effect for trial was investigated to determine if there were comprehension score differences among the four trials regardless of group. A significant main effect was found for trial, F (1.30, 184.44) = 4.193, p < .01, η2 = .055. This indicated potential differences between the mean comprehension scores of the four trials. Therefore, Bonferroni post hoc comparisons were conducted to examine specific differences between trials. Post hoc comparisons failed to discover any significant differences, indicating that there was no significant comprehension score differences between trials, although the differences between trials 1 and 2 and between trials 2 and 4 approached significance (Table 4.2). 102 Table 4.5 Comprehension Scores Significance between Trials Comparison Significance Value Trial 1 vs. Trial 2 .053 Trial 1 vs. Trial 3 .156 Trial 1 vs. Trial 4 1.00 Trial 2 vs. Trial 3 1.00 Trial 2 vs. Trial 4 .051 Trial 3 vs. Trial 4 .474 Finally, a main effect for group was assessed using between-subjects ANOVA to determine whether there were differences between the two groups. The ANOVA failed to find a significant effect for group, F (1, 72) = .256, p = .614, η2 = .004. Overall, the results of the hypothesis 1 analysis provided evidence that there were no significant group differences in comprehension scores. However, differences between trials 1 and 2 and between trials 2 and 4 approached significance regardless of group. This analysis clearly indicates that hypothesis 1 was not supported and participants in the Reading Tools group did not have significantly higher article comprehension scores for trials 2, 3, and 4. The Reading Tools and Article Evaluation/Critique Hypothesis 2 There will be an interaction between the Reading Tool group, the Without Reading Tool Access group and Trials. In Trial 1 there should not be any significant difference in evaluation scores between groups; however, for trials 2, 3 and 4 the Reading Tool Access condition should have significantly higher article evaluation scores. 103 Analysis of the relevant data gathered to test this hypothesis was submitted to a mixed design ANOVA. This ANOVA tested for an interaction between the within-subjects (repeated measure) variable (four trials) and the between-subjects variable (group). The students’ critique score was the dependent variable. The critique score was derived from two independent markers scoring the student talking points by using the rubric described in Appendix B to assess how well each student critiqued the articles. The scale used with the rubric provided students with a possible critique score between 0 and 40 inclusively. To test hypothesis 2, three effects were investigated using a mixed design ANOVA to determine whether students who were provided access to the Reading Tools had higher critique scores than students without Reading Tools access (refer to Table 4.3 for descriptive statistics). First, an analysis was conducted to investigate whether there was an interaction between trial and group. The repeated measures ANOVA indicated that the pattern of mean critique scores for the Reading Tools group across the trials was not significantly different from that of the Without Reading Tools group. This outcome supported that there was no significant interaction between trial and group, F (3, 216) = .926, p = .429, η2 = .013. 104 Table 4.6 Critique Means and Standard Deviation Scores by Group Group Mean Standard Deviation Trial 1 27.78 5.32 Reading Toolsª 28.95 4.76 Without Reading Tools 26.55 5.66 Trial 2 28.75 5.39 Reading Toolsª 30.11 4.71 Without Reading Toolsb 27.31 5.75 Trial 3 27.69 5.36 Reading Toolsª 28.41 5.40 Without Reading Toolsb 26.93 5.29 Trial 4 26.96 4.72 Reading Toolsª 27.65 4.76 Without Reading Toolsb 26.23 4.64 an = 38. bn = 36 Second, a main effect for trial was investigated to determine if there were critique score differences among the four trials regardless of group. A significant main effect was found for trial, F (3, 216) = 4.140, p < .01, η2 = .054. Therefore, post hoc comparisons were conducted to explore the specific differences between trials. Post hoc comparisons found a significant difference between trial 2 and trial 4, indicating that scores on trial 2 were significantly higher than scores on trial 4 (Table 4.4). 105 Table 4.7 Critique Scores Significance between Trials Comparison Significance Value Trial 1 vs. Trial 2 .507 Trial 1 vs. Trial 3 1.00 Trial 1 vs. Trial 4 .857 Trial 2 vs. Trial 3 .190 Trial 2 vs. Trial 4 .001** Trial 3 vs. Trial 4 .574 *p < .05. **p < .0083. Finally, a main effect for group was investigated using between-subjects ANOVA to determine whether there were differences between the two groups. The ANOVA results approached significance, but failed to find a significant main effect for group, F (1, 72) = 3.93, p = .051, η2 = .052. Since the effect approached significance, a Pairwise comparison using t tests was conducted to determine if significant critique score differences occurred between the two groups at any of the trials. Results indicated trial 1 approached significance and trial 2 was significant (Table 4.5). However, the alpha score needed to be adjusted from .05 to .0125 because multiple comparisons were used, and also to protect against type I errors. After adjusting the alpha level, results indicated that none of the Pairwise comparisons were significant. 106 Table 4.8 Critique Score Trial and Group Relationships Group Mean Standard deviation t df Significance Trial 1 Reading Toolsª 28.95 4.76 1.98 73 .051 Without Reading Toolsb 26.55 5.66 Trial 2 Reading Toolsª 30.11 4.71 2.41 73 .018* Without Reading Toolsb 27.31 5.75 Trial 3 Reading Toolsª 28.41 5.40 1.19 72 .239 Without Reading Toolsb 26.93 5.29 Trial 4 Reading Toolsª 27.65 4.76 1.40 73 .167 Without Reading Toolsb 26.23 4.64 an = 38. bn = 36 *p < .05. **p < .0125. Overall, results failed to support hypothesis 2. There were no significant differences between the Reading Tools group critique scores and those of the Without Reading Tools group. However, group differences approached significance in trial 1 and 2. In addition, there were no significant critique scores among the four trials with the exception of all participants having higher scores on trial 2 compared to trial 4. 107 The Reading Tools and the Student’s Perceived Ability to Utilize the Article Hypothesis Three There will be an interaction between the Reading Tools group and the Without Reading Tool group such that in trial one there should not be any significant difference in the article utilization scores between groups, and in trials 2, 3, and 4 the Reading Tool Access condition should have significantly higher article utilization scores as defined by each student’s perception regarding the following: H3A) How comfortable he or she would feel using this article in a class presentation, whether to criticize or draw on its perspective, and H3B) How comfortable he or she would feel using the article in the context of working with other teachers in a school, whether to criticize or draw on its perspective. Analysis of the relevant data gathered to test H3A and H3B hypotheses was analyzed separately using a mixed design ANOVA analysis. Hypothesis 3A Analysis of the relevant data gathered to test hypothesis 3A was submitted to a mixed design ANOVA. The ANOVA tested for interaction between the within-subjects (repeated measure) variable (four trials) and the between-subjects variable (group). Student perception of utilization of the article was the dependant variable. Student perception of utilization was derived from the students’ survey response scores. After completing the critiquing assignment for each trial, participants answered survey questions related to their perception 108 of utilization of the article. For H3A, the results of the survey question “How comfortable would you feel using this article in a class presentation, whether to criticize or draw on its perspective?” served as the dependant variable. The scale used for the survey question was a scale ranging from 1 No Contribution to 5 High Contribution. To test hypothesis 3A, three effects were investigated: an interaction between trial and group, main effect for trial, and main effect for group. The repeated measures ANOVA indicated there was a significant interaction between trial and group, F (3, 204) = 3.684, p = .013, η2 = .051. In addition, a significant main effect was found for trial, F (3, 204) = 13.98, p < .001, η2 = .171. Analyzing the between-subjects (group) effects, the ANOVA revealed no main effect for group, F (1, 68) = .221, p = .640, η2 = .003. Because a significant interaction between trial and group was found, follow-up analyses were conducted to look for simple main effects. To assess for a simple main effects for trial within group, separate one-way repeated measures ANOVAs were conducted, first for the Reading Tools group and second for the Without Reading Tools group. First, the results indicated a simple main effect for trial within the Reading Tools group, F (3, 102) = 8.931, p < .001, η2 = .208. Since there was a significance effect, a Pairwise comparison using t tests was conducted to determine in which trials there was a significant difference in utilization score. Bonferroni post hoc analyses found significant difference between Trials 1 and 2, 2 and 3, and 2 and 4 (Table 4.6). In all cases participants within the Reading Tools group reported more comfort in trial 2 (Mean = 4.17) compared to the other trials (Table 4.7). 109 Table 4.6 Hypothesis 3A Significance between Trials within Reading Tools Group Comparison Significance Value Trial 1 vs. Trial 2 .0001** Trial 1 vs. Trial 3 1.00 Trial 1 vs. Trial 4 1.00 Trial 2 vs. Trial 3 .001** Trial 2 vs. Trial 4 .001** Trial 3 vs. Trial 4 1.00 *p < .05 or **p < .0083 Table 4.7 Hypothesis 3A the Mean and SD for the Trials within the Reading Tools Group Trial Mean Standard D 1 3.314 .187 2 4.17 .119 3 3.34 .169 4 3.31 .204 Second, a test within the Without Reading Tool group was performed to establish whether or not there were differences in scores among the trials for the Without Reading Tool group. The repeated measures ANOVA results indicated there was a simple main effect for trial within the Without Reading Tools group, F (3, 102) = 8.678, p < .001, η2 = .203. The post hoc analysis found significant differences between Trials 1 and 3, and 2 and 3. Trial 3 scores were significantly lower than Trials 1 and 2 (Table 4.8). The significance values were 110 adjusted from .05 to .0083 according to the Bonferroni procedure to control for multiple comparisons and to protect against type I errors. Table 4.9 Hypothesis 3A Trial Comparisons within the Without Reading Tool Group Comparison Significance Value Trial 1 vs. Trial 2 1.00 Trial 1 vs. Trial 3 .006** Trial 1 vs. Trial 4 .300 Trial 2 vs. Trial 3 .001** Trial 2 vs. Trial 4 .084 Trial 3 vs. Trial 4 .634 *p < .05. **p < .0083. To examine the simple main effects of group within trial, four independent t tests with group as the independent variable and each of the trials as a dependent variable were performed (Table 4.9). Results indicated Trial 1 was significant. However, the alpha score was adjusted from .05 to .0125 because multiple comparisons were used and also to protect against type I errors. After adjusting the alpha level, results indicated that none of the Pairwise comparisons were significant. 111 Table 4.10 Hypothesis 3A Simple Main Effects of Group within Trial Group Mean Standard Deviation t df Significance Trial 1 Reading Toolsª 3.38 1.11 2.03 58.48 .047* Without Reading Toolsb 3.81 .66 Trial 2 Reading Toolsª 4.18 .69 1.43 72 .158 Without Reading Toolsb 3.94 .75 Trial 3 Reading Toolsª 3.39 1.02 .858 71 .394 Without Reading Toolsb 3.19 .97 Trial 4 Reading Toolsª 3.29 1.18 .761 72 .449 Without Reading Toolsb 3.47 .84 an = 38. bn = 36 *p < .05. **p < .0125. Results indicate no support for an interaction between the Reading Tools group and the Without Reading Tools group. In trial one an interaction would not be expected since neither group had access to the Reading Tools. However, an interaction would have been expected in trials 2, 3, and 4 in which the Reading Tool Access condition should have significantly higher article utilization scores. Results indicated no significant differences in scores between the two groups for any of the four trials. Hypothesis 3B Analysis of the relevant data gathered to test hypothesis 3B was submitted to a mixed design ANOVA. This ANOVA tested for an interaction between the within-subjects (repeated measure) variable (four trials) and the between-subjects variable (group). The students’ perception of utilization of the article is the dependant variable. Each student’s 112 perception of utilization of the article was derived from that student’s survey response score. After completing the critiquing assignment for each trial, participants answered survey questions related to their perception of utilization of the article. For H3B, the results of the survey question “How comfortable would you feel using this article working with other teachers in a school, whether to criticize or draw on its perspective?” served as the dependant variable. The scale used for the survey question was a scale ranging from 1 No Contribution to 5 High Contribution. To test hypothesis H3B, three effects were investigated: an interaction between trial and group, main effect for trial, and main effect for group. The repeated measures ANOVA indicated there was no significant interaction between trial and group, F (3, 207) = 1.11, p = .346, η2 = .016 (refer to Table 4.10 for descriptive statistics). However, a significant main effect was found for trial, F (3, 207) = 13.180, p < .001, η2 = .160. Analysis of the between- subjects effects revealed no main effect for group, F (1, 69) = .005, p = .944, η2 = .000. 113 Table 4.11 Hypothesis 3B Means and Standard Deviation Utilization Scores by Group Group Mean Standard Deviation Trial 1 3.62 .87 Reading Toolsª 3.56 1.00 Without Reading Toolsb 3.69 .72 Trial 2 4.07 .72 Reading Toolsª 4.19 .67 Without Reading Toolsb 3.94 .76 Trial 3 3.27 1.01 Reading Toolsª 3.28 1.16 Without Reading Toolsb 3.26 .85 Trial 4 3.51 1.01 Reading Toolsª 3.42 1.20 Without Reading Toolsb 3.60 .77 an = 38. bn = 36 Because a significant main effect for trial was found, further tests were conducted. The alpha score was adjusted from .05 to .0125 because multiple comparisons were used and also to protect against type I errors. After adjusting the alpha level, results indicated that Trials 1 and 2 were significantly different, where Trial 2 was significantly higher than Trial 1. Trials 2 and 3 were significantly different, where Trial 2 was significantly higher than Trial 3. In addition, Trials 2 and 4 were significantly different, where Trial 2 was significantly higher than Trial 4 (Table 4.11). All three significant values indicate that 114 participants regardless of group rated Trial 2 a higher utilization score than the other three trials. Table 4.12 Hypothesis 3B Significance between Trials Comparison Significance Value Trial 1 vs. Trial 2 .001** Trial 1 vs. Trial 3 .083 Trial 1 vs. Trial 4 1.00 Trial 2 vs. Trial 3 .001** Trial 2 vs. Trial 4 .001** Trial 3 vs. Trial 4 .403 *p < .05. **p < .0083. Results indicated no support for Hypothesis 3B. The pattern of differences among trials was different for the Reading Tools group compared to the pattern for the Without Reading Tools group. However, there were no significant differences between groups within any given trial. This shows that the participants were more comfortable with certain trials, however, there were no significant differences between the Reading Tools group and the Without Reading Tools group. Other Results Survey II consisted of two parts (Appendix B). All participants completed the first part (Part A) after submitting and discussing the last article as part of the assignment. Part A explored the students’ online reading versus print reading habits. Descriptive statistics indicate that for each trial the majority of students did not print out the article for each trial (Table 4.12). Sixty-two percent of the students did not print out any of the articles, while 115 only twelve percent printed out all four articles (Table 4.13). Students were given the opportunity to provide reasons for printing the articles or reading online. Those answers were tabulated and categorized. The most popular response regarding why students printed articles was that printing provided the ability to interact with the article, for example., annotate, and the flexibility to read anywhere and anytime (Table 4.14). The most popular response regarding why students read the articles online was that it was “easier” and more environmentally friendly (Table 4.15). Table 4.13 Number of Participants Printing the Article - Survey II Part A Students Online Versus Print Reading Habits Print Article 1 Article 2 Article 3 Article 4 No 53 53 60 58 Yes 21 21 14 16 Table 4.14 Survey II Part A Number of Articles Students Printed Number of articles students printed Number of Participants (% of total) N = 74 None 46 (62.1) One article 7 (9.5) Two articles 7 (9.5) Three articles 5 (6.8) Four Articles 9 (12.2) 116 Table 4.15 Survey II Part A Reasons Students Printed Reasons for printing Number of participant comments supporting the rationale (N=28) Interactivity – ability to highlight or annotate 8 Ability to read anywhere (beach or home) 7 Screen reading causes negative side effects (eye strain or headaches) 6 Limited access to technology (Internet or Computer) 6 Read better on paper 5 Out of habit 2 Reading enjoyment 2 Table 4.16 Survey II Part A Reasons Students Read Online Reasons for Reading Online # of participant comments supporting the Rationale (N=55) “Easier” 21 Save paper – environmental reasons 19 Economic reasons i.e., save money 15 Interactivity – ability to copy/paste, highlight 13 Lack of access to printing 8 Reading Tools 4 The second Part of Survey II (Part B) was created specifically for the participants in the Reading Tools group. The survey consisted of three questions with an optional space at the end to add any additional comments. The first two questions used a 5 point Likert scale 117 ranging from 1 No Contribution to 5 High Contribution. The first question investigated the students’ perceptions of how using the Reading Tools contributed to completing the assignment. Results indicated that there was no clear view on whether the Reading Tools assisted students in completing the assignment (Table 4.16). Further analysis indicated that some students believed the Reading Tools contributed positively; however, a slightly lower number of students believed there was little or no contribution (Table 4.17). The second question investigated whether the Reading Tools increased or decreased students’ ability to focus on the position of the article’s author. Results indicate that there is little evidence to support that the Reading Tools either increase or decrease students’ ability to focus (Table 4.18). 118 Table 4.17 Survey II Part B Question 1 Results To
what
degree,
if
at
all,
in
reading
the
assigned
article
did
the
Reading
Tools
 contribute
to…
 Scale
–
1
Low
Contribution
to
5
High
Contribution
 
 Mean
 Standard
Deviation
 Ability
to
learn
 3.30
 1.33
 Ability
to
assess
argument
value
 3.35
 1.42
 Ability
to
assess
argument
strength
 3.22
 1.55
 Confidence
in
presenting
 2.65
 1.27
 Engagement
with
issue
 2.54
 1.12
 119 Table 4.18 Reading Tools Access Group Perceptions of Reading Tools Contribution To what degree, if at all, in reading the assigned article did the Reading Tools contribute to… Scale – 1 Low Contribution to 5 High Contribution Count (%) Rated 1 Count (%) Rated 2 Count (%) Rated 3 Count (%) Rated 4 Count (%) Rated 5 Count (%) Rated either 4 or 5 Ability to Learn 6 (15.8) 4 (10.5) 6 (15.8) 15 (39.5) 6 (15.8) 21 (55.3) Ability to Assess 7 (18.4) 3 (7.9) 5 (13.2) 14 (36.8) 8 (21.0) 22 (57.8) Ability to Assess Argument Strength 8 (21.0) 5 (13.2) 6 (15.8) 7 (18.4) 11 (28.9) 18 (47.3) Confidence in Presenting 9 (23.7) 8 (21.0) 10 (26.3) 7 (18.4) 3 (7.9) 10 (26.3) Engagement with Issue 7 (18.4) 13 (34.2) 8 (21.1) 8 (21.1) 1 (2.6) 9 (23.7) 120 Table 4.19 Reading Tools Impact on Ability to Focus To
what
degree,
if
any,
did
the
Reading
Tools…
 Scale
–
1
Low
Contribution
to
5
High
Contribution
 
 Mean
 Standard
Deviation
 Increase
ability
to
focus
 2.97
 1.36
 Decrease
ability
to
focus
 1.46
 .87
 The third question of Survey II Part B asked students to rate each of the “Reading Tools.” These questions were asked to provide insight into which tools were used and how highly they were rated. This question used a 1 to 5 Likert Scale from 1 No Contribution to 5 High Contribution, with the option of “Did Not Use.” Results indicate a wide range of use and ratings for the Reading Tools (Table 4.19). 121 Table 4.20 Survey II Part B Question 3 Reading Tool Usefulness Rate the following tool on their usefulness to your completion of this assignment . . . Tool Mean SD # did not use Abstract 3.72 1.40 12 Review policy 2.93 1.07 23 About the author 3.08 1.56 25 How to cite item 3.25 1.24 21 Indexing metadata 2.86 1.35 30 Print version 3.36 1.63 26 Look up items 3.52 1.25 10 Notify colleague 2.25 1.50 33 Email the author 3.00 1.67 31 Author’s work 4.28 .96 5 Book searches 4.09 1.04 26 E-journals 3.52 1.19 12 Related theory 3.63 1.01 13 Related studies 3.92 1.09 11 Pay-per-view 1.50 1.00 33 Online forums 1.50 1.00 33 Multimedia 2.00 1.20 29 Teaching files 2.80 1.79 32 Relevant portals 3.14 1.35 30 Government policy 3.18 1.47 25 Media reports 3.79 1.12 23 Web search 3.76 1.22 16 Look up a word 3.79 1.22 13 Search journal 3.00 1.37 21 Note. The likert scale used to respond to the question was 1 Low Contribution to 5 High Contribution. In addition, the Mean is based on only those students who used the tool. 122 The final question of Survey II Part B was an open question that provided all students with the option to provide any additional feedback about the Reading Tools. Results indicated an overwhelmingly positive response toward the Reading Tools (Table 4.20). Results of this question are used to support the discussion in Chapter 5. Table 4.21 Survey II Part B Optional Open Comments – All Comments Reported Student Comments 1 I found them non-intrusive but extremely valuable when I required aid. 2 If you could incorporate tools like highlight and stickynote as in diigo you would have a complete package of reading tools. Even when I printed the article I kept the program open so I could use the Reading Tools while reading. I started my search for other references using the Reading Tools. Sometimes this satisfied my search, but with the information literacy paper and cyberbullying papers the searches didn't satisfy so I did further searches on my own using journal databases from the library. I think your journal searches would be more successful if more journals are searched, so I guess you would need more open access journals to get involved. Good Luck with everything:-) 3 When you pressed on one of the links, it took you to a new window and it was hard going back to it to refine your search. Ex. Searching for related articles. You would search one journal, but then to go back to the search engine you would have to open up a new link. This was a little inconvenient. The author's background info was extremely helpful. 4 The Reading Tools were useful in finding out about the author and his or her background. I believe an orientation with the tools would have swayed me to use them more when coming up with talking points. 5 I could see these tools being useful. For the talking points I only used them to find out about the author. I did not use the searching tools because I like using the library and google scholar. 6 While I didn't use most of the Reading Tools, I found that the ones I did use were helpful. Perhaps a short lesson on how to effectively use the Reading Tools would be a good idea. 123 Student Comments 7 I didn't know what the reading tools would do or how to use them. A tutorial (online) would have been helpful. 8 I did not get to use reading tools for these articles, but I wish that I did. The people sitting around me kept commenting how useful they found the reading tools, and that they thought the tools would help them get the assignment done faster. 9 The reading tools are kickass. But if an author has time he could create his article to pop up other research material that supports his or her work. This research tool definitely helped myself in further understanding the articles and increased my speed and efficiency. Thanks I really enjoyed them. 10 I loved the Reading Tools. After using them in the second article, I was worried about not having them on the next article (thinking that I would be crossed-over into the \"no tools\" group next time. Glad I wasn't! 11 I really liked Reading Tools - finding related articles and info related to author was very helpful. The government policy link was great for some of the issues related to Canada. It made me feel like I really understood the article because it gave me more credible resources. It was very helpful using look up terms, some of the more scientific words I needed help with. 12 Excellent way in linking information and resources from the article to information on the Internet without opening another browser. Convenient and easy to use. Linked to many resources, but also gives the opportunity to \"suggest another source\" that is not listed. Liked the idea of double-clicking the word, then it goes to the definition of the word. 13 I found the Reading Tools to be highly efficient in time management. That is, the word definitions, \"about the author\" and related links were very helpful to have just one click away. Overall, the most beneficial tools for me were the author's information/background and the word definitions, link to online dictionaries. 124 CHAPTER 5 DISCUSSION: DO THE READING TOOLS MAKE A DIFFERENCE? Introduction This chapter discusses the results presented in Chapter 4 with emphasis on the implications of this study and recommendations for further research. In addition, the limitations of the study are also discussed. This research study assesses the degree to which a specific set of online Reading Tools, are able to help students critique an article as part of an assignment. Specifically, the study assessed whether the set of Reading Tools used by Open Journal Systems assists university students in the context of an article critiquing assignment by contributing to the following: a) comprehension of an article; b) ability to evaluate the quality of the article; and c) augment their level of confidence in using the article, critically or supportively, as part of their future work as teachers or university students. This study used a convenience sample of 75 pre-service teacher education students who comprised two sections of the compulsory course entitled “Information Communication Technology for Secondary Teachers.” Each section of students (section 1 n = 41, section 2 n=34) were randomly assigned to either one of two groups based on access to the Reading Tools. To establish a baseline for comparison within the Reading Tools group, no students had access to the Reading Tools for the first article. For articles 2 – 4 the respective groups either had or did not have access to the Reading Tools. As part of the assignment students were asked to create a list of 10-12 talking points for each article prior to the class. They used the points as a basis for discussing the article with their groups (Appendix C). These 125 talking points were then marked by two independent markers using a rubric (Appendix D) to arrive at a “comprehension” and “critique” score. After submitting the talking points and before discussing each article, students completed an online survey with a Likert rating system that asked, “How comfortable would you feel using this article in a class presentation, whether to criticize or draw on its perspective?” and “How comfortable would you feel using this article in working with other teachers in a school, whether to criticize or draw on its perspective?” These survey results were used to provide a quantitative score related to whether or not the Reading Tools increased the students’ perceived ability to utilize the article in their work in schools. Students in the treatment groups were compared to students who did not have access to the Reading Tools. A mixed design ANOVA was used to analyze the data for all three hypotheses. The between variables included students’ access to the Reading Tools (treatment) and students without access to the Reading Tools (control). The repeated measures were the four articles (trials). The analysis indicated there were no significant differences found between the two groups of students in the areas of comprehension, ability to critique, and utilization of the Reading Tools. However, student survey feedback indicated a positive perception that the Reading Tools provided value to the online research reading environment. Discussion Introduction No support emerged from the current study for any of the three hypotheses. The study provided no support for hypothesis 1, which predicted students who had access to the Reading Tools while completing the assignment would have higher comprehension scores. In 126 addition, the study provided no support for hypothesis 2, which predicted students who had access to the Reading Tools while completing the assignment would have higher critique scores. Furthermore, the results of the current study provided no support for hypothesis 3, which predicted that students with the Reading Tools access would have significantly higher article utilization scores. Although there was no evidence for support for any of the three hypotheses, results from Surveys I and II indicated that students perceived the Reading Tools as useful, although that judgment was made without a basis of comparison. The following section will discuss how the results of this study align with the literature reviewed in Chapter 2, how student interaction with the tools may have impacted the results and how the findings of this study can inform the research and development of online reading tools. Literature Review Response This study attempted to build on the findings of other studies that investigated reading tools that reside next to online journal articles. Willinsky (2004), Willinsky & Quint- Rapoport (2007), and Blake et al. (2007) focused their research on determining to what degree the Reading Tools used in Open Journal Systems or Pub Med’s tools and services, enhance readers’ experience in online scholarly research environments. Willinsky’s (2004) investigation exposed the need to refine the Reading Tools and to determine if the refined Reading Tools are less confusing. Subsequently, the Reading Tools were reorganized and new databases were added to search for related materials. This study used the new version of the Reading Tools to test the three hypotheses. Blake et al. (2007) investigated whether or not Reading Tools assist humanities scholars and computing scholars in comprehending, evaluating and utilizing the research 127 literature in their respective disciplines. The author’s study reinforced the need to investigate other populations. They hypothesized that other populations, besides humanities and computing scholars, who may not have online critical reading procedures in place, may benefit from using the Reading Tools to assist in finding related studies. Furthermore, participants in Blake et al.’s study identified students as one population that may find the tools useful. The sample of users in this study presented here consisted of students who were novices in the specific knowledge domain. As discussed later in this chapter, survey results indicate that many of the students who participated in the present study may not find the Reading Tools very useful because they already use other online information retrieval procedures. Willinsky & Quint-Rapoport (2007) investigated how three types of PubMed Tools and services (information retrieval, access, and management) contributed to the scholarly research article reading experience. Participant feedback indicated that the tools contributed to engagement and comprehension of the research article. Based on the results, the authors recognized that further study is needed to assess the impact of this type of tool on comprehension, ability to evaluate, and utilization of the article. The present study attempts to measure quantitatively whether or not Reading Tools affect comprehension, the ability to evaluate the author’s contribution to the field, and the utilization of the article. However, as discussed, study results do not support the Reading Tools’ positive effect, even if participants in both studies felt that the tools were useful. As discussed in Chapter 2, this study attempted to build upon a number of important lessons. The literature indicates that the use of navigational tools may enhance knowledge acquisition in complex nonlinear hypertext environments. Results of this study did not 128 support the benefits of Reading Tool access in terms of significantly higher comprehension of an article; ability to evaluate the quality of the article; or augment the level of confidence in using the article critically or supportively as part of future work as teachers or university students. However, as will be discussed later in this chapter, students provided positive feedback on the usefulness of the Reading Tools. In addition to indicating that the use of navigational tools may enhance knowledge acquisition in complex nonlinear hypertext environments, the literature suggests that the Reading Tools would increase the nonlinearity of text, learner control, and the complexity of the reading task. Consequently, the Reading Tools would increase disorientation and cognitive load, both of which may impede knowledge acquisition. Students were surveyed to determine overall perception as to whether or not the Reading Tools increased or decreased their ability to complete the assignment. Students from the Reading Tools group completed Survey II (Appendix B) after all four trials (note that survey I addressed information retrieval strategies and will be discussed later in this chapter). As part of survey II, two questions using a 1 to 5 Likert scale (1 No Contribution to 5 High Contribution) addressed: “To what degree, if any, did the Reading Tools increase your ability to focus on the author’s position?” and “To what degree if any, did the Reading Tools decrease ability to focus . . . on the author’s position?” Results of the two survey questions (Table 5.1) indicate that students did not perceive that the Reading Tools either increase disorientation or cognitive load sufficiently to impede their knowledge acquisition, or increase their ability to focus. There was also little evidence to support that the Reading Tools actually increased the ability to focus. 129 Table 5.1 Participant Reading Tool Access Group Descriptive Results Survey II Part B Question 2 To what degree, if any, did the Reading Tools . . . Mean Standard Deviation Increase Ability to Focus 2.97 1.36 Decrease Ability to Focus 1.46 .87 Note. The Likert scale used to respond to the question was 1 Low Contribution to 5 High Contribution. Discussion of Results The findings of this study suggest that students who had access to the Reading Tools did not have significantly higher scores than students who did not have access to the Reading Tools regarding comprehension of an article, ability to evaluate the quality of the article; and level of confidence in using the article; critically or supportively, as part of their future work as teachers or university students. However, Survey I and II results indicated that many of the students used the Reading Tools and perceived the tools as useful. In Survey II Part B some students rated the Reading Tools as contributing highly to their ability to learn, their ability to assess argument strength, and their confidence in presenting and engaging with the issue (Table 5.2). The majority of students rated the Reading Tools contribution to their ability to learn and their ability to assess as either 4 or 5 (scale 1 Low Contribution to 5 High Contribution). However, the results also indicated that fewer students felt the Reading Tools contributed to their confidence in presenting and in their engagement with the issue. 130 Table 5.2 Reading Tools Access Group Perceptions of Reading Tools Contribution To what degree, if at all, in reading the assigned article did the Reading Tools contribute to… Count (%) Rated 1 Count (%) Rated 2 Count (%) Rated 3 Count (%) Rated 4 Count (%) Rated 5 Count (%) Rated either 4 or 5 Ability to learn 6 (15.8) 4 (10.5) 6 (15.8) 15 (39.5) 6 (15.8) 21 (55.3) Ability to assess 7 (18.4) 3 (7.9) 5 (13.2) 14 (36.8) 8 (21.0) 22 (57.8) Ability to assess argument strength 8 (21.0) 5 (13.2) 6 (15.8) 7 (18.4) 11 (28.9) 18 (47.3) Confidence in presenting 9 (23.7) 8 (21.0) 10 (26.3) 7 (18.4) 3 (7.9) 10 (26.3) Engagement with issue 7 (18.4) 13 (34.2) 8 (21.1) 8 (21.1) 1 (2.6) 9 (23.7) Note. The Likert scale used to respond to the question was 1 Low Contribution to 5 High Contribution. Student comments from Survey II provide further support for students’ positive perception of the Reading Tools. For example, one student commented, The reading tools are kickass. But if an author has time he could create his article to pop up other research material that supports his or her work. This research tool definitely helped myself in further understanding the articles and increased my speed and efficiency. Thanks I really enjoyed them. A second student commented, I really liked Reading Tools - finding related articles and info related to author was very helpful. The government policy link was great for some of the issues related to Canada. It made me feel like I really understood the article because it gave me more 131 credible resources. It was very helpful using look up terms, some of the more scientific words I needed help with. A third student commented, I found the Reading Tools to be highly efficient in time management. That is, the word definitions, ‘about the author’ and related links were very helpful to have just one click away. Overall, the most beneficial tools for me were the author's information/ background and the word definitions, link to online dictionaries. Students may have perceived the Reading Tool as useful even if the tools did not make a difference to their classroom assignments. Student Interaction with the Reading Tools: Access versus Actual Use An important part of this study was to create an authentic learning activity that simulates a scenario in which a student naturally comes across an article while completing an assignment and discovers the Reading Tools residing adjacent to the article. We structured the experiment to closely as possible approximate this ideal scenario. The tools were brought to the participants’ attention; however, they were not taught or demonstrated in any way. In addition, students were not told they had to use the tools. This environment was designed to test whether access to the Reading Tools would enhance a student’s comprehension, their ability to critique, and their perception of the utilization of an article in the context of an authentic university assignment. The following section discusses insights into the participants’ information retrieval strategies, including whether or not students used the tools, how they were used, and some of the potential barriers to fully leveraging the tools. 132 Did Students Use the Tools? Providing access to the Reading Tools for the Reading Tools group did not guarantee that participants would leverage the tools in a meaningful way or even use them at all. As part of the online Survey I (Appendix A), participants from both groups answered two questions each time they submitted their talking points. They are: “If there were points or concepts in the article that you were not familiar with, what did you, if anything, do to figure them out?” and “What strategies did you use to evaluate the article, to establish the degree to which you should trust its claims?” The answers to these questions, the Reading Tool ratings, and optional comments that the students provided as part of Survey II (Appendix B) give insight into whether or not students with access to the Reading Tools used the tools, and in what way. Table 5.3 Reading Tools use from the Reading Tools Group Type of Reading Tools use Number of students (Percent) No use 4 (10.5%) Minimal use 12 (31.5%) Used Reading Tools 22 (58%) Analysis of the survey responses indicated that 34 of the 38 students who were given Reading Tool access used the Reading Tools in some way (Table 5.3). Out of the four students who did not use the Reading Tools at all, one did not even realize there was access available to the Tools. The remaining 34 participants used the Reading Tools either minimally or in a meaningful way. Out of the 34 students only 12 used the tools minimally (Table 5.3). These students typically used simple tools that provided a specific function. For 133 example, some used the “Look Up Words” and “About the Author” tools. The “About the Author” tool provides a biography of the author and links to where a reader could find more biographical information. This tool provided a quick way to answer one of the important parts of the assignment: “Why the author should or should not be judged as having a contribution to make to this issue.” Students who used the Reading Tools minimally typically used either Library Databases such as ERIC or Academic Premier, Google, or Google Scholar to find additional articles, instead of using the Reading Tools for that function. One student commented, “I could see these tools being useful. For the talking points I only used them to find out about the author. I did not use the searching tools because I like using the library and Google Scholar.” Another student commented on using “About the Author, in the Reading Tools and my own searches on Academic Search Premier, ERIC and Google Scholar.” A third student commented, “The tool bar had a link to find out more about the author which was helpful. I also used ERIC and Google Scholar and found more articles the author had written.” In total, 22 out of the 38 students from the Reading Tools group used the Reading Tools to look up words and to find out about the author, and search for additional articles (Table 5.3). With 16 out of 38 students (42%) not using the Reading Tools (or using them minimally), it is less likely that an interaction would be found between the students with Reading Tools group and the Without Reading Tools group. Moreover, the question is raised as to what may have prevented those 16 students from using the Reading Tools more extensively. In the next section, barriers to Reading Tools use are discussed. Barriers to Reading Tool Use The results of this study indicate that a number of factors may have inhibited the students from fully leveraging the Reading Tools to complete the assignment. From student 134 feedback, lack of training in how to use the tools was identified as one potential barrier to using the Reading Tools. Students also indicated that they already had information retrieval strategies and did not have the time to, or recognize the value of, switching strategies and trying the Reading Tools. In addition, some of the students who tried the tools found that they were not comprehensive enough and that they needed other tools in conjunction with or instead of the Reading Tools. Lack of Training. The tools were not taught or demonstrated in any way. This allowed the researcher to create an authentic learning environment that would serve the students who are reading online and outside of the classroom to simulate a student who naturally came across the article while completing an assignment and discover the Reading Tools residing adjacent to the article. However, some students felt assistance in using the Reading Tools may have been beneficial. One student commented, “While I didn't use most of the Reading Tools, I found that the ones I did use were helpful. Perhaps a short lesson on how to effectively use the Reading Tools would be a good idea.” Another student commented, “The Reading Tools were useful in finding out about the author and his or her background. I believe an orientation with the tools would have swayed me to use them more when coming up with talking points.” This comment points to the fact that some of the tools may be intuitive and others more complex. A third student commented, “I didn't know what the reading tools would do or how to use them. A tutorial (online) would have been helpful.” This perception of a need for training suggests that online tutorials and/or more intuitive tools for some of the students would be beneficial. 135 Students Already Have Information Retrieval Strategies. The majority of students in this sample had already completed four years of university and established strategies for completing similar assignments. Students commented that they already had strategies and did not want to take the time to learn new tools. In addition, students were not given access to the Reading Tools for the first article. This created a condition that encouraged the students to establish a way of completing the assignment satisfactorily without the Tools. One student who already had established information retrieval strategies commented, “I could see these tools being useful. For the talking points I only used them to find out about the author. I did not use the searching tools because I like using the library and Google Scholar.” Another student tried the new tools; however, this student fell back to familiar strategies when the Reading Tools were not successful. The student commented “I used the Reading tools, and if they did not provide what I needed… I went to my normal article search through ERIC or Academic Search premier for a paper on the subject.” Another student commented verbally about having critiqued many articles before and having experience with the Library databases and did not see any need to change strategies. Limitations of the Reading Tools. Students would truly need to see added benefits for them to adopt a new tool that would require a change in their information retrieval strategies. As discussed above, some students reported how the Reading Tools assisted them with the assignment. However, limitations of the Reading Tools may have influenced whether students used the tools and whether they would enhance a student’s comprehension, ability to critique, and perception of utilization of an article. Students made comments about some of the limitations of the 136 Reading Tools, and, consequently, as a result of these limitations many found that they needed to use other tools to compensate for the perceived short comings of the Reading Tools. Student responses to Surveys I and II indicate that 22 students did not use the Reading Tools exclusively and felt that they should be used in conjunction with other search tools. The following comments reveal the lack of comprehensiveness of the tools, the need for additional information retrieval tools, and explain the reason why many students needed to fall back on “tried and true” information retrieval strategies when the Reading Tools were not, in their minds, successful. One student commented, “I researched the author and the journal to see their credentials. I used both a Google search as well as used the links in the reading tools.” Another student commented, I used the link in the Reading Tools, Related Articles, to search for related work from other authors. However, 4 out of 6 times, there were no results given or the articles only contained abstracts and not the full article. I logged into the UBCO Library instead and searched indexes and databases. Also used Google Scholar. A third student commented, I started my search for other references using the Reading Tools. Sometimes this satisfied my search, but with the information literacy paper and cyberbullying papers the searches didn't satisfy so I did further searches on my own using journal databases from the library. I think your journal searches would be more successful if more journals are searched, so I guess you would need more open access journals to get involved. 137 A fourth said, I used the info on the Author given through reading tools and used the authors works reading tool. However I was not overly impressed with the results the Authors Works reading tool got for me, so I continued further search on ERIC on my own. In addition, some students offered advice on how the Reading Tools might be improved. One student suggested, “If you could incorporate tools like highlight and stickynote as in Diigo you would have a complete package of reading tools.” Rating of the Tools. Results of Survey II indicate that students found some of the tools more useful than others in this particular context (Table 5.4). For example, “Look Up Items,” “Author’s Work,” and “Related Studies” were the most popular tools, and “Author’s Work” and “Book Searches” were the most highly rated. Although more research is needed, this data may provide insight into the development of future types of tools for the enhancement of the online scholarly environment. 138 Table 5.4 Reading Tools Group Rating of Each Tool Rate the following tools on their usefulness to your completion of this assignment… Tool Mean SD # did not use Look up items 3.52 1.25 10 Authors work 4.28 .96 5 Book searches 4.09 1.04 26 E-journals 3.52 1.19 12 Related theory 3.63 1.01 13 Related studies 3.92 1.09 11 Media reports 3.79 1.12 23 Web Search 3.76 1.22 16 Look up a Word 3.79 1.22 13 Note. The Likert scale used to respond to the question was 1 Low Contribution to 5 High Contribution. Peer Influence. There may have been an unnatural peer influence impacting whether or not students used the Reading Tools. Reading Tools use may have been influenced by classmates who either promoted or demoted their use. There may have been some students who used the tools and promoted them. For example, one student from the Without Reading Tools group commented, 139 I did not get to use Reading Tools for these articles, but I wish that I did. The people sitting around me kept commenting how useful they found the Reading Tools, and that they thought the tools would help them get the assignment done faster. This promotion may have positively impacted whether the Reading Tools were used, and may in part explain an adoption rate difference between the two sections of students participating in the study. Section two of the course seemed to have a higher adoption rate of the Reading Tools than section one (Table 5.5). A subsequent analysis of data between the two sections was completed; however, no significant difference in scores between sections was found. Table 5.5 Reading Tool Use by Course Section Type of Reading Tool Use Section 1 Section 2 No Use 4 0 Minimal Use 11 1 Used the Tools 6 16 The Power of Google and Google Scholar. Participants’ use of Google Scholar may well have reduced the value of the Reading Tools. If students already use Google Scholar successfully, why would they change and use the Reading Tools? Do the Reading Tools provide features that Google Scholar does not? The Reading Tools were created prior to Google Scholar and may now be obsolete. Google Scholar has continually improved its database and features. Furthermore, Google Scholar provides the ability to search with context sensitive linking that can display search results 140 that contain an additional link to the user’s library’s database. The question remains: What benefit does the Reading Tools have over tools like Google Scholar? Students were not directly asked whether they used Google or Google Scholar; however, both search engines were often mentioned when students answered the following two Survey I questions: “If there were points or concepts in the article that you were not familiar with, what did you, if anything, do to figure them out?” and “What strategies did you use to evaluate the article, to establish the degree to which you should trust its claims?” In the Reading Tools group, 24 out of the 38 students (63%) mentioned using Google or Google Scholar. Google Scholar is a comprehensive tool which offers features that are very beneficial when critiquing an article. For example, Google Scholar provides a forward referencing tool that offers a good mechanism for checking two of the guiding questions that made up part of the assignment that was the focus of this study: 1) why the author should or should not be judged as having a contribution to make to this issue and 2) how the author’s position relates to other work in the field, and stances on the issue including, but not restricted to, the reader’s ideas. One student commented that he/she had used Google Scholar to “look up the author and checked out his other articles and if anyone had cited him.” In addition, another student said, “I used Google scholar to check the number of citations the article has received.” These comments indicate that some of the students had the ability to leverage Google Scholar in a meaningful way. Information Retrieval Strategies for the Without Reading Tools Group Another factor contributing to the results is that the students who were members of the Without Reading Tools group had the skill and the resources to complete the assignment 141 just as well even though they did not have access to the Reading Tools. In online Survey I (Appendix A) students from both groups were asked the following two questions: “If there were points or concepts in the article that you were not familiar with, what did you, if anything, do to figure them out?” and “What strategies did you use to evaluate the article, to establish the degree to which you should trust its claims?” Responses to these questions provided insight into how the Without Reading Tools group were able to complete the assignment just as well as the Reading Tools group. Analysis indicated that students in the Without Reading Tools group used a combination of resources to complete their assignment. Google, Google Scholar, online dictionaries, Wikipedia, and Library databases such as EBSCO, JSTOR, and ERIC were used. 19 out 37 (51.4%) of the students mentioned Google or Google Scholar by name. Summary of Discussion Although 34 out of 38 of the students in the Reading Tools group used the tools in some way, the majority rated the contribution of the Reading Tools highly and commented on how they liked using the tools, although the tools had no impact on their assignments. Possible reasons why the results did not reach significance are that many of the students in the Reading Tools group did not fully leverage the tools and that students in the Without Reading Tools group were able to complete the assignments using a variety of other online tools. Possible reasons why the Reading Tools group did not take greater advantage of the tools, in addition to inherent weaknesses in the tools’ design and conception, include participants’ lack of training with the Tools, participants already having information retrieval 142 strategies, and the availability of other tools such as Google Scholar that may be more efficient. Limitations of the Study Several possible limitations in this study may have influenced the results. Inter- marker reliability, researcher bias, and differences between the articles are limitations that were presented in Chapter III. Moreover, designing an authentic experience for the readers also presented limitations. In an effort to design a study to emulate a natural experience with the Reading Tools, there was a trade off between obtaining a direct measure of the usefulness of the Reading Tools and providing an authentic reading experience. The study asked students to create talking points as preparation to discuss an article in a scholarly manner. These talking points existed as an indirect measure of the contribution of the Reading Tools. However, students were also able to complete the assignment using other means. Furthermore, the students thought highly of the tools, but this positive feedback may have occurred because the tools made it more convenient to do the assignment. Students were still able to complete the assignment using databases such as Academic Search Premier and ERIC; Internet based search engines such as Google and Google Scholar; and tools such as online dictionaries and Wikipedia. Future Research Based on the results of this study a number of lessons emerge that may inform the research and development of tools that would enhance the online research article reading 143 environment. As discussed, the results indicated no support for any of the three hypotheses; however, students did have a positive perception of the usefulness of the Reading Tools. This may in part be attributed to the fact that students found using the tools a convenient way to complete the assignment. However, this convenience factor requires further exploration. The developers of online research reading environments need to assess the purpose of including the tools and ask the question whether it is the purpose of the tools to enhance the reader’s learning, to save the reader time, or to assist in a new form of reading online research. The way students read online is changing. Online research readers now “bounce” between articles as they skim an article and move on to the next article of interest. “Average times that users spend on e-journal sites are very short: typically four and eight minutes respectively” (Centre for Information Behaviour and the Evaluation of Research, 2008, p.10). The new form of reading relies on power browsing horizontally through titles, content pages, and abstracts looking quickly for results (Centre for Information Behaviour and the Evaluation of Research, 2008). The Reading Tools in this study would support this new form of reading. It is important to note that further research is required to explore whether or not these or other types of tools have value for university level readers. An important consideration for any set of online tools is their intuitiveness. If readers cannot operate the tools they will not be able to leverage them properly. A few of the participants commented that it would be useful to have an orientation session with the tools. Although this study purposefully created an authentic test of the Reading Tools (by not requiring students to use the tools) it would be helpful to know how useful the Reading Tools could be if the students knew how to fully leverage them. In Survey II Part B one student 144 commented that, “I didn't know what the reading tools would do or how to use them. A tutorial (online) would have been helpful.” Blake et al. (2007) found that participants in this study who already had methods of finding additional research did not find the Reading Tools useful. Informal feedback received in this study indicates that the participants had similar experiences. Students informally commented that they already had ways to find additional resources using the online library databases, and therefore did not feel the need to learn a new way to access additional resources. However, this issue needs to be formally tested. It may be useful to assess readers from a different target audience who do not have formal information retrieval strategies. This audience may find the tools helpful in locating additional resources. For example, teachers who are going back to graduate school to complete a Masters of Education after years in the field may find the tools more beneficial than current students who have more experience using information retrieval tools. Another group to test is a sample of general public users. This population does not normally access academic library databases and could benefit from the open access literature that is available through the Reading Tools. For example someone who is looking for medical information related to an illness and may find the links to open access literature useful. The medical information they retrieve could be complex and the related materials supplied by the Reading Tools could potentially aid the reader’s comprehension. Based on findings in this study it is recommended that an analysis of all potential reading tools should be conducted. This would include tools such as those provided by PloS One and HighWire Press that leverage Google Scholar to help readers locate additional related materials that might aid the reading process. The study undertaken here did not 145 directly investigate participant use of Google Scholar, but it would be useful to determine whether or not the Reading Tools have been “remediated” by Google Scholar. It is recognized that there is a link from the Reading Tools to Google Scholar; however, an issue that needs to be investigated is whether there are beneficial value added tools that Google Scholar can not offer. In addition, the HighWire Press and Public Library of Science (Plos One) tools use different individual tools than OJS. These tools may be better for students with the level of education described in this study because they include forward referencing. The forward referencing feature assists a reader in finding other article’s that have cited the article being read. This feature provides information that can help readers judge whether the author of an article is in a position to write on a particular subject. A study comparing the tools offered by various organizations may assist with designing a new version of reading tools. This study reinforces the need to assess the effectiveness of new versions of reading tools with a diverse number of online scholarly reading audiences. 146 REFERENCES Alexander, P. A. (2000). Toward a model of academic development: Schooling and the acquisition. Educational Researcher, 4, 28-33. Alexander, P. A., & Jetton, T. L. (2000). Learning from text: A multidimensional and developmental perspective. In M. L. Kamil, P. B. Mosenthal, P. D. Pearson, & R. Barr (Eds.), Handbook of reading research (Vol. 3, pp. 285-310). Mahwah, NJ: Erlbaum. Alexander, P. A., Kulikowich, J. M., & Jetton, T. L. (1994). The role of subject-matter knowledge and interest in the processing of linear and nonlinear texts. Review of Educational Research, 64, 201–252. Altun, A. (2000). Patterns in cognitive processes and strategies in hypertext reading: A case study of two experienced computer users. Journal of Educational Multimedia and Hypermedia, 9(2), 35-55. Beit-Arie, O., Blake, M., Caplan, P., Flecker, D., Ingoldsby, T., Lannom, L.W., Mischo, W.H., Pentz, E., Rogers, S. and Van de Sompel, H. (2001). Linking to the appropriate copy. D-Lib Magazine, 7(9). Blake, A., Leitch, C., Siemens, R., & Willinsky, J. (2007). It may change my understanding of the field: New reading tools for scholars. Paper presented at the PKP Scholarly Publishing Conference, Vancouver, Canada. 147 Bolter, J. D. (2001). Writing space: Computers, hypertext, and the remediation of print [2nd edition]. Mahwah, NJ: Lawrence Erlbaum. Bolter, J. D., & Grusin, R. (1999). Remediation: Understanding new media. Cambridge, MA: MIT Press. Boyce, P., Owens, E., & Biemesderfer, C. (1997). Electronic publishing: Experience is telling us something. Serials Review, 23 (3) 1-9. Branch J. L. (2000). The trouble with think alouds: Generating data using concurrent verbal protocols (Ed.). CAIS 2000: Dimensions of a Global Information Science. Canadian Association for Information Science Proceedings of the 28th Annual Conference. Bresciani, M. J., Zelna, C. L., & Anderson, J. A. (2004). Criteria and rubrics. In: Assessing Student Learning and Development: A Handbook for Practitioners, Washington, DC: National Association of Student Personnel Administrators, 29–37. Brinkerhoff, J. D., Klein, J. D., & Koroghlanian, C. M. (2001). Effects of overviews and computer experience on learning from hypertext. Journal of Educational Computing Research, 25, 427–440. Calisir, F., & Gruel, Z. (2003). Influence of text structure and prior knowledge of the learner on reading comprehension, browsing, and perceived control. Computers in Human Behavior, 135-145. 148 Calisir, F., Eryazici, M., & Lehto, M. (2008). The effects of text structure and prior knowledge of the learner on computer-based learning. Computers in Human Behavior, 24, 439-450. Centre for Information Behaviour and the Evaluation of Research (CIBER). (2008). Information behaviour of the researcher of the future. Scholarly Communications Report, 12(1), 1-35. Retrieved February 12, 2008, from http://www.ucl.ac.uk/slais/research/ciber/downloads/ Chen, C., & Rada, R. (1996). Interacting with hypertext: A meta-analysis of experimental studies. Human-Computer Interaction, 11, 125-156. Chen, S., Jing-Ping, F., & Macredie, R. (2004). Navigation in hypermedia learning systems: Experts vs. novices. Computers in Human Behavior. Available online at http://people.brunel.ac.uk/~csstsyc/science.pdf. Choo, C., Furness, C., Paquette, S., van den Berg, H., Detlor, B., Bergeron, P., & Heaton, L. (2006). Working with information: Information management and culture in a professional services organization. Journal of Information Science, 32(6), 491-510. Conklin, J. (1987). Hypertext an introduction and survey. IEEE Computer, 20(9), 17-41. Cotton, D., & Gresty, K. (2006). Reflecting on the think-aloud method for evaluating e- learning. British Journal of Educational Technology, 37(1), 45-54. Cress, U. & Knabel, O. (2003). Previews in hypertext: Effects on navigation and knowledge acquisition. Journal of Computer Assisted Learning, 19, 517-527. 149 Cuddy, C. (2005). HighWire press and its journey to become the world’s largest full-text stm online journal collection, Journal of Electronic Resources in Medical Libraries, 20 (1), 1-13. DeStefano, D., & LeFevre, J. A. (2007). Cognitive load in hypertext reading: A review. Computers in Human Behavior, 23(3), 1616-1641. De Jong, T., & Van der Hulst, A. (2002). The effects of graphical overviews on knowledge acquisition in hypertext. Journal of Computer Assisted Learning, 18, 219-231. Eason, K., Harker, S., Apps, A. & MacIntyre, R. (2004). Towards an integrated digital library: Exploration of user responses to a ‘Joined-Up’ service. Lecture Notes in Computer Science, 3232, 452-463. Eveland, W.P., & Dunwoody, S. (2002). An investigation of elaboration and selective scanning as mediators of learning from the Web versus print. Journal of Broadcasting and Electronic Media. 46, 34-53. Eveland, W.P., Marton, K., & Seo, M. (2004). Moving beyond “just the facts”: The influence of online news on the content and structure of public affairs knowledge. Communication Research, 31, 82 - 108. Farrell, I, & Moore, D. (2000/2001). The effect of navigation tools on learners’ achievement and attitude in a hypermedia environment. Journal of Educational Technology Systems. 29(2), 169-181. 150 Ford, N., & Chen, S. (2000). Individual differences, hypermedia navigation and learning: An empirical study. Journal of Educational Multimedia and Hypermedia, 9, 281-311. Garson, G. David (n.d.). Intraclass Correlations (ICC), from Statnotes: Topics in Multivariate Analysis. Retrieved April 8, 2008 from http://www2.chass.ncsu.edu/garson/pa765/index.htm. Graff, M. (2002). Learning from Hypertext and the Analyst-Intuition Dimension of Cognitive Style. In G. Richards (Ed.), Proceedings of World Conference on E- Learning in Corporate, Government, Healthcare, and Higher Education 2002 (pp. 361-368). Chesapeake, VA: AACE. Harter, S. (1998). Scholarly communication and electronic journals: An impact study. Journal of the American Society for Information Science. 49(6), 507-516. Hofman, R., & Van Oostendorp, H. (1999). Cognitive effects of a structural overview in a hypertext. British Journal of Educational Technology, 30, 129-140. Jenkins, C., Corritore, C.L., & Wiedenbeck, S. (2003). Patterns of information seeking on the web: A qualitative study of domain expertise and Web expertise. Information Technology and Society, 1(3), 64-89. Keyhani, A. (1993). The online journal of current clinical trials: An innovation in electronic journal publishing. Database, 16, 14-23. Landow, G. P. (1997). Hypertext 2.0: The Convergence of Contemporary Critical Theory and Technology. Baltimore: Johns Hopkins University Press. 151 Lapid, M. & Kung, S. (2006). Getting too many e-alerts? Customizing searches, subscribing wisely can prevent email clutter. The Journal of Family Practice, 5 (7). Landis, J. & Koch, G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174. Last, D., O’Donnell, A., & Kelly, A. (2001). The effects of prior knowledge and goal strength on the use of hypertext. Journal of Educational Multimedia and Hypermedia, 10(1), 3-25. Lawless, K.A., & Brown, S.W. (1998). Knowledge, interest, and recall of a hypermedia environment. Paper presented at the annual meeting of the Society for the Scientific Study of Reading, San Diego, CA. Lee, M. J. (2005). Expanding hypertext: Does it address disorientation? Depends on individuals’ adventurousness. Journal of Computer-Mediated Communication, 10(3), article 6. Lee, M, & Baylor, A. L (2006). Designing metacognitive maps for web-based learning. Educational Technology and Society, 9 (1), 344-348. Liu, Z. (2005). Reading behavior in the digital environment: Changes in reading behavior over the past ten years. Journal of Documentation, 61(6), 700 -713. Luzon, M. (2007). The added value of features of online scholarly journals. Journal of Technical Writing and Communication, 37(1), 59-73. 152 Macedo-Rouet, M., Rouet, J., Epstein, I., & Fayard, P. (2003). Effects of online reading on popular science comprehension. Science Communication, 25, 99 - 128. Madrid, I. R., Van Oostendorp, Herre, & Melguizo, M. C. (2009). The effects of the number of links and navigation support on cognitive load and learning with hypertext: The mediating role of reading order. Computers in Human Behavior, 25, 66–75. Maki, P. L. (2004). Assessing for learning: Building a sustainable commitment across the institution. Sterling, Virginia: Stylus, 2004. Maes, A., van Geel, A., & Cozijn, R. (2006). Signposts on the digital highway: The effect of semantic and pragmatic hyperlink previews. Interacting with Computers, 18, 265- 282. McDonald, S. & Stevenson, R. J. (1996). Disorientation in hypertext: The effects of three text structures on navigation performance, Applied Ergonomics 27, 61–68. McDonald, S., & Stevenson, R.J. (1998a). Effects of text structure and prior knowledge of the learner on navigation in hypertext. Human Factors, 40(1), 18-27. McDonald, S., & Stevenson, R. J. (1998b). Navigation in hyperspace: An evaluation of the effects of navigation tools and subject matter expertise on browsing and information retrieval in hypertext. Interacting with Computers, 10, 129-142. McDonald, S. & Stevenson, R. J. (1999). Spatial versus conceptual maps as learning tools in hypertext. Journal of Educational Multimedia and Hypermedia, 8(1), 43-64. 153 McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30-46. Mishra, P. Spiro, R. J. & Feltovich, P. J. (1996). Technology, representation and cognition: The prefiguring of knowledge in Cognitive Flexibility Hypertexts. In H. van Oostendorp (Ed.), Cognitive aspects of electronic text processing. (pp.287-306). Norwood, NJ: Ablex Publishing. Mitchell, T., Chen, S., & Macredie, R., (2005). Hypermedia learning and prior knowledge: Domain expertise vs. system expertise. Journal of Computer Assisted Learning, 21, 53-64. Moskal, B. M., & Leydens, J. A. (2000). Scoring rubric development: Validity and reliability. Practical Assessment, Research and Evaluation, 7(10). Mogge, D. (1999). Seven years of tracking electronic publishing: The ARL. Library Hi Tech, 17(1), 17-25. Mueller-Kalthoff, T., & Moeller, J. (2003). The effects of graphical overviews, prior knowledge, and self-concept on hypertext disorientation and learning achievement. Journal of Educational Multimedia and Hypermedia, 12, 117-134. Mueller-Kalthoff, T., & Moeller, J. (2006). Browsing while reading: Effects of instructional design and learners’ prior knowledge. ALT-J, Research in Learning Technology, 14(2), 183-198. 154 Nonaka, I. (1994). A dynamic theory of organizational knowledge creation. Organization Science, 5(1), 14-36. Niederhauser, D. S., Reynolds, R. E., Salmen, D. J., & Skolmoski, P. (2000). The influence of cognitive load on learning from hypertext. Journal of Educational Computing Research, 23(3), 237–255. Nicholas, D., Huntington, P., & Watkinson, A. (2004). Scholarly journal usage: The results of deep log analysis. Journal of Documentation, 61(2), 248 – 280. Oppenheim, C., Greenhalgh, C., & Rowland, F. (2000). The future of scholarly journal publishing. Journal of Documentation, 56 (4), 361-398. Peek, R. P., & Newby, G. B. (1996). Scholarly publishing: The electronic frontier. MIT Press. Potelle, H., & Rouet, J.-F. (2003). Effects of content representation and readers’ prior knowledge on the comprehension of hypertext. International Journal of Human- Computer Studies, 58, 327-345. Rouet, J.-F., Vidal-Abarca, E., Bert-Erboul, A. & Millogo, V. (2001). Effects of information search tasks on the comprehension of instructional text. Discourse Processes, 31(2), 163–186. Rowland, F. (1997). Print journals: Fit for the future?, Ariadne, 7. Sack, J. (2005). HighWire press: Ten years of publisher-driven innovation. Learned Publishing, 18 (2), 131–42. 155 Schwartz, N., Andersen, C., Hong, N., Howard, B., & McGee, S. (2004). The influence of metacognitive skills on learners’ memory of information in a hypermedia environment. Journal of Educational Computing Research, 31(1), 77–93. Schwen, T.M., Kalman, H.K., Hara, N., & Kisling, E.L. (1998). Potential knowledge management contributions to human performance technology research and practice. Educational Technology Research and Development, 46(4), 73-89. Shapiro, A. M. (1999). The relevance of hierarchies to learning biology from hypertext. Journal of the Learning Sciences, 8(2), 215–243. Shapiro, A. M. (2000). The effect of interactive overviews on the development of conceptual structure in novices learning from hypermedia. Journal of Educational Multimedia and Hypermedia, 9(1), 57-78. Shapiro, A. M., & Niederhauser, D. (2004). Learning from hypertext: Research issues and findings. In D. H. Jonassen (Ed.), Handbook of research on educational communications and technology (2nd ed, pp. 605–620). Mahwah, NJ: Erlbaum. Siemens, R., Willinsky, J., Blake, A. (2006). A study of professional reading tools for computing humanists. Electronic Textual Cultures Laboratory, University of Victoria. Tenopir, C. (2004). Online scholarly journals: How many? The Library Journal, 2 http://www.libraryjournal.com/index.asp?layout=articlePrint&articleID=C%A374956 . 156 Tierney, R., & Simon, M. (2004). What's still wrong with rubrics: Focusing on consistency of performance criteria across scale levels. Practical Assessment, Research & Evaluation, 9(2). Tremayne, M., & Dunwoody, S. (2001). Interactivity, information processing, and learning on the World Wide Web. Science Communication, 23, 111-134. Willinsky, J. (2004). As open access is public access, can journals help policymakers read research. Canadian Journal of Communication, 29(3/4), 381-394. Willinsky, J. (2006a). The access principle: The case for open access to research and scholarship. Cambridge, MA: MIT Press. Willinsky, J. (2006b). When the research's over, don't turn out the lights. In K. Tobin & J. Kincheloe (Ed.), Doing educational research: A handbook. Rotterdam, Netherlands: Sense Publishers.439-456. Willinsky, J., & Quint-Rapoport, M., (2007). How complementary and alternative medicine practitioners use PubMed. Journal of Medical Internet Research, 9(2). Yang, S.C. (1997). Information seeking as problem-solving using a qualitative approach to uncover the novice learners’ information – seeking processed in a perseus hypertext system. Library and Information Science Research. 19(1), 71–92. Zumbach, J. (2006). Cognitive overhead in hypertext learning reexamined: Overcoming the myths. Journal of Educational Multimedia and Hypermedia, 15(4), 411-432. 157 APPENDIX A SURVEY I 158 APPENDIX B SURVEY II Name: _____________________ Part A (All Students) Mark your response beside any of the assigned articles, if any, which you printed.  Laptops in the Classroom  Cyberbullying  Digital Divide  Information Literacy Reasons for printing: ________________________________________________________________________ ________________________________________________________________________ Reasons for reading online: ________________________________________________________________________ ________________________________________________________________________ 159 160 N      4. In the following space please provide any additional feedback you wish to convey about the Reading Tools. ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 161 ________________________________________________________________________ _______________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 162 APPENDIX C ASSIGNMENT Assignment – Educational Technology Issues: Critique and Discussion Read and critically evaluate the contribution which the following authors have made to four of the pressing technology issues facing teachers today. Discuss and engage with the author’s contributions to these issues in small-groups. Issues and Articles 1. Laptop Classrooms/Schools 1. Intro: Tuesday, July 10 2. Due: Friday, July 13 2. Cyber Bullying 1. Intro: Friday, July 13 2. Due: Tuesday, July 17 3. Digital Divide 1. Intro: Tuesday, July 17 2. Due: Friday, July 20 4. Information Literacy 1. Intro: Friday, July 20 2. Due: Wednesday, July 25 *Note authors & articles will be provided to students during the “Intro” Procedure Each article will be discussed in small groups during class for twenty minutes over four class sessions. In preparation for this discussion, students will create a list of 10 – 12 “talking points” for each article prior to class. These points are intended to guide a student leading a class discussion of the article or to participate, as in this case, in a small-group discussion on the article. Groups will then have an opportunity to share their discussion with the class. 163 The talking points should summarize the ideas to be presented on (a) What are the author’s key points; (b) why the author should or should not be judged as having a contribution to make to this issue; (c) where and why you are inclined to support or challenge one or more aspects of the author’s position; and (d) how the author’s position relates to other ideas, work, and stances on this issue including, but not restricted to, your own. Each talking point should be approximately three sentences, i) issue identification; ii) take a position; iii) present basis for position. Prior to the small group discussion students will submit an electronic copy of their talking points through WebCT and complete an online reflective survey. Assessment: This assignment is Pass or Non-Pass. All talking points will be read by the instructor and feedback will be given back to the class rather than individual. Students who receive a grade of Non-Pass will need to re-work the assignment to Pass standards within a time set by the instructor 164 APPENDIX D ASSIGNMENT RUBRIC Talking Points Rubric Article: _____________ Quality of Talking Points - Exceptional Above and Beyond 9-10/10 Excellent 8-9/10 Very Good 7-8/10 Good 6-7/10 Poor Below 6/10 Name: DISPLAYS COMPREHENSI ON Shows nuanced comprehensio n of the article’s thesis and primary claims; the writer’s assessment of the article displays substantial depth, fullness and complexity of thought Shows sufficient comprehension of the article’s thesis and primary claims; the writer’s assessment of the article displays some depth and complexity of thought Shows sufficient comprehension of the articles thesis and primary claims; the assessment of the article displays only a little depth and complexity of thought Shows insufficient comprehension of the article’s thesis or little awareness of its complexity; may treat the article simplistically or repetitively Lacks focus or demonstrates confused or simplistic thinking; writer may demonstrate no overall conception of article’s thesis and primary claims /10 CRITIQUE Ability to identify strengths and weaknesses of author’s argument Identified key strengths and weaknesses of author’s argument. Challenged author’s claims where applicable Identified most of the author’s key strengths and weaknesses of author’s argument Identify a few of the author’s key strengths and weaknesses of author’s argument Distracted by minor points and missed some of the author’s key points Focused on either just strengths or just weaknesses. Missed the author’s message and/or misinterpreted author’s argument Supported argument with quality citations or relevant experience/prior knowledge Supported all talking points with relevant quality citation or valid experience 11 or 12 =10 10 = 9 Most of talking points are supported with relevant quality citation or valid experience 8 = 8 9 = 8.5 Some of the talking points are only supported with opinion 7 =7 Most of the points not well supported with relevant citations/ arguments 3 to 5 = 6 6 = 6.5 Confusing; points not well supported or supported with irrelevant citations or experiences Below 3 = 4 Number and Quality of external Sources to support 4 or more - Variety of excellent 3 or 4 - At least three academic 2 or 3 - two academic source, and one 1 – no academic sources, just one non-academic 0 - Not well supported used author’s points 165 argument Academic sources = peer review or edited book Non academic source = government site, article, organization site academic sources 4 =9 5 = 9.5 6 or more 10 sources 3 academic sources =8 3 academic + 1 non academic – 8.5 nonacademic source 2 academic =7 2 academic +1 non-academic =7.5 source 1 academic 6 1 academic and 1 non academic = 6.5 to support his/her own points or no external sources at all. >5 Critiqued whether author is in a position to write on the topic Supported view of whether author is in a position to write on the topic with exceptional arguments Supported with Author’s academic credentials (degree, institution), practical experience or knowledge in the area, published related articles Cover at least two of the following: Author’s academic credentials (degree, institution), practical experience or knowledge in the area, published related articles Cover at least one of the following: Author’s academic credentials (degree, institution), practical experience or knowledge in the area, published related articles Supported view of whether author is in a position to write on the topic with poor arguments i.e. from Yale and therefore an authority Critique TOTAL /40 Total /50 166 APPENDIX E STUDENT CONSENT FORM Department of Language & Literacy Education 2125 Main Mall Vancouver, BC, Canada V6T 1Z4 Tel: (604) 822-5788 Tel: (604) 822-3154 July 9, 2007 GENERAL INFORMATION AND INFORMED CONSENT Reading Tools: The Enhancement of an Online Research Environment? Principal Investigator: Dr. John Willinsky, Professor, Language and Literacy Education Co-investigator: Peter Arthur, Language and Literacy Education You have been invited to participate in this study because you comprise a segment of the population central to using online literature in the pursuit of learning and in your future career. This research is being conducted by Peter Arthur as part of his doctoral program at UBC. Purpose and Process: The overall purpose of this study is to assess the degree to which an online “Reading Tool” will assist students with critiquing an article as part of an assignment. The Reading Tool resides next to an online peer reviewed article and provides a number of contextual resources. The assignment asks students to read and critically evaluate the contribution of the authors of four different peer reviewed journal articles that focus on pressing technology issues facing teachers today. Each article will be discussed in small groups during class for twenty minutes over four class sessions. In preparation for this discussion students will create a list of 10 – 12 “talking 167 points” for each article prior to class. A copy of these points will be handed in and accessed for purposes of this study in terms of comprehension and ability to evaluate the article. In addition students will complete an online survey prior to each small group discussion. A second survey will be completed by students who used the Reading Tools, prior to their last small group discussion. The research is intended to determine whether the reading tool assists students in critiquing a research article with respect to comprehension, ability to evaluate the quality of the article and whether the tools increase the level of confidence in using the article, critically or supportively, as part of their future work as teachers in a school or work as university students. Participation and confidentiality: Access to this data will be restricted to Arthur, Willinsky, and two research assistants, and one statistician. Selected data will be made public in the form of publications, web sites, public forums such as conferences and focus groups, but with this data, the anonymity of all participants will be maintained by using pseudonyms, unless participants indicate on the survey that they wish to be identified with the views expressed. All data will be destroyed at the completion of the research project. Participants may refuse to participate or withdraw at any time from any portion of these research activities, without prejudice to one's standing in this course, even after signing this consent form. At any stage in the involvement in these activities, participants may request clarification on any issue regarding the research. Finally, this research does not involve any risk to the participants in this research project. Should you have any more concerns or questions about your rights or your treatment in this research project you may contact the Research Subject Information Line in the UBC Office of Research Services +1 604 822 8598. Further clarification on any issues regarding this particular project may be obtained from Professor John Willinsky (john.willinsky@ubc.ca or +1-604-822-3950). 168 Consent I have read the above and give my consent to participate in this project and acknowledge receipt of a copy of this document. _____________________________________________________ Signature 169 APPENDIX F RESEARCH ETHICS APPROVAL 170 "@en ; edm:hasType "Thesis/Dissertation"@en ; vivo:dateIssued "2009-05"@en ; edm:isShownAt "10.14288/1.0067043"@en ; dcterms:language "eng"@en ; ns0:degreeDiscipline "Language and Literacy Education"@en ; edm:provider "Vancouver : University of British Columbia Library"@en ; dcterms:publisher "University of British Columbia"@en ; dcterms:rights "Attribution-NonCommercial-NoDerivatives 4.0 International"@en ; ns0:rightsURI "http://creativecommons.org/licenses/by-nc-nd/4.0/"@en ; ns0:scholarLevel "Graduate"@en ; dcterms:title "Reading tools : the enhancement of an online scholarly research environment"@en ; dcterms:type "Text"@en ; ns0:identifierURI "http://hdl.handle.net/2429/5733"@en .