Arts, Faculty of
Information, School of
DSpace
UBCV
Novin, Alamir
Doctor of Philosophy - PhD
University of British Columbia
According to Goffman’s (1974) Framing Theory, information is always presented in a frame (i.e., parameters that both limit and contextualize information). For example, a photograph is often framed both to communicate essential information (e.g., by focusing on a subject) and to remove information deemed unnecessary (e.g., by cropping a scene). However, when people look at the information, they might not always have the frame's parameters in mind (e.g., what information did a photographer crop out of the photograph?). In addition, many topics are open problems, meaning they are subject to multiple frames (e.g., how to solve climate change). How do people reconcile these multiple frames in an open problem when they seek online information? To explore this question, I conducted an experiment to understand how frames affect the search process of students working to solve open-ended information problems. Participants were asked to research on Google the use of cellphones on campus, and divided into two conditions: one condition was provided with a task framed around health, while the other condition had a task with no frame. The study found a significant difference between conditions both in the behaviors students exhibit with selecting information and their activities when using information. I used a think-aloud to observe abductive reasoning (i.e., the reasoning people use to find information outside of a frame). An analysis of the think-aloud data identified eight themes whereby people reference frames, which can assist abductive reasoning. My findings have implications for both information systems design and the end-users of those systems (e.g., from information design to teachers creating information curricula). As our search systems become more automated and more reliant on advanced AI, the role of frames will become essential to our mediation and evaluation of information.
HOW TASK FRAMES AFFECT THE PROCESS AND PRODUCTS OF INQUIRY by ALAMIR NOVIN B.A., The University of British Columbia, 2009 MA., Concordia University, 2012 A DISSERTATION SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY in THE FACULTY OF GRADUATE AND POSTDOCTORAL STUDIES (Library, Archival and Information Studies) THE UNIVERSITY OF BRITISH COLUMBIA (Vancouver) September 2023 © Alamir Novin, 2023 ii The following individuals certify that they have read, and recommend to the Faculty of Graduate and Postdoctoral Studies for acceptance, the dissertation entitled: How task frames affect the process and products of inquiry submitted by Alamir Novin in partial fulfilment of the requirements for the degree of Doctor of Philosophy in Library, Archival and Information Studies Examining Committee: Eric Meyers, Associate Professor, Library, Archival and Information Studies, UBC Supervisor Heather O’Brien, Associate Professor, Library, Archival and Information Studies, UBC Supervisory Committee Member Richard Arias-Hernández, Associate Professor, Library, Archival and Information Studies, UBC Supervisory Committee Member Luanne Sinnamon, Associate Professor, Library, Archival and Information Studies, UBC University Examiner Alfred John Hermida, Professor, School of Journalism, UBC University Examiner iii Abstract According to Goffman’s (1974) Framing Theory, information is always presented in a frame (i.e., parameters that both limit and contextualize information). For example, a photograph is often framed both to communicate essential information (e.g., by focusing on a subject) and to remove information deemed unnecessary (e.g., by cropping a scene). However, when people look at the information, they might not always have the frame's parameters in mind (e.g., what information did a photographer crop out of the photograph?). In addition, many topics are open problems, meaning they are subject to multiple frames (e.g., how to solve climate change). How do people reconcile these multiple frames in an open problem when they seek online information? To explore this question, I conducted an experiment to understand how frames affect the search process of students working to solve open-ended information problems. Participants were asked to research on Google the use of cellphones on campus, and divided into two conditions: one condition was provided with a task framed around health, while the other condition had a task with no frame. The study found a significant difference between conditions both in the behaviors students exhibit with selecting information and their activities when using information. I used a think-aloud to observe abductive reasoning (i.e., the reasoning people use to find information outside of a frame). An analysis of the think-aloud data identified eight themes whereby people reference frames, which can assist abductive reasoning. My findings have implications for both information systems design and the end-users of those systems (e.g., from information design to teachers creating information curricula). As our search systems become more automated and more reliant on advanced AI, the role of frames will become essential to our mediation and evaluation of information. iv Lay Summary In a search task, studies show that the search page can influence people’s understanding of a topic. That is because many search pages focus people’s attention on the top results. This is problematic if the bottom result provides context that challenges the top results. Can people recognize when the bottom results’ context is important? How? Perhaps if the search task itself is contextualized can it influence people to focus on the bottom results? In this study, I gave a similar search task question to two groups of people. However, the two groups had different contextual wordings. The participants did not know that regardless of what they queried, I instructed the computer to produce the same search page for both groups. Designing the experiment this way allowed me to observe whether the wording of the task could influence people more than the search page. v Preface This dissertation is an original, unpublished, independent work by the author, Alamir Novin. The fieldwork reported in Chapters 2- 4 was covered by UBC Ethics Certificate titled USER-STUDY-3 and number H21-00554-A001. vi Table of Contents Abstract .......................................................................................................................................... iii Lay Summary ................................................................................................................................. iv Preface............................................................................................................................................. v Table of Contents ........................................................................................................................... vi List of Tables ................................................................................................................................. ix List of Figures ................................................................................................................................. x Glossary ........................................................................................................................................ xii Acknowledgements ...................................................................................................................... xix Dedication ..................................................................................................................................... xx 1 Introduction ............................................................................................................................. 1 1.1 Frames and the Search Process ....................................................................................... 3 1.2 Problem Space and Research Question ........................................................................... 8 1.3 Contribution to Prior Research on Information Tasks .................................................. 14 1.4 The Structure of this Dissertation ................................................................................. 15 1.5 Summary of the Chapter ............................................................................................... 16 2 Literature Review.................................................................................................................. 17 2.1 Frames and Sense-Making ............................................................................................ 17 2.1.1 Frame Theory ............................................................................................................ 18 2.1.2 How Information Scholars Approach Sense-Making ............................................... 20 2.2 Search As Learning ....................................................................................................... 25 2.2.1 Information Selection, Ideation, and Query-Construction in SAL ........................... 28 2.2.2 Information Use –Studying the SERP Interactions .................................................. 29 2.2.3 Reconciling Frames – Studying Cognition During the ISP ...................................... 32 vii 3 Methods................................................................................................................................. 37 3.1 Study Design ................................................................................................................. 37 3.1.1 Experimental Conditions .......................................................................................... 39 3.1.2 Task Design .............................................................................................................. 41 3.1.3 SERP Layout ............................................................................................................. 44 3.1.4 Pilot ........................................................................................................................... 49 3.2 Adjustments for a global pandemic .............................................................................. 51 3.3 Participant Task ............................................................................................................ 52 3.3.1 Recruitment ............................................................................................................... 54 3.3.2 Data Collection ......................................................................................................... 57 3.4 Data Analysis ................................................................................................................ 59 3.5 Ensuring Research Quality ........................................................................................... 62 3.6 Chapter Summary ......................................................................................................... 64 4 Findings................................................................................................................................. 66 4.1 Information Selection & Query-Constructions ............................................................. 68 4.1.1 Observations on Query-Constructions: ..................................................................... 69 4.1.2 Number of Keywords Used in Query: ...................................................................... 70 4.1.3 Observations on Clicking Behavior: ......................................................................... 73 4.1.4 Analysis and Limitations of Click-Through Data ..................................................... 74 4.1.5 Observations on Time-On-Site: ................................................................................ 77 4.1.6 Analysis and Limitations of Time-On-Site Data ...................................................... 78 4.2 Information Use & SERP Interactions .......................................................................... 81 4.2.1 Observations on Sites Used ...................................................................................... 81 4.2.2 Analysis and Limitations of Site-Use Data ............................................................... 85 4.2.3 Task Responses ......................................................................................................... 85 4.2.4 Rubric and Health-Frame Analysis:.......................................................................... 87 4.3 Think-Aloud Analysis ................................................................................................... 95 4.3.1 Triangulating the Concurrent and Retrospective Think-Alouds with Observed Search Actions. ..................................................................................................................... 96 4.3.1.1 Inquiring: Making an inquiry about a frame ..................................................... 97 4.3.1.2 Conceptualizing: Conceptualizing a frame ....................................................... 99 4.3.1.3 Orientating: Orientation to Acquire more Perspectives about a Frame .......... 101 4.3.1.4 Guiding: Comparing and Contrasting Different Frame Perspectives ............. 102 viii 4.3.1.5 Numerositizing: Validating Frames Using a Sense of Logic or Probability. . 105 4.3.1.6 Anthropologizing: Aligning Frames with Personal or Cultural Norms .......... 107 4.3.1.7 Transferring: Validating Frames Based on Expertise ..................................... 109 4.3.1.8 Expressing: Sharing Frames with Others ........................................................ 112 4.4 Summary of Findings .................................................................................................. 117 5 Discussion ........................................................................................................................... 119 5.1.1 Task Framing and Information Selection ............................................................... 120 5.2 Inferences on how the task frame affects the selection process .................................. 122 5.2.1 Task Framing and Information Use ........................................................................ 123 5.3 Inferences on how the task frame affects the search use ............................................ 126 5.3.1 Framing Interactions and the Challenge of Task Frames ....................................... 126 5.4 Updating the Information Seeking Process................................................................. 130 5.5 Updates to Methods of Studying the ISP .................................................................... 136 5.5.1 Chapter Summary ................................................................................................... 137 6 Conclusion .......................................................................................................................... 139 6.1 Contribution to Prior Theories on Systems and Sense-Making .................................. 139 6.2 Contribution to Pedagogy and Information Literacy .................................................. 141 6.3 Contribution to Researching the ISP Using ICOGNATE........................................... 141 6.4 Future studies .............................................................................................................. 142 6.5 Limitations .................................................................................................................. 143 6.6 Conclusion .................................................................................................................. 145 Works Cited ................................................................................................................................ 147 Appendix 1 – Consent Form ....................................................................................................... 164 Appendix 2 – Task Script and Interview Questions ................................................................... 167 ix List of Tables Table 1 The associated concepts with each of the research questions .......................................... 12 Table 2 Relationship between the measures and the instruments in the dissertation ................... 58 Table 3 Rubric for detecting the level of influence of health results in writing output ................ 60 Table 4 The associated concepts with each of the research questions .......................................... 66 Table 5 Demographics of Participants .......................................................................................... 67 Table 6 Comparison of Health Terms Mentioned in Queries ....................................................... 69 Table 7 Comparison of participants who conducted breadth-first or depth-first searches within the first 10 seconds of encountering a SERP ................................................................................ 80 Table 8 Number of participants who satisfied each of the rubric’s criteria. ................................. 91 Table 9 Number of participants awarded a final rubric score ....................................................... 92 Table 10 Coding table for ICOGNATE ...................................................................................... 116 x List of Figures Figure 1 Diagram Showing the Original Photograph of the Five Climate Activists Before Vanessa Nakate was Cropped Out by AP. Nakate's Tweet is also featured on the right................ 4 Figure 2 Diagram of the study design’s relationship to RQ1, 2, and 3. 1) Participants were assigned randomly to whether they would receive a task with the health frame. 2) A video recorded participants' interactions with the search process and the product output. 3) To answer RQ1, Queries, Click-throughs, and Time-on-Site was collected. 4) For RQ2, Sites-used, Terms-Used, and a Rubric was collected. 5) Throughout the process participants thought aloud. Ericsson and Simon’s (1993) Think-Aloud Protocol was used to collect both concurrent and retrospective (post-task think-aloud). 6) Both types of think-alouds were triangulated with the search actions to answer RQ3. .................................................................................................................................. 42 Figure 3 Order of results on the SERP page retrieved by students. Note that Google’s interface may change by the time of publication. ........................................................................................ 46 Figure 4 Screenshot of the webpage participants used to enter main points and submit final answers. Note that the textbox can be expanded to the participants’ desired size. ....................... 54 Figure 5 Comparison of total clicks (i.e., the aggregate of sites clicked by each group). The moving average is represented by the dashed line. ....................................................................... 73 Figure 6 Participants would click, highlight, copy, and paste information from SERP snippets without clicking through to the website. ....................................................................................... 76 Figure 7 Box chart of average time spent on site with minimum and maximum ranges and the average .......................................................................................................................................... 78 Figure 8 Aggregate of sites used by each group for the writing output and the moving average 83 Figure 9 Participants for each group condition are clustered by the depth of their site-use. ........ 84 xi Figure 10 Graph of the number of participants who satisfied each of the rubric’s criteria. ......... 91 Figure 11 Final rubric score for each participant .......................................................................... 92 Figure 12 Comparison of word frequencies for HF and NHFP tasks ........................................... 94 Figure 13 Three Visual Metaphors Distinguishing a Frame, Multiple Frames, and Indefinite Frames ......................................................................................................................................... 132 Figure 14 The relationship between an intended and distributed metaframe as a person frames an open problem .............................................................................................................................. 134 Figure 15 Model of how participants moved between intending to perform information-seeking stages (via the think-aloud) and distributing those tasks (via SERP interactions, notes, and task output). ........................................................................................................................................ 135 xii Glossary Abductive Reasoning: The pragmatic approach of recognizing multiple perspectives on prior data that requires a hypothesis to acquire novel data (Peirce, 1906). Seeking new information and creating hypotheses are qualities of abductive reasoning (Peirce, 1906). However, the process by which this occurs is not well understood. Answer Box: An extended snippet (see: Snippet) of a search result that appears at the top of a search page (Miklošík & Daňo, 2016). Bias: When random noise around information is erroneously characterized as a pattern (Kahneman et al., 2016). In an information-seeking process, people depend on three factors, heuristics, algorithms, or sources, that can lead to Kahneman’s definition of bias if they are overly depended on (Novin & Meyers, 2017). For example, a searcher depending on Google’s top result can cause associated noise (e.g., Google’s algorithm accidentally provides a weak answer) to be erroneously characterized as a pattern (e.g., the result is perceived to be the best answer simply because it is at the top). Breadth-First Search: A search behavior by a user that involves skimming all the SERP results before clicking (Klöckner et al., 2004). The term is useful for describing a system’s algorithm, but for humans the terms “breadth” and “first” leave ambiguity. I define “breadth-first” as the action that takes place within the first 10 seconds a participant encounters a SERP. xiii Click-Throughs: When a user clicks and visits a hyperlink on a search page (Ong et al., 2017). Depth-First Search: A search behavior by a user that involves iteratively clicking and reading individual links (Klöckner et al., 2004). I define “depth-first” as which action takes place within the first 10 seconds a participant encounters a SERP. Ecological Model: A model of all the main stakeholders within a defined system including the dynamics of their interactions (Fidel, 2012). Frame: The demarcations used to communicate meaningful information from the ubiquity of environmental data (Goffman, 1974). Frame Analysis: An analysis of frames as “definitions of the situation [that] are built up in accordance with the principles of organization which govern events—at least social ones—and our subjective involvement in them.” (Goffman, 1974, pp. 10-11). Frame Theory: A theory that reconceptualizes Aristotle’s concept of “frame of thought” as a collective psychological phenomenon whereby humans can frame two paradoxical thoughts (unlike mathematical sets that exclude paradox) (Bateson, 1973, p.177-193). I use Bateson’s (1973) definition of frame theory to explain how humans can also frame less paradoxical but open problems with indefinite frames. xiv Heuristic: A cognitive shortcut (e.g., or “rule of thumb) that people adopt to reduce the cognitive effort required in a task (Kahneman, 2013). For example, a person who trusts Google can rely on the Answer Box without double-checking with other search results. Human Information Interaction (HII): The ecology of interactions that take place with information behaviors and human-computer interactions (i.e., both the inner cognitive and external social interactions) (Fidel, 2012). Information-Searching: A subset of Information-Seeking. It involves a person actively and directly retrieving information via a channel, such as a search engine, or sources, such as an article (Byström & Hansen, 2005). Information-Seeking: A superset of search tasks, whereby an information-seeking task may contain multiple instances of information seeking. Seeking involves a user orientating their behavior to address a general search need (Byström & Hansen, 2005). Information-Seeking Process (ISP): A subset of information behaviors that may include searching. Carol Kuhlthau (1993) developed the most well-known ISP. I use the term ISP as a more general reference to the process of information seeking. Inquiry: A key component of most Information Search Processes in the Information Sciences. The concept dates back to at least Socrates. The definition of inquiry goes beyond the act of xv asking questions and involves the pragmatic “process of acquiring and changing beliefs about the world” (Stalnaker, 1984). Metacognition: Thinking recursively about thinking or more accurately “cognition about cognitive phenomena” (Flavell, 1979). The concept of metacognition emerges in Search-As-Learning (See: Search-As-Learning) (Smith & Rieh, 2019). People do not just search for information, they often use metacognition to think about how, why, and the way they are searching (Smith & Rieh, 2019). Metacognition enables people to monitor, regulate, and reflect on what they are learning to make more effective search actions. Moving Average: A measurement to compare participants’ responses to two conditions. In statistics, “a moving average is a time series constructed by taking averages of several sequential values of another time series” (Hyndman, 2011). Open problem: A problem or puzzle that is approachable but unsolvable (Pehkonen, 1997). In terms of framing, I use this definition to explain how an open problem may be approached using multiple frames but the exact number of frames required is indefinite. Priming: A general term for a mechanism that initiates a response that can lead to bias (Kahneman, 2013). I distinguish between priming from framing: A key difference between a frame and priming is that a frame is a parameter including and excluding information while a prime is merely an initial catalyst. A frame can be used to communicate or filter context and meaning, whereas a prime does not need to filter context, just initiate a stimulant. xvi Quasi-Live Corpus: A live corpus that controls for certain variables. This dissertation controlled for the document-order of the results on the SERP but the SERP results themselves were live. Outside of the SERP’s order of results, the query box interactions, the SERP features (i.e., clicking, highlighting, and scrolling), and the websites were not controlled for. Search-As-Learning (SAL): “A user-driven, technology supported process of human information interaction (HII). In this process, people engage intellectually, physically and emotionally with digital information” (Freund et al., 2014, p.1). The field of SAL studies a form of self-regulated learning that requires greater information literacy, such as developing critical thinking skills when exploring a SERP. Search Engine Results Page (SERP): A page of results that a user sees after submitting a query to a search engine (Höchstötter & Lewandowski, 2009). Self-Regulated Learning: The process by which people actively control their learning process while information seeking (Urgo & Arguello, 2022). Snippet: The blurb which appears below the title of a search result on a search page (Bink et al., 2022). Task: Any process that has an input and output (Steiner, 1972, p. 89). xvii Task Description: A task’s definitions of a requirement and goal (Wildemuth & Freund, 2012). Task Frame: The parameters surrounding an objective’s input, process, or output (Neale et al., 1987). These parameters include risks that indirectly lead people to bias (Neale et al., 1987). For example, a teacher might intend to frame the task of students cleaning chalkboards as “disciplinary lessons” while students frame it as a “needless chore”. Task Process: The behaviors that manifest when a task is performed (Wildemuth & Freund, 2012). Think-Aloud: An experimental protocol that requires participants to make utterances about what they are thinking as they complete a task (Ericsson & Simon, 1993). Think-alouds can qualify as what Ericsson and Simon (1993) refer to as a “concurrent think-aloud” (i.e., utterances made while a task is being completed) or as a “retrospective think-aloud” (i.e., Asking participants to retrospectively walk through the decisions they made to complete a task). Time-on-Task: The number of seconds participants spend on the entire task (Karweit, 1984). Time-on-Site: The number of seconds a person has a website open (Van Voorhees et al., 2009). Total Clicks: Aggregate of sites clicked by each group (Skiera et al., 2010). xviii Utterance: An extended pause in their think-aloud or a gap in a person’s speech (MacGregor et al., 2010; Traum & Heeman, 1997). Utterances can be collected in accordance with Anders Ericsson and Herbert Simon’s (1993) Think-Aloud Protocol. Work Task: A task designated under an institutional context (e.g., as opposed to a pleasurable task). A work task is a common ground for two levels of information-related subtasks: search tasks (i.e., directly retrieving information via a channel, such as a search engine) and seeking tasks (i.e., a superset of search tasks that involves an orientation towards a general search need) (Byström & Hansen, 2005). xix Acknowledgements I am grateful for the continual and strong support of my supervisor, Dr. Eric Meyers, and my committee, Dr. Heather O’Brien and Richard Arias-Hernández during the writing of this dissertation. Their many insights guided me towards the completion of this dissertation. I also thank Dr. Luanne Sinnamon, Dr. Alfred Hermida, Dr. Catherine L Smith, and Professor Rhea Tregebov for their feedback during a lively dissertation defense. I extend my gratitude to the faculty at UBC’s School of Information and my past and present colleagues in the graduate program. There are too many to mention but this dissertation was truly a group effort. xx Dedication My deepest gratitude is for the limitless support provided to me by my family; both of my parents and both of my brothers. My amazing life-partner, Nastia Armilde Campos-Rivera motivated and inspired me throughout the doctoral process. Most of all, I dedicate this dissertation to my older brother, Sal. He had faith in me from the beginning. He supported me for both my Masters and my Doctorate. He will continue to inspire me because he will never leave my heart. He passed on so many lessons throughout our time together that I am still learning from his ways here on Earth. 1 1 Introduction Search engines are a primary means of retrieving information via the Internet for users around the world. Search engine result pages (SERPs) continue to evolve in how they display links, descriptions, and sponsored content in diverse ways (Oliveira & Lopes, 2023). Despite Americans using search engines frequently, most (70%) see the spread of false information online as a major threat to well-being – more so than infectious diseases or climate change (Silver, 2020). Furthermore, the nature of navigating misinformation via search tools continues to evolve as people use more diverse online tools (e.g., Siri/Alexa, TikTok, and Chat-GPT) to find information and develop solutions to everyday problems. For example, while Google crawls billions of websites and presents the top ten results on its first page, Siri/Alexa does the same crawl but returns a single vocal answer (potentially over-simplifying more complex or nuanced topics) (Desai & Twidale, 2022)). For some young adults, TikTok is an emerging strategy for finding information, such as health topics, that bypasses traditional search practices (Song et al., 2022). Search will continue to evolve as artificial intelligence tools, like OpenAI’s ChatGPT, integrate with search engines. While the technological evolution of search tools makes the search process more convenient, the algorithms used for filtering information also become opaquer. These developments have implications for the processes and outcomes of information seeking, whether for academic work or everyday problem-solving with web-based information. This dissertation is rooted in the assumption that information is always presented within a frame (i.e., the parameters that both limit and contextualize information). Frame theory was put 2 forward by the Canadian philosopher Erving Goffman (1974) and the theory developed in the fields of sociology (Tannen, 1993), artificial intelligence (J. Fodor, 2006), and cognitive science, notably the experiments carried out by Daniel Kahneman and Amos Tversky aggregated in the book “Thinking, Fast and Slow” (2013). Frames emerge in both the technical parameters and personal decisions that give information context. For example, people need to account for various embedded frames when they retrieve news from a search engine: journalists filter which information sources they need to frame a story, and the story may then be framed again by an editor’s headline and a stock photograph. The headline itself may be framed by a search engine algorithm that places it at a specific area of a page accompanied by related searches and images. Thus, one way of defining frames is as a tool for communicating how meaningful information can be demarcated from the ubiquity of data. However, this demarcation process is subjective and not precise; thus, frames always lead to different kinds of biases. My motivation is to explore whether these multiple frames affect how people search for and make sense of information. Are users aware of biases that emerge from framing actions, and can they question or resist these biases? Exploring this question led to insights on how people sense-make information when searching. Understanding these cognitive mechanisms behind how people sense-make information led to scholarly work that can assist with improving information literacy instructions and designing information systems. I am interested in exploring whether or not frames affect biases when users encounter a problematic situation or open-ended information problem. Open-ended information problems are problems that do not have a specific or “correct” answer, unlike fact-finding (Surif et al., 2014; Oliver & Hannafin, 2000). As such, I sought to observe a problematic situation that occurs when 3 people find information about an open-ended topic but are unaware of the frames influencing them. A person may trust a search engine’s algorithm without considering if the system's designers are framing for or against a topic. Part of the complexity of addressing this problem is that recognizing bias is more complicated than distinguishing outright false statements. That is because biases do not always emerge from the validity of the information itself, but from the inclusion and exclusion of contextual information in a web of relationships. For example, the claim that wood-knocking causes lottery wins is demonstrably false. However, suppose a person conducting a search on lottery ticket winners finds that lottery winners also knock on wood. In that case, they may build a web of relationships between lottery winners and wood-knocking. If the person makes the statement, “I found that lottery winners always knock on wood,” it is biased. The statement has a bias because the search strategy has a bias. The search strategy excluded both searching for lottery winners who do not knock on wood and lottery losers who do knock on wood. Thus, the user retrieved accurate information but searched for an incomplete frame. There are many names for similar types of biases (e.g., “cherry picking data” (Howard, 2018), “stacking the deck” (Giles, 2006), or “selection bias” (Kahneman, 2013)). However, the root mechanisms for the biases often emerge because a frame excluded information. 1.1 Frames and the Search Process A typical example of framing occurs when photojournalists frame a photograph: The photographer chooses which scenic information to keep inside the frame of the photograph and which to exclude. The problem is that what may seem to be a straightforward process can lead to large consequences. For example, the diagram below features a photograph by the Associate Press in 2020 to report a story about young climate activists protesting at the World Economic 4 Forum (Evelyn, 2020). The AP news editor, Sally Buzbee, claimed they wanted to frame the photo to focus on Greta Thurnberg. However, in doing so a young Ugandan woman, Vanessa Nakate, was cropped out. Later, Nakate said it was her first experience with racism and tweeted the AP “erased a continent” (Vanessa Nakate [@vanessa_vash], 2020). The AP stuck with the reasoning behind the photo’s framing and did not apologize until there was a public uproar. Figure 1 Diagram Showing the Original Photograph of the Five Climate Activists Before Vanessa Nakate was Cropped Out by AP. Nakate's Tweet is also featured on the right. Interestingly, Nakate also noted how even the caption of AP News’ photograph did not mention her presence. Nakate is touching on how frames can also influence frames where a photograph’s frame influences the caption’s frame within an article’s frame. Frames can also be embedded within each other. A photojournalist’s image is embedded within a journalist’s textual framing of a story, which is embedded by the framing of the editor-in-chief, which may be influenced by the newspaper’s investors and stakeholders. The newspaper’s reader must then navigate within these series of embedded frames set up by multiple actors (i.e., from the photojournalist to the journalist, editor-in-chief, and then the investors). Each actor’s frame introduces a new form of bias. The journalist may write an article as though their story is the most important story of the 5 day, while an editor-in-chief may want readers to divide their attention amongst various stories throughout the newspaper to engage the reader to read more sections and encounter more advertisements. A photograph’s frame was used as a metaphor by Gregory Bateson (1973) and Erving Goffman (1974) to develop theories on how humans create more abstract frames. An example of a more abstract frame is the ‘agenda building’ that occurs in mainstream media when an editor decides what to print. Regardless of whether the frame is physical or abstract, this dissertation argues the demarcation mechanism is subjective and can lead to biases. Erving Goffman (1974) is credited for developing framing theory with his work on frame analysis, which builds on the definition of frames (See: Frames) as a concept for sociological analysis. Goffman (1974) defines frames as “definitions of the situation [that] are built up in accordance with the principles of organization which govern events—at least social ones—and our subjective involvement in them.” (Goffman, 1974, pp. 10-11). Goffman (1974, pp. 139 & pp, 241) restates the sociological theory that humans are performers within the Shakespearean idea that “all the world is a stage” to argue that all information is communicated via frames (i.e., we perform on our own stage but observe other people’s performances too). Goffman also argues there is an interaction between two “primary” frames: The “natural” and the “social.” The individual frames what they perceive as the natural world and communicates within a societal frame. Since Goffman’s original work, scholars have provided more robust and concrete methodologies for frame analysis. Scholars not only expanded on Goffman’s work but also framed framing theory and analysis. For example, Daniel Kahneman (2013) frames framing as a cognitive bias issue whereby the analysis is often experimental and psychological. Deborah Tannen (1993) frames frame analysis as a sociological issue that requires more sociological 6 approaches. Tannen (1993) herself acknowledges in her book that her framing of frames is different than what Goffman initially intended. Both Erving Goffman’s and Deborah Tannen’s credit frame theory to Bateson’s (1973) philosophy on frames. Gregory Bateson built on the philosophical concept of “frame of thought” from Aristotle to Gilbert Ryles) to re-characterize a collective of “frames of thought” as a psychological phenomenon (1973, pp. 177-193). Bateson (1973) distinguishes frames from mathematical sets, by arguing that humans can frame two paradoxical thoughts. For example, Bateson (1973, pp.177-193) argues that while the sentence ‘This statement is false’ is too paradoxical to be categorized as a mathematical set, humans can frame it at the very least as a paradox. These paradoxes do not inhibit communication but enable people within a society to communicate conflicting concepts within an “ecology” of relationships. Bateson (1973, pp. 177-193) uses the example of how people can play, fight, but also play-fight and also love and hate each other. Bateson’s (1973) ecology of frames also serves as a precursor to Tannen’s and Goffman’s assertion that all communication requires a frame. This dissertation uses Bateson’s definition of frame theory to explain how humans can also frame less paradoxical but open problems. Goffman’s theory is also applied to search engines (Andersson, 2017). Andersson (2017) applied Goffman’s theory to students and their experience with Google. Andersson (2017) found that students frame Google as either an authority, a neutral infrastructure, or a fact-finding tool. This latter frame means some “participants view Google in terms of a fact-finding machine through which they can locate a certain type of information. Frames are necessary to communicate 7 information for any medium, including the SERP. Search engines embed the sources’ frames within multiple additional frames for users (Novin & Meyers, 2017b). In the discussion section of this dissertation, I expand on this prior research to argue that when a search involves a person using a SERP to retrieve sources, each of these actors (i.e., person, SERP, source) occurs with three major frames, respectively: A subject’s frame, a medium frame, and an environmental frame. For example, suppose a student is searching for cellphones in a classroom. The student may frame cellphones as a classroom aid, in a medium frame Google frames search results on cellphone topics with ads to sell Android phones, and in an environment frame, academia may perceive cellphones as a distraction. These three major frames can cause the actors to frame information in different ways: First, the search engine retrieves sources with frames that are dependent on factors such as the document’s genre (e.g., a blog, a discussion forum, or a preprint scholarly article). Henceforth, the source’s framing within an environmental frame is referred to as the source frame. Second, the search engine's algorithm requires a frame to decide which documents to present, in what order to display them, and which to exclude. Henceforth, the SERP’s framing within a medium frame is referred to as the SERP’s frame. Third, there is a person’s own cognition that operates within a ‘frame of mind’ on how the answers to a query are perceived. For example, a person’s cognition may have developed heuristics (or “rules of thumb”) that are biased towards reading familiar sources (e.g., Wikipedia) while automatically excluding less familiar or less accessible sources (e.g., academic articles) (Novin & Meyers, 2017b). Henceforth, a person’s cognitive framing within the subject frame is referred to as the cognitive frame. These frames within the search process are continuously evolving. For example, technology and algorithms may improve the relevancy of results, people may expect to find answers faster, and source results evolve to satisfy these expectations. My findings will suggest 8 that the three frames are interdependent and form within a metaframe. Within this metaframe is a fourth frame that Novin & Meyers (2017a, 2017b) did not extensively study: The work task framing a search. Work tasks provide a common ground for two levels of information-related subtasks: search tasks (i.e., directly retrieving information via a channel, such as a search engine) and seeking tasks (i.e., a superset of search tasks that involves an orientation towards a general search need) (Byström & Hansen, 2005). A work task is developed by including contextual and situational descriptions alongside topical descriptions for a query (Byström & Hansen, 2005). Byström and Hansen’s (2005) example of a work task is how Kuhlthau provided students with an information-seeking task as a class assignment. Greater research in the information sciences needs to account for more contextual factors in a work task via established methods such as using a pilot (Borlund & Schneider, 2010). I used a pilot to design an experiment to examine a work task’s frame. I define a task frame as the parameters surrounding an objective’s input, process, or output. For example, a teacher might intend to frame the task of students cleaning chalkboards as “disciplinary lessons” while students frame it as a “needless chore”. How this fourth frame, the task frame, interacts with the other three frames (i.e., the cognitive frame, SERP’s frame, or source frame) during information seeking is an area for potential investigation. 1.2 Problem Space and Research Question The Internet remains the public’s primary source for retrieving scientific information (National Science Board, 2020). A decade ago, the National Science Board (2014) cautioned that even though the public has greater access to scientific information on the Internet, they are also at greater risk of encountering misinformation. This risk has grown to become more concerning (National Science Board, 2020). With controversial science issues, it is plausible that about half of the self-declared ‘scientific information’ on social media sites like YouTube is not just misinformation, but disinformation (i.e., disinformation deliberately misinforms, often with a 9 hidden agenda) (Allgaier, 2019). Even with media trusted to provide quality information, science literacy faces challenges. For example, climate change contrarians receive 49% more coverage than regular climate change scientists in digital mainstream media (Petersen et al., 2019). With so much information to balance, it is difficult to blame users for letting Google’s search algorithms do their filtering and framing. However, search algorithms should not be uncritically trusted as they can influence opinions and decisions with their biases (Belkin et al., 2009; Crawford, 2021; Epstein et al., 2017; Noble, 2018; Novin & Meyers, 2017a). Given these challenges, information scholars are interested in studying the concept of Search As Learning (SAL), such as how people learn via search engines to inform themselves on diverse topics that require a high level of critical-thinking skills (Urgo & Arguello, 2023; Vakkari, 2016). Scholars looking at SAL are interested in how people use self-regulated learning to actively control their learning process while information seeking (Urgo & Arguello, 2022) instead of passively leaving the regulation to a third-party (e.g., a course syllabus requires less self-regulated learning than a person learning via Google searches). One of the challenges SAL research explores is when learners trust lists produced by search algorithms more than their analytical skills (Meyers, 2012). Educating the public on the effects of online misinformation requires understanding and developing information literacy. Information literacy is an evolving term that encompasses three main perspectives: “1) information literacy as the acquisition of ‘information age’ skills, 2) information literacy as the cultivation of habits of mind, and 3) information literacy as engagement in information-rich social practices” (Addison & Meyers, 2013). Ultimately, educating the public requires developing search strategies that learners can utilize (Meyers, 2012) in tandem with developing critical thinking skills (Urgo & Arguello, 10 2023). The cognitive difference between the two is that, while critical thinking requires an analysis of existing information via inductive or deductive reasoning, search strategies require more abductive reasoning. Abductive reasoning focuses on hypothesis creation and the reasoning people use to seek novel information instead of inductive and deductive reasoning, which focuses more on the information that has already been obtained (Peirce, 1905; Dewey, 2013). Bateson (1973) and Goffman (1974) theorize that people communicate by performing within different frames. The frames people present may also affect a search task. For example, Byström and Hansen (2005) write about how a search task about building a bridge would be approached differently by an engineer or a mathematician. In addition, a search page provides a list of different sources communicating different messages, leading a person to encounter multiple different frames. If a person encounters multiple similar framing of a topic on a search page, the frames can compound on each other to also influence how a user understands a topic (Epstein et al., 2017). Conversely, each of the different framings of a topic can carry its own individual influence on how a person perceives a topic (Kahneman, 2013). My prior study with Eric Meyers (2017a) found that a person can be influenced by the information in the subject frame (i.e., cognitive bias), the medium (i.e., a SERP), or source frames (i.e., the documents in a SERP). The combination of these biases can cause different groups of people to summarize topics with different biases (thereby spreading a bias). For example, even though the participants we studied read SERP results about both corn biofuels and algae biofuels, their summaries focused on one type over another depending on the presentation of the order and genre of the documents (Novin & Meyers, 2017). We found that if corn biofuels appeared at the top of a SERP and algae biofuels at the bottom, people would summarize the topic of “biofuels” to focus on corn biofuels 11 and often neglect even mentioning algae. In the discussion section of that study, we suggested that this finding could be troublesome if the SERP topic is not as innocuous as “biofuel” types and more related to social justice issues (e.g., the minority perspectives of less-dominant groups in a political debate that favors a status quo) (Novin & Meyers, 2017). We also suggested that a way to mitigate some of the SERP’s influence would be to create greater awareness of how a SERP frames a topic (Novin & Meyers, 2017). However, this also raises the question of how beholden a user is to a SERP’s frame. Notably, Novin & Meyers (2017) also observed that a minority of participants are not as influenced by the SERP’s frame and represented both sides of “biofuels.” How and why did some participants search deeper down the SERP? Perhaps the context surrounding a search task itself can usurp some of the SERP’s influence? Could a task’s frame influence people enough to look beyond the SERP’s bias and focus on the information near the bottom of the SERP? Answering these questions provided insight into why certain participants search to the bottom of the SERP for more perspectives, the rationale for their decision, and the importance behind such behavior. The overarching question posed by this research inquiry is: how do frames influence the processes and products of information seeking and use? This question is elaborated on further as: RQ1: How does task framing affect the information selection processes of students working on open-ended information problems? RQ2: How does task framing affect the information use of students working on open-ended information problems? RQ3: How do task, medium, and source frames interact in the process of making sense of open-ended information problems? 12 RQ1 relates to the thought process in selection during information seeking (i.e., information query constructions and site selections). RQ2 relates to direct actions involving information use (i.e., how far down a SERP do people use sites and which sites did people use in a task output). Finally, RQ3 builds on RQ1 and RQ2’s findings to explain a searcher’s cognitive processes during their general information search process (ISP) when interacting with multiple frames (i.e., do people recognize frames and how do they reconcile conflicting ones?). Research Question Associated Concepts Analytics Interest RQ1: Information Selection Ideation (i.e., planning for a task) during ISP Query-constructions Frame effects on selection RQ2: Information Use Behaviors/ interactions with SERP Click-through, time-on-site, & writing outputs Frame effects on usage RQ3: Cognition & Synthesis Evaluating frames & cognitive biases Think-aloud during ISP & interviews Reconciling with different frames Table 1 The associated concepts with each of the research questions To answer the research questions, I employed a mixed-method design to analyze and explain the physical and cognitive processes of 40 undergraduate students using a simulated search interface with ten pre-selected documents to write a short position paper on the topic of cellphones on campus. The analysis sought to uncover: 1) the extent to which, if at all, task framing affected the student’s process of inquiry, as captured during task performance and think-aloud protocols; and 2) the extent to which the effects of framing were represented in the short position paper (henceforth referred to as the task output). 13 Prior research provides evidence that biases in a SERP’s presentation (e.g., order effects) affect how searchers conceptualize topics (Epstein & Robertson, 2015). However, the effects are often combatted by a person’s own prior experience, knowledge, and emotions towards a topic – “there is no ‘average user’” (Bar-Ilan et al., 2007). People’s faith in search tools like Google to return relevant and useful results (Meyers, 2011) contributes to the powerful frame (Novin & Meyers, 2017). In the current study, I experimented with the influence of a task’s frame on the search process. I manipulated the task description’s preamble to create two experimental conditions: a group with a specific inquiry frame and a group with a more general inquiry frame. I also manipulated the search engine so that both groups retrieved the same search results regardless of what they queried. I then measured whether the task’s description influenced how students selected and used information on the SERP. I also used a think-aloud to understand how the participants made sense of multiple frames. More information on these experiment’s methods is explained in Chapter 3. While earlier work in Human-Information Interaction emphasized the processes of finding and selecting information (Fidel, 2012), we are now equally concerned with how that information affects decision-making processes and the actions one takes with new knowledge (Meyers, 2018). When information was a challenge to come by, the quality of that information was paramount (Meyers, 2018). In the current age, information is plentiful, but assessing relevance and utility has become a more pressing challenge (Meyers, 2018). My dissertation addresses the effects of frames on information selection and use. It fits within an emerging focus in HII that seeks to understand how search is a liminal space (Fidel, 2012), where human comprehension of 14 a topic evolves as one performs inquiry work. My dissertation has implications for how scholars conceptualize the search process and information use. It further supports the development of interventions to improve the quality of information literacy instruction for secondary and post-secondary students. 1.3 Contribution to Prior Research on Information Tasks Prior research by information scholars has long established that differences in tasks lead to large effects on search behavior and outcomes (Turpin & Scholer, 2006). The work presented here situates itself within this rich history. Jiqun Liu (2021) coded 76 task-based IR papers from 2000 to 2020 on common task predictors. Liu (2021) found that task complexity, type, and structure (e.g., such as how well-defined a task is) are all strong predictors (i.e., often yielding statistically significant results) in objective search tasks (Liu, 2021). However, understanding the nuances behind how tasks affect search is still an open question (Liu, 2021). One approach to understanding the nuances requires that research accounts for multiple task predictors, especially cognitive dimensions such as intentionality, which can assist with furthering understanding (Liu, 2021). I contribute to this line of research over the last two decades on two fronts: 1) The experiment presented here accounted for cognitive factors, such as intentionality, via the think-aloud, and 2) This dissertation made observations on what I believed to be a more nuanced condition than task complexity, type and structure by looking at the mere framing of a task description. That is, the participants in two task conditions were provided similar task questions that were not complex, but familiar and similarly defined. However, one group was provided a sentence preamble that communicated a frame without affecting the complexity or task definition. Participants were informed the task was open without any "correct" answers. Nor was I looking for "correct" answers or even frames. Instead, I wanted to observe whether and how people with almost identical tasks respond to a communicated frame. For example, one potential 15 outcome of this experiment could have been that participants in both conditions in this experiment dismiss the frames I provide because people are inundated with multiple frames and do not react to every frame. The choice of observing competing frames aligns with Liu’s (2021) observation that future research should make observations on user intentions and the nuances of task objectives. Due to competition between multiple frames, it could very well be that a frame is so nuanced that they are not influential enough to be observable in a user’s decisions. Conversely, whether framing a task objective decreases or increases task complexity requires more research and experimentation. As my findings will show, people must balance multiple indefinite frames within a frame, which I will later refer to as a metaframe. I will also argue that there are eight themes involved in the process of balancing these frames within a metaframe, which I refer to as the ICOGNATE model that is presented in Chapter 5: Discussion. 1.4 The Structure of this Dissertation Six chapters comprise this dissertation. Chapter 2 further develops the notion of framing introduced at the beginning of the dissertation. The concept of Search As Learning (SAL) and its connection to this study are explained in greater depth in Chapter 2. Chapter 3 elaborates on the mixed method design and data collection procedures. Chapter 4 presented the findings, consisting of quantitative and qualitative analyses of the experiment. The findings are linked back to the theoretical framework in Chapter 5, which returns to the research questions and discussed their importance. Finally, Chapter 6 outlines the study’s contributions and limitations. 16 1.5 Summary of the Chapter Currently, there is a concerning rise in misinformation occurring in the public sphere. Addressing this problem requires understanding both the systems delivering information and users’ interactions with information. In this chapter, I explained the evolution of Goffman’s (1974) frame analysis and the need for further research in this area regarding the information-seeking process. I also explained how prior studies in the information sciences argued that scholars should expand their focus on search and retrieval to include the context of a work task (Byström & Hansen, 2005). In addition, more observations on a person’s cognition when searching and learning are required (Urgo & Arguello, 2023; Vakkari, 2016). My prior study with Eric Meyers found that three types of frames affect how people search (i.e., the subject, medium, and source frame). I further proposed that a fourth frame, the task frame, may influence searchers. I briefly outline an experiment to investigate this fourth frame amongst the various sources of bias influencing their search process. Chapter 2 goes into greater depth regarding sense-making and how an important and growing body of literature, called Search As Learning (SAL), addresses the new developments in search engine design and use. 17 2 Literature Review This chapter first reviews the literature on the concept of frames and sense-making. It then uses literature in Search as Learning (SAL) to elaborate on three concepts: Information Selection, Information Use, and Cognition during the Information Search Process (ISP). Each of these concepts is defined within its own respective section. 2.1 Frames and Sense-Making With all the scientific information available online, how do students inform themselves about the sciences today in the “Information Age”? In many ways, access to large volumes of information benefits learning. For example, scientific knowledge is widely available on the Internet and online social networks disseminate scientific communication. However, amidst the growth in access to information, humans also require a way to make sense of and interpret scientific information beyond access to massive amounts of data. The philosophy of science espouses that scientific progress depends on social consensus and conflicts (Longino, 2018) – not just access to data. In parallel with social consensus and conflicts, society requires a way to make sense of the data via processes known as sense-making and its counter sense-unmaking. Dervin (1998) describes the relationship between the two processes as: Sense making and sense unmaking is a mandate for the human condition. Humans, Sense making [sic] assumes, live in a world of gaps: a reality that changes across time and space and is at least in part ‘gappy’ at a given time-space; a human society filled with difference manifested in madness, culture, personality, inventiveness, tentativeness, and capriciousness; a self that that is sometimes centered, sometimes muddled, and always becoming. (p.36) 18 It is common in academia for multiple disciplines to come together to interpret scientific data ad hoc and inconclusively (Edwards et al., 2011). The interdisciplinary collaborations in the sciences eventually evolve to address complex issues (Williams, 2023). In tandem, the massive sense-making that science requires can also reach international political debates. For example, multiple nations debated back and forth over the politics of where, how, and why the Large Hadron Collider ought to be built (Giudice, 2012). Of course, collaborations will create massive amounts of data, but the information that is useful to the majority of scholars and the public is derived from the interactions to interpret the data, more so than the data itself. Human interactions escalate well past any presentation of data because the sciences often contain multiple perspectives. The public also increases discourse when the science is controversial. Moreover, when the science conflicts with a corporate agenda, such as in the case of global warming and fossil fuel profits or cigarette carcinogens and the tobacco industry, then corporations might spend billions to purposefully produce decades of deliberately hair-splitting debates that overwhelm the public discourse required for action (See: Oreskes’ (2019) “Why Trust Science?” for a summary of this form of deliberate information overload). In the end, people have access to a plethora of information on the Internet, but they still require an ability to make value judgments on the quality of the information. 2.1.1 Frame Theory Erving Goffman’s (1974) frame theory argues that frames organize an individual’s perception of an experience. Goffman (1974) uses the example of how a picture frame structures a photograph that captures an experience. Thus, an otherwise forgetful experience may be photographed and 19 given meaning because the frame holds it up for display. Ultimately, what is included in a frame's set can be either set by an agent (e.g., a newspaper editor who builds an agenda for which information qualifies as news) or interpreted by an agent (e.g., a skeptical reader looking for a newspaper's bias). In her book Framing and Discourse, Deborah Tannen (1993) uses Goffman’s (1974) frame theory to explore how frames occur at multiple levels of relationships in a community. Tannen observed framing in her study of a mother bringing her child to a doctor’s hospital. Tannen noticed that the doctor would simultaneously juggle multiple different frames of interactions between the patient, the mother in the room, and even the cameras set up by a researcher. Information science scholars, such as danah boyd (2008) and Zeynep Tufecki (2008), also use Goffman’s (1974) framing theory to explain the various frames people use on social media or when they author online documents (e.g., a blog). Alongside the framing of documents, Goffman’s (1974) theory is also applied to search engines (Andersson, 2017). Andersson (2017) applied Goffman’s theory to students and their experience with Google. Andersson (2017) found that students frame Google as either an authority, a neutral infrastructure, or a fact-finding tool. This latter frame is explained by the research as: Participants view Google in terms of a fact-finding machine through which they can locate a certain type of information. This framing can be interpreted as support for Rieh et al.’s (2016) argument that search engines put factual learning in focus over other types of learning (Andersson, 2017). Therefore, how students frame a search engine will affect learning. Understanding how the students’ frame affects learning can improve pedagogy. For example, in one study, youths were asked to draw their mental model of how a Google search engine worked within a user's search 20 process (Kodama et al., 2017). The findings revealed that youths would often anthropomorphize system-side elements of Google. The findings led the study's authors to conclude that instructors need to revisit how the search process is taught in classes (Kodama et al., 2017). Framing a task’s description can also lead to biased outcomes (Kahneman, 2013). In these types of experiments the task process is mostly fixed but changing the wording of the task description can influence people (Kahneman, 2013; Kühberger, 1998), For example, a task frame claiming an action has a 90% probability of success biases participants to act on a decision as opposed to telling them it has a 10% probability of failure (Kahneman, 2013). Also, task frames that hint people towards what role they should play can influence people For example, hinting at whether participants should play the role of a buyer or seller influences people’s outputs (Neale et al., 1987). A meta-analysis of the various ways of framing a task suggested that questions remain about how nuanced the wording of a task frame can be to influence behavior (Kühberger, 1998). For example, should the wording of the task itself be different for two groups of participants or can the preamble before a task’s question frame the task? 2.1.2 How Information Scholars Approach Sense-Making In the Information Sciences, early scholars examined how users search and access information using system-centric study designs, such as the Cranfield tests (Ellis et al., 1993). System-centric benefits do not always translate to a user’s benefits. If the tools measuring the effectiveness of IR systems focus on system benefits, then the user’s needs may be missed (Turpin & Scholer, 2006). Information retrieval scholars, such as Nick Belkin (1984) and Peter Ingwerson (1996) later adopted the “cognitive viewpoint” (Ingwerson, 1984). This viewpoint centers on human 21 sense-making over the decisiveness of humans (Kuhlthau, 1993). For example, Carol Kuhlthau (1993) writes that within the bibliographic paradigm, “systems and intermediaries are presently directed to answering well-defined questions, not ill-defined ones reflecting uncertainty.” As another example, Dervin (1998) associated information seeking with a human’s movement through space and time and the person formulating a point of view. Nick Belkin (1982) suggested that the state of a problem and a person’s position regarding solving the problem should be included in any definition of sense-making. Kuhlthau (1993) expanded on these definitions to include a person’s actions, experiences, and cognitive rationalizations. While Nick Belkin’s (1982) concept of the Anomalous State of Knowledge, Brenda Dervin’s (1998) concept of Sense-making during the information-seeking process, and Carol Kuhlthau’s (1993) concept of the Information Search Process present different models of seeking, the models generally agree that uncertainty arises when a person recognizes a gap or an anomaly between the given context at hand and their desired situation (Wilson, 1999). As information scholars adopted the cognitive viewpoint, academia’s viewpoint of “cognition” also developed to include the concept of distributed cognition and embodied cognition. Distributed cognition theorizes that a person’s thoughts can be distributed in a system so that they can be used at different times or even by different people who engage with a system (e.g., writing down where someone left off in a task so that others may pick up on the task later) (Salomon, 1997). Embodied cognition theorizes that a person’s thoughts are not isolated from their environment, but can embody them as extensions of the body (Dourish, 2001). For example, a person using a computer’s mouse does not need to actively be conscious of the parameter between their hand and mouse and can subconsciously perceive the entire wire as an extension of their arm to the cursor. 22 Task-based approaches are one way for scholars to research the role of context. Vakkari (2003) writes that scholars need to account for task-based approaches to information seeking. A work task approach allows researchers to observe for information seeking and information searching via behavioral and cognitive approaches (Byström & Hansen, 2005). A person will alter their mental model and perception of the relevancy of the information they seek and retrieve based on the goal of a task (Vakkari, 2003). Building on Kuhlthau's model of the information-seeking process, Vakkari (2001) found correlations between a student's mental model, the information sought, the search tactics used, and the assessment of the relevancy and usefulness of the information. With a seeking task, the relevancy of information is not the most important variable for scholars to account for. In such situations, information scholars espouse studying information usefulness more closely than relevancy (Cole et al., 2009). As Cole et al., (2009) write: The purpose of an information retrieval (IR) system is to help users accomplish a task. IR system evaluation should consider both task success and the value of support given over the entire information-seeking episode. Relevance-based measurements fail to address these requirements (p.1). A person’s task-based information needs can lead them to pull information from a variety of sources (Byström & Kumpulainen, 2020). Decisions are made primarily to progress through a task than to adhere to a prior formulation of how to complete a task (Byström & Kumpulainen, 2020). In a task, information is sought due to a vertical relationship (between general and specific, from a workplace context to an information source) (Byström & Kumpulainen, 2020). A work task is developed by accounting for these vertical information needs, such as contextual 23 and situational descriptions alongside topical descriptions for a query (Byström & Hansen, 2005). The complexity of the task question can also affect how participants search and comprehend a task (Rouet et al., 2001). A task question that demands a higher level of learning can influence participant search patterns to focus on reviewing information and integrating it into their output (Rouet et al., 2001). Conversely, a task question with a lower level of complexity can influence participants’ search patterns to focus on locating and memorizing information (Rouet et al., 2001). However, not all questions fall under the dichotomy of low-level and high-level questions: An open-problem task can have seemingly low levels of complexity, but without any clear answers can be more demanding on cognitive load. In such cases, a pilot study can help determine the type of search patterns to expect (Borlund & Schneider, 2010). With the emergence of the Internet, scholars faced the challenge of observing how a user’s ISP functions alongside the dynamic evolutions of search tools. Scholars are responding to the challenges in our current Information Age by examining the evolution of human-information interactions (HII) when searching (Oliveira & Lopes, 2023). HII is defined as the ecology of interactions that take place with information behaviors and human-computer interactions (i.e., both the inner cognitive and external social interactions) (Fidel, 2012). The term ecology here refers to all the main stakeholders within a defined system and the dynamics of their interactions (Fidel, 2012). For example, an individual searching online includes the interactions of multiple stakeholders, from the search engine designers to the authors of the websites retrieved. Scholars need to focus on the process of seeking and sense-making that takes place by individuals in 24 relation to their society (Fidel, 2012). Seeking and sense-making are cognitive processes that are both internal to the mind and externally distributed in the minds of other individuals, family members, and communities (i.e., distributed cognition) (Salomon, 1997). Information is not just communicated between people; individuals must share similar frames to understand the meaning of the information. During the search process, people require the ability to search the context, such as metadata, as well as the desired information. A person’s ability to search and use context is essential for SAL (Smith & Rieh, 2019). People are not only thinking about information but are metacognitively thinking about their learning. Systems can support a user’s ability to search contextual information during searching and learning (Smith & Rieh, 2019). Therefore, understanding the search process requires recognizing that it consists of learning-centric elements, user elements (i.e., cognition), and system elements (i.e., computer interactions). Smith & Rieh (2019) argued that users engage in transactive memory during their search process by sharing their memory of information split between their cognition and a system. For example, people might remember where they found the information they searched without memorizing the content since they can always re-find it, strengthening the bond between cognition and a system. More nuanced searched interactions can be further elicited via exploratory search tasks with an open-ended problem (i.e., a problem that lacks an easy or clear resolution) (Wildemuth & Freund, 2012). Exploratory tasks are oriented towards learning, the context and situation of the task are clearly specified, multi-faceted (e.g., results with multiple perspectives), multi-staged (e.g., including note-taking and a writing output in a task), and the data collection should be attuned to record these nuances (Wildemuth & Freund, 2012). Similarly, I used a search task on an open-problem 25 with multiple perspectives and stages to elicit more nuanced interactions between a system and cognition. The next section presents the SAL literature that informed me on how to observe such nuanced search interactions. 2.2 Search As Learning Search As Learning (SAL) is a concept in the Information Sciences that concerns itself with how search affects learning and comprehension (Urgo & Arguello, 2023). SAL can be defined as “search as a user-driven, technology supported process of human information interaction (HII). In this process, people engage intellectually, physically and emotionally with digital information” (Freund et al., 2014, p.1). A particular niche of SAL studies is a form of self-regulated learning that requires greater information literacy, such as developing critical thinking skills when exploring a SERP (Rieh et al., 2016). Furthermore, greater information-literacy allows users to sense-make using metacognition (i.e., thinking about thinking) (Smith & Rieh, 2019). For example, users need the ability to not just think about information, but reflect on how they are thinking about the information (e.g., are they thinking critically about the information or are they biased towards the information?). As a field, SAL is evolving in tandem with the technological developments of the Internet, such as the complexity of a search algorithm or a SERP’s visualization. However, the technology may limit some of the offline embodiments that humans have developed for searching and learning (H. Zhang et al., 2022). These limitations include browsing the same way that humans do in the physical world (H. Zhang et al., 2022) (i.e., using our peripheral vision in a triaging manner). Another limitation is technology’s lag in simulating physical interactions (Nichols & Twidale, 2011). For example, a worn-out book with annotations in a library may not be as “pristine” as a 26 digital book, but it immediately alerts other patrons that the book is often read (Nichols & Twidale, 2011). The virtual features added to traditional media designed for learning also require caution. Today, children are raised with a variety of media interconnected via the Internet (i.e., data analytics connecting various toys to create “an Internet of Toys” (Holloway & Green, 2016)), and little research has studied whether children consent to the datafying of their learning process as they mature and whether profit-driven companies affect their maturation (Holloway & Green, 2016). In addition, little research has studied the psychological comfort of growing-up with physical media where there is no virtual lag, no interruptions, no data collection, or no profit-driven agenda by the stakeholders in control of the medium. At its core, SAL is interested in understanding and supporting the ISP (Vakkari, 2016). Vakkari (2016) writes in his survey of the SAL literature that the root model structuring searching as learning is based on Kuhlthau’s articulation of the ISP. Modern developments in society and technology may affect how users proceed through an ISP. For example, Vakkari (2016, p.16) argues that there is “an urgent need to develop tools that support users to interact with sources for constructing an end-product by structuring and manipulating information.” Doing so requires developing cognitive tools and evaluating a user’s progress in learning (Vakkari, 2016). Vakkari’s (2016) survey of the SAL literature addresses three areas where the field of SAL requires greater exploration: The first is how modern social and technological progress affects our traditional model of the ISP. Second, is the cognitive development that occurs during SAL. And third, is a more philosophical development of learning. 27 Addressing these components of SAL requires adopting a new approach to examine learning. O’Brien et al. (2022) studied participants with varying domain-knowledge and knowledge of search techniques. The authors found that “search expertise, more so than domain expertise, played an important mediating role in learning during the search process” (O’Brien et al., 2022). Thus, domain expertise does not always determine how well people will learn from a search task. SAL scholars found that self-regulated learning (i.e., how people control their learning process) is essential to information literacy (Urgo & Arguello, 2022) . Previously established theories in information science, psychology, and cognitive science identify various observable think-aloud variables corresponding with self-regulated learning (Urgo & Arguello, 2022). Notably, Urgo and Arguello (2022) use Greene et al., (2015) observations prior work with observing self-regulated learning amongst participants tasked with informing themselves about science topics (Greene et al., 2015). Based on work by Greene and others, Urgo and Arguello (2022) defined a Self-Regulated Learning Coding Framework. Urgo and Arguello (2022) aggregated the cognitive and behavioral variables studied in prior literature on self-regulated learning with SAL experiments. The authors then used their collection to highlight which variables pertain to an ISP and can be informed by a think-aloud. The authors found five significant components from the Self-Regulating Learning Coding Framework that apply to Search as Learning: Planning, Strategy-Use, Monitoring, Task Difficulty & Demands and Interest (for a more extensive list see: Urgo & Arguello, 2022). Drawing on these prior studies, I experimented with the participants' ideation, selection, site-use, task outputs, and metacognition. I chose these components in particular because my experiment looked specifically at how participants wrestle with an open problem with multiple frames. 28 Specifically, the designed task was an exploratory search task. Wildemuth and Freund’s (2012) definition of this type of search task explains: Exploratory search tasks focus on learning and investigative search goals; they are general (rather than specific), open-ended, and often target multiple items/documents; they involve uncertainty and are motivated by ill-defined or ill-structured problems; they are dynamic and evolve over time; they are multi-faceted and may be procedurally complex; and they are often accompanied by other information or cognitive behaviors, such as sensemaking (Wildemuth & Freund, 2012, p.1). The following sub-sections present prior literature on these concepts. The measures for the data are derived and operationalized from the recommendation by Urgo and Arguello (2022b), including their work with Capra (2020). Section 2.2.1 relates to ideation and information selection, Section 2.2.2 relates to site-use and task outputs, and Section 2.2.3 relates to metacognition. 2.2.1 Information Selection, Ideation, and Query-Construction in SAL The first research question in my dissertation is about exploring how task frames affect people when they ideate and select information (i.e., RQ1: How does task framing affect the information selection processes of students working on open-ended information problems?) Part of ideating how to carry out a task includes the query-constructions (Urgo & Arguello, 2022b). Studying query-construction can inform understanding of both SAL processes and learning outcomes (Urgo & Arguello, 2022b). For example, the number of query terms and the diversity of terms are variables that SAL researchers can study to identify impacts on learning outcomes (Bhattacharya & Gwizdka, 2019; Collins-Thompson et al., 2016; Palani et al., 2021). A starting point for analyzing query variables involves looking at the frequency of words used 29 and removing the stop words (e.g., the, and, and but) (Chapman et al., 2013). Inferences can then also be made with how participants summarize a topic. For example, Bhattacharya & Gwizdka (2019) found that query terms with uncommon vocabulary are correlated with greater learning outcomes. Urgo and Arguello (2020) suggested that SAL researchers interested in observing the preliminary steps of ideation should look for the variables that can be coded (especially during a think-aloud): writing preliminary notes, restating the goal, creating subgoals, and time-planning (Urgo & Arguello, 2002). Having note-taking features enabled in the first place contributes to SAL research as it helps facilitate a searcher’s understanding and remembering of a searched topic (Roy et al., 2021). In addition to note-taking and query-constructions, I will be looking at what Ong et al., (2017) define as the click-throughs (i.e., which sites are visited) and the participants’ time-on-site (i.e., the number of seconds participants spend on the website clicked) and time-on-task (i.e., the number of seconds participants spend on the entire task). 2.2.2 Information Use –Studying the SERP Interactions This study is particularly interested in the variables that relate to strategy use. The second research question for my dissertation is RQ2: How does task framing affect the information use of students working on open-ended information problems? Analyses of site-use variables on a SERP can begin with whether users can choose to spend their time making depth-first searches (i.e., iteratively clicking and reading individual links) or a breadth-first search (i.e., skimming all the SERP results before clicking) (Klöckner et al., 2004). Kelly & Azzopardi (2015) and Yamamoto & Yamamoto (2020) looked at the search-depth (i.e., how far down the SERP did the participants click) by scoring the number of links a user clicked through a SERP and comparing 30 different groups of participants. In their survey of prior literature on SAL and self-regulated learning, Urgo and Arguello (2022) collected and analyzed several variables for observing strategy-use. These include observing how participants compare and contrast, coordinate informational sources, corroborate sources, hypothesize, and infer (Urgo & Arguello, 2022c). In addition, observations can be made on how people use the information on the SERP. For example, Li et al. (2009) observed when participants made “good abandonments” of SERP pages when they were satisfied enough with a SERP’s thumbnail, title, and snippet to use the information to make a decision without needing to click on a link for it to count as a click-through. Therefore, it is important for the researcher to not only gather data on the trace-data (e.g., mouse clicks and site visits) of computer interactions, but also to observe the participants’ interactions to make observations to support the quantitative data analysis (Borlund & Schneider, 2010). Furthermore, mislearning can affect search. Meyers (2018) observed that even with quality information, there is the question of whether someone learns information simply via direct access. He argues that task failures are not deficits in a student's inquiry process, but areas for pedagogical interventions. Meyers (2018) also argues that a student's reading skills and ability to evaluate information may require more scholarly focus than the quality of information they retrieve and engage with. Observing how well participants understand the topic they searched from search results is challenging for all scholars (Salmerón & García, 2011). Experimenting with SAL can meet some of these challenges. SAL emerged from "a growing trend in studies to expand and re-conceptualize the search process from querying and inspecting search results to cover also working with search results for creating the outcome of a search" (Vakkari, 2016). Thus, some 31 further inferences can be made by analyzing user writing outputs. Evaluating participant outputs can also involve coding users’ task outputs in a search task (O’Brien et al., 2020). Another method of observing a participant’s learning outcomes includes providing a test (e.g., multiple-choice or true or false) (X. Zhang et al., 2011). Researchers can also analyze writing outputs using a content analysis alongside a rubric. Wildemuth (2016) outlines a step-by-step approach for conducting content analysis for information science researchers. Content analysis involves several stages, including data preparation, identification of the unit of analysis, creation of categories and a coding scheme, testing of the coding scheme on a sample of text, coding of the full text, assessment of the coding's consistency, analysis of the coded data to draw conclusions, and reporting of the methods and findings (Y. Zhang & Wildemuth, 2009). A content analysis can also assist with looking at writing outputs (Y. Zhang & Wildemuth, 2009). Furthermore, Bali & Ramadan (2007) found that when a rubric is used in tandem with a content analysis, both test measures complement each other when assessing online learning outcomes. Using a content analysis alongside a rubric provided qualitative nuance to analyzing the outputs of both individuals and a group learning online (Bali & Ramadan, 2007). In addition, a rubric introduces more objective analysis and measures to apply the themes that emerged in a content analysis (Bali & Ramadan, 2007). Studying strategy-use variables during SAL can also lead to better system designs. For example, one study experimented with a system feature that would re-rank the document order to correspond with set demographics based on learning needs (Weingart & Eickhoff, 2016). As 32 another example, a software feature, ULearn, was developed to assist users’ understanding when SAL takes place within a medical context (Alfano et al., 2019). One SAL study experimenting with system features found that including highly relevant pictures alongside text assists the cognitive effort required for participants to complete a search task (Karanam et al., 2012). Another similar tool, SaR-Web-a, supports SAL by providing users more with information on the missing cultural context surrounding their search (Fulantelli et al., 2016). SAL research also assists with considerations on how a search task relates to a work task. For example, X-Site is an enterprise search system that takes into account how corporate document-genres (e.g., white papers, technical reports, and emails) pertain to an employee's search needs within a work task (Yeung, 2007). 2.2.3 Reconciling Frames – Studying Cognition During the ISP The third research question in my dissertation is interested in how people interact with multiple frames (RQ3: How do task, medium, and source frames interact in the process of making sense of open-ended information problems?). The human ability to reconcile multiple frames is a human trait that is difficult to quantify. However, this ability requires greater research from scholars because it is an important part of communication, but there are no system designs for it. For example, AI has a problem with replicating how humans frame – a phenomenon AI scholars refer to as the ‘framing problem.’ (Chow, 2013; Dennett, 1984; J. A. Fodor, 2008). Suppose an AI system is simulating the role of a doctor (i.e., working with the frames of a doctor). In that case, it might have a problem reframing its parameters to simulate a less professional but more compassionate role if a patient begins to cry. Conversely, a human is able to recognize when to work with the frames of a formal but professional doctor or an informal but compassionate 33 human. The doctor and patient can manage multiple frames, from the doctor having a formal professional frame to a friendlier informal one. AI scholars argue that the ability to recognize multiple frames is fundamental to an agent’s (both humans and computers) search for information (J. Fodor, 2006). Conceptualizing abductive reasoning as a process for approaching frames can be useful. How abductive reasoning leads individuals to select frames for information is a problem not very well understood by scholars (J. Fodor, 2006) but deserves greater attention. Seeking new information and creating hypotheses are qualities of abductive reasoning (i.e., abductive reasoning is the pragmatic approach of recognizing multiple perspectives on prior data. It requires a hypothesis to acquire novel data). For example, two children pushing each other need to learn to differentiate whether a push was in a play-frame or an aggressive-frame (Tannen, 1993). How do the children make that differentiation? The children learn to differentiate between these two frames via another frame of searching for and building hypotheses. In this search frame, two children hitting each other with their hands may raise questions that lead to different hypothetical situations: Are the actions occurring in a game of tag? Then the children need to understand the information they are sharing by tagging one another in a “play” frame. Does a child not want to be pushed? If so, then these actions lead to a more violent frame. People may use communication and discourse to resolve these forms of conflicting frames (Tannen, 1993). People may also make sense of situations by individually making a hypothesis about conflicting information (Peirce, 1905). Researching abductive reasoning requires exploring the concept of search, learning, and cognition. Research on the concept of SAL finds that it is intertwined with cognition. For 34 example, users may structure their learning schemas in accordance with a SERP's resources (Reynolds et al., 2019). Experimenting with these variables can lead to making predictions on cognitive outcomes. For example, one study carried out an experiment to observe whether predictions can be made on the number of visited websites and the learning outcome (Sharit et al., 2008). As another example, one study found that more comprehensive summaries written after a search are associated with greater working memory (Choi et al., 2019). Paying attention to cognitive factors can help reveal nuanced context (Weingart & Eickhoff, 2016). In one study, participants were observed by researchers to determine when they interacted with text results and video results (Pardi et al., 2019). The researchers asked the participants to then summarize their findings and discovered that some participants used the document-genre (i.e., text or video) of the search results to justify their findings. The experiment concluded that the document-genre of the results affected how a user summarized their findings (Pardi et al., 2019). Studying the cognitive nuances can also reconcile and further scholarly research. For example, it is generally agreed that longer reaction times are indications of greater cognitive effort (Kellogg, 1987). Similarly, one SAL experiment found that a user's time-on-site can lead to greater knowledge gains (Yu et al., 2018). However, these findings conflict with a similar SAL experiment where the researchers found that the length of time correlates with less knowledge gains (Bhattacharya & Gwizdka, 2018). The conflicting results spurred on greater research: Bhattacharya and Gwizdka (2019) reconciled the differences they had with Yu et al.’s (2019) study by looking further into the nuances of the inferences being made. In a post-hoc analysis, Bhattacharya and Gwizdka (2019) suggested that a possible reason they witnessed less knowledge gains with students who spent more time on websites is because their participants experienced more 35 difficulty comprehending the content of websites and therefore required more time than participants in Yu et al.’s (2019). Urgo and Arguello (2022) suggest that a deep understanding of cognition and SAL requires an investigation of what exactly participants are thinking during a search task. Another method of understanding what people think during SAL involves collecting the utterances during SAL (Urgo & Arguello, 2022a). In their review of the literature, Urgo and Arguello (2022) observed that the advantage of using the think-aloud protocol during SAL is that it can capture the dynamics of self-regulated learning (i.e., a user actively monitoring and taking control of what they are learning to achieve a goal (Urgo & Arguello, 2022a). Anders Ericsson and Herbert Simon (1993) developed the think-aloud protocol to assist researchers with making cognitive insights into a person’s thought process during a task. In a think-aloud, a participant utters the often-fragmented thoughts in their mind (Ericsson & Simon, 1993). Each utterance can be determined by when the participant made pauses in their think-aloud, which is a common definition of an utterance (MacGregor et al., 2010; Traum & Heeman, 1997). One way of observing a think-aloud is to combine a person’s concurrent think-aloud (i.e., people think-aloud almost simultaneously with the actions they take) and their retrospective think-aloud (i.e., people reflect on the action they take) (Ericsson & Simon, 1993). While not every human has a “voice in their head,” the phenomenon is common enough that collecting such utterances will provide insights into the narrative that accompanies a person’s cognitive state. While some of these actions can be observed without a think-aloud (e.g., note-taking), a better understanding of how people are self-regulating their learning during SAL emerges with a think- 36 aloud (Urgo & Arguello, 2022). In addition, what is of interest in an analysis are the differences that emerge between a person’s concurrent and retrospective think-aloud, such as when participants misremember the process they used to complete a task (Ericsson & Simon, 1993). Ericsson and Simon (1993) suggest one way to analyze the think-aloud is to make a 1) Qualitative analysis of the utterances for major themes, 2) Observations of the utterances that can qualify as a theme, and then 3) Compare the concurrent and retrospective think-alouds for discrepancies that affect these themes. Information scholars understand that people have access to a large amount of information (including misinformation) today. How people frame this information accurately requires sense-making to make value judgments on the quality of information. Understanding how users sense-make is just as crucial as understanding information access. Sense-making requires users to engage with their situation and social surroundings. These engagements can be a form of distributed cognition (Salomon, 1997), whereby a user can distribute who and when others can engage with a system. If sense-making is essential to individuals and cognition can be distributed, then the public plays a vital role in how people engage with quality information. I ground this concept in the Pragmatic philosophy put forth by John Dewey (2013) and Charles Peirce (1905). For Dewey’s Pragmatism, intelligence is a method to seek knowledge and never a final state. In addition, Peirce (1905) argues that knowledge is retrieved through a method of applying a type of search heuristic (i.e., abductive reasoning) that consists of a set of biological “admissible hypotheses.” As the next chapter outlines, by using a think-aloud that collects data on how participants use abductive reasoning, I can then observe if it also helped people look for, recognize, and reconcile multiple frames. 37 3 Methods 3.1 Study Design Framing is an essential part of understanding how people seek and understand information that they find online. My study builds on prior research by Novin and Meyers (2017a), both in its conceptual underpinnings and methods. Novin and Meyers (2017) observed students as they examined a Search Engine Results Page (SERP) that presented multiple perspectives on a science topic. My earlier work found that the SERP creates a frame that affects how searchers understand and write about a topic. In the current study, I explore this phenomenon further by looking at how the task description affects the synthesis of multiple perspectives and the cognitive reasoning that users employ in this process. The cognitive reasoning that leads a person to examine and synthesize multiple perspectives is referred to as abductive reasoning. To observe abductive reasoning, I selected a mixed-method approach, employing a task-based experiment with two conditions during which participants perform a think-aloud protocol. I then conducted an interview with each participant. Finally, I debriefed the participants about the experiment. A key difference between prior work and my current study was the development of a subtle difference between task descriptions for the two groups of participants, which I anticipated would create a framing effect. The research questions for my dissertation are: RQ1: How does task framing affect the information selection processes of students working on open-ended information problems? RQ2: How does task framing affect the information use (synthesis) of students working on open-ended information problems? 38 RQ3: How do task, medium, and source frames interact in the process of making sense of open-ended information problems? The experimental portion of my study aimed to make inferences on how framing a task question affects the information-seeking process. The experiment used a search task with an open-ended problem, as this can elicit more nuanced interactions (Wildemuth & Freund, 2012). Similar to Wildemuth and Freund’s (2012) suggestions for a task design, this task was oriented towards learning, and the context and situation of the task were specified (i.e., participants were asked to research and report a topic as though it was for a group assignment), multi-faceted (the SERP results had multiple perspectives), the task was multi-staged (participants could include notes, query and interact with a SERP (under controlled conditions), and then write output in a task), and the data collection methods were attuned to record these interactions (Wildemuth & Freund, 2012). Two of the main components of a task include: A task description that defines a requirement and goal and the task processes, which are the behaviors manifested when a task is performed (Wildemuth & Freund, 2012). Both groups received the same task objective, which involved reading a task description and conducting a search online for information that can help answer the task question, and then writing the answer in a task output. One group received a preamble in the task description that I anticipated would make a more precise frame for the task. Methodologically, my study used a Mixed-Methods design. Mixing data risks producing ordering effects that affect the analysis. Moving from quantitative data analysis to qualitative data analysis is generally more suited for explanatory results where the qualitative data is used to explain the quantitative data (Creswell, 2011). On the other hand, moving from qualitative to 39 quantitative data is generally suited for exploratory results where the qualitative data is collected, and then the frequency of concepts is quantitatively measured (Creswell, 2011). In both cases, the secondary data collection is used to explain/explore the initial accumulation of data. While there are different ways researchers can mix data based on the explanatory or exploratory approach (Creswell, 2011), I use a sequential explanatory strategy so that after testing for the frequency of search behaviors (e.g., clicks used and the frequency of query terms) with a quantitative analysis, a qualitative analysis provides context on the themes that occurred during my experiment (e.g., what participants uttered during their think-aloud) (Creswell, 2014). Section 3.1.1 explains how I developed a one-factor, two-condition experimental design. Section 3.2 will explain interruptions to the study due to the pandemic. Section 3.3 presents the participant task and how the data was collected (i.e., observing the participant’s interactions with a computer (i.e., trace data), the utterances in the think-aloud, and the construction of their writing task output). The methods for data analysis are then presented in Section 3.4. 3.1.1 Experimental Conditions Prior research has shown that the top five results on a SERP receive far more engagement from users than the bottom five results (on a SERP presenting ten results) (Epstein & Robertson, 2015). The principle that describes these patterns is sometimes called “search depth” (Yamamoto & Yamamoto, 2020). To demonstrate how the two groups of students in this study were compared, suppose a SERP’s topic has two perspectives, X and Y, where X is presented at the top of the SERP while Y is at the bottom. The prior study by Novin and Meyers (2017) found that students will favor mentioning X over Y because they are more biased toward information 40 that appears at the top of the SERP. Some students will mention both X and Y perspectives. However, the majority of users draw most heavily from the top search results (Novin & Meyers, 2017; Epstein & Robertson, 2015). One goal of my experiment was to see if this pattern of use could be disrupted through the task description. In Condition 1, students were provided with a brief research question (13 words) and a set of 10 SERP results related to the question. Condition 2 provided the same task question and SERP resources, but added a short 14-word preamble before the task. I anticipated that Condition 2’s preamble would focus the students' attention and orient them to the bottom SERP results. While both conditions received identical SERPs, Condition 2’s preamble used a similar framing of the topic as the results at the bottom of the SERP. Both conditions were to produce a task output answering the task question. Participants were asked to frame their answers as though they were working on a group project for an instructor’s class assignment. In addition, they were asked to write their notes and answers as though their audience were their group members in the class. While Google users often do not have their task descriptions and objectives written out for them (e.g., as often depicted in Brenda Dervin's bridge metaphor or Nick Belkin's anomalous state of knowledge), students in group projects often do have the main objectives of their task written down in an assignment format by an instructor. Therefore, both presenting the task frame as a written objective for a ‘classroom assignment’ with an intended audience increased the external validity. 41 3.1.2 Task Design The topic for the task was an open science problem. Novin and Meyers (2017) and Huang (2020) showed that framing occurs with scientific controversies related to climate change. However, Novin & Meyers (2017) looked at biofuels while Huang (2020) look at the complexities of climate change – both of the topics chosen were less familiar to students. It is much easier to observe metacognition or “thinking about thinking” during a think-aloud when a person is somewhat familiar with the topic they need to think about (Smith et al., 2022). Certain types of bias like a media bias (i.e., SERP bias) are lessened when participants have greater familiarity with a topic (Bink et al., 2022). Familiarity with the search task also leads to a better understanding of a participant’s search process (Borlund & Schneider, 2010). For example, in an educational search task, students should be informed of the purpose of the task assignment (Smith et al., 2022). My study aimed to better understand how students used cognition, such as using their prior knowledge to judge the quality and completeness of the information presented. With this in mind, I chose a task question for this experiment focused on a controversy surrounding a topic that is both familiar to students and affects their daily activities: cellphones on campus. Debates over cellphones often appear in the media and Britannica’s ProCon (2020), a website dedicated to debate coverage, lists cellphones as a highly debated topic with various perspectives. Cellphones are not only a commonly understood controversy (Draws, 2021), but a prior study by Draws (2021) showed that strong signs of cognitive biases emerge when students explore SERPs on the topic of cellphones. Thus, I created the following tasks for participants. One task question had a preamble about health and is henceforth referred to as the “Health-framed task” (HF) and the second question had no preamble and is henceforth referred to as the “Non-Health-Framed Preamble” (NHFP) task. Participants will be distinguished by whether they 42 were in the HF group or the NHFP group. To re-iterate, not having a health frame preamble does not mean there was no frame, just no health frame preamble. Which frame participants responded to is left for the Findings and Discussion section. I did not know if the participants would see the Health frame (or if the participants would respond to another frame). Task Topic: Cellphones HF task (i.e., health preamble): The health and safety of the students at UBC should be a top priority. Should UBC restrict the use of cellphones on school premises? Please explain why. NHFP task (i.e., no health preamble): Should UBC restrict the use of cellphones on UBC’s premises? Please explain why. I anticipated that the question’s preamble (i.e., “the health and safety…”) might create a focus for the students’ inquiry. The extent to which these words might make a difference depends on whether the focus requires different resources to answer the question. Figure 2 Diagram of the study design’s relationship to RQ1, 2, and 3. 1) Participants were assigned randomly to whether they would receive a task with the health frame. 2) A video recorded participants' interactions 43 with the search process and the product output. 3) To answer RQ1, Queries, Click-throughs, and Time-on-Site was collected. 4) For RQ2, Sites-used, Terms-Used, and a Rubric was collected. 5) Throughout the process participants thought aloud. Ericsson and Simon’s (1993) Think-Aloud Protocol was used to collect both concurrent and retrospective (post-task think-aloud). 6) Both types of think-alouds were triangulated with the search actions to answer RQ3. Participants were asked to make utterances about what they were thinking as they completed the task. An utterance is when a person makes an extended pause in their think-aloud or a gap in a person’s speech (MacGregor et al., 2010; Traum & Heeman, 1997). The utterances were collected in accordance with Anders Ericsson and Herbert Simon’s (1993) Think-Aloud Protocol. The utterances collected during this portion of the study design were qualified as what Ericsson and Simon (1993) refer to as a “concurrent think-aloud.” For example, participants often made utterances akin to “I’ll search this” or “I’ll click this.” After the task, I asked participants to retrospectively walk me through the different stages of their search process from start to finish. Here the participants would use their memory to make utterances akin to: “I aimed to find”, “I searched that”, or “I intended to click that.,” in what Ericsson and Simon (1993) refer to as a “retrospective think aloud.” The think-aloud protocol was also tested in the pilot (See section 3.1.4). After the pilot, one adjustment made was requesting participants to read aloud what they were reading during the pilot. Alongside the video recording of participants’ faces and behavior, this adjustment allowed me to determine where participants were focusing their attention, such as whether they were skimming a webpage. I used the video recordings to triangulate both types of think-alouds (concurrent and retrospective) with the participants’ search moves to answer the third research question, which was about how participants make sense of the multiple different frames of information they encounter. 44 3.1.3 SERP Layout To complete the task, the participants in both conditions were provided with a simulated SERP (i.e., a webpage designed to look like a Google search page but with predetermined sources and ordering). The simulated SERP presented the same results in the same order to both groups of participants. Novin and Meyers (2017) also employed a simulated SERP to control the presentation and order of the materials. In this experiment, I arranged the materials so that the top of the SERP presented general information about cellphones that was unrelated to health issues and the bottom of the SERP presented resources that related to cellphones and health issues. While developing this layout, forty different test searches were made using three different browsers and searched anonymously (i.e., using a browser’s incognito mode) for the most common search results provided by SERPs. After accumulating the most frequent results, they were re-ordered into the simulated SERP as shown in Figure 3. In the simulated SERP’s layout, the top four topics contained the most frequent results retrieved that were not about health. These results were primarily about academic concerns, such as students cheating or being distracted by cellphones. The next two results were somewhat health-related (i.e., they were not framed to be about health but did contain some information that relates to health within the article). The bottom four results on the SERP were the most frequently retrieved health-related results. The main health concern with these latter results was whether cell phones cause cancer. When SERPs are constructed for experimental designs testing for biases, a balance is required between not creating artificial lab settings yet controlling for enough factors to detect a bias (Liu, 2023). The simulated-SERP created was a quasi-live corpus version of Google and the 45 Internet: Despite controlling for the document-order of SERP results, almost all other features of Google were interactive. Users could enter queries in Google’s search box and a SERP that looked like Google would appear. Similar to many modern SERPs, participants would see the first three results on screen and would have to scroll down to find the remaining seven websites. Participants could click on any of the sites and retrieve a fully live version of any of the sites they clicked. Thus, participants can click on hyperlinks to search deeper within websites and view personalized advertisements on respective sites. 46 Figure 3 Order of results on the SERP page retrieved by students. Note that Google’s interface may change by the time of publication. 47 The quasi-living corpus of Google was used to increase external validity. While any search engine could be used, Google is the most popular search engine and controls 85.53% of the market (Statista, 2023), Since at least the time of Cranfield tests, information scientists have attempted to simulate the realism behind a user’s search environment while still controlling for key SERP interactions of interest. Information scholars, such as Gwizdka and Bilal (2017), Epstein and Robertson (2018), and Novin and Meyers (2017) often recreate Google’s SERP features -- due to the domination of Google on the search engine market (Statista, 2023). While these authors mention the limitation of creating a simulated environment, the extent of this limitation is rarely stated. However, in an insightful paper by Buhler et al., (2023), the authors argue that this leaves the participants’ perception of being limited as a presumption. Therefore, the researchers conducted a study on the “participants’ perception of the experimental environment itself” (Buhler et al., 2023). In their study, the researchers found that most participants do not find the features of simulated SERP tasks to be unrealistic (Buhler et al., 2023). Within the minority of participants who found certain features of simulated search unrealistic, the most unrealistic portion is the “starting point” of a task: 16% of their participants said they may not choose Google, or any search engine, to begin a search task (Buhler et al., 2023). Instead, students may choose tools designed for their specialization (there is no clear pattern of which tools are preferred) (Buhler et al., 2023). While alternative forms of seeking information were outside the scope of this paper, it speaks to the importance of designing a search task question that justifies the use of Google. Interestingly, the researchers found that within the minority of participants who did not find certain features of simulated SERPs realistic, the most minor criticism was abiding by Ericsson and Simon's (1993) think-aloud protocol (Buhler et al., 2023). As the authors explain, "the very low number of comments about the think- 48 aloud protocol suggests that it is an effective data collection technique when paired with a simulation because it gathers rich data without significantly disrupting the realism of the simulation" (Buhler et al., 2023). Buhler et al., (2023, p 204) conclude that simulated search environments create a "naturalistic environment in which to capture rich, real-time quantitative and qualitative data. The use of behavioral simulation as a library and information science research method can provide researchers with a realistic view into information behavior." The authors suggest that future studies can make simulations more realistic by allowing participants to open SERP links as new tabs or windows (Buhler et al., 2023). My study did enable these features of a clickable SERP and the websites themselves were live. All that was controlled for was the SERP order. Due to the fixed document-order of the SERP, all participants would have to first perceive the top three SERP results which were more focused on academic topics than health information before having to scroll down to find sources that focus on health information. Therefore, the participants in the NHFP group had the option to not view the bottom health-related results while the HF group viewed the academic-focused sources first and then chose whether to click or interact with the sources before scrolling down to the health-related sources. However, the influence of perceiving the top results first was not presumed. For example, Novin and Meyers (2017) found that placing a website with an undesired document-genre at the top of a SERP would often lead to the website being skipped by participants or not read closely enough for the information to be retained. In addition, while the presentation was fixed (and therefore what participants would perceive was fixed) the methods did not make presumptions on the interactions and engagement (e.g., what is read, clicked, or used). For example, the HF 49 participants had the option to choose not to scroll down the SERP. Similarly, the NHFP group had the option to not interact with the top of the SERP and focus on the bottom of the SERP. How I tuned my measurements to observe these processes alongside the think-aloud is presented in the next section. The next section will also explain how I test the experiment’s search task, simulated-SERP, and think-aloud process with 15 participants in a pilot. 3.1.4 Pilot A pilot was conducted with 15 students recruited via convenience sampling to test the designed script and ensure that the study design, instruments for measurement, and simulated search page were working. The pilot test helped me with practicing the script and task procedure (See Section 3.2). Practicing the pilot also assisted with making the script’s instructions and explanations clear and concise. I asked the pilot participants if the task description was clear and then observed if they could complete the task without requiring my assistance. If the participants required clarification on how to complete the task, I was ready to adjust the task script to be clearer. The pilot confirmed that participants will find the search task to be straightforward and while the think-aloud requirement was not a natural requirement, it did not hinder the completion of the task in any substantial way. The pilot also allowed me to confirm whether data was collected correctly by the recording instruments. One adjustment made with the pilot was that I instructed participants to also read aloud as part of their think-aloud. This adjustment enabled the video recording to capture when participants were skimming information: Participants would scroll down the page at a pace faster than they could read off words that captured their attention or comment on during their think-aloud (whether participants subconsciously noticed results were not of concern to this study). 50 The pilot also assisted me with familiarizing myself with user interactions with the simulated-SERP. For example, participants would sometimes open links in new tabs or in separate side-by-side windows. Originally, I thought of controlling for this behavior, but then decided that it would not greatly affect my analysis. A more determining factor that the pilot assisted me with was choosing which demographics the study will focus on: The pilot contained 6 participants who did not have a university degree (prior research suggested that education in digital literacy may affect information behavior (Hango, 2014)). Despite the small sample-size, I was concerned that education demographics affected the likelihood of whether certain document-genres would be interacted with. Therefore, I decided to control for this factor by limiting my demographic to students with a reasonably high level of digital literacy until a future study. The rationale was that if I find framing affects students with a high level of digital literacy then this would be an argument for re-evaluating how digital literacy is taught. Conversely, if framing effects were observed in a population with low digital literacy then it would raise the question of whether the students simply need a standard level of digital literacy lessons. This argument is elaborated on in the Recruitment Section 3.3.1. Another one of the major changes to the task was the realization that my prompts (i.e., iteratively prompting participants to explain what they are thinking (Ericsson & Simon, 1993)) during the think-aloud were rarely required. Ericsson and Simon’s (1993) think-aloud protocol suggests that prompts may be required at times, but are less required when the stages to complete a task occur in familiar step-wise intervals of action (e.g., as opposed to longitudinal moments of meditation). The experiment serves as an example of a familiar work task step-wise intervals: Participants were familiar with iteratively reading search results, scrolling for sites, clicking sites, skimming 51 for information within the site, and noting that information. Participants could think aloud during these actions and provide context for their rationale behind these actions without me prompting them. Finally, the pilot allowed for a rubric to be further developed to analyze the writing outputs [Table 3]. It was noticed that while participants in both groups included points that relate to health, it seemed the participants with the health-frame task made an additional effort of making points related to health in their introduction, body, and/or conclusion. At times, these participants would focus most of their writing output on health and even search for multiple health-related themes (i.e., mental and physical). To distinguish and analyze the content of these writing outputs more, the rubric was developed based on the magnitude of health information being incorporated [Table 3]. 3.2 Adjustments for a global pandemic Originally, my study’s proposal was set to occur face-to-face with participants inside a lab. However, before the experiment could take place, the setting was disrupted due to the social-distancing regulations set by the government of British Columbia to address the COVID-19 pandemic. My study was conducted during the height of the pandemic risks for COVID-19, specifically summer and autumn of 2020. A few of the notable adjustments included: 1) I had to work remotely with participants and could not collect as much environmental data (e.g., how loud a participant’s background is), 2) Participants had to also be isolated enough to not have to wear a mask around others and not obstruct facial observations, 3) Data collection from the participants’ computer and camera was remote, 4) Extra computer hardware was required to collect, store, and backup data, and 5) New software was required to stream, record, and collect information from participants. 52 Therefore, Zoom was chosen as the video call application based on the recommendation of the UBC administration. The observations by the researcher were made via a Zoom call, where both participants' screens and facial expressions were captured. The new setting for the study was the participant’s personal rooms, computers, keyboards, and mice. Two of the unforeseen advantages of using Zoom were that the participant’s webcam could be used to record facial expressions up front (instead of from a separate camera) and the utterances by participants were not only recorded by Zoom but automatically transcribed as a backup. 3.3 Participant Task A time and date for the study to take place on Zoom were set for each participant who consented to the research [Consent form in Appendix 1]. Participants were informed that should they feel uncomfortable with the task, and that they could stop participating at any time. A script explaining the study and its objectives was read to the participants [Appendix 2]. The participant’s permission to begin recording the study was then requested. When permission was granted, a Zoom call began to record the students’ actions as they answered a brief questionnaire about their demographic, prior digital literacy skills, and their level of confidence in their digital literacy skills. Participants were asked two questions about their prior knowledge of a scientific topic. Students were then assigned to one of two conditions randomly (via coin-flip) until twenty participants filled each group. The participants were then provided with the task for their condition and asked to open two tabs or windows for two different URLs. The first URL is linked to a webform’s textbox for students to write their notes and their final answers to the task questions. The second URL linked to the simulated-search engine to be used (students were not told the SERP results would be manipulated on the simulated page). They were then asked to think-aloud while they attempted to answer the question by synthesizing what they learned. Participants were provided with a simulated Google page for searching and answering, and a 53 textbox to jot down notes on the main points they wished to share and submit their final answer. After writing a query in the simulated Google query box, a simulated-Google SERP would load 10 predetermined search results linking to live websites that could be clicked-through and investigated. At any point, students could jot down notes and then submit their final answer to the task question. Students were reminded that the task would take about 20 minutes, but participants were welcome to take extra time if needed. Participants were also informed that there were no right or wrong answers and may simply provide their opinion about the science topic. Students were asked to write their findings as if they would be shared with group members working on a class assignment on the topic. Students were instructed to write about their research findings using the simulated Google page and to then answer the task question. The task was to share both their notes on the main points of their findings and their final answer for the group. After completing the task, students were asked three more questions in an interview about their ISP. Participants were debriefed on how the study was set up and that the search engine that they used was a simulated-Google SERP that I created and contained results that I pre-determined. Participants were also informed that their answers may help design future search page interfaces, such as the Google search engine. Finally, participants were asked if they had any further questions. Immediately after any questions, students were sent a gift card for ice cream as compensation. 54 Figure 4 Screenshot of the webpage participants used to enter main points and submit final answers. Note that the textbox can be expanded to the participants’ desired size. 3.3.1 Recruitment When the pilot study portion recruited 15 participants, it was with people with a diverse range of educational achievements that might affect the quality of answers. However, I chose to control for the diversity of the educational achievements. According to Canada’s Statistical Agency, inequalities in the literacy of people living in Canada is a long-standing issue that requires greater attention (Hango, 2014). However, for the purpose of my study’s research questions, participants with less variability in their literacy were sought so that the experiment could attribute variances between two different group conditions to the intervening variable over demographics. A Canadian university degree is a greater determiner of literacy skills than gender, race, or place of birth: less than 21% of people with a Canadian university degree (regardless of where a person is born) have low literacy and numeracy skills compared to Canadians without a degree (88%) (Hango, 2014). This is largely attributed to the selectivity of the university (Hango, 2014). In addition, individuals in highly selective schools have less variance in their information skills (Delacre et al., 2017). UBC is a highly selective university (the mean for direct acceptance is 90% (Mukherjee-Reed & Szeri, 2019)). Undergraduate 55 students with similar levels of domain-knowledge, information literacy education, and confidence in their information literacy were sought from UBC to study how frames affected search as learning and information literacy. I briefly provide three ways that findings from the chosen demographic can be generalized. The chosen demographic is generalizable under Baskerville and Lee’s (1999; 2003) definition of generalizability for participant interactions with information systems. In addition, the authors demonstrate how “increasing sample size” does not lead to generalizability (Baskerville & Lee, 1999). In addition, Baskerville and Lee (1999) note: “It is incorrect and even harmful that many information systems researchers typically criticize their own intensive (qualitative, interpretive, critical, and case) research as lacking ‘generalizability.’” The authors provide different ways to generalize a finding with a demographic, one of which includes looking for falsifiable variables in a theory – especially conflicts between empirical evidence and theory (Lee & Baskerville, 2003). As a brief example, a claim such as “all people click the top result in a SERP” need only to be refuted by a single person who does not (i.e., regardless of their demographic, the word “all” can be refuted by a single person). A second way this experiment can generalize the findings with its chosen demographic is by showing instances when students with generally high critical-thinking skills will exhibit low critical-thinking skills. A third way this demographic can be generalized is how “even very narrowly defined study samples may provide widely generalizable results if conducted with an eye to rigorous internal validity” (Kukull & Ganguli, 2012). For example, a 1954 article by Doll and Hill excluded demographics, such as young people and women, to focus only on how old White male doctors who smoked were prone to lung cancer. This result proved to be very generalizable, but it relied on narrowing the 56 demographics from other variables that might confound the power of their findings (e.g., young people were less likely to report lung cancer) to increase the internal validity of cause and effect (Kukull & Ganguli, 2012). Generalizability requires demonstrating the impact of the effect. For example, Doll and Hill's (1954) narrow sample size of old White men enabled a high “power” (i.e., not just statistical significance, but the magnitude of the effect (Sullivan & Feinn, 2012)) linking cigarettes and lung cancer (Kukull & Ganguli, 2012). Using an online calculator (Soper, 2018), I determined the estimated sample size needed to experiment with two independent groups receiving a single treatment. The inputted values are: effect size (f2) = 0.95 (medium), power = 0.80, and alpha = 0.05. Based on these parameters, the minimum required sample size was N = 38. In order to allow for participant attrition and potential data issues, I recruited 40 people. The recruitment strategy used the UBC student listservs to post requests for participants. Information flyers at local points of interest such as public libraries were also posted. Due to the pandemic causing many students to work-from-home, UBC communities found on the social networks, Reddit, Facebook, and Discord were also used. Both online and offline posts informed students that they would be searching on the Internet for 30 minutes with an ice cream gift card from a popular ice cream shop offered as compensation. Students who contacted the researcher were first asked to read an email that explained the study’s general requirements and expectations in greater detail. The email also informed students about their rights, such as their right to leave the study if they feel uncomfortable. If students were still interested in participating 57 in the study, they were asked to consent in writing. 3.3.2 Data Collection The measures for the data are derived and operationalized from the recommendation by Urgo and Arguello (2022b), including their work with Capra (2020), mentioned in the Literature Review. The data collected during the experiment include: 1) The utterances from students as they think aloud and interactions with the search engine that is collected by the video recordings. 2) the final summaries created by the students are collected by a web application, and 3) a post-task interview is also collected via video. The operationalized measures are presented in Table 2. Two different techniques were used to gather the data. The screen-capture video feature from Zoom collected the interactions between the student and the computer’s interface, such as their search queries, scrolling, highlighted texts, visited websites, and clicks within websites. Screen and think-aloud recordings were also used by Urgo, Arguello, and Capra (2020) during their task study on SAL. The participant’s webcam also recorded the student’s facial expressions. An advantage of video is the ability to gather a large stream of data for both quantitative and qualitative analysis (Roschelle, 2000). Along with the captured video, the final task outputs by students were collected for textual analysis. The trace data included the queries and sources visited. Finally, the interview conducted with the participants were also recorded. Videos were transcribed for every word, pause, and filler words (e.g., Um, Uh, Hmm… etc.) to determine utterances (i.e., an utterance occurs when pauses are made (MacGregor et al., 2010; Traum & Heeman, 1997)). On a second pass of the 58 videos, I added notes on the search actions including: Time-on-task, scrolling behavior (such as how far a participant scrolled down a page, sites-clicked, time-on-site, and information used for writing outputs -- including direct quotes by copy/paste or by memory. In addition, noteworthy behaviors were also documented, such as when participants highlighted text to read a SERP or used a split-screen. The measures and variables are organized in Table 2. The collected data was then stored on a dedicated hard drive and backed-up on a second hard drive. Each participant had a respective folder that stored their videos, transcripts, outputs, and my notes. Measured Variables Data Collection Instrument Measures Demographics Entry Questionnaire • Preferred Pronouns (only if they were comfortable sharing) • Discipline (i.e., STEM or Arts) • Residency (i.e., Domestic or International) • Prior training in digital literacy (i.e., yes or no) • Self-reported level of confidence in search (i.e., Multiple-choice: Very low, low, average, high, or very high). Search Behaviors Video capture using Zoom • Queries typed (terms and number of terms) • Task completion time (stopwatch) • Click-throughs • Number of search results pages viewed • Time spent viewing search results pages (stopwatch) • Scrolling behavior • Highlights made to SERP or website text on the screen (screen capture) • Task/Search Outcomes Video capture using Zoom and web form submissions • Sites-used (information copied/pasted or paraphrased in the final answer) • Terms used in notes and final answer made in the textbox • Rubric to quantify the prevalence of terms Search Decision-Making and Sense-Making Video capture using Zoom and researcher’s notes • The above search behaviors and outcomes are qualitatively triangulated with the utterances in the concurrent think-aloud and retrospective think-aloud into themes. Table 2 Relationship between the measures and the instruments in the dissertation 59 3.4 Data Analysis Analyses were conducted after all the experimental information was collected. The independent variable is the task frame, while the dependent variables were search behaviors [Table 2]. Analyses included descriptive data and statistical tests (i.e., Chi-square and t-tests) for the quantitative data and a content analysis of the qualitative data. RQ1: Data from the query-constructions, SERP usage, and task responses were analyzed to make inferences on framing. The objective of the analysis was to compare how the task’s frame affected the participants by comparing the HF and NHFP groups. For RQ1, the following data were collected and analyzed for selection behavior: 1. Observations on query-constructions – using both qualitative observations on the type of terms to count their frequency for quantitative analysis according to methods by Chapman et al. (2013). 2. Time spent on query-construction. 3. Number of keywords used in the query. 4. Click-throughs (i.e., which sites were clicked and visited) 5. Time-on-site (i.e., how long participants spent on sites) RQ2: Site-use between participants in each condition was then analyzed and compared. To determine site-use, a content analysis of the task outputs was used to see what information from the sources also appeared in the task output (i.e., the written summaries completed by each participant during the search session). In addition, the frequency of terms in the summaries was also examined. Finally, the addition of a scoring rubric [Table 3] alongside content analysis provided a secondary way of assessing online learning outcomes (Bali & Ramadan, 2007): Site-use measures include: 60 1. Breadth and depth of search (i.e., whether participants decided to click the search results iteratively or if they scanned the entire SERP first). 2. Site-use depth (i.e., how far down the site did participants go to use information for their task output, not just click-through) 3. Sites used in the writing outputs 4. Word frequency of the write outputs 5. The participants' task outputs were analyzed via a rubric [Table 3]. In addition, the participants’ responses to the questions were analyzed to see how well the students support their answers in the two conditions (with or without a health-frame). Part of the objective of this examination was to determine the magnitude of the effect the health frame had on participants. For example, did the health-framed task influence people to include merely a sentence about health or did they frame their written output to focus on frames?). Therefore, I used the rubric to determine how widespread the use of health terms emerged, whereby the rubric awarded points for when health terms were used in the introduction, body, and conclusion. In addition, the rubric awarded points on whether multiple points about health were made. The rubric was developed with the pilot to detect the level of influence of health results in the writing output. Rubric Scores per Criteria Introduction includes a point about Health Notes include a point about Health Includes multiple notes about Health 50% or more of the output phrases are about Health Conclusion includes a point about Health Table 3 Rubric for detecting the level of influence of health results in writing output 61 RQ3: The think-aloud was analyzed and compared to see how frames affected the participants’ synthesis of the SERP information. RQ3 builds on RQ1 and RQ2 to understand how people make sense of different competing frames (i.e., subject, medium, environment, and task frame) qualitatively. After analyzing the query-constructions, SERP usage, and task responses, the think-aloud was analyzed to see if participants referenced the different competing frames (either directly or indirectly). Also of interest was if participants recognized the frames were competing and tried to reconcile the competition. Between the recognizing and reconciling of the competing frames, participants may use both deductive and inductive logic. I was also interested in whether and how participants used abductive reasoning. Abductive reasoning is the pragmatic approach of recognizing multiple perspectives on prior data that requires a hypothesis to acquire novel data (Peirce, 1906). Seeking new information and creating hypotheses are qualities of abductive reasoning. However, the process by which this occurs is not well understood. For example, did abductive logic lead the participants to strategize a way to find potentially useful information, and how? By triangulating the think-alouds with the search moves, I identified themes on how people approach frames that I qualify as abductive reasoning. Finally, another scenario of interest was if the competing frames caused participants to change their minds. One of the post-interview questions was whether participants felt a SERP result changed their minds on the topic and how it came about [Appendix 2 – Questions 3]. The think-aloud analysis consisted of analyzing the following information: 1. Qualitative content analysis of utterances for major themes. This analysis included a content analysis. Zhang and Wildemuth (2009) provide the following method for content analysis (1) prepare the data, (2) define the unit of analysis, (3) develop categories and the 62 coding scheme, (4) test the coding scheme in a text sample, (5) code the text, (6) assess the coding’s consistency, (7) draw conclusions, and (8) report the findings 2. Observations that might lend insight into interactions of the different frames (i.e., task, subject, medium, and source frame). 3. Triangulating the concurrent and retrospective think-alouds (i.e., the post-session interview) with observed search actions. The patterns of utterance and behavior that emerge from these processes were compared with the SAL literature on information seeking and use, especially regarding SERP use during the ISP. Qualitative coding looking for clusters and patterns was validated with an intercoder reliability test (See Codebook: [Table 10]). 3.5 Ensuring Research Quality Threats to the validity of my research may emerge from different areas. This subsection explains the steps taken to ensure high-quality research data. Notably, mixed-methods are conducted to ensure that the qualitative and quantitative data can be triangulated and support each other when inferences are made. In addition, the validity of when a synthesis is determined to be ‘framed’ needs to be defined clearly. To address this threat, my study used a 6-point rubric that awarded a point for each sign of framing [Table 3]. The rubric was tested and validated using pilot data. Another threat to validity occurs when identifying common themes that emerge from the think-aloud. Therefore, to address this threat, an intercoder reliability for categorizing the think-aloud utterances was conducted with a researcher unaffiliated with this study. Training and a codebook on how to categorize the utterances were provided to the second coder (Presented in Chapter 4 63 [Table 10]). A strong Cohen’s Kappa with a second coder not connected to the work was established before I proceeded to make inferences about the data (See Section 4.3.1 for Cohen’s kappa). To help me to determine the correct statistical tests, the statistical software Laerd Statistics was used (Laerd.com, 2023). Certain assumptions required a Chi-Square test and other assumptions required an unpaired Student t-test. For the following dependent variables (i.e., breadth-first vs. depth-first searches, frequency of terms used in query-constructions, the frequency of terms used in writing-output, and framing of the writing-output) the following two assumptions were made that justified the Chi-Square test: 1. The variable studied is nominal (i.e., the text of the query and writing output). 2. There are two or more nominal groups (i.e., comparing 2 or more word frequencies). For the following dependent variables (i.e., click-throughs, time-on-site, site-usage, and depth of sites-use, time-on-query, output word-count, and rubric scores), the following six assumptions were made, which justified using the unpaired student t-test: 1. The dependent variable was either interval, continuous, or ordinal, but not categorical. 2. There was only one independent variable, which was the framing condition (i.e., the health-framed task), and two independent groups (i.e., groups either had the health-framed task or did not). 3. There is an independence of observations, whereby there is no relationship between the observations in each group (i.e., there are different participants for each group that do not influence each other). A t-test was used to determine if the intervening variable (i.e., framing) affected two groups differently in a randomized and controlled experiment 64 (Delacre et al., 2017; Mishra et al., 2019). A one-sample t-test can be used with any sample size (Mishra et al., 2019). 4. The two groups are similar enough that there are no significant outliers that will affect the task (e.g., a Ph.D. student whose dissertation is on the health effects of cellphones). A t-test was used to compare the difference in means for both groups. 5. The dependent variables used for the t-tests (i.e. click-throughs, time-on-site, site-usage, and depth of sites-used) were assumed to be “approximately normally distributed” for each group. “Approximately normal” means that the t-test only requires “approximately normal data because it is quite’ robust’ to violations of normality, meaning that this assumption can be a little violated and still provide valid results” (Laerd.com, 2023). Given the similar academic credentials of the students, there was little reason to believe that one group would skew towards faster or slower searchers and both would be approximately normal. 6. There is assumed to be a “homogeneity of variances” between the two groups in terms of academic abilities and search skills. As will be shown in the next chapter, most students also rank their digital literacy skills as above average, which will further justify the use of the student t-test. 3.6 Chapter Summary This chapter presented my dissertation’s study design and methods used to answer the research question: How does task framing affect the information-seeking processes of students working to solve open-ended information problems? Answering the research question was conducted by a mixed-methods methodology because it triangulates the quantitative data with qualitative data to make inferences about the participants’ cognition and its limitations (Borlund & Schneider, 65 2010). The mixed-methods included a study where students were provided with a search task, a questionnaire, and an interview. The search task data were recorded by computer trace data and by webcam using Zoom. These methods were then analyzed by comparing the students’ think-aloud based on the health-frame and non-health frame conditions. The framing condition was designed to observe how participants recognize competing frames. A qualitative analysis of the summaries produced by the participants was also conducted. Ultimately, my study was designed to gather evidence and analyze the cognitive processes of individual students using search engines to tackle open-ended problems. The main analysis sought to understand: 1) the extent to which, if at all, task framing affected the student’s process of inquiry, as captured during task performance and think-aloud protocols; 2) the extent to which the effects of framing were represented in the task-output (i.e., the post-search writing or summary writing), and 3) understanding how a task frame affects the way information sources are perceived, reasoned through, and used. 66 4 Findings Overview: The following chapter presents the findings to address the three-research question: RQ1: How does task framing affect the information selection processes of students working on open-ended information problems? RQ2: How does task framing affect the information use of students working on open-ended information problems? RQ3: How do task, medium, and source frames interact in the process of making sense of open-ended information problems? To investigate these questions, this chapter provides the findings from the conducted experiment for each RQ in three parts: Research Question Variable of Interest Statistical Test Inference RQ1: Information selection & sense making Query-constructions, Click-through Chi-Square Test Frame effects on planning RQ2: Information Use Sites used & task outputs Student T-Test Frame effects on usage RQ3: Cognition & Synthesis Think-Aloud during ISP Interviews Cohen-Kappa Reconciling with different frames Table 4 The associated concepts with each of the research questions 67 As explained in the Methods section, I use a sequential explanatory strategy so that after testing for the frequency of search behaviors (e.g., clicks used and the frequency of query terms) with a quantitative analysis, a qualitative analysis provides context on the themes that occurred during this experiment (e.g., what participants uttered during their think-aloud) (Creswell, 2014). Demographics: The experiment recruited 40 participants. The participants’ ages ranged from 18-29 years old. Thirty-six of 40 (90%) participants ranked their search skills as high or very high. Similarly, 36 of the participants reported receiving prior training in digital literacy within an academic setting. None of the participants ranked their search skills as below average. 19 participants’ preferred pronoun was “her,” 18 of the participants identified as “him,” and three participants identified as “they.” 24 of the students studied in STEM fields and 16 studied in the Arts. 26 of the students were domestic (i.e., held Canadian citizenship or permanent resident status) and 14 were international. This information is summarized in Table 5: Age Range 18-29 Preferred Pronouns Her: 47.5% Him: 45% They: 7.5% Discipline STEM (Science, Engineering, Math, and Computer Science): 60 Arts: 40% Residency Domestic: 65% International: 35% Prior Training in Digital Literacy? Yes: 90% No: 10% (Self-reported) Confidence in Search Skills Very High: 27.5% High: 60% Average: 12.5% Low: 0 Very Low: 0 Table 5 Demographics of Participants 68 Section 4.1 presents finding on the query-constructions (time-on-query, number of keywords, word frequencies, and content analysis), click-throughs, and the time-on-site. Section 4.2 presents the SERP interactions (sites-used and depth of site-use) and the task responses (rubric, word frequencies, and content analysis). Section 4.3 presents findings on the think-aloud analysis for major themes (including Cohen’s kappa) and the triangulation of the concurrent and retrospective think-alouds. 4.1 Information Selection & Query-Constructions RQ1: How does task framing affect the information selection processes of students working on open-ended information problems? This sub-section will show the data collection for how the participants within each condition selected information. Specifically, the query-constructions (i.e., the frequency of terms), which site participants spent time on, the time participants spent making query-constructions, the number of keywords chosen, the click-throughs, and whether participants made breadth-first or depth-first searches are all measured. Students were told the task would take approximately 20 minutes. 16 students stayed within the time-frame of 13-20 minutes, but 4 students used above 20 minutes. There was no significant difference between the two group conditions on the time spent on the task. The average time on task for students in the Health-framed task (HF) group was 17:34 or 1054 seconds (SD = 362) and 18:03 or 1083 seconds (SD=476) for the Non-Health-framed task (NF) group. Similarly, the time spent on query-constructions also did not show a significant difference between the two groups. The average time constructing a query for students in the HF group was 84 seconds (SD = 9) and 79 seconds (SD=11) for the NHFP group. 69 4.1.1 Observations on Query-Constructions: A starting point for analyzing a query involves looking at the frequency of words used and removing the stop words (e.g., and, but, it… etc.) (Chapman et al., 2013). Using NVivo's Word Frequency calculator, the most frequent terms for both groups of participants are compared. Project stop words, such as “And,” “but” and “or,” were excluded from the word frequency (listed in Table 6). As Table 6 shows, all participants included the word “cellphones” (or “phone”) in their query-construct. The next most frequently used terms by both groups were based on the education institution (including “post-secondary,” “classrooms,” “university,” “universities,” “campus,” and “UBC”). However, the main difference between the two groups was whether or not health-related words were used in the query. 16 of the participants in the HF group used the terms “Health” or “Safety” in their query while none of the participants in the other group did so [Table 6]. Therefore, the framing of the task influenced participants to construct their queries. A Chi-Square test was conducted and the difference between the two groups was statistically significant, χ2 (1,40) =6, p=0.01. Query terms Health-Framed Group Non-Health-Framed Group “Cellphone” or “Phone” 20/20 20/20 “Post-Secondary”, “Campus”, “Classroom”, “University,” “Universities,” or “UBC” 10/20 14/20 “Health” or “Safety” 16/20 0/20 Table 6 Comparison of Health Terms Mentioned in Queries 70 If the queries were made in Google – as opposed to the simulated search page in my experiment – the query alone would cause a Search Engine’s algorithm to retrieve results that were further framed by the inclusion of the keyword “health.” However, in this experiment, the query had no such effect on the results retrieved as both groups were provided with identical SERP pages. Table 6 shows how the task frame may have influenced how participants framed their queries. From the analysis, two observations are made that apply to both groups that will be presented in the proceeding sections [4.1.2 & 4.1.3]: 1) participants sometimes used queries to evoke inspiration on what to write, rather than searching directly for information; 2) while 6 participants wrote out full questions, 34 used fragmented phrases. Within this latter group, the participants did not truncate their queries to save time but used the keywords as ‘hubs’ connecting their thoughts to the keywords found in the SERP. These findings will be elaborated on further in the chapter. 4.1.2 Number of Keywords Used in Query: The average query was 6 words in length (M=6, SD=2) for both the HF and the NHFP groups, and therefore no statistically significant differences were found. Six participants in both groups wrote full-sentence questions as their queries. Examples for the NH group included: Participant-31’s query was “what are the advantages and disadvantages of cellphones” and Participant-34’s query was “should universities restrict the use of cellphones on campus”. An example of a query in question format from the HF group is Participant-27’s query: “does the use of cellphones affect the health and safety of students ubc.” Participant-25’s query also asked, “do cellphones effect [sic] health”. However, 34 participants did not write their answers as complete sentences. These latter participants wrote in free text that focused only on the keywords. For example, 71 Participant-36 in the HF group made the following short query: “effects of cellphones on health” (5 words). As another example, Participant-38 wrote a longer query that included the location of interest: “student cellphone use at ubc policy health and safety” (9 words). An example of a short query in the NHFP group is Participant-33 who wrote: “cellphones downside at university” (4 words). A longer query by Participants-21 is “cellphones pros and cons in public places” (7 words). These truncated queries were more than just shortcuts to acquiring information. Utterances in the think-aloud demonstrate that participants sometimes spent time considering which keywords to use and characterized them more as hubs in a network connecting them to keywords provided by the SERP. For example, Participant-14 spent time thinking before her query about which keywords would link her to similar opinions by students than keywords that would link to the opinions of university administrators. The expectation for a network of keywords was expected even when the SERP did not present a network that responded or reflected to the query’s keywords (due to the simulated-SERP presenting fixed results). As Participant-38 explained during her concurrent think-aloud, “I’m not really concerned with asking a full sentence on Google because I want a lot of hits on the keywords I’m using.” Participant-38 continued to search for the keywords after their query. For example, Participant-38 entered her keywords “student cellphone uses at ubc policy health and safety”. She then saw the keywords were not mentioned in Google’s Answer Box. Immediately after, Participant-38 pressed CTRL-F (to activate her browser’s “Find” feature) and searched again for any highlighted keywords on the SERP itself. 72 While the framing of the task influenced participants’ queries, the next sections will investigate whether this influence affected how participants interacted with the actual search page. On the other hand, Participant-25 was less careful with her keywords when she wrote “do cellphones effect [sic] health”. The query contained a grammatical error by misspelling ‘affect’. During the think-aloud, she said she was aware she made a grammatical mistake but that it would not matter to Google. Participant-25 thought Google will understand which keywords were important and not be affected by extra keywords or her incorrectly spelled words. Participant-25’s query is less than half (4 words) of Participant-38’s query (9 words). Participant-38 was looking for exact factual information while Participant-25 was looking for more broad information for inspiration. One of the limitations of the query process was that when participants re-queried they were provided the fixed SERP page again. Seven of the participants re-constructed their queries (four from the HF group and three from NHFP). Five of the participants (two from HF and 3 from NHFP) restructured their queries to retrieve UBC results. After the seven participants re-queried they found the same SERP results, five of the seven participants continued their search. The other two of the participants mentioned that the SERP did not seem to be responding and were prompted by the researcher to continue with their search. 73 4.1.3 Observations on Clicking Behavior: Figure 5 Comparison of total clicks (i.e., the aggregate of sites clicked by each group). The moving average is represented by the dashed line. As the above figure shows, clicks on the first three websites were relatively high for both groups of participants. The moving average (marked by the dashed line in the above figure) tends to decline per site for both groups. This behavior was expected as it was previously observed by Novin & Meyers (2017) amongst other studies ((Andersson, 2017; Bar-Ilan et al., 2007; Cano-Orón, 2019)) that participant clicks gravitate towards the top results of a SERP. This finding also demonstrates that participants in both groups did not limit themselves to the top three results at 74 the top of the SERP that were immediately presented on screen, but would also take the extra step to scroll to the seven bottom results off screen. Due to the fixed document-order of the SERP, all participants would first perceive the top three SERP results which were more focused on academic topics than health information before having to scroll down to find sources that focus on health information. Therefore, the participants in the NHFP group had the option to not engage with the bottom health-related results while the HF group had to first view the academic-focused sources (and then choose whether to click or interact with the sources) before scrolling down. However, by the fifth to eighth site, there is an increase in clicks by the HF group that raises the moving average when compared to the NH group. The main reason for this difference in trends is due to the health-focused sites residing precisely after the fifth SERP result. The ninth site was a health site with a more academic document-genre and received only two clicks from the NH group. By the last site, more than twice as many participants are clicking on the website in the HF group than in the NHFP group. The mean of total clicks by Health participants (M=10.6, SD=3.1) was slightly higher than the NHFP participants (M=8.6, SD=3.66), but there was no statistically significant difference, t(1.8)=38, p = 0.07. 4.1.4 Analysis and Limitations of Click-Through Data While Figure 5 illustrates how the task frame can affect SERP interactions, click-throughs do not tell the whole story. Click-throughs indicate whether sites are opened. Unfortunately, this definition excludes much of the nuance that comes with selecting sites. For example, the first site in Figure 5 has more clicks by the HF group than the NHFP group. However, as will be demonstrated in the following figures, the first site was used more by participants in the NF group because they used its information directly from Google’s Answer box. Conversely, the HF 75 group clicked on the first results more often to search within the website itself for any health-related information that did not appear in the Google Answer box. In comparison to Figure 4’s aggregate of sites clicked, Figure 8 shows an aggregate of sites used by each group. The difference between the two figures demonstrates that while some participants in the NHFP group did not click on Health search results, they still used the information via the SERP’s snippet. Users may be satisfied enough by a SERP’s thumbnail, title, and snippet to never click on a link (i.e., a behavior Li et al. (2009) call “good abandonments”). However, even the categorization of good abandonment does not capture the various reasons why a participant may not click on a link yet still find it useful. As mentioned in Section 4.1.1, participants sometimes construct queries to elicit responses from the SERP that they can use for inspiration rather than factual information. For example, Participants-3 used a site’s snippet for their task output, but never bothered clicking through to the site. Their stated reason for doing so was not due to satisfaction with the information. Instead, Participant-3 saw the word “cheating” in the SERP’s snippet and thought of related points (from her past experiences) to cellphones in the classroom being used for academic performance. Thus, a more correct description would be that Participant-3 used the title and snippet as inspiration to create points for their task output without actually using information directly. In certain cases of good abandonment, participants could still technically click a SERP result, but only to highlight and read the snippet – not click through. Thus, clicking itself can also take place without counting as a click-through. For example, Figure 6 shows how Participant-26 clicked, highlighted, copied, and pasted information from the SERP’s answer box without ever 76 clicking on the actual website. Figure 6 Participants would click, highlight, copy, and paste information from SERP snippets without clicking through to the website. Clicks on websites are not always related to relevancy. For example, they can be used to remove ambiguity from the SERP’s snippet. In one situation, Paricipant-16 clicked on a non-health-related website while she was looking for health-related information because, as she noted herself, the language of the snippet was ambiguous enough that it could be related. In other cases, participants would sometimes click through to a website, but only to confirm the information they already acquired from the initial snippet. For example, Participant-9 clicked on the non-academic site with the URL pointing to the university McGill.ca only to verify that the document-genre was not intended for an academic audience. On the other hand, Participant-9 performed the same action as the previous Participant-5, but only because they misjudged the document-genre as academic (again, due to the URL’s association with McGill University) – only to find out the information was not in the desired academic genre. More rarely, participants would click the site to open it in a tab, but then forget about visiting the tab. This would mean that a site was both clicked through and left open for a long time but never used, which raises the issues that are covered in the next sub-section about defining a user’s time-on-site. 77 Given the nature of the task and the accessibility of the results, participants were generally not motivated to search much deeper within websites. While participants could click through the live websites, they rarely did so because scanning the first web page sufficed to answer the task question. However, I observed four participants click-through hyperlinked websites with different purposes: 1) Participant-32 did so to read more about the author’s biography. 2) Participant-9 verified if the fifth result from McGill.ca was in fact from McGill.ca. 3) Participant-22 intended to find more context on a point they did not originally consider. 4) Participant-16 clicked on the website’s own search function. This latter participant primarily used the search function to determine if there was information that related to Canada, but could not find any such information. 4.1.5 Observations on Time-On-Site: In total, the participants in the NHFP task spent slightly less time-on-sites (M=141, SD=73 seconds) than the participants in the HF task (M=161, SD=57). However, this difference was not statistically different t(38) =0.69, p=0.5). Looking at only the health results (past the sixth result), the participants in the NHFP task spent much less time (M=101, SD=48) than the HF participants (M=210, SD=24). The difference is statistically significant, t(38)=47, p=0.002). Figure 7 shows the time spent on each website for both groups. The information is presented in a box chart that includes the minimum and maximum times users spent on each website. The moving average of the time-on-site for both groups is represented by different colored lines. Comparing the lines, we see that the NHFP group spent more time on average on the first four websites. However, after the fifth site, the HF group begins to spend more time on each website than the NHFP group. The HF group’s higher activity after the fifth website is parallel to their 78 clicking behavior [Figure 5] whereby they clicked on more websites past the fifth website than the NHFP group. Figure 7 Box chart of average time spent on site with minimum and maximum ranges and the average 4.1.6 Analysis and Limitations of Time-On-Site Data While the breadth-first (i.e., skimming all the SERP results before clicking (Klöckner et al., 2004)) vs. depth-first (i.e., iteratively clicking and reading individual links (Klöckner et al., 2004)) search strategy dichotomy is meaningful in computer science system designs (e.g., a robot or web-crawler), the dichotomy is less observant when humans search. Participants switched between searching between breadth and depth far too often to categorize their behavior as one or another. This experiment emphasizes the term “first” in breadth-first and depth-first 79 because this heuristic can only be tracked within the first moment a participant encounters a SERP. Afterward, tracking such behavior becomes complicated when participants constantly switch between breadth and depth type searches depending on the information they encounter. Participants often skimmed the first three sites and then chose whether to keep skimming an indeterminate number of sites. In this latter case, the participant would find a thumbnail, title, snippet, or URL that “catches their eye” (a situation often reported by participants in their retrospective think-aloud (See: Sub-section 4.3.1.2)), visit the site, and then choose to either do a breadth-second or depth-second search within the site, constantly switching between these two approaches. To be more precise, I will define “first” as which action takes place within the first 10 seconds a participant encounters a SERP. As Table 7 shows, this experiment still found little difference in which approach members of each group chose to perform regardless of a more precise definition. The results were not statistically significant, χ2 (1,40) =2.5, p=0.11. More users in the HF group skimmed the SERP for results. As the think-aloud analysis will show later, the type of search used seems to be a heuristic that is more dependent on a user’s habits or preferences with searching, thus dampening what the SERP can influence. For example, some health participants still preferred to review each SERP result iteratively to see why Google recommended those results. Conversely, some HF participants wanted to do a breadth-first search out of a habit of scanning the range of results before clicking any. However, as Figure 7 shows, there was a difference between both groups’ time spent on each site. Furthermore, this difference is more apparent when comparing the Site-Use Depths (i.e., how far down the SERP participants used information instead of just clicking/viewing) [Figure 8]. 80 Breadth-First Search Depth-First Search HF Participants 12/20 8/20 NHFP Participants 7/20 13/20 Table 7 Comparison of participants who conducted breadth-first or depth-first searches within the first 10 seconds of encountering a SERP Time-on-site was measured by how long the site was actively open (i.e., the site was actively in use) as opposed to not being actively used (e.g., if the site was left open in the background while the searcher viewed other sites for an extended period). However, there are contextual variables that are not captured by time alone. A person who keeps a site open and actively reads it for 2 minutes has a different search behavior than a person who keeps a site open for 2 minutes while looking at an image for inspiration. Behavioral differences were observed by tracking the participants’ screens and observing their facial expressions and eyes. Often two participants would use a website for about the same amount of time, but one participant might scan a site and then read slowly while another might read the introduction of a site slowly and then scan for more information further down the page). NOTE: My study did not use a formal eye-tracking method, but the web camera confirmed screen-based behaviors. However, face tracking alone will still miss some of the additional nuances captured via the think-aloud data collection. The think-aloud was especially helpful in observing how participants used the websites. The think-aloud will be analyzed more closely in Section 4.3 as it addresses RQ3. Finally, the use of technological affordances affects a participant’s time-on-site and limits a scholar’s analysis. For example, Participant-34 looked at SERP results in a split-screen (i.e., side by side) with the window where they were to write their notes and final task output. The participant actively looked back and forth between the different windows. That participant’s 81 time-on-site might be similar to a participant who has the same windows in split-screen but keeps the information in their head —only using the second window as a backup to double-check what they will write down. Similarly, participants often opened tabs and kept them open for different uses. For example, Participant-4 kept one site open in the background to act as a reminder of the information (i.e., Participant-4 said they “might need” the website later), while the second tab was in the background open for the same amount of time but mostly neglected. Even more complex interactions occurred when the participant kept the information of the first site in their mind for later use but closed the tab, while the second site stayed open but was never used again. Such scenarios show the limitations of only looking at the time-on-site and clicking behavior. A more robust analysis of search behavior will emerge by observing the sites used. Section 4.3 will show that such scenarios are also explained more comprehensively with a concurrent think-aloud (as a retrospective think-aloud might miss this if the participant forgets to mention what information they kept inside their mind). 4.2 Information Use & SERP Interactions RQ2: How does task framing affect the information use of students working on open-ended information problems? The following measures for information use are collected and analyzed: site-use (i.e., which individual sites are used), site-use depth (i.e., how far down a SERP a person will use information), how much of the site’s content is used in the task output, and the frequency of terms used in the writing task output. 4.2.1 Observations on Sites Used . Usage was measured by whether the participants used information from the SERP results to write content into their respective outputs. Site-usage was identified when participants 82 copy/pasted information from a site or paraphrased the site’s information into their task output. Most participants would accompany these actions with utterances during their think-aloud as they interacted with the site (e.g., while reading a website, Participant-34 claimed “I should say something about this” and then wrote a related note in the task-output). This finding also demonstrates that participants in both groups did not limit their usage of information to the top three results at the top of the SERP that were immediately presented on screen, but would also take the extra step to scroll to the seven bottom results off screen. However, as Figure 8 shows, while the NHFP group focused mostly on the top of the SERP, the participants in the HF group included the sources from the bottom that relates to health. The line in Figure 8 shows the moving average of the number of participants using each site. The HF groups’ activity overtakes the NHFP group after the fifth site — similar to the time-on-task and click-through graphs. 83 Figure 8 Aggregate of sites used by each group for the writing output and the moving average Figure 9 shows the site-use depth, which represents how deep down the SERP list participants used information from the sites. In accordance with other SERP experiments, such as Kelly & Azzopardi (2015) and Yamamoto & Yamamoto (2020), the number of links a user moved down a SERP represented search depth. Therefore, the first link in a SERP has a search depth of 1 and the last link has a search depth of 10. Figure 9 represents participants for each group condition clustered by the depth of their site-use: The first bar on the left indicates three participants from the NHFP only used the first result to complete their answer while none of the participants from the HF did so. The bars on the furthest right indicate that six participants from the HF used 84 information while only two participants did so from the NHFP group. Figure 9 shows that 3 participants from the NHFP only used the first result to complete their answer while none of the participants from the HF group did so. All participants in the HF would at least use information from the first five SERP results. None of the participants from either group would end their site-use on the ninth website. However, six participants from the HF group used information as far down as the tenth site while only two participants did so from the NHFP group. As Figure 9 shows, there was an observable difference between the groups on the search depth of the sources used. The HF task group had to search deeper down the SERP page. The 20 participants who received the HF task searched deeper (M = 7.91, SD = 2.71) compared to the 20 participants in the NHFP group’s search depth (M = 2.90, SD = 1.64) at a statistically significant level, t(38)=7.07, p=0.0001. Therefore, the depth of the site-usage was statistical significance. NHFP Figure 9 Participants for each group condition are clustered by the depth of their site-use. 36340101020 0 0 0143406012345671 2 3 4 5 6 7 8 9 10Number of ParticipantsSite Order (First to Last)Non-Health Health 85 4.2.2 Analysis and Limitations of Site-Use Data As shown in prior figures, the difference between the two groups became more significantly pronounced when considering the usage of sources. These findings suggest that the influence of framing is more observable when the search-depth usage of websites is measured rather than other variables, such as click-throughs and site-usage alone. On the other hand, both site-use and site-use depth show whether sites are used or not, but not how frequently they are used. For example, participants sometimes copy-paste several points into a task output for one site while using another less frequently. However, measuring this frequency is complex and does not always reflect how useful the site is because that factor depends on what the site was used for. Notably, many participants used a site to copy/paste several points of information directly from the site, such as statistics. In contrast, other participants used sites to inspire several points that come to mind from past experiences. For example, Participant-31 did not have an HF task. However, when she saw a health result she exclaimed, “Oh there’s health!” and wrote down a point from prior-knowledge about health in her task output. The health site’s mere existence was enough to inspire a new avenue of thought that was not considered before. This form of usefulness is difficult to quantify. In addition, how multiple concepts from different sites are synthesized is difficult to quantify. Most importantly, participants not only noticed agreements and disagreements between different sites but would actively look for them and then synthesize these points together differently. 4.2.3 Task Responses I instructed participants to use the simulated-Google page and linked sites in the quasi-live corpus to answer the task question. I asked the participants to write their findings as they would 86 for a group project. The task output included both the participants’ notes on their findings and a final answer to the task question. Most task outputs consisted of point-form notes (7 points on average) followed by a paragraph as a final answer. Due to the note-taking nature of the task, there was some variation in the task outputs. The word count for the shortest task output was 151 words while the longest task output was 937 words (M=242, SD=32). On average, NHFP wrote 234.3 words (SD=25) while HF wrote 249.7 (SD=26). While HF outputs were slightly longer on average (15.4 words) there was no statistically significant difference in the length of the task outputs, t(38)=1.7, p=0.1. -Part of the reason the HF group wrote slightly more is due to their inclusion of more points from the sources that were not health-related than the NHFP group used health-related terms (as will be elaborated on with the rubric). A content analysis guided the data analysis of the task outputs using Zhang and Wildemuth’s (2009) suggested procedure: (1) prepare the data, (2) define the unit of analysis, (3) develop categories and the coding scheme, (4) test the coding scheme in a text sample, (5) code the text, (6) assess the coding’s consistency, (7) draw conclusions, and (8) report the findings. The software used to prepare and assist with the data analysis was NVivo 2022. The content analysis of the task outputs was conducted on both groups to determine how participants framed their responses. Each sentence was analyzed for common themes. The two main themes were health and academic performance. These themes emerged based on whether the sentence used health-related terms or academic-related terms. Examples of health-related terms include: “posture,” “cancer,” and “anxiety.” Examples of academic performance-related terms include: “cheating,” “learning,” and “distraction.” Both groups used a mix of these terms but with vastly different frequencies: While 90% of the HF group made more than a single point that related to academics, only 30% of the NFHP made more than a single 87 point about health. As the rubric will elaborate on further, the HF group made and used health-related points far more extensively than the NHFP group: 15 of the participants with the HF task dedicated 50% or more of their task outputs to health topics, while zero of the NHFP task outputs dedicated more than 50% [Table 3]. The next sub-section presents a six-level rubric (provided in Methods) to further measure differences in the task output composition. 4.2.4 Rubric and Health-Frame Analysis: Using a rubric alongside a content analysis can provide ways of assessing the online learning outcomes of both individuals and a group (Bali & Ramadan, 2007). The six-point rubric (0-5 points) compared how the notes and task outputs contain evidence of a health frame. A point was given to each participant’s task response if they: 1) Introduced the topic as one that relates to health in their task output, 2) If they included a point related to health topics in their task output, 3) If they had more than one point related to health (e.g., physical and mental), 4) If 50% or more of the points related to health, and 5) if the task outputs' conclusion mentioned health. Below, Box 1 provides an example of a task output from a participant in the HF group who was awarded 5 points using the rubric, and Box 2 provides an example of a second task output from a participant who is also from the HF group and was awarded 1 point: Main Points: -Studies have shown that cellphone overuse can have a negative impact on your posture, eyesight, and hearing, not to mention distract drivers and pedestrians. [Rubric awarded one point for referring a point to health] -Greater incidences of depression and anxiety in users. [Rubric awarded one point for including multiple points about health] -Cell phone and technology addiction manifests in one or more of the following ways: Choosing to use your device even in "dangerous or prohibited contexts; “losing interest in other activities; 88 feeling irritable or uneasy if separated from your phone; or feeling anxiety or loneliness when you use information and communication technologies.” -Four times greater chance of an accident, when mobile phones (either handheld or with a kit) are used while driving due to distraction. Children have the potential to be at greater risk than adults for developing brain cancer from cell phones. -Their nervous systems are still developing and therefore more vulnerable to factors that may cause cancer. But this means providing criteria and procedures for students that enable their learning.- and also learning when they need to put their phones down and switch off. These are skills and knowledge that are essential for life in today quite the contrary. [Rubric awarded one point because more than 50% (9 of 12 sentences) of the notes refer to health] Final Answer: Cell phones can negatively impact students physical and mental health. [Rubric awarded one point for introduction referring to health] However, banning the use of cellphones only encourages students to use them more. Prohibiting them from access to their phones during class time would not stop them when they are not in the classroom so it would not make the biggest difference. Rather, UBC should focus on teaching us the cons to cellphone use. Our generation suffers the most from mental health issues like anxiety and depression, if UBC were to teach us how our cell phones could be contributing to that, some people would surely listen. Or in the very least, they would be aware. [Rubric awarded one point for conclusion referring to health (i.e., UBC should inform students about health issues)] Box 1: participant’s task response that was awarded 5-points The following is a sample from a participant with an HF task who received only one point in the rubric for mentioning “nomophobia,” (i.e., the anxiety of not having a phone): Main Points: -Students could be distracted, or spend more time playing video games or texting friends than paying attention in class (Top Hat website) -On the flip side, cellphones are being used to teach, making lessons more engaging and interactive. Rather than banning them, some teachers are looking at ways of incorporating cell phones into their curriculum to provide a better learning environment. (Top Hat website) Benefits: - Learn more effectively - Used for academic purposes 89 - Find answers to classroom questions on Google (from Student Pulse Survey) - Access Digital textbooks - Take pictures of lecture slides (very relevant when the prof. is really fast at talking) - Profs using Menti to look at polls and stuff - Even take an exam because some people can't afford to buy a laptop or PC or something -Cellphones and other mobile devices can significantly enrich their education through active learning. Robert Shuter, founder of The Center for Intercultural New Media Research says --> Instructors and universities (boomer profs don't know how to use tech so they ban it lol) Matthew Numer at Dalhousie Uni says it's on me if my lectures are not compelling enough... should be compelling enough for students to pay attention - Students tended to perform worse on the quiz when they had their cell phone and when they scored higher on a measure of nomophobia s cell phone (, was authored by Jessica S. Mendoza, Benjamin C. Pody, Seungyeon Lee, Minsung Kim, and Ian M. McDonough) [Rubric awarded one point for referring to a point on health] Final Answer: Not surprisingly, students also found that their ability to concentrate in class and while writing papers improved (without the use of devices). I don't agree because now a lot of students use like cellphones, laptops, and other devices to take notes which could be really beneficial. Cellphones can help students communicate with their classmates, especially if they miss notes or something so it makes it really easy to do that. University is different than high school. Students pay for their education so it's really their fault if they continue to use their phone in class and distract themselves. UBC as an institution should not take control of that. Cheating on exams with using cellphones. If you're going to cheat, just don't get caught. If you get caught, that's on you. That's your fault. You're getting suspended. Plagiarisms? Just don't get caught. Box 2: Participant’s task response that was awarded 1-point Similar to Box 2, NHFP participants generally did not focus the framing of their task response on health. As a result, their scoring on the rubric had low numbers because the participants did not make many references to health-related points [Figure 10]. Most participants in this group produced task responses that provided evidence that the SERP influenced participants to frame cellphones as an academic concern. In addition, while the HF group focused most of their points 90 on health, they also used a more diverse set of points by mentioning both academic-related and health-related points. Figure 10 shows that the HF group scored higher than the NHFP group. More than 95% of the HF group mentioned health in the introduction of their task output. All participants in the HF group had health-related points in the body of their notes and 85% brought up different health-related aspects (e.g., physical issues, such as cramping, and mental health issues, such as anxiety). 75% of the HF group had 50% or more of their notes related to health topics and 85% concluded their task output by referring to health. In comparison, only 10% of the NHFP group referred to health in their introduction and conclusion. Fewer than half of the NHFP group mentioned a point about health and fewer than a third mentioned any secondary points about health. None of the NHFP group members dedicated 50% or more of their task output to health-related points. A t-test for the two independent groups is used where the NHFP group (M= 1.45, SD=1.05) scored lower on the rubric than the HF group is 3.80 (M=3.80, SD=1.15), t(38) = 6.75, p = 0.0001. 91 Figure 10 Graph of the number of participants who satisfied each of the rubric’s criteria. Rubric Scores per Criteria 1 (Introduction includes a point about Health) 2 (Body Includes a note about Health) 3 (Includes multiple notes about Health) 4 (50% or more of task output is about Health) 5 (Concludes with a point about Health). Task outputs (NHFP Group) 2/20 9/20 6/20 0/20 2/20 Task outputs (HF Group) 19/20 20/20 17/20 15/20 17/20 Table 8 Number of participants who satisfied each of the rubric’s criteria. The final score in the 6-point rubric (from 0 to 5) for each participant was then collected and the scores for both groups are then compared in Figure 11. 92 Figure 11shows both groups’ rubric scores graphed on the same axis, whereby two peaks emerge: The mean for the NHFP group is 1.45 (SD=1.05), and for the HF group is 3.80 (SD=1.15), t(38) = 6.7, p =0.0001. Figure 11 Final rubric score for each participant Rubric Final Score 0 1 2 3 4 5 NHFP Group 4 7 5 4 0 0 HF Group 0 1 2 3 8 6 Table 9 Number of participants awarded a final rubric score As Chapter 3 explained, the purpose of the rubric is to determine if a task output is framed in combination with the content analysis. A task output was considered health-framed if 3 to 5 points on the rubric were satisfied and not health-framed if only 0 to 2 points were scored. For 93 example, a task output that has the introduction, body, and conclusion all mention health-related points will be considered health-framed. Conversely, a task output that only mentions a single health-related point is not health-framed. The rubric revealed: 85% (N=17) of the HF group’s task output also had a health-frame (i.e., they scored 3-5 points on the rubric). The other 15% (N=3) in the HF group did not use a health-frame (i.e., they scored 0-2 points on the rubric). Inversely, 80% (N=16) of the NHFP group did not have a health-frame, while 20% (N=4) had a health-frame, χ2 (1, N=40) =16.94, p=0.001). Therefore, the task frame led the two groups to produce significantly different frames for their outputs. Using NVivo’s Word Frequency calculator, I compared the most frequent terms in the task output for both groups of participants. Figure 12 includes the variations and frequency of the terms. Project stop words listed extensively in Table 6 are excluded (e.g., a, the, and but). For both groups’ variations of the words “students,” “cellphones,” and “using” were the most frequently used words, respectively. A difference in word frequency emerged between both groups. The most frequent words for the participants with the HF task were variations of the word “Health” followed by “Learning,” while for the NHFP group, it is simply “Learning.” Figure 12 shows that the HF group used health-related terms more frequently in their task output (123 times in aggregate) than the NHFP group (26 times in aggregate). 94 Figure 12 Comparison of word frequencies for HF and NHFP tasks In summary, both the qualitative and quantitative analysis found that most of the participants provided with the HF task focused on health-related topics and framed their entire task output to be about health. Conversely, the NHFP group framed their task output to align with the top results on the SERP. Comparing the statistically significant differences between both groups regarding the SERP usage [Figure 8] and the task output in Table 9 suggests a task’s frame has the potential to influence a participant more than the frame caused by a search engine’s document-order. The next section looks at how this difference in framing amongst participants emerged by looking at the think-aloud. 95 4.3 Think-Aloud Analysis RQ3: How do task, medium, and source frames interact in the process of making sense of open-ended information problems? In this section, data is collected and analyzed regarding how the think-aloud references the task, medium, and source frame. Using the data from RQ1 and RQ2, this experiment collected quantitative evidence that suggests the influence of the task frame (i.e., the HF) superseded the SERP’s frame (i.e., the SERP) for most participants. RQ3 is interested in how this occurred: How did participants make sense of the interacting frames? Answering this question required a qualitative analysis of think-aloud. While RQ1 and RQ2 provided evidence that a strong task frame has the potential to dominate information selection and use patterns, the think-aloud was analyzed to understand the sense-making occurring in both conditions for RQ3. In the introduction of this dissertation I argued that when an ISP involves a person using a SERP to retrieve sources, each of these actors (i.e., person, SERP, and source) occurs with three major frames, respectively: A subject’s frame, a medium frame, and an environmental frame. These three major frames can cause the actors to frame information in different ways that lead to the creation of either a cognitive frame, SERP frame, or source frame. I questioned whether a fourth type of framing can occur via the task itself. Addressing RQ1 and R2 indicated that the task description itself can create a frame. In the sub-section below, I show how the task also occurs within a fourth major frame (alongside the subject, medium, and environment frame). Henceforth, this fourth major frame that task-frames occur within is referred to as the metaframe. Using the search actions with the concurrent and retrospective think-aloud, I triangulated the data by grouping them into one of the four major frames (i.e., the subject, medium, environment, or metaframe). 96 4.3.1 Triangulating the Concurrent and Retrospective Think-Alouds with Observed Search Actions. A content analysis of both the concurrent and retrospective think-alouds was conducted with 40 participants. Data coded that relate to their search strategy included 1551 minutes of search tasks, 40 task outputs, and 40 interviews, 5065 utterances. The triangulation of the concurrent think-aloud, retrospective think-aloud, and observation of the participant behavior resulted in 32 codes. The 32 codes were further grouped into eight themes, presented in Table 10. I consulted a second coder (an external professor unassociated with this dissertation) to establish inter-coder reliability. Table 10 was provided as a coding-table to guide the coder. The second coder found Table 10 relatively straightforward, so the training session required 7 minutes of instructions, which mostly consisted of reading the themes in Table 10. Both coders practiced the coding together for 10 more minutes to ensure the training was understood. Due to time constraints and the enormity of the 5065 utterances, the second coder was then asked to code 10% of the data independently. The statistical software, STATA, was then used to measure Cohen’s Kappa, which held almost perfect agreement (Kappa= 0.891, SE of kappa = 0.017, 95% confidence interval: From 0.857 to 0.924). Drawing on the work of Peirce (1906), the pragmatic approach of recognizing multiple perspectives on prior data requires a hypothesis to acquire novel data. I link abductive reasoning to how people managed multiple frames. As participants were seeking new information and creating hypotheses about different frames, they should demonstrate characteristics of abductive reasoning. I identified eight themes that illustrate the abductive approach to frames. The eight themes are presented in the ICOGNATE model. In brief, eight themes suggest that people 97 interacted with frames by 1) Asking direct questions about the frame, 2) Orientated themselves to acquire more perspectives about a frame, 3) Compared and contrasted different perspectives in a frame, 4) Validated their frames based on logic and probabilities, 5) Aligned frames with their own personal, social, or cultural norms, 6) Validate frames based on expertise, 7) Or pondered how to share their own frame. For the convenience of briefly referencing the eight themes as symbols, they were re-worded into names identifying each of the themes to form the acronym ICOGNATE: Inquiring, Conceptualizing, Orientating, Guiding, Numerositizing, Anthropologizing, Transferring, or Expressing [Table 10]. Each of the eight themes in ICOGNATE is provided below alongside data from the: 1) retrospective think-aloud; 2) concurrent think-aloud; and 3) observation of the participant behavior. The qualitative analysis of ICOGNATE is presented below. 4.3.1.1 Inquiring: Making an inquiry about a frame Definition of this category: When the retrospective think-aloud, concurrent think-aloud, and search behavior were triangulated, inquiry was a major theme that emerged. Inquiry is a key component of most ISPs in the Information Sciences, but the concept dates back to at least Socrates. The definition of inquiry goes beyond the act of asking questions and involves the pragmatic “process of acquiring and changing beliefs about the world” (Stalnaker, 1984). I use the term "Inquiring” to refer to when people make an inquiry about a frame. Retrospective Think-Aloud: During the Retrospective Think-Aloud, participants referred to the act of inquiry as part of their search process. As Participant-35 explained: “I just thought about what are my main questions and like how should I put this into a search engine.” Alongside asking questions, participants 98 also mentioned that they would take time to reflect before typing in a query. For example, Participant-7 mentioned: “First, I brainstormed a little bit through related topics or concerns, and from that pool of words, I picked a set of words [for the query] that would be reasonably common on sites.” Similarly, Participant-33 mentioned: “I didn't first straight go off to search, like I did write down some of my points that I had. So then after I did try to look [with my query] for more concrete evidence or more details that could add to my already outlined answer.” Participants also emphasized the importance of the keywords in both their questions and their eventual query in the SERP. As Participant 14 explained: “My first step would be picking the keywords that I think are going to be relevant. After that, if I still haven't found what I'm looking for, I'll come up with a different question and I'll search again.” Concurrent Think-Aloud: During the Concurrent Think-Aloud, inquiries were made throughout the participants’ ISP processes. The inquiry emerged in four ways related to frames: 1. Subject: Participant utterances made references to internal questions about the self (e.g.,” How do I personally feel about this topic?”). 2. Medium: Participant utterances made references to questions about the features or functions (e.g., “Why did these search results show up?”) 3. Environment: Participant utterances made references to questions about the source’s content (e.g., “I wonder what others say?”) 4. MetaFrame: Participant utterances made references to thinking about the task inquiry itself or questioned the task frames themselves (e.g., “How do I go about answering this question?”). Observed actions: These utterances often coincided with a person’s actions. Notably, this occurred when people were making a query on the SERP, using the search features on a source’s webpage, or a browser’s ‘Find’ feature. However, questions were asked throughout the search task as well. For 99 example, one participant ended their search task with an inquiry: They first asked themselves a question, then asked themselves again whether it was worth pursuing that line of inquiry, and then decided that the lack of value from the inquiry was a sufficient reason to end their task. 4.3.1.2 Conceptualizing: Conceptualizing a frame Definition of this category: When the retrospective think-aloud, concurrent think-aloud, and search behavior were triangulated, conceptualization was a major theme that emerged. I liken conceptualization to Herbert Simon’s (1998) chunking (i.e., bits of information that serve as “cues that, when recognized, evoke access to heuristic suggestions”) and a process that cognitive scientists call feature-matching that involves pattern recognition (Gallistel & King, 2009). However, I use the broader term conceptualization to emphasize the psychological reactions during this process reported by participants. As the triangulation of the participant data will show, part of chunking/feature-matching data involves how people emotionally react to it. I use the term "Conceptualizing” to refer to when people conceptualize a frame. Retrospective Think-Aloud: Participants described immediate conceptualizations or emotional reactions to different stimuli as part of their search process. Participants would describe looking for information that is “eye-catching” because not all information can be understood at once. For example, as Participant 5 explained: “[After] I searched it up, I just went through the links to see what caught my eye.” For example, an image, bolded text, or a paragraph emphasized by a colored border would invite reactions. For example, Participant-14 explained: “Once I'm on the website, I'm looking for passages and quotes.” Participants often referred to letting their eyes move to information – even statistical numbers or an insightful quote could draw a reaction. Conversely, this same process could be used to avoid information. As Participant-19 explained: “One of the first things that I 100 look out for are ads and distracting stuff if I just made a Google search. And I’m going to have to know the site [at the top of the SERP] or I’ll scroll down a couple of results.” Participants depended on these reactions to conceptualize the probability that a bit of information might be useful. Participant-12 explained that the language of a website needs to first be conceptualized before engaging with it: “I look to see if [the heading] is kind of useless. Like does it matter? And like do I know exactly what the paragraphs is talking about?” In other words, participants would psychologically react to information while suspending evaluation about its actual usefulness. As Participant-6 explained: “When it tends to catch my eye it's usually not garbage but at times it is good. So, you save it to the side [for later use].” Concurrent Think-Aloud: During the Concurrent Think-Aloud, conceptualizations were made throughout the participant’s ISP process. The conceptualizations emerged in four ways related to frames. 1. Subject: Participant utterances made references to immediate psychological reactions to a piece of text (e.g., uttering “Oh here’s a stat!” in an excited tone) 2. Medium: Reaction to SERP features (e.g., “This quote seems like it can lead to useful information later.”). 3. Environment: Source information recognized (e.g., uttering “It’s mostly ads” in a frustrated manner). 4. MetaFrame: Thinking about how the task itself is conceptualized. Reactions to task frames (e.g., uttering “This should be a simple task” in a confident tone). Observed Behavioral Actions: These utterances would occur concurrently with emotional reactions. Alternatively, participants would naturally go through the cognitive process known as feature-matching whereby they use their memory of visual stimuli to know what to focus on or avoid. On the SERP, participants would automatically focus on SERP features, such as the answer box, bolded keywords in SERP results, and the top of the SERP page. Conversely, participants would also avoid certain features, 101 such as advertisements or Google’s doodles. Similarly, on a search result’s webpage, participants would focus on titles, images, or highlighted text. They would also skim the webpages and react to certain types of text, such as statistics. Participants would often read the first few words in a paragraph to get an understanding of the content of an article. If the content sounded interesting, participants would read the paragraph in full and slowly. 4.3.1.3 Orientating: Orientation to Acquire more Perspectives about a Frame Definition of this category: I use the term orientation to refer to both the physical act of orientating oneself and the cognitive concept of figuring out ‘where’ one is in their ISP. Prior studies observing metacognition during SAL found participants “described thinking about and orienting themselves towards the task they were to perform” (Crescenzi, 2016). The triangulation of the data led to the theme of Orientation emerging as well. I use the term "Orientating” to refer to when people use orientation to acquire more perspectives about a frame. Retrospective Think-Aloud: Participants referred to various ways of orientating themselves during their information-seeking process. Participants referred to various browser features they could use, such as tabs. Participants also explained that orientation often takes the form of planning how to search to find new information. For example, when deciding on how to consume information some participants would read each article iteratively while others would skim the entire SERP page, and still others might open links in new tabs. Participant-8 explained that he would save his search strategy until after first skimming down what a SERP’s algorithm offered: “I try my own heuristic. After doing a first pass.” How participants navigated webpages they retrieved was also strategized. For example, participants would look directly for snippets, statistics, or menus that would allow them to navigate further. As Participant-14 explained: “Basically, I just repeat the step of clicking on 102 links that I think are trustworthy and finding the passage. If there's a link that's on the website that links out to another thing that's more specialized, then I would click on that.” As another example, Participant-12 said that that she takes “a long time skimming because usually how I look at pages is that I [first] look at the heading.” Concurrent Think-Aloud: Participants oriented themselves throughout the ISP process in four ways in reference to frames: 1. Subject: Prior knowledge of features is used to navigate (e.g., “I’m looking for a familiar website.”). 2. Medium: Using their notes or SERP navigation (e.g.,” I’m scrolling down to see what else Google suggests.”). 3. Environment: Source website navigation (e.g., “I’m skimming down this article.”). 4. MetaFrame: Thinking about how to orient around the task. Navigating actions based on task frame (e.g., “I’ll focus on these parts of the task first.”). Observed Behavioral Actions: These utterances would occur concurrently with many actions. For example, the affordances of a SERP invited certain Orientation actions, such as highlighting text with a cursor or opening a link in a new browser-tab. Similar actions would be observed on a search result’s webpage. Alongside orienting themselves on the SERP and webpages, participants would also orient themselves by using their task outputs to write information down or double-check what they previously wrote. Participants also used website features to orient themselves, such as navigating through an interactive infographic or data visualization. 4.3.1.4 Guiding: Comparing and Contrasting Different Frame Perspectives Definition of this category: In prior work, I found that when participants are unfamiliar with a topic they move from general references to a topic (e.g., Wikipedia) to more specific references (e.g., academic articles) (Novin & Meyers, 2017). But how does one move from the general to 103 the specific when they are unaware of a source’s document-genre? I observed in this study that participants move from the general to the specific by actively looking for a diverse set of information to compare and contrast. In some sense, this is a very similar process to what Socrates, Hegel, and Marx term as the dialectic (a thesis and anti-thesis come together to form a synthesis). However, I observed in this analysis there is an added component of people actively searching for diversity and not just a singular counterargument. In other words, people do not look for two sides of a story, but for many sides to compare and guide them. I use the term "Guiding” to refer to when people are comparing and contrasting different frame perspectives. Retrospective Think-Aloud: Participants mentioned guiding themselves to a general understanding by comparing and contrasting the information they retrieved. This strategy required first comparing their own prior knowledge to a generalization of the topic they were searching for. It then involved comparing results with each other to get a generalization. As Participant-15 explained, generalizations are sought first, but the intention is to narrow down as more information is acquired: “I start off with like the broad topic and then narrow down after getting some more information.” Participant-4 explained that his search strategy required diversifying the information retrieved and comparing them: “I get a lot of different general articles so that I could like find different information – Get various information.” Participant-6 explained that comparing diverse information was necessary “just so I understand both sides of the coin because both of them tend to be valid.” Similarly, Participant-14 explained that retrieving points and counterpoints ended a key part of her search 104 process: “I might look for evidence for both sides, and then once I found like a couple of points for both sides, I'll consider that done.” Concurrent Think-Aloud: Participants compared information throughout the ISP process to guide how they search. Guiding occurred in four ways in reference to frames: 1. Subject: Prior knowledge on a topic is compared and contrasted, such as the pros and cons of a debate (e.g., “I see both sides of the issue.”) 2. Medium: SERP results are compared and contrasted against each other (e.g., “These sites provide pros but I’m looking for the cons.”). 3. Environment: Source information is compared and contrasted with information from other sources (e.g., “This article is providing the other side of the story.”). 4. MetaFrame: Thinking about guiding. Task frames are compared and contrasted (e.g., “The task question focuses on the negatives but what about the positives?)”. Observed Behavioral Actions: Participants would look for general information, such as the information provided in a SERP’s answer box or a referential website (e.g., a government website that lists the dangers of cellphones). From here participants would try to make further comparisons (e.g., looking for a website that lists the pros and cons of a topic). Participants not only uttered that they are looking for pros and cons on a topic but would go as fa r to write a table list of “Pros” and “Cons” in their notes. Alternatively, participants would make the comparisons more directly in the sentences of their task outputs. For example, participants may write about the safety of cellphones and compare and contrast it with sources about their dangers. Participants would even use their task outputs on both sides of a topic to further guide their search. For example, after Participant-17 wrote a list comparing the “Benefits” and the “Risks” of cellphones on campus he noticed that he wrote down more points about the benefits than risks and needed more points 105 about the risks. While double-checking through his task output he also noticed that the benefits were more anecdotal compared to the risks he listed and said he should search for benefits with more supporting research. 4.3.1.5 Numerositizing: Validating Frames Using a Sense of Logic or Probability. Definition of this category: Numerical Cognition refers to a “sense of numerousness” or “quantical sense” (Dos Santos, 2022) rather than numbers. For example, people can perceive whether something is greater or lesser in quantity without having to count. This “fuzzy” type of numerousness extends to ‘fuzzy logic’ (Yamashita, 1989), whereby people can use flexible forms of ideas and logic (Zadeh, 1988). For example, people do not need to count the exact number of grains of sand to determine if something is a pile of sand, the concept of a pile has ‘fuzzy’ boundaries. As a more complex example, if someone is told there is highway construction with 30 KM/hour speed limits taking place on a short-cut they can estimate that a longer highway path with 100 KM/hour speed limits can be faster with little calculations. In addition, Cognitive scholars have argued that the brain carries out some form of computation (Gallistel & King, 2009). Computational thinking can lead to complex forms of algorithmic thinking (Katai, 2015), which itself includes logical thinking and probabilities. One of the themes that emerged from the data was that people refer to some form of computation when describing how they think while searching. I found sometimes people go beyond numerical cognition (i.e., a sense of numerousness without numbers) and have a sense of calculations and algorithms (e.g., a sense of statistical power without having to understand the calculations). I settled on using the term numerosity (i.e., the sense of a numerical difference) as a catch-all phrase for these types of utterances. I use the term "Numerositizing” to refer to when people validate frames based on a sense of logic and probabilities. 106 Retrospective Think-Aloud: Participants said that making logical choices based on probabilities was important for their search strategy. Various forms of using probabilities and logic were mentioned. One of the most relied-on strategies was how frequently information is repeated. For example, Participant-8 focused on the frequency with which a particular source was mentioned: “First of all is the repeatability of the source.” However, other forms of logic were also relied on. A few participants mentioned strategies that were akin to a Bayesian approach. For example, Participant-36 looked for the most recent statistics: “Obviously the more recent it is the more relevant it is” but at times she also used her prior knowledge to evaluate the validity of statistics: “I thought those were kind of high numbers.” Participant-6 argued he used logic for every line of information he encountered: “I basically have like three switches in my brain: It's the ‘Attack,’ it's the ‘Counter-Point,’ and then the ‘Garbage.’ And so, I break down every sentence. I'm like ‘All right, is this for me or against me regarding my stance [for the final task output]. And when I find something particularly interesting, I look back into my notes and say, ‘All right, is this part of a point that I already stated? Or is this a rebuttal to a counterpoint that I have?’” Concurrent Think-Aloud: During the Concurrent Think-Aloud, participants thought about the probabilities and logic of information throughout the ISP process in four ways in reference to frames: 1. Subject: The overall validity and soundness is considered (e.g., “I find this to be the most logical conclusion.”). 2. Medium: The logic in the SERP algorithm and computation are considered (e.g., “Google’s showing mostly websites in agreement.”). 3. Environment: The source’s probability and logic are considered (e.g., “This article backs their claims with statistics.”. 4. MetaFrame: Logic behind the task’s frames is considered (e.g., “This is the most logical way to approach the task.”). 107 Observed Behavioral Actions: Participants would make future decisions based on numerosity. For example, participants used keywords that searched for probabilities and statistics, clicked on websites providing numerical analysis, and wrote the exact numbers down in their task outputs to back up their stance. For example, Participant-16 found statistics supporting her stance on cellphone restrictions on a website. However, she explained that the study supporting the statistics seemed faulty, so she decided to avoid making the claim. Participants also wrote down simple logical statements based on the information they retrieved, such as IF-THEN statements. Participants also tried to use logic in relation to a SERP to determine the next steps in their ISP. For example, it was common for participants to reason that if cellphones are very dangerous there would be more studies establishing a link. 4.3.1.6 Anthropologizing: Aligning Frames with Personal or Cultural Norms Definition of this Category: Hofstadter (2001) famously claimed that analogies are the “core of cognition.” I build on the importance of analogies as they emerged in the ISP. I found that anecdotes and cultural norms can serve an equal purpose as numerosity when people search for information. For example, as the triangulation of the data shows, participants are more likely to make sense of new information by relating to the information via anecdotes or a cultural understanding than to create an analogy. One possible explanation for this is that relating information via recalling prior knowledge expends less cognitive effort than having to think of a new example. Stories are also a powerful way of remembering and relating arguments. During this task, participants sought anecdotes or opinions of people rather than looking for analogies. Therefore, I use the term anthropological to refer to the personal anecdotes, analogies, or cultural norms that participants used during their ISP. I use the term "Anthropologizing” to refer to when people align frames with their own personal, social, or cultural norms. 108 Retrospective Think-Aloud: Participants said that part of their search process involved aligning newly acquired information with their prior knowledge using anecdotes, analogies, or cultural norms. For example, Participant-15 began her search using her prior experience that cellphones impacted the health of herself and the people she knew: “I already had my own opinion from my own experience and from people I know. So, I wanted to search out first, like, what the dangers were and how bad is it, you know? Like is it something small?” Similarly, Participant-40 used her sense of UBC’s student culture to further her search. She argued that UBC’s administration was “overextending their regulations on student rights,” so she searched for student opinions to support this point. Using Anthropology as a search strategy was not only used to find opinions or confirm biases, but also to find information that supported those opinions. For example, Participant-14 explained that she wanted “opinion websites with useful sources included rather than just an opinion column.” Concurrent Think-Aloud: During the Concurrent Think-Aloud, Information was aligned with anecdotes or cultural norms throughout the participant’s ISP process in four ways in reference to frames: 1. Subject: Anecdotes and past experiences are used (e.g., “I already know what this feels like.”). 2. Medium: SERP’s cultural position is considered (e.g., “Google employees probably didn’t want their algorithm to show other results”). 3. Environment: Analogies or cultural concepts are recognized (e.g., “I can use this article’s anecdote.”). 4. MetaFrame: Thinking about cultural aspects of the task’s frames (e.g., “Answering this task depends on the city.”). Observed Behavioral Actions: 109 These utterances would occur concurrently with actions. Participants used keywords that search for opinions, clicked on websites providing relatable or appealing arguments, and quote anecdotes in their task output to back up their stance. For example, participants would actively look for sources that referred to student culture in Canada. Participants also wrote their own analogies or experiences in the task outputs to support their position. Participants referred to specific stories about their experiences, such as details of witnessing texting and driving hazards. Participant-13 used an example of cellphones being used for family emergencies. Similarly, when Participant-15 could not find much information related to safety, she recalled stories she read about other people: “So I'm going to say, safety kind of reminds me of a lot of stories you see where it's like about [a person’s] mental health. You know, how it leads someone to suicide and stuff like that.” 4.3.1.7 Transferring: Validating Frames Based on Expertise Definition of this Theme: Transfer learning “reuses knowledge from past related tasks to ease the process of learning to perform a new task. The goal of transfer learning is to leverage previous learning and experience to more efficiently learn novel, but related, concepts, compared to what would be possible without this prior experience” (Yang et al., 2013). While Vygotsky’s socio-cultural approach to learning was not always in agreement with Piaget’s psychological approach, the two scholars’ theories would be in agreement that some form of transferring of knowledge takes place between novices and experts (Billett, 1998). Building from the concept of transfer learning, this dissertation will use the term transform because information was not only transferred but it would transform the learner in how they see themselves. The novice no longer sees themselves as completely uninformed, but as someone that has gained a degree of expertise. I found that for knowledge to transfer, participants must be comfortable with how they perceive 110 their self in relation to society. Trust in the expert was important but participants trusted their own expertise in judging when certain experts should be trusted over others. At times, it may be argued, participants saw themselves as more of an expert than the source they were reading. Therefore, participants transferred information when they saw themselves in relation to society’s network (e.g., a biologist may not trust a science journalist’s article on biology but may still rely on them for astronomy news). When participants found experts they trust, they would gain confidence, and then write with a new sense of expertise. I use the term "Transferring” to refer to when people validate frames based on expertise. Retrospective Think-Aloud: Participants said that their search process involved trusting the various societal roles. Doing so meant not only retrieving expert advice or trusted administrations, but also understanding their own relationship with their society and a network of trusted experts. For example, Participant-39 said: “I knew that if cancer.org says it doesn’t cause cancer then I don’t need to look at other people’s opinions.” However, participants did not need to have prior trust in the reputability of a site. Participants explained that new sources can also present themselves as reputable if they are transparent about how they acquired information. As Participant-1 explained, sources “have more weight if there was like a scientific thing done so like, observed and surveyed, like cause-and-effects.” Similarly, Participant-8 also looked to see if sources are “peer-reviewed or if they're just someone talking. And then that can inform [readers] on how correct or relevant an entire article may be.” Concurrent Think-Aloud: 111 Participants aligned themselves with information from experts-in-society throughout the participant’s ISP process in four ways in reference to frames: 1. Subject: The subject’s own confidence or expertise is used (e.g., “I actually took a course on this.”) 2. Medium: Expertise in SERP algorithm is considered. (e.g., “Google put this as the first result because it thinks it is the most helpful.”) 3. Environment: Information by experts in society is considered (e.g., “This website is written by a professor on the subject.”). 4. MetaFrame: Thinking about transferring itself. The influence of society’s experts on a task is considered (e.g., “I wonder how the experts would go about this task.”). Observed Behavioral Actions: Typically, these utterances would occur concurrently with actions. Participants used keywords that search for established expertise, clicked on websites providing scientific backing, and the retrieved information would be added to the notes or task output. Participants would seek experts, gain confidence, and then write their answers with a sense of expertise. Information was transferred based on where participants saw themselves in relation to society’s network. For example, Participant-15, a psychology student, mentioned during her search process that her background helped her pay attention to and understand the websites about mental health and cell phones. However, she recognized that her domain of expertise may bias her toward only mental health issues and so decided to change the course of her search process toward exploring how the topic of “safety of cellphones” can also refer to privacy: “I'm into mental health studies and that kind of stuff, but also, it could be like privacy, which could be unsafe.” Participants also wrote with greater confidence after gaining some form of expertise on a topic from their search than when they initially wrote their notes. For example, Participant-28’s task output said cellphones “potentially cause health risks? Not for sure? I don’t think so?” But then concluded “A lot of 112 studies were done but no results are shown [...] I don't think this actually impacts long-term health.” 4.3.1.8 Expressing: Sharing Frames with Others Definition of this Theme: I define expression as a person’s framing of an output of information. While prior ISPs often place similar stages at the end, e.g., evaluation or reflection, this dissertation found that participants were iteratively editing and evaluating how their output was expressed from the beginning and throughout the ISP. The output can either be a form of distributed cognition (Salomon, 1997) (e.g., writing notes to remember later) or a form of communication (e.g., writing an email for someone else). As mentioned in the findings, participants sometimes felt the need to express their opinion on a topic or sub-topic by outputting information before retrieving information from sources. Expression also refers to the style participants used to share the information they searched online as part of a dialectic; people tailor their communication to an audience. The expression theme that emerged from the data refers to not just the output but the rhetoric participants used to frame that output. There are three appeals to rhetoric, logos (i.e., demonstrating one is logical), ethos (i.e., demonstrating authority over a topic), and pathos (i.e., demonstrating one is sympathetic or empathetic) (Aristotle, 2019). Similarly, participants felt the need to communicate to the audience of their writing output when they felt satisfied by a diversity of search results, accomplished a thoughtful answer, or even bored enough by repetitive information. Participants do not always ‘finalize’ ideas but write as though they are part of a larger discussion and might even expect feedback. I use the term "Expressing” to refer to when people share frames with others. Retrospective Think-Aloud: 113 Participants described the strategy by which they expressed the information they found as a key element of their ISP. Participants explained the importance of communicating this information as a dialectic between the different perspectives and their own. Participant-6 explained that “it’s important, especially in presentations, to have both the point that I'm talking about and the counterpoint.” Participants also described the role of sentiments when contemplating information and how to communicate affect in their writing in different ways. First, participants referred to various sentiments as signals that the information they acquired should be expressed at all (by writing it down as notes or for their task output). Second, participants were continuously concerned with whether or not what they were communicating about the information they retrieved was both accurate and comprehensive. Participant-40 explained she felt the information she was writing down was complete after she felt “happy” that what she was expressing was “sound” and accurate: “When I was like happy with my answer already. I felt like it was sound and not too like far out. I wasn't stating anything that was, like, false.” Similarly, Participant-8 described needing to feel “comfortable” that the information he was expressing was comprehensive. “[I wrote information] based on whether I feel comfortable that it is an acceptable answer. What I mean is, when I feel that I've seen enough information to give a good …a reasonable answer to the question.” Participants explained the sense of comprehensiveness was achieved when the information retrieved “felt” redundant or the repetitive information was becoming boring. As Participant-16 explained: “Eventually it just got to the point where it felt like I was seeing the same thing over and over again. Reading the same thing over and over. With the rare oddball [piece of information]. So, I felt this [the final task output] is fine.” Similarly, Participant-14 explained redundancy caused her to send her answer to group members: “When I felt [my search] was mostly getting redundant results I sent my answer.” Before sending 114 the task output to group members, participants often used the Expression theme to double-check and finalize what they were communicating. Participant 8 explained, “You go write it down, and then afterwards you look at the conclusion, make sure that everything’s in line, if there's any sources to cite, etcetera.” Concurrent Think-Aloud: Participants made utterances that referred to expressing information throughout the participant’s ISP process in four ways about frames. 1. Subject: Participants’ utterances referred to using information for future hypotheses (e.g., “This fact is good to remember for later.”). 2. Medium: Participant utterances referred to providing feedback to SERP designers (e.g., “This should be reported to Google as unhelpful.”) 3. Environment: Participant utterances made references to a dialectical engagement with others (e.g., “This is the stance I’ll present to my group members.”) 4. MetaFrame: Participant utterances made references to thinking about how information is expressed itself or the task frame is expressed (e.g., “This how I’d answer the task.”) Observed Behavioral Actions: These utterances were observed to occur concurrently with actions. For example, participants would direct their task outputs to their group members by preparing the information to be presented. Preparing their task output consisted of finalizing the writing of a summary of their findings, making final edits, and evaluating their final draft. Framing Themes Description Examples of Intended Utterances Example of Actions Inquiring User poses questions about: 1. Subject: Internal questions about the self. 2. Medium: Questions about the features or functions. 3. Environment: Questions about the source’s content. 1.“How do I personally feel about this topic?” 2. “Why did these search results show up?” 3. “I wonder what others say?” User queries “cellphone effects on health and safety” on SERP. 115 4. MetaFrame: Thinking about the inquiry itself. Question about the task frames themselves. 4. “How do I go about answering this question?” Conceptualizing User conceptualizes stimulus: 1. Subject: Immediate psychological reaction. 2. Medium: Reaction to SERP features. 3. Environment: Source information recognized. 4. MetaFrame: Thinking about how the task is conceptualized. Reactions to task frames. 1.“Oh! I like this idea!” 2. “I’m seeing a lot of ads.” 3. “This seems like useful information.” 4. “This should be a simple task.” User naturally focuses on top of SERP. Reads Google result that states, “cellphones are unsafe.” Orientating User orients to find new information: 1. Subject: Prior knowledge is used to navigate. 2. Medium: SERP navigation 3. Environment: Source website navigation. 4. MetaFrame: Thinking about how to orient around the task. Navigating actions based on task frame. 1.“I’m looking for a familiar website.” 2. “I’m scrolling down to see what else Google suggests.” 3. “I’m skimming this article.” 4. “I’ll focus on these parts of the task first.” User orients their network of prior knowledge on cellphones being safe and trustworthy document-genres to find more sources. Guiding Information is compared and contrasted: 1. Subject: Prior knowledge on a topic is compared and contrasted (e.g., pros and cons). 2. Medium: SERP results are compared and contrasted against each other. 3. Environment: Source information is compared and contrasted with information from other sources. 4. MetaFrame: Thinking about guiding. Task frames are compared and contrasted. 1.“I read the opposite is true.” 2. “These sites provide pros but I’m looking for the cons.” 3. “This article is providing the other side of the story.” 4. “The task question focuses on the negatives but what about the positives?” Information about the safety of cellphones is compared with sources about their dangers. Numerositizing Probabilities and logic of information are used: 1. Subject: Strength of validity and soundness is considered. 1.“I find this to be the most logical conclusion.” User reasons that if cellphones are very dangerous 116 2. Medium: The algorithm’s variables are considered. 3. Environment: Source’s probability and logic are considered. 4. MetaFrame: Logic behind the task’s frames is considered. 2. “Google’s showing mostly websites in agreement.” 3. “This article backs their claims with statistics.” 4. “This is the most logical way to approach the task.” there would be more studies establishing a link. Anthropologizing Information is aligned with anecdotes or cultural norms: 1. Subject: Anecdotes and past experiences are used. 2. Medium: SERP’s cultural position is considered. 3. Environment: Analogies or cultural concepts are used. 4. MetaFrame: Thinking about cultural aspects of task frames. 1. “I experienced this once.” 2. “Google employees probably don’t want that.” 3. “This article sounds mostly anecdotal.” 4. “This part of the task relates to my community.” Info provided by trusted sources is accepted as sufficient or factual. Transferring Information is aligned with “experts-in-society.”: 1. Subject: Subject’s own confidence or expertise is used. 2. Medium: Expertise in SERP algorithm is considered. 3. Environment: Information by experts in society is considered. 4. MetaFrame: Thinking about transferring itself. The influence of society’s experts on the task is considered. 1. “I actually took a course on this.” 2. “This is Google’s first result because it’s the most helpful.” 3. “This website is written by a professor on the subject.” 4. “I wonder how the experts would go about this task.” User gains expertise. Aligns new findings on cell safety with their own expertise on cellphones. Expressing Expression of information: 1. Subject: User uses information for future hypotheses. 2. Medium: SERP feedback features are used. 3. Environment: Dialectical engagements. 4. MetaFrame: Thinking about how information is expressed itself. Task frame is expressed. 1. “This fact is good to remember for later.” 2. “This should be reported to Google as unhelpful.” 3. “This is what I think about the topic.” 4. “This is how I’d answer the task.” User writes a summary of their findings or evaluates and edits a final draft. Table 10 Coding table for ICOGNATE 117 4.4 Summary of Findings The findings are presented in the order of the research questions. RQ1 inquired about the task frames’ effect on the selection process. Addressing this question required observations on query term constructions, click-throughs, and time-on-site. In their query constructions, the participants in HF used health terms far more often than NHFP by a statistically significant margin. There was a small but not significant difference in click-throughs as participants often clicked-through sites to search for information within sites. However, the participants in HF also spent more time on the health results than the NHFP group by a statistically significant margin. RQ2 inquired about the task frames’ effect on information use. The HF group used the health-related search results located near the bottom more frequently by a statistically significant margin. The participants’ search-use depth was also different between the two groups by a statistically significant margin, whereby more participants in the HF group would use information deeper down the search page while the NHFP group mostly stopped using information by the fourth result. For RQ2, a rubric was used to compare the spread and focus of the health-related terms throughout the write-up. The HF group scored higher on the rubric than the NHFP group. The participants in the HF group wrote more about health in the introduction, body, and conclusion. RQ3 inquired about how the task, medium, and source frames affected sense-making. Addressing this question required triangulating the think-aloud with the search moves. Eight themes characterized the frame interactions in the ICOGNATE model: 1) Participants would make an inquiry about a frame, 2) A way to conceptualize/recognize a frame, 3) Orientation to acquire more perspectives about a frame, 4) Compare and contrast different perspectives in a frame, 5) Validate frames based on logic and probabilities, 6) Align frames with their own personal, social, or cultural norms, 7) Validate frames based on expertise, 8)And finally, thinking 118 of how to share the frame. The subsequent chapter connects these findings to the broader literature of information seeking and put the ICOGNATE model in the context of other ISPs. 119 5 Discussion My dissertation investigated the overarching research question: “How does framing affect the search process of students working to solve open-ended information problems?” In Chapter 1, I operationalized this larger question in three sub-questions: RQ1: How does task framing affect the information selection processes of students working on open-ended information problems? RQ2: How does task framing affect the information use of students working on open-ended information problems? RQ3: How do task, medium, and source frames interact in the process of making sense of open-ended information problems? In the following sections, I will explain how the data provides insight into each of these questions and builds on the existing body of literature in Information Science and the interdisciplinary focus area of Search As Learning (SAL). Beyond SAL, each of these research questions provides philosophical insight for the pragmatist. At the beginning of this dissertation, I grounded my study in the pragmatic philosophy put forth by John Dewey (2013) and Charles Peirce (1905). Dewey argued that intelligence is a method to seek knowledge and never a final state. Peirce (1905) argued that knowledge is retrieved through abductive reasoning, whereby a person filters through multiple perspectives for a set of “admissible hypotheses.” As this chapter explains, by looking at how people ideate and select information (RQ1) to use (RQ2) while reconciling for various frames (RQ3), a model of abductive reasoning may emerge. 120 5.1.1 Task Framing and Information Selection RQ1: How does task framing affect the information selection processes of students working on open-ended information problems? Comparing the participants within each condition, I conclude that: Task framing affects how people select information. Specifically, the query-constructions (i.e., the frequency of terms) and sites selected differed due to the presence or absence of a specified frame. Prior research on information-seeking and cognitive biases focused on how technology can lead to biases (Belkin et al., 2009; Crawford, 2021; Epstein et al., 2017; Noble, 2018; Novin & Meyers, 2017a). For example, a search engine can influence people to exhibit cognitive biases (Epstein et al., 2017; Novin & Meyers, 2017a). However, by looking at the liminal moments before and after the selection process (i.e., when people ideate about their queries or spent time thinking about the websites they selected) more nuance emerges. The findings illustrate that providing a frame for an information task, in this case focusing the searcher’s attention on health and safety, changed how participants ideate and use query terms. In other words, the search engine was not the dominant determiner of how people interacted with information; rather, – it was the human communicating a frame in a task or the participants communicating their answers for their peers. Chapter 4’s Table 6 compares the query terms used and found the task frame influenced how most participants framed their queries. Comparing the click-throughs between participants in the two conditions found the selection of information on the SERP was not significantly different [Chapter 4’s Figure 5]. For both groups of participants, clicks on the first three websites were relatively high. This behavior was expected as it was previously observed by prior scholarship (Andersson, 2017; Bar-Ilan et al., 2007; Cano-Orón, 2019; Novin & Meyers, 2017) that participant clicks gravitate towards the top results of a SERP. However, with the latter sites, there was an increase in clicks by the HF 121 group. The last site had more than twice as many clicks by the HF group compared to the NHFP group. While isolating a technology in an experiment can reveal its influence on people’s decision-making (Belkin et al., 2009; Crawford, 2021; Epstein et al., 2017; Noble, 2018; Novin & Meyers, 2017a), accounting for human context and communication in an experiment can also reveal how a task’s frame can supersede that influence. Investigating RQ1 revealed that certain dependent variables used to study the ISP selection process (i.e., click-through behavior, number of keywords chosen, time on query constructions, and breadth-first vs. depth-first search) were not significantly affected by frames. Click-through behavior (Ong et al., 2017; Urgo & Arguello, 2022) and whether participants use a breadth-first or depth-first search (Klöckner et al., 2004) can vary in different conditions. However, in this study, these variables did not differ by a statistically significant margin between the two task conditions. Nor did the time participants spent making query-constructions or the number of keywords chosen differ significantly between the two conditions. Instead, a task frame affected how much time participants spent on a site to select for information. Prior studies found that the time spent on a site can be an indicator of knowledge gains or comprehension difficulties (Bhattacharya & Gwizdka, 2018; Yu et al, 2019). The experiment in this study suggests two more potential reasons for varying times on a site: 1) People might spend less time on the site to ideate or examine new frames. Or 2) people are metacognitively examining and evaluating the framing of the information they read. For example, while Figure 5 showed how the task frame could affect SERP interactions, click-throughs, and time-on-site did not tell the whole story. As Figure 8 demonstrated, a site can be used more by participants despite not being clicked when participants used its information directly from Google’s Answer box. Thus, the NHFP group 122 clicked through the first result on the SERP less often than the HF group but used it more often. The reason for this unexpected behavior emerged from the think-aloud and observing search behaviors: The NHFP group found the information in the Answer box useful enough to use without clicking through to the website, while the HF group would click-through in search of any health information. Since there was no health information on the SERP’s first site, the HF group used it less despite the higher number of click-throughs. Using these findings to build on prior research, this dissertation makes three inferences regarding RQ1. First, a task frame always exists but can influence participants based on how explicitly the frame is communicated (e.g., including the health preamble in a task description vs. excluding one). Second, with an explicit task frame, the task frame can influence participants’ selection of sources from the search page. And third, when the task frame is less explicit, the search page’s framing of the topic can be more influential on the participants’ selection process. 5.2 Inferences on how the task frame affects the selection process This study demonstrated that frames affect document selection. In particular, students in the two conditions not only selected different documents but spent different amounts of time on them. However, participants in both conditions still clicked on many sites that did not correspond with their task frame. Reflecting on the justifications provided by the think-aloud of participants, I infer that participants will still rely on the heuristic that a site is worth being clicked-through if Google provides the search result (i.e., they justify how or why Google’s algorithm included the site in the SERP). However, after clicking through the website, most participants were able to discern that the visited website did not particularly meet the frame they were seeking and would then explore via clicking for other websites. For example, participants in the HF group would 123 still seek health-related information with results at the top of the search page, but after not finding any relevant information they would spend less time on the website. Despite this general behavior, some participants would sometimes include a few information points in their writing output to share with peers. I believe this behavior shows a distinction between relevant information and 'framed' information: Participants will seek information that fits into an overall frame, but may still find relevant information that is external to that frame and attempt to fit it into a frame. This does not mean that every piece of relevant information will fit into a frame. As the findings show, the HF group chose to not spend their time on the information that did not fit within the health frame they sought and so did not fit most of the relevant information at the top of the SERP into their write-up. 5.2.1 Task Framing and Information Use RQ2: How does task framing affect the information use of students working on open-ended information problems? Task framing can affect information use in several ways, including: 1) The site-use depth (i.e., how far down a SERP a person will use information), 2) How much of the site’s content ended up being used in the writing output, and 3) The frequency of terms used in a writing output. The findings for RQ2 supported prior research that technology can lead to biases (Belkin et al., 2009; Crawford, 2021; Epstein et al., 2017; Noble, 2018; Novin & Meyers, 2017a) because the participants in the HF group still used the information from the top SERP results often. The findings addressing RQ2 provided further evidence to just how influential SERPs can be. The method of accumulating further evidence on a SERP’s influence was derived from the information scholars cited in Chapter 2 who espoused that researchers should study information usefulness more closely than relevancy. Notably, the importance of usefulness becomes apparent 124 with well-defined tasks (Cole et al., 2009). Supporting this prior literature, my findings show that while the previous section found little statistical difference between both groups on the click-throughs, this difference is significant when the sites used for the task were compared [Chapter 4, Figure 9]. When I looked at how far down the SERP the HF group was willing to use information, I found a difference between both conditions that was statistically significant. Prior studies on task frames found that changing the wording of a task, such as using negative or positive terms, will influence people (Kahneman, 2013; Neale et al., 1987). For example, a task frame claiming an action has a 90% probability of success can influence participants to complete the action as opposed to telling them it has a 10% probability of failure (Kahneman, 2013). In this study, the measures used on the students’ task outputs (i.e., the rubric and comparison of word frequency) also showed high statistically significant differences [Chapter 4: Figure 10, Figure 11, Figure 12]. I interpret these findings to mean that the two conditions’ task descriptions enabled participants to perceive two different task frames that required two different levels of health information to be completed. The two frames were influential enough to direct the participants writing outputs to include two different levels of health information. Just as prior studies found that tasks that frame the success-rate of an output can influence participants’ outputs (Kahneman 2013; ), this study found that even when presenting a task as an open problem with “no right answers” the mere referencing of one perspective of a task (i.e., health) still influenced participants towards different outputs. Prior scholars (O’Brien et al., 2020; Vakkari, 2016) found that contextual analyses of writing outputs can highlight key differences between participants in two conditions. This study supports these findings. For the HF group, all the participants had health-related points in the body of their 125 notes, with 85% mentioning multiple health-related aspects (e.g., physical issues and mental health issues). In comparison, fewer than half of the participants in the NHFP group mentioned a point about health and less than a third mentioned any secondary points about health. None of the NHFP group participants dedicated 50% or more of their task output to the topic of health. Instead, they used a more diverse set of points provided by the SERP, such as giving equal attention to mentioning academic and privacy issues. Therefore, we might infer that in this situation the influence of the task frame can be greater than the medium’s frame for most participants. This finding serves as an argument against just how technologically deterministic a SERP’s document-order can be. Using these findings to build on prior research, this dissertation makes three inferences regarding RQ2 that build on and support RQ1. The data used to address RQ2 indicate that a task frame always exists, but can influence participants based on how explicitly one is communicated. More specifically, with an explicit task frame, the task frame can influence participants’ selection (as found with RQ1’s findings) and will continue to affect participants’ usage of sources from the search page (as found with RQ2’s findings). Therefore, when the task frame is less explicit, the search page’s framing of the topic was more influential on not only selection (as found with RQ1’s findings) but will also have a continuous effect on site usage for the duration of the search task. However, how long this effect lingers after the completion of the search task would require a longitudinal study. For example, in a study by Herbert Simon,(Simon & Stern, 1955) he found that framing effects (e.g., the framing of political issues) only influence people’s perceptions for a few days before they consider alternative perspectives. 126 5.3 Inferences on how the task frame affects the search use The task frame affected the search use and the focus of the overall writing output regarding this information. Both groups received the same relevant results on the top and the same wording for the question on whether should cellphones should be restricted by UBC. However, the relevancy of that information was affected by the preamble of the health frame. In other words, the relevancy of the top SERP to the direct question in isolation remained fixed. If participants in the HF group were merely searching and retrieving answers to the question without regarding the health frame, these top results would have sufficed. However, the frame affected the entire seeking process of what qualified as relevant and useful in the first place. 5.3.1 Framing Interactions and the Challenge of Task Frames RQ3: How do task, medium, and source frames interact in the process of making sense of open-ended information problems? The task, medium, and source frame can all influence participants. However, the task frame can have a stronger influence than the SERP’s frame. For both Dewey (2013) and Peirce (1905), the reconciliation of multiple perspectives was an essential component of pragmatic thinking. An outcome of exploring the process of ideation and selection for RQ1 and information usage for RQ2 was that the findings lent some understanding to the process of abductive reasoning. Using RQ1 and RQ2, this experiment showed quantitative evidence that the influence of the task frame (i.e., the health-frame) can supersede the medium’s frame (i.e., the SERP) for most participants. RQ3 is interested in how did this occur? How did participants make sense of the interacting frames? Answering this question required a more subjective and qualitative analysis. A strong task frame has the potential to influence information selection and use patterns more so than framing from a SERP. Looking at the think-aloud, it seems that the SERP itself can set the framing of an issue if a clear 127 task frame is absent. My interpretation of the think-aloud data is that participants thought about how the task frame aligned with the search page frame. The NHFP group seemed to interact mostly with what the top of the Google page suggested were answers. However, I inferred from their writing outputs that some of the participants occasionally tried to justify the use of the health results in their answers. Conversely, the HF group attempted justifications for the use of health results more often because I infer that they perceived that the health frame surrounding their task superseded the search page. How the participants reconciled all three frames (the subject, medium, and environment) inspired me to look back at my findings on the think-aloud analysis to make further inferences. Looking back at my analysis that led to ICOGNATE, I found participants often remembered information that could be used for what may first seem to be a ‘potentially relevant frame.’ In turn, this potential frame would influence participants and even be used later in the ISP. The importance of keeping frames in mind extended to keeping stages of the ISP in mind. Participants would sometimes explicitly distinguish between how they intended their search strategy to be and how their search strategy played out. The frequency of this observation and analysis from think-aloud suggests that there are two ISPs, the intended ISP, and the distributed ISP. The intended ISP framing interacts with the stimulus of the problem (for example, how the person frames an open problem to understand it). The distributed ISP interacts with the synthesis addressing the problem (for example, how the person frames an open problem for a task output). During their concurrent think-aloud, participants would often make references to what they intended to do during their search process, I refer to these utterances as an “intended ISP.” During their retrospective think-aloud, participants almost always explicitly referred to their ISP and framed it 128 as a much more linear process, I refer to these utterances as a “distributed ISP.” For example, almost half of the participants began their search process with an intended ISP of framing their query and notes towards a local frame using terms such as “UBC.” However, a “UBC” frame was unavailable to participants on the SERP. In their think-aloud, participants provided evidence to suggest they were altering their strategy to explore the frames provided in the SERP. In their retrospective think-aloud, they would mainly refer to their reframed distributed ISP while neglecting the limitation they experienced with their intended frame (i.e., a local frame, such as UBC). An example of a person thinking about different frames within a metaframe involved Participant-3 who thought about a site’s author to think about multiple stakeholders behind a website. The participant first read the content from a website claiming the risks of cellphones in a classroom that disagreed with their personal opinion, the participant then questioned the data of the site, then questioned the metadata (i.e., the author) and used the fact that the author was a teacher to think about all the stakeholders. The participant commented that the teacher’s website should be understood in the context of the education industry pushing teachers towards alternative agendas (e.g., more ad revenue or selling a book) to pay bills. Thus, the participant was not only thinking about the framing of the content by the teacher, but also how other stakeholders frame the role of a teacher. Very little of this context was directly available in either the data or metadata to the participant – leaving them to only assume about hidden agendas in the education industry. Building on Erving Goffman's (1974) and Deborah Tannen’s (1993) work on framing theory, my study confirms that participants can interact, reconcile, challenge, or even create their own frames. However, I also extend their argument by suggesting these actions take place within a metaframe, whereby participants can think about indefinite frames at once. Participants who 129 worked with conflicting frames would sometimes even change their minds. Participant-16 wanted to focus on statistical frames and also wanted their stance to be informed by that frame. The hierarchy of a statistical frame superseded but also was influenced by the other frames. The participant's heuristics (i.e., looking for statistics) was upheld and applied for the task frame (i.e., health), but the statistics did not manage to undo the influence of the health-frame (e.g., the participant did not look for only statistics or the statistics of cheating). Within this metaframe, participants would keep in mind other websites they encountered and compare and contrast them. Thus, not only were the sites kept in mind, but the juxtaposition of the different types of sites was kept in mind during the ISP. Within this metaframe, the think-aloud revealed that participants can think about frames as situations and even metacognate about the ISP itself. Participants can hold sites in mind, they can hold multiple ISP stages themselves in mind together in parallel. It was by working with different frames that allowed people to change their minds. Thus, alongside the distributed ISP, is an internal ISP with the participant's own intentions on how to carry out the ISP. It is via this intended ISP that people can reframe information. A metaphor for metaframing indefinite frames. Returning to my prior study (Novin & Meyers, 2017), I previously found that a person can be influenced by the information in the subject frame (i.e., cognitive bias), the medium (i.e., a SERP), or source frames (i.e., the documents in a SERP). The combination of these biases can cause different groups of people to summarize topics with different biases (potentially spreading a bias). For example, even though the participants I studied would read SERP results about both corn biofuels and algae biofuels, their summaries would focus on one type over another, depending on the order and genre of the results presented (Novin & Meyers, 2017). I found that if corn biofuels appeared at the top of a 130 SERP and algae biofuels at the bottom, people would summarize the topic of “biofuels” to focus on corn biofuels and often neglect even mentioning algae. In the discussion section of that previous study, I suggested that the finding could be troublesome if the SERP topic is not as innocuous as “biofuel” types and more related to social justice issues (e.g., the minority perspectives of less-dominant groups in a political debate that favors a status quo) (Novin & Meyers, 2017). I also suggested that a way to mitigate some of the SERP bias is greater awareness of how a SERP frames a topic (Novin & Meyers, 2017). However, this also raises the question of how beholden a user is to a SERP bias. Perhaps the context surrounding a search task itself can usurp some of the SERP bias? For example, if a task was structured to represent a less-dominant view that would only appear at the bottom of a search, would participants still favor the results a SERP presents at the top? I now suggest that the task itself can influence people’s ISP. 5.4 Updating the Information Seeking Process While no one can create a search model that captures all human-computer interactions, there are general mechanisms that need to be accounted for. I believe one of those mechanisms is how people frame context and communicate context with frames. Early search models in the 20th century by information scholars, such as Belkin, Elis, Kuhlthau, and Dervin, laid the foundation for representing search as non-linear and dynamic (Wilson, 1999; Vakkari, 2006). However, these models often relied on observing search behaviors and interviews. By using a framed work task to look for signs of abductive reasoning with a think-aloud I feel I expanded on these models. Focusing on indicators of abductive reasoning provided insight into how people ideate about hypotheses and use heuristics when they encounter multiple indefinite frames. In doing so I expanded some of the past stages identified by prior ISP models. 131 In particular, the Anthropologizing and Numerositizing stages are not mentioned in prior ISP models, although evidence from my study suggests they may play key roles in how people rationalize and make sense of frames: Numerositizing was identified as when participants validate frames based on logic and probabilities. For example, in the concurrent think aloud Participant-8 uttered: “This article backs their claims with statistics.” and Participant-36: “[This] is the most logical way to approach the task.” Likewise, in the retrospective think aloud Participant-6 provided an analysis of how he uses logic during his ISP: “I basically have like three switches in my brain: It's the ‘Attack,’ it's the ‘Counter-Point,’ and then the ‘Garbage.’ And so, I break down every sentence. I'm like, ‘All right, is this for me or against me regarding my stance [for the final task output]. And when I find something particularly interesting, I look back into my notes and say, ‘All right, is this a part of a point that I already stated? Or is this a rebuttal to a counterpoint that I have?’” Similarly, Numerositizing was identified as when participants validated frames based on logic and probabilities. For example, in the concurrent think aloud Participant-8 uttered: “This article backs their claims with statistics.” and Participant-36: “[This] is the most logical way to approach the task. Chapter 2’s literature review began by mentioning past metaphors used by information scholars to describe the ISP process, such as Marcia Bates’ “berry-picking” model or Brenda Dervin’s description of information seeking as a “gap” that needed to be bridged. However, during my process of conceptualizing people’s think-alouds, I wanted a metaphor of the information search process that focused on how people sense-make not just by framing a topic, nor by framing multiple frames, but by juxtaposing multiple frames of an indefinite open problem via a 132 metaframe. I use MC Escher’s “impossible cube,” a metaphorical cube with indefinite frames, as that metaphor. The three images below provide visualizations distinguishing a frame, multiple frames, and indefinite frames. The first image is the metaphorical picture frame used by Goffman (1974). It is a photograph of the tower of Pisa, but it is framed to create an optical illusion about its height. The second image is still a photograph, but its framing breaks the optical illusion to demonstrate that different people will frame for different perspectives. The photograph itself demonstrates that people can frame for multiple frames and thereby multiple perspectives. However, some topics will remain open and contain indefinite frames. The third image, the “impossible cube” is not a cube that can be comprehensively framed. The viewer can see multiple frames but, unlike the second image, the viewer cannot break the optical illusion into clear multiples. Figure 13 Three Visual Metaphors Distinguishing a Frame, Multiple Frames, and Indefinite Frames The process of juxtaposing an indefinite number of sub-frames was found to be a challenge for participants, leading them to not always account for all of the frames they encountered in their think-aloud or writing output. This observation was fully apparent when I was trying to triangulate people’s concurrent think-alouds with their retrospective think-alouds for the 133 ICOGNATE model. When people described their search process to me, they did not just recall the stages, but created a narrative on what they intended to do during their search process. However, that narrative did not always match the concurrent think-aloud on how the search process played out. This was an expected outcome as Ericsson and Simon (1993) wrote extensively about how the concurrent and retrospective think-alouds are often dissonant. While this dissonance was characterized as a limitation of the think-aloud protocol, I find it to be a feature exemplifying the persistence of metaframing; people retrospectively frame how they encountered and accounted for frames. For example, returning to Participant 6’s retrospective think-aloud, he provided a narrative: He says he has 3 switches in his brain that he uses to frame information as either supportive, a counter-point, or as garbage. However, when I revisited the video of his actual search strategy, his narrative was more about how he intended to carry out his information search process than what happened. While to some extent I believe he used this tactic as a general guide for his search process, not all information can be framed as neatly into his three categories as he intended. As the construction of his write-up revealed, at different moments the participant would uncritically evaluate information (all participants did so at different moments). He exhibited the same cognitive biases I observed in other participants such as relying more heavily on results just because they appeared at the top of a search page. Thus, the participant created a frame in his writing output that was influenced by how the simulated search page framed the topic. I suggest the sense-making of an open problem involves a person simultaneously metacognating about how they intend to deal with framing a topic (e.g., Participant 6’s 3 switches) and communicating or distributing their own framing of the information (e.g., Participant 6’s framing of his write-up was influenced by the search page. 134 Figure 14 The relationship between an intended and distributed metaframe as a person frames an open problem The relationship between the four types of frames (i.e., subject, medium, source, and a metaframe) are visualized below using the following analogy: Suppose people who have never seen a geometric cube before need to reconcile different perspectives of a cube using sources from the Internet. However, different sources only provide a single perspective (i.e., creating a source frame). The medium frame would be a SERP listing different cube perspectives iteratively, where the order of the sources (or cube perspectives) may influence how people perceive the cube. 135 Figure 15 Model of how participants moved between intending to perform information-seeking stages (via the think-aloud) and distributing those tasks (via SERP interactions, notes, and task output). My dissertation observed that participants recognized that these frames could sometimes compete and tried to reconcile them. From my analysis of participant utterances and search behaviors, I found that people recognize, interact, or reconcile multiple indefinite frames via the themes identified in ICOGNATE. ICOGNATE is the process by which people address indefinite frames. In brief, the eight themes suggest frames can be interacted with as an inquiry, concept, orientation, guide, numerosity, anthropologically, transferring, or expression [Chapter 4’s Table 10]. ICOGNATE builds on prior work by information scholars mapping the information-seeking process (ISP), such as Kuhlthau’s ISP. During the ISP, there are multiple frames surrounding information that people interact with that can affect information seeking. ICOGNATE identified the themes where people think or metacognate about these frames. Identifying the themes of ICOGNATE was important because people can challenge the influence of frames by thinking about them or using metacognition (e.g., thinking about how someone’s thinking framed information [Figure 15]). 136 5.5 Updates to Methods of Studying the ISP One of the contributions I make to future methods for studying the ISP is via the creation and use of the Quasi-Live Corpus [Figure 3]. I created a live corpus that controls for certain variables in a SERP but keeps the corpus live for realism to increase external validity. This dissertation controlled for the document-order of the results on the SERP but the SERP results themselves were live. Outside of the SERP’s order of results, the query box interactions, the SERP features (i.e., clicking, highlighting, and scrolling), and the websites were not controlled for. By doing so, the simulated search page allowed me to observe participants’ natural behaviors during a task. It also addressed one of Buhler et al.’s (2023) suggestions for future research about simulating search tasks being more realistic by allowing participants to open SERP links as new tabs or windows. I felt giving participants greater freedom in tuning their search setup for the task allowed them to be more comfortable with simulated SERP. In addition, by asking participants to create writing-outputs, I feel the experiment captured enough information about participants’ understanding of the topic without burdening them with any kind of “test.” An important aspect of ISPs for information scholars, particularly those studying SAL, is measuring the learning outcomes (O’Brien et al., 2020; Vakkari, 2016; X. Zhang et al., 2011). This dissertation supports prior scholars (Bali & Ramadan, 2007; Y. Zhang & Wildemuth, 2009) argument that using a contextual analysis on a writing output can assist with evaluating learning outcomes well enough. 137 5.5.1 Chapter Summary I used this chapter to connect my findings to prior literature both within and outside of the information sciences. Doing so allowed me to elaborate on my inferences from my findings and then create a model of how the findings come together. I began by confirming the usefulness of the methods developed in SAL to measure the participants' ISP interactions, notably the selection and usage of information, while accounting for the more abstract concept of metacognition. I used this chapter to discuss 1) Where these measurements were effective for comparing frames, 2) Some further inferences I can make measuring framing, and then 3) Built a model that accounted for how framing emerged in the ISP. Past Information scholars found that users often select for information at the top of the SERP, even to the point of exhibiting signs of cognitive biases (Bar-Ilan et al., 2007; Fabris et al., 2020; Robertson et al., 2018). However, the findings for RQ1 found different outcomes for information selection may emerge depending on the task frames. This argument was based on my own inference of the findings. Stronger evidence supporting my inference about a frame’s influence was found while addressing RQ2 and the information people use to complete a task output. Prior literature from both within the information field (Andersson, 2017; Liu, 2023; Urgo & Arguello, 2022a) and outside of the information field (Kahneman, 2013; Kühberger, 1998; Neale et al., 1987; Tannen, 1993) also allowed me to compare my manipulation of task frame variables (i.e., the inclusion and exclusion of a preamble to an open problem) with other manipulations, such as directly changing the wording of a task question or hinting at the probability of beneficial task outputs. Compared to other studies that framed tasks by hinting at the outputs, the findings here found evidence that frames that do not need to hint at probability outputs to influence people’s 138 search with open problems. In addition, based on the framing theory of Goffman (1974) and Bateson (1973), framing can influence users from the query terms they use to their writing-outputs. Finally, my discussion made inferences that go beyond observing the various ways people are influenced by frames by looking at how people openly interacted with frames or addressed frames during their ISP via ICOGNATE. Building on the past literature on ISPs and accounting for the various signs of framing (i.e., ICOGNATE) I suggested that participants wrestled with an intended ISP and distributed ISP when they were presented with multiple frames. The indefinite frames of an open problem can cause people to reason through them abductively. ICOGNATE provided a model of the eight different themes that can be observed when people use abductive reasoning. While this model has its limitations, the ICOGNATE themes that emerged during this experiment provided observable evidence of abductive reasoning in an ISP that extends the philosophical concept of abductive reasoning that the pragmatists, John Dewey and Charles Peirce espoused. 139 6 Conclusion This dissertation is built on prior theories on how systems influence sense-making, particularly SAL. The findings provide nuance to our prior conceptualization of how search engines may influence our heuristics and our knowledge of a topic. Particularly, it showed how context and task frames can significantly affect the behaviors of people conducting research – regardless of the SERP’s influence. It also provided a model, ICOGNATE, for researching and identifying framing during the information search process. The following subsections conclude this dissertation by elaborating on several contributions the study makes to the field of Information Science. The contributions are clustered in three broad areas: 1) prior theories on how systems influence sense-making, 2) pedagogy and the importance of information literacy, and 3) a method of researching the information search process of participants using ICOGNATE. I provide two areas where this dissertation may foster further research and outline some of the limitations that temper the findings and conclusions. 6.1 Contribution to Prior Theories on Systems and Sense-Making This dissertation contributes to the field of Human Information Interactions and Search As Learning (SAL). More specifically, it provides additional depth related to how people use metacognition when framing information and introduces the concept of abductive reasoning’s (i.e., the selection, use, and evaluation of hypotheses) role with an open problem. An open problem cannot be ‘solved’ by some grand deductive or inductive reasoning. Therefore, abductive reasoning requires selecting, using, and reconciling the multiple frames a person encounters. To abductively select, use, and reconcile frames, people use the eight themes in 140 ICOGNATE to abductively approach open problems. Prior research in SAL found that metacognition enables humans to both complete a search task (Crescenzi, 2016) and conceptualize search as a learning experience (O’Brien et al., 2022). Much of the prior SAL research made significant advances in learning in terms of how the information on a SERP is acquired by users. However, this dissertation began by philosophically re-evaluating the concepts of search and learning via a pragmatic philosophical lens emphasizing the importance of abductive reasoning. Abductive reasoning is ever more important when the problem is open-ended, as the solutions to a hypothesis may occur long after an immediate search task – if a solution is ever found at all. If solutions are not foreseeable, then what is being “searched” to be “learned”? In these open-problem situations, this dissertation found that the parameters of both search and learning are expanded enough to present a problem when defining search as learning. That is, learning involves much more than just acquiring the knowledge contained in documents. People can download medical textbooks online but require years of practice before becoming a doctor. Similarly, learning involves more than just the knowledge acquired by a query, a SERP, and its documents. With open problems, searchers employ abductive reasoning when solutions are lacking to create hypotheses that are continuously evaluated throughout a search process while framing multiple perspectives (i.e., abductive reasoning). More specifically, participants were observed to sense-make open problems by looking for where society's thinking on a problem “left off” when addressing the open problem, instead of what society “knows.” In other words, a person navigates for how society's understanding of a problem is distributed amongst sources, just as Hutchins' observed a Navy team distributes their cognition of a ship's navigation tasks so that any member can pick up where others “left off” before the task is completed (Hutchins, 1994). Search situations requiring distributed cognition are common; there are far 141 more open-ended problems that require framing multiple perspectives than there are definitive answers. 6.2 Contribution to Pedagogy and Information Literacy First, the findings of this dissertation make a scholarly contribution to SAL in terms of understanding how people search and how frames affect the ISP. This scholarly work also complements past research that focused on information quality, “fake news,” misinformation, and disinformation (Afful-Dadzie et al., 2023; Hameleers, 2023; Miró-Llinares & Aguerri, 2023). The findings show that even high-quality information can be misfit, such as when participants use information despite its irrelevancy (e.g., the HF-group tried to fit in information from the top results of the SERP even when the bottom results were more relevant). The findings from this dissertation show that frames influence how people assess the usefulness of information. Second, my dissertation contributes to our understanding of information literacy in a democratic society where many communities may have competing frames of information. Public discourse benefits when citizens are educated on digital literacy and how to think critically about framing. Since publishing the paper, Novin & Meyers (2017), other papers citing my work have expanded on my original findings to observe how a SERPs influence can propagate sexism, racism, homophobia, and other forms of oppression (Cano-Orón, 2019; Fabris et al., 2020; Robertson et al., 2018; Urman et al., 2022). While these studies expanded on the effects of framing, I wanted this dissertation to contribute an explanation of how frames affect information selection and use. 6.3 Contribution to Researching the ISP Using ICOGNATE Finally, the ICOGNATE model opens the path for building on methods of researching the ISP with abductive reasoning in mind. To further understand how frames, affect information seeking, 142 I suggested that scholars explore how people interact with multiple frames in the themes of ICOGNATE. ICOGNATE provided the foundations for eight themes that may emerge and can be identified. One of these foundations comes from the strong inter-coder reliability of the eight themes. The strong inter-coder reliability with the ICOGNATE model suggests that if a similar study is reproduced then the eight themes of ICOGNATE should be observable. If replicability is the case, then it may be useful to use more quantitative methods that study the frequency or duration with which each of these themes emerges and their relationships. I have already begun preliminary work in this area which I will briefly touch on in the next section on future studies. 6.4 Future studies The Search As Learning area of inquiry is growing swiftly as scholars recognize the importance of learning processes to how people interact with search engines and other information access and inquiry tools. I envision two future paths for this research: 1) a focus on abductive reasoning with different user populations and search systems; and 2) further refinement and testing of the ICOGNATE approach to enhance scholarly understanding of search system interactions. This study employed a simulated-SERP presentation to control the order of search results to ascertain how the task description would affect how participants evaluated the usefulness of a specific set of documents. It could be that a live corpus of Google will lead students to frame their results differently. I am in the process of replicating this experiment using a live corpus. Preliminary analysis indicates that the influence of task frames persists. As mentioned in the Introduction, new search tools are being developed even as I type, particularly the rush to incorporate AI and chat-based search in existing tools, based on the public infatuation with OpenAI’s ChatGPT. For example, Google’s Bard, Microsoft’s Bing, and Baidu’s ERNIE all seek 143 to exploit the interest in moving from search engines to “answer engines” that generate responses in a conversational manner. Whether the tasks used in this study would translate to differences in output for different search tools could be another extension to this inquiry. By understanding how framing affects a person’s ISP, scholars can hopefully adopt this method into understanding how people alter their search behavior to address different framings of situations and how this may lead to different cognitive biases. Finally, I introduced the ICOGNATE model that emerged from my analysis of participant utterances and search behaviors and extends prior work on mapping the information-seeking process. I observed the ISP to be non-linear and dynamic, whereby different themes of the process can be thought of in parallel. Additional research is needed to validate and operationalize it as a model that privileges information behavior's non-linear nature. 6.5 Limitations Several limitations challenge the validity of certain claims made in this dissertation. The most important limitation was that the page was a simulated Google page that controlled for the documents presented: It was not responsive to queries. This control could lead to participants being slightly dissatisfied or disoriented with the results. Another major limitation is the operationalization of the task frame. This was done by changing the task prompt and essentially specifying the search topic to a greater extent. First, the demographic that was used for the experiment does not represent the general population. The demographic is both younger and more academically educated than the average person in Canada and the World. However, biases are a problem that affects everyone. If highly educated students (90% of whom claim to have received some form of digital literacy education and are highly confident in their skills) can exhibit cognitive-biases, then the level of digital literacy imparted to students in a curriculum can 144 be improved. Second, the experiment in this dissertation used a simulated-Google page, which does not represent all the features Google provides. Third, measurements on time-on-site and time-on-task are not perfect representations of cognitive-effort. While there is a long scholarly tradition of making inferences on cognitive-effort via the time spent carrying out a cognitive action, it has long been acknowledged that this is an inexact tool. For this reason, this dissertation attempted to triangulate this data with think-aloud data. However, this raises a fourth limitation: As useful as a think-aloud can be in providing insights into a student's thoughts, it does not replicate their internal thoughts. A think-aloud merely represents the fragmented thoughts communicated and not their subconscious or hidden thoughts. It is extremely likely that students did not consciously communicate certain thoughts. To strengthen the inferences on what was communicated, my dissertation created the ICOGNATE coding-table [Table 10] for identifying common themes. However, this presents the fifth limitation: ICOGNATE is merely a model built via a strong inter-coder reliability between two coders. It could very well be that two other coders strongly agree on another coding scheme. Finally, the observations made are based on an experiment, which requires controlling for certain variables that would normally not be if the observations were made “in the wild.” This is a common trade-off that all information retrieval studies face when wishing to control for variables they wish to make statistical inferences on (Liu, 2023). In such scenarios, the researcher needs to justify a balance between their controls and mimicking real-world scenarios (Liu, 2023). This study presented a search task whereby participants were asked to frame their answers as though they were working on a group project for an instructor’s class assignment. In addition, they were asked to write their notes and answers as though their audience were their group members in the class. While these instructions helped increase some of the external validity, it is still a simulated scenario and Google users, 145 especially students outside of the classroom, often do not have their task descriptions and objectives written out for them. 6.6 Conclusion This dissertation makes two essential contributions to understanding how people search and make meaning with web-based information. The first is its advancements when answering the general question, “How does framing affect the search process of students working to solve open-ended information problems?” This study found that a hierarchy of framing that can emerge to influence a person's search experience, whereby a task's frame can potentially supersede a SERP's influence. This finding expands on prior research that found people use metacognition (i.e., thinking about thinking) during search (Cole, 2002; Smith & Rieh, 2019) by looking at the frames people metacognate about. Examining the think-aloud data, I concluded that just as cognition requires framing, metacognition requires a metaframe (i.e., where an agent can metacognate using multiple frames). The second contribution was the dissertation's method to analyze think-aloud to observe abductive reasoning. To look at abductive reasoning the participants' hypotheses and inquiry were tracked and analyzed more closely. The method introduced the ICOGNATE coding-table [Table 10]. For this dissertation, the novel method was used to track the effects of framing on the think-aloud and how people challenge frames or re-frame topics. Frames exert an enormous influence on what we do on how we think through everyday problems. Yet, frames are often invisible during our information-seeking activities (e.g., reading the news, reading a book, watching a tv show). In this study, I tried to forefront frames in an information search task to see how students would use resources to address an everyday campus problem (i.e., the use of cellphones on campus). The study found a significant difference 146 between conditions both in the behaviors that students exhibit with selecting information and their activities when using information. The findings from this study have implications for information systems design and end-users of those systems (e.g., from information design to teachers creating information curricula). As our search systems become more automated and more reliant on advanced AI, the role of frames will become essential to our mediation and evaluation of information. 147 Works Cited Addison, C., & Meyers, E. (2013). Perspectives on information literacy: A framework for conceptual understanding. Information Research: An International Electronic Journal, 18(3), n3. Afful-Dadzie, E., Afful-Dadzie, A., & Egala, S. B. (2023). Social media in health communication: A literature review of information quality. Health Information Management Journal, 52(1), 3–17. Alfano, M., Lenzitti, B., Taibi, D., & Helfert, M. (2019). ULearn: Personalized Medical Learning on the Web for Patient Empowerment. 217–228. Allgaier, J. (2019). Science and Environmental Communication on YouTube: Strategically Distorted Communications in Online Videos on Climate Change and Climate Engineering. Frontiers in Communication, 4, 36. https://doi.org/10.3389/fcomm.2019.00036 Andersson, C. (2017). “Google is not fun”: An investigation of how Swedish teenagers frame online searching. Journal of Documentation, 73(6), 1244–1260. Aristotle. (2019). Aristotle’s Art of Rhetoric. University of Chicago Press. Bali, M., & Ramadan, A. R. (2007). Using rubrics and content analysis for evaluating online discussion: A case study from an environmental course. Journal of Asynchronous Learning Networks, 11(4). Bar-Ilan, J., Keenoy, K., Yaari, E., & Levene, M. (2007). User rankings of search engine results. Journal of the American Society for Information Science and Technology, 58(9), 1254–1266. https://doi.org/10.1002/asi.20608 148 Baskerville, R., & Lee, A. S. (1999). Distinctions among different types of generalizing in information systems research. New Information Technologies in Organizational Processes: Field Studies and Theoretical Reflections on the Future of Work, 49–65. Bateson, G. (1973). Steps to an Ecology of Mind: Collected Essays in Anthropology, Psychiatry, Evolution and Epistemology. Paladin. Belkin, N. J. (1984). Cognitive models and information transfer. Social Science Information Studies, 4(2–3), 111–129. https://doi.org/10.1016/0143-6236(84)90070-X Belkin, N. J., Cole, M., & Liu, J. (2009). A model for evaluation of interactive information retrieval. 7–8. Belkin, N. J., Oddy, R. N., & Brooks, H. M. (1982). ASK for information retrieval: Part I. Background and theory. Journal of Documentation, 38(2), 61–71. Bhattacharya, N., & Gwizdka, J. (2019). Measuring Learning During Search: Differences in Interactions, Eye-Gaze, and Semantic Similarity to Expert Knowledge. Proceedings of the 2019 Conference on Human Information Interaction and Retrieval, 63–71. https://doi.org/10.1145/3295750.3298926 Billett, S. (1998). Appropriation and ontogeny: Identifying compatibility between cognitive and sociocultural contributions to adult learning and development. International Journal of Lifelong Education, 17(1), 21–34. https://doi.org/10.1080/0260137980170103 Bink, M., Zimmerman, S., & Elsweiler, D. (2022). Featured Snippets and their Influence on Users’ Credibility Judgements. 113–122. Borlund, P., & Schneider, J. W. (2010). Reconsideration of the simulated work task situation: A context instrument for evaluation of information retrieval interaction. 155–164. 149 Buhler, A. G., Brannon, B., Cataldo, T. T., Faniel, I. M., Connaway, L. S., Valenza, J. K., Elrod, R., & Cyr, C. (2023). How real is real enough? Participant feedback on a behavioral simulation used for information-seeking behavior research. Journal of Librarianship and Information Science, 55(1), 191–207. https://doi.org/10.1177/09610006211067799 Byström, K., & Hansen, P. (2005). Conceptual framework for tasks in information studies. Journal of the American Society for Information Science and Technology, 56(10), 1050–1061. https://doi.org/10.1002/asi.20197 Byström, K., & Kumpulainen, S. (2020). Vertical and horizontal relationships amongst task-based information needs. Information Processing & Management, 57(2), 102065. Cano-Orón, L. (2019). Dr. Google, what can you tell me about homeopathy? Comparative study of the top10 websites in the United States, United Kingdom, France, Mexico and Spain. Profesional de La Informacion, 28(2). Chapman, S., Desai, S., Hagedorn, K., Varnum, K., Mishra, S., & Piacentine, J. (2013). Manually classifying user search queries on an academic library web site. Journal of Web Librarianship, 7(4), 401–421. Choi, B., Capra, R., & Arguello, J. (2019). The Effects of Working Memory during Search Tasks of Varying Complexity. Proceedings of the 2019 Conference on Human Information Interaction and Retrieval, 261–265. https://doi.org/10.1145/3295750.3298948 Chow, S. J. (2013). What’s the Problem with the Frame Problem? Review of Philosophy and Psychology, 4(2), 309–331. Cole, M., Liu, J., Belkin, N. J., Bierig, R., Gwizdka, J., Liu, C., Zhang, J., & Zhang, X. (2009). Usefulness as the criterion for evaluation of interactive information retrieval. 1–4. 150 Collins-Thompson, K., Rieh, S. Y., Haynes, C. C., & Syed, R. (2016). Assessing Learning Outcomes in Web Search: A Comparison of Tasks and Query Strategies. Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval, 163–172. https://doi.org/10.1145/2854946.2854972 Crawford, K. (2021). Atlas of AI: Power, politics, and the planetary costs of artificial intelligence. Yale University Press. Crescenzi, A. (2016). Metacognitive knowledge and metacognitive regulation in time-constrained in information search. SAL@ SIGIR. Creswell, J. (2014). Research Design—Qualitative, Quantitative, and Mixed Methods Approach (Vol. 3). Sage. Delacre, M., Lakens, D., & Leys, C. (2017). Why psychologists should by default use Welch’s t-test instead of Student’s t-test. International Review of Social Psychology, 30(1). Dennett, D. C. (1984). Cognitive wheels: The frame problem of AI. Minds, Machines and Evolution, 129–151. Dervin, B. (1998). Sense‐making theory and practice: An overview of user interests in knowledge seeking and use. Journal of Knowledge Management, 2(2), 36–46. https://doi.org/10.1108/13673279810249369 Desai, S., & Twidale, M. (2022). Is Alexa like a computer? A search engine? A friend? A silly child? Yes. 4th Conference on Conversational User Interfaces, 1–4. https://doi.org/10.1145/3543829.3544535 Dewey, J. (2013). Logic—The Theory of Inquiry. Read Books Ltd. Dos Santos, C. F. (2022). Re-establishing the distinction between numerosity, numerousness, and number in numerical cognition. Philosophical Psychology, 1–29. 151 Dourish, P. (2001). Where the action is: The foundations of embodied interaction. MIT press. Edwards, P. N., Mayernik, M. S., Batcheller, A. L., Bowker, G. C., & Borgman, C. L. (2011). Science friction: Data, metadata, and collaboration. Social Studies of Science, 41(5), 667–690. https://doi.org/10.1177/0306312711413314 Ellis, D., Cox, D., & Hall, K. (1993). A Comparison of the Information Seeking Patterns of Researchers in the Physical and Social Sciences. Journal of Documentation, 49(4), 356–369. https://doi.org/10.1108/eb026919 Epstein, R., Robertson, R. E., Lazer, D., & Wilson, C. (2017). Suppressing the Search Engine Manipulation Effect (SEME). Ericsson, K. A., & Simon, H. A. (1993). Protocol Analysis, revised edition: Verbal Reports as Data. MIT Press. Evelyn, K. (2020, January 29). “Like I wasn’t there”: Climate activist Vanessa Nakate on being erased from a movement. The Guardian. https://www.theguardian.com/world/2020/jan/29/vanessa-nakate-interview-climate-activism-cropped-photo-davos Fabris, A., Purpura, A., Silvello, G., & Susto, G. A. (2020). Gender stereotype reinforcement: Measuring the gender bias conveyed by ranking algorithms. Information Processing & Management, 57(6), 102377. Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry. American Psychologist, 34(10), 906. Fodor, J. (2006). How the mind works: What we still don’t know. Daedalus, 135(3), 86–94. Fodor, J. A. (2008). LOT 2: The language of thought revisited. Clarendon Press ; Oxford University Press. 152 Freund, L., O’Brien, H., & Kopak, R. (2014). Getting the big picture: Supporting comprehension and learning in search. Proceedings SAL. Fulantelli, G., Marenzi, I., Ahmad, Q. A. I., & Taibi, D. (2016). SaR-Web-A Tool to Support Search as Learning Processes. SAL@ SIGIR. Gallistel, C. R., & King, A. P. (2009). Memory and the computational brain: Why cognitive science will transform neuroscience. Wiley-Blackwell. Giles, J. (2006). Stacking the deck. Nature, 440(7082), 270–272. https://doi.org/10.1038/440270a Giudice, G. F. (2012). Big Science and the Large Hadron Collider. Physics in Perspective, 14(1), 95–112. https://doi.org/10.1007/s00016-011-0078-1 Gobet, F., & Simon, H. A. (1998). Expert chess memory: Revisiting the chunking hypothesis. Memory, 6(3), 225–255. Goffman, E. (1974). Frame analysis: An essay on the organization of experience. Harper & Row. Greene, J. A., Bolick, C. M., Jackson, W. P., Caprino, A. M., Oswald, C., & McVea, M. (2015). Domain-specificity of self-regulated learning processing in science and history. Contemporary Educational Psychology, 42, 111–128. https://doi.org/10.1016/j.cedpsych.2015.06.001 Gwizdka, J., & Bilal, D. (2017). Analysis of children’s queries and click behavior on ranked results and their thought processes in google search. 377–380. Hameleers, M. (2023). Disinformation as a context-bound phenomenon: Toward a conceptual clarification integrating actors, intentions and techniques of creation and dissemination. Communication Theory, 33(1), 1–10. 153 Hango, D. (2014). University graduates with lower levels of literacy and numeracy skills (75-006–X). Statistics Canada. https://www150.statcan.gc.ca/n1/en/pub/75-006-x/2014001/article/14094-eng.pdf Höchstötter, N., & Lewandowski, D. (2009). What users see–Structures in search engine results pages. Information Sciences, 179(12), 1796–1812. Hofstadter, D. R. (2001). Analogy as the core of cognition. The Analogical Mind: Perspectives from Cognitive Science, 499–538. Holloway, D., & Green, L. (2016). The Internet of toys. Communication Research and Practice, 2(4), 506–519. Howard, J. (2018). Cognitive Errors and Diagnostic Mistakes: A Case-Based Guide to Critical Thinking in Medicine. Springer. Hyndman, R. J. (2011). Moving Averages. Ingwersen, P. (1996). Cognitive perspectives of information retrieval interaction: Elements of a cognitive IR theory. Journal of Documentation, 52(1), 3–50. Kahneman, D. (2013). Thinking, fast and slow (First). Farrar, Straus and Giroux. Kahneman, D., Rosenfield, A. M., Gandhi, L., & Blaser, T. (2016). Noise: How to overcome the high, hidden cost of inconsistent decision making. Harvard Business Review, 94(10), 38–46. Karanam, S., van Oostendorp, H., & Indurkhya, B. (2012). Evaluating CoLiDeS + Pic: The role of relevance of pictures in user navigation behaviour. Behaviour & Information Technology, 31(1), 31–40. https://doi.org/10.1080/0144929X.2011.606335 Karweit, N. (1984). Time-on-task reconsidered: Synthesis of research on time and learning. Educational Leadership, 41(8), 32–35. 154 Katai, Z. (2015). The challenge of promoting algorithmic thinking of both sciences- and humanities-oriented learners: Promoting algorithmic thinking. Journal of Computer Assisted Learning, 31(4), 287–299. https://doi.org/10.1111/jcal.12070 Kellogg, R. T. (1987). Effects of topic knowledge on the allocation of processing time and cognitive effort to writing processes. Memory & Cognition, 15(3), 256–266. Kelly, D., & Azzopardi, L. (2015). How many results per page?: A Study of SERP Size, Search Behavior and User Experience. Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, 183–192. https://doi.org/10.1145/2766462.2767732 Klöckner, K., Wirschum, N., & Jameson, A. (2004). Depth-and breadth-first processing of search result lists. 1539–1539. Kodama, C., St. Jean, B., Subramaniam, M., & Taylor, N. G. (2017). There’sa creepy guy on the other end at Google!: Engaging middle school students in a drawing activity to elicit their mental models of Google. Information Retrieval Journal, 20, 403–432. Kühberger, A. (1998). The Influence of Framing on Risky Decisions: A Meta-analysis. Organizational Behavior and Human Decision Processes, 75(1), 23–55. https://doi.org/10.1006/obhd.1998.2781 Kuhlthau, C. C. (1993). A Principle of Uncertainty for Information Seeking. Journal of Documentation, 49(4), 339–355. https://doi.org/10.1108/eb026918 Kukull, W. A., & Ganguli, M. (2012). Generalizability. Neurology, 78(23), 1886–1891. https://doi.org/10.1212/WNL.0b013e318258f812 155 Laerd.com. (2023, April). Independent t-test in SPSS Statistics—Procedure, output and interpretation of the output using a relevant example | Laerd Statistics. https://statistics.laerd.com/spss-tutorials/independent-t-test-using-spss-statistics.php Lee, A. S., & Baskerville, R. L. (2003). Generalizing generalizability in information systems research. Information Systems Research, 14(3), 221–243. Li, J., Huffman, S., & Tokuda, A. (2009). Good abandonment in mobile and PC internet search. 43–50. Liu, J. (2021). Deconstructing search tasks in interactive information retrieval: A systematic review of task dimensions and predictors. Information Processing & Management, 58(3), 102522. Liu, J. (2023). Implications and New Directions for IR Research and Practices. In J. Liu, A Behavioral Economics Approach to Interactive Information Retrieval (Vol. 48, pp. 181–201). Springer International Publishing. https://doi.org/10.1007/978-3-031-23229-9_7 Longino, H. E. (2018). Essential tensions—Phase two: Feminist, philosophical, and social studies of science. In A mind of one’s own (pp. 93–109). Routledge. MacGregor, L. J., Corley, M., & Donaldson, D. I. (2010). Listening to the sound of silence: Disfluent silent pauses in speech have consequences for listeners. Neuropsychologia, 48(14), 3982–3992. Meyers, E. M. (2018). When search is (mis) learning: Analyzing inference failures in student search tasks. Proceedings of the Association for Information Science and Technology, 55(1), 357–366. Miklošík, A., & Daňo, F. (2016). SEARCH ENGINE OPTIMISATION AND GOOGLE ANSWER BOX. Communication Today, 7(1). 156 Miró-Llinares, F., & Aguerri, J. C. (2023). Misinformation about fake news: A systematic critical review of empirical studies on the phenomenon and its status as a ‘threat.’ European Journal of Criminology, 20(1), 356–374. Mishra, P., Singh, U., Pandey, C., Mishra, P., & Pandey, G. (2019). Application of student’s t-test, analysis of variance, and covariance. Annals of Cardiac Anaesthesia, 22(4), 407. https://doi.org/10.4103/aca.ACA_94_19 Mukherjee-Reed, A., & Szeri, A. (2019). University of British Columbia 2018/19 Annual Enrolment Report. https://academic.ubc.ca/sites/vpa.ubc.ca/files/documents/2018-19%20Enrolment%20Report.pdf Neale, M. A., Huber, V. L., & Northcraft, G. B. (1987). The framing of negotiations: Contextual versus task frames. Organizational Behavior and Human Decision Processes, 39(2), 228–241. https://doi.org/10.1016/0749-5978(87)90039-2 Nichols, D. M., & Twidale, M. B. (2011). Recommendation, collaboration and social search. http://researchcommons.waikato.ac.nz/handle/10289/5659 Noble, S. U. (2018). Algorithms of oppression: How search engines reinforce racism. New York University Press. Novin, A., & Meyers, E. (2017a). Making Sense of Conflicting Science Information: Exploring Bias in the Search Engine Result Page. CHIIR ’17 Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval, 175–184. https://doi.org/10.1145/3020165.3020185 Novin, A., & Meyers, E. M. (2017b). Four Biases in Interface Design Interactions. In A. Marcus & W. Wang (Eds.), Design, User Experience, and Usability: Theory, Methodology, and 157 Management (Vol. 10288, pp. 163–173). Springer International Publishing. https://doi.org/10.1007/978-3-319-58634-2_13 O’Brien, H. L., Cole, A., Kampen, A., & Brennan, K. (2022). The Effects of Domain and Search Expertise on Learning Outcomes in Digital Library Use. ACM SIGIR Conference on Human Information Interaction and Retrieval, 202–210. https://doi.org/10.1145/3498366.3505761 O’Brien, H. L., Kampen, A., Cole, A. W., & Brennan, K. (2020). The Role of Domain Knowledge in Search as Learning. Proceedings of the 2020 Conference on Human Information Interaction and Retrieval, 313–317. https://doi.org/10.1145/3343413.3377989 Oliveira, B., & Lopes, C. T. (2023). The Evolution of Web Search User Interfaces—An Archaeological Analysis of Google Search Engine Result Pages. ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR’23), Austin, TX, USA. https://doi.org/10.1145/3576840.3578320 Oliver, K., & Hannafin, M. J. (2000). Student Management of Web-Based Hypermedia Resources During Open-Ended Problem Solving. The Journal of Educational Research, 94(2), 75–92. https://doi.org/10.1080/00220670009598746 Ong, K., Järvelin, K., Sanderson, M., & Scholer, F. (2017). Using Information Scent to Understand Mobile and Desktop Web Search Behavior. Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, 295–304. https://doi.org/10.1145/3077136.3080817 Oreskes, N. (2019). Why Trust Science? Princeton University Press. 158 Palani, S., Ding, Z., MacNeil, S., & Dow, S. P. (2021). The “Active Search” Hypothesis: How Search Strategies Relate to Creative Learning. Proceedings of the 2021 Conference on Human Information Interaction and Retrieval, 325–329. https://doi.org/10.1145/3406522.3446046 Pardi, G., Kammerer, Y., & Gerjets, P. (2019). Search and Justification Behavior During Multimedia Web Search for Procedural Knowledge. Companion Publication of the 10th ACM Conference on Web Science, 17–20. https://doi.org/10.1145/3328413.3329405 Pehkonen, E. (1997). Use of Open-Ended Problems in Mathematics Classroom. Research Report 176. ERIC. Peirce, C. S. (1905). What Pragmatism Is. The Monist, 15(2), 161–181. Petersen, A. M., Vincent, E. M., & Westerling, A. L. (2019). Discrepancy in scientific authority and media visibility of climate change scientists and contrarians. Nature Communications, 10(1), 3502. https://doi.org/10.1038/s41467-019-09959-4 Rieh, S. Y., Collins-Thompson, K., Hansen, P., & Lee, H.-J. (2016). Towards searching as a learning process: A review of current perspectives and future directions. Journal of Information Science, 42(1), 19–34. Robertson, R. E., Lazer, D., & Wilson, C. (2018). Auditing the personalization and composition of politically-related search engine results pages. 955–965. Roschelle, J. (2000). Choosing and using video equipment for data collection. Handbook of Research Design in Mathematics and Science Education, 709–731. Rouet, J.-F., Vidal-Abarca, E., Erboul, A. B., & Millogo, V. (2001). Effects of information search tasks on the comprehension of instructional text. Discourse Processes, 31(2), 163–186. 159 Roy, N., Torre, M. V., Gadiraju, U., Maxwell, D., & Hauff, C. (2021). Note the highlight: Incorporating active reading tools in a search as learning environment. 229–238. Salmerón, L., & García, V. (2011). Reading skills and children’s navigation strategies in hypertext. Computers in Human Behavior, 27(3), 1143–1151. https://doi.org/10.1016/j.chb.2010.12.008 Salomon, G. (1997). Distributed Cognitions: Psychological and Educational Considerations. Cambridge University Press. Sharit, J., Hernández, M. A., Czaja, S. J., & Pirolli, P. (2008). Investigating the Roles of Knowledge and Cognitive Abilities in Older Adult Information Seeking on the Web. ACM Transactions on Computer-Human Interaction, 15(1), 1–25. https://doi.org/10.1145/1352782.1352785 Silver, L. (2020). Americans see different global threats facing the country now than in March 2020. Pew Research Center. https://www.pewresearch.org/fact-tank/2022/06/06/americans-see-different-global-threats-facing-the-country-now-than-in-march-2020/ Simon, H. A., & Stern, F. (1955). The effect of television upon voting behavior in Iowa in the 1952 presidential election. American Political Science Review, 49(2), 470–477. Skiera, B., Eckert, J., & Hinz, O. (2010). An analysis of the importance of the long tail in search engine marketing. Electronic Commerce Research and Applications, 9(6), 488–494. Smith, C., Urgo, K., Arguello, J., & Capra, R. (2022). Learner, Assignment, and Domain: Contextualizing Search for Comprehension. 191–201. 160 Song, S., Zhao, Y. C., Yao, X., Ba, Z., & Zhu, Q. (2022). Serious information in hedonic social applications: Affordances, self-determination and health information adoption in TikTok. Journal of Documentation, 78(4), 890–911. Stalnaker, R. C. (1984). Inquiry (Issue 145). Cambridge University Press. Statista. (2023, March). Global search engine desktop market share 2023. Statista.Com. https://www.statista.com/statistics/216573/worldwide-market-share-of-search-engines/ Steiner, I. D. (1972). Group process and productivity. Academic Press. Sullivan, G. M., & Feinn, R. (2012). Using Effect Size—Or Why the P Value Is Not Enough. Journal of Graduate Medical Education, 4(3), 279–282. https://doi.org/10.4300/JGME-D-12-00156.1 Surif, J., Ibrahim, N. H., & Dalim, S. F. (2014). Problem Solving: Algorithms and Conceptual and Open-ended Problems in Chemistry. Procedia - Social and Behavioral Sciences, 116, 4955–4963. https://doi.org/10.1016/j.sbspro.2014.01.1055 Tannen, D. (1993). Framing in Discourse. Oxford University Press. The Britannica Group. (2020, August). ProCon. Britannica Procon.Org. www.procon.org Traum, D. R., & Heeman, P. A. (1997). Utterance units in spoken dialogue. In E. Maier, M. Mast, & S. LuperFoy (Eds.), Dialogue Processing in Spoken Language Systems (Vol. 1236, pp. 125–140). Springer Berlin Heidelberg. https://doi.org/10.1007/3-540-63175-5_42 Turpin, A., & Scholer, F. (2006). User performance versus precision measures for simple search tasks. 11–18. Urgo, K., & Arguello, J. (2022a). Capturing Self-Regulated Learning During Search. 161 Urgo, K., & Arguello, J. (2022b). Learning assessments in search-as-learning: A survey of prior work and opportunities for future research. Information Processing & Management, 59(2), 102821. Urgo, K., & Arguello, J. (2022c). Learning assessments in search-as-learning: A survey of prior work and opportunities for future research. Information Processing & Management, 59(2), 102821. Urgo, K., & Arguello, J. (2023). Goal-setting in support of learning during search: An exploration of learning outcomes and searcher perceptions. Information Processing & Management, 60(2), 103158. https://doi.org/10.1016/j.ipm.2022.103158 Urgo, K., Arguello, J., & Capra, R. (2020). The Effects of Learning Objectives on Searchers’ Perceptions and Behaviors. 77–84. Urman, A., Makhortykh, M., & Ulloa, R. (2022). Auditing the representation of migrants in image web search results. Humanities and Social Sciences Communications, 9(1), 1–16. Vakkari, P. (2003). Task-based information searching. Annual Review of Information Science and Technology (ARIST), 37, 413–464. Vakkari, P. (2016). Searching as learning: A systematization based on literature. Journal of Information Science, 42(1), 7–18. https://doi.org/10.1177/0165551515615833 Van Voorhees, B. W., Fogel, J., Pomper, B. E., Marko, M., Reid, N., Watson, N., Larson, J., Bradford, N., Fagan, B., & Zuckerman, S. (2009). Adolescent dose and ratings of an Internet-based depression prevention program: A randomized trial of primary care physician brief advice versus a motivational interview. Journal of Cognitive and Behavioral Psychotherapies: The Official Journal of the International Institute for the Advanced Studies of Psychotherapy and Applied Mental Health, 9(1), 1. 162 Vanessa Nakate [@vanessa_vash]. (2020, January 24). You didn’t just erase a photo You erased a continent But I am stronger than ever https://t.co/J34WMXvPAo [Tweet]. Twitter. https://twitter.com/vanessa_vash/status/1220852476355731458 Weingart, N., & Eickhoff, C. (2016). Retrieval Techniques for Contextual Learning. SAL@ SIGIR. Wildemuth, B. M. (2016). Applications of social research methods to questions in information and library science. Abc-Clio. Wildemuth, B. M., & Freund, L. (2012). Assigning search tasks designed to elicit exploratory search behaviors. 1–10. Williams, P. (2023). A route to greener Big Science. Nature Physics, 1–2. https://doi.org/10.1038/s41567-022-01933-0 Wilson, T. D. (1999). Models in information behaviour research. Journal of Documentation, 55(3), 249–270. Yamamoto, Y., & Yamamoto, T. (2020). Personalization Finder: A Search Interface for Identifying and Self-controlling Web Search Personalization. Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020, 37–46. https://doi.org/10.1145/3383583.3398519 Yamashita, T. (1989). Fuzzy Logical Approach to Perception of Dot Numerosity. Perceptual and Motor Skills, 69(3_suppl), 1319–1329. https://doi.org/10.2466/pms.1989.69.3f.1319 Yang, L., Hanneke, S., & Carbonell, J. (2013). A theory of transfer learning with applications to active learning. Machine Learning, 90(2), 161–189. Yeung, P. C. K. (2007). Weighting Document Genre in Enterprise Search [Master Thesis, University of Waterloo]. https://uwspace.uwaterloo.ca/handle/10012/3160 163 Zadeh, L. A. (1988). Fuzzy logic. Computer, 21(4), 83–93. Zhang, H., Buchanan, G., Kelly, R., Twidale, M., Chang, S., & McKay, D. (2022). It’s Still Rock and Roll to Me: A Model of Online Browsing Behaviour. Proceedings of the Association for Information Science and Technology, 59(1), 381–392. https://doi.org/10.1002/pra2.761 Zhang, X., Cole, M., & Belkin, N. (2011). Predicting users’ domain knowledge from search behaviors. 1225–1226. Zhang, Y., & Wildemuth, B. M. (2009). Unstructured interviews. Applications of Social Research Methods to Questions in Information and Library Science, 222–231. 164 Appendix 1 – Consent Form School of Information Vancouver Campus Irving K. Barber Learning Centre Vancouver, BC Canada V6T 1Z1 Email [redacted] Consent Form for the following study: “How Do We Search and Summarize Topics?” Principal Investigator: Dr. Eric Meyers, Associate Professor, School of Information (UBC), [redacted] Co-Investigator(s): Alamir Novin, PhD candidate, School of Information (UBC) [redacted] Thank you for your interest in participating in this study. Before you begin, you should read the following information so that you are aware about the study and your rights. At the end of this consent form, if you still agree to participate in this study then you can provide your explicit consent. What is the purpose of this study? This study is being conducted to inform a graduate degree. The purpose of this research is to observe your behavior when you read search engine results on a science topic and which ones you use to summarize the topic. The method will observe how you search and use an interview to understand your knowledge on a scientific topic. What will happen during the study? (Study Procedures): The study will take place on a recorded Zoom call where you will be provided with a search task to research a science topic. First, you’ll be asked two questions about your previous knowledge on a scientific topic. Next, you will be provided a search page to 165 research the topic. You’ll then be asked to read and rank the search results and use whichever you need in order to summarize the topic. During your searching you’ll be asked to “think aloud” (i.e., saying what you’re thinking out loud as you interact with the search page). The whole process should take about 30 minutes, but you’re welcome to take extra time if you need to. In return, you’ll be compensated with a Rain or Shine gift card for an ice cream. There are no right or wrong answers, and you may simply provide your opinion about the science topic and the usefulness of the search results. You probably won’t experience discomfort during the procedures but if at any time you feel uncomfortable, you may stop participating. Your answers will inform the education system and help design future interfaces, such as the Google search engine. What are the risks and benefits? Risk: The study’s researcher will be present as you search and will be asking what you’re thinking. Otherwise, the risk is no greater than the same discomfort you may find when searching Google about science topics. You will also be asked for your opinions, but you do not have to provide it if you are uncomfortable. Your information will be kept confidential; your name, image or other identifiable information will not be linked with the data you provide, and no personally identifiable information will be published in reports, articles, or other research products. Benefit: The benefits of the research are generally academic; they support informed public dialogue and suggestions for search engine design. During the Zoom call you should be in a private or quiet space. If there will be any surrounding people, you should make them aware that you’ll be in a recorded call as well. You can also turn off your camera, and you can mute your microphone (if it is not needed). You should also log in using only a nickname or a substitute name. Confidentiality: Information will be used in aggregate (mixed in with other responses) to identify themes from all the answers. Your individual case will not be used in a way that may identify you. If your words or descriptions on the task are used then your identity and confidentiality will be maintained. Your answers will be held on protected servers that meet BC's strict privacy legislations and University of British Columbia's community concerns. Research findings will be distributed via publications. Some of these publications will be open access for the public. Contact for information about the study: If at any time you have questions about your participation in the project, please contact, Alamir Novin, in the iSchool Department at the University of British Columbia, at [redacted] or by email at [redacted]. Who can you contact if you have complaints or concerns about the study? 166 If you have any concerns or complaints about your rights as a research participant and/or your experiences while participating in this study, contact the Research Participant Complaint Line in the UBC Office of Research Ethics at [redacted] or if long distance e-mail [redacted] or call toll free [redacted] 167 Appendix 2 – Task Script and Interview Questions Study Introductory Encounter Script 1.1. GREETINGS & INTRODUCTIONS !I’m Alamir and I’m a doctoral student at UBC’s School of Information. [CO]: I know student life can be really busy... [CO]: This session will last about 30 minutes. I will briefly explain my study so that you can begin your search. [CO]: Is it ok if we continue? 1.2 EXPLANATION ABOUT THE STUDY [CO]: As you read on the information sheet, I’ll be observing you while you search and then conduct a brief interview. I’m interested in learning how undergraduate students, such as yourself, use search results to understand science topics. For example, I’ll be interested in which sites you find to be useful. This will help researchers develop better search strategies for students. [CO]: Do you have any questions? If yes, attend to their inquiries and concerns. If not, move to section 1.3. 1.3 CONSENT [CO]: Thank you for reading the consent forms. I just want to remind you that you can leave the study at any time you want without any consequence. You’ll receive a rain or shine gift certificate for your participation. The gift certificate will be sent to your email after the study. 1.4 INTRODUCTION TO STUDY TASK AND DATA COLLECTION [CO]: I’d like to first introduce you to what the study’s task is like. I’ll be asking you to search during this Zoom call. I’ll also be interested in which pieces of information you use from the sites you visit. To help me get a better idea of what you’re thinking, I’ll ask you to think aloud as you search. To give you an example I’ll screen share a google search. One query. [CO] While you search I will be taking notes based on which search results you find to be useful and which ones you don’t. After you complete the task, I will begin the interview portion. Throughout this process I will record the Zoom call as you share your screen and store the information on a secure hard drive. [CO]: Do you have any questions so far about how the data will be collected? If yes, answer their questions and concerns. If no, move to the following paragraph. 1.5 Task Set-Up [CO]: Just a reminder, the recording may capture some events in the background as I’ll be recording what is happening on the webcam and on your computer screen. Is this location/position comfortable for you? 168 If yes, proceed to the following paragraph. If not, [CO] informs that the participant may position/move their computer anywhere they feel most comfortable. [CO]: Before you start sharing your screen, please open your browser and open a blank tab. Then enter the following URL I messaged you. In a second blank tab please enter the second URL I’ll message you. [CO] will message a link to blogs.ubc.ca/news and blogs.ubc.ca/news/submit. [CO]: I’d like you to use the first tab to read the search task’s question – you’ll also be using this space to submit your answer. The second tab will be where you will use Google’s search. Is that clear? [CO] will pay attention whether the participant requires further assistance [CO]: Great. Is it ok with you if I begin the recording? If yes, [CO] will proceed to next paragraph If no, [CO] will ask if it’s ok to proceed with a demonstration without recording the screen. Depending on the participant’s answer, [CO] will ask if the participants have any concerns or inquiries and attend to their concerns/inquiries. 1.6 Observation [CO]: Before we begin. I’d just like to briefly ask about your familiarity with the topic of cell phones on school campus. Do you recall coming across any issues on the topic, such as from the news media or in any of your classes? [CO] will record the answer. [CO]: Thank you. Now I would like you to answer the task question using the Google page you see on your screen while thinking out loud. Every now and then I might prompt you on what you’re thinking. [CO] will video record the activities is doing to complete their task. [CO]: Thank you! I now have only 6 brief interview questions. [CO]Conduct interview with following questions: 1. How difficult did you find the task questions? What made them easy or difficult? 2. Can you recap your search process for me. From when you queried to how you came up with your answer? 3. Did any search page results change your mind on a topic? 4. Were certain page results found to be more useful than others? 169 5. Did you notice any sources that conflict with each other? Did you feel a need to resolve the conflict? Why and how? 6. Were there any parts of the study that left you feeling dissatisfied? The study is complete. Thank you! You’ve helped me tremendously. Debrief the session for about 5 minutes. Explain that the Google page was a mock-up and the search results were hand-picked by myself. Explain that I chose the topic because I myself am not sure what the right or wrong answer to cell phones on campus is – it is more likely a situational issue. 1.7 Compensation [CO]: Thank you again for your time! I hope you enjoyed this session. Can I send your gift certificate to [redacted]? Allow some time for the participant to confirm their email address. [CO]: I’ll be emailing a rain or shine gift certificate to your email. Just a moment. [CO] sends the gift certificate and certificate of completion to the participant. [CO]: Could you please confirm the receipt of the gift certificate? Allow some time for the participant to confirm the receipt of the gift certificate and certificate of completion. [CO]: Thank you! 1.8 Wrapping up [CO]: Once again, I really appreciate your interest in my study. Thank you! Do not hesitate to contact me if you have any questions or concerns. [CO]: Thank you again! Your participation will be valuable! Goodbye!
Thesis/Dissertation
eng
Library, Archival and Information Studies
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Meyers, Eric M. (Professor in information science)
How task frames affect the process and products of inquiry
Text