Applied Science, Faculty of
Electrical and Computer Engineering, Department of
DSpace
UBCV
Mavali, Sahar
Master of Applied Science - MASc
University of British Columbia
In-class multitasking is a very common practice among university students as a strategy to tackle their many goals and commitments in life. However, research shows that in-class multitasking affects students' learning outcomes and academic performance negatively. Even though many students are aware of these potential drawbacks, in-class multitasking remains persistent in in-person and even more so in online classes. To learn more about students' motivations, perceptions, and challenges in in-class multitasking in online classes and to be able to propose solutions to their challenges, we conducted a small formative study (N=10) where we asked students to keep multitasking diaries in their online classes and followed up with semi-structured interviews. Through this study, we categorized students' challenges with in-class multitasking into 1. cognitive consequences which consist of content loss, context loss, and the overhead in time spent after the class to compensate for multitasking, and 2. metacognitive consequences which consist of the inhibition of time and attention management due to multitasking. Our insights from the preliminary study lead to the overarching requirement in design which is to support in-class multitasking by reducing its adverse effects on learning. Based on the cognitive consequences, our first design requirement is to enable immediate recovery of missed content in the class time frame. To satisfy this requirement, we propose a bichronous learning environment that enables accelerated guided viewing of past/missed content immediately. In an experimental evaluation (N=20) of these design elements, our prototype showed significant improvement in students' learning outcomes when they multitask, compared to a baseline system. Based on the metacognitive consequences, we define our second design requirement as providing assistance in metacognitive monitoring in in-class multitasking by enabling cognitive offloading. We propose topic, activity, and time-based offloading to help students manage their time and attention when multitasking under different circumstances. Our qualitative evaluation (N=9) of these design elements shows promise in their effectiveness in assisting students' self-regulation in in-class multitasking.
Supporting Multitasking in OnlineSynchronous ClassesbySahar MavaliB.Sc., Sharif University of Technology, 2020A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFMASTER OF APPLIED SCIENCEinThe Faculty of Graduate and Postdoctoral Studies(Electrical and Computer Engineering)THE UNIVERSITY OF BRITISH COLUMBIA(Vancouver)October 2023© Sahar Mavali 2023The following individuals certify that they have read, and recommend tothe Faculty of Graduate and Postdoctoral Studies for acceptance, the thesisentitled:Supporting Multitasking in Online Synchronous Classessubmitted by Sahar Mavali in partial fulfillment of the requirements forthe degree of Master of Applied Science in Electrical and ComputerEngineering.Examining Committee:Sid Fels, Professor, Electrical and Computer Engineering, UBCSupervisorLuanne Sinnamon, Associate Professor, School of Information, UBCSupervisory Committee MemberKonstantin Beznosov, Professor, Electrical and Computer Engineering, UBCSupervisory Committee MemberiiAbstractIn-class multitasking is a very common practice among university studentsas a strategy to tackle their many goals and commitments in life. How-ever, research shows that in-class multitasking affects students’ learningoutcomes and academic performance negatively. Even though many stu-dents are aware of these potential drawbacks, in-class multitasking remainspersistent in in-person and even more so in online classes. To learn moreabout students’ motivations, perceptions, and challenges in in-class multi-tasking in online classes and to be able to propose solutions to their chal-lenges, we conducted a small formative study (N=10) where we asked stu-dents to keep multitasking diaries in their online classes and followed upwith semi-structured interviews. Through this study, we categorized stu-dents’ challenges with in-class multitasking into 1. cognitive consequenceswhich consist of content loss, context loss, and the overhead in time spentafter the class to compensate for multitasking, and 2. metacognitive conse-quences which consist of the inhibition of time and attention managementdue to multitasking.Our insights from the preliminary study lead to the overarching require-ment in design which is to support in-class multitasking by reducing itsadverse effects on learning. Based on the cognitive consequences, our firstdesign requirement is to enable immediate recovery of missed content inthe class time frame. To satisfy this requirement, we propose a bichronouslearning environment that enables accelerated guided viewing of past/missedcontent immediately. In an experimental evaluation (N=20) of these designelements, our prototype showed significant improvement in students’ learn-ing outcomes when they multitask, compared to a baseline system. Basedon the metacognitive consequences, we define our second design requirementiiiAbstractas providing assistance in metacognitive monitoring in in-class multitaskingby enabling cognitive offloading. We propose topic, activity, and time-basedoffloading to help students manage their time and attention when multi-tasking under different circumstances. Our qualitative evaluation (N=9)of these design elements shows promise in their effectiveness in assistingstudents’ self-regulation in in-class multitasking.ivLay SummaryUniversity students engage in a substantial amount of multitasking in on-line classes despite being aware of its negative impacts on their learning.Depending on the learner’s goals, in-class multitasking can be a positivestrategic behavior to increase productivity. In a small formative study, weexplored students’ challenges in in-class multitasking and identified a num-ber of design elements. We propose to facilitate multitasking in online syn-chronous classes by providing a novel bichronous (a blending of synchronousand asynchronous) learning environment named Time-Turner that providesasynchronous guided accelerated viewing of past content during synchronousclasses. Our summative evaluation of our prototype showed a significant im-provement in learning outcomes when multitasking (N=20). Furthermore,95% of users found the design helpful and expressed interest in having itin their online classes. Furthermore, we designed metacognitive assistingalerts and evaluated them qualitatively. Our findings show the potential ofsupporting positive multitasking in synchronous online classes.vPrefaceAll the research in this thesis was conducted in the Human CommunicationTechnologies (HCT) laboratory at the University of British Columbia. Theexperiments and user studies reported in this thesis were approved by theUniversity of British Columbia’s Research Ethics Board (certificate numberH22-00642). A significant portion of this thesis has been submitted to atop-tier conference.I, Sahar Mavali, was the lead investigator in this research. Dr. Sid Felsprovided supervision in formulating research questions, conducting exper-iments, and editorial feedback on the manuscript. Dr. Dongwook Yooncontributed to refining the design guidelines and iterating on the proposedprototype. Dr. Luanne Sinnamon provided feedback on the research direc-tions, experiment design, and the design guidelines. Members of the HCTlab contributed by participating in pilot studies and providing valuable feed-back on the experiment design.viTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Research Questions . . . . . . . . . . . . . . . . . . . . . . . 31.2 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . 41.3 Thesis Structure . . . . . . . . . . . . . . . . . . . . . . . . . 52 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.1 In-class Multitasking . . . . . . . . . . . . . . . . . . . . . . 62.2 Positive/productive multitasking . . . . . . . . . . . . . . . . 72.3 Existing Designs for Multitasking . . . . . . . . . . . . . . . 93 Exploring Students’ In-class Multitasking . . . . . . . . . . 103.1 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103.1.1 Participants . . . . . . . . . . . . . . . . . . . . . . . 113.1.2 Procedure . . . . . . . . . . . . . . . . . . . . . . . . 113.2 Insights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133.2.1 Insights from the Diary Study . . . . . . . . . . . . . 13viiTable of Contents3.2.2 Insights from the Interview . . . . . . . . . . . . . . . 173.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244 Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254.1 Designing for Cognitive Consequences of In-class Multitask-ing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264.1.1 Design Requirements and Rationale . . . . . . . . . . 264.1.2 Cognitive Design Elements . . . . . . . . . . . . . . . 284.1.3 Time-Turner Prototype Design . . . . . . . . . . . . . 304.2 Designing for Metacognitive Consequences of In-class Multi-tasking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364.2.1 Design Requirements and Rationale . . . . . . . . . . 364.2.2 Designing for Metacognitive Consequences . . . . . . 374.2.3 Metacognitive Assistance Prototype . . . . . . . . . . 385 Evaluating the Design Elements . . . . . . . . . . . . . . . . 425.1 Evaluating Cognitive Design Elements (Time-Turner) . . . . 425.1.1 Participants . . . . . . . . . . . . . . . . . . . . . . . 425.1.2 Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . 435.1.3 Material . . . . . . . . . . . . . . . . . . . . . . . . . 435.1.4 Procedure . . . . . . . . . . . . . . . . . . . . . . . . 455.1.5 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . 475.1.6 Results . . . . . . . . . . . . . . . . . . . . . . . . . . 475.2 Evaluating Metacognitive Design Elements . . . . . . . . . . 525.2.1 Participants . . . . . . . . . . . . . . . . . . . . . . . 525.2.2 Procedure . . . . . . . . . . . . . . . . . . . . . . . . 525.2.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . 536 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 616.1 Supporting Students’ Autonomy . . . . . . . . . . . . . . . . 616.2 Facilitating In-class Multitasking . . . . . . . . . . . . . . . . 626.3 Benefits of Self-paced Learning . . . . . . . . . . . . . . . . . 636.4 Implications for Teaching . . . . . . . . . . . . . . . . . . . . 646.4.1 Beyond Online Learning . . . . . . . . . . . . . . . . 65viiiTable of Contents6.5 Limitations and Future Work . . . . . . . . . . . . . . . . . . 666.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69AppendicesA Consent Forms . . . . . . . . . . . . . . . . . . . . . . . . . . . 75B Diary Template . . . . . . . . . . . . . . . . . . . . . . . . . . . 78C Follow-up Interview . . . . . . . . . . . . . . . . . . . . . . . . 79D Materials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81D.1 Lecture Videos . . . . . . . . . . . . . . . . . . . . . . . . . . 81D.2 Secondary Task . . . . . . . . . . . . . . . . . . . . . . . . . 81D.3 Prototype . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82D.4 Quiz . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82E Survey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83ixList of Tables3.1 Consequences of in-class multitasking affecting students’ learn-ing outcomes. . . . . . . . . . . . . . . . . . . . . . . . . . . . 22xList of Figures3.1 Snapshot of a diary template . . . . . . . . . . . . . . . . . . 123.2 Medium and time information for the multitasking entries instudents’ diaries. . . . . . . . . . . . . . . . . . . . . . . . . . 143.3 Students’ perceptions of in-class multitasking. . . . . . . . . . 164.1 Findings, requirements, and design elements . . . . . . . . . . 274.2 An overview of the cognitive design elements prototyped inTime-Turner. . . . . . . . . . . . . . . . . . . . . . . . . . . . 314.3 Pause, skip, scrub: Three ways to switch modes . . . . . . . . 334.4 Augmented progress bar in Time-Turner . . . . . . . . . . . . 344.5 Guided accelerated viewing . . . . . . . . . . . . . . . . . . . 354.6 Interaction flowchart . . . . . . . . . . . . . . . . . . . . . . . 364.7 Topic-based cognitive offloading . . . . . . . . . . . . . . . . . 394.8 Time-based cognitive offloading . . . . . . . . . . . . . . . . . 414.9 Activity-based cognitive offloading . . . . . . . . . . . . . . . 415.1 Self-perception of multitasking ability . . . . . . . . . . . . . 485.2 Participants’ perceived frequency of in-class multitasking andwatching recordings with a faster speed. . . . . . . . . . . . . 495.3 TimeTurner improves students learning outcomes when mul-titasking. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 505.4 Participants found Time-Tuner helpful for their learning. . . 515.5 Ease of use . . . . . . . . . . . . . . . . . . . . . . . . . . . . 545.6 Helpfulness of features . . . . . . . . . . . . . . . . . . . . . . 555.7 Interest in access to and use of the features. . . . . . . . . . 595.8 Usage frequency . . . . . . . . . . . . . . . . . . . . . . . . . 60xiList of FiguresA.1 Consent form, page 1 . . . . . . . . . . . . . . . . . . . . . . . 76A.2 Consent form, page 2 . . . . . . . . . . . . . . . . . . . . . . . 77B.1 Template for diary study . . . . . . . . . . . . . . . . . . . . . 78C.1 Interview script page 1 . . . . . . . . . . . . . . . . . . . . . . 79C.2 Interview script page 2 . . . . . . . . . . . . . . . . . . . . . . 80D.1 Reading Comprehension Question from Task . . . . . . . . . 81D.2 Post-lecture Quiz . . . . . . . . . . . . . . . . . . . . . . . . . 82E.1 Survey for Evaluating Metacognitive Design Elements . . . . 83xiiChapter 1IntroductionMultitasking is defined as interleaving multiple tasks at the same time frameand switching between them [1]. Improving efficiency is one of the main mo-tivations behind adopting multitasking in day-to-day activities [19]. There-fore, multitasking is an extremely common practice among university stu-dents who struggle in tackling their high workloads and time constraints.Research shows that university students engage in a notable amount of mul-titasking when attending their classes [21]. Furthermore, in-class multi-tasking is reported to be significantly greater in online classes compared totraditional in-person classes [2, 29].While many students engage in in-class multitasking aiming to increasetheir productivity, it does not come without its own disadvantages. There isa large body of research proving the negative effects of in-class multitaskingon students’ learning outcomes and performance [14, 28]. Several theories ofattention such as the bottleneck theory, the information processing theory,and the scattered attention hypothesis support and explain these findingsby presenting humans’ attention and cognitive capacity as a limited resource[43]. With the increased transition to online synchronous delivery of univer-sity lectures, the adverse effects of in-class multitasking in online classes arebecoming a more serious issue.The potential negative consequences of multitasking are recognized bymost students [34]. However, students still continue to multitask in theirclasses. This shows that students find that the benefits of multitaskingoutweigh the detriments in certain situations. It is important to note thatsome in-class multitasking can lead to more productivity for the studentsor simply have a higher priority at the moment. Therefore, eliminating in-class multitasking completely is not a reasonable solution for reducing the1Chapter 1. Introductionnegative effects of in-class multitasking on learning outcomes.By acknowledging the potential benefits of in-class multitasking for learn-ers and its inevitable nature, we propose to support in-class multitaskingby mitigating its adverse effects on students’ academic performance. Inthis thesis, we explore students’ motivations, perceptions, and challenges inmultitasking in online synchronous classes. Based on our findings, we elicitrequirements and come up with design elements that facilitate students’in-class multitasking and measure their effectiveness in an evaluative userstudy.We conducted a small formative study with 10 university students whowere enrolled in classes that were delivered synchronously online. The goalof this formative study was to gain insights into students’ motivations, per-ceptions, and challenges in in-class multitasking and organize these insightsas the first step of our design process. This study was conducted in theformat of a diary study to help students track and reflect on their in-classmultitasking habits followed by a semi-structured interview. To elicit ourdesign requirements, we organized the discovered consequences of in-classmultitasking into two groups: the resulting content loss and context lossleading to extra time spent after the class fall under the cognitive conse-quences of in-class multitasking, and the inhibition of the learner’s time andattention management fall under the metacognitive consequences of in-classmultitasking. Our findings motivate us to support in-class multitasking inorder to mitigate its consequences for students’ learning while preserving itsbenefits.To address the cognitive consequences of in-class multitasking we defineour first requirement R1. Enable immediate recovery of missed content inthe class time frame.. To satisfy R1, we propose a bichronous learning [33]environment that enables accelerated guided viewing of past content asyn-chronously in online synchronous classes immediately after multitasking toprevent the content loss, context loss, and the need for extra time spent onthe lecture content asynchronously. In an experimental evaluative user study(N=20) we evaluated the proposed cognitive design elements implementedin a prototype called Time-Turner. Our results show that compared to21.1. Research Questionsa baseline system, Time-Turner shows significant improvement in students’learning outcomes when they multitask in online synchronous classes. Qual-itative feedback from our participants showed high levels of interest in accessto these design elements in online synchronous classes.To address the metacognitive consequences of in-class multitasking, weelicit our second requirement, R2. Provide assistance in metacognitive mon-itoring in in-class multitasking by enabling cognitive offloading.. To satisfyR2, we propose time, topic, and activity-based cognitive offloading thatassist with metacognitive monitoring when multitasking. Integrating lec-ture slides with the live class provides topic sections that can be used asbookmarks. Interactive components of the lecture such as discussions andQ&As are also presented as sensitive events that can trigger an alert andbring students back to the live class. Finally, time-based cognitive offload-ing allows students to plan their multitasking and receive aid in regulatingtheir actions accordingly. Our qualitative evaluation of introducing con-tent and time-based cognitive offloading to improve students’ self-regulationin in-class multitasking shows evidence that supports their effectiveness inpossible scenarios.This thesis reports the process and output of a need-finding study, fol-lowed by a design process and an experimental study for its evaluation.Furthermore, based on the discovered potentials of the design, we providediscussion points and future directions for further enhancing students’ learn-ing experience when engaged in in-class multitasking.1.1 Research QuestionsOur research begins by exploring students’ attitudes towards multitaskingand discovering their needs and challenges in in-class multitasking.• RQ1.a: What are students’ motivations and perceptions of multitask-ing in online synchronous classes?• RQ1.b: What are students’ challenges when multitasking in onlinesynchronous classes?31.2. ContributionsBased on our findings in the need-finding study, we aim to facilitatein-class multitasking by addressing students’ challenges when attempting it.• RQ2: What are design elements that can mitigate the consequencesof in-class multitasking for learning while maintaining its potentialbenefits for students?1.2 ContributionsThe main contribution of this thesis is:• Empirically proven design elements that effectively mitigatethe adverse effects of in-class multitasking on learning out-comes: A novel bichronous learning environment that provide real-time asynchronous access to past lecture content, and demonstratesits statistically significant positive effect on learning outcomes whenmultitasking in the class.Additionally, we provide the following contributions.• Creating Time-Turner, a proof of concept prototype for eval-uating the proposed design elements: We create a proof of con-cept prototype with features that comply with our proposed designelements and use the prototype in our evaluative studies.• Empirical evidence on demand for technological support forstudent multitasking in online synchronous classes: Our find-ings from the formative study led us towards designing to facilitatein-class multitasking instead of designing to eliminate it by provid-ing a categorization of the consequences of in-class multitasking forstudents and eliciting requirements accordingly.• Supplementary design elements to address the metacognitiveconsequences of in-class multitasking: We propose topic, activity,and time-based cognitive offloading to help students in regulating theiractions and collected qualitative feedback on these elements.41.3. Thesis Structure1.3 Thesis StructureThe remainder of this thesis is structured as follows. Chapter 2 providesa literature review around in-class multitasking and its effects on learning,positive and productive multitasking, and the existing designs and interven-tions for multitasking. Chapter 3 describes the methodology and findings inour preliminary need-finding study setting up the motivations for our design.Chapter 4 presents the process of deriving requirements and design elementsfrom our prior findings and the implementation of a proof of concept pro-totype called “Time-Turner”. Chapter 5 presents the experiment design forevaluation purposes consisting of methodology and results. Chapter 6 pro-vides discussion points around our approach, mentions the limitations andfuture directions, and concludes our work by presenting a summary of ourcontributions.5Chapter 2Related Work2.1 In-class MultitaskingIn the context of learning, multitasking can be defined as dividing attentionbetween two or more tasks and performing non-sequential task switching in alearning situation [22]. In this thesis, we are focusing on in-class multitaskingwhich is limiting the learning situation to a lecture or class.Studies show that students engage in a significant amount of in-classmultitasking. Using a combination of observation and surveys among 3000students, Ragan et al. [39] showed that students engage in off-task computeractivities in the class almost two-thirds of the time. In another study byFried [16], in-class laptop users reported multitasking for 17 out of 75 min-utes of the class. Of those students 81% reported that they checked emailduring the lectures, 68% reported that they used instant messaging, 43%reported surfing the net, 25% reported playing games, and 35% reporteddoing “other” activities.Looking into the literature, we find strong evidence for the negativeimpacts of in-class multitasking on students’ learning and academic perfor-mance. A study by Kuznekoff et al. [26] shows the effects of mobile in-classmultitasking on learning across three groups; control, low-distraction, andhigh-distraction. The participants in all groups watched a video lectureand took notes. Their learning was measured in two assessments after theywatched the videos. A significant difference was observed between the con-trol and high-distraction groups with the students in the control group writ-ing down 62% more information in their notes and scoring 1.5 letter gradeshigher. In a similar study Macdonald [35] show that there is a negative cor-relation between students’ in-class texting and their final grade and GPA.62.2. Positive/productive multitaskingIn a different study by Conard et al., [10], interrupting students with textmessages in the class led to a significant margin in learning that was affectedbut not fully moderated by the student’s interests in the topic. Consistentwith the previous studies, Brooks [8] found that personal social media usagenegatively affects task performance.Laptop multitasking has similarly been researched and shown to be re-ducing students’ learning outcomes. Hembrooke et al. [18] conducted acomparison study between in-class laptop users and non-users. Lap-top usereduced students’ recall and recognition scores on the lecture content. Whenit comes to online classes specifically, research shows that students engagein a significantly larger amount of multitasking in online classes comparedto in-person classes [29].Analyzing the vast research on this topic, we observe that their findingsall point to in-class multitasking being detrimental to students’ learningwith much of a focus on text messaging and social media usage as secondarytasks done in the classroom. However, looking at the substantial presence ofmultitasking in students’ habits, it is imperative that we look into students’perceptions of multitasking and its effects on their performance.Elder [13] shows that students believe that cellphone use in college classesinterferes with their learning and concentration. However, these beliefs donot prevent them from multitasking. Another study shows that studentswere able to accurately predict their decline in learning in in-class textingconditions compared to non-texting conditions [17]. This further confirmsthat students are aware of how their in-class multitasking affects their aca-demic outcomes to an extent.This leads us to ask this question: If students are indeed aware of theadverse effects of in-class multitasking, why do they persist in engaging init?2.2 Positive/productive multitaskingSzumowska et al. [42] research the origins and motivations behind multi-tasking. In their experiments, they show that the frequency of multitasking72.2. Positive/productive multitaskingcorrelates with the number of active goals that people pursue. They alsodemonstrate how the relative importance of goals makes a difference in mul-titasking where the degree of multitasking decreased in a condition with onegoal being significantly more important than other goals compared to allgoals being equally important. To confirm their results, they experimentedin a learning setting with 138 university students and measured their mo-bile in-class multitasking and the relative importance of the learning goaland showed that the greater the importance of the learning goal, the fewerstudents engaged in mobile multitasking.If students are engaging in a lot of in-class multitasking that means thatthe relative importance of learning in the class is not higher than their otheractive goals, either because of the increase in other goal’s importance, or adecrease in the learning goal’s importance for the student.It is worthwhile to consider that depending on the student’s active goalsand their importance, in-class multitasking can be beneficial to them overall.Poplawska et al. [38] suggest that many people use media multitasking ineducational or occupational domains to satisfy their professional, social, andemotional goals. Depending on what goals we use as reference and whatconsequences we analyze, media multitasking can be a beneficial strategicbehavior to achieve goals or a shortcoming in self-regulation.Even though the existing research on multitasking in the domain of ed-ucation is mostly pointing toward its effects on reducing performance, somestudies show the potential benefits of multitasking in online workplace meet-ings. Cao et al. [9] perform a large-scale analysis of multitasking in remoteonline meetings. They suggest that multitasking in online meetings can leadto a boost in productivity. Other studies found that the benefits of mul-titasking are part of a trade-off. This means that while multitasking canhelp increase productivity, it can bring side effects such as stress and therequirement for extra effort [44].While there is no doubt that in-class multitasking impairs students learn-ing and academic performance, taking up a narrow view and focusing onlearning goals as the only outcome is not sufficient. It is important to un-derstand that students pursue multiple goals of equal importance and even82.3. Existing Designs for Multitaskingthough in-class multitasking might affect their learning goals adversely, theirincreased productivity in their other tasks can make it worthwhile for them.2.3 Existing Designs for MultitaskingThere are many interventions and solutions for self-control and self-regulationthat can be used to reduce students’ multitasking. In a review of digitalself-control tools, Lyngs et al. [31] divides these interventions into 4 maincategories, block/removal of distractions, self-tracking of activities, goal ad-vancement, and reward/punishment. An example of self-tracking interven-tions, Lottridge et al. [30] designed a browser plug-in that divides browsertabs into work and non-work and tracks the time spent on each category anddisplays it to the user to reduce time spent on non-work tasks. PomodoLock[25] is an example of a blocker intervention that blocks interruption sourcesacross multiple devices to avoid self-interruption.In a completely different approach in a recent study, Son et al. [41]embrace the inevitability of multitasking and accept distractions as a naturalcomponent of online meetings. They prove the effectiveness of real-timetranscription for facilitating catching up in multitasking in online meetingsqualitatively and quantitatively.In this literature review, we discovered that students tend to multitask intheir online classes despite being aware of its negative effects on their learn-ing. Observing in-class multitasking as an inevitable part of the student’slearning experience and considering its potential benefits, we searched forexisting work on supporting students who engage in in-class multitasking.However, there is a gap in the existing research on facilitating multitaskingin online classes and assisting students in mitigating its adverse effects onlearning outcomes.9Chapter 3Exploring Students’ In-classMultitaskingIn this chapter, we address the first research question, RQ1. What arestudents’ motivations, perceptions, and challenges in multitasking in onlinesynchronous classes?. Most of the existing research on in-class multitaskingviews learning as the reference goal at all times. However, observing thelarge amount of multitasking that occurs in college and university classes,especially in online synchronous classes, we believe that there are otherfactors in play. We are interested in finding out what students think abouttheir in-class multitasking habits. If they are already aware of the negativeeffects of in-class multitasking on their learning, why do they still engagein it, and finally what are the challenges and problems they face when theymultitask in their classes?The user study in this chapter is a preliminary step to learn more aboutstudents’ needs and propose solutions to improve their learning experiencewhen they engage in in-class multitasking in online synchronous classes.Collecting and organizing the insights gained in this formative study is anessential part of our design process. However, we recognize the limitationsof this formative study and do not make any claims on the generalizabilityof these findings.3.1 MethodTo answer these questions, we designed a formative study with 10 universitystudents who were taking online synchronous classes. We used diary studiesto learn more about the nature of students’ multitasking and allow them103.1. Methodto reflect on it, followed by semi-structured interviews to dig deeper intostudents’ opinions about in-class multitasking and how it can be improved.3.1.1 ParticipantsWe recruited 10 participants (3 male, 7 female) between the ages of 18-24.Our inclusion criteria required participants to be current undergraduate orgraduate university students. Of our participants, 6 were undergraduate and4 were graduate students enrolled in different programs such as Engineering,Arts, and History. All 10 participated in at least one session of the diarystudy and 8 out of 10 participants agreed to have a follow-up semi-structuredinterview with the investigator.3.1.2 ProcedureWhile the research in in-class multitasking reports the amount of mobile orlaptop multitasking students do in their classes, there is little to no mentionof why students engage in those activities during the class. To find out, wedesigned a diary study template for students to fill out during their onlinesynchronous classes. The benefit of using a diary study in this situation isits high ecological validity [11] and more reliable data collection because ofthe in situ nature of recording data. However, there are certain drawbacksto this method. Keeping diaries can disrupt students’ normal flow of in-classactivities. We understand that in order to observe students’ in-class multi-tasking behaviors, we are introducing another form of in-class multitaskingto their actions by asking them to keep a diary simultaneously. Despite this,we believe that obtaining reliable and ecologically valid data is beneficial toour understanding of in-class multitasking and therefore find this approachreasonable.The students were given a diary template to fill out during their onlinesynchronous class. They were asked to record all the tasks they did duringthe class that were not a part of the main class activity in the diary template.For example, if the instructor asks the students to fill out a survey form,they are not supposed to record that as an in-class multitasking instance.113.1. MethodFigure 3.1: An example of a diary template filled by one of our participants inone session of an online class.For each task, the students were asked to write the title, the device theyused for performing the task (mobile, laptop, notebook, ...), why they choseto do that task, and their time estimate for how long they spent on the task(less than 1 minute, 1-5 minutes, more than 5 minutes).To collect data on students’ perceptions of their multitasking in retro-spect, the students were asked to answer 3 Likert-scale questions after theclass ends. The questions were about how related their activity was to theclass, how satisfied they are with choosing to do this activity during theclass, and if they find the benefits of doing the task in the class outweighingits detriments. Figure 3.1 shows a snapshot of the diary template given tothe students in our diary study.The students were asked to keep a multitasking diary for 2 or 3 sessionsof their online synchronous classes. This gave us some insight into students’multitasking activities and their perceptions and also allowed them to gainmore self-reflection in terms of how often and how long they multitask intheir online classes. This provided a good ground for discovering more inthe follow-up interviews.After 2 or 3 sessions of participation in the diary study, the partici-pants were invited to a remote interview session via Zoom. The interviewswere held in a semi-structured manner where the interviewer reviewed the123.2. Insightsstudents’ diary entries with them, and asked clarifying questions about in-teresting or intriguing entries. The participants are asked questions abouttheir in-class multitasking habits, their perceptions of their in-class multi-tasking activities, and the challenges they faced when trying to multitask intheir classes.3.2 Insights3.2.1 Insights from the Diary StudyDiary EntriesIn our diary study, 10 students kept multitasking diaries in 22 sessions ofonline synchronous classes. The session lengths were between 1-3 hours.Overall 106 diary entries were recorded across these 22 sessions. On average,each participant had 5.08 tasks entered per session with a minimum of 2tasks/session and a maximum of 8.5 tasks/session. Of the 106 multitaskingdiary entries, 22% were estimated to take longer than 5 minutes, 38% wereestimated to take between 1-5 minutes and 40% were estimated to take lessthan 1 minute to finish. In terms of device and medium for multitasking, outof 106 diary entries, 43 entries (41%) were done on a computer, 21 entries(20%) were done on a phone, 35 (33%) used neither computer nor phone(e.g. talking to my sister). Finally, 7 entries (7%) did not specify whichmedium they used (e.g. got a reminder on my calendar so went to check it).See Figure 3.2. In 75 out of 106 diary entries (75%), students disagreed toan extent with the statement “This activity was related to the ongoing classcontent.”.Reasons for In-class MultitaskingThe students were asked to write down why they decided to do that task atthat time for each diary entry. To discover common themes among students’reasons, we performed reflexive thematic analysis [6] on the diary entries anddiscovered the following themes.133.2. InsightsFigure 3.2: Medium and time information for the multitasking entries in students’diaries.High importance or urgency of task (T1): One of the main themesobserved in students’ multitasking entries was an emphasis on the high im-portance of their task. Students used words such as “urgent”, “necessary”,“needed to”, and “had to” to express the importance of the task being themain reason behind doing it. The reasons for 26 entries ( 25%) fell underthis theme.Low importance of learning content (T2): Students put down thelow importance of the learning goal as the reason for their multitasking inmany instances. This theme was observed in 20 entries (19%). Studentsexpressed their reasons for expressing the class or the topic as “slow”, “bor-ing”, “not interested”, “I already knew the content”, etc.External interruptions (T3): External interruptions are interrup-tions that are caused externally[23]. These types of interruptions were an-other main culprit for students’ in-class multitasking in 15% of the diaryentries. Students reported receiving notifications, getting calls, and otherpeople speaking to them as their reasons for multitasking under this theme.Self-interruptions (T4): Self-interruptions are defined as interrup-tions and are initiated internally [23]. In 16% of the diary entries, themain reason for students’ multitasking fell under the self-interruption theme.Some examples of self-interruptions in students’ diaries are expecting a mes-143.2. Insightssage or an event so checking to see if they received any (e.g. “I was curiousif I had gotten any new emails.”), mental or physical fatigue, hunger, andthirst.It is important to mention that these themes are not mutually exclusive.It is very likely that students had more than one of these factors drivingthem to multitask during their online classes. However, it is expected thatthe students wrote the most prominent reason for their multitasking, andtherefore our analysis is based on students’ emphasis. This can explain thereason behind having two separate themes T1 and T2 instead of combiningthem. We separated our entries based on the high importance of the task(in T1) and the low importance of the learning goal (in T2) based on thestudents’ emphasis in their answers.While external and internal interruptions (T3 and T4) were written inthe multitasking diaries as reasons for students’ multitasking, they can beconsidered reasons in a different level compared to the relative importanceof tasks (T1 and T2). For each multitasking activity to occur in the class-room, it requires a trigger. This trigger can be external (e.g. an emailnotification), or internal (e.g. student remembering the need to reply toan email). An interruption is an immediate and necessary cause for multi-tasking. However, an interruption alone does not necessarily lead to in-classmultitasking. For example, a student might receive a notification but decideto ignore it because the lecture content is more important to them. There-fore, the perceived importance of the task and the learning goal works handin hand with the internal and external interruptions to determine whetheror not the student engages in in-class multitasking.Perceptions of MultitaskingStudents are mostly aware of the adverse effects of in-class multitasking onlearning. However, we still observe a significant amount of multitasking inonline classes. It is possible that students are not being considerate of theconsequences of multitasking at the moment and multitask in a lapse ofjudgment. However, it is also possible that students choose to multitask153.2. InsightsFigure 3.3: For each multitasking entry in their diary students had to answertwo Likert-scale questions (A, B in the figure). Their answers shown above tell usthat they are mostly satisfied with choosing to multitask during the class and findit beneficial in the majority of cases.because they find it beneficial for them despite being aware of its potentialdetriments.In our diary study, we asked students to answer these two questions ona 7-point Likert scale (1 strongly disagree - 7 strongly agree) at the end ofthe class.• 1. I am satisfied with choosing to do this activity during the class.• 2. I believe the benefits of doing this activity during the class, out-weighed its detriments.In Figure 3.3 you can see the results for student’s answers to this ques-tion. In 70.76% of the entries, the student had agreed to statement 1 I amsatisfied with choosing to do this activity during the class. to an extent,meaning that they had chosen somewhat agree, agree, or strongly agree. In62.26% of the entries, the students had agreed to statement 2 I believe thebenefits of doing this activity during the class, outweighed its detriments. toan extent. These numbers show that even when reviewing their multitask-ing after the class and in retrospect, the participants found the majority oftheir multitasking beneficial.163.2. Insights3.2.2 Insights from the InterviewIn our analysis of students’ multitasking diaries, we found interesting resultsfor why our participants choose to multitask in class and what their per-ceptions of their multitasking activities look like. In order to gain a morecomprehensive answer to these questions, we conducted semi-structured in-terviews with 8 of the 10 original participants. In addition, we asked aboutwhat challenges students face with in-class multitasking in their online syn-chronous classes.We believe that after filling out the diaries, the participants are moreaware of their in-class multitasking habits and therefore able to provide morereliable and thorough answers to these questions.Why do students multitask in online synchronous classes?Our interviews confirm that the one main factor leading to students’ in-class multitasking is the high perceived importance of the task and/or thelow perceived importance of the learning goal. Our participants went intomore detail in their reasons for perceiving the task as more important or thelearning goal as less important to further support this point.A. They are not learning effectively. Lecture content was the mostagreed-upon factor influencing students’ multitasking. All eight participantsin the interview mentioned lecture content as a reason for their multitasking.Confirming our insights in the diary studies, students provided more detailsfor why they found the lecture content and therefore the learning goal ofless importance and therefore chose to multitask.Our participants found the learning goal less important when they be-lieved that they were not learning effectively. This can have multiple reasons.One of the main reasons is students’ prior knowledge of the topic. For exam-ple, a participant who was a graduate student mentioned that they decidedto multitask in undergraduate-level courses because they already knew thematerial.“I don’t miss anything because I know the content and it’s easy for me”.173.2. InsightsThis factor goes hand-in-hand with students’ wrong choice of course se-lection. Three out of eight participants mentioned that they had to makethose wrong selections because of required credits for their program or othersimilar reasons.While the class content being too easy or the lecture pace being tooslow was one of the main reasons for students’ in-class multitasking, a highlevel of difficulty was also mentioned as another reason. One participantmentioned this “I couldn’t understand the topic during the class and it wastoo hard and I couldn’t catch up. So I gave up and stopped listening anddid something else.”. If the student cannot learn from the lecture contentbecause they find it too difficult or they think the learning pace is too slowand not efficient, it is most likely that they choose to do other tasks becausethey do not find their learning effective.Our participants also mentioned that their interest in the topic beingdiscussed played an important role. “Depending on the content, if the topicdoes not apply to my situation and isn’t interesting to me I won’t listen tothose and choose to do some other thing.”. This can show us that studentsseek autonomy and control over what they want to listen to and learn andwhat they do not. Therefore, they evaluate the importance of the topicsbeing discussed and choose their actions accordingly. “A lot of professorsgo on tangents and that’s not related to the main goal of the class so theystray from the main goal and make me feel like it’s not important and I payless attention”.A common reason discussed for assigning lower importance to the classcontent and choosing to multitask was peer activity. Six out of eight par-ticipants talked about how they assign a lower priority and importance totheir peer’s activities in the class compared to the instructor’s teaching andtherefore choose to multitask. Three out of eight participants reported thatthey usually multitask when other students are asking questions.“Sometimesstudents ask questions that I find irrelevant and something that I’m not in-terested in.”. Another activity mentioned by two participants was classdiscussions.“[I multitask more] When it’s open-discussion vs actual instruc-tion”. Overall it seems like some students do not find their peer input in the183.2. Insightslecture as important as the teaching done by the instructor and multitaskmore during activities such as Q&A sessions and discussion groups. Thisprovides interesting implications for teaching as these methods are presumedto increase students’ engagement and decrease their distraction. However,it is apparent that in some cases they are not successful and lead to evenmore in-class multitasking.B.They have more/more important commitments.Seven out of all eight participants mentioned that when there are morecommitments and a busier schedule, they tend to multitask more.“It is Apriland I have a very busy schedule so that happens more often. So the busieryou are more you multitask”. University students tend to have a lot ofresponsibilities and time constraints. Close deadlines and time shortageswill increase the priority and importance of the tasks and lead to morein-class multitasking.Even without considering time constraints, some tasks are considered“necessary” by students and therefore they choose to do them during theclass. Two participants talked about situations where they chose to do amore important task over learning in the class. “ Sometimes I have to finishsomething else. I want to listen but I have to work on it. The deadline ismore important and I can catch up on the class.”, “They were things thatwere necessary. For those things, I don’t care what is happening in the class.I prefer to do those things”.C. They get distracted.Habitual internal interruptions are a main factor that leads to in-classmultitasking. One participant mentioned that even when the class contentis interesting and engaging, they still check their phone at least twice duringthe class out of habit.“ It’s inevitable for me to not check my phone so Ithink that would happen at some point.”. Another participant mentions thatinternal interruptions are more common when they use their phone in class.External interruptions such as notifications are another important triggerfor multitasking. Additionally, the environment in which students attend193.2. Insightstheir online classes seems to affect their multitasking habits. One studentmentioned that they multitask more when they attend their class from theirhome compared to when they are in their research lab at the university andattend their online class. “.. at home there’s a lot more that I can do. Moreopportunities to multitask at home which could be a good thing or a bad thingdepending on what I’m doing.”. The learning environment can increase bothinternal and external interruptions. For example, at home, students havemore tasks to do and might be distracted by external factors such as familymembers or friends speaking to them.What is students’ perception of multitasking in onlinesynchronous classes?A. In-class multitasking can save time.Six out of eight participants believed that multitasking can help them getmore things done and save time as a result. Students emphasized that whenthey are not learning much from the lecture, being able to do something elseat the same time helps them use their time more efficiently. P1: “..I willuse my time in an efficient manner rather than just listening to the subjectthat I already know about.” .Students found more opportunities to multitask during online classescompared to online classes and believed that they could take advantage ofin-class multitasking more. Social norms and classroom etiquette make mul-titasking seem inappropriate and even rude in traditional in-person classes.In online classes, the ability to turn your video and audio off makes it possi-ble to be able to engage in other activities more freely.“In in-person classesI don’t feel comfortable or appropriate to check my phone when I’m wait-ing for my classmates to finish their work but at home it’s easier.”. If thestudents choose to use their time better, online classes can provide a greatopportunity for more productivity during class time. P2: “I prefer online,because, for a class like this, I think if I had to do it in person I would loseso many hours [by not multitasking]”203.2. InsightsB. In-class multitasking can help students concentrate better.Two participants mentioned how doing a second task when attendingtheir online lectures will help them focus better and get less distracted.“Long and discussion-based classes make it very hard for me to pay atten-tion, so when I’m multitasking I am able to pay attention better because Iam stimulated in other ways rather than just auditory stimulation.”C. In-class multitasking can cause learning overhead.Some participants thought that in-person classes allowed for better focusand learning because it was more difficult to multitask in them. Being able tomultitask more during online classes can distract students from the learninggoal and lead to lower learning outcomes. Two out of eight participants saidthat they would prefer it if they multitasked less often during their onlineclasses and paid more attention to the lecture.“I think before the pandemic in in-person classes, it was better learningbecause I didn’t do other things during the class.”.Multitasking affects students’ learning negatively and requires them tospend more time after class to learn the content they missed. P1: “If I wasmultitasking in a class that was very new and I needed to know the topic,I would need to spend more time after the class to learn it or ask the profor classmates.”. So even though students might choose to multitask to savetime, they need to spend extra time learning the content that they missedfrom the lecture.D. In-class multitasking can lead to fatigue.Two participants mentioned that multitasking sometimes makes themfeel mentally tired. Doing another task while trying to listen to the class atthe same time can make students feel a lot of pressure and become mentallyfatigued.P1: “Sometimes [multitasking] makes me exhausted because I need to do toomuch and use my brain too much.”P10: “Sometimes I feel like too much is going on and [I am] overextendingmyself by dividing my time between everything. ”213.2. InsightsType Consequence DescriptionCognitiveIn-class multitaskingleads to contentloss.Students miss content from the lecture whenthey are distracted with doing another task.In-class multitaskingleads to contextloss.Students have a hard time learning the fol-lowing content after they switch back to thelearning goal from their second task. .Recovering fromin-class multitaskingrequires extra timeafter the class.Students need to watch lecture recordings orstudy the missed material to compensate fortheir multitasking. This counters increased ef-ficiency as a benefit of multitasking.MetacognitiveIn-class multitaskinginhibits students’time manage-ment.Students are not able to execute their plannedtiming in multitasking and tend to spendlonger on tasks than they expected.In-class multitaskinginhibits student’sattention man-agement.Students have difficulty recognizing thechanges in topic and activity in the classroomwhen they multitask. As a result they tend tomiss return points to the class.Table 3.1: Consequences of in-class multitasking affecting students’ learning out-comes.What are the challenges students face when multitasking inonline synchronous classes?Based on the insights from the formative study, we divided students’ chal-lenges with in-class multitasking into two categories, cognitive and metacog-nitive consequences of multitasking. Cognitive consequences cover chal-lenges in cognition and learning and metacognitive consequences cover chal-lenges in metacognition in in-class multitasking. In Table 3.1, you can finda summary of the identified challenges and their categorization.A. Cognitive Consequences of In-class MultitaskingAccording to the cognitive load theory, human beings have a limitedcognitive capacity in working memory[12]. When engaged in more than onetask at a time our ability to learn and process is lower compared to doing223.2. Insightsa single task due to higher cognitive load. Based on our insights, students’main challenge with in-class multitasking, is its cognitive consequences. Thismeans that dividing their mind between learning and another task leads tolower performance in both. P9 talks about how they cannot give “100%” toeither thing they are doing when they multitask in class. P3 mentions thateven though sometimes they have to work on other courses in the class, theyare not able to do any of the tasks well.Another problem comes from switching between the learning goal andan unrelated task being difficult for students. Students have a hard timecatching up to the lecture content when they come back and find their mindsoccupied with other tasks. P8 talks about their experience switching backto the class from another task: “When you switch your attention to anothertask and then go back to the class, I lose focus and find it difficult to con-centrate. Since the topics are very different it’s hard to switch attention.”.With students’ attention and mind divided between two tasks, not onlydo they miss content from the lecture, but they also have worse performancein the other task because they want to pay attention to the class at the sametime. P10 talks about this: “ I can’t do it because 1. I feel like I can’t focuson it and 2. I feel like I should be focusing on my class.”. While the actof missing class content hinders students’ learning, their fear of missing outmakes their multitasking even more challenging and does not allow them toperform their other tasks to the best of their ability. P9 said: “[the problemis] Feeling like I’m missing out on the class even though I chose to multitaskbecause I felt like I wouldn’t be missing out.”.To address this challenge in their multitasking, students spend extra timeafter class to watch lecture recordings. P2, P3, and P8 mentioned watchinglecture recordings as a way to recover missed information. P2 mentionedthat she prefers to watch the recordings because she watches them with afaster speed and is more time-efficient, “ I prefer to watch the recordingswith faster speed and save time”.B. Metacognitive Consequences of In-class MultitaskingWhile in-class multitasking affects cognition to a great extent, it also233.3. Summaryinfluences students’ metacognitive abilities. Too much distraction does notallow for reliable planning and self-regulation. Students often find them-selves not being able to stick to their plans for multitasking. Our partici-pants reported that they sometimes spend too long on their tasks and forgetto get back to class in time. “ .. and you think you’ve been playing 1 or2 minutes but in reality 10 minutes have passed.”. Students have difficultywith task and attention management when they are distracted doing a task.Task wandering is another consequence of students’ lack of metacognitiveregulation. The student decides to leave the class for another task but endsup getting more distracted and doing more unplanned tasks. P9 said: “OnceI get distracted and check one email and then read another email and beforeI know it I’m not listening to the class anymore.”Students reported that they use auditory cues from the classroom thatsignal a change in topic or activity to know when to switch from doing an-other task back to the class. However, sometimes they miss those cues andare unable to get back to the class on time. P3 said: “If I’m not completelydistracted if I hear key words like: “OK, let’s move on” I go back to the classbut sometimes I miss that and I realize that the part that wasn’t interestingto me is done.”.3.3 SummaryThe main goal behind our preliminary study was to gain insights that driveour design process. There are several key takeaways in our results that di-rectly influenced our design in the next chapter. First, we observed thatparticipants were satisfied with choosing to multitask in their online classesand found them beneficial most of the time. We also observed that our par-ticipants were motivated to multitask based on the relative importance oftheir goals. Finally, by multitasking in their online classes, they faced conse-quences for their learning which we categorized into cognitive and metacog-nitive consequences. In Chapter 4, we discuss how our design requirementsare derived based on these insights.24Chapter 4DesignIn the previous chapter, we learned that students multitask in class whenthey find their other goal more important than their learning goal and findit beneficial. However, learning remains one of the main active goals that astudent is pursuing, and in-class multitasking can be seriously detrimental tostudents’ learning outcomes. In-class multitasking affects students’ learningnegatively and requires students to spend more time after class to learn thecontent they missed. Therefore, our goal in this chapter is to come up withdesign solutions that support in-class multitasking by mitigating itsadverse effects on students’ learning instead of eliminating it.To achieve this, we review our insights from the preliminary study. Wecategorized students’ perceived challenges in in-class multitasking into thefollowing categories, as seen in table 3.1.• Cognitive Consequences: Content loss and context loss are miti-gated by the student spending extra time after the class on the learningcontent.• Metacognitive Consequences: Poor time and attention manage-ment due to lower metacognitive monitoring preventing a timely returnto the learning task.In this chapter, our goal is to answer RQ2: What are design elementsthat can mitigate the consequences of in-class multitasking for learning whilemaintaining its potential benefits for students?Our approach is to elicit design requirements based on the identifiedconsequences for learning and come up with design elements that bring uscloser to our goals of supporting in-class multitasking. In the following254.1. Designing for Cognitive Consequences of In-class Multitaskingsections, we describe the rationale and the outputs of the design process.We also describe an implementation of the proposed design elements in aproof-of-concept prototype for the evaluation process in Chapter 5.4.1 Designing for Cognitive Consequences ofIn-class Multitasking4.1.1 Design Requirements and RationaleBased on our insights from the formative study, students find content lossto be a serious challenge with multitasking and that it leads to lower perfor-mance in learning and secondary tasks. This is consistent with the findings ofBellur et al. [4] on how in-class multitasking (synchronous) is different fromhomework multitasking (asynchronous) in the sense that in synchronousmultitasking students often miss critical information and do not know whatthey missed.In online synchronous classes, when students miss a part of the lecturedue to multitasking, they sometimes make up for it by watching lecturerecordings after the class. However, watching recordings after class is notan ideal solution. University students are tight on time as they are andthat is one of their main reasons for multitasking in their online classes. Sospending extra time watching the lecture once more counters their effortsto save time by multitasking. Another problem with this approach is thatmany students simply do not do it. If students were distracted with anothertask for a part of the class it is likely that they do not remember the timeinformation for that section later. Most students will not bother with goingthrough the recording to find a short section that they might have missed.It is worth mentioning that missing a part of the lecture content whenyou are away from the class is not the only penalty for in-class multitasking.When students switch back from doing that task to listening to the lecture,they often have difficulty catching up as they are missing context and there-fore might not learn the following content very well. As an example, it isvery possible that a student leaves the class for 5 minutes to check their264.1. Designing for Cognitive Consequences of In-class MultitaskingFigure 4.1: Based on our insights from the preliminary study on students’ percep-tions, motivations, and challenges in in-class multitasking, we define our overarchingrequirement as supporting in-class multitasking instead of eliminating it. Based onour categorization of student’s challenges into cognitive and metacognitive conse-quences, we elicit R1 and R2 as our design requirements and define D1-D3 as ourcognitive design elements and D4-D6 as our metacognitive design elements.emails but ends up not learning 10 minutes worth of the lecture contentbecause they couldn’t make sense of the following content when they gotback to the class because of their lacking context.To address these problems, we define our main requirement asR1. Enable recovery of missed content and context in the timeframe of the class.In conventional online synchronous classes, immediately going back to watcha section you missed is not possible. If we compare this with an easy,nchronousonline class where the student has the ability to rewind, pause or skip for-ward the video, it is apparent that providing these capabilities can providean advantage by reducing the content and context loss. To allow students torewind and replay the sections they missed in an online synchronous class,we provide real-time access to the video recording to students. This waystudents can choose to watch the past content asynchronously if they miss274.1. Designing for Cognitive Consequences of In-class Multitaskingsomething.By enabling asynchronous viewing during a synchronous class, we cre-ate a unique bichronous learning environment. Bichronous online learningis combining and blending synchronous and asynchronous modes of learn-ing [33]. Most existing research on bichronous learning blends these modesby introducing synchronous events and activities into an otherwise mostlyasynchronous learning environment (e.g. holding synchronous discussionsessions for an asynchronous online course). However, in our design, wepropose to do the opposite. We are embedding asynchronous learning com-ponents inside a mostly synchronously delivered online course to allow forthe freedom and control over content viewing that exists in asynchronousclasses while taking advantage of the regulation and interactive componentsof a synchronous online class.To allow for seamless switching between these two modes and help stu-dents minimize the content loss resulting from in-class multitasking in thetime frame of the class, students should watch the missed content with afaster playback speed. Watching the past content asynchronously at a fasterspeed allows students to catch up to the live lecture eventually and be ableto learn the new content easily as they are now updated with the contentand the context of what they had missed. To optimize the time overheadrequired for reviewing missed content and ensure timely catching up to thelive class, guidance on the watching speed and time needed to catch up tothe live class should be provided to the user.Finally, when allowing temporal freedom in a synchronous class, it isimportant for students to be aware of their temporal location relative tothe synchronous live stream of the class. Furthermore, detection and visu-alization of missed and watched content help ensure effective viewing of therequired content.4.1.2 Cognitive Design ElementsBased on the identified cognitive consequences of multitasking for learning,we defined the first design requirement as R1. Enable immediate recovery284.1. Designing for Cognitive Consequences of In-class Multitaskingof missed content in the class time frame. Based on R1, we propose a setof cognitive design elements (See Figure 4.1). In the following section, thesecognitive design elements are described.D1. Asynchronous Viewing in Synchronous SettingsResearch shows an inherent difference between synchronous (e.g. in-classmultitasking) and asynchronous (e.g. homework multitasking) settings [4].When multitasking in synchronous online classes, students often miss criticalinformation and are not able to recover the missed information until afterthe class. However, in asynchronous settings, it is easier to continue thepreliminary task from where you left off after returning from multitasking.We propose to create a unique bichronous learning environment [33] byintroducing asynchronous viewing during an otherwise fully synchronousonline class. We provide real-time access to the lecture recordings and allowstudents to rewind, pause, and skip forward in the recording as they would inan asynchronous class, immediately after they return to the class from theirmultitasking. Recovering missed content immediately will solve content lossand context loss due to in-class multitasking.D2. Visualization of Temporal InformationBellur et al. [4] also mentioned that a main challenge for students with multi-tasking in synchronous settings is not knowing what they missed. Therefore,it is important to provide information about what parts of the class havebeen missed and what parts have been watched so that students review themissed content efficiently. Furthermore, having the freedom to access pastcontent asynchronously in a live class means that the user needs to knowtheir temporal location and that of the live class at the same time.D3. Guided Accelerated Asynchronous ViewingA consequence of in-class multitasking is the need to spend extra time afterclass to compensate for its negative effects on learning. This is contraryto multitasking to save time and increase efficiency. While watching the294.1. Designing for Cognitive Consequences of In-class Multitaskingmissed content normally will solve the problem of context loss as opposed towatching the content after the class, it will still require extra time and meansthat the students’ in-class multitasking might not end up increasing theirefficiency. Accelerated asynchronous viewing is a good solution to ensurethat students recover the missed content in the class and catch up to thelive lecture eventually. Students are able to watch content with a fasterplayback speed set to a default value of x1.5 at the start. The user is able toadjust the speed based on the content and their comfort to maximize theirefficiency.However, choosing the right playback speed might present a new chal-lenge for students if they wish to be caught up to the lecture before a targettime. We define a target time as a point in the lecture when the studentanticipates an event that requires them to be caught up and present in thelive lecture. We anticipate that in most situations students want to set thetarget time to the estimated end of the lecture. This way they would havebeen able to watch all the lecture content and do their secondary task inthe time frame of the class. Alternatively, if there are events in the classthat require students’ live participation, such as a quiz in the middle of thelecture, students can set that as their target time. Based on the set tar-get time, the interface can calculate if the user will be able to view all theremaining content before the target time with their current speed or notand guide the user in adjusting their pace to ensure the user’s synchronouspresence by the desired time.4.1.3 Time-Turner Prototype DesignWe prototyped Time-Turner as a proof of concept for the proposed designelements addressing the cognitive consequences of in-class multitasking. InFigure 4.2, you can see a snapshot of Time-Turner. It includes several addi-tional components compared to a conventional platform for online classes.In the top left corner, “LIVE” shows that the user is currently watching thelive lecture in synchronous mode as per D1 we create a bichronous learningenvironment where the students are able to switch between the two viewing304.1. Designing for Cognitive Consequences of In-class MultitaskingFigure 4.2: An overview of our cognitive design elements prototyped in Time-Turner. A) The mode label shows Recorded/Live when the student is watching thepast/real-time content respectively. B) In the recorded mode, the live class is shownin picture-in-picture view to let students be aware of the events in the live classif needed. C) A set of controls (pause, skip back, skip forward, change playbackspeed) is provided. D) The student is able to set a target time for when theyneed to be caught up to the live class. E) An augmented progress bar providestemporal information to students about the watched/missed content, and theirlocation relative to the live class.314.1. Designing for Cognitive Consequences of In-class Multitaskingmodes.Following D2, the progress bar, shows information about the synchronousheader marked as “Live” and the asynchronous header. When the user is at-tending the lecture in synchronous mode, the synchronous and asynchronousheaders overlap with each other and move at the same pace. However, oncethe user switches to the asynchronous mode, the two headers are separated.The synchronous header which represents the temporal location of the livelecture, keeps moving forward at its original pace and cannot be modifiedin any way. On the other hand, the asynchronous header which shows thetemporal location of the user can be moved to any point on the left side ofthe live header, and its moving pace can be adjusted by the user by changingthe playback speed.You can see the controls provided for navigating the video lecture in thegray bar. These controls include skip back, pause, skip forward, and adjustplayback speed. In synchronous mode, skipping forward and increasing theplayback speed are disabled as the live feed is immutable. However, theuser can skip back if they need to view missed sections for a second time orpause the lecture if they need to step away from the lecture to multitask. Inboth situations, the interface will switch to the asynchronous mode as theuser will be watching the past content relative to the live class. In additionto pressing pause and skipping backward, scrubbing on the progress bar isanother way to switch to the asynchronous learning mode. Clicking on anypoint on the progress bar on the left side of the synchronous header (in thepast) will start playing the recorded content starting from there.The progress bar also gives the user information about the missed andwatched content. The progress bar fills up in gray as the user watchesthe lecture in synchronous mode. If the user switches to another tab orwindow to multitask during the class, the progress bar will stop filling andthe user will see a gap upon their return to the class (Figure 4.4). The usercan then scrub to the beginning of the gap and watch the missed sectionasynchronously. The missed parts that are watched asynchronously are filledin a different color (green) to create a distinction between the content thatwas watched synchronously. In Figure 4.3, you can see pause, skip, and324.1. Designing for Cognitive Consequences of In-class MultitaskingFigure 4.3: Top: If the user wants to multitask off-device, they can pause thestream before switching to another task. Once they are back, they can continuewatching asynchronously from where they left off. Middle: If the user misses a shortsection of the lecture, they can skip back to view the missed content. Bottom: Ifthe user is multitasking on-device, they can leave the lecture window and once theycome back, they can see the gap showing the duration of the missed content. Theycan scrub the progress bar to the gap and watch it asynchronously.334.1. Designing for Cognitive Consequences of In-class MultitaskingFigure 4.4: The augmented progress bar in Time-Turner (enlarged), shows theusers where they are in time (asynchronous header), where the live class is in time(synchronous header), and how long it takes for the user to catch up to the liveclass. The progress bar provides information on the content that was watched inlive mode, watched in recorded mode, and was missed due to multitasking.scrub, the three ways the user can switch to the asynchronous mode andwatch the past content.To comply with D3, Time-Turner provides guidance in adjusting thewatching speed. Once the user uses one of the three methods (pause, skip,scrub) to switch to the asynchronous mode, a two-sided arrow shows upbetween the synchronous and the asynchronous headers. The label on thisarrow shows the temporal distance between these two headers based on thecurrent playback speed in the asynchronous viewing. This temporal distanceis the time it takes the user to catch up to the live lecture at the currentspeed. You can see this label in Figure 4.3. This piece of information willhelp the user make a more informed decision about their pace and adjust itbased on their ideal catch-up time.If there is a target time when the user has to be fully caught up to thelecture by that point, such as an upcoming quiz, discussion, or the end ofthe class, the user can click on the flag button on the right bottom cornerand set the flag on the right time on the progress bar. The label on top of344.1. Designing for Cognitive Consequences of In-class MultitaskingFigure 4.5: Top: The student is watching the lecture asynchronously with 1.3xspeed. With this speed, they will catch up to the live lecture 3.9 minutes laterthan the target time they had set. Time-Turner prompts them to increase theirspeed by showing the time difference in red color and blinking. Bottom: After thestudent increases the playback speed to 1.5x, they are now able to catch up to thelive lecture before the target time and the time difference shows in green.the flag shows its temporal location relative to the beginning of the class.For example, in Figure 4.2, the user expects to be caught up to the lecture10 minutes after the start of the class, so they set the flag in minute 10 onthe progress bar as indicated by the label on top. Based on the target time,Time-Turner calculates whether or not the user is able to catch up on timewith the current speed. The time difference with the target time shows upnext to the target flag as seen in Figure 4.5. For example, if the user willcatch up to the live class 2 minutes later than the target time, the differenceis shown as “+ 2 min” in red text. After getting this information the usercan increase their viewing playback speed or skip forward until the textsturn green indicating that they will catch up before the target time.In Figure 4.6, you can find a flowchart showing the interaction flow ofthe user with the prototype.354.2. Designing for Metacognitive Consequences of In-class MultitaskingFigure 4.6: Interaction flow of the user with the prototype4.2 Designing for Metacognitive Consequences ofIn-class Multitasking4.2.1 Design Requirements and RationaleResearch shows that multitasking inhibits metacognition []. Our insightsfrom the formative study support this statement as the lack of sufficientmetacognition leads to poor time and attention management. In order toaddress these challenges, we define our second design requirement asR2. Provide assistance in metacognitive monitoring in in-classmultitasking by enabling cognitive offloading. Going over our re-sults from our interviews, we can see that students’ main challenge in thiscategory comes from the flawed execution of their plans for multitasking.Students leave the class expecting to be back quickly but end up spending364.2. Designing for Metacognitive Consequences of In-class Multitaskingtoo long on their tasks. Sometimes, they don’t find the current topic usefulso decide to skip it, but because they miss when the next topic starts, theymiss some new content without meaning to. Another concern for studentsis that when they are distracted with other tasks, they might miss partici-pation opportunities in class activities such as discussions or answering theinstructor’s questions. Our proposed solution to these challenges is to enabletime, topic, and activity-based cognitive offloading.4.2.2 Designing for Metacognitive ConsequencesTo address R2, we propose supplementary design elements that focus onenhancing students’ metacognitive monitoring in in-class multitasking byproviding cognitive offloading.D4. Integrated Lecture Slides as Topic Handles for Topic AlertsThe class topic is an important factor driving students’ multitasking. Stu-dents choose to multitask when they find their learning not effective. Inour interviews, long introductions and tangential topics that do not concernthe student were mentioned as times when the students prefer to do moreproductive tasks. When distracted with another task, students’ awarenessof the lecture topic is reduced and they are not able to detect a topic changeoccurring. As a result, missing more content than they had intended is anunwanted consequence of in-class multitasking.We propose to provide information about future topics to students tohelp them with their planning and monitoring in the process of multitasking.Providing access to lecture slides before the class is common practice withmost instructors and is shown to improve students’ overall attendance andparticipation [3] and more efficient encoding of the lecture [32]. We recognizethat lecture slides are a good representation of the topics discussed in theclassroom and therefore choose to integrate them into the lecture as alerthandles. Using the future lecture slides, the students can set an alert onan upcoming topic that they want to be present for. Using topic-basedcognitive offloading, allows the students to ensure their timely presence for374.2. Designing for Metacognitive Consequences of In-class Multitaskingthe desired topic. This will lead to improved focus on the second task astheir attention is not split between the two tasks anymore.D5. Activity AlertsMultitasking inhibits students’ awareness of class activities that might re-quire a switch back to the learning task. To make sure that they are presentfor Q&A sessions, discussions, or other class activities, students need tocheck on the class content frequently and therefore are not able to focus ontheir secondary tasks. To assist with their monitoring of class activities, wepropose activity-based cognitive offloading. Receiving reliable notificationsin case of activity changes such as Q&A and discussions happening in theclass helps students focus on their secondary task and make sure they canswitch to the learning task when they need to.D6. Time AlertsTo address the inhibition in time management due to multitasking and dis-traction, we provide students with time-based cognitive offloading. Settingtimers is an extremely common example of cognitive offloading in our every-day life. However, we want to provide a quick and easy way with minimalcognitive load to ensure its effectiveness.4.2.3 Metacognitive Assistance PrototypeTo implement D4 in our prototype we proceed as follows. Assuming that theuser uploads the lecture slides to the interface before each class, the futureslides are shown as thumbnails underneath the progress bar in a discreetequally spaced format. By hovering over each thumbnail, it will increase insize and show a clear view of the upcoming slide. As the lecture proceeds, thepast and current thumbnails change to a continuous bar with their lengthsvisualizing the time spent on that lecture slide proportional to the full classduration. The thumbnail bar and the progress bar are coupled in the sensethat the part of the progress bar right on top of each of the current and384.2. Designing for Metacognitive Consequences of In-class MultitaskingFigure 4.7: Before a student starts to multitask, they can hover over the upcomingslide thumbnails to learn what the future topics are. If they want to be back whena certain topic comes up, they can click on the thumbnail for the slide to bookmarkit. When that slide shows on the screen, they will get a notification to pay attentionto the class.past lecture slides is the time spent on that slide, allowing students quickscrubbing to each topic if needed.To enhance students’ metacognitive monitoring, we allow users to “book-mark” future slides that they need to be present for (topic alert). For ex-ample, if the students find the current topic not useful for them and wantto increase their productivity by doing a different task, they can bookmarkthe next topic by clicking on the upcoming slide thumbnail before switchingto their task. When the said topic/slide comes up, they will receive a noti-fication to let them know about the topic change. You can see an overviewof the implementation of D4 in Figure 4.7.To implement a proof of concept for D5, we created an activity alertbutton. When the user clicks on this button, activity alerts will be enabledand a red bar will appear on the button. That means that when a discussionstarts or a question is asked, the user will receive a notification. We providean option to have this alert disabled as some users might not be interestedin the interactive component of the live class and therefore do not need to394.2. Designing for Metacognitive Consequences of In-class Multitaskinguse this feature. In Figure 4.9, you can see an instance of the activity alertgoing off when the instructor asks a question in the live class while the useris in the asynchronous mode. After being notified, the user can switch tosynchronous mode by clicking on the picture-in-picture window.We create a built-in timer to satisfy D6. To ensure that they spendonly a short time away from the class, users can click on the time alertbutton once to set a quick 1-minute timer that shows up on the progressbar. Clicking on the button twice will set it to two minutes and so on.When the time is up, the user will hear a notification reminding them toget back to the class. This is a quick-to-use implementation of time-basedcognitive offloading that can be useful for people who tend to spend longeron multitasking than they originally anticipated. In Figure 4.8, you can seea timer set by clicking on the time alert button once. The timer icon is set 1minute away on the progress bar and the label on top shows the remainingtime until the timer goes off.404.2. Designing for Metacognitive Consequences of In-class MultitaskingFigure 4.8: Time-based activity offloading. A) The student can click on the timerbutton to quickly set a 1-minute timer, click twice to set a 2-minute timer, and soon. B) The timer shows up on the progress time at the temporal location that it isset for.Figure 4.9: Activity-based offloading. C) The student can enable activity alertsby clicking on its button. D) When a question is asked in the live class, the studentis notified and can read the question on the screen.41Chapter 5Evaluating the DesignElementsIn this chapter, our goal is to evaluate the effectiveness of our design. Wedesigned Time-Turner to address the cognitive consequences of in-class mul-titasking and time and content-based alerts to assist with the metacognitiveconsequences of in-class multitasking for students’ learning outcomes. Thefollowing describes our evaluation process.5.1 Evaluating Cognitive Design Elements(Time-Turner)5.1.1 ParticipantsThe inclusion criteria for this evaluative study were as follows: We screenedparticipants for being university students (undergraduate, graduate, post-doc). In order to eliminate the effects of the language barrier on learning,we screened participants for being native English speakers. Our final partic-ipant list consisted of 20 university students between the ages of 18-44 (10men, 8 women, 2 non-binary/transgender) from a variety of majors such ascomputer science, law, mathematics, religion and folklore, criminology andetc. We did not include participants from our previous study to preventbias. Our recruitment was done through Prolific (www.prolific.co).425.1. Evaluating Cognitive Design Elements (Time-Turner)5.1.2 TasksOur goal is to evaluate the effectiveness of Time-Turner in overcoming thecognitive consequences of in-class multitasking. In order to achieve this,we conduct a within-subject user study where we ask our participants toperform in-class multitasking across two conditions (Time-Turner and base-line) and measure their immediate recall through a test after the class. Theparticipants were asked to attend a simulated synchronous lecture as theirlearning task. At a certain point in time (same for all participants), the par-ticipants received a notification asking them to work on an unrelated taskduring the class. The duration of the secondary task is controlled and thesame for all participants. After the class ended, the participants were askedto take a test to measure their immediate recall as an indication of theirlearning outcomes. Participants repeated the same process twice under thetwo conditions (Time-Turner and baseline).5.1.3 MaterialIn Bloom’s taxonomy [5], learning outcomes can be categorized into 6 differ-ent levels: knowledge, comprehension, application, analysis, synthesis, andevaluation. Of these 6 levels, classroom learning mostly consists of knowl-edge, or the ability to remember what you have learned (memorization),and comprehension, or the ability to grasp the meanings and being ableto interpret and translate concepts. In this study, the first level referredto as knowledge or memorization was chosen as the measure of students’learning outcomes. The reason for this decision was twofold. First, thelength of the lecture videos used in this study was short (10 minutes each)because of the controlled nature of a within-subject study. In a short time,comprehension is not a sensitive comparison measure across the two condi-tions, as comprehension learning is of higher granularity. Second, measuringand quantifying knowledge as a measure of the student’s learning outcomecan be done reliably in a recall test, whereas measuring comprehension canpresent as more challenging in a short-term experiment. To further confirmour decision, we looked into the existing literature and our findings confirm435.1. Evaluating Cognitive Design Elements (Time-Turner)our choice of measuring memorization vs. comprehension. Jamet et al. [20]measured the effects of in-class multitasking on students’ memorization andcomprehension. They found that multitasking affected students’ memoriza-tion negatively but showed no significant effect on comprehension. Theydiscuss that comprehension does not have enough discriminatory power andmight not be affected by in-class multitasking to the same extent.We chose a recorded video lecture from an Introduction to Biology courseto use for simulating a live lecture in our study. We chose “Cell Biology”as a topic because of the high amount of knowledge transferred in it thatprovides us with a rich comparison measure between the two conditions ina short time span.Since we are using two different 10-minute videos for each condition, itis important that both videos are similar in terms of information density,difficulty, and delivery. Therefore, we chose two 10-minute sections of thesame lecture. Being taught by the same instructor, on the same topic andsession, makes the two videos close in terms of difficulty and density. Tomeasure students’ learning outcomes from these lectures, we created testsfrom the lecture content. The tests were in the format of true/false state-ments with +1 points allocated to correct answers, −13 points allocated towrong answers, and 0 points allocated to choosing “I don’t know”. Thiswas done to discourage students from guessing the answer with no priorknowledge.The number of true/false statements that can be extracted from eachlecture, gave us a good idea of the video’s information density and helpedus decide on two 10-minute sections with the same amount of content. Totest whether or not our video-quiz combinations are fair and close to eachother, we conducted a pilot test (N=4) in which participants were asked towatch the lectures, take the quizzes, and rate them in terms of difficulty.Our pilot tests showed that there was no significant difference in terms ofdifficulty between the video and quiz pairs.Our criteria for designing the secondary task in students’ in-class mul-titasking is as follows: We are trying to find a task that does not requireprior training and is doable by everyone. We are also looking for a task445.1. Evaluating Cognitive Design Elements (Time-Turner)with a high cognitive load to make sure that students’ in-class multitaskinginduces the cognitive consequences that were previously discussed. To addecological validity, we designed this task to look like an English assignmentwith reading comprehension questions. The students were initially asked toread a paragraph and answer two comprehension questions from the pas-sage. The problem discovered with this approach in our pilot tests was thatstudents experienced a significant learning effect in answering the readingcomprehension questions. Our pilot participants reported that the morequestions they answered, the less the cognitive load of the tasks became andtherefore led to a less reliable comparison across the two conditions. In orderto prevent the learning effect, we changed the second question to a differentformat of counting questions instead of a comprehension question. Someexamples of the counting questions used in the assignment are: What is thefifth word in the third sentence of the passage? How many words are therein the first sentence of the passage? Write all the words in the passage thatstart with the letter ’s’. Adding these counting questions to the assignmentproved to reduce the learning effect and their variety kept students engagedand distracted from the lecture.Finally, to conduct a reliable comparison across the two conditions, weneed the students to spend the same amount of time on the secondary taskand away from the learning task when using Time-Turner and the baseline.To make sure this happens, we put time limits on the questions. Eachreading passage comes with a 30-second count-down timer (time to answerone comprehension question and one counting question) and moves on tothe next question after the 30 seconds are over. This guarantees the sameamount of time spent multitasking and makes sure that the students areexposed to new questions at a steady rate.5.1.4 ProcedureThe study was conducted remotely over a 1-hour session. The participantswere given a survey link and followed the steps in the survey based on theinstructions given to them. The participants first filled out a demographics455.1. Evaluating Cognitive Design Elements (Time-Turner)survey and answered questions about their perceived multitasking abilityand habits. The participants were then introduced to the Time-Turnerinterface and its features through an instructional video. To make sure theparticipants were familiar with the interface, they were asked to try outdifferent features of the prototype themselves.Since our study is conducted in a within-subject manner, it is importantto minimize the learning effect as much as possible as it can lead to increasedperformance in the second condition because of more familiarity and prac-tice with the experiment setup. To do so, we included two shorter trialversions of the tasks to make sure that the users were already familiar withusing the interface before participating in the main experiment. Each trialtask consists of the student watching a short 3-minute section of a Biologyclass simulating an online synchronous lecture. At a certain point in time,the students receive a notification telling them that they have an Englishassignment due soon and need to finish it now. The student will open theEnglish assignment (which is in the reading comprehension format describedin the previous section) and answers the questions. After 90 seconds, theassignment is submitted and the participants get back to listening to therest of the lecture. After the lecture is over, the participant will answer10 true/false questions. The participant is asked to do this task twice, onceusing the Time-Turner prototype and once using the baseline system. Whenusing Time-Turner, the participants have the ability to rewind and watchthe missed section.After the two trial versions, the participants are familiar with the formatof the lectures, tasks, tests, and the study flow. They are asked to repeatthe same procedure with the 10-minute videos for the main experiment. Inthis experiment, the students are asked to spend 2.5 minutes on the secondtask, increased proportionally to the length of the video. The tests used tomeasure students’ learning outcomes each include 30 true/false statements.The participants will repeat the task once using the prototype and once withthe baseline system. To remove the order effect and to minimize the effectsof using two different videos in a within-user study, we counterbalanced thevideo material and the order of conditions and had four different variations465.1. Evaluating Cognitive Design Elements (Time-Turner)of material-condition in our experiments: 1. prototype-lecture1 & baseline-lecture2, 2. prototype-lecture2 & baseline-lecture1, 3. baseline-lecture1 &prototype-lecture2, 4. baseline-lecture2 & prototype lecture1.In the last step, the participants are asked to complete a Likert-scalequestionnaire that asks about their opinions on Time-Turner and how ef-fective they find the prototype. Additionally, two open-ended questions askabout their likes and dislikes in the prototype.5.1.5 AnalysisFor determining the effects of using Time-Turner on learning outcomes weperformed a two-tailed paired-comparison t-test on students’ grades ob-tained from the post-lecture test with the null hypothesis being H0 : P < B(P: prototype, B: baseline). We used Cohen’s D to measure the effect size.We repeated the comparison for the students’ grades in the multitaskingsection (the times when they were busy with doing the assignment), and thenon-multitasking sections (the times when they were not busy with anothertask) to understand how each part of their learning is affected by usingTime-Turner. Of the total 30 questions of the quiz, 9 questions were relatedto the multitasking section and 21 were related to the non-multitasking sec-tions. To further investigate the effectiveness and usability of Time-Turnerqualitatively, we included a Likert-scale questionnaire that inquired aboutstudents’ perceptions of the prototype and its performance.5.1.6 ResultsIn this section, we will present the qualitative and quantitative findingsobtained from different stages of the study. We report the findings fromthe pre-test questionnaire, the within-subject experiment, and the post-testquestionnaire.Pre-study QuestionnaireIn this questionnaire, we collected the participants’ demographic data aswell as their perceived multitasking abilities. Inspired by Sanbonmatsu’s475.1. Evaluating Cognitive Design Elements (Time-Turner)Figure 5.1: Self-perception of multitasking ability compared to other people(top), and other students (bottom).measure of perceived multitasking ability [40], we asked students about howthey compare their multitasking abilities to the general population and otherstudents in a Likert-scale format. You can see the results for students’perceptions of their multitasking abilities and their level of comfort with itin Figure 5.1. We can see that the majority of the participants perceivedtheir multitasking abilities to be about the same as the general populationand other university students.To learn more about our participants’ multitasking habits and their ex-perience with watching lecture recordings at a faster speed, we asked themthe two following questions: How often do you multitask in your classes?,How often do you watch lecture recordings or videos at a faster speed? Youcan find the results for these two questions in Figure 5.2.Multitasking ExperimentTime-Tuner, improves students’ total learning outcomes when theyengage in in-class multitasking. We observed a significant improvementin students’ percent grades in the whole test that was taken after watchingthe lecture and multitasking when using Time-Turner compared to whenusing the baseline system (p < 0.05, t(19) = 2.53). The mean grade across485.1. Evaluating Cognitive Design Elements (Time-Turner)Figure 5.2: Perceived frequency of in-class multitasking (top), perceived fre-quency of watching lecture videos with faster speed (bottom)the whole test increased by 7.5% when using Time-Turner compared to thebaseline (Figure 5.3. To find the effect size we calculated Cohen’s D to be0.46 showing a close to moderate effect.To get more detailed, we performed the same comparison for the mul-titasking section grades and the non-multitasking section grades. Our two-tailed paired t-test analysis (H0 : Pmult < Bmult) on the multitasking sec-tions’ grades also showed significant improvement when using Time-Turnercompared to the baseline (p < 0.05, t(19) = 2.12). The mean grade acrossthe multitasking section increased by 13.3% when using Time-Turner com-pared to the baseline (Figure 5.3. This shows that students were able tomitigate the content loss that occurred because of their engagement in thesecond task to some extent and proved Time-Turner to be an effective in-tervention.When the students use Time-Turner’s accelerated viewing to watch theparts that they missed due to multitasking, they are able to watch whatthey missed. However, to make up for the time loss and until they arecaught up to the live class, they need to watch the lecture at a faster pacefor some time following the missed section. This leads us to think aboutwhether or not using Time-Turner will reduce the learning outcome for the495.1. Evaluating Cognitive Design Elements (Time-Turner)Percent Grade 0255075100Multitasking TotalPrototype BaselineFigure 5.3: Left: Students’ mean percent grade in the multitasking sectionincreased by 13.3% when multitasking using Time-Turner compared to the baseline(p < 0.05, t(19) = 2.12). Right: Students’ mean percent grade across the whole testincreased by 7.5% when multitasking using Time-Turner compared to the baseline(p < 0.05, t(19) = 2.53).non-multitasking sections due to watching them at a faster speed on aver-age. To test this, we perform a two-tailed paired t-test with the null hy-pothesis being no significant difference between the two. Our results showthat there is no significant difference observed across the two conditions(p = 0.65, t(19) = 1.58) and therefore show that Time-Turner does not re-duce students’ learning outcomes in the non-multitasking sections becauseof the faster average playback speed.Post-test QuestionnaireAfter the participants had the chance to use the features in Time-Turnerin two trial lectures and two main lectures, they were asked to fill out aquestionnaire about their opinion on the prototype and its features. Ourresults show strong positive attitudes towards Time-Turner. Out of 20 par-ticipants, 10 strongly agreed, and 9 somewhat agreed that they find Time-Turner helpful for their learning. Only 1 participant somewhat disagreedwith this statement. Thirteen participants strongly agreed and 6 somewhat505.1. Evaluating Cognitive Design Elements (Time-Turner)Figure 5.4: Top: 19 out of 20 participants somewhat or strongly agreed thatthey would use Time-Turner if it was available in their classes. Bottom: 19 outof 20 participants somewhat or strongly agreed that they find Time-Turner helpfulfor their learning.agreed that if Time-Turner was accessible in their online classes, they woulduse it. One participant strongly disagreed with this statement, see Figure5.4.In our open-ended questions about participants’ likes and dislikes, theymentioned that they liked: “That I knew what portions I missed”, “replayinginformation in a more time-efficient way”, “[knowing] if I would be able tocatch up with the live class based on the playback speed”, “being able togo back in the live [class] when missing or miss understanding something”,and “the ability to fast forward through bits I am already familiar with”.One participant mentioned that they have ADHD and believe that a lot ofneurodiverse students will enjoy using Time-Turner.In terms of participants’ dislikes of Time-Turner, the main concern broughtup by 5 participants was that speeding up the lecture might result in lowerlearning gain. One participant said: “If the content is very dense, having itrepeated while sped up may not be ideal to retaining the information.”. An-other participant mentioned that setting the speed correctly was stressfuland distracting for them. “I found it a bit stressful trying to find the rightspeed to use to catch up, it was a bit distracting in and of itself and I hadan urge to catch up quickly and probably put it on too high a speed.”.515.2. Evaluating Metacognitive Design Elements5.2 Evaluating Metacognitive Design ElementsIn this section, we evaluate the design guidelines proposed to address themetacognitive consequences of in-class multitasking. To do so, we use thetime-based, topic-based, and activity-based alerts implemented in the proto-type and collect information about university students’ perceptions of theireffectiveness in a survey.5.2.1 ParticipantsIn this study, our goal is to understand how students perceive the proposedmetacognitive design guidelines and how they compare their effectivenessto each other and the Time-Turner. In the quantitative multitasking studywhere we evaluated Time-Turner, participants spent a significant amountof time interacting with the prototype. Therefore, they are now very fa-miliar with the problem space, the scenarios that motivate these designelements, and using interventions that support in-class multitasking to re-duce its adverse effects on learning outcomes. As a result, we decided torecruit participants from the original participation list in the multitaskingstudy. Out of 20 university students participating in that study, 9 partici-pants (5 women, 4 men) between the ages of 18-44 chose to participate inthe survey to evaluate the metacognitive assisting features in the prototype.Our recruitment was done through Prolific (www.prolific.co).5.2.2 ProcedureThe survey study has 4 main sections, Time-Turner, topic-based alerts,activity-based alerts, and time-based alerts. We included Time-Turner to beable to compare the perceived effectiveness of metacognitive assisting alertsto the design guidelines proposed in Time-Turner.In each section, the participant is asked to watch a video that introducesthe feature and explains how to use it. After watching a video, they are givena scenario that represents a situation where the student is multitasking inthe class, and using the feature can be beneficial for them. For example, the525.2. Evaluating Metacognitive Design Elementsscenario given for topic-based alerts is:You are in an online Biology class. You know the professor always as-signs the first slide to a recap of the previous session at the beginning of theclass and you have already studied the previous session’s content. You wantto use the time to reply to an email but want to make sure you are back forthe new teaching content on “Cell Theory”.The participant is asked to imagine themselves in this scenario and usethe feature. They were then asked open-ended questions about their likesand dislikes in this feature and other scenarios they would find this featureuseful. Following the open-ended questions, they were asked four 5-pointLikert scale questions: 1. I find this feature easy to use., 2. I find thisfeature helpful., 3. I would like to have this feature in my online classes.,4. I would use this feature if it was available in my online classes. Thelast feature-specific question asked about how often students would use thisfeature in their online classes with the following 5 options: always, most ofthe time, about half of the time, sometimes, and never.After the participant answers the questions for all four features, they areasked to fill out the System Usability Scale (SUS)[7], a 10-item questionnairein the 5-point Likert scale format that is a reliable measure for measuringthe usability of the prototype as a system.5.2.3 ResultsI find this feature easy to use.In answer to I find this feature easy to use, the questionnaire results showthat our participants find all four features relatively easy to use. In order tocompare the results, we find the mean score in the Likert item questions (5strongly agree, 4 somewhat agree, 3 neither agree nor disagree, 2 somewhatdisagree, 1 strongly disagree). Time-based alerts had the highest meanscore of 4.78, followed by Time-Turner (4.67), Topic-based alerts (4.45),and Activity-based alerts (4.23). The result for this question is visualizedin Figure 5.5.535.2. Evaluating Metacognitive Design ElementsFigure 5.5: In ease of use, time-based alerts had the highest mean score of 4.78,followed by Time-Turner (4.67), Topic-based alerts (4.45), and Activity-based alerts(4.23).I find this feature helpful.In answer to I find this feature helpful, our results show that participantsfound Time-Turner most helpful among the four features with a mean scoreof 4.78. Topic-based and activity-based alerts got positive feedback with amean score of 4.22, where 8 out of 9 participants agreed to their helpfulnessto an extent and 1 participant neither agreed nor disagreed. Finally, time-based alerts had a mean score of 3.89 in terms of helpfulness showing thatsome participants did not find this feature helpful. You can find the resultsvisualized in Figure 5.6.I would use this feature if it was available in my online classes.Similar to participants’ views on the helpfulness of the features, our partic-ipants showed the most interest in having access to Time-Turner in theironline classes and using it and the least interest in having time-based alertsand using it. In answer to I would like to have this feature in my onlineclasses., Time-Turner had a mean score of 4.89, followed by topic-based545.2. Evaluating Metacognitive Design ElementsFigure 5.6: In helpfulness, Time-Turner had the highest mean score of 4.78,followed by activity and topic-based alerts (4.22), and time-based alerts (3.89).alerts (4.44), activity-based alerts (4.33), and time-based alerts (3.89).In answer to I would use this feature if it was available in my onlineclasses., Time-Turner had a mean score of 4.67, followed by topic-basedalerts and activity-based alerts (4.11), and time-based alerts (4.0). You canfind the results for these two questions in Figure 5.7How often would you use this feature in your online classes?To compare the perceived frequency of use for the proposed features, weassign scores to each option as follows: always= 5, most of the time= 4,about half of the times= 3, some times= 2, never = 1 and calculate themean score for each feature. Time-Turner had the highest mean score of3.56, followed by activity-based alerts (3.11), topic-based alerts (3.0), andtime-based alerts (2.56). You can find the results visualized in Figure 5.8.555.2. Evaluating Metacognitive Design ElementsReflectionsBased on the comparison of results in the Likert item questionnaire, we foundout that the metacognitive assisting features are in general less interestingto students and are perceived to be less helpful compared to Time-Turnerwhich was a proposed design solution to cognitive consequences of in-classmultitasking. To understand the reasons behind the results in this compar-ison we analyze the students’ responses to the open-ended questions in thequestionnaire asking about their likes and dislikes for each feature.Our results for topic-based alerts showed positive feedback from studentswith 8 out of 9 participants finding this feature helpful and all 9 participantsexpressing interest in having this feature accessible in their online classes.Participants mostly expressed interest in being able to multitask in partsof the lecture that are not beneficial to them. Another participant wentinto more detail describing a situation where all the lecture content is notrelevant to them: “If the entire lecture wasn’t relevant to me. Maybe, ifit was a lecture where the students were coming from different courses anddifferent backgrounds. Maybe in that case some things would be more rel-evant to students from another area than they would from another.”. Oneparticipant remarked on how easy it is to find different sections of the classwith this feature: “I like that this feature minimizes the amount of time astudent needs to spend looking for the required lecture sections.”. Finally,an interesting comment highlighted how lecturers’ involvement can increasethe benefits of this feature by giving pings at the start of different topicsbased on the content: “I like that you can set the bookmarks to plan yourattention during the lecture according to the slides but it may also be usefulto allow lecturers to give different pings at the start of different topics toallow students to tune in.”.Time-based alerts had the lowest average popularity among the proposedfeatures. Out of 9 participants one participant strongly disagreed and oneparticipant somewhat disagreed with this feature being helpful. Only 6 outof 9 participants agreed to an extent with wanting to have this feature intheir online classes. Exploring students’ responses to the questionnaire in565.2. Evaluating Metacognitive Design Elementssearch of the reason behind the relative unpopularity of this feature broughtus the following results. Six out of 9 participants found the time-based alertsuseful as a reminder to get back to the class on time ( “It would be usefulif you have a quick task to complete but may be easily sidetracked and needa reminder to return to the lesson.”). One participant also mentioned thatthey liked how easy it was to set up and use. However, the other 3 par-ticipants did not find this feature helpful because they did not find this arealistic solution. One participant said that they think it doesn’t make sensefor them to use this feature as it will take longer to plan ahead compared tomultitasking quickly without any planning. Another participant expressedhow using a timer creates extra pressure and is counter to the other designelements provided to support flexibility. They said: “Perhaps setting rigidtime limits on a program that is designed around flexibility and productivitymay be counter-intuitive. the addition of a timer may be needlessly pressur-izing.”. For many students, in-class multitasking is something that occursunplanned and unexpectedly. Therefore, providing means to recover fromtheir in-class multitasking after they do it is more reasonable than providingmeans for planning for it ahead of time.Activity-based alerts were perceived to be helpful by most participantswith 8 out of 9 agreeing to an extent to their helpfulness, and 8 out of 9expressing interest in having them accessible in their online classes. Par-ticipants were interested in the ability to engage and participate in classactivities even when they were multitasking. One participant mentioned“I like that this could help with engagement in online classes. For instance,when a poll has to be taken but a student is busy with other tasks.”. However,two participants mentioned that even when notified of the class activity, it isdifficult to learn the context quickly enough to be able to participate in thediscussion. So a mechanism to provide quick context for students in this sit-uation is required. Another factor that was brought up was whether or notclass participation was part of the lecture. One participant disagreed with“I would like to have this feature in my online classes.” because they didnot need to participate in class activities in their online classes and thereforedid not find a need for this feature.575.2. Evaluating Metacognitive Design ElementsTo conclude, an important factor in the effectiveness of metacognitiveassisting features in in-class multitasking is whether or not students engagein multitasking consciously, meaning that they are aware of their actions andplan for them. In these cases, providing assistance in metacognitive moni-toring can help students execute their multitasking plans better. However,for students who do not engage in in-class multitasking consciously and in-tentionally, setting alerts in the planning phase is not realistic. Furthermore,topic-based alerts can be enhanced by receiving input from the instructorsbased on their ideas on the importance of different topics. To increase theeffectiveness of activity-based alerts and to ensure students’ ability to par-ticipate effectively, further investigation of quick catching-up methods isrequired to provide context and essential information upon switching atten-tion to the new activity.585.2. Evaluating Metacognitive Design ElementsFigure 5.7: a. Top: Time-Turner had a mean score of 4.89, followed by topic-based alerts (4.44), activity-based alerts (4.33), and time-based alerts (3.89)., b.Bottom: Time-Turner had a mean score of 4.67, followed by topic-based alerts andactivity-based alerts (4.11), and time-based alerts (4.0).595.2. Evaluating Metacognitive Design ElementsTimeTurnerTopic alertsTime alertsActivity alerts0 2 4 6 8 10always most of the time about half of the time sometimes neverHow often do you think you would use this feature in your online classes?Figure 5.8: In usage, Time-Turner had the highest mean score of 3.56, followed byactivity-based alerts (3.11), topic-based alerts (3.0), and time-based alerts (2.56).60Chapter 6Discussion6.1 Supporting Students’ AutonomySupporting university students’ multitasking in online classes can be a con-troversial topic. Research shows that university students’ academic per-formance is negatively affected by in-class multitasking and suggests thatstudents reduce or eliminate their in-class multitasking. However, we pro-pose to support in-class multitasking instead of eliminating it. One mightask: Is supporting an activity that reduces students’ learning outcomes agood idea?In a study by Niterman et al. [37], university students were asked whosejob they think it is to minimize off-task technology use in the classroom.They found that students view off-task and on-task technology use in theclassroom as an informed and personal choice. The authors discuss thatstudents want to have autonomy over their learning and be treated as adultsas they are consumers who are paying for their education.While academic goals are some of the most important goals in a univer-sity student’s life, it is extremely important to recognize that they are nottheir only goals. University students are adults with many different socialand personal goals as well as their academic and professional goals. While inan ideal world, the student is able to separate these goals and pursue themone by one, in reality, it is very common for different goals to overlap. Ourfindings in our formative study showed that students are aware of the pricethey pay in learning when they pursue other goals in the classroom and aregenerally satisfied with their choices to multitask. Therefore, providing asense of autonomy and freedom to make these choices will eventually leadto higher satisfaction and improve their learning experience.616.2. Facilitating In-class MultitaskingIt is imperative for HCI researchers to take this into consideration andprovide solutions for the challenges of in-class multitasking without disre-garding students’ autonomy over their learning and their decisions to pursueother goals when in the classroom.6.2 Facilitating In-class MultitaskingStudents’ multitasking in online classrooms is an inevitable part of theirlearning and brings with it a trade-off. Students choose to do other tasksbecause they deem them of higher priority than the learning goal at the time.However, the resources spent on other tasks take away from the resourcesassigned to the learning goal and therefore lead to lower learning outcomes.It can be argued that this recovery process in which the students useTime-Turner to watch the missed content asynchronously and at a fasterspeed is a form of multitasking itself and introduces a new kind of trade-off.Even though the students are learning the lecture content, they are awayfrom the current lecture activity and therefore their activity is consideredmultitasking. In response, unlike other in-class multitasking activities, usingTime-Turner in the class does not lead to further content loss as the stu-dents are able to watch all the future content. However, it is possible thatwatching the lecture content with a faster playback speed leads to cognitiveoverloading and impairs students’ learning to some extent.To learn more about the effects of learning at a faster speed, we lookedinto the existing research in this area. A study by Murphy et al. [36] com-pares students’ immediate and delayed learning outcomes when watchinglecture videos with 1x, 1.5x, and 2x speeds. The findings show that increas-ing the speed up to 2x led to minimal costs but watching with a speed fasterthan 2x caused a decline in learning. In another study by Lang et al. [27],it is shown that students who watch accelerated content with an increase to1.25x speed, are more likely to have better grades. This is evidence that notonly watching the lecture content asynchronously and with a faster playbackspeed does not impair students’ learning outcomes, but it might also leadto better learning and performance for them.626.3. Benefits of Self-paced LearningIt can be argued that to bypass the multitasking trade-off, students canchoose to watch their lectures in the recorded format. In our interviews inthe formative user study, we learned that some students prefer to watch theirlectures asynchronously and in the recorded format because they can watchthem at a faster speed and save time. Rewinding and re-watching difficultcontent is also possible in asynchronous classes. Although asynchronouslearning can provide students with the ability to learn at their own paceand in their own time, students who struggle with self-regulation in theirstudies, find this type of learning problematic [36]. A study by Fabriz etal. [15] shows that students have more satisfaction with synchronous classesbecause of the increased social aspects of learning such as feedback andinteractions. Our prototype, allows students to take advantage of the self-paced learning provided in asynchronous mode while being able to benefitfrom the regulated nature of synchronous classes and the social componentof learning present in them.6.3 Benefits of Self-paced LearningIn our study, using Time-Turner helped students mitigate the content lossthat occurred because of their in-class multitasking. However, the benefits ofthis interface for students’ learning are not limited to multitasking. Studentscan use this interface for rewinding and re-watching sections of the class thatthey do not understand or learn very well when first listening to for anyreason. They are able to review content as many times as they need untilthey feel confident that they have learned it well. While self-paced learningis mostly possible in asynchronous online classes where students take controlof how long they wish to spend on each topic, combining asynchronous andsynchronous modes and creating a unique bichronous environment in Time-Turner lets them take advantage of opportunities for self-paced learning insynchronous online classes.As a result, this interface can potentially facilitate education accessi-bility. About 4.4% of adults between the age of 18-44 are diagnosed withAttention Deficit/Hyperactivity Disorder (ADHD)[24]. Students with this636.4. Implications for Teachingcondition often have difficulty staying focused and paying attention whenattending class. These distractions from listening to the lecture lead tosignificant content loss and impair their learning. Providing access to theproposed design elements prototyped into Time-Turner can allow studentswith ADHD or similar conditions to mitigate content loss due to attentiondeficit in the classroom.While students who struggle with learning a topic can choose to rewind tospend more time on it before moving on to the rest of the lecture, self-pacedlearning provided in our solution also allows students to watch the lecturecontent that they are already familiar with or find easy in a more time-efficient manner and use their time more productively. Guided acceleratedviewing helps students start their asynchronous viewing on time and settheir pace correctly to be able to catch up to the live class before a targettime (e.g. end of the class).6.4 Implications for TeachingThere is no doubt that introducing these features into students’ learningexperience influences teaching methods as well. We showed that mixingsynchronous and asynchronous learning environments together can help stu-dents learn at their own pace as well as benefit from the advantage of syn-chronous learning. But a question might arise: How does a bichronous learn-ing environment affect an instructor’s teaching? Should teaching methodsremain the same, or change and adapt to these new conditions?In a classroom where a similar bichronous setting to Time-Turner isimplemented for all students, students can adjust their pace to optimizetheir learning. To facilitate students’ self-paced learning in this setting,it is important to reflect on the types of content in the lecture. We cancategorize a lecture’s content into two main groups, 1. deferrable content:content that can be watched asynchronously without losing its effectiveness2. non-deferrable content: content that cannot be watched asynchronouslywithout losing effectiveness. In a synchronous online class, lecture deliverywould count as deferrable content and social and interactive components of646.4. Implications for Teachingthe class would be considered as non-deferrable content since watching themasynchronously does not allow participation.In our supplementary toolbar, we included activity alerts, to notify stu-dents of these interactive activities and to let them participate in non-deferrable content of the classroom synchronously. While this can helpstudents’ participation, abrupt switches from the asynchronous mode to thesynchronous mode can be stressful for students. Instructors can address thisproblem easily by understanding and accepting the differences between thedeferrable and non-deferrable content and allowing students to watch thedeferrable content at their own pace while providing them with informationabout the timing of non-deferrable content to ensure students’ synchronousparticipation in those activities. For example, an instructor can plan a 5-minute Q&A session for one hour after the class starts. They can let thestudents know in advance that the first hour of the class will be pure instruc-tions and after one hour they will have a chance to ask for clarifications orfeedback. Using the guided asynchronous viewing feature in Time-Turner,students can set the critical point to one hour later and rest assured thatthey will catch up to the live classroom by that time by receiving guidanceon how to adjust their pace.With instructors’ cooperation, students have the chance to watch thedeferrable content at their preferred pace, while ensuring synchronous pres-ence for non-deferrable content of the lecture to maximize the benefits of amixed bichronous learning environment.6.4.1 Beyond Online LearningOur research explored students’ in-class multitasking and proposed solutionsto support it by mitigating its negative outcomes. However, multitaskingis a very common human behavior that is observed in many other settings.Existing research on multitasking in online workplace meetings points tosimilar issues [9]. Therefore, it is worth exploring these new contexts andthe potential for supporting productive multitasking in them. Our proposeddesign elements can be applied in other similar situations where digital con-656.5. Limitations and Future Worktent and context loss are the consequences of multitasking.Considering the positive results in our study for in-class multitasking inonline synchronous classes, it is worth contemplating their effectiveness inin-person classes. Real-time streaming and recording of in-person lecturesis a common practice. Considering the prevalence of laptop use by studentsin in-person classes, creating a bichronous learning environment where theyare able to rewind and re-watch the recording of the missed content on theircomputers while present in the live in-person class is feasible. This providesopportunities for positive multitasking and self-paced learning in in-personclasses.6.5 Limitations and Future WorkThere are several limitations to this project. First of all, while our quanti-tative multitasking study provided valuable results in terms of the effects ofcognitive design elements on students’ learning outcomes in in-class multi-tasking, due to the limited time and relatively small number of participants,the study conditions were very well-controlled. Our current findings mo-tivate a longer-term longitudinal study with higher ecological validity andmore participants to observe the effects of these interventions on a largerscale.In our evaluation of the cognitive design elements, we measured stu-dents’ learning outcomes on a memorization-based learning task. We usedtrue/false statement tests to ensure full coverage of the learning content.Even though our results showed promise in improving students’ learningoutcomes, there are limits to these circumstances. Therefore, there is aneed for future work to measure the effectiveness of our proposed interven-tions with other types of learning through more holistic methods of learningand performance evaluation.The qualitative evaluation of the metacognitive assisting design elementswith a small sample size can be considered a preliminary step that motivatesfurther research on these measures and how they affect students’ quality ofin-class multitasking and its effectiveness. Future research can study these666.6. Conclusionsfactors quantitatively and qualitatively in realistic learning conditions.In our evaluative experiment, our proposed design elements proved tobe successful in mitigating the negative effects of in-class multitasking onlearning. Students showed strong interest in having access to similar tech-nologies in their online classes and believed they would benefit greatly fromit. However, as with many other tools, our solutions are not effective in allsituations. Interactivity plays an important role in the effectiveness of ourcognitive and metacognitive design elements. In an online class with a sig-nificant amount of synchronous interactive components, switching betweenthe synchronous and the asynchronous modes can create stress and cogni-tive overhead for students. This provides grounds for more explorations infinding ways to support students’ in-class multitasking in mostly interactivesettings.Finally, it is worth mentioning that our research on students’ in-classmultitasking was solely centered around students’ opinions and perceptionsof this activity in light of the absence of this perspective in the existingresearch. However, instructors and lecturers are the other actors in thelearning experience and their inputs and opinions on students’ multitaskingcan provide new insights that can improve students’ learning and multitask-ing. Studying the instructors’ perspectives can be a valuable addition to thefindings in this work.6.6 ConclusionsIn this thesis, we provided new insights into students’ motivations, percep-tions, and challenges in in-class multitasking in online synchronous classes.Motivated by these insights, we elicit requirements and propose multiple de-sign elements to support in-class multitasking by addressing its discoveredcognitive and metacognitive consequences for students’ learning outcomes.Our evaluation of the design elements shows statistically significant improve-ment in students’ learning outcomes and promising qualitative feedback. Wepropose to take on a new perspective towards in-class multitasking by ac-cepting it as a natural part of the learning experience and providing solutions676.6. Conclusionsthat address the negative consequences of in-class multitasking while main-taining its potential benefits for learners. Our results prove that supportingin-class multitasking can effectively mitigate its adverse effects on learningand pave the way for more research to facilitate in-class multitasking insteadof eliminating it.68Bibliography[1] Rachel F. Adler and Raquel Benbunan-Fich. Self-interruptionsin discretionary multitasking. Computers in Human Behavior,29(4):1441–1449, Jul 2013.[2] Kamer Ainur Aivaz and Daniel Teodorescu. College students’ distrac-tions from learning caused by multitasking in online vs. face-to-faceclasses: A case study at a public university in romania. InternationalJournal of Environmental Research and Public Health, 19(18):11188,Sep 2022.[3] Kimberley A. Babb and Craig Ross. The timing of online lecture slideavailability and its effect on attendance, participation, and exam per-formance. Computers Education, 52(4):868–881, May 2009.[4] Saraswathi Bellur, Kristine L. Nowak, and Kyle S. Hull. Make it ourtime: In class multitaskers have lower academic performance. Comput-ers in Human Behavior, 53:63–70, Dec 2015.[5] Benjamin Samuel Bloom. Taxonomy of Educational Objectives: TheClassification of Educational goals. Handbook 1, Cognitive Domain.Longman, New York, 1956.[6] Virginia Braun and Victoria Clarke. Using thematic analysis in psy-chology. Qualitative Research in Psychology, 3(2):77–101, 2006.[7] John Brooke. “SUS-A quick and dirty usability scale.” Usability evalu-ation in industry. CRC Press, June 1996. ISBN: 9780748404605.[8] Stoney Brooks. Does personal social media usage affect efficiency andwell-being? Computers in Human Behavior, 46(46):26–37, May 2015.69Bibliography[9] Hancheng Cao, Chia-Jung Lee, Shamsi Iqbal, Mary Czerwinski,Priscilla Wong, Sean Rintel, Brent Hecht, Jaime Teevan, and LongqiYang. Large scale analysis of multitasking behavior during remote meet-ings. arXiv:2101.11865 [cs], Jan 2021.[10] Maureen A. Conard and Robert F. Marsh. Interest level improves learn-ing but does not moderate the effects of interruptions: An experimentusing simultaneous multitasking. Learning and Individual Differences,30:112–117, Feb 2014.[11] Mary Czerwinski, Eric Horvitz, and Susan Wilhite. A diary study oftask switching and interruptions. Proceedings of the 2004 conferenceon Human factors in computing systems - CHI ’04, 2004.[12] Ton de Jong. Cognitive load theory, educational research, and in-structional design: some food for thought. Instructional Science,38(2):105–134, Aug 2009.[13] Anastasia D. Elder. College students’ cell phone use, beliefs, and effectson their learning. College Student Journal, 47(4):585–592, 2013.[14] Yvonne Ellis, Bobbie Daniels, and Andres Jauregui. The effect of mul-titasking on the grade performance of business students. Research inHigher Education Journal, 8(1):1–10, 2010.[15] Sabine Fabriz, Julia Mendzheritskaya, and Sebastian Stehle. Impact ofsynchronous and asynchronous settings of online teaching and learningin higher education on students’ learning experience during covid-19.Frontiers in Psychology, 12, Oct 2021.[16] Carrie B. Fried. In-class laptop use and its effects on student learning.Computers Education, 50(3):906–914, Apr 2008.[17] Arnold D. Froese, Christina N. Carpenter, Denyse A. Inman, Jessica R.Schooley, Rebecca B. Barnes, Paul W. Brecht, and Jasmin D. Chacon.Effects of classroom cell phone use on expected and actual learning.College Student Journal, 46(2):323–332, Jun 2012.70Bibliography[18] Helene Hembrooke and Geri Gay. The laptop and the lecture: Theeffects of multitasking in learning environments. Journal of Computingin Higher Education, 15(1):1–18, Sep 2003.[19] Yoori Hwang, HyoungJee Kim, and Se-Hoon Jeong. Why do me-dia users multitask?: Motives for general, medium-specific, andcontent-specific types of multitasking. Computers in Human Behav-ior, 36:542–548, Jul 2014.[20] Eric Jamet, Corentin Gonthier, Salomeacute Cojean, Tiphaine Colliot,and Se´verine Erhel. Does multitasking in the classroom affect learn-ing outcomes? a naturalistic study. Computers in Human Behavior,106:106264, Jan 2020.[21] Reynol Junco. In-class multitasking and academic performance. Com-puters in Human Behavior, 28(6):2236–2243, Nov 2012.[22] Reynol Junco and Shelia R. Cotten. No a 4 u: The relationship be-tween multitasking and academic performance. Computers Education,59(2):505–514, Sep 2012.[23] Ioanna Katidioti, Jelmer P. Borst, Marieke K. van Vugt, and Niels A.Taatgen. Interrupt me: External interruptions are less disruptive thanself-interruptions. Computers in Human Behavior, 63:906–915, Oct2016.[24] Ronald C Kessler, Lenard Adler, Russell Barkley, Joseph Biederman,C Keith Conners, Olga Demler, Stephen V Faraone, Laurence L Green-hill, Mary J Howes, Kristina Secnik, Thomas Spencer, T Bedirhan Us-tun, Ellen E Walters, and Alan M Zaslavsky. The prevalence and cor-relates of adult adhd in the united states: results from the nationalcomorbidity survey replication. The American journal of psychiatry,163(4):716–23, 2006.[25] Jaejeung Kim, Chiwoo Cho, and Uichin Lee. Technology supportedbehavior restriction for mitigating self-interruptions in multi-device en-71Bibliographyvironments. Proceedings of the ACM on Interactive, Mobile, Wearableand Ubiquitous Technologies, 1(3):1–21, Sep 2017.[26] Jeffrey H. Kuznekoff and Scott Titsworth. The impact of mobile phoneusage on student learning. Communication Education, 62(3):233–252,Jul 2013.[27] David Lang, Guanling Chen, Kathy Mirzaei, and Andreas Paepcke. Isfaster better? Proceedings of the Tenth International Conference onLearning Analytics Knowledge, Mar 2020.[28] Rosen Larry D., Lim Alex F., Carrier L. Mark, and Cheever Nancy A.An empirical examination of the educational impact of text message-induced task switching in the classroom: Educational implicationsand strategies to enhance learning. Revista de Psicolog´ıa Educativa,17(2):163–177, 2011.[29] Andrew Lepp, Jacob E. Barkley, Aryn C. Karpinski, and Shweta Singh.College students’ multitasking behavior in online versus face-to-facecourses. SAGE Open, 9(1):215824401882450, Jan 2019.[30] Danielle Lottridge, Eli Marschner, Ellen Wang, Maria Romanovsky,and Clifford Nass. Browser design impacts multitasking. Proceed-ings of the Human Factors and Ergonomics Society Annual Meeting,56(1):1957–1961, Sep 2012.[31] Ulrik Lyngs, Kai Lukoff, Petr Slovak, Reuben Binns, Adam Slack,Michael Inzlicht, Max Van Kleek, and Nigel Shadbolt. Self-control incyberspace. Proceedings of the 2019 CHI Conference on Human Factorsin Computing Systems, May 2019.[32] Elizabeth J. Marsh and Holli E. Sink. Access to handouts of presenta-tion slides during lecture: Consequences for learning. Applied CognitivePsychology, 24(5):691–706, Apr 2009.[33] Florence Martin, Swapna Kumar, Albert D. Ritzhaupt, and Drew Polly.Bichronous online learning: Award-winning online instructor practices72Bibliographyof blending asynchronous and synchronous online modalities. The In-ternet and Higher Education, page 100879, Aug 2022.[34] Kaitlyn E. May and Anastasia Elder. Efficient, helpful, or distracting?a literature review of media multitasking in relation to academic per-formance. International Journal of Educational Technology in HigherEducation, 2018.[35] Sylvia E. McDonald. The effects and predictor value of in-class textingbehavior on final course grades. College Student Journal, 47(1):34–40,Mar 2013.[36] Dillon H. Murphy, Kara M. Hoover, Karina Agadzhanyan, Jesse C.Kuehn, and Alan D. Castel. Learning in double time: The effect oflecture video speed on immediate and delayed comprehension. AppliedCognitive Psychology, Nov 2021.[37] Elena Neiterman and Christine Zaza. A mixed blessing? students’ andinstructors’ perspectives about off-task technology use in the academicclassroom. The Canadian Journal for the Scholarship of Teaching andLearning, 10(1), May 2019.[38] Agnieszka Pop lawska, Ewa Szumowska, and Jakub Kus´. Why do weneed media multitasking? a self-regulatory perspective. Frontiers inPsychology, 12, Feb 2021.[39] Eric D. Ragan, Samuel R. Jennings, John D. Massey, and Peter E.Doolittle. Unregulated use of laptops over time in large lecture classes.Computers Education, 78:78–86, Sep 2014.[40] David M. Sanbonmatsu, David L. Strayer, Nathan Medeiros-Ward, andJason M.Watson. Who multi-tasks and why? multi-tasking ability, per-ceived multi-tasking ability, impulsivity, and sensation seeking. PLoSONE, 8(1):e54402, Jan 2013.[41] Seoyun Son, Junyoug Choi, Sunjae Lee, Jean Y Song, and Insik Shin. It73is okay to be distracted: How real-time transcriptions facilitate onlinemeeting with distraction. Apr 2023.[42] Ewa Szumowska and Arie W. Kruglanski. The psychology of gettingbusy: Multitasking as a consequence of goal activation. Journal ofExperimental Psychology: General, 151(1):137–160, Jan 2022.[43] Winneke A. van der Schuur, Susanne E. Baumgartner, Sindy R. Sumter,and Patti M. Valkenburg. The consequences of media multitasking foryouth: A review. Computers in Human Behavior, 53:204–215, Dec2015.[44] Fred R. H. Zijlstra, Robert A. Roe, Anna B. Leonora, and Irene Krediet.Temporal factors in mental work: Effects of interrupted activities. Jour-nal of Occupational and Organizational Psychology, 72(2):163–185, Jun1999.74Appendix AConsent Forms75Appendix A. Consent FormsFigure A.1: Consent form, page 176Appendix A. Consent FormsFigure A.2: Consent form, page 277Appendix BDiary TemplateFigure B.1: Template for diary study78Appendix CFollow-up InterviewFigure C.1: Interview script page 179Appendix C. Follow-up InterviewFigure C.2: Interview script page 280Appendix DMaterialsD.1 Lecture VideosMultiple sections were selected from the Biology 1010 lecture 6 as the lecturevideos for training and experiments. https://www.youtube.com/watch?v=gt_raJgxKis&t=2652sD.2 Secondary TaskThe following are two reading comprehension questions from the assignmentsgiven to students during the lecture to do as a form of multitasking:Figure D.1: Reading Comprehension Question from Task81D.3. PrototypeD.3 PrototypeThe link to the prototype used in one condition of the experiment for eval-uating the proposed cognitive design elements can be found here. https://sahmav.github.io/Digestive-P/D.4 QuizThe quiz created to evaluate students’ learning outcomes from the lecturevideo was in the below format:Figure D.2: Post-lecture Quiz82Appendix ESurveyTo evaluate the proposed metacognitive assisting design elements we usedthe following survey format:Figure E.1: Survey for Evaluating Metacognitive Design Elements83
Thesis/Dissertation
eng
Electrical and Computer Engineering
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Fels, Sidney
Supporting multitasking in online synchronous classes
Text