Applied Science, Faculty of
Electrical and Computer Engineering, Department of
DSpace
UBCV
Jaddigadde Srinivasa, Ranjitha
Master of Applied Science - MASc
University of British Columbia
Handwritten note-taking with pen and paper is still the preferred medium to achieve information seeking and comprehension from diverse learning objects. But students, especially in video-based learning settings, exercise laborious practices to re-find the corresponding video context when reviewing notes. We propose the orchestration of students' handwritten notebook content as interoperable links to retrieve previously watched instructional videos. This work articulates the research objectives in two phases. In phase 1, we analyzed the characteristic features of notebook content of watched videos. And, in phase 2, we investigated student expectations and requirements of the proposed video retrieval system. Analysis of quality handwritten notebook samples and the related video materials in a lab study with ten engineering students revealed distinctive characteristic representations of note content such as text, formula, figures, and chiefly a hybrid of all the 3. A box plot interpretation of notes and the watched video content confirmed that at least 75% of the identified note samples demonstrated a verbatim overlap of 50% or more with the related video content, hinting at its potential use as a query artifact. Additionally, the video references to collected note samples exhibited referencing at three temporal levels: point, interval, and whole video. A 12-student lab study indicated higher satisfaction for video matches returned at the `interval' level and showcased students' existing workarounds for linking back to videos. Overall, students rated a positive Mean score for the system's usability to re-find note-specific video context. A medium-fidelity prototype was built, using off-the-shelf computer vision algorithms, to deduce technology requirements associated with the proposed approach. When tested on the 181 identified note samples, the prototype system matched 77.5% of the samples to corresponding watched videos. The proposed method worked exceptionally well to find suitable videos for textual notes --- yielding a 98% accuracy. The note content overlap with the video results further highlights the fragmented nature of the evaluated accuracy across all three temporal levels. Overall, the presented work ascertains the prospect of augmenting prevalent Personalized learning (PL) strategies, such as handwriting notes for future reference, to easily re-find and connect to the watched videos.
Leveraging Students’ Handwritten Notes to link to watchedInstructional VideosbyRanjitha Jaddigadde SrinivasaB.E., Sapthagiri College of Engineering, 2014a thesis submitted in partial fulfillmentof the requirements for the degree ofMaster of Applied ScienceinThe Faculty of Graduate and Postdoctoral Studies(Electrical and Computer Engineering)The University of British Columbia(Vancouver)April 2021© Ranjitha Jaddigadde Srinivasa, 2021The following individuals certify that they have read, and recommend to the Faculty ofGraduate and Postdoctoral Studies for acceptance, the thesis entitled:Leveraging Students’ Handwritten Notes to link to watched InstructionalVideossubmitted by Ranjitha Jaddigadde Srinivasa in partial fulfillment of the require-ments for the degree of Master of Applied Science in Electrical and ComputerEngineering.Examining Committee:Dr. Sidney Fels, Department of Electrical and Computer EngineeringSupervisorDr. Dongwook Yoon, Department of Computer ScienceSupervisory Committee memberDr. Julia Rubin, Department of Electrical and Computer EngineeringSupervisory Committee memberiiAbstractHandwritten note-taking with pen and paper is still the preferred medium to achieveinformation seeking and comprehension from diverse learning objects. But students, es-pecially in video-based learning settings, exercise laborious practices to re-find the corre-sponding video context when reviewing notes. We propose the orchestration of students’handwritten notebook content as interoperable links to retrieve previously watched in-structional videos. This work articulates the research objectives in two phases. In phase1, we analyzed the characteristic features of notebook content of watched videos. And,in phase 2, we investigated student expectations and requirements of the proposed videoretrieval system.Analysis of quality handwritten notebook samples and the related video materials in alab study with ten engineering students revealed distinctive characteristic representationsof note content such as text, formula, figures, and chiefly a hybrid of all the 3. A boxplot interpretation of notes and the watched video content confirmed that at least 75%of the identified note samples demonstrated a verbatim overlap of 50% or more with therelated video content, hinting at its potential use as a query artifact. Additionally, thevideo references to collected note samples exhibited referencing at three temporal levels:point, interval, and whole video.A 12-student lab study indicated higher satisfaction for video matches returned atthe ‘interval’ level and showcased students’ existing workarounds for linking back tovideos. Overall, students rated a positive Mean score for the system’s usability to re-find note-specific video context. A medium-fidelity prototype was built, using off-the-shelf computer vision algorithms, to deduce technology requirements associated withthe proposed approach. When tested on the 181 identified note samples, the prototypeiiisystem matched 77.5% of the samples to corresponding watched videos. The proposedmethod worked exceptionally well to find suitable videos for textual notes — yieldinga 98% accuracy. The note content overlap with the video results further highlights thefragmented nature of the evaluated accuracy across all three temporal levels. Overall,the presented work ascertains the prospect of augmenting prevalent Personalized learning(PL) strategies, such as handwriting notes for future reference, to easily re-find andconnect to the watched videos.ivLay SummaryWhen learning from instructional videos, students often take handwritten notes to im-prove recall and comprehension. When reviewing their notes, it can be difficult forstudents to return to the corresponding part of the video. In this thesis, we present theidea of using content available in student notebooks to connect to relevant video contextin a video archive. A set of notebook samples were analyzed to explore distinct types ofcontent and references made to videos. A prototype was designed to test the resourcesand requirements to develop a fully working system. Experimental studies conducted tovalidate the proposed system demonstrated that it is feasible for implementation. Thefindings indicated that the students find the proposed approach usable.vPrefaceThe data collection section of the experimental study discussed in Chapter 3 of this workwas approved by UBC Behavioural Research Ethics Board(BREB), Study Number: H13-01589-A022.I was the primary investigator for the work contained within this thesis. The idea ofaddressing interoperability between student notebooks and video medium was presentedto me by my supervisor Dr. Sidney Fels at the beginning of my degree. Further iterationswere my designs. All the ideation, design, implementation, and testing for this projectwas carried out at the Human Communication Technologies (HCT) Laboratory in theElectrical and Computer Engineering department at the University of British Columbia(UBC) (Vancouver campus).A condensed version of Chapter 2, 3, 4, and 5 is submitted to a conference, ti-tled “NoteLink: A Point-and-Shoot Linking Interface between Students’ HandwrittenNotebooks and Instructional Videos”. I was the lead investigator for this work andwas solely responsible for the ideation, software development, conduction of user stud-ies, data analysis, and manuscript composition. Samuel Dodson(School of Information,UBC) and Dr. Kyoungwon Seo(Electrical and Computer Engineering Department) wereco-investigators. Dr. Sidney Fels (Electrical and Computer Engineering Department)and Dr. Dongwook Yoon (Department of Computer Science) provided valuable feed-back towards the experimental design. Samuel Dodson contributed to the presentationof background work and data collection. Dr. Kyoungwon Seo contributed with resultsarticulation.viTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiList of Abbreviations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xivAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvi1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Thesis Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.2 Research questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2.1 Characteristics of Watched Video Notes . . . . . . . . . . . . . . 41.2.2 Student Expectations, Requirements of a Watched Video Re-findingSystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.3 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61.4 Thesis Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102.1 In-house Video Functionalities for Note-taking . . . . . . . . . . . . . . . 102.2 Cross-Media Interaction with Paper Notes . . . . . . . . . . . . . . . . . 142.3 Searchable-Video Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172.4 Note Content Characteristics . . . . . . . . . . . . . . . . . . . . . . . . 182.5 Handwriting Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 Characteristics of Watched Video Notes . . . . . . . . . . . . . . . . . . 213.1 Notes Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21vii3.1.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213.1.2 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223.1.3 Description of the Captured Notes . . . . . . . . . . . . . . . . . 243.2 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253.2.1 Content-based Categorization of Note Content . . . . . . . . . . . 253.2.2 Unfolding the Hybrid Type . . . . . . . . . . . . . . . . . . . . . 283.2.3 Verbatim Overlap . . . . . . . . . . . . . . . . . . . . . . . . . . . 293.2.4 Watched Video Timepoint . . . . . . . . . . . . . . . . . . . . . . 333.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344 Rendering Watched Video Matches . . . . . . . . . . . . . . . . . . . . . 364.1 Experiment Design and Procedure . . . . . . . . . . . . . . . . . . . . . . 374.1.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374.1.2 Study Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . 384.2 Study Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444.2.1 Video in “Interval” Temporal Level . . . . . . . . . . . . . . . . . 444.2.2 Selection-based Use of Video Surrogates . . . . . . . . . . . . . . 464.2.3 Usability reports . . . . . . . . . . . . . . . . . . . . . . . . . . . 484.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 504.3.1 Link to Note vs Note of Links – Linking Strategies . . . . . . . . 524.3.2 Impact on the Existing Note-taking Practices . . . . . . . . . . . 524.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 535 Pragmatics of Linking Notes to Video . . . . . . . . . . . . . . . . . . . 555.1 Note Recognition Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 555.2 Finding Video Matches . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575.3 Smartphone as a Pedagogical Interface . . . . . . . . . . . . . . . . . . . 595.4 Accuracy of matching videos to notes . . . . . . . . . . . . . . . . . . . . 61viii5.5 Content Overlap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 645.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 675.7 Implementation requirements . . . . . . . . . . . . . . . . . . . . . . . . 695.7.1 Automated Content Type Segmentation . . . . . . . . . . . . . . 695.7.2 Recognition Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . 715.7.3 Computing Time . . . . . . . . . . . . . . . . . . . . . . . . . . . 726 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 736.1 Limitations and Future Work . . . . . . . . . . . . . . . . . . . . . . . . 746.2 Extension of Application Interactions . . . . . . . . . . . . . . . . . . . . 76References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80Appendices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89A Participant data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90A.1 Notebook Samples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90B Study Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109B.1 Accuracy-Overlap results . . . . . . . . . . . . . . . . . . . . . . . . . . . 109B.2 Friedman & Wilcoxon test results . . . . . . . . . . . . . . . . . . . . . . 177B.3 Interview Transcripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184B.4 Survey results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184C Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189ixList of TablesTable 3.1 Demographic data of Participants P1 to P10 . . . . . . . . . . . . . . 23Table 4.1 Demographic data of the 12 students . . . . . . . . . . . . . . . . . . . 38Table 4.2 Identified current linking techniques . . . . . . . . . . . . . . . . . . . 53Table 5.1 The number of count (n) of note-video matches, and accuracy of retriev-ing matching video is reported for each content type (i.e., text, formula,and drawing) overall and at the three levels of temporal specificity (i.e.,point, interval, and whole video). . . . . . . . . . . . . . . . . . . . . . 64Table A.1 Notebook Sample details . . . . . . . . . . . . . . . . . . . . . . . . . 90Table B.1 Accuracy-Overlap Evaluation Results . . . . . . . . . . . . . . . . . . 110Table B.2 Likert-scale data for temporal preference in Point, Interval and wholevideo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185Table B.3 Likert-scale data for video surrogate preference . . . . . . . . . . . . . 186Table B.4 Likert-scale survey results for Effectiveness in Usability Evaluation . . 187Table B.5 Likert-scale survey results for Efficiency in Usability Evaluation . . . . 188Table B.6 Likert-scale survey results for Satisfaction in Usability Evaluation . . . 188xList of FiguresFigure 2.1 Behaviors of active and passive viewing, categorized using the ICAPframework [22]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12Figure 3.1 An example of a student-labelled ground truth of a scanned note,recording 1) the name of the matching video, 2) the timestamp withinthe matching video, and 3) the students’ Remember, Know, Guessconfidence interval. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24Figure 3.2 Notes with text-only content . . . . . . . . . . . . . . . . . . . . . . . 26Figure 3.3 Notes with formula-only content . . . . . . . . . . . . . . . . . . . . . 27Figure 3.4 Notes with figure-only content . . . . . . . . . . . . . . . . . . . . . . 27Figure 3.5 Notes with a combination of textual and non-textual content . . . . . 28Figure 3.6 Hybrid link consisting of connected text, non-text components . . . . 30Figure 3.7 Hybrid link with semantically broad text content . . . . . . . . . . . 30Figure 3.8 Hybrid link with highlighted content . . . . . . . . . . . . . . . . . . 30Figure 3.9 Box plot comparison of content overlap in the text, formula and figurerepresentations respectively. . . . . . . . . . . . . . . . . . . . . . . . 32Figure 4.1 An example video surrogate, comprised of a thumbnail (a), title (b),keywords (c), and summary (d). . . . . . . . . . . . . . . . . . . . . . 42Figure 4.2 Coded mapping of preference levels (i.e., very good, good, neutral,bad, and very bad) to each level of temporal specificity (i.e., point,interval, and whole video). . . . . . . . . . . . . . . . . . . . . . . . . 46Figure 4.3 Stacked bar chart showing likert-scale observations for Effectiveness. . 49Figure 4.4 Stacked bar chart showing likert-scale observations for Efficiency. . . 49xiFigure 4.5 Stacked bar graph showing likert-scale observations for Satisfaction. . 50Figure 5.1 Matching flow for Text and Formula content. . . . . . . . . . . . . . . 57Figure 5.2 Matching pictorial content for figures . . . . . . . . . . . . . . . . . . 58Figure 5.3 Mobile interface, is presented with the option to take a picture ofstudents’ handwritten note content (top) to find the correspondingvideo with topically similar content, read to be played back within theapp (bottom) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60Figure 5.4 The pie charts showing the total accuracy of finding the matched andthe breakdown across point, interval and whole video temporal levels 63Figure 5.5 The bar chart showing the total number of matched note samples ofthe 3 content types; Text, Formula, and figure . . . . . . . . . . . . . 63Figure 5.6 Results showing overlap between video results and text-only content. 65Figure 5.7 Boxplot results showing overlap between video results and formula-only content. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66Figure 5.8 Link samples with low legibility . . . . . . . . . . . . . . . . . . . . . 68Figure B.1 Temporal preference: a) It is easy to find the information I need fromthis point . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178Figure B.2 Temporal preference: b) The information is effective in helping mecomplete the tasks and scenarios for learning . . . . . . . . . . . . . . 179Figure B.3 Temporal preference: c) I’m satisfied with the retrieved video time point180Figure B.4 Temporal preference: a) It is easy to find the information I need fromthis point . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181Figure B.5 Preference for Video Surrogates: b) This design has all the functionsand capabilities I expect it to have . . . . . . . . . . . . . . . . . . . 182xiiFigure B.6 Preference for Video Surrogates: c) The organization of informationon the layout screen is clear . . . . . . . . . . . . . . . . . . . . . . . 183xiiiList of AbbreviationsAPI Application Programming InterfaceCAD Computer-Aided DesignCC Connected ComponentECAR Educause Center for Applied ResearchFLANN Fast Library for Approximate Nearest NeighborsGAN Generative Adversarial NetworksHCT Human Communication TechnologiesHWR Handwriting Word RecognitionICAP Interactive Constructive Active and PassiveISEE Interactive Shared Education EnvironmentLBP Local Binary PatternOCR Optical Character RecognitionOpenCV Open Computer VisionPIM Personal Information Management (PIM)PL Personalized LearningROI Region of InterestSD Standard DeviationSIFT Scale Invariant Feature TransformxivSSIM Structural SimilarityTRECVidTREC Video Retrieval EvaluationUBC University of British ColumbiaURL Uniform Resource AllocatorxvAcknowledgementsI would like to thank my supervisor, Dr. Sidney Fels, for giving me the opportunity towork on this project. It was an exciting idea from the start, and I thoroughly enjoyedthe ideation process to shape my work to what it looks like today. I continue to remainexcited about the future of this work. I am incredibly grateful for the guidance andunwavering support, even when things did not seem to go my way. Thank you for beingso patient and encouraging throughout my degree.I want to thank Dr. Dongwook Yoon for his insights in the course of building my work.His views on the user study methods, procedures have greatly helped the work addresscrucial questions. A big thank you is extended to other members of my team—SamuelDodson and Dr. Kyoungwon Seo—for allowing me to constantly pick at their brainswhenever I was stuck and needed help. Samuel deserves a special mention for all hisassistance with the thesis organization. I would also like to specifically thank SameerSunani for helping me with the initial configurations in the prototype development.This work was supported by the University of British Columbia Teaching LearningEnhancement Fund (TLEF), the Natural Sciences and Engineering Research Council ofCanada (NSERC), and Microsoft. I also thank ViDeX team for their feedback on thework, particularly Gokberk Ozsoy for his contributions in note recognition, especially forformulas. 1Lastly, I owe everything to my parents and my husband for being the most significantsupporters they are. My parents have always given me the best of opportunities, butthis time they encouraged me to move all the way to Canada. Words cannot express mygratitude to my husband, Kiran Kadaba, for all he had done to support me, even when1https://matheqrecognition.blogspot.com/xvithings were not going my way. I would not be half the person I am without the twosupport systems.xviiCHAPTER 1IntroductionStudents frequently engage in note-taking to improve recall and comprehension [40] [85].With paper-based notebooks, in particular, students maintain the notes in one placeand leverage a versatile platform for information gathering practices [63][77]. Pen andpaper medium enables students to handwrite flexibly and review notes in many learningsituations, such as during lectures, labs, or tutorials. However, it cannot be easy to returnto a specific source of the notes, for example, a related chapter in a specific textbook whenreviewing for an upcoming exam. This is fundamentally an issue of re-retrieval in pen-paper note-taking medium when information spans across a diverse collection of mediaservices. Re-finding of information, a salient learning activity under the umbrella term ofpersonalized learning, can be challenging, especially as students learn from instructionalvideos.The adoption of personalized learning approaches has increased significantly in recentyears [18]. Personalized learning facilitates an efficient way of connecting instructionto students’ preferences, interests, and needs [35] [75]. For instance, when a studenttakes notes watching an instructional video, he is enabling himself to prepare in case heforgets essential parts of the lecture. He makes a note of equations, diagrams, etc. thatare relevant to the discussion, such that when he returns to the part of the discussionlater, he can comprehend with relative ease. His learning activity is a conscious or anunconscious effort to be equipped with the required knowledge in future need. This isa classic illustration of personalized learning, which aligns well with the findings from1prior research, that emphasizes the intentions of note-taking in video-based learning aspredominantly to review the notes afterward [47][18].Thus, there is an imperative need to better understand the critical attributes ofreviewing handwritten notes captured when learning from instructional videos and ex-ploring design opportunities to facilitate effective linking to video information withouthindering students’ natural note-taking practices.1.1 Thesis ObjectiveWhen learning from notebooks, students often operate re-finding of the source with thehelp of navigational cues [79][97]. With physical information objects, such as textbooks,navigational cues include properties of the artifact (e.g., its size, color, thickness), para-text (e.g., page numbers and headings), and the relative positions of the informationsought (e.g., “about half-way through the textbook on the top of a left-hand page”). Ine-books, while some previously-mentioned contextual cues are lost, students commonlyuse text-search, highlighting, and scrolling to find information [56]. Bookmarks and an-notations are other widely used cues to identify important parts of the book [68] [79].With audio learning materials, indexing the transcripts converted from the speech is rel-atively easy, similar to e-documents. These types of contextual cues can be less salientin the video.In the case of instructional videos, as the video content is visually changing with time,students need to navigate the video to find the relevant content, which can be difficult andtime-consuming. Fewer and less rich cues are extracted from the visual, auditory, textual,and temporal information such as slide transitions, talking head instructor. Current videointerfaces leverage these cues to facilitate navigation and re-finding of video information,through supporting time-linked video-based annotation [44], [62] or table of contents[101], for example. However, re-finding relevant information in a collection of videos2whose cues are to be identified from handwritten paper-based notes is an under-exploredarea of research.In handwritten notes of instructional video content, students can write down time-codes and slide titles as links/cues to associate to a video [23]. Unfortunately, when thesecues are present in a notebook, it can be difficult to look up the related video using atimestamp or slide title when learning from an extensive video collection. Moreover, theselinks are not always recorded by students because they are time-consuming to create andcan become a distraction to learning.When learning from videos, students often transcribe or record verbatim content intheir notebooks rather than paraphrasing [8]. Comprehending, summarizing key points,paraphrasing, and noting them down simultaneously might be too challenging, given thetime and efforts in scrolling back and forth. This is especially evident in courses involvingtechnical learning such as mathematics, engineering as pointed by researchers on note-taking strategies [85]. Facts, definitions, and graphical representations are recorded whenstudents believe copying word to word will help recognize source material. This salientfeature of notes structure, often termed as Verbatim transcriptions by researchers [63][41] is leveraged in the current work to augment reviewing in students’ learning process.Hence, the focus of the thesis is to evaluate the effectiveness of exercising content of notesthat are verbatim reference(s) to a specific part(s) of watched videos as query elementsto aid content-based video retrieval.1.2 Research questionsThe primary research objective of the current work is to glean the distinctive features andrequirements of linking notes to watched videos. First, we investigate the distinguishingtraits in the notebook content, such as the types of note representations and the extentof verbatim occurrence of video content in the notebooks. Second, we showcase the3requirements of rendering the found matching videos. Students’ attitudes towards theways of rendering videos, along with their expectations, are captured. The followingresearch questions address the above-defined objectives:1. What are the characteristics of the note content captured when learning from in-structional videos?2. What are the requirements of a watched video rendering system to meet students’reviewing expectations?Throughout this work, watched video notes are the notes captured when learning frominstructional videos, and a video timepoint as the topically related timestamp in aninstructional video.1.2.1 Characteristics of Watched Video NotesTo answer research question 1, we investigated the distinct types of representations inwatched video note content, along with the nature of the reference made, if done manually,to a certain time in the video source. To this end, we conducted a lab study with tenundergraduate students. Examining a total of 181 identified note samples from the datacollected led to the following findings:• Students use four distinctive representations of content types in engineering notes:text, formula, figure, and the hybrid types.• Most (atleast 75%) of the notes demonstrated a verbatim overlap of some amountwith the watched videos.• The note reference to video is often made to distinct temporal levels in a video.The recorded observations yielded subsequent exploration of the student expectationsand requirements of watched video re-finding mechanism that follows in the next section.41.2.2 Student Expectations, Requirements of a Watched VideoRe-finding SystemTwo research elements can reflect the ability of the proposed mechanism to re-findwatched instructional videos from the already captured notes. First, students’ expec-tations of a prototype system that transfers their existing re-finding strategies. Second,implementation requirements to utilize notebook content as a linking artifact. We brieflyoutline the incorporated methodology to deduce insights on both the expectations andrequirements below:1. A 12-student study elicits the existing workarounds employed by students to linknotes to corresponding video sources. Students’ expectations towards a) videotimepoint retrieval in the conceptualized temporal levels, b) to-be-rendered videosurrogates, the elements of a video to help quick recollection of the included con-tent, were recorded with the help of survey-based questions. Usability evaluationfollowed by the qualitative analysis of the interview transcripts reports the useful-ness and ability of the proposed approach to link note-book content to relevantvideo materials.2. Design and development of a a mid-fidelity prototype present the implementationchallenges and requirements of using Computer Vision to link handwritten notes tovideo content. The implementation process employed readily available off-the-shelfOptical Character Recognition (OCR) techniques and image matching algorithms.Testing the prototype against a set of 181 identified note samples revealed salienttakeaways of the prototype’s linking feasibility.51.3 ContributionsFirstly, the proposed research work provides a new design solution to link physical noteswith corresponding digital video sources. The novelty of the proposed approach is inemploying note content instead of special markers or pre-recorded indices to aid videoretrieval, which is the primary contribution of this work. The empirical pieces of evidencefrom controlled lab studies further support the contributions of the entailed work.1. Firstly, this work recognizes, applies, and articulates the idea of video retrievalusing note content. Identifying an application domain, that is, technical notes cap-tured in selected engineering courses, and testing out a design solution to extractdesign recommendations for that domain is the core contribution of this work. Theconcept also aims to seamlessly bridge the gap between paper and digital learn-ing materials to better user experience in linking different learning materials. Thedesign application of a medium-fidelity prototype combines paper and mobile de-vice merits where students can comfortably scan and access digital video sourcesrelated to the note document. Testing the prototype against a set of note samplesalso demonstrated the ability of the proposed approach by retrieving the correctvideo matches for 77.5% of the total note samples. In particular, 97.5% of the text-only notes matched to their corresponding videos also highlighted the advancementin handwritten recognition of textual elements. The design challenges and impli-cations showcase the feasibility of bringing the proposed approach to use. The factthat early users reported that they find the system usable shows promise.2. Content analysis of watched video notes, an important processing step to exten-sively explore the recognition capacity, is a significant empirical contribution of thework. A controlled lab study investigating ten students’ previously captured note-books reported students’ increased use of textual descriptions in notebooks over the6non-textual representations such as figures and formulas. A special case of hybridtype addresses the complexities of recognizing closely connected text and non-textcomponents. Additionally, the percentage of content overlap between the capturednotes and the watched videos showed that most (at least 50%) of the note sam-ples comprising of formulas and figures showed 100% overlap. In contrast, textualnotes exhibited a more considerable variance of paraphrasing or noting additionalinformation not available in the video materials. Interestingly, box plot distributionindicated that at least 75% of the observed notes with both textual and non-textualcontent types comprised some amount of content overlap. The findings also suggestthat the notes with as low as 25% overlap with the video content also retrieve ex-pected videos. Thus corroborating the design choice of this work, that is to utilizethe notebook content to re-find watched videos3. Outlines inferences on the timepoint in the video rendering. This work specificallyaddresses the difference between the expected video timestamp and the retrievedvideo timestamp by leveraging its temporal nature in point, interval, and wholevideo. Experimental results from a 12-student lab study substantiated the pref-erence for video result playing from any point in time inside the section/intervalwhere the note content occurs. A significantly lower satisfaction score was recordedfor a timepoint anywhere in the video than the timepoint in the relevant section/in-terval in a Wilcoxon signed-rank test (Z = -2.240; p = 0.011). While the apparentexpectation of a video retrieval system is to render the exact matching slide/pointas expected, qualitative analysis of the interview transcripts confirmed a highertemporal preference for interval level with 0 negative codes. The overall choicefor video rendering in the interval temporal level paves the way for a whole newexploration in the video context to be rendered.In sum, note-taking is an important learning activity. This paper discusses the uti-7lization of technology support to prevailing note-taking habits employed by students.Specifically, it describes the characteristics and requirements of a technique that letsstudents link to a video source from which notes are captured. The goal is to allowstudents to continue to produce notes in natural ways without heeding to efforts that goto re-finding or reviewing later, so having the potential to serve interoperability betweenhandwritten notes and video materials for review purposes.1.4 Thesis OverviewThe thesis is organized as follows:Chapter 1 Introduction: Introduces the impediments associated with linking of notesto watched video as well as the motivation and objectives of the work contained in thisthesis.Chapter 2 Background: Describes past work done in learning ecologies, video hyper-links and utilizing handwritten notes.Chapter 3 Characteristics of Watched Video Note Content: Describes the experimen-tal user study conducted to explore the attributes of content types in notes and how avideo timepoint is employed in the linking process.Chapter 4 Rendering Watched Video Matches: Includes study methods and findingsrelated to preferences for video rendering in temporal levels and related surrogates. Ar-ticulates findings on the existing workarounds to link to videos, usability evaluation ofthe proposed approach.Chapter 5 Pragmatics of Linking Notes to Videos: Introduces the design and im-plementation of note recognition, matching, and rendering of video, explains the re-sources necessary for its implementation. Also, synopses of technology requirementsassociated with implementation are covered. Accuracy analysis of matching notes to8videos is demonstrated.Chapter 6 Conclusions and Future Work: Presents the concluding remarks for thisthesis highlighting a summary of contributions and limitations and discusses possiblefuture directions for this work.9CHAPTER 2BackgroundThis section covers some of the foundational studies related to linking paper-based hand-written notes with digital materials such as e-documents, video and audio sources. Learn-ing occurs in coordination with a collection of media types. Dodson et al. found thatundergraduates in flipped classrooms interact with heterogeneous information ecologies,composed of learning materials that span text, video, and audio [23]. These learningmaterials are accessed through a variety of platforms, such as video players, learningmanagement systems, and communication backchannels. A challenge for learners is thelimited interoperability within information ecologies, which can result in a kind of infor-mation archipelagos [43]. The focus of this work is to build on the previous work thathighlights fundamental mechanisms to augment cross-linking between notes and videomaterials in specific. An overview of handwritten notes’ characteristics when learningfrom digital media and the current state of handwritten content recognition is also con-tained in this section.2.1 In-house Video Functionalities for Note-takingNowadays, increasingly, instructional videos are being embedded in traditional courses.Also, video materials are an essential means of information delivery in massive onlineopen courses (MOOCs) and flipped classrooms [21]. Students often overt distinct learn-ing behaviors in video-based learning settings, such as bursts of asynchronous engagementacross various information source materials and(or) selective sampling of video content10for future review. Such characteristic video learning behaviors are classified by the Inter-active, Constructive, Active, and Passive (ICAP) framework [13]. Behaviors that includephysical information manipulation for effective seeking or generation of additional in-formation for comprehension fall under the active learning category. The affordancesavailable in the video learning objects, such as browsing, highlighting, annotating ornote-taking, are administered to control active learning.Studies that operationalize the ICAP framework in relation to video-based learninghave emphasized positive learning outcomes when students engage cognitively and mean-ingfully with the materials rather than passively viewing video content. Mitrovic et al.[60] incorporated ICAP framework behaviours in an active viewing system to learn softskills. The focus was to evaluate conceptual understanding and reflection of video ma-terials by leveraging commenting/rating on videos over passive viewing. Although thefindings indicated higher conceptual understanding, the participants found commentingon the videos and rating comments cognitively demanding.Dodson et al. [22] described students’ behaviors in video-based learning using aViDeX framework that is grounded in the model of active viewing. Common behaviorsidentified in the previous work and their observations were categorized and assignedto distinct groups of ICAP framework as seen in the figure 2.1. They discuss variousaffordances exercised in active and constructive behaviors, such as highlighting transcripts(constructive) and seeking subsequent highlighted instances (active).A growing number of video platforms have included active-viewing affordances tofacilitate video-based learning. Pausing, browsing, replay, and archive mechanisms arenow standard in-built features. As video players are revamping active viewing, newaffordances for constructive behaviors are now a priority.In the learning activities pertaining to constructive behavior, learners construct theirmeaning to expand and extend beyond the video content itself [22]. Video platforms11Figure 2.1: Behaviors of active and passive viewing, categorized using the ICAPframework [22].administer supporting mechanisms both in-house and external information mediums tofacilitate constructive actions. In-house mechanisms support annotation activities suchas adding notes to specific points in the video, highlight/underline a video frame ortranscript, or marking/tagging a video scene. Overall, they focus on elevating the syn-chronization between the notes and the video context for easy review. Marshall [58]articulated various functions of annotations from paper to digital note-taking forms,which can transfer to relevant in-house video actions.First, highlighting or underlining is performed to signal note-taker for future attentionon the specific part of the notes. Fong et al., in a video platform ViDeX, demonstratedtext-book style highlighting of video content [27]. The subsequent findings corroboratedthat video transcripts will allow users to highlight, search, and review the video moreeasily. Second, tagging acts as placeholders to personalize the reflection of an important12point in the video. TagLecture [30] introduced a game to motivate the need to increasethe number of tags associated with video lectures. Consequent investigation revealedthat youth and almost 50% of the adults did not prefer playing the tagging game due tothe type of labels: “they think they are too personal and subjective to tag video lecturesproperly”.Next, in-video note-taking allows recording reflections or interpretations of the infor-mation present. The interfaces today support in-house note-taking that is timestampedto the current video time for efficient review. A tool called Interactive Shared EducationEnvironment (ISEE) [62] explored the issues in video annotation. It generates hyper-linked timestamps, Smartlinks, to associate the notes with video contents. Similarly,Dorn et al., examined the use of spatiotemporal anchored collaboration affordances toenable collaborative annotation and discussion of video content as a first-order learningactivity [24]. Results indicated that students take advantage of the system’s affordancesto interact in meaningful ways, though overall student annotation authoring is restricted.The limitations of in-video note-taking mechanisms are the linear-only capability, mean-ing there is less room for physical non-linear formatting like adding doodlings/markingsof reflections around the notes taken.NoteStruct [52], in-video note-taking system, prompted learners to perform a seriesof note-taking activities. First, learners highlight, comment and add questions whilewatching the video. Second, learners walk through every highlight they made and elab-orate or merge key points. In the final phase, learners review all the notes generatedin steps 1 and 2 in a free-form text editor and are free to edit parts of the notes. Thefree-form editor partly allocates non-linear formatting but fails to transfer the free-formformatting affordances found in the paper medium. Also, the 3-step note-taking is bothtime-consuming and demands additional effort when learning from videos.Furthermore, Note-taking with a video annotation system was compared with hand-13written note-taking when learning from videos by Bargeron et al., [2]. Results indicatedthat note-taking with MRAS, a video annotation system, consumed a lot more time thantraditional note-taking, even though the number of words captured was the same on aver-age. However, it is important to note that participants preferred MRAS over handwrittennotes because MRAS was able to contextualize the notes with the corresponding videotimestamp. This very observation indicates that retrieving the context of captured notesis an important activity to support students’ constructive behaviors.The preceding discussion pointed to several video platforms with features for textbook-style highlighting, note-taking, and tagging, along with their contributions to transferphysical annotating practices. Nevertheless, previous work suggests that students oftentake notes with paper-based notebooks, even when provided with video annotation tools[36] [94]. As Sellen and Harper [77] points out, the paperless office is a myth. Samet al. found that notebooks are used to synthesize, organize and orchestrate importantinformation from multiple class materials in one place for use [23]. The authors alsoemphasize that students prefer interoperability over medium-specific tools. Thus, in thefollowing section, significant work of cross-referencing video materials from paper notesis covered.2.2 Cross-Media Interaction with Paper NotesThe key characteristic of students learning from various information objects is to acquireand keep items of value, as defined in an umbrella term Personal Information Management(PIM) [92] [4]. The onus is often on the students to manage their information in theright place, in the suitable form to meet their learning needs. Microsoft’s OneNote, forexample, provides interactive features for note-taking but also constrains the use of atabbed platform to aid the organization of notes and references to other informationsources. Several researchers have highlighted the general lack of progress within HCI14towards closing the gap between the affordances of paper and the benefits of digitalmedia [5] [4].Hypertext is often thought of digital environments’ content, particularly the Web;however, hypertext can also exist in non-digital environments [50]. For example, Mar-shall’s studies of students’ textbook annotations suggest that annotations are conceptualhyperlinks within and between Lexia [58]. Literary theorists have argued that hyperlinkscan have various extents or scopes [50], allowing links to reference objects at differentlevels of granularity. For example, a student can create a ‘word level’ link by defining anunfamiliar word from her textbook or a ‘collection level’ link by writing a note that sum-marises a chapter. Other scholars have explored the linking between media. Augmentingpaper-based documents within digital information [34], in particular. For example, alepidopterist’s paper-based fieldnotes can be linked to photographs of butterflies [99][54].Embedded Media Markers (EMM) are indicators, frequently employed in terms ofglyph codes, barcodes, or transparent marks, signifying the availability of additional me-dia associated with that part of the document. Post-captured in-video notes are printedon paper to include place markings or marginal markings that promote incidental reflec-tions of material circumstances. Lynn D et al. developed a system for video access fromnotes on a paper medium [93]. The invention includes a note-taking system that allows auser to select keyframes or make annotations during a note-taking session (meeting, pre-sentation, or other activity); the captured annotation indexes a position of a video. Thecaptured keyframes and annotations are printed on a paper sheet with glyph encoding.The printed sheet is used to access the relevant parts of the video by scanning the glyphcodes associated with the target keyframe or annotation. The printed sheet can thenbe annotated or elaborated in free-form, allowing non-linear formatting of the capturednotes.15PaperChains [69] connects paper-based material with digital audio using low-tech,easily accessible equipment. The authors make use of standard camera phone handsets tophotograph physical content and then allow the photographed item to be augmented withdigital audio in multiple locations via interaction with the image. The approach uses twoprecisely placed QR (quick response) codes printed on paper, allowing the PaperChainssystem to detect the item, its orientation, and dimensions, with any camera-enabled mo-bile handset. This method requires no additional specialist hardware (such as a dock orcamera-pen) – users interact directly with the photograph using the phone as a proxy.When reviewing the audio files, touching anywhere on the photograph plays the audiofor that location. Although time-consuming, utilization of EMM-type indicators syn-chronizes notes contextually with the corresponding video material. But the limitationhere is the onus in maintaining media-only special paper notes and also accomodatingthe associated technology requirements.More recent work in this area has adopted technology such as Anoto, which usesa special dot-marked paper and a camera to recognize document areas while writing.ChronoViz [28] integrates researcher’s paper notes into the composite time-coded dataset of video files. It exploits Anoto digital pen technology to support the integration ofpaper-based digital notes. The pen includes an onboard infrared camera and tracks itsposition on the paper in real-time by reading the dot pattern and, in turn, making itpossible to navigate data in more flexible and powerful ways.Today, this work continues with research exploring how novel tools can support paper-based note-taking practices for active, constructive learning when interacting with thevideo. We address the imperative need to nurture interoperability between students’paper-based notebooks, central to information weaving of videos and other media, andthe corresponding context. The integration of interactive navigation techniques withexisting note-taking practices enables students to navigate to moments noted during16initial learning activity quickly.2.3 Searchable-Video DataCurrent work on instructional video indexing and matching gathers the visual, textual,and auditory streams of data from videos, opening up the content of instructional videofor topic matching through traditional information retrieval approaches [45] [83] [87] [97][96]. TRECVid [1], a workshop series, has been pioneering content-based video analysis,retrieval, and detection since 2001. Of special importance is their semantic indexing task,assigning semantic tags to video samples, that aims at evaluating methods and systemsfor detecting visual, auditory, or multi-modal concepts in video shots. The methods followa “bag of visual words” matching approach or more elaborate aggregation methods likegenerating Fisher Vectors or SuperVectors. In VCenter [37], for example, Hsiao andWang segment video into a series of frames from which only the most representativeframes are used for indexing. iVIEW [53] is another system that supports full-contentsearching of multilingual text and audio extracted from the video.Temporal linking appears to be a widespread approach in video platforms for enter-tainment too. In a study of YouTube comments, Yarmand et al. found that commentsreference a variety of temporal aspects of video: from single points to intervals to wholevideos [98]. Our approach utilizes all the visual, textual, and auditory data availablefrom the video materials to deduce matched results of videos from their correspondingnotes. This work also explores the effectiveness of linking note-based video content totemporal aspects, as suggested in the previous studies.172.4 Note Content CharacteristicsNotebooks fulfill a vital function for students because free-form writing, in particular,allows them to leverage a greater breadth of expressions [77]; especially when, in ourcase, writing notes on the content that appear in the videos, which include equations,figures, and tables. Real-time notes are unwieldy: most contain mixed cursive and graph-ical writing, often written at angles, in various sizes, and at multiple locations all overthe page. These constraints do not change the problem statement but require a moreprofound investigation into the content of the notes.Analyses of the lexical structure of notes by Piolet et al., [72] showed three charac-teristics of notes. First, abbreviating procedures, for instance writing down ‘poss’ for‘possibility’. Second, symbols substituting for syntax such as equals, star, arrows. Third,the physical formatting of the notes in a non-linear way. As it is possible that eachindividual uses different abbreviations/formatting techniques, or in some cases the sameindividual can use different abbreviations/substitutes for the same word in various partsof his notes, the current work does not observe the stated characteristics. Rather, thefocus of the work is to examine the features of notes in relation to the video materials.In the context of recall performance evaluation, several characteristics of notebookcontent have been identified in empirical studies, including note quantity [63], note quality[70], and verbatim overlap [63]. Note quantity is the number of words while note qualityis measured by counting the number of factual statements in the notes compared to thelecture video, which can be judged as either right or wrong, present in the notes. Verbatimoverlap stands for word-for-word overlap between the video content and the content ofthe notes. In this work, we focus on the content of notes that can be overlapped andmatched with the video content.182.5 Handwriting RecognitionThe notebook content, written in relevance to instructional videos, must be translatedto machine-readable information to assist the semantic lookup of the video source. Rec-ognizing handwriting content remains an active area of research [73] [86]. A major areaof focus has been on handwriting recognition through the analysis of stroke data, suchas the direction and order letters are drawn [81] [19]. This information is readily cap-tured by devices that support digital ink, such as mobile phones and tablets. However,students continue to use pen and paper for taking notes [36], meaning stroke data isnot always available. Applications in off-line handwriting recognition through variousdomains include recognizing postal addresses [80], bank checks [29], writer identification[65], historical document recognition [65], and calligraphy imitation [102]. Various stud-ies comparing on-line versus off-line handwritten recognition [73] [86] have articulatedthe underlying methods and gaps in the respective categories.Yuan and Seales [100] proposed a semi-automatic solution to reading manuscriptsthat involves a document analysis (DA) module and a graphical user interface. The DAmodule detects and ranks regions of interest in an image using which users can manuallyconfigure the parameters of the DA module, thereby ranking candidate regions for linking.Our work, on the contrary, proposes a fully automated solution that identifies regions ofinterest for each image in every video from the library and delivers a ranked list of videoswithout user intervention.Alternative approaches to handwriting word recognition (HWR) have demonstratedsubstantial improvements in recognition accuracy by employing a combination of con-volutional and recurrent neural networks [12] [71] [74] [102]. Deep-learning approachesrequire a large amount of training data to discern the most important features for char-acter recognition. Collecting and annotating a sufficiently large dataset of handwrittennotes with connected components in different settings remains expensive and laborious.19To our knowledge, no such dataset is publicly available. Therefore, in this work, themedium-fidelity prototype is built upon readily available off-the-shelf Optical CharacterRecognition (OCR) engines whose application can be scalable to different note types.20CHAPTER 3Characteristics of Watched Video NotesThis chapter aims to investigate how and what types of notebook content link to whattimepoint in a video. The diversity of the captured note content and the reference pointin the video material are surveyed and discussed. Identifying students’ notes’ distinctcharacteristics is especially trivial to inform the selection of effective recognition tech-niques. We also deduce a taxonomy of handwritten expressions and the temporal levels ofreferencing the corresponding video timepoint. Additionally, a dataset of watched videonotes is created, which is later used for testing the performance of an automated linkingprototype.3.1 Notes CollectionIn this section, previously taken student notes on instructional videos are analyzed forany information that could connect to the video. We inspect the identified informationto uncover the attributes of 1) the content type of the notes and 2) the correspondingdestination within the video, allowing us to compare the students’ notes to the correctpoint in the video.3.1.1 ParticipantsWe recruited ten university students — four from an electromagnetics course, two frommachine learning, three from a data analysis course, and one from a crash course onsoftware engineering. The study targeted students from engineering courses specifically21to increase the likelihood of finding diverse types of technical notes. 4 of the 12 studentswere undergraduates and 6 graduates, all of them between 18 to 28 years old. Table3.1 shows the demographic data of the ten students, their level of education at the timeof the study, and the course from which the notes. The table also includes the numberof videos and the average length of the videos in each course. The inclusion criteriawere that each unit of sample possessed pre-written notes recorded on paper notebookswatching instructional videos. Before the study, each participant confirmed they had atleast eight pages of handwritten notes they had taken while learning with video thatensured a sufficient amount of data in each session.3.1.2 ProcedureStudents located pages in their notebooks containing handwritten notes related to apoint in an instructional video. Each instance of a video-related note was recorded as alink to populate a dataset of notebook-video mappings. A flatbed scanner recorded notepages to maintain the resolution, luminance, and color quality of the data collected. Thenotes collected were full-page copies of students’ notebooks. Given that we made copiesof students’ actual notes, they might have included personally identifiable informationor content that they would like to exclude from the database. Therefore, each partici-pant was provided with a plain piece of paper to conceal all the content they were notcomfortable sharing.Ground truthing was a crucial step for understanding the linking patterns and val-idating the core idea of note-content-based video access. We collected student-labeledground truth observations along with the actual copies of video-related notebook content.The course videos that students learned from were available in a video learning platformViDeX [27] prior to the study. We used a laptop to view the scanned notebook pagesand the videos to which the notes belonged. Once the copy of relevant notebook pages22Table 3.1: Demographic data of Participants P1 to P10Participant Gender Level Course No. ofVideos/-CourseAveragevideolength(hh:mm:ss)P1 Female 2nd year,Undergradu-ateEngineeringElectromagnetics(ELEC211)15 0:07:56P2 Female 2nd year,Undergradu-ateCircuits & Elec-tromagnetics(BMEG220)16 0:08:03P3 Male 1st year,MAScMachine Learn-ing (CPSC340)12 0:50:43P4 Male 2nd year,Undergradu-ateCircuits & Elec-tromagnetics(BMEG220)16 0:08:03P5 Female 2nd year,Undergradu-ateEngineeringElectromagnetics(ELEC211)15 0:07:56P6 Male 1st year,MAScReact, JS 1 2:25:26P7 Male 1st year,MAScMachine Learn-ing (CPSC340)12 0:50:43P8 Male 2nd year,MengData Analysis(EOSC510)13 0:18:41P9 Male 1st year,MAScData Analysis(EOSC510)13 0:18:41P10 Female 2nd year,MAScData Analysis(EOSC510)13 0:18:41was taken, participants marked a rectangular box around the content that can be linkedto the video on each page using editing software like paint, PowerPoint slides. Eachmarking included the following details as shown in 3.1.1. Name/title of the video.2. Timestamp of the referenced video material that links to the note content.3. A confidence interval representing their level of confidence, measured using the23Figure 3.1: An example of a student-labelled ground truth of a scanned note, record-ing 1) the name of the matching video, 2) the timestamp within the matching video,and 3) the students’ Remember, Know, Guess confidence interval.Remember, Know, Guess paradigm [25], of their responses. Participants reportedif they remembered the video occurrence or knew about the occurrence or guessedif they took some time to go through the videos and guessed the occurrence.We returned students’ notebooks after completing the data collection session, whichtook approximately 45 minutes, during which we gathered as many links as possible.Pilot testing with four students prior to the data collection recorded any impedimentsthrough the process that needs attention.3.1.3 Description of the Captured NotesFive to ten pages of notes per participant were recorded, comprising a minimum of 10to a maximum of 35 video links. An important aspect to note is that each student tookconsiderable time to scroll through the videos and the video timeline to record the relevanttimestamp. However, the time taken to identify links did not indicate a general pattern.Also, we did not see any consistency in the number of notes per student and their capacityof recalling the related video context as Remember, Know, or Guess. For instance, twostudents from the same course had contrasting patterns of link identification. P1 marked9 note links to 9 different videos, whereas P5 marked 20 links to 6 different videos, withat least 2 links pointing to each video.Students marked most video links with a confidence level of Remember when asked to24indicate the basis on which they judged that they had studied the video previously. Theapparent reason could be as students had the videos included as part of their ongoingcurriculum. They marked two links as Know and four as Guess. One student tagged apart of the note page as video-related content but did not remember the video and markedthe confidence level as ‘Unsure’. In total, we collected 181 video links which can be seenin detail in A.1 and analyzed for various characteristics so that we can apply appropriaterecognition analysis in the later phase. It is particularly imperative to identify the typesof note content that exist to aid the recognition of handwritten content and thereby elicitdesign ideas to support the robust linking of notes to videos.3.2 Results and DiscussionThe identified set of notes and their corresponding video references set the stage for threemain subtasks:1. Understanding the similarities and differences between the various content types ofthe notes.2. Developing a taxonomy of similar notes.3. Understanding the temporal nature of the video references.3.2.1 Content-based Categorization of Note ContentWe observed the links for collective identities whose characteristics were derived based onthe subject matter in the note content. Previous work has demonstrated the distinctionbetween textual and non-textual content types in handwritten notes [20] [38]. Theyemphasize the need for assigning a specific content type to corresponding specializedrecognition systems for high-quality accuracy. Accordingly, the following four main unitscategorize the links based on the content type:25Figure 3.2: Notes with text-only content1. Text: A link that has plain textual content. This type includes groups that havetextual descriptions such as comparison table, a summary of a concept, definitions,bulletin of important pointers, freestyle flowcharts, piece of code. An example foreach of these groups is shown in 3.2.2. Formula: A link that includes mathematical symbols, equations as shown in figure3.3. One line equation or a block of derivation were both regarded as formulas.3. Figure: A link that comprises handwritten illustrations of graphical elements.This includes pictorial representations of electromagnetic elements, graphs, circuitswhose examples can be seen in figure 3.4.4. Hybrid: A link that uses a combination of the above three types in close connectionas shown in figure 3.5. An example could be a figure with one or more lines oftextual details of the figure.The categorization was devised based on the examples described in the hierarchicalannotation of online handwritten documents [38]. Mapping video-related note content26Figure 3.3: Notes with formula-only contentFigure 3.4: Notes with figure-only contentto the categories mentioned above showed that most links were text-based. 39.22% ofthe references (n = 71) were written letters or words, 34.25% (n = 62) were formulas,and 4.97% (n = 9) were drawings of figures/drawings/circuits, not surprising given thatour participants are student engineers. We also found 21.54% (n = 39) of hybrid casesrepresenting a combination of textual and non-textual content. It is important to notethat textual content type accounted for the highest usage of the video links data collected,indicating its broader use over other content types to link to videos. This is consistent27Figure 3.5: Notes with a combination of textual and non-textual contentwith the findings from previous work [38] that reports text as the frequently used contenttype in online handwritten documents.3.2.2 Unfolding the Hybrid TypeIn hybrid content types, textual and non-textual contents haphazardly amalgamate tostructure a reference to some point in a video. This poses a challenge in assigning arecognition technique to hybrid types found in notes. Further scrutiny into the samplesof this type displayed that one of the 2 or 3 types dominate the other in obvious ways,as listed below. We leveraged this factor of the predominance of one type over the othersto unfold and assign the hybrid type to text, formula, or figure.1. The content load of one type exceeds the other types. This can be seen in thesample shown in figure 3.6. The link comprises all three types; text, formula, and28figure. However, it has more textual characters and letters when compared tomathematical symbols and pictorial representations. So, the content type chosenfor this case is text.2. The topical similarity between one content type and the video far exceeding theother(s) distinguishes the hybrid types. Figure 3.7 shows one such example. Al-though both content types have about an equal number of characters, the text‘sample mean’ here is not unique and can be semantically related to several videos.However, the equation can be a finer query that could be semantically close to theexpected video. Hence, the formula was the choice in this case.3. Sometimes, the significance of a content type is manually highlighted in hybridnotes. In the sample shown in figure 3.8, equations are highlighted in textual boxesto indicate its significance. Hence the accentuated type was chosen, in this case,formula.We applied the above-discussed directions to all the hybrid cases to assign eachcase to one of the other three content types. In the end, 98 of the total recordswere tagged as text type, 65 as formula, and 18 as figure.3.2.3 Verbatim OverlapThe Verbatim overlap is a measure of the word-for-word overlap between the lecturecontent and the content of the notes. Mueller and Oppenheimer [63] explored the effectsof verbatim overlap in three studies (n = 65, n = 149, and n = 109). In all three studies,all participants took notes, either by hand or with a computer. The authors scored thesenotes on their verbatim overlap. They compared a fixed chunk of text in the notes witha chunk of text in the lecture transcript and reported a percentage of match for each.We followed a similar procedure to calculate the percentage of matching for each29Figure 3.6: Hybrid link consisting of connected text, non-text componentsFigure 3.7: Hybrid link with semantically broad text contentFigure 3.8: Hybrid link with highlighted content30recorded link in the notes. As the content of the notes included both textual and non-textual components, each chunk of the components in the notes was systematically com-pared with the matching chunk in the video with the following guidelines.• Firstly, for text, each word that is a noun or a term that has lexical significance andis not a stop word was considered as a chunk of data. Stop words are commonlyused words, such as “how”, “what”, “or”, that are excluded from searches to parseinformation objects faster. We identified the chunks discretely irrespective of thesequence of their appearance.• For formulas, defining a chunk was not straightforward as the length of equationscould vary from a simple two-term equation to a number of lines. Therefore, anequation connected by less than or equal to 3 terms on either side of the equals signwas regarded as a chunk. If an equation exceeded more than 3 terms, we assignedit to the next chunk.• In figures, a graphical representation that has connecting nodes from one end toanother was considered one chunk. As we looked into freehand paper drawings,if the figure was structurally similar to the one in the video, the overlap was trueotherwise false.• For the hybrid type, we identified the chunks according to the type assigned. Forexample, if the hybrid representation has a higher number of graphical components,we followed the figures’ procedure to identify and match the chunks.Therefore, we computed the percentage of content overlap as:% of overlap = # of matching chunkstotal # of chunks in the notes ∗ 100 (3.1)31Figure 3.9: Box plot comparison of content overlap in the text, formula and figurerepresentations respectively.The box plot in figure 3.9 visualizes the overall distribution of the content overlapacross text, formula, and figure representations in the watched video note content.A glance at the plots suggests that the Inter Quartile Regions (IQRs) of all the threeboxes are above the 50% overlap. This very fact implies that at least half of the notelinks identified in each type, that is, text, formula, and figure, matched half or more ofthe content chunk for chunk as found in the videos. We outline some of the observationsspecific to each type below:• In the case of texts and figures, at least 25% of the note samples showed an overlap ofless than 50% giving rise to two possible interpretations. One, students completely32paraphrased the content in the notes in one’s own words or two, the matching videotimestamp as indicated in the ground truth did not exactly match the context ofthe notes. The IQR for the text type also exhibits a larger variance in the 50-100%window of overlap compared to the other two types. This indicates that with thetext type, there is more paraphrasing when compared to formula and figures.• The median is the lowest for the text type pointing to 75% of overlap and is at100% for the formula and figure. Formulas in specific manifest 100% overlap in thecase of at least half of the formula-based notes. This implies that when learningfrom videos, formulas are transferred literally to notes for reviewing purposes.• Furthermore, for both text and figure, there are cases when there is no overlapat all, with a minimum overlap of 0%. But, it is extreme in the case of formula,indicating some amount of obvious overlap for most cases.3.2.4 Watched Video TimepointInvestigating how students marked the expected timestamp of video to each part of noteswas crucial in determining the predicted outcome of searching through the videos. Weexamined the timestamp — i.e., the link destination — recorded as the ground truth foreach collected note page and registered the following observations:1. The note content did not always point to the video slide as indicated in the datacollected.2. Expected timestamp did not always indicate the start of an interval. The videocontent of the notes spread across a stretch in the video. When a software code isexplained, for instance.33The recorded timestamp with the video title failed to indicate if a student is trying tore-find the whole video, an interval of similar content in the video, or a particular slide.Thus, there is added ambiguity in the video timestamp to be rendered that correspondsto expected. It is crucial to rationalize the video timepoint rendering evaluation toarticulate the ability of an automated linking approach.To this end, we conceptualized the timestamp to be rendered on the temporal at-tributes in the video content. Previous work has emphasized the provision of temporalcontext within search results in video [9] [90]. Yarmand et al., [98] presented a taxon-omy for classifying video references in YouTube comments by temporal specificity. Theauthors identified three temporal characteristics of video referencing as Point that refer-ences distinct moment, Interval that references a span/section of time, and Whole Videothat references the entire video. Accordingly, we define the retrieved video timepoint asa timepoint in the three temporal levels:1. Point: when a video timestamp is pointing to specific content in a video,2. Interval: when a video timestamp is pointing to an interval of the related videosection,3. Video: when a video timestamp is anywhere in the video3.3 SummaryWe performed a needs assessment by collecting previously taken notes of watched videosthrough a lab study with ten students to crucially analyze the types of notes and theirattributes to being used as query elements. First, we articulated the outlook of identifyingvarious information types: text, formula, drawing and hybrid, in the watched video notes.It is important to note that textual content type accounted for about half of the videolink data collected, indicating its broader use over other content types to link to videos.34It is also important to note that the extension of hybrid type to one of the other threetypes also lead to a majority of text-dominant links; 26 out of 38 hybrid notes, to beprecise. This is consistent with the findings from previous work [38] that reports text asthe frequently used content type in handwritten documents.In addition to the identified types of notes, investigation of the verbatim overlap factoralso yielded interesting insights in relation to each type. The box plot interpretationsimplied that students tend to copy important formulas and figures as it is from the videoslides, but paraphrase or summarize the textual explanations into their notes. Overall, atleast 75% of the notes in each type showed some amount of verbatim overlap, confirmingthe ability of watched video notes as efficient query elements to retrieve related videos.While the 100% overlap of content between notes and corresponding video in at leasthalf of the identified formula and figure note samples, and in 25% of the text samples, isa promising finding to aid the proposed approach, 0% overlap for text and figure samplesposes a trivial challenge.We conceptualized the timestamp in the resulting video on the temporal scale to atimepoint in point, section, or whole video. We believe the problem of accounting for adifference between the retrieved to expected video timestamp has been addressed withthe temporal attributes through this approach. However, student reviews on the videoretrieval in the 3 levels and other requirements in the video viewing end are importantto determine the comprehensive features in the future systems. On the whole, the estab-lished characteristics of both note content and the corresponding watched video addressthe first research question of the work. In the following chapters, the emphasis is onexploring how to link back to the video source material in one of the three temporal lev-els. Also, we address the optimal temporal level among the three groups in the retrievedvideo.35CHAPTER 4Rendering Watched Video MatchesStudents’ attitude on employing a linking device as an educational tool is vital to design,evaluate and roll out the right technology. As discussed in chapter 3, the timestamprendered in one of the temporal levels can influence the likelihood of determining thevideo result as a potential match or not. Therefore, we evaluate the acceptable temporaldifference between the retrieved video and the expected in this chapter. In turn, thechapter also draws on the inclusion of other video-related objects, along with the videoresult playing from an expected timestamp, to conveniently establish the match. Drawinginferences on these points are imperative in determining the optimization parameters forthe system’s ability to predict video matches. Therefore, the objective of this chapter isthree-fold:1. To learn the temporal specificity, the preferred temporal level among point, interval,and whole video levels where the temporal difference between retrieved to expectedis acceptable.2. To examine the preferences for video surrogates along with the video when renderedon search.3. To evaluate the usability of the proposed approach, that is the idea of an automatedlinking interface over the existing linking approaches.364.1 Experiment Design and ProcedureWe conducted a user study with 12 Engineering students to demonstrate the preferencefor temporal specificity in point, interval, and whole video, along with the expectations forvideo surrogates in search results. With approval from the University of British Columbia(UBC) Behavioural Research Ethics Board, we obtained informed consent was obtainedfrom all the participants. A pilot study with 2 participants confirmed any setbacks in theprocedure of the study. The study began with only 4 participants to determine the totalsample size. An a-priori power analysis was performed with the data received from the 4participants using the G*Power application [26]. The chosen test family was the t-test,and the chosen statistical test was the ‘Means: Difference between two dependent means(matched pairs)’ test. We calculated the effect size based on the Likert-scale data for aquestion comparing the information effectiveness among the three temporal levels ‘Theinformation is effective in helping me complete the tasks and scenarios for learning’. Thepoint level condition was the control group compared with the interval and whole videolevels as the treatment groups. The calculated effect size was dz = 1.03 for point-intervalgroup and α = 1.59 for the point-whole video using the difference of means and standarddeviations. The computed sample size was 6, 12 choosing an input error probability of0:05 and desired power of 0:95. We finalized the sample size as 12 and recruited studentsaccordingly.4.1.1 ParticipantsWith a sample comprising 12 university students, the controlled lab-based study includedstudents from a range of engineering disciplines and degree levels. Table 4.1 outlines thestudents’ basic information. We reached undergraduate and graduate engineering stu-dents with convenience sampling. Inclusion criteria were that students were over eighteen37years of age and had experience learning with video. We scheduled half-hour appoint-ments with each participant. The entire session was audio-recorded, which enabled us tofocus on the verbal prompts. A verbatim transcript of the interview was later generated.Table 4.1: Demographic data of the 12 studentsParticipants Gender LevelP1 Female MASCP2 Female PhDP3 Female MASCP4 Male PhDP5 Male MASCP6 Male MASCP7 Male MASCP8 Male MASCP9 Male 2nd year, UndergraduateP10 Male 2nd year, UndergraduateP11 Female 2nd year, UndergraduateP12 Female 4th year, Undergraduate4.1.2 Study ProcedureThe conducted study was a within-subject study in which each subject performed thesame tasks in all the conditions. Before beginning with the key tasks of the study,each participant went through an introductory session where they shared their existingworkarounds when learning from the videos. The session was followed by a mock video-learning task to help students familiarize themselves with the video-learning settings andalso generate sufficient real-time notes to guide the subsequent tasks. The remainder ofthe procedure was designed to address the key objectives of this chapter individually.The study began with a few experience-based questions to learn about students’current video learning, reviewing, and note-taking habits. The questions followed a semi-structured interview pattern to allow students to express their reviews related to learningfrom videos. We performed a 2-student pilot testing before the experiment to formulate38the questions that sensitively addressed video-based learning reviews. Although thesession included exploratory questions, key questions asked were as follows:1. How often do you learn from videos? What kinds?2. What measures do you take to help you remember the things that you have learnedfor reviewing later?3. How do these measures help you re-find the source efficiently?As a next step, we gave participants a realistic task scenario: to learn from an instruc-tional video and complete a comprehension quiz to improve ecological validity, in linewith Borlund’s work on simulated work tasks in lab studies [6]. We modeled the task togenerate real-time notes whose purpose was to deduce inferences on the experience andtraits involved in the process of video-based learning. Additionally, participants used thenotes to reflect on the various conditions in the later part of the study.Initially, we selected one 5-minute video to learn. The pilot study showed that stu-dents have varied interests and might get frustrated to learn unrelated content. Hence,we selected 3 videos of different genres to accommodate the curriculum-based choice ofpreferred video. Three instructional videos that spanned approximately 5 minutes eachwere chosen beforehand. Each student was free to choose any video from the three. Thetopic of the videos were: 1) Introduction to capacitors, 2) Nervous system, and 3) Re-inforcement learning. The motive of the study was not revealed to the students at thispoint to avoid leading participants to perform a certain way. We told them that it is nec-essary to remember the video content after the viewing session and connect back to thevideo during the study. Participants were given blank sheets of paper to take notes whilethey watched the video. Students were asked to identify parts of notes that pointed tovideo-related content after completing the video learning session. Their general learningand linking habits were discussed, which later followed to a discussion on the conditions39that supported the objective of the study. After gleaning on various existing linkingstrategies carried out, we explained the intent of our proposed linking approach to thestudents.4.1.2.1 Temporal SpecificityThe first objective of this study was to investigate the acceptable temporal differencein the video rendered to that of expected. We considered three conditions of deliveringa timepoint in point level, interval level, and whole video level. The conditions weredemonstrated relative to the participant’s notes captured in the previous step and therelated video. For instance, participants noted a video timestamp that connects to partof their notes as mm:ss in video time. In the case of point condition, the timestamp to beretrieved was demonstrated in two to three timepoints, say mm:ss+0:02 or mm:ss-0:02in the same frame that pointed to the noted timestamp. If a frame spanned for about 30seconds and the noted timestamp is inside the frame, the retrieved timestamp is eitherthe start, middle, or end of the 30 seconds window. Thus, the temporal difference isnow in this window condition. We repeated the procedure for the other two temporalconditions, that is, various timepoints in the corresponding section and the whole video,the order counterbalanced. Students discussed the pros and cons of all three conditionsin detail to observe the acceptable difference. Additionally, they provided the preferencefor each condition based on the following after-scenario questionnaire on a Likert scaleof 1 (strongly disagree) to 5 (strongly agree):1. It is easy to find the information I need from this point.2. The information is effective in helping me complete the tasks and scenarios forlearning.3. I’m satisfied with the retrieved video time point.404.1.2.2 Re-scan, list of search resultsSubmitting a meaningful query is essential for video search effectiveness from a large videoarchive [91]. Taking the example of the Google search engine, a user often reformulatesor rephrases the topical content of queries to retrieve the desired information. Similarly,students expressed their reviews on performing re-search operations with other relevantvideo-related queries from their notes. The discussion also covered traditional ranked-listpresentation of more than one to two video results in the mobile scenario.Along with the video and the timepoint at the acceptable temporal level, the videosurrogates’ rendering is significant in the video retrieval process. The next step was toglean on the video surrogate’s expectations when rendering the predicted video.4.1.2.3 Preference for Video SurrogatesPopular search engines such as Google provide a set of keywords along with the heading,URL in search results to highlight query relevant pages. This page-related informationpresented are surrogates for a web platform that assists searchers in deciding whether toexamine the full web page. Similarly, surrogates for video are a condensed representationof the video as a whole while summarizing key features to cue a quick sense of the expectedvideo. They facilitate rapid sense-making. Textual and non-textual video surrogates wereused in Marchionini’s work [57] to aid rapid gist determinations from the list of resultsto select which video to watch. While there are numerous types, four fundamental andfrequently exercised video surrogates were selected for the study: 1) Title of the video,2) Thumbnail, a small image that exemplifies video content, 3) Keywords to indicatesignificant video substance and, 4) A summary of the video content. We demonstratedthree sample designs with the four surrogates to the students:1. Video with title and thumbnail41Figure 4.1: An example video surrogate, comprised of a thumbnail (a), title (b),keywords (c), and summary (d).2. Video with title, thumbnail, and keywords3. Video with title, thumbnail, keywords, and summary.Figure 4.1 shows the design sample for the third condition. Students’ preferences foreach sample were discussed and rated based on a set of Likert-scale questions with a scaleof 1 (strongly disagree) to 5 (strongly agree) in three scenarios:1. It is easy to find the information I need from this point.2. This design has all the functions and capabilities I expect it to have.3. The organization of information on the layout screen is clear.4.1.2.4 UsabilityEach participant talked about the usability of the proposed system as a working applica-tion, recorded in terms of three key elements; efficiency, effectiveness, and satisfaction [9].The ratings for the three elements were captured through the following set of Likert-scalequestions (1; strongly disagree to 5; strongly agree).421. Effectiveness is indicated by the quality of solution, accuracy, and completenesswith which users complete specific tasks. Following questions were used to reporton the effectiveness of our approach:(a) I think the system is easy to use.(b) I found the various functions in this system were well integrated.(c) I think that I would need the support of a technical person to be able to usethis system.2. Efficiency is shown by task completion time or learning time which was capturedusing the following questions in our study:(a) I believe I could become productive quickly using this system.(b) I would imagine that most people would learn to use this system very quickly.(c) I needed to learn a lot of things before I could get going with this system.3. Satisfaction is confirmed by users’ comfort and positive attitudes towards usingthe system. The study reported on users’ satisfaction rates based on the belowquestions:(a) I think that I would like to use this system frequently.(b) I found the system unnecessarily complex.(c) Overall, I am satisfied with this system.The items for each usability element were phrased with two positive and one negativequestion to avoid extreme response bias and acquiescent bias. Interview data was tran-scribed and de-identified before analysis.434.2 Study FindingsIn this section, we refer to the findings from the study discussed in the previous section,particularly with respect to preferences in terms of video retrieval and attitudes towardsusing the proposed system.4.2.1 Video in “Interval” Temporal LevelWe compared the difference in preference for the temporal levels of video timepoints(i.e., retrieved video timepoint in point, interval, or full video) based on the data fromthe three Likert-scale questions. Since the data collected was ordinal, it failed to meet theassumptions for parametric tests. Thus, we performed the Friedman test, a parametricalternative, that confirms whether there are overall differences between the groups ofconsideration. But it does not pinpoint which groups, in particular, differ from eachother. To achieve this, one needs to run a post-hoc analysis. We conducted Wilcoxonsigned-rank test with a Bonferroni correction applied, resulting in a significance level setat α < 0.017. Here we used an exact p-value considering the small sample size (n = 12).We analyzed the three questions, respectively.1. First, participants’ perceived ease in finding information from the retrieved timepoint was statistically significantly different among the three timepoint groups(χ2(2) = 7.4, p = 0.021). However, the post-hoc test did not locate significantdifferences between the combinations of the three groups despite an overall reduc-tion in the whole video group compared to the point and interval groups.2. Participants’ perceived effectiveness in completing tasks for learning was statisti-cally and significantly different among the groups (χ2(2) = 6.897, p = 0.029). Butno statistical difference was observed in the combinations.443. Lastly, participants’ satisfaction with the retrieved timepoint was significantly dif-ferent in the groups (χ2(2) = 9.190, p = 0.008). Additionally, the post-hoc testshowed significant difference in the video versus interval groups (Z = −2.240; p =0.011). Participants felt higher satisfaction when the retrieved video timepoint wasin the interval when compared to anywhere in the whole video. The Ranks tableprovided some interesting data on the comparison of participants’ satisfaction forthe two groups. Ten participants had a higher satisfaction score for video playingin the interval of the expected content occurrence over a timepoint anywhere in thewhole video. However, only 1 participant had a higher score for the whole videocondition, and one participant saw no change in terms of satisfaction.To learn more about the preference for the video timepoint retrieval in one of thethree temporal groups, interview transcripts were examined for explanatory quotes. Thequotes from the 12 participants were coded with five different preference levels, very good,good, neutral, bad, very bad as seen in figure 4.2. For example, P13 had the followingreviews for each category:• Whole video; That can be really inconvenient because some videos are like twohours → coded to very bad• Interval; it is somewhat helpful I’ll probably be pretty satisfied → coded to good• Point; Well, then I’ll just be pretty happy. Yeah. Gets me to where I want to go→ coded to very good.Quotes were coded as neutral if preference changed with contexts. For example, P1reported that the length of the video influences preference when timepoint retrieved isanywhere in the video; ‘If it’s a 15-minute video. Yeah, I would be OK with it becauseit’s just 15 minutes you would want to devote like a solid understanding and if it’s a day45Figure 4.2: Coded mapping of preference levels (i.e., very good, good, neutral, bad,and very bad) to each level of temporal specificity (i.e., point, interval, and wholevideo).before an exam. Yeah, but if it’s an hour or two hours then it would become slight bit ofan inconvenience to find it.’Following the coded mapping of preference to each temporal group across the 12participants, the point-level temporal group presented 9 - very good, 1 - good, and 1 -negative code tagged to bad. The interval-level temporal group displayed no negativecodes, 8 - good and 4 - very good codes. Six participants exhibited negative codes, bad,in the case of the video-level temporal group with five participants still indicating positivecodes, 2 - very good and 3 - good.4.2.2 Selection-based Use of Video SurrogatesTo examine the video metadata cues to be rendered along with the video, data fromLikert-scale questions was analyzed to compare the three conditions: 1) Title + Thumb-nail, 2) Title + Thumbnail + Keywords, and 3) Title + Thumbnail + Keywords +Summary. Once again, we performed the Freidman test along with the Wilcoxon signed-46rank test to investigate the difference between the three conditions.Participants’ perceived ease in finding information from the retrieved timepoint showedno statistically significant difference between the conditions (χ2(2) = 2.889, p = 0.240).Participants’ expectations of the design (i.e., design has all the expected functions) alsoshowed no statistically significant difference between the conditions (χ2(2) = 2.8, p =0.272). Participants’ perceived information clarity was statistically significantly differ-ent among the groups (χ2(2) = 8.914, p = 0.008). Post hoc test indicated a significantdifference in the clarity of information organization among the summary and keywordsurrogates (Z = −2.762; p = 0.004). Ranks showed that 9 participants had a higherclarity score for keywords than the summary. No participant preferred summary overkeywords. However, three participants saw no change in the organization with the twogroups.Since there was no significant difference recorded in 2 out of the 3 likert-scale ques-tions, we could not deduce preference for each of the selected surrogates. Although therewas the influence of the video surrogate combination in the organization of informa-tion, it was not clear about the ease of finding information. Interview excerpts furthersupported this evidence. Participant P3 preferred keywords; ‘Yeah. I would prefer thekeywords most because that one is short. For the abstract one, if it’s one or two sen-tences, I think that will be better.’ Participant P10 pointed to the need for summary;‘I think more the information more it will be easy for me. So ok from here it is likemore information so I can just read through and recall if that’s the video I’m looking for ’.Therefore, Select or Deselect action items can be included in an expandable list of videosurrogate choices. When a student first views the video result on search, the defaultvalue, title + thumbnail, is the choice that appears in the video control and selects theother surrogate options that they want.The inclusion of transcripts also seemed to be influenced by the author’s accent, video47content type, compact video viewing in mobile applications. An additional observationwas that all the participants except one (P3) preferred seeing a thumbnail that showedthe video content that matches the note content. For example, P9 informed that ‘ifthat slide would be in the thumbnail, Yeah that would be like the most optimal scenario’.However, P3 reported that it might not be as useful: ‘I think it’s because, with the typicalvideo thumbnail, I think for a video they always have a thumbnail selected for that videoin the system and I remember that thumbnail. And if you change the thumbnail, duringthe search, I’ll probably get confused.’4.2.3 Usability reportsWe analyzed the final survey data on usability. The figures 4.3, 4.4, and 4.5 belowshow the stacked bar charts constructed from Likert scale data for 3 different usabilityelements: Effectiveness, Efficiency and Satisfaction. The counts of participants on eachrow who agree with the statement are shown to the right in dark blue color; the countswho disagree are shown to the left in dark red color. Participants who neither agree nordisagree are shown in a neutral color.Participants thought that NoteLink was easy to use (M = 4.42, SD = 0.49) and wereable to become productive quickly using NoteLink (M = 4.33, SD = 0.62). Overall,participants were satisfied with our NoteLink (M = 4.33, SD = 0.47). However, thereports did not indicate complete agreement in terms of the system’s functioning. 2participants marked ‘neutral’ when asked about the system’s integration of functions. 2Participants also agreed that there is lot to learn. All the 12 participants asserted thatthe system would be very useful while learning from videos if it were a real applicationand if it could scope well to their expectations. P1 said, ‘If it had things I want, yes.Very useful. Saves so much time, and maybe I’ll watch more videos then.’ P2 agreed:‘Definitely, for videos that have to be bookmarked. This is very useful.’48Figure 4.3: Stacked bar chart showing likert-scale observations for Effectiveness.Figure 4.4: Stacked bar chart showing likert-scale observations for Efficiency.49Figure 4.5: Stacked bar graph showing likert-scale observations for Satisfaction.Lastly, qualitative analysis of the interview transcripts regarding the traditional listof search results showed that 11 out of 12 participants preferred having a list of probablevideo matches (in particular, three videos on average). For instance, P1 preferred havingat most three options; ‘Options are definitely good. But how many options are there? Idon’t want something like google where it has pages of matches. Maybe three, not morethan that’. Therefore, we decided to go with three options with an option to view morewhen required.4.3 DiscussionThe video timestamp to be retrieved was conceptualized and evaluated on a temporalscale to support semantic evaluation of the system’s retrieval prediction. We establishedan overall difference in the preference for the predicted video timepoint in the three condi-tions; point/interval/Whole-video. The interval group, in particular, showed a significantincrease in satisfaction compared to anywhere in the whole video. The reason is likelycontextual, involving several aspects of the viewer’s information need at the time that50might influence the preferred temporal level. For instance, P9 had his preference changewith the length of the video and complexity of the topic: ‘If it’s a short video, I’d beokay with just anywhere in the video. If it’s a long video, I wouldn’t be okay with just thevideo. If it’s a very complex topic, I would definitely not prefer being taken to the point.But, if it’s a rather simple topic, of course, I’d be okay with just, you know, like beingtaken to the point. ...The sweet spot would be to use the middle one — interval.’ Thecoded preference mapping in figure 4.2 also showed that interval level match point is themost opted with 0 negative codes. The point-based temporal specificity also showed themost positive codes (10 out of 12). However, 6 out of 12 participants shared negativecodes for the whole video level, indicating lower preference compared to the other twolevels.Interview transcripts revealed students’ preference to view a list of probable videomatches seen in the traditional queries to digital libraries. Christel [14] discussed twomajor challenges in locating needed information from the list of results: 1) Informationreturned can be too much, and 2) Information needs are different for different users.Similar trends were observed in the preference for video surrogates: title, thumbnail,keywords, and summary. Therefore, the enable/disable feature-based selection of videosurrogates from the ranked list of video matches is suggested in this work. Furthermore,emulating design recommendations from literature [10], [82] on effective mobile informa-tion retrieval can boost the performance of our system in the human-computer interactiveperspective.Analysis of interview excerpts along with the notes captured during the study led tointeresting themes towards linking and note-taking habits.514.3.1 Link to Note vs Note of Links – Linking StrategiesWe analyzed the current conventional workarounds that students perform to link theirnotes to videos in the interview excerpts. The findings led to a list of linking strategiesthat can compile identical methods employed in digital and paper-based physical medi-ums. This is a unique observation over the traditional categorization of digital versusphysical linking techniques. We classified the collection of linking methods into two maingroups whose elements are listed in the table 4.2:1. One category comprises of links or video cues such as title, timestamp, contentheadings, and many more adjacent to the relevant part of the notes, that is termedas link to Notes.2. Another category includes creating a document of links such as word or excel doc-ument of pointers to videos that is named as Note of Links.Each participant expressed their views of an existing approach versus the proposedapproach. P7 reported that ‘I never thought it would be easier or difficult. If I hadanother choice like another process that makes things easier then definitely I would havefelt that this would be difficult.’ The participant also found the system’s approach to beeasy. ‘I would definitely like it. Instead of searching myself every time I go to YouTubeand saving a link somewhere and forgetting things. Yeah, scanning and it gets to the rightvideo, Yeah. That’s really good’. Thus, the proposed system hints at a great potential insupporting automatic linking to videos over the existing manual inclusion of links.4.3.2 Impact on the Existing Note-taking PracticesTechnologies that create new environments for note-taking have shown to bring change inthe way students take notes. For instance, a study on the annotation of digital documents52Table 4.2: Identified current linking techniquesLink to Notes Note of Links1Title, interval/contentheadings, in-video text,author nameExcel sheet of video to notepointers2 Timestamp, video link ondigital notesWord document of slidescreenshots3Highlight, color code, ‘x’identification marks tonotesVideo Bookmarks[59] demonstrated significant changes in the note content and anchoring style. On thesame lines, students discussed the possible changes/influence on their existing note-takinghabits with the introduction of the proposed linking system. Interview quotes indicatedthe proposed approach might not change students’ note-taking habits; ‘My note takingwouldn’t change but would be far more helpful’ said P13. The possibility of extending theapplication of our system on digital notes also played a role in indicating no influence;‘...because I write electronic notes I can imagine that could also be something that I canuse this snipping tool to take pictures and do that.’ Overall the ability of the systemto retrieve video without adding links or pointers at the time of capturing notes assurespreserving students’ current note-taking style. ‘I don’t think my note taking process wouldchange as much. No, I don’t. I would still write my notes like this. Because it workseven now, if I scan this word’ said participant P12.4.4 SummaryThis chapter conducted an experimental study to draw on the students’ expectationsof mechanizing their existing workarounds in linking notes to videos to make informedand inspired design solutions. The qualitative analysis of interview transcripts bolstered53the evidence demonstrated by the qualitative data from Likert-scale questions. Resultsindicated that students prefer video rendered with a timepoint in the interval level, whilepoint level matches also showed higher satisfaction. Rendering video with a timepointanywhere in the related interval/section provides a considerably large window of timedifference between the expected and the retrieved video timepoint. This is an importantfinding as it aids future video retrieval systems to conceptualize the timepoint differencein determining the right video context.The results also showed that no participants had higher priority for summary thankeywords, although they showed no significant priority for thumbnail. A deeper inves-tigation into the effects of video surrogates on video retrieval systems is necessary forthis work in the future. However, students indicated that the search results could go upto 3 possible matches. Ultimately, the usability evaluation revealed that students wouldfind the proposed linking approach helpful when learning from instructional videos. Thestudents’ positive attitude towards the proposed method is paramount to advance intothe design and development of watched video retrieval systems.Overall, results established the requirements of the video rendering in terms of tempo-ral specificity and the types of surrogates. The next chapter moves discusses the pragmat-ics of linking handwritten notes to corresponding videos with the help of a medium-fidelityprototype design. We can perform an informed flow of recognition analysis with the es-tablished content types used in watched video notes. Also, the preferences for temporallevels can enable the prototype design to find the right video matches. The prototypedevelopment and testing outline the technical requirements in the next chapter, which,with the findings in this chapter, comprehensively answers the second research objectiveof this work.54CHAPTER 5Pragmatics of Linking Notes to VideoFrom the previous chapters, three conclusions are apparent: 1) students make links tovideo content in their notebooks using a variety of content representations (text, formulas,and figures), 2) in all the identified content types of watched video notes, there is someamount of verbatim overlap, and 3) expected timestamp in a video can be scaled tothree temporal levels in a video; timestamp in a point, interval, and whole video. Amedium-fidelity prototype was devised on top of the above hit inferences to evaluate theeffectiveness of exercising watched video notes to link to corresponding videos that:1. Recognizes the three handwritten content types,2. Matches these to a collection of videos and,3. Presents three best matching videos with timepoint in one of the three temporallevels.5.1 Note Recognition AnalysisTo limit the implementation efforts and focus on the implementation gaps associatedwith the proposed approach, we explored readily available off-the-shelf Optical Charac-ter Recognition (OCR) technologies to recognize handwritten images. The process ofselecting the best OCR model directed us to employ different recognition models fordifferent handwritten content types. The goal here was not to compare or evaluate therecognition accuracy or capacity of various models.55OCR Application Programming Interfaces (API) from PixLab1, Google cloud vision2,and Microsoft’s computer vision read handwritten text API3 were used to compare andidentify the optimal recognition API. A random number of text-based samples from thecollected video notes in phase I were sampled and then compared for the word matchingcount. The words matched did not include stop-words such as ‘the’, ‘at’, ‘is’, and soon. With nearly similar results, Microsoft’s API worked slightly better with rotatedcharacters and produced relatively fewer false positives when compared to the otherAPIs. However, it did not reliably recognize the special characters and symbols oftenused in formulas. It also could not interpret lines in figures that did not contain anymeaningful text. Thus, we used Microsoft API to read text-only notes, and exploredother OCR APIs for non-textual components.Our choice to extract special symbols like scientific notations from a given image,without stroke data, was Mathpix OCR API4. The API’s performance was evaluated witha few randomly picked formulas from the pilot study dataset and recorded the numberof matched characters from the original equation. The result object returned with aconfidence field which demonstrated a probability of more than 70% for all the testedcases. Hence, Mathpix was used for reading mathematical content, that is, formulas.A two-step recognition process facilitated reading lines in pictorial representations.First, we used Scale Invariant Feature Transform (SIFT)5 algorithm from Open SourceComputer Vision Library (OpenCV) to extract feature-based keypoints and descriptorsfrom a scaled and slightly rotated image. Second, we studied structural information thatpredicts the perceived quality of an image as Structural Similarity (SSIM) index[89].Next, the Matcher block incorporates finding the corresponding video from the col-1https://pixlab.io/api2https://cloud.google.com/vision/docs/reference/rest/3https://docs.microsoft.com/en-us/azure/cognitive-services/computer-vision/concept-recognizing-text4https://mathpix.com/ocr5https://docs.opencv.org/3.4/da/df5/tutorial py sift intro.html56Figure 5.1: Matching flow for Text and Formula content.lection of videos.5.2 Finding Video MatchesVisual information from slides and the transcribed auditory text was compared withthe recognized handwritten content. Figure 5.1 illustrates the approach used to pick thematching video for the handwritten units’ text and formula. Extracted data is the outputfrom OCRing text and formula handwritten input. Stop-words were filtered out beforefeeding into the matching block.A quick search of data occurrence in the transcripts of videos provided a list ofpossible matches. We carefully examined the video slides of these matches to find thebest matching video timepoint. Finding similar textual components was straightforwardas the entire slide can be OCRed to extract all the text-only data. However, instructionalslides are often content-heavy and include various types of information like equations,figures, and more content types in a slide. The non-textual objects, such as, formulashave to be identified and highlighted before reading them. Region of Interest (ROI)facilitates filtering or operating on unique portions of an image. Using OpenCV, ROIsfor each video slide equipped the highlighting and extracting borders of mathematicalcontent.57Figure 5.2: Matching pictorial content for figuresThe next step directly applied the chosen OCR techniques on the slides for text typeand the extracted ROIs for the formula type. The resulting data from this function wascompared with the input handwritten data to arrive at the best matching ROI, whichgives the best matching video with the timestamp where the data occurs.The process flow for pictorial data is different from the text and formula as there is noquantifiable information that can easily compare in this case. A flowchart can be seen inthe 5.2. Here, again, ROIs were extracted from each video slide. Feature descriptors weredetected using SIFT from ROIs and matched with input image descriptors. For featurematching OpenCV’s Fast Library for Approximate Nearest Neighbors (FLANN)6 algo-rithm was used. Here, to eliminate possible false matches, a ratio test was performed thatselects only the most promising matches for further screening. Following this approachthrough the collection of videos, a list of possible matches was captured. Later, the SSIMof the note image was compared with the list of ROIs from the filtered videos to get thefinal matching video with a timestamp where the pictorial information exists.In each case, the matcher block picked 3 best possible video matches, in line with theinferences drawn from the students in the previous chapter. In the subsequent section, amobile interface application that provides a platform to submit the watched video notesand play the retrieved videos is discussed.6https://docs.opencv.org/3.4/dc/dc3/tutorial py matcher.html585.3 Smartphone as a Pedagogical InterfaceEnabling an interaction platform to feed the note content and consume related videos isan important step in designing the prototype. The goal here is to employ an interface thatexpects minimum effort from a user to provide the desired input and delivers video resultsthat convey the information necessary to choose a match. Mobile devices allocate newpedagogical affordances to students in varied temporal conditions and learning settings[16]. They have proven to possess great potential in engaging students with effortlessaccess to needed content and delivering information ‘just-in-time’ [95], [32], [42], [48].The annual survey by the Educause Center for Applied Research [ECAR] on MobileIT in higher education has tracked mobile technology usage since 2012 [31], [17]. Thereports demonstrate a positive correlation among student use, the importance placed onthese technologies, and students’ academic success. Additionally, considering students’robust use of paper notebooks for their coursework, it is better to design a mechanismin the mobile world that adapts to the attributes of paper. The attempt here is to aidthe ability to capture heterogeneous handwritten information in student notebooks andquickly retrieve required video sources in their own devices.To this end, we leveraged the ability to use a camera to take pictures and videoviewing in smartphones to design a mobile interface in this work. This interface employsa mobile camera to point to video-related notes and re-find corresponding video in amobile-based video platform. Figure 5.3 illustrates the use of the interface in an examplecase, where a student points to a textual description in notes to retrieve the correspondingvideo. First, a student opens the application and is presented with the two options tofeed the formula, take a picture, or upload an image(top). For either option, the usercan crop/adjust the image containing the equation to delimit the note content that canbe linked to the video. The corresponding video, that is, the video watched while notingdown the content, is presented to the user, played back within the application (down).59Figure 5.3: Mobile interface, is presented with the option to take a picture of students’handwritten note content (top) to find the corresponding video with topically similarcontent, read to be played back within the app (bottom)60The mobile application was built using Nativescript plugin API7 for using devicecamera8, background-http plugin9 for enabling http methods and video player plugin10that uses the native video players to play remote. At the back end, the image input isfed to Recognizer and Matcher blocks to find the corresponding video from a collectionof videos.5.4 Accuracy of matching videos to notesWe assessed the ability of the prototype system to retrieve watched videos on the note-book data sample identified in chapter 3. Additionally, instructional videos that studentslearned to make notes were also collected to compare the video result with the groundtruth video marked with the note data. We transcribed the spoken content of the videosto facilitate the matching between the note content and the video. The prototype testeda total of 181 watched video notes. The evaluation metric used was accuracy which is afraction of the number of matched note samples to the total number of note samples:% of Accuracy = # of matched notestotal # of notes ∗ 100 (5.1)Following guidelines were used to choose a video result as matched or not-matched:1. As the search result gives only three different videos relevant to the note content,we treated any video result of the three as a match irrespective of their ranks.2. Timestamp retrieved was compared with the observed timestamp collected as groundtruth. In both cases, the video content was checked and marked if they belonged to7https://www.nsplugins.com/8https://market.nativescript.org/plugins/nativescript-camera/9https://market.nativescript.org/plugins/nativescript-background-http/10https://market.nativescript.org/plugins/nativescript-videoplayer/61the same point, the same interval where the video content marked as ground truthor as the whole video if they were from different intervals.3. If any of the three matches did not match the expected video, it was marked asNo-Match which is 0 relevance.4. If the recognizer block threw an error or returned no information, it was markedas No-Data.After analyzing all the video notes, 38 of the 181 notes were regarded as no-data andexcluded from the set. Of the remaining 143 notes, one was marked as a special case asit did not include ground truth, that is, timestamp and the video. Therefore, the datafrom the Matcher block that was available for comparison was 142. About 77.5% of thetotal notes (n = 142) returned video(s) that matched the expected videos that studentsreported as ground truth. The pie chart in figure 5.4 illustrates the accuracy breakdownacross all the note samples. 29.58% of the notes samples were matched to the exact pointin video as that of the expected. 27.46% of the samples returned match results playingin the same interval or section where the students’ note content is present. 20.42% of thesamples produced the right videos but did not play in the same point or interval as wasexpected.Figure 5.5 demonstrates the accuracy percentage for each of the types breaking downthe accuracy of finding the video matches across the three different content types, thatis, text, formula, and figure. The system matched textual content with an accuracy of97.5%, formulas with 56.8%, and drawings with 38.9%. Table 5.1 shows the exact numberof matches and accuracy predicted for the three content types on the temporal scale ofpoint, interval, and video.62Figure 5.4: The pie charts showing the total accuracy of finding the matched andthe breakdown across point, interval and whole video temporal levelsFigure 5.5: The bar chart showing the total number of matched note samples of the3 content types; Text, Formula, and figure63Table 5.1: The number of count (n) of note-video matches, and accuracy of retrievingmatching video is reported for each content type (i.e., text, formula, and drawing)overall and at the three levels of temporal specificity (i.e., point, interval, and wholevideo).Temporal LevelNote type Point Interval Whole video OverallCount (n) Count (n) Count (n) Count (n) Notes (N) Accuracy(n/N%)Text 36 27 15 78 80 97.5Formula 5 10 10 25 44 56.8Drawing 1 2 4 7 18 38.9Total 42 39 29 110 142 77.55.5 Content OverlapWe computed the percentage of content overlap between the note content and the videoslide deduced from the results of our system. Doing this will give us more insights intohow the system performs concerning student’s manual process of linking their previouslywritten notes to corresponding video context. The proposed approach matched most ofthe textual components in student notes related to watched videos with a 97.5% accuracy.Thus, we first considered the general characteristics of the overlap results of the matchedvideo for the text-only content and made the following observations from the boxplotshown in figure 5.6.1. Atleast 50% of the video results that played from the exact video slide/pointmatched the note content word to word with more than 90% overlap. Minimumoverlap is at 25%, and the IQR spans between 65-100%, indicating that video re-sults playing from the same point as expected had an overlap greater than 65% formost of the note samples.2. IQR of the matched content in the case of interval condition also manifests a con-siderable amount of overlap, that is 55-80%, with at least 75% of the note samples64Figure 5.6: Results showing overlap between video results and text-only content.matched in this condition showing an overlap greater than 50%. This hints thatthe note content in the case of textual components summarize an interval of a videoand can point to multiple frames in a video interval3. In the case of the whole video condition, the matched results still show some overlap.However, it offers a decreased percentage of overlap. An interesting factor to noteis, more than 75% of the notes showed overlap between 20-40% indicating that thenote content might re-appear in various parts of the video.Similarly, figure 5.7 also demonstrates the percentage of overlap for the formula-basedcontent.65Figure 5.7: Boxplot results showing overlap between video results and formula-onlycontent.1. Similar to the case of textual note samples, the point condition demonstrated anoverlap greater than 65% for at least 75% of the note samples.2. Interval condition confirmed a wide range of overlap in the IQR region. However,it also had matches with 0% overlap.3. In the whole video condition, most cases showed an overlap of less than 30%, hintingthat the video results started playing randomly.In the case of figure-type notes, the accuracy was predominantly low and recordedonly seven matches. Hence, we did not include the box plot representation of this type.665.6 DiscussionWe showcased the accuracy of retrieving watched videos on both temporal levels andcontent types. The system matched approximately 78% of the note data to expectedvideos on one of the temporal levels. More than half of the returned videos exhibited atimepoint in point level or interval temporal level as preferred among the participants.The system did exceptionally well in matching textual note content to videos as finerlegibility in textual content and matured attempts in word-spotting exists [88]. Analysisof content overlap between notes and the video results further highlighted interestingfindings respective to retrieval in the three temporal levels. Close to half of the text-onlysamples were matched with timepoint in the point level and more importantly returneda relevant context with a good amount of overlap for timepoint in the interval level.Although a video point that is semantically closer to the note content exists, students’choice of topical relevance was at another timepoint in the same section or interval wherethe note content is present. The presented work has indeed conceptualized the differenceof timestamp between the result to that of the expected video. This is an importantfinding in aiding the future video retrieval systems to allow a wider window of differencein the retrieved video timepoint.However, in terms of formulas and drawings, the system matched less than half ofthe total links. The prototype system returned a few videos with point level matchesthat had a content overlap of 100%. Additionally, more than half of the equations andgraphs that yielded no matching videos had an overlap of 100% in the actual observations.One of the reasons for low accuracy in the two non-textual content types could be thegap in generating non-textual Region of Interest (ROIs) in the Matcher block. Thestart and end of outlines for ROI bounding boxes are not obvious in equations andfigures, which influences the matching effectiveness. ROI segmentation must be improvedwith training models prior to matching the note image to address this gap. Moreover,67Figure 5.8: Link samples with low legibilityrecognition of handwritten equations does not always yield success. Recent reports fromCompetitions on Recognition of Online Handwritten Mathematical Expressions (ICDAR2019 CROHME) [55] have shown an accuracy of 77% for image-based recognition ofequations. Similarly, advancements in feature matching approaches must find soundcorrespondences, especially with a higher proportion of false-positive matches in the caseof free-form drawings.We shouldnote that 38 video notes were excluded, marked as No-data, from the ac-curacy calculation. This was because the recognizer failed to detect the content beforethe note was passed to the matcher block due to legibility concerns, such as bad imagequality or the information in the link might have been too less to detect content mean-ingfully. It can be evident in the sample data shown in Figure 5.8 where the legibility iscompromised or the data to be recognized is too narrow for meaningful interpretation.Additionally, in the proposed approach, the content types are physically categorized innotebooks for the hybrid type before feeding into the recognition module. The distinctionbetween textual and non-textual content types is not always obvious. Of the 39 hybridnotes assigned to one of the three content types, one was dropped as a special case withno ground truth. Thus, 30 out of 38 hybrid notes were matched to the right videos,leading to an accuracy of 78.94%. This corroborates the right choice of content typeassignment for improved accuracy in retrieving the video match.685.7 Implementation requirementsBy designing the medium-fidelity prototype, a number of technology impediments wereidentified that hinders the ability to recognize and match notes to videos effectively. Thefollowing sections outline a few essential requirements.5.7.1 Automated Content Type SegmentationThe primary step of the proposed system is to interpret the text and non-text elementsseparately to carry out the subsequent recognition and matching processes. We must as-sign the mobile-scanned note content to the appropriate content labels with the identifiedtypes: text, formula, and figure before the corresponding recognition module assignment.Several researchers have proposed heuristic methods to separate text/non-text segmentsin online handwritten documents that utilize a combination of spatial and temporal in-formation available from stroke level data [19], [20].In classroom notebooks, however, this process is an arduous task without stroke-levelinformation. Furthermore, the content of paper-based handwritten notes varies in size,shape, and orientation with different writing styles [76]. Most often, all the content typesare closely connected and merge haphazardly, which is a significant issue in distinguishingthem. Additional challenges include arbitrary layout, inconsistency in the note qualityas they can be written on a variety of paper and can be old.Previous work has introduced techniques that allow content segmentation, and layoutanalysis of a page in three main methods: top-down, bottom-up, and hybrid techniques[66]. The top-down techniques first detect the highest level of structures and then pro-ceed to the bottom layer by training a classifier model that successively splits text, andnon-text components [7], [38]. The top-down approach has shown a good (about 95%)prediction rate for identifying textual information, but not when non-textual elements69like formulas and graphs are present. On the contrary, bottom-up methods begin withprimary components and merge them into a segment [19]. These methods mostly focus ononline handwritten documents and do not fit our case as the knowledge of digital strokesor individual pixel data are not available in paper-based documents. Hybrid methodscombine the two approaches [78]. Florent et al. [61] proposed a hierarchical combinationof Conditional Random Fields(CRFs) to extract the document layout of handwrittenletters. Sarkar et al. [76] presented a typical connected components(CCs) classification-based method to separate text from non-text components. A similar method is proposedin [3] where the LBP operator is used to classify the CCs. Kundu et al. [49] recentlyproposed a method that applies Generative Adversarial Networks (GANs) where theyconsidered ‘text line extraction’ in handwritten documents as an image-to-image trans-lation task. Many more methods have considered the complex type of documents buthave not addressed the most common issue of handwritten documents, i.e., overlappingcomponents.Recent advancement has introduced a few commercial/non-commercial modules thatsupport text/non-text segmentation through pre-trained classifiers. Azure Custom vi-sion11 is one such service that facilitates the use of a machine-learning algorithm to trainand classify the images. During the process of prototype implementation, we built atraining model with hundreds of handwritten note samples by tagging labels pointingto the three content types. However, the test run with the data collected yielded us anutmost accuracy of 60%. The performance of the classifier significantly dropped for theimages that had overlapping components of different types, that is, the hybrid type.In the case of Hybrid images that consist of closely connected text and non-textcomponents, assigning a label involves more work than classifying an image into anappropriate content type. Consider the example shown in Figure 3.7. Here, although the11https://docs.microsoft.com/en-us/azure/cognitive-services/Custom-Vision-Service/overview70text and non-textual data elements are arbitrarily constructed, the textual informationconveys semantically rich interpretations that can aid the accurate topical search of therelated video. The pictorial data, in this case, might not be as helpful. Therefore, theclassifier must point the following case to a text label instead of pointing to 3 independenttypes, which otherwise becomes taxing to process both in time and space. This poses animperial need to possess prior knowledge of the content in the classifier.Along with the content segmentation at the scanning end of the mobile interface, videoslides also contain both text and non-text components. As seen in the implementationsection, the Region of Interests (ROIs) extracted per image must be accurately mappedto non-textual components for efficient matching. Thus, there is an imperative needto develop an advanced content classification system that efficiently identifies differentcontent types in both handwritten paper notebooks and the video slides as a mandatorypre-processing step.5.7.2 Recognition AccuracyGenerally factors which affect accuracy of OCR can be disparate language classes [103],character merging and fragmentation [11], resolution (e.g. [84], [64]), illumination, skewand noise [46]. Handling each of these issues is significant in improving the overallrecognition accuracy. As we collected the data was collected under controlled conditions,skew, blurriness and illumination are less likely to contribute towards OCR inaccuracy.Character segmentation for multi-lingual handwritten recognition is now a significantarea of research. The proposed system has been developed with recognizers to accom-modate English, mathematical characters, and pictorial representations. However, therecognizer should be able to scale to notes of multiple languages. For instance, for noteswith a mix of English and Korean/Chinese language, the OCR accuracy might signifi-cantly drop. Tesseract has proven to show very high accuracy for Latin characters. Park71et al. [67] proposed a framework to tackle multi-lingual scripts. The words are split intomultiple character images fed to a set of pre-trained classifier models. Unlike English,languages like Korean, Chinese have thousands of character classes. Thus, independentrecognizers with explicit character segmentation and individual character recognition areimportant to map videos of interest.Poor resolution of the notebook content scanned can greatly influence the OCR accu-racy. Notes can be old or ill-maintained, which degrades the quality of content present inthe notebook. Image interpolation as a pre-processing step in the Recognition pipelinecan lessen the overall accuracy. Several researchers have proposed to tackle resolutiondegradation in printed document images [39], [15] [33]. Ankit et al. [51] showed anaccuracy improvement of up to 21% in OCRing document images by employing a super-resolution-based pre-processing step.5.7.3 Computing TimeResearch suggests that even slightly higher retrieval latency by search engines can sig-nificantly decline in users’ perceptions of result quality and engagement with the searchresults. The time between query submission to search response should be reduced asmuch as possible. In the case of the proposed system for video search, the processingtime is dependent on the number of videos to be looked up.In textual links, the processing time is directly proportional to the number of videos asthe content is searched in the video transcript directly. However, the time taken dependson the number of ROIs extracted per video slide for formulas and figures. As the ROIextraction becomes more precise, the time factor increases with a potential improvementin the matching accuracy. A more grouped ROI can produce good speed, which mightaffect the accuracy of finding the right video match. Hence deeper work to address thistrade-off is necessary to aid the system’s ability to retrieve videos of interest efficiently.72CHAPTER 6ConclusionsThe work in this thesis investigated the use of engineering students’ handwritten notecontent, captured when learning from educational videos, as a linking artifact to connectback to originally watched video context when reviewing. We observed distinct charac-teristics of notes, references to related videos, and verbatim overlap, a pivotal elementto find relevant video content, in 10 engineering students’ video notes. Results indi-cated overlap of some amount in at least 75% of the notes in all the identified types ofnote content. The retrieved video timepoint was conceptualized on the temporal scale,a timepoint in the same point, interval, or the whole video as that of the expected. Webelieve this approach accounts for the difference between the retrieved to expected videotimestamp.The findings from the 12-student lab study showed an overall preference for videotimepoint retrieval in the ‘interval’ level. Also, a medium-fidelity prototype informedinsights regarding the existing technology in designing a fully working application. Eval-uating the prototype system against a set of 181 identified note samples, 77.5% of thesamples were matched with the right videos as expected, with the video result playing inone of the temporal levels. Also, 97.5% of the total textual notes were matched to thevideos observed in the ground truth. This is especially interesting given that more thanhalf of the links identified were textual content indicating its broader use over other con-tent types. The thematic analysis of interview transcripts in the study revealed that theproposed system of linking notebook content with corresponding videos was perceived73as usable, effective, and easy to use. and can improve both the learning experience andlearning outcomes. The implementation gaps given the existing technology such as au-tomated content separation, recognition inaccuracy and computation time were derivedin prototype development.Along with the established results on the temporal specificity and video surrogatesassociated with matching video rendition, it was interesting to note how it affects users’linking behaviors. Students’ response to the system’s usability showcased its potentialuse as a supplement to bring context back to their notes rather than modeling notebookas a platform to include video-related annotations. Also, the existing linking practicesas seen in the study transcripts in Table 4.2 are mostly manual approaches that confirmthe proposed approach’s novelty.6.1 Limitations and Future WorkAlthough the results of our initial evaluation are positive, we recognized limitations withinthe study. With regard to the notes collection elaborated in chapter 3, the focus was onlyon technical notes to generate varied types of content types. The observations were basedon a small corpus of class notes gathered from Engineering students at the Universityof British Columbia. The intention was to consider diverse content types accessible inengineering notes as confirmed during the pilot study. It is important to note that thereis a great deal of individual variation in note-taking practices. So the characteristics ofvideo-based notes described in this work are applicable only generally, not universally.Additionally, the participants in the latter study, that reported on the systems’ us-ability and requirements were recruited based on convenience sampling. All the studentswere part of varied degree levels pursuing different majors. Also, they were in no wayconnected and randomly picked. While an ample amount of qualitative data was col-lected that led to interesting inferences, the study might be vulnerable to selection bias74and other influences that go beyond the control of the study.Next, the collected notes benefited us with distinctive content types, data from otherdomains fall out of scope in this work, music-based courses or notes from a chemistrycourse, for example. The extension of this work in the future must address notes of othergenre containing unique characters and different languages.Along with the diverse genres, lecture characteristics such as the modality and thelecture structure is another critical factor that needs deeper investigation concerningthe note-taking styles. Lectures can be seen to be presented with a classroom stylewritten content or spoken content with talking heads. Previous work, however, shows noindications of lecture modality influencing the amount of lecture material students recordor remember. Furthermore, the structure of lectures in a specific modality can haveclosely connected or disconnected/discontinuous segments stitched together dependingon the complexity of the lecture topic. The temporal specificity of such lecture materialsdramatically varies and is an important factor to consider in rendering the matchingvideos.While the current work has confirmed students’ positive attitude towards the in-strumentation of notebook content-based video retrieval, the current work falls short ofevidence in various learning settings, as shown in the examples below:1. First, how a linking interface functions when working on assessments and preparingfor exams. A detailed review of the expectations of a linking system’s use whenapproaching class tests or an examination can contradict some of the observationshere, chiefly pertaining to temporal specificity and surrogates of the retrieved video.2. Second, how does the system perform on old notes, say older than a year or more.An interesting factor would be evaluating the validity of the matched video results.Although the prototype designed for this work was of sufficient fidelity to draw in-75ferences on the requirements, incorporating additional image correction methods cansignificantly improve performance. For instance, 38 records were dropped as the recog-nizer failed to translate the note content. Since the intention was to employ off-the-shelfrecognition algorithms to implement a prototype of sufficient fidelity, a pre-processingstep to improve the resolution was not a part of this work. Particularly for handwrit-ten drawings, image warping as a mandatory pre-processing step can significantly aidthe matching accuracy. Future work needs to address the optimization of matchingnon-textual data and more comparative experimental designs with more extensive datacollections.Lastly, the focus of this work emphasizes the interoperable nature between notebooksand video materials. However, nowadays, most paper documents have an electroniccounterpart in a variety of media that is accessible on the World Wide Web or from someother online database or document corpus. While the video plays a central role in flippedclassrooms and other video-based pedagogies, video is not the only media type studentsengage with when learning with video. Students make use of a diverse collection ofinformation objects from within their information ecology. A typical engineering student,for example, will use her notebook during class, in labs, when doing homework, andtutorials, to name a few everyday academic tasks. Thus, as part of future work, the aimis to evaluate how notebooks can become more interoperable with other media types.6.2 Extension of Application InteractionsWe have demonstrated a set of primary user interactions with the presented prototypeto associate a piece of watched-video note in a paper document with a particular video.The future can build on numerous extensions on that association, whose illustrations webriefly outline in this section.At the most superficial level, when a user scans a part of notes, the mobile app could76provide more options over the regular edit/crop features. Suppose the user feels that theinformation in the scanned part of notes could be insufficient in fetching the videos ofinterest. In that case, an option to add more information by typing/drawing free-handcan strengthen the context. This way, the actual notes can remain unchanged. Advancedcollaborative actions such as inviting friends, colleagues, or peers to add on the scannednote can help locate the video counterpart better. Audio-augmented action items mayalso be designed on the edit page of scanned notes to provide bonus information.The strength of the scanned query in fetching suitable videos may be improved, takinginto account the types of errors likely to occur in the particular captured note data. Oneexample of this is an indication of suspected errors in recognizing specific characters suchas a mathematical symbol or highlighting a word as important; in this instance, a searchengine may assign them a lower priority or treat highlighted characters as wildcards.Additionally, the same video(s) can often be searched using multiple scans of notes,pages of notes that entail a mathematical derivation, for example. The mobile appcan implement these and many other extensions of “paper/digital integration” withoutrequiring changes to the current writing processes, editing and sharing documents, givingsuch conventional paper documents a whole new layer of digital functionality.The ability and time to recognize the scanned notes can also be improved by keepingfrequently occurring or important records of capturing-retrieving video results. One cankeep track of a map of a key portion of notes such as words, characters, or descriptorsto the corresponding videos so that the occurrence of the same corpus can be used toenhance the recognition process, eventually increasing the likelihood of bringing up thesame video results. To aid indexing of capture-retrieve process maps, students may choosemarkers such as emoticons, stars, or the capture time. Doing this, the future capture-retrieve actions on the same set of notes can be faster and may not require additionalcueing to fetch the video match.77User feedback can be trivial in identifying a set of one or more candidate matchingvideos. Candidate videos can be weighted according to their probable relevance (forexample, based on the number of other users who have scanned to fetch these videos ortheir popularity on the Internet). These weights can be applied in the iterative matchingprocess. If there are likely to be delays or cost associated with processing a note query orreceiving the results, this data can improve the performance of the local device, reduceprocessing costs, and provide helpful and timely user feedback.Adding overlays or markups on the retrieved videos can also supplement the feedbackprocess and, in turn, future lookup of the videos. For example, a student can attachtext or an audio recording of his/her thoughts about a particular video segment for laterretrieval as annotations. As another example, a user may also attach a picture(s) ofnotes used to retrieve the video that might assist other users when shared with the classgroup. Another possible functionality that may benefit users is to enable the lookup andsubsequent inclusion of other related electronic counterparts of a specific video section.Say, a text document that details the functioning of a model in a PDF document ora podcast introduced in the video section. Thus, we believe that the retrieved videoand its digital counterparts as markups can bolster the prospect of added note-relatedinformation, eventually creating a rich, interactive platform to connect to a wide rangeof information sources in a single scan.Lastly, user-specific actions and history may also enhance many aspects of the systemoperation. Say, if the previous capture was within the last few minutes, it is very likelyto be from the same video or area of study. Similarly, it is more likely that a note isbeing recorded in start-to-finish order. Or a student might frequently capture text-onlyor mathematics notes, as he/she finds it important for future review. Such user-specificfactors can help the system establish the location of relevant videos in cases of ambiguityand also reduce the effort in looking up through a large video archive.78In summary, the proposed approach has established a new baseline for linking paper-based notes to instructional videos with room for future improvement. The design,implementation, and evaluation of the system have led to several interesting implicationsand extensions, including note-type scalability, recognition optimization, and improvinginteraction with the system. In turn, they are paving the way for future trials to inves-tigate similar approaches in augmenting interoperability between notebook content andcorresponding reference media.79References[1] Awad George, Cees GM Snoek, Alan F. Smeaton and G. Que´not. Trecvid semanticindexing of video: A 6-year retrospective. ITE Transactions on Media Technology andApplications, pages 187–208, 2016.[2] D. Bargeron, A. Gupta, J. Grudin, and E. Sanocki. Annotations for streaming video onthe web: system design and usage studies. Computer Networks, 31(11-16):1139–1153,1999.[3] S. Bhowmik, R. Sarkar, and M. Nasipuri. Text and non-text separation in handwrittendocument images using local binary pattern operator. In Proceedings of the FirstInternational Conference on Intelligent Computing and Communication, pages 507–515.Springer, 2017.[4] R. Boardman and M. A. Sasse. ” stuff goes into the computer and doesn’t come out” across-tool study of personal information management. In Proceedings of the SIGCHIconference on Human factors in computing systems, pages 583–590, 2004.[5] R. Boardman, R. Spence, and M. A. Sasse. Too many hierarchies? the daily struggle forcontrol of the workspace. In Proc. HCI International 2003, volume 1, pages 616–620.CRC Press: Taylor & Francis Group, 2003.[6] P. Borlund. The IIR evaluation model: A framework for evaluation of interactiveinformation retrieval systems. Information Research, 8(3), 2003.[7] T. M. Breuel. Two geometric algorithms for layout analysis. In International workshopon document analysis systems, pages 188–199. Springer, 2002.[8] D. C. Bui, J. Myerson, and S. Hale. Note-taking with computers: Exploring alternativestrategies for improved recall. Journal of Educational Psychology, 105(2):299, 2013.[9] D. Byrne, P. Wilkins, G. J. F. Jones, A. F. Smeaton, and N. E. O’Connor. Measuringthe Impact of Temporal Context on Video Retrieval. In Proceedings of the 2008International Conference on Content-Based Image and Video Retrieval, pages 299–308,New York, {NY}, 2008. ACM.[10] C. Carpineto, S. Mizzaro, G. Romano, and M. Snidero. Mobile Information RetrievalWith Search Results Clustering: Prototypes and Evaluations. Journal of the AmericanSociety for Information Science and Technology, 60(5):877–895, 2009.[11] R. G. Casey and E. Lecolinet. A survey of methods and strategies in charactersegmentation. IEEE transactions on pattern analysis and machine intelligence, 18(7):690–706, 1996.[12] D. Castro, B. L. D. Bezerra, and M. Valenc¸a. Boosting the Deep Multidimensional80Long-Short-Term Memory Network for Handwritten Recognition Systems. In 2018 16thInternational Conference on Frontiers in Handwriting Recognition (ICFHR), pages127–132, Piscataway, {NJ}, 2018. IEEE.[13] M. T. Chi. Active-constructive-interactive: A conceptual framework for differentiatinglearning activities. Topics in cognitive science, 1(1):73–105, 2009.[14] M. G. Christel. Evaluation and User Studies With Respect to Video Summarization andBrowsing. In Multimedia Content Analysis, Management, and Retrieval 2006, page60730M, Bellingham, {WA}, 2006. International Society for Optics and Photonics,SPIE.[15] F. Cloppet, N. Vincent, and Others. Adaptive fuzzy model for blur estimation ondocument images. Pattern Recognition Letters, 86:42–48, 2017.[16] H. Crompton, D. Burke, and K. H. Gregory. The use of mobile learning in PK-12education: A systematic review. Computers & Education, 110:51–63, 2017.[17] E. Dahlstrom, J. D. Walker, and C. Dziuban. ECAR study of undergraduate studentsand information technology. Technical report, 2013, 2013.[18] S. Deb and P. Bhattacharya. Enhancing personalized learning with interactive notetaking on video lectures–an analysis of effective hci design on learning outcome. InInternational Conference on Next Generation Computing Technologies, pages 558–567.Springer, 2017.[19] A. Delaye and C.-L. Liu. Text/non-Text Classification in Online HandwrittenDocuments With Conditional Random Fields. In Chinese Conference on PatternRecognition, pages 514–521, Berlin, Germany, 2012. Springer.[20] A. Delaye and C.-L. Liu. Contextual Text/non-Text Stroke Classification in OnlineHandwritten Notes With Conditional Random Fields. Pattern Recognition, 47(3):959–968, 2014.[21] S. J. DeLozier and M. G. Rhodes. Flipped classrooms: a review of key ideas andrecommendations for practice. Educational psychology review, 29(1):141–151, 2017.[22] S. Dodson, I. Roll, M. Fong, D. Yoon, N. M. Harandi, and S. Fels. An active viewingframework for video-based learning. In Proceedings of the Fifth Annual ACMConference on Learning at Scale, pages 1–4, 2018.[23] S. Dodson, I. Roll, N. M. Harandi, S. Fels, and D. Yoon. Weaving Together Media,Technologies, and People: Students’ Information Practices in Flipped Classrooms.Information and Learning Sciences, 120(7/8):519–540, 2019.[24] B. Dorn, L. B. Schroeder, and A. Stankiewicz. Piloting {TrACE}: ExploringSpatiotemporal Anchored Collaboration in Asynchronous Learning. In Proceedings ofthe 18th {ACM} Conference on Computer Supported Cooperative Work & Social81Computing, pages 393–403, New York, {NY}, 2015. ACM.[25] J. C. Dunn. Remember-Know: A Matter of Confidence. Psychological Review, 111(2):524–542, 2004.[26] F. Faul, E. Erdfelder, A.-G. Lang, and A. Buchner. G* power 3: A flexible statisticalpower analysis program for the social, behavioral, and biomedical sciences. Behaviorresearch methods, 39(2):175–191, 2007.[27] M. Fong, S. Dodson, X. Zhang, I. Roll, and S. Fels. {ViDeX}: A Platform forPersonalizing Educational Videos. In Proceedings of the 18th {ACM/IEEE} JointConference on Digital Libraries, pages 331–332, New York, {NY}, 2018. ACM.[28] A. Fouse, N. Weibel, E. Hutchins, and J. D. Hollan. {ChronoViz}: A System forSupporting Navigation of Time-Coded Data. In Extended Abstracts on Human Factorsin Computing Systems, pages 299–304. ACM, New York, {NY}, 2011.[29] K. Franke and M. Ko¨ppen. Towards an universal approach to background removal inimages of bankchecks. In in Proceedings 6th International Workshop on Frontiers inHandwriting Recognition (IWFHR), Tajon, Korea. Citeseer, 1998.[30] M. Furini, S. Mirri, and M. Montangero. {TagLecture}: The Gamification of VideoLecture Indexing Through Quality-Based Tags. In 2017 {IEEE} Symposium onComputers and Communications ({ISCC}), pages 122–127, Piscataway, {NJ}, 2017.IEEE.[31] J. Galanek, D. Gierdowski, and D. Christopher Brooks. ECAR study of undergraduatestudents and information technology. Technical report, 2018, 2018.[32] R. Gangaiamaran and M. Pasupathi. Review on Use of Mobile Apps for LanguageLearning. International Journal of Applied Engineering Research, 12(21):11242–11251,2017.[33] U. Garain, A. Jain, A. Maity, and B. Chanda. Machine reading of camera-held lowquality text images: an ICA-based image enhancement approach for improving OCRaccuracy. In 2008 19th International Conference on Pattern Recognition, pages 1–4.IEEE, 2008.[34] F. Guimbretie`re. Paper Augmented Digital Documents. In Proceedings of the 16thAnnual {ACM} Symposium on User Interface Software and Technology, pages 51–60,New York, {NY}, 2003. ACM.[35] K. Gynther. Design Framework for an Adaptive MOOC Enhanced by BlendedLearning: Supplementary Training and Personalized Learning for Teacher ProfessionalDevelopment. Electronic Journal of e-Learning, 14(1):15–30, 2016.[36] N. M. Harandi, F. Agharebparast, L. Linares, S. Dodson, I. Roll, M. Fong, D. Yoon,and S. Fels. Student Video-Usage in Introductory Engineering Courses. Proceedings of82the Canadian Engineering Education Association (CEEA), pages 1–8, 2018.[37] J.-H. Hsiao and Y.-Z. Wang. Vcenter: A digital video management system with mobilesearch service. In Proceedings of the 7th ACM/IEEE-CS Joint Conference on DigitalLibraries, page 508, New York, NY, 2007. Association for Computing Machinery.[38] E. Indermu¨hle, H. Bunke, F. Shafait, and T. Breuel. Text Versus Non-Text Distinctionin Online Handwritten Documents. In Proceedings of the 2010 {ACM} Symposium onApplied Computing, pages 3–7, New York, {NY}, 2010. ACM.[39] M. Irani and S. Peleg. Improving resolution by image registration. CVGIP: Graphicalmodels and image processing, 53(3):231–239, 1991.[40] R. S. Jansen, D. Lakens, and W. A. IJsselsteijn. An integrative review of the cognitivecosts and benefits of note-taking. Educational Research Review, 22:223–233, 2017.[41] R. S. Jansen, D. Lakens, and W. A. IJsselsteijn. An integrative review of the cognitivecosts and benefits of note-taking. Educational Research Review, 22:223–233, 2017.[42] S. Ja¨rvela¨, P. Na¨ykki, J. Laru, and T. Luokkanen. Structuring and RegulatingCollaborative Learning in Higher Education With Wireless Networks and Mobile Tools.Journal of Educational Technology & Society, 10(4):71–79, 2007.[43] W. Jones. Personal Information Management. Annual Review of Information Scienceand Technology, 41(1):453–504, 2007.[44] M. Kam, J. Wang, A. Iles, E. Tse, J. Chiu, D. Glaser, O. Tarshish, and J. Canny.Livenotes: A System for Cooperative and Augmented Note-Taking in Lectures. InProceedings of the {SIGCHI} Conference on Human Factors in Computing Systems,pages 531–540, New York, {NY}, 2005. ACM.[45] L. S. Kate, M. M. Waghmare, and A. Priyadarshi. An approach for automated videoindexing and video search in large lecture video archives. In 2015 internationalconference on pervasive computing (ICPC), pages 1–5. IEEE, 2015.[46] A. Khotanzad and Y. H. Hong. Invariant image recognition by Zernike moments. IEEETransactions on pattern analysis and machine intelligence, 12(5):489–497, 1990.[47] K. Kobayashi. Combined effects of note-taking/-reviewing on learning and theenhancement through interventions: A meta-analytic review. Educational Psychology,26(3):459–477, 2006.[48] A. Kukulska-Hulme and L. Shield. An overview of mobile assisted language learning:From content delivery to supported collaboration and interaction. ReCALL, 20(3):271–289, 2008.[49] S. Kundu, S. Paul, S. K. Bera, A. Abraham, and R. Sarkar. Text-line extraction fromhandwritten document images using GAN. Expert Systems with Applications, 140:112916, 2020.83[50] G. Landow. Hypertext 3.0: Critical Theory and New Media in an Era of Globalization.Johns Hopkins University Press, Baltimore, {MD}, 2006.[51] A. Lat and C. V. Jawahar. Enhancing OCR Accuracy with Super Resolution. In 201824th International Conference on Pattern Recognition (ICPR), pages 3162–3167, 2018.[52] C. Liu, C.-L. Yang, J. J. Williams, and H.-C. Wang. Notestruct: Scaffolding note-takingwhile learning from online videos. In Extended Abstracts of the 2019 CHI Conference onHuman Factors in Computing Systems, pages 1–6, 2019.[53] M. R. Lyu, E. Yau, and S. Sze. A multilingual, multimodal digital video library system.In Proceedings of the 2nd ACM/IEEE-CS Joint Conference on Digital Libraries, pages145–153, New York, NY, 2002. Association for Computing Machinery.[54] W. E. Mackay, G. Pothier, C. Letondal, K. Bøegh, and H. E. Sørensen. The missinglink: augmenting biology laboratory notebooks. In Proceedings of the 15th annual ACMsymposium on User interface software and technology, pages 41–50, 2002.[55] M. Mahdavi, R. Zanibbi, H. Mouche`re, C. Viard-Gaudin, and U. Garain. {ICDAR}2019 {CROHME}+ {TFD}: Competition on Recognition of Handwritten MathematicalExpressions and Typeset Formula Detection. In 15th {IAPR} International Conferenceon Document Analysis and Recognition, pages 1533–1538, Piscataway, {NJ}, 2019.IEEE.[56] A. Mangen. Hypertext Fiction Reading: Haptics and Immersion. Journal of Research inReading, 31(4):404–419, 2008.[57] G. Marchionini, Y. Song, and R. Farrell. Multimedia Surrogates for Video Gisting:Toward Combining Spoken Words and Imagery. Information Processing &Management, 45(6):615–630, 2009.[58] C. Marshall. Toward an Ecology of Hypertext Annotation. In Proceedings of the Ninth{ACM} Conference on Hypertext and Hypermedia, pages 40–49, New York, {NY}, 1998.ACM.[59] C. Marshall and A. J. B. Brush. Exploring the Relationship Between Personal andPublic Annotations. In Proceedings of the 4th {ACM}/{IEEE-CS} Joint Conference onDigital Libraries, pages 349–357, New York, {NY}, 2004. ACM.[60] A. Mitrovic, V. Dimitrova, L. Lau, A. Weerasinghe, and M. Mathews. Supportingconstructive video-based learning: requirements elicitation from exploratory studies. InInternational Conference on Artificial Intelligence in Education, pages 224–237.Springer, 2017.[61] F. Montreuil, S. Nicolas, E. Grosicki, and L. Heutte. A new hierarchical handwrittendocument layout extraction based on conditional random field modeling. In 2010 12thInternational Conference on Frontiers in Handwriting Recognition, pages 31–36. IEEE,2010.84[62] X. Mu. Towards Effective Video Annotation: An Approach to Automatically LinkNotes With Video Content. Computers & Education, 55(4):1752–1763, 2010.[63] P. A. Mueller and D. M. Oppenheimer. The Pen Is Mightier Than the Keyboard:Advantages of Longhand Over Laptop Note Taking. Psychological Science, 25(6):1159–1168, 2014.[64] A. Namane and M. A. Sid-Ahmed. Character scaling by contour method. IEEETransactions on Pattern Analysis and Machine Intelligence, 12(6):600–606, 1990.[65] A. J. Newell and L. D. Griffin. Writer identification using oriented basic image featuresand the delta encoding. Pattern Recognition, 47(6):2255–2265, 2014.[66] O. Okun, D. Dœrmann, and M. Pietikainen. Page segmentation and zone classification:the state of the art. Technical report, OULU UNIV (FINLAND) DEPT OFELECTRICAL ENGINEERING, 1999.[67] J. Park, E. Lee, Y. Kim, I. Kang, H. I. Koo, and N. I. Cho. Multi-Lingual OpticalCharacter Recognition System Using the Reinforcement Learning of CharacterSegmenter. IEEE Access, 8:174437–174448, 2020.[68] J. Pearson, G. Buchanan, and H. Thimbleby. The Reading Desk: Applying PhysicalInteractions to Digital Documents. In Proceedings of the {SIGCHI} Conference onHuman Factors in Computing Systems, pages 3199–3202, New York, {NY}, 2011. ACM.[69] J. Pearson, S. Robinson, and M. Jones. Paperchains: Dynamic sketch+ voiceannotations. In Proceedings of the 18th ACM Conference on Computer SupportedCooperative Work & Social Computing, pages 383–392, 2015.[70] S. T. Peverly, P. C. Vekaria, L. A. Reddington, J. F. Sumowski, K. R. Johnson, andC. M. Ramsay. The relationship of handwriting speed, working memory, languagecomprehension and outlines to lecture note-taking and test-taking among collegestudents. Applied Cognitive Psychology, 27(1):115–126, 2013.[71] V. Pham, T. Bluche, C. Kermorvant, and J. Louradour. Dropout Improves RecurrentNeural Networks for Handwriting Recognition. In 2014 14th International Conferenceon Frontiers in Handwriting Recognition, pages 285–290, Piscataway, {NJ}, 2014. IEEE.[72] A. Piolat, T. Olive, and R. T. Kellogg. Cognitive effort during note taking. Appliedcognitive psychology, 19(3):291–312, 2005.[73] R. Plamondon and S. N. Srihari. Online and Off-Line Handwriting Recognition: AComprehensive Survey. {IEEE} Transactions on Pattern Analysis and MacHineIntelligence, 22(1):63–84, 2000.[74] J. Puigcerver. Are multidimensional recurrent layers really necessary for handwrittentext recognition? In 2017 14th IAPR International Conference on Document Analysisand Recognition (ICDAR), volume 1, pages 67–72. IEEE, 2017.85[75] S. Redding. Through the Student’s Eyes: A Perspective on Personalized Learning andPractice Guide for Teachers. Center on Innovations in Learning, Temple University,2013.[76] R. Sarkar, S. Moulik, N. Das, S. Basu, M. Nasipuri, and M. Kundu. Suppression ofnon-text components in handwritten document images. In 2011 InternationalConference on Image Information Processing, pages 1–7. IEEE, 2011.[77] A. Sellen and R. H. R. Harper. The Myth of the Paperless Office. {MIT} Press,Cambridge, {MA}, 2001.[78] R. W. Smith. Hybrid page layout analysis via tab-stop detection. In 2009 10thInternational Conference on Document Analysis and Recognition, pages 241–245. IEEE,2009.[79] Y.-C. Song, G.-D. Chen, and L.-Y. Li. Improving E-Book Reading With InformationCues: An User Investigation and Suggestion. In 2013 {IEEE} 13th InternationalConference on Advanced Learning Technologies, pages 261–263, Piscataway, {NJ}, 2013.IEEE.[80] S. N. Srihari. Recognition of handwritten and machine-printed text for postal addressinterpretation. Pattern recognition letters, 14(4):291–302, 1993.[81] T. F. Stahovich and H. Lin. Enabling Data Mining of Handwritten Coursework.Computers & Graphics, 57:31–45, 2016.[82] S. Sweeney and F. Crestani. Effective Search Results Summary Size and Device ScreenSize: Is There a Relationship? Information Processing & Management, 42(4):1056–1074, 2006.[83] T. Tuna, J. Subhlok, L. Barker, V. Varghese, O. Johnson, and S. Shah. Developmentand evaluation of indexed captioned searchable videos for stem coursework. InProceedings of the 43rd ACM technical symposium on Computer Science Education,pages 129–134, 2012.[84] R. A. Ulichney and D. E. Troxel. Scaling binary images with the telescoping template.IEEE transactions on pattern analysis and machine intelligence, (3):331–335, 1982.[85] P. Van Meter, L. Yokoi, and M. Pressley. College Students’ Theory of Note-TakingDerived From Their Perceptions of Note-Taking. Journal of Educational Psychology, 86(3):323–338, 1994.[86] P. C. Vashist, A. Pandey, and A. Tripathi. A Comparative Study of HandwritingRecognition Techniques. In 2020 International Conference on Computation, Automationand Knowledge Management (ICCAKM), pages 456–461, Piscataway, {NJ}, 2020. IEEE.[87] F. Wang, C.-W. Ngo, and T.-C. Pong. Structuring low-quality videotaped lectures forcross-reference browsing by video text analysis. Pattern Recognition, 41(10):3257–3269,862008.[88] K. Wang, B. Babenko, and S. Belongie. End-To-End Scene Text Recognition. In 2011International Conference on Computer Vision, pages 1457–1464, Piscataway, {NJ},2011. IEEE.[89] Z. Wang, A. C. Bovik, and H. R. Sheikh. Structural similarity based image qualityassessment. Digital Video image quality and perceptual coding, pages 225–241, 2005.[90] M.-F. Weng and Y.-Y. Chuang. Multi-Cue Fusion for Semantic Video Indexing. InProceedings of the 16th ACM International Conference on Multimedia, pages 71–80,New York, {NY}, 2008. ACM.[91] R. W. White and D. Morris. Investigating the Querying and Browsing Behavior ofAdvanced Search Engine Users. In Proceedings of the 30th Annual International ACMSIGIR Conference on Research and Development in Information Retrieval, pages255–262, New York, {NY}, 2007. ACM.[92] S. Whittaker. Personal information management: from information consumption tocuration. Annual review of information science and technology, 45(1):1, 2011.[93] L. D. Wilcox, P. Chiu, M. Sasaoka, J. Miyazaki, D. L. Hecht, and L. N. Flores. Systemand method for video access from notes or summaries, Jan. 12 2010. US Patent7,647,555.[94] A. E. Witherby and S. K. Tauber. The Current Status of Students’ Note-Taking: Whyand How Do Students Take Notes? Journal of Applied Research in Memory andCognition, 8(2):139–153, 2019.[95] N. Yamamoto. An Interactive Learning System Using Smartphone: Improving Students’Learning Motivation and Self-Learning. In 2014 Ninth International Conference onBroadband and Wireless Computing, Communication and Applications, pages 428–431.IEEE, 2014.[96] H. Yang and C. Meinel. Content based lecture video retrieval using speech and videotext information. IEEE transactions on learning technologies, 7(2):142–154, 2014.[97] S.-J. Yang, G.-D. Chen, and L.-Y. Li. How Students Use Contextual Cues in FindingInformation in Paper and Electronic Textbooks. In 2011 {IEEE} 11th InternationalConference on Advanced Learning Technologies, pages 302–304, Piscataway, {NJ}, 2011.IEEE.[98] M. Yarmand, D. Yoon, S. Dodson, I. Roll, and S. S. Fels. “Can you believe [1:21]?!”:Content and time-based reference patterns in video comments. In Proceedings of the2019 {CHI} Conference on Human Factors in Computing Systems, pages489:1—-489:12, New York, {NY}, 2019. ACM.[99] R. Yeh, C. Liao, S. Klemmer, F. Guimbretie`re, B. Lee, B. Kakaradov, J. Stamberger,87and A. Paepcke. {ButterflyNet}: A Mobile Capture and Access System for FieldBiology Research. In Proceedings of the {SIGCHI} Conference on Human Factors inComputing Systems, pages 571–580, New York, {NY}, 2006. ACM.[100] C. J. Yuan and W. B. Seales. Guided linking: Efficiently making image-to-transcriptcorrespondence. In Proceedings of the 1st ACM/IEEE-CS Joint Conference on DigitalLibraries, page 471, New York, NY, 2001. Association for Computing Machinery.[101] B. Zhao, S. Lin, X. Luo, S. Xu, and R. Wang. A Novel System for Visual Navigation ofEducational Videos Using Multimodal Cues. In Proceedings of the 25th {ACM}International Conference on Multimedia, pages 1680–1688, New York, {NY}, 2017.ACM.[102] B. Zhao, J. Tao, M. Yang, Z. Tian, C. Fan, and Y. Bai. Deep imitator: handwritingcalligraphy imitation via deep attention networks. Pattern Recognition, page 107080,2020.[103] H. Zheng, J. Wang, Z. Huang, Y. Yang, and R. Pan. Chinese/english mixed charactersegmentation as semantic segmentation. arXiv preprint arXiv:1611.01982, 2016.88Appendices89APPENDIX AParticipant dataA.1 Notebook SamplesThe notebook samples are grouped under each participant number, example P1 and canbe accessed here1. Further details about each of the notebook sample is detailed in thefollowing table A.1.Table A.1: Notebook Sample details. Notes ActualTypeGroupedTypeCourse Video,timepoint.Over-lap. NoOver-lap% ofOver-lap1 P1r1 hybrid equation ELEC211 Coloumb’slaw, 3.234 6 402 P1r2 hybrid text ELEC211 Electric flux,5.113 3 503 P1r3 hybrid equation ELEC211 ElectricDipole, 3.482 9 18.184 P1r4 hybrid text ELEC211 Biot-Savartlaw, 3.057 2 77.785 P1r5 equation equation ELEC211 MagneticForce, 1.064 7 36.361https://ubcca-my.sharepoint.com/:f:/g/personal/ranjs92 student ubc ca/EtvgSRjJKktPqKHcXUxUauIBynZU9R26ePuSuwC-5YvZpA?e=VN1Kei906 P1r6 hybrid text ELEC211 Magnetic ma-terials, 4.5416 5 76.197 P1r7 equation equation ELEC211 MagneticFlux, 1.413 6 33.338 P1r8 equation equation ELEC211 Current, 4.17 2 7 22.229 P1r9 hybrid text ELEC211 Unsure10 P2r1 equation equation BMEG220 PotentialEnergy, 1:43,1:56, 2:092 4 33.3311 P2r2 graph graph BMEG220 Electric flux,2.103 1 75.0012 P2r3 equation equation BMEG220 Current, 3.05 3 4 42.8613 P2r4 graph graph BMEG220 Conductingmaterials,1.121 1 50.0014 P2r5 text text BMEG220 Classificationof Electricalnetworks,1.253 2 60.0015 P2r6 graph graph BMEG220 ElectricDipole, 1.201 1 50.0016 P2r7 equation equation BMEG220 Capacitance,1.321 0 100.0017 P2r8 graph graph BMEG220 Ideal op-amps, 7.400 1 0.009118 P2r9 equation equation BMEG220 Biot-Savartlaw, 4.051 0 100.0019 P2r10 graph graph BMEG220 MagneticFlux, 2.010 1 0.0020 P2r11 equation equation BMEG220 MagneticForce, 5.013 0 100.0021 P2r12 graph graph BMEG220 Solenoids &Toroids, 7.571 0 100.0022 P2r13 hybrid text BMEG220 Magnetic ma-terials, 4.5115 0 100.0023 P2r14 equation equation BMEG220 Faraday’slaw, 6.471 0 100.0024 P2r15 hybrid text BMEG220 Coulomb’slaw, 1.514 0 100.0025 P3r1 text text CPSC340 Neural Net-works -Prediction,1:273 5 37.5026 P3r2 hybrid text CPSC340 Neural Net-works -Prediction,4.252 1 66.679227 P3r3 text text CPSC340 Neural Net-works -Prediction,7.253 1 75.0028 P3r4 text text CPSC340 Neural Net-works -Training,22.195 1 83.3329 P3r5 text text CPSC340 Neural Net-works -Training,22.574 0 100.0030 P3r6 text text CPSC340 Neural Net-works -Training,23.563 3 50.0031 P3r7 text text CPSC340 Convolutionalnetworks,2.046 0 100.0032 P3r8 equation equation CPSC340 Convolutionalnetworks,2.311 0 100.0033 P3r9 text text CPSC340 K-NearestNeighbours,32.277 0 100.009334 P3r10 text text CPSC340 K-NearestNeighbours,36.507 0 100.0035 P3r11 text text CPSC340 K-NearestNeighbours,37.533 3 50.0036 P3r12 text text CPSC340 K-NearestNeighbours,40.372 0 100.0037 P3r13 hybrid text CPSC340 OrdinaryLeastSquares,29.082 1 66.6738 P3r14 text text CPSC340 OrdinaryLeastSquares,33.082 0 100.0039 P3r15 hybrid text CPSC340 NormalEquations,3.314 0 100.0040 P3r16 equation equation CPSC340 NormalEquations,11.401 0 100.009441 P3r17 text text CPSC340 ExploratoryData Analy-sis, 0.355 6 45.4542 P3r18 text text CPSC340 ExploratoryData Analy-sis, 2.367 4 63.6443 P3r19 text text CPSC340 ExploratoryData Analy-sis, 5.453 2 60.0044 P4r1 hybrid graph BMEG220 Coulomb’sLaw, 0:141 2 33.3345 P4r2 equation equation BMEG220 Coulomb’sLaw, 1:181 5 16.6746 P4r3 hybrid text BMEG220 Potential En-ergy, 0:203 7 30.0047 P4r4 equation equation BMEG220 Potential En-ergy, 1:351 6 14.2948 P4r5 hybrid text BMEG220 Potential En-ergy, 2:193 1 75.0049 P4r6 equation equation BMEG220 Potential En-ergy, 3:042 4 33.3350 P4r7 text text BMEG220 Classificationof ElectricalNetworks,0:279 1 90.009551 P4r8 text text BMEG220 Classificationof ElectricalNetworks,0:441 2 33.3352 P4r9 text text BMEG220 Classificationof ElectricalNetworks,1:042 2 50.0053 P4r10 text text BMEG220 Classificationof ElectricalNetworks,2:164 3 57.1454 P4r11 text text BMEG220 Classificationof ElectricalNetworks,3:094 4 50.0055 P4r12 text text BMEG220 Classificationof ElectricalNetworks,4:054 6 40.0056 P4r13 graph graph BMEG220 Capacitance,0:331 0 100.0057 P4r14 equation equation BMEG220 Capacitance,1:131 1 50.009658 P4r15 text text BMEG220 Capacitance,1:320 4 0.0059 P4r16 text text BMEG220 IdealOpAmps,0:402 2 50.0060 P4r35 text text BMEG220 IdealOpAmps,1:362 0 100.0061 P4r17 hybrid graph BMEG220 IdealOpAmps,1:181 0 100.0062 P4r18 text text BMEG220 IdealOpAmps,2:345 2 71.4363 P4r19 hybrid text BMEG220 IdealOpAmps,5:011 2 33.3364 P4r20 hybrid text BMEG220 IdealOpAmps,7:257 3 70.0065 P4r21 hybrid text BMEG220 MagneticFlux, 0:155 1 83.3366 P4r22 hybrid text BMEG220 MagneticFlux, 0:392 6 25.009767 P4r23 equation equation BMEG220 MagneticFlux, 1:322 0 100.0068 P4r24 equation equation BMEG220 MagneticFlux, 2:001 5 16.6769 P4r25 hybrid text BMEG220 MagneticFlux, 3:063 0 100.0070 P4r26 hybrid text BMEG220 MagneticFlux, 3:161 2 33.3371 P4r27 hybrid text BMEG220 MagneticFlux, 4:351 3 25.0072 P4r28 hybrid text BMEG220 Solenoids andToroids, 0:261 3 25.0073 P4r29 hybrid text BMEG220 Solenoids andToroids, 3:031 5 16.6774 P4r30 text text BMEG220 Solenoids andToroids, 6:140 2 0.0075 P4r31 hybrid graph BMEG220 Solenoids andToroids, 6:451 1 50.0076 P4r32 hybrid text BMEG220 Solenoids andToroids, 8:300 2 0.0077 P4r33 hybrid graph BMEG220 Solenoidsand Toroids,10:212 0 100.009878 P4r34 equation equation BMEG220 Solenoidsand Toroids,13:092 1 66.6779 P5r1 equation equation ELEC211 Biot-Savartlaw, 2.201 0 100.0080 P5r2 equation equation ELEC211 Biot-Savartlaw, 3.291 0 100.0081 P5r3 equation equation ELEC211 Biot-Savartlaw, 5.121 0 100.0082 P5r4 equation equation ELEC211 Biot-Savartlaw, 6.051 0 100.0083 P5r5 equation equation ELEC211 Capacitance,1.201 0 100.0084 P5r6 equation equation ELEC211 Capacitance,2.101 0 100.0085 P5r7 equation equation ELEC211 Capacitance,3.101 0 100.0086 P5r8 equation equation ELEC211 Capacitance,3.421 0 100.0087 P5r9 equation equation ELEC211 Capacitance,4.151 0 100.0088 P5r10 equation equation ELEC211 Coulomb’slaw, 1.501 0 100.0089 P5r11 equation equation ELEC211 MagneticForce, 3.530 1 0.009990 P5r12 equation equation ELEC211 MagneticFlux, 1.351 0 100.0091 P5r13 equation equation ELEC211 MagneticFlux, 2.151 0 100.0092 P5r14 equation equation ELEC211 MagneticFlux, 2.513 0 100.0093 P5r15 equation equation ELEC211 MagneticFlux, 3.201 0 100.0094 P5r16 equation equation ELEC211 MagneticForce, 0.401 1 50.0095 P5r17 equation equation ELEC211 MagneticForce, 2.431 0 100.0096 P5r18 equation equation ELEC211 MagneticForce, 6.500 1 0.0097 P5r19 equation equation ELEC211 Magnetic Ma-terials, 7.141 0 100.0098 P5r20 equation equation ELEC211 Magnetic Ma-terials, 7.401 0 100.0099 P6r1 text text React 0.1 3 0 100.00100 P6r2 text text React 0.17 0 1 0.00101 P6r3 text text React 0.47 5 0 100.00102 P6r4 text text React 3.23 3 0 100.00103 P6r5 hybrid graph React 3.29 2 2 50.00104 p6r6 text text React 3.29 2 3 40.00105 P6r7 hybrid graph React 5.07 2 0 100.00100106 P6r8 text text React 6.38 3 1 75.00107 P6r9 text text React 10.07 2 6 25.00108 P6r10 text text React 11.34 3 0 100.00109 P6r11 text text React 13.43 1 4 20.00110 P6r12 text text React 16.35 2 1 66.67111 P6r13 text text React 28.31 4 4 50.00112 P6r14 text text React 39.3 3 1 75.00113 P7r1 text text CPSC340 CS3 Designin Comput-ing, 12.096 3 66.67114 P7r2 text text CPSC340 CS3 Designin Comput-ing, 12.402 2 50.00115 P7r3 text text CPSC340 Fundamentalsof learning,35.363 6 33.33116 P7r4 text text CPSC340 Fundamentalsof learning,41.437 0 100.00117 P7r5 text text CPSC340 Fundamentalsof learning,42.487 0 100.00118 P7r6 text text CPSC340 k-nearestneigh-bors,6.373 1 75.00101119 P7r7 text text CPSC340 k-nearestneighbors,22.147 2 77.78120 P7r8 text text CPSC340 k-nearestneighbors,29.497 0 100.00121 P7r9 text text CPSC340 L2 Regu-larization,12.527 0 100.00122 P7r10 equation equation CPSC340 L2 Regu-larization,13.302 0 100.00123 P7r11 hybrid graph CPSC340 PCA Intu-ition, 4.131 0 100.00124 P7r12 hybrid text CPSC340 PCA train-ing, 27.2211 3 78.57125 P8r1 hybrid text EOSC510 Mean & Vari-ance, 9.031 0 100.00126 P8r2 text text EOSC510 Mean & Vari-ance, 13.113 0 100.00127 P8r3 text text EOSC510 Non-linearoptimization,0.022 0 100.00102128 P8r4 text text EOSC510 Non-linearoptimization,4.306 0 100.00129 P8r5 text text EOSC510 Non-linearoptimization,13.004 2 66.67130 P8r6 hybrid equation EOSC510 Non-linearoptimization,15.402 3 40.00131 P8r7 hybrid text EOSC510 Eigen vectorapproach,0.084 2 66.67132 P8r8 text text EOSC510 Classification:k nearestneighbouringclassifier ,0.055 0 100.00133 P8r9 text text EOSC510 Classification:k nearestneighbouringclassifier,9.363 1 75.00103134 P8r10 text text EOSC510 Classification:k nearestneighbouringclassifier,13.026 0 100.00135 P8r11 text text EOSC510 Mean & Vari-ance, 4.172 0 100.00136 P8r12 hybrid equation EOSC510 Mean & Vari-ance, 6.281 0 100.00137 P9r1 text text EOSC510 Correlation& Regression,0.340 1 0.00138 P9r2 text text EOSC510 Correlation& Regression,0.560 1 0.00139 P9r3 equation equation EOSC510 Mean & Vari-ance, 2.151 0 100.00140 P9r4 equation equation EOSC510 Mean & Vari-ance, 4.261 0 100.00141 P9r5 equation equation EOSC510 Mean & Vari-ance, 4.441 0 100.00142 P9r6 equation equation EOSC510 Mean & Vari-ance, 6.402 0 100.00143 P9r7 equation equation EOSC510 Mean & Vari-ance, 7.281 0 100.00104144 P9r8 text text EOSC510 Mean & Vari-ance, 8.395 0 100.00145 P9r9 equation equation EOSC510 Mean & Vari-ance, 9.154 0 100.00146 P9r10 equation equation EOSC510 Mean & Vari-ance, 11.061 0 100.00147 P9r11 equation equation EOSC510 Mean & Vari-ance, 11.431 0 100.00148 P9r12 text text EOSC510 Mean & Vari-ance, 13.114 0 100.00149 P9r13 equation equation EOSC510 Mean andVariance,14.281 0 100.00150 P9r14 equation equation EOSC510 Mean & Vari-ance, 16.151 0 100.00151 P9r15 hybrid text EOSC510 Mean & Vari-ance, 18.253 0 100.00152 P9r16 equation equation EOSC510 Mean & Vari-ance, 19.121 0 100.00153 P9r17 text text EOSC510 Linear Re-gression,0.084 0 100.00154 P9r18 equation equation EOSC510 Linear Re-gression,0.561 0 100.00105155 P9r19 equation equation EOSC510 Linear Re-gression,2.201 0 100.00156 P9r20 equation equation EOSC510 Linear Re-gression,4.192 0 100.00157 P9r21 equation equation EOSC510 Linear Re-gression,5.052 0 100.00158 P9r22 text text EOSC510 Geometricapproach,00:272 2 50.00159 P9r23 text text EOSC510 Geometricapproach,6.417 0 100.00160 P9r24 text text EOSC510 PCA appliedto real data00:221 0 100.00161 P9r25 text text EOSC510 PCA appliedto real data00:261 0 100.00162 P9r26 text text EOSC510 PCA appliedto real data00:544 0 100.00106163 P9r27 hybrid graph EOSC510 Rotated PCA00:171 0 100.00164 P9r28 hybrid graph EOSC510 Rotated PCA00:291 0 100.00165 P9r29 graph graph EOSC510 Rotated PCA00:411 0 100.00166 P9r30 graph graph EOSC510 Rotated PCA2.441 0 100.00167 P9r31 text text EOSC510 Rotated PCA3.255 0 100.00168 P10r1 equation equation EOSC510 Mean andVariance,0.011 0 100.00169 P10r2 equation equation EOSC510 Mean andVariance,4.171 0 100.00170 P10r3 equation equation EOSC510 Mean andVariance,4.171 0 100.00171 P10r4 equation equation EOSC510 Mean andVariance,15.511 0 100.00172 P10r5 equation equation EOSC510 MLP, 0.02 1 0 100.00173 P10r6 text text EOSC510 MLP, 8.06 1 0 100.00107174 P10r7 text text EOSC510 PCA –Geometricapproach0:016 0 100.00175 P10r8 text text EOSC510 PCA – Eigenvector ap-proach 0:085 0 100.00176 P10r9 equation equation EOSC510 PCA – Eigenvector ap-proach 0:081 0 100.00177 P10r10 equation equation EOSC510 ComplexData 7:451 0 100.00178 P10r11 equation equation EOSC510 ComplexData 9:171 0 100.00179 P10r12 hybrid text EOSC510 Scaling 9:35 2 0 100.00180 P10r13 text text EOSC510 KNN classi-fier 2:282 2 50.00181 P10r14 text text EOSC510 KNN classi-fier 4.363 3 50.00108APPENDIX BStudy ResultsB.1 Accuracy-Overlap resultsIn this section, the results of the accuracy calculation are outlined in the table B.1. Thecolumns show the note sample No. along with the type. 3 best matches are listed in theresults column which can be compared with the ground truth observations. The leveldemonstrates the temporal level in which the results were retrieved. The percentage pfoverlap for both the actual ground truth video and the results videos are listed.109Table B.1: Accuracy-Overlap Evaluation ResultsNote NoteTypeResults Ground Truth Level ActualOver-lap(%)ResultOver-lap(%)P1r1 formula {”name”: ”6- Con-duction Materi-als”, ”timestamp”:”00:03:48”},{”name”: ”9- Mag-netic Flux”, ”times-tamp”: ”00:06:24”},{”name”: ”11-Magnetic Materi-als”, ”timestamp”:”00:05:22”}Coloumb’slaw, 3.23No match 40.00 NAP1r2 text {”name”: ”8- Biot-Savart Law”, ”times-tamp”: ”00:04:07”},{”name”: ”9- Mag-netic Flux”, ”times-tamp”: ”00:00:44”},{”name”: ”3- ElectricFlux”, ”timestamp”:”00:03:28”}Electric flux,5.113rd,Section50.00 33.33110P1r3 formula {”name”: ”9- Mag-netic Flux”, ”times-tamp”: ”00:03:05”},{”name”: ”4- ElectricDipole”, ”timestamp”:”00:03:40”},{”name”: ”8- Biot-Savart Law”, ”times-tamp”: ”00:00:30”}ElectricDipole, 3.482nd,frame18.18 18.18P1r4 text {”name”: ”8- Biot-Savart Law”, ”times-tamp”: ”00:02:24”},{”name”: ”10- Mag-netic Force”, ”times-tamp”: ”00:02:14”},{”name”: ”12- Fara-day’s Law”, ”times-tamp”: ”00:00:00”}Biot-Savartlaw, 3.051st,Frame77.78 77.78111P1r5 formula {”name”: ”11-Magnetic Materi-als”, ”timestamp”:”00:07:12”},{”name”: ”9- Mag-netic Flux”, ”times-tamp”: ”00:00:20”},{”name”: ”12- Fara-day’s Law”, ”times-tamp”: ”00:05:45”}MagneticForce, 1.06Nomatch36.36 NAP1r6 text {”name”: ”11-Magnetic Materi-als”, ”timestamp”:”00:04:59”},{”name”: ”12- Fara-day’s Law”, ”times-tamp”: ”00:00:10”},{”name”: ”8- Biot-Savart Law”, ”times-tamp”: ”00:00:20”}Magnetic ma-terials, 4.541st,Frame76.19 76.19112P1r7 formula {”name”: ”9- Mag-netic Flux”, ”times-tamp”: ”00:02:55”},{”name”: ”11-Magnetic Materi-als”, ”timestamp”:”00:07:12”},{”name”: ”8- Biot-Savart Law”, ”times-tamp”: ”00:04:47”}Magnetic Flux,1.411st,section33.33 66.67P1r8 formula {”name”: ”5- Cur-rent”, ”timestamp”:”00:01:43”},{”name”: ”7- Capaci-tance”, ”timestamp”:”00:01:38”},{”name”: ”10- Mag-netic Force”, ”times-tamp”: ”00:02:24”}Current, 4.17 1st,video22.22 11.11113P1r9 text {”name”: ”10- Mag-netic Force”, ”times-tamp”: ”00:00:34”},{”name”: ”2- PotentialEnergy”, ”timestamp”:”00:00:50”},{”name”: ”11-Magnetic Materi-als”, ”timestamp”:”00:03:09”}Unsure Specialcase.DIV/0! .DIV/0!P2r1 formula {”name”: ”2- PotentialEnergy”, ”timestamp”:”00:02:05”},{”name”: ”13 - Mag-netic Force”, ”times-tamp”: ”00:00:20”},{”name”: ”5- Con-ducting Materi-als”, ”timestamp”:”00:02:05”}Potential En-ergy, 1:43,1:56, 2:091st,frame33.33 33.33114P2r2 figure {”name”: ”8- ElectricDipole”, ”timestamp”:”01:10”},{”name”: ”9- Capaci-tance”, ”timestamp”:”00:35”},{”name”: ”14 -Solenoids andToroids”, ”times-tamp”: ”09:30”}Electric flux,2.10Nomatch75.00 NAP2r3 formula {”name”: ”4- Cur-rent”, ”timestamp”:”00:01:43”},{”name”: ”15 -Magnetic Materi-als”, ”timestamp”:”00:03:09”}Current, 3.05 1st,section42.86 28.57115P2r4 figure {”name”: ”2- PotentialEnergy”, ”timestamp”:”01:45”},{”name”: ”1 -Coulomb’s Law”,”timestamp”:”02:20”},{”name”: ”8- Biot-Savart Law”, ”times-tamp”: ”00:50”}Conductingmaterials, 1.12Nomatch50.00 NAP2r5 text ”name”: ”6- Clas-sification of Electri-cal Networks”, ”times-tamp”: ”00:01:20”},{”name”: ”2- PotentialEnergy”, ”timestamp”:”00:05:19”},{”name”: ”9- Capac-itance”, ”timestamp”:”00:04:45”}ClassificationofElec-tri-calnet-works,1.251st,frame60.00 60.00116P2r6 figure {”name”: ”14 -Solenoids andToroids”, ”times-tamp”: ”06:00”},{”name”: ”1 -Coulomb’s Law”,”timestamp”:”02:20”},{”name”: ”9- Capaci-tance”, ”timestamp”:”00:35”}ElectricDipole, 1.20Nomatch50.00 NAP2r7 formula Capacitance,1.32Nodata100.00 NAP2r8 figure {”name”: ”2- PotentialEnergy”, ”timestamp”:”01:45”},{”name”: ”1 -Coulomb’s Law”,”timestamp”:”02:15”},{”name”: ”9- Capaci-tance”, ”timestamp”:”00:35”}Ideal op-amps,7.40Nomatch0.00 NAP2r9 formula Biot-Savartlaw, 4.05Nodata100.00 NA117P2r10 figure {”name”: ”8- Biot-Savart Law”, ”times-tamp”: ”01:50”},{”name”: ”9- Capaci-tance”, ”timestamp”:”00:35”},{”name”: ”5- Con-ducting Materials”,”timestamp”: ”04:15”}Magnetic Flux,2.01Nomatch0.00 NAP2r11 formula {”name”: ”17 - Fara-day’s Law”, ”times-tamp”: ”00:05:45”},{”name”: ”12- Mag-netic Flux”, ”times-tamp”: ”00:00:20”},{”name”: ”8- Biot-Savart Law”, ”times-tamp”: ”00:02:43”}MagneticForce, 5.01Nomatch100.00 NA118P2r12 figure {”name”: ”1 -Coulomb’s Law”,”timestamp”:”02:25”},{”name”: ”8- Biot-Savart Law”, ”times-tamp”: ”02:15”},{”name”: ”14 -Solenoids andToroids”, ”times-tamp”: ”06:00”}Solenoids &Toroids, 7.573rd,section100.00 0.00P2r13 text {”name”: ”15 -Magnetic Materi-als”, ”timestamp”:”00:04:59”},{”name”: ”14 -Solenoids andToroids”, ”times-tamp”: ”00:01:56” ,{”name”: ”13 - Mag-netic Force”, ”times-tamp”: ”00:00:00”}Magnetic ma-terials, 4.511st,Frame100.00 100.00119P2r14 formula {”name”: ”17 - Fara-day’s Law”, ”times-tamp”: ”00:07:35”},{”name”: ”12- Mag-netic Flux”, ”times-tamp”: ”00:00:20”},{”name”: ”8- Biot-Savart Law”, ”times-tamp”: ”00:02:43”}Faraday’s law,6.471st,video100.00 0.00P2r15 text {”name”: ”1 -Coulomb’s Law”,”timestamp”:”00:02:02”},{”name”: ”18 -Displacement Cur-rent \u00a9CarolJaeger”, ”timestamp”:”00:07:09”},{”name”: ”12- Mag-netic Flux”, ”times-tamp”: ”00:00:20”}Coulomb’slaw, 1.511st,frame100.00 100.00120P3r1 text {”name”: ”NeuralNetworks - Predic-tion”, ”timestamp”:”00:03:09”},{”name”: ”K-NearestNeighbors”, ”times-tamp”: ”00:32:12”},{”name”: ”PCA”,”timestamp”:”00:37:40”}Neural Net-works - Predic-tion, 1:271st,section37.50 87.50P3r2 text {”name”: ”Con-volutional NeuralNetworks”, ”times-tamp”: ”02:40”},{”name”: ”CS3:Design in Comput-ing”, ”timestamp”:”01:15”},{”name”: ”Neural net-works”, ”timestamp”:”10:50”}Neural Net-works - Predic-tion, 4.25Nomatch66.67 NA121P3r3 text {”name”: ”NeuralNetworks - Predic-tion”, ”timestamp”:”00:08:01”},{”name”: ”PCA”,”timestamp”:”00:02:30”},{”name”: ”NormalEquations”, ”times-tamp”: ”00:28:40”}Neural Net-works - Predic-tion, 7.251st Sec-tion75.00 75.00P3r4 text {”name”: ”Neural net-works”, ”timestamp”:”00:23:26”},{”name”: ”K-NearestNeighbors”, ”times-tamp”: ”00:18:16”},{”name”: ”CS3:Design in Comput-ing”, ”timestamp”:”00:38:54”}Neural Net-works - Train-ing, 22.191st sec-tion83.33 100.00122P3r5 text {”name”: ”Funda-mentals of Learn-ing”, ”timestamp”:”00:15:18”},{”name”: ”NeuralNetworks - Predic-tion”, ”timestamp”:”00:37:24”},{”name”: ”K-NearestNeighbors”, ”times-tamp”: ”00:09:02”}Neural Net-works - Train-ing, 22.57Nomatch100.00 NAP3r6 text {”name”: ”Neural net-works”, ”timestamp”:”00:24:40”},{”name”: ”NeuralNetworks - Predic-tion”, ”timestamp”:”00:11:32”},{”name”: ”Ordinaryleast squares”, ”times-tamp”: ”00:20:27”}Neural Net-works - Train-ing, 23.561st sec-tion50.00 100.00123P3r7 text {”name”: ”Con-volutional NeuralNetworks”, ”times-tamp”: ”00:02:28”},{”name”: ”PCA -Training”, ”times-tamp”: ”00:02:21”},{”name”: ”Neural net-works”, ”timestamp”:”00:00:00”}Convolutionalnetworks, 2.041st Sec-tion100.00 100.00P3r8 formula Convolutionalnetworks, 2.31Nodata100.00 NAP3r9 text {”name”: ”K-NearestNeighbors”, ”times-tamp”: ”00:32:47”},{”name”: ”NormalEquations”, ”times-tamp”: ”00:08:17”},{”name”: ”Ordinaryleast squares”, ”times-tamp”: ”00:19:29”}K-NearestNeighbours,32.271stframe100.00 100.00124P3r10 text {”name”: ”K-NearestNeighbors”, ”times-tamp”: ”00:37:32”},{”name”: ”Ex-ploratory Data Anal-ysis”, ”timestamp”:”00:38:39”},{”name”: ”NormalEquations”, ”times-tamp”: ”00:44:44”}K-NearestNeighbours,36.501stframe100.00 100.00P3r11 text {”name”: ”K-NearestNeighbors”, ”times-tamp”: ”00:39:06”},{”name”: ”PCA -Training”, ”times-tamp”: ”00:01:00”},{”name”: ”NormalEquations”, ”times-tamp”: ”00:35:58”}K-NearestNeighbours,37.531st Sec-tion50.00 100.00P3r12 text K-NearestNeighbours,40.37Nodata100.00 NA125P3r13 text {”name”: ”Ordinaryleast squares”, ”times-tamp”: ”00:00:00”},{”name”: ”NormalEquations”, ”times-tamp”: ”00:01:13”},{”name”: ”K-NearestNeighbors”, ”times-tamp”: ”00:23:31”}Ordinary LeastSquares, 29.081stvideo66.67 33.33P3r14 text Ordinary LeastSquares, 33.08Nodata100.00 NAP3r15 text {”name”: ”K-NearestNeighbors”, ”times-tamp”: ”00:03:04”},{”name”: ”NormalEquations”, ”times-tamp”: ”00:20:55”},{”name”: ”Neural net-works”, ”timestamp”:”00:06:30”}Normal Equa-tions, 3.311stvideo100.00 25.00P3r16 formula [] Normal Equa-tions, 11.40Nodata100.00 NA126P3r17 text {”name”: ”K-NearestNeighbors”, ”times-tamp”: ”00:09:02”},{”name”: ”Ex-ploratory Data Anal-ysis”, ”timestamp”:”00:22:03”},{”name”: ”Funda-mentals of Learn-ing”, ”timestamp”:”00:32:37”}ExploratoryData Analysis,0.352ndvideo45.45 27.27P3r18 text {”name”: ”Ex-ploratory Data Anal-ysis”, ”timestamp”:”00:05:16”},{”name”: ”Regular-ization”, ”timestamp”:”00:02:18”},{”name”: ”Funda-mentals of Learn-ing”, ”timestamp”:”00:47:27”}ExploratoryData Analysis,2.361st sec-tion63.64 54.55127P3r19 text {”name”: ”Ordinaryleast squares”, ”times-tamp”: ”00:02:27”},{”name”: ”Ex-ploratory Data Anal-ysis”, ”timestamp”:”00:46:34”},{”name”: ”K-NearestNeighbors”, ”times-tamp”: ”00:23:31”}ExploratoryData Analysis,5.452ndvideo60.00 60.00P4r1 figure {”name”: ”17 - Fara-day’s Law”, ”times-tamp”: ”00:00:00”},{”name”: ”8- Biot-Savart Law”, ”times-tamp”: ”00:00:50”},{”name”: ”3- ElectricFlux”, ”timestamp”:”00:00:20”}Coulomb’sLaw, 0:14Nomatch33.33 NA128P4r2 formula {”name”: ”1 -Coulomb’s Law”,”timestamp”:”00:02:02”},{”name”: ”5- Con-ducting Materi-als”, ”timestamp”:”00:02:55”},{”name”: ”12- Mag-netic Flux”, ”times-tamp”: ”00:00:00”}Coulomb’sLaw, 1:181st,section16.67 50.00P4r3 text {”name”: ”2- PotentialEnergy”, ”timestamp”:”00:00:50”},{”name”: ”14 -Solenoids andToroids”, ”times-tamp”: ”00:01:56”},{”name”: ”17 - Fara-day’s Law”, ”times-tamp”: ”00:00:32”}Potential En-ergy, 0:201st,section30.00 40.00129P4r4 formula {”name”: ”2- PotentialEnergy”, ”timestamp”:”00:02:05”},{”name”: ”15 -Magnetic Materi-als”, ”timestamp”:”00:07:02”},{”name”: ”17 - Fara-day’s Law”, ”times-tamp”: ”00:06:05”}Potential En-ergy, 1:351st,section14.29 85.71P4r5 text {”name”: ”8- Biot-Savart Law”, ”times-tamp”: ”00:02:43”},{”name”: ”2- PotentialEnergy”, ”timestamp”:”00:00:25”},{”name”: ”13 - Mag-netic Force”, ”times-tamp”: ”00:00:00”}Potential En-ergy, 2:192nd,video75.00 50.00130P4r6 formula {”name”: ”12- Mag-netic Flux”, ”times-tamp”: ”00:00:20”},{”name”: ”4- Cur-rent”, ”timestamp”:”00:01:43”},{”name”: ”15 -Magnetic Materi-als”, ”timestamp”:”00:03:19”}Potential En-ergy, 3:04Nomatch33.33 NAP4r7 text {”name”: ”6- Clas-sification of Electri-cal Networks”, ”times-tamp”: ”00:00:30”},{”name”: ”17 - Fara-day’s Law”, ”times-tamp”: ”00:04:06”},{”name”: ”2- PotentialEnergy”, ”timestamp”:”00:00:25”}Classificationof ElectricalNetworks, 0:271st,frame90.00 90.00131P4r8 text {”name”: ”8- Biot-Savart Law”, ”times-tamp”: ”00:02:54”},{”name”: ”6- Classi-fication of ElectricalNetworks”, ”times-tamp”: ”00:00:00”},{”name”: ”14 -Solenoids andToroids”, ”times-tamp”: ”00:00:00”}Classificationof ElectricalNetworks, 0:442nd,video33.33 0.00P4r9 text {”name”: ”6- Clas-sification of Electri-cal Networks”, ”times-tamp”: ”00:01:10”},{”name”: ”8- Biot-Savart Law”, ”times-tamp”: ”00:02:43”},{”name”: ”13 - Mag-netic Force”, ”times-tamp”: ”00:00:00”}Classificationof ElectricalNetworks, 1:041st,Frame50.00 50.00132P4r10 text {”name”: ”6- Classi-fication of ElectricalNetworks”, ”times-tamp”: ”00:02:47”},{”name”: ”8- Biot-Savart Law”, ”times-tamp”: ”00:02:43”},{”name”: ”3- ElectricFlux”, ”timestamp”:”00:05:58”}Classificationof ElectricalNetworks, 2:161st,Section57.14 71.43P4r11 text {”name”: ”6- Classi-fication of ElectricalNetworks”, ”times-tamp”: ”00:03:17”},{”name”: ”8- Biot-Savart Law”, ”times-tamp”: ”00:06:38”},{”name”: ”14 -Solenoids andToroids”, ”times-tamp”: ”00:00:00”}Classificationof ElectricalNetworks, 3:091st,Frame50.00 50.00133P4r12 text {”name”: ”6- Classi-fication of ElectricalNetworks”, ”times-tamp”: ”00:04:54”},{”name”: ”10- IdealOpAmps”, ”times-tamp”: ”00:10:58”},{”name”: ”15 -Magnetic Materi-als”, ”timestamp”:”00:05:22”}Classificationof ElectricalNetworks, 4:051st,Section40.00 80.00P4r13 figure {”name”: ”8- Biot-Savart Law”, ”times-tamp”: ”02:15”},{”name”: ”13 - Mag-netic Force”, ”times-tamp”: ”06:20”},”name”: ”12- Mag-netic Flux”, ”times-tamp”: ”02:25”}Capacitance,0:33Nomatch100.00 NAP4r14 formula [] Capacitance,1:13Nodata50.00 NA134P4r15 text {”name”: ”18 -Displacement Cur-rent \u00a9CarolJaeger”, ”timestamp”:”00:09:20”},{”name”: ”9- Capaci-tance”, ”timestamp”:”00:02:45”},{”name”: ”2- PotentialEnergy”, ”timestamp”:”00:00:25”Capacitance,1:322nd,video0.00 50.00P4r16 text Ideal OpAmps,0:40Nodata50.00 NAP4r35 text {”name”: ”10- IdealOpAmps”, ”times-tamp”: ”00:01:40”},{”name”: ”6- Classi-fication of ElectricalNetworks”, ”times-tamp”: ”00:01:37”}Ideal OpAmps,1:361stframe100.00 100.00135P4r17 figure {”name”: ”14 -Solenoids andToroids”, ”times-tamp”: ”06:00”},{”name”: ”2- PotentialEnergy”, ”timestamp”:”03:25”}{”name”: ”18 - Dis-placement Current\u00a9Carol Jaeger”,”timestamp”: ”02:40”}Ideal OpAmps,1:18Nomatch100.00 NAP4r18 text {”name”: ”10- IdealOpAmps”, ”times-tamp”: ”00:01:40”},{”name”: ”17 - Fara-day’s Law”, ”times-tamp”: ”00:00:00”},{”name”: ”12- Mag-netic Flux”, ”times-tamp”: ”00:00:00”}Ideal OpAmps,2:341st sec-tion71.43 57.14136P4r19 text {”name”: ”15 -Magnetic Materi-als”, ”timestamp”:”00:03:09”},{”name”: ”10- IdealOpAmps”, ”times-tamp”: ”00:03:36”},{”name”: ”17 - Fara-day’s Law”, ”times-tamp”: ”00:00:00”}Ideal OpAmps,5:012nd,video33.33 66.67P4r20 text {”name”: ”10- IdealOpAmps”, ”times-tamp”: ”00:08:03”},{”name”: ”8- Biot-Savart Law”, ”times-tamp”: ”00:02:43”},{”name”: ”14 -Solenoids andToroids”, ”times-tamp”: ”00:04:30”}Ideal OpAmps,7:251st sec-tion70.00 90.00137P4r21 text {”name”: ”12- Mag-netic Flux”, ”times-tamp”: ”00:00:20”},{”name”: ”18 -Displacement Cur-rent \u00a9CarolJaeger”, ”timestamp”:”00:07:09”},{”name”: ”15 -Magnetic Materi-als”, ”timestamp”:”00:05:22”}Magnetic Flux,0:151stframe83.33 83.33P4r22 text {”name”: ”12- Mag-netic Flux”, ”times-tamp”: ”00:01:00”},{”name”: ”13 - Mag-netic Force”, ”times-tamp”: ”00:00:34”},{”name”: ”17 - Fara-day’s Law”, ”times-tamp”: ”00:00:32”}Magnetic Flux,0:391st sec-tion25.00 75.00P4r23 formula Magnetic Flux,1:32Nodata100.00 NA138P4r24 formula {”name”: ”12- Mag-netic Flux”, ”times-tamp”: ”00:00:20”}Magnetic Flux,2:001st,video16.67 33.33P4r25 text {”name”: ”12- Mag-netic Flux”, ”times-tamp”: ”00:03:45”},{”name”: ”17 - Fara-day’s Law”, ”times-tamp”: ”00:02:08”},{”name”: ”3- ElectricFlux”, ”timestamp”:”00:01:48”}Magnetic Flux,3:061st Sec-tion100.00 100.00P4r26 text {”name”: ”12- Mag-netic Flux”, ”times-tamp”: ”00:03:15”},{”name”: ”17 - Fara-day’s Law”, ”times-tamp”: ”00:03:38”},{”name”: ”3- ElectricFlux”, ”timestamp”:”00:01:00”}Magnetic Flux,3:161stframe33.33 33.33139P4r27 text {”name”: ”18 -Displacement Cur-rent \u00a9CarolJaeger”, ”timestamp”:”00:06:28”},{”name”: ”17 - Fara-day’s Law”, ”times-tamp”: ”00:00:32”},{”name”: ”12- Mag-netic Flux”, ”times-tamp”: ”00:00:44”}Magnetic Flux,4:353rd,video25.00 25.00P4r28 text {”name”: ”14 -Solenoids andToroids”, ”times-tamp”: ”00:00:00”}Solenoids andToroids, 0:261stframe25.00 25.00140P4r29 text {”name”: ”14 -Solenoids andToroids”, ”times-tamp”: ”00:05:30”},{”name”: ”17 - Fara-day’s Law”, ”times-tamp”: ”00:03:38”},{”name”: ”15 -Magnetic Materi-als”, ”timestamp”:”00:01:09”}Solenoids andToroids, 3:031st sec-tion16.67 50.00P4r30 text Solenoids andToroids, 6:14Nodata0.00 NAP4r31 figure {”name”: ”14 -Solenoids andToroids”, ”times-tamp”: ”06:00”},{”name”: ”18 -Displacement Cur-rent \u00a9CarolJaeger”, ”timestamp”:”03:05”},{”name”: ”5- Con-ducting Materials”,”timestamp”: ”00:15”}Solenoids andToroids, 6:451st Sec-tion50.00 0.00141P4r32 text {”name”: ”14 -Solenoids andToroids”, ”times-tamp”: ”00:10:01”},{”name”: ”12- Mag-netic Flux”, ”times-tamp”: ”00:00:54”},{”name”: ”17 - Fara-day’s Law”, ”times-tamp”: ”00:04:06”}Solenoids andToroids, 8:301st sec-tion0.00 50.00P4r33 figure {”name”: ”2- PotentialEnergy”, ”timestamp”:”01:45”},{”name”: ”8- Biot-Savart Law”, ”times-tamp”: ”00:50”},{”name”: ”14 -Solenoids andToroids”, ”times-tamp”: ”05:00”}Solenoids andToroids, 10:213rdvideo100.00 50.00142P4r34 formula {”name”: ”8- Biot-Savart Law”, ”times-tamp”: ”00:00:30”},{”name”: ”12- Mag-netic Flux”, ”times-tamp”: ”00:00:20”}Solenoids andToroids, 13:09Nomatch66.67 NAP5r1 formula {”name”: ”8- Biot-Savart Law”, ”times-tamp”: ”00:06:07”},{”name”: ”12- Fara-day’s Law”, ”times-tamp”: ”00:07:30”},{”name”: ”1 -Coulomb’s Law”,”timestamp”:”00:01:52”}Biot-Savartlaw, 2.201stvideo100.00 100.00P5r2 formula {”name”: ”12- Fara-day’s Law”, ”times-tamp”: ”00:04:06”},{”name”: ”5- Cur-rent”, ”timestamp”:”00:01:58”},{”name”: ”8- Biot-Savart Law”, ”times-tamp”: ”00:00:30”}Biot-Savartlaw, 3.293rdvideo100.00 0.00143P5r3 formula {”name”: ”8- Biot-Savart Law”, ”times-tamp”: ”00:06:07”},{”name”: ”12- Fara-day’s Law”, ”times-tamp”: ”00:07:30”},{”name”: ”5- Cur-rent”, ”timestamp”:”00:01:58”}Biot-Savartlaw, 5.121st sec-tion100.00 0.00P5r4 formula {”name”: ”12- Fara-day’s Law”, ”times-tamp”: ”00:07:30”},{”name”: ”5- Cur-rent”, ”timestamp”:”00:01:58”},{”name”: ”8- Biot-Savart Law”, ”times-tamp”: ”00:02:58”}Biot-Savartlaw, 6.053rdvideo100.00 0.00P5r5 formula {”name”: ”7- Capac-itance”, ”timestamp”:”00:01:38”}Capacitance,1.201st sec-tion100.00 100.00P5r6 formula [] Capacitance,2.10Nodata100.00 NAP5r7 formula [] Capacitance,3.10Nodata100.00 NA144P5r8 formula {”name”: ”7- Capac-itance”, ”timestamp”:”00:03:31”}Capacitance,3.421st sec-tion100.00 100.00P5r9 formula {”name”: ”7- Capac-itance”, ”timestamp”:”00:01:38”}Capacitance,4.151stvideo100.00 0.00P5r10 formula {”name”: ”7- Capac-itance”, ”timestamp”:”00:03:23”},{”name”: ”8- Biot-Savart Law”, ”times-tamp”: ”00:02:59”},{”name”: ”9- Mag-netic Flux”, ”times-tamp”: ”00:00:15”}Coulomb’slaw, 1.50Nomatch100.00 NA145P5r11 formula {”name”: ”12-Faraday’sLaw”,”timestamp”:”00:07:30”},{”name”: ”8- Biot-Savart Law”, ”times-tamp”: ”00:02:43”},{”name”: ”11-Magnetic Materi-als”, ”timestamp”:”00:07:07”}MagneticForce, 3.53Nomatch0.00 NAP5r12 formula {”name”: ”12- Fara-day’s Law”, ”times-tamp”: ”00:04:21”},{”name”: ”9- Mag-netic Flux”, ”times-tamp”: ”00:05:00”},{”name”: ”8- Biot-Savart Law”, ”times-tamp”: ”00:00:30”}Magnetic Flux,1.352ndvideo100.00 100.00146P5r13 formula {”name”: ”9- Mag-netic Flux”, ”times-tamp”: ”00:05:00”},{”name”: ”12- Fara-day’s Law”, ”times-tamp”: ”00:04:06”},{”name”: ”8- Biot-Savart Law”, ”times-tamp”: ”00:00:30”}Magnetic Flux,2.151stvideo100.00 0.00P5r14 formula {”name”: ”9- Mag-netic Flux”, ”times-tamp”: ”00:03:05”},{”name”: ”8- Biot-Savart Law”, ”times-tamp”: ”00:00:30”},{”name”: ”5- Cur-rent”, ”timestamp”:”00:00:43”}Magnetic Flux,2.511stframe100.00 100.00147P5r15 formula {”name”: ”8- Biot-Savart Law”, ”times-tamp”: ”00:04:28”},{”name”: ”1 -Coulomb’s Law”,”timestamp”:”00:01:37”},{”name”: ”12- Fara-day’s Law”, ”times-tamp”: ”00:05:45”}Magnetic Flux,3.20Nomatch100.00 NAP5r16 formula MagneticForce, 0.40Nodata50.00 NAP5r17 formula {”name”: ”12- Fara-day’s Law”, ”times-tamp”: ”00:06:00”},”name”: ”11- MagneticMaterials”, ”times-tamp”: ”00:07:07”},{”name”: ”5- Cur-rent”, ”timestamp”:”00:01:48”}MagneticForce, 2.43Nomatch100.00 NA148P5r18 formula {”name”: ”8- Biot-Savart Law”, ”times-tamp”: ”00:00:30”},{”name”: ”5- Cur-rent”, ”timestamp”:”00:00:43”},{”name”: ”9- Mag-netic Flux”, ”times-tamp”: ”00:00:15”MagneticForce, 6.50Nomatch0.00 NAP5r19 formula {”name”: ”12- Fara-day’s Law”, ”times-tamp”: ”00:07:30”},{”name”: ”11-Magnetic Materi-als”, ”timestamp”:”00:07:07”},{”name”: ”8- Biot-Savart Law”, ”times-tamp”: ”00:02:43”}Magnetic Ma-terials, 7.142ndframe100.00 100.00149P5r20 formula {”name”: ”11-Magnetic Materi-als”, ”timestamp”:”00:07:07”},{”name”: ”8- Biot-Savart Law”, ”times-tamp”: ”00:00:30”},{”name”: ”5- Cur-rent”, ”timestamp”:”00:01:48”}Magnetic Ma-terials, 7.401st sec-tion100.00 0.00P6r1 text {”name”: ”Re-act”, ”timestamp”:”00:00:00”}0.1 1st Sec-tion100.00 66.67P6r2 text 0.17 Nodata0.00 NAP6r3 text {”name”: ”Re-act”, ”timestamp”:”00:01:00”}0.47 1st sec-tion100.00 80.00P6r4 text {”name”: ”Re-act”, ”timestamp”:”00:03:09”}3.23 1st sec-tion100.00 66.67P6r5 figure {”name”: ”React”,”timestamp”: ”19:30”}3.29 video 50.00 0.00150p6r6 text {”name”: ”Re-act”, ”timestamp”:”00:03:42”},{”name”: ”Re-act”, ”timestamp”:”00:01:00”}3.29 1st sec-tion40.00 80.00P6r7 figure {”name”: ”React”,”timestamp”: ”3:10”}5.07 1stvideo100.00 0.00P6r8 text [] 6.38 Nodata75.00 NAP6r9 text {”name”: ”Re-act”, ”timestamp”:”00:09:57”},{”name”: ”Re-act”, ”timestamp”:”00:03:42”},{”name”: ”Re-act”, ”timestamp”:”00:00:00”}10.07 1st sec-tion25.00 75.00P6r10 text 11.34 Nodata100.00 NA151P6r11 text {”name”: ”Re-act”, ”timestamp”:”00:13:16”},{”name”: ”Re-act”, ”timestamp”:”00:10:17”},{”name”: ”Re-act”, ”timestamp”:”00:03:42”}13.43 1stvideo20.00 0.00P6r12 text {”name”: ”Re-act”, ”timestamp”:”00:15:02”},{”name”: ”Re-act”, ”timestamp”:”00:00:00”},{”name”: ”Re-act”, ”timestamp”:”01:75:35”}16.35 1st sec-tion66.67 66.67152P6r13 text {”name”: ”Re-act”, ”timestamp”:”00:27:57”},{”name”: ”Re-act”, ”timestamp”:”00:01:00”},{”name”: ”Re-act”, ”timestamp”:”00:01:09”}28.31 1st sec-tion50.00 50.00P6r14 text {”name”: ”Re-act”, ”timestamp”:”00:29:48”},{”name”: ”Re-act”, ”timestamp”:”00:02:29”},{”name”: ”Re-act”, ”timestamp”:”00:01:00”}39.3 1st sec-tion75.00 75.00153P7r1 text {”name”: ”CS3:Design in Comput-ing”, ”timestamp”:”00:12:14”},{”name”: ”Funda-mentals of Learn-ing”, ”timestamp”:”00:26:28”},{”name”: ”Neural net-works”, ”timestamp”:”00:26:22”}CS3 Design inComputing,12.091stframe66.67 66.67P7r2 text {”name”: ”CS3:Design in Comput-ing”, ”timestamp”:”00:13:14”},{”name”: ”Neural net-works”, ”timestamp”:”00:30:12”},{”name”: ”NeuralNetworks - Predic-tion”, ”timestamp”:”00:37:04”}CS3 Design inComputing,12.401st sec-tion50.00 50.00154P7r3 text {”name”: ”K-NearestNeighbors”, ”times-tamp”: ”00:09:02”},{”name”: ”PCA -Training”, ”times-tamp”: ”00:15:46”},{”name”: ”Funda-mentals of Learn-ing”, ”timestamp”:”00:47:27”}Fundamentalsof learning,35.363rdvideo33.33 11.11P7r4 text {”name”: ”K-NearestNeighbors”, ”times-tamp”: ”00:24:46”},{”name”: ”Funda-mentals of Learn-ing”, ”timestamp”:”00:27:13”},{”name”: ”CS3:Design in Comput-ing”, ”timestamp”:”00:03:24”}Fundamentalsof learning,41.431stvideo100.00 14.29155P7r5 text {”name”: ”Funda-mentals of Learn-ing”, ”timestamp”:”00:43:37”},{”name”: ”K-NearestNeighbors”, ”times-tamp”: ”00:04:07”},{”name”: ”Regular-ization”, ”timestamp”:”00:16:20”}Fundamentalsof learning,42.481stframe100.00 100.00P7r6 text {”name”: ”Funda-mentals of Learn-ing”, ”timestamp”:”00:49:06”},{”name”: ”Neural net-works”, ”timestamp”:”00:27:24”},{”name”: ”K-NearestNeighbors”, ”times-tamp”: ”00:21:38”}k-nearestneighbors,6.373rdvideo75.00 50.00156P7r7 text {”name”: ”K-NearestNeighbors”, ”times-tamp”: ”00:22:11”},{”name”: ”Funda-mentals of Learn-ing”, ”timestamp”:”00:45:47”},{”name”: ”Regular-ization”, ”timestamp”:”00:25:10”}k-nearestneighbors,22.141stframe77.78 77.78P7r8 text {”name”: ”K-NearestNeighbors”, ”times-tamp”: ”00:30:04”},{”name”: ”NormalEquations”, ”times-tamp”: ”00:35:58”},{”name”: ”Funda-mentals of Learn-ing”, ”timestamp”:”00:01:40”}k-nearestneighbors,29.491stframe100.00 100.00157P7r9 text {”name”: ”Regular-ization”, ”timestamp”:”00:12:47”},{”name”: ”PCA”,”timestamp”:”00:02:33”},{”name”: ”Neural net-works”, ”timestamp”:”00:21:59”}L2 Regulariza-tion, 12.521stframe100.00 100.00P7r10 formula {”name”: ”Funda-mentals of Learn-ing”, ”timestamp”:”00:14:09”}L2 Regulariza-tion, 13.30Nomatch100.00 NAP7r11 figure {”name”: ”NormalEquations”, ”times-tamp”: ”39:40”},{”name”: ”PCA”,”timestamp”:”15:20”},{”name”: ”K-NearestNeighbors”, ”times-tamp”: ”09:00”}PCA Intuition,4.132ndvideo100.00 0.00158P7r12 text {”name”: ”PCA -Training”, ”times-tamp”: ”00:28:35”},{”name”: ”Funda-mentals of Learn-ing”, ”timestamp”:”00:28:34”},{”name”: ”Convo-lutional Neural Net-works”, ”timestamp”:”00:19:26”}PCA training,27.221stframe78.57 78.57P8r1 text Mean & Vari-ance, 9.03Nodata100.00 NAP8r2 text {”name”: ”Mean andVariance”, ”times-tamp”: ”00:14:45”},{”name”: ”PCA- Geometric ap-proach”, ”timestamp”:”00:00:07”},{”name”: ”Multi-ple Linear Regres-sion”, ”timestamp”:”00:13:01”}Mean & Vari-ance, 13.111stframe100.00 0.00159P8r3 text Non-linearoptimization,0.02Nodata100.00 NAP8r4 text {”name”: ”Non-linearoptimization”, ”times-tamp”: ”00:04:38”},{”name”: ”Multi-ple Linear Regres-sion”, ”timestamp”:”00:12:00”},{”name”: ”PCA- Eigenvector ap-proach”, ”timestamp”:”00:00:36”}Non-linearoptimization,4.301stframe100.00 100.00P8r5 text {”name”: ”Non-linearoptimization”, ”times-tamp”: ”00:11:43”},{”name”: ”k-meansclustering”, ”times-tamp”: ”00:02:50”},{”name”: ”Multi-ple Linear Regres-sion”, ”timestamp”:”00:08:03”}Non-linearoptimization,13.001stframe66.67 66.67160P8r6 formula {”name”: ”LinearRegression”, ”times-tamp”: ”00:00:00”},{”name”: ”PCA- Geometric ap-proach”, ”timestamp”:”00:08:00”},{”name”: ”PCA- Eigenvector ap-proach”, ”timestamp”:”00:00:00”}Non-linearoptimization,15.40Nomatch40.00 NAP8r7 text {”name”: ”PCA- Eigenvector ap-proach”, ”timestamp”:”00:00:20”},{”name”: ”LinearRegression”, ”times-tamp”: ”00:00:20”},{”name”: ”Multi-ple Linear Regres-sion”, ”timestamp”:”00:04:40”}Eigen vectorapproach, 0.081stframe66.67 66.67161P8r8 text {”name”: ”KNN clas-sifier”, ”timestamp”:”00:00:20”},{”name”: ”k-meansclustering”, ”times-tamp”: ”00:02:14”},{”name”: ”Mean andVariance”, ”times-tamp”: ”00:00:20”}Classification:k nearestneighbouringclassifier , 0.051stframe100.00 100.00P8r9 text {”name”: ”KNN clas-sifier”, ”timestamp”:”00:09:48”},{”name”: ”MSEand max likeli-hood”, ”timestamp”:”00:04:05”},{”name”: ”Mean andVariance”, ”times-tamp”: ”00:02:37”}Classification:k nearestneighbouringclassifier, 9.361stframe75.00 75.00162P8r10 text {”name”: ”KNN clas-sifier”, ”timestamp”:”00:13:03”},{”name”: ”Multi-ple Linear Regres-sion”, ”timestamp”:”00:09:43”},{”name”: ”PCA- Geometric ap-proach”, ”timestamp”:”00:00:20”}Classification:k nearestneighbouringclassifier, 13.021stframe100.00 100.00P8r11 text Mean & Vari-ance, 4.17Nodata100.00 NAP8r12 formula {”name”: ”Mean andVariance”, ”times-tamp”: ”00:09:49”},{”name”: ”LinearRegression”, ”times-tamp”: ”00:01:45”},{”name”: ”Multi-ple Linear Regres-sion”, ”timestamp”:”00:08:29”}Mean & Vari-ance, 6.281st sec-tion100.00 0.00163P9r1 text Correlation& Regression,0.34Nodata0.00 NAP9r2 text Correlation& Regression,0.56Nodata0.00 NAP9r3 formula {”name”: ”LinearRegression”, ”times-tamp”: ”00:04:48”}Mean & Vari-ance, 2.15Nomatch100.00 NAP9r4 formula [] Mean & Vari-ance, 4.26Nodata100.00 NAP9r5 formula [] Mean & Vari-ance, 4.44Nodata100.00 NAP9r6 formula {”name”: ”Mean andVariance”, ”times-tamp”: ”00:04:08”},{”name”: ”LinearRegression”, ”times-tamp”: ”00:04:42”}Mean & Vari-ance, 6.401st sec-tion100.00 50.00P9r7 formula {”name”: ”LinearRegression”, ”times-tamp”: ”00:04:48”}Mean & Vari-ance, 7.28Nomatch100.00 NA164P9r8 text {”name”: ”Mean andVariance”, ”times-tamp”: ”00:07:37”},{”name”: ”Multi-ple Linear Regres-sion”,”timestamp”:”00:08:24”},{”name”: ”LinearRegression”, ”times-tamp”: ”00:09:32”}Mean & Vari-ance, 8.391stframe100.00 100.00P9r9 formula [] Mean & Vari-ance, 9.15Nodata100.00 NAP9r10 formula [] Mean & Vari-ance, 11.06Nodata100.00 NAP9r11 formula {”name”: ”MSEand max likeli-hood”, ”timestamp”:”00:05:31”},{”name”: ”LinearRegression”, ”times-tamp”: ”00:04:48”}Mean & Vari-ance, 11.43Nomatch100.00 NA165P9r12 text {”name”: ”Mean andVariance”, ”times-tamp”: ”00:09:17”},{”name”: ”LinearRegression”, ”times-tamp”: ”00:12:05”},{”name”: ”PCA- Eigenvector ap-proach”, ”timestamp”:”00:05:47”}Mean & Vari-ance, 13.111st sec-tion100.00 75.00P9r13 formula {”name”: ”LinearRegression”, ”times-tamp”: ”00:10:30”},{”name”: ”MSEand max likeli-hood”, ”timestamp”:”00:05:31”}Mean andVariance,14.28Nomatch100.00 NAP9r14 formula [] Mean & Vari-ance, 16.15Nodata100.00 NA166P9r15 text {”name”: ”Mean andVariance”, ”times-tamp”: ”00:17:15”},{”name”: ”PCA- Geometric ap-proach”, ”timestamp”:”00:11:52”},{”name”: ”PCA- Eigenvector ap-proach”, ”timestamp”:”00:22:23”}Mean & Vari-ance, 18.251stframe100.00 100.00P9r16 formula [] Mean & Vari-ance, 19.12Nodata100.00 NAP9r17 text {”name”: ”LinearRegression”, ”times-tamp”: ”00:00:20”},{”name”: ”Multi-ple Linear Regres-sion”, ”timestamp”:”00:00:20”},{”name”: ”PCA- Eigenvector ap-proach”, ”timestamp”:”00:08:33”}Linear Regres-sion, 0.081stframe100.00 100.00167P9r18 formula [] Linear Regres-sion, 0.56Nodata100.00 NAP9r19 formula [] Linear Regres-sion, 2.20Nodata100.00 NAP9r20 formula {”name”: ”Mean andVariance”, ”times-tamp”: ”00:04:07”},{”name”: ”PCA- Geometric ap-proach”, ”timestamp”:”00:00:15”}Linear Regres-sion, 4.19Nomatch100.00 NAP9r21 formula {”name”: ”Mean andVariance”, ”times-tamp”: ”00:04:07”},{”name”: ”LinearRegression”, ”times-tamp”: ”00:04:48”}Linear Regres-sion, 5.052ndframe100.00 100.00168P9r22 text {”name”: ”PCA- Geometric ap-proach”, ”timestamp”:”00:00:20”},{”name”: ”KNN clas-sifier”, ”timestamp”:”00:16:33”},{”name”: ”PCA- Eigenvector ap-proach”, ”timestamp”:”00:05:55”}Geometric ap-proach, 00:271stframe50.00 50.00P9r23 text {”name”: ”LinearRegression”, ”times-tamp”: ”00:00:20”},{”name”: ”MSEand max likeli-hood”, ”timestamp”:”00:04:05”},{”name”: ”PCA- Geometric ap-proach”, ”timestamp”:”00:00:07”}Geometric ap-proach, 6.413rdvideo100.00 42.86169P9r24 text PCAap-pliedtorealdata00:22Nodata100.00 NAP9r25 text PCAap-pliedtorealdata00:26Nodata100.00 NAP9r26 text {”name”: ”PCA onreal data”, ”times-tamp”: ”00:01:40”},{”name”: ”PCA- Geometric ap-proach”, ”timestamp”:”00:00:07”},{”name”: ”Multi-ple Linear Regres-sion”, ”timestamp”:”00:10:10”}PCAap-pliedtorealdata00:541stframe100.00 100.00170P9r27 figure {”name”: ”MSEand max likeli-hood”, ”timestamp”:”02:20”},{”name”: ”Non-linearoptimization”, ”times-tamp”: ”18:20”},{”name”: ”k-meansclustering”, ”times-tamp”: ”05:00”}RotatedPCA00:17Nomatch100.00 NAP9r28 figure {”name”: ”RotatedPCA”, ”timestamp”:”00:02”},{”name”: ”MSEand max likeli-hood”, ”timestamp”:”02:40”},{”name”: ”Non-linearoptimization”, ”times-tamp”: ”13:20”},RotatedPCA00:291stframe100.00 100.00171P9r29 figure {”name”: ”MSEand max likeli-hood”, ”timestamp”:”01:00”},{”name”: ”Non-linearoptimization”, ”times-tamp”: ”06:20”},{”name”: ”k-meansclustering”, ”times-tamp”: ”05:00”}RotatedPCA00:41Nomatch100.00 NAP9r30 figure {”name”: ”MSEand max likeli-hood”, ”timestamp”:”05:00”},{”name”: ”Non-linearoptimization”, ”times-tamp”: ”06:20”},{”name”: ”KNN clas-sifier”, ”timestamp”:”01:00”}RotatedPCA2.44Nomatch100.00 NA172P9r31 text {”name”: ”RotatedPCA”, ”timestamp”:”00:04:09”},{”name”: ”k-meansclustering”, ”times-tamp”: ”00:08:26”},{”name”: ”PCA- Geometric ap-proach”, ”timestamp”:”00:06:29”}RotatedPCA3.251stframe100.00 100.00P10r1 formula {”name”: ”LinearRegression”, ”times-tamp”: ”00:04:48”}MeanandVari-ance,0.01Nomatch100.00 NAP10r2 formula [] MeanandVari-ance,4.17Nodata100.00 NAP10r3 formula [] MeanandVari-ance,4.17Nodata100.00 NA173P10r4 formula {”name”: ”ComplexData”, ”timestamp”:”00:00:48”},{”name”: ”LinearRegression”, ”times-tamp”: ”00:04:48”}MeanandVari-ance,15.51Nomatch100.00 NAP10r5 formula [] MLP,0.02Nodata100.00 NAP10r6 text MLP,8.06Nodata100.00 NAP10r7 text {”name”: ”PCA- Geometric ap-proach”, ”timestamp”:”00:00:10”},{”name”: ”PCA- Eigenvector ap-proach”, ”timestamp”:”00:05:55”},{”name”: ”KNN clas-sifier”, ”timestamp”:”00:16:33”}PCA–Ge-o-met-ricap-proach0:011stframe100.00 100.00174P10r8 text {”name”: ”PCA- Eigenvector ap-proach”, ”timestamp”:”00:00:10”},{”name”: ”Non-linearoptimization”, ”times-tamp”: ”00:00:10”},{”name”: ”LinearRegression”, ”times-tamp”: ”00:00:10”}PCA–Eigenvec-torap-proach0:081stframe100.00 100.00P10r9 formula [] PCA–Eigenvec-torap-proach0:08Nodata100.00 NAP10r10 formula {”name”: ”LinearRegression”, ”times-tamp”: ”00:04:48”},{”name”: ”ComplexData”, ”timestamp”:”00:00:48”}ComplexData7:452ndvideo100.00 0.00175P10r11 formula [] ComplexData9:17Nodata100.00 NAP10r12 text Scaling9:35Nodata100.00 NAP10r13 text {”name”: ”MSEand max likeli-hood”, ”timestamp”:”00:03:28”},{”name”: ”KNN clas-sifier”, ”timestamp”:”00:03:20”},{”name”: ”Multi-ple Linear Regres-sion”, ”timestamp”:”00:10:10”}KNNclas-si-fier2:282nd frame 50.00 50.00176P10r14 text {”name”: ”KNN clas-sifier”, ”timestamp”:”00:13:03”},{”name”: ”Multi-ple Linear Regres-sion”, ”timestamp”:”00:08:24”},{”name”: ”PCA- Eigenvector ap-proach”, ”timestamp”:”00:15:28”}KNNclas-si-fier4.361st video 50.00 100.00B.2 Friedman & Wilcoxon test resultsThe friedman and wilcoxon test results for the preference levels in temporal conditionsand video surrogates is shown in the figures B.1, B.2, B.3, B.4, B.5, and B.6.177Figure B.1: Temporal preference: a) It is easy to find the information I need fromthis point178Figure B.2: Temporal preference: b) The information is effective in helping mecomplete the tasks and scenarios for learning179Figure B.3: Temporal preference: c) I’m satisfied with the retrieved video time point180Figure B.4: Temporal preference: a) It is easy to find the information I need fromthis point181Figure B.5: Preference for Video Surrogates: b) This design has all the functionsand capabilities I expect it to have182Figure B.6: Preference for Video Surrogates: c) The organization of information onthe layout screen is clear183B.3 Interview TranscriptsThe interview transcripts of the user study with 12 participants can be accessed here1.B.4 Survey resultsThis section shows the survey data for the preference of temporal level and video sur-rogates in the retrieved video results as can be seen in the tables B.2, B.3. Later, theusability data in terms of Effectiveness, Efficiency and Satisfaction is also covered in thesection in the tables. The results are derived from the 12-student experimental studydiscussed in chapter 4. The likert-scale data in all the results range from 1-Stronglydisagree to 5-Strongly agree.1https://ubcca-my.sharepoint.com/:f:/g/personal/ranjs92 student ubc ca/EtvgSRjJKktPqKHcXUxUauIBynZU9R26ePuSuwC-5YvZpA?e=Zl0y1u184Table B.2: Likert-scale data for temporal preference in Point, Interval and wholevideoPoint Interval VideoParticipant Name a b c a b c a b cP1 Fan 5 5 5 4 4 5 4 4 3P2 Qian 3 4 4 3 3 3 3 3 3P3 Rui 5 5 5 5 5 5 3 4 4P4 Mazoud 3 4 3 2 2 2 4 4 4P5 Praneeth 5 5 5 5 5 5 5 5 4P6 Taslim 4 5 4 4 4 4 3 4 3P7 Pramit 5 5 3 4 5 4 3 3 2P8 Deb 5 3 2 5 5 5 5 4 4P9 Achinth 5 5 3 5 5 4 4 5 2P10 Shaunak 5 4 5 4 4 3 4 4 2P11 Munira 5 5 5 3 3 4 1 2 2P12 Lakshmi 4 5 3 5 5 5 4 4 3• a - It is easy to find the information I need from this point• b - The information is effective in helping me complete the tasks and scenarios forlearning• c - I’m satisfied with the retrieved video time point185Table B.3: Likert-scale data for video surrogate preferenceTitle +ThumbnailTitle +Thumbnail+KeywordsTitle +Thumbnail+Keywords+SummaryParticipant Name a b c a b c a b cP1 Fan 4 5 5 4 4 3 3 3 2P2 Qian 4 4 4 4 4 4 3 3 4P3 Rui 4 4 5 5 5 5 4 4 3P4 Mazoud 4 4 4 5 5 5 3 3 3P5 Praneeth 5 5 4 5 5 5 5 5 4P6 Taslim 4 3 4 3 3 4 4 3 3P7 Pramit 4 3 4 4 5 5 5 5 5P8 Deb 3 4 5 5 4 5 5 5 5P9 Achinth 2 3 5 4 4 4 5 5 2P10 Shaunak 4 5 5 4 4 2 3 3 1P11 Munira 2 3 5 5 5 5 5 5 4P12 Lakshmi 1 2 2 4 4 4 3 3 3• a - It is easy to find the information I need from this point• b - This design has all the functions and capabilities I expect it to have.• c - The organization of information on the layout screen is clear186Table B.4: Likert-scale survey results for Effectiveness in Usability EvaluationEffectivenessEasy touseFunctionswell inte-gratedNeedtechnicalsupport5 4 25 5 14 4 14 4 34 4 14 5 14 3 45 4 24 4 15 4 14 3 25 4 1187Table B.5: Likert-scale survey results for Efficiency in Usability EvaluationEfficiencyQuick pro-ducitivityQuickLearningLot tolearn5 4 35 5 13 5 24 4 15 3 45 5 14 4 44 4 24 4 15 5 24 4 24 5 1Table B.6: Likert-scale survey results for Satisfaction in Usability EvaluationSatisfactionFrequentuseOverallsatisfac-tionComplexity5 4 25 5 54 4 23 4 24 5 25 5 14 4 25 4 24 4 24 5 24 4 25 4 2188APPENDIX CImplementation DetailsThe implementation details of the note recognition of each type that is text, formula,and figure and the mobile application interface design can be accessed here 1.1https://learning.github.ubc.ca/ranjs92/NoteLink.git189
Thesis/Dissertation
eng
Electrical and Computer Engineering
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Leveraging students' handwritten notes to link to watched instructional videos
Text