Science, Irving K. Barber Faculty of (Okanagan)
Computer Science, Mathematics, Physics and Statistics, Department of (Okanagan)
DSpace
UBCO
Foss, Sarah
Master of Science - MSc
University of British Columbia
Interactive question systems improve student engagement and provide opportunities for increased practice and skill mastery. Developing database design diagrams is a key skill for database courses, but providing evaluation feedback is time-consuming for instructors and accurate auto-grading is challenging due to the variability of student answers especially when labeling diagram components. This work presents a system for the automatic creation and real-time evaluation of database design questions using Unified Modeling Language (UML) diagrams. Students directly interact with the question text, and the system continuously generates a visual representation of their answer as well as provides immediate feedback at any time. By utilizing a web-based, customizable user interface, the system supports precise marking and the ability to practice variants of design questions to mastery. Classroom evaluations demonstrate high student satisfaction compared to traditional UML design questions and preference for using the software to improve their learning outcomes. Experiments also measured how submission limits, such as maximum submission limits and applying regression penalties, significantly improve student behavior.
AutoER: A System for the AutomaticGeneration and Evaluation of UMLDatabase Design DiagramsbySarah FossBCIS, Okanagan College, 2019A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFMASTER OF SCIENCEinThe College of Graduate Studies(Computer Science)THE UNIVERSITY OF BRITISH COLUMBIA(Okanagan)October 2022© Sarah Foss, 2022The following individuals certify that they have read, and recommendto the College of Graduate Studies for acceptance, a thesis/dissertation en-titled:AutoER: A System for the Automatic Generation andEvaluation of UML Database Design Diagramssubmitted by Sarah Foss in partial fulfilment of the requirements of thedegree of Master of ScienceDr. Ramon Lawrence, I. K. Barber Faculty of ScienceSupervisorDr. Patricia Lasserre, I. K. Barber Faculty of ScienceSupervisory Committee MemberDr. Bowen Hui, I. K. Barber Faculty of ScienceSupervisory Committee MemberDr. Stephen W. McNeil, I. K. Barber Faculty of ScienceUniversity ExamineriiAbstractInteractive question systems improve student engagement and provideopportunities for increased practice and skill mastery. Developing databasedesign diagrams is a key skill for database courses, but providing evalua-tion feedback is time-consuming for instructors and accurate auto-gradingis challenging due to the variability of student answers especially when la-beling diagram components. This work presents a system for the automaticcreation and real-time evaluation of database design questions using UnifiedModeling Language (UML) diagrams. Students directly interact with thequestion text, and the system continuously generates a visual representationof their answer as well as provides immediate feedback at any time. By uti-lizing a web-based, customizable user interface, the system supports precisemarking and the ability to practice variants of design questions to mastery.Classroom evaluations demonstrate high student satisfaction compared totraditional UML design questions and preference for using the software toimprove their learning outcomes. Experiments also measured how submis-sion limits, such as maximum submission limits and applying regressionpenalties, significantly improve student behavior.iiiLay SummaryThis thesis proposes and implements a new system, AutoER, for the cre-ation and automatic evaluation of database design diagrams. This systemallows students to build visual representations of diagrams by directly in-teracting with question text. Students can, at any time, request automatedfeedback to check their work. Instructors can choose to load the systemwith pre-written questions, or they can have the system auto-generate ran-dom questions. The ability to generate questions allows students to practicequestions until mastery and also reduces academic dishonesty by generat-ing unique questions for each student during examinations. Surveys fromclassroom evaluations revealed high student satisfaction and that studentsprefer to use the software over traditional methods. To limit over-relianceon the auto-grading feature, limitations on the number of student attemptsper question were explored and measured in the classroom studies.ivPrefaceThe study in this thesis was conducted with the approval of the UBCOkanagan Behavioural Research Ethics Board (BREB) under the certificatenumber H21-01600.Parts of this thesis were published in the research-article titled Auto-matic Generation and Marking of UML Database Design Diagrams in theProceedings of the 53rd ACM Technical Symposium on Computer ScienceEducation V. 1, Pages 626–632, in February 2022.vTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . viList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xGlossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . xivDedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvChapter 1: Introduction . . . . . . . . . . . . . . . . . . . . . . . 11.1 Thesis Contributions . . . . . . . . . . . . . . . . . . . . . . . 11.2 Thesis Organization . . . . . . . . . . . . . . . . . . . . . . . 3Chapter 2: Background . . . . . . . . . . . . . . . . . . . . . . . 42.1 Database Design and Diagram Creation . . . . . . . . . . . . 42.1.1 Simplified Educational Database Design Tools . . . . 42.2 Automated Evaluation . . . . . . . . . . . . . . . . . . . . . . 72.2.1 Pattern Matching . . . . . . . . . . . . . . . . . . . . . 72.2.2 Machine Learning . . . . . . . . . . . . . . . . . . . . 72.2.3 String Comparison . . . . . . . . . . . . . . . . . . . . 82.3 Generating Questions . . . . . . . . . . . . . . . . . . . . . . 102.4 Limitations on Submissions . . . . . . . . . . . . . . . . . . . 102.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12viTABLE OF CONTENTSChapter 3: AutoER System Development and Design . . . . . 133.1 System Overview . . . . . . . . . . . . . . . . . . . . . . . . . 133.2 System Architecture . . . . . . . . . . . . . . . . . . . . . . . 143.2.1 Student Frontend . . . . . . . . . . . . . . . . . . . . . 153.2.2 Backend . . . . . . . . . . . . . . . . . . . . . . . . . . 153.2.3 Database . . . . . . . . . . . . . . . . . . . . . . . . . 173.2.4 Reverse Proxy . . . . . . . . . . . . . . . . . . . . . . 173.3 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193.3.1 Login . . . . . . . . . . . . . . . . . . . . . . . . . . . 193.3.2 Question Interface . . . . . . . . . . . . . . . . . . . . 193.3.3 Submission Policies . . . . . . . . . . . . . . . . . . . . 213.3.4 Student Mark Retrieval . . . . . . . . . . . . . . . . . 233.4 AutoER Question Creation . . . . . . . . . . . . . . . . . . . 233.4.1 Question Markup . . . . . . . . . . . . . . . . . . . . . 273.4.2 Answer Format . . . . . . . . . . . . . . . . . . . . . . 283.4.3 User Interface Template . . . . . . . . . . . . . . . . . 303.4.4 Automatic Question Evaluation . . . . . . . . . . . . . 303.4.5 Automatic Question Generation . . . . . . . . . . . . 31Chapter 4: Case Studies: Undergraduate Database Course . 344.1 Participant Recruitment . . . . . . . . . . . . . . . . . . . . . 344.2 System Use Data Collection . . . . . . . . . . . . . . . . . . . 344.2.1 Assignments . . . . . . . . . . . . . . . . . . . . . . . 364.2.2 Midterm Exam . . . . . . . . . . . . . . . . . . . . . . 384.2.3 Final Exam . . . . . . . . . . . . . . . . . . . . . . . . 384.3 Student Survey . . . . . . . . . . . . . . . . . . . . . . . . . . 394.4 Course Administrator Survey . . . . . . . . . . . . . . . . . . 40Chapter 5: Results . . . . . . . . . . . . . . . . . . . . . . . . . . 425.1 System Evaluation . . . . . . . . . . . . . . . . . . . . . . . . 425.1.1 System Use . . . . . . . . . . . . . . . . . . . . . . . . 425.1.2 System Usability . . . . . . . . . . . . . . . . . . . . . 445.1.3 Free-form Survey Results . . . . . . . . . . . . . . . . 445.1.4 Administrator Evaluation . . . . . . . . . . . . . . . . 475.2 Student Performance Evaluation . . . . . . . . . . . . . . . . 475.2.1 Assignment Performance . . . . . . . . . . . . . . . . . 475.2.2 Midterm Generated Question Performance . . . . . . 485.2.3 Final Exam Performance . . . . . . . . . . . . . . . . 485.2.4 Summer 2021 Study . . . . . . . . . . . . . . . . . . . 495.2.5 Fall 2021 Study . . . . . . . . . . . . . . . . . . . . . . 52viiTABLE OF CONTENTS5.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61Chapter 6: Conclusion . . . . . . . . . . . . . . . . . . . . . . . . 666.1 Limitations and Threats to Validity . . . . . . . . . . . . . . 686.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . 69Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71Appendices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77Appendix A: Consent Forms . . . . . . . . . . . . . . . . . . . . . . 78A.1 Administrator Consent . . . . . . . . . . . . . . . . . . . . . . 78A.2 Student Consent . . . . . . . . . . . . . . . . . . . . . . . . . 80Appendix B: Surveys . . . . . . . . . . . . . . . . . . . . . . . . . . 82B.1 Student Survey . . . . . . . . . . . . . . . . . . . . . . . . . . 82B.2 Administrator Survey . . . . . . . . . . . . . . . . . . . . . . 84Appendix C: Assignments . . . . . . . . . . . . . . . . . . . . . . . 86C.1 Assignment 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . 86C.1.1 Question 1 (10 marks) . . . . . . . . . . . . . . . . . . 87C.1.2 Question 2 (10 marks) . . . . . . . . . . . . . . . . . . 88C.2 Assignment 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . 89C.2.1 Question 1 (15 marks) . . . . . . . . . . . . . . . . . . 89C.2.2 Question 2 (35 marks) - Project Deliverable . . . . . . 91viiiList of TablesTable 3.1 AutoER Question Components . . . . . . . . . . . . . 25Table 5.1 Description of Common Metrics in AutoER Evaluations 43Table 5.2 AutoER Usage Metrics . . . . . . . . . . . . . . . . . . 44Table 5.3 Assignment Upload Metrics . . . . . . . . . . . . . . . 44Table 5.4 System Usability Scale Survey Results . . . . . . . . . 45Table 5.5 AutoER Performance Metrics . . . . . . . . . . . . . . 47Table 5.6 Student Profile Categories . . . . . . . . . . . . . . . . 50Table 5.7 Summer Student Profile Metrics . . . . . . . . . . . . 52Table 5.8 Limiting Attempts Change . . . . . . . . . . . . . . . . 53Table 5.9 Comparison of Profiles Across Studies . . . . . . . . . 54Table 5.10 User Count and Grades with One Submission . . . . . 55Table 5.11 Performance Comparison of Submission Policies . . . . 56Table 5.12 Limiting Attempts Across Student Profiles . . . . . . . 58Table 5.13 Limiting Attempts: Student Preference and Performance 61ixList of FiguresFigure 2.1 Database Design Text and UML Class Diagram . . . 5Figure 3.1 AutoER System Architecture . . . . . . . . . . . . . . 14Figure 3.2 Administration Interface Question Creation . . . . . . 17Figure 3.3 AutoER Design Diagram . . . . . . . . . . . . . . . . 18Figure 3.4 AutoER Login Screen . . . . . . . . . . . . . . . . . . 19Figure 3.5 AutoER User Interface . . . . . . . . . . . . . . . . . 20Figure 3.6 Adding a Key to an Attribute . . . . . . . . . . . . . 20Figure 3.7 Editing a Relationship between Entities . . . . . . . . 21Figure 3.8 Example Student Feedback . . . . . . . . . . . . . . . 22Figure 3.9 Evaluation Details of a Student’s Submitted Answer . 24Figure 3.10 Question Base . . . . . . . . . . . . . . . . . . . . . . 27Figure 3.11 Question Markup . . . . . . . . . . . . . . . . . . . . 27Figure 3.12 Original Answer . . . . . . . . . . . . . . . . . . . . . 29Figure 3.13 String Representation of the Answer . . . . . . . . . . 29Figure 3.14 Check Mark Button . . . . . . . . . . . . . . . . . . . 30Figure 3.15 Generated Question Example with Letter Placeholders 33Figure 4.1 Assignment 1 Question 1 . . . . . . . . . . . . . . . . 37Figure 4.2 Final Exam Question and Answer . . . . . . . . . . . 41Figure 5.1 Student Performance Grouped By Total Submissions 50Figure 5.2 Example Student Performance Profiles . . . . . . . . 51Figure 5.3 Example Fall Student Performance Profiles . . . . . . 55Figure 5.4 Midterm Exam Students by Category and SubmissionPolicy . . . . . . . . . . . . . . . . . . . . . . . . . . . 57Figure 5.5 Final Exam Students by Category and SubmissionPolicy . . . . . . . . . . . . . . . . . . . . . . . . . . . 57Figure 5.6 Midterm Exam Students by Category and SubmissionPolicy . . . . . . . . . . . . . . . . . . . . . . . . . . . 59xLIST OF FIGURESFigure 5.7 Final Exam Students by Category and SubmissionPolicy . . . . . . . . . . . . . . . . . . . . . . . . . . . 60Figure C.1 Assignment 1 Question 1 . . . . . . . . . . . . . . . . 87Figure C.2 Assignment 1 Question 2 . . . . . . . . . . . . . . . . 88Figure C.3 Assignment 2 Question 1 . . . . . . . . . . . . . . . . 90Figure C.4 Assignment 2 Question 2 . . . . . . . . . . . . . . . . 92xiGlossaryApplication Programming Interface (API) A software intermediary with aset of definitions and protocols to allow applications to interact. 15–17,30Data Definition Language (DDL) The syntax for creating and modifyingdatabase objects. 91Entity Relationship (ER) A database design model that describes how en-tities relate to each other. 3, 4, 6, 7, 9, 30, 43, 47, 66, 86, 91Hypertext Markup Language (HTML) A markup language used to struc-ture web documents. 14, 30Internet Protocol (IP) The set of rules for routing and addressing packetsof data. 35JavaScript Object Notation (JSON) A text-based format for representingstructured data. 15, 16Learning Management System (LMS) A software application designed forthe distribution, management, and automation of educational materi-als. 14, 26Massive Online Open Course (MOOC) An open online course designed forlarge numbers of geographically dispersed students. 1Representational state transfer (REST) An architectural style defining stan-dards used across web-based applications. 15Structured Query Language (SQL) A programming language used to ma-nipulate and manage databases. 34, 37, 89, 91xiiSystem Usability Scale (SUS) A standardized survey tool used for measur-ing system usability. 39, 44, 62, 68teaching assistant (TA) An individual that assists an instructor with in-structional activities. 1, 23, 40, 48, 63, 67Unified Modeling Language (UML) A modeling language used to expressand design software systems. iii, 1, 4, 6–9, 12, 13, 23, 25, 34, 36, 48,66–69, 86, 91Uniform Resource Locator (URL) A web address. 15–17, 19, 25, 26, 32, 67xiiiAcknowledgementsI would like to express my deepest gratitude to Dr. Ramon Lawrence. Ithas been a privilege to work with and learn from such a dedicated educatorand researcher. His passion for teaching and commitment to his students’success is inspiring, and I hope to bring it with me and emulate it in my ownacademic career. Without his wealth of experience, guidance, and patiencethis work would not have been possible.Special thanks to Tatiana Urazova. Her invaluable assistance and dedica-tion helped ensure the success of this work. I would also like to acknowledgeDr. Youry Khmelevsky for his continued support of the project.Thank you to my supervisory committee, Dr. Bowen Hui and Dr. Pa-tricia Lasserre, for reviewing my work and helping me ensure that it is thehighest quality possible.This endeavor would not have been possible without the ongoing supportof the faculty and administration at Okanagan College, and in particular,the support of my colleagues Ken Chidlow, Dr. Jim Nastos, Dr. ScottFazakerley, and Alan Kennedy.Finally, I could not have undertaken this journey without the support ofmy family. Thank you to my amazing husband, Jeramie, and my two incred-ible children, Gabrielle and Gavin. Their unending patience and unwaveringsupport mean the world to me.xivDedicationFor Anne and Clair Foss.I’m so incredibly fortunate to have had your love and unconditional supportin my life. I know you would be proud.xvChapter 1IntroductionClassrooms can now take many different formats ranging from traditionalon-campus lecture halls to online learning platforms offering Massive OnlineOpen Courses (MOOCs). Many courses, regardless of the format, can havehundreds of students, creating time-consuming workloads for course admin-istrators. These large class sizes can also present issues with precise markingas it is difficult for humans to grade a large quantity of assignments or ex-ams consistently, particularly when the marking workload is divided acrossmultiple instructors or teaching assistants (TAs).Automated learning tools can help alleviate these issues by providingconsistent, instant feedback to students. These tools can further benefit stu-dents by providing opportunities for increased practice with learned conceptsand skill mastery. Several systems support the generation and auto-markingof questions [WHZ15, EP08, BE15] for practice and improving learning out-comes.Database design and modeling are key concepts taught in database andsoftware engineering courses. Design exercises are often challenging for stu-dents and additional practice helps learning. This work focuses on questionswhere the goal is to produce a design diagram for a database that capturesthe requirements discussed in a written text. Evaluating design diagramsis complex due to the lack of testing frameworks, diversity of correct an-swers, and interpreting the semantics of the diagram and its concepts. Priorwork examined automatic evaluation of design diagrams using image pro-cessing [TWS06], syntactic, structural, and semantic analysis [BAK19], andmachine learning [BMM20]. These systems provide evaluations similar tohuman markers for instructor developed questions.1.1 Thesis ContributionsThe contribution of this thesis is a question generation and evaluationsystem for Unified Modeling Language (UML) database design diagramsand its evaluation in a database course. The system, called AutoER, sup-ports both instructor-created and automatically generated questions with11.1. THESIS CONTRIBUTIONSinstructor configurable marking. Automatically generating question vari-ants allows for students to practice questions until mastery and is beneficialfor online exams where the potential for student collaboration and academicdishonesty exists. AutoER provides flexibility for instructors to change allaspects of its behavior including the user interface, marking and feedback,question generation, and operation as a stand-alone system or integratedwith a learning management system.A unique aspect of the user interface is that the student builds the designdiagram by interacting with the question text directly and requesting thesystem add components to the diagram rather than drawing them manu-ally. This improves the speed of diagram construction and helps beginnerslearn fundamental design challenges. Consistent naming also enables precisemarking and matching with correct solutions.The system was evaluated in two offerings of an undergraduate databasecourse at the Okanagan campus of the University of British Columbia. Theresearch questions were:− RQ1: Do students prefer to use the AutoER system compared topaper and other design software?− RQ2: Is the AutoER system efficient and usable for students?− RQ3: Does automatic grading increase student performance on databasedesign assignments?− RQ4: Is random generation of questions effective for use in examevaluations?− RQ5: Does limiting submissions to the system impact student sub-mission behaviour in a positive way?− RQ6: Which of the submission policies limiting student submissionsis most effective?− RQ7: What submission policy is preferred by students?To address these questions, students were given an opportunity to usethe system for assignments, examinations, and additional practice at theirown discretion. Extensive data was collected regarding the system usageand student performance. The students were surveyed after several weeksof usage to gain insight into the perceived usability and effectiveness of thesystem.21.2. THESIS ORGANIZATION1.2 Thesis OrganizationThis thesis examines the development and evaluation of an interactivepedagogical tool for the generation and evaluation of Entity Relationship(ER) diagrams. Chapter 2 presents a background on the database designproblem and a discussion of existing systems for diagram creation and auto-matic evaluation. The AutoER system architecture and system features aredescribed in Chapter 3. Chapters 4 and 5 detail the experimental setup andresults using the system in an undergraduate database course. Finally, theconclusion of the thesis and potential future work are discussed in Chapter6.3Chapter 2Background2.1 Database Design and Diagram CreationA fundamental learning objective in an undergraduate database course isthe ability to design a database for a particular problem domain. Databasedesign is a challenging problem for students as they must learn the particulardiagram syntax and be able to translate the written text into the appropriatediagram elements.Database diagrams are constructed using either ER notation [Che76] orUML [Gro17]. UML class diagrams represent database entities as classes,relationships as associations, and attributes may be associated with entitiesand relationships. To manually construct a database design problem, an in-structor would describe a database schema using plain language and wouldalso construct one or more solution diagrams. A set of grading criteria wouldbe defined to standardize the evaluation of answers. Student’s diagramswould then manually graded by visual comparison between the student’sanswers and the solution diagrams. Evaluation and providing feedback istime-consuming as students may not use the syntax consistently, misinter-pret the question, use inconsistent naming and labels, and not understandfundamental modeling constraints.Figure 2.1 contains a database design question text and the answer as aUML class diagram.2.1.1 Simplified Educational Database Design ToolsGiven a particular database design question, an instructor allows a stu-dent to produce the diagram on paper or using UML database design soft-ware. Surveys of modeling software for instruction [ALS19, AL17] indicatepreference for easy-to-use systems to allow students to focus on learning theconcepts rather than the software. Other important aspects in preferredsystems are that they are perceived to give good feedback, are simple to in-stall, and that they should be low cost or free to use. Commercial productsused by experts are often costly, large programs that contain more features42.1. DATABASE DESIGN AND DIAGRAM CREATION(a) Question Text(b) Example UML Class DiagramFigure 2.1: Database Design Text and UML Class Diagram©Sarah Foss (2022)52.1. DATABASE DESIGN AND DIAGRAM CREATIONand complexity than required for educational use.Several low-cost educational systems have been developed to supportthe construction and evaluation of database designs in ER or UML classdiagrams.QuickUML [AV03] was developed in response to existing systems thatprovide too much UML notation to beginner students, over-complicatingthe learning process. This system simplifies beginner level learning by onlyproviding basic UML functions to allow students to begin to apply newlylearned concepts without being slowed down with more advanced aspects ofdomain modeling. This tool was intentionally kept small and lightweight toallow for easy installation on a number of systems. Although this system islightweight and easy to use, feedback to students is not a system feature.The tool MinimUML [TPE05] takes a similar approach and only providesa subset of UML notation with a focus on high-level abstract design. Thedevelopers of this tool noted that some existing simple tools were lackingsome of the desirable features that more complex systems have and createda system that supports features like printing and undo/redo. The systemalso supports annotations that allows instructors to provide feedback, butautomated feedback is not supported.DiagrammER [GM20] is another simplified educational tool, however, itis a web-based application which allows users to avoid installation of thetool altogether. This system allows beginner computer science students tocreate and interact with ER diagrams by dragging and dropping elements inthe diagram creation area. A novel feature of the system allows instructorsto upload example questions with the corresponding correct ER diagram forstudents to retrieve and view. Unfortunately, automated feedback is nota feature that is implemented in the system; however, it is planned futurework.EERTutor [Zak04] is a tutoring system for ER design that provides inter-active diagram construction and tutorials to help students during diagramcreation. When an error is made, the toolbar and editing area are disabledand a dialogue with common-sense feedback is presented to the student.There are multiple levels of feedback available. The first level of feedbacksimply informs the student whether the answer is correct. Students can thenchoose the option for a “hint” or a “detailed hint” for more detailed infor-mation about the error and guidance to improve their answer. If the studentis still unable to answer the question, they can request to view the solutiondiagram. These prompts help the students identify and correct errors whenthey may not have enough experience or knowledge to do so on their own.This level of in-depth feedback is beneficial for formative assessments, but62.2. AUTOMATED EVALUATIONmay not be appropriate for summative evaluation.2.2 Automated EvaluationProviding automatic feedback and evaluation for modeling and designtasks has received increased attention in recent research. Similar to grad-ing and evaluation systems for questions on programming [Lee21, MNS+20,EP08, MSR21], providing real-time feedback to students allows for formativelearning, increased engagement, and consistency and efficiency in evaluation.Automatic evaluation of design diagrams must match student solutions withone or more instructor solutions while handling issues with naming, struc-tural, and semantic differences.There are several approaches for automatic grading and feedback fordiagrams.2.2.1 Pattern MatchingMarking of student drawn ER diagrams and other diagram types us-ing image processing techniques and recognizing patterns that match withanswers was discussed in [TWS06, STW13]. This approach does not re-quire systems that enforce precise diagrams, meaning diagrams where thenames of diagram components are pre-defined. To perform the evaluation,the student diagram and the answer diagram are compared for similarityby identifying components that convey specific meaning in that domain andcomparing them to the matching components in the solution. Similarity be-tween entities is measured by the weighted sum of their label identifiers andattributes and similarity between relationships is measured by the weightedsum of the labels of objects they relate and their cardinalities. The resultis used to generate a grade and feedback for the student. Thresholds mustbe defined to determine whether two components are sufficiently similar tobe considered a correct match. Challenges of using pattern matching withimprecise student diagrams include accurately grading images due to miss-ing or extra information, and inconsistent terminology including synonyms,abbreviations and misspelling, and determining optimal thresholds for sim-ilarity.2.2.2 Machine LearningIn [SvdPSC19], machine learning was used to grade UML class diagrams.In their approach, multiple classification models and a regression model72.2. AUTOMATED EVALUATIONwere trained with extracted features from student answer files from a singleassignment along with associated grades from expert markers. Experimentswere done using both a 10-point scale (with grades ranging from 1-10) anda 5-point scale. Results showed that accuracy was not sufficient with a topaccuracy of 42% with the 10-point scale and 46% with the 5-point scale.Machine learning automatic assessment in [BMM20] produced gradingby letter grade. This research compared five common machine learningmodels trained with student answers along with grades produced from aheuristic string parsing grading algorithm. The output was then comparedto manual instructor-graded solutions. Results showed that both LogisticRegression and Naive Bayes models were accurate within a letter grade tothe baseline. However, only letter grades were provided, and there wasno feedback for improvement to students. Another big drawback was thatthe dataset had to be extensively manually cleaned to correct misspelling,missing syntax, and duplicate entries in order to be used with the simpleheuristic algorithm.2.2.3 String ComparisonMost commonly, student and instructor diagrams are compared using astring representation of the diagram contents.UMLGrader [Has11], which is based on UMLint [HR11], a tool used toidentify defects in UML diagrams, is a web-based application that evaluatesUML class diagrams developed using IBM Rational Rose. Each diagramin Rational Rose has an associated model file that describes the diagramin text. Using the model files from a solution diagram and a student dia-gram, the text is compared using string comparison. For the comparison,spaces and underscores are stripped and capitalization is ignored. Entitynames require exact matches, and attributes and associations require par-tial matches with the associated names from the solution. Because studentsmanually enter component names for the diagrams into the system, theyare required to name their diagram precisely otherwise it may be marked asincorrect. The system does flag common errors like misspelling and invalidmultiplicities using UMLint, however, the tight constraints of the systemalong with manual name entry pose some challenges to the auto-evaluationsystem.[SBP+10] presents a similar UML database design tool allowing studentsto create diagrams with string comparison auto-marking. This tool recordsall of the parameters of an answer into a file and compares it directly withthe solution file. In this system, specific attribute names, denoted with82.2. AUTOMATED EVALUATIONparenthesis, must appear in the problem description to be valid for use in theproblem. This addresses one of the challenges present in UMLGrader, butalso unfortunately eliminates the exercise of the identification of attributesfor the student. The constraint of manually entering exact names of entityclasses is still present in this system.Another system using string comparison for automatic evaluation isMonstER park [Sch20]. This tool uses gamification to encourage studentlearning by interactively developing a diagram corresponding with a story.As the game is played, both a visual diagram display for the student andan underlying structure comprising of JavaScript arrays that represent theelements of the ER diagram are built. The user inputs are normalized by re-moving all non-alphabetic characters and then converted to lowercase text.The auto-evaluation is performed by comparing the string contents of theanswer arrays to the strings in solution array. Although exact matching isrequired for this system as well, there are a number of acceptable matchesin each solution array to allow for slightly more flexibility in the student’sanswer.A system was developed [BAK19] that attempts to address the problemof exact naming in auto-grading software. This tool use syntactic, structural,and semantic matching of string representations of UML diagrams to allowstudents to enter a wide range of answers that may be evaluated as correct bythe system. The system uses the Levenshtein distance [L+66] to determine asyntactic match for component names. Names are also semantically matchedusing a combination of three algorithms to determine a similarity score.The classes are then subject to a structural match by comparing componentnames and the matched classes are then compared and graded. To evaluatethe auto-grading system, 20 diagrams were graded manually by an instructorand then assessed again with the tool. The automated grades were within14% of the instructor’s grades.The auto-grading accuracy was improved by further customized gradingcriteria [BAK20]. In this study, experiments were conducted on two differ-ent groups to measure the auto-grading effectiveness of the system comparedto manual evaluation. There were two sets of evaluations done. First, thediagrams were graded without the marking customization. Then, they wereevaluated again with the instructor’s custom grading criteria. In the firstevaluation, the first group had results that had an average difference of6.5% to the instructor’s grading and the other group’s results had an av-erage difference of 24.6%. After adding the custom marking configuration,the difference in the first group was reduced to 4.8% and the difference inthe second group was reduced to 13%. This addition to the system in-92.3. GENERATING QUESTIONScreased accuracy for the auto-grader to be within 5 to 15% of instructorgrades. However, although this seems to be relatively high accuracy, anygrade deviation from the instructor’s grade is undesirable and because ofthe variation of inputs accepted into the system, higher marking accuracymay be difficult to achieve with this approach.2.3 Generating QuestionsA barrier to providing a large quantity of database design questions toallow students to practice concepts to mastery is that instructor-generatedquestions and marking criteria are time-consuming to produce. The capabil-ity to generate question variants based on parameters is valuable as shown bydomain independent systems such as PrairieLearn [WHZ15] and Web-CAT[EP08]. Generated question variants support repetitive student learning andare useful in evaluation situations to reduce student collaboration and copy-ing [CWZ18]. To the author’s knowledge, there has been no prior researchdone in the specific area of database design question generation.2.4 Limitations on SubmissionsAuto-grading systems provide students the opportunity to utilize feed-back to help guide their answers while practicing new concepts. However,if a system does not have a submission policy to limit attempts, studentsmay develop an over-reliance on the feedback which may negatively impacttheir learning. Most automated assessment systems support either unlim-ited submissions or a maximum number of attempts per question as thedefault. Systems may also enforce a time limitation between submissions[EP08]. Another technique is to penalize mistakes where a student’s submis-sion receives a lower mark than the previous submission, called a regression[BZHH21]. Limiting the number of submissions is widely used to counteractpoor behaviors. Students are less likely to guess and more likely to spendmore time thinking about a problem with limits applied. The goal of sub-mission policies is to encourage independent student learning and reflectionand reduce reliance on feedback and hints provided by the system. Auto-mated assessment systems have the potential to improve positive studentmetrics [LSTA21] such as performance, engagement, learning, and retentionand decrease procrastination and failures. Prior results have discussed howstudents may use feedback to change their solution without thinking aboutthe problem and stop working on a question once the mark is “good enough”102.4. LIMITATIONS ON SUBMISSIONS[HBTN21]. Systems vary the amount of feedback and hints provided to re-duce reliance on the system. Submission policies are also a key componentin insuring positive outcomes.Edwards [Edw04] argues for encouraging students to move from trial-and-error strategies to reflection in action. Prior research by Malmi et al.[MKKN05] demonstrates how introducing limits on attempts versus unlim-ited submissions encouraged students to spend more time on each submis-sion and their first attempts had higher performance. Students often usetrial-and-error near the end of an assessment to improve their grade butnot their learning. In [KKM06], students were classified into five categoriesbased on their number of submissions and final grade: passers - few at-tempts/low grades, ordinaries - few attempts/medium grades, talented - fewattempts/high grades, ambitious - high attempts/high grades, and iterators -high attempts. This classification did not consider the time between studentsubmissions or any grade difference between submissions. The classificationgoal was to identify user profiles and trial-and-error behaviour.Detecting and penalizing regressions in student submissions that resultedin lower marks was explored in [BZHH21]. In this study, student perfor-mance on automatically graded programming assignments with an imposedregression penalty in a software engineering course were compared with thesame assignments from the previous year without the regression penalty.Introducing the regression penalty was shown to dramatically reduce thenumber of submissions by 50%, decrease regressions by 90%, and increasethe time between submissions. Regression penalties encouraged students tospend more time thinking about their answers before submission. However,it should be noted that some students indicated a perceived compromise totheir grades as well as increased stress-levels related to the penalties. Furtherto that, the overall average grade fell by 10% and analysis on the assign-ment submissions revealed a reduced number of submissions after studentsachieved a high, but not perfect mark. A number of students reported thatstopped submitting on the assignment after achieving 95% out of concernof lowering their grade.The key challenge is identifying the appropriate submission policy thatleads to desirable outcomes. Unlimited attempts and flexibility are commonfor formative assessments [MBF+18] where the focus is on student practiceand learning. Appropriate feedback allows students to improve their under-standing [MBF+18], increases engagement [MGF+20], and reduces bias andvariation in grading. For formative assessments, the submission strategyshould discourage poor behaviors [Auv15, MDP20] such as the trial-and-error strategy and procrastination. Students may be more likely to “game112.5. DISCUSSIONthe system” [Pie13] compared to learning the material with an automatedsystem compared to instructor grading. Multiple attempts improve studentperformance in summative assessments [FGF21] but determining the num-ber of attempts may be a harder issue to address. A survey of assessmentsystems [IAKS10] explicitly mentions limiting the number of submissionsand the amount of feedback as considerations for these systems, but thereis a lack of recommendations and best practices.2.5 DiscussionAs noted above, there are several tools for pedagogical diagram creationand evaluation, however, no existing system contains all of the features thatis presented in this work. Like a number of the tools described above, Au-toER is a simplified database diagram creation tool that is easy to use,requires no installation, and is free for student use. It also provides auto-mated feedback to students that can be requested at any time. This workis different from the aforementioned tools in two main regards:1. Students do not manually enter component names. Instead, they inter-act with the question text directly to identify the diagram elements.This ensures that the system’s string comparison approach to auto-evaluation of diagrams is accurate and precise. Since the class namesand attributes are predefined in the question, there is no need for syn-tactic or semantic matching which means that student answers can bedirectly compared with solutions without ambiguities and the markingwill match the instructor’s rubric exactly.2. The AutoER system also is capable of automatically generating UMLdiagram design questions and the associated solution diagrams. Tothe author’s knowledge, this is a novel feature that is not present inany other database design learning system.12Chapter 3AutoER SystemDevelopment and Design3.1 System OverviewThe AutoER system supports students learning database design dia-grams by providing real-time visualization and feedback during questionanswering. It is a simplified diagram creation tool that targets users firstlearning the design concepts and process of extracting model elements fromwritten text. Students can answer instructor-created or randomly gener-ated questions by interacting with the question text to develop the design(see Figure 3.5). Unlike other existing UML design software, the studentselects the modeling construct to add (entity, attribute, relationship) fromthe question text but does not directly draw the construct on the diagramitself. The diagram is drawn based on the student answer dynamically. Be-cause the answer is built with predefined keywords, the system is able toaccurately and precisely evaluate the student’s answer by string comparisonto instructor solutions, avoiding the need for syntactic and semantic match-ing and other types of normalization on element names. In addition to this,there are also no other current systems that allow for the auto-generation ofUML diagram design questions and solutions, which makes direct compar-isons between AutoER and other existing systems difficult.The key features are:− Ability to create a question type that represents a particular questionstructure− Support for uploading user interface code file and marking code filefor a question type− Creation of generated questions with solutions using uploaded code− Storing, evaluating, retrieving student questions and answers− Ability to optionally enforce attempt limiting submission policies133.2. SYSTEM ARCHITECTUREFrom the instructor perspective, the AutoER system is completely con-figurable including the user interface presented to students, the evaluationand feedback, and the parameters and approach for question generation.The system consists of a backend web server and database for generating,evaluating, and storing questions and answers. The JavaScript/HypertextMarkup Language (HTML) user interface allows students to interact withquestions on any browser or Learning Management System (LMS), avoidingthe hassle of program installation on their systems.3.2 System ArchitectureThe server is implemented using Django, React, Traefik, and PostgreSQLas a stand-alone system deployed using Docker Compose. Porting the serverfunctionality to use other question systems such as [WHZ15, EP08] is futurework.The system is comprised of four Docker containers: Student Frontend,Backend, Database, and Reverse Proxy as shown in Figure 3.1. These con-tainers and their functionality are discussed below.Figure 3.1: AutoER System Architecture©Sarah Foss (2022)143.2. SYSTEM ARCHITECTURE3.2.1 Student FrontendThe Student Frontend container hosts a React JS website that primarilyhandles and renders the question interface. After a user is authenticated, aReact page retrieves and displays the instructor-defined template and ques-tion associated with the Uniform Resource Locator (URL). It then attemptsto retrieve the last answer submitted by the authenticated user and associ-ated auto-grading information for the question. If no answer is found, it willsend an empty object to the displayed page denoting that it is the user’sfirst attempt.When a student submits an answer for evaluation, it sends the student’sanswer information to the Django component for evaluation and then awaitsthe marking information to send to the page for display.This container also allows for course administrator authenticated accessto pages that offer in-depth details about specific student answers and allowsadministrators to access a student’s last answers for each question.3.2.2 BackendThe Backend container hosts a Django application to handle the server-side processing for the AutoER system. The server’s duties include the struc-turing of data, interfacing with the PostgreSQL database, communicatingwith the Frontend container, executing the auto-marking code, executingthe question generation code, and providing the interface for administratorsto manage questions and users.REST APIThis application allows for communication with the Frontend containerthrough its Representational state transfer (REST) Application Program-ming Interface (API) built with the Django REST Framework. The fron-tend site will initiate a communication with an authenticated GET requestat the server’s /api endpoint and the server will respond by transferring therequested data as a JavaScript Object Notation (JSON) object.Auto-Marking AnswersWhen the server is requested to evaluate a student’s answer, it will re-trieve the question, the custom Python auto-marking code associated withthe question, and all the possible answers from the database. It will thencompile and execute the code using the question and answers as arguments153.2. SYSTEM ARCHITECTUREand return a JSON object containing the evaluation results. The resultinggrade information is then stored in the database and the results are sent aspart of the API’s response JSON object.Question GenerationThe process for question generation is slightly more complex. When theserver receives a request for a generated question type, it will first attemptto retrieve the generated question associated with the user and questiontype denoted by the URL from the database. If it does not exist, then thequestion generation code will be run. The steps are as follows:− Generate a random seed for the question to ensure uniqueness.− Retrieve the associated generation type information from the database.This contains the generated code description (i.e., midterm exam ques-tion, final exam question, practice question, etc.), a reference to theassociated question template, the maximum grade that the generatedcode should allow, and the custom Python question generation code.− Execute the question generation code, passing the seed as an argument.− Store the resulting generated question to the database using the seedas a title.− Store the resulting answer to the database with a reference to theassociated question.− Add an associative entry to the database containing a reference to thequestion type, a reference to the generated question, and the currentuser.− The generated question is sent as part of the API’s request fulfillment.Administration InterfaceCurrently, course administrators use Django’s integrated administrator’sinterface to upload and update questions, question templates, and updateall other question related information. Figure 3.2 shows an example of ques-tion creation in the administration portal. They also use the administrationinterface to create and update student user profiles. Future work includesbuilding a custom course administrator’s front-end to increase administra-tors’ ease of use.163.2. SYSTEM ARCHITECTUREFigure 3.2: Administration Interface Question Creation©Sarah Foss (2022)3.2.3 DatabaseThe Database container hosts a PostgreSQL database server. As shownin Figure 3.1 this container only interfaces with the Backend container be-cause Django effectively manages all the incoming and outgoing data usingObject Relational Mapping.Figure 3.3 shows the design diagram of the AutoER system.3.2.4 Reverse ProxyThe Reverse Proxy container hosts a Traefik reverse proxy server. Thisserver is the entry point into the AutoER system’s multi-container envi-ronment and is the only accessible container from outside Docker. Thiscontainer will map external URLs to internal Docker container services androute inbound traffic to the correct container.The URL https://autoed.ok.ubc.ca/ is the student endpoint andany requests made to that address will be routed to the Student Fron-tend container. The API and administration interface endpoint is the URLhttps://autoed.ok.ubc.ca/api/ and the reverse proxy server will routethose respective requests to the Backend container.173.2. SYSTEM ARCHITECTUREFigure 3.3: AutoER Design Diagram©Sarah Foss (2022)183.3. FEATURES3.3 Features3.3.1 LoginA student accesses an AutoER question through a question URL that isprovided by the instructor as part of an assignment or quiz. Each questionhas its own unique URL. Before they have access to any question, they mustenter their preassigned username and password into the secure login screen,as shown in Figure 3.4. Any failed attempts will redirect the user back to thelogin screen with a notification of the incomplete sign-in. After a successfullogin, an authentication token is stored in local storage to allow the studenta persistent logged in state.Figure 3.4: AutoER Login Screen©Sarah Foss (2022)3.3.2 Question InterfaceOnce logged in for the first time, the student will see the question textand an empty diagram. The question text contains markup that allows stu-dents to select particular words and phrases to add to the diagram. Context-sensitive pop-up menus allow for adding entities and attributes. Figure 3.5shows the user interface and a student answering the question. As the stu-dent builds the diagram, it is dynamically updated in real time. Diagramsare drawn using the open source nomnoml library1.There are two side menus that allow for editing entities and relationshipsbetween entities. The manage entities menu enables users to remove entitiesand attributes and add or edit key properties on attributes. Figure 3.6 showsa user using the manage entities menu to indicate a particular attribute isa key for an entity.1https://nomnoml.com/193.3. FEATURESFigure 3.5: AutoER User Interface©Sarah Foss (2022)Figure 3.6: Adding a Key to an Attribute©Sarah Foss (2022)203.3. FEATURESRelationships can be added and edited between entities using the managerelationship menu. Relationships can be between two distinct entities orcan be self-referential. The following cardinalities can be optionally addedto relationships: zero or one (0..1), one (1..1), zero to many (0..*), and oneto many (1..*). An example of a user editing a relationship between entitiescan be seen in Figure 3.7.Figure 3.7: Editing a Relationship between Entities©Sarah Foss (2022)When a student wants to request feedback, they click on the Check Markbutton in the top left-hand corner as shown in Figure 3.5. This submits atext representation of the diagram for evaluation to the server and feedbackis returned immediately to the student. The granularity of the feedback,marking strategy, and number of times a student can submit are control-lable by the instructor. An example of feedback provided to students isshown in Figure 3.8. This high-level feedback provides only grading infor-mation at the level of diagram components (entities, relationships, etc.). Itis suitable for exam situations, whereas feedback on specific errors may bemore appropriate for formative assessments.3.3.3 Submission PoliciesA recent improvement on the AutoER system is the addition of newsubmission policies to limit the number of submissions per student. Theattempts can be limited by either restricting the number of attempts orwith a grade regression penalty. This is chosen by the instructor duringquestion creation.When the restricted number of attempts policy is enforced, the systemwill increment the attempt number after each submission made by the stu-dent. When the maximum number of attempts has been reached, the systemwill disable editing of the diagram and show the student the mark of the213.3. FEATURESEntity name marks: 1.2 / 1.2Entity attribute marks: 0.5 / 0.6Entity primary key marks: 0.8 / 1.2Weak entity key marks: 0.5 / 1.0Extra entities: 3.0 / 4.0Total entity marks: 3.0 / 4.0Relationship entity marks: 2.5 / 3.5Relationship cardinalities marks: 2.5 / 3.5Extra relationships: -0.25Total relationship marks: 4.75 / 7.0Total marks: 7.75 / 11.0Total scaled marks: 7.05 / 10.0Figure 3.8: Example Student Feedback©Sarah Foss (2022)most recent attempt. The maximum number of attempts is customizableby the instructor.The regression penalty submission policy, based on the regression penaltyintroduced in [BZHH21], applies a permanent penalty to the student’s gradewhen a student submits an answer that is graded lower than the previousattempt. Where P is the previous higher mark and C is current lower mark,the regression penalty R is calculated as:R =(P − C)2(3.1)For example, if a student’s mark decreased from 94 to 90 from one sub-mission to the next, then the regression penalty is 2 marks. The highestmark they would be able to receive would be limited to 98. Any regres-sion penalty accrued is permanent and cumulative. The regression penaltycould also be modified with different values, which may be investigated infuture work. Although the regression penalty policy allows for an unlimitednumber of attempts by the student, it is intended to limit trial and errorsubmissions.During question creation, the instructor can choose from the followingpossible schemes:− Unlimited attempts− Restricted number of attempts− Regression penalty223.4. AUTOER QUESTION CREATION− Allow the student to choose regression penalty or limited attempts− Randomly assign a limited number of attempts set by the instructoror a regression penaltyIn previous iterations of the system only the unlimited attempts submis-sion policy was offered. As discussed in the results chapter, during the initialevaluation of the system, a high average number of submissions by studentswas noted. These policies were developed to limit undesirable submissionbehaviour.3.3.4 Student Mark RetrievalInstructors and TAs are able to access a detailed breakdown of the eval-uation of a student’s submitted answer. Figure 3.9 shows an example ofthe course administrator’s view of a student’s answer. This page shows thestudent’s final mark, the text representation of their answer, the visual rep-resentation of their answer, the feedback the student received, and a MarkerFeedback section with more in-depth information about the grading notavailable to the student.The Marker Feedback section lists any missed or incorrect entities, at-tributes, relationships, and cardinalities and shows the correct diagram con-structs from the answer. This allows any course administrator to verify theauto evaluation done by the AutoER system and answer student inquiriesabout their automated grade on a question.3.4 AutoER Question CreationInstructors define questions in the Django administration portal. Courseadministrators can also upload custom evaluation code, question generationcode, and user interfaces. Table 3.1 summarizes the different customizablecomponents involved in question creation and customization and the overalldesign of the database is shown in Figure 3.3.Two types of database design questions can be created in the system:instructor designed questions and generated questions.Instructor Defined QuestionsExisting UML questions that have been written by instructors are con-vertible for use in the AutoER system. The steps for developing instructor-created database design questions are:233.4. AUTOER QUESTION CREATIONFigure 3.9: Evaluation Details of a Student’s Submitted Answer©Sarah Foss (2022)243.4. AUTOER QUESTION CREATIONQuestion Component DescriptionQuestion Type The type of question in the system. Cur-rently, the system only accepts UMLdatabase diagram design type questions,but support for other types of questions isfuture work.Question Template The user interface for questions. Each ques-tion template also holds the marking codeand the marker’s feedback page and is as-sociated with a question type.Question The marked-up text of a question, maxi-mum grade, penalty type, and other ques-tion details. Each question is assigned toan existing question template.Answer An answer to an associated question. Theremay be multiple answers for one question.Generated Question Type A description for a question that may berandomly generated by a user and the codefor generation. Every generated questiontype created has a URL for a user to visitto generate a random question. Each gen-erated question type is associated with anexisting question template.Generated Question Detail Details regarding a question that has beenautomatically generated including the gen-erating user, date, and a reference to thegenerated question.Table 3.1: AutoER Question Components©Sarah Foss (2022)− The UML database design question texts are marked up with AutoERnotation.− The answers to the questions are converted to the required string rep-resentations.− If the proper template does not already exist, a question type andtemplate is created in the AutoER system using the administrationinterface. Each template contains the customizable code for: the ques-253.4. AUTOER QUESTION CREATIONtion interface page, the auto-marking code, and the marker feedbackpage.− The database design question is created in the system referencing theassociated question template. Each question contains the marked upquestion text and specific marking criteria.− The answers are created in the system referencing the associated ques-tion. Each answer contains the converted string representations of theanswer diagram.− The question is then made available to students using the questionURL https://autoed.ok.ubc.ca/questions/{question_id} allow-ing access with any web browser or integration into a LMS. The systemsupports multiple question URLs that students can visit anytime.Automatically Generated QuestionsIn addition to instructor-designed questions, AutoER is capable of au-tomatically generating question variants. They allow students to generateany number of questions to allow them to practice until mastery. Generatedquestions are also useful for online exams to combat academic dishonesty aseach student will be assigned a unique variant.The steps for developing automatically generated database design ques-tions are:− If the proper template does not already exist, a question type andtemplate is created in the AutoER system using the administrationinterface. Each template contains the customizable code for: the ques-tion interface page, the auto-marking code, and the marker feedbackpage.− A generated question type is created in the system referencing theassociated question template. Each generated question type containsthe description of the question, the maximum grade for the generatedquestion and the customizable code for the generation of a databasedesign question.− A question is made available to students using the question URLhttps://autoed.ok.ubc.ca/generatedQuestion/{gen_ques_id} al-lowing access with any web browser or integration into a LMS.263.4. AUTOER QUESTION CREATION3.4.1 Question MarkupIn many cases, instructors already have a selection of questions that theywould like to assign their students. These pre-defined questions can be usedwith AutoER with the proper formatting.A question can be converted for use by the system by marking up key-words that should be selectable by the student with square brackets. Theset of marked-up keywords must include all of the entities and attributesthat should be present in the final answer. It should be noted that everymarked-up keyword can be added by the user as a possible entity and apossible attribute to any and all entities so other keywords that are not partof the answer can be marked up to increase the difficulty of the question.If there are keywords that are not appropriate for the graph (i.e., long key-words, multiple word keyword, etc.) alternate words can be substituted byincluding them in parentheses after the keyword entry. Figure 3.10 shows apre-defined question before conversion and Figure 3.11 shows the questionafter conversion for use with the AutoER system.Figure 3.10: Question Base©Sarah Foss (2022)Figure 3.11: Question Markup©Sarah Foss (2022)273.4. AUTOER QUESTION CREATIONThe question is loaded into the system by creating a question in theadministrator’s portal. The instructor is able to upload a formatted ques-tion, set the total marks for the question (which will scale automatically),set the type of limiting penalty to be used and the maximum attempts ifnecessary. The question needs to reference the associated question templateduring question creation.3.4.2 Answer FormatEvery question entered into the AutoER system must have at least oneassociated answer. Multiple correct answers are also accepted by the system.Answers, which are typically in diagram form, must be converted into astring representation with the following rules:− An entity is represented by enclosing the name in square brackets.− Attributes are associated with an entity by adding a pipe character(‘|’) after the entity name and listing the attributes delimited by semi-colons.− Keys are denoted after the attribute name with either a {PK} or{PPK}, representing a primary key or partial primary key respec-tively.− Define relationships between entities by specifying the two entities insquare brackets and separating them with a dash.− Cardinality constraints can be added to a relationship by includingthem on the appropriate side of the dash. Acceptable cardinalities are0..1, 1..1, 0..*, and 1..*.− Flexible cardinality constraints are also supported for questions withmultiple correct answers. AutoER’s flexible cardinality constraints are0@1, identifying that either 0..1 or 1..1 is correct and * identifying thateither 0..* or 1..* is correct.Figure 3.12 shows an example of an answer before conversion into theAutoER answer format. Figure 3.13 shows the AutoER string represen-tation of that answer. The answers are loaded into the system by creatinganswers with reference to the associated question in the administrator’s por-tal. Future work includes the creation of an instructor front-end that wouldallow a question-and-answer string to be built dynamically by interactingwith context menus, similar to the student question interface.283.4. AUTOER QUESTION CREATIONFigure 3.12: Original Answer©Sarah Foss (2022)Figure 3.13: String Representation of the Answer©Sarah Foss (2022)293.4. AUTOER QUESTION CREATION3.4.3 User Interface TemplateThe customizable user interface template of AutoER is implemented ina single JavaScript/HTML file. It is formatted as a single file to allow forsimple uploading to the server.When the file is first loaded by a browser, the file receives a marked-upquestion, any previous grade information and answer strings for the currentuser from the backend API via the React application. It will then parsethe received marked-up question and reformat it with HTML for AutoERfunctionality. The question will be rendered with event-listeners on the key-words to add the interactivity with the question. If it is not the student’sfirst attempt on the question, the last submission information will be dis-played, and the previous attempt’s diagram will be drawn to the HTMLcanvas element. Otherwise, the text “Your ER Diagram is empty” will bedisplayed on the canvas and it will display that it is the user’s first attempt.The system uses the nomnoml library to draw the visualizations to thecanvas HTML based on a string representation of the answer. As the userinteracts with the text of the question and menus, the program builds theanswer string dynamically. After any updates to the string, the graph visu-alization is redrawn.When the user clicks on the check mark button, as shown in Figure 3.14,the page submits the user’s information to the Backend API via the Reactapplication for auto-evaluation and awaits the response. When the responseis received, it displays the mark and feedback for the student. If there areremaining attempts, the template file allows the student to continue editingtheir diagram.Figure 3.14: Check Mark Button©Sarah Foss (2022)3.4.4 Automatic Question EvaluationWhen a student requests feedback on a design question, the text repre-sentation of their answer is sent to the server for evaluation. The evaluationprocess performs the following steps:− Matches entities in the student answer by name with instructor solu-tion(s).303.4. AUTOER QUESTION CREATION− For each entity, matches attributes within the entity.− Detects relationships in student answer matching instructor solutionusing names of entities and cardinality constraints (0..1, 1..1, 0..*,1..*). Flexible cardinality constraints are supported for questions withmultiple correct answers.− Verifies cardinality constraints match solution.− If more than one instructor solution is available, student receives high-est mark when comparing with each solution.The automatic question evaluation using the text representation of theanswer has similar characteristics as other systems [BAK20, Has11]. Marksare assigned based on a student answer matching the solution’s diagram ele-ments (entities, relationships, attributes), and the instructor assigns weightsto each of the diagram elements. The default marking scheme assigns equalweight for all elements of the same type (e.g., all entities have same markvalue), but instructors are free to modify marking criteria. Assigning marksequally makes it more efficient for instructors creating marking criteria. Thesystem automatically scales the mark to a target question mark. An in-structor can configure the weights of diagram elements, and even modifythe Python code for individual questions for specific use cases.The matching process has very high precision as the names and labels inthe diagram are extracted from the question markup rather than free-formentry by students. This avoids issues with naming conflicts and matchingambiguity that reduce accuracy in other systems. The disadvantage is thatthe question markup provides students hints on the key information in thequestion. Instructors control the question markup and are encouraged to addmarkup for words that are not part of the solution to increase the difficultyof the question. Given a particular markup word phrase, the student mustperform the key design activity of determining how to model as an entity,attribute, or relationship.3.4.5 Automatic Question GenerationAutoER is capable of automatically generating question variants givenparameters such as the number of entities, attributes, and relationships.Python code generates the question text and question answer using theseparameters. Randomization allows different diagrams to be generated foreach student.Question generation has the steps:313.4. AUTOER QUESTION CREATION− Generate a random number of entities in range [min,max] given byinstructor.− Generate a random number of weak entities.− Generate relationships of particular association cardinalities (0..1, 1..1,0..*, 1..*). Rules limit the number of relationships between entities andnumber of recursive relationships.− Names of entities, attributes, and relationships used in the answer andquestion text are randomly generated. Figure 3.15 shows an example ofan automatically generated question with letter placeholders for entityand attribute names. Other placeholders, like “gibberish” words, maybe used instead.− Vary question text by randomizing phrase descriptions. For example,a 0..* relationship between entities A and B may be described as “Amay be related to many B” or “A has multiple relationships with B”or other similar phrases.− Question text statements are re-ordered to mimic real questions. Forexample, relationship information may appear near the associated en-tities or later in the description.The server supports generated questions by allowing the upload of a codefile for question generation. When the student accesses the question URL,the generation code is run to create a random question specific for that userthat is stored in the database. Then the student interacts with the questionas if created by an instructor.323.4. AUTOER QUESTION CREATIONFigure 3.15: Generated Question Example with Letter Placeholders©Sarah Foss (2022)33Chapter 4Case Studies: UndergraduateDatabase CourseThe AutoER system was evaluated in two sessions of the same under-graduate database course, COSC 304 Introduction to Databases, at theUniversity of British Columbia Okanagan.4.1 Participant RecruitmentThe first session was in the summer 2021 term with 44 students enrolled,and the second session was in the fall 2021 term with 180 students enrolled.After receiving consent from the instructor (see Appendix A.1), the studentswere given an introduction to the system at the beginning of the courseby the author and were given the choice to participate in the study. Thestudents that were interested in participating were asked to read and signthe consent form (see Appendix A.2). Although this study took place in aComputer Science course at the University of British Columbia Okanagan,no course administrators or participants of the study were involved in thedevelopment of the system or in the investigation of the studies. Specifically,the author or supervisor were not involved as instructors in the course.4.2 System Use Data CollectionThe database design module is about 20% of the COSC 304 course con-tent and is taught after students have learned Structured Query Language(SQL) and the relational model. Students have limited prior experience withUML, which they are exposed to briefly in prior programming courses.Data CollectedThe following is a list of all the data collected from each submission tothe AutoER system:344.2. SYSTEM USE DATA COLLECTION− User ID− Question ID− Submission number− Student answer− Automatically assessed grade− Generated feedback for the student− Detailed feedback for course administrators− Date and time submitted− Source Internet Protocol (IP) address− Penalty type (second study only)− Regression penalty (second study only)Marking Scheme UsedAlthough the marking scheme is customizable, each diagram question inthe study was marked with the following default marking scheme:− 0.2 marks for each correct entity name− 0.1 marks for each correct attribute list for an entity− 0.2 marks for identifying each primary key− 0.5 marks for identifying each weak entity− -0.25 marks for each extra entity in diagram− 0.5 marks for identifying relationship between two entities− 0.25 marks for identifying each correct cardinality on relationships− -0.25 marks for each extra relationship in diagram354.2. SYSTEM USE DATA COLLECTIONIf the total marks did not match the instructor defined mark of thequestion, the question’s total calculated score was automatically scaled tomatch.A question in AutoER may also be subjected to a submission policy ofeither a regression penalty, which applies a permanent penalty to a student’smarks when a submission with grade decrease occurs, or a maximum attemptlimitation. Like the marking scheme, the number of maximum submissionsis customizable, but the default maximum number of submission used inthe system for the policy was seven. The maximum attempt limitation ofseven submissions was chosen as a baseline as it was a reasonable numberdesigned to limit guessing but fewer attempts could be investigated in futurecomparison and study.4.2.1 AssignmentsIn the course, there are two assignments involving design questions, aquestion on an online midterm exam, and a question on an online finalexam. Students participating in the study were given the option of usingthe AutoER system for any of these components, or they could continue toanswer the questions on paper or using the UML software previously usedin the course. The assignments consisted of human generated questionsconsistent with prior years. The instructions and questions were presentedon GitHub with a link to AutoER and also links to other software thatstudents may choose to use to complete the questions.The fall session study added some additional experimentation from theinitial summer session study. In the second study, there were additionalsubmission policies enforced in the system that were not present in the firststudy. This allowed for further analysis on the submission behaviour notedin the summer.The first assignment was done in pairs and consists of two databasedesign questions. Each question was marked out of 10. Figure 4.1 shows thefirst question of the first assignment and the associated solution diagram.See Appendix C for the detailed assignment questions and solutions used inthe study.The second assignment (Appendix C.2) was the first deliverable of thecourse project developing a database-driven, online store and was done ingroups of up to four students. The second assignment has an advanceddatabase design question involving multiple weak entities for the first ques-tion, and a large design question (11 entities, 13 relationships) for the sec-ond question. This assignment also requires using data types for attributes364.2. SYSTEM USE DATA COLLECTION(a) Question 1(b) Question 1 AnswerFigure 4.1: Assignment 1 Question 1©Sarah Foss (2022)and mapping to the relational model, which are not supported by AutoER.Students using AutoER must manually convert their diagrams into SQLCREATE TABLE statements. The AutoER portion of the questions were374.2. SYSTEM USE DATA COLLECTIONworth 10 marks and 21 marks respectively.The participants in the fall study and summer study both had unlimitedattempts with no penalties on the assignment questions.4.2.2 Midterm ExamThe midterm exam question used the random question generation fromthe AutoER system, but students were still able to answer the questionhowever they wished. In COSC 304, the midterm exam is 90 minutes, openbook, and students may use any software and course resources. Overall,50% of the exam marks are related to database design with the other halfon database programming. The randomly generated question was worth20% of the exam.In the summer study, participants who chose to use the system hadunlimited attempts with no penalties on the midterm question. Studentparticipants in the fall study were randomly assigned a submission policyof either a maximum of seven attempts or regression penalty. However, dueto an error with the exam deployment on the learning management system,the split was not even. 29% of participants were assigned the maximumattempts limitation and 71% were assigned the regression penalty. As thiswas the first usage of the submission policies in the system, it is difficult togauge the impact of the uneven split.Before the summer exam, a practice midterm with an automatically gen-erated question was available with the same format. For the fall exam, apractice midterm with three automatically generated questions were avail-able: a practice question with a regression penalty, a practice question witha maximum of seven attempts limitation, and a practice question where thestudents could choose the submission policy.4.2.3 Final ExamIn both sessions, the final exam was open book, and the final examquestion in the AutoER system was an instructor defined question used inprevious offerings of the course. Figure 4.2 shows the final exam questionused in both studies. The question was presented on the online exam webpage with a link to AutoER as well as instructions to upload an image ofthe diagram.In the summer, the final exam was 3 hours and the database designquestion was worth 10% of the exam. Participants who chose to use the384.3. STUDENT SURVEYAutoER system had unlimited attempts with no penalties. A practice examwas available with the same format.The final exam was 2.5 hours in the fall, and the database design questionwas worth 12% of the overall exam. For this exam, participants were able tochoose the submission policy: they could either have a maximum of sevenattempts or they could have unlimited attempts with a regression penalty.They were also able to choose their submission policy on the practice exam.4.3 Student SurveyStudents were asked to complete a survey (see Appendix B.1) after work-ing with the system for several weeks. The survey contained the questionsfrom the System Usability Scale (SUS) [Bro96], an established and widely-used questionnaire that measures ease of use in systems. It is comprisedof 10 questions with Likert scale answers. To calculate a SUS score, eachanswer is converted into numerical data with the following scoring:− Strongly Disagree: 0 points− Disagree: 1 point− Neutral: 2 points− Agree: 3 points− Strongly Agree: 4 pointsEach score is adjusted by subtracting 1 from each odd-numbered scoreand subtracting each even-numbered score from 5. The sum of the adjustedscores is computed and multiplied by 2.5 to get the standard SUS scorewhich will be in the range of 0-100. Equation 4.1 shows a more concise wayof calculating the SUS score.SUS = 2.5((SUS01 + SUS03 + SUS05 + SUS07 + SUS09)− 5)+(25− (SUS02 + SUS04 + SUS06 + SUS08 + SUS10))) (4.1)Results of empirical evaluation of the System Usability Scale [BKM08]indicate that systems that score between 60 - 70 on the scale can be consid-ered marginally acceptable and systems that score above 70 are consideredacceptable, with better systems scoring in the high 70s and above.394.4. COURSE ADMINISTRATOR SURVEYThe student survey also contained a multiple-choice question about theiruse of the system and five open-ended questions asking about why they usedthe software, positives and negatives with the software, and general com-ments. The survey was made available after the midterm and was completedby 27 students out of 44 students (61%) in the summer offering and wascompleted by 50 out of 180 (28%) in the fall session.4.4 Course Administrator SurveyInstructors and TAs were asked to complete a survey (see AppendixB.2) after the end of the course. This survey was designed to gauge theeffectiveness of the system from the course administrator’s perspective. Itwas comprised of 4 multiple choice questions and 6 open-ended questionsabout their decision to continue to use the system, what they liked aboutthe system, what they thought could be improved, and general comments.404.4. COURSE ADMINISTRATOR SURVEY(a) Final Exam Question(b) Final Exam QuestionFigure 4.2: Final Exam Question and Answer©Sarah Foss (2022)41Chapter 5ResultsThe results from the evaluations of the AutoER system are broken intotwo sections, System Evaluation and Student Performance Evaluation. InSection 5.1, the evaluations are focused on perceived system usability andstudent system preference. Section 5.2 focuses on student academic per-formance and submission behaviour patterns within the system. Commonmetrics used in the evaluations and their descriptions are listed in in Table5.1.5.1 System Evaluation5.1.1 System UseThe first analysis looked at if students prefer using AutoER comparedto paper and other design software. The summary of usage for the twoassignments and midterm and final exams is in Table 5.2.Students collaborated in groups for both Assignment 1 and 2. As aresult, the number of system users for the assignments was lower than thenumber of users during the examinations. This is because groups of studentswould interact with the system using a single student account. Table 5.3shows the number of students that uploaded diagrams created with AutoERas their final answer compared to the number of students that uploadeddiagrams created by hand or with other systems. The number of studentsusing AutoER was determined by counting all students in a group if thegroup submitted an answer created with AutoER.Most students used AutoER for the first assignment, with an 89% up-load rate in the summer and an 87% upload rate in the fall. AutoER doesnot support mapping to the relational model as required for the part of thesecond assignment, so usage was expected to be lower. Interestingly, the us-age was still high as students appreciated checking their design model beforeadding data types and conversion. The average number of submissions for aquestion per user was high, and individual student traces indicated studentsused the feedback to improve their understanding and performance on the425.1. SYSTEM EVALUATIONMetric DescriptionUsers The count of the users interacting with theAutoER system.% of Participants The percentage of total participants perstudy.Submits The average number of submissions per userto the AutoER system.AutoER Grade The average top grade of diagrams createdand evaluated with AutoER.Grade The average grade of all participant’s dia-grams including grades of diagrams createdwith AutoER and all other methods.Regressions The average number of grade regressions peruser. A grade regression occurs when a stu-dent submits a diagram that is graded lowerthan the previous attempt.Seconds The average number of seconds between sub-mission to the AutoER system per user.First Mark The average grade of the first submission tothe AutoER system.% of Users The percentage of total users interacting withthe AutoER system.Table 5.1: Description of Common Metrics in AutoER Evaluations©Sarah Foss (2022)questions that would not have happened with a single evaluation point. Thiswas also common for the large ER diagram in assignment 2.98% of students in the summer and 96% in the fall used the AutoERsystem for the database design midterm exam question. A higher numberwas expected given that the system generated the question. Even thoughstudents did not have to use AutoER to answer the question, almost all didfor reasons of efficiency and marking feedback.Although the final exam question was an instructor-defined question andstudents were given the choice to complete this question manually or to useany diagram software, this question saw a similar usage to the midterm, with93% and 96% usage in the summer and fall, respectively. This indicates thestudents’ strong preference for the software and the ability to submit multi-ple times and have their answers automatically evaluated during the exam.435.1. SYSTEM EVALUATIONAssign 1 Assign 2 Midterm FinalS F S F S F S FUsers 30 141 27 130 43 172 41 172% of Participants 67% 78% 61% 72% 98% 96% 93% 96%Submits 83 32 108 37 25 5 57 6Table 5.2: AutoER Usage Metrics©Sarah Foss (2022)Assign 1 Assign 2S F S FCreated with AutoER 39 157 34 170% of Participants 89% 87% 77% 94%Created with Other Methods 2 14 10 4% of Participants 5% 8% 23% 2%Table 5.3: Assignment Upload Metrics©Sarah Foss (2022)Note that the number of submissions allowed during the exam depends onthe policy enforced. This is discussed in detail in Section 5.2.5.5.1.2 System UsabilityThe next analysis evaluated the system’s usability and efficiency. Theresponses to the SUS questions are in Table 5.4. A SUS score reflects a user’ssubjective ranking of a system’s usability. A score greater than 70 indicatesa system is moderately user-friendly [BKM08]. AutoER had a SUS score of77.01 in the summer study and a SUS score 76.95 in the fall study, showingthat the system had good usability.5.1.3 Free-form Survey ResultsThe free-form survey answers had a relatively small sample size and var-ied in depth of detail and because of this, the coding of feedback was deemedunnecessary. Instead, a subjective and qualitative analysis was performedto identify perceived strengths of the tool and to help guide improvementsto the system.The feedback contained many positive comments about the ability tobuild a diagram by interacting with the question text rather than drawing445.1. SYSTEM EVALUATIONSD D N A SAQuestion S F S F S F S F S FI think I wouldlike to use thistool frequently.4% 2% 0% 4% 8% 16% 27% 38% 62% 40%I found the toolunnecessarilycomplex.50% 32% 31% 56% 12% 8% 0% 2% 8% 2%I thought thetool was easy touse.0% 2% 4% 0% 4% 10% 35% 46% 58% 42%I think that Iwould need thesupport of atechnical personto be able to usethis system.65% 54% 12% 34% 8% 4% 0% 4% 15% 4%I found the vari-ous functions inthis tool werewell integrated.4% 2% 8% 6% 19% 10% 35% 50% 35% 30%I thought therewas too muchinconsistency inthis tool.31% 24% 31% 52% 15% 12% 8% 8% 15% 2%I would imaginethat most peo-ple would learnto use this toolvery quickly.4% 2% 4% 2% 4% 10% 27% 44% 62% 42%I found the toolvery cumber-some to use.35% 26% 31% 58% 19% 4% 4% 6% 12% 2%I felt very con-fident using thetool.0% 2% 12% 4% 12% 4% 31% 50% 46% 38%I needed to learna lot of thingsbefore I couldget going withthis tool.54% 16% 15% 50% 15% 22% 8% 6% 8% 6%Table 5.4: System Usability Scale Survey ResultsCategories - SD: Strongly Disagree, D: Somewhat Disagree, N: Neutral, A:Somewhat Agree, SA: Strongly Agree©Sarah Foss (2022)455.1. SYSTEM EVALUATIONdirectly. Students indicated this helped them understand how to extractkey concepts and add them to the model (which is a critical aspect of thedesign task). The majority of the feedback involved positive comments onAutoER’s ease of use, which is a critical factor for learning design software[ALS19]. Students also commented that the marking feedback helped themlearn from and fix their mistakes, and several students indicated a desire formore detailed feedback when practicing.A selection of quotes from students when asked what they found usefulin the software include:− “It was easy to use, especially in exams.”− “It really helped me grasp the content.”− “It was a good way to get feedback on diagrams.”− “I think having AutoER is extremely useful and beneficial to studentsas it is helpful being able to interact with the diagram and receive realtime feedback.”Areas for improvement included the display of the diagrams, especiallypositioning of cardinalities on relationships, improved display of recursiverelationships, the ability to add other types of relationships, and the abilityto re-arrange diagrams.A selection of quotes from students when asked what additional featuresthey would like to see in the system:− “I think it would be helpful to be able to drag the relationships intodifferent order. This would make it easier to get a better lookingdiagram.”− “Sometimes the cardinality gets harder to read as the number of enti-ties increase. If this is solved that’ll be great!”− “A better implementation of the way the relationships link up, a lot ofthe time the multiplicities are actually hidden behind entities. Othertimes when you make a relationship between 2 items the relationshipline freaks out.”− “More tools for testing relational algebra, sql, json, xpath.”465.2. STUDENT PERFORMANCE EVALUATION5.1.4 Administrator EvaluationThe survey for course administrators indicated that the software wasfound to be very useful and easy to use by participants. Administrators alsocommented positively about the time saved because of the grading precisionof the software.Reflecting similar feedback from the student survey, administrators gen-erally suggested improvements in the display of the diagrams. They alsowould like the ability to search student attempts and a more robust markerfeedback page.5.2 Student Performance EvaluationThere were several metrics evaluated including the percentage of stu-dents using the software, the mean number of submissions per user, averagetime between submissions, and average final mark. A summary of the per-formance metrics is in Table 5.5.Student performance was compared to the course offering in winter 2020with 180 students that used the same instructional content, assignments,and final exam, but was taught by a different instructor.Assign 1 Assign 2 Midterm FinalS F S F S F S FUsers 30 141 27 130 44 172 41 172AutoER Grade 91% 89% 76% 71% 95% 94% 91% 69%Grade 93% 92% 94% 96% 93% 92% 88% 75%Regressions 18.39 6.33 22.13 6.57 3.43 0.41 10.24 1.22Submits 83 32 108 37 25 5 57 7Seconds 1445 17276 808 10446 120 N/A2 327 804Table 5.5: AutoER Performance Metrics©Sarah Foss (2022)5.2.1 Assignment PerformanceThe 2020 offering of COSC 304 had performance on both assignment 1and 2 of 85%. The performance using AutoER was higher for both assign-ments in both studies. Only a portion of assignment 2 involves designing ER2The average time between submissions per user has been removed from the analysisdue to a deployment error.475.2. STUDENT PERFORMANCE EVALUATIONdiagrams with the rest of the assignment requiring mapping to the relationalmodel. Assignment 2 was the first lab for the project involving designingand building a database, so it is less predictive of AutoER’s usability.This course was taught by a different instructor, creating a differentinstructional environment, which prevents making a conclusive statementregarding student performance.The number of assignment submissions cre-ated with other methods was not sufficient for comparison with submissionscreated with AutoER. However, it is evident that student performance wasequivalent or slightly improved from 2020, as students actively engaged withthe system through multiple submissions, and survey feedback indicated astrong preference for using AutoER compared to Astah or other UML toolsused previously.5.2.2 Midterm Generated Question PerformancePerformance on the generated midterm question was high, with an av-erage grade of 93% and 92% on the summer and fall midterms, respectively.In the 2020 offering of the course, there was no comparable design questionon the exam, however, the average grade from the 2019 COSC 304 midtermquestion was 70%. Unfortunately, the summer and fall midterms are alsonot directly comparable with the question from the 2019 midterm as thatquestion was an instructor defined paper exam question.The effectiveness of the auto-generated questions and marking were alsoinformally evaluated by TAs after manually marking each midterm questionsubmission. They noted that each student’s answer requires about two min-utes to grade manually when submitting on paper or using other software,whereas the AutoER tool did it instantly.The AutoER system’s auto-marking eliminated bias, mistakes, and vari-ation between TAs which often occurs for large classes. The system providesdetailed marking feedback to the TAs to verify correctness. This feedbackcan also be provided to students after the exam is complete.5.2.3 Final Exam PerformanceThe final exam question used in both studies was an instructor-definedquestion that had been used in the previous offering of the course. In thatprevious offering, the average grade on the final exam question was 74%.The summer study’s final question average grade was 88% which was anincrease of 14% from the previous offering. The performance on the fallfinal question was significantly lower at 75%, though still slightly higher485.2. STUDENT PERFORMANCE EVALUATIONthan the 2020 offering.It is difficult to compare the final exam question results between thesummer and fall studies as there were substantial differences in exam en-vironments. In addition to the enforced submission policies not present inthe summer, the students were only given 2.5 hours to complete the fallfinal exam versus the summer exam’s 3-hour time limitation. The markweightings were different between the exams as a result of this.5.2.4 Summer 2021 StudyFurther data analyzed relates to student performance and submissionbehaviour during the exam. In the summer study, students were allowedunlimited submissions to measure how they interact with the system with norestrictive submission policies enforced. The initial profile analysis comparedthe average number of submissions per question to the top grade achieved.Figure 5.1 displays student performance by number of submissions withquartile groupings by total number of submissions during the midterm exam.First quartile students submitted the lowest number of times and achievednear perfect marks. These students solved a substantial amount of theproblem before checking their answers. The fourth quartile of students hadat least 31 submissions and demonstrated some undesirable behaviour suchas trial-and-error. In the diagram, the dot represents the cut-off point ofnumber of submissions to be in that quartile. The graph line continues tothe end to make it easier to compare across quartiles. For example, thecutoff for the first quartile is 13 submissions.Although categorizing students by their average number of submissionsper question showed interesting results, it was determined that the averagenumber of regressions (grade from submission decreases from last submis-sion) per question better classified the students into submission behaviourprofiles.Student ProfilesThere are several distinctive student profiles (see Tables 5.6 and 5.7)that are visible in the data, and sample traces for each profile are shownin Figure 5.2. The profiles are distinguished by the number of regressionsthat were present in their answers. Note that although students are catego-rized by number of regressions, during the summer study, regressions werenot displayed to the user and did not limit how the student answered thequestion. The regressions were calculated during post-analysis. While these495.2. STUDENT PERFORMANCE EVALUATIONFigure 5.1: Student Performance Grouped By Total Submissions. Markershows maximum quartile submissions cutoff.©Sarah Foss (2022)categories are based on the number of regressions present in the submissionhistory, other behavioural patterns emerged from the analysis. Most notably,some students demonstrated behaviours like nearly completing the questionbefore the first submission, iterative development, and others showing lessdesirable trial and error exploration.Category Midterm % Final % Description1 25.0 14.6% No regressions2 25.0 17.1% 1 to 2 regressions3 25.0 17.1% 3 to 5 regressions4 25.0 51.2% 6+ regressionsTable 5.6: Student Profile Categories©Sarah Foss (2022)The top students in Category 1 achieved high scores with few attemptswith one student achieving 100% on the first attempt on the midterm, andtwo students achieving 100% on the first attempt on the final exam. Thesestudents would have similar performance if the number of attempts werereduced, or regression penalties were introduced [BZHH21]. Students inCategory 2 have a low number of regressions, however, they may or maynot have a low number of submissions. As shown in Figure 5.2, the example505.2. STUDENT PERFORMANCE EVALUATIONFigure 5.2: Example Student Performance Profiles©Sarah Foss (2022)student in Category 2 used iterative development and feedback from theauto-grader to verify they are on the right track, and only had one regression.In Category 3, more regressions are present, which typically occur nearthe middle and the end of the problem. For the database design problem,identifying entities and attributes is the logical first step, and regressions inthe middle may have allowed students to use trial-and-error along with themarking feedback to finalize the entities first. Regressions near the end seemto indicate that students may be using the same trial-and-error behaviour tofind small mistakes (often missing relationships and incorrect cardinalities)before completing the question.Category 4 students demonstrate numerous regressions throughout theirattempts, in no particular area of the problem. This seems to indicate thesestudents are using trial-and-error to solve the entire question rather thanrelying on course knowledge. All the highest submitting students were inCategory 4.Overall, the data and student profiles from the summer study validateresearch in [BZHH21] on reducing the number of submissions or introducingregression penalties to limit guessing. This motivated the development ofthe limiting attempts constraints for the system introduced in the fall study.515.2. STUDENT PERFORMANCE EVALUATION1 2 3 4Mid Final Mid Final Mid Final Mid FinalUsers 11 6 11 7 11 7 11 21AutoER Grade 99.6% 99.4% 95.0% 81.4% 92.5% 93.3% 91.5% 91.0%First Mark 53% 48% 22% 24% 32% 11% 34% 14%Regressions 0.00 0.00 1.45 1.29 4.00 4.14 8.27 18.19Submits 8 12 17 17 30 37 45 90Seconds 273 1854 68 257 73 103 79 62Table 5.7: Summer Student Profile Metrics©Sarah Foss (2022)5.2.5 Fall 2021 StudySubmission Policy ImpactThe first study revealed that some of the students that used the AutoERsystem in the summer seemed to be utilizing trial-and-error for solving ques-tions. In response to that, submission policies as discussed in [BZHH21] wereintroduced to the system for the fall study. These policies were only enforcedfor the AutoER examination questions. For the fall midterm question, thestudents were assigned either the regression penalty or were limited to 7 at-tempts on the question. For the fall final exam, they were given the optionof choosing either of the two policies.Table 5.8 shows an overview of the changes to the submission behaviourof the users of the system. The number of average grade regressions de-creased by 82% and the number of regressions dropped 88% on the fallmidterm. The regressions and submissions were reduced by 88% and theaverage seconds between submissions per user was increased by 146% onthe final exam question.These changes demonstrate that students in the second study were morethoughtful about how they approached submitting answers, ensuring thatthe answer is as accurate as possible before submission rather than repeat-edly checking and changing their answers during development. This in-dicates that adding limitation constraints positively affect how users ap-proached solving problems in the system.Submission Policy Impact By ProfileThe data was categorized into the same regression profiles as examinedin the summer study for comparison. The result of the analysis is shown inTable 5.9. One of the most obvious changes between the studies is that thepercentage of students in Category 1 (0 regressions) was increased by over525.2. STUDENT PERFORMANCE EVALUATIONMidterm FinalS F Chg S F ChgGrade 93% 92% -1% 88% 75% -13%Regressions 3.43 0.41 -88% 10.24 1.22 -88%Submits 25 5 -82% 57 7 -88%Seconds 327 804 146%Table 5.8: Limiting Attempts Change©Sarah Foss (2022)180% for both the midterm and final fall exam. The number of students inCategory 2 (1 to 2 regressions) was increased by over 170% for the final fallexams. The users that were classified into these categories during the firststudy showed desirable behaviours in their approach to solving questions inthe system. Similarly, most users in these top categories in the fall seemto use strategies that demonstrate topic knowledge rather than relying onguessing.When examining the individual performances in Category 2, there seemsto be a behaviour change in that most users seem to no longer be using it-erative development but are behaving more similarly to a Category 1 usersfrom the summer, demonstrating a high first mark with few submits. Theonly difference between the two categories in the second study is that thereare one or two grade regressions present in a Category 2 user’s answer his-tory. Sample traces for each profile in the fall study are shown in Figure5.3.The positive changes were not limited to the top categories as usersin every category show improvement in some metrics. All categories showa decrease in the average number of submissions and an increase to theaverage of the first mark submitted, as shown in Table 5.9. The decreasein Category 4 (6+ regressions) users is the most substantial, with a 98%decrease on the midterm and a 97% decrease on the final exam. However,upon examination of individual submissions, the trial-and-error behaviourseems to be still visible in the submission history for a few students.The individual submission history for users in Category 3 (3 to 5 re-gressions) tend to display an iterative development procedure, like the firststudy. However, they also seem to display a more randomized regressionpattern indicating the possibility of more guessing at answers rather thanrelying entirely on prior knowledge.One surprising finding was that Category 1 users’ final exam grade av-535.2. STUDENT PERFORMANCE EVALUATIONCategory 1Midterm FinalS F Chg S F ChgUsers 11 127 6 69% of Users 25.0% 73.8% 195.3% 14.6% 41.3% 182.1%AutoER Grade 99.6% 95.2% -4.4% 99.4% 69.4% -30.1%First Mark 52.6% 81.2% 54.3% 48.3% 53.6% 11.0%Regressions 0.0 0.0 0.0% 0.0 0.0 0.0%Submits 8 3 -65.1% 12 3 -73.1%Seconds 1854 1211 -34.7%Category 2Midterm FinalS F Chg S F ChgUsers 11 40 7 80% of Users 25.0% 23.3% -7.0% 17.1% 46.5% 172.4%AutoER Grade 95.0% 92.4% -2.7% 81.4% 71.7% -11.9%First Mark 21.9% 70.7% 222.3% 23.5% 49.2% 109.3%Regressions 1.45 1.20 -17.5% 1.29 1.41 9.9%Submits 17 7 -57.3% 17 6 -63.4%Seconds 257 589 129.3%Category 3Midterm FinalS F Chg S F ChgUsers 11 4 7 20% of Users 25.0% 2.3% -90.7% 17.1% 11.6% -31.9%AutoER Grade 92.5% 82.7% -10.6% 93.3% 61.3% -34.3%First Mark 32.4% 37.9% 17.0% 11.1% 36.3% 225.5%Regressions 4.0 4.0 0.0% 4.1 3.5 -15.5%Submits 30 26 -13.8% 37 14 -62.0%Seconds 103 360 247.5%Category 4Midterm FinalS F Chg S F ChgUsers 11 1 21 3% of Users 25.0% 0.6% -97.7% 51.2% 1.7% -96.6%AutoER Grade 91.5% 87.2% -4.7% 91.0% 62.6% -31.3%First Mark 34.1% 67.8% 98.9% 14.0% 28.2% 101.8%Regressions 8.3 7.0 -15.4% 18.2 8.7 -52.4%Submits 45 25 -44.3% 90 38 -57.3%Seconds 62 104 68.9%Table 5.9: Comparison of Profiles Across Studies©Sarah Foss (2022)545.2. STUDENT PERFORMANCE EVALUATIONFigure 5.3: Example Fall Student Performance Profiles©Sarah Foss (2022)erages were lower than Category 2 users in the second study. Upon morein-depth analysis, there was a new behaviour not noted in the first study’sexaminations: an increase of one attempt submission histories. The analysisof one-attempt submissions is shown in Table 5.10. This behaviour couldbe a result of the 30-minute decrease in time given for the final exam witha similar number of questions. This increase of one-attempt submission his-tories lowers the overall grade average of users in Category 1 from 70.0% to69.4%, however, it does not entirely explain the finding as the grade averagefrom the Category 2 users is 71.7%.Users AutoER GradeSummer Midterm 1 100%Summer Final 1 100%Fall Midterm 45 96.3%Fall Final 19 68.1%Table 5.10: User Count and Grades with One Submission©Sarah Foss (2022)Overall, there is strong evidence that limiting attempts in the systemalters user behaviours in positive ways, from reducing grade regressions andnumber of submission attempts to increasing time spent on answers.555.2. STUDENT PERFORMANCE EVALUATIONComparing Submission PoliciesThe submission policies attempt to solve the problem of over-relianceon the automatic marker with different approaches. The maximum attemptlimitation allows the instructor to set a maximum number of attempts on aquestion, while the regression penalty imposes a permanent cost on submit-ting answers that are graded lower than the previous submission.The data was analyzed to examine which of the two submission policiesis more effective at limiting trial-and-error behaviour with the least impacton student performance. Comparisons of means from the collected datawere performed using two-sample t-Tests assuming unequal variances withsignificance level α = 0.05, and the p-values are provided.Table 5.11 shows that the difference between the midterm grades wasnot significant (p = 0.84) between the two strategies, but the final examdata shows that students who had the maximum attempts submission policyachieved a grade that was nearly 10% higher (p = 0.04). The students whoused the maximum attempt policy also have a statistically significant loweraverage submits and longer submit times, possibly indicating that thoseusers spent more time taking into consideration the quality and quantity oftheir answers before submission.Midterm FinalMaxAtt Regr MaxAtt RegrUsers 50 122 137 35AutoER Grade 95% 94% 71% 62%Regressions 0.320 0.451 0.934 2.314Submits 3 5 5 14Seconds 854 607Table 5.11: Performance Comparison of Submission Policies©Sarah Foss (2022)Interestingly, the number of regressions is higher when the question ismarked with a regression penalty. This makes sense for several reasons. Ifthere is a limited number of attempts, then there are a limited number ofregressions that can happen. It also stands to reason that if the user knowsthat they have an unlimited number of submissions, they may be more likelyto submit with a trial-and-error strategy. The fact that the average numberof submissions and the average number of seconds between submissions washigher for the users with the maximum attempts policy also support thisargument.565.2. STUDENT PERFORMANCE EVALUATIONSubmission Policies Across Student ProfilesThe average grade of students in each category was lower for the regres-sion penalty compared to maximum attempts limitation. The first submis-sion mark was generally considerably lower as well. Pie charts comparingthe number of students per category for unlimited submissions, maximumattempts policy, and regression penalty for the midterm and final exams arein Figures 5.4 and 5.5 respectively.(a) Unlimited Attempts (b) Regression Penalty (c) Maximum AttemptsFigure 5.4: Midterm Exam Students by Category and Submission Policy©Sarah Foss (2022)(a) Unlimited Attempts (b) Regression Penalty (c) Maximum AttemptsFigure 5.5: Final Exam Students by Category and Submission Policy©Sarah Foss (2022)Examining the submission policies over student profiles further illus-trates the maximum attempts restriction’s effectiveness. Table 5.12 showsthe comparison of submission policies across student profiles. The average575.2. STUDENT PERFORMANCE EVALUATIONCategory 1 Category 2 Category 3 Category 4MaxAtt Regr MaxAtt Regr MaxAtt Regr MaxAtt RegrMidtermUsers 39 88 11 29 0 4 0 1AutoER Grade 95% 95% 93% 92% 83% 87%First Mark 92% 77% 64% 73% 38% 68%Regressions 0.00 0.00 1.45 1.10 4.00 7.00Submits 2 3 6 8 26 25FinalUsers 58 11 69 11 10 10 0 3AutoER Grade 70% 66% 73% 61% 63% 59% 63%First Mark 55% 44% 52% 32% 52% 20% 28%Regressions 0.00 0.00 1.41 1.45 3.10 3.90 8.67Submits 3 4 6 10 7 21 38Seconds 1249 1014 584 621 424 296 104Table 5.12: Limiting Attempts Across Student Profiles©Sarah Foss (2022)first submission mark was higher in Category 1 and Category 3 with themaximum attempt limitation. It varied between exams in Category 2 with10% lower average grade for the first attempt on the midterm and 20%higher first grade on the final exam. The average seconds between attemptsin Category 2 were also lower, while the average time was higher for bothCategory 1 and Category 3. Overall, most measures favour the maximumattempt restriction over the regression penalty.The maximum attempt policy with the restriction of 7 submissions alsoprevented any user from being classified as Category 4 with either themidterm or final exam. This is because it would be improbable for a userto have 6 regressions with a maximum of 7 submissions.Visualizations of student submission profiles are in Figures 5.6 and 5.7.The lines are colored based on the category the student was assigned to(category 1 - green, category 2 - blue, category 3 - yellow, and category4 - red). Note that the Figure 5.6a has been reduced from showing 91submissions to showing a maximum of 60 and Figure 5.7a has been reducedfrom 199 to 120. An obvious impact of the submission policy is the dramaticreduction in the number of submissions. There was a hard cutoff of sevensubmissions for the maximum attempt policy, and many students did notreach that cutoff. Although the regression penalty reduces the number ofsubmissions, there was still a percentage of students with a high numberof submissions (and regressions). There were more regressions for studentsthat had a regression penalty versus a limited number of submissions. Thisis an interesting result as the regression penalty on the student’s grade was585.2. STUDENT PERFORMANCE EVALUATION(a) Unlimited Submissions(b) Regression Penalty(c) Maximum AttemptsFigure 5.6: Midterm Exam Students by Category and Submission Policy©Sarah Foss (2022)595.2. STUDENT PERFORMANCE EVALUATION(a) Unlimited Submissions(b) Regression Penalty(c) Maximum AttemptsFigure 5.7: Final Exam Students by Category and Submission Policy©Sarah Foss (2022)605.3. DISCUSSIONonly applied for students in the regression penalty submission policy. Theresults were similar for both exams. The final exam with submission penal-ties had a much more significant reduction in submissions and regressions.Student PreferenceThe analysis examined what the fall students chose for their final examsubmission policy given the policy that they were assigned for the midterm.The results are shown in Table 5.13. Overall, given the choice of the max-imum attempts restriction or the regression penalty, 80% of students chosethe maximum attempt policy. The students that chose this for the finalexam did 10% better overall on the exam question in the AutoER system.Interestingly, students that chose this option for the final exam also tendedto do better in the course, scoring 2% higher on the randomly assigned,automatically generated midterm question, and 3% higher overall.Users % of Users Mid Final Dif CourseRegr Choice 34 20% 92% 61% -31% 81%Regr ->Regr 31 18% 92% 61% -32% 81%MaxAtt ->Regr 3 2% 89% 65% -24% 79%MaxAtt Choice 137 80% 94% 71% -23% 84%Regr ->MaxAtt 90 53% 94% 70% -24% 83%MaxAtt ->MaxAtt 47 27% 95% 73% -21% 85%Total 171 100% 94% 69% -25% 83%Table 5.13: Limiting Attempts: Student Preference and Performance©Sarah Foss (2022)Further analysis revealed that 74% of users that were assigned the regres-sion penalty during the midterm chose to use maximum attempts limitationon the final exam. Nearly all the users that were assigned the maximumattempts policy on their midterm chose that same policy on the final exam.It appears that although users could have unlimited attempts with feedbackon a final exam question with a regression penalty, the maximum attemptssubmission policy may have been considered to be the safer option as itdoesn’t penalize the overall marks for mistakes along the way.5.3 DiscussionThe AutoER system is a simple pedagogical tool designed to reinforcelearned concepts in an introductory database design course. For a learning615.3. DISCUSSIONtool to be deemed effective, there must be some motivation for studentsto choose to use the system over traditional methods. The first researchquestion (RQ1) examined whether students preferred to use the AutoERsystem compared to paper or other existing design software. Because of theunique features of AutoER, there was no direct experimental comparisonwith other existing software, however, students were given the option touse paper or any other system for all database design questions. The usagedata demonstrates that the students preferred using the AutoER systemcompared to other options with a high number of users for all assignmentsand examination questions.Another metric to gauge a learning tool’s effectiveness is the perceivedusability of the system. A usable system is easy to learn and allows thestudent to focus on the task at hand, rather than having to spend timefamiliarizing themselves with the tool itself. AutoER’s usability was ex-plored with the second research question (RQ2). The system usability sur-vey [Bro96] completed by the students revealed that the system had SUSscores of around 77% in both studies. The free-form comments were alsopositive, highlighting that interacting with text rather than drawing helpedthem learn concepts, especially with feedback. Although there were areasfor improvement noted, such as issues with the display of relationships ofthe diagram, overall, the survey results indicate that the system has goodusability and that the students perceived the tool to be effective and helpful.The third research question (RQ3) investigated if the AutoER system’sautomatic grading approach increased student performance on assignments.In both studies, student performance on assignment questions was higherthan the previous offering. The average assignment grades were in the range92-96% compared to about 85% in the 2020 fall offering of the course. How-ever, due to the differences in instructional environments, a definitive answercould not be reached. Although there was evidence of increased performanceand high system usage, further study is required to make a conclusive state-ment.The question auto generation capability is a unique feature not availablein any other system to the author’s knowledge. Although question genera-tion in general has been extensively researched, the domain-specific area ofdatabase design diagrams has been overlooked. RQ4 investigated whetherthis novel feature was effective for use in exam evaluations. More than 95%of students chose to use AutoER for the midterm exam question in bothstudies. Teaching assistants manually validated each question generatedand the associated auto-marking, and they noted that there were no issueswith unsolvable questions or marking that was inconsistent with established625.3. DISCUSSIONguidelines. They also remarked that each student answer requires abouttwo minutes to grade manually when submitting on paper or using othersoftware. The auto-marking saved time as the evaluation is immediate, andit eliminates bias, mistakes, and variation between TAs which often occursfor large classes. This indicates that AutoER’s question generation featureand auto evaluation approach is effective for use in examinations.The data from the first study revealed some distinctive behaviour pat-terns in the submission history of the users. These users were categorizedinto 4 profiles based on the number of regressions in the answers. Studentsin the Category 1 displayed positive submission behaviours such as highmarks on their first submission and a low number of overall submissions.Users in Category 4 had a higher number of average submissions and re-gressions throughout their history, potentially demonstrating trial-and-errorbehaviour. To attempt to limit this type of reliance on the auto-grader, at-tempt limiting submission policies were added to the system for the secondstudy. The hypothesis that student submission behavior would be positivelyaffected by submission limiting policies is confirmed by the collected data.Both strategies of either using a maximum number of submissions or a re-gression penalty are effective at reducing the number of submissions andregressions by 82% to 88% and increasing the time between submissions by146%. There is strong evidence supporting RQ5 on improving student be-haviors when interacting with automatic assessment systems. Specifically,the amount of trial-and-error behavior is greatly reduced.Comparing students based on regressions rather than submissions is abetter approach, regardless if the system penalizes students for regressionmistakes. Since a regression occurs when a student’s mark goes down, poorstudent behaviors of trial-and-error, exploiting system feedback, and randomguessing are much more likely when a student makes a mistake. If a studenthas multiple submissions, but the grade continues to increase, it is an in-teresting debate whether that is a poor behavior that should be prevented.However, many regressions clearly indicate a pattern of not understandingthe question sufficiently to improve the mark after every submission. Thenumber of regressions is closely related with the number of submissions andis a better indicator of student mistakes.The sixth research question (RQ6) was designed to identify what sub-mission limiting policy is the most effective. Although both maximum at-tempts and regression penalty show positive improvements overall, the max-imum attempt policy data displayed higher student performance and bettersubmission behavioural patterns. Student performance on the final examwas 10% higher with the maximum attempts limitation, and these students635.3. DISCUSSIONhad lower number of attempts and regressions. There were students withthe regression penalty limitation that still had a high number of regressionsand submissions, which are behaviours that should be avoided.High performing students excel on either submission policy as they usevery few submissions with limited mistakes. As more submissions are per-formed, more regressions occur for all students. However, the bounded num-ber of submissions results in students spending more time between submis-sions which decreases the chance of regressions.However, it should be noted that the difference in behaviour and per-formance was more notable in the final exam, where students chose theirpolicy, than in the midterm exam, where students were assigned their pol-icy. This indicates the possibility of a selection effect, where more of thehigher performing students may have chosen the maximum attempt policyand more of the weaker performing students may have chosen the regressionpenalty. It is possible that stronger-performing students may have viewedthe possibility of a permanent penalty to their final mark as a riskier choice,while weaker performing students may have viewed unlimited attempts asa better strategic choice. If this is the case, it could have lead to the higherperformance and improved behavioural patterns viewed in the data for themaximum attempt policy compared to the regression penalty policy for thefinal exam.When considering the load on the automatic assessment system, a maxi-mum attempts policy is the clear winner as students make fewer submissionsoverall and the absolute number of submissions is bounded. These resultsare in-line with the results in [BZHH21] that studied the impact of regres-sions for reducing the load on the assessment system. The additional resultscomparing with maximum attempts and student preference are importantcontributions.The final research question (RQ7) tested student preference for the sub-mission policy. Given the choice on a high stakes final exam where theirmarks were directly impacted, the vast majority (80%) selected maximumattempts as their preferred submission policy. It appears the maximum at-tempts restriction may have been perceived to be less risky by the studentsas it doesn’t permanently penalize the overall marks, however, there wasno survey data collected to confirm this assumption. Although it can bedefinitively stated that the students prefer the submission policy of maxi-mum attempts over the regression penalty, more research is required on theexploration of the motivation of the preference.Overall, the results demonstrate that the AutoER system is an easy touse, effective database design learning tool with novel features. Allowing645.3. DISCUSSIONstudents to interact with question text directly to build diagrams ensuresthat the system is able to provide consistent, accurate, and immediate eval-uation of diagrams regardless of class size. The ability to generate randomdomain-specific questions is also beneficial as it allows for the creation ofunique variants of questions in an examination where the potential of aca-demic dishonesty is high, particularly during online exams. The additionof attempt-limiting submission policies to the system encourages studentsto be thoughtful about their diagram creation and reduces over-reliance onthe automatic grader. Of the two submission policies implemented in thesystem, limiting the number of attempts has shown to be more effectiveand preferred by students, although both policies have a positive impact onstudent behaviour.65Chapter 6ConclusionThis thesis sought to demonstrate that utilizing the AutoER systemin an introductory database course for UML design questions can supportstudent learning and improve course administrators’ efficiency by provid-ing automated real-time feedback to students. A general background onthe challenge of learning diagram creation and software for database de-sign used in classrooms in database courses was explored in Section 2.1.Database design can be difficult for students at first as they must learn theproper syntax and learn to translate the written text into the appropriatediagram elements. Diagram creation can use ER or UML notation and canbe drawn on paper or created with the help of diagram software. Surveysof UML software indicate the preference for easy-to-use systems focusingon learning rather than commercial products used by experts. Whetherdesigned on paper or created with the help of software, consistent manualevaluation of the diagrams can be time-consuming, particularly for largeclasses, because students can use inconsistent naming and labels, can havetrouble understand fundamental modeling constraints, and can misinterpretthe question.As discussed in Section 2.2, automatic diagram evaluation with real-timefeedback allows for increased student engagement and increased consistencyand efficiency in grading for course administrators. Automatic generation ofquestions allows practicing until mastery and for online exams to reduce aca-demic dishonesty. Prior work examined automatic diagram marking usingimage processing, structural and semantic analysis, and machine learning.Although evaluation with some systems can be accurate to supplied an-swers within about 10%, challenges remain with achieving higher accuracy.Currently, there is no support for auto-generation of questions within thesesystems.The AutoER system architecture and available features are describedin Chapter 3. The web-based system allows students to build diagrams byinteracting with question text directly rather than drawing diagram com-ponents manually. This improves speed and guarantees consistent namingfor precise marking. As the student is building their answer, the system66continuously generates a visual representation of their answer.AutoER supports both instructor-created and automatically generatedquestions and is flexible allowing modification to its user interface, markingand feedback, and question generation parameters. The marking processis precise as students use standardized names when building a diagram andthe feedback from the auto-grader is provided to a student by request at anytime. Randomized questions are generated given parameters on the numberof entities, attributes, and relationships. The randomized questions havea single URL for use in a testing system, but each student gets their ownquestion variant. To combat relying on the auto-grader to answer questions,two types of penalties that restricted attempts were added to the system:maximum attempts, which limits the number of submissions and a regressionpenalty, which allows unlimited attempts but applies a permanent penaltywhenever a student’s mark goes down.The system was evaluated in two offerings of an undergraduate databasecourse in 2021 at the University of British Columbia Okanagan. Presentedin Chapter 4, the studies collected course data from two assignments, anautomatically generated question on a midterm exam, and an instructor-defined question on the final exam. Students could use the AutoER forany of these questions or answer questions on paper or with other UMLsoftware previously used. The first study allowed for unlimited attempts onall questions. The second study randomly assigned either maximum attemptlimitation or regression penalty submission policy on the midterm questionand allowed the student to choose their submission policy on the final exam.After using the system for several weeks, students completed a usabilitysurvey and provided feedback (see Appendix B.1) regarding their choice touse the system.The results of the studies were examined in Chapter 5. The usage andsurvey data demonstrates high student satisfaction compared to traditionalUML design question formats and a preference for using the software toimprove their learning outcomes. The generation of question and automaticevaluation proved to be effective for use in examinations with TAs notingthat the system reduced time spent marking and ensured consistency andaccuracy in evaluation.Data from the first study revealed some distinctive behavioural patternsin the submission history of the users with some students demonstratingan over-reliance on the auto-grader denoted with a high number of submis-sions and grade regressions. To attempt to limit this type of behaviour,two types of submission policies were added to the system, a regressionpenalty and a maximum attempt limitation, for the second study. With the676.1. LIMITATIONS AND THREATS TO VALIDITYsubmission policies enforced, the average number of regressions and sub-missions decreased significantly indicating that the users were being morethoughtful about their answers, relying on course knowledge rather than Au-toER’s evaluation feedback to answer questions. A comparison between thetwo submission policies revealed that student performance was higher andtrial-and-error behaviour was lower when the students used the maximumattempt limitation. The data also revealed that the majority of studentschose the maximum attempt submission policy when given an option.6.1 Limitations and Threats to ValidityThe research evaluation was designed to determine the usability of Au-toER for question generation and evaluation. The SUS evaluation demon-strated good usability metrics. There was no comparator tool taught in thecourse or an evaluation with any other auto-grading UML software. Sur-vey results demonstrate that the students perceived AutoER to be effective,but there is no data to determine how it compares to other auto-gradingsystems. Further, it is not possible to directly compare with such systemsas AutoER allows students to build answers using consistent terminologyrather than drawing the diagram and entering their own names. This ap-proach to question answering ensures precise marking according to criteriaunlike other approaches that have more matching ambiguity.Although the system statistics indicate a high number of student sub-missions and strong overall performance on the assignments and exams, thisdata is insufficient to argue conclusively there is increased engagement andlearning. Students clearly preferred using AutoER, and their grades werehigher than the previous offering without it. However, the differences ininstructional environments may have had an impact on this result.To evaluate the impact of the AutoER system on learning and engage-ment, students from the same environment could be randomly separatedinto two groups for two assignments: For the first assignment, one groupcould be designated an experimental group utilizing the AutoER system fordiagram creation and evaluation and the other a control group using tradi-tional methods as a baseline. For the second assignment, the groups couldswap roles. For each of the assignments, a pre-test and post-test could beconducted to measure the learning gains of the assignment. This could allowfor a more conclusive assessment of learning with the AutoER system. How-ever, this type of experimentation may offer an unfair advantage to somestudents and was deemed unsuitable for these studies.686.2. FUTURE WORKThe evaluation of submission policies was designed to determine theeffect of submission limits in realistic student experiences. The compar-ison between maximum attempts and regression was done within a singlecourse section allowing for an identical instructional environment. Althoughthe random assignment between the two submission limits on the midtermexam was not equal due to an error, there were sufficient numbers of stu-dents in both categories. Allowing students to select their approach on thefinal exam was fair and did not affect grouping in any way, however, it mayhave led to a selection effect, where higher performing students and lowerperforming students chose different policies based on their needs. The stu-dent self-selection data allows for statements on student preference in anactual situation, but without a survey after the final exam there is no datato determine why students selected one technique over the other.Comparing the data collected from the second course section with sub-mission limits with the first section with no limits is reasonable, with theacknowledgement that the two course sections are not identical instructionalenvironments. Although the instructor and all materials were the same, thefirst course section had a smaller class size. Ideally, some students may havebeen randomly selected to have no submission limits in the second coursesection, but this was determined to be an unreasonable advantage comparedto other students.6.2 Future WorkFuture work will extend the evaluation for more course offerings andintegrate the system into learning management systems using the LearningTools Interoperability (LTI) standard. The system will be improved by theaddition of an instructor’s frontend to allow for easier question creation.The question interface will also be improved to allow for more editing ofthe diagrams and improved visual representation. Although the system wastested for database design diagrams using UML, it is also applicable to otherdatabase diagram notations and UML diagrams for other use cases such asdeveloping an object-oriented class diagram. Other planned work includesdevelopment of interfaces for the aforementioned use-cases, explorations ofdifferent types of feedback, and the examination of variants of the submis-sion limits such as combining regressions and maximum attempts, and theexploration of different maximum attempt limits.Further evaluation of student submission behaviour in discipline-agnosticauto-grading systems is also future work planned beyond the further devel-696.2. FUTURE WORKopment of the AutoER system. Proposed work includes researching theimpact of different types of submission limitations on varying problem typesacross academic disciplines and exploring the correlation of performance andstudent submission policy preference.70Bibliography[AL17] Luciane Telinski Wiedermann Agner and Timothy C. Leth-bridge. A survey of tool use in modeling education. In 20thACM/IEEE International Conference on Model Driven Engi-neering Languages and Systems, MODELS 2017, Austin, TX,USA, September 17-22, 2017, pages 303–311. IEEE ComputerSociety, 2017. → pages 4[ALS19] Luciane Telinski Wiedermann Agner, Timothy C. Lethbridge,and Inali Wisniewski Soares. Student experience with softwaremodeling tools. Softw. Syst. Model., 18(5):3025–3047, 2019. →pages 4, 46[Auv15] Tapio Auvinen. Harmful study habits in online learning en-vironments with automatic assessment. In 2015 InternationalConference on Learning and Teaching in Computing and En-gineering, pages 50–57, 2015. → pages 11[AV03] Carl Alphonce and Phil Ventura. Quickuml: a tool to supportiterative design and code development. In Companion of the18th Annual ACM SIGPLAN Conference on Object-OrientedProgramming, Systems, Languages, and Applications, OOP-SLA 2003, October 26-30, 2003, Anaheim, CA, USA, pages80–81. ACM, 2003. → pages 6[BAK19] Weiyi Bian, Omar Alam, and Jo¨rg Kienzle. Automated grad-ing of class diagrams. In 22nd ACM/IEEE International Con-ference on Model Driven Engineering Languages and SystemsCompanion, MODELS Companion 2019, Munich, Germany,September 15-20, 2019, pages 700–709. IEEE, 2019. → pages1, 9[BAK20] Weiyi Bian, Omar Alam, and Jo¨rg Kienzle. Is automated grad-ing of models effective? assessing automated grading of classdiagrams. In Proceedings of the 23rd ACM/IEEE International71BibliographyConference on Model Driven Engineering Languages and Sys-tems, MODELS ’20, page 365–376, New York, NY, USA, 2020.ACM. → pages 9, 31[BE15] Kevin Buffardi and Stephen H. Edwards. Reconsidering auto-mated feedback: A test-driven approach. In Proceedings of the46th ACM Technical Symposium on Computer Science Educa-tion, SIGCSE 2015, Kansas City, MO, USA, March 4-7, 2015,pages 416–420. ACM, 2015. → pages 1[BKM08] Aaron Bangor, Philip T. Kortum, and James T. Miller. Anempirical evaluation of the system usability scale. InternationalJournal of Human–Computer Interaction, 24(6):574–594, 2008.→ pages 39, 44[BMM20] Younes Boubekeur, Gunter Mussbacher, and Shane McIntosh.Automatic assessment of students’ software models using asimple heuristic and machine learning. In MODELS ’20:ACM/IEEE 23rd International Conference on Model DrivenEngineering Languages and Systems, Canada, 18-23 October,2020, Companion Proceedings, pages 20:1–20:10. ACM, 2020.→ pages 1, 8[Bro96] John Brooke. Sus: A quick and dirty usability scale. In Usabil-ity Evaluation in Industry, pages 189–194, 1996. → pages 39,62[BZHH21] Elisa L. A. Baniassad, Lucas Zamprogno, Braxton Hall, andReid Holmes. STOP THE (AUTOGRADER) INSANITY: re-gression penalties to deter autograder overreliance. In MarkSherriff, Laurence D. Merkle, Pamela A. Cutter, Alvaro E.Monge, and Judithe Sheard, editors, SIGCSE ’21: The 52ndACM Technical Symposium on Computer Science Education,Virtual Event, USA, March 13-20, 2021, pages 1062–1068.ACM, 2021. → pages 10, 11, 22, 50, 51, 52, 64[Che76] Peter P. Chen. The entity-relationship model - toward a unifiedview of data. ACM Trans. Database Syst., 1(1):9–36, 1976. →pages 4[CWZ18] Binglin Chen, Matthew West, and Craig B. Zilles. How muchrandomization is needed to deter collaborative cheating on72Bibliographyasynchronous exams? In Proceedings of the Fifth Annual ACMConference on Learning at Scale, London, UK, June 26-28,2018, pages 62:1–62:10. ACM, 2018. → pages 10[Edw04] Stephen H. Edwards. Using software testing to move studentsfrom trial-and-error to reflection-in-action. In Proceedings ofthe 35th SIGCSE Technical Symposium on Computer ScienceEducation, SIGCSE ’04, page 26–30, New York, NY, USA,2004. Association for Computing Machinery. → pages 11[EP08] Stephen H. Edwards and Manuel A. Pe´rez-Quin˜ones. Web-cat: automatically grading programming assignments. In Pro-ceedings of the 13th Annual SIGCSE Conference on Innovationand Technology in Computer Science Education, ITiCSE 2008,page 328. ACM, 2008. → pages 1, 7, 10, 14[FGF21] Emily Faulconer, J. C. Griffith, and H. Frank. If at first you donot succeed: student behavior when provided feedforward withmultiple trials for online summative assessments. Teaching inHigher Education, 26(4):586–601, 2021. → pages 12[GM20] Carlos R. Jaimez Gonza´lez and Jazmı´n Mart´ınez-Samora. Di-agrammer: A web application to support the teaching-learningprocess of database courses through the creation of E-R dia-grams. iJET, 15(19):4–21, 2020. → pages 6[Gro17] Object Management Group. Omg® unified modeling lan-guage® (omg uml®), 2017. Retrieved: 2021-08-13. → pages4[Has11] Robert W Hasker. Umlgrader: an automated class diagramgrader. Journal of Computing Sciences in Colleges, 27(1):47–54, 2011. → pages 8, 31[HBTN21] Georgiana Haldeman, Monica Babes-Vroman, Andrew Tjang,and Thu D. Nguyen. CSF: formative feedback in autograding.ACM Trans. Comput. Educ., 21(3):21:1–21:30, 2021. → pages11[HR11] Robert W Hasker and Mike Rowe. Umlint: Identifying de-fects in uml diagrams. In 2011 ASEE Annual Conference &Exposition, pages 22.1558.1 – 22.1558.14, 2011. → pages 873Bibliography[IAKS10] Petri Ihantola, Tuukka Ahoniemi, Ville Karavirta, and OttoSeppa¨la¨. Review of recent systems for automatic assessmentof programming assignments. In Proceedings of the 10th KoliCalling International Conference on Computing Education Re-search, Koli Calling ’10, page 86–93, New York, NY, USA,2010. Association for Computing Machinery. → pages 12[KKM06] Ville Karavirta, Ari Korhonen, and Lauri Malmi. On the use ofresubmissions in automatic assessment systems. Comput. Sci.Educ., 16(3):229–240, 2006. → pages 11[L+66] Vladimir I Levenshtein et al. Binary codes capable of correctingdeletions, insertions, and reversals. In Soviet physics doklady,volume 10, pages 707–710. Soviet Union, 1966. → pages 9[Lee21] Haden Hooyeon Lee. Effectiveness of real-time feedback andinstructive hints in graduate CS courses via automated gradingsystem. In SIGCSE ’21: The 52nd ACM Technical Symposiumon Computer Science Education, Virtual Event, USA, March13-20, 2021, pages 101–107. ACM, 2021. → pages 7[LSTA21] Madeleine Lor˚as, Guttorm Sindre, Hallvard Trætteberg, andTrond Aalberg. Study behavior in computing education—a sys-tematic literature review. ACM Trans. Comput. Educ., 22(1),oct 2021. → pages 10[MBF+18] Marina Marchisio, Alice Barana, Michele Fioravera, Sergio Ra-bellino, and Alberto Conte. A model of formative automaticassessment and interactive feedback for stem. In 2018 IEEE42nd Annual Computer Software and Applications Conference(COMPSAC), volume 01, pages 1016–1025, 2018. → pages 11[MDP20] Samiha Marwan, Anay Dombe, and Thomas W. Price. Un-productive help-seeking in programming: What it is and howto address it. In Proceedings of the 2020 ACM Conferenceon Innovation and Technology in Computer Science Education,ITiCSE ’20, page 54–60, New York, NY, USA, 2020. Associa-tion for Computing Machinery. → pages 11[MGF+20] Samiha Marwan, Ge Gao, Susan Fisk, Thomas W. Price, andTiffany Barnes. Adaptive immediate feedback can improvenovice programming engagement and intention to persist in74Bibliographycomputer science. In Proceedings of the 2020 ACM Confer-ence on International Computing Education Research, ICER’20, page 194–203, New York, NY, USA, 2020. Association forComputing Machinery. → pages 11[MKKN05] Lauri Malmi, Ville Karavirta, Ari Korhonen, and Jussi Nikan-der. Experiences on automatically assessed algorithm simu-lation exercises with different resubmission policies. J. Educ.Resour. Comput., 5(3):7–es, sep 2005. → pages 11[MNS+20] Hamza Manzoor, Amit Naik, Clifford A. Shaffer, Chris North,and Stephen H. Edwards. Auto-grading jupyter notebooks. InJian Zhang, Mark Sherriff, Sarah Heckman, Pamela A. Cut-ter, and Alvaro E. Monge, editors, Proceedings of the 51stACM Technical Symposium on Computer Science Education,SIGCSE 2020, Portland, OR, USA, March 11-14, 2020, pages1139–1144. ACM, 2020. → pages 7[MSR21] Mostafa Mohammed, Clifford A. Shaffer, and Susan H. Rodger.Teaching formal languages with visualizations and auto-gradedexercises. In SIGCSE ’21: The 52nd ACM Technical Sympo-sium on Computer Science Education, Virtual Event, USA,March 13-20, 2021, pages 569–575. ACM, 2021. → pages 7[Pie13] Vreda Pieterse. Automated assessment of programming as-signments. In Proceedings of the 3rd Computer Science Ed-ucation Research Conference on Computer Science EducationResearch, CSERC ’13, page 45–56, Heerlen, NLD, 2013. OpenUniversiteit, Heerlen. → pages 12[SBP+10] Josep Soler, Imma Boada, Ferran Prados, Jordi Poch, andRamo´n Fabregat. A formative assessment tool for conceptualdatabase design using UML class diagram. iJET, 5(3):27–33,2010. → pages 8[Sch20] Johannes Schildgen. Monster park - the entity-relationship-diagram learning game. In ER Forum, Demo and Posters 2020co-located with 39th International Conference on ConceptualModeling (ER 2020), Vienna, Austria, November 3-6, 2020,volume 2716 of CEUR Workshop Proceedings, pages 150–157.CEUR-WS.org, 2020. → pages 975Bibliography[STW13] Neil Smith, Pete G. Thomas, and Kevin G. Waugh. Automaticgrading of free-form diagrams with label hypernymy. In 2013Learning and Teaching in Computing and Engineering, LaT-iCE 2013, Macau, Macao, March 21-24, 2013, pages 136–142.IEEE Computer Society, 2013. → pages 7[SvdPSC19] Dave R. Stikkolorum, Peter van der Putten, Caroline Speran-dio, and Michel Chaudron. Towards automated grading ofUML class diagrams with machine learning. In Proceedings ofthe 31st Benelux Conference on Artificial Intelligence (BNAIC2019) and the 28th Belgian Dutch Conference on MachineLearning (Benelearn 2019), volume 2491 of CEUR WorkshopProceedings. CEUR-WS.org, 2019. → pages 7[TPE05] Scott A. Turner, Manuel A. Pe´rez-Quin˜ones, and Stephen H.Edwards. minimuml: A minimalist approach to UML diagram-ming for early computer science education. ACM J. Educ. Re-sour. Comput., 5(4):1:1–1:28, 2005. → pages 6[TWS06] Pete G. Thomas, Kevin G. Waugh, and Neil Smith. Usingpatterns in the automatic marking of er-diagrams. In Pro-ceedings of the 11th Annual SIGCSE Conference on Innovationand Technology in Computer Science Education, ITiCSE 2006,Bologna, Italy, June 26-28, 2006, pages 83–87. ACM, 2006. →pages 1, 7[WHZ15] MatthewWest, Geoffrey L. Herman, and C. Zilles. Prairielearn:Mastery-based online problem solving with adaptive scoringand recommendations driven by machine learning. In 122ndASEE Annual Conference and Exposition, pages 26.1238.1 –26.1238.14, 2015. → pages 1, 10, 14[Zak04] Konstantin Zakharov. Feedback micro-engineering in eer-tutor.2004. → pages 676Appendices77 Version: 1.1 June 29, 2021 1 Project Title: AutoER - ER/UML Diagram Designer and Evaluator Human Ethics - H21-01600 Principal Investigator: Dr. Ramon Lawrence Email: ramon.lawrence@ubc.ca Department of Computer Science, Physics, Mathematics, Statistics UBCO Co-Investigator: Sarah Foss Email: sarafoss@mail.ubc.ca INFORMED CONSENT: COURSE ADMINISTRATOR PARTICIPATION Introduction You are invited to participate in an evaluation of an experimental educational tool for generating and assessing ER diagrams. As a participant in this study, you will be asked to complete a questionnaire and interact with this tool as needed. This statement contains information about the present study that is intended to help you decide whether you wish to participate in it. You may refuse to sign this form and not participate in this study. You should be aware that even after you signed, you are free to withdraw at any time. If you withdraw from this study, it will not affect your relationship with this unit, the people involved in the study, the services it may provide to you, or the University of British Columbia Okanagan. If you have any questions about this study before, during, or after your participation, please feel free to direct them to: Dr. Ramon Lawrence Email: ramon.lawrence@ubc.ca Study Objectives The objective of this work is to explore the utility of allowing students to interact with entity-relationship diagram questions in a visual way that will allow the system to generate a diagram that can be assessed automatically and consistently. Procedure: ER Diagram Designer and Evaluator Total expected time: 3 hours 1. The co-investigator will visit your class to explain the study and the consent form to the students. 2. For the assignments and exams that require students to draw ER diagrams, you will log into the tool and enter the questions for the assignment/exam and the answers in the required format. 3. As part of the course, your students will work on questions related to ER diagrams in the software directly. 4. After their work has been submitted, you will retrieve their answer and manually grade the student’s work as usual. If necessary, you may override the feedback or grades generated by the system. 5. At the end of the course, you will be asked to complete a short questionnaire based on the utility and usability of the tool. This is to help us identify what has been useful for you and what changes we need to make in the future. Appendix AConsent FormsA.1 Administrator Consent78 Version: 1.1 June 29, 2021 2 Risks There is a risk that the software will malfunction and negatively impact or delay student's grades. To mitigate this, the course administrator will manually mark each submission and, if necessary, they will have the ability to override marks and add feedback for the students in the software Benefits The student participants may gain a better understanding of the concepts of entity-relationship diagrams through immediate visual representation. They will also benefit from consistent marking from the software. The course administrators will benefit from the time-reducing auto marking capabilities of the software. Privacy and confidentiality Questionnaires are anonymous. Questionnaire data, consent data, and information collected from the system will all be encrypted and stored separately. Since we are only interested in average responses and behaviors, reports about individual students will not be released. Participants have a right to access their own results as long as their names and numeric tags are still available. The data obtained from this study will be retained for at least 5 years after publication in secured, electronic form, which only researchers associated with this research will have access to. No audio or video tapes will be used. General results of the research study may be available through publications. A summary of the results will be made available to participants upon request. Concerns about participants’ rights If you have any concerns or complaints about your rights as a research participant and/or your experiences while participating in this study, contact the Research Participant Complaint Line in the UBC Office of Research Ethics toll free at 1-877-822-8598 or the UBC Okanagan Research Services Office at 250-807-8832. It is also possible to contact the Research Complaint Line by email (RSIL@ors.ubc.ca). When doing so, include the study number H21-01600 when contacting their staff so they can better assist you. Participant Certification I have read this Consent and Authorization form. I have had the opportunity to ask, and I have received answers to, any questions I had regarding the study and the use and disclosure of information about me for the study. I agree to take part in this study as a research participant. By my signature I affirm that I have received a copy of this Consent form. Participant Name: Date: Participant Signature: 79 Version: 1.1 June 29, 2021 1 Project Title: AutoER - ER/UML Diagram Designer and Evaluator Human Ethics - H21-01600 Principal Investigator: Dr. Ramon Lawrence Email: ramon.lawrence@ubc.ca Department of Computer Science, Physics, Mathematics, Statistics UBCO Co-Investigator: Sarah Foss Email: sarafoss@mail.ubc.ca INFORMED CONSENT: STUDENT PARTICIPATION Introduction You are invited to participate in an evaluation of an experimental educational tool for generating and assessing ER diagrams. As a participant in this study, you will be asked to complete a questionnaire and interact with this tool as needed. This statement contains information about the present study that is intended to help you decide whether you wish to participate in it. You may refuse to sign this form and not participate in this study. You should be aware that even after you signed, you are free to withdraw at any time. If you withdraw from this study, it will not affect your relationship with this unit, the people involved in the study, the services it may provide to you, or the University of British Columbia Okanagan. If you have any questions about this study before, during, or after your participation, please feel free to direct them to: Dr. Ramon Lawrence Email: ramon.lawrence@ubc.ca Study Objectives The objective of this work is to explore the utility of allowing students to interact with entity-relationship diagram questions in a visual way that will allow the system to generate a diagram that can be assessed automatically and consistently. Procedure: ER Diagram Designer and Evaluator Expected course work time: 4 hours Expected additional time: 0.5 hours Total expected time: 4.5 hours 1. As part of the course, you will work on questions related to ER diagrams in the software directly. 2. After your work has been submitted and automatically graded, your instructor will also manually grade your answers to verify they were marked correctly. If necessary, they are able to override the feedback or grades. 3. At the end of the course, you will be asked to complete a short questionnaire based on the utility and usability of the tool. This is to help us identify what has been useful for you and what changes we need to make in the future. Risks There is a risk of the software malfunctioning and negatively impacting your grades in the course. To mitigate this, the course administrator will also manually mark each submission and, if necessary, they will have the ability to override marks and add feedback in the software. There is a risk that you may become frustrated with the software. To mitigate this, you will have the option of using traditional methods of creating entity-relationship diagrams at any time. A.2. Student ConsentA.2 Student Consent80 Version: 1.1 June 29, 2021 2 Benefits You may gain a better understanding of the concepts of entity-relationship diagrams through immediate visual representation. You will also benefit from consistent marking from the software. Privacy and confidentiality Questionnaires are anonymous. Questionnaire data, consent data, and information collected from the system will all be stored separately and encrypted. Since the interest of this study is to identify average responses and behavior of the entire group of participants, you will not be identified in any way in any written reports of this research. Participants have a right to access their own results so long as their names and numeric tags are still available. The data obtained from both components of the study will be retained for at least 5 years after publication in secured, electronic form, which only researchers associated with this research will have access to. No audio or video tapes will be used. General results of the research study may be available through publications. A summary of the results will be made available to participants upon request. Access to course work and associated grades By checking the boxes below, you are providing consent for the researchers in this study to analyze your entity-relationship diagram course work and associated grades. The grades will only be used for research analysis purposes without including any of the participants’ personal information. Concerns about participants’ rights If you have any concerns or complaints about your rights as a research participant and/or your experiences while participating in this study, contact the Research Participant Complaint Line in the UBC Office of Research Ethics toll free at 1-877-822-8598 or the UBC Okanagan Research Services Office at 250-807-8832. It is also possible to contact the Research Complaint Line by email (RSIL@ors.ubc.ca). When doing so, include the study number H21-01600 when contacting their staff so they can better assist you. Participant consent and signature I have read this Consent form. I have had the opportunity to ask, and I have received answers to, any questions I had regarding the study and the use and disclosure of information about me for the study. I agree to take part in this study as a research participant. By my signature I affirm that I have received a copy of this Consent form. Yes No I am willing to provide access to my submitted entity-relationship diagram course work Yes No I am willing to provide access to my course grades associated with the submitted work. Whether or not you choose to participate, you will receive the same assignments and exam questions in this course. Your decision to participate or not participate in this study will in no way impact your grades or your relationship with UBC. Participant Name: Date: Participant Signature: 81 Version: 1.2 July 27, 2021 1 Project Title: AutoER - ER/UML Diagram Designer and Evaluator Human Ethics - H21-01600 Principal Investigator: Dr. Ramon Lawrence Email: ramon.lawrence@ubc.ca Department of Computer Science, Physics, Mathematics, Statistics UBCO Co-Investigator: Sarah Foss Email: sarafoss@mail.ubc.ca Participant ID: _______________________________ Date: ___________________________ Questionnaire - AutoER – ER/UML Diagram Designer and Evaluator Software Thank you for your voluntary participation in the AutoER - ER/UML Diagram Designer and Evaluator research study under the direction of Dr. Ramon Lawrence. In order to improve the software used in this study, this questionnaire has been developed to gather feedback regarding your experiences using the software. We value your honest and detailed responses. Your responses are confidential. Strongly Disagree Somewhat Disagree Neutral Somewhat Agree Strongly Agree 1. I think I would like to use this tool frequently. 2. I found the tool unnecessarily complex. 3. I thought the tool was easy to use. 4. I think that I would need the support of a technical person to be able to use this system. 5. I found the various functions in this tool were well integrated. 6. I thought there was too much inconsistency in this tool. 7. I would imagine that most people would learn to use this tool very quickly. 8. I found the tool very cumbersome to use. 9. I felt very confident using the tool. 10. I needed to learn a lot of things before I could get going with this tool. Appendix BSurveysB.1 Student Survey82 Version: 1.2 July 27, 2021 2 11. Select all that apply. I used the software for the following tasks: - At least one lab - More than one lab - Midterm exam - Final exam - Did not use the software 12. If applicable, why did you choose to keep using the software? 13. If applicable, why did you choose to stop using the software? 14. What aspects of the software did you find helpful? 15. What additional features would you like to see with this software? 16. Please provide any general comments about the software and its use in the course. 83 Version: 1.1 June 29, 2021 1 Project Title: AutoER - ER/UML Diagram Designer and Evaluator Human Ethics - H21-01600 Principal Investigator: Dr. Ramon Lawrence Email: ramon.lawrence@ubc.ca Department of Computer Science, Physics, Mathematics, Statistics UBCO Co-Investigator: Sarah Foss Email: sarafoss@mail.ubc.ca Participant ID: _______________________________ Date: ___________________________ Questionnaire - AutoER – ER/UML Diagram Designer and Evaluator Software Thank you for your voluntary participation in the AutoER - ER/UML Diagram Designer and Evaluator research study under the direction of Dr. Ramon Lawrence. In order to improve the software used in this study, this questionnaire has been developed to gather feedback regarding your experiences using the software. We value your honest and detailed responses. Your responses are confidential. 1. Select all that apply. I used the software for the following tasks: - At least one lab - More than one lab - Midterm exam - Final exam - Did not use the software 2. Select one. I found the software easy to use. - Strongly agree - Somewhat agree - Neither agree nor disagree - Somewhat disagree - Strongly disagree 3. Select one. I found the software useful. B.2. Administrator SurveyB.2 Administrator Survey84 Version: 1.1 June 29, 2021 2 - Strongly agree - Somewhat agree - Neither agree nor disagree - Somewhat disagree - Strongly disagree 4. Select one. The software reduced the time spent grading ER diagrams. - Strongly agree - Somewhat agree - Neither agree nor disagree - Somewhat disagree - Strongly disagree 5. If applicable, why did you choose to keep using the software? 6. If applicable, why did you choose to stop using the software? 7. What aspects of the software did you find helpful? 8. What additional features would you like to see with this software? 9. Do you have an estimate on how much time the software saved for grading? 10. Please provide any general comments about the software and its use in the course. 85Appendix CAssignmentsC.1 Assignment 1Building ER diagrams using an UML modeling toolThis lab designs ER diagrams in UML notation. Students participatingin the AutoER study can use the AutoER Designer and Evaluator softwarethat performs real-time marking and feedback.It is also possible to use a variety of drawing software such as astah.Note: Astah no longer has a free community version. Request a studentlicense or use the 30-day trial. diagrams.net is an online drawing tool thatcan also be used but it does not have as good support for database modelingin UML.86C.1. Assignment 1C.1.1 Question 1 (10 marks)(a) Question(b) AnswerFigure C.1: Assignment 1 Question 1©Sarah Foss (2022)87C.1. Assignment 1C.1.2 Question 2 (10 marks)(a) Question(b) AnswerFigure C.2: Assignment 1 Question 2©Sarah Foss (2022)88C.2. Assignment 2C.2 Assignment 2C.2.1 Question 1 (15 marks)Note: In addition to the diagram, submit a document containing themapping to the relational model. SQL CREATE TABLE statements arenot required. Just provide relation names, attributes, keys and foreign keys.89C.2. Assignment 2(a) Question(b) AnswerFigure C.3: Assignment 2 Question 1©Sarah Foss (2022)90C.2. Assignment 2C.2.2 Question 2 (35 marks) - Project DeliverableThe project will build an online store like Amazon.com selling whateverproducts you want. The first step is to develop a database design and convertthat design into SQL Data Definition Language (DDL).Deliverables:1. Draw the ER/UML diagram for this database. (21 marks)2. Convert the UML diagram into SQL DDL. Make sure to define primarykeys and foreign keys. Your SQL DDL must run on either MySQL,Microsoft SQL Server or Oracle APEX (preferable). (11 marks)3. Determine what you are going to sell in your store. Write a missionstatement (1 mark) and executive summary paragraph (2 marks) de-scribing your idea.Items to submit: your UML diagram, a text file containing your SQL DDL,a document with your mission statement and executive summary.91C.2. Assignment 2(a) Question(b) AnswerFigure C.4: Assignment 2 Question 2©Sarah Foss (2022)92
Thesis/Dissertation
eng
Computer Science
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Lawrence, Ramon
AutoER : a system for the automatic generation and evaluation of UML database design diagrams
Text