Science, Irving K. Barber Faculty of (Okanagan)
Computer Science, Mathematics, Physics and Statistics, Department of (Okanagan)
DSpace
Hunt, Bridgette
This thesis explores the role of generative artificial intelligence (AI), specifically chatbots, in higher education by examining student sentiment, engagement, and the impact on instructor workload. The study focuses on the HelpMe system, a web application help center incorporating a Retrieval- Augmented Generation (RAG)-based chatbot adapted from the ChatEd architecture. Implemented in an upper-year computer science course at UBC Okanagan, the chatbot was designed to assist students with asynchronous questions using a knowledge base built from course materials and instructorprovided answers. To ascertain the chatbot’s effectiveness and students’ perception of it, survey data was collected from 83 participants, accompanied by backend system usage data and instructor-reported email correspondence. The survey gauged attitudes toward AI in education, willingness to use chatbots, preference for course-provided versus external tools, and comparative usefulness against traditional help-seeking methods like email and office hours. Results indicate that a majority of students are open to using chatbots for academic assistance, mainly if provided directly by the institution. While many participants responded neutrally to questions about the chatbot’s capabilities, a significant portion acknowledged its utility and expressed support for broader integration of such tools in other courses. However, usage data revealed that most students interacted with the chatbot only once or twice, with a sharp decline in engagement after the initial weeks of the term. Despite its potential, there was insufficient evidence to determine if the chatbot significantly reduced the volume of student emails or demands on instructor time. The findings suggest that while students are receptive to AI-based tools in the classroom, sustained engagement requires further improvements in usability, integration, and trust. The HelpMe system’s chatbot may serve best as a supplemental resource rather than a replacement for traditional support channels. Ultimately, this thesis highlights both the promise and limitations of generative AI in education, offering insight into how such systems can be refined to better support students and educators alike.
Chatbots in the Classroom: A Surveyof Student Sentiment Toward AI inEducationbyBridgette HuntA THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFB.SC. COMPUTER SCIENCE HONOURSinIrving K. Barber Faculty of Science(Computer Science)THE UNIVERSITY OF BRITISH COLUMBIA(Okanagan)April 2025© Bridgette Hunt, 2025AbstractThis thesis explores the role of generative artificial intelligence (AI),specifically chatbots, in higher education by examining student sentiment,engagement, and the impact on instructor workload. The study focuses onthe HelpMe system, a web application help center incorporating a Retrieval-Augmented Generation (RAG)-based chatbot adapted from the ChatEd ar-chitecture. Implemented in an upper-year computer science course at UBCOkanagan, the chatbot was designed to assist students with asynchronousquestions using a knowledge base built from course materials and instructor-provided answers.To ascertain the chatbot’s effectiveness and students’ perception of it,survey data was collected from 83 participants, accompanied by backend sys-tem usage data and instructor-reported email correspondence. The surveygauged attitudes toward AI in education, willingness to use chatbots, pref-erence for course-provided versus external tools, and comparative usefulnessagainst traditional help-seeking methods like email and office hours.Results indicate that a majority of students are open to using chatbotsfor academic assistance, mainly if provided directly by the institution. Whilemany participants responded neutrally to questions about the chatbot’s ca-pabilities, a significant portion acknowledged its utility and expressed sup-port for broader integration of such tools in other courses. However, usagedata revealed that most students interacted with the chatbot only once ortwice, with a sharp decline in engagement after the initial weeks of theterm. Despite its potential, there was insufficient evidence to determine ifthe chatbot significantly reduced the volume of student emails or demandson instructor time.The findings suggest that while students are receptive to AI-based toolsin the classroom, sustained engagement requires further improvements inusability, integration, and trust. The HelpMe system’s chatbot may servebest as a supplemental resource rather than a replacement for traditionalsupport channels. Ultimately, this thesis highlights both the promise andlimitations of generative AI in education, offering insight into how suchsystems can be refined to better support students and educators alike.iiTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vChapter 1: Introduction . . . . . . . . . . . . . . . . . . . . . . . 1Chapter 2: Background . . . . . . . . . . . . . . . . . . . . . . . 42.1 Generative Artificial Intelligence . . . . . . . . . . . . . . . . 42.2 Retrieval-Augmented Generation . . . . . . . . . . . . . . . . 62.3 Artificial Intelligence in Education (AIEd) . . . . . . . . . . . 8Chapter 3: Methodology . . . . . . . . . . . . . . . . . . . . . . 123.1 Questionnaire . . . . . . . . . . . . . . . . . . . . . . . . . . . 133.2 Chatbot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13Chapter 4: Results . . . . . . . . . . . . . . . . . . . . . . . . . . 174.1 Survey Results . . . . . . . . . . . . . . . . . . . . . . . . . . 174.2 Chatbot Usage & Performance . . . . . . . . . . . . . . . . . 244.3 Demands on Instructor Time . . . . . . . . . . . . . . . . . . 27Chapter 5: Conclusion . . . . . . . . . . . . . . . . . . . . . . . . 295.1 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295.2 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315.3 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . 31Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32iiiList of TablesTable 3.1 Survey Questions regarding the HelpMe system’s Chat-bot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14Table 3.2 Survey Questions regarding the HelpMe system’s Asyn-chronous Question Forum . . . . . . . . . . . . . . . . 15ivList of FiguresFigure 3.1 An example of a document chunk in the chatbot’svector store . . . . . . . . . . . . . . . . . . . . . . . . 16Figure 4.1 Responses to the survey question: “I am willing touse a chatbot or any available AI system for help ina course.” . . . . . . . . . . . . . . . . . . . . . . . . . 18Figure 4.2 Responses to the survey question: “Chatbot answerswere relevant and accurate.” . . . . . . . . . . . . . . 18Figure 4.3 Responses to the survey question: “Chatbot conver-sation was human-like.” . . . . . . . . . . . . . . . . . 19Figure 4.4 Responses to the survey question: “The chatbot in-cluding source material links was helpful.” . . . . . . 19Figure 4.5 Responses to the survey question: “The chatbot as-sisted me in meeting my learning outcomes.” . . . . . 19Figure 4.6 Responses to the survey question: “Chatbots shouldbe used in other courses.” . . . . . . . . . . . . . . . . 19Figure 4.7 Responses to the survey question: “I would prefer touse a course-provided chatbot rather than a privateoption like ChatGPT.” . . . . . . . . . . . . . . . . . 20Figure 4.8 Responses to the survey question: “I would rather usethe chatbot than use office hours or email to commu-nicate with teaching staff.” . . . . . . . . . . . . . . . 20Figure 4.9 Responses to the survey question: “I would recom-mend a struggling peer to use the chatbot for assis-tance.” . . . . . . . . . . . . . . . . . . . . . . . . . . 20Figure 4.10 Responses to the survey question: “How often didyou use the chatbot?” . . . . . . . . . . . . . . . . . . 21Figure 4.11 Responses to the survey question: “How often didyou use the Anytime Question feature?” . . . . . . . 21Figure 4.12 Responses to the survey question: “I prefer email overasking Anytime Questions via HelpMe.” . . . . . . . . 22vLIST OF FIGURESFigure 4.13 Responses to the survey question: “Public AnytimeQuestion answers were helpful.” . . . . . . . . . . . . 22Figure 4.14 Responses to the survey question: “A single placefor help (office hours, chatbot, Anytime Questions) isbeneficial.” . . . . . . . . . . . . . . . . . . . . . . . . 22Figure 4.15 Responses to the survey question: “How would yourate the quality and relevance of the answers providedby the AI in Anytime Questions?” . . . . . . . . . . . 23Figure 4.16 Responses to the survey question: “I asked more ques-tions to the chatbot or Anytime Questions than I sentemails in the course.” . . . . . . . . . . . . . . . . . . 23Figure 4.17 An aggregation of the survey responses, determinedto be positive, negative, or neutral toward the chatbotand AI. . . . . . . . . . . . . . . . . . . . . . . . . . . 23Figure 4.18 Distribution of interactions between students and chat-bot . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24Figure 4.19 Student questions for the chatbot broken down bytype (course, technical, material, irrelevant, and ma-licious) separated by re-used answers. . . . . . . . . . 25Figure 4.20 Chatbot interactions throughout the term . . . . . . 26Figure 4.21 Percentage of unique answers vs. verified (stored)answers . . . . . . . . . . . . . . . . . . . . . . . . . . 27Figure 4.22 Helpfulness vs. harmfulness of chatbot answers givenrelevance . . . . . . . . . . . . . . . . . . . . . . . . . 27Figure 4.23 Reasons for student emails to professor . . . . . . . . 28Figure 4.24 Time (minutes) needed to address emails throughoutthe term . . . . . . . . . . . . . . . . . . . . . . . . . 28viChapter 1IntroductionArtificial intelligence (AI) is an umbrella term in computational sciencesused to refer, in general, to the development of computer algorithms thatare capable of performing tasks that would otherwise require human intelli-gence. The increasing usage of artificial intelligence (AI) tools has becomea demonstrable issue in higher education scenarios, with the public releaseof ChatGPT1 in 2022, students at all levels of education have been usingit, among other Large Language Models (LLMs). Chatbots, with their abil-ity to generate comprehensible text, can act as assistants in numerous usecases, including within education. However, many universities have imposedrestrictions on the usage of generative AI tools to curb student usage andaddress the pedagogical concerns of teachers that students may fail to meetlearning outcomes due to reliance on AI in their academic pursuits.Successful implementations of generative AI began with the introductionof the Transformer architecture [25], which utilizes the attention mechanism[2], a process that finds patterns with high correlation in sentences com-posed of natural language. The Transformer architecture is an evolutionfrom previous encoder-decoder architectures, which used recurrent neuralnetworks (RNNs) [25]. Performing natural language processing (NLP) withRNNs was marked by the structures being difficult to parallelize, leading tosuboptimal architectures that made rapid NLP infeasible. The Transformerarchitecture uses a feed-forward neural network architecture, making use ofthe self-attention mechanism [25]. Later adaptations of the Transformer ar-chitecture are the basis of such technologies as the Generative Pre-trainedTransformer (GPT) [18], and other technologies such as the Bi-directionalEncoder Representations from Transformers (BERT) [7].Chatbots, a form of generative AI, specifically an application of LargeLanguage Models, are question-and-answer machine learning models thathave seen skyrocketing usage by the public, and, prevalently, by students.Large Language Models are a type of machine-learning model that aretrained on a massive corpus of text-based information, such as books, web1”Introducing ChatGPT” OpenAI, 20221pages, and various other sources. The fundamental basis of all modern largelanguage models is the Transformer architecture by Vaswani et al. [25].Recent developments in generative AI, primarily in chatbot improve-ment, have led to increased specificity in query responses and a better under-standing of the context. Retrieval-Augmented Generation (RAG) [10, 21, 27]is a method developed to improve an LLMs response to queries. By pro-viding domain-specific knowledge to an LLM through some means, RAGimbues the response with more relevant information. With RAG, questionsto the chatbot are used to search through a vector store with embedded‘chunks’ of knowledge, composed of sectioned parts of a source document,in an attempt to improve the answers generated by the chatbot [10].A motivating factor behind the development of RAG techniques is thatgenerative AI has a propensity to be error-prone, being capable of givingmisleading or harmful answers to users’ queries [13, 30, 31]. This is espe-cially of concern in the domain of education, where false information can bedetrimental to students utilizing AI tools for course assistance [12]. Otherconcerns are that the trained knowledge base of the AI can become out ofdate [27], with the chatbot being unable to give correct information beyonda certain time frame, especially for time-sensitive information.In the domain of education, the question of whether generative AI isan acceptable and/or useful technological addition has been questioned bymany researchers, with many studies positing that it can be effective inproviding personalized assistance to students while simultaneously reducingdemands on instructor time [9, 29]. Chatbots can provide on-demand an-swers and synchronous, one-on-one help sessions with students, which areotherwise difficult to facilitate in an era where students outnumber teachersone hundred to one [19, 29].It has been promoted by many researchers that generative artificial in-telligence can provide a valuable service to instructors and students alike.However, the adoption of generative AI tools by instructors has been slow,with one study by Wang et al. [28] suggesting that their lack of confidencein using these new tools effectively is partly responsible. In other studies,researchers have found that students themselves are more likely to utilizeartificial intelligence for assistance if they have greater trust in it, and aresimultaneously more likely to view it as more useful than otherwise [1, 17].Many studies have been conducted using a structural equation modeling(SEM) approach to understand the adoption behaviour of AI technologies,especially by students [1, 15, 17]. Other studies have used similar approachesto determine the reasons that instructors may or may not adopt AI tech-nologies [28]. In general, studies that use an SEM approach will often use2the Technology Acceptance Model [5, 6], which attempts to model an in-dividual’s acceptance of technology with four variables: behavioural intent(BI), attitude towards use (ATU), perceived usefulness (PU), and perceivedease of use (PEU).Several implementations of artificial intelligence assistants have been de-veloped and introduced in the classroom. In the English First Language(EFL) context, artificial intelligence improved sentence structure and ap-proval by participating learners [11]. Other researchers have developed tech-niques that utilize retrieval-augmented generation and similar techniques tocreate chatbots that are capable of answering queries with more context[21, 27]. Introducing such augmentations to an LLM can create highly spe-cialized agents for student assistance.Generative artificial intelligence has been extensively tested to ensureits accuracy and usability, especially by model developers. Human-centeredstudies are often more focused on whether potential users intend to adoptthese technologies [1, 15, 17, 28], and what factors can be identified to impactthe behavioural intent of an individual to adopt AI in their routine. There isa lack of emphasis on implementation and feedback testing from actual users,especially in the higher education context. This thesis aims to investigateand answer questions regarding the usability and application of LLMs andchatbots in university-level education:1. Do students accept and engage with the LLM or use alternate helpmethods?2. Do demands on instructor time change when utilizing a chatbot forstudent assistance?Students in an upper-year computer science course have interacted witha chatbot deployed within a live-service help system [26, 27] with their in-teractions with the chatbot cataloged. An optional survey with a series ofquestions related to this chatbot was offered to obtain specific user feedbackand sentiment toward the technology. The instructor of this course docu-mented the emails that they had received throughout the term, includinginformation regarding the date these emails were addressed, how long ittook to address the concern within the email, and the context area that theemail’s concerns corresponded to.3Chapter 2Background2.1 Generative Artificial IntelligenceArtificial intelligence with generative capabilities has been a staple ofdeep-learning contributions for the past decade. In Vaswani et al.’s [25]landmark contribution “Attention Is All You Need,” the Transformer wasintroduced. This pioneering technology allowed research in deep learningnetworks to move beyond recurrent and convolutional networks, instead fo-cusing on improving the attention mechanism. The concept of utilizing neu-ral networks to convert natural language strings to a fixed-length vector rep-resentation was demonstrated by multiple studies, particularly the studies ofSutskever et al. [22] and Cho et al. [4], known commonly as a “sequence-to-sequence” (Seq2seq), and specifically a form of encoder-decoder operationfor natural language processing (NLP).The Attention mechanism, introduced by Bahdanau et al. [2], is de-signed to find patterns with high correlation in natural language sentences.Bottlenecks that would otherwise occur in sentence-to-vector architectures[22] are avoided by instead encoding an input sentence into a sequence ofvectors, typically a vector encoding of the words within the sentence, andchoosing only a subset of these during decoding. This prevents issues re-lated to the generated fixed-length vectors being too large for the model tosupport.A primary issue of the original encoder-decoder architectures found inearly works is that they utilized recurrent neural networks (RNNs) to per-form their operations. RNNs are difficult to parallelize, leading to a bot-tleneck in performance [25]. However, with the introduction of the Trans-former, instead of RNNs, a feed-forward neural network is utilized. Boththe encoder and decoder stacks of the transformer architecture make use ofan intra-attention, or self-attention, mechanism.In the encoder stack of the transformer, the self-attention layers dependentirely on the previous layer’s outputs, with the current layer being able toattend to all positions of the previous layer. The self-attention layers in thedecoder stack are limited to interacting only with the matching positions in42.1. GENERATIVE ARTIFICIAL INTELLIGENCEprevious layers, with this masking ensuring predictions at a given positioni only depend on known outputs at positions less than i [25]. The generalpurpose of the feed-forward neural network is to make predictions, subse-quently performing back-propagation to adjust the network’s layers’ weightsgiven the error calculated from the predicted values [25].The Attention mechanism and Transformer architecture, among otheradvances in NLP, allow the network, composed of decoder-only Transform-ers, to perform robust natural language generation. The underlying mecha-nisms of self-attention make the network capable of capturing complex re-lationships between words in various contexts [2]. Early NLP architecturesstruggled with performance due to the use of difficult-to-parallelize RNNs.Modern transformer architectures perform prompt encoding and decodingsteps through feed-forward network layers in parallel for a more efficientalgorithmic response of the model [25].Various implementations of the transformer architecture have been em-ployed to facilitate natural language generation by machine learning models;the most significant of these is the models developed by OpenAI, specifically,the models using the Generative Pre-trained Transformer (GPT) architec-ture. OpenAI’s GPT models, starting with GPT-1 [18], are the foremostfamily of LLMs. OpenAI has improved upon the transformer architectureand attention mechanism, especially in terms of efficiency. The implemen-tation of a “sparse transformer,” specifically, with an improved “sparse” at-tention algorithm broke records with a 30 times performance improvementin extracting information from inputs [3].Generative pre-training is a fine-tuning technique, as opposed to feature-based approaches, which uses a selection of weights from a pre-trained modelto train new, additional task-specific models [7]. Fine-tuning techniquesfine-tune all pre-trained parameters, rather than focusing on task-specificparameters as in feature-based approaches. Generative pre-training is asemi-supervised machine learning technique, which combines unsupervisedand supervised machine learning to build generative AI models. The gen-erative pre-training stage is unsupervised, where a large set of data is usedto initialize the weights of the network, and is followed by a supervised fine-tuning stage to adapt network parameters to the task the model is intendedto perform [18].While GPT models are exclusively developed by OpenAI, contempo-raries which use similar architectures exist. For example, Meta’s LLaMAmodels are based upon the GPT architecture, as introduced by Touvronet al. [24]. Alterations from the GPT architecture in the first version ofLLaMA included the addition of pre-normalization and alteration to em-52.2. RETRIEVAL-AUGMENTED GENERATIONbedding positioning and the activation function used by the network [24].Google researchers Devlin et. al [7] introduced the Bidirectional EncoderRepresentations from Transformers (BERT) in 2019, the year after Radfordet al.’s [18] pioneering work in the development of GPT models. The BERTmodel likewise uses generative pre-training but diverges from OpenAI’s im-plementations of the task by utilizing bi-directional transformer encoders,based on the original specification of Vaswani et al [25]. BERT utilizes twounsupervised tasks in its pre-training stage, which differs from the GPTimplementation by OpenAI, which performs a single unidirectional task. Inthe fine-tuning stage, BERT utilizes the self-attention mechanism that isinherent to the transformer architecture. The BERT model was eventuallyused in Google’s search engine for processing user inputs and making searchpredictions2.Later, researchers at Google would develop LaMDA [23], a prototypeLLM that would eventually become the basis of Bard, Google’s response toOpenAI’s ChatGPT. Bard would later become known as Gemini. LaMDAdiffers from OpenAI’s GPT models in that it is a decoder-only Transformerarchitecture and does not perform a generative pre-training step. Instead, itis trained on a large dataset composed of texts and fine-tuned on responseswith manually produced annotations [23].OpenAI researchers would later develop InstructGPT3, where they aug-mented the model by reinforcement learning from human feedback (RLHF)[16]. This utilized human feedback to fine-tune the model and improve re-sponses. The RLHF technique has been utilized in OpenAI’s releases sinceGPT-44. Other techniques to improve GPT’s language generation capabil-ities in terms of accuracy have been attempted, including the use of Crit-icGPT5, a GPT-4-based model used to supervise and indicate mistakes toassist the RLHF process.2.2 Retrieval-Augmented GenerationWhile large language models like ChatGPT are some of the most effec-tive tools for natural language processing and generation, various issues arevisible in the frameworks of current LLMs. Hallucination, a phenomenonwhere an LLM’s output contains information not based on factual informa-2“Understanding searches better than ever before”, Pandu Nayak, October 25, 20193Aligning language models to follow instructions, OpenAI (2022)4GPT-4, OpenAI (2023)5Finding GPT-4’s mistakes with GPT-4, OpenAI (2024)62.2. RETRIEVAL-AUGMENTED GENERATIONtion, can create a situation where users of the AI are given false informationthat is contrary to reality [13, 30, 31]. Some researchers have posited thatAI’s potential to respond with false information can impede students’ learn-ing if not directly harmful [12].An additional issue with LLMs is that due to the necessity of a well-known corpus of training data to fine-tune the weights of LLMs, the knowl-edge base of the LLM may become out-of-date as new information is notintegrated immediately. Another key issue is that some LLMs, includingChatGPT, do not by default share where their information comes from asthat is often lost in the training process. This makes it difficult to determinewhether or not an LLM is sharing factual information without manual hu-man investigation, which reduces the effectiveness and usability of LLMs asa method of information retrieval. This is a potential threat in an era wherestudents are widely using ChatGPT and other LLMs to retrieve informationthat search engines would have been required for before effective chatbotslike ChatGPT were introduced.An attempt to resolve many of these issues is Retrieval-Augmented Gen-eration (RAG), a technique in which, given an LLM prompt, additionalinformation is retrieved based on said prompt. This requires the implemen-tation of an external store of contextual data as well as a method for retrieval[10]. Implementations of RAG will typically contain sections, often referredto as “document chunks”, of extracted information from contextual datawith a generated encoding, an “embedding”. The query is then encodedusing the same algorithm as the source documents, and this encoding iscompared against the source documents’ encodings [10] to retrieve a chunkof information from the context. After the chunk is retrieved, it is then pro-vided to an LLM through some means, such as by appending the originalquery with the information and passing it to the LLM [27]. The benefits ofan RAG model include that the source of the data is known and may bestored alongside the document chunks and that the information used by theLLM is up-to-date and reliable.The source of information for an RAG model can range from a well-known and accessible corpus of data to a vector-embedding database. WhenRAG is desired to augment an LLM’s capability to perform general-purposelanguage generation, it is more attractive to choose from a set of data thatrequires less maintenance, such as that of Wikipedia. In cases where moredomain-specific knowledge is required, a vector-embedding database con-structed from specific documents may be more desirable and has been im-plemented in some education contexts [21, 27]. The use of domain-specificknowledge can serve to improve chatbot responses to student queries and72.3. ARTIFICIAL INTELLIGENCE IN EDUCATION (AIED)lower the risk of hallucination and sharing of out-of-date information [27].In Sanz et. al’s [21] implementation of a chatbot improved through RAG,the RAG model’s knowledge base was composed of information regardinguniversal learning design. Sanz et al.’s implementation utilized an exter-nal API to retrieve source documents and transform them into plain-textdocuments. These plain-text documents were then converted into vector em-beddings and stored in a vector database. Given queries to the chatbot, theFacebook AI Similarity Search (FAISS) [8] library was used to perform vec-tor similarity search operations, which allowed for retrieval of the embeddedsource documents via embeddings extracted from a query. Upon receiving astudent question, the query would go through the same embedding step asthe source documents and is subsequently used to perform a semantic searchthrough the knowledge base. The ranked results from the vector similaritysearch would then be sent to a generative AI model to produce the final,contextualized output.Wang et al. [27] proposed the ChatEd architecture, which utilizes andbuilds upon the RAG framework to further improve chatbot responses, andfine-tune their usage in education. The ChatEd architecture aims to sat-isfy three main requirements in the chatbot question-and-answer framework:factual correctness of information, relevancy to the domain, and informationverifiability. As with ChatGPT and other LLM chatbot implementations,the ChatEd chatbot is used by posing questions, receiving answers, andcontinuing the back-and-forth as long as the user wishes. As with the im-plementation of Sanz et al. [21], the ChatEd architecture uses the FAISSlibrary [8] to match prompts to document chunks.2.3 Artificial Intelligence in Education (AIEd)The evolution of pedagogy is inextricably intertwined with the evolutionof technology, with the adoption of new technologies continuously improvingeducation for both educators and students alike. The introduction of artifi-cial intelligence into education has, however, been controversial. In a surveyof 3800 students performed by the Digital Education Council (DEC)6, itwas found that the majority of students use ChatGPT. Despite using AIas a tool for information retrieval, more than half of the students surveyedstated a fear of AI being used for evaluation in the classroom, indicating anunderstanding that AI can be an unreliable judge of factual correctness.6What Students Want: Key results of the DEC Global AI Student Survey, DigitalEducation Council (2024)82.3. ARTIFICIAL INTELLIGENCE IN EDUCATION (AIED)Educational researchers have debated the concept that chatbots can bea benefit to the classroom, even before the introduction of state-of-the-arttechnologies such as ChatGPT and other LLMs. An identified benefit ofchatbots in the classroom are their ability to act as a low-cost, low-effortassistant in educational contexts [9]. Winkler & So¨llner [29] argue thatwhile the effectiveness of chatbots is subject to a variety of factors, theycan solve the problem of individualized support for students in large classes.They identify four core advantages with the introduction of chatbots in thebroader sense, those being:− A reduction in service costs, as chatbots can replace human assistantsin some domains.− An increase in user satisfaction due to rapid response times and con-stant availability.− A general advantage of AI tools is that chatbots can anticipate ques-tions and have fine-tuned interactions with users.− Prevalent business analysis capabilities from chatbots allow automaticprocessing of conversations and subsequent improvements to be made.While some of these advantages are not necessarily relevant to educa-tion, Winkler & So¨llner argue that the capability of students receiving rapidresponses from an agent available 24/7 holds promise in easing demands forone-on-one time with instructors in large courses, and can be achieved withminimal resource investment.Implementations of AI in the classroom have been received with positivesentiment. Gayed et al. [11] developed “AI KAKU,” an artificial intelli-gence tool that provided word suggestions to English First Language (EFL)learners. In addition, a reverse translation of the user’s English text to theirnative language was provided beneath their English input. The experimen-tal group was found to have improved performance when compared with thecontrol group participants, with improved sentence fluency. In addition tothe improved performance, the study’s experimental group participants hada largely positive reaction to the AI KAKU tool.In Sanz et al.’s [21] pilot study of a customized chatbot for the classroom,the researchers polled 21 student participants, with 20 reporting satisfactionwith the chatbot’s responses. Additionally, before they implemented theirchatbot, the researchers surveyed 49 teachers and 107 students to investigatewhat both groups desire out of artificial intelligence in the classroom, and92.3. ARTIFICIAL INTELLIGENCE IN EDUCATION (AIED)their thoughts on its introduction to education. The responses from bothgroups were largely open to the idea of using chatbots to support students,but teachers demonstrated some degree of distrust in technology.Despite various successful implementations of AI in the classroom, adop-tion rates of AI by educators themselves have been low, with many re-searchers aiming to identify factors that can explain adoption behaviour inboth teachers and students. Wang et al. [28] used a structural equationmodeling (SEM) approach to explain instructor intent, or lack thereof, toutilize AI in their teaching. They investigated factors such as anxiety (AN)toward the new technologies, self-efficacy (SE), the teachers’ faith in theirabilities to utilize the new technologies, attitude towards use (ATU) itself,perceived ease of use (PEU), and perceived usefulness (PU), and surveyed311 teachers with a questionnaire. After analyzing and applying the teach-ers’ responses to their structural model, it was found that around 70% ofbehavioural changes related to teachers’ adoption intent could be explainedby the factors of AN, SE, ATU, PEU, and PU. Wang et. al’s findings sup-ported several of their hypotheses, including that improving an instructor’sSE, or confidence in their abilities, would have a positive effect on theirperceived ease of use (PEU) and attitude towards AI use (ATU). Addition-ally, a negative correlation between self-efficacy and anxiety was observed,indicating that by improving a teacher’s self-efficacy, their anxiety towardutilizing AI tools would be reduced. These findings imply that despite initialreservations about AI technologies, instructors’ intent to adopt the technolo-gies can be improved by improving their trust in the technology and theirconfidence in themselves when wielding it.Similar to the study of Wang et al., Pillai et al. [17] used an SEM ap-proach to survey 1380 participants regarding the same factors of PEU andPU, but introduced other measures: the interactivity (INV), personalization(PSN), perceived trust (PTR), anthropomorphism (APM), or, similarity toa human, perceived intelligence (PEI), adoption intention (ADI), “sticki-ness” to learn from human teachers (SKN), or the students’ indicated biastowards learning from human teachers, and finally, actual usage (ATU). Pil-lai et al. found evidence to support many of their hypotheses, such as thatPEU, PU, PSN, INV, APM, and PTR all positively impact the adoptionintention. Malik et al.’s [15] study, also utilizing an SEM approach, simi-larly found that PEU, PU can alter students’ sentiments towards chatbotadoption. Another study by Ayanwale & Ndlovu [1] uses the factors ofPEU, PU, as well as others from the Innovation Diffusion Theory (DTI)[20] which describes the variables of relative advantage (RA), compatibility(CM), complexity (CO), trialability (TR), trust (TRU), and observability102.3. ARTIFICIAL INTELLIGENCE IN EDUCATION (AIED)(OB). Ayanwale & Ndlovu surveyed 842 undergraduate student participants,and found evidence supporting their hypotheses that the factors RA, CO,TR, and TRU all positively impact the intention to adopt. These findingsindicate that students view chatbots as advantageous and feel compatiblewith the technologies in the context of education. Additionally, the findingsregarding trust (TRU) indicate that more trustworthy chatbots are morelikely to be adopted by students.In general, the studies above can be described using the Technology Ac-ceptance Model [5, 6], which introduces the variables of behavioral intent(BI), attitude towards use (ATU), perceived usefulness (PU), and perceivedease of use (PEU). Each study provides evidence that supports, to a lesseror greater degree, the posited hypotheses. In general, these supported hy-potheses indicate both instructors and students alike are more likely to adoptchatbots and other AI technologies in an educational context when the toolsare seen as useful and/or easy to use. Wang et al.’s evidence that anxietytowards AI tools can be curbed by improving potential users’ confidence intheir abilities to use said tools is a significant indicator that the success ofAI tools in education may be limited at this time purely due to lack of in-struction regarding how to effectively make use of them. Other particularlyinteresting findings are that students, at large, are a demographic that ismore open to the idea of using chatbots in an educational context [1, 15, 17].The ChatEd architecture of Wang et al. [27] was targeted toward usagein education, with a focus on ensuring the correctness, relevancy, and veri-fiability of answers. Moreover, the ChatEd architecture was designed withinstructors in mind, providing a simple utility interface for instructors toupload documents to be used as sources in the vector store, which are thenused as contextual information sources by the underlying RAG model andconnected LLM chatbot.11Chapter 3MethodologyThis study was conducted in an upper-level computer science course atthe University of British Columbia Okanagan (UBCO) over the 2024 wintersemester from September to December 2024. Students were provided accessto an asynchronous help center known as HelpMe [26], a web application thatprovides a live queue for facilitating online office hours, and asynchronousquestion-and-answer with the assistance of a RAG-model chatbot. The chat-bot used in the HelpMe system is the chatbot from Wang et al.’s [27], andfollows its architecture identically, with minimal upgrades that have beenmade following its debut.The chatbot was accessible in the main course landing page of theHelpMe system and available through a floating element present on eachpage. Students could also interact with the chatbot via a question fo-rum, where students could pose questions to the instructional team asyn-chronously and receive (human) answers later. All asynchronous questionswould receive an initial answer from the chatbot which the students coulddeem sufficient or request further help from the course’s instructional team.The HelpMe system was presented to students as the primary location pointto request assistance with the course materials, primarily to reduce the num-ber of emails received by instructional team members. Additionally, thequeue system aims to allow students to seek course-related support withouthaving to arrange travel for in-person office hours outside of regular classtime [26].Students were encouraged to use the HelpMe system throughout thecourse with grade bonus incentives provided for submitting assignmentsearly through the office hours system. Students who chose to submit their as-signments early via office hours would regularly encounter the chatbot uponnavigating to the course landing page within the HelpMe system. The asyn-chronous question forum was also presented as an option for seeking coursehelp, where students would also receive initial answers to their queries fromthe chatbot.123.1. QUESTIONNAIRE3.1 QuestionnaireAt the end of the course, students were encouraged to fill out a survey.This survey was a general evaluation of the HelpMe system’s user accep-tance and contained questions regarding the usage statistics and feedbackregarding the office hours system, chatbot, and anytime question forum, inaddition to usability criteria of the HelpMe system as a whole, and questionsconcerning whether students prefer the HelpMe system over other methodsof seeking help in courses.The survey contained a total of 40 questions, with the majority beingpart of multiple-choice Likert-scale-answer [14] sections. The Likert scaleused is the common five-answer form: strongly agree, agree, neutral (orN/A), disagree, strongly disagree. The primary focus of this research is thequestions seen in Tables 3.1 and 3.2, which list the survey questions relatedto the chatbot and asynchronous question components of the system.The survey questions for the chatbot generally aimed to gather and iden-tify student sentiment towards the HelpMe chatbot implementation, butwere also suited to gather information regarding general student sentimenttoward chatbots in general, and in the education domain. Due to the asyn-chronous question hub featuring interactions with the chatbot, as it respondsto each question made by default, questions regarding the asynchronousquestion forum are relevant to measuring the chatbot’s performance. Thesequestions aim to retrieve the same usability metric evaluations as chatbot-specific questions but do not focus on general sentiment toward AI in theclassroom. Both question suites fielded quantitative and qualitative feed-back in the form of general feedback paragraph answer questions at the endof their corresponding page.3.2 ChatbotThe vector store used to improve the chatbot’s answers with RAG waspopulated with various forms of course information, primarily composed ofdocument chunks formed from teaching materials, such as lecture slides, andmanually inserted Q&A items. From the source documents, over eight hun-dred document chunks were formed. The mechanism used to form documentchunks would generally parse portable document format (PDF) and othertext-based file formats to create relevant contextual information that couldbe used to later augment the LLM’s responses.133.2. CHATBOTPrompt Possible AnswersHow often did you use the chat-bot?Multiple Choice:• Never• Once• Two to five times• Six to ten times• Over 10 timesI am willing to use a chatbot orany available AI system for helpin a course.Multiple Choice:• Likert 5-ScaleChatbot answers were relevantand accurate.Multiple Choice:• Likert 5-ScaleChatbot conversation was human-like.Multiple Choice:• Likert 5-ScaleThe chatbot including source ma-terial links was helpful.Multiple Choice:• Likert 5-ScaleThe chatbot assisted me in meet-ing my learning outcomes.Multiple Choice:• Likert 5-ScaleChatbots should be used in othercourses.Multiple Choice:• Likert 5-ScaleI would prefer to use a course-provided chatbot rather than aprivate option like ChatGPT.Multiple Choice:• Likert 5-ScaleI would rather use the chatbotthan use office hours or email tocommunicate with teaching staff.Multiple Choice:• Likert 5-ScalePlease provide any general com-ments about the chatbot and itsuse in the course.Paragraph AnswerTable 3.1: Survey Questions regarding the HelpMe system’s Chatbot143.2. CHATBOTPrompt Possible AnswersHow often did you use the Any-time Question feature?Multiple Choice:• Never• Once• Two to five times• Six to ten times• Over 10 timesI prefer email over asking AnytimeQuestions via HelpMe.Multiple Choice:• Likert 5-ScalePublic Anytime Question answerswere helpful.Multiple Choice:• Likert 5-ScaleA single place for help (officehours, chatbot, Anytime Ques-tions) is beneficial.Multiple Choice:• Likert 5-ScaleHow would you rate the qualityand relevance of the answers pro-vided by the AI in Anytime Ques-tions?Multiple Choice:• Excellent• Very Good• Average• PoorI asked more questions to the chat-bot or Anytime Questions than Isent emails in the course.Multiple Choice:• Yes• No• About the samePlease provide any general com-ments regarding how the Any-time Question support comparesto email, the AI chatbot, or tra-ditional in-person office hours.Paragraph AnswerTable 3.2: Survey Questions regarding the HelpMe system’s AsynchronousQuestion Forum153.2. CHATBOT5 Relational Model Formal DefinitionThe relational model is formally defined in terms ofsets and set operations. A relation schema R (A1 , A 2 , ..., A n ) has each attribute A i witha name and a domain dom(A I ).A relation instance denoted r(R) is a set of n-tuples <d 1 , d 2 , ..., d n > where each d i isan element of dom(A i ) or is null.* The relation instance is the extension of therelation.* A value of null represents a missing or unknownvalue.Example: Product (id , name , supplierId , categoryId ,price)R = Product (relation name)* Set A= {id, name , supplierId , categoryId , price}* dom(price) is set of all possible positivecurrency values* dom(name) is set of all possible strings thatrepresent people ’s namesFigure 3.1: An example of a document chunk in the chatbot’s vector storeIn addition to the manually uploaded source documents, the chatbotRAG architecture features a secondary vector store that contains embeddedstudent questions for the chatbot and their corresponding answer so that thequestion and answer pair may be referenced by the chatbot later. A uniquefeature is the “verified” answer, where the instructor can denote a questionas having a particular answer. If a question’s embedding matches a questionwith a verified answer’s embedding, that answer will be the one used by thechatbot, rather than processing the request in full, reducing the utilizationof expensive components (i.e., using the LLM to produce the answer).In addition to data gathered via the usability survey, data from theHelpMe system itself was used to study the chatbot’s performance and usagemetrics. The HelpMe system uses a relational DBMS to store system anduser-related data. Course and chatbot interaction data were extracted fromthe system’s database to perform an analysis of the overall usage of thechatbot.16Chapter 4Results4.1 Survey ResultsThe survey received a total of 83 responses from students. Responseswere processed, analyzed, and visualized. Invalid responses, where the stu-dent left the question empty, were not processed, and incomplete surveyswere also discarded from the analysis. For the primary survey questions ofinterest, Questions 21 through 25 and Question 27, each received around60 responses. At least a quarter of the participants did not complete thesurvey in full and left many of the response areas blank, including parts ofthe survey where the HelpMe system’s chatbot and anytime question featurewere evaluated. Further, many of the questions were answered with neutralresponses by the participants. The results of each of the survey questionsare visualized in this section.Question 21.1, had the majority of participants agree with the statement“they were willing to use a chatbot or any available AI system for help in acourse,” as can be seen in Figure 4.1. Participants largely demonstrated anopenness to using AI systems for educational assistance, with few outrightindicating they are unwilling to use AI tools for help in education.However, for the following Question 21.2, most participants did not havean opinion regarding the statement “Chatbot answers were relevant andaccurate,” as seen in Figure 4.2. Most of the remaining responses heldan affirming opinion. Given usage statistics seen in Figures 4.10 and 4.18,it makes sense that most participants reported a neutral opinion, as mostparticipants did not use the chatbot. A slim majority of the students who didreport an opinion, however, agreed that the chatbot’s answers were relevantand accurate.174.1. SURVEY RESULTSFigure 4.1: Responses to the sur-vey question: “I am willing to usea chatbot or any available AI sys-tem for help in a course.”Figure 4.2: Responses to the sur-vey question: “Chatbot answerswere relevant and accurate.”The following four part-questions of Question 21 follow a similar patternto Question 21.2. Questions 21.3 and 21.5 had most participants eitherreporting no opinion or disagreeing that the chatbot is anthropomorphic orthat it assisted them in meeting their learning outcomes, which can be seenin Figures 4.3 and 4.5. Conversely, Questions 21.4 and 21.6 each receivedmore agreeable sentiment from participants, as seen in Figures 4.4 and 4.6.Responses to Question 21.4 indicate that several participants thought thatthe presence of source material links was useful. Question 21.6 receiveda more pronounced enthusiastic response, indicating that participants are,in general, largely open or unbothered by the introduction of AI in theclassroom. Responses to these questions indicate that most participants donot hold an opinion regarding the statements which is consistent with usagestatistics seen in Figures 4.10 and 4.18.184.1. SURVEY RESULTSFigure 4.3: Responses to the sur-vey question: “Chatbot conversa-tion was human-like.”Figure 4.4: Responses to the sur-vey question: “The chatbot in-cluding source material links washelpful.”Figure 4.5: Responses to the sur-vey question: “The chatbot as-sisted me in meeting my learningoutcomes.”Figure 4.6: Responses to the sur-vey question: “Chatbots shouldbe used in other courses.”Responses to Question 21.7 indicate that participants are nearly equallydivided when it comes to their preference in using course-provided AI toolswhen compared with private options, like ChatGPT, visualized in Figure 4.7.It cannot be said decisively whether participants would prefer to use course-provided AI tools or private AI tools. A significant portion of participantsindicated they hold no preference either way. Of the non-neutral responses,there is a slight leaning of participants toward preferring a course-providedchatbot, implying there is a somewhat viable use case for instructors pro-viding students with AI assistance tools.Question 21.8 has similarly divided responses, with the majority of par-ticipants being against the idea of using a chatbot in place of contacting194.1. SURVEY RESULTScourse instructional teams via email or office hours, depicted in Figure 4.8.Figure 4.7: Responses to the sur-vey question: “I would preferto use a course-provided chatbotrather than a private option likeChatGPT.”Figure 4.8: Responses to the sur-vey question: “I would ratheruse the chatbot than use officehours or email to communicatewith teaching staff.”Figure 4.9: Responses to the sur-vey question: “I would recom-mend a struggling peer to use thechatbot for assistance.”Responses to Question 21.9 did not show that participants conclusivelywould recommend (or not recommend) the chatbot to peers who are strug-gling with course materials, as shown in Figure 4.9. Interestingly, valencedopinions from participants outnumbered neutral responses for this question.Non-neutral responses were only slightly positive-leaning, indicating thata sizable portion of participants would recommend their peers utilize thechatbot if they required help.Question 22 asked participants to report how much they used the chatbot204.1. SURVEY RESULTSthroughout the term, which showed that participants did not use the chatbotmuch throughout the term, as seen in Figure 4.10. Most participants in thesurvey never used it, and very few used it more than two to four times. Thesubsequent Question 23 asked a similar usage metric but instead concernsthe anytime question feature, with similar results to the chatbot’s usage,visualized in Figure 4.11. The majority of responses indicated no usage atall. Of those who used it, most used it one time throughout the term.Figure 4.10: Responses to the sur-vey question: “How often did youuse the chatbot?”Figure 4.11: Responses to thesurvey question: “How often didyou use the Anytime Question fea-ture?”Question 24 is similarly split into 3 parts as Question 21. It presentedstatements and asked participants to respond with their degree of agree-ment with the statement in a multiple-choice Likert-scale answer with fiveoptions, exactly as seen in Question 21. Participant responses to Question24.1 indicated that participants either hold no opinion or are nearly evenlysplit on whether they prefer email over using the anytime questions forumon HelpMe, as seen in Figure 4.12. Question 24.2, likewise, received an out-sized portion of neutral responses. Opinionated responses were, however,dominated by positive sentiment. This implies that participants who usedthe anytime questions forum found publicly available answers helpful, as vi-sualized in Figure 4.13. The responses that Questions 24.1 and 24.2 receivedare dominated by non-opinions. This is consistent with the usage statisticsfor the anytime question forum, which are depicted in Figure 4.11.Question 24.3 received a majority positive sentiment from participants,indicating that participants agree there is a use case for all-in-one help sys-tems in education, with responses shown in Figure 4.14.214.1. SURVEY RESULTSFigure 4.12: Responses to thesurvey question: “I prefer emailover asking Anytime Questionsvia HelpMe.”Figure 4.13: Responses to the sur-vey question: “Public AnytimeQuestion answers were helpful.”Figure 4.14: Responses to the sur-vey question: “A single place forhelp (office hours, chatbot, Any-time Questions) is beneficial.”Question 25 asked participants how “they would rate the quality andrelevance of the answers provided by the AI in Anytime Questions?” Par-ticipants could choose from the following options: “poor,” “average,” “verygood,” and “excellent.” Responses indicate that participants, on average,would view the AI’s responses as mediocre; still, only a few participantsopined that the AI’s responses were poor in quality, with a significant chunkstating they were very good or excellent, shown in Figure 4.15.Question 27 posed: “I asked more questions to the chatbot or AnytimeQuestions than I sent emails in the course.” It received a total of 57 validresponses, and could be responded to by the answers of “yes,” “no,” or“about the same.” Responses to Question 27 indicate that when seeking224.1. SURVEY RESULTSassistance with the course, participants did not utilize the anytime questionsfeature more than they emailed the professor throughout the term. Theresponses to Question 27 are visualized in Figure 4.16. These results areconsistent with the usage statistics depicted in Figure 4.11 and the numberof emails the instructor received throughout the term, seen in Figure 4.23.Figure 4.15: Responses to the sur-vey question: “How would yourate the quality and relevance ofthe answers provided by the AI inAnytime Questions?”Figure 4.16: Responses to the sur-vey question: “I asked more ques-tions to the chatbot or AnytimeQuestions than I sent emails in thecourse.”An aggregation of survey responses regarding the chatbot and AI, ingeneral, is in Figure 4.17. Participant responses were split practically in halfbetween those who expressed an opinion and those who did not, while mostof those who did express an opinion would express one that was affirmingto the chatbot.Figure 4.17: An aggregation of the survey responses, determined to be pos-itive, negative, or neutral toward the chatbot and AI.234.2. CHATBOT USAGE & PERFORMANCE4.2 Chatbot Usage & PerformanceWhen analyzing the extracted data from the HelpMe system, severalindicators can be found that corroborate the survey data. In Figure 4.18, ahistogram displaying the frequency of total interactions students had withthe chatbot is shown. Of those students who interacted with the chatbotdirectly, the majority of them would have only interacted with it once ortwice. There were a total of 103 unique students who interacted with theHelpMe system’s chatbot throughout the term. Very few students had morethan 10 interactions with the chatbot, with only one student being recordedto have interacted with the chatbot 30 times throughout the term.Figure 4.18: Distribution of interactions between students and chatbotThe students who interacted with the chatbot asked a variety of ques-tions. A visual breakdown of the questions asked to the chatbot is shown inFigure 4.19. The majority of questions asked were looking for informationregarding course policy. The next highest category of questions were thoserequiring technical help, particularly when students were having trouble withthe tools used to complete the course assignments. Following technical help,students then requested most often help with the material; this could includedirectly asking for help with questions from the assignment and verifying in-formation concerning the course material. The last two categories includedirrelevant questions (i.e., did not concern the course material, policies, or244.2. CHATBOT USAGE & PERFORMANCEtechnical difficulties) and malicious questions (i.e., attempting to obtain theAPI key used by the course chatbot through prompting it). Additionallyshown in the figure is a report of the percentage of these questions that wereanswered with a unique, newly generated answer.Figure 4.19: Student questions for the chatbot broken down by type (course,technical, material, irrelevant, and malicious) separated by re-used answers.Figure 4.20 shows a visualization of the number of interactions studentshad with the chatbot throughout the term. There were a total of 452 in-teractions with the chatbot. The majority of these interactions occurredwithin the first month, with the peak number of interactions occurring inlate September. After this, interactions were limited for most of the termuntil mid-November, when usage spiked briefly.254.2. CHATBOT USAGE & PERFORMANCEFigure 4.20: Chatbot interactions throughout the termFigure 4.21 visualizes how many answers the chatbot gave which werenewly generated, unique answers, and those which were verified, repeat an-swers. Just under 80 percent of answers the chatbot gave throughout theterm were unique, with the remaining nearly 20 percent of its responses be-ing a verbatim repeat of what was specified as the verified answer for thegiven query. Additionally, the responses the chatbot gave throughout theterm were manually labeled to determine whether the answers were helpfuland/or harmful and whether the question was relevant or irrelevant to thecontext of the course. In this breakdown, it was found that the vast major-ity of questions were relevant and provided a helpful answer, but nearly 9percent of answers the chatbot gave to relevant questions were deemed ashaving harmful responses. Around 3 percent of answers the chatbot gavewere helpful, but for an irrelevant question, and a marginal percentage ofquestions received answers that were both helpful and harmful (where theanswer would have some correct, but some incorrect/harmful information),whether the question was relevant in the first place or not.264.3. DEMANDS ON INSTRUCTOR TIMEFigure 4.21: Percentage of uniqueanswers vs. verified (stored) an-swersFigure 4.22: Helpfulness vs.harmfulness of chatbot answersgiven relevance4.3 Demands on Instructor TimeThroughout the term, the instructor of the course received 114 emailsfrom students. In Figure 4.23, a breakdown of the reasoning for the receivedemails can be seen. The majority of emails received (45) were regardingassignments, followed by 41 emails regarding the midterm or final exams, 18for marking-related concerns, 7 for course administration issues, and 3 re-garding group member/work concerns. The time taken throughout the termby the instructor to address each of these emails is visualized in Figure 4.24.The instructor offered online office hours via the HelpMe system throughoutthe term, where the chatbot was available to answer quick queries.Many of the questions asked in emails by the students would not havebeen given appropriate answers by the chatbot, such as questions that con-cerned teammates in group work and marking. However, many of the ques-tions asked which concerned assignment materials had answers that wouldbe contained within the chatbot’s vector store of contextual information. Asseen in Figure 4.24, the instructor had to spend a significant amount of timeresponding to emails during a few dates late into the term, in early Novem-ber. This influx of emails corresponds to the occurrence of the students’second midterm exam and the due dates of several assignments, indicatingwhy there is an influx of emails regarding marking, exams, and assignmentsaround this period.274.3. DEMANDS ON INSTRUCTOR TIMEFigure 4.23: Reasons for student emails to professorFigure 4.24: Time (minutes) needed to address emails throughout the term28Chapter 5Conclusion5.1 DiscussionThe chatbot survey results do not imply a decisive general opinion bystudents, whether positive or negative, regarding the use of AI technology inthe classroom. The majority of participants did not report a strong opinionto most of the survey questions, most likely due to the fact the chatbot wasnot utilized by many of them throughout the term, as seen in Figures 4.10and 4.18. In addition, the questions regarding the anytime question forumalso received an out-sized portion of neutral responses, which is consistentwith the feature’s usage statistics, depicted in Figure 4.11.Question 21.1 (Figure 4.1) received widely positive responses in a ques-tion that aimed to gauge general participant opinion toward AI as an educa-tional assistance tool. This is consistent with previous research into studentadoption behaviour toward AI, which demonstrated students are mostlyopen to using AI for assistance in education [1, 15, 17].Question 21.7 (Figure 4.7) showed participants divided between whetherthey would prefer course-provided AI tools over private options, however,discounting neutral responses, a small majority of participants indicated thatthey would prefer to use a chatbot provided by the instructor or institution,rather than a private option. This is supporting evidence for the usability ofthe ChatEd architecture [27], which is implemented on a case-by-case basisfor education assistance systems, such as the HelpMe system.Question 21.8 (Figure 4.8) showed likewise divided responses - but over-all negative - when participants were asked whether they’d prefer to use thechatbot over direct communication with teaching staff. This is consistentwith previous findings from studies that report that many students do notview, or wish, chatbots to be replacements for teachers [17, 21]. However,previous researchers have stated that the chatbot’s ability to provide indi-vidual support in large classrooms [19, 29], and while these findings do notnecessarily disprove this theory, they do show some evidence that studentsmay not support the replacement of instructional staff with chatbots. How-ever, due to students’ willingness to utilize AI tools to assist them in courses,295.1. DISCUSSIONchatbots can still be a way to provide additional help when one-on-one timewith instructional staff is not readily available.Question 24.3 (Figure 4.14) demonstrated a largely positive sentimenttoward the HelpMe system as a whole, with the majority of participantsagreeing that an all-in-one system like HelpMe is beneficial.In Section 4.2, a more granular approach is used to gauge the true metricsof the chatbot’s usage and performance. It is shown in Figure 4.18 that themajority of students who utilized the chatbot only used it once or twice, withan almost exponential decrease in the number of students as the number ofchatbot interactions increased. This generally indicates that the chatbotwas not well-used by students, and it may not have piqued their interestor seemed useful to them. In Figure 4.20, the number of interactions thatoccurred, by date, is shown. It is visible here that the number of interactionswas much higher at the start of the term, peaking particularly on one earlyday when the HelpMe system was introduced to the class, and one otherday, around the time of the course’s second midterm occurred. Otherwise,the usage of the chatbot declined sharply around the end of the first monthof the course. The underutilization of the chatbot may be partly due tothe under-utilization of the HelpMe system, as it was not a requirement topass the course and would be used exclusively for obtaining bonus markson assignments and seeking help, whether through online office hours, theasynchronous question feature, or the chatbot itself.In terms of performance, the chatbot generally performed well when itcame to providing helpful and harmful answers, as seen in Figure 4.22. Thevast majority of questions were both relevant and had been given a helpfulanswer, followed by a small minority being relevant and being given harmfulanswers. Other cases, where the question was irrelevant and the answer washelpful, or the answer was both helpful and harmful and the question wasirrelevant or relevant, can be treated essentially as noise. This is consistentwith the survey sections (see Figure 4.2) which requested opinions regardingthe chatbot’s answers’ relevancy and accuracy, where, other than the major-ity of neutral responses, the general sentiment was that participants agreedwith the statement that the chatbot’s answers were relevant and accurate.With the number of emails received by the course’s instructor, it is notevident that the chatbot reduced work for the instructor (see Figures 4.23and 4.24). However, there was no baseline of email volume before the intro-duction of the chatbot. The majority of questions asked via email regardedassignments and the chatbot was well-prepared with context regarding thecourse material to respond to. However, for several questions asked bystudents, such as those concerning marking, those that involve discussions305.2. CONCLUSIONregarding group members, and sometimes specific exam information, thechatbot would not have been capable of answering those queries in the firstplace. It is not conclusive whether the chatbot saved the instructor time byanswering questions and avoiding unnecessary emails to the instructor.In general, it appears that students were much more likely to utilize al-ternative help methods, such as directly emailing the instructor or otherwisecontacting teaching staff, using online- or in-person office hours, and possi-bly using private AI tools such as ChatGPT throughout the term. However,there is evidence to support that many students will accept and engage withan LLM in an educational context.5.2 ConclusionThe HelpMe system’s chatbot did not receive a decisive evaluation giventhe responses to the survey. Participants in the survey did demonstrate awillingness to use AI tools for assistance in education, showing a degree ofacceptance and engagement with either a private or course-provided LLMtool. However, results from the survey questions regarding its effectivenessas an accurate and reliable feature were generally inconclusive. The chat-bot was under-utilized throughout the study duration and this is a likelycontributor to the number of non-opinionated responses from survey partic-ipants. Investigation into questions the chatbot answered throughout thestudy indicated that it was able to give helpful responses to student queriesalmost nine times out of ten. Without a baseline number of emails, it cannotbe conclusively determined if the chatbot saved the instructor time by be-ing available to respond to student queries. The HelpMe system’s chatbotdemonstrates the potential to be an effective assistance tool but requiresmore investigation to be evaluated properly.5.3 Future WorkFurther investigations into the efficacy of the HelpMe system’s chatbotcould include further effort in surveys, with more incentive for students toutilize the tool throughout the study duration. Additional contributionscould include studying student opinions on existing private AI systems suchas ChatGPT to provide a benchmark. Investigating the reasons studentsprefer to directly contact their instructors and instructional teams via emailor office hours would provide insights into student behaviour and metricsthat could be used to improve systems like HelpMe.31Bibliography[1] Musa Adekunle Ayanwale and Mdutshekelwa Ndlovu. Investigatingfactors of students’ behavioral intentions to adopt chatbot technolo-gies in higher education: Perspective from expanded diffusion theoryof innovation. Computers in Human Behavior Reports, 14:100396,2024. ISSN 2451-9588. doi: https://doi.org/10.1016/j.chbr.2024.100396. URL https://www.sciencedirect.com/science/article/pii/S2451958824000290. → pages 2, 3, 10, 11, 29[2] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural Ma-chine Translation by Jointly Learning to Align and Translate, 2014.URL https://arxiv.org/abs/1409.0473. → pages 1, 4, 5[3] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever.Generating Long Sequences with Sparse Transformers. ArXiv,abs/1904.10509, 2019. URL https://api.semanticscholar.org/CorpusID:129945531. → pages 5[4] Kyunghyun Cho, Bart Merrienboer, Caglar Gulcehre, Fethi Bougares,Holger Schwenk, and Y. Bengio. Learning Phrase Representations usingRNN Encoder-Decoder for Statistical Machine Translation. Conferenceon Empirical Methods in Natural Language Processing, June 2014. doi:10.3115/v1/D14-1179. → pages 4[5] Fred D. Davis. Perceived Usefulness, Perceived Ease of Use, and UserAcceptance of Information Technology. MIS Quarterly, 13(3):319–340, 1989. ISSN 02767783, 21629730. URL http://www.jstor.org/stable/249008. → pages 3, 11[6] Fred D. Davis, Richard P. Bagozzi, and Paul R. Warshaw. Extrinsicand intrinsic motivation to use computers in the workplace. Journalof Applied Social Psychology, 22(14):1111–1132, 1992. doi: 10.1111/j.1559-1816.1992.tb00945.x. → pages 3, 11[7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.BERT: Pre-training of Deep Bidirectional Transformers for Lan-32Bibliographyguage Understanding. In North American Chapter of the Asso-ciation for Computational Linguistics, 2019. URL https://api.semanticscholar.org/CorpusID:52967399. → pages 1, 5, 6[8] Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson,Gergely Szilvasy, Pierre-Emmanuel Mazare´, Maria Lomeli, Lucas Hos-seini, and Herve´ Je´gou. The Faiss library, January 2024. URLhttps://arxiv.org/abs/2401.08281. → pages 8[9] Fotos Frangoudes, Marios Hadjiaros, Eirini Schiza, Maria Matsangidou,Olia Tsivitanidou, and Kleanthis Neokleous. An Overview of the Use ofChatbots in Medical and Healthcare Education, pages 170–184. SpringerInternational Publishing, July 2021. ISBN 978-3-030-77942-9. doi: 10.1007/978-3-030-77943-6 11. → pages 2, 9[10] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, YuxiBi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval-Augmented Generation for Large Language Models: A Survey, 2024.URL https://arxiv.org/abs/2312.10997. → pages 2, 7[11] John Maurice Gayed, May Kristine Jonson Carlon, Angelu Mari Oriola,and Jeffrey S. Cross. Exploring an AI-based writing Assistant’s impacton English language learners. Computers and Education: ArtificialIntelligence, 3:100055, 2022. ISSN 2666-920X. doi: https://doi.org/10.1016/j.caeai.2022.100055. URL https://www.sciencedirect.com/science/article/pii/S2666920X22000108. → pages 3, 9[12] Adrian Groza and Anca Marginean. BRAVE NEW WORLD: AI INTEACHING AND LEARNING. In ICERI2023 Proceedings, 16th an-nual International Conference of Education, Research and Innovation,pages 8706–8713. IATED, November 2023. ISBN 978-84-09-55942-8. doi: 10.21125/iceri.2023.2221. URL https://doi.org/10.21125/iceri.2023.2221. → pages 2, 7[13] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu,Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Surveyof Hallucination in Natural Language Generation. ACM Comput. Surv.,55(12), March 2023. ISSN 0360-0300. doi: 10.1145/3571730. URLhttps://doi.org/10.1145/3571730. → pages 2, 7[14] Rensis Likert. A Technique for the Measurement of Attitudes. Archivesof Psychology, 140:1–55, 1932. → pages 1333Bibliography[15] Reena Malik, Ambuj Shrama, Sonal Trivedi, and Rikkee Mishra. Adop-tion of Chatbots for Learning among University Students: Role of Per-ceived Convenience and Enhanced Performance. International Journalof Emerging Technologies in Learning (iJET), 16(18):200–212, Septem-ber 2021. doi: 10.3991/ijet.v16i18.24315. → pages 2, 3, 10, 11, 29[16] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wain-wright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, KatarinaSlama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E.Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul FrancisChristiano, Jan Leike, and Ryan J. Lowe. Training language models tofollow instructions with human feedback. ArXiv, abs/2203.02155, 2022.URL https://api.semanticscholar.org/CorpusID:246426909. →pages 6[17] Rajasshrie Pillai, Brijesh Sivathanu, Bhimaraya Metri, and NeerajKaushik. Students’ adoption of AI-based teacher-bots (T-bots) forlearning in higher education. Information Technology & People, 37:328–355, January 2023. doi: 10.1108/ITP-02-2021-0152. → pages 2, 3,10, 11, 29[18] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.Improving language understanding by generative pre-training. OpenAI,2018. → pages 1, 5, 6[19] Mohamed Ragheb, Passent Tantawi, Nevien Farouk, and AhmedHatata. Investigating the acceptance of applying chat-bot (Artificialintelligence) technology among higher education students in Egypt. In-ternational Journal of Higher Education Management, 8:1–13, February2022. doi: 10.24052/IJHEM/V08N02/ART-1. → pages 2, 29[20] Everett M. Rogers. Diffusion of Innovations: Modifications ofa Model for Telecommunications, pages 25–38. Springer BerlinHeidelberg, Berlin, Heidelberg, 1995. ISBN 978-3-642-79868-9.doi: 10.1007/978-3-642-79868-9 2. URL https://doi.org/10.1007/978-3-642-79868-9_2. → pages 10[21] Enrique Sanz, Gonzalo Mariscal, Sara Redondo Duarte, Eva Jime´nez-Garc´ıa, and Sonia Requejo. AI-Driven Student Assistance: ChatbotsRedefining University Support. In 18th International Technology, Edu-cation and Development Conference, pages 617–625, March 2024. doi:10.21125/inted.2024.0221. → pages 2, 3, 7, 8, 9, 2934Bibliography[22] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequencelearning with neural networks. In Proceedings of the 28th Interna-tional Conference on Neural Information Processing Systems - Volume2, NIPS’14, page 3104–3112, Cambridge, MA, USA, 2014. MIT Press.→ pages 4[23] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer,Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, LeslieBaker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng,Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun,Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, ZhifengChen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung-ChingChang, Igor Krivokon, Will Rusch, Marc Pickett, Kathleen S. Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos San-tos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prab-hakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina,Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, AlenaButryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Co-hen, Rachel Bernstein, Ray Kurzweil, Blaise Agu¨era y Arcas, ClaireCui, Marian Croak, Ed H. Chi, and Quoc Le. LaMDA: LanguageModels for Dialog Applications. CoRR, abs/2201.08239, 2022. URLhttps://arxiv.org/abs/2201.08239. → pages 6[24] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet,Marie-Anne Lachaux, Timothe´e Lacroix, Baptiste Rozie`re, NamanGoyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin,Edouard Grave, and Guillaume Lample. LLaMA: Open and EfficientFoundation Language Models, 2023. URL https://arxiv.org/abs/2302.13971. → pages 5, 6[25] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, LlionJones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attentionis All you Need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach,R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neu-ral Information Processing Systems, volume 30. Curran Associates, Inc.,2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf. →pages 1, 2, 4, 5, 6[26] Kevin Wang and Ramon Lawrence. HelpMe: Student Help Seeking us-ing Office Hours and Email. In Proceedings of the 55th ACM Technical35BibliographySymposium on Computer Science Education V. 1, SIGCSE 2024, page1388–1394, New York, NY, USA, 2024. Association for Computing Ma-chinery. ISBN 9798400704239. doi: 10.1145/3626252.3630867. URLhttps://doi.org/10.1145/3626252.3630867. → pages 3, 12[27] Kevin Wang, Jason Ramos, and Ramon Lawrence. ChatEd: A ChatbotLeveraging ChatGPT for an Enhanced Learning Experience In HigherEducation. In INTED2024 Proceedings, 18th International Technol-ogy, Education and Development Conference, pages 6580–6589. IATED,March 2024. ISBN 978-84-09-59215-9. doi: 10.21125/inted.2024.1722.URL https://doi.org/10.21125/inted.2024.1722. → pages 2, 3, 7,8, 11, 12, 29[28] Youmei Wang, Chenchen Liu, and Yun-Fang Tu. Factors Affecting theAdoption of AI Based Applications in Higher Education: An Analysisof Teachers Perspectives Using Structural Equation Modeling. Educa-tional Technology & Society, 24(3):116–129, 2021. → pages 2, 3, 10[29] Rainer Winkler and Matthias So¨llner. Unleashing the Potential of Chat-bots in Education: A State-Of-The-Art Analysis. Academy of Manage-ment Proceedings, 2018:15903, April 2018. doi: 10.5465/AMBPP.2018.15903abstract. → pages 2, 9, 29[30] Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu,Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang,Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. Siren’s Song inthe AI Ocean: A Survey on Hallucination in Large Language Models,2023. URL https://arxiv.org/abs/2309.01219. → pages 2, 7[31] Terry Yue Zhuo, Yujin Huang, Chunyang Chen, and Zhenchang Xing.Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliabilityand Toxicity, 2023. URL https://arxiv.org/abs/2301.12867. →pages 2, 736
Graduating Project
eng
Unreviewed
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Undergraduate
University of British Columbia. COSC_O 449
Chatbots in the Classroom : A Survey of Student Sentiment Toward AI in Education
Text