Open Collections

Abstract

The emergence of conversational AI tools in late 2022 practically changed the face of software engineering and software engineering education. Contemplating the question of how to best prepare and evaluate students in this new reality, we experimented with systematically introducing a conversational AI tool, ChatGPT, into the 2023 offering of an upper-level undergraduate project-based course on Software Engineering. In this course, 20 groups of four students each had to design and implement a project of their choice, with an Android-based mobile client and a Node.js-based cloud server. This thesis discusses our goals, approach, and lessons learned from introducing ChatGPT into the first phase of the project development: scoping the work and defining the project requirements. Our experience shows that students can achieve comparable results using a variety of ChatGPT interaction modes and the success of each mode largely depends on students’ preferences, learning styles, and the invested effort. Yet, in any of the modes, with moderate effort, students can produce artifacts of a mid-range quality level of around 80%. Moving above this range requires substantial investment, which can be spent on brainstorming, crafting high-quality prompts, or critically assessing ChatGPT’s output. We also observe low prompting proficiency of the students: students can improve their prompting strategies by providing a more adequate description of their course and project setup, examples, and expected output format for their requests. Interestingly, students can often be “swayed” by ChatGPT’s projected confidence, even when their original ideas are, in fact, more appropriate than the proposed refinements. In a follow up study, we further evaluate the latest version of ChatGPT on the original students’ prompts, and discover that while ChatGPT can produce artifacts that are better in some attributes, the results are still mediocre and would need substantial effort to get 95%+ scoring artifacts. We hope that our experience and lessons learned will help spark further discussions on how to best embed AI tools into the software engineering curriculum and work practices.