Large language models (LLMs) are revolutionizing the field of computing education with their powerful code-generating capabilities. Traditional pedagogical practices have focused on code writing tasks, but there is now a shift in importance towards reading, comprehending and evaluating LLM-generated code. Alongside this shift, an important new skill is emerging -- the ability to solve programming tasks by constructing good prompts for code-generating models. In this work we introduce a new type of programming exercise to hone this nascent skill: 'Prompt Problems'. Prompt Problems are designed to help students learn how to write effective prompts for AI code generators. A student solves a Prompt Problem by crafting a natural language prompt which, when provided as input to an LLM, outputs code that successfully solves a specified programming task. We also present a new web-based tool called Promptly which hosts a repository of Prompt Problems and supports the automated evaluation of prompt-generated code. We deploy Promptly in one CS1 and one CS2 course and describe our experiences, which include student perceptions of this new type of activity and their interactions with the tool. We find that students are enthusiastic about Prompt Problems, and appreciate how the problems engage their computational thinking skills and expose them to new programming constructs. We discuss ideas for the future development of new variations of Prompt Problems, and the need to carefully study their integration into classroom practice.
Generative artificial intelligence tools (Generative AI), such as ChatGPT, allow users to interact with them in intuitive ways (e.g., conversational) and receive (mostly) good-quality answers. In education, such systems can support students' learning objectives by providing accessible explanations and examples even when students pose vague queries. But, they also encourage undesired help-seeking behaviors, such as by providing solutions to the students' homework. Therefore, it is important to better understand how students approach such tools and the potential issues such approaches might present for the learners.In this paper, we present a case study for understanding student-AI collaboration to solve programming tasks in the CS1 introductory programming course. To this end, we recruited a gender-balanced majority non-white set of 15 CS1 students at the University of Houston, a large public university in the US. We observed them solving programming tasks. We used a mixed-method approach to study their interactions as they tackled Python programming tasks, focusing on when and why they used ChatGPT for problem-solving. We analyze and classify the questions submitted by the 15 participants to ChatGPT. Additionally, we analyzed user interaction patterns, their reactions to ChatGPT's responses, and the potential impacts of Generative AI on their perception of self-efficacy.Our results suggest that, in about a third of the cases, the student attempted to complete the task by submitting the full description of the tasks to ChatGPT without making any effort on their own. We also observed that few students verified their solutions. We discuss the potential implications of these results.
It is well known that programming error messages can be notoriously difficult for novices to understand, hampering progress and leading to frustration. In response, researchers have explored various approaches for enhancing such messages, yet results from this active strand of research are currently mixed. Direct comparisons of results between studies is challenging as these typically investigate different kinds of message enhancements and report results using different metrics. In addition, many prior studies have involved code writing tasks. In such cases, not all students encounter the same errors and messages, and it is difficult to isolate the time spent interpreting messages and resolving errors from the time spent writing code. In this research, we explore the effects of presenting novices with compiler error messages designed using the most recent collection of published guidelines - specifically, more easily readable, short, positive messages containing resolution hints. To accurately determine the time and effort required to read and respond to the messages, we utilise a debugging task where all students are presented the same code and therefore encounter the same errors. We present results of a randomised controlled experiment (n > 700) which shows that, compared to standard error messages, the messages we tested resulted in significantly shorter debugging times and higher self-reported scores of message usefulness for students in the very early stages of learning a new language.
Novice programmers often struggle through programming problem solving due to a lack of metacognitive awareness and strategies. Previous research has shown that novices can encounter multiple metacognitive difficulties while programming. Novices are typically unaware of how these difficulties are hindering their progress. Meanwhile, many novices are now programming with generative AI (GenAI), which can provide complete solutions to most introductory programming problems, code suggestions, hints for next steps when stuck, and explain cryptic error messages. Its impact on novice metacognition has only started to be explored. Here we replicate a previous study that examined novice programming problem solving behavior and extend it by incorporating GenAI tools. Through 21 lab sessions consisting of participant observation, interview, and eye tracking, we explore how novices are coding with GenAI tools. Although 20 of 21 students completed the assigned programming problem, our findings show an unfortunate divide in the use of GenAI tools between students who accelerated and students who struggled. Students who accelerated were able to use GenAI to create code they already intended to make and were able to ignore unhelpful or incorrect inline code suggestions. But for students who struggled, our findings indicate that previously known metacognitive difficulties persist, and that GenAI unfortunately can compound them and even introduce new metacognitive difficulties. Furthermore, struggling students often expressed cognitive dissonance about their problem solving ability, thought they performed better than they did, and finished with an illusion of competence. Based on our observations from both groups, we propose ways to scaffold the novice GenAI experience and make suggestions for future work.
An abstract is not available for this content so a preview has been provided. Please use the Get access link above for information on how to access this content.
Background and Context. Metacognitive skills are important for all students learning to program and interest in applying pedagogical approaches in early programming courses that focus on metacognitive aspects is growing. However, most studies of such approaches are not rigorously based in theory, and when they are, almost always utilize foundational education and psychology theories from as far back as the 1970s. More recent theory is less tested, and not all relevant metacognitive theories have been explored in the computing education research literature.
'/%3e%3cpath d='M200 752.362h200v300H200z' style='fill:%23fff;fill-opacity:1;stroke:none' transform='translate(0 -752.362)'/%3e%3cpath d='M400 752.362h200v300H400z' style='fill:%23ff883e;fill-opacity:1;stroke:none' transform='translate(0 -752.362)'/%3e%3c/svg%3e)
'/%3e%3cuse xlink:href='%23a' transform='rotate(72 0 0)'/%3e%3cuse xlink:href='%23a' transform='rotate(-72 0 0)'/%3e%3cuse xlink:href='%23a' transform='scale(-1 1) rotate(72)'/%3e%3c/g%3e%3c/defs%3e%3cpath fill='%23012169' d='M0 0h1200v600H0z'/%3e%3cpath stroke='%23FFF' stroke-width='60' d='m0 0 600 300M0 300 600 0' clip-path='url(%23b)'/%3e%3cpath stroke='%23C8102E' stroke-width='40' d='m0 0 600 300M0 300 600 0' clip-path='url(%23c)'/%3e%3cpath stroke='%23FFF' stroke-width='100' d='M300 0v300M0 150h600' clip-path='url(%23b)'/%3e%3cpath stroke='%23C8102E' stroke-width='60' d='M300 0v300M0 150h600' clip-path='url(%23b)'/%3e%3cuse xlink:href='%23d' fill='%23FFF' transform='matrix(45.4 0 0 45.4 900 120)'/%3e%3cuse xlink:href='%23d' fill='%23C8102E' transform='matrix(30 0 0 30 900 120)'/%3e%3cg transform='rotate(82 900 240)'%3e%3cuse xlink:href='%23d' fill='%23FFF' transform='rotate(-82 519.022 -457.666) scale(40.4)'/%3e%3cuse xlink:href='%23d' fill='%23C8102E' transform='rotate(-82 519.022 -457.666) scale(25)'/%3e%3c/g%3e%3cg transform='rotate(82 900 240)'%3e%3cuse xlink:href='%23d' fill='%23FFF' transform='rotate(-82 668.57 -327.666) scale(45.4)'/%3e%3cuse xlink:href='%23d' fill='%23C8102E' transform='rotate(-82 668.57 -327.666) scale(30)'/%3e%3c/g%3e%3cuse xlink:href='%23d' fill='%23FFF' transform='matrix(50.4 0 0 50.4 900 480)'/%3e%3cuse xlink:href='%23d' fill='%23C8102E' transform='matrix(35 0 0 35 900 480)'/%3e%3c/svg%3e)
