PaCon: a symbolic analysis approach for tactic-oriented clustering of programming submissions

Enrollment in programming courses increasingly surges. To maintain the quality of education in programming courses, instructors need ways to understand the performance of students and give feedback accordingly at scale. For example, it is important for instructors to identify different problem-solving ways (named as tactics in this paper) used in programming submissions. However, because there exist many abstraction levels of tactics and high implementation diversity of the same tactic, it is challenging and time-consuming for instructors to manually tackle the task of tactic identification. Toward this task, we propose PaCon, a symbolic analysis approach for clustering functionally correct programming submissions to provide a way of identifying tactics. In particular, PaCon clusters submissions according to path conditions, a semantic feature of programs. Because of the focus on program semantics, PaCon does not struggle with the issue of an excessive number of clusters caused by subtle syntactic differences between submissions. Our experimental results on real-world data sets show that PaCon can produce a reasonable number of clusters each of which effectively groups together those submissions with high syntax diversity while sharing equivalent path-condition-based semantics, providing a promising way toward identifying tactics.