Assessing Learning Analytics Systems Impact by Summative Measures

This paper introduces a randomized study conducted among a group of 48 student Java programmers to assess the impact of learning analytics (LA) on their academic performance. The LA system design incorporated both cognitive and metacognitive tools to help learners take possession of their learning processes. Participation was voluntary and data about potential confounding factors were also collected to minimize bias by blocking on two or more factors (future work). This paper summarily explores the relationships between students' programming expertise, coding assignments, user experience and satisfaction, and academic performance. The results of this preliminary exploration are inconclusive as to whether the LA system made a difference in academic performance. Nevertheless, they seem to indicate that LA was beneficial to student programmers and that summative measures such as grades are not a proper metric to measure the usefulness of LA systems.