What Do We Know about Assessing Computational Thinking? A New Methodological Perspective from the Literature
9
Citation
54
Reference
10
Related Paper
Citation Trend
Abstract:
Developing computational thinking (CT) assessment methods appropriate for elementary students is attracting growing attention as CT research in elementary education progresses. To review the current elementary CT assessments for potential gaps, and seek additional methodologies to expand our understanding of CT, an integrative literature review of 75 research papers was performed in two phases. In Phase One, we conducted a critical analysis of existing elementary CT assessment studies. Key results include: 1) Artifact analysis, CT assessment items, and interviews are the most common methods utilized to assess CT in elementary grades; 2) Existing CT assessments primarily focus on students' computational artifacts and performance on CT tests; however, strategies to study students' thought processes during CT problem-solving are limited and under-utilized. Guided by the results of phase one, along with the theoretical perspective that connected CT to visual processing ability, in phase two we performed a survey of literature in the area of understanding cognitive processes through eye-tracking (i.e., visual attention) and think-aloud methodologies (i.e., verbalization). We focused on eye-tracking and think-aloud methodologies as these have been used to understand students' cognitive processes during problem-solving in other areas. Based on these findings, we proposed that in addition to current established methodologies, eye-tracking with the think-aloud technique can provide new insights into students' CT.Keywords:
Think aloud protocol
Protocol analysis
Artifact (error)
Tracking (education)
Simulations and games bring the possibility to research complex processes of managerial decision-making. However, this modern field requires adequate methodological procedures. Many authors recommend the use of a combination of concurrent think-aloud (CTA) or retrospective think-aloud (RTA) with eye-tracking to investigate cognitive processes such as decision-making. Nevertheless, previous studies have little or no consideration of the possible differential impact of both think-aloud methods on data provided by eye-tracking. Therefore, the main aim of this study is to compare and assess if and how these methods differ in terms of their impact on eye-tracking. The experiment was conducted for this purpose. Participants were 14 managers who played a specific simulation game with CTA use and 17 managers who played the same game with RTA use. The results empirically prove that CTA significantly distorts data provided by eye-tracking, whereas data gathered when RTA is used, provide independent pieces of evidence about the participants' behavior. These findings suggest that RTA is more suitable for combined use with eye-tracking for the purpose of the research of decision-making in the game environment.
Think aloud protocol
Protocol analysis
Tracking (education)
Cite
Citations (26)
Think-aloud protocols are among the most standard methods for usability evaluation, which help to discover usability problems and to examine improvements because they provide direct information on a user’s thinking and cognitive processes; however, it is often difficult to determine how to analyze the data to identify usability problems because there is no formulaic analysis procedure for textual data. Therefore, the analysis is time-consuming, and the quality of the results varies depending on an analyst’s skills. In the present study, the author proposes a formulaic analysis think-aloud protocol method that specifies the procedure for analyzing participants’ verbal responses during usability tests. The aim of the proposed think-aloud protocol method was to deliver an explicit procedure using step coding (SCAT) and 70 design items for textual data analysis, and then, the method was applied to a case study of usability evaluation to confirm that the method could extract the target system’s problems. By using step coding and 70 design items, the process of extracting usability problems from textual data was made explicit, and the problems were extracted analytically. In other words, the proposed method was less ambiguous. Once a formulaic analysis procedure was established, textual data analysis could be performed easily and efficiently. The analysis could be performed without hesitation after data acquisition, and there were fewer omissions. In addition, it is expected that the procedure would be easy to use, even for novice designers.
Think aloud protocol
Protocol analysis
Cognitive walkthrough
Cite
Citations (8)
Think aloud protocol
Protocol analysis
Identification
Empirical Research
De facto
Cite
Citations (19)
This paper presents preliminary findings from an eyetracking study that investigates the validity of responses provided by participants who have been instructed to use think-aloud protocol during a Website usability study. Ten participants were given four information-finding scenarios to perform on the Washington State Department of Licensing website and they were instructed to think aloud. While performing these tasks, an eyetracking system determined where on the screen the subjects looked. Videotapes of the sessions containing the participant's computer screen, mouse and eye movements, and verbalizations were analyzed to determine the degree validity offered think-aloud protocol. Initial findings suggest that some subtle cues indicating participants' expectations of information and link location, and their confidence in link names may be lost when using think-aloud protocol and observation alone.
Think aloud protocol
Protocol analysis
Cite
Citations (33)
The present study examined different think-aloud protocols with eye tracking technology in usability testing in an exploratory and comparative way. A focused selection was made from a list of usability methods and usability techniques to deepen the understanding of think-aloud protocols and eye tracking technology. The concurrent and retrospective think-aloud protocols have been used together with a gaze-measuring (classic) eye tracker and an (cued) eye tracker with a vision bubble to create four conditions. To compare the four conditions, a review from previous research with similar usability tests had been done. From the review, six criteria have been selected by which the four conditions can be compared. The six criteria have been used with the questionnaire UMUX-lite and the instrument Rating Scale Mental Effort to do the comparative study. The results of the comparative study suggest that the retrospective think-aloud protocol which utilizes the classic eye tracker fits the criteria better than the three other conditions. Furthermore, the results suggest that concurrent and retrospective think-aloud protocols which use the cued eye tracker fit the criterion regarding the participants’ experience better than the two think-aloud protocols which use the classic eye tracker. The results also suggest that the two think-aloud protocols which use the cued eye tracker fit the criteria regarding the questionnaire UMUX-lite and the time spent on a task worse than the two think-aloud protocols which use the classic eye tracker. In conclusion, the present study can be seen as an in-depth exploration into the world of usability testing and encourages investigating think-aloud protocols and eye tracking further.
Think aloud protocol
Protocol analysis
Cued speech
Exploratory research
Cite
Citations (0)
Think Aloud (TA) studies provide rich verbal data about reasoning during a problem solving task. Using TA and protocol analysis, investigators can identify the information that is concentrated on during problem solving and how that information is used to facilitate problem resolution. From this, inferences can be made about the reasoning processes that were used during the problem-solving task. In the past, the validity of data obtained from TA studies has been suspect because of inconsistencies in data collection and the inability to verify findings obtained from the slow, laborious process of protocol analysis. This article describes a means of obtaining more accurate verbal data and analyzing it in a standardized step-by-step manner.
Think aloud protocol
Protocol analysis
Suspect
Cite
Citations (600)
The aim of this qualitative study is to uncover the composing processes of a male Pakistani student writer tentatively with the help of cognitive discourse analysis. The study qualitatively analyzed the verbal transcripts to come to the tentative conclusions about the writing behaviors of the L2 student. An analysis of the participant's writing think-aloud protocol verbalized data show that writing is a recursive process. The writer applies different meta-cognitive, cognitive, social and affective strategies in a recursive pattern while writing.
Protocol analysis
Think aloud protocol
Qualitative analysis
Participant Observation
Cite
Citations (0)
The article explores what students are really thinking when they engage in research projects or when they solve information intensive problems using think aloud verbal protocols. Topics covered include Verbal Protocol Analysis (VPA), the applications of the think-aloud protocol in the library literature and getting complementary information by using think-aloud protocols in combination with other techniques.
Think aloud protocol
Protocol analysis
Cite
Citations (0)
School assessments too often ignore the process of individuals' cognitions. As part of a systematic effort to develop an assessment procedure focused on the tactics used to facilitate reading comprehension, this research investigated Lytle's (1985) approach to think-aloud protocol analysis. Data are presented for fourth- and fifth-graders that show the degree to which they used reasoning, elaboration, signaling understanding, analysis, judging, and monitoring doubts while they were reading three fictional passages. The percentage of each move used by individual subjects was stable across the three stories that were administered. Significant correlations were obtained between comprehension of the passage used to obtain the think-aloud protocols and two of the six categories of moves: Reasoning and Signaling Understanding. These students used Elaboration, Signaling Understanding, and Reasoning most frequently; they used Monitoring Doubts less frequently; and they rarely used Analysis and Judging. Unique contributions of this research include the extension of previous work with this method to elementary school students, demonstration of stability in the use of moves across the three stories, and detection of correlations between this system and comprehension of the passage used to obtain the think-aloud protocol. Suggestions are made for future research that concerns the validity of think-aloud protocol analysis as a diagnostic technique and an approach to facilitate comprehension skills.
Protocol analysis
Think aloud protocol
Elaboration
Reading aloud
Cite
Citations (47)
The protocols of Think Aloud and Eye Tracking, in their own unique way have proven to be great methods to understand users’ thought processes, and their mental models when interacting with interfaces. However the effectiveness of the combination of the two protocols in discovering usability problems has not been explored. This study aimed to discover if the addition of Eye Tracking data (fixations and scan movements) to the traditional protocol of Think Aloud can uncover more usability problems. Web users were split into three groups: Eye Tracking Only (ET), Think Aloud Only (TA), and Eye Tracking and Think Aloud Only (ET+TA). Participants in all conditions were asked to complete two tasks on two websites each. Along with questionnaires, eye movement data was collected for conditions with the Eye Tracking aspect and verbalizations were collected for conditions with the Think Aloud aspect. The analysis of the data showed that the total number of usability problems (not unique) identified by the participants in the ‘Eye Tracking and Think Aloud’ (ET + TA) condition was higher than the other two conditions. However, a Tukey HSD post-hoc test revealed that the differences between ‘ET + TA’ and the ‘Eye Tracking Only’ (ET) conditions was non-significant. The analysis also which resulted in non-significant differences between the conditions ‘Eye Tracking’ (ET) and ‘Eye Tracking and Think Aloud’ (ET + TA) led to inconclusive results on whether the Think Aloud method is disruptive or not. This may lead future researchers to develop robust practice sessions to help participants verbalize and create evaluation rules for eye movement data. THINK ALOUD & EYE TRACKING! 5 A Paruchuri | Thesis
Think aloud protocol
Protocol analysis
Tracking (education)
Cite
Citations (0)