Assessing the Language of Chat for Teamwork Dialogue

Introduction In technology enhanced language learning, many pedagogical activities involve students in online discussion such as synchronous chat, in order to help them practice their language skills (e.g., Hedayati & Foomani, 2015). Such online team chat increases students' participation and evenly distributes their participation (Dennis & Garfield, 2003). In today's global and complex environment, besides developing the language competency of students, it is also important to nurture their teamwork and collaboration competencies (Rychen & Salganik, 2003). Moreover, language communication is an important glue in teamwork that helps build the team together and propel it forward in various tasks (Baker et al., 2005). Thus, beyond evaluating the academic competency of the language, it is meaningful to examine the teamwork discourse of students. Providing an assessment of teamwork dialogue helps students gain a better awareness of their teamwork competency and become better team players (Erkens & Janssen, 2008; Koh et al., 2014). Advancements in technology like learning analytics have allowed such assessments to be made more automated. To develop such automated assessment systems, training models using human evaluation (involving computational linguistics and text classification) have the highest potential for efficiency and scalability. These models use computers to code the data instead of humans, which is faster; they can also be developed to examine different texts which allows for scalability (Erkens & Janssen, 2008; Rose et al., 2008). In other words, language assessment can be made more efficient, and this facilitates more immediate feedback for students and teachers which can help improve students' learning (Anjewierden et al., 2007). However, a major challenge in automatically assessing online chat discourse is that chat texts have many irregularities in structure, short lengths and contextual complexities. This makes the identification of codes (such as teamwork dimensions) more difficult. Past research suggests that pre-processing the text to organize and take into account desired features would be helpful for analysis. Despite the importance of pre-processing, the steps in pre-processing tend to be vague. There seems to be a black box of pre-processing. Another problem is that there are varying approaches to train models using human evaluation such as natural language processing and supervised machine learning. This area of analysis has not been widely documented and techniques also depend on the nature of the coding scheme and the purpose of classification. Furthermore, there are many machine learning algorithms that could be used. Finding out the most effective method would be the key to provide an automatic analysis of the language of chat, which serves as formative assessment for students. Therefore, the focus of the paper is on the methods of using learning analytics for assessing online chat discourse, in particular, to measure the dimensions of teamwork. The research questions are: * How can text be effectively pre-processed to assess the language of chat for teamwork dialogue? * What approach works best to assess the language of chat for teamwork dialogue? * Which algorithms are the most effective in classifying teamwork dimensions? This study is part of a larger research project exploring the 21st century competency of teamwork in technology enhanced learning. Based on previous literature and pilot studies, six dimensions of teamwork were conceptualized (Koh et al., 2014). A coding scheme for these dimensions was developed and chat log data was manually coded. Chat log data of the study was obtained from 14 year old students who were collaborating on an online collaborative problem-solving activity as part of their project work curriculum. Besides providing students with the opportunity to engage in teamwork, the activity was designed for English communication, which also helps students practice their language skills. …