Measuring Knowledge Gains in an SMS m-Learning Intervention: The Case of ChildConnect South Africa

ChildConnect is an early childhood development programme which aims to use mobile learning to support and educate parents and other caregivers. The programme is based on an SMS curriculum focused on early childhood development content, underpinned by six core learning outcomes. In a pilot in 2017, 899 primary and secondary caregivers signed up to receive the messages. Caregivers were randomly assigned to either a treatment or control group. The treatment group received 3 messages a week for 24 weeks while the control group received one message every two weeks. Pre and post-tests were administered using ‘Unstructured Supplementary Service Data’ (USSD) surveys on mobile phones. Results of the tests were combined into a composite learning outcome score. This paper explores preliminary evidence of knowledge gains, comparing pre-test outcome scores to post-test outcome scores across treatment and control groups. The treatment group performed better on their learning outcome score than those in the control group by ~0.7 points (Cohen’s d = 0.21, small effect size, but non-trivial). This shows that there was a small but measurable positive effect on the knowledge gains of the treatment group compared to the control.