Method in the madness: the design of games as valid and reliable scientific tools

Video games have great value for collecting rich data that can be used to study fundamental research questions in fields such as psychology and human-computer interaction. There is an increased use of games as scientific tools to assess and train the cognitive, as well as non-cognitive traits and abilities of players. However, games designed as research tools have a unique and challenging set of considerations so that they meet both the requirements of a scientific protocol, while also maintaining an acceptable gaming experience. Additionally, the success of a research game requires extensive planning and co-creation support for trans-disciplinary teams. This paper describes five critical considerations for the design and implementation of custom-built games as scientifically valid and reliable experimental test beds. These findings resulted from several years of work creating game artifacts to study cognitive training and assessment of individual differences. This class of games presents unique design challenges that are often at odds with traditional game design conventions. However, there are many promising applications and use cases for thoughtfully instrumented games that are capable of generating data with comparable accuracy to traditional and previously validated measures.