Supporting systematic assessment of digital interventions: A framework for Analysing and Measuring Usage and Engagement Data (AMUsED)

Abstract Trials of digital interventions (DIs) can yield extensive, in-depth usage data, yet usage analyses tend to focus on broad descriptive summaries of how an intervention has been used by the whole sample. This paper proposes a novel framework to guide systematic, fine-grained usage analyses that better enables understanding of how an intervention works, when and for whom. Framework for Analysing and Measuring Usage and Engagement Data description: The framework comprises three stages to assist: 1) familiarisation with the intervention and its relationship to the captured data; 2) identification of meaningful measures of usage and specifying research questions to guide systematic analyses of usage data; 3) preparation of datasheets, and consideration of available analytical methods with which to examine the data. Framework application: The framework can be applied to inform data capture during the development of a DI and/or in the analysis of data after the completion of an evaluation trial. We will demonstrate how the framework shaped preparation and aided efficient data capture for a DI to lower transmission of cold and flu viruses in the home, and informed a systematic in-depth analysis of usage data collected from a separate DI designed to promote self-management of colds and flu. Conclusions: The AMUsED framework guides systematic and efficient in-depth usage analyses that will support standardized reporting with transparent and replicable findings. These detailed findings may also enable examination of what constitutes effective engagement with particular interventions. Keywords: behavioral research; internet; health; patient engagement; data analysis