Digital Phenotyping Using Multimodal Data

Digital phenotyping involves the quantification of in situ phenotypes using personal digital devices and holds the potential to dramatically reshape how serious mental illnesses (SMI) assessment is conducted. Despite promise, few, if any, digital phenotyping efforts for SMI have garnered the support necessary for clinical implementation. In this paper, we highlight how digital phenotyping efforts can be improved by integrating data from multiple channels (i.e., “multimodal” data integration). We begin by arguing that “multimodal” integration is critical for digital phenotyping of many, possibly most, SMI symptoms. Next, we consider computational approaches that can accommodate multimodal data. We conclude by considering next steps for multimodal data for research and clinical applications. We punctuate this paper with examples of multimodal digital phenotyping using natural language processing (NLP) to measure speech disorganization in psychosis.