Classification of Sex and Alzheimers Disease via Brain Imaging-Based Deep Learning on 85,721 Samples

Beyond detecting brain damage or tumors, little success has been attained on identifying individual differences and brain disorders with magnetic resonance imaging (MRI). Here, we sought to build industrial-grade brain imaging-based classifiers to infer two types of such inter-individual differences: sex and Alzheimers disease (AD), using deep learning/transfer learning on big data. We pooled brain structural data from 217 sites/scanners to constitute the largest brain MRI sample to date (85,721 samples from 50,876 participants), and applied a state-of-the-art deep convolutional neural network, Inception-ResNet-V2, to build a sex classifier with high generalizability. In cross-dataset-validation, the sex classification model was able to classify the sex of any participant with brain structural imaging data from any scanner with 94.9% accuracy. We then applied transfer learning based on this model to objectively diagnose AD, achieving 88.4% accuracy in cross-site-validation on the Alzheimer9s Disease Neuroimaging Initiative (ADNI) dataset and 91.2% / 86.1% accuracy for a direct test on two unseen independent datasets (AIBL / OASIS). Directly testing this AD classifier on brain images of unseen mild cognitive impairment (MCI) patients, the model correctly predicted 63.2% who eventually converted into AD, versus predicting 22.1% as AD who did not convert into AD during follow-up. Predicted scores of the AD classifier correlated significantly with illness severity. By contrast, the transfer learning framework was unable to achieve practical accuracy for psychiatric disorders. To improve interpretability of the deep learning models, occlusion tests revealed that hypothalamus, superior vermis, thalamus, amygdala and limbic system areas were critical for predicting sex; hippocampus, parahippocampal gyrus, putamen and insula played key roles in predicting AD. Our trained model, code, preprocessed data and an online prediction website have been openly-shared to advance the clinical utility of brain imaging.