Tractostorm: Rater reproducibility assessment in tractography dissection of the pyramidal tract

Investigative studies of white matter (WM) brain structures using diffusion MRI (dMRI) tractography frequently require manual WM bundle segmentation, often called "virtual dissection". Human errors and personal decisions make these manual segmentations hard to reproduce, which have not yet been quantified by the dMRI community. The contribution of this study is to provide the first large-scale, international, multi-center variability assessment of the "virtual dissection of the pyramidal tract (PyT). Eleven (11) experts and thirteen (13) non-experts in neuroanatomy and "virtual dissection" were asked to perform 30 PyT segmentation and their results were compared using various voxel-wise and streamline-wise measures. Overall the voxel representation is always more reproducible than streamlines ($\approx$70\% and $ \approx$35\% overlap respectively) and distances between segmentations are also lower for voxel-wise than streamline-wise measures ($\approx$3mm~and~$\approx$6mm respectively). This needs to be seriously considered before using tract-based measures (e.g. bundle volume versus streamline count) for an analysis. We show and argue that future bundle segmentation protocols need to be designed to be more robust to human subjectivity. Coordinated efforts by the diffusion MRI tractography community are needed to quantify and account for reproducibility of WM bundle extraction techniques in this era of open and collaborative science.