An end-to-end deep learning model can detect the gist of the abnormal in prior mammograms as perceived by experienced radiologists

This study investigated the possibility of building an end-to-end deep learning-based model for the prediction of a future breast cancer based on prior negative mammograms. We explored whether the probability of abnormal class membership given by the model was correlated with the gist of the abnormal as perceived by radiologists in negative prior mammograms. To build the model, an end-to-end network, previously developed for breast cancer detection, was fine-tuned for breast cancer prediction by using a dataset containing 650 prior mammograms from women, who were diagnosed with breast cancer in a subsequent screening and 1000 cancer-free women. On a set of 630 test images, the model achieved an AUC of 0.73. For extracting gist responses, 17 experienced radiologists were recruited, viewed mammograms for 500 milliseconds and gave a score showing whether they would categorize the case as normal or abnormal on the scale of 0- 100. The image set contained 40 normal, 40 current cancer images along with 72 prior mammograms from women who would eventually develop a breast cancer. We averaged the scores from 17 readers and produced a single score per image. The network achieved an AUC of 0.75 for differentiating prior images from normal images. For 72 prior mammograms, the output of the network was significantly correlated with the strength of the gist of the abnormal as perceived by experienced radiologists (Spearman’s correlation=0.84, p<0.01). This finding suggested that the network successfully learned the representation of the gist of the abnormal in prior mammograms as perceived by experienced radiologists.