Multimodal feature learning and fusion on B-mode ultrasonography and sonoelastography using point-wise gated deep networks for prostate cancer diagnosis

B-mode ultrasonography and sonoelastography are used in the clinical diagnosis of prostate cancer (PCa). A combination of the two ultrasound (US) modalities using computer aid may be helpful for improving the diagnostic performance. A technique for computer-aided diagnosis (CAD) of PCa is presented based on multimodal US. Firstly, quantitative features are extracted from both B-mode US images and sonoelastograms, including intensity statistics, regional percentile features, gray-level co-occurrence matrix (GLCM) texture features and binary texture features. Secondly, a deep network named PGBM-RBM2 is proposed to learn and fuse multimodal features, which is composed of the point-wise gated Boltzmann machine (PGBM) and two layers of the restricted Boltzmann machines (RBMs). Finally, the support vector machine (SVM) is used for prostatic disease classification. Experimental evaluation was conducted on 313 multimodal US images of the prostate from 103 patients with prostatic diseases (47 malignant and 56 benign). Under five-fold cross-validation, the classification sensitivity, specificity, accuracy, Youden's index and area under the receiver operating characteristic (ROC) curve with the PGBM-RBM2 were 87.0%, 88.8%, 87.9%, 75.8% and 0.851, respectively. The results demonstrate that multimodal feature learning and fusion using the PGBM-RBM2 can assist in the diagnosis of PCa. This deep network is expected to be useful in the clinical diagnosis of PCa.