Similarity Steered Generative Adversarial Network and Adaptive Transfer Learning for Malignancy Characterization of Hepatocellualr Carcinoma

Deep learning with Convolutional Neural Network (CNN) has exhibited high diagnostic performance for lesion characterization. However, it is still challenging to train powerful deep learning systems for lesion characterization, because there are often limited samples in different malignancy types and there exist considerable variabilities across images from multiple scanners in clinical practice. In this work, we propose a similarity steered generative adversarial network (SSGAN) coupled with pre-train and adaptive fine-turning of data from multiple scanners for lesion characterization. Specifically, SSGAN is based on adding a similarity discriminative measure in the conventional generative adversarial network to effectively generate more discrepant samples, while the adaptive fine-tune strategy is adopted to optimally make decisions on whether to use the pre-train layers or the fine-tune layers. Experimental results of pathologically confirmed malignancy of clinical hepatocellular carcinoma (HCCs) with MR images acquired by different scanners (GE, Philips and Siemens) demonstrate several intriguing characteristics of the proposed end-to-end framework for malignancy characterization of HCC as follows: (1) The proposed SSGAN remarkably improves the performance of lesion characterization and outperforms several recently proposed methods. (2) The adaptive fine-tuning combined with the proposed SSGAN can further improve the performance of lesion characterization in the context of limited data. (3) Clinical images acquired by one MR scanner for pre-train can be used to improve the characterization performance of images acquired by another MR scanner, outperforming the pre-train with ImageNet.