Transfer Language Space with Similar Domain Adaptation: A Case Study with Hepatocellular Carcinoma

Transfer learning is a common practice in image classification with deep learning where the available data is often limited for training a complex model with millions of parameters. However, transferring language models requires special attention since cross-domain vocabularies (e.g. between news articles and radiology reports) do not always overlap as the pixel intensity range overlaps mostly for images. We present a concept of similar domain adaptation where we transfer an interinstitutional language model between two different modalities (ultrasound to MRI) to capture liver abnormalities. Our experiments show that such transfer is more effective for performing shared targeted task than generic language space transfer. We use MRI screening exam reports for hepatocellular carcinoma as the use-case and apply the transfer language space strategy to automatically label thousands of imaging exams.