Segmentation, Detection, and Classification of Cell Nuclei on Oral Cytology Samples Stained with Papanicolaou

Although oral cancer is considered a global health issue with 350,000 people diagnosed over a year, it can successfully be treated if diagnosed at early stages. Papanicolaou is an inexpensive and non-invasive method, generally applied to detect cervical cancer, but it can also be useful to detect cancer on oral cavities. The manual process of analyzing cells to detect abnormalities is a time-consuming cell analysis and is subject to variations in perceptions from different professionals. This paper compares three different deep learning (DL) approaches: segmentation, object detection, and image classification. Our results show that the binary object detection with Faster R-CNN is the best approach for nuclei detection and localization (0.76 IoU). Since ResNet 34 had a good performance on abnormal nuclei classification (0.86 $$F_1$$ score), we concluded that these two models can be used in combination to perform a reliable localization and classification pipeline. This work reinforces that the automated analysis of oral cytology to build a pipeline for nuclei classification and localization using DL can contribute to minimize the subjectivity of the human analysis and also support the detection of cancer at early stages.