Automated Nuclei Segmentation in Dysplastic Histopathological Oral Tissues Using Deep Neural Networks

Dysplasia is a common pre-cancerous abnormality that can be categorized as mild, moderate and severe. With the advance of digital systems applied in microscopes for histological analysis, specialists can obtain data that allows investigation using computational algorithms. These systems are known as computer-aided diagnosis, which provide quantitative analysis in a large number of data and features. This work proposes a method for nuclei segmentation for histopathological images of oral dysplasias based on an artificial neural network model and post-processing stage. This method employed nuclei masks for the training, where objects and bounding boxes were evaluated. In the post-processing step, false positive areas were removed by applying morphological operations, such as dilation and erosion. This approach was applied in a dataset with 296 regions of mice tongue images. The metrics accuracy, sensitivity, specificity, the Dice coefficient and correspondence ratio were employed for evaluation and comparison with other methods present in the literature. The results show that the method was able to segment the images with accuracy average value of \(89.52 \pm 0.04\%\) and Dice coefficient of \(84.03 \pm 0.06\%\). These values are important to indicate that the proposed method can be applied as a tool for nuclei analysis in oral cavity images with relevant precision values for the specialist.