Semantic segmentation of pollen grain images generated from scattering patterns via deep learning

Pollen can lead to individuals suffering from allergic rhinitis, with a person's vulnerability being dependent on the species and the amount of pollen. Therefore, the ability to precisely quantify both the number and species of pollen grains in a certain volume would be invaluable. Lensless sensing offers the ability to classify pollen grains from their scattering patterns, with the use of very few optical components. However, since there could be 1000s of species of pollen one may wish to identify, in order to avoid having to collect scattering patterns from all species (and mixtures of species) we propose using two separate neural networks. The first neural network generates a microscope equivalent image from the scattering pattern, having been trained on a limited number of experimentally collected pollen scattering data. The second neural network segments the generated image into its components, having been trained on microscope images, allowing pollen species identification (potentially allowing the use of existing databases of microscope images to expand range of species identified by the segmentation network). In addition to classification, segmentation also provides richer information, such as the number of pixels and therefore the potential size of particular pollen grains. Specifically, we demonstrate the identification and projected area of pollen grain species, via semantic image segmentation, in generated microscope images of pollen grains, containing mixtures and species that were previously unseen by the image generation network. The microscope images of mixtures of pollen grains, used for training the segmentation neural network, were created by fusing microscope images of isolated pollen grains together while the trained neural network was tested on microscope images of actual mixtures. The ability to carry out pollen species identification from reconstructed images without needing to train the identification network on the scattering patterns is useful for the real-world implementation of such technology.