Fruit detection for strawberry harvesting robot in non-structural environment based on Mask-RCNN

Abstract Deep learning has demonstrated excellent capabilities for learning image features and is widely used in image object detection. In order to improve the performance of machine vision in fruit detection for a strawberry harvesting robot, Mask Region Convolutional Neural Network (Mask-RCNN) was introduced. Resnet50 was adopted as backbone network, combined with the Feature Pyramid Network (FPN) architecture for feature extraction. The Region Proposal Network (RPN) was trained end-to-end to create region proposals for each feature map. After generating mask images of ripe fruits from Mask R-CNN, a visual localization method for strawberry picking points was performed. Fruit detection results of 100 test images showed that the average detection precision rate was 95.78%, the recall rate was 95.41% and the mean intersection over union ( MIoU ) rate for instance segmentation was 89.85%. The prediction results of 573 ripe fruit picking points showed that the average error was ±1.2 mm. Compared with four traditional methods, the method proposed demonstrates improved universality and robustness in a non-structural environment, particularly for overlapping and hidden fruits, and those under varying illumination.