Detection of strawberries with varying maturity levels for robotic harvesting using YOLOv4

Abstract. Inaccurate detection and/or localization of strawberries will cause fruit injury or failed attempt during robotic picking. In this work, a method is proposed to accurately detect and localize strawberries combining an object detection network, YOLOv4, and a classification network, Alexnet. YOLOv4 model was used to detect strawberries into various maturity groups (flower, immature, nearly mature, mature, and overripen berries) and provide their location information whereas Alexnet model was used for assessing if the detected matured strawberries were completely or partially visible. Compared to the same achieved by YOLOv2, YOLOv3 and YOLOv4 trained for regular objects,YOLOv4 model was specifically trained to detect small objects, which achieved the highest mean average precision of 80.7% and F1 score of 0.80 with an average precision (AP) of 91.7% in detecting mature strawberries. YOLOv4 achieved a high processing speed of 55 ms on single image (resolution: 1200x1000 pixels). The model was further validated with the RGB images generated from point cloud, where it achieved an AP of 90.15% showing that the model was robust to detect berries in images collected with different settings. Similarly, Alexnet model achieved an accuracy of 90.0% in classifying matured strawberries into completely and partially visible groups with processing speed of 3 ms per image (resolution: 227x227x3 pixels). This technique also provided 2D to 3D mapping of strawberries for the harvesting robots. This method showed a strong potential as a means for providing accurate strawberry detection desirable for robotic harvesters, particularly with a collaborating dual-manipulator system.