A fine recognition method of strawberry ripeness combining Mask R-CNN and region segmentation

Can Tang; Du Chen; Xin Wang; Xindong Ni; Yehong Liu; Yihao Liu; Xu Mao; Shumao Wang

doi:10.3389/fpls.2023.1211830

A fine recognition method of strawberry ripeness combining Mask R-CNN and region segmentation

Front Plant Sci. 2023 Jul 28:14:1211830. doi: 10.3389/fpls.2023.1211830. eCollection 2023.

Authors

Can Tang¹, Du Chen^{1

2}, Xin Wang¹, Xindong Ni¹, Yehong Liu¹, Yihao Liu¹, Xu Mao^{1

3}, Shumao Wang^{1

3}

Affiliations

¹ College of Engineering, China Agricultural University, Beijing, China.
² State Key Laboratory of Intelligent Agricultural Power Equipment, Henan, China.
³ Beijing Key Laboratory of Optimized Design for Modern Agricultural Equipment, Beijing, China.

Abstract

As a fruit with high economic value, strawberry has a short ripeness period, and harvesting at an incorrect time will seriously affect the quality of strawberries, thereby reducing economic benefits. Therefore, the timing of its harvesting is very demanding. A fine ripeness recognition can provide more accurate crop information, and guide strawberry harvest management more timely and effectively. This study proposes a fine recognition method for field strawberry ripeness that combines deep learning and image processing. The method is divided into three stages: In the first stage, self-calibrated convolutions are added to the Mask R-CNN backbone network to improve the model performance, and then the model is used to extract the strawberry target in the image. In the second stage, the strawberry target is divided into four sub-regions by region segmentation method, and the color feature values of B, G, L, a and S channels are extracted for each sub-region. In the third stage, the strawberry ripeness is classified according to the color feature values and the results are visualized. Experimental results show that with the incorporation of self-calibrated convolutions into the Mask R-CNN, the model's performance has been substantially enhanced, leading to increased robustness against diverse occlusion interferences. As a result, the final average precision (AP) has improved to 0.937, representing a significant increase of 0.039 compared to the previous version. The strawberry ripeness classification effect is the best on the SVM classifier, and the accuracy under the combined channel BGLaS reaches 0.866. The classification results are better than common manual feature extraction methods and AlexNet, ResNet18 models. In order to clarify the role of the region segmentation method, the contribution of different sub-regions to each ripeness is also explored. The comprehensive results demonstrate that the proposed method enables the evaluation of six distinct ripeness levels of strawberries in the complex field environment. This method can provide accurate decision support for strawberry refined planting management.

Keywords: Mask R-CNN; deep learning; image processing; ripeness recognition; strawberry.

Grants and funding

This work was funded by the National Key Research and Development Program of China [No. 2022YFD2001203], the project of agricultural machinery R & D, manufacturing, promotion, application and the integration, and the National Natural Science Foundation of China [No. 32201687]. This work was supported in part by College of Engineering, China Agricultural University (CAU).