Real-time dense small object detection algorithm based on multi-modal tea shoots

Luyu Shuai; Ziao Chen; Zhiyong Li; Hongdan Li; Boda Zhang; Yuchao Wang; Jiong Mu

doi:10.3389/fpls.2023.1224884

Real-time dense small object detection algorithm based on multi-modal tea shoots

Front Plant Sci. 2023 Jul 18:14:1224884. doi: 10.3389/fpls.2023.1224884. eCollection 2023.

Authors

Luyu Shuai^{1

2}, Ziao Chen³, Zhiyong Li^{1

2}, Hongdan Li^{1

2}, Boda Zhang^{1

2}, Yuchao Wang⁴, Jiong Mu^{1

2}

Affiliations

¹ College of Information Engineering, Sichuan Agricultural University, Ya'an, China.
² Ya'an Digital Agricultural Engineering Technology Research Center, Sichuan Agricultural University, Ya'an, China.
³ College of Law, Sichuan Agricultural University, Ya'an, China.
⁴ College of Mechanical and Electrical Engineering, Sichuan Agricultural University, Ya'an, China.

Abstract

Introduction: The difficulties in tea shoot recognition are that the recognition is affected by lighting conditions, it is challenging to segment images with similar backgrounds to the shoot color, and the occlusion and overlap between leaves.

Methods: To solve the problem of low accuracy of dense small object detection of tea shoots, this paper proposes a real-time dense small object detection algorithm based on multimodal optimization. First, RGB, depth, and infrared images are collected form a multimodal image set, and a complete shoot object labeling is performed. Then, the YOLOv5 model is improved and applied to dense and tiny tea shoot detection. Secondly, based on the improved YOLOv5 model, this paper designs two data layer-based multimodal image fusion methods and a feature layerbased multimodal image fusion method; meanwhile, a cross-modal fusion module (FFA) based on frequency domain and attention mechanisms is designed for the feature layer fusion method to adaptively align and focus critical regions in intra- and inter-modal channel and frequency domain dimensions. Finally, an objective-based scale matching method is developed to further improve the detection performance of small dense objects in natural environments with the assistance of transfer learning techniques.

Results and discussion: The experimental results indicate that the improved YOLOv5 model increases the mAP50 value by 1.7% compared to the benchmark model with fewer parameters and less computational effort. Compared with the single modality, the multimodal image fusion method increases the mAP50 value in all cases, with the method introducing the FFA module obtaining the highest mAP50 value of 0.827. After the pre-training strategy is used after scale matching, the mAP values can be improved by 1% and 1.4% on the two datasets. The research idea of multimodal optimization in this paper can provide a basis and technical support for dense small object detection.

Keywords: RGB-D-IR; attention mechanism; dense small object detection; frequency domain; multimodal image fusion; scale matching; tea shoots.

Grants and funding

This work was funded by the Research and application of key technologies for intelligent spraying based on machine vision (key technology research project) of Sichuan Provincial Department of Science and Technology (grant number 22ZDYF0095).