Study on the detection of water status of tomato (Solanum lycopersicum L.) by multimodal deep learning

Zhiyu Zuo; Jindong Mu; Wenjie Li; Quan Bu; Hanping Mao; Xiaodong Zhang; Lvhua Han; Jiheng Ni

doi:10.3389/fpls.2023.1094142

Study on the detection of water status of tomato (Solanum lycopersicum L.) by multimodal deep learning

Front Plant Sci. 2023 May 31:14:1094142. doi: 10.3389/fpls.2023.1094142. eCollection 2023.

Authors

Zhiyu Zuo^{1

2}, Jindong Mu¹, Wenjie Li¹, Quan Bu¹, Hanping Mao^{1

2}, Xiaodong Zhang¹, Lvhua Han¹, Jiheng Ni²

Affiliations

¹ School of Agricultural Engineering, Jiangsu University, Zhenjiang, China.
² Key Laboratory of Modern Agricultural Equipment and Technology, Ministry of Education/High-tech Key Laboratory of Agricultural Equipment and Intelligence of Jiangsu Province, Jiangsu University, Zhenjiang, China.

Abstract

Water plays a very important role in the growth of tomato (Solanum lycopersicum L.), and how to detect the water status of tomato is the key to precise irrigation. The objective of this study is to detect the water status of tomato by fusing RGB, NIR and depth image information through deep learning. Five irrigation levels were set to cultivate tomatoes in different water states, with irrigation amounts of 150%, 125%, 100%, 75%, and 50% of reference evapotranspiration calculated by a modified Penman-Monteith equation, respectively. The water status of tomatoes was divided into five categories: severely irrigated deficit, slightly irrigated deficit, moderately irrigated, slightly over-irrigated, and severely over-irrigated. RGB images, depth images and NIR images of the upper part of the tomato plant were taken as data sets. The data sets were used to train and test the tomato water status detection models built with single-mode and multimodal deep learning networks, respectively. In the single-mode deep learning network, two CNNs, VGG-16 and Resnet-50, were trained on a single RGB image, a depth image, or a NIR image for a total of six cases. In the multimodal deep learning network, two or more of the RGB images, depth images and NIR images were trained with VGG-16 or Resnet-50, respectively, for a total of 20 combinations. Results showed that the accuracy of tomato water status detection based on single-mode deep learning ranged from 88.97% to 93.09%, while the accuracy of tomato water status detection based on multimodal deep learning ranged from 93.09% to 99.18%. The multimodal deep learning significantly outperformed the single-modal deep learning. The tomato water status detection model built using a multimodal deep learning network with ResNet-50 for RGB images and VGG-16 for depth and NIR images was optimal. This study provides a novel method for non-destructive detection of water status of tomato and gives a reference for precise irrigation management.

Keywords: depth images; multimodal deep learning; nondestructive detection; tomato; water status.

Grants and funding

This work was supported by the earmarked fund for CARS (CARS-23-D03) and the Agricultural Independent Innovation Fund of Jiangsu province (CX (20)1005).