OSC-CO2: coattention and cosegmentation framework for plant state change with multiple features

Rubi Quiñones; Ashok Samal; Sruti Das Choudhury; Francisco Muñoz-Arriola

doi:10.3389/fpls.2023.1211409

OSC-CO²: coattention and cosegmentation framework for plant state change with multiple features

Front Plant Sci. 2023 Oct 31:14:1211409. doi: 10.3389/fpls.2023.1211409. eCollection 2023.

Authors

Rubi Quiñones^{1

2}, Ashok Samal¹, Sruti Das Choudhury^{1

3}, Francisco Muñoz-Arriola^{3

4}

Affiliations

¹ School of Computing, University of Nebraska-Lincoln, Lincoln, NE, United States.
² Computer Science Department, Southern Illinois University Edwardsville, Edwardsville, IL, United States.
³ School of Natural Resources, University of Nebraska-Lincoln, Lincoln, NE, United States.
⁴ Department of Biological Systems Engineering, University of Nebraska-Lincoln, Lincoln, NE, United States.

Abstract

Cosegmentation and coattention are extensions of traditional segmentation methods aimed at detecting a common object (or objects) in a group of images. Current cosegmentation and coattention methods are ineffective for objects, such as plants, that change their morphological state while being captured in different modalities and views. The Object State Change using Coattention-Cosegmentation (OSC-CO²) is an end-to-end unsupervised deep-learning framework that enhances traditional segmentation techniques, processing, analyzing, selecting, and combining suitable segmentation results that may contain most of our target object's pixels, and then displaying a final segmented image. The framework leverages coattention-based convolutional neural networks (CNNs) and cosegmentation-based dense Conditional Random Fields (CRFs) to address segmentation accuracy in high-dimensional plant imagery with evolving plant objects. The efficacy of OSC-CO² is demonstrated using plant growth sequences imaged with infrared, visible, and fluorescence cameras in multiple views using a remote sensing, high-throughput phenotyping platform, and is evaluated using Jaccard index and precision measures. We also introduce CosegPP+, a dataset that is structured and can provide quantitative information on the efficacy of our framework. Results show that OSC-CO² out performed state-of-the art segmentation and cosegmentation methods by improving segementation accuracy by 3% to 45%.

Keywords: cosegmentation; high-throughput plant phenotyping; image analysis; image sequences; multiple dimensions; multiple features; object state change; segmentation.

Grants and funding

This material is based upon work supported by the National Science Foundation under Grant No. DGE-1735362. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. Also, the authors acknowledge the support provided by the Agriculture and Food Research Initiative Grant number NEB-21-176 and NEB-21-166 from the USDA National Institute of Food and Agriculture, Plant Health and Production and Plant Products: Plant Breeding for Agricultural Production.