Multimodal Gated Mixture of Experts Using Whole Slide Image and Flow Cytometry for Multiple Instance Learning Classification of Lymphoma

Noriaki Hashimoto; Hiroyuki Hanada; Hiroaki Miyoshi; Miharu Nagaishi; Kensaku Sato; Hidekata Hontani; Koichi Ohshima; Ichiro Takeuchi

doi:10.1016/j.jpi.2023.100359

Multimodal Gated Mixture of Experts Using Whole Slide Image and Flow Cytometry for Multiple Instance Learning Classification of Lymphoma

J Pathol Inform. 2023 Dec 29:15:100359. doi: 10.1016/j.jpi.2023.100359. eCollection 2024 Dec.

Authors

Noriaki Hashimoto¹, Hiroyuki Hanada¹, Hiroaki Miyoshi², Miharu Nagaishi², Kensaku Sato², Hidekata Hontani³, Koichi Ohshima², Ichiro Takeuchi^{1

4}

Affiliations

¹ RIKEN Center for Advanced Intelligence Project, Furo-cho, Chikusa-ku, Nagoya, 4648603, Japan.
² Department of Pathology, Kurume University School of Medicine, 67 Asahi-machi, Kurume, 8300011, Japan.
³ Department of Computer Science, Nagoya Institute of Technology, Gokiso-cho, Showa-ku, Nagoya, 4668555, Japan.
⁴ Department of Mechanical Systems Engineering, Nagoya University, Furo-cho, Chikusa-ku, Nagoya, 4648603, Japan.

Abstract

In this study, we present a deep-learning-based multimodal classification method for lymphoma diagnosis in digital pathology, which utilizes a whole slide image (WSI) as the primary image data and flow cytometry (FCM) data as auxiliary information. In pathological diagnosis of malignant lymphoma, FCM serves as valuable auxiliary information during the diagnosis process, offering useful insights into predicting the major class (superclass) of subtypes. By incorporating both images and FCM data into the classification process, we can develop a method that mimics the diagnostic process of pathologists, enhancing the explainability. In order to incorporate the hierarchical structure between superclasses and their subclasses, the proposed method utilizes a network structure that effectively combines the mixture of experts (MoE) and multiple instance learning (MIL) techniques, where MIL is widely recognized for its effectiveness in handling WSIs in digital pathology. The MoE network in the proposed method consists of a gating network for superclass classification and multiple expert networks for (sub)class classification, specialized for each superclass. To evaluate the effectiveness of our method, we conducted experiments involving a six-class classification task using 600 lymphoma cases. The proposed method achieved a classification accuracy of 72.3%, surpassing the 69.5% obtained through the straightforward combination of FCM and images, as well as the 70.2% achieved by the method using only images. Moreover, the combination of multiple weights in the MoE and MIL allows for the visualization of specific cellular and tumor regions, resulting in a highly explanatory model that cannot be attained with conventional methods. It is anticipated that by targeting a larger number of classes and increasing the number of expert networks, the proposed method could be effectively applied to the real problem of lymphoma diagnosis.

Keywords: Digital pathology; Flow cytometry; Mixture of experts; Multiple instance learning; Whole slide image.