GPT-4 Turbo with Vision fails to outperform text-only GPT-4 Turbo in the Japan Diagnostic Radiology Board Examination

Yuichiro Hirano; Shouhei Hanaoka; Takahiro Nakao; Soichiro Miki; Tomohiro Kikuchi; Yuta Nakamura; Yukihiro Nomura; Takeharu Yoshikawa; Osamu Abe

doi:10.1007/s11604-024-01561-z

GPT-4 Turbo with Vision fails to outperform text-only GPT-4 Turbo in the Japan Diagnostic Radiology Board Examination

Jpn J Radiol. 2024 May 11. doi: 10.1007/s11604-024-01561-z. Online ahead of print.

Authors

Yuichiro Hirano^{1

2}, Shouhei Hanaoka³, Takahiro Nakao⁴, Soichiro Miki⁴, Tomohiro Kikuchi^{4

5}, Yuta Nakamura⁴, Yukihiro Nomura^{4

6}, Takeharu Yoshikawa⁴, Osamu Abe³

Affiliations

¹ Department of Radiology, The International University of Health and Welfare Narita Hospital, 852 Hatakeda, Narita, Chiba, Japan. yhirano-tky@umin.ac.jp.
² Department of Radiology, the University of Tokyo Hospital, 7-3-1 Hongo, Bunkyo-Ku, Tokyo, Japan. yhirano-tky@umin.ac.jp.
³ Department of Radiology, the University of Tokyo Hospital, 7-3-1 Hongo, Bunkyo-Ku, Tokyo, Japan.
⁴ Department of Computational Diagnostic Radiology and Preventive Medicine, the University of Tokyo Hospital, 7-3-1 Hongo, Bunkyo-Ku, Tokyo, Japan.
⁵ Department of Radiology, School of Medicine, Jichi Medical University, 3311-1 Yakushiji, Shimotsuke, Tochigi, Japan.
⁶ Center for Frontier Medical Engineering, Chiba University, 1-33 Yayoicho, Inage-Ku, Chiba, Japan.

PMID: 38733472
DOI: 10.1007/s11604-024-01561-z

Abstract

Purpose: To assess the performance of GPT-4 Turbo with Vision (GPT-4TV), OpenAI's latest multimodal large language model, by comparing its ability to process both text and image inputs with that of the text-only GPT-4 Turbo (GPT-4 T) in the context of the Japan Diagnostic Radiology Board Examination (JDRBE).

Materials and methods: The dataset comprised questions from JDRBE 2021 and 2023. A total of six board-certified diagnostic radiologists discussed the questions and provided ground-truth answers by consulting relevant literature as necessary. The following questions were excluded: those lacking associated images, those with no unanimous agreement on answers, and those including images rejected by the OpenAI application programming interface. The inputs for GPT-4TV included both text and images, whereas those for GPT-4 T were entirely text. Both models were deployed on the dataset, and their performance was compared using McNemar's exact test. The radiological credibility of the responses was assessed by two diagnostic radiologists through the assignment of legitimacy scores on a five-point Likert scale. These scores were subsequently used to compare model performance using Wilcoxon's signed-rank test.

Results: The dataset comprised 139 questions. GPT-4TV correctly answered 62 questions (45%), whereas GPT-4 T correctly answered 57 questions (41%). A statistical analysis found no significant performance difference between the two models (P = 0.44). The GPT-4TV responses received significantly lower legitimacy scores from both radiologists than the GPT-4 T responses.

Conclusion: No significant enhancement in accuracy was observed when using GPT-4TV with image input compared with that of using text-only GPT-4 T for JDRBE questions.

Keywords: Artificial intelligence (AI); ChatGPT; GPT-4 Turbo; GPT-4 Turbo with Vision; Japan Diagnostic Radiology Board Examination (JDRBE); Large language model (LLM).