Performance of ChatGPT on the National Korean Occupational Therapy Licensing Examination

Si-An Lee; Seoyoon Heo; Jin-Hyuck Park

doi:10.1177/20552076241236635

Performance of ChatGPT on the National Korean Occupational Therapy Licensing Examination

Digit Health. 2024 Feb 29:10:20552076241236635. doi: 10.1177/20552076241236635. eCollection 2024 Jan-Dec.

Authors

Si-An Lee¹, Seoyoon Heo², Jin-Hyuck Park³

Affiliations

¹ Department of ICT convergence, The Graduate School, Soonchunhyang University, Asan, Republic of Korea.
² Department of Occupational Therapy, Kyungbok University, Namyangju, Republic of Korea.
³ Department of Occupational Therapy, College of Medical Science, Soonchunhyang University, Asan, Republic of Korea.

Abstract

Background: ChatGPT is an artificial intelligence-based large language model (LLM). ChatGPT has been widely applied in medicine, but its application in occupational therapy has been lacking.

Objective: This study examined the accuracy of ChatGPT on the National Korean Occupational Therapy Licensing Examination (NKOTLE) and investigated its potential for application in the field of occupational therapy.

Methods: ChatGPT 3.5 was used during the five years of the NKOTLE with Korean prompts. Multiple choice questions were entered manually by three dependent encoders, and scored according to the number of correct answers.

Results: During the most recent five years, ChatGPT did not achieve a passing score of 60% accuracy and exhibited interrater agreement of 0.6 or higher.

Conclusion: ChatGPT could not pass the NKOTLE but demonstrated a high level of agreement between raters. Even though the potential of ChatGPT to pass the NKOTLE is currently inadequate, it performed very close to the passing level even with only Korean prompts.

Keywords: ChatGPT; artificial intelligence; large language models; licensing examination; occupational therapy.