Approaches to Sampling for Quality Control of Artificial Intelligence in Biomedical Research

S F Chetverikov; K M Arzamasov; A E Andreichenko; V P Novik; T M Bobrovskaya; A V Vladzimirsky

doi:10.17691/stm2023.15.2.02

Approaches to Sampling for Quality Control of Artificial Intelligence in Biomedical Research

Sovrem Tekhnologii Med. 2023;15(2):19-25. doi: 10.17691/stm2023.15.2.02. Epub 2023 Mar 29.

Authors

S F Chetverikov¹, K M Arzamasov², A E Andreichenko³, V P Novik⁴, T M Bobrovskaya⁵, A V Vladzimirsky⁶

Affiliations

¹ Head of the Sector of the Development of Systems for the Implementation of Intelligent Medical Technologies; Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health care Department, 24/1 Petrovka St., Moscow, 127051, Russia.
² Head of the Department of Medical Informatics, Radiomics and Radiogenomics; Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health care Department, 24/1 Petrovka St., Moscow, 127051, Russia.
³ Leading Researcher, Department of Medical Informatics, Radiomics and Radiogenomics; Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health care Department, 24/1 Petrovka St., Moscow, 127051, Russia; Head of the Group of Artificial Intelligence; K-SkAI LLC, 17 Naberezhnaya Varkausa, Petrozavodsk, the Republic of Karelia, 185031, Russia; Leading Researcher; ITMO National Research University, 49 Kronverksky Pr., Saint Petersburg, 197101, Russia.
⁴ Researcher, Sector of the Development of Systems for the Implementation of Intelligent Medical Technologies; Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health care Department, 24/1 Petrovka St., Moscow, 127051, Russia.
⁵ Junior Researcher, Sector of the Development of Systems for the Implementation of Intelligent Medical Technologies; Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health care Department, 24/1 Petrovka St., Moscow, 127051, Russia.
⁶ Deputy Director for Research; Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health care Department, 24/1 Petrovka St., Moscow, 127051, Russia; Professor, Department of Information and Internet Technologies; First Moscow State Medical University named after I.M. Sechenov (Sechenov University), 8/2 Trubetskaya St., Moscow, 119991, Russia.

Abstract

The aim of the study is to evaluate the efficacy of approaches to sampling during periodic quality control of the artificial intelligence (AI) results in biomedical practice.

Materials and methods: The approaches to sampling based on point statistical estimation, statistical hypothesis testing, employing ready-made statistical tables, as well as options of the approaches presented in GOST R ISO 2859-1-2007 "Statistical methods. Sampling procedures for inspection by attributes" have been analyzed. We have considered variants of sampling of different sizes for general populations from 1000 to 100,000 studies.The analysis of the approaches to sampling was carried out as part of an experiment on the use of innovative technologies in computer vision for the analysis of medical images and their further application in the healthcare system of Moscow (Russia).

Results: Ready-made tables have specific statistical input data, which does not make them a universal option for biomedical research. Point statistical estimation helps to calculate a sample based on given statistical parameters with a certain confidence interval. This approach is promising in the case when only a type I error is important for the researcher, and a type II error is not a priority. Using the approach based on statistical hypothesis testing makes it possible to take account of type I and II errors based on the given statistical parameters. The application of GOST R ISO 2859-1-2007 for sampling allows using ready-made values depending on the given statistical parameters.When evaluating the efficacy of the studied approaches, it was found that for our purposes, the optimal number of studies during AI quality control for the analysis of medical images is 80 items. This meets the requirements of representativeness, balance of the risks to the consumer and the AI service provider, as well as optimization of labor costs of employees involved in the process of quality control of the AI results.

Keywords: AI quality control; artificial intelligence; sampling; statistical methods.

MeSH terms

Artificial Intelligence
Biomedical Research*
Female
Humans
Interior Design and Furnishings
Labor, Obstetric*
Pregnancy
Quality Control

Grants and funding

Study funding. This article was prepared by the team of authors as part of a scientific and practical project in the field of medicine (No.EGISU: 122112400040-1) “Reference data sets for the sustainable development of artificial intelligence technologies in medical diagnostics aimed at minimization of the long-term effects of the coronavirus pandemic for the public health in the city of Moscow”.