Approaches to Sampling for Quality Control of Artificial Intelligence in Biomedical Research

The aim of the study is to evaluate the efficacy of approaches to sampling during periodic quality control of the artificial intelligence (AI) results in biomedical practice.

Materials and Methods. The approaches to sampling based on point statistical estimation, statistical hypothesis testing, employing ready-made statistical tables, as well as options of the approaches presented in GOST R ISO 2859-1-2007 “Statistical methods. Sampling procedures for inspection by attributes” have been analyzed. We have considered variants of sampling of different sizes for general populations from 1000 to 100,000 studies.

The analysis of the approaches to sampling was carried out as part of an experiment on the use of innovative technologies in computer vision for the analysis of medical images and their further application in the healthcare system of Moscow (Russia).

Results. Ready-made tables have specific statistical input data, which does not make them a universal option for biomedical research. Point statistical estimation helps to calculate a sample based on given statistical parameters with a certain confidence interval. This approach is promising in the case when only a type I error is important for the researcher, and a type II error is not a priority. Using the approach based on statistical hypothesis testing makes it possible to take account of type I and II errors based on the given statistical parameters. The application of GOST R ISO 2859-1-2007 for sampling allows using ready-made values depending on the given statistical parameters.

When evaluating the efficacy of the studied approaches, it was found that for our purposes, the optimal number of studies during AI quality control for the analysis of medical images is 80 items. This meets the requirements of representativeness, balance of the risks to the consumer and the AI service provider, as well as optimization of labor costs of employees involved in the process of quality control of the AI results.

1. Vasyuta E.A., Podolskaya T.V. Challenges and prospects for the introduction of artificial intelligence in medicine. Gosudarstvennoe i municipal’noe upravlenie. Ucenye zapiski 2022; 1: 25–32, https://doi.org/10.22394/2079-1690-2022-1-1-25-32.
2. Yarmukhametov R.R. Overview of usages of artificial intelligence in medicine. Naukosfera 2020; 12–2: 172–178.
3. Alekseeva M.G., Zubov A.I., Novikov M.Yu. Artificial intelligence in medicine. Mezdunarodnyj naucno-issledovatel’skij zurnal 2022; 7–2: 10–13, https://doi.org/10.23670/irj.2022.121.7.038.
4. Karpov O.E., Andrikov D.A., Maksimenko V.A., Hramov A.E. Explainable artificial intelligence for medicine. Vrac i informacionnye tehnologii 2022; 2: 4–11, https://doi.org/10.25881/18110193_2022_2_4.
5. Afimina K.G., Kushnirchuk I.I. Application of artificial intelligence methods in medicine. Izvestia Rossijskoj voenno-medicinskoj akademii 2021; 40(S1–S3): 17–19.
6. Elizarova M.I., Urazova K.M., Ermashov S.N., Pronkin N.N. Artificial intelligence in medicine. International Journal of Professional Science 2021; 5: 81–85.
7. Gusev A.V., Vladzymyrskyy A.V., Sharova D.E., Arzamasov K.M., Khramov A.E. Evolution of research and development in the field of artificial intelligence technologies for healthcare in the Russian Federation: results of 2021. Digital Diagnostics 2022; 3(3): 178–194, https://doi.org/10.17816/dd107367.
8. Morozov S.P., Vladzimirskiy A.V., Klyashtornyy V.G., Andreychenko A.E., Kul’berg N.S., Gombolevskiy V.A., Sergunova K.A. Klinicheskie ispytaniya programmnogo obespecheniya na osnove intellektual’nykh tekhnologiy (luchevaya diagnostika). Seriya “Luchshie praktiki luchevoy i instrumental’noy diagnostiki”. Vyp. 57 [Clinical trials of software based on intelligent technologies (radiology). Series “Best practices of radiological and instrumental diagnostics”. Issue 57]. Moscow; 2019; 51 p.
9. Prikaz Ministerstva zdravookhraneniya RF ot 15.09.2020 No.980n “Ob utverzhdenii Poryadka osushchestvleniya monitoringa bezopasnosti meditsinskikh izdeliy” [Order of the Ministry of Health of the Russian Federation of September 15, 2020 No.980n “On approval of the Procedure for monitoring the safety of medical devices”].
10. Reshenie Kollegii Evraziyskoy ekonomicheskoy komissii ot 22.12.2015 No.174 “Ob utverzhdenii Pravil provedeniya monitoringa bezopasnosti, kachestva i effektivnosti meditsinskikh izdeliy” [Decision of the Board of the Eurasian Economic Commission of December 22, 2015 No.174 “On approval of the Rules for monitoring the safety, quality and efficiency of medical devices”].
11. World Health Organization. Guidance for post-market surveillance and market surveillance of medical devices, including in-vitro-diagnostics. WHO; 2021. URL: https://www.who.int/docs/default-source/essential- medicines/in-vitro-diagnostics/draft- public-pmsdevices.pdf?sfvrsn=f803f68a_2.
12. European Commission. Guidance on Clinical Evaluation (MDR)/Performance Evaluation (IVDR) of Medical Device Software. Luxembourg; 2020. URL: https://ec.europa.eu/health/system/files/2020- 09/md_mdcg_2020_1_guidance_clinic_eva_md_software_en_0.pdf.
13. U.S. Food and Drug Administration. Postmarket Surveillance Under Section 522 of the Federal Food, Drug, and Cosmetic Act. Guidance for Industry and Food and Drug Administration Staff. 2021. URL: https://www.fda.gov/media/81015/download.
14. Federal’nyy zakon ot 21.11.2011 No.323-FZ “Ob osnovakh okhrany zdorov’ya grazhdan v Rossiyskoy Federatsii” (v red. ot 01.01.2022) [Federal Law of November 21, 2011 No.323-FZ “On the basics of protecting the health of citizens in the Russian Federation” (as amended on January 1, 2022)].
15. Benjamens S., Dhunnoo P., Meskó B. The state of artificial intelligence-based FDA-approved medical devices and algorithms: an online database. NPJ Digit Med 2020; 3: 118, https://doi.org/10.1038/s41746-020-00324-0.
16. Kelly C.J., Karthikesalingam A., Suleyman M., Corrado G., King D. Key challenges for delivering clinical impact with artificial intelligence. BMC Med 2019; 17(1): 195, https://doi.org/10.1186/s12916-019-1426-2.
17. U.S. Food and Drug Administration. Proposed Regulatory Framework for Modifications to Artificial Intelligence Machine Learning (AI ML)-Based Software as a Medical Device (SaMD). Discussion Paper and Request for Feedback. 2019. URL: https://www.fda.gov/media/122535/download.
18. IMDRF Software as a Medical Device (SaMD) Working Group. “Software as a Medical Device”: Possible Framework for Risk Categorization and Corresponding Considerations. 2014. URL: https://www.imdrf.org/sites/default/files/docs/ imdrf/final/technical/imdrf-tech-140918-samd- framework-risk-categorization-141013.pdf.
19. Park Y., Jackson G.P., Foreman M.A., Gruen D., Hu J., Das A.K. Evaluating artificial intelligence in medicine: phases of clinical research. JAMIA Open 2020; 3(3): 326–331, https://doi.org/10.1093/jamiaopen/ooaa033.
20. Article 78 — post-market surveillance system of the manufacturer. URL: https://lexparency.org/eu/32017R0746/ART_78.
21. Florey С.D. Sample size for beginners. BMJ 1993; 306(6886): 1181–1184, https://doi.org/10.1136/bmj.306.6886.1181.
22. Adler Yu.P. Sample: “all or nothing”. Kontrol’ kacestva produkcii 2015; 8: 26–32.
23. Adler Yu.P. Is your sample representative? Kontrol’ kacestva produkcii 2016; 5: 39–43.
24. Bandarenkо N.N., Pisaryk V.M., Atrashkevich T.I., Novik I.I. Forming of the representative sample for steps-survey in the republic of Belarus. Voprosy organizacii i informatizacii zdravoohranenia 2018; 2: 30–38.
25. Burmeister E., Aitken L. Sample size: how many is enough? Aust Crit Care 2012; 25(4): 271–274, https://doi.org/10.1016/j.aucc.2012.07.002.
26. Naing L., Winn T., Rusli B.N. Practical issues in calculating the sample size for prevalence studies. Arch Orofac Sci 2006; 1: 9–14.
27. Braganza O. Economically rational sample-size choice and irreproducibility. arXiv; 2019; URL: https://arxiv.org/pdf/1908.08702v2.pdf.
28. Singh A.S., Masuku M.B. Sampling techniques & determination of sample size in applied statistics research: an overview. Int J Economics Commerce Manag 2014; 2(11): 1–22.
29. Lwanga S.K., Lemeshow S. Sample size determination in health studies. World Health Organization; 1991; 80 p.
30. Kim J., Seo B.S. How to calculate sample size and why. Clin Orthop Surg 2013; 5(3): 235–242, https://doi.org/10.4055/cios.2013.5.3.235.
31. Sharafutdinova N.Kh., Kireeva E.F., Nikolaeva I.E., Pavlova M.Yu., Khalfin R.M., Sharafutdinov M.A., Borisova M.V., Latypov A.B., Galikeeva A.Sh. Statisticheskie metody v meditsine i zdravookhranenii [Statistical methods in medicine and public health]. Ufa: FGBOU VO BGMU Minzdrava Rossii; 2018; 131 p.
32. Skaff P.A., Sloan J. Design and analysis of equivalence clinical trials via the SAS system. Proc SUGI 1998; 23: 1166–1171.
33. Koichubekov B.K., Sorokina M.A., Mkhitaryan X.E. Sample size determination in planning of scientific research. Mezdunarodnyj zurnal prikladnyh i fundamental’nyh issledovanij 2014; 4: 71–74.
34. Noordzij M., Tripepi G., Dekker F.W., Zoccali C., Tanck M.W., Jager K.J. Sample size calculations: basic principles and common pitfalls. Nephrol Dial Transplant 2010; 25(5): 1388–1393, https://doi.org/10.1093/ndt/gfp732.
35. Cody J. Sample size calculation using SAS®, R, and nQuery software. SAS Global Forum; 2020. URL: https://www.sas.com/content/dam/SAS/support/en/ sas-global-forum-proceedings/2020/4675-2020.pdf.
36. Tarakanova V.V., Naumkin B.I. Formation of the sample population. Eksperiment i innovacii v skole 2009; 3: 46–49.
37. Israel G.D. Determining sample size. Florida: University of Florida, IFAS extension; 2012.
38. Kadam P., Bhalerao S. Sample size calculation. Int J Ayurveda Res 2010; 1(1): 55–57, https://doi.org/10.4103/0974-7788.59946.
39. Dell R.B., Holleran S., Ramakrishnan R. Sample size determination. ILAR J 2002; 43(4): 207–213, https://doi.org/10.1093/ilar.43.4.207.
40. Lakens D. Sample size justification. Collabra Psychol 2022; 8(1): 1–32.
41. Kirby A., Gebski V., Keech A.C. Determining the sample size in a clinical trial. Med J Aust 2002; 177(5): 256–257, https://doi.org/10.5694/j.1326-5377.2002.tb04759.x.
42. Jones S.R., Carley S., Harrison M. An introduction to power and sample size estimation. Emerg Med J 2003; 20(5): 453–458, https://doi.org/10.1136/emj.20.5.453.
43. Rebrova O.Yu., Gusev A.V. Sample size calculation for clinical trials of medical decision support systems with binary outcome. Sovremennye tehnologii v medicine 2022; 14(3): 6, https://doi.org/10.17691/stm2022.14.3.01.
44. Schilling Е.G., Neubauer D.V. Acceptance sampling in quality control. Taylor & Francis Group, LLC; 2008; 709 p.
45. Polunina N.V. Obshchestvennoe zdorov’e i zdravo­okhranenie [Public health and healthcare]. Moscow: Meditsinskoe informatsionnoe agentstvo; 2010; 544 p.
46. Narkevich A.N., Vinogradov K.A. Methods for determining the minimum required sample size in medical research. Social’nye aspekty zdorov’a naselenia 2019; 65(6): 10.
47. Paniotto V.I., Maksimenko V.S. Kolichestvennye metody v sotsiologicheskikh issledovaniyakh [Quantitative methods in sociological research]. Kiev; 2003. URL: https://www.kiis.com.ua/materials/books/ 376072_C6170_paniotto_v_i_maksimenko_v_s_ kolichestvennye_metody_v_sociolo.pdf.
48. Syrtsova L.E., Kosagovskaya I.I., Avksent’eva M.V. Osnovy epidemiologii i statisticheskogo analiza v obshchestvennom zdorov’e i upravlenii zdravookhraneniem [Fundamentals of epidemiology and statistical analysis in public health and health management]. Moscow; 2003; 91 p.
49. Agisheva D.K., Zotova S.A., Matveeva T.A., Svetlichnaya V.B. Matematicheskaya statistica [Mathematical statistics]. Volgograd: VPI (filial) VolgGTU; 2010; 159 p.
50. Rumyantsev P.O., Saenko V.A., Rumyantseva U.V., Chekin S.Yu. Statisticheskie metody analiza v klinicheskoy praktike [Statistical methods of analysis in clinical practice]. 2009. URL: https://medstatistic.ru/articles/StatMethodsInClinics.pdf.
51. Taherdoost H. Determining sample size; how to calculate survey sample size. Int J Econ Manag Syst 2017; 2: 237–239.
52. Blackwelder W.C. Equivalence trials. In: Encyclopedia of biostatistics. Volume 2. New York: John Wiley and Sons; 1998; p. 1367–1372.
53. Chow S.C., Shao J., Wang H. Sample size calculations in clinical research. 2nd Edition. Florida: Chapman & Hall/CRC Biostatistics Series; 2008.
54. GOST R ISO 2859-1-2007. Statisticheskie metody. Protsedury vyborochnogo kontrolya po al’ternativnomu priznaku. Chast’ 1. Plany vyborochnogo kontrolya posledovatel’nykh partiy na osnove priemlemogo urovnya kachestva [Statistical methods. Sampling procedures for inspection by attributes. Part 1: sampling schemes indexed by acceptance quality limit for lot-by-lot inspection]. Moscow: Standartinform; 2007; 101 p.
55. Sharashkina T.P. Statisticheskie metody v upravlenii kachestvom [Statistical methods in quality management]. Saransk: Mordovskiy gosudarstvennyy universitet; 2013; 91 p.
56. Borodachev S.M. Statisticheskie metody v upravlenii kachestvom [Statistical methods in quality management]. Ekaterinburg: Izdatel’stvo Ural’skogo universiteta; 2016; 87 p.
57. Klyachkin V.N. Statisticheskie metody v upravlenii kachestvom [Statistical methods in quality management]. Ul’yanovsk: UlGTU; 2013; 156 p.
58. Efimov V.V. Osnovy berezhlivogo proizvodstva [Fundamentals of lean manufacturing]. Ul’yanovsk: UlGTU; 2011; 160 p.
59. Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department. Experiment on the use of innovative computer vision technologies for analysis of medical images in the Moscow healthcare system. URL: https://www.clinicaltrials.gov/ct2/show/NCT04489992.
60. Andreychenko A.E., Logunova T.A., Gombolevskiy V.A., Nikolaev A.E., Vladzymyrskyy A.V., Sinitsyn V.E., Morozov S.P. A methodology for selection and quality control of the radiological computer vision deployment at the megalopolis scale. medRxiv; 2010, https://doi.org/10.1101/2022.02.12.22270663.
61. Mezhdunarodnaya klassifikatsiya bolezney 10-go peresmotra (MKB-10) [International Classification of Diseases of the 10th Revision (ICD-10)]. 2021. URL: https://mkb-10.com.
62. Prikaz Departamenta zdravookhraneniya goroda Moskvy ot 24.02.2022 No.160 “Ob utverzhdenii Poryadka i usloviy provedeniya eksperimenta po ispol’zovaniyu innovatsionnykh tekhnologiy v oblasti komp’yuternogo zreniya dlya analiza meditsinskikh izobrazheniy i dal’neyshego primeneniya v sisteme zdravookhraneniya goroda Moskvy” [Order of the Department of Health of the city of Moscow dated February 24, 2022 No.160 “On approval of the Procedure and conditions for conducting an experiment on the use of innovative technologies in the field of computer vision for the analysis of medical images and further application in the health care system of the city of Moscow”].
63. Bavrina A.P. Basic concepts of statistics. Medicinskij al’manah 2020; 3: 101–111.
64. Koshevoy O.S., Karpova M.K. Sample size determination in the course of regional sociological research. Izvestia vyssih ucebnyh zavedenij. Povolzskij region 2011; 2: 98–104.
65. Fox N., Hunn A., Mathers N. Sampling and sample size calculation. Sheffield: Trent RDSU; 2007; 41 p.

Chetverikov S.F., Arzamasov K.M., Andreichenko A.E., Novik V.P., Bobrovskaya T.M., Vladzimirsky A.V. Approaches to Sampling for Quality Control of Artificial Intelligence in Biomedical Research. Sovremennye tehnologii v medicine 2023; 15(2): 19, https://doi.org/10.17691/stm2023.15.2.02