Russian English

Volume 18, Issue 3 (2026)

page 5
Integration of Transformer-Based Architecture and Large Language Models for Optical Coherence Tomography Data Analysis to Improve the Accuracy of Differential Diagnosis of Retinal Diseases
Integration of Transformer-Based Architecture and Large Language Models for Optical Coherence Tomography Data Analysis to Improve the Accuracy of Differential Diagnosis of Retinal Diseases

Konshina O.V., Pershin A.D., Kulyabin M.K., Borisov V.I.

Key words: OCT biomarker classification; multiclass OCT classification; LLM in ophthalmology; deep learning; retina; ViT; DeepSeek.

2026, Volume 18, Issue 3, page 5
DOI: https://doi.org/10.17691/stm2026.18.3.01
Full text of the article: HTML PDF
285
127
  • Abstract
  • References
  • How to cite

The aim of the present study was to improve the differential diagnosis accuracy of retinal diseases combining a transformer model to classify the biomarkers on OCT images and the large language model DeepSeek-V3.

Materials and Methods. Two datasets were collected and annotated: a training set (3288 central retinal OCT images annotated for 8 biomarkers) and a validation set (50 clinical cases from octcases.com). For biomarker classification, we compared ResNet, DenseNet, EfficientNet, and Vision Transformer (ViT-Tiny-Patch16-224) architectures. The ViT-Tiny model demonstrated the highest performance, its F1 macro — 0.84±0.03. An integration algorithm was developed to combine predicted biomarker labels with clinical history data via the DeepSeek-V3 LLM API.

Results. Combining OCT biomarkers and the medical history significantly improved the diagnostic accuracy: Top-1 Accuracy — 78%, Top-3 Accuracy — 94%, MRR — 84%, representing 10–44% improvement over using either data type alone.

Conclusion. The suggested approach enabled to automate the detection of biomarkers on OCT images and enhance the differential diagnosis accuracy of eye diseases, reducing the image interpretation time and supporting clinical decision-making.

  1. Sitnova A.V., Valitov E.R., Svetozarskiy S.N. Application of deep learning algorithms based on the multilayer Y0L0v8 neural network to identify fungal keratitis. Sovremennye tehnologii v medicine 2024; 16(4): 5–13, https://doi.org/10.17691/stm2024.16.4.01.

  2. Gliznitsa P.V., Takhchidi Kh.P., Svetozarskiy S.N., Bursov A.I., Shusterzon K.A. Machine learning in the diagnosis and treatment of ophthalmic diseases. Head and Neck 2022; 10(1): 83–90.

  3. Takhchidi K., Gliznitsa P., Svetozarskiy S., Bursov A., Shusterzon K. Labelling of data on fundus color pictures used to train a deep learning model enhances its macular pathology recognition capabilities. Bulletin of Russian State Medical University 2021; 2021(4), https://doi.org/10.24075/brsmu.2021.040.

  4. Bahir D., Zur O., Attal L., Nujeidat Z., Knaanie A., Pikkel J., Mimouni M., Plopsky G. Gemini AI vs. ChatGPT: a comprehensive examination alongside ophthalmology residents in medical knowledge. Graefes Arch Clin Exp Ophthalmol 2025; 263(2): 527–536, https://doi.org/10.1007/s00417-024-06625-4.

  5. Mullappilly S.S., Kurpath M.I., Pieri S., Alseiari S.Y., Cholakkal S., Aldahmani K.M., Khan F.S., Anwer R.M., Khan S., Baldwin T., Cholakkal H. BiMediX2: Bio-Medical EXpert LMM for diverse medical modalities. In: Findings of the Association for Computational Linguistics: EMNLP 2025. 2025; p. 14051–14071, https://doi.org/10.18653/v1/2025.findings-emnlp.756.

  6. Gao J., Li C., Liu H., Naumann T., Poon H., Usuyama N., Wong C., Yang J., Zhang S. LLaVA-Med: training a large language-and-vision assistant for biomedicine in one day. In: Advances in Neural Information Processing Systems 36. 2023; p. 28541–28564, https://doi.org/10.52202/075280-1240.

  7. Zhou J., He X., Sun L., Xu J., Chen X., Chu Y., Zhou L., Liao X., Zhang B., Gao X. SkinGPT-4: an interactive dermatology diagnostic system with visual large language model. medRxiv 2023. URL: https://www.medrxiv.org/content/10.1101/2023.06.10.23291127v1.

  8. Avetisov S.E., Kats M.V. Using optical coherent tomography in diagnosis of retinal diseases (review of literature). Universum: meditsina i farmakologiya 2017; 4(38). URL: http://7universum.com/ru/med/archive/item/4561.

  9. Pandya B.U., Grinton M., Mandelcorn E.D., Felfeli T. Retinal optical coherence tomography imaging biomarkers: a review of the literature. Retina 2024; 44(3): 369–380, https://doi.org/10.1097/IAE.0000000000003974.

  10. Yang M., Du J., Lv R. CRAT: advanced transformer-based deep learning algorithms in OCT image classification. Biomedical Signal Processing and Control 2025; 104: 107544, https://doi.org/10.1016/j.bspc.2025.107544.

  11. Wang X., Tang F., Chen H., Luo L., Tang Z., Ran A.R., Cheung C.Y., Heng P.A. UD-MIL: uncertainty-driven deep multiple instance learning for OCT image classification. IEEE J Biomed Health Inform 2020; 24(12): 3431–3442, https://doi.org/10.1109/JBHI.2020.2983730.

  12. Üzen H., Fırat H., Alperen Özçelik S.T., Yusufoğlu E., Çiçek İ.B., Şengür A. Central serous retinopathy classification with deep learning-based multilevel feature extraction from optical coherence tomography images. Optics & Laser Technology 2025; 184: 112519, https://doi.org/10.1016/j.optlastec.2025.112519.

  13. Khudhur A.M. Optical coherence tomography retinal classification using deep neural network. J Opt 2025, https://doi.org/10.1007/s12596-025-02561-6.

  14. Abbas Y., Hadi H.J., Aziz K., Ahmed N., Akhtar M.U., Alshara M.A., Chakrabarti P. Reinforcement-based leveraging transfer learning for multiclass optical coherence tomography images classification. Sci Rep 2025; 15(1): 6193, https://doi.org/10.1038/s41598-025-89831-2.

  15. Diao S., Su J., Yang C., Zhu W., Xiang D., Chen X., Peng Q., Shi F. Classification and segmentation of OCT images for age-related macular degeneration based on dual guidance networks. Biomedical Signal Processing and Control 2023; 84: 104810, https://doi.org/10.1016/j.bspc.2023.104810.

  16. Hamid L., Elnokrashy A., Abdelhay E.H., Abdelsalam M.M. A deep learning LSTM-based approach for AMD classification using OCT images. Neural Computing and Applications 2024; 36(31): 19531–19547, https://doi.org/10.1007/s00521-024-10149-7.

  17. Katalevskaya E.A., Sizov A.Yu., Gilemzianova L.I. Artificial intelligence algorithm for segmentation of pathological structures on optical coherence tomography scans. Russian Journal of Telemedicine and E-Health 2022; 8(3): 21–27, https://doi.org/10.29188/2712-9217-2022-8-3-21-27.

  18. Salehi A., Lee G.C., Ren H., Yu S., Singh R., Talcott K., Melo A., Greenlee T., Chen E., Conti T., Manivannan N. Multi-pathology classification of retinal OCT scans. Investigative Ophthalmology & Visual Science 2023; 64: 1086.

  19. Arzamastsev A.A., Fabrikantov O.L., Kulagina E.V., Zenkova N.A. Classification of optical coherence tomography images using deep machine-learning methods. Digital Diagnostics 2024; 5(1): 5–16, https://doi.org/10.17816/DD623801.

  20. Kulyabin M., Zhdanov A., Pershin A., Sokolov G., Nikiforova A., Ronkin M., Borisov V., Maier A. Segment anything in optical coherence tomography: SAM 2 for volumetric segmentation of retinal biomarkers. Bioengineering 2024; 11(9): 940, https://doi.org/10.3390/bioengineering11090940.

  21. Gholami P., Roy P., Parthasarathy M.K., Lakshminarayanan V. OCTID: optical coherence tomography image database. Computers & Electrical Engineering 2020; 81: 106532, https://doi.org/10.1016/j.compeleceng.2019.106532.

  22. Kulyabin M., Zhdanov A., Nikiforova A., Stepichev A., Kuznetsova A., Ronkin M., Borisov V., Bogachev A., Korotkich S., Constable P.A., Maier A. OCTDL: optical coherence tomography dataset for image-based deep learning methods. Sci Data 2024; 11(1): 365, https://doi.org/10.1038/s41597-024-03182-7.

  23. Yamauchi T. Spatial sensitive Grad-CAM++: improved visual explanation for object detectors via weighted combination of gradient map. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. 2024; p. 8164–8168.

Konshina O.V., Pershin A.D., Kulyabin M.K., Borisov V.I. Integration of transformer-based architecture and large language models for optical coherence tomography data analysis to improve the accuracy of differential diagnosis of retinal diseases. Sovremennye tehnologii v medicine 2026; 18(3): 5, https://doi.org/10.17691/stm2026.18.3.01