Assessment of Accuracy of Spatial Object Localization by Means of Mono and Stereo Modes of Visual-to-Auditory Sensory Substitution in People with Visual Impairments (a Pilot Study)

The aim of the study is to assess the accuracy of spatial object localization in mono and stereo modes of visual-to-auditory sensory substitution by means of the developed system tested on persons with normal or corrected-to-normal vision.

Materials and Methods. A prototype of a visual-to-auditory sensory substitution device based on a video camera with two lenses was prepared. Software to convert the signal from a video camera into an audio signal in mono and stereo modes was developed.

To assess the developed system, an experimental study with 30 blindfolded sighted participants was conducted. 15 persons were tested in mono mode, 15 — in stereo mode. All persons were trained to use the visual-to-auditory sensory substitution system. During the experiment, participants were to locate a white plastic cube with dimensions of 4×4×4 cm³ on a working surface. The researcher placed the cube in one of 20 positions on the working surface in a pseudo-random order.

Results. To assess the accuracy of the cube localization, deviations along the X- and Y-axes and absolute deviations were calculated. The general dynamics of localization accuracy was positive both in mono and stereo modes. Absolute deviation and X-axis deviation were significantly higher in stereo mode; there was no significant difference in Y-axis deviation between modes. On average, participants tended to underestimate the distance to the cube when it was on the left, right, or far side of the working surface, and overestimate the distance to the cube when it was on the near side of the working surface.

Conclusion. Tests demonstrated that the accuracy of object localization in stereo mode can be improved by increasing the time for training the participants and by showing them more presentations. The results of the study can be used to develop assistive techniques for people with visual impairments, to manufacture medical equipment, and create brain-computer interfaces.

1. World Health Organization. Blindness and vision impairment. URL: https://www.who.int/news-room/fact-sheets/detail/blindness-and-visual-impairment.
2. Federal State Statistics Service (Rosstat). Incidence in the population by main classes of diseases in 2000–2022. URL: https://rosstat.gov.ru/storage/mediabank/zdr2-1.xls.
3. Erickson-Davis C., Korzybska H. What do blind people “see” with retinal prostheses? Observations and qualitative reports of epiretinal implant users. PLoS One 2021; 16(2): e0229189, https://doi.org/10.1371/journal.pone.0229189.
4. Koponen L.M., Peterchev A.V. Transcranial magnetic stimulation: principles and applications. In: He B. (editor). Neural Engineering. Springer, Cham; 2020, https://doi.org/10.1007/978-3-030-43395-6_7.
5. Nguyen T., Gao J., Wang P., Nagesetti A., Andrews P., Masood S., Vriesman Z., Liang P., Khizroev S., Jin X. In vivo wireless brain stimulation via non-invasive and targeted delivery of magnetoelectric nanoparticles. Neurotherapeutics 2021; 18(3): 2091–2106, https://doi.org/10.1007/s13311-021-01071-0.
6. Farnum A., Pelled G. New vision for visual prostheses. Front Neurosci 2020; 14: 36, https://doi.org/10.3389/fnins.2020.00036.
7. Bach-y-Rita P., Kercel S. W. Sensory substitution and the human-machine interface. Trends Cogn Sci 2003; 7(12): 541–546, https://doi.org/10.1016/j.tics.2003.10.013.
8. Brooks J., Kristjánsson Á., Unnthorsson R. Sensory substitution: visual information via haptics. In: Holmes N.P. (editor). Somatosensory Research Methods. Neuromethods. Vol 196. Humana, New York, NY; 2023, https://doi.org/10.1007/978-1-0716-3068-6_14.
9. Meijer P.B. An experimental system for auditory image representations. IEEE Trans Biomed Eng 1992; 39(2): 112–121, https://doi.org/10.1109/10.121642.
10. Lebedeva S.A. Zvukovoe zrenie kak sovremennyy sposob reabilitatsii nezryachikh lyudey. V kn.: Cognitive Neuroscience — 2021 [vOICe sound vision as a modern method for rehabilitation of blind people. In: Cognitive Neuroscience — 2021]. Ekaterinburg: Izdatel’stvo Ural’skogo universiteta; 2022; p. 176–180. URL: https://elar.urfu.ru/handle/10995/109097.
11. Capelle C., Trullemans C., Arno P., Veraart C. A real-time experimental prototype for enhancement of vision rehabilitation using auditory substitution. IEEE Trans Biomed Eng 1998; 45(10): 1279–1293, https://doi.org/10.1109/10.720206.
12. Abboud S., Hanassy S., Levy-Tzedek S., Maidenbaum S., Amedi A. EyeMusic: introducing a “visual” colorful experience for the blind using auditory sensory substitution. Restor Neurol Neurosci 2014; 32(2): 247–257, https://doi.org/10.3233/RNN-130338.
13. Bologna G., Deville B., Pun T., Vinckenbosch M. Transforming 3D coloured pixels into musical instrument notes for vision substitution applications. EURASIP Journal on Image and Video Processing 2007; 2007: 1–14, https://doi.org/10.1155/2007/76204.
14. Kim M., Park Y., Moon K., Jeong C.Y. Deep learning-based optimization of visual–auditory sensory substitution. IEEE Access 2023; 11: 14169–14180, https://doi.org/10.1109/ACCESS.2023.3243641.
15. Neugebauer A., Rifai K., Getzlaff M., Wahl S. Navigation aid for blind persons by visual-to-auditory sensory substitution: a pilot study. PLoS One 2020; 15(8): e0237344, https://doi.org/10.1371/journal.pone.0237344.
16. Ward J., Meijer P. Visual experiences in the blind induced by an auditory sensory substitution device. Conscious Cogn 2010; 19(1): 492–500, https://doi.org/10.1016/j.concog.2009.10.006.
17. Łukowska M., Kałwak W., Osiński D., Janik J., Wierzchoń M. How to teach a blind person to hear colours? Multi-method training for a colour-to-sound sensory substitution device — design and evaluation. International Journal of Human-Computer Studies 2022; 168: 102925, https://doi.org/10.1016/j.ijhcs.2022.102925.
18. Richardson M., Thar J., Alvarez J., Borchers J., Ward J., Hamilton-Fletcher G. How much spatial information is lost in the sensory substitution process? Comparing visual, tactile, and auditory approaches. Perception 2019; 48(11): 1079–1103, https://doi.org/10.1177/0301006619873194.
19. Butorova A.S., Sergeev A.P. Experience of using non-invasive sensory substitution in localization of objects in space. Traektoriya issledovaniy — chelovek, priroda, tekhnologii 2023; 2(6): 16–27, https://doi.org/10.56564/27825264_2023_2_16.
20. Caraiman S., Zvoristeanu O., Burlacu A., Herghelegiu P. Stereo vision based sensory substitution for the visually impaired. Sensors (Basel) 2019; 19(12): 2771, https://doi.org/10.3390/s19122771.
21. International Classification of Diseases, 10th revision (ICD-10). URL: https://mkb-10.com/.
22. Seeing with Sound. A tutorial on using The vOICe program. URL: https://www.seeingwithsound.com/manual_ru/The_vOICe_Training_Manual_ru.htm.
23. Froese T., Ortiz-Garin G.U. where is the action in perception? An exploratory study with a haptic sensory substitution device. Front Psychol 2020; 11: 809, https://doi.org/10.3389/fpsyg.2020.00809.
24. Yun O.T., Deng Q. Concerns regarding sensory substitution: overload and conflict. Proceedings of the 2022 International Conference on Social Sciences and Humanities and Arts (SSHA 2022) 2022, https://doi.org/10.2991/assehr.k.220401.185.
25. Kiss D.S., Toth I., Jocsak G., Barany Z., Bartha T., Frenyo L.V., Horvath T.L., Zsarnovszky A. Functional aspects of hypothalamic asymmetry. Brain Sci 2020; 10(6): 389, https://doi.org/10.3390/brainsci10060389.
26. Hart S.G., Staveland L.E., Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. Advances in Psychology 1988; 52: 139–183, https://doi.org/10.1016/S0166-4115(08)62386-9.

Butorova А.S., Koryukin Е.А., Khomenko N.М., Sergeev А.P. Assessment of Accuracy of Spatial Object Localization by Means of Mono and Stereo Modes of Visual-to-Auditory Sensory Substitution in People with Visual Impairments (a Pilot Study). Sovremennye tehnologii v medicine 2024; 16(4): 29, https://doi.org/10.17691/stm2024.16.4.03