Performance efficiency in a computerized task with different types of auditory interfaces

Context and relevance. Auditory interfaces are frequently studied as a way of lessening the cognitive load on the operator in conditions when the visual interface is too demanding in terms of time and cognitive resources. However, despite data on the efficiency of auditory interfaces in comparison to visual ones, different types of auditory interface have not been compared against each other, e.g., parametric mapping of data to the qualities of sound vs. spatial modelling of sound location. The current study proposes a novel approach to comparing spatial and or non-spatial interfaces in two conditions — when the information they communicate corresponds to the type of the interface, and when it doesn’t. Objective: to investigate how using spatial and non-spatial interfaces affects the accuracy and reaction time in the task with information that either corresponds to the type of the interface or doesn’t. Hypotheses. The task with the congruent auditory interfaces was expected to be performed more accurately and quicker; a positive effect of characteristics related to processing auditory information was also expected. Methods and materials. The sample consisted of 52 university students and instructors (39 women, mean age — 23.41 ± 8.05 years; 13 men, mean age — 35.08 ± 17.68 years). The study used a novel computerized task about managing the “orbital station”, the questionnaire of multi-modality in perception (Bandurka, 2004) and Plymouth Sensory Image Questionnaire (Andrade et al., 2014; Razvaliaeva, 2024). Results. The interaction between the type of the auditory interface and the type of the task contributed to the accuracy of performance (it was lower in incongruent conditions). Reaction times were also slower in incongruent blocks. Sequence effects were observed (the accuracy improved as the experiment continued), and different block sequences had an effect as well: the performance in non-spatial tasks was poorer if they were the first in the sequence. The effects of specific sounds varied for different tasks: spatial sounds were distinguished better in the spatial task. Vividness of sound imagery contributed to accuracy only for non-spatial auditory interface. Conclusions. The characteristics of the auditory interface and the correspondence between the auditory scale and the type of information relevant to the task contribute to the efficiency of the performance in the task.

1. Бандурка, Т.Н. (2005). Полимодальность восприятия в обучении. Как раздвинуть границы познания. Иркутск: Оттиск.
   Bandurka, T.N. (2005). Multimodality of perception in education. How to broaden the horizons of learning. Irkutsk: Ottisk. (In Russ.).
2. Носуленко, В.Н. (2020). Звук в интерфейсах взаимодействия человека и техники. В: В.И. Панов (Ред.), Экопсихологические исследования — 6: экология детства и психология устойчивого развития (с. 155—159). М.: ФГБНУ «Психологический институт РАО»; Курск: Университетская книга, 2020. https://doi.org/10.24411/9999-044A-2020-00035
   Nosulenko, V.N. (2020). Sound in human-machine interfaces. In: V.I. Panov (Ed.), Eco-psychological studies — 6: ecology of childhood and psychology of stable development (pp. 155—159). Moscow: FGBNU “Psikhologicheskii institut RAO”; Kursk: Universitetskaya kniga, 2020. https://doi.org/10.24411/9999-044A-2020-00035
3. Разваляева, А.Ю. (2024а). Адаптация Плимутского опросника сенсорных образов на российской выборке. Экспериментальная психология, 17(3), 144—157. https://doi.org/10.17759/exppsy.2024170310
   Razvaliaeva, A.Yu. (2024a). Validating the Plymouth Sensory Imagery Questionnaire in the Russian Sample. Experimental Psychology (Russia), 17(3), 144—157. (In Russ.). https://doi.org/10.17759/exppsy.2024170310
4. Разваляева, А.Ю. (2024б). Сонификация для управления пространственным положением объекта: систематический обзор. Познание и переживание, 5(2), 86—109. https://doi.org/10.51217/cogexp_2024_05_02_04
   Razvaliaeva, A.Yu. (2024b). Systematic review of sonification for controlling an object’s position in space. Cognition and Experience, 5(2), 86—109. (In Russ.). https://doi.org/10.51217/cogexp_2024_05_02_04
5. Разваляева, А.Ю. (2025). Субъективные оценки, роль музыкального обучения и воспринимаемое качество слуховых интерфейсов в экспериментальной задаче. Экспериментальная психология, 18(4), 122—135. https://doi.org/10.17759/exppsy.2025180407
   Razvaliaeva, A.Yu. (2025). Subjective ratings, musical training and the perceived quality of auditory interfaces in an experimental task. Experimental Psychology (Russia), 18(4), 122—135. (In Russ.). https://doi.org/10.17759/exppsy.2025180407
6. Разваляева, А.Ю., Носуленко, В.Н. (2023). Пространственная локализация цифрового звука в научном эксперименте и практике. Экспериментальная психология, 16(2), 20—35. https://doi.org/10.17759/exppsy.2023160202
   Razvaliaeva, A.Yu., Nosulenko, V.N. (2023). Spatial Localization of Digital Sound in Scientific Experiment and Practice. Experimental Psychology (Russia), 16(2), 20—35. (In Russ.). https://doi.org/10.17759/exppsy.2023160202
7. Andrade, J., May, J., Deeprose, C., Baugh, S.J., Ganis, G. (2014). Assessing vividness of mental imagery: the Plymouth Sensory Imagery Questionnaire. British Journal of Psychology, 105(4), 547—563. https://doi.org/10.1111/bjop.12050
8. Bălan, O., Moldoveanu, A., Moldoveanu, F., Morar, A., Ivaşcu, S. (2017). Perceptual feedback training for improving spatial acuity and resolving front-back confusion errors in virtual auditory environments. In: 40th International Conference on Telecommunications and Signal Processing (TSP) (Barcelona, Spain, 5—7 July 2017) (pp. 334—337). Brno: Brno University of Technology. https://doi.org/10.1109/TSP.2017.8075999
9. Baldwin, C.L. (2012). Auditory Cognition and Human Performance: Research and Applications. Boca Raton, FL: CRC Press.
10. Bates, D., Maechler, M., Bolker, B., Walker, S. (2015). Fitting Linear Mixed-Effects Models Using lme4. Journal of Statistical Software, 67(1), 1—48. https://doi.org/10.18637/jss.v067.i01
11. Ben-Shachar, M., Lüdecke, D., Makowski, D. (2020). effectsize: Estimation of Effect Size Indices and Standardized Parameters. Journal of Open Source Software, 5(56), 2815. https://doi.org/10.21105/joss.02815
12. Black, D., Hettig, J., Luz, M., Hansen, C., Kikinis, R., Hahn, H. (2017). Auditory feedback to support image-guided medical needle placement. International Journal of Computer Assisted Radiology and Surgery, 12(9), 1655—1663. https://doi.org/10.1007/s11548-017-1537-1
13. Couture-Beil, A. (2024). rjson: JSON for R. R package version 0.2.23. URL: https://CRAN.R-project.org/package=rjson (viewed: 01.02.2025).
14. de Leeuw, J.R., Gilbert, R.A., Luchterhandt, B. (2023). jsPsych: Enabling an open-source collaborative ecosystem of behavioral experiments. Journal of Open Source Software, 8(85), 5351. https://doi.org/10.21105/joss.05351
15. Dubus, G., Bresin, R. (2013). A systematic review of mapping strategies for the sonification of physical quantities. PloS One, 8(12), e82491. https://doi.org/10.1371/journal.pone.0082491
16. Gelding, R.W., Thompson, W.F., Johnson, B.W. (2015). The Pitch Imagery Arrow Task: Effects of Musical Training, Vividness, and Mental Control. PLoS One, 10(3), e0121809. https://doi.org/10.1371/journal.pone.0121809
17. Guerra, J., Smith, L., Vicinanza, D., Stubbs, B., Veronese, N., Williams, G. (2020). The use of sonification for physiotherapy in human movement tasks: A scoping review. Science & Sports, 35(3), 119—129. https://doi.org/10.1016/j.scispo.2019.12.004
18. Houtenbos, M., de Winter, J.C., Hale, A.R., Wieringa, P.A., Hagenzieker, M.P. (2017). Concurrent audio-visual feedback for supporting drivers at intersections: A study using two linked driving simulators. Applied Ergonomics, 60, 30—42.
19. Hubbard, T.L. (2010). Auditory imagery: Empirical findings. Psychological Bulletin, 136(2), 302—329. https://doi.org/10.1037/a0018436
20. Matinfar, S., Salehi, M., Suter, D., Seibold, M., Dehghani, S., Navab, N., Wanivenhaus F., Fürnstahl P., Farshad M., Navab, N. (2023). Sonification as a reliable alternative to conventional visual surgical navigation. Scientific Reports, 13(1), 5930.
21. Parseihian, G., Gondre, C., Aramaki, M., Ystad, S., Kronland-Martinet, R. (2016). Comparison and evaluation of sonification strategies for guidance tasks. IEEE Transactions on Multimedia, 18(4), 674—686.
22. Peirce, J.W., Hirst, R.J. MacAskill, M.R. (2022). Building Experiments in PsychoPy. 2nd Ed. London: Sage.
23. Peres, S.C., Best, V., Brock, D., Frauenberger, C., Hermann, T., Neuhoff, J.G., Nickerson, L.V., Shinn-Cunningham, B., Stockman, T. (2008). Auditory interfaces. In: P. Kortum (Ed.), HCI beyond the GUI: design for haptic, speech, olfactory, and other nontraditional interfaces (pp. 147—196). Amsterdam: Elsevier/Morgan Kaufmann. https://doi.org/10.1016/B978-0-12-374017-5.00005-5
24. R Core Team (2024). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. URL: https://www.R-project.org/ (viewed: 01.02.2025).
25. Revelle, W. (2024). psych: Procedures for Psychological, Psychometric, and Personality Research. Northwestern University, Evanston, Illinois. URL: https://CRAN.R-project.org/package=psych (viewed: 01.02.2025).
26. Walker, B.N. (2023). The past, present, and promise of sonification. Arbor, 199(810), a728. https://doi.org/10.3989/arbor.2023.810008