Application of Kohonen self-organizing maps for identification of acoustic stress predictors in complex technical system operators

This article presents the development and validation of a method for identifying acoustic predictors of stress in operators of complex technical systems, based on Kohonen Self-Organizing Maps (SOM). The relevance of this research is driven by the need to enhance the operational safety of complex technical systems — particularly aircraft — through objective, non-contact monitoring of personnel's psychoemotional state. The study was conducted as part of a research project aimed at designing an advanced civil aircraft cockpit equipped with an intelligent crew support system. A key requirement for this cockpit is the provision of continuous, non-invasive monitoring of pilots' functional state to enable timely detection of critical changes associated with stress and fatigue. The proposed approach involves extracting a nine-dimensional acoustic feature vector from the operator's speech signal — comprising fundamental frequency (F0), spectral centroid, spectral bandwidth, spectral roll-off, and five peak frequencies of the power spectrum — followed by classification of the current state as either "normal" or "stress" using a trained Kohonen map. To verify the proposed approach, a software model called "AudioStressPredictor" was developed, implementing the full processing pipeline: from creating a personalized operator profile and training the model to real-time stress level monitoring. Experimental testing was carried out, and the results confirm the viability of the proposed approach and its potential for application in aviation training centers, as well as as one of the key components of onboard decision support systems for advanced aircraft.

1. Грешников И.И. Моделирование элементов информационно-управляющего поля кабины и действий экипажа воздушного судна на универсальном стенде прототипирования: дис. ... канд. техн. наук. М.: МАИ, 2022.
   Greshnikov, I.I. (2022). Modelirovanie elementov informatsionno-upravlyayushchego polya kabiny i deistvii ekipazha vozdushnogo sudna na universal'nom stende prototipirovaniya [Modeling elements of the cockpit information and control field and aircraft crew actions on a universal prototyping bench] (Doctoral dissertation). Moscow Aviation Institute, Moscow, Russia.
2. Куравский Л.С., Баранов С.Н., Малых С.Б. Нейронные сети в задачах прогнозирования, диагностики и анализа данных: учеб. пособие. М.: РУСАВИА, 2003. 100 с.
   Kuravsky, L.S., Baranov, S.N., & Malykh, S.B. (2003). Neironnye seti v zadachakh prognozirovaniya, diagnostiki i analiza dannykh [Neural networks in forecasting, diagnostics and data analysis tasks]. Moscow: RUSAVIA.
3. Куравский Л.С., Козырев А.Д., Грешников И.И. Математическая модель сопутствующей деятельности пилота и её применение для объективной оценки его состояния и профессиональной подготовки // Экспериментальная психология. 2024. Т. 17. № 1. С. 161–180. DOI: 10.17759/exppsy.2024170111.
   Kuravsky, L.S., Kozyrev, A.D., & Greshnikov, I.I. (2024). Matematicheskaya model' soputstvuyushchei deyatel'nosti pilota i ee primenenie dlya ob"ektivnoi otsenki ego sostoyaniya i professional'noi podgotovki [Mathematical model of the pilot's concurrent activity and its application for objective assessment of their state and professional training]. Eksperimental'naya psikhologiya [Experimental Psychology (Russia)], 17(1), 161–180. https://doi.org/10.17759/exppsy.2024170111
4. Куравский Л.С., Малых С.Б., Кравчук Т.Е., Кузнецова И.В., Семаго Н.Я. Методы классификации в психодиагностических исследованиях // Вопросы психологии. 2006. № 1. С. 157–168.
   Kuravsky, L.S., Malykh, S.B., Kravchuk, T.E., Kuznetsova, I.V., & Semago, N.Ya. (2006). Metody klassifikatsii v psikhodiagnosticheskikh issledovaniyakh [Classification methods in psychodiagnostic research]. Voprosy psikhologii [Questions of Psychology], (1), 157–168.
5. Куравский Л.С., Мармалюк П.А., Юрьев Г.А., Баранов С.Н., Полещук Г.Н., Смирнов А.А., Шишов А.Н. Диагностика лётного состава по результатам работы на авиационных тренажёрах // Нейрокомпьютеры: разработка, применение. 2014. № 10. С. 14–23.
   Kuravsky, L.S., Marmalyuk, P.A., Yuryev, G.A., Baranov, S.N., Poleshchuk, G.N., Smirnov, A.A., & Shishov, A.N. (2014). Diagnostika letnogo sostava po rezul'tatam raboty na aviatsionnykh trenazherakh [Diagnostics of flight personnel based on the results of work on aviation simulators]. Neirokomp'yutery: razrabotka, primenenie [Neurocomputers: Development and Application], (10), 14–23.
6. Куравский Л.С., Юрьев Г.А., Златомрежев В.И., Грешников И.И., Поляков Б.Ю. Оценка действий экипажа самолёта по данным видеоокулографии // Экспериментальная психология. 2021. Т. 14. № 1. С. 204–222. DOI: 10.17759/exppsy.2021140110.
   Kuravsky, L.S., Yuryev, G.A., Zlatomrezhev, V.I., Greshnikov, I.I., & Polyakov, B.Yu. (2021). Otsenka deistvii ekipazha samoleta po dannym videookulografii [Assessment of aircraft crew actions based on video-oculography data]. Eksperimental'naya psikhologiya [Experimental Psychology (Russia)], 14(1), 204–222. https://doi.org/10.17759/exppsy.2021140110
7. Куравский, Л.С., Грешников, И.И., Глухова, Э.Д., Юрьев, Г.А., Юрьева, Н.Е., Поляков, Б.Ю., Соколов, А.В., Махортов, И.А., Кислицын, Е.Д. (2025). Комплексная оценка информационно-управляющего поля кабины экипажа гражданского воздушного судна: исследуемые показатели и методы их анализа. Экспериментальная психология, 18(2), 236–260. https://doi.org/10.17759/exppsy.2025180214
   Kuravsky, L.S., Greshnikov, I.I., Glukhova, E.D., Yuryev, G.A., Yuryeva, N.E., Polyakov, B.Y., Sokolov, A.V., Makhortov, I.A., Kislitsyn, E.D. (2025). Comprehensive assessment of the information and control field of the cockpit of a civil aircraft: the studied indicators and methods of their analysis. Experimental Psychology (Russia), 18(2), 236–260. (In Russ.). https://doi.org/10.17759/exppsy.2025180214
8. Куравский, Л.С., Грешников, И.И., Левонович, Н.И., Юрьева, Н.Е., Глухова, Э.Д., Махортов, И.А., Кислицын, Е.Д., Соколов, А.В., Захарьчева, А.А. (2026). Количественная оценка компонентов интегральной когнитивной нагрузки пилотов воздушных судов. Экспериментальная психология, 19(1), 167–185. https://doi.org/10.17759/exppsy.2026190111
   Kuravsky, L.S., Greshnikov, I.I., Levonovich, N.I., Yuryeva, N.E., Glukhova, E.D., Makhortov, I.A., Kislitsyn, E.D., Sokolov, A.V., Zakharcheva, A.A. (2026). Quantitative assessment of the components of the integral cognitive load of aircraft pilots. Experimental Psychology (Russia), 19(1), 167–185. (In Russ.). https://doi.org/10.17759/exppsy.2026190111
9. de Cheveigné A., Kawahara H. YIN, a fundamental frequency estimator for speech and music // The Journal of the Acoustical Society of America. 2002. Vol. 111. No. 4. P. 1917–1930. DOI: 10.1121/1.1458024.
10. Feigh K.M., Dorneich M.C., Hayes C.C. Toward a characterization of adaptive systems: A framework for researchers and system designers // Human Factors. 2012. Vol. 54. No. 6. P. 1008–1024. DOI: 10.1177/0018720812443983.
11. Frampton M., Sripada S., Bion R.A.H., Peters S. Detection of time-pressure induced stress in speech via acoustic indicators // Proceedings of the SIGDIAL 2010 Conference. Tokyo, Japan, 2010. P. 253–256.
12. Giddens C.L., Barron K.W., Byrd-Craven J., Clark K.F., Winter A.S. Vocal indices of stress: A review // Journal of Voice. 2013. Vol. 27. No. 3. P. 390.e21–390.e29. DOI: 10.1016/j.jvoice.2012.12.010.
13. ICAO. Doc 9859: Safety Management Manual (SMM). 4th ed. Montreal: International Civil Aviation Organization, 2018.
14. Kohonen T. Self-organized formation of topologically correct feature maps // Biological Cybernetics. 1982. Vol. 43. No. 1. P. 59–69. DOI: 10.1007/BF00337288.
15. Kohonen T. Self-Organizing Maps. 3rd ed. Berlin: Springer, 2001. 501 p. DOI: 10.1007/978-3-642-56927-2.
16. Kuravsky L.S., Baranov S.N. Application of self-organizing feature maps for diagnostics of vibroacoustic systems // Proc. Condition Monitoring 2001. Oxford, UK, 2001. P. 79–89.
17. Kuravsky L.S., Yuryev G.A. A novel approach for recognizing abnormal activities of operators of complex technical systems // International Journal of Advanced Research in Engineering and Technology. 2020. Vol. 11. No. 4. P. 119–136.
18. Kuravsky L.S., Yuryev G.A., Zlatomrezhev V.I., Yuryeva N.E. Assessing the Aircraft Crew Actions with the Aid of a Human Factor Risk Model // Experimental Psychology (Russia). 2020. Vol. 13. No. 2. P. 153–181. DOI: 10.17759/exppsy.2020130211.
19. McFee B., Raffel C., Liang D. et al. librosa: Audio and Music Signal Analysis in Python // Proc. 14th Python in Science Conference. 2015. P. 18–25.
20. Sharma N., Gedeon T. Objective measures, sensors and computational techniques for stress recognition and classification: A survey // Computer Methods and Programs in Biomedicine. 2012. Vol. 108. No. 3. P. 1287–1301. DOI: 10.1016/j.cmpb.2012.07.003.
21. Virtanen P., Gommers R., Oliphant T.E. et al. SciPy 1.0: Fundamental algorithms for scientific computing in Python // Nature Methods. 2020. Vol. 17. P. 261–272. DOI: 10.1038/s41592-019-0686-2.