Evaluating Chatbot usability: Adaptation of the BUS-11 Questionnaire for a Russian-speaking sample

Context and Relevance. Chatbots powered by generative artificial intelligence (AI) are becoming a crucial tool in education, healthcare, and service industries. A key factor in their effectiveness is usability; however, specialized assessment tools for Russian-speaking users have been lacking until now. Objective: to adapt and validate a Russian-language version of the short questionnaire BUS for evaluating the usability of generative AI-based chatbots. Hypothesis. The Russian-language version of the tool demonstrates reliability and validity, and replicates the key factors of the original methodology: Perceived Dialogue and Information Quality, Perceived Bot Function Quality, and Perceived Function Accessibility. Methods and Materials. The study involved 207 participants (aged 18—60); test-retest reliability was assessed on a sub-sample of 98 respondents with an interval of 1—2 months. Procedures included linguistic adaptation, exploratory and confirmatory factor analysis, Cronbach's alpha coefficient, and correlation analysis. Results. The questionnaire demonstrated high internal consistency (α = .81) and test-retest reliability (r = .83). The three-factor structure was confirmed: Dialogue Quality, Functionality, and Function Accessibility. Convergent validity was established through high correlations with UMUX-LITE (r = .845) and AttrakDiff scales; divergent validity was confirmed by a low correlation with the Satisfaction with Life Scale (r = .12). Conclusions. The adapted version of the short BUS is a reliable tool for assessing the usability of chatbots on a Russian-speaking sample and can be used in both scientific and applied research.

1. Власян, Г.Р., Кожухова, И.В. (2019). Формальные и неформальные приглашения в русском языке: контекст и стратегии вежливости. Russian Journal of Linguistics, 23(4), 994—1013. https://doi.org/10.22363/2687-0088-2019-23-4-994-1013
   Vlasyan, G.R., Kozhukhova, I.V. (2019). Formal and Informal Russian Invitation: Context and Politeness Strategies. Russian Journal of Linguistics, 23(4), 994—1013. (In Russ.). https://doi.org/10.22363/2687-0088-2019-23-4-994-1013
2. Косова, Е., Горбунова, Е. (2023). Как подсказки веб-интерфейса индуцируют онлайн-доверие: обзор литературы. Психологические исследования, 16(87), 1—41. https://doi.org/10.54359/ps.v16i87.1353
   Kosova, E., Gorbunova, E. (2023). Exploring web-interface clues inducing e-trust: literature review. Psychological Studies, 16(87), 1—41. (In Russ.). https://doi.org/10.54359/ps.v16i87.1353
3. Лихачева, А.Б. (2017). Вербальные особенности городского пространства как часть национальной коммуникативной культуры. Slavistica Vilnensis, 61, 161—172.
   Likhacheva, A.B. (2017). Verbal Features of Urban Space as a Part of National Communicative Culture. Slavistica Vilnensis, 61, 161—172. (In Russ.).
4. Осин, Е.Н., Леонтьев, Д.А. (2008). Апробация русскоязычных версий двух шкал экспресс-оценки субъективного благополучия. Сборник Материалов III Всероссийского социологического конгресса (c. 117—142). М.: Институт социологии РАН, Российское общество социологов.
   Osin, E.N., Leontyev, D.A. (2008). Approbation of Russian versions of two subjunctive well-being scales. Proceedings of the Third all Russia Congress of Sociology (pp. 117—142). Moscow: Institute of Sociology of the RAS, Russian Society of Sociologists. (In Russ.).
5. Роговец, А.Ю., Мазуркевич, А., Суворова, И.Ю., Винник, М., Приходько, А.А. (2023). Чат-бот как способ поддержки жертв школьной травли. Современная зарубежная психология, 12(3), 103—114. https://doi.org/10.17759/jmfp.2023120310
   Rogovets, A.Yu., Mazurkevich, A., Suvorova, I.Yu., Vinnik, M., Prikhodko, A.A. (2023). Chat-bot as a way to support victims of school bullying. Journal of Modern Foreign Psychology, 12(3), 103—114. (In Russ.). https://doi.org/10.17759/jmfp.2023120310
6. Abd-Alrazaq, A.A., Alajlani, M., Ali, N., Denecke, K., Bewick, B.M., Househ, M. (2021). Perceptions and opinions of patients about mental health chatbots: scoping review. Journal of Medical Internet Research, 23(1), e17828. https://doi.org/10.2196/17828
7. Al-Abdullatif, A.M., Al-Dokhny, A.A., Drwish, A.M. (2023). Implementing the Bashayer chatbot in Saudi higher education: Measuring the influence on students’ motivation and learning strategies. Frontiers in Psychology, 14, 1129070. https://doi.org/10.3389/fpsyg.2023.1129070
8. Balderas, A., García-Mena, R.F., Huerta, M., Mora, N., Dodero, J.M. (2023). Chatbot for communicating with university students in emergency situation. Heliyon, 9(9), e19517. https://doi.org/10.1016/j.heliyon.2023.e19517
9. Borsci, S., Malizia, A., Schmettow, M., Van Der Velde, F., Tariverdiyeva, G., Balaji, D., Chamberlain, A. (2022). The Chatbot Usability Scale: The design and pilot of a Usability Scale for interaction with AI-based conversational agents. Personal and Ubiquitous Computing, 26(1), 95—119. https://doi.org/10.1007/s00779-021-01582-9
10. Borsci, S., Prati, E., Malizia, A., Schmettow, M., Chamberlain, A., Federici, S. (2023a). Ciao AI: The Italian adaptation and validation of the Chatbot Usability Scale. Personal and Ubiquitous Computing, 27(6), 2161—2170. https://doi.org/10.1007/s00779-023-01731-2
11. Borsci, S., Schmettow, M., Malizia, A., Chamberlain, A., Van Der Velde, F. (2023b). A confirmatory factorial analysis of the Chatbot Usability Scale: A multilanguage validation. Personal and Ubiquitous Computing, 27(2), 317—330. https://doi.org/10.1007/s00779-022-01690-0
12. Brin, D., Sorin, V., Vaid, A., Soroush, A., Glicksberg, B.S., Charney, A.W., Nadkarni, G., Klang, E. (2023). Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessments. Scientific Reports, 13(1), 16492.
13. Brooke, J. (1996). SUS: A “quick and dirty” usability scale. In: P.W. Jordan, B. Thomas, B.A. Weerdmeester, A.L. McClelland (Eds.), Usability Evaluation in Industry (pp. 4—7). London: Taylor and Francis.
14. Chagas, B.A., Pagano, A.S., Prates, R.O., Praes, E.C., Ferreguetti, K., Vaz, H., Reis, Z.S.N., Ribeiro, L.B., Ribeiro, A.L.P., Pedroso, T.M., Beleigoli, A., Oliveira, C.R.A., Marcolino, M.S. (2023). Evaluating User Experience With a Chatbot Designed as a Public Health Response to the COVID-19 Pandemic in Brazil: Mixed Methods Study. JMIR Human Factors, 10, e43135. https://doi.org/10.2196/43135
15. Cheng, X., Yin, L., Lin, C., Shi, Z., Zheng, H., Zhu, L., Liu, X., Chen, K., Dong, R. (2024). Chatbot dialogic reading boosts comprehension for Chinese kindergarteners with higher language skills. Journal of Experimental Child Psychology, 240, 105842. https://doi.org/10.1016/j.jecp.2023.105842
16. Dhiman, N., Jamwal, M. (2023). Tourists’ post-adoption continuance intentions of chatbots: Integrating task-technology fit model and expectation-confirmation theory. Foresight, 25(2), 209—224. https://doi.org/10.1108/FS-10-2021-0207
17. Elyoseph, Z., Hadar-Shoval, D., Asraf, K., Lvovsky, M. (2023). ChatGPT outperforms humans in emotional awareness evaluations. Frontiers in Psychology, 14, 1199058. https://doi.org/10.3389/fpsyg.2023.1199058
18. Finstad, K. (2010). The Usability Metric for User Experience. Interacting with Computers, 22(5), 323—327. https://doi.org/10.1016/j.intcom.2010.04.004
19. Gunnam, G.R., Inupakutika, D., Mundlamuri, R., Kaghyan, S., Akopian, D. (2022). Chatbot integrated with machine learning deployed in the cloud and performance evaluation. Electronic Imaging, 34(3), 205-1-205-5. https://doi.org/10.2352/EI.2022.34.3.MOBMU-205
20. Hassenzahl, M., Monk, A. (2010). The Inference of Perceived Usability From Beauty. Human-Computer Interaction, 25(3), 235—260. https://doi.org/10.1080/07370024.2010.500139
21. Janson, A. (2023). How to leverage anthropomorphism for chatbot service interfaces: The interplay of communication style and personification. Computers in Human Behavior, 149, 107954. https://doi.org/10.1016/j.chb.2023.107954
22. Lewis, J.R., Utesch, B.S., Maher, D.E. (2013). UMUX-LITE: When there’s no time for the SUS. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2099—2102). https://doi.org/10.1145/2470654.2481287
23. Lin, C.-C., Huang, A.Y.Q., Yang, S.J.H. (2023). A Review of AI-Driven Conversational Chatbots Implementation Methodologies and Challenges (1999–2022). Sustainability, 15(5), 4012. https://doi.org/10.3390/su15054012
24. Lopes, E., Jain, G., Carlbring, P., Pareek, S. (2023). Talking Mental Health: A Battle of Wits Between Humans and AI. Journal of Technology in Behavioral Science, 9(4), 628—638. https://doi.org/10.1007/s41347-023-00359-6
25. Mathur, A., Tiwari, S. (2023). The Impact of Artificial Intelligence on Customer Relationship Management in the Indian Banking Industry. In: P. Mittal, R. Bansal (Eds), Social Development and Governance: Innovations in Education, Technology and Management (pp. 107—116). QTanalytics India. https://doi.org/10.48001/978-81-966500-9-4_9
26. Mohamad Suhaili, S., Salim, N., Jambli, M.N. (2021). Service chatbots: A systematic review. Expert Systems with Applications, 184, 115461. https://doi.org/10.1016/j.eswa.2021.115461
27. Nguyen, M., Casper Ferm, L.-E., Quach, S., Pontes, N., Thaichon, P. (2023). Chatbots in frontline services and customer experience: An anthropomorphism perspective. Psychology & Marketing, 40(11), 2201—2225. https://doi.org/10.1002/mar.21882
28. Paliwal, S., Bharti, V., Mishra, A.K. (2020). AI Chatbots: Transforming the Digital World. In: V.E. Balas, R. Kumar, R. Srivastava (Eds.), Recent Trends and Advances in Artificial Intelligence and Internet of Things (Vol. 172, pp. 455—482). Springer International Publishing. https://doi.org/10.1007/978-3-030-32644-9_34
29. Prince, C., Omrani, N., Schiavone, F. (2024). Online privacy literacy and users’ information privacy empowerment: The case of GDPR in Europe. Information Technology & People, 37(8), 1—24. https://doi.org/10.1108/ITP-05-2023-0467
30. Rane, N.L., Mallick, S.K., Kaya, Ö., Rane, J. (2024). Role of machine learning and deep learning in advancing generative artificial intelligence such as ChatGPT. In: Applied Machine Learning and Deep Learning: Architectures and Techniques (pp. 96—111). Deep Science Publishing. https://doi.org/10.70593/978-81-981271-4-3_5
31. Sahata Sitanggang, A., Syafariani, R.F., Sari, F.W., Wartika, W., Hasti, N. (2023). Relation of Chatbot Usage Towards Customer Satisfaction Level in Indonesia. International Journal of Advances in Data and Information Systems, 4(1), 86—96. https://doi.org/10.25008/ijadis.v4i1.1261
32. Schillaci, C.E., de Cosmo, L.M., Piper, L., Nicotra, M., Guido, G. (2024). Anthropomorphic chatbots’ for future healthcare services: Effects of personality, gender, and roles on source credibility, user satisfaction, and intention to use. Technological Forecasting and Social Change, 199(3), 123025. https://doi.org/10.1016/j.techfore.2023.123025
33. Tsai, W.-H.S., Liu, Y., Chuan, C.-H. (2021). How chatbots’ social presence communication enhances consumer engagement: The mediating role of parasocial interaction and dialogue. Journal of Research in Interactive Marketing, 15(3), 460—482. https://doi.org/10.1108/JRIM-12-2019-0200
34. Valério, F.A.M., Guimarães, T.G., Prates, R.O., Candello, H. (2018). Chatbots Explain Themselves: Designers’ Strategies for Conveying Chatbot Features to Users. Journal on Interactive Systems, 9(3). https://doi.org/10.5753/jis.2018.710
35. Van De Vijver, F., Hambleton, R.K. (1996). Translating Tests. European Psychologist, 1(2), 89—99. https://doi.org/10.1027/1016-9040.1.2.89
36. Weeks, R., Sangha, P., Cooper, L., Sedoc, J., White, S., Gretz, S., Toledo, A., Lahav, D., Hartner, A.-M., Martin, N.M., Lee, J.H., Slonim, N., Bar-Zeev, N. (2023). Usability and Credibility of a COVID-19 Vaccine Chatbot for Young Adults and Health Workers in the United States: Formative Mixed Methods Study. JMIR Human Factors, 10, e40533. https://doi.org/10.2196/40533
37. Xue, J., Zhang, B., Zhao, Y., Zhang, Q., Zheng, C., Jiang, J., Li, H., Liu, N., Li, Z., Fu, W., Peng, Y., Logan, J., Zhang, J., Xiang, X. (2023). Evaluation of the Current State of Chatbots for Digital Health: Scoping Review. Journal of Medical Internet Research, 25, e47217. https://doi.org/10.2196/47217