Skip to Main Content

Experimental Psychology (Russia)
2025. Vol. 18, no. 3, 194–210
doi:10.17759/exppsy.2025180313
ISSN: 2072-7593 / 2311-7036 (online)

Evaluating Chatbot usability: Adaptation of the BUS-11 Questionnaire for a Russian-speaking sample

 
Audio is AI-generated
52

,

PDF (RUS)
Online translation in english
GS
Information
in Google Scholar
Metrics

Abstract

Context and Relevance. Chatbots powered by generative artificial intelligence (AI) are becoming a crucial tool in education, healthcare, and service industries. A key factor in their effectiveness is usability; however, specialized assessment tools for Russian-speaking users have been lacking until now. Objective: to adapt and validate a Russian-language version of the short questionnaire BUS for evaluating the usability of generative AI-based chatbots. Hypothesis. The Russian-language version of the tool demonstrates reliability and validity, and replicates the key factors of the original methodology: Perceived Dialogue and Information Quality, Perceived Bot Function Quality, and Perceived Function Accessibility. Methods and Materials. The study involved 207 participants (aged 18—60); test-retest reliability was assessed on a sub-sample of 98 respondents with an interval of 1—2 months. Procedures included linguistic adaptation, exploratory and confirmatory factor analysis, Cronbach's alpha coefficient, and correlation analysis. Results. The questionnaire demonstrated high internal consistency (α = .81) and test-retest reliability (r = .83). The three-factor structure was confirmed: Dialogue Quality, Functionality, and Function Accessibility. Convergent validity was established through high correlations with UMUX-LITE (r = .845) and AttrakDiff scales; divergent validity was confirmed by a low correlation with the Satisfaction with Life Scale (r = .12). Conclusions. The adapted version of the short BUS is a reliable tool for assessing the usability of chatbots on a Russian-speaking sample and can be used in both scientific and applied research.

General Information

Keywords: questionnaire, method adaptation, chatbots, artificial intelligence, usability

Journal rubric: Psychodiagnostics

Article type: scientific article

DOI: https://doi.org/10.17759/exppsy.2025180313

Funding. The study was funded by Russian Science Foundation (RSF), project number 24-28-00364.

Received 06.12.2024

Revised 17.02.2025

Accepted

Published

For citation: Rafikova, A.S., Voronin, A.N. (2025). Evaluating Chatbot usability: Adaptation of the BUS-11 Questionnaire for a Russian-speaking sample. Experimental Psychology (Russia), 18(3), 194–210. (In Russ.). https://doi.org/10.17759/exppsy.2025180313

© Rafikova A.S., Voronin A.N., 2025

License: CC BY-NC 4.0

References

  1. Власян, Г.Р., Кожухова, И.В. (2019). Формальные и неформальные приглашения в русском языке: контекст и стратегии вежливости. Russian Journal of Linguistics, 23(4), 994—1013. https://doi.org/10.22363/2687-0088-2019-23-4-994-1013
    Vlasyan, G.R., Kozhukhova, I.V. (2019). Formal and Informal Russian Invitation: Context and Politeness Strategies. Russian Journal of Linguistics, 23(4), 994—1013. (In Russ.). https://doi.org/10.22363/2687-0088-2019-23-4-994-1013
  2. Косова, Е., Горбунова, Е. (2023). Как подсказки веб-интерфейса индуцируют онлайн-доверие: обзор литературы. Психологические исследования, 16(87), 1—41. https://doi.org/10.54359/ps.v16i87.1353
    Kosova, E., Gorbunova, E. (2023). Exploring web-interface clues inducing e-trust: literature review. Psychological Studies, 16(87), 1—41. (In Russ.). https://doi.org/10.54359/ps.v16i87.1353
  3. Лихачева, А.Б. (2017). Вербальные особенности городского пространства как часть национальной коммуникативной культуры. Slavistica Vilnensis, 61, 161—172.
    Likhacheva, A.B. (2017). Verbal Features of Urban Space as a Part of National Communicative Culture. Slavistica Vilnensis, 61, 161—172. (In Russ.).
  4. Осин, Е.Н., Леонтьев, Д.А. (2008). Апробация русскоязычных версий двух шкал экспресс-оценки субъективного благополучия. Сборник Материалов III Всероссийского социологического конгресса (c. 117—142). М.: Институт социологии РАН, Российское общество социологов.
    Osin, E.N., Leontyev, D.A. (2008). Approbation of Russian versions of two subjunctive well-being scales. Proceedings of the Third all Russia Congress of Sociology (pp. 117—142). Moscow: Institute of Sociology of the RAS, Russian Society of Sociologists. (In Russ.).
  5. Роговец, А.Ю., Мазуркевич, А., Суворова, И.Ю., Винник, М., Приходько, А.А. (2023). Чат-бот как способ поддержки жертв школьной травли. Современная зарубежная психология, 12(3), 103—114. https://doi.org/10.17759/jmfp.2023120310
    Rogovets, A.Yu., Mazurkevich, A., Suvorova, I.Yu., Vinnik, M., Prikhodko, A.A. (2023). Chat-bot as a way to support victims of school bullying. Journal of Modern Foreign Psychology, 12(3), 103—114. (In Russ.). https://doi.org/10.17759/jmfp.2023120310
  6. Abd-Alrazaq, A.A., Alajlani, M., Ali, N., Denecke, K., Bewick, B.M., Househ, M. (2021). Perceptions and opinions of patients about mental health chatbots: scoping review. Journal of Medical Internet Research, 23(1), e17828. https://doi.org/10.2196/17828
  7. Al-Abdullatif, A.M., Al-Dokhny, A.A., Drwish, A.M. (2023). Implementing the Bashayer chatbot in Saudi higher education: Measuring the influence on students’ motivation and learning strategies. Frontiers in Psychology, 14, 1129070. https://doi.org/10.3389/fpsyg.2023.1129070
  8. Balderas, A., García-Mena, R.F., Huerta, M., Mora, N., Dodero, J.M. (2023). Chatbot for communicating with university students in emergency situation. Heliyon, 9(9), e19517. https://doi.org/10.1016/j.heliyon.2023.e19517
  9. Borsci, S., Malizia, A., Schmettow, M., Van Der Velde, F., Tariverdiyeva, G., Balaji, D., Chamberlain, A. (2022). The Chatbot Usability Scale: The design and pilot of a Usability Scale for interaction with AI-based conversational agents. Personal and Ubiquitous Computing, 26(1), 95—119. https://doi.org/10.1007/s00779-021-01582-9
  10. Borsci, S., Prati, E., Malizia, A., Schmettow, M., Chamberlain, A., Federici, S. (2023a). Ciao AI: The Italian adaptation and validation of the Chatbot Usability Scale. Personal and Ubiquitous Computing, 27(6), 2161—2170. https://doi.org/10.1007/s00779-023-01731-2
  11. Borsci, S., Schmettow, M., Malizia, A., Chamberlain, A., Van Der Velde, F. (2023b). A confirmatory factorial analysis of the Chatbot Usability Scale: A multilanguage validation. Personal and Ubiquitous Computing, 27(2), 317—330. https://doi.org/10.1007/s00779-022-01690-0
  12. Brin, D., Sorin, V., Vaid, A., Soroush, A., Glicksberg, B.S., Charney, A.W., Nadkarni, G., Klang, E. (2023). Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessments. Scientific Reports, 13(1), 16492.
  13. Brooke, J. (1996). SUS: A “quick and dirty” usability scale. In: P.W. Jordan, B. Thomas, B.A. Weerdmeester, A.L. McClelland (Eds.), Usability Evaluation in Industry (pp. 4—7). London: Taylor and Francis.
  14. Chagas, B.A., Pagano, A.S., Prates, R.O., Praes, E.C., Ferreguetti, K., Vaz, H., Reis, Z.S.N., Ribeiro, L.B., Ribeiro, A.L.P., Pedroso, T.M., Beleigoli, A., Oliveira, C.R.A., Marcolino, M.S. (2023). Evaluating User Experience With a Chatbot Designed as a Public Health Response to the COVID-19 Pandemic in Brazil: Mixed Methods Study. JMIR Human Factors, 10, e43135. https://doi.org/10.2196/43135
  15. Cheng, X., Yin, L., Lin, C., Shi, Z., Zheng, H., Zhu, L., Liu, X., Chen, K., Dong, R. (2024). Chatbot dialogic reading boosts comprehension for Chinese kindergarteners with higher language skills. Journal of Experimental Child Psychology, 240, 105842. https://doi.org/10.1016/j.jecp.2023.105842
  16. Dhiman, N., Jamwal, M. (2023). Tourists’ post-adoption continuance intentions of chatbots: Integrating task-technology fit model and expectation-confirmation theory. Foresight, 25(2), 209—224. https://doi.org/10.1108/FS-10-2021-0207
  17. Elyoseph, Z., Hadar-Shoval, D., Asraf, K., Lvovsky, M. (2023). ChatGPT outperforms humans in emotional awareness evaluations. Frontiers in Psychology, 14, 1199058. https://doi.org/10.3389/fpsyg.2023.1199058
  18. Finstad, K. (2010). The Usability Metric for User Experience. Interacting with Computers, 22(5), 323—327. https://doi.org/10.1016/j.intcom.2010.04.004
  19. Gunnam, G.R., Inupakutika, D., Mundlamuri, R., Kaghyan, S., Akopian, D. (2022). Chatbot integrated with machine learning deployed in the cloud and performance evaluation. Electronic Imaging, 34(3), 205-1-205-5. https://doi.org/10.2352/EI.2022.34.3.MOBMU-205
  20. Hassenzahl, M., Monk, A. (2010). The Inference of Perceived Usability From Beauty. Human-Computer Interaction, 25(3), 235—260. https://doi.org/10.1080/07370024.2010.500139
  21. Janson, A. (2023). How to leverage anthropomorphism for chatbot service interfaces: The interplay of communication style and personification. Computers in Human Behavior, 149, 107954. https://doi.org/10.1016/j.chb.2023.107954
  22. Lewis, J.R., Utesch, B.S., Maher, D.E. (2013). UMUX-LITE: When there’s no time for the SUS. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2099—2102). https://doi.org/10.1145/2470654.2481287
  23. Lin, C.-C., Huang, A.Y.Q., Yang, S.J.H. (2023). A Review of AI-Driven Conversational Chatbots Implementation Methodologies and Challenges (1999–2022). Sustainability, 15(5), 4012. https://doi.org/10.3390/su15054012
  24. Lopes, E., Jain, G., Carlbring, P., Pareek, S. (2023). Talking Mental Health: A Battle of Wits Between Humans and AI. Journal of Technology in Behavioral Science, 9(4), 628—638. https://doi.org/10.1007/s41347-023-00359-6
  25. Mathur, A., Tiwari, S. (2023). The Impact of Artificial Intelligence on Customer Relationship Management in the Indian Banking Industry. In: P. Mittal, R. Bansal (Eds), Social Development and Governance: Innovations in Education, Technology and Management (pp. 107—116). QTanalytics India. https://doi.org/10.48001/978-81-966500-9-4_9
  26. Mohamad Suhaili, S., Salim, N., Jambli, M.N. (2021). Service chatbots: A systematic review. Expert Systems with Applications, 184, 115461. https://doi.org/10.1016/j.eswa.2021.115461
  27. Nguyen, M., Casper Ferm, L.-E., Quach, S., Pontes, N., Thaichon, P. (2023). Chatbots in frontline services and customer experience: An anthropomorphism perspective. Psychology & Marketing, 40(11), 2201—2225. https://doi.org/10.1002/mar.21882
  28. Paliwal, S., Bharti, V., Mishra, A.K. (2020). AI Chatbots: Transforming the Digital World. In: V.E. Balas, R. Kumar, R. Srivastava (Eds.), Recent Trends and Advances in Artificial Intelligence and Internet of Things (Vol. 172, pp. 455—482). Springer International Publishing. https://doi.org/10.1007/978-3-030-32644-9_34
  29. Prince, C., Omrani, N., Schiavone, F. (2024). Online privacy literacy and users’ information privacy empowerment: The case of GDPR in Europe. Information Technology & People, 37(8), 1—24. https://doi.org/10.1108/ITP-05-2023-0467
  30. Rane, N.L., Mallick, S.K., Kaya, Ö., Rane, J. (2024). Role of machine learning and deep learning in advancing generative artificial intelligence such as ChatGPT. In: Applied Machine Learning and Deep Learning: Architectures and Techniques (pp. 96—111). Deep Science Publishing. https://doi.org/10.70593/978-81-981271-4-3_5
  31. Sahata Sitanggang, A., Syafariani, R.F., Sari, F.W., Wartika, W., Hasti, N. (2023). Relation of Chatbot Usage Towards Customer Satisfaction Level in Indonesia. International Journal of Advances in Data and Information Systems, 4(1), 86—96. https://doi.org/10.25008/ijadis.v4i1.1261
  32. Schillaci, C.E., de Cosmo, L.M., Piper, L., Nicotra, M., Guido, G. (2024). Anthropomorphic chatbots’ for future healthcare services: Effects of personality, gender, and roles on source credibility, user satisfaction, and intention to use. Technological Forecasting and Social Change, 199(3), 123025. https://doi.org/10.1016/j.techfore.2023.123025
  33. Tsai, W.-H.S., Liu, Y., Chuan, C.-H. (2021). How chatbots’ social presence communication enhances consumer engagement: The mediating role of parasocial interaction and dialogue. Journal of Research in Interactive Marketing, 15(3), 460—482. https://doi.org/10.1108/JRIM-12-2019-0200
  34. Valério, F.A.M., Guimarães, T.G., Prates, R.O., Candello, H. (2018). Chatbots Explain Themselves: Designers’ Strategies for Conveying Chatbot Features to Users. Journal on Interactive Systems, 9(3). https://doi.org/10.5753/jis.2018.710
  35. Van De Vijver, F., Hambleton, R.K. (1996). Translating Tests. European Psychologist, 1(2), 89—99. https://doi.org/10.1027/1016-9040.1.2.89
  36. Weeks, R., Sangha, P., Cooper, L., Sedoc, J., White, S., Gretz, S., Toledo, A., Lahav, D., Hartner, A.-M., Martin, N.M., Lee, J.H., Slonim, N., Bar-Zeev, N. (2023). Usability and Credibility of a COVID-19 Vaccine Chatbot for Young Adults and Health Workers in the United States: Formative Mixed Methods Study. JMIR Human Factors, 10, e40533. https://doi.org/10.2196/40533
  37. Xue, J., Zhang, B., Zhao, Y., Zhang, Q., Zheng, C., Jiang, J., Li, H., Liu, N., Li, Z., Fu, W., Peng, Y., Logan, J., Zhang, J., Xiang, X. (2023). Evaluation of the Current State of Chatbots for Digital Health: Scoping Review. Journal of Medical Internet Research, 25, e47217. https://doi.org/10.2196/47217

Information About the Authors

Antonina S. Rafikova, Candidate of Science (Psychology), Institute of Psychology, Russian Academy of Sciences, Moscow, Russian Federation, ORCID: https://orcid.org/0000-0001-9831-6027, e-mail: antoninarafikova@gmail.com

Anatoly N. Voronin, Doctor of Psychology, Professor, Head of the Laboratory of Psychology of Speech and Psycholinguistics, Institute of Psychology, Russian Academy of Sciences, Moscow, Russian Federation, ORCID: https://orcid.org/0000-0002-6612-9726, e-mail: voroninan@bk.ru

Contribution of the authors

Antonina S. Rafikova — idea of the research; data collection, writing and formatting of the manuscript; research planning; supervision of the research.

Anatoly N. Voronin — idea of the research, application of statistical, mathematical and other data analysis methods; visualization of research results, manuscript writing.

Both authors participated in the discussion of the results and approved the final text of the manuscript.

Conflict of interest

The authors declare no conflict of interest.

Ethics statement

The study was approved by the local ethics committee of the Institute of Psychology of the Russian Academy of Sciences (protocol No. 25-18, 15.05.2024).

Metrics

 Web Views

Whole time: 280
Previous month: 191
Current month: 32

 PDF Downloads

Whole time: 52
Previous month: 28
Current month: 10

 Total

Whole time: 332
Previous month: 219
Current month: 42