The Application of Item Response Theory (IRT) for AUDEE Scale Psychometric Assessment

143

Abstract

Using item response theory (IRT), we examine AUDEE Scale that measures six different aspects of university digital educational environment (DEE): DEE Learning Process Satisfaction, DEE Communication satisfaction and Learning Motivation, DEE Stress Tension, Need for Support in DEE Learning Activity, DEE Dishonest Strategy Prevalence, and DEE Accessibility. Five-point Likert scale is used for all 38 questions of the AUDEE Scale. The study involved 406 MSUPE students who completed e-courses in the online format (90.1 % female). The average age of the participants was 28,7 ± 9,6 years (median = 24 years, minimum = 19 years, maximum = 72 years). The application of the graded response model (GRM) separately to each of the six subscales of the AUDEE Scale showed that all the items have at least moderate item discrimination, and most of the items have high item discrimination. The threshold values for each question of the AUDEE Scale, corresponding to the relative indicator of difficulty, increase monotonically. Test information functions for each subscale of the AUDEE Scale indicate that almost all measurements of the components of the AUDEE Scale are quite accurate in the range of mean values, and only for the scale 6 “DEE Accessibility” the estimation accuracy is high for the lowest values and drops sharply for medium and high values.

General Information

Keywords: digital educational environment (DEE), AUDEE scale, Item Response Theory (IRT), item difficulty, item discrimination, test information function

Publication rubric: Modeling and Data Analysis for Digital Education

Article type: scientific article

Funding. The reported study was funded by the Moscow State Univer¬sity of Psychology and Education (MSUPE) in the framework of the re¬search project “Digital Technologies in Higher Education: Development of Technology for Individualizing Education Using E-Courses

For citation: Radchikova N.P., Sorokova M.G., Odintsova M.A., Gusarova E. The Application of Item Response Theory (IRT) for AUDEE Scale Psychometric Assessment . Digital Humanities and Technology in Education (DHTE 2021): Collection of Articles of the II All-Russian Scientific and Practical Conference with International Participation. November 11-12, 2021 / V.V. Rubtsov, M.G. Sorokova, N.P. Radchikova (Eds). Moscow: Publishing house MSUPE, 2021., pp. 557–570.

References

  1. Avanesov V.S. Istoki i osnovnye ponyatiya matematicheskoi teorii pedagogicheskikh izmerenii (Item Response Theory) [The Origins and Basic Concepts of Item Response Theory] // Pedagogichesk­ie Izmereniya = Pedagogical measurements, 2007, no. 3, pp. 3–36. (In Russ.)
  2. Kroker L., Algina D. Vvedenie v klassicheskuyu i sovremennuyu te­oriyu testov [Introduction to classical and modern test theory]. Mos­cow: Logos, 2010. 668 p. (In Russ.)
  3. Neiman Yu.M., Khlebnikov V.A. Vvedenie v teoriyu modelirovaniya i parametrizatsii pedagogicheskikh testov [Introduction to the Theo­ry of Modeling and Parametrization of Educational Tests]. Moscow: Prometei, 2000. 168 p. (In Russ.)
  4. Sorokova M.G., Odintsova M.A., Radchikova N.P. Shkala otsenki tsifrovoi obrazovatel’noi sredy (TsOS) universiteta [Scale for Assess­ing University Digital Educational Environment (AUDEE Scale)] // Psikhologicheskaya nauka i obrazovanie = Psychological Science and Education, 2021, Vol. 26, no. 2, pp. 52–65. DOI: https://doi.org/10.17759/pse.2021260205 (In Russ., Abstr. in Engl.).
  5. Baker F.B. The Basis of Item Response Theory, 2nd ed. ERIC Clear­inghouse on Assessment and Evaluation, College Park, 2001. 186 p.
  6. Burnham K.P., Anderson D.R. Model selection and multimodel in­ference: A practical information-theoretic approach (2nd ed.). New York: Springer-Verlag, 2002. 488 p.
  7. Kang T., Chen T.T. Performance of the generalized S-X2 item fit in­dex for polytomous IRT models // Journal of Educational Measure­ment, 2008, no. 45 (4), pp. 391–406.
  8. Muraki E. A generalized partial credit model: Application of an EM algorithm. ETS Research Report Series, 1992. pp. 159–176.
  9. Orlando M., Thissen D. Likelihood-based item-fit indices for dichot­omous item response theory models // Applied Psychological Meas­urement, 2000, no. 24 (1), pp. 50–64.
  10. Samejima F. Graded Response Model. In W.J. van der Linden, & R.K. Hambleton (Eds.). Handbook of Modern Item Response The­ory. New York, NY: Springer, 1997. pp. 85–100.
  11. Sorokova, Marina; Odintsova, Maria; Radchikova, Nataly (2021): Scale for Assessing University Digital Educational Environment (AUDEE Scale): validation database. Psychological Research Data & Tools Repository. Dataset. https://doi.org/10.25449/ruspsyda­ta.14742816

Information About the Authors

Nataly P. Radchikova, PhD in Psychology, Leading Researcher of Scientific and Practical Center for Comprehensive Support of Psychological Research «PsyDATA», Moscow State University of Psychology & Education, Chief Specialist of the Laboratory of Biophysics of Excitable Media, Institute of Theoretical and Experimental Biophysics, Russian Academy of Sciences, Pushchino;, Moscow, Russia, ORCID: https://orcid.org/0000-0002-5139-8288, e-mail: nataly.radchikova@gmail.com

Marina G. Sorokova, Doctor of Education, PhD in Physics and Matematics, docent, Head of Scientific and Practical Center for Comprehensive Support of Psychological Research "PsyDATA", Head of the Department of Digital Education, Moscow State University of Psychology and Education, Moscow, Russia, ORCID: https://orcid.org/0000-0002-1000-6487, e-mail: sorokovamg@mgppu.ru

Maria A. Odintsova, PhD in Psychology, Docent, Head of the Department of Psychology and Pedagogy of Distance Learning, Faculty of Distance Learning, Moscow State University of Psychology & Education, Moscow, Russia, ORCID: https://orcid.org/0000-0003-3106-4616, e-mail: mari505@mail.ru

Elena Gusarova, MA in Psychology, Moderator, Scientific and Practical Center for Comprehensive Support of Psychological Research PsyDATA, Moscow State University of Psychology & Education, Moscow, Russia, ORCID: https://orcid.org/0000-0001-6968-5253, e-mail: bondarevaes@fdomgppu.ru

Metrics

Views

Total: 164
Previous month: 7
Current month: 5

Downloads

Total: 143
Previous month: 13
Current month: 3